provcat.catalog

The Catalog joins a configuration, a resolver, the stores and the cache.

cat = Catalog.open() # provcat.yaml, PROVCAT_CONFIG or defaults cat.resolve(“data+table://acme/shared/prod/agreements/qa_outputs?at=stage:production”) path = cat.fetch(“github://rahuldave/provcat/README.md?at=branch:main”)

A catalog for one tenant, Catalog.open(tenant="acme") or PROVCAT_TENANT, sees only the shared configuration and that tenant’s part, and refuses the logical URIs of other tenants.

A URI whose scheme is a store is already physical, so resolve returns it as it is. A URI whose scheme is a logical scheme goes through the resolver once, and the result must name a store.

Writes take a logical or a physical URI:

cat.put("data+table://acme/shared/dev/agreements/qa_outputs/part-0.parquet", "out.parquet")
pinned = cat.commit("data+table://acme/shared/dev/agreements/qa_outputs", "qa run 42")

A snapshot pins a URI, fetches the pinned version, and copies it into a read-only folder of the cache named by its content:

snap = cat.snapshot("data+table://acme/shared/prod/agreements/qa_outputs")
snap.pinned, snap.path, snap.digest

A write goes to a branch: a URI with no ?at=, which writes to the scheme’s default branch, or one with ?at=branch:NAME. Any other selector names a fixed version, which cannot change, so it is refused.

Every method runs through a lifecycle spot, where the hooks of the app run (see provcat.hooks). A caller can pass the hooks, as an orchestrator does: Catalog(config, hooks=..., app=..., principal=...). Otherwise the app comes from PROVCAT_APP, and its hooks from the proxy configuration. close runs the session spot again.

class Snapshot

source

Snapshot(uri: Uri, pinned: Uri, path: Path, digest: str, kind: str)

A read-only copy of one pinned version of a file or a folder.

Parameter Type Default Description
uri Uri required The URI as asked for, logical or physical
pinned Uri required The physical URI, pinned to a fixed version; a file:// URI pins to itself
path Path required The read-only copy, at .provcat/snapshots/KIND/DIGEST/NAME in the cache
digest str required The SHA-256 of the file, or of the folder (see cache.tree_digest)
kind str required “file” or “folder”

class Catalog

source

Catalog(config: dict | None = None, *, cache_root: str | Path | None = None, offline: bool = False, resolver: Resolver | None = None, tenant: str | None = None, app: str | None = None, hooks: Hooks | Proxy | dict | None = None, principal: str | None = None)

Resolve, read and write catalog URIs, through the stores, the cache and the hooks.

Most code opens one with Catalog.open, which reads a configuration source. The constructor takes the configuration as a dict.

Parameter Type Default Description
config dict | None None The configuration: stores, schemes, stages, entries, cache, tenants; None for the default stores alone
cache_root str | Path | None None The cache folder; None takes the app’s cache_root, then cache.root, then the default
offline bool False Work from the cache alone: what it does not hold raises OfflineMiss
resolver Resolver | None None The resolver of logical URIs; None makes a TemplateResolver from the configuration
tenant str | None None The tenant whose view of the configuration to use; None for the admin
app str | None None The app whose hooks run. Without hooks= or principal=, it must be the process’s app, from PROVCAT_APP, or None for that app. With hooks= or principal=, it names the caller’s app; an app that the hooks file names needs its Proxy from hooks.load
hooks Hooks | Proxy | dict | None None A caller’s own hooks: a Hooks, a Proxy from hooks.load, or a dict of spot to hooks; the process’s hooks run after them
principal str | None None Who uses the catalog, as the hooks see it; None for the user of the process. As with hooks=, a process launched with PROVCAT_APP refuses it with ConfigError

Catalog.close

source

Catalog.close() -> None

End the session: the hooks at the session spot see event close.

Catalog.open

source

Catalog.open(spec: str | ConfigSource | None = None, *, tenant: str | None = None, **kw: Any) -> Catalog

Open a catalog over a configuration source: a YAML file, etcd, or the defaults.

The other keyword arguments go to Catalog, such as offline, cache_root, resolver, app, hooks and principal.

Parameter Type Default Description
spec str | ConfigSource | None None A YAML path, an etcd:// URL or a ConfigSource; None takes PROVCAT_CONFIG, then ./provcat.yaml, then the defaults
tenant str | None None The tenant; None takes PROVCAT_TENANT, and with none the catalog is the admin’s
kw Any The other arguments of Catalog, such as offline, cache_root, app and hooks
returns Catalog A catalog over the configuration that the source loads for the tenant

Catalog.resolve

source

Catalog.resolve(uri: str | Uri) -> Resolution

Find the physical URI that a URI names.

A logical URI runs the resolve spot; a physical one is its own answer.

Parameter Type Default Description
uri str | Uri required The URI to resolve, logical or physical
returns Resolution The logical URI, the physical URI it names, and the steps between

Catalog.pin

source

Catalog.pin(uri: str | Uri) -> Uri

Resolve uri and replace its moving name with a fixed version. It runs the pin spot.

Parameter Type Default Description
uri str | Uri required The URI to pin, logical or physical
returns Uri The physical URI, with any branch or time replaced by a fixed version; StoreError where the store cannot pin, as for a file in a bucket without versioning or a bucket folder with no provcat commit

Catalog.stat

source

Catalog.stat(uri: str | Uri) -> Entry

Describe one file or folder: its kind, size and version. It runs the stat spot.

Parameter Type Default Description
uri str | Uri required The URI of a file or a folder, logical or physical
returns Entry The file or the folder; NotFound when there is none

Catalog.list

source

Catalog.list(uri: str | Uri) -> list[Entry]

List a folder, one level deep. It runs the list spot.

Parameter Type Default Description
uri str | Uri required The URI of a folder, logical or physical
returns list[Entry] The files and folders directly inside it

Catalog.fetch

source

Catalog.fetch(uri: str | Uri) -> Path

Return a local path that holds the file or folder.

Parameter Type Default Description
uri str | Uri required The URI of a file or a folder, logical or physical
returns Path A local path that holds it: a copy in the cache, or for file:// the file itself

Catalog.snapshot

source

Catalog.snapshot(uri: str | Uri) -> Snapshot

Pin a URI, fetch that version, and copy it into a read-only folder of the cache.

No later fetch changes the copy. The URI must be pinnable: a file or folder in lakeFS or GitHub, a file in a versioned bucket, a folder with a provcat commit, or a local file or folder. The file store pins a URI to itself; the digest still names the bytes. An object store without versioning cannot pin, so it raises a StoreError. The pin and the fetch run their own spots inside the snapshot spot.

Parameter Type Default Description
uri str | Uri required The URI of a file or a folder, logical or physical
returns Snapshot The pinned URI, the read-only copy, its digest and its kind

Catalog.read

source

Catalog.read(uri: str | Uri) -> bytes

Fetch a file and return its bytes.

Parameter Type Default Description
uri str | Uri required The URI of a file, logical or physical
returns bytes The bytes of the file; IsADirectoryError for a folder

Catalog.open_file

source

Catalog.open_file(uri: str | Uri, mode: str = 'rb', autocommit: bool = True) -> CatalogReader | CatalogWriter

Open a file to read (“rb”) or to write (“wb”).

A read looks up and pins uri once, at open, and then reads only that version. It runs the pin and open spots, read_block for each read, and close with the byte ranges read. A write goes to a temporary file, and close runs the close spot around a put; with autocommit off, commit() does. Nothing is committed to the history of the store. If a hook at open refuses after the handle was made, the handle is dropped unseen.

Parameter Type Default Description
uri str | Uri required The URI of the file, logical or physical; a write takes a branch
mode str 'rb' “rb” to read, “wb” to write; any other mode raises ValueError
autocommit bool True For a write: True puts the file at close; False puts nothing until the caller calls the handle’s commit()
returns CatalogReader | CatalogWriter A CatalogReader for “rb”, a CatalogWriter for “wb”

Catalog.fs

source

Catalog.fs(register: bool = True, replace: bool = False) -> CatalogFileSystem

Make an fsspec file system over this catalog; it needs provcat[fs].

Once the logical schemes are fsspec protocols, fsspec.open and pandas take logical URIs (see provcat.fs).

Parameter Type Default Description
register bool True Register the logical schemes of the catalog as fsspec protocols
replace bool False Take a scheme over from a catalog that registered it with another context or other hooks
returns CatalogFileSystem A file system that reads and writes through this catalog

Catalog.put

source

Catalog.put(uri: str | Uri, src: str | Path | bytes) -> list[Entry]

Write a local file, a local folder or bytes at a URI on a branch.

A folder is written file by file under the URI, and each file runs the put spot.

Parameter Type Default Description
uri str | Uri required Where to write, logical or physical, on a branch
src str | Path | bytes required A local file, a local folder, or the bytes to write
returns list[Entry] One Entry for each file written

Catalog.delete

source

Catalog.delete(uri: str | Uri) -> None

Delete a file on a branch.

Parameter Type Default Description
uri str | Uri required The file to delete, logical or physical, on a branch

Catalog.commit

source

Catalog.commit(uri: str | Uri, message: str, metadata: dict[str, str] | None = None) -> Uri

Commit the branch that a URI is on. It runs the commit spot.

The metadata records where the commit came from: a logical URI adds provcat.uri. A store without commits raises StoreError.

Parameter Type Default Description
uri str | Uri required A URI on the branch to commit, logical or physical
message str required The commit message
metadata dict[str, str] | None None Key-value pairs stored with the commit
returns Uri The physical URI of uri, pinned to the new commit

Catalog.branch

source

Catalog.branch(uri: str | Uri, source: str) -> Uri

Make the branch that a URI names. A store without branches raises StoreError.

Parameter Type Default Description
uri str | Uri required A URI on the branch to make, logical or physical
source str required The branch, tag or commit that the new branch starts from
returns Uri The physical URI of the new branch

Catalog.merge

source

Catalog.merge(uri: str | Uri, source: str, message: str = '', metadata: dict[str, str] | None = None, strategy: str | None = None) -> Uri

Merge a branch, a tag or a commit into the branch that a URI is on.

As with commit, a logical URI adds provcat.uri to the metadata. A store without merges raises StoreError.

Parameter Type Default Description
uri str | Uri required A URI on the branch to merge into, logical or physical
source str required The branch, tag or commit to merge from
message str '' The message of the merge commit
metadata dict[str, str] | None None Key-value pairs stored with the merge commit
strategy str | None None How a conflict is settled: None fails, “source-wins” or “dest-wins”
returns Uri The physical URI of uri, pinned to the merge commit