Design of provcat

Configuration sources

A value can read the environment with ${oc.env:NAME,default}. Each source has two methods. raw() returns the configuration as written. load() fills in every ${...} from the environment of the reader.

YamlConfig reads a YAML file. EtcdConfig keeps each section as JSON under <prefix>/<section>. It talks to the etcd v3 HTTP gateway with the standard library, so it needs no etcd client package. It does not support TLS or etcd user accounts yet.

provc config push copies the raw form into etcd, so each reader fills in its own home folder and endpoints. The push is one etcd transaction. It writes the sections that the file has, and deletes the sections that the file does not have.

The template resolver

Each logical scheme has these settings:

• parts: names for the first part and each segment, in order.
• rest: an optional name that takes all extra segments, joined with /.
• target: a physical URI with {name} fields. A field can be a part name, the rest name, ref, version or at.
• ref: the default branch for {ref} when the URI has no branch:, tag: or commit:. It can name parts, for example "{env}", so that each environment is a branch. A version: selector keeps the default.
• pass_selector: if true, the physical URI gets the selector that the resolver used. A stage becomes the selector it points at. A scheme with a ref default passes branch:<ref> when the URI has no selector. With version:, only the version is passed.
• match: an optional regular expression for each part. provcat refuses a URI whose part does not match the whole expression.

A template can use only plain {name} fields, and only the names in the list above. provcat checks the fields when it loads the configuration. It refuses attribute access, indexes and format specs.

The resolver percent-encodes each value in the template, so a value can never add a path segment or a ?at= to the physical URI. A ref that contains / fails when the ref fills one path segment, as it does for lakeFS.

Entries

An entry is a rule for one logical name or one prefix, in the entries section. The resolver tries the entries before the template of the scheme, and the longest key wins. The reference gives the rules. The resolver checks each key against the parts and match of its scheme when it loads the configuration. The catalog checks that the store of each target exists.

An entry has two fields, {path} and {ref}. The resolver encodes them as it encodes the fields of a template. {ref} is there because lakeFS keeps the branch in the path. Without it, an entry for lakeFS could not take ?at=commit:ID, and a run could not fix the version of such a name.

Stores

A driver class has four methods for reading. Each method takes the cache.

• pin(uri, cache): returns the URI with a fixed version. GitHub, Hugging Face and lakeFS replace a branch, a tag or a time with a commit. They record the pin in the cache, so that --offline can use it. An object store pins a file to its version ID, and a folder to its newest provcat commit. Both need versioning on the bucket. The file driver returns the URI as it is.
• stat(uri, cache): returns one Entry, a file or a folder, with a size and a version.
• list(uri, cache): returns the entries directly inside a folder.
• fetch(uri, cache): returns a local path that holds the file or folder.

A driver that can be written also has two methods for writing:

• put(uri, src, cache): writes the local file src at the URI, and returns its Entry.
• delete(uri, cache): deletes one object.

The file, s3, gcs, azure and lakefs drivers can be written. The github and hf drivers cannot, and the base class raises a StoreError. The lakefs driver has commit, branch and merge. The s3, gcs and azure drivers have commit and branch, and a commit needs versioning on the bucket. “Writes” below explains each.

To add a driver, write a subclass of Store, set driver, give its options and their types in option_types(), and decorate the class with register_driver. A store entry with driver: <name> then uses it.

S3, Google Cloud Storage and Azure Blob Storage share one base class, ObjectStore in packages/provcat/src/provcat/stores/objects.py. It holds stat, list, folder fetch, the cache and offline mode. A driver for one more object store gives only six primitives:

• locate(uri): the bucket and the key that a URI names.
• _head: the size and version of one object.
• _pages: the pages of a listing, with folder prefixes and objects.
• _get: writes the bytes of one object, and returns their version.
• _put: writes a local file as one object, and returns its version.
• _delete: deletes one object, or raises NotFound.

Each driver uses its cloud’s Python SDK, never a command-line tool. The version of a file is the ETag on S3 and Azure, and the generation on GCS. The GCS and Azure SDKs are optional extras, so the drivers import them only when a store is used. An ADLS Gen2 account keeps folders as marker blobs, and the Azure driver leaves them out of listings.

The cache

LocalCache writes every file to a temporary name under .provcat/tmp and then renames it, so a reader never sees half a file. The records under .provcat are named by a hash of the store and the key, so two keys never share or block a record:

• meta: the version of each cached file, with its size, modification time, inode and SHA-256. If the time or the inode changed, as after a copy of the cache, provcat hashes the file and keeps it if the hash matches. If the size or the hash differs, provcat fetches the file again. provcat does not notice a rewrite in place that keeps the size, the time and the inode. It hashes the file only when the time or the inode changed. On Linux a tick of the clock can be a few milliseconds, so the inode tells apart two files of one size from one tick. Two writes of one file take turns, so its record describes the last one. If the record changes while a read checks the file, the read gets no file and fetches it again.
• done: a folder that was fetched whole.
• refs: a branch, tag or default branch pinned to a commit. The record includes the kind, so tag:v1 and branch:v1/x do not touch each other.
• locks: one lock file for each folder and each file. Fetches of one folder, and writes of one file, run one at a time, in threads and in processes.

Some disks, for example the default disks of macOS and Windows, store Guide.md and guide.md as one file. The same is true of one name in two Unicode forms. The GitHub and Hugging Face drivers and the object stores refuse to fetch a folder that holds two such names. They compare every level of each path.

They refuse such a folder on Linux too, so that every cache can move to any disk. The check also treats straße and strasse as one name. A single file with such a name still works, because the hash in its record shows when the other name overwrote it.

Hooks

Each method of Catalog runs through a lifecycle spot, and the hooks of the app run there, as middleware. The design is in the access notes. What is built differs from it in these ways:

• Pins before a fetch. The catalog pins before a fetch only for github and hf, whose drivers already pin inside fetch. A store says so with fetch_pins = True. lakeFS and the object stores still fetch a branch without a pin. A pin first would change the path in the cache, and a bucket without versioning cannot pin.

• The fetch answer. The hooks at fetch get the local path. They get no SHA-256, because a digest of a folder reads every file. A hook can compute it.

• No cache layers yet. Every hook runs before the base cache. A hook that returns an answer without calling next skips the cache and the store. A layer between the base cache and the store needs the drivers to call it, and that is not built.

• The principal. It is the user of the process until profiles come. A caller can name it with principal=.

• Hooks from a caller. A program that no launcher started can pass hooks to each Catalog, so several apps can share one process. The hooks of the process run last, next to the store. So they see the request that runs. A caller can add hooks, but it cannot skip a hook of the process.

• Files. open, read_block and close run in Catalog.open_file. A read fetches the whole pinned file at open, so read_block runs on the copy in the cache. The cache of blocks is not built.

• The snapshot spot. It gets the physical URI before the pin, because the pin runs inside it.

• The end of a session. The session spot runs at the end only when code calls Catalog.close() or uses the catalog in a with block. A library that never closes its catalog, as provrun does now, runs it only at the start.

A snapshot runs pin and fetch inside its own spot, with the logical URI. A folder put runs the put spot once for each file. provc runs the session spot when it opens the catalog and when it ends.

The GitHub driver

The GitHub driver checks the kind of a ref through the git refs API, so tag:main reads the tag and not the branch. It follows an annotated tag to its commit. It fetches a folder as one tarball of the commit. It skips links, devices and names with .. in the tarball, and unpacks with the Python data filter. Offline mode serves a file inside a folder that was fetched whole. The token goes only to the host of api_url, and the driver asks gh auth token for the token of that host.

The GitHub driver also reads files that Git LFS keeps. Git LFS is the Git extension that keeps large files outside the repository, with a small pointer file in their place. The contents API and the tarball both give the pointer file. So after each download, the driver looks for pointer files and asks the Git LFS batch API for their content, 100 at a time. It checks the size and the SHA-256 of each object against its pointer, and it keeps nothing that does not match.

The LFS server is on the web host, for example github.com for api.github.com, so the token goes there too. A download link must use HTTPS, and a redirect can go only to HTTP or HTTPS. The download links from the batch API carry their own headers, and the token never goes to them.

The Hugging Face driver

The Hugging Face driver reads models, datasets and Spaces. Its URI starts with the kind of repository, models, datasets or spaces, as in the paths of HfFileSystem. An example is hf://datasets/owner/repo/path. The kind names the place, not the use, so a logical name of either scheme can point at any of them. The next two parts are the owner and the name of the repository. Without a selector, the driver uses main, the default branch of every Hugging Face repository.

The driver asks the Hub for the branches and the tags, so tag:main reads the tag and not the branch. A commit: must be 7 to 40 hex digits, because the Hub also takes a branch or a tag in that place.

The driver calls the Hub API itself, through huggingface_hub, with the timeout of the store. It tries a call again after a 429, a 5xx or a network error. Each call names the pinned commit.

The driver downloads each file with the functions under hf_hub_download of huggingface_hub. Xet is the storage under Hugging Face, and it splits files into chunks. The functions read a file that Xet keeps with the hf_xet package, or over plain HTTP if hf_xet is not installed. A download over plain HTTP waits HF_HUB_DOWNLOAD_TIMEOUT seconds, 10 by default, not the timeout of the store. The driver fetches a folder file by file, from one listing of the folder at the commit.

The driver checks each file it downloads. A file that Xet or Git LFS keeps must match the SHA-256 that the Hub lists, and any other file its git blob ID. The driver keeps nothing that does not match. It writes each file only into the provcat cache, not into the cache of huggingface_hub under ~/.cache/huggingface, and it keeps no listings in memory. So the provcat cache holds the only copy. huggingface_hub and the Xet library still write small files of their own under ~/.cache/huggingface, such as logs.

The token goes to the endpoint of the store, so the endpoint must use HTTPS. Only the test servers on this machine may use plain HTTP. A page link of the Hub must point at the endpoint too. The token does not go to the download hosts that the Hub sends a download to. hf_xet sends it only to the Hub, for a Xet token. Without token_env, huggingface_hub finds the token in HF_TOKEN or in the file that hf auth login writes.

Folders in an object store

An object store fetches a folder object by object, and then deletes cached files whose objects no longer exist. It removes the folder marker first, and writes it again only after the whole folder is current. It refuses a folder that holds keys no URI can name, for example a//b. It records the version that came with the downloaded bytes. The version is the ETag on S3 and Azure, and the generation on GCS. The S3 driver does not send If-Match, because S3-compatible servers do not all support it.

A configuration for each tenant

One configuration holds every tenant. The schemes are shared, so every tenant maps URIs by the same rules. Two sections hold the parts that are not shared:

provc --tenant acme, or PROVCAT_TENANT=acme, opens acme’s view. It holds the shared sections and acme’s part. provcat cuts the view before it fills in ${...} values, so another tenant’s part is never read. Without a tenant, provc opens the admin’s view, which holds every part. provc config show --tenant acme prints a view.

A scheme names the part that says whose a URI is, with tenant: org. In acme’s view, a URI of such a scheme with another org is refused. A scheme without tenant, such as code, serves every tenant.

These rules keep one tenant from passing for another:

• Store names. A tenant’s store names end with -<tenant>, as in lakefs-acme and s3-acme. So no tenant can define lakefs-globex, which globex’s URIs resolve to.
• Stages. A tenant’s stages must be URIs of that tenant. So acme cannot point globex’s stage:production anywhere.
• Entries. A tenant’s entries must be URIs of that tenant. Each must go to a store of that tenant, whose name ends with -<tenant>, and to a bucket or repository named <tenant> or <tenant>-.... So an access list that grants acme-* still holds. A tenant’s entry has no ${...} values, because the check reads the text before provcat fills them in.
• One name, one place. A store, stage or entry named in two parts is an error. Two ways of writing one stage URI, such as with a trailing /, count as one name.
• The tenant picks the store. In a scheme whose store name has a field, as in lakefs-{org}, the tenant part must be that field.

In etcd, each part has its own keys:

A reader for acme reads the shared keys and the keys under <prefix>/tenants/acme/ only. It does not read the list of names. So etcd’s own access control can keep the tenants apart. Give acme’s etcd user read access to those keys only. Grant the prefix with its trailing /, as <prefix>/tenants/acme/, or the grant also covers acme2. The configuration holds no keys, only profile names, but it does show other tenants’ stores and endpoints.

The views are not access control on their own. A tenant who can read the YAML file, or the etcd keys of another tenant, can read that part. The boundary stays the keys each tenant holds.

The lakeFS driver

The lakefs driver is the S3 driver plus the lakeFS REST API. Data comes through the S3 gateway, where the first key segment is a ref. A ref is a branch, a tag or a commit. The REST API answers what the gateway cannot:

• Pin. pin turns lakefs://repo/main/path into lakefs://repo/<commit>/path. The commit is the head of the branch. Data that is not committed shows on the branch, but not on the pinned URI.
• Time. lakefs://repo/main/path?at=ts:TIME names the newest commit of main made at or before TIME. stat, ls and fetch pin it first, and then read that commit. A time in the future is refused, because its commit could still change.
• Refs. The API says whether a ref exists. So ls of an empty branch gives [], and a missing branch or repository gives an error that names it. ls lakefs://repo lists the branches.

A logical URI keeps its default branch with ts:, and the resolver passes ?at=ts:TIME on to the physical URI. So data+table://acme/shared/prod/...?at=ts:2026-10-01 reads branch prod as it was on that day. A stage can point at a time, for example audit: "ts:2026-03-31T23:59:59Z".

Each pin goes into the cache, so --offline uses the commit that the last online pin found. A pin of a time less than 10 minutes ago is not kept. A commit under way, or a server clock that runs behind, can still add a commit before that time.

A time search reads the first-parent log of the branch, 1000 commits a page, from the newest commit back. The first-parent log holds the commits of the branch itself. A commit of a feature branch that was merged in is left out, because the branch never pointed at it. The search takes the first commit made at or before TIME. lakeFS records when each commit was made, not where a branch pointed, so the answer can differ from the history in two cases:

• A reset. A branch that was reset to an older commit pointed at that commit with no new commit to show it.
• A set date. The lakeFS commit API can set the date of a commit, so a commit can claim an earlier time.

The API takes the same keys as the S3 gateway. provcat never follows a redirect from the API, because a redirect would send the keys to the host that it names.

Time on GitHub

github://owner/repo/path?at=ts:TIME names a commit of the default branch, and ?at=ts:BRANCH@TIME a commit of BRANCH. A time never holds an @, so the last @ ends the branch name.

GitHub has no log of where a branch pointed. So the driver walks the branch from its head, along first parents, and takes the first commit whose committer date is at or before TIME. It reads the history 100 commits a page, and stops once it has read 10,000 commits. As on lakeFS, a commit made on another branch and merged later with a merge commit is left out. A future time is refused, and a pin of a time less than 10 minutes ago is not kept for --offline.

Git records when a commit was made, not when it was pushed. So the answer can differ from what the branch showed at TIME:

• A late push. A commit made on Monday and pushed on Wednesday counts from Monday.
• A rewrite. A force push or a rebase replaces commits, and their dates go with them.
• A wrong clock. The committer’s clock sets the date.
• A fast-forward merge. The branch takes the other branch’s commits as its own, with their old dates.
• A merge from a pull. git pull on a branch can make a local commit the first parent, so the branch’s own earlier commits become the second parent.

--offline keeps the answer that the last online pin found. A late push or a rewrite can change the online answer later, and the offline one stays as it was.

Time on Hugging Face

hf://...?at=ts:TIME and ?at=ts:BRANCH@TIME work as on GitHub, with main in place of the default branch. The Hub lists the commits of a branch newest first, 100 a page. The driver takes the first one whose date is at or before TIME. It stops once it has read 10,000 commits.

The Hub gives no parents of a commit. So the driver cannot keep to first parents, and a commit that someone made on another branch and merged with a merge commit counts too. The Hub itself merges a pull request as one new commit, so the extra commits come only from history that someone pushed with git.

A branch in a logical URI

A logical URI keeps its branch in its path, for example code://org/repo/dev/path. A GitHub or Hugging Face target has no {ref}, so a plain ts:TIME would lose dev. So the resolver keeps the branch, and the catalog gives a GitHub or Hugging Face store ?at=ts:dev@TIME. Other stores get ts:TIME, as before. An @ in the time of a logical URI is refused, so it cannot name a second branch.

Merges

provc merge lakefs://repo/main --from feature merges feature into main with the lakeFS merge API. It merges the last commit of the source. So writes on feature that are not committed stay on feature. Commit them first.

lakeFS refuses some merges, and provcat says why:

• A conflict. Both branches changed the same file since they split. Unless --strategy picks a side, lakeFS refuses the merge. source-wins keeps the source’s file, and dest-wins keeps the branch’s file.
• Nothing to merge. The branch already has every change in the source.
• Uncommitted changes. The branch that receives the merge has writes that are not committed. Commit them, or delete them, first.

A merge through a logical URI adds provcat.uri to the merge commit’s metadata, as a commit does. lakeFS adds its own key, .lakefs.merge.strategy.

The cache needs no change for writes. An online read asks the store for the version first, so it never serves a copy that a write has replaced.

Access control is the store’s. With the shared lakeFS, each org’s keys write, commit, branch and merge only its own repositories. The system test packages/provcat/tests/system/test_lakefs_acl.py checks this.

The layout

A bucket holds one project. Its first folders are the branches, main among them:

s3://acme-sales/main/tables/agreements/part-0.parquet
s3://acme-sales/dev/tables/agreements/part-0.parquet

A scheme maps a logical URI onto that layout with {ref}. The layout needs no new code in the resolver:

schemes:
  data:
    parts: [org, project]
    rest: path
    target: "s3://{org}-{project}/{ref}/{path}"
    ref: main
    match: {org: "[a-z0-9]+", project: "[a-z0-9-]+"}

So data://acme/sales/tables/agreements?at=branch:dev is s3://acme-sales/dev/tables/agreements. An org has many buckets, one for each project, and the cloud’s access control works on each bucket.

Fixed versions

A bucket with versioning on keeps every version of an object, and gives each an ID. Two selectors name fixed versions:

• ?at=version:ID names one version of one file. provc pin of a file gives it.
• ?at=commit:ID names a provcat commit of a folder. It works on the folder and on every file and folder in it.

A pin of a folder gives the newest commit of that folder, or of a folder above it, that holds the folder. A folder with no commit cannot be pinned. A bucket without versioning cannot pin or commit, and provcat says so.

Fixed versions of a logical URI

A logical URI names a fixed version with the same selectors. The resolver keeps the branch in the path and puts the fixed version in ?at=:

files://acme/sales/t?at=commit:ID          -> s3-acme://acme-sales/main/t?at=commit:ID
files://acme/sales/t/a.csv?at=version:ID   -> s3-acme://acme-sales/main/t/a.csv?at=version:ID
files://acme/sales/t?at=branch:dev         -> s3-acme://acme-sales/dev/t

The branch in the path is the default branch of the scheme, its ref, even when the target places the selector with {at}. A stage that names a commit resolves the same way. A bucket has no tags, so ?at=tag:X is refused. lakeFS takes a commit in the path, so a lakeFS target still puts a commit in {ref}. Each store says which way it works with the attribute refs_in_path.

A commit records its folder, such as main/t, and the first folder is the branch. A read at a commit checks that its URI is in that folder. So a logical URI reads a commit only on the default branch of the scheme. A commit made on another branch, such as dev, fails through the logical URI. The error names the physical URI to read, such as s3-acme://acme-sales/dev/t?at=commit:ID, and provc commit prints that URI too. A logical URI cannot name a branch and a commit at once, and the resolver does not read the commit to find its branch.

What a commit is

provc commit s3://acme-sales/main/tables -m "qa run 42" writes two objects at the top of the bucket:

A reader checks that the commit object hashes to its ID, so a changed commit is refused. A read at a commit gets each file’s listed version. The cache keeps these versions apart from the live files.

provc branch s3://acme-sales/dev --from main copies every file of main to dev. --from <commit ID> copies the versions in a commit. S3 and GCS copy inside the cloud. Azure downloads and uploads.

What a commit in a bucket guarantees

A bucket writes one object at a time. It never writes several objects in one step. So a commit is made safe in two ways:

• The file list is a snapshot. A commit lists every version under the folder twice, deleted versions too. Every write makes a new version ID. So if the two listings are equal, nothing changed between them, and every listed version was current at one moment. If they differ, the commit lists again, up to five times, and then fails.
• Each folder’s commits have one order. The commit creates log entry N only if no object has that name. S3 and Azure do this with If-None-Match: *, and GCS with ifGenerationMatch=0. Of two commits at once, one gets N. The other lists the files again and takes N + 1, or finds nothing to commit. Each commit names its parent.

lakeFS works differently. It keeps each branch pointer in its database, and a commit moves the pointer with one compare-and-swap. Transactions has the details, and compares Nessie, Delta Lake, Iceberg and SlateDB.

What a bucket commit does not give:

• Several files written as one step. Writers write files one by one, and a reader of the live folder can see half a change. Read at a commit to get one moment. Write through lakeFS to make several files appear at once.
• Commits that outlive their versions. A lifecycle rule that deletes old versions, or a person who deletes one, breaks the commits that name it. A read of such a file fails with “not found”. Keep old versions while you still need to read their commits.
• One order across folders. Each folder has its own log. A pin compares the commits of a folder and of the folders above it by their time, which comes from each writer’s clock. A commit records its time to the microsecond. If a folder and a folder above it have the same time, the folder above counts as newer. So a pin gives the same answer every time, but the answer is wrong when the folder below came second. When two writers’ clocks differ, a later commit can also get an earlier time and sort as older.
• ADLS Gen2. An account with a hierarchical namespace keeps no blob versions, so it can neither pin nor commit.
• An atomic branch. A branch copies file by file. If it fails halfway, the new branch is half made. Delete it and copy again.