Notes on how code reads and writes data through provcat and provrun

Terms

How code names and opens data

Code names data only by logical URI. It opens a URI, reads it and closes it, as it does a local file. The lookup decides where the bytes are, and any other facts go in the catalog, not in the URI.

fs = catalog.fs()                      # an fsspec file system
with fs.open("data+table://acme/nlp/prod/corpora/wiki/part-0.parquet", "rb") as f:
    table = pyarrow.parquet.read_table(f)

df = pandas.read_parquet("data+table://acme/nlp/prod/corpora/wiki/part-0.parquet")

One target can be code under one name and data under another. The scheme of a name says what the thing is to its users. The lookup says where it is.

There is one catalog and one file system. The file system is a thin fsspec adapter over the catalog, and it has no logic and no cache of its own. Every method of the catalog runs through the lifecycle spots. So the catalog has no method that reads data without the hooks.

The trust boundary

The hooks control the code that uses the catalog API. They do not stop code that goes around the API in the same process. Such code can call a store object of the catalog or open a file in the cache folder. It can also use boto3 or HfFileSystem with the same keys.

So a gate is a security control only when the gated code cannot reach the keys or the cache folder:

• Now. provrun runs the agent in its own process. The sandbox is a set of paths, /input, /output and /scratch, that the tools of the agent see. The keys and the cache folder are in the same process. So the sandbox holds only while the agent has no tool that runs code.
• Step 6 of the plan of pull requests below. The agent moves to its own process or container, which holds no keys and no cache folder. Code there asks the provrun process to open a URI, and the provrun process runs the catalog with the gate. Undeclared opens are a security control only after this move.
• Later. The access layer gives short-lived keys for each grant, so the keys themselves carry the limits.

Outside a sandbox, for example in a notebook, the hooks are a policy for code that cooperates, and not a security control.

The lifecycle spots

Each operation of the catalog passes through fixed spots:

Each spot also gets the logical URI that started the operation, even at fetch and snapshot. So a hook can decide by the logical name at every spot.

The catalog calls the pin spot before every snapshot. It calls it before a fetch from GitHub or Hugging Face, whose drivers pin inside fetch. lakeFS and the object stores still fetch a branch without a pin, because a pin first would change the path in the cache. “Hooks” in the design lists this and the other ways that step 4 differs from these notes.

stat and list do not pin first. A file that a session put on a lakeFS branch, and did not commit yet, is on the branch but in no commit. So a stat or a list of the branch must still find it.

Hooks

A hook at a spot wraps the work of the catalog, as middleware wraps a request in a web framework. The hook gets the request and a function next, which runs the rest of the chain. One shape covers every kind of hook:

def broker_gate(ctx, req, next):
    # At resolve, req.physical is None, because the lookup has not run yet.
    if not broker.allows(ctx.principal, req.spot, req.logical, req.physical):
        raise AccessDenied(req.logical)
    return next(req)

The order of a chain is recorder, then gate, then the base cache, then the cache layers, then the store. So the recorder sees every request, even one that the gate refuses. provrun records access_denied today, and it keeps doing so. A gate sees every request that reaches the data, and a read from the base cache or a cache layer passes the gate too. A principal that may not read a URI cannot read, through the catalog, a copy that another principal fetched.

The base cache

The base cache is always there, right after the gates in every hook chain. A proxy configuration cannot remove it or move it. The drivers need it. A fetch returns a path inside it, and a folder fetch installs into it under a lock. Offline mode reads the record of pins from it. The drivers now call the cache in about 30 places, and those calls stay as they are.

A cache layer adds a level between the base cache and the store. A cache in S3 is such a layer. A fetch then looks in the base cache first, then in the shared cache, and then in the store. A pinned version is safe to share between machines, because it never changes. The record of pins stays in the base cache, because each machine pins at its own time.

The base cache has these levels, and the notes add the third one:

1. A copy by version. The cache keeps a pinned URI in a path that names the version. No later fetch rewrites the copy.
2. A snapshot. provrun stages each declared input from a read-only snapshot, which is named by its SHA-256.
3. Blocks of a large file (new). A reader of a Parquet file often reads a few byte ranges of a large file. The cache keeps each block under the pinned URI. The fsspec adapter keeps no blocks of its own, so the cache and its hooks see every block.

cache.root stays in the configuration, as the root of the base cache for users who have no hooks file. The proxy configuration can set another root.

The proxy configuration

Hooks come later from the profile of the principal and the access layer. Until then, a proxy configuration gives each app its hooks. It is a local file, and it is never in etcd. A configuration in etcd describes the data. The hooks describe who reads it and how, so they belong to the machine and the app.

# ~/.config/provcat/hooks.yaml, or the file in PROVCAT_HOOKS
apps:
  default: {}
  provrun:
    cache_root: /var/cache/provcat-runs
    "*": [provrun.evidence, provrun.broker]
    fetch: [cache.s3]
    read_block: [cache.s3]

• The hooks. Each name is a Python entry point in the group provcat.hooks. A package registers its hooks when it is installed. So the file names hooks, and it holds no code and no path to code. If two packages register one name, provcat refuses to start.
• The spots. The hooks under "*" run at every spot, first. The hooks under a spot run after them. A name can be in a chain only once.
• The app. The launcher sets the app name and the path of the hooks file when it starts the process, through PROVCAT_APP and PROVCAT_HOOKS. provcat reads them once, when the process opens its first catalog. A later Catalog.open(app=...) can name only that app. Code in the process cannot pick an app with fewer hooks. A process that no launcher started, such as an orchestrator, can instead pass the hooks to each Catalog, as “Hooks from a program” in the reference shows.
• The writer. The person who runs the machine writes the hooks file. The principal whose reads it controls must not be able to write it.
• The root of the base cache. cache_root in the section of an app takes the place of cache.root in the configuration. Two apps with two roots keep two records of pins, so --offline in one app does not see the pins of the other.
• No app. If PROVCAT_APP is not set and the file exists, the process gets the default section.
• No file. If PROVCAT_APP is not set and the file does not exist, every app gets the base cache and no hooks, as provcat works now. If PROVCAT_APP names an app and the file is missing or has no section for it, provcat refuses to start. So a missing file never turns a gate off without an error.

Later, the profile and the permissions give the hook chain of each principal. Then the proxy configuration goes away, and the spots and the hooks stay as they are.

The lookup

A scheme template sends every URI of a scheme to one place. So today every code:// URI goes to GitHub, and every data+table:// URI goes to lakeFS. A catalog entry adds a rule for one name or one prefix:

entries:
  "code://acme/tokenizer/prod":
    target: "github://acme/tokenizer/{path}?at=branch:prod"
  "code://acme/tokenizer/prod/weights":
    target: "hf://models/acme/tokenizer/{path}"

These rules apply to entries:

• Longest prefix. The longest prefix wins. A prefix matches whole path segments only, so code://acme/tok does not match code://acme/tokenizer. If no entry matches, the scheme template applies, as now.
• Fields. A target can use {path}, which is the rest of the URI after the prefix. provcat encodes each segment of {path} as the templates do now, so a path cannot add a selector or climb with ... The build added a second field, {ref}, with a ref default. lakeFS keeps the branch in the path, so without {ref} an entry for lakeFS could not take ?at=commit:ID.
• Selectors. A selector on the URI, for example ?at=commit:ID, takes the place of the selector in the target. So a run can fix a version of any name. A ts: selector keeps the branch of the target, as the templates do now. So ?at=ts:2026-01-01 on the first entry above becomes ts:prod@2026-01-01 on GitHub.
• Stages. A stage of a URI comes before an entry, as it comes before a template now.
• The scheme rules. The prefix must match the parts and the match patterns of its scheme, up to its length. A key cannot name a store, because the catalog does not look up a physical URI.
• Tenants. Entries live in the shared part or in a tenant’s part of the configuration. provcat checks an entry of a tenant as it checks a stage. The scheme names one part of the key with its tenant: key, and that part must be the tenant. The store of the target must end with - and the tenant. The tenant view of the configuration takes in entries for this. The build added two rules. A tenant entry has no ${...} values. Its bucket or repository is named for the tenant, as acme or acme-....

Physical URIs

The catalog also takes a physical URI, for example github://owner/repo/path. It does no lookup for one, so a hook sees no logical name. A gate can refuse physical URIs. The provrun broker already does so outside the dev environment.

The fsspec adapter registers the logical schemes as fsspec protocols. fsspec keeps one registration for each protocol in a process. A later catalog with the same app, principal, tenant and hooks takes the registration over. provcat refuses one with another app, principal, tenant or hooks, so fsspec never runs one caller’s reads through another’s hooks. Other catalogs in the process work through their own catalog.fs(register=False). catalog.fs(replace=True) takes the registration over on purpose, for example when a notebook cell runs again.

The adapter does not register the stores. fsspec and huggingface_hub already register github and hf for their own file systems, and provcat does not replace them. Code reaches a physical URI through catalog.fs() only.

Reads pin at open

A read must give the bytes that the evidence names. So open does the lookup once and pins once. Every read through that handle reads the pinned version, even if the branch moves during the read.

A hook at pin can hold the pins of a session. Then two opens of the same URI give the same version. provrun puts such a hook on a run, so the evidence of a run names one version for each URI. A run that must see a newer version opens the URI with a selector, for example ?at=commit:ID, and the evidence shows the change.

The record of a read holds the logical URI, the physical URI, the pinned URI and an identity of the bytes:

• A whole file. The identity is the SHA-256.
• Blocks of a file. The identity is the version that the store gives, for example a lakeFS checksum, an S3 version ID or a Git commit. The record also holds the byte ranges that the reader read.
• A store without versions. file:// pins a URI to itself. Then the identity is the size and the time of the last change. Such a read is not fixed, because the bytes can change during the read. The record says so. A bucket without versioning cannot be pinned, so it cannot be read in a run, as now.

Code in a run can open a URI that the run did not declare. The gate decides at that time, and the evidence records the open as it records a declared input. Declared inputs are still staged before the agent starts, as now.

Writes

open(uri, "wb") writes to a temporary file. close runs the put spot, so the hooks see the write before the file goes into the store. Nothing is committed until the code calls commit, as with provc put now. In a run, the evidence hook records each written file as an output, and the commit of the bundle takes them all.

fsspec under the drivers

fsspec has implementations for most of the stores that provcat reads:

A driver that uses an fsspec implementation stays behind the lifecycle spots. So the hooks apply to it as to every other driver. The hf driver does not use the cache of huggingface_hub under ~/.cache/huggingface. So the base cache is the only cache of its files.

The GitHub driver reads the content of a Git LFS file through the Git LFS batch API, as the design explains.

Plan of pull requests

1. Git LFS in the GitHub driver. Done in commit 8609a4c. The driver replaces each pointer file with its content, in a single file and in a folder. It gets the content through the Git LFS batch API, and it checks the size and the SHA-256 against the pointer.
2. The hf driver. Done in commit 780052f. It reads Hugging Face models, datasets and Spaces through the Hub API. It pins to a commit, and it takes branch:, tag:, commit: and ts:. It is read-only at first.
3. Catalog entries. Done. The lookup takes entries by the rules above, in the YAML file and in etcd.
4. The lifecycle spots and the hooks. Done. The catalog runs each operation through its spots. The proxy configuration and the entry points come in, with two hooks, provcat.log and provcat.logical_only. Nothing changes for users who have no hooks file. The catalog pins before a fetch only where the driver already did, and cache layers are not built yet.
5. The fsspec file system. Done. catalog.fs() gives the adapter, with a pin at open. The logical schemes register as fsspec protocols. A read fetches the whole pinned file at open, and the cache of blocks is not built yet. A bucket without versioning cannot be opened, because it has no version to pin.
6. provrun on the hooks. The broker becomes a gate hook, and the evidence becomes a recorder hook. The agent moves to its own process or container, and undeclared opens go from it to the provrun process. This step changes provrun, so it needs agreement on provrun’s design first. provrun now reads pinned physical URIs to check a bundle, so a gate that refuses physical URIs would stop it.

Steps 1 to 3 do not depend on each other. Step 5 needs step 4, and step 6 needs steps 4 and 5.

Decisions

• Undeclared opens. Code in a run can open a URI that the run did not declare. The gate decides at that time.
• No gates in etcd. Hooks come from the proxy configuration now, and from the profile and the permissions later.
• The identity of a partial read. The evidence of a read of blocks holds the version from the store and the byte ranges that the reader read. A hash of the whole file would make the reader download the whole file.
• The cache root. cache.root stays in the configuration, as the default for users who have no hooks file. The proxy configuration can set another root.
• The kind of a Hugging Face repository. It goes in the physical URI, as in hf://datasets/..., which is the form that HfFileSystem uses. A store option would need one store for each kind.