provcat.stores.hf

Hugging Face repositories: hf://datasets/owner/repo/path?at=branch:main.

It needs the hf extra: pip install provcat[hf].

The authority is the kind of repository: models, datasets or spaces, as in the paths of HfFileSystem and the Hub API. The next two segments are the owner and the name of the repository, and the rest is the path in it.

The selector names a git ref with branch:, tag: or commit:. Without a selector the store uses main, the default branch of every Hugging Face repository. ?at=ts:TIME names the newest commit of main whose date is at or before TIME, and ?at=ts:BRANCH@TIME that of BRANCH. The Hub lists the commits of a branch newest first, but it does not give their parents. So a commit that someone made on another branch and merged with a merge commit counts too. The Hub itself merges a pull request as one new commit, so this matters only for history that someone pushed with git.

Every read first pins the ref to a commit. The cache key holds the commit::

<cache>/hf/<kind>/<owner>/<repo>/<commit>/<path>

The store downloads each file with the functions that hf_hub_download of huggingface_hub uses, which also read files that Xet keeps. It writes the file only into the provcat cache, not into the cache of huggingface_hub, and it keeps no listings in memory. It checks each file it downloads: a file that Xet or Git LFS keeps against the SHA-256 that the Hub gives, and any other file against its git blob ID. A folder comes file by file, from one listing of the folder at the commit.

The store cannot write.

Options:

endpoint: the Hub root. The default comes from huggingface_hub, which reads HF_ENDPOINT and otherwise uses https://huggingface.co. The token goes to it, so it must use HTTPS.
token_env: the environment variables to read a token from, in order. If none is set, the store sends no token. Without this option, huggingface_hub finds the token itself: in HF_TOKEN, or in the file that hf auth login writes.
timeout: seconds for each call to the Hub API. The default is 60. A download of a file over plain HTTP waits HF_HUB_DOWNLOAD_TIMEOUT seconds instead, 10 by default, and a Xet download has its own timeouts.