provcat.cache

The local cache.

The cache keeps a readable copy of remote objects, so a folder of it can be mounted into a container. The layout is::

<root>/<store>/<key...>                the cached files
<root>/.provcat/meta/<hash>.json       the version of one cached file
<root>/.provcat/done/<hash>.json       a folder that was fetched whole
<root>/.provcat/refs/<hash>            a moving name pinned to a version
<root>/.provcat/tmp/                   files and folders being written
<root>/.provcat/locks/<hash>.lock      one lock for each key being written
<root>/.provcat/snapshots/<kind>/<sha>/<name>  a read-only copy, named by its content

A store decides the key. GitHub keys hold the commit, so they never change. S3 keys are bucket and object key, and the meta file holds the ETag.

The records under .provcat are named by a hash of store and key, so two keys can never share or block a record. A meta record also holds the size, the modification time, the inode and the SHA-256 of the file it describes. If the time or the inode differs, as after the cache was copied, the file is hashed again and kept if the hash matches. On Linux a time’s tick can be a few milliseconds, so the inode tells apart two files of one size from one tick. Writes of one key take its lock, so the last record describes the last file, and a read that saw the record change while it looked returns nothing. If the size differs, or the bytes differ from the hash, the file is fetched again. A rewrite in place that keeps the size, the time and the inode, as rsync --inplace -t makes, is not noticed. The locks need a filesystem with flock, such as a local disk. On a share without it, every write fails.

Two names that differ only in case or in Unicode form land in one file on macOS and Windows. The stores refuse a folder that holds such names.

Every write goes to a temporary name under .provcat/tmp and then is renamed, so a reader never sees half a file.

A fetch can rewrite a cached file in place when a branch moves, so a cached path is not safe to mount while something reads it. A snapshot, made by Catalog.snapshot, copies a file or a folder into a folder named by the SHA-256 of its content, and makes the copy read-only. Nothing in provcat writes to a snapshot again, so a mount of one never changes.

class LocalCache

source

LocalCache(root: str | Path | None = None)

The local cache of fetched files and folders, their versions, the pinned refs and the snapshots.

A store driver calls it to find and to write what it fetches. Each key is a list of path segments under the store’s name.

Parameter	Type	Default	Description
`root`	str \| Path \| None	`None`	The cache folder; None takes PROVCAT_CACHE, then XDG_CACHE_HOME/provcat, then ~/.cache/provcat

LocalCache.path

source

LocalCache.path(store: str, key: Iterable[str]) -> Path

Return the cache path of a key, whether or not anything is there.

Parameter	Type	Default	Description
`store`	str	required	The store’s name
`key`	Iterable[str]	required	The path segments under the store
returns	Path		Where the file or folder of the key lives in the cache; UriError for a bad segment

LocalCache.get

source

LocalCache.get(store: str, key: list[str], version: str | None) -> Path | None

Return the cached file if it is there, unchanged, and has this version.

Parameter	Type	Default	Description
`store`	str	required	The store’s name
`key`	list[str]	required	The path segments under the store
`version`	str \| None	required	The version wanted; None accepts any cached version, as offline mode does
returns	Path \| None		The cached file; None when it is missing, changed, or of another version

LocalCache.put

source

LocalCache.put(store: str, key: list[str], fill: Callable[[Path], str | None], version: str | None = None) -> Path

Write a file with fill, then record its version.

The version that fill returns wins over the version argument, so the record always describes the bytes on disk. With neither, StoreError.

Parameter	Type	Default	Description
`store`	str	required	The store’s name
`key`	list[str]	required	The path segments under the store
`fill`	Callable[[Path], str \| None]	required	Writes the file at the temporary path it is given, and may return its version
`version`	str \| None	`None`	The version to record, when fill returns none
returns	Path		The cached file

LocalCache.folder_done

source

LocalCache.folder_done(store: str, key: list[str], version: str | None) -> Path | None

Find a folder that was fetched whole.

Parameter	Type	Default	Description
`store`	str	required	The store’s name
`key`	list[str]	required	The path segments of the folder
`version`	str \| None	required	The version wanted; None accepts any
returns	Path \| None		The cached folder when it was fetched whole at that version; else None

LocalCache.covered

source

LocalCache.covered(store: str, key: list[str], version: str, min_len: int) -> Path | None

Return the cached file or folder if a folder at or above key was fetched whole.

Parameter	Type	Default	Description
`store`	str	required	The store’s name
`key`	list[str]	required	The path segments of the file or folder
`version`	str	required	The version the whole folder must have been fetched at
`min_len`	int	required	The shortest key prefix to look at, such as 1 to stop at the bucket
returns	Path \| None		The cached file or folder; None when no such folder covers it

LocalCache.mark_folder

source

LocalCache.mark_folder(store: str, key: list[str], version: str) -> None

Record that a folder was fetched whole at a version.

Parameter	Type	Default	Description
`store`	str	required	The store’s name
`key`	list[str]	required	The path segments of the folder
`version`	str	required	The version it was fetched at

LocalCache.unmark_folder

source

LocalCache.unmark_folder(store: str, key: list[str]) -> None

Forget that a folder was fetched whole.

Parameter	Type	Default	Description
`store`	str	required	The store’s name
`key`	list[str]	required	The path segments of the folder

LocalCache.prune

source

LocalCache.prune(store: str, key: list[str], keep: set[tuple[str, ...]]) -> None

Delete cached files under key whose relative keys are not in keep.

Parameter	Type	Default	Description
`store`	str	required	The store’s name
`key`	list[str]	required	The path segments of the folder
`keep`	set[tuple[str, …]]	required	The paths under the folder, as tuples of segments, to keep

LocalCache.lock

source

LocalCache.lock(store: str, key: list[str]) -> Iterator[None]

Hold an exclusive lock for one key, across threads and processes.

Parameter	Type	Default	Description
`store`	str	required	The store’s name
`key`	list[str]	required	The path segments to lock
returns	Iterator[None]		A context manager that holds the lock while its block runs

LocalCache.install_folder

source

LocalCache.install_folder(store: str, key: list[str], version: str, fill: Callable[[Path], None]) -> Path

Build a folder with fill(tmp_dir), then move it into place whole.

Installs of one key run one at a time. If another thread or process installed the same version first, its folder stays.

Parameter	Type	Default	Description
`store`	str	required	The store’s name
`key`	list[str]	required	The path segments of the folder
`version`	str	required	The version of the folder, recorded once it is in place
`fill`	Callable[[Path], None]	required	Writes the folder’s contents into the temporary folder it is given
returns	Path		The cached folder

LocalCache.get_ref

source

LocalCache.get_ref(store: str, name: list[str]) -> str | None

Read the version that a moving name was last pinned to, as offline mode does.

Parameter	Type	Default	Description
`store`	str	required	The store’s name
`name`	list[str]	required	The moving name, as segments, such as a repository, a branch and a time
returns	str \| None		The version it was pinned to; None when it was never pinned

LocalCache.set_ref

source

LocalCache.set_ref(store: str, name: list[str], value: str) -> None

Record the version that a moving name points at.

Parameter	Type	Default	Description
`store`	str	required	The store’s name
`name`	list[str]	required	The moving name, as segments
`value`	str	required	The fixed version it points at now

tree_digest

source

tree_digest(p: Path) -> str

The SHA-256 of a file, or of a folder.

A folder's digest covers every entry under it, folders included, so an
empty folder counts. Each entry is ``F�path�sha256

for a file or D�path� for a folder, in sorted order, after the line provcat-tree-v1``. A file name holds no NUL, so the form is unambiguous. A file and a folder can still have equal digests, because a file can hold any bytes; snapshots keep them apart by kind. A link or any other special file is refused, so the digest always describes plain bytes.

Parameter	Type	Default	Description
`p`	Path	required	A file or a folder
returns	str		The hex SHA-256; StoreError for a link or a special file