provcat.stores.objects

The shared logic of object stores: S3, Google Cloud Storage and Azure Blob.

An object store keeps flat keys such as a/b/c.csv in a bucket, and “/” makes the folders. ObjectStore turns that into stat, list and fetch, with the cache, folder fetches and offline mode. A driver gives six primitives:

locate(uri): the bucket and the key that a URI names.
_head(uri, bucket, key): the size and version of one object, or NotFound.
_pages(uri, bucket, prefix, delimiter, limit): pages of a listing. Each page is a pair of lists, the folder prefixes and the objects as (key, size, version).
_get(uri, bucket, key, out): write the object’s bytes to the open file out, and return the version of the bytes written.
_put(uri, bucket, key, src): write the local file src as the object, and return the version of the bytes written.
_delete(uri, bucket, key): delete the object, or raise NotFound.

The version is whatever changes when the bytes change: the ETag on S3, the generation on GCS, the ETag on Azure. The cache keeps it, and fetches a file again when it changes.

A bucket with versioning on keeps every version of an object, and gives each an ID. Then a URI can name a fixed version:

?at=version:ID names one version of one file. pin of a file gives it.
?at=commit:ID names a provcat commit. commit lists every file under a folder with its version ID, and writes the list as one object, .provcat/commits/<ID>.json, at the top of the bucket. The ID is the SHA-256 of that object, so a reader checks the list it got. A folder, or a file in it, at a commit reads the versions in the list. pin of a folder gives its newest commit, or the newest commit of a folder above it.

A bucket writes one object at a time, so a commit takes two steps that make it safe while other programs write:

The file list is a snapshot. A commit lists every version under the folder, deleted ones too, twice. Each write makes a new version ID, so two equal listings mean that nothing changed between them. Then each listed version was current at one moment. If they differ, it tries again, and then gives up.
The commits of a folder have one order. Commit N of a folder is the object .provcat/log/<SHA-256 of the folder>/<10^15 - N>, which holds its ID. The commit creates it only if it is not there yet, so of two commits at once one gets N and the other tries again with N + 1. The newest commit lists first, and each commit names the one before it as its parent.

branch copies a branch, the first folder in the bucket, or a commit, to a new branch. The copy is file by file, so a branch that fails halfway is left half made.

A driver with versions gives six more primitives:

_version_of(uri, bucket, key): the version ID of the current object.
_history(uri, bucket, prefix): every version under the prefix, deleted ones too, as (key, version ID, size, current). A delete marker has the size None.
_head_version and _get_version: _head and _get of one version.
_create(uri, bucket, key, src): _put only if no object has the key; else it raises Exists.
_copy(uri, bucket, key, version, to): copy an object, or one version of it.

_version_of and _history raise a StoreError when the bucket keeps no versions.

A folder fetch refuses a folder that holds keys no URI can name, for example a//b, and names that one disk could store as one, for example Dir/a and dir/b. Fetches of one folder run one at a time. It marks the folder done only after every file is current and every deleted object is gone from the cache.

class ObjectStore

source

The base class of a driver for an object store, which keeps flat keys in buckets.

It gives stat, list, fetch, put and delete, with the cache, folder fetches and offline mode, and on a bucket with versioning pin, commit and branch. It takes the constructor of Store. A driver subclasses it, sets driver, and gives six primitives: locate, _head, _pages, _get, _put and _delete. For versions it gives more: _version_of, _history, _head_version, _get_version, _create and _copy. The module’s docstring says what each returns.

ObjectStore.locate

source

ObjectStore.locate(uri: Uri) -> tuple[str, str]

Find the bucket and the key that a URI names. A driver gives it; a URI it cannot name raises UriError.

Parameter	Type	Default	Description
`uri`	Uri	required	A physical URI of this store
returns	tuple[str, str]		The bucket and the key, which is the URI’s path joined by “/”, empty for the bucket

ObjectStore.stat

source

ObjectStore.stat(uri: Uri, cache: LocalCache) -> Entry

Describe one file or folder. A key with keys below it is a folder; a bucket always is.

Without ?at=, offline mode raises OfflineMiss.

Parameter	Type	Default	Description
`uri`	Uri	required	A physical URI of this store, with or without ?at=
`cache`	LocalCache	required	The cache, which holds the commits read
returns	Entry		The file or the folder; NotFound when there is neither

ObjectStore.list

source

ObjectStore.list(uri: Uri, cache: LocalCache) -> list[Entry]

List a folder, one level deep. Without ?at=, offline mode raises OfflineMiss.

Parameter	Type	Default	Description
`uri`	Uri	required	A physical URI of a folder, with or without ?at=
`cache`	LocalCache	required	The cache, which holds the commits read
returns	list[Entry]		The files and folders directly inside it; NotFound for a file or nothing

ObjectStore.walk

source

ObjectStore.walk(uri: Uri) -> list[Entry]

List every file below a folder, at any depth.

Keys that end in “/” are folder markers and are left out. Any other key that no URI can name is an error, so a fetched folder is never short.

Parameter	Type	Default	Description
`uri`	Uri	required	A physical URI of a folder, without ?at=
returns	list[Entry]		One Entry for each file, at any depth

ObjectStore.fetch

source

ObjectStore.fetch(uri: Uri, cache: LocalCache) -> Path

Copy a file or a whole folder into the cache, or find it there current.

Offline, a URI without ?at= gives what the cache holds at any version, or raises OfflineMiss.

Parameter	Type	Default	Description
`uri`	Uri	required	A physical URI of a file or a folder, with or without ?at=
`cache`	LocalCache	required	The cache to copy it into
returns	Path		The file or folder in the cache

ObjectStore.put

source

ObjectStore.put(uri: Uri, src: Path, cache: LocalCache) -> Entry

Write a local file as one object. A bucket, a folder, or a key under .provcat is refused.

Parameter	Type	Default	Description
`uri`	Uri	required	The physical URI of the object to write, without ?at=
`src`	Path	required	The local file to write
`cache`	LocalCache	required	The cache; a put does not use it
returns	Entry		The file written, with its size and version

ObjectStore.delete

source

ObjectStore.delete(uri: Uri, cache: LocalCache) -> None

Delete one object.

Parameter	Type	Default	Description
`uri`	Uri	required	The physical URI of the object to delete, without ?at=
`cache`	LocalCache	required	The cache; a delete does not use it

ObjectStore.pin

source

ObjectStore.pin(uri: Uri, cache: LocalCache) -> Uri

Pin a file to its version, and a folder to the newest commit that holds it.

A bucket without versioning raises StoreError for a file, and a folder with no provcat commit raises StoreError.

Parameter	Type	Default	Description
`uri`	Uri	required	A physical URI of a file or a folder
`cache`	LocalCache	required	The cache, which holds the commits read
returns	Uri		The URI with ?at=version:ID for a file, or ?at=commit:ID for a folder; a URI with ?at= is checked and returned

ObjectStore.commit

source

ObjectStore.commit(uri: Uri, message: str, metadata: dict[str, str] | None = None, allow_empty: bool = False) -> Uri

Record the version of every file under a folder as a new provcat commit.

It needs versioning on the bucket. A folder with no files raises StoreError, and so does one with no change, unless allow_empty.

Parameter	Type	Default	Description
`uri`	Uri	required	The physical URI of a folder, without ?at=
`message`	str	required	The commit message; it must not be empty
`metadata`	dict[str, str] \| None	`None`	Key-value pairs stored with the commit
`allow_empty`	bool	`False`	Commit even when no file changed since the last commit
returns	Uri		The folder pinned to the new commit, with ?at=commit:ID

ObjectStore.branch

source

ObjectStore.branch(uri: Uri, source: str) -> Uri

Make a branch by copying another branch or a commit, file by file.

A branch that already exists raises StoreError. A copy that fails halfway leaves the branch half made.

Parameter	Type	Default	Description
`uri`	Uri	required	The physical URI of the new branch: a folder at the top of the bucket, without ?at=
`source`	str	required	A branch, which is a folder at the top of the bucket, or a commit ID
returns	Uri		uri itself