provcat.fs

An fsspec file system over a Catalog. It needs provcat[fs].

fs = catalog.fs() with fs.open(“data+table://acme/nlp/prod/corpora/wiki/part-0.csv”) as f: rows = f.read() with fsspec.open(“data+table://acme/nlp/prod/corpora/wiki/part-0.csv”) as f: …

The file system is a thin adapter. Each method calls a Catalog method, so the hooks of the app run at every spot, and the base cache is the only cache. catalog.fs() registers the logical schemes of the catalog as fsspec protocols, so fsspec.open, and libraries built on fsspec, take logical URIs. A scheme that fsspec already knows, such as s3 or data, is refused. Physical URIs work through the file system object, but they are not registered, so fsspec’s own s3, gcs and file systems stay as they are.

Paths are whole URIs, with the scheme and any ?at=. A name from ls or find is a URI under the one that was listed, with any / in its selector written %2F. A path with ?at= is taken as it is, not as a glob pattern.

A writer puts its file at close. fsspec’s text mode and compression close the file even when a with block fails, so the partial file is put. Inside with fs.transaction: each write is put only when the block ends without an error. fsspec’s caching layers, such as simplecache::, keep copies that the hooks do not see, so do not use them over a catalog.

class CatalogFileSystem

source

CatalogFileSystem(catalog: Catalog | None = None, **kw: Any)

fsspec over a Catalog. Use catalog.fs() to make one.

The call is CatalogFileSystem(catalog=None, **kw); fsspec’s metaclass hides it from the signature above.

The keyword arguments, other than catalog, go to fsspec’s AbstractFileSystem, as its storage options.

Parameter	Type	Default	Description
`catalog`	Catalog \| None	`None`	The catalog to read and write through; None takes the bound one, or Catalog.open()
`kw`	Any		Options for fsspec’s AbstractFileSystem

CatalogFileSystem.over

source

CatalogFileSystem.over(catalog: Catalog, register: bool = True, replace: bool = False) -> CatalogFileSystem

Make a file system over a catalog; by default, register its logical schemes with fsspec.

A later catalog with the same app, principal, tenant and hooks takes the scheme over. One with another context, or other hooks, is refused, so fsspec never runs one caller’s reads through another’s hooks; it uses register=False, or replace=True to take the scheme over on purpose, for example when a notebook cell runs again.

Parameter	Type	Default	Description
`catalog`	Catalog	required	The catalog to read and write through
`register`	bool	`True`	Make each logical scheme of the catalog an fsspec protocol that opens this catalog
`replace`	bool	`False`	Take a scheme over from a catalog with another context or other hooks
returns	CatalogFileSystem		A file system over the catalog; ConfigError when a scheme clashes

CatalogFileSystem.unstrip_protocol

source

CatalogFileSystem.unstrip_protocol(name: str) -> str

Return the path unchanged, because a path here is a whole URI.

Parameter	Type	Default	Description
`name`	str	required	A path, which is a whole URI
returns	str		The same path: it already has its scheme

CatalogFileSystem.info

source

CatalogFileSystem.info(path: str, **kw: Any) -> dict

Describe one file or folder through Catalog.stat.

provcat ignores the keyword arguments that fsspec passes.

Parameter	Type	Default	Description
`path`	str	required	A URI, logical or physical
`kw`	Any		Ignored; fsspec passes options here
returns	dict		name, size, type (“file” or “directory”), version and uri; FileNotFoundError when there is none

CatalogFileSystem.ls

source

CatalogFileSystem.ls(path: str, detail: bool = True, **kw: Any) -> list

List a folder through Catalog.list. Each name is a URI under path.

provcat ignores the keyword arguments that fsspec passes.

Parameter	Type	Default	Description
`path`	str	required	A URI, logical or physical
`detail`	bool	`True`	Return the info dict of each entry, not only its name
`kw`	Any		Ignored; fsspec passes options here
returns	list		For a folder, its entries; for a file, the file itself. FileNotFoundError when there is none

CatalogFileSystem.expand_path

source

CatalogFileSystem.expand_path(path: str | list[str], recursive: bool = False, maxdepth: int | None = None, assume_literal: bool = False, **kw: Any) -> list[str]

Expand glob patterns as fsspec does, except that a path with ?at= is taken literally.

The other keyword arguments go on to fsspec’s expand_path.

Parameter	Type	Default	Description
`path`	str \| list[str]	required	A URI or glob pattern, or a list of them
`recursive`	bool	`False`	Include everything under each folder
`maxdepth`	int \| None	`None`	How many levels deep a recursive expansion goes; None for all
`assume_literal`	bool	`False`	Take the paths as they are, not as glob patterns; a path with ?at= always is
`kw`	Any		Passed on to fsspec
returns	list[str]		The matching paths

CatalogFileSystem.rm_file

source

CatalogFileSystem.rm_file(path: str) -> None

Delete one file. FileNotFoundError when there is none.

Parameter	Type	Default	Description
`path`	str	required	The URI of a file on a branch

CatalogFileSystem.cp_file

source

CatalogFileSystem.cp_file(path1: str, path2: str, **kw: Any) -> None

Copy a file by reading it through the catalog and writing it back.

provcat ignores the keyword arguments that fsspec passes.

Parameter	Type	Default	Description
`path1`	str	required	The URI of the file to copy
`path2`	str	required	The URI to write the copy at, on a branch
`kw`	Any		Ignored; fsspec passes options here

CatalogFileSystem.mkdir

source

CatalogFileSystem.mkdir(path: str, create_parents: bool = True, **kw: Any) -> None

Do nothing: a folder exists when a file is in it.

provcat ignores create_parents and the keyword arguments that fsspec passes.

Parameter	Type	Default	Description
`path`	str	required	The URI of the folder
`create_parents`	bool	`True`	Ignored
`kw`	Any		Ignored

CatalogFileSystem.makedirs

source

CatalogFileSystem.makedirs(path: str, exist_ok: bool = False) -> None

Do nothing: a folder exists when a file is in it.

Parameter	Type	Default	Description
`path`	str	required	The URI of the folder
`exist_ok`	bool	`False`	Ignored

CatalogFileSystem.ukey

source

CatalogFileSystem.ukey(path: str) -> str

A key that changes when the file changes.

Parameter	Type	Default	Description
`path`	str	required	A URI, logical or physical
returns	str		The version that the store gives, or fsspec’s own key when it gives none