provevi.experiments

An experiment: a question a team asks again and again in a project.

An experiment is a folder in the catalog, at a logical URI such as experiment://acme/risk/dev/credit/q, holding one file, experiment.yaml. A commit of the folder is a version, so the folder must be in a store with commits. The file says what every run of the experiment shares::

kind: agent-eval
description: Does a retrieval step raise accuracy on the revenue questions?
inputs:
  - data+table://acme/risk/dev/evals/revenue
metrics: [accuracy]

kind is agent-eval, an agentic experiment for evals, or ml, a machine-learning experiment. The file holds no agent spec, so one experiment can try many agents. write_experiment writes and commits the file, and read_experiment reads it at a pin. A run names its experiment beside its agent, and records the pin and the file’s hash in run_created.

class Experiment

source

Experiment(kind: str, description: str, inputs: tuple[str, ...] = (), metrics: tuple[str, ...] = ())

What every run of an experiment shares.

Raises ProveviError for a kind, a description, inputs or metrics that are not valid, and TypeError for inputs or metrics that are not a list or a tuple of texts and cannot be looped over.

Parameter Type Default Description
kind str required One of KINDS: agent-eval or ml
description str required The question the experiment asks; not empty
inputs tuple[str, …] () The logical URIs that each run reads, such as an eval set
metrics tuple[str, …] () The names of the metrics each run reports, such as accuracy

Experiment.to_dict

source

Experiment.to_dict() -> dict[str, Any]

The experiment as the mapping its file holds.

Parameter Type Default Description
returns dict[str, Any] kind, description, inputs and metrics, with lists for the tuples

Experiment.to_bytes

source

Experiment.to_bytes() -> bytes

The file, as YAML with its keys in a fixed order.

Parameter Type Default Description
returns bytes The YAML, in UTF-8

Experiment.from_bytes

source

Experiment.from_bytes(raw: bytes, where: str = 'the experiment') -> Experiment

Read the experiment in a file.

Raises ProveviError for a file that is not an experiment, or has a key it does not know.

Parameter Type Default Description
raw bytes required The bytes of an experiment.yaml
where str 'the experiment' What to call the file in an error, such as its pinned URI
returns Experiment The experiment the file holds

class PinnedExperiment

source

PinnedExperiment(uri: str, pin: str, sha256: str, experiment: Experiment)

An experiment as one version holds it.

Parameter Type Default Description
uri str required The experiment’s URI, with no selector
pin str required The URI pinned to the version, with ?at=commit:
sha256 str required The SHA-256 of experiment.yaml at that version
experiment Experiment required The experiment that file holds

write_experiment

source

write_experiment(catalog: Catalog, uri: str, experiment: Experiment, message: str) -> PinnedExperiment

Write an experiment’s file in its folder, and commit the folder.

The catalog’s hooks see the write and the commit. Raises ProveviError, and provcat’s errors, as for a store with no commits.

Parameter Type Default Description
catalog Catalog required The catalog to write through
uri str required The experiment’s folder, such as experiment://acme/risk/dev/credit/q, with no ?at=
experiment Experiment required The experiment to write
message str required The commit message
returns PinnedExperiment The new version, as read_experiment reads it

read_experiment

source

read_experiment(catalog: Catalog, uri: str) -> PinnedExperiment

Read an experiment at a pinned version.

The version is the newest, or the one that the URI’s ?at= names. Raises ProveviError for a store whose pin names no version, as a file store’s does.

Parameter Type Default Description
catalog Catalog required The catalog to read through
uri str required The experiment’s folder, with ?at= to name a version
returns PinnedExperiment The experiment, its pin and its file’s hash