provrun.telemetry
In a controlled run, the evidence is the only record of the run, unless the
operator’s telemetry policy approves another. provevi reads and checks the
policy (provevi.telemetry); this module applies it to the process.
The policy file comes from the policy argument of a run, or else from the
environment variable PROVRUN_TELEMETRY_POLICY. With neither, the policy
approves nothing. A run file cannot name the policy file. Its telemetry
section can only narrow the policy.
For each run, provrun:
- sets a few settings before the backend is imported, so that a library reads
values that keep it quiet (
SETTINGS), and puts them back when the last run that is going on ends. It leaves every other variable as the user set it, and records the telemetry variables (variables): where they point, never a credential; - sets the global OpenTelemetry tracer, meter and logger providers to its own, once for the process. Its tracer and logger providers refuse a processor that provrun did not add;
- turns off MLflow’s autolog and tracing when MLflow is loaded, empties
LiteLLM’s callback lists, and holds the loggers that can log request bodies
at WARNING. It does this as each run starts and around each model call
(
recheck), and puts LiteLLM’s lists and the logger levels back when the last run ends; - finds a configured Logfire and an OpenTelemetry auto-configuration, which send past the policy and cannot be turned off;
- builds a PydanticAI agent with instrumentation off, or sent to provrun’s
providers when the policy approves it (
pydantic_ai_instrument), and takes a model out of an InstrumentedModel (pydantic_ai_model); - sends the run’s spans only to the places that the policy approves, and
keeps an approved library’s spans as evidence (
destinations).
Each refusal, and each thing turned off, is written to each open run’s
evidence as telemetry_blocked, with the distribution that did it.
The providers stay for the rest of the process: OpenTelemetry lets a process set each global provider only once. MLflow’s autolog and tracing stay off too. So a program that runs provrun, and sets its own providers later, keeps provrun’s.
load_policy
load_policy(policy: TelemetryPolicy | str | Path | None = None) -> TelemetryPolicyFind the operator’s telemetry policy.
It is the policy given, else the file that PROVRUN_TELEMETRY_POLICY
names, else the policy that approves nothing. Raises RunSettingError if
the file is bad.
| Parameter | Type | Default | Description |
|---|---|---|---|
policy |
TelemetryPolicy | str | Path | None | None |
A policy, or the path of its file; None reads PROVRUN_TELEMETRY_POLICY |
| returns | TelemetryPolicy | The operator’s policy |
variables
variables(environ: Mapping[str, str]) -> dict[str, str | None]Say which telemetry variables are set, and where they point, never a credential.
The variables are those of FAMILIES, less provrun’s SETTINGS. A variable
that names a place, such as an endpoint or a URI, maps to its value
without credentials; a protocol or service name maps to its value when
that is a plain name, else to “
| Parameter | Type | Default | Description |
|---|---|---|---|
environ |
Mapping[str, str] | required | The environment variables, such as os.environ |
| returns | dict[str, str | None] | Each telemetry variable, in name order, with what may be recorded of its value |
providers
providers() -> _ProvidersGive provrun’s OpenTelemetry providers, made and set as the global ones on the first call.
The tracer provider takes no span processor but provrun’s router, which the first call adds.
| Parameter | Type | Default | Description |
|---|---|---|---|
| returns | _Providers | The providers: tracer, meter and logger, and foreign, each global one someone else set first |
recheck
recheck(telemetry: Telemetry | None = None, *, fail: bool = True) -> list[str]Turn off again what a library switched on during a run.
That is MLflow, a LiteLLM callback or a logger level. Then look for what cannot be turned off, such as a configured Logfire. A backend calls it around each model call. Each problem goes to each open run once, as Telemetry.found says. Outside a run it does nothing.
| Parameter | Type | Default | Description |
|---|---|---|---|
telemetry |
Telemetry | None | None |
The telemetry of the run that calls, which fail checks |
fail |
bool | True |
Raise RunSettingError if telemetry is strict and has found a problem |
| returns | list[str] | The problems found; empty outside a run |
pydantic_ai_instrument
pydantic_ai_instrument(policy: TelemetryPolicy) -> AnyGive the value for agent.instrument of an agent that provrun builds.
It is False unless the policy approves PydanticAI’s telemetry; then it is
settings that send its spans to provrun’s providers. The policy may name PydanticAI as
pydantic-ai or as pydantic-ai-slim. The spans hold prompts and replies only
when every one of its destinations receives bodies. The value set on the agent
wins over Agent.instrument_all.
| Parameter | Type | Default | Description |
|---|---|---|---|
policy |
TelemetryPolicy | required | The run’s policy |
| returns | Any | False, or PydanticAI’s InstrumentationSettings on provrun’s providers |
pydantic_ai_model
pydantic_ai_model(model: Any, telemetry: Telemetry | None = None) -> AnyTake the InstrumentedModel off a model, which would send spans by its own settings.
Each one taken off is written as telemetry_blocked. Raises ValueError for an InstrumentedModel inside another model, such as a FallbackModel: PydanticAI looks only at the outer model, so the inner one’s spans would go to its own provider, and provrun does not change the caller’s model.
| Parameter | Type | Default | Description |
|---|---|---|---|
model |
Any | required | A PydanticAI model, or anything else, which comes back as it is |
telemetry |
Telemetry | None | None |
The run whose model it is, which gets the events; None sends them to each open run |
| returns | Any | The model, with each InstrumentedModel around it taken off |
class Telemetry
Telemetry(policy: TelemetryPolicy, mode: str, variables: dict[str, str | None] = <factory>, overridden: dict[str, str] = <factory>, problems: list[str] = <factory>, imported_before: list[str] = <factory>, recorder: Any = None, pending: list[tuple[str, str | None, str]] = <factory>, seen: set[str] = <factory>, failed: list[str] = <factory>, baseline: dict[str, tuple[Any, bool]] | None = None, recorded: int | None = None, ended: bool = False, checked: bool = False, routes: Any = None)The telemetry control of one run: its policy, and what it found.
| Parameter | Type | Default | Description |
|---|---|---|---|
policy |
TelemetryPolicy | required | The operator’s policy, narrowed by the run file’s telemetry section |
mode |
str | required | The evidence mode, “strict” or “lenient”; strict fails the run on a problem |
variables |
dict[str, str | None] | <factory> |
The telemetry variables, as variables() records them |
overridden |
dict[str, str] | <factory> |
Each setting that the user had set otherwise, with the user’s value |
problems |
list[str] | <factory> |
The problems found before the run opened, which run_created records |
imported_before |
list[str] | <factory> |
The libraries imported before the run set the environment |
recorder |
Any | None |
The run’s EvidenceRecorder, once the run is open |
pending |
list[tuple[str, str | None, str]] | <factory> |
Blocked before it was open |
seen |
set[str] | <factory> |
Each problem, reported once |
failed |
list[str] | <factory> |
In strict mode, problems found during the run |
baseline |
dict[str, tuple[Any, bool]] | None | None |
Each function at the run’s start |
recorded |
int | None | None |
How many problems run_created’s record holds |
ended |
bool | False |
The end checks have run |
checked |
bool | False |
The start’s check is past; a problem now fails or warns as it is found |
routes |
Any | None |
Where the run’s spans go, a destinations.Routes |
Telemetry.prepare
Telemetry.prepare(policy: TelemetryPolicy, section: Any, mode: str) -> TelemetryNarrow the policy by the run file’s telemetry section, and check the installed libraries.
Raises RunSettingError if the section widens the policy. A problem
found here fails a strict run when applied starts.
| Parameter | Type | Default | Description |
|---|---|---|---|
policy |
TelemetryPolicy | required | The operator’s policy |
section |
Any | required | The run file’s telemetry section, or None |
mode |
str | required | The evidence mode, “strict” or “lenient” |
| returns | Telemetry | The run’s telemetry, with the problems the installed libraries give |
Telemetry.applied
Telemetry.applied(store: Any = None, *, close: bool = True) -> Iterator[Telemetry]Set the environment and the providers for the length of the run.
A strict run with a problem raises RunSettingError as it starts; a lenient one warns. Sending the last spans as routes close can take seconds.
| Parameter | Type | Default | Description |
|---|---|---|---|
store |
Any | None |
The run’s evidence store; None is the file store |
close |
bool | True |
Close routes at the end; the runner passes False and closes them in a thread |
| returns | Iterator[Telemetry] | This telemetry, inside a with block |
Telemetry.record
Telemetry.record() -> dictGive what run_created records: the policy, and what this run changed and found.
| Parameter | Type | Default | Description |
|---|---|---|---|
| returns | dict | The policy’s digest and source, the variables, the overridden settings, the problems, imported_before and the routes |
Telemetry.found
Telemetry.found(problem: str) -> NoneTake a problem found during the run.
It is written as a telemetry_problem event; a strict run then fails at its next check, and a lenient one warns. Each problem counts once. Before the start’s check, it goes into run_created’s problems, and that check fails or warns.
| Parameter | Type | Default | Description |
|---|---|---|---|
problem |
str | required | What was found, in words |
Telemetry.fail_if_found
Telemetry.fail_if_found() -> NoneRaise RunSettingError if a strict run found a problem.
Telemetry.end
Telemetry.end(*, fail: bool = True) -> NoneRun the checks at the run’s end: a last recheck, and the full scan.
The scan finds a function wrapped during the run. Each check runs once.
| Parameter | Type | Default | Description |
|---|---|---|---|
fail |
bool | True |
Raise RunSettingError if a strict run found a problem; a run that has failed passes False, to keep its own error |
Telemetry.attach
Telemetry.attach(recorder: Any, trace_id: str | None = None) -> NoneWrite the run’s telemetry events to its evidence from now on.
The telemetry_blocked events found before the evidence was open come first. A problem found after run_created’s record was taken is written as a telemetry_problem event.
| Parameter | Type | Default | Description |
|---|---|---|---|
recorder |
Any | required | The run’s EvidenceRecorder, now open |
trace_id |
str | None | None |
The run’s trace ID in hex, whose spans go to the run’s destinations from now on |
Telemetry.blocked
Telemetry.blocked(path: str, *, library: str | None = None, detail: str = '') -> NoneWrite a telemetry_blocked event.
Before the run’s evidence is open, the event waits for attach;
after it closes, it is a warning.
| Parameter | Type | Default | Description |
|---|---|---|---|
path |
str | required | What was blocked, such as pydantic_ai.instrumented_model or destination |
library |
str | None | None |
The distribution that did it, when known |
detail |
str | '' |
What happened, in words |