provevi.telemetry

The telemetry policy: what may record or send data from a run, besides the evidence.

During a run, provrun turns off each path by which a library records or sends data, such as MLflow’s autolog or LiteLLM’s callbacks. The operator approves exceptions in a policy file, which this module reads and checks::

version: 1
libraries:
  pydantic-ai:
    versions: ">=2.64,<2.65"    # the versions that were tested
    telemetry: [logfire]        # the destinations its instrumentation may feed
destinations:
  logfire:
    kind: logfire               # logfire, otlp, mlflow or live
    receives: usage             # names, usage or bodies
    endpoint: https://logfire-us.pydantic.dev
hosts:
  - logfire-us.pydantic.dev     # a host that a destination may send to
  - .amazonaws.com              # a leading dot: any host under the domain

A run file’s telemetry section can narrow the policy with narrow, but never widen it. attributes_for gives the span attributes that a destination receives at its level, and split_body takes the bodies out of a span’s attributes, so that the evidence keeps them as a payload. match_destination finds the destination that a telemetry variable, such as OTEL_EXPORTER_OTLP_ENDPOINT, points at. otlp_traces_target gives the protocol and the address to which provrun sends traces over OTLP.

The hosts list names the hosts that destinations may send to. The policy checks each destination’s endpoint against it, and the list is also a record for the firewall. Neither provevi nor provrun limits a run’s connections. That belongs to the firewall or the container that runs it.

provrun applies the policy. This module only reads, checks and describes it.

attributes_for

source

attributes_for(attributes: Mapping[str, Any], level: str) -> dict[str, Any]

Cut a span’s attributes to those that a destination at a level receives.

Span events, such as an exception with its message, are bodies, and this function does not see them. Raises ProveviError for a level that is not one.

Parameter Type Default Description
attributes Mapping[str, Any] required A span’s attributes
level str required The destination’s level, one of LEVELS
returns dict[str, Any] The attributes it receives at that level

distribution_of

source

distribution_of(module: str) -> str | None

Find the distribution that installed a module, to name a library in a report.

When it cannot tell, it guesses: a module that no distribution installed gives its own top name, as a program may name its own. To decide what the policy approves, use span_libraries, which does not guess.

Parameter Type Default Description
module str required A module’s dotted name, such as opentelemetry.sdk.trace
returns str | None The distribution’s name, as pip knows it, such as opentelemetry-sdk, or a guess; None for an empty name

span_libraries

source

span_libraries(scope: str) -> tuple[str, ...]

Find the names that the policy may know a span’s library by.

It gives the scope name only if a distribution of that name is installed and the name has no dot, and the distribution whose files hold the scope’s module. It does not guess: a scope whose module no installed distribution holds, or any scope when the installed packages cannot be read, gives no name, so the policy approves none of its spans. A process reads the installed packages once, and does not see a change to them after that; span_libraries.cache_clear() and the same call on _shipped and _files make it read them again.

Parameter Type Default Description
scope str required A span’s instrumentation scope name, such as pydantic_ai
returns tuple[str, …] The library names, as canonical distribution names; empty when none can be shown

plain_name

source

plain_name(value: Any, *, declared: Collection[str] = ()) -> bool

Tell whether a value is a plain name, which a place below bodies may get.

A plain name is a name, as the levels say, that holds no URL, no user and password, no setting such as token=x, no path from /, no path that looks like one of a machine’s files, such as home/someone/.ssh/id_rsa or x/key.pem, no part before a / that is a URL’s scheme, a host or an email, such as https:/host/x or x/host.com/a, and no more than one /, and none that starts like a common API key, such as sk- or ghp_. A secret or a path can pass the levels’ test of a name, so a place below bodies should get only plain names. A name such as ns:model@tag looks like user:password@host, so it is not plain. A whole number and a yes or no are plain, as they are names; a number with a fraction is not, nor is an empty list.

org/model is plain, and tmp/a/b is not. A model name such as openrouter/meta-llama/llama-3-70b has two / too, so a text in declared may have more. It must pass every other test. Raises TypeError when declared is one text.

Parameter Type Default Description
value Any required A value from a span or an event: a text, a whole number, a yes or no, or a list of texts
declared Collection[str] () Texts that may have more than one /, such as a model’s name; not one text
returns bool Whether a place below bodies may get the value

plain_attribute

source

plain_attribute(key: str, value: Any, *, model_keys: Collection[str] = frozenset({'gen_ai.request.model', 'gen_ai.response.model', 'mlflow.llm.model'})) -> bool

Tell whether a value is a plain name for its key.

A model’s name, under one of model_keys, may have more than one /, as openrouter/meta-llama/llama-3-70b or a Vertex model’s path has, and so may each name in a list of them. Any other value is a plain name as plain_name says.

Parameter Type Default Description
key str required The attribute’s or the data’s key
value Any required Its value
model_keys Collection[str] frozenset({'gen_ai.request.model', 'gen_ai.response.model', 'mlflow.llm.model'}) The keys whose value is a model’s name; RUN_MODEL_KEYS for provrun’s own events
returns bool Whether a place below bodies may get the value

plain_experiment

source

plain_experiment(value: Any) -> bool

Tell whether an experiment’s URI or pin is one that a place below bodies may get.

One such is experiment://acme/risk/dev/credit/q?at=commit:7311e8.... Its path, and each part of it, is a plain name, though the path may have more than one /, and the pin names a commit. A URI with a branch, a host or any other setting is not plain.

Parameter Type Default Description
value Any required An experiment’s URI or pin, from an event’s data
returns bool Whether a place below bodies may get the value

plain_span_name

source

plain_span_name(name: Any) -> bool

Tell whether a span’s name is one that a place below bodies may get.

It must be one or two words of name characters, such as chat gpt-4o, each a plain name.

Parameter Type Default Description
name Any required A span’s name
returns bool Whether a place below bodies may get the name

split_body

source

split_body(attributes: Mapping[str, Any]) -> tuple[dict[str, Any], dict[str, Any]]

Split a span’s attributes into what an event may keep and the bodies.

An event keeps the names and the usage; the evidence keeps the bodies as a payload.

Parameter Type Default Description
attributes Mapping[str, Any] required A span’s attributes
returns tuple[dict[str, Any], dict[str, Any]] The names and usage, then the bodies

blocked_data

source

blocked_data(path: str, *, library: str | None = None, detail: str = '') -> dict

The data of a telemetry_blocked event. Raises ProveviError for an empty path.

Parameter Type Default Description
path str required What was blocked, such as litellm.callbacks or network; not empty
library str | None None The distribution that tried it, when known; it is stored in its canonical form
detail str '' More about the attempt
returns dict path, detail, and library when it is given

class Library

source

Library(name: str, versions: str, telemetry: tuple[str, ...] = ())

A library whose telemetry the policy approves.

Parameter Type Default Description
name str required The distribution’s canonical name, such as pydantic-ai
versions str required The tested versions, as a specifier such as >=2.64,<2.65
telemetry tuple[str, …] () The names of the destinations its instrumentation may feed

Library.allows_version

source

Library.allows_version(version: str) -> bool

Tell whether the policy names a version of the library.

A local build, such as 2.64.0+patched, passes only if the specifier names it. A version that is not one gives False.

Parameter Type Default Description
version str required An installed version, such as 2.64.0
returns bool Whether versions names it

class Destination

source

Destination(name: str, kind: str, receives: str, endpoint: str | None = None)

A place outside the evidence that spans may go to, and what it receives.

Parameter Type Default Description
name str required The destination’s name in the policy
kind str required logfire, otlp, mlflow or live
receives str required Its level, one of LEVELS
endpoint str | None None The URL it sends to; None for a live viewer, or an mlflow copy in the store’s own file

class TelemetryPolicy

source

TelemetryPolicy(libraries: Mapping[str, Library] = <factory>, destinations: Mapping[str, Destination] = <factory>, hosts: tuple[str, ...] = (), source: str = '<none>')

What a run may record or send besides its evidence.

A library or a destination that the policy does not name gets nothing. Each destination’s endpoint must be on a host that hosts names.

Parameter Type Default Description
libraries Mapping[str, Library] <factory> The approved libraries, by canonical name
destinations Mapping[str, Destination] <factory> The destinations, by name
hosts tuple[str, …] () The hosts that destinations may send to; a leading dot names each host under a domain
source str '<none>' Where the policy came from, such as its file’s path, for messages

TelemetryPolicy.deny_all

source

TelemetryPolicy.deny_all() -> TelemetryPolicy

The policy that approves nothing, for a run with no policy file.

Parameter Type Default Description
returns TelemetryPolicy A policy with no libraries, destinations or hosts

TelemetryPolicy.load

source

TelemetryPolicy.load(path: str | Path) -> TelemetryPolicy

Read a policy file, in YAML or JSON.

Raises ProveviError, with every problem the file has.

Parameter Type Default Description
path str | Path required The policy file; ~ is the home folder
returns TelemetryPolicy The policy, with the path as its source

TelemetryPolicy.from_dict

source

TelemetryPolicy.from_dict(doc: Any, source: str = '<dict>') -> TelemetryPolicy

A policy from a parsed document. Raises ProveviError, with every problem.

Parameter Type Default Description
doc Any required The parsed policy document: a mapping with version, libraries, destinations and hosts
source str '<dict>' Where the document came from, for messages
returns TelemetryPolicy The policy

TelemetryPolicy.to_dict

source

TelemetryPolicy.to_dict() -> dict

The policy as a document that from_dict reads back.

Parameter Type Default Description
returns dict version, libraries and destinations in name order, and hosts in the order given

TelemetryPolicy.digest

source

TelemetryPolicy.digest() -> str

The SHA-256 of the policy’s canonical JSON, for a run to record which policy it ran under.

Parameter Type Default Description
returns str 64 lowercase hex digits

TelemetryPolicy.library

source

TelemetryPolicy.library(name: str) -> Library | None

The approved library of that name.

Parameter Type Default Description
name str required A distribution’s name, in any form pip accepts, such as pydantic_ai
returns Library | None The library; None when the policy does not approve it

TelemetryPolicy.destinations_for

source

TelemetryPolicy.destinations_for(library: str) -> list[Destination]

The destinations that the library’s instrumentation may feed.

Parameter Type Default Description
library str required A distribution’s name
returns list[Destination] The destinations; empty for a library the policy does not approve

TelemetryPolicy.live_level

source

TelemetryPolicy.live_level() -> str | None

The level that a viewer of a run’s live feed receives.

None if the policy names no live viewer, so a run shows nothing live.

Parameter Type Default Description
returns str | None One of LEVELS, or None

TelemetryPolicy.allows_host

source

TelemetryPolicy.allows_host(host: str) -> bool

Tell whether the policy’s hosts name a host.

An entry with a leading dot names each host under that domain, but not the domain itself.

Parameter Type Default Description
host str required A host name; case and a trailing dot are ignored
returns bool Whether the host may be sent to

TelemetryPolicy.match_destination

source

TelemetryPolicy.match_destination(kind: str, url: str | None) -> Destination | None

Find the destination of a kind whose endpoint is a URL.

Both are made regular first by normal_endpoint. An mlflow destination with no endpoint, the copy in the store’s own file, matches url None. A url that cannot be made regular matches nothing.

Parameter Type Default Description
kind str required The destination’s kind, such as otlp or mlflow
url str | None required The address a telemetry variable names; None for the store’s own file
returns Destination | None The destination; None when none matches

TelemetryPolicy.check_installed

source

TelemetryPolicy.check_installed(versions: Mapping[str, str] | None = None) -> list[str]

Find each approved library that is installed at a version the policy does not name.

Parameter Type Default Description
versions Mapping[str, str] | None None Each distribution’s version; None for the versions installed in this Python
returns list[str] One problem each; empty when every approved library is at a version the policy names, or is not installed

TelemetryPolicy.narrow

source

TelemetryPolicy.narrow(section: Any, source: str = 'the run file') -> TelemetryPolicy

Narrow the policy by a run file’s telemetry section.

A key that the section leaves out keeps the policy’s value. Raises ProveviError if the section would widen the policy in any way.

Parameter Type Default Description
section Any required A run file’s parsed telemetry section, with libraries, destinations and hosts; None keeps the policy
source str 'the run file' Where the section came from, for messages and the new source
returns TelemetryPolicy The narrowed policy

load_yaml

source

load_yaml(text: str) -> Any

Parse YAML safely, and refuse a key that a mapping names twice.

Plain YAML loaders allow such a key, and keep the last value.

Parameter Type Default Description
text str required A YAML or JSON document
returns Any The parsed document

normal_endpoint

source

normal_endpoint(url: str) -> str

Write a URL in one regular form, so that two ways of writing one address are equal.

The scheme and the host are lowercased, and the default port is added. A trailing / is dropped, except from a sqlite path. So are a database driver such as +psycopg, the user and password of a database URL, and the allowed settings of its query but sslmode, which stays, so that a policy can ask for TLS. A sqlite URL keeps only its path.

Raises ProveviError for anything that two URL parsers could read as two different places: a backslash, a space, a control character or a #; a host that is not ASCII, because two standards of international names read some hosts differently; a user in an http or https URL; a % in the host; a port of 0 or an empty port; a path with . or .. segments or //; a query, which can name another server, as ?host= does, except the settings in _SAFE_QUERY, such as sslmode=require; a host list; a relative sqlite path; an mssql URL with no port or no database, which ODBC can read as the name of a data source. Also for a URL that is not http, https or a database.

Parameter Type Default Description
url str required An http, https or database URL, such as a destination’s endpoint
returns str The URL in its regular form, such as https://host:443/path

otlp_traces_target

source

otlp_traces_target(environ: Mapping[str, str]) -> tuple[str, str] | None

Find the protocol and the address to which provrun’s OTLP exporter sends traces.

It reads the variables in environ as the OpenTelemetry specification gives them. With neither endpoint variable set, provrun makes no OTLP exporter.

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT wins, and is used as it is. Otherwise OTEL_EXPORTER_OTLP_ENDPOINT is used, with /v1/traces added over HTTP but not over gRPC. A variable that is set but empty counts as not set. The protocol comes from OTEL_EXPORTER_OTLP_TRACES_PROTOCOL, then OTEL_EXPORTER_OTLP_PROTOCOL, then OTEL_TRACES_EXPORTER (otlp_proto_grpc or otlp_proto_http), and is http/protobuf, the specification’s default, otherwise. provrun must make the exporter of that protocol. A gRPC endpoint written as host:port gets https://, or http:// when OTEL_EXPORTER_OTLP_TRACES_INSECURE or OTEL_EXPORTER_OTLP_INSECURE is true. Raises ProveviError for a protocol that is not grpc, http/protobuf or http/json.

Parameter Type Default Description
environ Mapping[str, str] required The environment variables, such as os.environ
returns tuple[str, str] | None The protocol and the address; None when neither endpoint variable is set