Library telemetry in provrun runs

The callback lists

LiteLLM keeps its callbacks in module globals. All of them are empty after import, as the test t1_litellm_import.py printed:

The list _custom_logger_compatible_callbacks_literal at litellm/__init__.py:102 names 48 built-in integrations. Among them are “otel”, “logfire”, “mlflow”, “langfuse”, “langfuse_otel”, “langsmith”, “datadog”, “arize”, “braintrust”, “posthog”, “s3_v2” and “gcs_bucket”. Each string turns into a logger object in _init_custom_logger_compatible_class at litellm/litellm_core_utils/litellm_logging.py:3654.

Ways the callbacks get switched on

• Assignment or append. Any code in the process can assign or append to the globals above.
• The per-call callbacks= argument. function_setup at litellm/utils.py:786 pops it and appends each entry to input_callback, success_callback, failure_callback and the two async lists. The test t2_litellm_calls.py passed a callback on one call. The callback then saw the next call, which had no callbacks= argument.
• The per-call success_callback= and failure_callback= arguments. litellm/utils.py:899 to litellm/utils.py:923 use them for that call only.
• The per-call logger_fn= argument. litellm/utils.py:943 stores it in a module global, user_logger_fn.
• Named integrations from environment variables. No environment variable adds a callback by itself. But a named integration reads its settings from the environment when it starts. For example, “logfire” needs LOGFIRE_TOKEN (litellm/litellm_core_utils/litellm_logging.py:3994), and “otel” reads OTEL_EXPORTER, OTEL_ENDPOINT, OTEL_HEADERS and the OTEL_EXPORTER_OTLP_* variables (litellm/integrations/opentelemetry.py:112 to litellm/integrations/opentelemetry.py:117).
• The proxy configuration file. A LiteLLM proxy config.yaml sets litellm_settings.callbacks. provrun does not run the proxy, so this path applies only if provrun calls the proxy code.
• Other libraries. mlflow.litellm.autolog() adds an MLflow callback. The page on MLflow autolog covers it.

The “otel” integration and the global tracer provider

The “otel” integration decides its provider in _get_or_create_provider at litellm/integrations/opentelemetry.py:248. If the global provider is an SDK TracerProvider, it uses that provider and adds no exporter. Otherwise it makes a new provider with its own exporter and sets it as global (litellm/integrations/opentelemetry.py:296 to litellm/integrations/opentelemetry.py:319). The exporter defaults to the console exporter.

The test t12_litellm_otel.py showed both cases:

• No provider set. LiteLLM set its own provider as global. The console exporter wrote two spans to stdout, and both held the prompt. provrun prints its JSON on stdout, so this output breaks that JSON.
• provrun’s provider set first. The spans litellm_request and raw_gen_ai_request went to provrun’s provider, with the prompt in their attributes.

provrun already sets USE_OTEL_LITELLM_REQUEST_SPAN=true, so these loggers make their own span and do not write onto provrun’s step span (packages/provrun/src/provrun/backends/litellm.py:94).

Network access by LiteLLM itself

• The cost map fetch at import. litellm/__init__.py:457 calls get_model_cost_map. That function does an httpx.get of the URL in LITELLM_MODEL_COST_MAP_URL, which defaults to raw.githubusercontent.com (litellm/litellm_core_utils/get_model_cost_map.py:153). The test t1_litellm_import.py caught the lookup of raw.githubusercontent.com port 443. With LITELLM_LOCAL_MODEL_COST_MAP=True, the import made no lookup and the source was “local” (litellm/litellm_core_utils/get_model_cost_map.py:258).
• The Anthropic beta headers fetch. litellm/anthropic_beta_headers_manager.py:122 fetches a configuration file the first time LiteLLM needs it. LITELLM_LOCAL_ANTHROPIC_BETA_HEADERS=True stops the fetch (litellm/anthropic_beta_headers_manager.py:142). A mock Anthropic call in t2_litellm_calls.py made no fetch, so I did not see when the fetch happens.
• The blog posts fetch. litellm/litellm_core_utils/get_blog_posts.py:66 fetches an RSS feed. Only the proxy calls it.
• Hugging Face tokenizers. The token counter can download a tokenizer for some models. litellm.disable_hf_tokenizer_download = True stops it (litellm/__init__.py:1315, litellm/utils.py:2174). The tiktoken files come with the package (litellm/litellm_core_utils/default_encoding.py:18).
• The telemetry flag. litellm.telemetry = True at litellm/__init__.py:214 is not read anywhere in the package. LiteLLM 1.83.0 sends no usage pings.

The .env file

litellm/__init__.py:18 calls dotenv.load_dotenv() when LITELLM_MODE is “DEV”, which is the default. If the main program is a script file, python-dotenv searches upward from the folder of the caller, which is the litellm package (dotenv/main.py:366 to dotenv/main.py:377). If the main program is interactive or python -c, it searches from the current folder.

The test used a virtual environment in dotproj/ with a .env file beside .venv. A script in another folder imported litellm, and the variable from that .env file appeared in os.environ. With LITELLM_MODE=PRODUCTION, it did not appear. The variables do not override variables that are set already.

Message redaction

litellm.turn_off_message_logging = True replaced the messages with “redacted-by-litellm” in both kwargs["messages"] and the standard logging object that a callback receives (t3_litellm_debuglog.py). It is off by default (litellm/__init__.py:194). The response text was still present in the standard logging object in the test output, so I did not confirm that the response is redacted.

How to clear and guard the callbacks

litellm.logging_callback_manager._reset_all_callbacks() at litellm/litellm_core_utils/logging_callback_manager.py:359 clears six lists. It does not clear _async_input_callback, service_callback or audit_log_callbacks, so provrun must clear those itself. After the reset, the callback from the earlier test saw no more calls.

To guard the lists, provrun can record every list before the call and compare after the call. Any entry that is not on the approved list is a failure.

Ways instrumentation gets switched on

PydanticAI makes no spans unless one of these paths is on. A default agent made 0 spans, even with a global SDK provider set (t5_pydanticai.py).

• Agent.instrument_all(settings_or_bool). It sets the class variable Agent._instrument_default for every agent (pydantic_ai/agent/__init__.py:1330). The default is False (pydantic_ai/agent/__init__.py:585).
• agent.instrument = settings_or_bool. This property sets one agent. The 2.64.0 constructor has no instrument= parameter.
• The Instrumentation capability. Agent(capabilities=[Instrumentation(settings)]) turns on spans for that agent (pydantic_ai/capabilities/instrumentation.py:113).
• An InstrumentedModel. A model wrapped in InstrumentedModel turns on spans, and its settings win over the agent settings (pydantic_ai/agent/__init__.py:1929 to pydantic_ai/agent/__init__.py:1942).
• logfire.instrument_pydantic_ai(). It calls Agent.instrument_all with no argument, sets agent.instrument for an agent, or returns an InstrumentedModel for a model (logfire/_internal/integrations/pydantic_ai.py:59 to logfire/_internal/integrations/pydantic_ai.py:63).

Agent.instrument_all(False) and agent.instrument = False do not stop an explicit Instrumentation capability. The test made 4 spans in both cases.

Which tracer provider it uses

InstrumentationSettings() takes tracer_provider and meter_provider arguments (pydantic_ai/models/instrumented.py:102). Without them, it reads the global providers when the settings object is made (pydantic_ai/models/instrumented.py:167). Before any global provider is set, the global provider is a proxy, and the tracer follows the provider that is set later.

logfire.instrument_pydantic_ai() passes Logfire’s own providers explicitly (logfire/_internal/integrations/pydantic_ai.py:30). So its spans go to Logfire’s provider, even when provrun owns the global provider.

PydanticAI also records a metric, gen_ai.client.token.usage, through the meter provider.

Spans and attributes

The test agent used TestModel and one tool. It made 4 spans, all with the instrumentation scope “pydantic-ai”:

With include_content=True, which is the default, the spans held the prompt, the instructions and the tool result. With include_content=False, they held none of these markers. But the message attributes stayed, with roles and part types only. gen_ai.tool.definitions and model_request_parameters still held the tool description and the JSON schema (t5b.py).

include_binary_content defaults to True, so images and files go into the spans as data unless provrun turns it off.

Baggage

The Instrumentation capability puts gen_ai.agent.name, gen_ai.agent.call.id and the conversation ID into OpenTelemetry baggage (pydantic_ai/capabilities/instrumentation.py:311). Baggage is context that a propagator copies into outgoing request headers.

The test t9_sdks.py ran an agent with an OpenAIChatModel against the local server, under opentelemetry-instrument. The request carried a traceparent header and a baggage header. The baggage header was gen_ai.agent.name=SECRET-AGENT-NAME,gen_ai.agent.call.id=...,gen_ai.conversation.id=.... Without the httpx instrumentor, the request had neither header.

How to force it off or point it at a provider provrun owns

• Off. Call Agent.instrument_all(False) and set agent.instrument = False. Then remove any Instrumentation capability, and unwrap any InstrumentedModel.
• Detect at run time. provrun’s evidence capability receives the run context in for_run. ctx.tracer is a NoOpTracer when the run has no instrumentation (pydantic_ai/agent/__init__.py:5277 to pydantic_ai/agent/__init__.py:5281). The test t13_detect_pa.py saw NoOpTracer for a plain agent. It saw ProxyTracer for an agent with Instrumentation() and after instrument_all(True).
• Approved. Add Instrumentation(InstrumentationSettings(tracer_provider=provrun_tp, meter_provider=provrun_mp, include_content=..., include_binary_content=False)). The test sent all 4 spans to the private provider and none to the global provider.

The global tracer provider

set_tracer_provider works once. A second call logs the warning “Overriding of current TracerProvider is not allowed” and changes nothing (opentelemetry/trace/__init__.py:544 to opentelemetry/trace/__init__.py:552). The test t6_logfire.py showed the warning when Logfire tried to set the provider after provrun.

get_tracer_provider loads a provider from OTEL_PYTHON_TRACER_PROVIDER if no provider is set and the variable is set (opentelemetry/trace/__init__.py:564 to opentelemetry/trace/__init__.py:573). With OTEL_PYTHON_TRACER_PROVIDER=sdk_tracer_provider, the global provider was an SDK TracerProvider before any code set it (t8_otel.py).

Anyone with a reference to a provider can add a span processor to it. MLflow did so to provrun’s provider (see the section on MLflow).

Environment variables that switch on export or instrumentation

opentelemetry-instrument and sitecustomize

The workspace has the opentelemetry-instrument command, because logfire-sdk[httpx] pulls in opentelemetry-instrumentation-httpx. The command puts the auto-instrumentation folder first on PYTHONPATH, so Python runs its sitecustomize.py. That file calls initialize(), which loads every installed instrumentor (opentelemetry/instrumentation/auto_instrumentation/__init__.py:83 to opentelemetry/instrumentation/auto_instrumentation/__init__.py:105).

The workspace has no distro, so no configurator runs. Under the command, httpx was instrumented, but the global provider stayed a proxy and OTEL_TRACES_EXPORTER=console printed nothing (t8_otel.py). If someone installs opentelemetry-distro, the command also sets a global provider with an OTLP exporter by default. OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=httpx,httpx2 left httpx not instrumented.

How provrun can own the provider

• Set its provider first. Call trace.set_tracer_provider(provrun_tp) before any library import that can set one. Then check trace.get_tracer_provider() is provrun_tp. If the check fails, another component set the provider first.
• Make the provider fixed. Pass an explicit sampler, an explicit Resource(...) and explicit SpanLimits, so that the OTEL_* variables do not change provrun’s spans.
• Refuse other processors. A subclass of TracerProvider can refuse add_span_processor after provrun has added its own. The test t14_guarded_provider.py refused MLflow’s MlflowV3SpanProcessor and recorded the attempt.
• Use a private provider. provrun can keep its provider private and pass it explicitly to PydanticAI. But opentelemetry/trace.get_tracer("provrun") in packages/provrun/src/provrun/otel.py reads the global provider, so that module must take the private provider too. A private provider does not stop other libraries from setting the global one.
• Own the meter and logger providers too. Logfire sets all three globals, and PydanticAI records a metric.
• Remove OTEL_SDK_DISABLED for provrun’s provider. The variable stops provrun’s own spans, so provrun should not use it as the off switch. If it is set, provrun should warn.

What logfire.configure does

• Global providers. For the default instance, configure() sets the global tracer, meter and logger providers once (logfire/_internal/config.py:1517 to logfire/_internal/config.py:1521). configure(local=True) does not set them. The test t6_logfire.py showed both cases.
• Global propagator. Both local=True and local=False set the global text map propagator to a wrapper, WarnOnExtractTraceContextPropagator (logfire/_internal/config.py:1546 to logfire/_internal/config.py:1555).
• Executors. Both forms call instrument_executors(), which patches ThreadPoolExecutor and ProcessPoolExecutor to carry context (logfire/_internal/config.py:1544).
• send_to_logfire. The default comes from LOGFIRE_SEND_TO_LOGFIRE, then from pyproject.toml [tool.logfire]. Otherwise it is True, or False under pytest (logfire/_internal/config_params.py:54). With the default and no token, configure() raised LogfireConfigError and asked the user to run logfire auth. With if-token-present, it sent nothing.
• Token. LOGFIRE_TOKEN gives the write token (logfire/_internal/config_params.py:62). A credentials file in .logfire/ also gives one. The exporter sends spans to {base_url}/v1/traces with the token in the Authorization header (logfire/_internal/config.py:1333 to logfire/_internal/config.py:1337). When a token is present, a thread also checks it against the Logfire API.
• OTLP from the environment. If OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is set, Logfire adds an OTLP exporter, whatever send_to_logfire says (logfire/_internal/config.py:1427 to logfire/_internal/config.py:1436). With send_to_logfire=False and the variable set to an example host, the net guard caught repeated lookups of that host.
• Console. The console exporter is on by default (logfire/_internal/config_params.py:78). console=False or LOGFIRE_CONSOLE=false turns it off.
• Additional processors. additional_span_processors=[...] adds processors to Logfire’s provider. The test used it to capture Logfire’s spans in memory.

Other Logfire paths

• The Pydantic plugin. logfire-sdk registers a Pydantic plugin through an entry point. Pydantic loads it for every model. It records validations only if LOGFIRE_PYDANTIC_PLUGIN_RECORD or [tool.logfire] sets the record mode, and the default is “off” (logfire/_internal/config_params.py:101). PYDANTIC_DISABLE_PLUGINS=logfire-plugin stops Pydantic from loading it.
• The pytest plugin. logfire-sdk registers a pytest plugin, logfire._internal.integrations.pytest. It runs only in test runs.
• logfire.instrument_httpx() and other instrument calls. They patch libraries in the same way as the OpenTelemetry instrumentors, and httpx then sends baggage headers.

Coexistence with MLflow tracing

MLFLOW_USE_DEFAULT_TRACER_PROVIDER defaults to True. Its name says the opposite of what it does. With True, MLflow uses an isolated provider of its own (mlflow/environment_variables.py:1055 to mlflow/environment_variables.py:1061). With False, MLflow uses the global provider and adds its processor to it.

The test t11_mlflow_provider.py set provrun’s provider first and then made an MLflow span with an OpenTelemetry child span:

So with false, MLflow exports every span of provrun’s provider, which includes PydanticAI and LiteLLM spans. Logfire and MLflow can both take the global provider. Logfire with local=True and MLflow in its default isolated mode do not compete for it.

An approved export to Logfire under provrun’s provider

There are two forms. I did not test either against the real Logfire service.

1. No Logfire SDK at all. provrun adds BatchSpanProcessor(OTLPSpanExporter(endpoint=f"{base_url}/v1/traces", headers={"Authorization": token})) to its own provider. LiteLLM’s “logfire” integration and Logfire itself export in the same way. provrun keeps the provider and the processor list, and Logfire is one approved destination.
2. A local Logfire instance. provrun calls logfire.configure(local=True, send_to_logfire=True, token=..., console=False) and passes that instance’s provider to InstrumentationSettings. This form leaves the global tracer provider alone. But it still changes the global propagator and patches the executors, and it still reads OTEL_EXPORTER_OTLP_*.

Form 1 gives provrun the most control. Form 2 adds Logfire’s scrubbing of secrets and its token check.