Hooks for provevi in PydanticAI, LiteLLM and OpenTelemetry

Turning on instrumentation

In 2.x the Agent constructor has no instrument argument. The constructor sets self._instrument = None. Three ways turn spans on:

• Agent.instrument_all(settings) sets the default for all agents.
• agent.instrument = settings sets it for one agent.
• capabilities=[Instrumentation(settings=...)] adds the capability explicitly.

An InstrumentedModel passed as the model also works. The run unwraps it and adds the capability. See pydantic_ai/agent/__init__.py, near the comment “Resolve instrumentation”.

InstrumentationSettings

The fields, from pydantic_ai/models/instrumented.py:

event_mode does not exist in 2.64.0. A search of the source for event_mode finds nothing. Messages go on span attributes only.

The format versions:

• Version 2 puts gen_ai.system_instructions, gen_ai.input.messages and gen_ai.output.messages on model request spans, and pydantic_ai.all_messages on the run span.
• Version 3 renames the spans to invoke_agent {name} and execute_tool {name}. It renames the tool attributes to gen_ai.tool.call.arguments and gen_ai.tool.call.result, and it adds thinking parts.
• Version 4 uses the uri and blob part types for media.
• Version 5, the default, leaves the span status unset for a deferred tool call.
• Version 6 sends tool results in a message with role='tool'.

The docs say that PydanticAI follows the GenAI conventions at version 1.37.0, and that they are still experimental. Source: https://pydantic.dev/docs/ai/integrations/logfire/.

Spans that PydanticAI emits

The run span sets the baggage keys gen_ai.agent.name, gen_ai.agent.call.id and gen_ai.conversation.id for the spans under it. Model request spans copy them as attributes. The key gen_ai.agent.call.id is a PydanticAI name. The current GenAI registry does not have it. Source: pydantic_ai/_instrumentation.py and pydantic_ai/capabilities/instrumentation.py.

Lifecycle hooks (capabilities)

A capability is a subclass of AbstractCapability, passed in Agent(capabilities=[...]) or in agent.run(..., capabilities=[...]). The Hooks capability registers plain functions with decorators, for example @hooks.on.before_tool_execute. Docs: https://pydantic.dev/docs/ai/capabilities/overview/ and https://pydantic.dev/docs/ai/core-concepts/hooks/.

The facts below affect provevi:

• wrap_model_request covers streamed requests. The docstring of after_model_request names streamed responses. A WrapperModel that overrides only request misses request_stream.
• before_tool_validate sees a call before PydanticAI checks its arguments. So the hook sees a tool call with bad arguments. call_tool on a toolset never sees such a call.
• before_tool_execute is the place for a policy gate. Raise ApprovalRequired there to defer the call before the tool runs. Raise SkipToolExecution(result) to give the model a result without running the tool.
• get_wrapper_toolset receives the combined, non-output toolset of the run. It includes tools that other capabilities add. Output tools are added separately and are not included.
• prepare_model_request runs once for each attempt, with the model that serves the attempt. So a FallbackModel or a retry gives one call for each attempt.

WrapperModel and WrapperToolset

WrapperModel (pydantic_ai/models/wrapper.py) delegates every method of Model. Besides request, it delegates request_stream, count_tokens, compact_messages, the batch methods and the methods for suspended responses. A wrapper that overrides only request does not see the other model calls.

WrapperToolset (pydantic_ai/toolsets/wrapper.py) delegates get_tools, call_tool, get_instructions, for_run, for_run_step and the context manager methods. call_tool(name, tool_args, ctx, tool) gets the arguments after validation.

Event stream

agent.run_stream(...), agent.iter(...) and event_stream_handler= give AgentStreamEvent values. The types are in pydantic_ai/messages.py:

• Model stream: PartStartEvent, PartDeltaEvent, PartEndEvent, FinalResultEvent.
• Tool handling: FunctionToolCallEvent, FunctionToolResultEvent, OutputToolCallEvent, OutputToolResultEvent, DeferredToolRequestsEvent, DeferredToolResultsEvent.
• Run changes: ToolAvailabilityDeltaEvent, CapabilityLoadDeltaEvent, ModelSettingsDeltaEvent, EnqueuedMessagesEvent.
• Other: CustomEvent, CapabilityEvent, and the realtime session events.

The stream is good for live display. It is a poor evidence source, because a consumer can stop reading it.

Tool calls, results, retries and usage

• Tool calls are ToolCallPart values in a ModelResponse. Each has tool_name, args and tool_call_id. Calls that the provider runs, for example a hosted web search, are NativeToolCallPart and NativeToolReturnPart. No toolset sees them.
• Tool results are ToolReturnPart values in the next ModelRequest.
• Retries are RetryPromptPart and RetryFeedbackPart. Tool spans carry pydantic_ai.tool.retry.cause, pydantic_ai.tool.retry.attempt and pydantic_ai.tool.retry.max_retries.
• Usage per request is ModelResponse.usage, a RequestUsage. Its fields include input_tokens, output_tokens, cache_read_tokens, cache_write_tokens, output_reasoning_tokens and cost.
• Usage per run is result.usage, a RunUsage. It adds requests and tool_calls.

Correlation with a provrun run

agent.run(..., run_id=...) takes a caller’s run ID. The docstring says an empty string, or an ID that is already in message_history, raises UserError. The run ID goes into the run span as gen_ai.agent.call.id, into baggage, and onto each message as run_id. The invoke_agent span opens under the current OTel context, so it nests under provrun’s own span automatically.

The supply chain incidents

• March 2026. litellm 1.82.7 and 1.82.8 were malicious. They were live on PyPI on 2026-03-24 from 10:39 UTC for about 40 minutes. Version 1.82.8 added a litellm_init.pth file that stole credentials. LiteLLM says the likely cause was the Trivy scanner in its CI. Both versions are gone from PyPI. Version 1.83.0, from a rebuilt pipeline, came on 2026-03-30. Sources: https://docs.litellm.ai/blog/security-update-march-2026, https://www.bitsight.com/blog/litellm-versions-1-82-7-1-82-8-supply-chain-compromise.
• October 2026. LiteLLM’s own pipeline built, signed and published a malicious litellm 1.107.0 on 2026-10-21. The pipeline used Trusted Publishing and cosign, and those checks did not stop it. LiteLLM says all earlier versions are safe. Sources: https://docs.litellm.ai/blog/security-incident-oct-21-2026, https://docs.litellm.ai/blog/security-changes-after-oct-21-2026. Reports from others: https://bleepingcomputer.com/news/security/litellm-release-pipeline-hacked-to-distribute-info-stealer-malware, https://cloudsmith.com/blog/litellm-1-107-0-compromised-on-pypi.md. The release is gone from PyPI, so the latest release that PyPI lists is 1.106.6.
• Proxy CVEs that affect 1.82.6. CVE-2026-35029, CVE-2026-35030 and GHSA-69x8-hrgq-fjj8 are fixed in 1.83.0. CVE-2026-42208, an SQL injection, affects 1.81.16 to 1.83.6. CVE-2026-30623 is fixed in 1.83.7-stable. CVE-2026-48710, a host header bypass, is fixed in 1.84.0. All of them are in the proxy server, not in the SDK call path that provrun uses. Source: https://docs.litellm.ai/blog/tags/security.
• Provenance. PyPI has no provenance attestation for litellm-1.82.6-py3-none-any.whl. It has one for 1.83.0. The first wheel of 1.106.6 also has none. The check used https://pypi.org/integrity/litellm/<version>/<file>/provenance.
• Packaging change. Version 1.106.6 ships compiled cp310-abi3 wheels for each platform, where 1.82.6 was a pure Python wheel.

The upper bound <=1.82.6 keeps out both bad releases. It also keeps out every fix since March.

CustomLogger hooks in 1.82.6

litellm/integrations/custom_logger.py defines the class. Register an instance in litellm.callbacks.

async_post_call_failure_deployment_hook is in the current docs, but not in 1.82.6. The docs are at https://docs.litellm.ai/docs/observability/custom_callback.

The success path is best-effort. litellm/litellm_core_utils/logging_worker.py describes the worker as “best-effort with bounded queues to prevent backpressure”. The queue holds 50,000 items, from LOGGING_WORKER_MAX_QUEUE_SIZE. Each coroutine has 20 seconds, from LOGGING_WORKER_MAX_TIME_PER_COROUTINE. A flush runs at exit through atexit. Also, a request with "no-log": True skips logging, unless litellm.global_disable_no_log_param is True.

StandardLoggingPayload

The success and failure hooks get kwargs["standard_logging_object"]. The table lists its fields in 1.82.6, from litellm/types/utils.py.

trace_id comes from litellm_session_id, else from litellm_trace_id, else from a new UUID. The function is StandardLoggingPayloadSetup._get_standard_logging_payload_trace_id.

The “otel” callback

Turn it on with litellm.callbacks = ["otel"]. The class is litellm/integrations/opentelemetry.py.

In 1.82.6:

• Parent. The span parent comes, in order, from metadata["litellm_parent_otel_span"], from an inbound traceparent header, or from the active span.
• Span shape. With no parent, it makes a litellm_request span and a raw_gen_ai_request child. A metadata["generation_name"] value replaces the name. With a parent, it makes no span of its own. It sets attributes on the parent and sets the parent status to OK. USE_OTEL_LITELLM_REQUEST_SPAN=true forces its own span.
• Attributes. It mixes the OpenLLMetry names (llm.request.type, llm.is_streaming and so on) with gen_ai.system, gen_ai.request.model, gen_ai.usage.*_tokens, gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions and gen_ai.framework=litellm.
• Content. litellm.turn_off_message_logging or message_logging=False removes message content. LITELLM_OTEL_INTEGRATION_ENABLE_EVENTS turns on log events.

The current docs add three things that 1.82.6 lacks. The first is OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental, which names the span {operation} {model} with kind CLIENT. The second is OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT. The third is a new integration behind LITELLM_OTEL_V2=true. A search of the 1.82.6 source finds none of these names. Source: https://docs.litellm.ai/docs/observability/opentelemetry_integration.

MLflow, Langfuse and the proxy

• MLflow. litellm.callbacks = ["mlflow"] loads MlflowLogger, a CustomLogger. It writes MLflow trace spans from the success and failure hooks, so it has the same best-effort delivery. Docs: https://docs.litellm.ai/docs/observability/mlflow.
• Langfuse. 1.82.6 has the Langfuse SDK logger (langfuse.py) and an OTel exporter (langfuse_otel.py). Docs: https://docs.litellm.ai/docs/observability/langfuse_otel_integration.
• Proxy. The proxy reads callbacks from litellm_settings.callbacks, success_callback and failure_callback in config.yaml. It reads a traceparent header into its spans. Each response has an x-litellm-call-id header. A client can send "no-log": true. Docs: https://docs.litellm.ai/docs/proxy/logging.

Where they live and their status

The GenAI conventions moved to https://github.com/open-telemetry/semantic-conventions-genai. The page https://opentelemetry.io/docs/specs/semconv/gen-ai/ says they are “no longer maintained in this repository”.

• Status. Every GenAI document says “Status: Development”.
• Releases. The new repository was created on 2026-05-05. On 2026-10-31 it has no release and no tag. Its changelog has only an “Unreleased” heading, and its README says “Schema URL: TODO”.
• Base version. The documents link to core semantic conventions v1.44.0, released 2026-08-04.
• Python package. In opentelemetry-semantic-conventions 0.66b2, every GEN_AI_* constant is marked “Deprecated: Moved to the OpenTelemetry GenAI semantic conventions repository”.

Span names

The documents also define spans for embeddings, fetch response, memory, agent skills and command execution. A span for an automatic retry covers the whole logical operation with all its retries.

Key attributes

• Operation: gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, gen_ai.response.id, gen_ai.response.finish_reasons.
• Agent: gen_ai.agent.id, gen_ai.agent.name, gen_ai.agent.version, gen_ai.conversation.id, gen_ai.workflow.name.
• Tool: gen_ai.tool.name, gen_ai.tool.call.id, gen_ai.tool.type, gen_ai.tool.call.arguments, gen_ai.tool.call.result, gen_ai.tool.definitions.
• Usage: gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_write.input_tokens, gen_ai.usage.reasoning.output_tokens.
• Content: gen_ai.system_instructions, gen_ai.input.messages, gen_ai.output.messages.
• Content references: gen_ai.system_instructions_ref, gen_ai.input.messages_ref, gen_ai.output.messages_ref, gen_ai.tool.definitions_ref.

gen_ai.system is gone from the current registry, and gen_ai.provider.name replaces it. The names gen_ai.usage.prompt_tokens and gen_ai.usage.completion_tokens are older forms.

How content is recorded

The rules are in docs/gen-ai/gen-ai-spans.md, section “Capturing instructions, inputs, and outputs”.

• Default. Instrumentations do not record content.
• On attributes. Content goes in gen_ai.system_instructions, gen_ai.input.messages and gen_ai.output.messages. A span holds them as JSON strings. An event holds them in structured form. JSON schemas define the message shape.
• On events. Inference content goes in the event gen_ai.client.inference.operation.details. The older events with one message per event are gone.
• The switch. The documents name the variable OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT, with the values NO_CONTENT, SPAN_ONLY, EVENT_ONLY and SPAN_AND_EVENT. But opentelemetry-util-genai 1.3b0 reads OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT.
• External storage. An instrumentation can call a content hook with the messages before they become JSON. The hook can store them elsewhere and put a *_ref attribute on the span. The documents leave the URI scheme open, and they allow content-derived IDs.

The external storage pattern matches provrun’s payload store, which names each payload by its SHA-256.

The context propagation rule changed on 2026-10-30. Instrumentations may pass gen_ai.conversation.id to nested instrumentations in the same process. They should not pass other attributes. Source: docs/gen-ai/README.md and pull request 556 in the same repository.

Instrumentation packages

The GenAI instrumentations moved from opentelemetry-python-contrib to https://github.com/open-telemetry/opentelemetry-python-genai. The copies in contrib get security fixes only.

opentelemetry-instrumentation-anthropic and opentelemetry-instrumentation-openai 0.62.4 belong to Traceloop OpenLLMetry, https://github.com/traceloop/openllmetry. They are not OTel projects.

opentelemetry-util-genai defines CompletionHook.on_completion(inputs, outputs, system_instruction, tool_definitions, span, log_record). It loads the hook from the entry point group opentelemetry_genai_completion_hook, named by OTEL_INSTRUMENTATION_GENAI_COMPLETION_HOOK. The wrapper _SafeCompletionHook logs and drops any exception, so the hook observes only.

Anthropic Python SDK 2.3.0

• Built-in OTel. The client makes one CLIENT span for each API call, named for example anthropic.messages.create, with gen_ai.operation.name=chat. It needs only opentelemetry-api. Spans are on by default when a tracer provider is registered.
• Options. The client takes open_telemetry={"tracer_provider", "propagation", "traces", "conversation_id"}. traces holds enabled, content_mode (metadata_only or content) and max_content_bytes (default 750,000). The environment variables start with ANTHROPIC_OPEN_TELEMETRY.
• Propagation. The SDK sends the trace context of the call span on each attempt, unless propagation is False.
• Tool runner. messages.tool_runner makes an anthropic.messages.tool_runner span with invoke_agent, and one execute_tool span for each tool.
• Middleware. Anthropic(middleware=[...]) or client.with_middleware(...) takes Middleware.handle and handle_async. The chain runs once for each HTTP attempt, inside the retry loop. A middleware can return without calling call_next, or raise, so it can block.

Source: anthropic/_tracing.py, anthropic/_middleware.py, anthropic/lib/_tracing/, and the README in the wheel metadata.

OpenAI Python SDK 3.42.0

• No built-in OTel. A search for opentelemetry and middleware in the package finds nothing.
• HTTP client. The SDK now uses httpx2 from Pydantic, https://github.com/pydantic/httpx2. A custom http_client with a transport or event hooks is the only hook in the SDK.
• Instrumentation. opentelemetry-instrumentation-genai-openai makes the spans. It observes only.

OpenAI Agents SDK 0.24.0

• Tracing processors. add_trace_processor(p) adds a processor, and set_trace_processors([...]) replaces the default exporter to OpenAI. A TracingProcessor has on_trace_start, on_trace_end, on_span_start, on_span_end, shutdown and force_flush. The provider catches each exception from a processor, so processors observe only.
• Span data. The types are AgentSpanData, TaskSpanData, TurnSpanData, FunctionSpanData, GenerationSpanData, ResponseSpanData, HandoffSpanData, GuardrailSpanData, MCPListToolsSpanData and the voice types.
• Sensitive data. RunConfig.trace_include_sensitive_data defaults to True, from OPENAI_AGENTS_TRACE_INCLUDE_SENSITIVE_DATA. RunConfig also takes trace_id, group_id and trace_metadata.
• Blocking. Tool input guardrails return ToolGuardrailFunctionOutput.reject_content(...) or raise_exception, so they can block. RunHooks and AgentHooks (on_llm_start, on_tool_start and others) are awaited in the run.
• OTel. The SDK traces are not OTel spans. opentelemetry-instrumentation-genai-openai-agents turns them into OTel spans through a processor.

Docs: https://openai.github.io/openai-agents-python/tracing/. Source: agents/tracing/, agents/tool_guardrails.py, agents/lifecycle.py, agents/run_config.py.