What MLflow autolog records and how to keep it out of a run
Research date: 2026-11-01. The versions are MLflow 3.17.2, pydantic-ai 2.64.0, litellm 1.83.0, openai 3.41.0, anthropic 2.3.0 and opentelemetry-sdk 1.45.1.
These notes follow MLflow tracing, which describes the trace data model, the SQLite schema and OTLP ingest. These notes do not repeat that material.
The facts come from two sources:
- Source. Citations such as
pydantic_ai/autolog_v2.py:85are relative to the MLflow 3.17.2 source tree. Citations that start withlitellm/point into the litellm 1.83.0 package. - Tests. The scripts were run in a scratch folder, and they are not in the repository. Each script runs with
./run.sh SCRIPT, which builds a throwaway environment with the versions above. The scripts call no model and use no key. PydanticAI usesFunctionModelorTestModel, LiteLLM usesmock_response, and the OpenAI and Anthropic clients talk to an httpxMockTransportinstubs.py.
Terms used on this page:
- Flavor. A flavor is an MLflow integration module for one library, for example
mlflow.openai. Each flavor has anautolog()function. - Autolog. Autolog is MLflow’s switch that makes a library write traces with no change to the calling code.
- Patch. A patch replaces a method on a class, so that MLflow code runs around the original method. MLflow keeps the original method on the class under another name, so that it can put it back.
- Gorilla. Gorilla is the small patching module in
utils/gorilla.py. It marks each patched attribute. - Tracer provider. A tracer provider is the OpenTelemetry (OTel) object that makes spans and sends them to span processors. A process has one global tracer provider. MLflow keeps a second, private one by default.
- Callback. A LiteLLM callback is a logger object or a name in one of the callback lists in the
litellmmodule. LiteLLM calls it after each model call. - Span type. The span type is MLflow’s kind of span, for example
LLM,CHAT_MODEL,TOOLorAGENT. It is in the attributemlflow.spanType.
Summary
- pydantic-ai 2.64.0 uses
autolog_v2. It patchesAgent, theInstrumentationcapability andMCPToolset. It also patchesAgent.__init__so that each new agent getsinstrument=True. With that setting, PydanticAI also writes its own OTel spans to the global tracer provider. - LiteLLM autolog is a callback. It adds
"mlflow"tolitellm.success_callbackandfailure_callback. LiteLLM then copies the logger into_async_success_callbackand_async_failure_callback. - Disable leaves a LiteLLM leak.
mlflow.litellm.autolog(disable=True)andmlflow.autolog(disable=True)do not clean the two async lists. After a disable, eachlitellm.acompletioncall still wrote a span. provrun’s LiteLLM backend usesacompletion. - The OpenAI and Anthropic flavors change the request. They add a W3C
traceparentheader to every call to the provider. The stub servers received the header on every call. - The generic switch does not undo a flavor switch. After
mlflow.openai.autolog(), a call tomlflow.autolog(disable=True)left the OpenAI patches on, and spans were still written. mlflow.tracing.disable()is the strongest single switch. It stopped every span and every header. Butmlflow.set_tracking_uri()with a new URI turns tracing on again.- No environment variable turns autolog on. Code must call
autolog(). Other code can do it for us. For example,mlflow.genai.evaluate()turns tracing on for every flavor that is not disabled by name. - Storage adds attributes. On write, MLflow adds
mlflow.spanType, inputs, outputs, token usage, tools, model, provider and cost from GenAI semantic convention attributes. It also fills the token, cost and session columns of the trace.
1. How autolog switches on
The common machinery
Every flavor goes through the same steps:
- Decorator.
@autologging_integration(name)wraps eachautolog()(utils/autologging_utils/__init__.py:385). It stores the call’s arguments in the global dictAUTOLOGGING_INTEGRATIONS[name](line 420). - Revert first. The wrapper calls
revert_patches(name)on every call, before anything else (line 431). - Stop on disable. If the flavor is not
"mlflow"anddisable=True, the wrapper returns (lines 436 to 437). The body of the flavor’sautolog()does not run. - Patch. The body calls
safe_patch(flavor, cls, method, patch_function)(utils/autologging_utils/safety.py:232). The patched method checks at each call whether the flavor is disabled, or whetherdisable_autologging()is active (lines 386 to 392). If so, it calls the original method. - Record.
_wrap_patch(line 806) applies agorilla.Patch, and_store_patch(line 825) records it in_AUTOLOGGING_PATCHES[flavor].
Gorilla marks each patch on the class. It sets _gorilla_active_patch_<name> to the patch and _gorilla_original_<name> to the original (utils/gorilla.py:62, 65, apply at 263). revert_patches (safety.py:710) reverts each stored patch and runs any cleanup callbacks in _AUTOLOGGING_CLEANUP_CALLBACKS (line 25).
Two flavors, litellm and openai, put the decorator on an inner _autolog() and keep work outside it. The decorated body does not run on disable, so, as the source says, the callback removal must sit outside (litellm/__init__.py:68 to 71, openai/autolog.py:113 to 116).
Each autolog() call also sends a usage event to MLflow’s telemetry service through _record_event(AutologgingEvent, ...), for example at pydantic_ai/__init__.py:256. Section 4 covers that service.
mlflow.autolog()
mlflow.autolog() is in tracking/fluent.py:3616. It patches nothing by itself.
- Library map. It maps library names to flavors.
LIBRARY_TO_AUTOLOG_MODULE(line 3759) lists the machine learning libraries.GENAI_LIBRARY_TO_AUTOLOG_MODULE(line 3777) listsanthropic,openai,litellm,pydantic_aiand 16 more. - Import hooks. It registers a post-import hook for each library (line 3874). A hook runs the flavor’s
autolog()when the library is imported. It also runs at once for a library that is already imported. - Indirect imports. In
generic_autolog.py,import pydantic_aiturned onpydantic_ai.pydantic_ai’s import ofopenaiturned on theopenaiflavor too. - Respect for a flavor call. The hook skips a flavor whose config came from a direct flavor call (line 3844). So
mlflow.autolog(disable=True)does not disable a flavor that someone turned on withmlflow.openai.autolog(). - Databricks. In the Databricks runtime, a call with
disable=Falseuses only the machine learning map. The GenAI flavors are left out (line 3804).
After mlflow.autolog(), _post_import_hooks held hooks for 32 libraries. The hooks stay after mlflow.autolog(disable=True). They then call each flavor with disable=True.
mlflow.pydantic_ai.autolog()
autolog() in pydantic_ai/__init__.py:133 reads the installed version. For pydantic-ai 2.x at 2.5.0 or newer, it calls setup_autologging() from autolog_v2.py and returns (lines 145 to 163). For 2.x older than 2.5.0, it logs a warning and patches nothing. The 1.x code below line 165 does not run for 2.64.0.
setup_autologging() (pydantic_ai/autolog_v2.py:422 to 476) applies these patches:
| Class | Method | Wrapper | Span |
|---|---|---|---|
Agent |
__init__ |
patched_agent_init (line 85) |
none |
Agent |
run |
patched_agent_run (line 93) |
Agent.run, AGENT |
Agent |
run_sync |
patched_agent_run_sync (line 105) |
Agent.run_sync, AGENT |
Agent |
run_stream |
patched_agent_run_stream (line 117), own wrapper |
Agent.run_stream, AGENT |
Agent |
run_stream_sync |
patched_agent_run_stream_sync (line 337), own wrapper |
Agent.run_stream_sync, AGENT |
Instrumentation |
wrap_model_request |
patched_capability_model_request (line 140) |
<ModelClass>.request, LLM |
Instrumentation |
on_tool_validate_error |
patched_capability_tool_validate_error (line 157) |
<tool>.validation, PARSER |
Instrumentation |
wrap_tool_execute |
patched_capability_tool_execute (line 184) |
<tool name>, TOOL |
MCPToolset |
list_tools |
patched_mcp_list_tools (line 210) |
MCPToolset.list_tools, TOOL |
MCPToolset |
direct_call_tool |
patched_mcp_direct_call_tool (line 221) |
MCPToolset.direct_call_tool, TOOL |
pa_autolog.py printed the same set of gorilla markers after the call.
The Agent.__init__ patch changes a global behaviour. patched_agent_init (line 85) sets self.instrument = True when the agent’s instrument is None. The Instrumentation capability exists only for an instrumented agent. So the LLM and tool spans depend on that patch. Also, an instrumented agent writes PydanticAI’s own OTel spans to the global tracer provider. otel_interplay.py showed both effects:
- Our provider gets spans. With a global SDK tracer provider in place, autolog made our exporter receive
invoke_agent agentandchat test. These spans carrygen_ai.input.messages,gen_ai.output.messagesandpydantic_ai.all_messages. Without autolog, the same agent wrote no PydanticAI span. - Old agents stay instrumented. After
mlflow.pydantic_ai.autolog(disable=True), an agent that was built while autolog was on still wrote PydanticAI spans. An agent built after the disable wrote none.
An agent with instrument = False set after construction gets only the Agent.run span (instrument_false.py). In pydantic-ai 2.64.0, instrument is a property and not an Agent() argument.
mlflow.litellm.autolog()
mlflow.litellm.autolog() (litellm/__init__.py:13 to 61) does three things when it is on:
- Callbacks. It adds the string
"mlflow"tolitellm.success_callbackandlitellm.failure_callback(lines 45 to 47). - Executor patch. It patches
submiton the LiteLLM thread pool executor instance (lines 81 to 107). The patchedsubmitruns a success handler at once, in the calling thread, in place of the pool. - The logger. LiteLLM resolves
"mlflow"tolitellm.integrations.mlflow.MlflowLogger. The logger, and not MLflow, writes the span (litellm/integrations/mlflow.py).
On the first call, LiteLLM replaces the strings with MlflowLogger objects. It also adds the logger to _async_success_callback and _async_failure_callback (seen in litellm_autolog.py). On disable, MLflow removes MLflow loggers from success_callback, failure_callback and callbacks only (lines 52 to 57). The async lists keep the logger.
mlflow.openai.autolog()
mlflow.openai.autolog() (openai/autolog.py:33 to 106) calls the decorated _autolog() (line 117). That function patches create on these classes, each sync and async:
- Chat completions.
chat.completions.CompletionsandAsyncCompletions, alsoparse. - Other APIs.
completions.Completions,embeddings.Embeddingsandimages.Images, wheregenerateis the patched method. - Responses.
responses.ResponsesandAsyncResponses, alsoparse. - Beta parse.
beta.chat.completionsparse, if the module exists. openai 3.41.0 has no such module.
openai_anthropic_autolog.py openai printed 14 patched methods. The wrappers are patched_call and async_patched_call (line 257).
If the OpenAI Agents SDK is installed, autolog() also patches AgentRunner.run and run_streamed (lines 86 to 87). That code sits outside the decorated function, so it runs even on disable=True. With the default disable_openai_agent_tracer=True, it calls clear_trace_processors() (line 99). That call removes the Agents SDK’s own trace processors. The workspace does not install the Agents SDK, so I did not test this path.
mlflow.anthropic.autolog()
mlflow.anthropic.autolog() (anthropic/__init__.py:16 to 65) patches Messages.create and AsyncMessages.create. If claude_agent_sdk is installed, it also patches ClaudeSDKClient.__init__ (line 55). That patch replaces query and receive_response on each client object (anthropic/autolog.py:21 to 72). The docstring says that streaming is not recorded, and messages.stream is not patched.
The traceparent header
The OpenAI and Anthropic wrappers add a W3C traceparent header to the request (openai/autolog.py:526, anthropic/autolog.py:145). The header holds the MLflow trace ID and span ID (tracing/distributed/__init__.py:71 to 82). The stub servers received traceparent: 00-<trace id>-<span id>-03 on every call. So autolog changes what goes to the model provider, and it gives the provider an ID that joins calls into one trace. The Gemini, Groq and Mistral flavors do the same.
mlflow.otel.autolog()
mlflow.otel.autolog() (otel/__init__.py:79 to 145) is a different switch. It is not in mlflow.autolog()'s map. It sets a global SDK tracer provider if there is none, and it adds an OTLP exporter to it. Every OTel span in the process then goes to the MLflow server. A disable only turns the processor off. The processor stays on the provider.
2. What each flavor records
The scripts wrote to sqlite:///<scratch>/<name>.db and printed every key of spans.content. The script helpers.py holds the dump code.
Fields that every trace gets
Every stored span has the keys trace_id, span_id, parent_span_id, name and status in content. It also has start_time_unix_nano, end_time_unix_nano, attributes, events and links. There is no kind key.
Every span has these attributes:
mlflow.traceRequestId, the MLflow trace ID in the formtr-<32 hex>.mlflow.spanType, a JSON string such as"LLM".mlflow.spanLogLevel, which was20in every run.mlflow.spanInputsandmlflow.spanOutputs, as JSON.
Each attribute value is JSON encoded, so a string value keeps its quotes.
Each trace also gets rows outside spans:
- Metadata. The table
trace_request_metadataholdsmlflow.traceInputsandmlflow.traceOutputs, cut to a preview length. It also holdsmlflow.trace_schema.version(3),mlflow.trace.sizeBytes,mlflow.trace.sizeStatsandmlflow.trace.infoFinalized. Last comemlflow.source.name,mlflow.source.type,mlflow.source.git.*andmlflow.user, which is the login name. - Tags. The table
trace_tagsholdsmlflow.trace.spansLocationandmlflow.artifactLocation. - Columns. The table
trace_infoholdsrequest_previewandresponse_preview. It also holds the totalsinput_tokens,output_tokens,total_tokens,cache_read_input_tokensandtotal_cost.
pydantic_ai
pa_autolog.py ran an agent named adder on a FunctionModel. The model called the tool add once and then answered. The span tree:
Agent.run AGENT (root)
FunctionModel.request LLM (first model call, returns the tool call)
add TOOL (tool call)
FunctionModel.request LLM (second model call, returns the answer)The tool span is a sibling of the model spans. It is not a child of the model call that asked for it.
Agent.run (autolog_v2.py:59 and 93):
- Format.
mlflow.message.formatis"pydantic_ai". - Agent fields. Every public, serializable field of the agent becomes an attribute, through
extract_safe_attributes(pydantic_ai/utils.py:23). The run gavename,deps_type,end_strategy,instrument,model,output_type,root_capabilityandtoolsets.modelis the model object’s__dict__, which held a functionreprin this run. - Inputs.
mlflow.spanInputsholds the bound call arguments withoutselfanddeps, for example{"user_prompt": "What is 2+3?"}(line 45). - Outputs.
mlflow.spanOutputsholdsdataclasses.asdict(result)and adds_new_messages_serialized, a list of the new messages (pydantic_ai/utils.py:52). The result also holds the full private_state, with the message history and the tool schemas. The stored span was 10.7 KB for a two-step run. - Usage.
mlflow.chat.tokenUsageis the run total.
<ModelClass>.request (autolog_v2.py:65 and 140):
- Name. The name is the class name of the model, for example
FunctionModel.request,OpenAIChatModel.requestorTestModel.request. - Model fields. Every public field of the model becomes an attribute, for example
model_name,model_id,system,canonical_system,labelandbatch_max_requests. - Model and provider.
mlflow.llm.modelismodel.model_name.mlflow.llm.providerismodel.system, for example"openai"or"function". - Inputs.
mlflow.spanInputsis{"messages", "model_settings", "model_request_parameters"}(pydantic_ai/utils.py:41). The messages are PydanticAIModelRequestandModelResponseobjects as dicts, withpart_kindon each part.model_request_parametersholds the tool definitions. - Outputs.
mlflow.spanOutputsis theModelResponseas a dict. - Usage.
mlflow.chat.tokenUsageis described below.
<tool name> (autolog_v2.py:184):
- Inputs.
mlflow.spanInputsis the tool arguments, for example{"a": 2, "b": 3}. - Outputs.
mlflow.spanOutputsis the return value, for example5. - Other. The span has no other attributes. It has no tool call ID.
The token usage comes from parse_usage (pydantic_ai/utils.py:75). It maps input_tokens to input_tokens and output_tokens to output_tokens, and it computes total_tokens. It adds cache_read_input_tokens and cache_creation_input_tokens only when they are not zero. The function calls usage() as a method, and pydantic-ai 2.64.0 warns that usage is a property.
openai
openai_anthropic_autolog.py openai made a chat call with a tool, an async chat call with the tool result, and a Responses call. Each call became one root span, so each call is its own trace:
| Span name | Type | Call |
|---|---|---|
Completions |
CHAT_MODEL |
chat.completions.create |
AsyncCompletions |
CHAT_MODEL |
async chat.completions.create |
Responses |
CHAT_MODEL |
responses.create |
The span name is the class name of the resource (openai/autolog.py:303). The span type comes from _get_span_type (line 172). The legacy completions API gives LLM, embeddings give EMBEDDING and images give TOOL.
The attributes:
- Inputs.
mlflow.spanInputsis the full keyword arguments of the call, for examplemodel,messages,toolsandtemperature. The messages are in the OpenAI chat format. - Plain keys. Every argument except
messagesandinputis also a plain attribute, for examplemodel,tools,temperatureandinstructions. - Format.
mlflow.message.formatis"openai"for chat and LLM spans. - Outputs.
mlflow.spanOutputsis theChatCompletionorResponseobject as a dict. - Tools.
mlflow.chat.toolsis a list in the form{"type": "function", "function": {"name", "description", "parameters"}}(openai/utils/chat_schema.py:26). - Model.
mlflow.llm.modelcomes from the response. If the response has no model, it comes from the request. - Usage.
mlflow.chat.tokenUsagemapsprompt_tokensandcompletion_tokenstoinput_tokensandoutput_tokens. It mapsprompt_tokens_details.cached_tokenstocache_read_input_tokens. - Cost.
mlflow.llm.costis added on storage, as section 5 says.
The flavor does not set mlflow.llm.provider. For a stream, the flavor adds one span event per chunk, and it builds the output from the chunks (openai/autolog.py:332).
anthropic
openai_anthropic_autolog.py anthropic made a sync call that returned a tool_use block, and an async call with the tool_result. The spans were Messages.create and AsyncMessages.create, both CHAT_MODEL and both roots. The name is <class>.<method> (anthropic/autolog.py:120).
The attributes:
- Inputs and outputs.
mlflow.spanInputsis the full keyword arguments, includingsystem,messages,toolsandmax_tokens.mlflow.spanOutputsis theMessageas a dict. There are no plain attribute copies of the arguments. - Format.
mlflow.message.formatis"anthropic". - Tools.
mlflow.chat.toolsholds the tools, converted to the OpenAI form above (anthropic/chat.py). - Model and provider.
mlflow.llm.modelcomes from the requestmodel.mlflow.llm.provideris always"anthropic"(line 140). - Usage.
mlflow.chat.tokenUsagehasinput_tokens,output_tokens,total_tokens,cache_read_input_tokensandcache_creation_input_tokens(line 182).
The flavor adds the cache tokens into input_tokens and total_tokens (lines 204 to 206). The stub returned input_tokens=15, cache_read_input_tokens=4 and cache_creation_input_tokens=1. The span recorded input_tokens=20. A rebuild from Anthropic’s raw usage must do the same sum to match.
litellm
litellm_autolog.py made an acompletion call with tools and a completion call. Each became one root span, named litellm-<call_type>, for example litellm-acompletion (litellm/integrations/mlflow.py:247). The type is LLM.
The attributes (litellm/integrations/mlflow.py:167 to 225):
- Inputs.
mlflow.spanInputsis{"messages", "tools"}in the OpenAI format. Optionalfunctions,stream,tool_choice,userandpredictionare added when they are set. - Outputs.
mlflow.spanOutputsis the LiteLLMModelResponseas a dict. - Call fields. The plain attributes
litellm_call_id,call_type,model,api_baseandcache_hitcome from the call. - Logging payload.
raw_llm_response,response_cost,saved_cache_costandrequest_tagscome from LiteLLM’s standard logging payload.request_tagscame frommetadata={"tags": [...]}. - Usage.
mlflow.chat.tokenUsageholdsinput_tokens,output_tokensandtotal_tokensonly. It has no cache keys. - Start time.
mlflow.spanStartTimeNsholds the start time, because the logger starts the span after the call with an explicit start time.
The logger tries to set chat messages and tools through mlflow.tracing.utils.set_span_chat_messages. That function does not exist in 3.17.2, so the logger skips the step in silence (litellm/integrations/mlflow.py:54 to 59). So LiteLLM spans have no mlflow.chat.tools and no mlflow.message.format. The storage step added mlflow.llm.model from the output, and mlflow.llm.cost where LiteLLM knew the price. The span has no mlflow.llm.provider.
The async logger runs on LiteLLM’s logging worker. In a script that ends the event loop at once, the worker did not run. Then the span was lost, with the warning “coroutine ‘Logging.async_success_handler’ was never awaited”. The scripts wait 1.5 seconds after each acompletion.
PydanticAI on an OpenAI model under mlflow.autolog()
generic_autolog.py turned on mlflow.autolog() and ran a PydanticAI agent on OpenAIChatModel with the stub client. Two flavors wrote spans into one tree:
Agent.run AGENT format pydantic_ai
OpenAIChatModel.request LLM format pydantic_ai, provider openai
AsyncCompletions CHAT_MODEL format openai, no provider
add TOOL
OpenAIChatModel.request LLM
AsyncCompletions CHAT_MODELEach model call is recorded twice, once in PydanticAI’s message format and once in OpenAI’s format. Both spans have mlflow.chat.tokenUsage. The trace total was input_tokens=24 and output_tokens=8, which is two calls of 12 and 4. So the roll up counted each call once, and it did not add the root’s own total either.
3. Paths that switch autolog on without our code
I found these paths. Each one needs some code to run in the process, but that code need not be ours.
| Path | Where | What it turns on |
|---|---|---|
A direct call to mlflow.<flavor>.autolog() or mlflow.autolog() by any imported package |
anywhere | the flavor, or hooks for 32 libraries |
Post-import hooks left by an earlier mlflow.autolog() |
tracking/fluent.py:3874 |
a flavor at the moment its library is imported, also by another library |
mlflow.genai.evaluate() |
genai/evaluation/base.py:354, models/evaluation/utils/trace.py:28 to 86 |
tracing for every supported flavor whose config is not disable=True, during the evaluation |
| Databricks runtime start | outside MLflow, documented at tracking/fluent.py:3801 to 3806 |
the machine learning flavors only, not the GenAI flavors |
mlflow.otel.autolog() |
otel/__init__.py |
an OTLP exporter on the global tracer provider |
LiteLLM proxy configuration, litellm_settings: success_callback: ["mlflow"] |
LiteLLM proxy | the LiteLLM logger, with no MLflow flavor call and no MLflow config |
litellm.success_callback, litellm.callbacks or the per-call success_callback= argument |
any code | the LiteLLM logger, which writes to MLflow’s current tracking URI |
mlflow autolog claude |
MLflow CLI | Claude Code hooks, outside our process |
These checks found no other path:
- Environment. No environment variable turns autolog on. The only
AUTOLOGvariable inenvironment_variables.pyisMLFLOW_AUTOLOGGING_TESTING(line 602), which changes error handling. - Import.
import mlflowturns nothing on. The audit after the import showed no config, no hook and no patch. - Entry points and startup files. The
mlflow3.17.2 distribution has noopentelemetry_instrumentorentry point and no.pthfile. Its entry points areconsole_scripts,mlflow.app,mlflow.app.clientandmlflow.deployments. - The workspace. Only
litellm/integrations/mlflow.pyrefers to MLflow in the workspace’s installed packages. I found no LiteLLM environment variable that adds callbacks in the SDK. The proxy reads them from its configuration file.
Two related paths are outside MLflow. Agent.instrument_all() in pydantic-ai, and logfire.instrument_pydantic_ai(), turn on PydanticAI’s own OTel spans. The opentelemetry-instrument wrapper loads opentelemetry_instrumentor entry points. The workspace has one, for httpx.
4. Suppression
Test results
suppress.py SCENARIO runs one scenario in a fresh process. The workload is a PydanticAI agent on the OpenAI stub with one tool call, one litellm.acompletion with mock_response, and one Anthropic call. A full run with everything on gives 8 spans and 3 traceparent headers.
| Scenario | Patches after the switch | Spans written after the switch | Headers |
|---|---|---|---|
baseline: generic and four flavor switches on |
26 methods, and the LiteLLM executor | 8 | 3 of 3 |
flavor_disable: each flavor autolog(disable=True) |
none | 1, litellm-acompletion |
0 of 3 |
global_disable_after_global: mlflow.autolog(), then mlflow.autolog(disable=True) |
none | 1, litellm-acompletion |
0 of 3 |
global_disable_after_flavor: mlflow.openai.autolog(), then mlflow.autolog(disable=True) |
14 OpenAI methods | 2, AsyncCompletions |
2 of 3 |
tracing_disable: everything on, then mlflow.tracing.disable() |
all, still applied | 0 | 0 of 3 |
tracing_disable_then_set_uri: as above, then mlflow.set_tracking_uri(other) |
all | 8 | 3 of 3 |
disable_autologging_ctx: everything on, workload inside disable_autologging() |
all | 1, litellm-acompletion |
0 of 3 |
sampling_zero: everything on, with MLFLOW_TRACE_SAMPLING_RATIO=0 |
all | 0 | 3 of 3 |
recipe: everything on, then guard.suppress_all() |
none | 0 | 0 of 3 |
Each switch and its effect
mlflow.<flavor>.autolog(disable=True). It reverts the flavor’s gorilla patches at once, and it recordsdisable=True. Later hooks andmlflow.genai.evaluate()respect that record. It does not resetinstrument=Trueon agents that exist already. For LiteLLM, it leavesMlflowLoggerin the async lists.mlflow.autolog(disable=True). It disables flavors thatmlflow.autolog()turned on. It does not touch a flavor that a direct flavor call turned on (tracking/fluent.py:3844). It leaves the import hooks in place, set to disable.mlflow.tracing.disable(). It replaces MLflow’s tracer provider with aNoOpTracerProvider(tracing/provider.py:938to 976). The patches stay. But each patched call makes a no-op span, so no span is stored and no header is sent. The LiteLLM logger also writes nothing, because it uses the same provider.disable_autologging(). The context manager (utils/autologging_utils/__init__.py:544) sets a module flag thatsafe_patchand the PydanticAI v2 wrappers read. The LiteLLM logger does not read it.MLFLOW_TRACE_SAMPLING_RATIO=0. The sampler drops every root span (tracing/provider.py:744). The flavors still make a span context and still send the header.- No environment variable disables tracing. The tracing variables in
environment_variables.pyset the destination, sampling, export and buffers. None of them is an off switch.
Ways to turn MLflow on again
- A new tracking URI.
mlflow.set_tracking_uri()with a new URI callstracing.provider.reset()(tracking/_tracking_service/utils.py:108to 121). The next span builds a real provider again. - Other calls.
mlflow.tracing.enable()andmlflow.tracing.reset()turn tracing on again (tracing/provider.py:979, 1083). - A flavor call. Any
mlflow.<flavor>.autolog()patches again.mlflow.genai.evaluate()turns on every flavor that is not disabled by name. - LiteLLM lists. Any code can add a logger to a LiteLLM callback list.
MLflow’s usage telemetry
MLflow sends usage events, such as each autolog() call, to config.mlflow-telemetry.io and the ingestion URL that it names (telemetry/constant.py:9, telemetry/client.py:274). MLFLOW_DISABLE_TELEMETRY=true or DO_NOT_TRACK=true turns it off (telemetry/utils.py:144 to 161). MLflow makes the decision once, at import (mlflow/__init__.py:456). telemetry_switch.py showed that a variable set after the import has no effect until a call to mlflow.telemetry.client.set_telemetry_client().
The recipe
guard.py holds a sketch of the recipe as suppress_all(). It passed the recipe scenario, and it has these steps:
- Set the environment. Set
MLFLOW_USE_DEFAULT_TRACER_PROVIDER=true,MLFLOW_DISABLE_TELEMETRY=trueandMLFLOW_TRACE_SAMPLING_RATIO=0, before MLflow is imported if possible. If MLflow is imported already, callset_telemetry_client(). - Disable by name. Call
mlflow.autolog(disable=True), and thenmlflow.<flavor>.autolog(disable=True, silent=True)for each flavor. The named calls revert patches that a flavor call made. They also recorddisable=True, so thatmlflow.genai.evaluate()and import hooks leave the flavor off. - Turn off MLflow’s provider. Call
mlflow.tracing.disable(). It stops spans and headers even if a patch comes back. - Clean LiteLLM. Remove every logger that configuration does not approve from all seven lists. The lists are
input_callback,success_callback,failure_callback,service_callback,callbacks,_async_success_callbackand_async_failure_callback. - Pin PydanticAI. Set
agent.instrument = FalseafterAgent(...), unless PydanticAI’s own OTel spans are approved. The patched__init__setsTrueonly when the value isNone.
Approved telemetry fits into step 4 and step 5. A LiteLLM logger whose name is in the approved list stays. An approved PydanticAI instrumentation gets an explicit InstrumentationSettings with provrun’s tracer provider.
Detection
guard.check() returns a list of problems. It reads these values:
AUTOLOGGING_INTEGRATIONS, for any flavor withdisablefalse._AUTOLOGGING_PATCHES, for any live patch.mlflow.tracing.provider.is_tracing_enabled().- The seven LiteLLM lists, for loggers that are not approved.
MLFLOW_USE_DEFAULT_TRACER_PROVIDER.
The check takes about 5 microseconds. It reads sys.modules and imports nothing. guard_reenable.py showed that it found the re-enable by set_tracking_uri(). The recipe scenario showed that it found a later mlflow.openai.autolog() and mlflow.litellm.autolog().
audit.py holds a wider check for reports. It scans every loaded module for gorilla markers, and it also finds patches on objects, such as LiteLLM’s executor. It took 650 milliseconds over 8,800 modules, so it suits the start and the end of a run.
Run guard.check() before each model call and each tool call. Run the full audit at the start and the end of a run. If a check finds a problem, record it as evidence and apply the recipe again. A library that patches through wrapt does not leave gorilla markers. To find it, compare the method objects with a list taken at start, or look for __wrapped__.
5. What MLflow adds when it stores a span
The translation step
SqlAlchemyStore._log_spans_once (store/tracking/sqlalchemy_store.py:5702) calls translate_span_when_storing(span) for each span (line 5729). The in-process SQLite path uses the same code, so autolog spans pass through it too. The function is in tracing/otel/translation/__init__.py:60 to 149. It adds an attribute only when the attribute is missing:
| Attribute | Source |
|---|---|
mlflow.spanType |
the span kind attribute of the first translator that knows it. It replaces only a missing type or UNKNOWN |
mlflow.spanInputs |
the translators’ input keys, then the span events |
mlflow.spanOutputs |
the translators’ output keys, then the span events |
mlflow.chat.tokenUsage |
the translators’ token keys. total_tokens is the sum when it is missing |
mlflow.message.format |
a translator that knows a format |
mlflow.chat.tools |
the translators’ tool definition keys |
mlflow.llm.model |
the translators’ model keys, then model in the inputs, then model in the outputs |
mlflow.llm.provider |
the translators’ provider key |
mlflow.llm.cost |
litellm.cost_per_token on the model, the provider and the usage (tracing/utils/__init__.py:364) |
MLflow tries the translators in the order of the list _TRANSLATORS (line 37). OpenInference comes first, then Gemini CLI, Claude Code and GenAI. Spring AI, Traceloop, Google ADK, Vercel AI, VoltAgent, LiveKit, Laminar and Langfuse follow.
The GenAI translator (tracing/otel/translation/genai_semconv.py:26 to 59) reads these keys:
| MLflow attribute | GenAI semantic convention key |
|---|---|
mlflow.spanType |
gen_ai.operation.name: chat to CHAT_MODEL, text_completion, generate_content and response to LLM, embeddings to EMBEDDING, execute_tool to TOOL, invoke_agent and create_agent to AGENT |
input_tokens |
gen_ai.usage.input_tokens |
output_tokens |
gen_ai.usage.output_tokens |
cache_read_input_tokens |
gen_ai.usage.cache_read.input_tokens |
cache_creation_input_tokens |
gen_ai.usage.cache_write.input_tokens, or the older gen_ai.usage.cache_creation.input_tokens |
mlflow.spanInputs |
gen_ai.input.messages, then gen_ai.tool.call.arguments, then the events gen_ai.system.message, gen_ai.user.message and gen_ai.assistant.message |
mlflow.spanOutputs |
gen_ai.output.messages, then gen_ai.tool.call.result, then the event gen_ai.choice |
mlflow.llm.model |
gen_ai.response.model, then gen_ai.request.model |
mlflow.llm.provider |
gen_ai.provider.name |
mlflow.chat.tools |
gen_ai.tool.definitions |
A test with PydanticAI’s own spans
translate.py ran an instrumented PydanticAI agent with no autolog. It sent the spans through OTLP protobuf, Span.from_otel_proto and SqlAlchemyStore.log_spans, which is the path of the server’s /v1/traces. MLflow added these attributes:
chat gpt-4o-mini. It gotmlflow.spanTypeCHAT_MODEL,mlflow.spanInputsandmlflow.spanOutputsin the GenAI message format withroleandparts,mlflow.chat.tokenUsagewith cache reads,mlflow.chat.tools,mlflow.llm.model,mlflow.llm.providerandmlflow.llm.cost.execute_tool add. It gotTOOL, with inputs{"a":2,"b":3}and output"5".invoke_agent adder. It got onlyAGENT. The run total ingen_ai.aggregated_usage.*is not read, so the root span has no usage.
No span got mlflow.message.format, because the GenAI translator names no format. The trace row got input_tokens=31, output_tokens=8, cache_read_input_tokens=2, a total_cost and a session_id from gen_ai.conversation.id. It got no trace_name and no previews. As mlflow-tracing.md found, the spans.type column stayed NULL, but model_name and model_provider were filled.
Roll ups on the trace
- Processor. When the root span ends, MLflow’s span processor writes
mlflow.traceInputsandmlflow.traceOutputsfrom the root span. It also writesmlflow.trace.tokenUsage, the sum of usage over the spans (tracing/processor/base_mlflow.py:342to 375). - Client cost. It writes
mlflow.trace.costonly for backends that need the cost from the client, such as Databricks (tracing/utils/__init__.py:943). - Store. For SQL, the store computes the totals, the cost columns, the session ID from
session.idorgen_ai.conversation.id, and the user fromuser.id(sqlalchemy_store.py:5715to 5790). In the SQLite runs, the totals were in thetrace_infocolumns.
Export in GenAI format
With MLFLOW_ENABLE_OTEL_GENAI_SEMCONV=true, OTLP export runs translate_span_to_genai (tracing/export/genai_semconv/translator.py:46 to 137). It does the reverse mapping:
- Span type.
mlflow.spanTypebecomesgen_ai.operation.name, withCHAT_MODELtochat,LLMtogenerate_content,EMBEDDINGtoembeddings,TOOLtoexecute_toolandAGENTtoinvoke_agent. - Model and usage.
mlflow.llm.modelbecomesgen_ai.request.model, andmlflow.llm.providerbecomesgen_ai.provider.name. The usage becomesgen_ai.usage.input_tokensandoutput_tokens. - Tools and agents. A
TOOLspan getsgen_ai.tool.name,gen_ai.tool.call.argumentsandgen_ai.tool.call.result. AnAGENTspan getsgen_ai.agent.name. - Messages. A converter for each
mlflow.message.formatwritesgen_ai.input.messages,gen_ai.output.messagesandgen_ai.system_instructions. It also writesgen_ai.request.temperature,max_tokens,top_p,stop_sequences,gen_ai.response.idandgen_ai.response.model(tracing/export/genai_semconv/converter.py). The formats areopenai,anthropic,geminiandbedrock. Any other format,pydantic_aiincluded, uses the OpenAI chat converter. - Names and cleanup. The span name becomes
<operation> <tool, agent or model>, for examplechat gpt-4o-mini. Othermlflow.*attributes are dropped, exceptmlflow.traceTag.*.
Names for a provevi trace database
A provevi trace database can use these names, so that MLflow and OTel tools read the same columns. The keys are in tracing/constant.py:56 to 157:
| Meaning | MLflow name | GenAI name |
|---|---|---|
| Span type | mlflow.spanType: AGENT, LLM, CHAT_MODEL, TOOL, EMBEDDING, PARSER, UNKNOWN |
gen_ai.operation.name |
| Inputs, outputs | mlflow.spanInputs, mlflow.spanOutputs |
gen_ai.input.messages, gen_ai.output.messages, gen_ai.tool.call.arguments, gen_ai.tool.call.result |
| Message format | mlflow.message.format: pydantic_ai, openai, anthropic |
none |
| Token usage | mlflow.chat.tokenUsage with input_tokens, output_tokens, total_tokens, cache_read_input_tokens, cache_creation_input_tokens, cache_creation_input_tokens_above_1hr |
gen_ai.usage.* |
| Model, provider | mlflow.llm.model, mlflow.llm.provider |
gen_ai.request.model, gen_ai.response.model, gen_ai.provider.name |
| Cost | mlflow.llm.cost with input_cost, output_cost, total_cost in USD |
none |
| Tools offered | mlflow.chat.tools, OpenAI function form |
gen_ai.tool.definitions |
| Session, user | trace metadata mlflow.trace.session, mlflow.trace.user. Span attributes session.id, user.id |
gen_ai.conversation.id |
| Trace totals | mlflow.trace.tokenUsage, mlflow.trace.cost, mlflow.traceInputs, mlflow.traceOutputs |
none |
The provrun module otel.py already writes gen_ai.operation.name, gen_ai.request.model, gen_ai.tool.name, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.agent.name. With those keys alone, MLflow’s storage step gives the span type, the model and the usage. For inputs, outputs and a provider, provrun must also write gen_ai.input.messages, gen_ai.output.messages and gen_ai.provider.name, or the mlflow.* keys.
To match autolog from provrun’s own events:
- One model call. A
model_calledevent maps to oneLLMorCHAT_MODELspan with the message format of its backend. For the Anthropic format, put the cache tokens intoinput_tokens. - One tool call. A
tool_calledevent maps to oneTOOLspan under the agent span, named by the tool, with the arguments and the result. The tool span is not under the model span. - The run. The run maps to one
AGENTroot span with the prompt and the final output.
Scripts
The scripts were run in a scratch folder, and they are not in the repository. Each one ran with ./run.sh.
| Script | What it shows |
|---|---|
run.sh |
Builds the environment, and sets MLFLOW_DISABLE_AGENT_HINT, MLFLOW_DISABLE_TELEMETRY and DO_NOT_TRACK |
helpers.py |
Fresh SQLite store, span dump, patch list |
stubs.py |
Stub OpenAI and Anthropic servers that record request headers |
pa_autolog.py |
PydanticAI patches, globals and span tree |
litellm_autolog.py |
LiteLLM callback lists, span tree and the async leak after a disable |
openai_anthropic_autolog.py openai or anthropic |
OpenAI and Anthropic patches, span trees and traceparent headers |
generic_autolog.py |
mlflow.autolog() hooks, and PydanticAI on OpenAI with two flavors |
suppress.py SCENARIO |
The suppression table |
otel_interplay.py, shared_provider.py |
Autolog with a global tracer provider, in both provider modes |
instrument_false.py |
agent.instrument = False under autolog |
telemetry_switch.py |
When the telemetry choice is made |
translate.py |
The storage translation of PydanticAI’s own OTel spans |
audit.py, guard.py, guard_reenable.py |
The full audit, the recipe and the fast check |
Open questions
- The OpenAI Agents SDK path. The SDK is not in the workspace, so I did not test the
AgentRunnerpatches or the clearing of its trace processors. From the source, these run even ondisable=True. - Claude Agent SDK. I did not test the
ClaudeSDKClientpatch. - Streams. I did not run streams for any flavor. The source says that OpenAI records chunk events, that Anthropic records no stream, and that LiteLLM adds
streaming_chunkevents. - Databricks. I did not test in a Databricks runtime. The claim about it comes from the comment and the code at
tracking/fluent.py:3801to 3806. - Other callers. I cannot list every third-party package that may call
mlflow.autolog(). The workspace has none today. The guard checks at run time for that reason.