> ## Documentation Index
> Fetch the complete documentation index at: https://www.adaline.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Advanced usage

> Multi-step workflows, session tracking, and production tracing patterns

Once you have basic instrumentation in place, these patterns help you model complex AI workflows — from multi-step RAG pipelines to conversational agents with tool calling.

## Span content: input and output

Every span's `content` object carries two keys — **`input`** and **`output`** — both stringified JSON. These represent the raw payload going into the operation and the raw result coming back out.

While `input` and `output` accept any valid stringified JSON, you get the most value by sending the **exact request payload you send to your AI provider** as `input` and the **full response** as `output`. When you use a supported provider — OpenAI, Anthropic, Google, and others — Adaline automatically parses these raw objects to:

* **Calculate cost** based on token counts and the model's pricing
* **Extract token usage** (prompt tokens, completion tokens, total)
* **Surface model metadata** such as stop reason, tool calls, and function invocations
* **Power continuous evaluations** with structured input/output pairs

The easiest way to achieve this is to build your request params as an object, pass it to the provider SDK, and stringify that same object as `input`:

<Tabs>
  <Tab title="TypeScript">
    ```typescript theme={null}
    const params = {
      model: "gpt-4",
      messages: [
        { role: "system", content: "You are a helpful assistant." },
        { role: "user", content: "Explain quantum computing in simple terms." },
      ],
      temperature: 0.7,
    };

    const response = await openai.chat.completions.create(params);

    span.update({
      status: "success",
      content: {
        type: "Model",
        provider: "openai",
        model: "gpt-4",
        input: JSON.stringify(params),
        output: JSON.stringify(response),
      },
    });
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    from adaline_api.models.log_span_content import LogSpanContent
    from adaline_api.models.log_span_model_content import LogSpanModelContent

    params = {
        "model": "gpt-4",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Explain quantum computing in simple terms."},
        ],
        "temperature": 0.7,
    }

    response = openai.chat.completions.create(**params)

    span.update({
        "status": "success",
        "content": LogSpanContent(
            actual_instance=LogSpanModelContent(
                type="Model",
                provider="openai",
                model="gpt-4",
                input=json.dumps(params),
                output=json.dumps(response.model_dump()),
            )
        ),
    })
    ```
  </Tab>
</Tabs>

This pattern works the same way for any supported provider — Anthropic, Google, etc. The key is that the `input` and `output` values are the raw, unmodified payloads.
You can also set the `input` and `output` fields to use [Adaline's own content schema](/docs/reference/sdk/v2/typescript/types/MessageContent), although this is more advanced and requires maintaining custom transformations to convert provider payloads into the Adaline format.

<Tip>
  If you're using a custom or unsupported provider, you can still send any valid JSON string for `input` and `output`. Adaline will store and display them — automatic enrichment simply won't apply.
</Tip>

## Group spans into a single trace

When your workflow makes multiple LLM calls (e.g., a RAG pipeline with an embedding call followed by a chat completion), group them under a single trace to see the full picture.

### With the Proxy

Use the `adaline-trace-reference-id` header with the same value across all requests:

```python theme={null}
import uuid

trace_id = str(uuid.uuid4())

# Step 1: Embedding
embedding = client.embeddings.create(
    model="text-embedding-3-small",
    input="User query",
    extra_headers={
        "adaline-api-key": os.getenv("ADALINE_API_KEY"),
        "adaline-project-id": os.getenv("ADALINE_PROJECT_ID"),
        "adaline-prompt-id": os.getenv("ADALINE_PROMPT_ID"),
        "adaline-trace-reference-id": trace_id,
        "adaline-trace-name": "rag-pipeline",
        "adaline-trace-status": "pending",
        "adaline-span-name": "query-embedding",
    },
)

# Step 2: Chat completion with context
response = client.chat.completions.create(
    model="gpt-4",
    messages=[
        {"role": "system", "content": f"Context: {retrieved_docs}"},
        {"role": "user", "content": "User query"},
    ],
    extra_headers={
        "adaline-api-key": os.getenv("ADALINE_API_KEY"),
        "adaline-project-id": os.getenv("ADALINE_PROJECT_ID"),
        "adaline-prompt-id": os.getenv("ADALINE_PROMPT_ID"),
        "adaline-trace-reference-id": trace_id,
        "adaline-trace-name": "rag-pipeline",
        "adaline-trace-status": "success",
        "adaline-span-name": "response-generation",
    },
)
```

### With the SDK

The SDK groups spans naturally through the trace object:

```typescript theme={null}
const trace = monitor.logTrace({ name: "rag-pipeline" });

const embeddingSpan = trace.logSpan({ name: "query-embedding" });
// ... do embedding call ...
embeddingSpan.update({ status: "success", content: { type: "Embeddings", ... } });
embeddingSpan.end();

const retrievalSpan = trace.logSpan({ name: "vector-search" });
// ... do retrieval ...
retrievalSpan.update({ status: "success", content: { type: "Retrieval", ... } });
retrievalSpan.end();

const llmSpan = trace.logSpan({ name: "response-generation" });
// ... do LLM call ...
llmSpan.update({ status: "success", content: { type: "Model", ... } });
llmSpan.end();

trace.update({ status: "success" });
trace.end();
```

## Session tracking

Group related traces by session to follow a user's full conversation or multi-request workflow:

### With the Proxy

```python theme={null}
session_id = "user-session-abc123"

response = client.chat.completions.create(
    model="gpt-4",
    messages=conversation_history,
    extra_headers={
        "adaline-api-key": os.getenv("ADALINE_API_KEY"),
        "adaline-project-id": os.getenv("ADALINE_PROJECT_ID"),
        "adaline-prompt-id": os.getenv("ADALINE_PROMPT_ID"),
        "adaline-trace-session-id": session_id,
        "adaline-trace-name": "chat-turn",
    },
)
```

### With the SDK

```typescript theme={null}
const trace = monitor.logTrace({
  name: "Chat Turn",
  sessionId: "user-session-abc123",
});
```

Session IDs let you [filter traces](/docs/monitor/analyze-log-traces#filter-traces) to see all activity within a single user session.

## Tool calling agent pattern

For agents that use tool calling, create nested spans that capture the decision -> tool execution -> response cycle:

```typescript theme={null}
const trace = monitor.logTrace({ name: "agent-request" });

// Step 1: Initial LLM call (may produce tool calls)
const decisionSpan = trace.logSpan({ name: "agent-decision" });
const llmResponse = await callLLM(messages);
decisionSpan.update({
  status: "success",
  content: { type: "Model", provider: "openai", model: "gpt-4", input: "...", output: "..." },
});
decisionSpan.end();

// Step 2: Execute tool calls
if (llmResponse.toolCalls) {
  for (const toolCall of llmResponse.toolCalls) {
    const toolSpan = trace.logSpan({
      name: `tool-${toolCall.function.name}`,
      tags: ["tool-call", toolCall.function.name],
    });
    
    const result = await executeToolCall(toolCall);
    
    toolSpan.update({
      status: "success",
      content: {
        type: "Tool",
        name: toolCall.function.name,
        input: toolCall.function.arguments,
        output: JSON.stringify(result),
      },
    });
    toolSpan.end();
  }

  // Step 3: Final response with tool results
  const responseSpan = trace.logSpan({ name: "final-response" });
  const finalResponse = await callLLM([...messages, ...toolResults]);
  responseSpan.update({
    status: "success",
    content: { type: "Model", provider: "openai", model: "gpt-4", input: "...", output: "..." },
  });
  responseSpan.end();
}

trace.update({ status: "success" });
trace.end();
```

## Error handling pattern

Capture errors at both the span and trace level for effective debugging:

```typescript theme={null}
const trace = monitor.logTrace({ name: "workflow" });

try {
  const span = trace.logSpan({ name: "llm-call" });
  
  try {
    const response = await callLLM(messages);
    span.update({ status: "success", content: { type: "Model", ... } });
  } catch (error) {
    span.update({
      status: "failure",
      attributes: {
        error: error instanceof Error ? error.message : String(error),
        errorType: error instanceof Error ? error.name : "Unknown",
      },
    });
    throw error;
  } finally {
    span.end();
  }

  trace.update({ status: "success" });
} catch (error) {
  trace.update({
    status: "failure",
    attributes: { error: String(error) },
  });
} finally {
  trace.end();
}
```

## Multi-provider pattern

When your workflow uses multiple AI providers (e.g., OpenAI for generation and Anthropic for review), capture each in separate spans within the same trace:

```typescript theme={null}
const trace = monitor.logTrace({ name: "content-pipeline" });

// OpenAI generation
const genSpan = trace.logSpan({ name: "content-generation" });
const draft = await openaiCall(messages);
genSpan.update({
  status: "success",
  content: { type: "Model", provider: "openai", model: "gpt-4", input: "...", output: draft },
});
genSpan.end();

// Anthropic review
const reviewSpan = trace.logSpan({ name: "content-review" });
const review = await anthropicCall(draft);
reviewSpan.update({
  status: "success",
  content: { type: "Model", provider: "anthropic", model: "claude-3-5-sonnet", input: draft, output: review },
});
reviewSpan.end();

trace.end();
```

## Passing variables for evaluation

When using [continuous evaluations](/docs/monitor/setup-continuous-evaluations), include variable values so they can be used to [build datasets](/docs/monitor/build-logs-from-dataset):

### With the Proxy

```python theme={null}
headers["adaline-span-variables"] = json.dumps({
    "user_question": {"modality": "text", "value": user_input},
    "context": {"modality": "text", "value": retrieved_context},
})
```

### With the SDK

Variables are set on the span's **content** object (specifically on `Model` or `ModelStream` content types), not on `logSpan()` directly:

```typescript theme={null}
const span = trace.logSpan({ name: "llm-call" });

span.update({
  status: "success",
  content: {
    type: "Model",
    provider: "openai",
    model: "gpt-4",
    variables: {
      user_question: { name: "user_question", value: { modality: "text", value: userInput } },
      context: { name: "context", value: { modality: "text", value: retrievedContext } },
    },
    input: JSON.stringify(params),
    output: JSON.stringify(response),
  },
});
```

## Override continuous evaluation sample rate

[Continuous evaluations](/docs/monitor/setup-continuous-evaluations) run on a configurable sample rate — not every span is evaluated. When you need to guarantee that a specific span is evaluated regardless of the sample rate, set `runEvaluation: true` on the span.

This is useful when you want to force evaluation on specific requests — high-value customers, flagged conversations, edge cases you're debugging, or canary deployments where every response matters.

### At span creation

<Tabs>
  <Tab title="TypeScript">
    ```typescript theme={null}
    const span = trace.logSpan({
      name: "chat-completion",
      runEvaluation: true,
      content: { type: "Model", provider: "openai", model: "gpt-4", input: JSON.stringify(params), output: "{}" },
    });
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    from adaline_api.models.log_span_content import LogSpanContent
    from adaline_api.models.log_span_model_content import LogSpanModelContent

    span = trace.log_span(
        name="chat-completion",
        run_evaluation=True,
        content=LogSpanContent(
            actual_instance=LogSpanModelContent(
                type="Model", provider="openai", model="gpt-4",
                input=json.dumps(params), output="{}",
            )
        ),
    )
    ```
  </Tab>
</Tabs>

### Via span update

You can also set it after creation — for example, based on a condition you only know after the LLM responds:

<Tabs>
  <Tab title="TypeScript">
    ```typescript theme={null}
    const response = await openai.chat.completions.create(params);

    const shouldEvaluate = user.tier === "enterprise" || response.choices[0].finish_reason === "content_filter";

    span.update({
      status: "success",
      runEvaluation: shouldEvaluate,
      content: {
        type: "Model",
        provider: "openai",
        model: "gpt-4",
        input: JSON.stringify(params),
        output: JSON.stringify(response),
      },
    });
    span.end();
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    response = openai.chat.completions.create(**params)

    should_evaluate = user.tier == "enterprise" or response.choices[0].finish_reason == "content_filter"

    span.update({
        "status": "success",
        "run_evaluation": should_evaluate,
        "content": LogSpanContent(
            actual_instance=LogSpanModelContent(
                type="Model",
                provider="openai",
                model="gpt-4",
                input=json.dumps(params),
                output=json.dumps(response.model_dump()),
            )
        ),
    })
    span.end()
    ```
  </Tab>
</Tabs>

### With the Proxy

Set the `adaline-span-run-evaluation` header to `"true"`:

```python theme={null}
response = client.chat.completions.create(
    model="gpt-4",
    messages=messages,
    extra_headers={
        "adaline-api-key": os.getenv("ADALINE_API_KEY"),
        "adaline-project-id": os.getenv("ADALINE_PROJECT_ID"),
        "adaline-span-run-evaluation": "true",
    },
)
```

<Note>
  Setting `runEvaluation: true` guarantees the span will be evaluated. It does not affect spans where the flag is omitted or `false` — those still follow the configured sample rate.
</Note>

## Parallel workflows

When your workflow runs operations concurrently — multiple tool calls at once, parallel retrieval from several sources, or fan-out to multiple LLM providers — create sibling spans that overlap in time. Adaline renders them correctly based on their `startedAt` and `endedAt` timestamps.

<Tabs>
  <Tab title="TypeScript">
    ```typescript theme={null}
    const trace = monitor.logTrace({ name: "parallel-workflow" });

    // Create all spans before starting the work
    const searchSpan = trace.logSpan({ name: "web-search", tags: ["parallel"] });
    const dbSpan = trace.logSpan({ name: "db-lookup", tags: ["parallel"] });
    const cacheSpan = trace.logSpan({ name: "cache-check", tags: ["parallel"] });

    // Run all operations concurrently
    const [searchResults, dbResults, cacheResults] = await Promise.all([
      webSearch(query).then((res) => {
        searchSpan.update({ status: "success", content: { type: "Retrieval", input: JSON.stringify({ query }), output: JSON.stringify(res) } });
        searchSpan.end();
        return res;
      }),
      dbLookup(query).then((res) => {
        dbSpan.update({ status: "success", content: { type: "Retrieval", input: JSON.stringify({ query }), output: JSON.stringify(res) } });
        dbSpan.end();
        return res;
      }),
      cacheCheck(query).then((res) => {
        cacheSpan.update({ status: "success", content: { type: "Function", input: JSON.stringify({ query }), output: JSON.stringify(res) } });
        cacheSpan.end();
        return res;
      }),
    ]);

    // Merge results and generate final response
    const llmSpan = trace.logSpan({ name: "generate-response" });
    const response = await callLLM(mergeContext(searchResults, dbResults, cacheResults));
    llmSpan.update({ status: "success", content: { type: "Model", provider: "openai", model: "gpt-4", input: JSON.stringify(params), output: JSON.stringify(response) } });
    llmSpan.end();

    trace.end();
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    from adaline_api.models.log_span_content import LogSpanContent
    from adaline_api.models.log_span_retrieval_content import LogSpanRetrievalContent
    from adaline_api.models.log_span_function_content import LogSpanFunctionContent
    from adaline_api.models.log_span_model_content import LogSpanModelContent

    trace = monitor.log_trace(name="parallel-workflow")

    search_span = trace.log_span(name="web-search", tags=["parallel"])
    db_span = trace.log_span(name="db-lookup", tags=["parallel"])
    cache_span = trace.log_span(name="cache-check", tags=["parallel"])

    async def do_search():
        res = await web_search(query)
        search_span.update({"status": "success", "content": LogSpanContent(
            actual_instance=LogSpanRetrievalContent(type="Retrieval", input=json.dumps({"query": query}), output=json.dumps(res)))})
        search_span.end()
        return res

    async def do_db():
        res = await db_lookup(query)
        db_span.update({"status": "success", "content": LogSpanContent(
            actual_instance=LogSpanRetrievalContent(type="Retrieval", input=json.dumps({"query": query}), output=json.dumps(res)))})
        db_span.end()
        return res

    async def do_cache():
        res = await cache_check(query)
        cache_span.update({"status": "success", "content": LogSpanContent(
            actual_instance=LogSpanFunctionContent(type="Function", input=json.dumps({"query": query}), output=json.dumps(res)))})
        cache_span.end()
        return res

    search_results, db_results, cache_results = await asyncio.gather(
        do_search(), do_db(), do_cache()
    )

    llm_span = trace.log_span(name="generate-response")
    response = await call_llm(merge_context(search_results, db_results, cache_results))
    llm_span.update({"status": "success", "content": LogSpanContent(
        actual_instance=LogSpanModelContent(type="Model", provider="openai", model="gpt-4", input=json.dumps(params), output=json.dumps(response)))})
    llm_span.end()

    trace.end()
    ```
  </Tab>
</Tabs>

The same pattern works for parallel tool calls inside an agent loop — create a span per tool call and run them with `Promise.all` / `asyncio.gather`. Each span records its own timing independently.

## Distributed logging

When a single user request flows through multiple services — an API gateway, a worker queue, a retrieval service — you can attach all spans to one trace using the [REST API](/docs/instrument/with-adaline-api). The main service creates the trace and passes its `referenceId` to downstream services. Those services don't create a new trace; they attach spans to the existing one using `traceReferenceId`, or update the trace using `referenceId`.

### Step 1: Main service creates the trace

The orchestrator creates the trace, logs its own span, and passes the `referenceId` to downstream workers:

```bash theme={null}
curl -X POST https://api.adaline.ai/v2/logs/trace \
  -H "Authorization: Bearer $ADALINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "projectId": "your-project-id",
    "trace": {
      "name": "user-request",
      "status": "unknown",
      "referenceId": "req-abc-123",
      "startedAt": 1700000000000,
      "endedAt": 1700000001000,
      "tags": ["api-gateway"]
    },
    "spans": [
      {
        "name": "route-and-dispatch",
        "status": "success",
        "referenceId": "span-gateway-001",
        "startedAt": 1700000000000,
        "endedAt": 1700000001000,
        "content": { "type": "Function", "input": "{...}", "output": "{...}" }
      }
    ]
  }'
```

The response returns the Adaline-generated `traceId`. But downstream services don't need it — they can reference the trace by `traceReferenceId` instead.

### Step 2: Worker attaches a span

A downstream worker picks up the job, reads the shared `referenceId`, and attaches its span to the same trace:

```bash theme={null}
curl -X POST https://api.adaline.ai/v2/logs/span \
  -H "Authorization: Bearer $ADALINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "projectId": "your-project-id",
    "traceReferenceId": "req-abc-123",
    "span": {
      "name": "retrieval-worker",
      "status": "success",
      "referenceId": "span-worker-001",
      "startedAt": 1700000001500,
      "endedAt": 1700000003000,
      "content": { "type": "Retrieval", "input": "{...}", "output": "{...}" }
    }
  }'
```

### Step 3: Another service attaches more spans

A second worker handles the LLM call, again using the same `traceReferenceId`:

```bash theme={null}
curl -X POST https://api.adaline.ai/v2/logs/span \
  -H "Authorization: Bearer $ADALINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "projectId": "your-project-id",
    "traceReferenceId": "req-abc-123",
    "span": {
      "name": "llm-worker",
      "status": "success",
      "referenceId": "span-worker-002",
      "startedAt": 1700000003000,
      "endedAt": 1700000005000,
      "content": { "type": "Model", "provider": "openai", "model": "gpt-4", "input": "{...}", "output": "{...}" }
    }
  }'
```

### Step 4: Any service can update the trace

Once the full pipeline completes, any service can update the trace status or add metadata using the same `referenceId`:

```bash theme={null}
curl -X PATCH https://api.adaline.ai/v2/logs/trace \
  -H "Authorization: Bearer $ADALINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "projectId": "your-project-id",
    "referenceId": "req-abc-123",
    "logTrace": {
      "status": "success",
      "tags": ["completed"]
    }
  }'
```

<Tip>
  The `referenceId` is any string you control — a request ID, a job ID, a correlation ID from your existing infrastructure. Pass it between services however you like: HTTP headers, message queue payloads, job metadata, or environment variables.
</Tip>

## Best practices

* **One trace per user request** — Each user interaction (API call, chat message, workflow trigger) should be a single trace.
* **Descriptive span names** — Use names that indicate the operation: `"query-embedding"`, `"vector-search"`, `"response-generation"` — not `"step-1"`, `"step-2"`.
* **Always set final status** — Update trace and span status to `"success"` or `"failure"` before ending.
* **Use tags for filtering** — Add tags like `["production", "v1.3"]` to make it easy to filter in the [Monitor](/docs/monitor/analyze-log-traces).
* **Include variables** — Pass variable values on spans so they can be captured into [datasets](/docs/monitor/build-logs-from-dataset) for evaluation.

## Next steps

<CardGroup cols={2}>
  <Card title="Log User Feedback" icon="message-circle" href="/docs/instrument/log-user-feedback">
    Attach user feedback signals to traces.
  </Card>

  <Card title="Analyze Log Traces" icon="list-tree" href="/docs/monitor/analyze-log-traces">
    View your traces in Monitor.
  </Card>
</CardGroup>