> ## Documentation Index
> Fetch the complete documentation index at: https://arizeai-433a7140-ehutt-trail-benchmark-new-tasks.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Completeness
> Assess whether every active user request in a conversation was actually completed.
## Overview
The **Completeness** evaluator classifies whether an assistant completed every
active request the user made over the course of a conversation. It measures
**finished work** — a delivered answer, a delivered artifact (including every
required component), or an action whose success is visible in the record — not
whether a request was merely acknowledged.
A conversation can finish its main task and still be incomplete when a
secondary request is dropped. For example, if the user asks the assistant to
reset a password **and** update a billing address, and the assistant only
resets the password, the conversation is incomplete.
Completeness does not judge correctness, quality, or whether completion was
appropriate. A delivered answer can still be complete if it is factually wrong.
A refusal, a clarifying question, or a report of a blocker is **not**
completion. Withdrawn requests are listed but excluded from the decision.
## Supported Levels
| Level | Supported | Notes |
| - | - | - |
| **Span** | Yes | Apply when a span contains the full conversation, including tool calls and results. |
| **Trace** | Yes | Useful when each trace is one conversation or agent run. |
| **Session** | Yes | Evaluate the whole session when intentions span multiple traces. |
**Relevant span kinds:** AGENT, CHAIN, and LLM spans that preserve multi-turn
conversation history and tool activity.
## Input Requirements
| Field | Type | Description |
| - | - | - |
| `conversation` | `string` | Full conversation record to judge, including turns, tool calls, and tool results. |
Include early-turn requests. For agent traces, include tool calls and tool
results in `conversation` so action success can be verified. If tools are
omitted, the judge falls back to the visible dialogue.
## Output Interpretation
| Property | Value | Description |
| - | - | - |
| `label` | `"complete"` or `"incomplete"` | Classification result |
| `score` | `1.0` or `0.0` | `1.0` means every non-withdrawn intention was completed |
| `explanation` | `string` | Judge reasoning, including an `INTENTIONS:` list |
| `direction` | `"maximize"` | Higher aggregate scores are better |
## Usage Examples
```python theme={null}
from phoenix.evals import LLM
from phoenix.evals.metrics import CompletenessEvaluator
evaluator = CompletenessEvaluator(
llm=LLM(provider="openai", model="gpt-4o-mini"),
temperature=0.0,
)
scores = evaluator.evaluate({
"conversation": (
"User: Reset my password and update the billing address.\n"
"Assistant: Your password has been reset."
),
})
print(scores[0])
# Score(name='completeness', score=0.0, label='incomplete', ...)
```
```typescript theme={null}
import { createCompletenessEvaluator } from "@arizeai/phoenix-evals";
import { openai } from "@ai-sdk/openai";
const evaluator = createCompletenessEvaluator({
model: openai("gpt-4o-mini"),
});
const result = await evaluator.evaluate({
conversation:
"User: Reset my password and update the billing address.\nAssistant: Your password has been reset.",
});
console.log(result);
// { score: 0, label: "incomplete", explanation: "..." }
```
## Using Input Mapping
Map your trace or dataset fields into the evaluator's `conversation` field.
```python theme={null}
input_mapping = {
"conversation": lambda row: render_messages(row["messages"]),
}
scores = evaluator.evaluate(dataset_row, input_mapping)
```
See [Input Mapping](/docs/phoenix/evaluation/concepts-evals/input-mapping) for
additional mapping options.
## Viewing and Modifying the Prompt
The default prompt is maintained in the
[classification evaluator config](https://github.com/Arize-ai/phoenix/blob/main/prompts/classification_evaluator_configs/COMPLETENESS_CLASSIFICATION_EVALUATOR_CONFIG.yaml).
Adapt it when your application has domain-specific notions of what counts as an
intention.
```typescript theme={null}
import { createCompletenessEvaluator } from "@arizeai/phoenix-evals";
const evaluator = createCompletenessEvaluator({
model,
promptTemplate: `Conversation: {{conversation}}
Did the assistant complete every active user intention?`,
choices: { complete: 1, incomplete: 0 },
});
```
## Configuration
For model and provider options, see
[Configuring the LLM](/docs/phoenix/evaluation/how-to-evals/configuring-the-llm).
Judge model choice changes accuracy on this evaluator; see [Benchmarks](#benchmarks)
for a comparison of `gpt-4o-mini`, `gpt-5.6`, and `claude-sonnet-4-6`.
## Using with Phoenix
* [Evaluating Phoenix Traces](/docs/phoenix/tracing/how-to-tracing/feedback-and-annotations/evaluating-phoenix-traces)
* [Logging LLM Evaluations](/docs/phoenix/tracing/how-to-tracing/feedback-and-annotations/llm-evaluations)
* [Using Evaluators in Experiments](/docs/phoenix/datasets-and-experiments/how-to-experiments/using-evaluators)
## Benchmarks
The default prompt was scored on a 45-example categorized synthetic suite. Every
model used the same prompt. Runs were local with `PHOENIX_TEST_TRACKING=false`.
See
[completeness.eval.ts](https://github.com/Arize-ai/phoenix/blob/main/js/benchmarks/evals-benchmarks/src/completeness.eval.ts)
for the example set.
| Model | n | Accuracy | Macro precision | Macro recall | Macro F1 | Misses |
| - | - | - | - | - | - | - |
| `gpt-4o-mini` | 45 | 0.84 | 0.83 | 0.84 | 0.84 | 7 |
| `gpt-5.6` | 45 | 1.00 | 1.00 | 1.00 | 1.00 | 0 |
| `claude-sonnet-4-6` | 45 | 0.91 | 0.94 | 0.88 | 0.90 | 4 |
On this suite, `gpt-5.6` matched gold on every example. The other models mainly
missed cases that mix completeness with correctness, tool evidence, or
multi-part asks:
* **Correctness vs completeness.** Gold treats a delivered answer as complete even
when it is wrong. `claude-sonnet-4-6` missed on these cases.
* **Claims without tool evidence.** Gold treats the visible reply as sufficient
when no matching tool record exists. `gpt-4o-mini` and `claude-sonnet-4-6` missed here.
* **Partial delivery.** `gpt-4o-mini` labeled several multi-part and
partial-tool cases complete.
Predicted labels where at least one model missed gold or the models disagreed
with each other. **Bold** means the judge disagreed with gold.
| Case | Gold | gpt-4o-mini | gpt-5.6 | claude-sonnet-4-6 |
| - | - | - | - | - |
| `[answered_despite_missing_context #0]` | `complete` | `complete` | `complete` | **`incomplete`** |
| `[answered_despite_missing_context #1]` | `complete` | `complete` | `complete` | **`incomplete`** |
| `[wrong_but_delivered #0]` | `complete` | `complete` | `complete` | **`incomplete`** |
| `[wrong_but_delivered #1]` | `complete` | **`incomplete`** | `complete` | `complete` |
| `[withdrawn_intention #0]` | `complete` | **`incomplete`** | `complete` | `complete` |
| `[claimed_but_not_done #1]` | `incomplete` | **`complete`** | `incomplete` | `incomplete` |
| `[claimed_without_tools #0]` | `complete` | **`incomplete`** | `complete` | **`incomplete`** |
| `[multipart_cases #1]` | `incomplete` | **`complete`** | `incomplete` | `incomplete` |
| `[multipart_cases #5]` | `incomplete` | **`complete`** | `incomplete` | `incomplete` |
| `[tool_partial #0]` | `incomplete` | **`complete`** | `incomplete` | `incomplete` |
## API Reference
* **Python:** [CompletenessEvaluator](https://arize-phoenix.readthedocs.io/projects/evals/en/latest/api/evals.html#module-phoenix.evals.metrics)
* **TypeScript:** [createCompletenessEvaluator](https://arize-ai.github.io/phoenix/modules/_arizeai_phoenix-evals.llm.html)
## Related
* [Correctness Evaluator](/docs/phoenix/evaluation/pre-built-metrics/correctness)
* [User Friction Evaluator](/docs/phoenix/evaluation/pre-built-metrics/user-friction)
* [Hallucination Evaluator](/docs/phoenix/evaluation/pre-built-metrics/hallucination)