Why this reading
A long-horizon agent needs more than a final-answer score.
The fresh workflow-aware benchmarking paper argues that real agents must preserve state, cite evidence, survive interruptions, and decide when to hand work back to a human.
For SP automation, those behaviors can matter as much as whether the agent eventually produces a correct sentence or code block.
Reading order
Your 30-minute plan.
Find state continuity, evidence traceability and escalation in the abstract.
Read episode-level evaluation, workflow state and cost-sensitive escalation scoring.
Compare protocol-level scoring with executable agent environments and end-state success.
Translate the benchmark into reproducibility, traceability, standards and measurable impact.
Design one multi-step benchmark episode for an SP reviewer.
Open-access sources
Fresh benchmark methodology plus executable-agent and CDISC context.
Brief background
Evaluate the path, not only the endpoint.
Static benchmarks usually provide all relevant context at once and score a single final answer. An agent workflow is different: information arrives over time, the agent uses tools, state changes, interruptions occur, and some cases should be escalated.
The main protocol therefore evaluates complete episodes and separates model, agent and simulated-workflow behavior. State continuity, evidence traceability and escalation are explicit evaluation targets.
HealthAgentBench adds executable environments. Its 54 tasks across seven categories require agents to inspect raw clinical artifacts and act through tools. The strongest reported agent achieved only about 42% task success, showing how much harder agent-native evaluation is than conventional QA.
CDISC adds the regulatory-development lens: reproducibility, edge-case handling, human-in-the-loop validation, standards integration, traceability and measurable impact all belong in the evaluation target.
For SP agents, a good benchmark episode should test: context → evidence retrieval → semantic interpretation → deterministic tool use → state update → evidence-linked decision → escalation or completion.
Key vocabulary
Fifteen terms for agent evaluation.
| Term | 中文 | Meaning / use |
|---|---|---|
| workflow-aware benchmark | 工作流感知基准 | An evaluation that tests an agent inside a multi-step workflow rather than on isolated prompts. |
| episode-level evaluation | 回合级 / 情境级评估 | Scoring a complete sequence of actions, state changes, evidence use, and handoffs as one unit. |
| state continuity | 状态连续性 | The ability to preserve and correctly update relevant information across multiple workflow steps. |
| evidence traceability | 证据可追溯性 | The ability to link a conclusion or action to the source evidence that supports it. |
| escalation decision | 升级处理决策 | A decision about when the agent should stop autonomous work and route a case to a human. |
| cost-sensitive evaluation | 成本敏感评估 | Scoring that assigns different costs to different types of errors, such as missed escalation versus unnecessary escalation. |
| handoff | 任务交接 | The transfer of responsibility or context from one role, agent, or reviewer to another. |
| longitudinal state | 纵向状态 | Information that accumulates or changes over time rather than existing in a single static snapshot. |
| workflow interruption | 工作流中断 | An unexpected event that forces the system to pause, revise, or redirect its current plan. |
| agent-native evaluation | 智能体原生评估 | Evaluation designed for agents that plan, use tools, inspect environments, and act over multiple steps. |
| task success rate | 任务成功率 | The proportion of complete tasks that satisfy the benchmark's end-state success criteria. |
| hidden ground truth | 隐藏金标准 | Reference answers or success criteria withheld from the agent during execution. |
| environment exploration | 环境探索 | The agent's process of inspecting files, data, tools, or system state before acting. |
| compositional reasoning | 组合式推理 | Combining multiple pieces of information and intermediate decisions to solve a complex task. |
| intermediate evaluation layer | 中间评估层 | A validation stage between static offline benchmarks and prospective real-world workflow studies. |
Useful phrases
Language for benchmark and validation discussions.
- move beyond static question answering - A production-oriented benchmark should move beyond static question answering.
- score the complete episode rather than the final answer alone - The benchmark scores the complete episode rather than the final answer alone.
- preserve state across interruptions and handoffs - The agent must preserve state across interruptions and handoffs.
- link every recommendation to inspectable evidence - Every recommendation should be linked to inspectable evidence.
- penalize missed escalation more heavily than unnecessary escalation - A safety-focused benchmark may penalize missed escalation more heavily than unnecessary escalation.
- evaluate tool use inside an executable environment - The system should be evaluated through tool use inside an executable environment.
- separate model capability from workflow behavior - The protocol separates model capability from workflow behavior.
- define success at the end of the workflow - Success should be defined at the end of the workflow.
- measure failure modes by workflow stage - Teams should measure failure modes by workflow stage.
- treat the benchmark as a bridge to prospective evaluation - The benchmark should be treated as a bridge to prospective evaluation.
Comprehension
Five questions.
- Why is final-answer accuracy insufficient for a long-horizon SP agent?
- What does state continuity mean in an ADaM or TFL review workflow?
- Why should missed and unnecessary escalation carry different costs?
- What does the HealthAgentBench success rate reveal about current agent systems?
- How do CDISC judging criteria broaden the benchmark beyond model accuracy?
Retelling
Say it three times.
- 30 seconds · Define workflow-aware benchmarking and name three behaviors.
- 45 seconds · Context → retrieve → interpret → tool → update state → evidence → decision.
- 60 seconds · Explain why escalation and traceability should be benchmarked before prospective deployment.
5-minute output task
Build one benchmark episode for an SP AI reviewer.
- Minute 1: Choose SAS log, SDTM mapping, ADSL/ADAE review, TFL reconciliation or code generation.
- Minutes 2-3: Create initial context, new evidence, a deterministic tool requirement and an ambiguity.
- Minute 4: Score correctness, state continuity, traceability, tool use and escalation separately.
- Minute 5: State the minimum benchmark evidence required before prospective deployment.
One sentence to keep
A useful benchmark for an SP agent should test not only whether it reaches the right answer, but whether it preserves context, uses the right evidence and tools, escalates uncertainty appropriately, and leaves a reviewable path to the result.