How to Measure Mercury 3 Latency Beyond Tokens per Second

Eva Wong

IceWhale author

Eva Wong is the Technical Writer and resident tinkerer at ZimaSpace. A lifelong geek with a passion for homelabs and open-source software, she specializes in translating complex technical concepts into accessible, hands-on guides. Eva believes that self-hosting should be fun, not intimidating. Through her tutorials, she empowers the community to demystify hardware setups, from building their first NAS to mastering Docker containers.

Mercury 3 official announcement cover artwork

Measure Mercury 3 from the moment a task starts until its output is usable. Record first-response time, full-response time, and total time including tools. Tokens per second describes part of generation; it can miss the wait before output and the work needed after the model finishes.

Use one small fixture with explicit acceptance criteria, and keep the prompt, input files, tool environment, and model settings recorded. That is enough to establish basic behavior and expose bottlenecks. Expand the trial only when a failure or a close decision needs more evidence.

Which timestamps should I collect?

Timestamp or duration	What it tells you
Request sent	The client-side starting point
First content received	Perceived responsiveness, when streaming applies
Complete response received	Full request duration
Tools completed	Execution time outside the model
Artifact accepted	Time until the task actually meets requirements

Use a monotonic client timer for durations and note whether streaming is enabled. Keep connection setup and reused-connection conditions consistent if comparing providers. Do not infer first-token behavior from a non-streaming request.

How do I choose a fair test prompt?

Choose a real, bounded task: extracting specified fields from an approved document or producing a small change in a copied repository. Keep the input and required output identical across runs. Record the model identifier, effort setting, and output limit so the results remain interpretable.

The Mercury 3 overview explains why vendor throughput figures are not a replacement for this task-level measurement.

How do tools affect the result?

Separate model waiting from tool execution. A slow NAS read, page navigation, or external search can dominate a multi-step task. Count retries, failed tool arguments, and repeated file reads rather than ignoring the time consumed by unsuccessful attempts.

Use the agent harness checklist to identify which component owns each delay. If the model call is a small fraction of the job, improving storage access or the task design may matter more than changing generation architecture.

How many runs do I need?

One run is useful for connectivity and obvious correctness problems. A few consistent repetitions can expose large practical differences, but they do not justify precise reliability claims or stable tail percentiles. Collect more observations only if the deployment stakes or a close performance decision warrant them.

Always keep failures visible. A fast successful run selected from several failed attempts is not representative evidence. The focused workflow check guide provides a lightweight acceptance approach.

What should I report beside latency?

Report whether the artifact passed review, relevant usage charges, tool calls, and manual corrections. Include the environment and date. If a run timed out, record it as a timeout rather than assigning it an attractive speed score.

For NAS files, preserve the fixture and accepted output in separate directories. Use the architecture comparison to decide what the measurements mean, and the file preparation guide to keep the pilot’s data exposure bounded.