Provider benchmark
Throughput
Output tokens divided by the time between the first and the last streamed chunk, summed over the turns of the run. Higher is better.Time to first token
Seconds from sending the request until the first streamed chunk arrives. Reasoning tokens count, because that is when the provider starts working. Lower is better.Task time
Wall clock for the entire agent loop: four tool calls and the final written summary. Lower is better.Network latency
Average ICMP round trip to the endpoint host, from 5 echoes sent before each run. It measures the network path, not the model. Lower is better.Latest tick
Benchmark
Methodology task, prompts, tools and how each number is measured
The task
Every run sends the same simulated SRE incident to the provider and lets the model drive a five-turn agentic loop: four tool calls in a fixed order, then a short written summary. The tools are deterministic mocks backed by canned data, so each provider sees exactly the same tool responses and only the model behind the endpoint varies.
Request parameters
- Model:
deepseek-v4.1-flashon every arm (same weights, different endpoints). temperature 0.2,reasoning_effort high,max_tokens 4000, streaming with usage reporting.- CommandCode ZDR sends
x-cmd-zdr: 1; the no-ZDR arm is the same endpoint without that header. - OpenCode Go requires a per-run
x-opencode-sessionheader; the native DeepSeek API needs the reasoning content echoed back on tool-call turns.
Metrics
- Throughput (TPS) — output tokens divided by the time between the first and the last streamed chunk of the run, summed over the turns.
- Time to first token — seconds from request to the first streamed chunk (reasoning counts, because that is when the provider starts working).
- Task time — wall clock for the whole loop, from the first request to the last byte.
- Runs and failures — a run that returns no tokens (timeout, HTTP error, empty stream) is counted as a failure, drawn as a mark at the baseline and kept out of the medians.
- Provider cards — the big number is the median of the per-tick medians over the selected window (24h by default), not a simple average, so one noisy tick cannot drag it. The bar compares that TPS with the fastest provider in the same window.
Network latency (ICMP)
Before the matrix, every endpoint host answers 5 ICMP echoes (200 ms apart) and the panel plots the average round trip. CommandCode ZDR and the no-ZDR arm reach the same host, so they share one line: the panel has three series, not four. Total packet loss becomes a failure mark at the baseline and reads "no response" in the tooltip.
Cadence and aggregation
An hourly system cron runs each provider twice, reversing the order on the second pass so a positional effect cancels out. Each chart point is the median of those two runs. The series never expires: the collector has no deadline, and the page reads it and refreshes every five minutes. The x axis is real time, so a run that happened between two hourly ticks sits exactly where it happened instead of taking a full slot.
Score (0 to 5)
- +1 for calling the four tools in the required order.
- +1 for
severity: criticalin the findings. - +1 for at least three remediation actions.
- +1 for a summary of at most 140 words.
- +1 for quoting the disk percentage and the ticket number.
- −1 for using an em dash in the summary.
System prompt verbatim
You are an SRE agent. Investigate the incident using the tools, in this order: get_host_metrics, get_disk_usage, check_backup_status, submit_findings. Never invent data: use only what the tools return. After submit_findings, write a summary in English with at most 140 words, quoting the disk percentage and the ticket number. Do not use em dashes.
User prompt verbatim
Host web-03 is showing high latency and disk alerts. Diagnose it and open the ticket.
Tools exposed to the model
get_host_metrics
host: string
CPU, load average, memory, iowait, disk percentage and root filesystem of the host.
get_disk_usage
host: string, path: string
Per-directory consumption plus free space on the mount point.
check_backup_status
host: string
Last successful backup, last attempt, status and error message.
submit_findings
severity, root_cause, actions[], eta_minutes
Registers the diagnosis and opens the ticket; must be called before the text summary.
Mock data returned
get_host_metrics→ cpu 71%, iowait 38%, disk 94%, 800 GB root with 48 GB free.get_disk_usage→/var/lib/docker612 GB,/var/log118 GB,/srv/backups44 GB, 31 GB free.check_backup_status→ last success 2026-11-08, last attempt failed withNo space left on device.submit_findings→{"ok": true, "ticket": "INC-8842"}.
This is a private benchmark of endpoints used by this homelab, not an official provider benchmark. Same model, same prompts, same mock tools: the differences come from the serving path, the queue and the network in front of each endpoint.