alde.sh llm stats
connecting

Provider benchmark

waiting for data

Throughput

Output tokens divided by the time between the first and the last streamed chunk, summed over the turns of the run. Higher is better.
decode tokens per second
loading…

Time to first token

Seconds from sending the request until the first streamed chunk arrives. Reasoning tokens count, because that is when the provider starts working. Lower is better.
seconds until the first token
loading…

Task time

Wall clock for the entire agent loop: four tool calls and the final written summary. Lower is better.
seconds for the whole task
loading…

Network latency

Average ICMP round trip to the endpoint host, from 5 echoes sent before each run. It measures the network path, not the model. Lower is better.
ICMP round trip to each endpoint
loading…

Latest tick

Benchmark

Methodology task, prompts, tools and how each number is measured

The task

Every run sends the same simulated SRE incident to the provider and lets the model drive a five-turn agentic loop: four tool calls in a fixed order, then a short written summary. The tools are deterministic mocks backed by canned data, so each provider sees exactly the same tool responses and only the model behind the endpoint varies.

Request parameters

  • Model: deepseek-v4.1-flash on every arm (same weights, different endpoints).
  • temperature 0.2, reasoning_effort high, max_tokens 4000, streaming with usage reporting.
  • CommandCode ZDR sends x-cmd-zdr: 1; the no-ZDR arm is the same endpoint without that header.
  • OpenCode Go requires a per-run x-opencode-session header; the native DeepSeek API needs the reasoning content echoed back on tool-call turns.

Metrics

  • Throughput (TPS) — output tokens divided by the time between the first and the last streamed chunk of the run, summed over the turns.
  • Time to first token — seconds from request to the first streamed chunk (reasoning counts, because that is when the provider starts working).
  • Task time — wall clock for the whole loop, from the first request to the last byte.
  • Runs and failures — a run that returns no tokens (timeout, HTTP error, empty stream) is counted as a failure, drawn as a mark at the baseline and kept out of the medians.
  • Provider cards — the big number is the median of the per-tick medians over the selected window (24h by default), not a simple average, so one noisy tick cannot drag it. The bar compares that TPS with the fastest provider in the same window.

Network latency (ICMP)

Before the matrix, every endpoint host answers 5 ICMP echoes (200 ms apart) and the panel plots the average round trip. CommandCode ZDR and the no-ZDR arm reach the same host, so they share one line: the panel has three series, not four. Total packet loss becomes a failure mark at the baseline and reads "no response" in the tooltip.

Cadence and aggregation

An hourly system cron runs each provider twice, reversing the order on the second pass so a positional effect cancels out. Each chart point is the median of those two runs. The series never expires: the collector has no deadline, and the page reads it and refreshes every five minutes. The x axis is real time, so a run that happened between two hourly ticks sits exactly where it happened instead of taking a full slot.

Score (0 to 5)

  • +1 for calling the four tools in the required order.
  • +1 for severity: critical in the findings.
  • +1 for at least three remediation actions.
  • +1 for a summary of at most 140 words.
  • +1 for quoting the disk percentage and the ticket number.
  • −1 for using an em dash in the summary.

System prompt verbatim

You are an SRE agent. Investigate the incident using the tools, in this order: get_host_metrics, get_disk_usage, check_backup_status, submit_findings. Never invent data: use only what the tools return. After submit_findings, write a summary in English with at most 140 words, quoting the disk percentage and the ticket number. Do not use em dashes.

User prompt verbatim

Host web-03 is showing high latency and disk alerts. Diagnose it and open the ticket.

Tools exposed to the model

get_host_metrics host: string

CPU, load average, memory, iowait, disk percentage and root filesystem of the host.

get_disk_usage host: string, path: string

Per-directory consumption plus free space on the mount point.

check_backup_status host: string

Last successful backup, last attempt, status and error message.

submit_findings severity, root_cause, actions[], eta_minutes

Registers the diagnosis and opens the ticket; must be called before the text summary.

Mock data returned

  • get_host_metrics → cpu 71%, iowait 38%, disk 94%, 800 GB root with 48 GB free.
  • get_disk_usage → /var/lib/docker 612 GB, /var/log 118 GB, /srv/backups 44 GB, 31 GB free.
  • check_backup_status → last success 2026-11-08, last attempt failed with No space left on device.
  • submit_findings → {"ok": true, "ticket": "INC-8842"}.

This is a private benchmark of endpoints used by this homelab, not an official provider benchmark. Same model, same prompts, same mock tools: the differences come from the serving path, the queue and the network in front of each endpoint.