Reliability benchmarks

The current, largest sample to date: 1,000 real sequential requests per line, measured after every reliability fix live on this page (stream+non-stream connection pooling, the Tier1 provider recovery, the alert-escalation fix) – the honest current baseline, replacing the smaller 2026-09-12 sample below as the headline figure.

Success rate by line

Time-to-first-token distribution

TTFT percentiles: Standard mean 644ms, median 197ms, p90 776ms, p95 1821ms. Super Deal mean 416ms, median 216ms, p90 1154ms, p95 1568ms. The long tail on both lines (visible in the >1s buckets) is almost entirely failover latency – a request that had to try a second or third tier after the first one stalled, not a slow response from a healthy provider.

Which tier actually served the request

The cheap Primary tier serves the large majority of real traffic on both lines – failover exists and is measured working (see the tables below), but is the exception, not the rule, under this realistic, non-adversarial load pattern.

Super Deal's 11 failures: 10 empty_body (a tier connected but delivered no content before every fallback tier was also exhausted) and 1 timeout – both classes are described in Methodology below. Standard had zero failures in this sample.

A quick confirmation, not a new standalone measurement of the same statistical weight as the 1,000/line figure above: real changes landed on the Gonka network in the two weeks since (new models observed: GLM-5.3-Flash, MiniMax-M2.7), so this check verifies the headline figure still holds rather than re-measuring from scratch. At n=200/line, read this as "still consistent with" the baseline, not a replacement for it. It also used a different, heavier prompt mix (long-context and code-generation tasks alongside short Q&A, not the original light-only bank) – TTFT here is not directly comparable to the figures above for that reason, so only success rate and tier routing are compared below.

Which tier actually served the request

Honest finding, not smoothed over: Standard's Primary-tier share dropped from 94.5% to 80.4% this day. Confirmed directly in production logs – 78 occurrences of a connection-level timeout to joingonka-deepseek specifically (∼39 of the 200 requests affected, matching the gap in Tier1 share). The cascade's backup tiers absorbed this correctly – overall success stayed at 99.5% – but it's a real, if not severe, degradation worth disclosing plainly rather than only reporting the headline success rate. Super Deal did not show the same drop in the same window. TTFT for this check: Standard median 375ms/p95 14.9s, Super Deal median 488ms/p95 3.9s – both tails reflect the connection-timeout-then-failover pattern above, not a slow healthy response.

This same finding was disclosed in the current OpenRouter application update, drawing from the same measurement – the site and the application are kept consistent, not telling two different stories.

Raw data: 2026-09-30-recheck-200-per-line/ in the repository below.

Not directly comparable to Phase 3 below – different date, different code (several real fixes landed today), and a different methodology (4 concurrent legs per round: the gateway plus each of its 3 underlying providers, instead of one paired direct-vs-gateway request). Read this as the current baseline and Phase 3 as the historical one it superseded, not as a before/after delta on the same measurement.

¹107 of 180 attempted – 73 excluded because the test account ran low on quota partway through (a testing artifact, not a gateway failure; disclosed, not smoothed over). *Standard light's 3.0% includes 4 requests lost to a disclosed self-collision between two of our own concurrent test processes sharing one test key – not a production-traffic pattern; 1.0% excludes it.

Combined: 652/667 successful (97.8%) across both lines and both workload levels, after excluding that disclosed self-collision (raw, uncorrected: 648/667, 97.2%) – full incident-level detail in the repository below.

A real, disclosed finding worth reading in full rather than glossing over: 4 of Super Deal's 8 heavy-workload failures were a genuine 3-tier cascade exhaustion (all providers failed in the same round), traced to this test run overlapping with a scheduled internal monitoring cycle hitting the same underlying provider accounts – an operational interaction between our own tooling, not an independent reliability defect. Full trace and root-cause detail in the repository below.

The same four providers as the older per-provider benchmark further below, re-measured today after the same fixes:

A striking reversal from the per-provider snapshot further below (2026-09-05): every Gonka-network broker's failure rate has come down sharply, and none show the network-wide simultaneous degradation seen in that earlier window. ²10 of proxygonka's 15 failures here are the same disclosed self-collision noted above, likely inflating this specific number above its true baseline. ³gonkaapiorg-deepseek's own window ran ~10–20 minutes before the other three started – not the exact same concurrent snapshot; see the repository for the full caveat.

Each pair sends the same prompt at the same instant twice: once straight to proxygonka-deepseek (bypassing the gateway), once through the SAGG gateway on the same line. Pairing controls for prompt difficulty and provider conditions changing over the 3-day run. "Requests saved" = direct failed, gateway succeeded, same prompt, same instant.

All two splits are statistically significant, but Standard's benefit is much larger than Super Deal's – Super Deal's own backup tier is measurably less reliable than Standard's. Full incident-by-incident log and raw data: see the repository below.

25 direct (non-gateway) requests to each of the four providers individually – measures each provider's own standalone reliability in this window, independent of the gateway's own failover.

All three Gonka-network brokers showed elevated failure in the same test window – a real, network-wide condition, not one broker having an isolated bad day. deepinfra-deepseek, the one non-Gonka provider, stayed at 96% success. This is exactly the failure mode a non-Gonka fallback tier is designed to survive – see the trace below for a real request that hit it.

Raw output of SAGG's own per-request tracing tool, for a genuine, unforced production request on the Standard line – not staged or simulated. Tier1 timed out, Tier2 also timed out, Tier3 succeeded.

{
    "request_id": "097c128a313f865f",
    "line": "standard",
    "steps": [
        { "step": "tier_attempt_start",  "elapsed_ms": 0,     "detail": { "provider": "proxygonka-deepseek", "tier": "primary" } },
        { "step": "tier_attempt_result", "elapsed_ms": 3001,  "detail": { "provider": "proxygonka-deepseek", "tier": "primary", "outcome": "connection_error", "phase": "pre_generation" } },
        { "step": "tier_attempt_start",  "elapsed_ms": 3001,  "detail": { "provider": "eterial-deepseek",    "tier": "backup"  } },
        { "step": "tier_attempt_result", "elapsed_ms": 15002, "detail": { "provider": "eterial-deepseek",    "tier": "backup",  "outcome": "connection_error", "phase": "pre_generation" } },
        { "step": "tier_attempt_start",  "elapsed_ms": 15002, "detail": { "provider": "deepinfra-deepseek",  "tier": "backup"  } },
        { "step": "tier_attempt_result", "elapsed_ms": 15438, "detail": { "provider": "deepinfra-deepseek",  "tier": "backup",  "outcome": "success" } }
    ],
    "final_outcome": "success",
    "total_ms": 15439
}

Slow (15.4s end-to-end) – but it succeeded instead of failing outright. The full, unedited trace file is in the repository below, alongside 3 more real traces covering the healthy path and a single-failover case.

The 1,000/line sample above: real sequential requests against the live production gateway, one request roughly every 10 seconds per line (not a burst/concurrency stress test - that is measured separately and is not part of these headline numbers), drawn from three separate, non-overlapping clean windows, all after this session's Tier1-provider recovery and connection-pooling fixes, with the test account's balance confirmed healthy throughout: 150 requests (2026-09-16 21:08-21:25 UTC), 500 requests (2026-09-17 19:05-22:32 UTC), 350 requests (2026-09-18 08:14-10:37 UTC). A dynamic per-request SLA (15s + generation time scaled to the request's own requested output length) rather than one fixed cutoff, so a legitimately long generation is never misclassified as a timeout. Unlike Phase 3 and the 2026-09-12 update below, this sample has no paired direct-provider comparison leg (the direct-comparison credentials were not part of this particular test run) - it measures the gateway's own end-to-end reliability, latency, and which tier actually serves each request, not a gateway-vs-direct delta. One honest, disclosed methodology nuance: the first (150-request) window ran from a different network origin than the other two (a local machine vs. the VPS the gateway itself runs on) - this cannot affect success/failure or tier routing (both are purely server-side facts) but plausibly explains a modest amount of the difference between that window's own TTFT and the other two's, folded into the combined percentiles above.

A request counts as a failure on a non-2xx status, a connection error/timeout, or a stream that started and then stalled – a fixed 15-second SLA per leg, applied identically to both paths in every pair. Because Phase 3's two legs share a prompt and a moment (not independent samples), the correct significance test is McNemar's exact test on the discordant pairs, not a two-proportion test – a two-sided exact binomial test on the "direct-failed/gateway-succeeded" vs. "gateway-failed/direct-succeeded" split.

Provider reliability on the Gonka network moves fast – in a separate check, proxygonka-deepseek went from 68% failure to 0% failure (16/16 real gateway calls succeeded on Tier1) within roughly 10–15 minutes. No single number here is a permanent verdict on any provider; every measurement is timestamped and describes that window.

Every number on this page is computed directly from raw, request-level JSONL records – not aggregated or hand-picked. The full dataset, dependency-free Python scripts that recompute every figure from scratch, and all 4 real trace files are published at:

github.com/privatedeskai/sagg-benchmark-data

For the 1,000-requests-per-line figures at the top of this page specifically, see the 2026-09-18-1000-per-line/ folder.