Reliability benchmarks

Each pair sends the same prompt at the same instant twice: once straight to proxygonka-deepseek (bypassing the gateway), once through the SAGG gateway on the same line. Pairing controls for prompt difficulty and provider conditions changing over the 3-day run. "Requests saved" = direct failed, gateway succeeded, same prompt, same instant.

All two splits are statistically significant, but Standard's benefit is much larger than Super Deal's – Super Deal's own backup tier is measurably less reliable than Standard's. Full incident-by-incident log and raw data: see the repository below.

25 direct (non-gateway) requests to each of the four providers individually – measures each provider's own standalone reliability in this window, independent of the gateway's own failover.

All three Gonka-network brokers showed elevated failure in the same test window – a real, network-wide condition, not one broker having an isolated bad day. deepinfra-deepseek, the one non-Gonka provider, stayed at 96% success. This is exactly the failure mode a non-Gonka fallback tier is designed to survive – see the trace below for a real request that hit it.

Raw output of SAGG's own per-request tracing tool, for a genuine, unforced production request on the Standard line – not staged or simulated. Tier1 timed out, Tier2 also timed out, Tier3 succeeded.

{
    "request_id": "097c128a313f865f",
    "line": "standard",
    "steps": [
        { "step": "tier_attempt_start",  "elapsed_ms": 0,     "detail": { "provider": "proxygonka-deepseek", "tier": "primary" } },
        { "step": "tier_attempt_result", "elapsed_ms": 3001,  "detail": { "provider": "proxygonka-deepseek", "tier": "primary", "outcome": "connection_error", "phase": "pre_generation" } },
        { "step": "tier_attempt_start",  "elapsed_ms": 3001,  "detail": { "provider": "eterial-deepseek",    "tier": "backup"  } },
        { "step": "tier_attempt_result", "elapsed_ms": 15002, "detail": { "provider": "eterial-deepseek",    "tier": "backup",  "outcome": "connection_error", "phase": "pre_generation" } },
        { "step": "tier_attempt_start",  "elapsed_ms": 15002, "detail": { "provider": "deepinfra-deepseek",  "tier": "backup"  } },
        { "step": "tier_attempt_result", "elapsed_ms": 15438, "detail": { "provider": "deepinfra-deepseek",  "tier": "backup",  "outcome": "success" } }
    ],
    "final_outcome": "success",
    "total_ms": 15439
}

Slow (15.4s end-to-end) – but it succeeded instead of failing outright. The full, unedited trace file is in the repository below, alongside 3 more real traces covering the healthy path and a single-failover case.

A request counts as a failure on a non-2xx status, a connection error/timeout, or a stream that started and then stalled – a fixed 15-second SLA per leg, applied identically to both paths in every pair. Because Phase 3's two legs share a prompt and a moment (not independent samples), the correct significance test is McNemar's exact test on the discordant pairs, not a two-proportion test – a two-sided exact binomial test on the "direct-failed/gateway-succeeded" vs. "gateway-failed/direct-succeeded" split.

Provider reliability on the Gonka network moves fast – in a separate check, proxygonka-deepseek went from 68% failure to 0% failure (16/16 real gateway calls succeeded on Tier1) within roughly 10–15 minutes. No single number here is a permanent verdict on any provider; every measurement is timestamped and describes that window.

Every number on this page is computed directly from raw, request-level JSONL records – not aggregated or hand-picked. The full dataset, a dependency-free Python script that recomputes the McNemar test from scratch, and all 4 real trace files are published at:

github.com/privatedeskai/sagg-benchmark-data