Benchmarks: a 24-hour test of version R1
version R1, 24 hours (2026-10-23 20:00 to 2026-10-24 19:59, UTC-7), synthetic set of five request classes, no external incident window. All figures on this page come from that single test unless a block says otherwise. Providers are not named (Provider A, B, C and an insurance provider outside the Gonka network).
Result by request class
Test traffic: short streaming chat, short non-streaming, long context of about 20,000 tokens, JSON mode and tool calls, every prompt unique. "Served" means an HTTP 200 answer that is complete.
Standard (Gonka providers first, with an insurance provider outside the Gonka network behind them)
| Request class | Requests | Served | First text, median / 95th pct | Total time, median / 95th pct |
|---|---|---|---|---|
| golden | 500 | 100.00% (500 of 500) | 1.2 s / 5.8 s | 1.8 s / 32.2 s |
| json mode | 447 | 100.00% (447 of 447) | 1.1 s / 5.1 s | 3.4 s / 32.2 s |
| long ctx | 485 | 100.00% (485 of 485) | 2.4 s / 9.4 s | 7.2 s / 327.9 s |
| short nonstream | 947 | 100.00% (947 of 947) | n/a (not streaming) | 3.3 s / 17.1 s |
| short stream | 1903 | 100.00% (1903 of 1903) | 1.3 s / 7.2 s | 2.9 s / 52.4 s |
| tools | 490 | 99.39% (487 of 490) | 1.3 s / 14.5 s | 2.4 s / 33.2 s |
Super Deal (economical tariff, depends on the Gonka network, no insurance outside Gonka)
| Request class | Requests | Served | First text, median / 95th pct | Total time, median / 95th pct |
|---|---|---|---|---|
| golden | 500 | 99.80% (499 of 500) | 1.2 s / 5.6 s | 2.2 s / 42.8 s |
| json mode | 488 | 99.59% (486 of 488) | 1.0 s / 5.3 s | 1.4 s / 53.4 s |
| long ctx | 472 | 92.58% (437 of 472) | 3.5 s / 10.4 s | 5.9 s / 377.3 s |
| short nonstream | 902 | 99.89% (901 of 902) | n/a (not streaming) | 2.5 s / 55.1 s |
| short stream | 1981 | 99.85% (1978 of 1981) | 1.3 s / 6.4 s | 3.0 s / 54.4 s |
| tools | 496 | 96.98% (481 of 496) | 1.4 s / 15.2 s | 2.5 s / 33.9 s |
The 99th percentile of Standard first text is 9.5 s (an estimate from one daily cycle); of Super Deal (economical tariff) 12.6 s (the same estimate).
The gateway and calling the provider directly
The same request at the same moment, once through the gateway and once straight to the first provider of the line, alternating which goes first. Version R1, this 24-hour window:
| Line | Paired requests | Served through the gateway | Served by calling the provider directly | First text via gateway, median / 95th | First text direct, median / 95th |
|---|---|---|---|---|---|
| Standard | 300 | 300 of 300 | 300 of 300 | 1.4 s / 6.5 s | 1.3 s / 7.2 s |
| Super Deal (economical tariff, depends on the Gonka network, no insurance outside Gonka) | 300 | 297 of 300 | 300 of 300 | 1.5 s / 5.2 s | 1.3 s / 7.8 s |
In this window the provider itself was healthy, so availability was the same on Standard (300 of 300 pairs on both paths); on Super Deal (economical tariff) the gateway lost 3 of the 300 pairs that the direct call served, which is not statistically significant (p = 0.25). The tail of first-text time (95th percentile) is lower through the gateway on both lines; for Standard the 95% intervals overlap, for Super Deal (economical tariff) they are close, and we do not call either difference significant.
On a day when the provider was failing (earlier version, with a date)
The acceptance test of an earlier version (be46d49), 2026-10-20 to 2026-10-21 (UTC-7), another provider condition: the direct provider failed part of the requests, and the gateway's failover covered them.
| Line | Paired requests | Served through the gateway | Served by calling the provider directly |
|---|---|---|---|
| Standard | 600 | 600 of 600 (100.0%) | 487 of 600 (81.2%) |
| Super Deal (economical tariff, depends on the Gonka network, no insurance outside Gonka) | 600 | 588 of 600 (98.0%) | 493 of 600 (82.2%) |
Answer content, measured on a strict extraction set
20 extraction tasks, ten passes spread over the 24 hours; a reply counts only if it is exactly the reference JSON. Same period.
| Path | Replies | Exactly the reference | 95% interval |
|---|---|---|---|
| Through the gateway, streaming | 200 | 92.0% | 87.4-95.0% |
| Through the gateway, non-streaming | 200 | 90.0% | 85.1-93.4% |
| Calling the provider directly | 200 | 72.0% | 65.4-77.8% |
On a strict extraction set; the answer is clean, without extra text; the cause is known: a Gonka-network provider called directly appends a repeated copy after a short JSON answer, because the network requires at least 64 generated tokens per request (gonka-ai/gonka issue 1955), and the gateway's JSON clean-up removes that copy. It is not a general statement about answer quality.
Agent scenario
Four sequential tool calls and a final answer, one hundred runs per line spread over the 24 hours.
| Line | Runs completed | 95% interval | Whole scenario time, median / 95th |
|---|---|---|---|
| Standard | 99 of 100 | 94.6-99.8% | 24 s / 78 s |
| Super Deal (economical tariff, depends on the Gonka network, no insurance outside Gonka) | 90 of 100 | 82.6-94.5% | 26 s / 106 s |
9 of the 10 failed Super Deal (economical tariff) runs ended with an HTTP error from the gateway (all providers unavailable or timed out) and one stopped without a tool call; Standard had 1 failed run.
What we know about the weak spots
Hours are UTC-7.
- Super Deal (economical tariff) had weaker hours: 24.10 02:00 served 95.1% of its requests (184 requests in that hour) and 24.10 11:00 served 97.1%.
- Super Deal (economical tariff) long-context requests (about 20,000 tokens of input) timed out 33 times out of 472; the economical tariff has no insurance outside the Gonka network.
- Standard's slowest hour for first text: 24.10 06:00, 95th percentile 9.4 s.
- Tool calling: in the agent scenario 58 of 100 Standard runs contained a duplicated tool call and 23 of 100 Standard runs had service markup of the provider inside the text; both come from the providers and were deliberately left outside this release.
- Pauses inside a stream: the gateway log of the 24 hours shows 349 streams (3.7% of 9,477 stream requests) that stopped sending data for the idle limit of about eight to ten seconds; 334 of them were continued transparently on another provider, 15 ended with an explicit error event and were not billed.
Measurement from a client place (Canada)
A separate measurement on version R1 from a computer in Canada over the public Internet: one short streaming request per line about every minute for three hours, at night local time, so it is indicative only. The first attempt in the evening was lost when the computer restarted; this one is the repeat. The earlier run before R1 is shown for reference, a different evening and different provider conditions.
| Measurement | Line | Requests | Served | First text, median / 95th |
|---|---|---|---|---|
| R1, three hours at night (22:40-01:40, UTC-7) | Standard | 178 | 178 of 178 | 1.9 s / 7.7 s |
| R1, same run | Super Deal (economical tariff, depends on the Gonka network, no insurance outside Gonka) | 178 | 178 of 178 | 1.8 s / 6.3 s |
| Before R1, evening of 2026-10-21 | Standard | 151 | 151 of 151 | 2.0 s / 9.3 s |
| Before R1, same run | Super Deal (economical tariff) | 151 | 147 of 151 | 2.0 s / 7.9 s |
Why the median is now 1.4 s and not 375 ms
Medians published in September (as low as 13 ms) were helped by the providers' response cache: the same short prompts were repeated. The R1 test puts a unique number into every prompt, so no request can be answered from a cache; the median of 1.4 s is the honest figure for new prompts.
What changed since September
| Published in September 2026 (older version, other providers' condition) | Figure then | Why it no longer describes the service |
|---|---|---|
| 3-5 September, 1,440 paired requests, gateway against calling a provider directly | failure rate 27.3% with the gateway against 36.2% direct | older gateway version and a period when the providers' network was failing; the new test below is of the current version |
| 16-18 September, 1,000 requests per line | 100.0% and 98.9% served; median time to first text 197 ms and 216 ms | older version; the very low medians were helped by caching of repeated prompts (see below) |
| 30 September, 200 requests per line | 99.5% and 99.0% served; median 375 ms, 95th percentile 14.9 s (Standard) | short check, different prompt mix, older version |
The full September page is kept as an archive with its date and version; nothing was deleted.
Method
- Version R1 (code 2eb32a1, image c16-2eb32a1), frozen during the test. Window: 2026-10-23 20:00 to 2026-10-24 19:59, UTC-7; no external incident window opened, so nothing was excluded.
- Served = HTTP 200 with a complete answer (for streams a finish reason was seen). First text = time to the first content token of served streaming requests.
- Proportions with 95% Wilson intervals; percentiles with 95% bootstrap intervals; paired comparisons with the exact McNemar test.
- The 99th percentile is an estimate from one daily cycle. Providers are anonymised.
- Billing was reconciled request by request: every served request had exactly one billing record and the token counters equal the sum of the records.
- Not repeated for R1: accuracy at 200,000 tokens (only measured on the earlier version be46d49).
Data and scripts
Anonymised request-level data and the scripts that recompute every number are published in the repository github.com/privatedeskai/sagg-benchmark-data, folder 2026-10-24-R1-24h/ (CSV files and make_package.py). No keys, no provider names.