| OS and kernel | Linux 6.17.0-1022-azure |
|---|---|
| CPU model | AMD EPYC 7763 64-Core Processor |
| Host logical cores | 4 |
| Effective cores | 4 |
| Affinity cores | 4 |
| Allowed affinity | 0-3 |
| Pinned CPU | 0 |
| CPU quota | |
| Host RAM | MemTotal: 16373452 kB |
| Available RAM | MemAvailable: 15189052 kB |
| Memory limit | |
| Test DB filesystem | type=ext4 total_kib=151263856 available_kib=86942268 |
| Rust | rustc 1.101.0-nightly (4ebd25825 2026-10-15) |
| SQLite | 3.53.2 |
| Engine revision | 6b1dcda388ef6e57acf2f07eb7b91852f426e5ca |
| Dirty source | false |
Scanner driver: scheduler
One CPU; file-backed SQLite with real migrations, WAL and synchronous=NORMAL. Plain custody and the production scanner run against a deterministic scripted daemon while independent SQLite and HTTP status readers plus an admin writer issue background work. A separate connection holds a SQLite write lock for a fixed interval each tick. The capacity sweep uses immediate RPC responses; the fault points below add scheduled RPC delay and failures, and a longer blocked writer followed by a drain period. A blocked writer is a contention proxy, not a real disk fault. Capacity below is observational for this workload and machine.
Observed capacity bracket: at least 128 tenants sustainable; no overload observed. Workload: open orders per tenant as recorded in run.json plus one tenant with 300 open orders, one transaction per block, one new block per tick, 2 database readers, 2 HTTP status readers, 1 admin writer, a 25 ms SQLite write lock per tick, 5 second tick budget, at most three blocks of oldest lag. Only this machine and schedule were tested.
| Tenants | Status | Measured time | Tick p50/p95/p99 | Timer delay | DB reads/writes | HTTP reads | Max DB read/write latency | Max queue wait read/write | Max query read/write | Max HTTP latency | WAL pending pages | Lag after drain | Raw data |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 32 | sustainable | 184 ms | 46/46/46 ms | 1845 µs | 80/19 | 42 | 1219/33531 µs | n/a/33453 µs | n/a/33572 µs | 1193 µs | 0 | 0 | JSON |
| 64 | sustainable | 221 ms | 55/56/56 ms | 2079 µs | 96/28 | 50 | 1301/33915 µs | n/a/33460 µs | n/a/33591 µs | 2620 µs | 0 | 0 | JSON |
| 128 | sustainable | 298 ms | 74/75/75 ms | 2055 µs | 130/40 | 68 | 1662/34488 µs | n/a/32943 µs | n/a/33940 µs | 3232 µs | 0 | 0 | JSON |
Blue: network high-water; orange: slowest tenant cursor. The gap is work still to catch up.
| Fault | Result | Fault detail | Final slowest cursor | Raw data |
|---|---|---|---|---|
| transient daemon RPC failures under SQLite contention | recovered | 6 injected RPC failures | 8 | JSON |
| blocked SQLite writer, then recovery | recovered | 1500 ms per tick for two ticks | 8 | JSON |
| single custody scan slot under concurrent tenant scans | recovered | 128 scans; max slot wait 107843 µs | 8 | JSON |
The custody fault uses one scan slot with a scripted 5 ms service delay while retaining real plain-custody matching. Actual slow disk commands and process-kill recovery are not measured by this fixture. The RPC fault point uses deterministic transient failures and a drain period; the capacity sweep has no RPC delay.