Enter the 143-bit cryptographic access key (or open with ?key=...) to unlock the empirical telemetry dashboard and interactive load simulator.
Empirical determination of optimal Ninja -l / user_load_average limits from live cgroup v2, PSI, and build progress telemetry.
Slide across 25-unit empirical load bins (112 to 738) to inspect real measured host throughput, per-build velocity, CPU core saturation, and stall rates.
In production /opt/ninja (src/build.cc:262), max_load_average (-l or Redis user_load_average:<user>) only gates new local-pool jobs when GetLoadAverage() >= max_load_average. Ready remote RBE edges (remote_pool / default pool) and critical-path local code generators continue launching unimpeded.
reproxy CAS hashing, gRPC streams, and kernel I/O on the 192-thread server.rewrapper processes and ~300β400 local compilation/linking threads across 4β5 concurrent builds, 125 active cores naturally generate a Linux load average of 420β460.Load / Active Core explodes from 3.6x to 8.5x, starving reproxy and collapsing RBE compile throughput by 89%.# models/build_monitor_components/resource_controller.py:325
# Replaces {"standard": 700.0, "upper_mid": 650.0, "mid": 600.0, "low": 550.0}
tier_load_map = {
"standard": 460.0, # Priority SLA: top sweet-spot knee
"upper_mid": 430.0, # Standard SLA: yields before Priority
"mid": 400.0, # Flex SLA: yields at physical saturation
"low": 360.0, # Starter SLA: opportunistic local slots
}
25-unit granular resolution across 23,224 active 8-second compilation windows. Shows peak contended server capacity at load 300β450 (~305β312 actions/sec), minimum stall rate at 350β450 (4.0%β4.5%), and the severe stall cliff above 480.
Separates Phase 1 (10β75%: 70% RBE Remote Compile), Phase 2 (75β95%: Mixed Link / Rust / Metalava / R8), and Phase 3 (95β99.8%: Local Packaging). Phase 2 peaks at load 350β450 (122β131 act/s) before all three phases collapse past 480.
Demonstrates why load > 480 is retrograde: actual CPU cores utilized by user slices peak at 115β127 mean cores (163β165 P90 cores) at load 350β460, then fall to 85 cores at load 680 as threads pile up in uninterruptible sleep (D-state) and memory PSI.
Across 232 completed full AOSP builds (β₯120k actions) and 668 large build launches: builds below P75 load 480 finish in ~45 min median (0%β5.4% abort rate), whereas builds in the 650+ regime drag to 85.1 min median (119.9 min P90) with a 36.3% unexpected stop rate.
Controlling for exact build concurrency confirms that the 460β480 cliff is not a workload-mix artifact: for every concurrency level from 2 to 6+ builds, aggregate server output peaks inside 250β450 and degrades above 480.
Left: Empirical split between concurrent remote RBE actions and local jobs. Right: Comparison of previous load limits (550β700, all above the thrashing cliff) against recommended sweet-spot ceilings (360β460).
End-to-end control flow showing how Redis dynamic limits, /opt/ninja Plan::FindWork(), and Linux kernel runqueues interactβallowing 70% remote RBE edges to bypass the load gate at 440.0 while deferring local pool launches before the 480 cliff.
Left: Median build velocity (actions/sec) across 8 progress stages and 9 system load bands. Right: Build stall probability (% < 5 act/s) across the same 2D phase Γ load grid.
Shows the full quartile and density distribution of per-build compile speed (left) and contended host aggregate output (right) from Light (<200) through Sweet Spot (350β450) to Old 680 Zone (580β750+).
Side-by-side comparison of two 225,000-action AOSP builds from live_build_tracing.db: a build under the 440 regime finishes its main Ninja graph in 38.8 minutes with zero stalls, whereas a build exposed to load 600β720 drags for 67.9 minutes.
Left: Normalized 6-axis radar comparison of Load 440, 350, 550, and 680. Right: Pareto Efficiency Frontier mapping Contended Host Output (X) vs. Per-Build Speed (Y), sized by Active CPU Cores and colored by System Load.
Tracks host kernel context switches/sec, active cgroup processes/threads (14,000+ above load 525), disk write bandwidth (MB/s), and host RAM utilization vs. Memory PSI across 25-unit load bins.
| Load Band | 8s Windows | Contended Host Mean (act/s) | Contended Host P50 (act/s) | Per-Build Compile Mean (act/s) | Per-Build Compile P50 (act/s) | Stall Rate (<5 act/s) | Crawl Rate (<25 act/s) | Active User Cores | Load / Core Ratio | Host CPU PSI | Host Mem PSI |
|---|