Estimate the inference working set.
Logical weight storage plus transformer KV cache, a runtime allowance and reserved headroom. The per-device result assumes weights and cache are evenly sharded across the selected accelerators.
parameters × bytes per weight × (1 + metadata allowance)2 × layers × KV heads × head dimension × tokens × concurrency × bytesper-device working set ÷ (1 - reserve)Primary references: NVIDIA NIM memory guidance ↗ Hugging Face cache structure ↗