LLM inference: VRAM and performance calculatorComfyUI image and video VRAM calculator

Your workload

Workload

Precision of the weights held in memory.

Longer contexts make cache precision more important.

Runtime settings32K-cache reference

Your hardware

Input settings

Sequences per batch. Serving concurrency and throughput require separate measurements.

Advanced runtime settings
Advanced · Model architecture
Layers
32
Attention heads
32
KV heads
8
Head dimension
128
Hidden size
4,096
Maximum context (tokens)
131,072
View model configuration

Performance and memory results

29.5%Memory use

Memory estimate

7.1 GiB needed

Fits16.9 GiB to spare24.0 GiB available

The estimated workload fits with the safety margin included.

Estimated GPU rental · $0.44/h · Per-GPU quote × devices

Generation speed
~74 token/s
Time per token
~13.6 ms
Time to first token
~0.528 s
Llama 3.1 8B Instruct

RTX 4090 24 GiB

Q4_K_M · FP16 KV

Model specifications
Memory breakdownGiB

Vertical line = usable hardware memory

  • Weights (Q4_K_M)4.58 GiB
  • KV cache (FP16)1.00 GiB
  • Runtime overhead0.75 GiB
  • CUDA Graph capture0.00 GiB · 0.0%
  • Safety margin0.76 GiB
Framework overhead breakdown

Planning allocations. Actual driver, library and allocator use varies by runtime.

Device runtime and driver
0.50 GiB
Operator workspaces
0.20 GiB
CUDA Graph
0.00 GiB
Allocator base
0.05 GiB
Other runtime buffers and communication
0.00 GiB
Compare context lengths
Compare GPU capacity

Inference simulation

Llama 3.1 8B Instruct · Q4_K_M · RTX 4090 24 GiB

Prompt processing
0.53 s
Token generation
1.72 s
Estimated duration
2.25 s
Ready
Timing playback · sample text0 / 128 · 0 s

Press play to preview how the sample streams.

Measured latency percentiles and request throughput

Frequently asked questions

How much VRAM does a 7B/8B model need at Q4_K_M?

As a concrete 8B example, Llama 3.1 8B at Q4_K_M, 8,192 tokens, batch 1 and FP16 KV cache needs an estimated 7.1 GiB including headroom. Other 7B/8B architectures and context lengths differ.

Why is the estimate larger than the model download?

GPU memory includes weights, KV cache or activations, runtime buffers and safety headroom. A file size alone does not represent the whole workload.

How much safety headroom is included?

We add 12% to the estimated subtotal. Green means the total fits; yellow means only the subtotal or a suggested lighter configuration fits; red means neither fits under our fallback checks.

Can I combine memory across multiple GPUs?

We sum usable memory across the selected count and include additional per-GPU overhead for LLMs. Your backend must support splitting the workload. ComfyUI requires component placement across cards, such as MultiGPU nodes; each component must fit individually.

Are GB and GiB the same here?

Memory estimates use GiB. One GiB is 1,073,741,824 bytes and one decimal GB is 1,000,000,000 bytes. Hardware names retain their published labels. When the catalog supplies only a nominal capacity, the calculator uses that number as the capacity in GiB.

Calculator changelog

All versions →
  • Inference now supports draft models, weight offload by layer or percentage, multi-GPU links and throughput estimates when no measurement is available.
  • Memory allocations use GiB. On unified-memory devices, the fit check includes both resident allocations and memory offloaded to the host.
  • The precision selector and advanced settings now use the same searchable menu as the model selector. Use the arrow keys to move between options and Enter to choose one.
  • Image and video workflows now show the memory peak at different resolutions. Set denoising steps and enter a measured step time to estimate sampling duration.