This chart showcases a range of benchmarks for GPU performance while running large language models like LLaMA and Llama-2, using various quantizations. The data covers a set of GPUs, from Apple Silicon M series chips to Nvidia GPUs, helping you make an informed decision if you’re considering using a large language model locally.

To switch GPUs on or off, use to the legend below the graph. When you hover over a data point, you’ll see additional details about each model, such as an estimated system price.

Memory Boundary Conditions for GPUs Running Large Language Models

Our benchmarks emphasize the crucial role of VRAM capacity when running large language models. Even if a GPU can manage specified model sizes and quantizations—for instance, a context of 512 tokens—it may struggle or fail with larger contexts due to VRAM limitations. Take the RTX 3090, which comes with 24 GB of VRAM, as an example. It handled the 30 billion (30B) parameter Airobors Llama-2 model with 5-bit quantization (Q_5), consuming around 23 GB of VRAM. However, expanding the context caused the GPU to run out of memory. This scenario illustrates the importance of balancing model size, quantization level, and context length for users.

Key Components of the Benchmark

GPUs Tested

We’ve included a variety of consumer-grade GPUs that are suitable for local setups. For instance, the Nvidia A100 80GB is available on the second-hand market for around $15,000. In contrast, a dual RTX 4090 setup, which allows you to run 70B models at a reasonable speed, costs only $4,000 for a brand-new setup.

Model Quantization

The benchmark includes model sizes ranging from 7 billion (7B) to 75 billion (75B) parameters, illustrating the influence of various quantizations on processing speed. Our tests were conducted on the LLaMA, Llama-2 and Mixtral MoE models; however, you can make rough estimates about the inference speed for other models, such as Mistral and Yi, based on the size of their weights in gigabytes. The table below displays the sizes of the models we used, categorized by their quantization.

Speed Measurement

Performance is quantified as tokens per second (t/s), representing the average speed after three tests with a 512 tokens context  and 1024 tokens generated.

Pricing Information

The chart includes estimated prices for computer systems equipped with the tested GPUs or Apple Silicon chips. Prices reflect the cost of complete systems, incorporating both new, second-hand, and refurbished units, to account for market availability.

Benchmarking Environment

Data was gathered from user benchmarks across the web and our personal benchmarks. We used Ubuntu 22.04,  CUDA 12.1, and llama.cpp (build: 8504d2d0, 2097).

For the dual GPU setup, we utilized both -sm row and -sm layer options in llama.cpp. With -sm row, the dual RTX 3090 demonstrated a higher inference speed of 3 tokens per second (t/s), whereas the dual RTX 4090 performed better with -sm layer, achieving 5 t/s more. However, it’s important to note that using the -sm row option results in a prompt processing speed decrease of approximately 60%.

Tests were conducted on systems sourced from two cloud GPU providers, vast.ai and runpod.io.

This GPU benchmark graph is work in progress and will be update with more GPUs regularly.