In recent years, large language models (LLMs) have revolutionized the field of artificial intelligence, offering unprecedented capabilities in programming, text summarization, role-playing, or serving as general AI assistants.

Among these, Llama 2 and the more recent Llama 3.1 stand out as powerful open-source alternatives to proprietary models. For enthusiasts and researchers alike, the ability to run these models locally presents an exciting opportunity to explore and harness their potential. However, the computational demands of these models necessitate careful consideration of hardware requirements

GPU Requirements for Llama 2 and Llama 3.1

At the heart of any system designed to run Llama 2 or Llama 3.1 is the Graphics Processing Unit (GPU). The parallel processing capabilities of modern GPUs make them ideal for the matrix operations that underpin these language models.  In our testing, We’ve found the NVIDIA GeForce RTX 3090 strikes an excellent balance between performance, price and VRAM capacity for running Llama. With its 24 GB of GDDR6X memory, this GPU provides sufficient VRAM to accommodate the substantial memory footprint of these models.

Using an RTX 3090 in conjunction with optimized software solutions like ExLlamaV2 and a 8-bit quantized version of Llama 3.1 13B, users can achieve impressive performance, with speeds up to 50 tokens per second. This level of performance brings near real-time interactions within reach for home users.

VRAM requirements for pure GPU inference for 4-Bit quantized Llama models

For smaller Llama models like the 8B and 13B, you can use consumer GPUs such as the RTX 3060, which handles the 6GB and 12GB VRAM requirements well. The LLaMA 33B steps up to 20GB, making the RTX 3090 a good choice.

When we scaled up to the 70B Llama 2 and 3.1 model, We quickly realized the limitations of a single GPU setup. A dual RTX 3090 or RTX 4090 configuration offered the necessary VRAM and processing power for smooth operation. This configuration allows for distribution of the model weights across the available VRAM, enabling faster token generation compared to setup where the models weights are split between the VRAM and the system memory (RAM).

For the massive Llama 3.1 405B, you’re looking at a staggering 232GB of VRAM, which requires 10 RTX 3090s or powerful data center GPUs like A100s or H100s. The hardware demands scale dramatically with model size, from consumer-friendly to enterprise-level setups.

Example of inference speed using llama.cpp,  RTX 4090, and Intel i9-12900K  CPU

Quantization: Balancing Performance and Accuracy

One of the key techniques enabling the use of these large models on consumer hardware is quantization (compression). This process reduces the precision of the model’s weights, significantly decreasing memory requirements and computational demands. The most common quantization levels for home use are 4-bit and 8-bit.

4-bit quantized models offer a higher level of compression, allowing larger models to fit within the constraints of consumer GPUs. These models run efficiently on GPUs with lower VRAM capacities, making them an excellent choice for users with more modest hardware. However, this compression comes at a cost of some reduction in model accuracy.

A typical Huggingface repository for Llama 3.1, where you can download various quantized versions of the model—in this case, Meta Llama 3.1 8B Instruct in GGUF file format.

8-bit quantized models strike a balance between compression and accuracy. They require more GPU memory and computational power than their 4-bit counterparts but generally offer higher accuracy. These models are well-suited for users with high-end consumer GPUs that have more VRAM.

CPU Considerations

While the GPU handles the bulk of the computational load, the CPU plays a crucial supporting role in the overall system performance. For users running Llama 2 or Llama 3.1 primarily on the GPU, the CPU’s main tasks involve data loading, preprocessing, and managing system resources. High-end consumer CPUs like the Intel Core i9-13900K or AMD Ryzen 9 7950X provide ample processing power for these tasks.

CPU-based inference is another popular approach for running large language models. Software solutions like llama.cpp (LM Studio, Ollama), combined with GGUF model formats, allow for split (VRAM and RAM) and pure CPU inference. This approach is particularly useful for users who may not have access to high-end GPUs or wish to run models that exceed their available VRAM.

When considering CPU-based inference, it’s important to note that performance is closely tied to memory bandwidth. This is because when generating a single token, the entire model needs to be read from memory once. That’s why a processors supporting higher memory bandwidth and more channels, such as those compatible with DDR5-6000 memory, can improve inference speeds.

This is an example of running llama.cpp with a Ryzen 7 3700X and 128GB RAM @ 3600 MHz.

RAM and Memory Bandwidth

The importance of system memory (RAM) in running Llama 2 and Llama 3.1 cannot be overstated. For GPU-based inference, 16 GB of RAM is generally sufficient for most use cases, allowing the entire model to be held in memory without resorting to disk swapping. However, for larger models, 32 GB or more of RAM can provide a performance boost.

These are the the memory (RAM) requirements for LLaMA model used on the CPU:

In CPU-based inference scenarios, the relationship between the CPU and memory becomes even more critical. The bandwidth between the CPU and RAM directly impacts inference speed. For instance, a system with a Core i9-10900X (supporting 4 memory channels) and DDR4-3600 memory can achieve a throughput of about 115 GB/s. With a 13 GB model, this translates to an inference speed of approximately 8 tokens per second, regardless of the CPU’s clock speed or core count.

Inference speed for 13B model with 4-bit quantization, based on memory (RAM) speed when running on CPU:

Storage: Fast Access to Model Data

While often overlooked, storage plays a crucial role in the overall performance of a system running Llama 2 or Llama 3.1. A high-speed NVMe SSD with PCIe 4.0 support is recommended for storing model files and datasets. This ensures rapid data transfer between storage and system RAM, minimizing bottlenecks during model loading and data preprocessing.

A minimum of 1 TB storage is advisable, but 2 TB or 4 TB configurations provide more flexibility for storing multiple models, datasets, and generated outputs. The sequential read and write speeds of modern NVMe SSDs contribute significantly to reducing model load times and enhancing overall system responsiveness.

Motherboard: The Foundation of Your AI Workstation

For users building a single-GPU system, a solid mid-range motherboard that matches the chosen CPU socket will suffice. However, those considering a dual-GPU setup, particularly with cards like the RTX 3090, need to pay special attention to motherboard selection.

Key considerations for dual-GPU setups include:

1. PCIe slot layout: Ensure there are two PCIe slots with adequate spacing between them, as the RTX 3090 is a 3-slot card.
2. PCIe bifurcation support: The motherboard should support splitting a single PCIe 16x slot into two 8x slots (8x/8x configuration) for optimal performance with dual GPUs.
3. Power delivery: Robust VRM design to handle the power demands of high-end CPUs and multiple GPUs.
4. Connectivity: Ample USB ports and M.2 slots for storage expansion.

Power Supply and Cooling: Keeping Your AI Rig Running Smoothly

Running Llama 2 or Llama 3.1 models, especially on high-end GPUs, can be power-intensive. A high-quality power supply unit (PSU) with sufficient wattage is crucial for system stability. For single GPU setups, an 750W or  850W PSU is generally sufficient. Dual-GPU configurations may require 1200W or higher PSUs to ensure stable operation under load.

Equally important is maintaining optimal thermal conditions. Large language models can put significant stress on GPUs and CPUs, leading to sustained high temperatures. A well-designed case with good airflow, coupled with effective CPU and GPU cooling solutions, is essential for maintaining performance and longevity of components.

Comparing Performance: Dual GPUs vs. Apple Silicon

While dual-GPU setups using RTX 3090 or RTX 4090 cards offer impressive performance for running Llama 2 and Llama 3.1 models, it’s worth considering alternative platforms. Apple’s M1, M2, and M3 series of processors, particularly in their Pro, Max, and Ultra configurations, have shown remarkable capabilities in AI workloads.

We were particularly impressed by the unified memory architecture of the M2 Max when testing Llama. The unified memory architecture (UMA) and the high bandwidth definitely contributed to efficient data handling. This architecture can be particularly beneficial for running large language models. For instance, the M2 Max chip supports up to 400 GB/s memory bandwidth, while the M2 Ultra pushes this to an impressive 800 GB/s.

However, when it comes to raw inference speed for the largest models, dual high-end NVIDIA GPUs still hold an edge. A dual RTX 4090 setup can achieve speeds of around 20 tokens per second with a 65B model, while two RTX 3090s manage about 15 tokens per second. In comparison, even the most powerful Apple Silicon chips struggle to match these speeds for the largest models.

Building Your Llama Rig

After exploring the hardware requirements for running Llama 2 and Llama 3.1 models, let’s summarize the key points and provide a step-by-step guide to building your own Llama rig.

Key Takeaways:

• GPU is crucial: A high-end GPU like the NVIDIA GeForce RTX 3090 with 24GB VRAM is ideal for running Llama models efficiently.
• CPU matters: While not as critical as the GPU, a strong CPU helps with data loading and preprocessing.
• RAM requirements: 32GB or more of fast RAM (DDR4-3600 or better) is recommended for optimal performance.
• Storage speed: An NVMe SSD with PCIe 4.0 support ensures fast model loading and data access.
• Cooling is essential: Proper cooling solutions are necessary to maintain performance during long inference sessions.
• Power supply: A high-quality PSU with sufficient wattage (850W+) is crucial for system stability.

LLM build based on RTX 3090 24GB

Expand and upgrade:

Consider adding more storage as you accumulate models and datasets
This build room for expiation in to double RTX 3090 setup. Just add a higher power PSU like Corsair HX1200 Platinum 1200 W 80+ Platinum and couple (~x5) of quiet! Pure Wings 2 87 CFM 120 mm case fans and you will be running 70B parameter models in no time.

Conclusion

Running Llama 2 and Llama 3.1 models locally opens up exciting possibilities for AI enthusiasts, researchers, and developers. While the hardware requirements may seem daunting, careful selection of components can result in a system capable of impressive performance. Whether opting for a high-end GPU setup or leveraging CPU-based inference, the key lies in balancing the various components to create a harmonious system tailored to your specific needs and budget.

As these models continue to evolve and new optimization techniques emerge, we can expect the hardware landscape to shift as well. Staying informed about the latest developments in both AI models and hardware will be crucial for those looking to push the boundaries of what’s possible with home-based AI systems.