Daydreamer LLM: Versions, Prompt Templates & Hardware Requirements

Updated: 2023-08-31 |

erp-plus

Explore all versions of the model, their file formats like GGML, GPTQ, and HF, and understand the hardware requirements for local inference.

Daydreamer is experimental model that expands certain regions of a smaller Llama-2 13B architecture to the scale of a larger variant 33B LLaMA model, filling in gaps with data from the bigger model. Through extensive finetuning leveraging beefy GPU hardware, it aims to adapt the smaller model to handle new capabilities. Coding data was excluded to keep the focus on conversational aptitude.

The observable output traits of Daydreamer suggest improved fluency, creativity and roleplaying ability thus far. The model can converse in both dry, informative styles and more vibrant personalities depending on the prompt. It handles various input formats gracefully.

VRAM:

Dual GPU

System Memory:

Test this model with:

Hardware requirements

The performance of an Daydreamer model depends heavily on the hardware it's running on. For recommendations on the best computer hardware configurations to handle Daydreamer models smoothly, check out this guide: Best Computer for Running LLaMA and LLama-2 Models.

Below are the Daydreamer hardware requirements for 4-bit quantization:

Memory speed

When running Daydreamer AI models, you gotta pay attention to how RAM bandwidth and mdodel size impact inference speed. These large language models need to load completely into RAM or VRAM each time they generate a new token (piece of text). For example, a 4-bit 7B billion parameter Daydreamer model takes up around 4.0GB of RAM.

Suppose your have Ryzen 5 5600X processor and DDR4-3200 RAM with theoretical max bandwidth of 50 GBps. In this scenario, you can expect to generate approximately 9 tokens per second. Typically, this performance is about 70% of your theoretical maximum speed due to several limiting factors such as inference sofware, latency, system overhead, and workload characteristics, which prevent reaching the peak speed. To achieve a higher inference speed, say 16 tokens per second, you would need more bandwidth. For example, a system with DDR5-5600 offering around 90 GBps could be enough.

For comparison, high-end GPUs like the Nvidia RTX 3090 boast nearly 930 GBps of bandwidth for their VRAM. The DDR5-6400 RAM can provide up to 100 GB/s. Therefore, understanding and optimizing bandwidth is crucial for running models like Daydreamer efficiently

Recommendations:

For Best Performance: Opt for a machine with a high-end GPU (like NVIDIA's latest RTX 3090 or RTX 4090) or dual GPU setup to accommodate the largest models (65B and 70B). A system with adequate RAM (minimum 16 GB, but 64 GB best) would be optimal.
For Budget Constraints: If you're limited by budget, focus on Daydreamer GGML/GGUF models that fit within the sytem RAM. Remember, while you can offload some weights to the system RAM, it will come at a performance cost.

Remember, these are recommendations, and the actual performance will depend on several factors, including the specific task, model implementation, and other system processes.

CPU requirements

For best performance, a modern multi-core CPU is recommended. An Intel Core i7 from 8th gen onward or AMD Ryzen 5 from 3rd gen onward will work well. CPU with 6-core or 8-core is ideal. Higher clock speeds also improve prompt processing, so aim for 3.6GHz or more.