繁體中文 Login Register

Running Local LLMs: A Comparison of Ollama, llama.cpp, LM Studio, and vLLM

DaDesktop Team ·

Deploying a Local Large Language Model (LLM) can be approached in various ways. Certain frameworks prioritise ease of access for beginners, whereas others offer granular control or are engineered to handle high-volume concurrent requests. The optimal selection hinges on your specific objectives: whether you seek a straightforward local chat interface, a highly configurable inference engine, or a robust production-level API.

Ollama

Ollama stands out as one of the most accessible entry points for local model execution. The process is streamlined: install the software, pull a model, and execute it via the command line. It also exposes a local API, facilitating integration with other applications and tools.

Pros:

  • Straforward installation and model management
  • User-friendly command-line workflow
  • OpenAI-compatible API
  • GPU acceleration support for NVIDIA, AMD, Apple Silicon, and Vulkan-based systems
  • Modelfiles enable customisation of models and parameters
  • Capability to handle concurrent requests, provided sufficient memory is available

Cons:

  • Limited low-level control compared to llama.cpp
  • Model management is tightly coupled with the Ollama ecosystem
  • Not the preferred choice when maximum serving throughput or distributed inference is required

Difficulty: Low. An ideal option for quickly deploying a model without navigating complex inference settings.

llama.cpp

llama.cpp is a lightweight C/C++ inference engine optimised for efficient model execution across diverse hardware environments. It utilises GGUF models and provides extensive control over the loading and execution processes.

Pros:

  • Precise control over context, GPU offloading, batching, threads, quantization, and other inference parameters
  • Extensive hardware support, including CUDA, HIP, Metal, Vulkan, and SYCL
  • Supports a wide range of quantization levels, from low-bit formats up to 8-bit
  • Ability to split models across multiple GPUs
  • Supports hybrid CPU and GPU execution when models exceed available VRAM
  • Includes llama-server for an OpenAI-compatible API

Cons:

  • Requires more configuration than Ollama or LM Studio
  • GGUF models must typically be downloaded and managed externally
  • Optimising performance often requires a deep understanding of inference parameters

Difficulty: Medium. Well-suited for users who require precise control over model execution or wish to experiment with performance and quantization strategies.

LM Studio

LM Studio is a desktop application dedicated to downloading, configuring, and executing local LLMs. It offers a graphical interface to simplify model discovery and the management of settings such as GPU offloading and context size.

Pros:

  • Intuitive graphical user interface
  • Model search and download capabilities via Hugging Face
  • Presents model and resource details prior to loading
  • OpenAI-compatible API server
  • Supports headless model execution via its llmster server
  • Compatibility with GGUF models via llama.cpp and MLX models on Apple Silicon

Cons:

  • Less granular control than direct usage of llama.cpp
  • The desktop-centric design may not suit all server deployment scenarios
  • Not primarily intended for large-scale, multi-user serving

Difficulty: Low. A strong choice for experimenting with local models while minimising time spent in the command line.

vLLM

vLLM is engineered specifically for serving LLMs to applications and multiple users. Its primary strength lies in efficient high-concurrency serving, leveraging techniques such as PagedAttention, continuous batching, prefix caching, and distributed inference.

Pros:

  • High throughput for handling multiple concurrent requests
  • Continuous batching and efficient KV-cache management
  • OpenAI-compatible API server
  • Direct compatibility with many Hugging Face models
  • Supports various quantization methods, including FP8, INT4, GPTQ, AWQ, and GGUF
  • Supports tensor, pipeline, expert, and other forms of parallelism
  • Designed for production-grade inference and serving

Cons:

  • More complex setup and configuration requirements
  • Primarily targeted at Linux environments
  • Often overkill for single-user interactive model execution
  • Hardware and model compatibility verification is essential prior to deployment

Difficulty: High. Best suited for deploying an inference service rather than simply running a model on a personal workstation.

Which one should you pick?

  • Seeking easy model execution: Ollama or LM Studio. Opt for Ollama for a command-line and simple API approach, or LM Studio for a graphical interface.
  • Desiring inference control: llama.cpp. It offers direct oversight of model loading, quantization, context, GPU offloading, and related settings.
  • Requiring a local API: Ollama, llama.cpp, or LM Studio. All three provide OpenAI-compatible APIs.
  • Serving multiple users: vLLM. Its continuous batching and distributed inference capabilities are tailored for this use case.
  • Experimenting with quantizations: llama.cpp or LM Studio.

Run it on DaDesktop

If local GPU hardware is insufficient, you can execute these tools on a DaDesktop cloud desktop. Select a GPU with adequate VRAM for your chosen model, launch the desktop environment, and install your preferred inference software.

Ollama and LM Studio are advantageous when a simple local environment is desired. llama.cpp provides greater control over hardware and inference configurations. vLLM serves as a viable option when you need to expose a model as a high-throughput API.

View available GPUs to compare VRAM and other specifications.