Why Hardware First for Local AI?
Local AI (local LLM) is a method of running AI models directly on your own managed PC, workstation, or server instead of external cloud APIs. The key advantages are that data doesn't leave your environment and there are no usage-based API costs.
However, "which model can run on which hardware" determines the success or failure of deployment. If the model size and hardware don't match, response speed drops below practical levels, or execution may be impossible altogether.
Model Size and Memory Relationship
The first thing to understand is the relationship between model parameter size and required memory (VRAM).
- Model weights generally occupy memory equal to parameter count × bytes per precision.
- Example: Running a 7B (7 billion parameter) model with 4-bit quantization (Q4) requires approximately 4GB for weights alone.
- Add input context (KV cache) and temporary inference data, so it's safe to secure 1.5–2× the weight size in VRAM.
| Model Size | Recommended VRAM (4-bit quantization) | Suitable Use Cases |
|---|---|---|
| 1–3B | 4–8GB | Summarization, classification, lightweight tasks |
| 7–8B | 8–16GB | Development assistance, document summarization, Q&A |
| 13–14B | 16–24GB | Workflow automation requiring quality |
| 30–34B | 24–48GB (multi-GPU possible) | High-difficulty reasoning, enterprise services |
| 70B+ | 48GB+ (multi-GPU) | Highest quality, large-scale context |
Maintaining full precision (FP16) increases required memory by 2–4×.
GPU Selection Criteria
1. VRAM Capacity Is Top Priority
You need to load the entire model into VRAM for speed. Insufficient VRAM causes offloading to CPU memory, making it tens of times slower. Look at "VRAM capacity" before "performance."
2. Choose Based on Purpose
- Personal development/experimentation: Consumer GPUs (24GB VRAM class) running 7–14B models is a realistic starting point.
- Team-level services: 1–2 data center GPUs (48–80GB VRAM class) can stably run 30B-class models.
- Large-scale/multi-user: Distributed inference across multiple GPUs, or parallel processing with inference server software (vLLM, etc.).
3. Inference Server Software
Ollama (simple local execution) and vLLM (high-performance server inference) are the leading options. Start with Ollama for small-scale experiments, and use dedicated inference servers like vLLM for services with multiple concurrent users.
Cost Structure Comparison: Local vs Cloud API
| Item | Local AI | Cloud API |
|---|---|---|
| Initial Investment | Hardware purchase (millions to tens of millions) | None |
| Operating Cost | Electricity, cooling, maintenance | Usage (token) based billing |
| Data Control | Full control within internal network | External transmission occurs |
| Scalability | Scale by hardware replacement | Instant scaling |
| Best For | High-volume, high-frequency use; sensitive data | Low-volume, intermittent use; quick start |
The key is usage volume. If you're sending hundreds of requests daily, API costs will quickly surpass hardware investment. Conversely, if it's only a few dozen requests per month, API is more economical.
When Hybrid Configuration Is the Practical Answer
Many companies choose a hybrid structure where sensitive tasks use local models and general tasks use external APIs. By routing based on data sensitivity and workflow characteristics, you can secure both control and performance.
Pre-Deployment Checklist
- Define use case: What model size is needed (reverse-engineer from required quality)
- Estimate usage: Concurrent users, daily request volume
- Budget range: Initial investment + 3-year operating costs
- Environment constraints: Power, cooling, space, network (air-gapped or not)
- Operations staff: Can you handle model swaps and performance monitoring?
If you want to design a local AI configuration for your environment, check the Local AI service page for deployment scope or contact us to share your current hardware and goals.