Research · Self-initiated project
What it costs to run AI privately
Eight measured experiments over two months on running open AI models on your own hardware instead of paying an AI provider per use.
- Eight experiments, published with their numbers and limits
- Same GPU, about 4× cheaper per answer by handling 8 requests at once
On a laptop · NVIDIA RTX 2050, 4 GB
How hot and power-hungry is a laptop GPU running AI?
Does a bigger model work the laptop harder?
What happens when more of the model runs on the GPU?
What if we count the CPU’s energy too?
In the cloud · One NVIDIA L4 on Google Cloud
Does this hold on a datacenter GPU?
Does handling more requests at once help?
Does the choice of model change that?
What do long documents cost?
The question
Businesses with sensitive data ask whether they should run their own AI model. Most of the answers available are opinions or sales claims. I wanted measured numbers.
How I ran it
- Start on a laptop GPU and measure heat and energy
- Move to a rented cloud GPU and measure cost, speed and memory
- Change one thing per experiment
- Publish each result, including the ones that contradicted the last
The setup
The laptop tests used an NVIDIA RTX 2050 with 4 GB of memory. The cloud tests used a single NVIDIA L4 on Google Cloud at about $0.71 an hour. The models were Llama 3.1 8B and Qwen3 8B, served with Ollama. The cloud infrastructure is defined in Terraform.
Built with
- Ollama
- Llama 3.1 8B
- Qwen3 8B
- NVIDIA L4
- Google Cloud
- Terraform
- Python
What I found
Handling 8 requests at once made each answer about 4× cheaper
Cost per million output tokens, same GPU
An idle GPU is the most expensive GPU
Cost per million output tokens, by how busy the GPU is
API price checked 27 September 2026.
- 96%
utilisation shown on the GPU’s meter while it had room for almost 5× more work
- 94%
of GPU memory used on long documents, which also more than doubled the cost per answer
My recommendation
Start with an API, measure real usage, and move specific workflows to private AI only where the data or the volume justifies it.
What this means for a small team
Most small teams should start with a standard AI service. When your data can’t leave your own systems, I can size a private setup from these measurements.
See what you could automate
Enter your website and get three workflows worth automating, with estimated hours saved.