Private AI Research: What 8 Experiments Showed

Should a business run its own AI model instead of paying OpenAI, Anthropic or Google per use? I spent two months answering that with experiments instead of opinions.

The series started on my laptop (an NVIDIA RTX 2050 with 4 GB of memory), measuring heat and energy. It moved to Google Cloud, on a single NVIDIA L4 GPU at about $0.71 an hour, measuring cost, speed and memory. The models were Llama 3.1 8B and Qwen3 8B, two popular open models you can run yourself.

This page sums up what the series found, in plain language. The technical write-ups for each experiment are linked at the bottom.

The short version

1. How you use the GPU matters more than which GPU you rent. Handling 8 requests at once instead of 1 made each answer about 4× cheaper.
2. An idle GPU is the most expensive GPU. A rented GPU bills every hour, busy or not.
3. Private AI is not automatically cheaper than an API. For a small model, pay-per-use APIs usually win.
4. Private AI wins on control, steady volume, and document-heavy work.
5. The GPU's "busy" meter misleads. It read 96% busy while the GPU had room for almost 5× more work.
6. Long documents fill memory fast. They more than doubled the cost and left the GPU 94% full.
7. Running your own AI is an ongoing operations job, not a one-time setup.

1. Same GPU, four times cheaper per answer

A rented GPU costs the same per hour whether it answers one question at a time or eight. When it handled eight at once, output per hour rose sharply and the cost per answer dropped about 4×.

Both models followed the same curve and finished almost tied ($1.07 vs $1.08 per million tokens). The takeaway for a business: choose between similar models on answer quality and licensing, not speed. How you run the model matters more.

A token is roughly three-quarters of a word. A million output tokens is about 750,000 words of AI-written text.

2. The GPU's "busy" meter misleads

At one request at a time, NVIDIA's standard utilisation reading said the GPU was 96% busy. By that number, there was no room left. Then it produced 4.8× more output at the same power draw, while the meter went down.

The meter only records whether the GPU is doing anything, not how much. For any AI system, track cost per result and how long users wait. Google's own guidance for scaling AI workloads says the same thing.

3. An idle GPU is the most expensive GPU

Every cost figure in this series assumes the GPU was working the whole time. Real businesses don't work that way. A GPU running around the clock costs about $518 a month, even through nights and weekends.

If your team only keeps it busy a quarter of the time, each answer costs about four times more. Below roughly 21% busy, even a premium API like Claude Haiku 4.5 becomes cheaper. Before anyone quotes you a "cheap" private AI setup, ask how busy they assumed it would be.

4. Private AI vs API: who wins, and when

Comparing a 24/7 GPU ($518/month) with pay-per-use APIs, counting output only:

Compared with	API price per 1M output tokens	Monthly output needed for the GPU to break even	Can one L4 GPU get there?
Claude Haiku 4.5	$5.00	~104 million tokens	Yes
Gemini 3.1 Flash-Lite	$1.50	~345 million tokens	Yes, if kept busy most of the day
GPT-5.6 Luna	$1.20	~432 million tokens	Only near full load
Hosted Llama 3.1 8B (AWS Bedrock)	$0.12	~4.3 billion tokens	No. That's about 7× what one L4 can produce

One L4 produced at most about 480–600 million output tokens a month at full load in these tests. API prices checked 27 September 2026 and may change.

Two things stand out:

• Renting the same small model through an API was 7–9× cheaper than running it myself, even with my GPU fully busy. Large providers spread their GPUs across thousands of customers, and a single business can't match that.
• Document-heavy work changes the math. APIs charge for every word you send in, not just what the AI writes back. Searching long contracts, manuals or policies sends in a lot of text, and on a rented GPU that input costs nothing extra. This is where private AI can come out ahead.

There's also a cost the table leaves out: someone has to maintain the system. For most small and mid-sized businesses, that labour costs more than the GPU.

5. Long documents fill the GPU and double the cost

When each request carried a longer document, the GPU needed much more working memory to hold it. At the longest setting tested (about 10,000 words per request, eight at a time), the GPU was 94% full and each answer cost more than twice as much.

If your use case involves large files, plan memory and budget around your longest documents, not your average ones.

6. Running your own AI is an operations job

In two months of testing, I ran into:

• GPU shortages. Whole Google Cloud zones ran out of the GPU I needed, and I had to retry in other zones.
• Discount machines shutting down. The cheaper "Spot" GPUs were taken back three times in under two hours.
• A routine software update that changed results by 25%. Nothing else in the setup changed.

Each has a known fix: reserved capacity, spreading across zones, and pinning software versions with a check after every upgrade. But someone has to own that work. "We'll just host it ourselves" includes that job.

A note for Philippine businesses: there is no GPU cloud region in the Philippines yet, so the nearest option is Singapore (about 23% pricier than the US region I tested in). Private AI gives you control over who touches your data. It does not keep the data inside the country.

When private AI makes sense

Situation	Better fit
Low or unpredictable usage	API
Getting started, still learning what you need	API
Data you can't send to a third party (legal, contractual, or policy reasons)	Private
Steady, high volume that keeps a GPU busy most of the day	Private
Heavy document work: searching contracts, manuals, policies, records	Private often wins
Mixed needs: some sensitive data, some general tasks	Both. Sensitive work on a private model, the rest on an API

My recommendation for most teams: start with an API, measure real usage for a few months, then move specific workflows to private AI where the data or the volume justifies it.

What these experiments don't prove

These results come from careful measurement, but they have limits:

• One setup: one cloud GPU type, one region, small 8-billion-parameter models, and Ollama, a popular tool built for running models on your own machine. Production-grade serving software is usually several times faster, which would lower the self-hosted cost.
• Best-case cost: every cost figure assumes a fully busy GPU and counts output tokens only.
• No quality check: I measured cost and speed, not how good the answers were. For many business tasks, answer quality decides the choice before cost does.
• Repeated test prompt: the same prompt was sent repeatedly, which likely made response times look faster than real-world use.

The experiments

#	Question	What I found
E001	How hot and power-hungry is a laptop GPU running AI?	Average temperature rose from 57°C to 75°C, and power draw rose about 18×
E002	Does a bigger model work the laptop harder?	It ran cooler, because most of it didn't fit on the GPU and ran on the slower CPU
E003	What happens when more of the model runs on the GPU?	2.5× faster, but GPU-only energy per answer looked worse
E004	What if we count the CPU's energy too?	Total energy per answer fell by half. The GPU-only number had been misleading
E005	Does this hold on a datacenter GPU?	Splitting the model between GPU and CPU was the worst case; full GPU was ~5× more efficient
E006	Does handling more requests at once help?	Cost per answer fell from $4.11 to $0.86 per million tokens
E007	Does the choice of model change that?	No. Llama and Qwen ended up tied, but a software update shifted results 25%
E008	What do long documents cost?	Cost more than doubled and the GPU ended up 94% full

Read more:

• The Metric That Kept Lying to Us — the E001–E006 synthesis, and the four times the obvious number pointed the wrong way
• Build vs. Buy for Private LLM Inference — the cost and operations decision in full
• One Config Change Cut Our Inference Cost 4.8× — E006, the concurrency result
• We Tested Our Concurrency Finding on a Second Model — E007, the same curve on Qwen3-8B
• Why We Didn't Upgrade to the Bigger GPU — E005, choosing hardware
• We Ran the Same Benchmark Twice and Got a Different Answer — E002, and why reproducibility matters

---

Weighing API vs private AI for your business?

I help businesses automate repetitive work with AI. When a workflow needs private AI, I set it up based on real numbers, not guesswork.

Book a call