Where Does Your Prompt Actually Go? API vs. Private Cloud vs. On-Prem, Compared
"Is it private?" is usually answered with a vibe, not a citation. Someone says an API provider is "basically fine" for sensitive data, or that self-hosting is "obviously more private" — and both of those can be true or false depending on which specific term, endpoint, and account tier is actually in play.
This piece skips the vibe and goes to the current, published terms: what OpenAI, Anthropic, and Google actually commit to for API traffic today, and how that compares to the two other deployment models we've measured directly in our own infrastructure work — renting a GPU in the cloud and running the model yourself, and running it on hardware you own outright.
Every claim below is sourced to each provider's own current documentation, checked in September 2026, with links. These terms change — Anthropic rewrote its consumer policy twice in the past year alone — so treat this as a snapshot, verify the live source before it informs a real decision, and don't take this as legal advice; it's a factual comparison, not a compliance opinion.
The default vs. the contract — the distinction that matters most
Before the provider-by-provider detail, one distinction explains most of the confusion in this space: a default is what happens if nobody changes a setting, and it can be changed by the provider at any time with a policy update. A contractual commitment — something in commercial or enterprise terms, a signed Data Processing Addendum, a Zero Data Retention agreement — is something the provider is bound to regardless of what changes for everyone else.
The clearest recent example: Anthropic built its reputation partly on not training on user data. In August 2025, it changed the default for consumer Claude accounts (Free, Pro, Max) — conversations now train future models unless the user actively opts out, with a longer retention window for those who allow it. That change did not touch commercial or API customers, who remain covered by separate terms that explicitly exclude training on customer content. Same company, two different promises, depending entirely on which product tier you're using. If you're evaluating a provider based on something you read a year ago, it's worth re-checking — not because providers are untrustworthy, but because "current policy" is a moving target industry-wide right now.
The three API providers, as of September 2026
OpenAI (API)
OpenAI's API has not used customer prompts or outputs to train its models by default since March 2023 — this is a longstanding, stable commitment, separate from the consumer ChatGPT Free/Plus tiers, where training is on by default unless the user opts out. For API traffic specifically:
- Retention: inputs and outputs are stored for up to 30 days for abuse monitoring, then deleted, unless legally required to retain longer.
- Zero Data Retention (ZDR): available for eligible customers on supported endpoints, on approval.
- A real-world caveat worth knowing: OpenAI was previously subject to a court order (stemming from litigation with the New York Times) to preserve consumer ChatGPT logs that would otherwise have been deleted — that order was lifted in September 2025, and it applied to consumer conversations, not API traffic under ZDR. It's a useful reminder that a stated retention policy can be overridden by legal process, for any provider, regardless of what the policy page says.
Source: OpenAI's API data usage documentation and enterprise privacy page.
Anthropic (API)
Anthropic's Commercial Terms of Service state that Anthropic does not train models on customer content from commercial products — the API, Claude for Work, Claude Gov. This is contractual, not a togglable default.
- Retention: API inputs and outputs are automatically deleted within 30 days of receipt or generation, unless a Zero Data Retention agreement is in place, a feature with longer retention is opted into, or longer retention is legally required.
- Zero Data Retention: available for eligible commercial customers; under ZDR, content isn't stored at rest beyond what's needed to return the response — though safety-classifier outputs are still retained even under ZDR, to enforce Anthropic's usage policy.
- A specific, current exception worth flagging: as of June 2026, Anthropic's "Covered Models" policy — which currently applies to Claude Fable 5 and Claude Mythos 5 — requires a mandatory 30-day retention window on every platform these models are offered on, and these specific models are not available under ZDR at all. If a workload needs true zero retention, this is the kind of model-specific exception worth checking before assuming a blanket policy applies to every model in a provider's lineup.
- Content flagged for a usage policy violation can be retained for up to two years, with associated safety classifier scores held for up to seven years, even under otherwise strict retention terms.
Source: Anthropic's Commercial Terms of Service and consumer terms update announcement.
Google (Gemini API / Vertex AI)
On paid Gemini API and Vertex AI services, Google does not use prompts or responses to improve its models — this is separate from the free-tier Google AI Studio, where different terms can apply, and separate from the consumer Gemini app, where conversations are retained in a user's Google Account for a default of 18 months (adjustable to 3 or 36 months).
- Retention: prompts and responses are logged for a limited period for safety, abuse detection, and legal compliance; specific windows depend on the deployment and contract.
- Zero Data Retention: available on the Gemini Developer API and Vertex AI for approved projects — but it requires action, not just enrollment: disabling data caching, and the request must be approved per project.
- An exception that's easy to miss: using Grounding with Google Search adds a mandatory 30-day retention window for the prompts, context, and output involved — and Google states plainly there is no way to disable this if the grounding feature is in use, even under an otherwise-approved ZDR configuration.
Source: Google's Gemini API zero data retention documentation and Vertex AI data governance docs.
The three, side by side
| OpenAI (API) | Anthropic (API) | Google (Gemini API / Vertex) | |
|---|---|---|---|
| Trains on API data by default | No (since March 2023) | No (contractual, commercial terms) | No (paid tiers) |
| Default retention | Up to 30 days | Up to 30 days | Varies by deployment; ≤30 days typical |
| Zero Data Retention available | Yes, eligible customers | Yes, eligible customers | Yes, approved projects; requires config |
| Known exception to watch | Legal holds can override retention policy | Newest "Covered Models" require 30-day retention, no ZDR | Search grounding forces 30-day retention, non-disableable |
The pattern across all three: none of the major API providers train on your data by default today, and all three offer contractual zero-retention paths for qualifying customers. The differences that actually matter are in the exceptions — the specific feature, model, or account tier that quietly carries different terms than the headline policy.
What changes with private cloud — running the model yourself, on rented hardware
This is the deployment model behind most of our own infrastructure series — renting a GPU (an NVIDIA L4 on Google Cloud, in our case) and running an open-weights model ourselves via Ollama, rather than calling anyone's API.
The structural difference: no LLM provider is in the data path at all. There's no OpenAI, Anthropic, or Google model receiving your prompts, because you're not sending prompts to their model — you're running your own inference server on infrastructure you're renting. The privacy question shifts entirely: it's no longer "what does the AI provider do with my prompt," because there isn't one. It becomes "what does the cloud infrastructure provider have access to."
That's a meaningfully smaller and more standard question. Google Cloud, AWS, and Azure have long-established data processing terms for their general infrastructure — the same terms that have governed databases, web servers, and file storage for over a decade, not AI-specific policies still being actively rewritten. A cloud provider hosting your VM can, in principle, access the underlying infrastructure (this is inherent to any rented compute), but it isn't running your prompts through a model of its own or logging them for its own product improvement — there's no product improvement loop to opt into or out of, because your model isn't theirs.
What this doesn't solve: you still don't own the physical hardware, so data residency, jurisdiction, and physical access assurances are governed by whichever cloud region and provider you chose — the same considerations that apply to any cloud-hosted workload, AI or not. And operationally, as we found running this exact setup, you take on real infrastructure responsibility: provisioning, driver installs, capacity planning, the occasional preemption or capacity stockout — detailed in our build-vs-buy piece.
What changes with on-prem — your own hardware, physically
The final tier removes the cloud provider from the equation too. If the model runs on hardware you physically own and control — in your own office, your own data center — there is no third party anywhere in the data path. Not an AI provider, not a cloud infrastructure provider. Nobody outside your organization has technical access to the prompts, the outputs, or the model weights, absent a physical breach or a deliberate integration you build yourself (internet access, telemetry, etc.).
This is the strongest privacy posture available, in a strict technical sense — and it's also the one with the fewest external guarantees to point to, because there's no vendor's compliance page to cite. Your own security practices are the entire privacy posture: physical access control, network isolation, patching, who on your team has access to the machine. For regulated industries with genuine data-residency or air-gap requirements — certain government, healthcare, or defense contexts — this is often the only model that actually satisfies the requirement, regardless of what any vendor's contract promises.
The tradeoff is real and worth stating plainly: this is also the model with the most ongoing operational burden, the least elasticity (no spinning up more capacity for a traffic spike), and the highest upfront cost. Our own hardware-selection findings are relevant here too — sizing on-prem hardware to the actual model and workload matters even more when there's no cloud elasticity to fall back on if the sizing is wrong.
A framework for which tier actually matches your requirement
Not every privacy concern needs the same answer, and matching the tier to the actual requirement — rather than defaulting to the most restrictive option available — usually saves real cost:
A contractual no-training, bounded-retention commitment is enough for most standard business confidentiality needs — client data, internal business information, anything where "a reputable vendor is contractually bound not to train on this and deletes it within 30 days" satisfies your actual obligation. This describes the current API terms from all three major providers, for their commercial/enterprise tiers specifically — not their consumer products.
Data residency or "no third-party AI vendor in the loop" requirements point toward private cloud — self-hosting removes the AI-specific data path entirely while still giving you cloud elasticity, at the cost of the operational overhead detailed in our build-vs-buy piece.
True air-gap, no-external-network, physical-control requirements — the kind seen in specific regulated or classified contexts — are the cases where on-prem is not a preference but a requirement, and where the operational cost is simply the price of the actual constraint.
A mismatch to watch for in either direction: treating every workload as if it needs on-prem when a properly-scoped ZDR agreement would satisfy the actual compliance requirement wastes real money and engineering time. The inverse — using a consumer-tier AI product with training-on-by-default for genuinely sensitive data because "it's just a chat" — is the more common and more consequential mistake.
What this doesn't cover
This piece is about where the data goes and who's contractually bound not to use it for what — it does not cover encryption specifics, sub-processor lists, individual certifications (SOC 2, HIPAA BAAs, FedRAMP), or region-by-region data residency guarantees, all of which matter for a real compliance decision and all of which are provider- and contract-specific beyond what a comparison article can responsibly generalize. For an actual regulated workload, the next step after this piece is reading the specific Data Processing Addendum or commercial agreement that would govern your account — not this article.
Where this fits
This piece sits alongside our infrastructure cost series — the same six experiments that measured what self-hosting actually costs also form the basis for the private-cloud and on-prem sections above. Start with The Metric That Kept Lying to Us for the full cost picture, or Build vs. Buy for Private LLM Inference for how the cost and privacy questions intersect in practice.
If you're weighing these tiers for a real workload with actual compliance requirements attached, book a call — matching the deployment model to the actual requirement, rather than defaulting to the most expensive one, is exactly the kind of question worth getting right before you build.