On-prem vs cloud AI.

The session

On-prem vs cloud AI.

The default answer to "where should I run my AI" in 2026 is "cloud, until it isn't." The cloud is faster to start, cheaper to scale, easier to update. For most workloads, it is the right answer. For a narrow and important set of workloads, it is not. The decision tree for that narrow set is what this session is about.

The on-prem case is built on three arguments: data residency, audit trail, and predictable cost. Data residency is the easy one — if your regulator requires that customer data never leave a specific jurisdiction, and your cloud provider does not have a region in that jurisdiction, on-prem is the answer. Audit trail is harder — if your auditor requires that the inference log, the model weights, and the prompt history all live on infrastructure you physically control, on-prem is the answer. Predictable cost is the hardest — at scale, the cloud bill for inference is variable and can spike, and some operators prefer the fixed cost of hardware to the variable cost of tokens.

The cloud case is built on three different arguments: time-to-value, scalability, and operational overhead. Time-to-value is the dominant one for most operators — the difference between deploying a model in an afternoon and deploying a model in six months is six months of business value. Scalability matters for workloads with bursty traffic. Operational overhead matters for operators without a dedicated infrastructure team.

The hybrid case is the most common in production. The model runs in the cloud for most queries. A small subset of queries (the ones that touch regulated data) are routed to an on-prem inference server. The on-prem server may be a smaller model (a fine-tuned 7B instead of a 70B), running on commodity hardware, inside the customer's data center. The cloud handles the long tail. The on-prem server handles the regulated slice. The Netwit runtime supports this pattern natively — the orchestration layer can route individual tool calls to on-prem or cloud endpoints based on data classification.

The cost math is worth doing. A 70B-parameter model on a single H100 GPU costs about $3/hour to run on a cloud provider, or about $30K to buy the hardware outright. The break-even is around 800 hours/month, or roughly 11 hours/day. Most production inference workloads run 24/7, so on-prem wins on cost above that threshold. The hidden costs are power, cooling, networking, and the engineers to keep the hardware alive — those typically add 30-50% to the headline hardware cost. The cloud bill is more honest about its total cost.

The data-residency argument is becoming more important, not less. Regulators in healthcare, finance, and government are tightening the rules. The EU AI Act, the US executive orders on AI, the UAE's data protection law, India's DPDP Act, and Canada's AIDA all touch on data residency in different ways. If you operate across borders, the right default is to assume that data residency is going to be a hard requirement for an increasing share of your workload, and to plan for hybrid from the start.

The audit-trail argument is the one that most operators under-weight until they are in front of an auditor. The auditor's question is not "does the model work" — it is "can you show me every decision the model made, with the input, the output, the model version, and the human who approved." Cloud providers will give you part of this. On-prem gives you all of it. For operators in regulated industries, on-prem (or a private cloud) is the default.

The session walks through the decision tree, the cost math, the data-residency rules, the audit-trail requirements, and three worked examples (a healthcare operator, a financial operator, a government operator) with the deployment model each one chose and why. The recording and the decision-tree template are sent to all registrants.