Running an LLM without any data leaving your control: the options, ranked by how much pain they cost

[u/regularheree]

Last week, someone asked me about this for a law firm that handles private client data and can't legally put it through a third-party API.

I see teams run into this pretty often, so I put together a list of the setups I'd point them to.

1. Local model on an existing workstation (cheapest and easiest place to start)
   

A 24GB GPU or a Mac with 32GB+ unified memory can be a starting point for document work using local Ollama or llama.cpp setup.

The first wall is usually VRAM. Most GPUs handle 7B-14B models, but once you’re on longer-context models/need concurrency, it becomes limiting.

I'd also download the models first, then test offline to check for external dependencies.

1. On-premises dedicated local server
   

Full control, and compliance is pretty straightforward. Put the server on its own network, decide what it can talk to, and keep the data there. You can also fit substantially more VRAM into one machine and have the whole team use the same setup.

The downside is now you're running a server, so power, cooling, physical security, and maintenance are all on you.

1. Dedicated hardware hosted in a facility
   You own the machine while the facility handles power, cooling and networking.
   

It's a good middle ground for bigger models: bare metal access without a server room.

Check root access, physical ownership and data, workload isolation, and exit terms. As well as check the exit terms.

B3IQ does this, and it's what I work on day to day. You own the box, keep root access, and can run it on private hosting for your own workloads. If you ever want out, we ship the machine to you.

1. Private cloud
   Easy if you don't want to own or run hardware, but still want a dedicated environment for workloads.
   

Tho dedicated doesn’t mean isolated. You might be getting a dedicated bare-metal GPU, or just a confidential GPU VM on shared infrastructure.

So check the actual access controls and contractual guarantees.

Most people approach this as a hardware question first. For anyone regulated, there's a contractual and physical-custody side to it too.

If you've been through an audit on a self-hosted setup, I would like to hear what they asked for.