An open-source LLM is a language model whose trained weights are published for anyone to download. You can run it on your own hardware, adapt it and build it into products without asking an external service on every call. That is the decisive difference from models reachable only through their provider’s API.
The term is imprecise, though, and systematically so. What blog posts call “open source” is in most cases an open-weight model: the weights are there, how they came about stays in the dark.
What “open” means in each case
Between a download link and genuine open source sit several tiers.
| Tier | What is available | What is missing |
|---|---|---|
| Open weights, bespoke licence | Model file to download, use subject to conditions | Free redistribution, sometimes restrictions for large providers or certain purposes |
| Open weights, free licence | Model file under Apache 2.0 or MIT, commercial use without special rules | Training data and training code |
| Fully open | Datasets, training scripts and intermediate checkpoints on top | Practically nothing, though repeating the training run remains a question of compute |
The first tier is the most common. Meta’s Llama models, for instance, ship with a bespoke licence containing usage conditions that classic open-source licences do not have. Mistral has released several models under Apache 2.0 and sits on the second tier. The third is rare and usually comes out of academia, such as the OLMo series from the Allen Institute for AI, which publishes data and training code alongside the weights.
The Open Source Initiative put forward its own definition of open AI in 2024, requiring information about data and training process in addition to the weights. By that standard a large share of what is called an open-source LLM today would not qualify. The distinction still matters in practice: it decides whether you may redistribute the model, embed it in a product and use it commercially without asking.
Why organisations self-host
- Data stays in house. Personal data, patient records, engineering drawings, contracts. If nothing leaves your own network, a whole block of reviews and contracts falls away.
- No model changes from outside. A provider can retire a model or quietly adjust it. What you run yourself behaves tomorrow the way it does today. For regulated processes with an audit obligation that is a real argument.
- A cost structure instead of a cost curve. Rather than billing per token you carry fixed infrastructure. At very high, steady throughput the arithmetic eventually tips towards your own hardware; at sporadic use it never does.
- Adaptability. An open model can be tuned on your own terminology, trimmed in size and packed into a device or an air-gapped environment.
- No network needed. Models run offline, on the shop floor, in a vehicle, on a laptop.
What running one takes
The download is free, the operation is not. Four points get underestimated regularly.
Hardware. Large models need graphics memory, more than a standard server configuration provides. Quantisation, meaning arithmetic at coarser numeric precision, cuts that requirement substantially and costs some quality depending on the task. For many applications a smaller, quantised model is the right answer.
Operations. Self-hosting means running a service: monitoring, updates, load balancing, failover. Tools such as Ollama, llama.cpp or vLLM take a lot of that off your hands, but they do not replace ownership.
Quality. Open models have caught up considerably and handle everyday tasks reliably. At the top end, on long chains of tool calls and hard reasoning, the big closed models are generally still ahead. The honest answer is that it depends on the task, and you measure it on your own case.
Law. Read the licence before the model goes into a product. The EU AI Act grants some relief for models released freely and openly, but that relief disappears the moment the system is used in one of the regulated high-risk areas. Responsibility sits with whoever puts the system into operation, not with whoever published the weights.
When the external API is the better fit
| Situation | Sensible route |
|---|---|
| Prototype, feasibility check, first version | External API, running in minutes |
| Fluctuating or low load | External API, no baseline cost |
| Hardest reasoning required | External API, currently usually stronger |
| Personal or strictly confidential data | Self-hosted |
| High, steady throughput | Self-hosted, above a certain volume |
| Offline or isolated operation | Self-hosted, no alternative |
A common middle path: the prototype is built against an external API, the application talks to it through a swappable layer, and moving to a self-hosted model stays a configuration question. That is considerably cheaper than a migration later on.
Frequently asked questions about open-source LLMs
Is an open model automatically GDPR compliant? No. Transfer to an external provider falls away, which is a big item. Everything else remains: legal basis, purpose limitation, deletion concept, logging, and the question of what happens to your users’ input.
Can I run a model like this on my laptop? Small models yes, at decent speed even. The larger variants need dedicated graphics hardware. For trying things out, a current notebook is plenty.
What does fine-tuning on my own data achieve? It shifts style, format and terminology, but it does not make the model more knowledgeable. For “the model should know our manuals”, retrieval augmented generation is almost always the better and cheaper route.
Can you trust an open model published abroad? The weights are a file, not a connection. They phone home to nobody. What remains to be checked is what a model actually outputs, where it has gaps or a slant, and whether the licence suits your plans. All of that is easier to check on an open model than on a closed one.
Conclusion
Open language models have separated the question “AI or not” from the question “whose servers”. For anything handling sensitive data or running without a network, that is the difference between feasible and not feasible. The price is operational responsibility: hardware, updates, measuring quality on your own case. Plan none of that in and you have merely swapped one bill for another. How we build components like these into existing systems is described on our page about custom software development, and our experience with AI in day-to-day development in the article AI in software development. Whether an open model ends up running on your own hardware or a service gets connected through an interface depends on the case; both routes are covered on our page about AI integration into existing software. For an assessment of your own plans there is a free consultation.