When ChatGPT came out in 2022, one of the most impressive things about interacting with it was that it felt like it could do anything.
How to design a good resume? You betcha.
The capital of France? Cake walk.
The life expectancy of a South American capybara? Absolutely.
This is part of the DNA (or rather, training data) of most of the big frontier models. ChatGPT, Claude and Gemini were trained on a wide array of data, and can generally handle most topics. But this is also part of what makes these models so massive to run, which begs the question; does your production system need to know about the capital of France?
You shouldn’t be paying for geography facts
LLMs are essentially large probability machines, capable of producing broad outputs based on broad inputs. But in reality, many use them for fairly specific needs; asking questions about a particular field, a particular feature in their product, or an autonomous workflow activated to carry out a particular task.
This produces a situation where many end up paying the expensive API token costs of an LLM where they use a fraction of its knowledge. This isn’t necessarily a problem of intelligence, but rather a problem of size - using a large model is expensive. But when using a frontier model provider, this quickly ends up becoming a trade-off between intelligence and cost, when the only option offered by them is simply choosing between smaller or larger generalist models.
What if there was an alternative?
Open-source models & domain specialization
In the world of open-source LLMs, you not only get to choose your base model, which in some cases can be focused towards particular use cases (such as coding), you also gain much more access into modyfing its behaviour in a number of ways, including modifying the system prompt and the agent’s capabilities.
At PineGrove, we sometimes refer to these as the “levers” of open-source models; all the things you can tweak to modify model behaviour. This probably deserves an entire post of its own, but broadly speaking these levers fall within a range of effectiveness and complexity of use:
These levers essentially allow you to take an open-source model, which is small enough to run on consumer hardware, and focus it to solve real problems in your business.
This process takes some tinkering, but at PineGrove we’ve seen good results with this approach, solving complicated tasks via one or more focused agents collaborating. And once you go down that road, you’re in control - and you will never need to pay for token margins again.
No free lunch
Now, the trade-off is perhaps obvious; if you’re using a smaller, more focused model, it won’t be as good as broad problem solving as e.g. ChatGPT. But personally, I have yet to see this be a real deal-breaker; most companies have pretty particular use cases in mind, which often rely more heavily on the integration and application of the model than its borader intelligence.
Additionally, this approach actually scales nicely across use cases, in ways which aren’t really possible when using provided frontier models; You can divide the work and run multiple focused models, all collaborating on a larger task. Not only can this let you handle the broader tasks, but it also comes with separation of concerns out of the box, and can scale across hardware in ways that make it easier to control contexts and data access.
In essence, this allows you to construct your AI flows more like a group chat of individual experts, at a size runnable on consumer hardware.
What’s even cooler about this is that you might not even need to run multiple models under the hood; if inference-time adjustments (like system prompts, RAG, skills, tools, steering etc.) are enough (and they often are), you can create multiple different specialized agent experiences at call-time, all while running a single base model on a single GPU.
Armed with this knowledge, the question becomes clear: which tech stack do you want to build the future of your business on?
Is it time to cancel my Claude subscription?
As excited as I am about the world of open-source models, the truth is that it does take some tinkering to get going, and in some cases, you will face challenges already solved by the big providers. However, the intention of this post isn’t to convince you to drop every AI subscription you’re currently using, but rather to start considering whether an open-source solution could work for some of your AI use-cases.
Maybe you like Claude’s integration into power-point, and that’s all fine.
But if you for example want to:
- Run complex financial flows where the integration between data, math (which LLMs are notoriously bad at), and your domain is the key
- Analyze documents containing sensitive data
- Go crazy with internal agentic workflows without incurring costs competing with the US national debt
Then maybe open-source models can open new doors for you, with a higher degree of control and insight into both behavaiour, data and costs.
Getting started with open-source LLMs
If this has piqued your interest in working with open-source LLMs, and you want to learn more, we wrote a blog post about the anatomy of open-source LLM stacks here.
Otherwise, you’re of course always welcome to reach out to us at PineGrove, and we’ll happily take a chat and spar with you on your setup - perhaps we even have existing solutions in place for your exact needs.
// Mikkel