Sovereign AI Coding With Local Open Models

In the past couple of years, I’ve been quite a vocal advocate for sovereignty of data and processing capacity. In conversations with engineering leaders, one of the more common questions I hear is “How do we use AI coding agents while we maintain control of our IP?” The answer to this question is elusive when everyone wants to use the frontier models and take advantage of their leading capabilities for reasoning. Each frontier model company leapfrogs the other in capability week by week, and the landscape is shifting daily.

Quickly following any sovereignty discussion comes the question “How can I control my token costs?” The frontier model companies are changing the economics with every release, and your token costs are becoming even more difficult to predict as their reasoning models seem to expand the multipliers of tokens consumed as they get more capable.

There are so many reasons these things are happening. Each frontier company is in an arms race, and they are quickly adjusting their models and agentic capabilities, expanding their inference platforms, making infrastructure deals, and adjusting their financial models. In addition to this, their user policies are often tilted in their favor, ensuring they can take advantage of the data that you send them to improve their platforms. Not to mention that if they decide your workload patterns are abusive or in violation of ever-changing terms, they can cut your access with little to no notice or recourse.

In my mind, this makes it more attractive to look for sovereign capabilities that are running in an environment that your organization controls. Your developers are embracing agentic coding. What do you do when you are worried about sending so much data and context out to the frontier labs? How do you stay in control? Open models are getting better and now there's a secure way to run them locally, so you have complete control. Not every task needs frontier lab model capabilities. I believe you should employ an intelligent routing capability for your agentic workflows to enable you to route work to the right model for the task at hand. For tasks that can run on open models, use open models on local capacity that you control. This will increase your sovereignty, increase your token efficiency, lower your risks, and reduce your costs.

Recently on August 5th, AMD, Supermicro and Spectro Cloud announced AMD Instinct Coder, a validated appliance for enterprise AI coding. Open models run locally on AMD Instinct GPUs inside pre-engineered systems. Spectro Cloud's PaletteAI Inference Launchpad sits in front and routes requests from your agent harness.

Coding has become a spec and a set of agent threads that drive the outcome

Teams have moved past single prompt coding. While that still works for the simplest of tasks, the emerging practice is to use agentic coding harnesses coupled with the IDE. This is called “spec-driven development” or SDD, popularized by the GitHub Spec Kit and AWS Kiro. You write a specification with your acceptance criteria, and the spec is the source of truth with which the agentic coding capabilities generate code and check the results against the spec.

This means that a single developer can now run teams of agents in parallel that are all working towards the goals outlined in the specification. The agents work on different requirements, and the result is coalesced into a working product. For example, a planner turns intent into the spec and a task list, implementers write code against the task list, reviewers open a clean context window and critique the results, and a tester runs it while a verifier checks the output against the acceptance criteria and sends work back when it fails. The reviewers have no memory of how the code was written so they can catch mistakes and hallucinations and ensure the code is of sufficient quality and matches the specifications. A lot of these threads are running in parallel, each with a narrow job, and each of them consuming token capacity. This is why AI coding is so token heavy.

Review is where the tokens go

The agentic coding workflow costs far more than a single prompt. Anthropic's engineering team reports that agents use roughly four times the tokens of a chat session, and multi-agent systems around fifteen times. A University of Illinois study across seven datasets put the multi-agent overhead between four and 220 times, depending on the architecture.

Developers are great at software engineering, but they don’t necessarily know how to estimate the needs of each task or understand the different token uses and per-token costs of different models, or how to choose the right model and level of effort for each task.

In a recent paper on Tokenomics, researchers at Concordia University measured token use across thirty development tasks in a multi-agent framework. Code review accounted for roughly 60 percent of all tokens while the initial design and coding together came to about 11 percent. This illustrates that writing the code is relatively cheap when compared to checking the result which is most of the token cost.

If your team has said something like, “We’re using Opus 5 Max”, and they use that for everything, that is not optimal because that means every task is costing you a lot of tokens with the most expensive model.

Discriminating tasks that do not require frontier reasoning

Some tasks are easy, some are hard. The frontier labs offer different models that suit different tasks. For example, Anthropic provides Haiku, Sonnet, Opus, and sometimes Fable. Each with increasing per-token costs because they are more compute intensive. Assigning tasks to the right model can help cut costs and increase efficiency all without reducing output quality.

Then we want to assign tasks to the right frontier model. OpenAI is known to be more token efficient than Anthropic but there are big differences in capability. Which one you use may be driven by an individual’s bias rather than the objective output quality and dollar efficiency.

You don’t need to pay a frontier lab premium for all tasks. Today’s landscape of open weight models comes in all shapes and sizes and are benchmarked to be extremely good, even frontier-class. They can be hosted at a much lower per-token cost either through a third party or on infrastructure you host yourself for near-zero token cost after your capex and power costs.

Verifying a solution is easier than producing one. Stanford's Weaver work showed an open model paired with an ensemble of open verifiers reaching accuracy comparable to a frontier reasoning model on math and reasoning benchmarks.

Agentic coding with a spec makes this better still because it includes acceptance criteria. Without acceptance criteria, a reviewer agent is asked whether code is good, which is a judgment call. With the acceptance criteria included in the spec, the verifier is asked whether the code satisfies an objective check. Objective checks are exactly the work open weight models handle well.

The highest volume work in your agentic coding pipeline is the critique and correct loop. This work can be run at a lower cost on your own hardware. Code generation and architectural tasks can still go to a frontier model, if necessary, though open models are improving quickly and obviating that need.

Route tasks automatically to the best destination

Instinct Coder routes tasks automatically to the best destination. Every request is read and matched to policy: sensitivity, task type, model capability, team, cost. Review and test threads run locally on the appliance. The few requests that need frontier capability go out through a metered path you control.

Spectro Cloud reports up to 70 percent lower token costs on the workload mixes they have modeled and when you use local inference the costs are going to be predictable.

As an additional benefit, local inference does not rate limit you or change the quality of responses as part of a cost optimization that is out of your control. Fan out multiple code review threads across a codebase and the only constraint is your hardware, not another platform’s rate limits, fees, and quotas. That hardware is scalable and you can expand your capacity at will by joining additional nodes.

Reduce exposure by keeping your decisions on-prem 

Every one of the agentic threads reads your code. The reviewer gets file contents. The explorer gets a map of your repository. The verifier gets your spec, which is the clearest statement of intent your company owns.

Agentic coding multiplies exposure the same way it multiplies cost. Keeping the routing decision and the inference on your own hardware keeps your architecture and business logic in your building. You decide what leaves, and you can show which requests did.

Predictable outcomes 

Metering, quotas and per team reporting tell you what AI costs the business this month, before the invoice arrives. And because you control the model version, your agents behave the same way next sprint as they do today. Anyone who has had a hosted model change under a running pipeline knows what that is worth.

There are well-known complaints about the frontier companies performing silent version changes, doing A/B testing, and “nerfing” the models when they are at their capacity limits. They also don’t have strong SLAs for availability. All of this creates frustration and inconsistent performance. These pains can be avoided by running the workloads on models that you host locally on engineered systems that are under your control and are designed to meet your capacity and availability requirements.

Sovereignty 

Data residency comes up in nearly every AI conversation with our Canadian, public sector and regulated clients. Sovereignty used to mean the geographic region your compute capacity and data sat in. Now it means knowing which model saw your code and exactly where the inference ran. Owning a solution like the AMD Instinct Coder answers both questions.

There are some agencies and regulated industry organizations that do not want their data and context going to ANY frontier model, regardless of cost implications. They may not even want the data leaving the building to go to a private hosted model. Open models are a great solution to this limitation when you run them locally, adjacent to the development team.

The AMD Instinct Coder can be deployed at air-gapped sites and is entirely self-contained. Out of the box it has secure defaults with web calls and egress turned off. No developer can reach the public internet with a prompt until a policy permits them to do so.

Where Arctiq fits 

We are a Spectro Cloud partner, and this is the work we do: platform engineering, security and cost control.

We can help you drive the decisions to support a Sovereign AI coding capability:

• Which models generate, which discriminate, and how you size for both
• What your routing policy says, and who approves an exception
• How your spec and agent workflow is structured so the loop terminates
• How this plugs into the existing IDEs and agent harnesses that your developers already use with no change to their workflows.
• How it operates on day two, gets patched, and sustained in the long-term.

We can scope and build this with you, and we can also manage it for you. 

Let us compare notes

If your coding token bill has your attention, or you cannot say today what your code review processes are sending outside, we want to hear how you are handling it. Get in touch and we will set up a call.

Arctiq can help. Talk to us today.