A merchant types: which products are almost out of stock?
The frontier model still talking to them hands that job to a GraphQL subagent: read the Admin API schema, search the docs, write the query, run it, and return the rows.
That handoff used to be expensive. Shopify Engineering describes the GraphQL subagent as context-heavy. It reads about 40 tokens for every one it writes, and a frontier-model API charges for every token in that window.
For that one job they post-trained NVIDIA Nemotron 3.5 Lightning into a task expert, then served the weights on their own GPUs.
On this GraphQL task the specialist scores slightly higher than the frontier model, responds 7× faster, and costs about 90% less to serve.
Call it the specialist split.
Introduction
The October write-up is the architecture for that split: rubric and judge, a raised frontier bar, the open-model choice, self-healing, Blackwell post-training, and the GraphQL scoreboard after six loops. Merchant decisions about Sidekick as a daily admin layer live in Shopify Sidekick's Continual Learning Flywheel: What Merchants Should Know. Reviewing a Canvas or Sidekick storefront edit before publish lives in Shopify Canvas: How to Review Sidekick Theme Changes Before They Go Live. External AI that writes through a connector is a different permission model, covered in Shopify Claude Connector: Setup, Benefits, and Risks.
Shopify published the loop on 21 October 2026 in How Shopify turns open models into task experts. Andrew McNamara and Logan Vegna adapted it from their NVIDIA GTC Berlin 2026 talk. GraphQL is their worked example.
What is the specialist split?
A frontier model still talks to the merchant, plans the work, and chooses the tool. Focused jobs go to subagents. Several of those subagents now run open models Shopify post-trains for one task.
Picture a floor manager who still greets the customer. Beside them sits a picking desk that only ever walks the warehouse. The manager plans. The desk fetches stock. That is the specialist split in one shop-floor picture.
A task expert is a post-trained open-model checkpoint they own, serve on their own GPUs, and update when the flywheel produces a better score. Because they control the weights, they also decide when the specialist changes.
| Layer | Frontier planner | GraphQL task expert |
|---|---|---|
| Job in Sidekick | Talk to the merchant, plan the work, choose the tool | Read schema and docs, write and run Admin GraphQL, return results |
| Model in the October write-up | Frontier model API | Post-trained NVIDIA Nemotron 3.5 Lightning |
| GraphQL judge score | 73 | 74.5 after six loops and 61,000 healed examples |
| Median full response (AIPerf replay) | 6.50 seconds | 0.87 seconds |
| Serving cost on that traffic | Frontier API pricing | About 90% lower; 90 to 98 percent lower across Shopify’s post-trained models |
| Who owns the weights | The frontier vendor | Shopify, served on Shopify GPUs |
Those rows are Shopify’s measurements for their GraphQL subagent. They describe how Sidekick is built. Your shop’s Admin GraphQL contract stays the same.
How do they raise the bar before the open model walks in?
Every task expert starts with a quality rubric written by product experts. For GraphQL the rubric covers completeness, execution, and response quality. How closely two experts agree on the same conversations sets the bar.
Shopify then tunes an LLM judge with the GEPA and ACE optimisers until the judge agrees with held-out human labels at the same level as those experts. From that point the judge scores every agent change, every candidate model, and every training run.
An open model waits. First an autoresearch loop raises the frontier-model version of the same agent. It proposes one small change at a time to prompts, tool descriptions, or control flow. It keeps the change only when the judge’s score improves.
The open model has to beat a raised bar.
That is why 74.5 against 73 reads as a win after the harness has already been improved. The open model is competing with a tuned agent rather than a raw API demo.
Which open model, and is it fast enough to bother training?
Shopify skips pretraining. They test open models such as NVIDIA Nemotron inside the real subagent, with the same tools and the same judge. They weigh starting quality, serving speed, and support in NVIDIA Megatron Bridge, the library they train with.
For GraphQL they chose Nemotron 3.5 Lightning. It is a mixture-of-experts model with three billion of its 30 billion parameters active per token.
Out of the box that checkpoint scored 37.7 on the GraphQL judge. The frontier model sat at 73. The gap is the work the flywheel has to close.
Serving is measured before post-training. A model that is too slow for production is not worth the training pass. Shopify benchmarks every workload with NVIDIA AIPerf, an open-source load tester in NVIDIA Dynamo. It replays production traces with synthetic text instead of customer data. Shopify contributed live-response replay, so multi-turn tests hit the server’s prefix cache the way production does.
Those measurements pick the serving stack. For Buyer Profile, a batch job, AIPerf compared vLLM, SGLang, and NVIDIA TensorRT-LLM. TensorRT-LLM did the most work per GPU.
In NVIDIA’s measurements, SimGym, Shopify’s shopping simulator, ran 4.5× faster on Blackwell B200 than on H200. Cost sat 1.48× lower.
Sidekick task-expert flywheel
Will Sidekick learn my store?
Only in a narrow sense.
Self-healing turns a failed production conversation into a training example. Every night the judge scores a sample of production GraphQL conversations. Anything short of a perfect score goes into the loop.
A frontier model acting as critic writes a hint. Never the answer. A simulated merchant then reruns the request with that hint against the same store. If the judge passes the rerun, Shopify removes the hint, anonymises the conversation, and adds it to the training set. If the rerun still fails, the critic tries again, up to three times.
Self-healing ran for 86 nights and repaired 59,535 GraphQL conversations, 69% of the failures it tried. Human annotators corrected the rest. Those corrections join the ground truth that calibrates the judge.
So the specialist improves on conversations that look like native Admin GraphQL. Stock questions. Schema reads. Queries that succeed or fail in a way the judge can score.
A dealer-inventory metaobject, a VIN-deduped CSV, or a private app webhook sits outside that curriculum until Sidekick can query it. Merchant data in the published write-up is described as anonymised before it lands in training.
The specialist inherits your schema. Broken variant titles still return a confident wrong row.
What do the GraphQL scores look like after six loops?
Post-training uses NVIDIA Megatron Bridge. It brings standard Hugging Face checkpoints onto NVIDIA’s Megatron Core training library without rewriting the model. It scales across nodes on Nebius by changing settings rather than code.
Blackwell makes each loop cheaper. On the same FP8 post-training job for Shopify’s Analytics model, eight B300 GPUs trained 1.86× faster than eight H200s. They used 46% fewer GPU-hours. The judge then scores the new checkpoint and the flywheel starts again.
On GraphQL, the scoreboard after six loops is the claim that matters for Sidekick. Out of the box, Nemotron 3.5 Lightning scored 37.7. The frontier model scored 73. The first loop took the open model to 61.5. Six loops later, the healed set held 61,000 examples, and the post-trained model scored 74.5.
Speed moved with quality. In an AIPerf replay of production GraphQL traffic, the post-trained Nemotron returned a full response in 0.87 seconds at the median, while the frontier model’s API sat at 6.50 seconds. The slowest 1% of requests finished 8× sooner.
The same checkpoint also beat another post-trained open model of comparable size. That peer was close at the median and more than twice as slow at p99.
Serving cost on that GraphQL traffic is about 90% lower than the frontier model.
Across Shopify’s post-trained models, self-hosting cuts serving cost by 90 to 98 percent compared with frontier-model pricing.
McNamara and Vegna close the architecture in one line: the frontier model keeps talking to the merchant and deciding what to call, while token-heavy specialist work moves to open models Shopify owns.
What should a merchant team do with the specialist split?
Native GraphQL questions get a specialist that can keep improving, because production traffic grows the training set, human corrections sharpen the judge, and each hardware generation makes the next loop cheaper to run.
Inventory, product, and order questions that already sit on clean Admin GraphQL fields are the workload this flywheel is built to serve. A lean beauty and skincare catalogue such as Filter by Molly-Mae, built for a focused product set a small team can merchandise, is the shape of data that kind of question can read.
Custom stack work stays outside the flywheel until it appears on a path Sidekick can query. RTR Vehicles is an auto and moto dealer-inventory build: a Shopify-native finder on metaobjects, VIN deduplication, a bi-weekly CSV upload, and GraphQL updates the team owns. Sidekick’s GraphQL specialist can ask native product and inventory questions. Dealer stock that lives in that metaobject model is only as visible as the schema Sidekick is allowed to read. The CSV workflow remains the freshness owner.
Admin GraphQL fluency is now part of how Shopify trains its own assistant. The October post is explicit that the specialist reads the API schema and the docs, then writes the query. Slow admin paths, the pattern in How to Fix Shopify Operational Friction When Simple Tasks Still Take Too Long, still shape what the picking desk can fetch. A metafield the team cannot find after the admin redesign is the same field a GraphQL specialist will miss or misread.
Shopper-tab agents are a different surface. Sequencing WebMCP against a staff Claude connector belongs in When Shopify WebMCP Matters vs the Claude Admin Connector. The task-expert loop is Shopify’s internal serving architecture for Sidekick subagents. It is not a merchant-facing model picker.
Clean native fields help the picking desk. Custom stack work still needs an owner on the floor.
If the ops question already lives on product and inventory fields, keep those fields tidy so the task expert can read them. If the truth still lives in a private app or a CSV the team uploads on its own cadence, name an owner for that path. Sidekick chat leaves that work on the floor.
Conclusion
The specialist split on the merchant side is now practical. Native GraphQL questions now sit on a specialist Shopify owns and retrains from production traffic. Custom apps, CSV-owned inventory, and checkout logic your team maintains still need an owner outside the chat.
When those answers still live in a custom app Sidekick cannot see, Shopify app development and a stack assessment map which questions belong on native GraphQL and which still need a maintainable build.



