推理工程大师课 — Philip Kiely 与 Ali Taha，Baseten

We first covered Baseten last year when DeepSeek mania was at peak hype. Now they have raised a monster $13B round and become one of the new cohort of AI Infra decacorns that are (with Nvidia, Intel, and the semis complex) chief beneficiaries of the Inference Inflection.

We return to Baseten at the peak of the 2026 edition of Open Weights debate. Ali has published a viral breakdown of Kimi K3:

And since you last saw him, Philip has spoken at AI Engineer and written the definitive book on Inference Engineering spotted all over SF:

Three years ago, inference engineering barely existed as a category.

Today, it is one of the most critical disciplines in AI. Inference engineering inherently tackles a different question than standard model training: “How do you turn those weights from training into a product that is fast, reliable, and affordable at scale?” Focusing on these creates an entirely new optimization problem.

In one recent GLM-5.2 experiment, quantizing more of the model actually preserved its benchmark quality while increasing throughput by 20%, because the errors introduced in different layers could cancel each other out.

Inference is no longer just the final step after training. It is becoming its own engineering discipline, with its own research problems, infrastructure, and increasingly specialized roles.

In this episode, Baseten’s Philip Kiely and Ali Taha join swyx and Vibhu to explain what actually happens after a new open model is released and what it takes to turn “we generated a token” into a fast, reliable, production-ready API.

We go deep on cache-aware routing, disaggregated prefill and decode, quantization, speculative decoding, KV-cache movement, model parallelism, GPU kernels, and the race to make frontier models up to 10× faster. Philip and Ali explain why inference optimizations can still produce gains of 20%, 100%, or even 200%; how quantization errors can cancel one another out; why identical weights can behave differently across clusters; and how Baseten grafted a Kimi vision encoder onto GLM-5.2 without changing the underlying language model.

The conversation then expands beyond LLMs into NVIDIA Dynamo, mega kernels, Rubin, AI-specific chips, local inference, video generation, diffusion versus autoregressive models, and the enormous compute barrier to generating coherent long-form video. Finally, we explore the convergence of training and inference, continual learning through persistent KV cache, and the emerging loop where models help optimize the infrastructure that runs them.

We discuss:

- What happens when a 200,000-token request enters an inference system

- Cache-aware routing and reusing previously computed KV cache

- Why prefill and decode are increasingly handled by different GPUs

- When dedicated deployments become cheaper and more reliable than shared APIs

- How speculative decoding uses a smaller model to accelerate a larger one

- Tool calling, structured outputs, and what LLMs actually do

- What it takes to support a new open model on day zero

- Grafting Kimi’s vision encoder onto GLM-5.2

- Retrofitting inefficient model layers with components from other architectures

- Why models sometimes collapse into repeating the same token

- How hardware, kernels, and race conditions create nondeterministic failures

- Preserving model fidelity while making inference faster

- How quantization errors can cancel each other out

- Why inference optimizations still deliver gains of 20%, 100%, and 200%

- How optimized serving can make a model up to 10× faster

- NVIDIA Dynamo, KV-aware routing, and distributed model serving

- Speculative decoding the speculative decoder

- Why local AI is about making models less dumb while data-center AI is about making them less slow

- Tensor, expert, and pipeline parallelism across GPUs

- Hardware-aware model design, auto-tuning, and the case against mega kernels

- Rubin and why inference is becoming a systems problem

- Whether modern GPUs are evolving into programmable AI ASICs

- Why enormous models like Kimi K3 require GB300-class hardware

- Why open-source video generation still trails Veo, Kling, and other closed models

- The quadratic attention bottleneck behind long-form AI video

- Autoregressive video, real-time generation, and compounding quality drift

- Why future video systems may combine autoregressive and diffusion architectures

- Training for inference and inference for training

- Continuous post-training, deployment, evaluation, and improvement loops

- How GLM-5.2 helped optimize the kernels serving GLM-5.2 itself

- Why faster networking could unlock dramatically faster decoding

- Continual learning, KV-cache compaction, and persistent model memory

Show Notes

- How to build a day-0 API for Kimi K3

- 22580: From GPT2 to Kimi3, Explained

Philip Kiely

- LinkedIn: https://www.linkedin.com/in/philipkiely

- X: https://x.com/philipkiely

- Inference Engineering: https://www.baseten.co/inference-engineering/

Ali Taha

- LinkedIn: https://www.linkedin.com/in/aliestaha/

- X: https://x.com/waterloointern

Timestamps

00:00:00 Introduction and the 200K-Token Prompt

00:03:18 Dedicated Deployments, Speculative Decoding, and Tool Calling

00:11:26 Launching Production-Ready Open Models

00:19:06 Model Retrofits, Failure Modes, and Nondeterminism

00:28:22 Quantization and Canceling Errors

00:32:15 The Race to 10× Faster Inference

00:40:48 Dynamo, Speculation, and Local vs. Data-Center AI

00:50:18 Model Parallelism, Auto-Tuning, and Mega Kernels

01:00:55 Rubin, GPUs vs. ASICs, and Custom AI Chips

01:10:03 Giant Models and the Limits of GPU Memory

01:12:42 AI Video, Quadratic Attention, and Autoregressive Generation

01:21:47 Audio, Images, and Diffusion Models

01:27:32 Training, Self-Optimizing Models, and Continual Learning

01:40:06 Closing Thoughts

Transcript

Introduction: Baseten, Waterloo Intern, and Inference Engineering

Swyx [00:00:00]: Okay, we’re here in the studio with Philip, old friend from Inference Engineering, the book, as well as Baseten and everything that you’ve done, you and I have done before, as well as Ali. Welcome.

Ali [00:00:15]: Pleasure to meet you.

Swyx [00:00:15]: Waterloo intern.

Ali [00:00:16]: Waterloo intern, always.

Swyx [00:00:17]: When did you get “Waterloo intern” as a handle?

Ali [00:00:19]: As a handle? Oh.

Ali [00:00:20]: I think the rebranding happened mid-March. When I saw it was open, I was like, “I have to take it. Up for grabs.”

Philip [00:00:26]: The problem is that Ali is really good at his job and is not gonna be an intern much longer.

Philip [00:00:30]: So we have to figure out who’s gonna get the handle.

Ali [00:00:33]: Well, I’ll pass the torch over to the next intern.

Swyx [00:00:34]: Oh, okay. It can be, like, you just pass it to another Waterloo grad.

Ali [00:00:37]: To another Waterloo intern. No, bruh.

Philip [00:00:39]: Yeah.

Ali [00:00:39]: Intern.

Swyx [00:00:40]: Intern, yeah.

Ali [00:00:40]: And no.

Philip [00:00:41]: You gotta get an intern from Waterloo.

Ali [00:00:42]: Yeah, I’ve gotta get an intern from Waterloo.

Swyx [00:00:44]: Right.

Ali [00:00:44]: But they have to follow the path.

Swyx [00:00:45]: Oh, it could, but it could come from Baseten, so it’s like whoever Baseten gets from Waterloo.

Ali [00:00:48]: Right.

Swyx [00:00:49]: Has the title of Waterloo.

Ali [00:00:50]: It stays in the ecosystem.

Philip [00:00:51]: Exactly.

Ali [00:00:52]: Halfway through the internship, you either get it or you’re out.

Philip [00:00:55]: You should also do, like, a big graduation ceremony where you change the handle.

Ali [00:00:59]: Just say it.

Philip [00:00:59]: For everybody.

Swyx [00:01:00]: You guys are good at ceremonies, clearly. We had a nice launch of the book, very successful. But before we get into all that, I wanna start off with a fun question for you. Okay, you’re an expert inference engineer. What happens when I send a long query, say two hundred thousand tokens into Baseten’s inference? What’s the process of query through GPU model routing, balancing, all that? What is all the stuff that we don’t think about?

Long Context Requests, KV Cache, and Cache-Aware Routing

Philip [00:01:26]: With a long query specifically, the first thing that I’m gonna ask is, “Have you sent me this query before, or at least part of it?” and I really hope you have, because it’s gonna be a lot easier for me and a lot cheaper for you. So the first thing that we’re gonna look at is some cache-aware routing, where we’re going to see, we probably have a number of instances, a number of replicas up serving whatever model you’re hitting. We want to send this one to something with, number one, available prefill workers, and number two, ideally some cached input already there so that we can skip prefill on at least part of these two hundred thousand tokens. If you’re doing two hundred thousand tokens, it’s probably coding or a multi-turn agent or something where you would expect to have that cached. If you don’t, we’re gonna have to send it to a prefill worker. We’ve at least on certain models disaggregated prefill and decode, so you’re going to have one set of GPUs that’s solely going to process the input, create the KV cache, and get you your first token, and then that’s going to be passed over to a separate set of GPUs, which is going to run decode. We’re going to iteratively make those tokens. We’re probably going to have some speculator model in front of that. I’m going to assume that you’re doing coding, and because of that, our speculator model, which assumes you’re doing coding, is gonna have a high draft token acceptance rate. If I’m wrong and you’re asking me to summarize every Harry Potter book, it’s gonna be slower. And then we stream that output to you and account for it, charge you, a couple of pennies and say, “Hey, would you like to send another one?”

Swyx [00:03:04]: Except Baseten doesn’t charge by pennies.

Philip [00:03:07]: Well, yeah, we charge. I’m assuming that we’re talking about the public model APIs. If you are setting up a dedicated deployment, then yeah, it’s not pennies.

Public APIs vs. Dedicated Deployments

Swyx [00:03:18]: Yeah, one of the key differentiators when I was talking with Baseten initially was that people who want very high volume just need to rent by the box, ‘cause then it’s up to you to figure out how to saturate the box.

Ali [00:03:31]: And more often than not, it’s, like, way cheaper if you’re pushing, like, millions of tokens per hour, if you just pay per hour instead of pay per token.

Philip [00:03:37]: Yeah, they do. I think that we’ve increasingly seen a lot of demand for the pay per token APIs, just because everyone wants to try open models, and then once they find a use case that’s really sticky, then they move over to dedicated.

Swyx [00:03:51]: Is there a best practice on when it’s time to swap over?

Philip [00:03:54]: Couple reasons. Yeah, reliability, that’s a big one, right?

Ali [00:03:57]: Like, if they have a very specific use case, they want you to train something specifically for them, like they want their own spec dec, for instance, for their own traffic.

Swyx [00:04:04]: Spec dec is speculative decoding.

Speculative Decoding and Custom Speculators

Ali [00:04:05]: Speculative decoding, yeah.

Swyx [00:04:07]: You have to explain.

Ali [00:04:07]: Sorry. Like, speculative decoding is like, if you have a huge model, right? And so the model is going to be generating one token at a time every single turn, every single forward pass. So we attach, like, this little, like, parasite, like this layer that goes on top of the model, and this model just has to predict. It does three very fast autoregressive forward passes, and it will predict, like, three certain tokens, and then you do one forward stage over the entire original model in order to see if those predictions were correct or not, and then you accept them or you reject them. Now, this draft model is traffic specific, so if you, like, Philip said, if you’re summarizing Harry Potter books, I can train exclusively that draft model on Harry Potter books, and I can guarantee you that I’m gonna accept the three tokens every single time. And so with that case, I increase your decode speed. I wouldn’t be able to provide this to you if you’re a shared endpoint

Swyx [00:04:53]: Yeah

Ali [00:04:53]: ‘cause I have no idea if you’re doing Harry Potter, if you’re doing coding, if you’re doing English. We don’t know. Also, there was a thing in the book that mentioned that if they really cared about a specific threshold, chapter four, I think. Do you remember that?

Philip [00:05:06]: Yeah. The things that you can do is you can set a specific, like, batch sizing, a specific, like, parallelism strategy if you’re trying to optimize for, like, throughput versus latency. You can. Maybe a NVFP4 quant doesn’t pass your benchmarks and you wanna run a model at higher precision, you could do that. There’s just a bunch of reasons why you might wanna have your own endpoint and the biggest one, of course, just being, like, you don’t have to deal with someone else doing a hundred million tokens of benchmarking traffic at the endpoint when you happen to be trying to serve your users.

Swyx [00:05:40]: Yeah. I think one thing that is. That is a classic journey. Like, it’s people is asking the, what happens when you type Google into the browser. Tool calling, is that just, you’re generating JSON or is there more complication beyond that?

Tool Calling, JSON, and Structured Outputs

Ali [00:05:58]: Certain customers that we have, they have their own post-trained models, and so they demand a tool calling that’s not just, like parse a file or go find the weather. It’s something that’s very specific and you have to do post-training on this. And if the post-training on the model is not good or if the quantization after the post-training to get the inference to be fast, the model will struggle reading the JSON file and reading the tool calling. But it doesn’t require its own like sandbox. It’s not like it’s going to use that tool calling to like escape a sandbox or like it doesn’t have to be contained. It can just be a normal dedicated deployment. The challenge with tool calling more and more seems to be that the companies want certain tool calling which is a very sensitive thing to train. And because you’re dealing with all of the JSON outputs, if it doesn’t like close the end of the request in a very certain manner, you end up with a model that did the tool calling and like the thinking and so as a result of that, it didn’t see the result and just hallucinated the result as it decoded. That seems to be the most challenging thing with tool calling, not really the sandboxes model.

Philip [00:06:56]: Yeah, that’s a challenge on the training side and then on the inference side, there’s work that you can do to scope the possible output. So we published this at this point close to two years ago, the solution to this problem which is you make a state machine and you use that to constrain the output to a specific format. So this is the structured output problem. If you remember back

Swyx [00:07:27]: Yeah, the specific grammar is,

Philip [00:07:29]: Yeah, exactly

Swyx [00:07:30]: GML had this thing.

Philip [00:07:31]: Yeah. So it’s like the old-school “make sure this is only JSON”, return only JSON or

Swyx [00:07:38]: Yeah

Philip [00:07:38]: Grandma’s gonna die type of prompts.

Swyx [00:07:39]: Is it BNF grammar? At some point OpenAI had released a thing that was like, yeah, if you want to constrain your output, write BNF grammar, back as NOR.

Philip [00:07:47]: In our inference system, it’s just a specified output format. And you get the guarantee that your output’s gonna be structured along that format. And so applying that to tool calls can like help cut down on. You can still call the wrong tool or call no tool. It doesn’t solve the certainty problem but it at least solves the output structuring problem

Swyx [00:08:10]: Yeah

Philip [00:08:10]: Within tool calls.

Swyx [00:08:12]: And MCP is just another form of tool, right.

Philip [00:08:14]: Yeah, exactly.

Swyx [00:08:15]: As far as there’s no special thing there.

Philip [00:08:16]: The thing I’m always like explaining to people is the LLM is not capable of doing anything. It’s only capable of making suggestions of what to do and then if those suggestions are formatted in a certain way and applied to a system that knows what to do with them, then an action occurs.

Swyx [00:08:32]: Yeah. Part of the fun stuff is, this is solved outside of tool calling too. Like in an agent loop if the output is not correct or you’re right, like reasoning, tool calling was done in the reasoning trace, just be like, “Oh, I don’t know what to do. Let me just try again.” And it might get there after a few tries. And on your point of training, sometimes this is harder in smaller models, so you don’t have the same exact quality output

Ali [00:08:56]: Right.

Swyx [00:08:57]: When you just swap from a big model, right?

Ali [00:08:59]: Yeah. I will say that, before, I think we need to go back to inference engineering proper.

Ali [00:09:04]: But, I had expected that something would replace JSON because it’s hard to stream JSON ‘cause JSON must be complete and you must have open and close brackets and everything. So it’s hard to parse something or validate something while it’s being streamed. So people invented all sorts of things that are like, I forget the name of some of these alternatives, but it’s something like TOML, something like YAML. But JSON seems to be dominant still.

Philip [00:09:30]: The JSON outputs aren’t that long, right? Like you could have a long-- ‘cause tool calls also contain the arguments in them and perhaps for a certain tool you might pass like a very long argument. But my impression of the median tool call is that it’s a relatively small number of tokens, right? So I would expect that speculators are generally fairly good at something as formatted as JSON. And so you would have like a pretty fast decode step there and that the streaming wouldn’t be as valuable, but maybe I’m wrong about that.

Ali [00:10:02]: I think you’re also bounded by the software or that the model is gonna integrate with if the software is built with JSON for the tool calls or if the company that you’- if your customer says that this is how our software works and our tools are interfaced with JSON, you can ask them to like, change their software and say like, “Yeah, this is gonna be better for the model.” but like with the right training shouldn’t be that much of a difference. Also more profitable if it outputs more tokens probably.

Swyx [00:10:25]: Depends on your business model.

Swyx [00:10:27]: It really depends. But I will say that, as a writer with like experience a lot with generated output, I do try to move from text to JSON text which is very long JSON, right? Like there’s paragraphs in every field because I’m trying to structure it, right?

Philip [00:10:44]: Right.

Swyx [00:10:44]: I want you to first make factual statements, then make opinions then make bullet point summaries, have dates, have entity references have your sources for references, all these things. Anyway, so these are things that like I think people who really experiment with structural output have to really care about. But, let’s, let’s recurse up the stack a little bit. Before we started recording, you mentioned something really cool, which is that there’s a lot of engineering that-- inference engineering that goes on when a new model provider releases a new model, right? So let’s call it GLM-5.2, Kimi K3. I had previously assumed, especially if it’s like, well, GLM 5 to 5.1 to GLM-5.2, like that you’ve supported them before. Is it that much work?

What It Takes to Support a New Open Model

Ali [00:11:26]: It’s a lot of work.

Swyx [00:11:28]: Yeah. Okay. So like, a lot of people, all you guys, right whenever a new model launch like, people rush to say like, “Oh, Hugging Face supports this, Fireworks supports this, Spacetime supports this,” and I’m like, “Yeah, of course we support it.” But what goes into that? What goes into

Philip [00:11:40]: I think it’s more than just support it too, right? It benefits the consumer a lot. Like I think it was with Kimi K2.5 or GLM-5.2 the latest, there was an inference war, right? X provider is at 90 tokens a second. The next day we’re at 150. The next

Swyx [00:11:55]: I kinda kicked that off with the GLM-5.2.

Swyx [00:11:58]: I wrote a Twitter article about. It got like half a million views,

Ali [00:12:02]: Based on being number

Swyx [00:12:03]: Yeah

Ali [00:12:04]: Or it’s for something else.

Swyx [00:12:05]: Yeah. Which,

Ali [00:12:06]: Oh my God

Swyx [00:12:07]: Which then got everyone really excited about, hey, how can we, bend tracks a little bit further and,

Philip [00:12:14]: There’s a difference between support the model, as in I can make a token out of this model, and support a model, as in I have a production-ready API from this model.

Philip [00:12:26]: Getting to the point of I can make a token out of this model is not that hard because generally the, open source inference engines, vLLM, SGLang of the world oftentimes even receive weights ahead of time, maintainers do, or the people making the model merge PRs to ensure support. So you generally can, just get it working on the standard open source stack without too much pain in most cases. The challenge is, every inference company is gonna have own proprietary stack. Some open source components, some in-house stuff. And for any arbitrary model, there’s going to be some new stuff. Sometimes you get lucky, like K, two five to two six was, like, pretty similar.

Quantization, Speculators, and Production Readiness

Ali [00:13:16]: Yeah. It was pure continued post-training

Philip [00:13:18]: Yeah

Ali [00:13:18]: If I remember correctly.

Philip [00:13:19]: Even in those cases, there’s still stuff you have to do. You have to redo the quantization work. You’re taking the model from. Generally, these models are not released in NVFP4, and we want them to be in NVFP4 for maximum Blackwell compatibility. So we have to perform that quantization, and, calibrate the quantization to make sure that we’re not causing any regression in the model’s intelligence. And then we also have to train the speculator, as we’ve talked about. Generally, we have. We have ZDR, zero data retention on our model APIs, so we don’t know exactly the traffic that people are sending us, but we know what’s popular. We know that coding use cases are popular. We know that agents, agentic use cases are popular. So we can get public data sets that are representative of that traffic and train general speculators. Now, with speculators today, you need to train the speculator using the base model itself because you’re getting hidden states out of the model from running inference on these specific prompts, and that is the training data you use to create the speculator. So there’s that process which you need the real model weights for. And then there’s of course just the process of, standing up all the infrastructure behind it, loading all this stuff, testing it. And then when there’s a new model with a newer architecture, I think that, like, the DeepSeek models tend to be the most challenging as they have, like, the most novel architectural stuff going on, model after model. But every new model has something. Kimi K2 had. Oh, sorry, GLM-5.2 had

Ali [00:14:53]: Sparse attention.

Philip [00:14:54]: Yeah,

Ali [00:14:54]: Yeah

Philip [00:14:54]: the DSA.

Ali [00:14:55]: Right. Which is brought from DeepSeek.

Philip [00:14:57]: Yeah. And

Ali [00:14:59]: So you can copy-paste then?

Philip [00:15:01]: It kind

Ali [00:15:01]: I don’t know how this works.

Philip [00:15:02]: So, like we had to, like, build support for that into our runtime. And you’re right, like it is really interesting the way that all of these open source labs borrow from each other. For example, like GLM-5.2 doesn’t have vision. So something that, Haley, a guy on our team, if we could take a look at this, he, like, grafted the Kimi vision encoder onto GLM-5.2.

Retrofitting Vision into GLM-5.2

Ali [00:15:27]: We’ll be training the projector.

Philip [00:15:28]: Exactly. So if you think about, like, the encoder, there’s the encoder, which is the part that looks at the image and turns it into latent information, and then there’s the projector which like

Ali [00:15:38]: You can say latent space. It’s okay.

Philip [00:15:41]: And then there’s the projector that maps it onto, the model itself, and then there’s the model weights. You don’t wanna mess with the model weights because you run a chance of making the model dumber at something else for the purpose of giving it vision. So instead, Haley started with just a projector, which is only a handful of millions of parameters.

Ali [00:16:02]: That would be, yeah.

Philip [00:16:02]: Yeah.

Ali [00:16:03]: Can you show the training one?

Ali [00:16:04]: Like the way it groks

Philip [00:16:05]: Yeah

Ali [00:16:06]: Very interesting.

Philip [00:16:06]: And maybe

Ali [00:16:07]: That right there

Philip [00:16:07]: Maybe Ali, you should take it from here. You’ve got a better

Ali [00:16:10]: Ooh, double the sand

Philip [00:16:11]: Understanding of this than I do.

Ali [00:16:11]: Yeah. You can see, like, he. The way he trained this is really cool. At the beginning, he was training it using just like, “Here’s a picture of a mountain. Can you describe what’s in this mountain?” And that caused it just like the first, learning walls. Like here you can see this all we’re trying to teach it is to translate the encoded. Like it’s already taken the encoder from Kimi K. It’s taken the image. It’

Philip [00:16:31]: Yeah. Frozen

Ali [00:16:31]: Frozen

Philip [00:16:32]: With adapter.

Ali [00:16:32]: Exactly.

Philip [00:16:33]: Yeah.

Ali [00:16:33]: So the brain is frozen and the eyes are frozen. It’s just we’re trying

Philip [00:16:37]: Align

Ali [00:16:38]: Interconnect between the eye and the brain, right? So the projector. And so you take the tokens and then he’s like, “Oh, can you describe what’s in this image?” And he’s like, “Oh, it’s a mountain,” or it’s a person or it’s a human, whatever the case is. But that didn’t cause complete understanding. So he changed it such that every image was associated with a data set of questions. Like, does this image have a white male? Does this image have birds in the top corner? Does this image have a scientist in it? All of that stuff. And it would have to answer questions correctly. And using not just training on describing an image, but being able to answer question, another question, answer over time. Like you can see the grokking, which is like genuinely insane, that retrofitting vision into a large LLM can learn to that extent. And even for images that it doesn’t perform well on, for instance, if you ask it a picture of like Stephen Hawking, “Who is this?” Maybe it doesn’t get it, but it will say something like, “This is Albert Einstein.” Like it still understands

Philip [00:17:25]: Close enough

Ali [00:17:26]: That this is a scientist who is a man who has, some significant achievements, all that stuff. So that’s like really cool.

Philip [00:17:32]: Yeah. So, we’ve covered Hao Tian before, who the author of the LLaVA paper that did this, a while ago. And I think that’s very foundational work for anyone who hasn’t done vision work before.

Ali [00:17:41]: Same with the CLIP and MetaCLIP, where you go from just captioning to building out questions

Philip [00:17:47]: Right

Ali [00:17:47]: Off the image and how much better you can get performance.

Philip [00:17:50]: Right. Right. Right. Yeah. But what’s, what’s so exciting about this is if you look at a model like this. Now, this is a little bit more of a research project. It’s not. It got to 56% on MMLU Pro, I think. So not quite frontier. But if you’re running this model, you haven’t suffered any loss on your GLM-5.2 quality. If you don’t have an image, it’ll just behave exactly the way it used to. And ultimately

Ali [00:18:14]: Which in the inference code you literally do not include the other part, right?

Philip [00:18:18]: Yeah. You would just skip the encoder if you don’t have an image input.

Ali [00:18:22]: Okay.

Philip [00:18:22]: Just confirming.

Philip [00:18:23]: Yeah

Ali [00:18:23]: Does it affect a lot on the overall inference side? Like you’re not adding much, you’re adding a very small vision encoder. These are typically like

Philip [00:18:30]: They’re super fine

Ali [00:18:31]: Less than a billion parameters, right?

Philip [00:18:32]: Yeah. It’s, - There’s a little bit less standardization among vision encoders

Swyx [00:18:37]: Yeah

Philip [00:18:37]: So the support matrix can be a little bit, sparser. But overall, yeah, it’s a pretty, it’s a pretty minor component of the overall system. And ultimately what you get out of the system is all of a sudden you have Kimi Vision, GLM weights, and DeepSeek attention all in one model.

Open Source Model Grafting and Franken-Merges

Philip [00:18:56]: And that’s, I think, a lot of the power and beauty of open source, is that you can take all of these different components and combine them together into a system that’s better than anyone

Swyx [00:19:05]: Yeah

Philip [00:19:05]: Can be individually.

Swyx [00:19:06]: People used to say that you would also do Franken-merges where you would take like

Philip [00:19:10]: Yeah

Swyx [00:19:10]: Layers from each model.

Swyx [00:19:11]: Does anyone do that anymore?

Ali [00:19:13]: Well, to your point previously when you were mentioning like, the work that goes into supporting a model when it first comes out, like GLM-5.2 or MiniMax M3 or whatever the case is. Sometimes you do have to like, you do have to switch out some things. Like, for instance, the MiniMax M3 head uses full attention, and with full attention you end up with this like insane bottleneck in spec dec ‘cause you’re doing auto-regressive token generation for three tokens, and you’re doing this like N squared over all of the tokens that are in your sequence. Your KV cache is like very large because it’s not sparse, it’s not top K. So we find it better to like, okay, we’re gonna replace this, we’re gonna replace this layer with a layer from another model that’s using like GQA, for instance. And then just with the right training, you can get it to have the same acceptance rate. So it is very possible to retrofit layers from other models and very much needed. If a layer is like inefficient, the training just becomes the challenge, like how do you ensure that you train it properly? Which again to your earlier point is like the mesh between training and inference. As in like you need very good training in order to do fast inference. That’s like, I feel like more and more becoming true.

Swyx [00:20:21]: Yeah. Anything else on the support side when you say like get it to fully production ready?

Loop Detection, Race Conditions, and Non-Determinism

Philip [00:20:26]: Yeah. I think that there’s also a question of just, we can test a model to a pretty extensive degree, but we’re trying to get it out quickly and then you see a bunch of other people test it and you get interesting results. There was an issue with, GLM briefly where we had some like mode collapses where it would just output the same token over and over again for certain prompts on certain temperatures. Like once you expose an endpoint to the real world, there’s going to be, so many more varieties of things given to it that you’re able to, discover and patch things. So it’s not just a, day zero process, it’s then like for the first week, for the first month, if a model remains popular, like how do you both fix bugs and then continue to push the envelope on performance?

Ali [00:21:21]: What do you mean you don’t want your model outputting S?

Swyx [00:21:24]: Is there loop detection on that stuff, by the way? It still happens like quite a lot, which is surprising.

Ali [00:21:30]: We have like we, in our endpoint, like if a model was to output the same token like four plus times, we just cut the generation. We say like, “Oh, sorry, this-- Like try again,” or like we will reprocess the request. ‘Cause we know then, like if it, like if, yeah, it’s four times the same token, it’s probably collapsed.

Swyx [00:21:45]: Yeah. Is there a way to opt out in case I really want that?

Ali [00:21:48]: You want that?

Ali [00:21:50]: I think there’s a way that we have to handle it. I’m not exactly certain, but I feel like in certain models, like when they output something like you can imagine, like a table for instance, and so they want, they wanna draw like 12 dashes and 12 dashes. Yeah, I think there’s a way for that to happen. I think we only do it on certain tokens. Like we exclude certain special characters.

Swyx [00:22:07]: Yeah.

Ali [00:22:07]: So we only do it on like certain like S is the most common almost. GLM-5.2

Swyx [00:22:11]: Oh

Ali [00:22:11]: And I think it was DSV 4 as well. Like you’d just have like looping issues where like you literally

Swyx [00:22:17]: It

Ali [00:22:17]: Just have like S.

Swyx [00:22:18]: Yeah. Is there a special, something special about S? No, just randomly

Ali [00:22:21]: It just seems to be the one token involved.

Swyx [00:22:23]: Yeah. And it’

Philip [00:22:24]: Is there

Swyx [00:22:24]: And it’s only temperature 0

Ali [00:22:27]: No

Swyx [00:22:27]: Even at other temperatures

Ali [00:22:27]: Even at like 0.9 or whatever, it will still, it will still collapse.

Swyx [00:22:30]: That’s weird, right?

Ali [00:22:30]: It’s, it is an inference problem to be honest, like a software problem. Like oftentimes, the image you run will-- like NVIDIA will release an image for instance, and if we will upstream the changes from their latest TensorRT-LLM image into our stack, we’ll find that it fixes it. Or oftentimes this will only happen in an inference engine that you’re using like SGLang. But if you were to switch to vLLM, that isn’t the case. So it seems to be like an extremely like deterministic software issue and not really a model issue. It’s not like a weights problem. Like I’- we’ll say like, “Oh, it’s a problem with the quant. We did PTQ wrong,” right? But that isn’t, that doesn’t make sense because the same weights used with a different inference engine does not repeat the problem. And sometimes it’s, the kernels that are being used in the backend have like these very subtle sometimes race conditions, where if you were to use this model hosted on one cluster, you will never get this problem.

Swyx [00:23:19]: Oh my God.

Ali [00:23:19]: But if you host it on a different cluster, you will. And the reason is the KV cache transfer from a node to node in that one cluster is using a slower interconnect than the node to node in another cluster. So that exposes the race, whereas in another cluster it doesn’t. So then you end up just like, okay, this model is not gonna be hosted on this cluster. We’re gonna host it on, another cluster because that cluster exposed that problem. But then it ends up with like, okay, is it the software? Is it the model weights or is it the hardware?

Swyx [00:23:42]: There is a thing about this with temperature 0 still not being deterministic, right?

Ali [00:23:46]: Right.

Swyx [00:23:46]: Mostly because of hardware. Even at temperature 0 same model, you won’t always get the same output.

Swyx [00:23:52]: Even-- But I’m surprised by the race condition one because, I thought PyTorch was a graph that like guarantees that you at least, execute things in the right order.

Ali [00:24:02]: Well, yeah, true. Like I’m not, I’m not saying that there is. Like well, you have things like PTL optimizations where like you can start a kernel before the end of the previous kernel, and that’s like ‘cause you want to do that because there’s

Swyx [00:24:12]: It’s like pipelining

Ali [00:24:12]: Expense. Exactly.

Swyx [00:24:13]: Yeah.

Ali [00:24:13]: But it’- But you don’t do it cleanly. Like you overlap a little bit of the execution. No, it is very possible that the kernel itself, like that one block that is supposed to be running in this instance of time, that kernel itself has a race condition. For instance, like a missing barrier. Like often if you’re designing a kernel and you want it to make it to be very fast, if you don’t test it extensively, you’ll, you’ll have certain threads access data points from registers before they’ve been written to by other threads

Swyx [00:24:36]: Yeah

Ali [00:24:36]: For example, because like your barrier is wrong or your synchronization was wrong. But yeah, like the testing itself is very difficult in those like, and

Swyx [00:24:42]: And there’s no like borrow checker

Ali [00:24:45]: What does that mean?

Swyx [00:24:46]: Like Rust. Like the. If you’re trying to have like memory safety It sounds like a comparable problem.

Ali [00:24:52]: Well, yes, but you’re working in CUDA, right, NVIDIA GPUs. Like- You just need a higher level language like modular Maybe that’s what modular is supposed to do. I don’t know.

Quantization Quality and Vendor Fidelity

Vibhu [00:25:00]: How do you see keeping quality of the model? So you talked about all these steps of, okay, you gotta do quantization, train your own speculative decoder

Ali [00:25:07]: Right

Vibhu [00:25:07]: Run on different hardware. Looking at other model providers, okay, you kicked off a inference speed race on the consumer end. What goes into keeping quality the same across them, right? Sure, you can run benchmarks

Ali [00:25:22]: Yeah

Vibhu [00:25:22]: But, like, how do you determine how much quantization are there standards? What goes into

Philip [00:25:27]: There’s a few things on quality. Most inference optimizations are lossless. KV caching, for example. You are just recomputing or preventing recomputing the same values. Speculation, of course, if a draft token is wrong, it gets rejected. The main lossy optimization is quantization. And that really comes down to, number one, data format, number two, which parts of the model you choose to quantize, which layers, and number three, like doing a lot of calibration on the quantized weights, to ensure that you’re preserving all the outliers. There’s other tricks that you can do, though. A big one is long context, ‘cause one thing you asked at, right at the beginning is, “Oh, what’s gonna happen if I send a 200,000 token request in?” So with a long input sequence, you need to, store a lot more information. You need to process a lot more tokens. And so even if a model has a context of a certain length, you might, as an inference provider, choose to build an API with a shorter context length, and of course a full length one as well. Because if someone doesn’t need the full million token context, for example, you can get them better performance. I don’t know if that’s exactly like quality of the model. The way that I think about quality is to what degree are we faithfully serving the original model? If you think of a golden implementation of a model that performs exactly the way the model is designed to perform, I think of quality as how close are we getting to that, 100% fidelity of the model.

Philip [00:27:13]: You can also, of course, think about quality from the training side and how do you push yourself past 100%. But when I think about purely inference optimizations, it’s getting faster while staying as close to that 100% fidelity mark as possible. And certainly our standard internally is that, like you should not be able to tell the difference between our API and a, official API. I think Kimi in particular does a good job of vendor benchmarking here

Ali [00:27:41]: Yes

Philip [00:27:41]: Where they have

Ali [00:27:42]: They released an actual vendor benchmark.

Philip [00:27:43]: Exactly, yeah.

Ali [00:27:44]: ‘Cause they accused, some people, Amazon? There was some provider that was not doing very well on Kimi’s benchmark.

Philip [00:27:50]: Yeah.

Philip [00:27:51]: So, with Reflect we probably

Vibhu [00:27:52]: This was a long time ago, right?

Philip [00:27:54]: No.

Ali [00:27:54]: Yeah, like three

Vibhu [00:27:55]: They also

Ali [00:27:55]: Four, five months ago

Vibhu [00:27:57]: This also happened with, I don’t remember which model, but they pulled out quite a few, and then they started a whole chart about this. It might have been

Philip [00:28:03]: Kimi Vendor Verifier.

Ali [00:28:04]: Yeah.

Philip [00:28:05]: Yeah.

Ali [00:28:05]: Yeah, ‘cause you, ‘cause you’d be pissed, right? Like if you’

Philip [00:28:07]: Yeah.

Ali [00:28:07]: If like if I’m a consumer and I’m using like Amazon’s endpoint for instance, and I’ve used Kimi and I’m like, “Oh my God, like this is bad,” I’m not gonna say, “Oh, Amazon quantized the model in a bad way.” I’m gonna say, “Oh, Kimi sucks.” Right?

Philip [00:28:17]: Yeah.

Ali [00:28:17]: So it seems like that makes sense.

Philip [00:28:19]: Yeah, they care. They care.

Vibhu [00:28:21]: Justifiably.

Ali [00:28:21]: Yeah, justifiably.

Vibhu [00:28:22]: This is probably a stupid question, but just checking, has anything improved from main quantization?

Philip [00:28:28]: Yeah.

Vibhu [00:28:28]: Like, is quantization always strictly worse?

Ali [00:28:30]: Well technically

Vibhu [00:28:32]: No

Ali [00:28:32]: It’s a lossy. Quantization

Philip [00:28:33]: Yeah

Ali [00:28:33]: Is a lossy, it’s a lossy implementation.

Philip [00:28:36]: Speed improves

Vibhu [00:28:36]: Speed improves.

Ali [00:28:37]: It the number, like

Vibhu [00:28:38]: No, I’ always look for inverse scaling laws.

Philip [00:28:40]: Yeah.

Ali [00:28:40]: Yeah.

Vibhu [00:28:40]: This is something I learned from Noam Brown, where like things that normally act in one direction sometimes do.

Philip [00:28:45]: Well, technically when you run a benchmark, because these models are deterministic, sometimes your,

Ali [00:28:52]: Yeah

Philip [00:28:52]: NVFP4 quant is like, two basis points higher than your

Ali [00:28:56]: No, it’s noise. It’s noise.

Philip [00:28:57]: Yeah, exactly. I’m like, yeah, it’s, it’s within. That’s why I always say within margin of error.

Philip [00:29:01]: And I stopped saying that because everyone assumes that what is, well, within some margin of error, we’re barely inside of that to the worst, so we’re saying. But yeah, sometimes it’s just like, gives you a higher output score. But like Ali said, that’s noise. To my knowledge, you’re not necessarily making the results better. You’re just trying to, again, like keep your fidelity as close to 100% to the original model.

Layer Selection, KL Divergence, and Better Quantization

Ali [00:29:27]: There is, to your point, research that we did on MP. I don’t know if you are able to pull

Philip [00:29:31]: Yeah

Ali [00:29:32]: A tweet we did. One of our research interns, Joshua, I think it’s a tweet on how we have 20% better quantized GLM-5.2 than NVIDIA. Essentially what we found throughout like this month research is, okay, quantization is a lossy. It’s. You’re compressing the data from, occupying 16 bits to occupying, four bits, for instance. And so you’re losing some information, and you’re trying to minimize that. And so when I say that I’m gonna quantize the model, my job becomes how do I find the layers that I can quantize, and how to find the layers to not. For instance, with image models, I don’t quantize modulation layers, and I don’t quantize out projections because those two are. Like out projection is what you see as the user. Modulation is what the model sees or understands. Right, exactly. And so to his paper, do you have the. It doesn’t have the. Yeah. It’s a long paper. I don’t know if I can find

Vibhu [00:30:25]: If there’s a part to search or it’s probably in the thread.

Ali [00:30:28]: It’s probably in the thread.

Vibhu [00:30:29]: Yeah.

Ali [00:30:29]: But the long and the short is it is very possible that quantizing more of the model makes the results. Like if I have a model that I quantize layers one, five, and 10, and another model where I only quantize layers one and It is possible that the model in which I quantized more information is going to perform better because the quantization errors have canceled out. And so what Joshua showed in his mathematical proof where he had like a verifier in, is that you can predict which layers are going to have quantization errors that will cancel out with each other, and you choose to quantize those layers. And so the result of doing this mathematical quantization is you end up with a model that’s 20% more quantized than another provider, so you get 20% more throughput of it because there’s more layers than running an NVFP4, and your quality is better than that other quant because the layers that you chose to quantize have their errors cancel out, like one layer skewed to the right one layer skewed to the left, one layer skewed to the right. Your final logits distribution is more similar to the original distribution of the model, so you have better fidelity. And so the way we proved this was with KL divergence. So instead of just scoring on the benchmarks, we scored the KL divergence between the logit distribution of the quantized model and the logit distribution of the original full precision model, and we showed that with this technique we get. If your probability distribution on the logits which token it wants to select is more of the same as the original model, you’re probably gonna end up staying true to the original model. So yeah, so it seems like previously before this, it seemed like the industry was, well, the more you quantize, the worse it’s gonna be, ‘cause the more loss you introduce. That’s not exactly, not necessarily true. So yeah, doesn’t improve it, but can cancel out.

Philip [00:31:57]: I think it might be this, but reminds me a good bit about pruning where you can prune off certain layers.

Philip [00:32:03]: But very interesting. Didn’t know this was a whole paper you guys put out.

Ali [00:32:06]: It’s. Fun fact, it was originally 72 pages, this paper, and then we decided

Philip [00:32:11]: Wow

Ali [00:32:11]: We can’t tell. We couldn’t release it. So it’s now 45.

Swyx [00:32:15]: Still 39 pages, so very substantive. We talked about evals and all these things and, like what’s possible in terms of speedup? Like it’s like probably like the number

Inference Speedups and Benchmarking

Swyx [00:32:25]: Thing that people do wanna care about, and it’s something that you wrote about in your post. Like official API is 70 tokens per second, and you push it up to 90. Is that like a normal thing?

Philip [00:32:36]: So what’s cool about working in inference, the reason that I think inference is going to be a useful place to do engineering for a long time, is that if you look at highly optimized domains like, say, finance, if you’re in finance, you measure how much better you got in basis points. It’s like, “Oh, I got five basis points better, like twentieth of 1% better,” that’s huge news because everything is so optimized. When we publish optimizations, it’s 20%, it’s 100% it’s 200%. So there’s still probably like a lot further to go, honestly. Like you’ll, you’ll know that inference is pretty much solved when researchers start publishing about how they got 1% faster at something.

Swyx [00:33:19]: Which by the way, because I am from the finance background, in the ‘70s, that was the margin at the time. When you did quantitative finance research, you would find

Ali [00:33:27]: And like 20%, tens of percent.

Swyx [00:33:29]: That’s. Yes.

Philip [00:33:29]: Yeah.