EP223：Ollama vs vLLM vs SGLang
ByteByteGo 通讯对比 Ollama、vLLM、SGLang 三种推理引擎，并介绍 Claude 文本水印原理。
中文处理结果
超过80%的容器支出被浪费。以下是如何解决。（赞助）
许多团队过度配置容器，未充分利用竞价实例，并且对哪些 Pod 在消耗预算没有可见性。获取 Datadog 的电子书，其中涵盖了针对 Kubernetes 和 ECS 环境的五种实用优化，并提供了您的团队今天即可应用的具体技术。
您将学习如何：
- 找出消耗云预算的空闲容器、过度配置的 Pod 和未使用的集群。
- 通过资源请求、限制和自动化成本建议来调整 CPU 和内存大小。
- 使用竞价实例和节省计划削减高达90%的成本，并确切知道何时使用它们。
获取电子书
本周的系统设计复习：
- Ollama vs vLLM vs SGLang
- Claude 的文本水印是如何工作的？
- 你应该知道的 12 项智能体技能
- Git 工作流：基本命令
- Apache Kafka vs. RabbitMQ
Ollama vs vLLM vs SGLang
要在您的机器上使用开放权重模型，您有三个主要选择：Ollama、vLLM 和 SGLang。但每个引擎处理请求的方式不同。下图显示了每个引擎之间的差异及其背后的主要技术。
Ollama：本地用户调用 OpenAI 兼容 API，请求在 FIFO 队列中排队。然后 Ollama 运行预量化的 GGUF 模型（一种它拉取的压缩格式），响应返回给用户。
Ollama 最适合本地开发、原型设计和笔记本电脑级硬件。
vLLM：许多用户同时访问服务器，连续批处理将新请求插入正在运行的批次中，而不是让它们等待批次完成。PagedAttention 存储 KV 缓存，即模型为已处理的 token 保留的内存。
vLLM 最适合高流量服务、最大化 GPU 利用率和数千个并发请求。
SGLang：智能体和多轮聊天发送的请求提示词重叠严重。前缀感知调度器通过 RadixAttention 缓存路由它们，该缓存是一个基数树，重用每个共享前缀而不是重新计算。
SGLang 最适合 AI 智能体和工具循环、多轮聊天以及 JSON/正则表达式输出。
Claude 的文本水印是如何工作的？
Anthropic 最近分享了他们对文本加水印的意图，以便识别 AI 生成的文本。这篇文章基于我对它如何工作的理解。
LLM 逐字生成文本。每一步，它们为下一个可能的词生成概率。水印技巧不是从这些词中随机采样，而是改变允许选择哪些词。
如何给响应加水印？
步骤 1：模型为下一个词生成概率。
步骤 2：通常，随机数生成器从好的候选中选择一个。使用水印，一个带密钥的函数接受一个密钥加上前几个词，并决定哪些候选是有效的。
步骤 3：对整个响应重复此过程。存在多个合理选择的地方携带水印信号。
如何检测带水印的文本？
原始正文摘录
EP223: Ollama vs vLLM vs SGLang
Over 80% of container spend is wasted. Here’s how to fix it. (Sponsored)
Many teams over-provision containers, underuse spot instances, and have no visibility into which pods are burning budget. Get the eBook from Datadog, which covers five practical optimizations for Kubernetes and ECS environments with specific techniques your team can apply today.
You’ll learn how to:
- Pinpoint idle containers, over-provisioned pods, and unused clusters draining your cloud budget.
- Right-size CPU and memory with resource requests, limits, and automated cost recommendations.
- Cut costs up to 90% with spot instances and savings plans and know exactly when to use each
Get the ebook
This week’s system design refresher:
- Ollama vs vLLM vs SGLang
- How does Claude’s text watermark work?
- Top 12 Agent Skills You Should Know
- Git Workflow: Essential Commands
- Apache Kafka vs. RabbitMQ
Ollama vs vLLM vs SGLang
To use open-weight models on your machine, you have three main options: Ollama, vLLM, and SGLang. But each engine handles requests differently. The diagram below shows the differences and the main techniques behind each engine.
Ollama: A local user calls the OpenAI-compatible API, and requests line up in a FIFO queue. Then Ollama runs a pre-quantized GGUF model, a compressed format it pulls, and the response comes back to the user.
Ollama is best for local dev, prototyping, and laptop-scale hardware.
vLLM: Many users hit the server at once, and continuous batching slots new requests into the running batch instead of making them wait for it to finish. PagedAttention stores the KV cache, the memory a model keeps for tokens it has already processed.
vLLM is best for high-traffic serving, max GPU utilization, and thousands of concurrent requests.
SGLang: Agents and multi-turn chats send requests whose prompts overlap heavily. A prefix-aware scheduler routes them through the RadixAttention cache, a radix tree that reuses every shared prefix instead of recomputing it.
SGLang is best for AI agents and tool loops, multi-turn chats, and JSON/regex outputs.
How does Claude's text watermark work?
Anthropic recently shared their intent to watermark text so they can identify AI-generated text. This post is based on my understanding of how it works.
LLMs produce text word by word. At each step, they generate probabilities for the next likely word. Instead of sampling randomly from those words, the watermarking trick changes which words are allowed to be picked.
How to watermark a response?
Step 1: The model produces probabilities for the next word.
Step 2: Normally a random number generator picks one of the good candidates. With watermarking, a keyed function takes a secret key plus the previous few words and decides which candidates are valid to pick from.
Step 3: This repeats for the whole response. Places where there are multiple plausible choices carry the watermark signal.
How to detect a watermarked text?
Step 1: For any candidate word in the text, we check whether it is a valid choice based on the secret key and the few preceding words. If the word is valid, that is counted as a match.
Step 2: Run this across the entire text. Watermarked text matches far more often. The overall match rate can be treated as an AI-generated score.
I’m personally getting quite annoyed by the false negatives from all these AI text detection techniques, especially for technical writing.
What's your thoughts on AI text detection? Do you think AI text detection is useful, or will it create more problems?
Top 12 Agent Skills You Should Know
Agent skills are instructions and scripts that teach your LLM agent a new skill. The diagram below shows the 12 most-starred skill repos on GitHub as of August 2026.
- Superpowers (obra/superpowers): This skill makes your agent plan before it writes code.
- skills (mattpocock/skills): Matt Pocock's personal skill set makes your agent challenge your plan first. This is useful as agents can sometimes be too soft.
- andrej-karpathy-skills: Multica AI distilled Karpathy's advice on AI coding pitfalls into one skill.
- everything-claude-code: Skills that help you set up your coding agent. This is useful when you are starting Claude Code from scratch.
- skills (anthropics/skills): This is Anthropic's official skills. It makes your agent capable of creating outputs like Word or PDF files.
- ui-ux-pro-max-skill: This has instructions that teach your agent how to prevent AI-like designs.
- caveman: Julius Brussee's skill makes your agent reply in short caveman speak.
- ponytail: Dietrich Gebert's skill teaches your agent how to write code that is simple and clean.
- agent-skills: Google's Addy Osmani included production-grade engineering practices in a skill
- graphify (safishamsi/graphify): This skill converts a codebase into a knowledge graph, so an agent can navigate easier.
- Understand-Anything: Egonex AI converts a codebase into visual maps to explore.
- impeccable (pbakaus/impeccable): This skill makes an agent better at UI polish.
Over to you: Which skill would you add to this list?
Git Workflow: Essential Commands
Git has a lot of commands. Most workflows use a fraction of them. The part that causes problems isn’t the commands themselves, it’s not knowing where your code sits after running one.
Working directory, staging area, local repo, remote repo. Each command moves code between these. Here’s what each one does.
- Saving Your Work: “git add” moves files from your working directory to the staging area. “git commit” saves those staged files to your local repository. “git push” uploads your commits to the remote repository
- Getting a Project: “git clone” pulls down the entire remote repository to your machine. “git checkout” switches you to a specific branch.
- Syncing Changes: “git fetch” downloads updates from remote but doesn’t change your files. “git merge” integrates those changes. “git pull” does both at once.
- The Safety Net: “git stash” is your undo button. It temporarily saves your uncommitted changes so you can switch contexts without losing work. “git stash apply” brings them back. “git stash pop” brings them back and deletes the stash.
Apache Kafka vs. RabbitMQ
Kafka and RabbitMQ both handle messages, but they solve fundamentally different problems. Understanding the difference matters when designing distributed systems.
Kafka is a distributed log. Producers append messages to partitions. Those messages stick around based on retention policy, not because someone consumed them. Consumers pull messages at their own pace using offsets. You can rewind, replay, reprocess everything. It is designed for high throughput event streaming where multiple consumers need the same data independently.
RabbitMQ is a message broker. Producers publish messages to exchanges. Those exchanges route to queues based on binding keys and patterns (direct, topic, fanout). Messages get pushed to consumers and then deleted once acknowledged. It is built for task distribution and traditional messaging workflows.
The common mistake is using Kafka like a queue or RabbitMQ like an event log. They’re different tools built for different use cases.
Over to you: If you had to explain when NOT to use Kafka, what would you say?