流式处理与批处理：数据处理的两种哲学
文章探讨批处理与流处理在完整性与延迟间的权衡，并概述微批、窗口、水印、Lambda/Kappa 架构及精确一次处理等策略。
中文处理结果
每个处理数据的系统最终都必须回答一个问题：数据何时才算足够完整，可以进入计算阶段？
一个汇总当天销售额的程序需要知道今天的所有销售数据是否真的都已到达。对于存储在文件中的数据，答案是显而易见的，因为文件有结尾。然而，对于持续到达且永不停歇的数据，则没有明确的答案；系统如何弥合这一差距，正是批处理与流式处理之间的区别。
批处理等待完整性。它收集数据直到一个自然边界、结束时间或完整文件，然后一次性对整个数据集进行计算。流式处理则为了速度而优先考虑完整性。它从仍在到达的数据中持续产生答案，这意味着它必须估计何时有足够的数据到达，并处理该估计错误的情况。完整性与延迟之间的这种权衡，是处理流式和批处理时需要考虑的关键因素。
在本文中，我们将介绍两端的策略以及各自的代价。
- 在批处理方面，包括全量加载、增量加载和大窗口聚合，微批处理介于两者之间。 - 在流式处理方面，涉及滚动窗口、滑动窗口和会话窗口，水印与迟到数据，Lambda 和 Kappa 架构，以及常常被误解的精确一次处理含义。
边界
阅读更多
原始正文摘录
Streaming vs Batch: Two Philosophies of Data Processing
Every system that processes data eventually has to answer one question. When is the data complete enough to be moved to the compute stage?
A program adding up a day’s sales needs to know whether all of today’s sales have actually arrived. For data stored in a file, the answer is trivial because the file has an end. However, for data that arrives continuously and never stops, there is no clean answer, and how a system resolves that gap is the difference between batch processing and streaming.
Batch processing waits for completeness. It collects data up to a natural boundary, a closing time, or a finished file, and then computes over the whole set at once. Streaming prioritizes completeness for speed. It produces answers continuously from data that is still arriving, which means it has to estimate when enough data has come in and handle the cases where that estimate is wrong. This trade-off between completeness and latency is the key consideration when dealing with streaming and batch.
In this article, we will cover the strategies on each side and what each one costs.
- On the batch side, that means full and incremental loads and large-window aggregation, with micro-batch sitting in between.
- On the streaming side, the territory runs through tumbling, sliding, and session windows, watermarks and late data, the lambda and kappa architectures, and the often-misunderstood meaning of exactly-once processing.
Boundaries
Read more