Event Ingestion: From Streaming to Your Lakehouse

A batch pipeline that runs every hour or every night works fine as long as nobody needs to know what's happening right now. The problem shows up when the business actually does need that: catching a fraudulent transaction before it settles, alerting on an IoT sensor that's drifted out of range, or feeding an operational dashboard that an on-call team is watching live. In those cases, waiting for the next batch window isn't a minor technical limitation — it's an outright failure to meet the requirement. The answer isn't "run the batch more often," since that just moves the problem downstream, but switching the ingestion model to one built for continuous event flows.

The publish-subscribe model: topics, partitions, and consumer groups

Event streaming platforms (Kafka, Kinesis, and their equivalents) solve one specific problem: decoupling whoever generates an event from whoever consumes it. A producer writes to a topic without knowing who's going to read it or how many readers there are; consumers subscribe to the topic without knowing who wrote to it. This lets a single event feed multiple downstream systems independently, with no direct coordination between them.

Internally, a topic isn't a single log — it's split into partitions. Each partition is an ordered, immutable log, and here's the important nuance: order is only guaranteed within a partition, not across the whole topic. If you need all events from the same customer or device to be processed in order, you have to partition by that key (for example, customer_id), so they always land in the same partition.

Parallelizing consumption is handled through consumer groups: multiple consumers are grouped under a shared identifier, and the system distributes partitions among them so that each partition is read by exactly one consumer in the group at a time. This gives you horizontal scalability almost for free, but it also sets a practical ceiling: there's no point running more active consumers than partitions — the extras just sit idle.

Delivery semantics: at-least-once, exactly-once, and idempotent consumers

Every messaging system has to answer an uncomfortable question: what happens if a consumer processes an event and fails before confirming it? There are three delivery semantics to choose from:

• At-most-once: receipt is acknowledged before processing. If something fails afterward, the event is lost. Rarely acceptable outside of non-critical metrics.
• At-least-once: acknowledgment happens after successful processing. If a failure occurs partway through, the event gets retried, which can produce duplicates. This is the default behavior in most production configurations.
• Exactly-once: each event is processed exactly one time, with no losses and no duplicates. It's the most desirable semantics on paper, but it requires transactional producers, consumers that participate in that transaction, and extra coordination between the broker and the destination sink. The cost is added latency and operational complexity.

Chasing exactly-once across the entire end-to-end chain usually costs more than it solves; designing idempotent consumers on top of at-least-once solves the same problem with far less friction.

An idempotent consumer assumes it will receive duplicates and neutralizes them at write time: it uses a natural identifier from the event as a key, performs an upsert instead of an insert, or deduplicates against a window of recent events before applying the change. This shifts the "exactly once" guarantee from the messaging infrastructure to the business logic — which is exactly where it matters most that there are no duplicate side effects (a bank transaction counted twice, an alert fired twice).

Micro-batch vs. continuous processing

Even within a streaming architecture, there's one design decision that drives a large share of the operational complexity: whether to process in micro-batches or in continuous mode.

Micro-batching accumulates events over a short window (seconds, not hours) and processes them as a small batch. It's the default model for most structured streaming engines: each trigger processes whatever accumulated since the previous trigger. The advantage is that it reuses all the existing batch-processing machinery (scheduling, fault tolerance, checkpoints) with minimal changes, and it's much easier to reason about and debug.

Continuous processing, by contrast, processes each event (or very small groups of events) as soon as it arrives, without waiting to close out a window. Latency drops from seconds to milliseconds, but the operational cost rises: backpressure has to be managed much more precisely, because if the consumer can't keep up with the producer there's no "next batch" to absorb the lag — pressure just builds up in real time. The usual response to backpressure is scaling out consumers, adding partitions, or, as a last resort, accepting a longer retention window on the topic while consumption catches up.

Schema management: the schema registry pattern

Event payloads change over time: a field gets added, another gets renamed, a data type changes. The problem is that producers and consumers are deployed independently, so a schema change on the producer side can silently break a consumer that wasn't updated at the same time.

The pattern that solves this is the schema registry: a central service that stores every schema's versions and validates the compatibility of each new change (backward, forward, or both) before allowing it to be published. Instead of embedding the full schema in every event, the serialized message (typically Avro or Protobuf) carries only a reference to the registered schema, and the consumer resolves it against the registry when deserializing.

This has two practical benefits: it reduces the size of each event (you're not repeating the schema in every message), and it turns contract changes into something validated automatically at publish time, rather than something discovered when a consumer starts failing in production.

When to actually use it

Real-time event ingestion isn't free: it means operating messaging infrastructure, monitoring consumer lag, managing partitioning, and absorbing the complexity of exactly-once semantics or idempotency. Before adopting it, it's worth checking three things:

1. The latency requirement is real, not aspirational. If the answer to "how fast do you actually need this data?" is "having it in tomorrow's daily report is fine," a scheduled batch is simpler, cheaper, and easier to debug. Save streaming for cases where a difference of seconds or hours changes a business decision: fraud, infrastructure monitoring, operational alerts.
2. Volume and load variability justify the infrastructure. Low, predictable volume doesn't need fine-grained partitioning or elastic consumer groups; a batch every 15 minutes can perform practically the same with a fraction of the operational effort.
3. The team can operate this sustainably. A poorly monitored streaming pipeline — no consumer lag alerts, no dead-letter queues, no schema management — is worse than a simple batch, because it fails silently and is much harder to diagnose.

In practice, the most common and most sensible pattern is hybrid: ingest the raw event in streaming mode into the base layer of the lakehouse (so you don't lose a single event and keep full traceability), and resolve the heavier transformations — aggregations, joins with dimensions, business models — in batch or micro-batch on top of that base layer, unless the final consumer itself (an operational dashboard, an alert) demands second-level latency. That way you get the no-data-loss guarantee that streaming provides, without paying the cost of continuous processing at every layer of the pipeline.