From Batch to Micro-Batch Streaming: Lessons Learned the Hard Way in a Delta Index Pipeline

"Many batch pipelines are limited more by scheduling and orchestration delays than by processing cost. Running micro-batches continuously can remove most of that latency without requiring record-level streaming."

"Record-level streaming is often proposed as the 'correct' solution, but in batch-oriented systems it introduces unnecessary operational risk without delivering meaningful benefits."

"For object store-based ingestion, especially in systems with eventual consistency, relying on success files or completion markers breaks down in practice, and deterministic, rate-based progress is often more reliable for micro-batch streaming."

"Long-running streaming jobs should be built to restart cleanly and regularly, treating restarts as a normal operational mechanism rather than a failure condition."

Many batch pipelines operate in a near-continuous mode, processing incremental data frequently to minimize freshness gaps. Migrating scheduled batch jobs to a micro-batch model using Spark Structured Streaming can eliminate scheduling delays and enhance operational predictability. Record-level streaming introduces unnecessary risks in batch systems without significant benefits. For object store ingestion, deterministic, rate-based progress is more reliable than success files. Long-running streaming jobs should be designed for clean restarts, treating them as normal operations rather than failures.

Read at InfoQ