The Illusion of Complexity: Why Your Brain Thinks Data Streaming is Hard (And Why It’s Wrong)
There’s a well-known saying: give a small boy a hammer, and everything he encounters will need pounding. For decades, the engineering world has been handed a powerful hammer: batch processing. It’s the first tool most of us learn, and it has shaped how we see our work.
We invented batch processing not because it was a natural fit for reality, but because it was a workaround for the technological limitations of the time. We couldn’t drink from the firehose, so we built dams to collect data in manageable chunks. The underlying reality, however, is that data has always been created in real time. A customer purchase, a stock trade, a sensor reading, these things have always happened in the moment, not in discrete batches.
Our technology has finally caught up with reality.
Stream processing lets us engage with the flow of data directly instead of waiting for it to accumulate. Yet, despite being a more faithful model of the world, it’s often shadowed by a persistent narrative that it’s just “too complex.”
This is a cognitive illusion. This is a symptom of a deep-seated “batch-first” mindset forged out of historical necessity. When our mental default is the workaround, the direct solution feels foreign. We mistake the friction of shifting our perspective for the complexity of the subject itself.
This article will deconstruct that illusion, exploring the psychological traps that make us cling to the familiar and showing how this pattern has repeated throughout history, from programming languages to the 19th-century “War of the Currents” over electricity. By the end, you’ll see that stream processing isn’t the problem; the hammer we’re all holding is.
How Your First Solution Blinds You
In an experiment in the 1940s, people were asked to measure a specific amount of water using three different-sized jars. The first few problems all required the same complex, multi-step formula. After establishing this pattern, the researchers presented a new problem that could be solved with a much simpler, two-step method. The result? The vast majority of participants who had been primed with the complex solution failed to see the simple one. They stuck with the only method they knew, even though it was inefficient.
This phenomenon, known as the Einstellung effect, is a cognitive trap where our brains get stuck on a familiar solution, preventing us from seeing better alternatives.
It’s a startlingly precise analogy for engineering. For the last 50 years, the industry has been built on databases and batch systems. It’s the first, and often only, “formula” we learn, becoming our default problem-solving set.
When a problem arises that is a natural fit for streaming, the Einstellung effect kicks in. Our minds default to the familiar batch-oriented path, and we try to frame the problem in terms of discrete jobs and bounded data. The streaming approach seems foreign and is dismissed as “complex,” not because it is, but because our brains are stuck on the first right answer we ever learned.
This initial bias is then reinforced by other psychological traps.
Confirmation bias leads us to seek out articles and opinions that validate our belief that streaming is hard, while ignoring evidence to the contrary. And the sunk cost fallacy makes us reluctant to devalue the decades of investment, in both skills and infrastructure, that we’ve poured into the batch world. Together, these forces create a powerful mental block that keeps the hammer of batch processing firmly in our hands.
You’ve Felt This Before: Parallels That Prove the Point
This mental resistance to a new paradigm isn’t unique to data engineering. It’s a recurring pattern in human innovation. Once you know what to look for, you can see it everywhere.
Imperative vs. Functional Programming
Anyone who has learned to code has felt this. Most programmers start with an imperative style, where you write a sequence of commands that modify the program’s state. It’s a step-by-step recipe.
Functional programming is a different world. It’s about describing a series of stateless data transformations. To an imperative programmer, functional code often feels “backwards” or like “solving a puzzle”. Yet, programmers who learn functional programming first often find it more intuitive.
The difficulty isn’t with the concep, it’s with unlearning.
In the context of data, batch processing is like imperative code, operating on a known, mutable state. Stream processing is like a pure function, a stateless transformation applied to data as it flows past.
Learning Mandarin as an English Speaker
English is built on a finite, 26-letter alphabet and uses pitch to convey emotion. Mandarin invalidates both of these rules.
It uses thousands of unique characters instead of an alphabet, and pitch (tones) determines a word’s fundamental meaning. The syllable ma can mean “mother,” “hemp,” “horse,” or “to scold,” depending entirely on the tone.
It’s not that Mandarin is necessarily more complex than English. It’s that to an English speaker, Mandarin forces the learner’s brain to operate on a completely new set of rules.
The War of the Currents
In the late 19th century, a fierce battle raged between Thomas Edison’s Direct Current (DC) and Nikola Tesla’s Alternating Current (AC).
Edison’s DC was the simple, established incumbent. It worked well for lighting up dense city blocks but couldn’t transmit power efficiently over long distances. Tesla’s AC was the more abstract, scalable challenger.
Edison waged a public fear campaign, electrocuting animals to frame AC as dangerously complex and protect his DC investments. Ultimately, AC’s superior ability to handle a distributed, high-scale world won out. DC was the batch processing of its day, effective for some problems but unscalable. AC was the stream processing paradigm, built for the world of tomorrow. The resistance was about protecting the status quo, not objective merit.
How Batch-First Thinking Holds Us Back
The batch-first mindset has real-world consequences that hold the industry back.
It leads to a kind of architectural myopia. When faced with a real-time problem, a batch-oriented mind defaults to what it knows, creating systems where streaming is awkwardly bolted on as an afterthought. This leads to brittle, overly complex systems that are a nightmare to maintain. A common anti-pattern is streaming events into a data warehouse, running a batch process to transform them, and then streaming the results back out to an operational database. This entire multi-hop, multi-technology process could have been done simply and efficiently in-stream, but the batch mindset forces the data into a familiar container, creating unnecessary latency and complexity.
The core conceptual error is that we’ve been taught to see the world backward. Batch processing isn’t the foundation; it’s a specialized form of streaming. A batch is simply a stream that has a beginning and an end. You can always slow down a real-time stream to create a batch, but you can never speed up a batch to create a real-time stream.
This bias is systematically installed from day one of our training. Engineering curricula, from university courses to professional bootcamps, almost universally teach databases and batch processing as the foundation. Core concepts like ETL, data warehousing, and SQL on static datasets come first. Stream processing, if it’s covered at all, is positioned as an “advanced topic” for the final modules, something to be learned only after the fundamentals are mastered. This educational sequencing is the primary engine that pre-programs the Einstellung effect into the next generation of engineers, ensuring the cycle continues.
Becoming Data Bilingual
The feeling that streaming is “too complex” is a cognitive illusion, an echo of the Einstellung effect born from decades of batch-first conditioning. Like a programmer fluent only in imperative code or an English speaker trying to grasp Mandarin, the friction we feel is the strain of unlearning, not the inherent difficulty of the subject itself.
The path forward isn’t to abandon batch processing. It remains a powerful and appropriate tool for many problems. The goal is to cultivate a “data bilingual” mindset.
This means developing equal fluency in both the language of bounded, static sets and the language of unbounded, continuous streams. An engineer with this mindset can approach a problem and choose the right paradigm based on its nature, not on their own cognitive comfort zone.
The systems we build must reflect the world they model, a world that is not a static repository of facts, but a continuous flow of events. By recognizing the hammer we’re holding, we can finally see that not every problem is a nail.
