The Future of AI Agents is Event-Driven
Note: Originally published by BigDataWire.
AI agents are set to transform enterprise operations with autonomous problem-solving, adaptive workflows, and scalability. But the real challenge isn’t building better models.
Agents need access to data, tools, and the ability to share information across systems, with their outputs available for use by multiple services — including other agents. This isn’t an AI problem; it’s an infrastructure and data interoperability problem. It requires more than stitching together chains of commands; it demands an event-driven architecture (EDA) powered by streams of data.
As HubSpot CTO Dharmesh Shah put it, “Agents are the new apps.” Meeting this potential requires investing in the right design patterns from the start. This article explores why EDA is the key to scaling agents and unlocking their full potential in modern enterprise systems.
To fully understand why EDA is essential for the next wave of AI, we must first look at how AI has evolved to this point.
The Evolution of AI
AI has progressed through two distinct waves and is now entering a third. The first two waves unlocked new possibilities but also suffered critical limitations.
The First Wave of AI: Predictive Models
The first wave of AI revolved around traditional machine learning, focusing on predictive capabilities for narrowly defined tasks.
Building these models required significant expertise, as they were crafted specifically for individual use cases. They were domain-specific, with their domain specificity embedded in the training data, making them rigid and tough to repurpose. Adapting a model to a new domain often meant starting from scratch — an approach that lacked scalability and slowed adoption.
The Second Wave of AI: Generative Models
Generative AI, driven by deep learning, marked a turning point.
Instead of being confined to single domains, these generative models were trained on vast, diverse datasets, giving them the ability to generalize across a variety of contexts. They could generate text, images, and even videos, opening up exciting new applications. However, this wave came with its own challenges.
Generative models are fixed in time — unable to incorporate new or dynamic information — and are difficult to adapt. Fine-tuning can address domain-specific needs, but it’s expensive and error-prone. Fine-tuning requires vast data, significant computational resources, and ML expertise, making it impractical for many situations. Additionally, since LLMs are trained on publicly available data, they don’t have access to domain-specific information, limiting their ability to accurately respond to questions that require context.
For example, suppose you ask a generative model to recommend an insurance policy tailored to a user’s personal health history, location, and financial goals.
In this scenario, you prompt the LLM and it generates a response. Clearly the model can’t deliver accurate recommendations because it lacks access to the relevant user data. Without it, the response will either be generic or flat-out wrong.
Compound AI Bridges the Gap
To overcome these limitations, Compound AI systems integrate generative models with other components like programmatic logic, data retrieval mechanisms, and validation layers. This modular design allows AI to combine tools, fetch relevant data, and tailor outputs in a way that static models cannot.
For instance, in the insurance recommendation example:
- A retrieval mechanism pulls the user’s health and financial data from a secure database.
- This data is added to the context provided to the LLM during prompt assembly.
- The LLM uses the assembled prompt to generate an accurate response.
This process, known as Retrieval-Augmented Generation (RAG), bridges the gap between static AI and real-world needs by dynamically incorporating relevant data into the model’s workflow.
While RAG effectively handles tasks like this, it relies on fixed workflows, meaning every interaction and execution path must be pre-defined. This rigidity makes it impractical to handle more complex or dynamic tasks where workflows cannot be exhaustively encoded. Encoding all possible execution paths manually is labor-intensive and ultimately limiting.
The limitations of fixed-flow architectures have led to the rise of the third wave of AI: agentic systems.
The Rise of Agentic AI
While AI has come a long way, we’re hitting the limits of fixed systems and even LLMs.
Google’s Gemini is reportedly failing to meet internal expectations despite being trained on a larger set of data. Similar results have been reported by OpenAI and their next-generation Orion model.
Salesforce CEO Marc Benioff recently said on The Wall Street Journal’s “Future of Everything” podcast that we’ve reached the upper limits of what LLMs can do. He believes the future lies with autonomous agents — systems that can think, adapt, and act independently — rather than models like GPT-4.
Agents bring something new to the table: dynamic, context-driven workflows. Unlike fixed paths, agentic systems figure out the next steps on the fly, adapting to the situation at hand. That makes them ideal for tackling the kinds of unpredictable, interconnected problems businesses face today.
Agents flip traditional control logic on its head.
Instead of rigid programs dictating every move, agents use LLMs to drive decisions. They can reason, use tools, and access memory — all dynamically. This flexibility allows for workflows that evolve in real time, making agents far more powerful than anything built on fixed logic.
How Design Patterns Shape Smarter Agents
AI agents derive their strength not only from their core abilities but also from the design patterns that structure their workflows and interactions. These patterns allow agents to tackle complex problems, adapt to changing environments, and collaborate effectively.
Let’s cover some of the common design patterns that enable effective agents.
Reflection: Improvement through Self-Evaluation
Reflection allows agents to evaluate their own decisions and improve their output before taking action or providing a final response. This capability enables agents to catch and correct mistakes, refine their reasoning, and ensure higher-quality outcomes.
Tool Use Expands Agent Capabilities
Interfacing with external tools extends an agent’s functionality, allowing it to perform tasks like retrieving data, automating processes, or executing deterministic workflows. This is particularly valuable for operations requiring strict accuracy, such as mathematical calculations or database queries, where precision is non-negotiable. Tool use bridges the gap between flexible decision-making and predictable, reliable execution.
Planning Turns Goals Into Actions
Agents with planning capabilities can break down high-level objectives into actionable steps, organizing tasks in a logical sequence. This design pattern is crucial for solving multi-step problems or managing workflows with dependencies.
Multi-Agent Collaboration: Modular Thinking
Multi-agent systems take a modular approach to problem-solving by assigning specific tasks to specialized agents. This approach offers flexibility: you can use smaller language models (SLMs) for task-specific agents to improve efficiency and simplify memory management. The modular design reduces complexity for individual agents by keeping their context focused on their specific tasks.
A related technique is Mixture-of-Experts (MoE), which employs specialized submodels, or “experts,” within a single framework. Like multi-agent collaboration, MoE dynamically routes tasks to the most relevant expert, optimizing computational resources and enhancing performance. Both approaches emphasize modularity and specialization — whether through multiple agents working independently or through task-specific routing in a unified model.
Just like in traditional system design, breaking problems into modular components makes them easier to maintain, scale, and adapt. Through collaboration, these specialized agents share information, divide responsibilities, and coordinate actions to tackle complex challenges more effectively.
In short, agents don’t just execute workflows; they reshape how we think about them. They’re the next step in building scalable, adaptable AI systems — moving past the constraints of traditional architectures and the current limitations of LLMs.
Agentic RAG: Adaptive and Context-Aware Retrieval
Agentic RAG evolves RAG by making it more dynamic and context-driven. Instead of relying on fixed workflows, agents can determine in real time what data they need, where to find it, and how to refine their queries based on the task at hand. This flexibility makes agentic RAG well-suited for handling complex, multi-step workflows that require responsiveness and adaptability.
For instance, an agent creating a marketing strategy might start by pulling customer data from a CRM, use APIs to gather market trends, and refine its approach as new information emerges. By retaining context through memory and iterating on its queries, the agent produces more accurate and relevant outputs. Agentic RAG brings together retrieval, reasoning, and action.
The Challenges with Scaling Intelligent Agents
Scaling agents — whether a single agent or a collaborative system — hinges on their ability to access and share data effortlessly. Agents need to gather information from multiple sources, including other agents, tools, and external systems, to make decisions and take action.
Connecting agents to the tools and data they need is fundamentally a distributed systems problem. This complexity mirrors the challenges faced in designing microservices, where components must communicate efficiently without creating bottlenecks or rigid dependencies.
Like microservices, agents must communicate efficiently and ensure their outputs are useful across the broader system. And like any service, their outputs shouldn’t just loop back into the AI application — they should flow into other critical systems like data warehouses, CRMs, CDPs, and customer success platforms.
Sure, you could connect agents and tools through RPC and APIs, but that’s a recipe for tightly coupled systems. Tight coupling makes it harder to scale, adapt, or support multiple consumers of the same data. Agents need flexibility. Their outputs must seamlessly feed into other agents, services, and platforms without locking everything into rigid dependencies.
What’s the solution?
Loose coupling through an event-driven architecture. It’s the backbone that allows agents to share information, act in real time, and integrate with the broader ecosystem — without the headaches of tight coupling.
Event-Driven Architectures: A Primer
In the early days, software systems were monoliths. Everything lived in a single, tightly integrated codebase. While simple to build, monoliths became a nightmare as they grew.
Scaling was a blunt instrument: you had to scale the entire application, even if only one part needed it. This inefficiency led to bloated systems and brittle architectures that couldn’t handle growth.
Microservices changed this.
By breaking applications into smaller, independently deployable components, teams could scale and update specific parts without touching the whole system. But this created a new challenge: how do all these smaller services communicate effectively?
If we connect services through direct RPC or API calls, we create a giant mess of interdependencies. If one service goes down, it impacts all nodes along the connected path.
EDA solved the problem.
Instead of tightly coupled, synchronous communication, EDA enables components to communicate asynchronously through events. Services don’t wait on each other — they react to what’s happening in real-time.
This approach made systems more resilient and adaptable, allowing them to handle the complexity of modern workflows. It wasn’t just a technical breakthrough; it was a survival strategy for systems under pressure.
The Rise and Fall of Early Social Giants
The rise and fall of early social networks like Friendster underscore the importance of scalable architecture. Friendster captured massive user bases early on, but their systems couldn’t handle the demand. Performance issues drove users away, and the platform ultimately failed.
On the flip side, Facebook thrived not just because of its features but because it invested in scalable infrastructure. It didn’t crumble under the weight of success — it rose to dominate.
Today, we risk seeing a similar story play out with AI agents.
Like early social networks, agents will experience rapid growth and adoption. Building agents isn’t enough. The real question is whether your architecture can handle the complexity of distributed data, tool integrations, and multi-agent collaboration. Without the right foundation, your agent stack could fall apart just like the early casualties of social media.
The Future is Event-Driven Agents
The future of AI isn’t just about building smarter agents — it’s about creating systems that can evolve and scale as the technology advances. With the AI stack and underlying models changing rapidly, rigid designs quickly become barriers to innovation. To keep pace, we need architectures that prioritize flexibility, adaptability, and seamless integration. EDA is the foundation for this future, enabling agents to thrive in dynamic environments while remaining resilient and scalable.
Agents as Microservices with Informational Dependencies
Agents are similar to microservices: they’re autonomous, decoupled, and capable of handling tasks independently. But agents go further.
While microservices typically process discrete operations, agents rely on shared, context-rich information to reason, make decisions, and collaborate. This creates unique demands for managing dependencies and ensuring real-time data flows.
For instance, an agent might pull customer data from a CRM, analyze live analytics, and use external tools — all while sharing updates with other agents. These interactions require a system where agents can work independently but still exchange critical information fluidly.
EDA solves this challenge by acting as a “central nervous system” for data. It allows agents to broadcast events asynchronously, ensuring that information flows dynamically without creating rigid dependencies. This decoupling lets agents operate autonomously while integrating seamlessly into broader workflows and systems.
Decoupling While Keeping Context Intact
Building flexible systems doesn’t mean sacrificing context. Traditional, tightly coupled designs often bind workflows to specific pipelines or technologies, forcing teams to navigate bottlenecks and dependencies. Changes in one part of the stack ripple through the system, slowing innovation and scaling efforts.
EDA eliminates these constraints. By decoupling workflows and enabling asynchronous communication, EDA allows different parts of the stack — agents, data sources, tools, and application layers — to function independently.
Take today’s AI stack, for example. MLOps teams manage pipelines like RAG, data scientists select models, and application developers build the interface and backend. A tightly coupled design forces all these teams into unnecessary interdependencies, slowing delivery and making it harder to adapt as new tools and techniques emerge.
In contrast, an event-driven system ensures that workflows stay loosely coupled, allowing each team to innovate independently.
Application layers don’t need to understand the AI’s internals — they simply consume results when needed. This decoupling also ensures AI insights don’t remain siloed. Outputs from agents can seamlessly integrate into CRMs, CDPs, analytics tools, and more, creating a unified, adaptable ecosystem.
Scaling Agents with Event-Driven Architecture
EDA is the backbone of this transition to agentic systems.
Its ability to decouple workflows while enabling real-time communication ensures that agents can operate efficiently at scale. As discussed here, platforms like Kafka exemplify the advantages of EDA in an agent-driven system:
- Horizontal Scalability: Kafka’s distributed design supports the addition of new agents or consumers without bottlenecks, ensuring the system grows effortlessly.
- Low Latency: Real-time event processing enables agents to respond instantly to changes, ensuring fast and reliable workflows.
- Loose Coupling: By communicating through Kafka topics rather than direct dependencies, agents remain independent and scalable.
- Event Persistence: Durable message storage guarantees that no data is lost in transit, which is critical for high-reliability workflows.
Data streaming enables the continuous flow of data throughout a business. A central nervous system acts as the unified backbone for real-time data flow, seamlessly connecting disparate systems, applications, and data sources to enable efficient agent communication and decision-making.
This architecture is a natural fit for frameworks like Anthropic’s Model Context Protocol (MCP).
MCP provides a universal standard for integrating AI systems with external tools, data sources, and applications, ensuring secure and seamless access to up-to-date information. By simplifying these connections, MCP reduces development effort while enabling context-aware decision-making.
EDA addresses many of the challenges MCP aims to solve. MCP requires seamless access to diverse data sources, real-time responsiveness, and scalability to support complex multi-agent workflows. By decoupling systems and enabling asynchronous communication, EDA simplifies integration and ensures agents can consume and produce events without rigid dependencies.
Event-Driven Agents Will Define the Future of AI
The AI landscape is evolving rapidly, and architectures must evolve with it.
And businesses are ready. A Forum Ventures survey found that 48% of senior IT leaders are prepared to integrate AI agents into operations, with 33% saying they’re very prepared. This shows a clear demand for systems that can scale and handle complexity.
EDA is the key to building agent systems that are flexible, resilient, and scalable. It decouples components, enables real-time workflows, and ensures agents can integrate seamlessly into broader ecosystems.
Those who adopt EDA won’t just survive — they’ll gain a competitive edge in this new wave of AI innovation. The rest? They risk being left behind, casualties of their own inability to scale.
