Event-Driven Architecture: An Honest Assessment

Event-driven systems are elegant in talks and brutal in production. After building and operating them across multiple companies, here is what nobody tells you before you commit to the pattern.

Every few years the industry rediscovers event-driven architecture and decides it is the answer. The talks are compelling. Services decoupled from each other. No direct dependencies. Producers that emit events and never think about who consumes them. Consumers that react to what happened and never worry about who caused it. The system as a whole becomes a collection of independent actors responding to a shared stream of facts about the world.

In the talk, this is clean. In production, it is one of the most operationally demanding patterns in software engineering, and the gap between how it is pitched and what it costs to run it well is wider than almost any other architectural pattern I can think of.

I have built event-driven systems that worked well and event-driven systems that were disasters. The difference was not the technology and it was not the team’s capability. It was whether the team went in with an accurate picture of what they were buying. Most teams do not get that picture before they commit. This article is an attempt to provide it.

What you actually get

Start with what is genuinely good, because there is genuine good here.

Decoupling that is real. When the order service publishes an OrderPlaced event and knows nothing about who consumes it, and when the inventory service consumes OrderPlaced and knows nothing about who published it, you have achieved something meaningful. Either service can be redeployed without the other. Either can evolve its internal implementation without negotiating with the other. A new service can start consuming OrderPlaced tomorrow without touching the order service at all.

This decoupling is the thing that makes large organisations with many teams possible. The team that owns the order service does not need to be in a meeting with every team that cares about orders. They publish the event. Every consumer team builds and maintains their own reaction to it. The coordination that would have been synchronous and blocking becomes asynchronous and independent.

Audit trails that emerge naturally. If your events are your source of truth, you have a complete record of everything that happened in your system in the order it happened. Not just the current state, but the history of how you got there. This is genuinely useful for debugging, for compliance, and for the class of bugs that are almost impossible to diagnose without knowing what sequence of events preceded them.

Load handling that is structural rather than bolted on. A consumer that reads from a queue processes work at the rate it can handle, regardless of the rate at which work arrives. The queue absorbs the spike. The consumer processes the backlog when capacity is available. This is structurally different from a synchronous system where a traffic spike hits the service directly and the service either handles it or falls over.

These are real benefits. They are worth having. They are also not free.

The cost nobody quotes you upfront

Eventual consistency is not a configuration option, it is a commitment.

When you move from synchronous calls to events, you give up the ability to know, at any given moment, that all parts of your system agree on the current state. The order was placed. The OrderPlaced event was published. The inventory service will consume it and reserve the stock. When? Soon. How soon? Depends on the consumer’s lag. What if the user queries their order status right now, before the inventory service has processed the event? The order exists in the order service’s view. The inventory has not yet been reserved. The system is in an intermediate state that is internally consistent but not yet globally consistent.

For many use cases this is acceptable. For some it is not. The teams that adopt event-driven architecture without thinking carefully about which of their use cases fall into which category discover the hard way that “eventually” can mean milliseconds, seconds, or minutes depending on what else is happening, and that users do not have the same patience for eventual consistency that architects do.

# The problem that bites teams who haven't thought this through:

# User places an order. Order service publishes event.
async def place_order(user_id: str, items: list) -> Order:
    order = Order.create(user_id=user_id, items=items)
    await db.save(order)
    await event_bus.publish(OrderPlaced(order_id=order.id, items=items))
    return order

# User immediately queries order status.
# Order service returns order with status "pending".
# Inventory service has not yet processed the event.
# Frontend shows "pending" with a spinner.
# User refreshes. Still pending. Refreshes again.
# Inventory event processes. Status updates to "confirmed".
# User has refreshed four times and is on the phone with support.

# The system was correct the entire time.
# The user experience was broken the entire time.
# These are not contradictory.

# What teams often do to address this:
# Read-your-own-writes consistency for the immediate response,
# combined with a clear UI state that communicates processing is happening.

async def place_order_with_ux_in_mind(user_id: str, items: list) -> dict:
    order = Order.create(user_id=user_id, items=items)
    await db.save(order)
    await event_bus.publish(OrderPlaced(order_id=order.id, items=items))

    return {
        "order_id": order.id,
        "status": "processing",
        "message": "Your order is being confirmed. This usually takes a few seconds.",
        "poll_url": f"/orders/{order.id}/status",
        # Give the client a signal about what to do next,
        # rather than leaving them in ambiguous pending state
    }

Debugging across services is a different discipline entirely.

In a synchronous system, a bug has a call stack. You look at the stack trace and you see exactly what called what in what order. The sequence is right there.

In an event-driven system, the equivalent of a call stack is a trace across multiple services, potentially across multiple events, potentially hours or days apart. OrderPlaced fires. InventoryReserved fires. PaymentProcessed fires. FulfillmentCreated fires. ShipmentCreated fires. The user reports that their order is stuck. You need to find where in this sequence something went wrong, knowing that each step is in a different service with different logs, possibly with events that were consumed out of order, possibly with a consumer that failed silently and moved on.

Without distributed tracing that propagates correlation IDs across every event, this debugging is archaeology. You are sifting through log files from multiple services trying to reconstruct what happened to a specific order that a specific user placed at a specific time.

# Event envelope that makes debugging survivable.
# Every event carries a correlation ID from the originating request.
# Every subsequent event in the chain inherits it.
# Every service logs it with every operation related to the event.
# When debugging, filter all service logs by correlation ID
# and you reconstruct the full sequence.

from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Any
import uuid


@dataclass
class Event:
    event_id: str = field(default_factory=lambda: str(uuid.uuid4()))
    event_type: str = ""
    correlation_id: str = field(default_factory=lambda: str(uuid.uuid4()))
    causation_id: str = ""    # ID of the event that caused this one
    occurred_at: str = field(
        default_factory=lambda: datetime.now(timezone.utc).isoformat()
    )
    schema_version: str = "1.0"
    payload: dict = field(default_factory=dict)

    def caused_by(self, parent_event: "Event") -> "Event":
        self.correlation_id = parent_event.correlation_id
        self.causation_id = parent_event.event_id
        return self


# When the inventory service handles OrderPlaced and emits InventoryReserved:
async def handle_order_placed(event: Event) -> None:
    order_id = event.payload["order_id"]

    log.info(
        "inventory.reservation.started",
        correlation_id=event.correlation_id,
        order_id=order_id,
    )

    await reserve_inventory(order_id)

    reserved_event = Event(
        event_type="InventoryReserved",
        payload={"order_id": order_id},
    ).caused_by(event)   # Inherits correlation_id, sets causation_id

    await event_bus.publish(reserved_event)

    log.info(
        "inventory.reservation.completed",
        correlation_id=event.correlation_id,
        order_id=order_id,
        caused_event_id=reserved_event.event_id,
    )

With this envelope, every event in a chain shares a correlation ID. Every log line from every service that handled any event in the chain includes that correlation ID. Debugging a stuck order is a single log query: show me every log line with this correlation ID, across all services, sorted by time.

Without this, you do not have an event-driven system you can operate. You have a system that works until it breaks and then you cannot find out why.

Consumer failures are invisible by default.

In a synchronous system, a failure is loud. The call throws an exception. The caller gets an error. Someone notices.

In an event-driven system, a consumer failure can be completely silent. The consumer reads the event, fails to process it, and depending on how it is configured, either requeues the event, moves it to a dead letter queue, or discards it and moves on. The producer never knows. The other consumers never know. The user whose order triggered the event never knows until they notice that something downstream has not happened.

Dead letter queues are the standard answer to this and they are the right answer, but they are only useful if someone is watching them. A dead letter queue that nobody monitors is not a safety net. It is a place where failed events go to be forgotten.

# Dead letter queue monitoring that actually alerts

import boto3
from prometheus_client import Gauge

sqs = boto3.client("sqs")

DLQ_DEPTH = Gauge(
    "sqs_dlq_message_count",
    "Number of messages in dead letter queue",
    ["queue_name"]
)


async def check_dlq_depths():
    queues = [
        ("order-processing-dlq", "https://sqs.region.amazonaws.com/account/order-processing-dlq"),
        ("inventory-dlq", "https://sqs.region.amazonaws.com/account/inventory-dlq"),
        ("payment-dlq", "https://sqs.region.amazonaws.com/account/payment-dlq"),
        ("fulfillment-dlq", "https://sqs.region.amazonaws.com/account/fulfillment-dlq"),
    ]

    for queue_name, queue_url in queues:
        response = sqs.get_queue_attributes(
            QueueUrl=queue_url,
            AttributeNames=["ApproximateNumberOfMessages"]
        )
        depth = int(response["Attributes"]["ApproximateNumberOfMessages"])
        DLQ_DEPTH.labels(queue_name=queue_name).set(depth)


# Alert rule: any DLQ with more than 0 messages is an incident.
# Not a warning. An incident.
# A message in the DLQ means an event failed to process.
# That means something the user expected to happen did not happen.
# That is always worth investigating immediately.

Any message in a dead letter queue is a symptom of a real problem. Not a might-be-a-problem. A real problem. Treating DLQ depth as a metric that alerts at zero normalises the expectation that failures are real and visible, rather than the expectation that failures are background noise to be managed.

The schema problem that grows until it bites you

Events are a public interface. Once a consumer is reading your events, the schema of those events is a contract. Changing the schema breaks the consumer.

In a synchronous API, schema evolution is managed through versioning. The producer runs V1 and V2 of the endpoint simultaneously. Consumers migrate at their own pace. When all consumers have migrated, V1 is deprecated.

In an event-driven system, the equivalent is possible but operationally harder. If you change the OrderPlaced event schema, you need every consumer of OrderPlaced to be updated before you change the schema, or the consumer needs to handle both old and new schemas simultaneously, or you need to maintain two event types in parallel during migration. None of these options is cheap, and they are cheaper if you planned for them than if you did not.

The teams that handle this well establish schema governance before they have a schema problem. Not after.

# Schema versioning that makes evolution manageable.
# Every event schema has an explicit version.
# Consumers declare which versions they can handle.
# The event bus routes accordingly.

from pydantic import BaseModel
from typing import Literal
from datetime import datetime


# Version 1 of OrderPlaced
class OrderPlacedV1(BaseModel):
    schema_version: Literal["1.0"] = "1.0"
    order_id: str
    user_id: str
    total: float
    occurred_at: datetime


# Version 2 adds line items and changes total to be in cents
# to avoid floating point issues that V1 had
class OrderPlacedV2(BaseModel):
    schema_version: Literal["2.0"] = "2.0"
    order_id: str
    user_id: str
    total_cents: int          # Changed: was "total: float"
    currency: str             # Added
    items: list[dict]         # Added
    occurred_at: datetime


# Consumer that handles both versions during migration period
class InventoryConsumer:
    async def handle(self, raw_event: dict) -> None:
        version = raw_event.get("schema_version", "1.0")

        if version == "1.0":
            event = OrderPlacedV1(**raw_event)
            order_id = event.order_id
            # V1 doesn't have items, so we have to fetch them
            items = await self.order_service.get_items(order_id)
        elif version == "2.0":
            event = OrderPlacedV2(**raw_event)
            order_id = event.order_id
            items = event.items
        else:
            log.error(
                "inventory.consumer.unknown_schema_version",
                version=version,
                event_id=raw_event.get("event_id"),
            )
            raise UnknownSchemaVersionError(version)

        await self.reserve_inventory(order_id=order_id, items=items)

Schema versioning adds boilerplate. The alternative is finding out during an incident that a schema change broke a consumer that nobody knew was depending on the old format.

When event-driven architecture is the wrong answer

The pattern is not universally appropriate. Teams adopt it when they should not, attracted by the elegance and the conference talks, and then spend years paying costs that were not necessary.

When you have one team and one service. Events are an organisational boundary mechanism. If there is no organisational boundary, you are paying the operational cost of distributed messaging for no architectural benefit. A function call is faster, simpler, and easier to debug. A modular monolith with internal domain events gives you the architectural thinking without the operational overhead.

When your operations require immediate consistency. Financial transactions. Inventory deduction that must be accurate at the moment of purchase. Medical record updates. Any situation where the user or the business cannot tolerate the state being temporarily inconsistent. Eventual consistency is not a technical property to be engineered around in these cases. It is a fundamental unsuitability.

When your team does not have the operational maturity for it. Event-driven systems require distributed tracing. They require DLQ monitoring. They require schema governance. They require expertise in at least one message broker technology. They require runbooks for consumer failure scenarios. Teams that are still establishing basic engineering practices should not add this operational surface area. Stabilise first. Adopt the pattern when you have the capacity to operate it correctly.

When the communication pattern is inherently synchronous. A user submits a form and expects a result. An API client makes a request and needs a response before it can proceed. A batch job reads data and produces a report. These patterns do not become better by adding events between the steps. They become more complex with no benefit. Forcing an asynchronous pattern onto an inherently synchronous workflow is an architecture astronaut move, not an engineering decision.

The systems that work

The event-driven systems that work well in production share a set of properties that are not negotiable.

Every event carries enough context to be processed without fetching additional data. A consumer that needs to make a synchronous call to process an event has a dependency on the producer that the event pattern was supposed to eliminate.

Every consumer is idempotent. Events can be delivered more than once. A consumer that is not idempotent will produce duplicate effects when this happens. Designing for idempotency upfront is straightforward. Retrofitting it after duplicate processing has caused data integrity issues is expensive.

# Idempotent consumer using a processed events log
async def handle_order_placed(event: Event) -> None:
    # Check if we have already processed this event
    already_processed = await processed_events.exists(event.event_id)
    if already_processed:
        log.info(
            "inventory.consumer.duplicate_event_skipped",
            event_id=event.event_id,
            correlation_id=event.correlation_id,
        )
        return

    # Process the event
    await reserve_inventory(event.payload["order_id"])

    # Record that we have processed it
    await processed_events.record(
        event_id=event.event_id,
        processed_at=datetime.now(timezone.utc),
        consumer="inventory-service",
    )

Every schema change goes through a review process before it is deployed. The review asks: which consumers will be affected? Have they been updated? Can the change be made backwards-compatible? If not, what is the migration plan?

Every dead letter queue has an alert and an owner. DLQ messages are investigated the same day they appear. Not triaged. Not backlogged. Investigated.

The teams that run event-driven systems well have built this infrastructure. It is not glamorous work. It does not appear in the conference talk about the elegant decoupling. It is the thing that makes the decoupling survivable in practice.

The honest version of the pitch

Event-driven architecture is worth adopting when you have multiple teams that need genuine autonomy, when your use cases can tolerate eventual consistency, when you have the operational maturity to run distributed messaging in production, and when the benefits of decoupling outweigh the costs of asynchrony and distribution.

In those circumstances it is genuinely powerful. The teams that use it well will tell you they cannot imagine going back. The operational complexity feels like a fair trade for the organisational flexibility.

In other circumstances it is an expensive mismatch between pattern and problem. The complexity of the pattern does not disappear because you chose it for the wrong reasons. It stays. You pay it.

The evaluation should be honest about both sides. Not “events are the modern way to build systems” which is a fashion statement. Not “events are always too complex” which ignores where they genuinely excel. The honest version: this pattern solves specific organisational and scalability problems at a specific operational cost. Do the problems apply to us? Can we afford the cost? Those are the questions worth asking before you commit.

The answer is sometimes yes. It is not always yes. And pretending otherwise is how teams end up with event-driven systems they cannot operate and cannot easily escape.