Someone came up to me at a conference and asked how they should implement the Saga pattern in Apache Flink and I think this is worth a written blogpost as well.

Their team came from a classic microservices architecture, where Sagas had become the standard answer whenever one business transaction crossed several services. Now they were moving parts of the system into Flink, so they did what experienced engineers naturally do: they brought the patterns that had kept the old architecture alive.

The instinct was reasonable. It was also where the trouble started.

Flink has state, timers, replay, checkpointing and transactional connectors. On paper, it looks almost purpose-built for Saga orchestration. You can key a stream by transaction ID, remember which steps have succeeded, wait for responses and emit compensating commands when something goes wrong.

Compared with maintaining the same logic across five microservices, this can even look beautifully simple.

The problem begins when Flink’s recovery model is confused with business rollback.

A Saga Is Not a Distributed Database Transaction

The Saga pattern was introduced for long-running transactions that cannot be executed as one atomic database transaction. Instead, the business operation is divided into multiple local transactions. Each participant commits its own changes, and when a later step fails, previously completed steps may need corresponding compensating actions.

Imagine an order process:

1. Create the order.
2. Reserve the inventory.
3. Capture the payment.
4. Arrange shipment.

If the payment fails, the inventory reservation may have to be released and the order marked as rejected.

That sounds like a rollback, but it is not one.

The inventory reservation was real. Other processes may have observed it. The payment provider may have created an authorization record. A cancellation might generate fees. An email cannot be pulled back from a customer’s inbox, and a parcel that has already left the warehouse does not respond particularly well to a database rollback command.

A compensation is a new business action that tries to reach an acceptable state. It does not erase history.

This distinction matters in every Saga implementation, but it becomes especially important in Flink because Flink is exceptionally good at restoring computational state. That can create a false sense that the whole business process has somehow become transactional.

It has not.

Flink Changes the Shape of the Problem

In a traditional microservice Saga, the process state is often scattered across service databases, message topics, retry queues and perhaps an orchestrator.

Flink offers a tempting alternative. Key the events by a Saga or business entity ID, and all relevant events for that key can be processed through the same logical state machine.

A KeyedProcessFunction, for example, can maintain:

• The current business state
• The steps that have already completed
• Correlation and idempotency identifiers
• Retry counters
• Deadlines and timers
• Information required for compensation

This is a strong model for event-driven coordination. Flink combines keyed state with event-time and processing-time timers, which makes it possible to react not only to incoming events but also to missing events and expired deadlines.

An order may enter WAITING_FOR_PAYMENT, register a timer for ten minutes and continue when either a PaymentConfirmed event arrives or the timer fires.

That is far cleaner than creating another database table and running a scheduler every minute to find abandoned processes.

However, clean state management does not mean that every business workflow belongs inside Flink.

Exactly Once Has a Jurisdiction

Flink’s exactly-once guarantee is frequently misunderstood.

When Flink recovers from a failure, it restores operator state from a checkpoint and replays source records from the corresponding positions. Exactly once therefore means that each event affects Flink-managed state exactly once.

End-to-end exactly-once processing requires more. The source must be replayable, and the sink must either participate transactionally in checkpointing or make repeated writes idempotent.

Exactly once has a jurisdiction, and it stops at the border of systems that cannot participate.

A Kafka sink can use transactions. A database sink may support upserts or transactional commits. An arbitrary REST endpoint usually does neither.

Consider this sequence:

1. Flink calls a payment API.
2. The payment provider successfully captures the money.
3. The Flink task fails before the result is durably included in a completed checkpoint.
4. Flink restores the previous checkpoint.
5. The payment request is executed again.

From Flink’s perspective, replaying the record is correct. From the customer’s perspective, paying twice is less impressive.

Flink’s asynchronous I/O operator stores records for in-flight requests in checkpoints and re-triggers those requests during recovery. This preserves Flink’s processing guarantees, but it also means the external operation must tolerate being called again.

The payment request therefore needs an idempotency key understood by the payment service. The same applies to inventory reservations, order creation, notifications and every compensating command.

Checkpointing does not replace idempotency. It makes idempotency more important.

Approach Zero: Check Whether You Need a Saga at All

Before deciding how to implement a Saga in Flink, I would first ask whether the problem is actually a Saga.

Not every multi-step event flow is a distributed transaction.

When all required changes can happen inside one database transaction, use the database transaction.

When a service must update its own database and reliably publish an event, a transactional outbox may solve the real problem without introducing a Saga. The database update and outbox entry are committed together, while change data capture later publishes the event.

When Flink is only updating derived views, aggregates or search indexes, correction events and idempotent upserts may be enough.

When all processing remains inside a transactional Kafka pipeline, Flink checkpointing and transactional Kafka sinks may already provide the required consistency boundary.

A Saga is useful when independent business actions have already committed and the system needs domain-specific continuation or compensation. It is not a general-purpose name for every pipeline with more than one step.

Approach 1: Model the Saga as a Flink State Machine

The most direct implementation places the Saga state inside keyed Flink state.

Every business process receives a stable identifier such as orderId, bookingId or claimId. Events are keyed by that identifier and processed by a stateful operator.

The state machine might look like this:

ORDER_CREATED
    -> INVENTORY_REQUESTED
    -> INVENTORY_RESERVED
    -> PAYMENT_REQUESTED
    -> PAYMENT_COMPLETED
    -> ORDER_CONFIRMED

Failure events lead to different transitions:

PAYMENT_REJECTED
    -> INVENTORY_RELEASE_REQUESTED
    -> ORDER_REJECTED

Timers handle missing responses. Commands and outcomes are written to separate topics, while completed or terminal Sagas are removed from active state.

Where this approach works well

It is particularly strong when the workflow is:

• Triggered and driven entirely by events
• High-volume and relatively uniform
• Bounded in duration
• Naturally partitioned by a business key
• Free from manual approval steps
• Based on participants that support idempotent commands
• Observable through events and state transitions

Flink provides scalable state, per-key ordering, timers, replay and fault recovery. Instead of treating Saga coordination as an external control plane, the coordination becomes part of the streaming topology.

For processes such as fraud evaluations, real-time order validation, automated reservation flows or machine-driven fulfilment decisions, this can be a very good fit.

Where it becomes uncomfortable

The Flink job now owns the process lifecycle.

Changing the workflow means changing and redeploying a stateful application. State schemas must remain compatible. Operator identifiers must remain stable across savepoints. Old Sagas may still be running according to rules that no longer match the new code.

You also need to build much of the operational experience yourself:

• Which step is a particular Saga currently waiting for?
• Why was a command retried?
• Which compensations have completed?
• Can an operator manually resume or override the process?
• Which workflow version created this state?
• How do you migrate thousands of active instances?

Flink can store all the required information. That does not mean it automatically provides a good case-management interface for it.

Approach 2: Use Event Choreography

In a choreographed Saga, no central component owns the complete process.

One participant publishes an event. Other participants react and publish their own outcomes. Flink may enrich, validate, correlate or transform these events, but each service still decides what to do next.

An OrderCreated event may trigger an inventory reservation. The resulting InventoryReserved event triggers payment processing. A PaymentRejected event triggers the release of the reservation.

This can work well for small and stable processes. Responsibilities remain distributed, and no single orchestrator needs to understand every participant.

The downside appears when the workflow grows.

The control flow becomes implicit. To understand one business process, you have to inspect the subscriptions and publishing behaviour of multiple services. Adding a new participant can create unexpected dependencies or event cycles. Integration tests begin to require half the platform.

Choreography usually looks wonderfully decoupled on an architecture diagram. Operationally, it can become a distributed treasure hunt.

This is not specifically a Flink problem. It is a characteristic of choreography itself. Official cloud architecture guidance similarly positions choreography as a good fit for workflows with only a few participants and warns that dependencies become harder to track as the number of participants grows.

Flink can make the event processing scalable, but it cannot make an implicit business process explicit.

Approach 3: Let Flink Orchestrate Through Events

A stronger Flink-centric design uses Flink as the central Saga coordinator but avoids directly invoking participant services wherever possible.

Instead, Flink emits commands to Kafka:

ReserveInventory
CapturePayment
ReleaseInventory
CancelOrder

The responsible services consume these commands, execute their local transactions and emit outcome events:

InventoryReserved
PaymentCaptured
InventoryReleaseFailed
OrderCancelled

Flink maintains the process state and decides which command comes next.

This architecture fits Flink much better than turning every operator into a REST client.

Commands can be written through a transactional Kafka sink, and the outcome stream is replayable. Participants can implement idempotency using the command ID or Saga ID. The event log also provides an audit trail of what was requested and what happened.

The important phrase here is “can implement idempotency.”

Even when Flink publishes a command exactly once into Kafka, the participant may crash after applying the local transaction but before acknowledging or publishing its result. The command may therefore be consumed again depending on the participant’s own processing model.

Every business action still needs a stable identity and a durable deduplication strategy.

The same is true for compensation. ReleaseInventory must be safe when the inventory was already released. RefundPayment must distinguish between no refund, a completed refund and a refund that is still being processed.

An event-driven orchestrator makes these situations manageable. It does not make them disappear.

Approach 4: Combine Flink With a Workflow Engine

The hybrid approach is often the most honest architecture.

Flink handles what Flink is good at:

• High-volume event ingestion
• Real-time enrichment
• Pattern detection
• Stateful correlation
• Continuous decisions
• Streaming analytics
• Triggering a business process when relevant conditions appear

A durable workflow engine handles what it is good at:

• Long-running business workflows
• Human approval tasks
• Durable retries over days or months
• Workflow versioning
• Explicit compensation paths
• Manual intervention
• Per-instance visibility
• Operational case management

For example, Flink may detect that a transaction requires an enhanced compliance review. Instead of keeping the complete review process inside a streaming job for three weeks, it starts a durable workflow and continues processing other events.

The workflow may request documents, wait for a human decision, contact an external provider and eventually emit a final result back into Kafka.

This introduces another platform component and another consistency boundary. That is a real cost.

However, hiding a workflow engine inside a KeyedProcessFunction does not remove that complexity. It only replaces a purpose-built workflow platform with custom code maintained by the streaming team.

Three Places to Put Saga Responsibility

Four approaches, but only three answers to the question that matters.

Approaches 1 and 3 place Saga responsibility in the same location: keyed state inside a Flink job. What separates them is transport. One calls participants directly, the other emits commands to Kafka and waits for outcome events. That choice has real consequences for idempotency, replay and auditability — which is why it deserved its own section — but it does not move the process state, and it does not move the pager. Flink still owns the lifecycle either way.

That leaves three genuine placements: the state lives in Flink, it lives implicitly across the participants, or it lives in a workflow engine built for it.

The difference is not technical elegance. It is where process state, retries, compensation and operational responsibility are expected to live — and, when a Saga is stuck at three in the morning, who is expected to know about it.

The chart below compares these three placements across those dimensions.

One thing does not vary across the three. Flink’s checkpoint recovery restores Flink’s state and replays its events, reliably and correctly, right up to the moment it calls something it does not own. Past that line there is no rollback, only compensation. Every placement above has to be designed for that line — it just moves who is responsible for noticing it.

The Limits You Need to Design For

Compensation can fail

A compensation is another distributed business action. It can time out, return an error or succeed without returning a response.

The Saga must therefore track the compensation itself, retry safely and eventually escalate cases that cannot be resolved automatically.

A system that can compensate only when everything works is not compensating. It is performing a demo.

Compensation may not restore the original state

Other processes may have modified the affected data after the original action.

Restoring an old value can overwrite legitimate newer changes. Compensation therefore requires business semantics, such as releasing the remaining reservation, issuing a partial refund or marking an order for manual review.

Compensations should usually be idempotent, auditable and designed with concurrent activity in mind. They may also need to run in an order different from the original forward steps.

Sagas do not provide isolation

Two Sagas may operate on the same customer, account, inventory item or credit limit at the same time.

One may observe temporary state created by the other. Updates can be lost, decisions may use stale values, and a compensation can conflict with a newer business operation.

Possible countermeasures include version checks, semantic locks, commutative updates, reservations and risk-based concurrency controls. The correct choice depends on the business risk, not merely on the technology being used.

State TTL is not a business timeout

Flink state can be configured with a time to live, but TTL cleanup happens on a best-effort basis. It is a storage-management feature, not a reliable trigger for business compensation.

Use explicit timers to detect business deadlines and emit the required state transition. Remove the state only after the process has reached a defined terminal condition and the necessary audit information has been persisted elsewhere.

Silently expiring an unfinished payment process is not failure handling. It is losing the paperwork.

Restoring Flink state does not restore the outside world

A savepoint can restore a consistent version of the Flink application state. It cannot revert writes that were already made to external systems.

The Flink documentation explicitly warns that restoring an application can cause external outputs to be emitted again because external writes are not rolled back with the savepoint.

This matters during failures, but also during deployments, migrations and incident recovery.

You must know exactly which side effects are transactional, which are idempotent and which require explicit reconciliation.

Every Saga needs a point of no return

A useful Saga design distinguishes between:

• Compensable steps, which can still be meaningfully reversed
• A pivot step, after which the process should move forward rather than backward
• Retryable steps, which must eventually complete after the pivot

For an order, payment capture or shipment handover might represent such a boundary.

Once the parcel has been collected, “undo shipment” may no longer be a realistic operation. The system may instead need to complete delivery and handle a return as a separate process.

This is a business decision disguised as an architecture decision, which is why developers should not design the compensation flow alone.

My Practical Decision Rule

I would implement the coordination inside Flink when the process is fundamentally a real-time event-driven state machine, runs at significant scale and interacts through replayable event streams with idempotent participants.

I would use choreography when the process is small, stable and genuinely understandable without a central model.

I would use a dedicated workflow engine when the process is long-running, frequently changed, operationally sensitive, includes human decisions or requires strong per-instance visibility.

I would use the outbox pattern when the actual problem is simply publishing an event reliably after a local database transaction.

Most importantly, I would never assume that a Saga must survive unchanged just because the previous microservice architecture had one.

The Question Is Not Whether Flink Can Do It

Apache Flink can implement a Saga.

With keyed state, timers, event processing and transactional connectors, Flink can implement almost any deterministic state machine. The more useful question is whether this business process should share the lifecycle, deployment model, failure semantics and operational tooling of a streaming job.

That is a much harder architectural question.

In this case, I did not tell them to throw away everything they had learned about Sagas. The pattern still described a real business problem: independent local transactions, partial failures and the need for compensation.

What needed to change was the assumption that the old implementation should simply be rebuilt inside Flink.

Flink should manage real-time state and decide what needs to happen next. Services should own their local transactions and make business commands idempotent. A workflow engine should own processes that require durable human and operational control.

The Saga pattern can cross all three boundaries.

The implementation should not.