Replay Guide

A guide to managing and performing replays for schema evolution, software evolution, and disaster recovery.

Replays within Concordance involve the manipulation of NATS JetStream consumers. These consumers can be thought of as something like server-stored high-watermark pointers. Replay involves manipulating those pointers.

Initiating Replays

Before you initiate a replay, make sure that you have purged whatever state you might need to purge. Aggregate internal state is stored in the CC_STATE key value bucket, and your projections will be wherever you have chosen to store them.

In the current version of the NATS server (2.9.15 at the time of this writing) and the corresponding NATS CLI, you cannot manipulate the "last seen" pointer of a durable consumer. This means you can't do something like reset AGG_CMD_bankaccount on the bank account aggregate's consumer. Instead, we need to delete and re-create the consumer.

Before we delete the consumer, we need to take a backup of its configuration (and ignore its state):

nats consumer info CC_EVENTS PROJ_bankaccount_projector --json | jq .config > config.json

Here we're taking a copy of the bank account projector consumer from the event stream and storing just the config JSON field/object in the config.json file. Now we can delete the consumer:

nats consumer rm CC_EVENTS PROJ_bankaccount_projector

And now that the projector's consumer has been deleted, we can create a new one:

nats consumer add --config config.json CC_EVENTS

This will add back the previous consumer, but the "last seen" marker will be set to the beginning of time, and thus replay all of the messages destined for that consumer. Note that all of the running consumers managed by Concordance are operating on a pull loop that is likely using an exponential backoff timer. This means you may have to wait a few seconds before you start seeing the messages run through the machinery again.

For information on when you might want to use replays for things like schema evolution and deploying new components, take a look at the Developer's Guide.

Further topics like deciding when you should pause incoming events during a replay and when or if things like blue/green deployment might be of assistance to you are beyond the scope of Concordance's documentation. Check the resources section for some good books on event sourcing that might cover that topic, including one from the original creator of Concordance.