Skip to main content

Rollback Procedures

A Cadence rollback restores the previous server binary and leaves the persistence schema at its newer version. Cadence has no rollback controller of its own. Operators roll back with the tool that deployed the release, such as Helm, Docker Compose, or their own manifests. Workflow state lives in the database, so rolling back a server process does not move or rewrite open executions.

This works because schema changes are applied before the new binary and are usually backward compatible, and the project aims to keep adjacent releases compatible, including patch releases. At startup, each server process checks that the installed schema is at least the version the binary expects, and the optional stricter schema verification applies the same rule. A newer schema is accepted on purpose, so the previous binary starts normally against it. See Upgrade and rollback testing for how this path is exercised.

For how a rollback can fail, see Failure scenarios. For the signals that should trigger one, see Rollback metrics.

What gets rolled back​

ComponentRollback actionNotes
Server binaryRedeploy the previous image or binary with a rolling restartSupported from N+1 to N. Rolling back further, from N+1 to N-1, is not maintained.
Persistence and visibility schemaNoneThe schema tools have no down migrations. The previous binary runs on the newer schema.
Dynamic configurationRevert the valueReloaded without a restart, except for file-based config on the Helm chart. See Dynamic configuration. Revert this before the binary when the regression came from a config change.
Static YAML configurationRestore the previous file and restartRequired when the release renamed or added static keys.
Application workersRedeploy the previous worker versionDeployed independently of the server. Workflow determinism applies. See Worker rollback.
Web UI and CLIRedeploy the previous image or binaryNeither stores state of its own.

A release can include a one-time data migration or special rollback instructions. Cluster maintenance requires reading the release notes for every version on the upgrade path, and those notes override the general procedure on this page.

Rolling back the server​

  1. Stop the rollout. Pause the Deployment or deployment pipeline so no more processes move to the new version.
  2. Revert recent dynamic config changes first. If the rollout shipped a dynamic config change, reverting it is faster than rolling back the binary and may be enough on its own.
  3. Redeploy the previous version. Use the same image tag, static configuration, and numHistoryShards the cluster ran before the upgrade. Replace processes gradually so History shards can move to surviving hosts between batches. Old and new binaries can share the cluster during the rollout, because the schema is already at least as new as every running binary requires.
  4. Leave the schema in place. Do not run a schema downgrade or restore a backup as part of a normal rollback.
  5. Validate. Check that workers are polling, existing workflows continue, new workflows start, and the rollback metrics return to their pre-upgrade levels.

Once a schema is applied, the next attempt only needs the binary rollout. There is no need to re-run the schema step.

Helm deployments​

helm rollback <release> <revision> restores the chart, values, and image tag of an earlier release revision. The chart's schema Job is a plain Job, not a Helm hook, with ttlSecondsAfterFinished: 60. Once Kubernetes has deleted the finished Job, helm rollback recreates it with the previous image. Because the database is already newer, update-schema exits with an error, since no schema directory in the image is newer than the database's version. The setup script catches that error, logs Rollback is not allowed, and exits successfully. The schema is not downgraded.

Each server pod in the chart also runs a wait-for-schema init container before Cadence starts. It reads the schema version bundled in the pod's image and waits until the database reports that version. Unlike the server's own check, this one checks that the database version matches the image's version and does not accept a newer schema. If the rollback crosses a release that raised the schema version, new pods remain in Init. Such releases include patch releases: the Cassandra schema moved from 0.45 in v1.4.0 to 0.47 in v1.4.1. The Deployment stops progressing when it reaches its maxUnavailable limit, and the remaining pods on the newer image keep serving.

Before running helm rollback, compare the schema version in the database with the version.go of the target image for your driver, for example schema/cassandra/version.go. The chart has no value that disables the init container. If the versions differ, one workaround is to roll the image back with the wait-for-schema init container removed, for example with helm upgrade --post-renderer. Restore the unmodified chart on the next upgrade.

Dynamic configuration​

Dynamic configuration is not versioned with the server binary, so redeploying the previous binary does not revert it. The exception is file-based config on Helm: helm rollback also restores dynamicConfig.values, and the pods recreated by the image change mount the restored file. Otherwise, revert it separately:

  • File-based client: restore the previous YAML file. Processes reload the file on their polling interval. On Helm, restore dynamicConfig.values, run helm upgrade, then restart the server pods, for example with kubectl rollout restart. The chart mounts the file with subPath, so a ConfigMap change does not reach running pods.
  • configstore client: use cadence admin config update to set the previous value, or cadence admin config restore to remove an override. Without --filter, restore removes the unfiltered value. With --filter, it removes only the matching filtered value. See Config store client.

Shipping dynamic config changes separately from binary changes keeps each one independently reversible.

Multi-cluster deployments​

With cross-cluster replication, upgrade and roll back one cluster at a time. If the new version misbehaves on one cluster, fail over the affected domains to a cluster still on the previous version, then roll back the upgraded cluster. Keep cluster names, initialFailoverVersion, and failoverVersionIncrement unchanged in both the upgrade and the rollback. initialFailoverVersion identifies each cluster, and changing it requires a separate migration.

Worker rollback​

Worker code is part of the adopter's application, not the Cadence server, and is rolled back through the application's own deployment. A worker rollback is only safe if the previous code can replay histories written by the new code. Guard workflow changes with the SDK versioning APIs, Go or Java, and replay recorded histories against both versions before deploying.

If a rolled-back worker hits nondeterminism on executions that started on the new code, recover them with cadence workflow reset-batch. See Reset workflow.

When a binary rollback is not enough​

Some changes are not undone by restoring the previous binary. They include:

  • A changed numHistoryShards
  • A domain archival URI, which can be set only once
  • Data already written by a one-time migration
  • A release note that marks a migration as incompatible

See Failure scenarios for their impact.

If the schema change itself is the problem, the recovery order is: fail over to a cluster still on the previous schema, reset affected workflows, and restore from backup as a last resort. Upgrade and rollback testing covers each step.