Skip to main content

Cadence Service Releases

Cadence follows semantic versioning and you can find more information about changes between version on the Cadence changelog.

Latest release for General Availability

v1.4.5 published on 2026-10-30

Major Releases​

v1.4.x​

v1.3.x​

v1.2.x​

v1.0.x​

v0.25.x​

v0.24.x​

v0.23.x​

v0.22.x​

v0.21.x​

v0.20.x​

v0.19.x​

v0.18.x​

v0.16.x​

v0.17.x​

v0.15.x​

v0.14.x​

v0.13.x​

v0.12.x​

v0.11.x​

v0.10.x​

v0.9.x​

v0.8.x​

v0.7.x​

v0.6.x​

v0.5.x​

v0.4.x​

v0.3.x​

v0.2.x​

v0.1.x​

Recent release notes​

Release v1.4.5 published by neil-xie on 2026-10-30​

Upgrade Notes

  • Cassandra schema: apply migrations through v0.55 before rollout (current is v0.52 in v1.4.2). v0.53 adds the semaphore_info type and executions.semaphore_map, v0.54 adds jitter_coefficient to workflow_execution and activity_info, and v0.55 adds semaphore_name to transfer_task. MySQL, PostgreSQL and SQLite schemas are unchanged. Schema updates that fail partway can now be re-run: duplicate DDL statements are logged and skipped (#8598).
  • If you're on a version older than v1.4.2, deploy v1.4.2 first, since it carries the read filter the tombstone reduction depends on. See Cassandra Tombstone Reduction below.
  • Default changes. These are on after upgrade unless you set them explicitly:
    • history.enableTimerQueueV2 and history.enableTransferQueueV2 now default to true (#8617).
    • history.timerProcessorCachedQueueReaderMode and history.transferProcessorCachedQueueReaderMode now default to enabled (#8565, #8592). In v1.4.2 both defaulted to disabled. Set disabled or shadow to go back.
  • FailoverDomain is now only accepted at the destination cluster (#8542). A request is rejected unless every target cluster it names is the cluster that received it. Scripts or tooling that fail a domain over from another cluster must pass --skip_destination_check (CLI, #8540) or set SkipDestinationClusterCheck on the request (#8539). The V2 failover workflows forward the flag (#8541). UpdateDomain-driven failovers are unchanged.
  • The server now refuses to start if the configured dynamic config client cannot be initialised (#8670, #8619). A syntax error, a missing file, an unreachable config store or a typo in dynamicconfig.client used to fall back to the no-op client and silently run on defaults. The no-op client is now used only when dynamic config is not configured, or when dynamicconfig.client: nop is set. Validate your dynamic config before upgrading.
  • Metrics: the workflow_execution_history_access gauge was replaced by the workflow_query_age_days histogram (#8559, #8566). Queue v2 cache metrics lost their shard_id tag and cached_queue_prefetch_window_span_ns was removed (#8551).
  • Go module language level is now 1.26.0 with toolchain Go 1.26.9 (#8675). Build environments must provide Go 1.26 or allow toolchain download.
  • v1.4.3 and v1.4.4 were tagged by mistake and have been removed. If you already have one of those tags, it is this same commit. v1.4.5 is the release to install.

Major Features

1. Cassandra Tombstone Reduction (#8318)

Cassandra creates a tombstone for every key-value deletion on a map column. Cadence tracks pending activities and pending timers in map columns, so every activity completion and every timer firing created one. Together with the read filter and unset-value work in v1.4.2, this release adds the write path for #8318: batch map deletion writes (#8522).

When enabled, deleting a pending activity or timer entry writes a sentinel value (activitySentinelScheduleID, timerSentinelTimerID) instead of issuing a delete. To keep the collections from filling with sentinels, a transaction that contains deletes will, with probability 1 in N, rewrite the whole pending activity or timer collection and drop the accumulated sentinels. Readers filter sentinels on load, so they are invisible to workflow logic.

This release also exposes Cassandra server warnings, such as tombstone overload on a read, from the persistence layer (#8678). It is groundwork for cleanup workflows that read, check, and skip the delete when a read hit too many tombstones.

Configuration

  • history.rewriteSampleRate (default 0, disabled): N means a roughly 1-in-N chance per transaction with deletes. 0 keeps the previous delete behaviour. 1 rewrites every time. The setting is dynamic and acts as the kill switch. A larger N means fewer rewrites, so more sentinels accumulate. Larger collections cost read latency on shard movement or cache eviction, when the whole collection is read from the database.
  • history.rewriteOptimizationBackends (default ["cassandra"]): backends the optimization applies to. Only Cassandra needs it. The value is read once at startup, so changing it requires a restart.

Migration Path

  • The feature is off by default, and upgrading changes no write behaviour.
  • The read filter from v1.4.2 must be running on every History host before you set history.rewriteSampleRate above 0. Sentinels written without the filter would show up as phantom pending activities and timers and break workflow executions.
  • After enabling, do not roll back to a build older than v1.4.2. Setting history.rewriteSampleRate back to 0 returns to real deletes, but sentinels already stored stay in place until the workflow completes (no rewrites run at 0), and they still need the read filter.

2. Retry Controls: FailureOptions and Backoff Jitter (#8238)

  • FailureOptions can override activity retry behaviour (#8554). A failure can now carry a next retry interval, which replaces the calculated backoff and may exceed the policy's maximumInterval. A value beyond the expiration interval fails the activity instead of retrying it. A FailureCategory of Fatal stops further retries. Workers opt in per failure, for example by returning a Go client CustomError built with cadence.WithFailureCategory and cadence.WithNextRetryDelay in client versions that include them.
  • Proportional jitter on retry backoff (#8583, #8604, #8605). RetryPolicy has a new jitterCoefficient. Each activity or workflow retry interval is scaled by a random factor in (1 - jitterCoefficient, 1], applied after the maximumInterval cap, so the maximum remains an upper bound. The value must be between 0 and 1. The default 0 means no jitter. Values up to 0.2 are recommended, since 1 allows a near-immediate retry. The coefficient is stored in workflow and activity mutable state and needs Cassandra schema v0.54. Rolling back is safe: older servers ignore the field and retry without jitter.

3. Queue v2 and Cached Queue Reader Enabled by Default (#7953, #8432)

History queue v2 is now the default for timer and transfer queues (#8617), and the cached queue reader introduced in v1.4.2 now defaults to enabled for both (#8565, #8592). The reader serves GetHistoryTasks from an in-memory per-shard cache instead of the database. In v1.4.2 the expected effect for timer tasks was about a 90% reduction in GetHistoryTasks requests.

Migration Path

  • The defaults changed, not the code paths. To keep the previous behaviour set history.enableTimerQueueV2 and history.enableTransferQueueV2 to false, and the two CachedQueueReaderMode settings to disabled.
  • shadow still runs the cache alongside database reads without serving from it, which is a lower-risk way to compare before committing.

Notable Bug Fixes & Enhancements

  • New cluster-redirection policies selected-apis-forwarding-v3 and all-domain-apis-forwarding-v3 forward StartWorkflowExecutionAsync and SignalWithStartWorkflowExecutionAsync to the active cluster before they are enqueued. The existing policy names and their behaviour are unchanged, so adopting v3 is opt-in (#8591, #8590).
  • Continue-as-new now inherits the current run's ActiveClusterSelectionPolicy. Previously, in an active-active domain whose default cluster differed from the run's cluster, the new run was built with the wrong failover version and the transaction failed with DomainNotActiveError (#8555).
  • A child start that finds a run with the same request ID now checks which cluster closed that run. If another cluster closed it, the start fails with WorkflowExecutionAlreadyStartedError instead of adopting the run (#8644).
  • Reset-and-reapply on a closed workflow now first checks for signals that have not yet been applied, and only resets when there are some (#8596).
  • Decision heartbeating no longer leaves a timer-fired event behind when the timer was cancelled in the same decision and the buffered event removal left no buffered events to flush (#8595).
  • task_list_kind is now kept when the full pending-activity map is rewritten in Cassandra, so activities no longer lose their sticky task list kind (#8626).
  • The global ratelimiter never limits a host below its local fallback limit (target / num-hosts). Before, some domains saw rejections at 30-60% of their configured RPS during bursts concentrated on a few frontend hosts (#8538).
  • RPC peer choosers for History and Matching are now refreshed every 30 seconds, not only on membership change events, so connections to hosts that died without a membership event are dropped (#8594).
  • The shard distributor spectator now stops with the service instead of leaking and flooding logs with "Failed to create stream" messages on shutdown (#8603).
  • Suppressed timer metrics are no longer registered. With histograms.default: histogram, timer series used to still appear in Prometheus with zero-valued _bucket, _count and _sum (#8667).
  • The development TLS certificate (config/credentials/keytest.crt) now has subject alternative names, so standard TLS verification works against ./cadence-server --zone tls (#8657).

Observability

  • New open-source Grafana dashboards, intended to replace the existing ones over time. The local docker-compose setups now use histogram timer metrics (#8558, #8578). Instructions for importing them into Cloud Monitoring (#8599).
  • workflow_query_age_days histogram, tagged by domain and workflow_close_status, records how old closed workflows are when queried through GetWorkflowExecutionHistory and DescribeWorkflowExecution. It helps tune domain retention (#8559, #8566).

CLI and Setup

  • cadence domain failover --skip_destination_check (#8540).
  • cadence admin config update accepts a JSON file as input, --value JSON parses correctly, and cadence admin config list takes an optional --name (#8627, #8608).
  • cadence-server has an auto-setup option that applies the latest schema and creates a default domain on start. The auto-setup docker image uses it instead of shell scripts, and a SQLite auto-setup image is available (#8437, #8569).

Full Changelog: https://github.com/cadence-workflow/cadence/compare/v1.4.2...v1.4.5


Release v1.4.3-prerelease06 published by fimanishi on 2026-10-16​

Schema Upgrades

  • Cassandra: from 0.54 to 0.55

What's Changed

New Contributors

Full Changelog: https://github.com/cadence-workflow/cadence/compare/v1.4.3-prerelease05...v1.4.3-prerelease06


Release v1.4.2 published by neil-xie on 2026-10-02​

Upgrade Notes

  • Cassandra schema: apply migrations through v0.52 before rollout. This release adds FailureOptions columns on activity_info (v0.48), history task DLQ tables (v0.49), and the new semaphore_metadata / semaphore_tokens / semaphore_tasks tables (v0.50–v0.52).
  • PostgreSQL schema: apply cadence v0.8 and visibility v0.9.
  • Prometheus metrics: several rollup series were renamed (they now end in _rollup) so they no longer share a name with a different label set. See Prometheus Metric Label Consistency below.
  • Go module language level is now 1.25.0 and the repository pins toolchain Go 1.25.14. Build environments must provide Go 1.25 or allow toolchain download.

Major Features

1. Cached Queue Reader (#7953, #8432)

Task processing is one of the most performance-critical parts of Cadence. Timer task reads are a large share of Cassandra traffic, and many of those reads repeat data the processor already received in a notification.

This release introduces CachedQueueReader, an in-memory per-shard cache that serves GetHistoryTasks from that notification instead of the database. For timer tasks, the expected effect is about a 90% reduction in GetHistoryTasks requests, about a 13% reduction in total Cassandra persistence requests, and about a 15% reduction in Cassandra CPU utilization.

Timer and transfer queues can keep recently created tasks in that cache. The caches are configured with history.timerProcessorCachedQueueReaderMode and history.transferProcessorCachedQueueReaderMode.

Migration Path

  • Both caches default to disabled in this release. Set the mode to shadow or enabled to opt in. No change is required to keep the previous read path.

2. Unified Schema Management

Cadence is introducing a unified, persistence-native framework for database setup, schema updates, and schema verification. Previously, operators used backend-specific schema commands and deployment scripts; this work moves the common lifecycle into the server’s persistence layer so a single configuration-driven workflow can manage every configured store. The new path understands separate default and visibility databases, multiple configured connections, and advanced-visibility stores, while existing schema CLI tools remain supported.

cadence-server update-schema

A new command creates missing databases/keyspaces and applies every pending versioned migration up to the latest schema bundled in that binary (#8414, #8496):

Before updating your server deployments, grab a binary from the new release and run the following command:

cadence-server update-schema

It’s recommended to run this via the published docker container, using all the same environment variables or configuration as your standard deployment.

This is safe to run while the previous version is running. Apply the schema update first: an older server can keep running on the new schema, and rolling back the binary does not roll the schema back.

It connects to each configured store, creates it when missing, initializes schema-version bookkeeping, sorts pending migrations by store and version, and applies them in order. It supports Cassandra, MySQL, PostgreSQL, SQLite, Elasticsearch, and OpenSearch. Pinot schema management is not included in this release.

Elasticsearch and OpenSearch do not use Cadence's version table. Their setup checks that the visibility index exists and adds mappings from the latest bundled template when needed. The configured index name must begin with cadence-visibility so it matches the template filter.

Startup Verification

Schema verification now delegates to each persistence backend and checks every configured default, visibility, and advanced-visibility store (#8445). It verifies that the store exists, schema versioning is initialized where applicable, and the installed version is at least the latest version bundled with the server.

The new verification path runs during server startup. By default, failures are logged as warnings and the existing Cassandra/SQL compatibility checks still determine whether startup may proceed. Setting dynamic config system.enforceSchemaVerificationV2=true makes any new verification failure block startup. This enforcement is disabled by default for a gradual rollout.

Migration Path

  • update-schema is opt-in in this release; existing schema CLI tools are unchanged and remain supported.
  • Apply migrations before deploying server processes. An older server can keep running on the new schema, but rolling back the binary does not roll the schema back.

3. Prometheus Metric Label Consistency (#7844)

Prometheus drops a metric if the same name is registered with different label keys. Persistence and Cadence request/latency/error series were emitted that way, so some scopes never appeared.

Fixes in this release:

  • Persistence rollups were renamed to persistence_requests_rollup, persistence_latency_rollup, and persistence_latency_ns_rollup. Direct persistence series retain their existing names and now consistently include labels such as is_retry and task_category="none" (#8064).
  • persistence_response_payload_size, persistence_response_row_size, and persistence_empty_response now use task_category="none" and no longer carry a domain label (#8338).
  • Per-task-list Cadence rollups were renamed to cadence_requests_rollup, cadence_errors_rollup, cadence_latency_rollup, cadence_latency_ns_rollup, and corresponding cadence_errors_*_rollup names. Frontend task-list errors no longer leak a tasklist label onto base cadence_errors*, and worker archiver latency uses domain="-1" for label consistency (#8270).

Migration Path

  • Point dashboards and alerts at the new *_rollup names. During a rolling upgrade, old processes emit aggregate series under the old base names while new processes emit the renamed rollups.
  • persistence_response_payload_size, persistence_response_row_size, and persistence_empty_response no longer have a domain label. Queries that group those series by domain need to be updated.

4. Cassandra Partition Size & Tombstone Reduction (#8318)

Cassandra creates a tombstone for every key-value deletion on a map column, and Cadence's pending-activity and pending-timer maps generate one on every activity completion or timer firing — a major source of partition-size and tombstone growth. This release lands the first step toward avoiding those deletes: sentinel-value read filters (activitySentinelScheduleID, timerSentinelTimerID) for the pending activity/timer maps, so a later write path can upsert sentinel values instead of deleting entries outright.

Migration Path

  • This change is read-side only and behavior-preserving — no operator action is needed for this release.
  • It must be fully deployed before the follow-up batch map-deletion write path ships (tracked separately, not yet merged as of this release), to guarantee backwards compatibility, and should not be rolled back once that write path is deployed and enabled via dynamic config.

Notable Bug Fixes & Enhancements

  • Matching task readers and writers now report fatal shutdown through the task-list manager, leaving teardown ownership with the manager and avoiding asynchronous child-triggered Stop races (#8347).
  • Cassandra task writes use gocql.UnsetValue instead of nil for unset fields, avoiding cell-level tombstones (#8317).
  • Treat DBUnavailableError as transient, preventing a transient QUORUM-unavailable read from being misclassified as missing history and avoiding a SignalWithStart ContinueAsNew path that could orphan the running execution (#8339).
  • SQL operations with an unexpected affected-row count now return a retryable error instead of EntityNotExistsError; a row-count mismatch does not necessarily mean the entity is absent (#8332).
  • Regenerate the activity retry timer on SyncActivity, preventing an activity retry from being permanently lost when domain failover occurs after the retry is recorded but before its backoff timer fires (#8425).
  • Schema setup treats a missing schema-version row as version zero so initialization can proceed (#8455).
  • Fixed visibility schema version bookkeeping for PostgreSQL and SQLite (#8345), and made MinCompatibleVersion non-decreasing across versioned manifests (#8468).
  • Domain deprecation now fails when any cluster still has active pollers (excluding cadence-scheduler); forced deprecation skips this check (#8399). Its activities use a 30-second heartbeat timeout so worker interruption is detected before the five-minute activity timeout (#8258).
  • Ringpop provider bootstrap failure no longer crashes the DaemonV2 process, although membership-resolver bootstrap failures remain fatal (#8398). Ringpop can also start as a one-node cluster when DNS initially returns no hosts (#8366).
  • The default admin.HeaderForwardingRules no longer forwards grpc- headers, preventing duplicate compression headers from breaking proxied RPCs. Explicit custom forwarding rules are unchanged (#8380).
  • Scheduler worker metrics and logs (#8473).
  • Feature-complete cadence schedule commands (#8297).
  • Batch start always uses v2; the old --v2 selector was removed (#8334).
  • Batch refresh to refresh workflow tasks matching a query (#8391).

Full Changelog: https://github.com/cadence-workflow/cadence/compare/v1.4.1...v1.4.2


Release v1.4.2-prerelease16 published by c-warren on 2026-10-01​

What's Changed

New Contributors

Full Changelog: https://github.com/cadence-workflow/cadence/compare/v1.4.2-prerelease15...v1.4.2-prerelease16


Release v1.4.1 published by neil-xie on 2026-06-30​

Migration Notes

  • Cassandra schema: transfer task and timer task tables require new columns for domain multi-tenancy task routing. Apply schema migrations before rolling out.
  • Histogram metrics: histograms config block must be explicitly set to opt into histogram emission. See Timer to Histogram Metric Migration section above for the full rollout path.

Major Features

1. Cadence Schedules

Cadence Schedules is now generally available. See the blog post and concepts documentation. Introduces a first-class scheduling primitive that replaces hand-rolled cron workflows. The full API surface (Create, Describe, Update, Delete, Pause, Unpause, List, Backfill) is implemented across the type system, proto mappers, Thrift transport, frontend handlers, and CLI. The scheduler runs as a per-domain durable workflow with ContinueAsNew, enforcing configurable overlap policies (SKIP, CONCURRENT, BUFFER, CATCH_UP_ALL), catch-up for missed fire times, concurrency limits, backfill with per-fire override policies, and search attribute propagation to target workflows. A per-domain scheduler worker manager uses membership-based routing and is enabled by default with a per-domain filter.

Migration Path

  • The scheduler worker is disabled by default. Operators can enable it per-domain via dynamic config.
  • Schedule search attributes must be added to your visibility index. They are included in the default key set automatically.

2. Active-Active Domains

Features

Added MySQL support for Active-Active Cadence Operations Failover workflow now supports passing the set of cluster-attributes that should be failed over

Bug Fixes

It is no longer possible to convert a domain to active-active via the FailoverDomain RPC

3. Domain Multi-Tenancy (in progress)

Extends Cadence's multi-tenancy guarantees below the domain level. Cadence already enforces rate limiting, task prioritization, and fair task scheduling at the domain level to prevent noisy-neighbor interference between customers. However, platform teams that consolidate multiple use cases into a single domain have historically had no isolation between those use cases. This release introduces task-list-level isolation primitives, including task routing, rate limiting, and scheduling priority, so that platform teams can get the same guarantees within a domain that domain-level isolation provides between domains.

Key Capabilities

  • Hierarchical weighted round-robin (IWRR) scheduler for fair task scheduling across task lists within a domain
  • Task list nice value for priority control between task lists within a domain
  • Task list rate limiting applied to both regular and sticky polls, preventing a single task list from starving others in the same domain
  • Task list name and kind stored in history and transfer tasks to enable task routing decisions
  • History task latency metrics tracked at the task list level for per-task-list observability

Migration Path

  • Cassandra schema upgrade required: transfer task and timer task tables gain new columns for task list name and kind. Apply schema migrations before rolling out this version.

Current Limitations

Tasklist-level isolation is currently in active development. Additional controls and enforcement points will be added in subsequent releases.

4. Timer to Histogram Metric Migration

Cadence is migrating from Tally's costly Timer metrics to ExponentialHistogram to reduce infrastructure costs and align with Prometheus and OpenTelemetry standards. The v1.4.1 release enables dual-emission for all timer metrics, alongside new GaugeMigration and CounterMigration frameworks.

Migration Path

Emission behavior is controlled per-metric or globally via the histograms block in your service config YAML. The valid modes are timer, histogram, and both.

  histograms:
    default: timer       # global default: "timer", "histogram", or "both"
    names:
      task_latency: true      # true = emit histogram, false = suppress
      persistence_latency: true

If histograms are not configured, or default is not set, behavior is the same as previous releases, only timers are emitted. Histogram emission must be explicitly opted into. Metrics not listed in the migration registry are always emitted regardless of this config. The recommended rollout path is:

  1. Set default: timer (or leave unset) to preserve existing behavior
  2. Opt individual metrics into both to validate histogram output alongside existing timers
  3. Set default: both once you are confident histograms are correct
  4. Set default: histogram to drop legacy timers when your dashboards and alerts are fully migrated

Current Limitations

Legacy timers will eventually be removed. The histograms config block is transitional and will become validation-only (rejecting explicit timer mode) once the migration is complete.

Performance & Scalability Improvements

  • Matching task list manager lookup is now O(1) by name (#7733).
  • Rate-limiter token waste and CPU spin under low task rates are fixed, significantly reducing idle CPU usage (#7977).
  • History queue processors are notified before shard lock release to reduce task processing latency (#8130).
  • Automatic mutable state corruption detection and repair (#7850).
  • Worker redundancy is improved (#7840).
  • Batcher RPS and concurrency can now be tuned at runtime via signals (#7824).

Notable Bug Fixes

  • Matching

    • Fix two separate getTasksPump deadlocks (#7855, #7930)
    • Fix rate-limiter token waste and CPU spin under low task rates (#7977)
    • Reclassify query-task-not-found as EntityNotExistsError (#7938)
    • Add timeout to notifyPartitionConfig during startup (#7833)
    • Limit context deadline when calling RecordTaskStarted (#7792)

  • Persistence / History

    • Use next_event_id column as source of truth when reading workflow execution from Cassandra (#7738)
    • Trim workflow timer tasks on workflow close and deletion (#7941)
    • Fix deadlock in workflow reset signal reapplication (#7913)
    • Convert replicator panics into errors rather than crashing the process (#8063)

Observability Enhancements

  • Add shard-id to persistence error logs (#8017)
  • Inject Start/Stop latency measurements in canary shards (#7985)
  • Emit correct is_retry tag on retried persistence and client calls (#8049)
  • Route PutReplicationTaskToDLQ metrics by domain (#8053)
  • Normalize cadence_authorization_latency labels for Prometheus (#8215)
  • Normalize cache metric labels for Prometheus (#8120)

CLI Enhancements

  • Add concurrency_limit support in CLI (#8028)
  • Add operational dynamic config CLI commands: get, update, delete (#8101)
  • Add --latest_time support for DecisionCompletedTime reset type (#8151)

Code Quality & Maintenance

  • Replace x/exp/maps with standard maps package (#8098)
  • Generalize LimiterFactory/Collection with type parameters (#7841)
  • Convert task schedulers to use Go generics (#7813)
  • Reduce log noise (#7915, #7910, #7903, #8071, #7901, #7991)

Infrastructure

  • Add domain audit log support for MySQL and SQLite
  • Add support for region-specific S3 access points in archival (#8015)
  • Update Docker image to support custom config file (#8155)
  • Fix Kafka default protocol version from 0.10.2.0 to 2.1.0 (#7890) — fixes startup crash with Sarama ≥ v1.45 against modern Kafka clusters
  • Reject requests that update both replication config and domain data simultaneously (#8252)

Full Changelog: https://github.com/cadence-workflow/cadence/compare/v1.4.0...v1.4.1


Release v1.4.0 published by neil-xie on 2026-02-27​

Major Features

1. Active-Active Domains

Cadence domains have been running in active-passive mode for years, which has been limiting for use cases requiring processing in all clusters (regions). Since late 2025, Cadence can process domains in both regions and distribute traffic based on users’ preferences within domains. This change will make your domains more flexible and more efficient due to utilizing resources in all regions.

Key Capabilities

  • Introduced ClusterAttribute as a flexible abstraction for defining cluster groupings beyond traditional region-based configurations
  • Active Cluster Selection Policy allows workflows to specify which cluster attributes they should run on

Migration Path

Active-active support is designed for backward compatibility:

  1. All domains will support active-active by default and without breaking existing behavior. While users setting “cluster attributes” while starting their workflows will be able to benefit from active-active processing
  2. Existing active-passive domains continue to work without changes if leaving “cluster attributes” empty

Current Limitations

This feature is currently implemented for Cassandra and the support for other DBs will come in Q1. We will also release a blog explaining how this improved related use cases, a wiki explaining how to use it and a code lab to help you try out. There is a risk for failover which causes workflows to be stuck if the schedule to start latency plus replication lag is more than 25 mins. We're working on a project to resolve the risk.

2. Replication Improvements

Cadence orchestrates its own replication, which allows us to seamlessly migrate from one DB technology to another, one cloud provider to another etc. The way it was working in the past was that the replication messages would be generated by reading workflow tasks from the database.

Given that replication is a continuous process between Cadence regions, we implemented a cache to keep the replication messages in memory until a replication poll message arrives so we could eliminate the DB calls due to replication. This came with a 99%+ cache hit rate, which almost entirely eliminated the DB calls due to replication, which used to be more than 20% of all DB calls. Another big benefit was for replication latencies; since we can directly serve the messages from memory, our replication latencies dropped from 13s to 2s.

Key Capabilities

  • Replication Budget Manager: New cache capacity control mechanism to prevent memory exhaustion during replication bursts
  • Improved task fetcher concurrency: Better handling of concurrent replication task fetching with enhanced metrics
  • BoundedAckCache optimizations: Generic cache implementation with improved ack handling

3. Shard Distributor Service (in progress)

The Shard Distributor is a new service component that provides dynamic shard assignment and load balancing across matching service instances. This enables better resource utilization, improved scalability, and operational flexibility for large-scale deployments.

Service Components

  • Leader Processor: Centralized controller for shard assignment decisions
  • Executor Client: Integration point for matching service instances to receive shard assignments
  • Spectator Client: Read-only monitoring interface for shard state
  • Canary: Health verification and ping protocol for shard ownership validation

Key Capabilities

  • Dynamic Shard Assignment
  • Dynamic spectator client control with enable/disable support
  • Load Balancing
  • Drain watching support for graceful shard handoff
  • Automatic retry on rebalancing loop failures
  • Migration Support: Migration mode for gradual rollout alongside existing static shard assignment
  • Integration with Matching Service: The matching service has been refactored to support shard distributor integration:

Configuration

The shard distributor can be enabled via dynamic config:

  shard-distributor:
      enabled: true
      loadBalancingMode: "shadow-mode"  # Options: naive, shard-stats, shadow-mode
      migrationMode: true  # Enable gradual migration

Monitoring

  • shard_handover_latency: Time taken for shard ownership transfer
  • active_shards_count: Number of active shards per executor
  • shard_assignment_conflicts: Concurrent assignment conflicts detected
  • executor_heartbeat_status: Executor health and liveness
  • ETCD watch event metrics for observability

4. Caller Type-Based Rate Limiting

A new caller type tracking and bypass mechanism has been introduced to allow granular rate limiting control for debugging and mitigation purposes. Key Capabilities:

  • Caller Type Header Propagation (#7644, #7653, #7638): Introduced cadence-caller-type header that propagates through service boundaries
    • Extracted at service inbound boundaries using middleware
    • Available in CLI via header support
    • Minimal performance impact (~150-300ns per request)
  • Persistence Rate Limiter Bypass (#7656): Dynamic config persistence.rateLimiterBypassCallerTypes allows specific caller types to bypass persistence rate limits
  • Frontend Regional Rate Limiter Bypass (#7662): Caller type-based bypass for regional frontend rate limiter to allow priority requests during high load

5. Visibility Enhancements - Cron and Execution Status (#7527)

Added comprehensive cron workflow visibility with new fields in visibility records:

New Fields:

  • CronSchedule: Display the cron schedule for workflows
  • ExecutionStatus: Show actual execution status (PENDING, STARTED, or close statuses) instead of just CONTINUED_AS_NEW for cron workflows
  • ScheduledExecutionTime: Track the actual scheduled execution time for cron workflows

Schema Changes Required: This feature requires database schema upgrades for all persistence stores (Cassandra, MySQL, PostgreSQL, SQLite, Elasticsearch)

CLI Updates:

  • New --print_cron flag to display cron-related fields in cadence workflow list
  • Shows execution status by default

Performance & Scalability Improvements

  • Database & Persistence

    • PostgreSQL timer task pagination (#7621): Improved pagination logic to handle large timer task queries efficiently
    • History node deletion (#7484): Configurable page size for history deletion via dynamic config
    • Snappy compression for history blobs (#7269): Reduced storage footprint and network transfer for history events
    • SQLite fixes (#7469): Resolved database locking issues for local/test deployments

  • Memory & Resource Usage

    • ETCD watch optimization (#7578): Removed WithPrevKV() to reduce memory overhead in shard manager
    • Reduced allocations in metrics (#7456): Optimized insertReportIntoSizes to minimize GC pressure
    • History deletion improvements (#7472): Fixed infinite loop in RangeCompleteHistoryTask when invalid page size provided

Notable Bug Fixes

  • History & Workflow Execution

    • Child workflow duplicate events (#7400): Proper handling of duplicate child workflow started events
    • Activity scheduled time on reset (#7597): Correctly update not-started activities scheduled time when resetting workflows
    • Restart workflow cron scheduling (#7247): Fixed bug where each restart skipped an additional cron scheduled run
    • History cleanup timeout handling (#7617): Avoid dangerous timeout conditions in history cleanup process
    • Workflow creation leak (#7523): Fixed resource leak during workflow creation in history service
    • Signal-with-start cleanup (#7540): Proper handling of signal-with-start in cleanup logic
    • Signal handling with DelayStart (#7702): Prevent signals from bypassing DelayStart configuration

  • Cross-Datacenter Replication

    • Domain ID usage in replication (#7550): Use domain ID instead of domain name for more reliable replication
    • Replication panic logging (#7396): Improved error handling and logging for replication stack panics
    • Database consistency error detection (#7573): More accurate detection of DB consistency errors

  • Active-Active Operations

    • Race condition in failover (#7587): Fixed race condition during active-active failover
    • Query workflow support (#7339): Proper query handling for active-active domains
    • StartWorkflow with terminate-if-running (#7361): Correct policy enforcement for active-active workflows
    • Auto-forwarding (#7356): Fixed cluster forwarding logic for active-active domains
    • Standby task handling (#7423): Prevent premature dropping of standby tasks in active-active scenarios

  • Matching Service

    • TaskList stop on shard stop (#7581): Properly stop task lists when stopping shard processor
    • TaskListActivitiesPerSecond enforcement (#7575): Correct rate limiting enforcement
    • Nil load hints handling (#7551): Added nil pointer checks for load hints
    • TaskList partition config invalidation (#7618): Properly invalidate TaskListPartitionConfig on attempted writes to read-only partitions
    • Domain not active error handling (#7676): Fixed domain not active error to be non-retryable for matching service in active-active scenarios
    • TaskList management with shard distributor (#7682): Properly handle shard processor lifecycle when onboarded to SD
    • Task list registry pattern (#7720): Introduced registry for better task list management

  • Persistence & Database

    • Host tag reversion (#7675): Reverted addition of host tag to persistence calls due to issues
    • History cleanup defaults (#7661): Changed defaults for history cleanup configuration
    • History cleanup error classification (#7627): Tightened error classifications for history cleanup operations
    • Visibility upsert optimization (#7693): Only upsert search attributes when advanced visibility is enabled

  • CLI

    • Rate limiter fix (#7585): Replaced token bucket with standard limiter in CLI for more reliable rate limiting
    • Admin CLI config parsing (#7726): Fixed parsing of multiple config values

Observability Enhancements

  • Metrics

    • Host tagging for persistence metrics (#7530): Better attribution of persistence operations to specific hosts
    • Shard handover latency (#7442, #7614): Track time taken for shard ownership transfers
    • Replication task fetcher metrics (#7462): Enhanced visibility into task fetcher performance
    • Workflow access tracking (#7331): New metrics to track workflow access patterns

  • Logging

    • CallerType and CallerInfo propagation (#7564, #7574, #7588): Context propagation for request tracing
    • Improved replication logging (#7584): Better logs for matching and active-active operations
    • Cluster redirection logs (#7333): Enhanced logging for cluster redirection handler
    • Error classification (#7466): Better categorization of errors in shard distributor

  • Monitoring Tools

    • Canary Grafana dashboard (#7464): Documentation for workflow success counter panels
    • Dynamic config observability (#7605): Standardized comments for all dynamic configs

CLI Enhancements

  • Failover domain command (#7295): New cadence domain failover command for active-active domains
  • Cluster attributes in start workflow (#7494): Support --cluster-attributes flag when starting workflows
  • Describe workflow improvements (#7461): Show ActiveClusterSelectionPolicy in describe output
  • Cluster attributes in domain describe (#7539): Display cluster attributes when describing domains
  • Failover history rendering (#7444, #7407): Improved rendering of domain failover history
  • Workflow refresh tasks command (#7657): New cadence workflow refresh-tasks command allows non-admin users to refresh workflow tasks

Code Quality & Maintenance

  • Refactoring

    • Domain handler refactoring (#7403, #7401, #7395): Multi-phase cleanup and modernization
    • History engine test refactoring (#7343, #7342): Improved test structure and maintainability
    • Matching engine refactoring (#7593, #7592, #7591, #7547): Replaced callbacks with explicit Registry pattern
    • Mapper nil handling (#7434): Comprehensive nil checking improvements across all mappers
    • Remove panics in type conversions (#7258): Safer error handling in proto/thrift mappers
    • QueueManager/Queue interfaces (#7652): Refactored queue manager and queue interfaces for better maintainability
    • Matching load calculation (#7647): Simplified load calculation for shards
    • Domain audit filter cleanup (#7725): Removed MaxCreatedTime from DomainAuditLogFilter

  • Testing

    • Simulation test improvements (#7303, #7283): Better active-active simulation testing
    • Integration tests (#7344): New DescribeCluster integration test
    • Conventional commit enforcement (#7278): CI validation of commit message structure

  • Deprecation & Cleanup

    • Removed deprecated dynamic configs (#7613): Cleanup of unused configuration properties
    • Removed deprecated fields (#7388, #7365, #7357, #7275): Cleanup of legacy active-active fields
    • Commented code removal (#7592): Removed obsolete code in matching engine
    • ExternalEntityProvider removal (#7292): Removed deprecated active-active provider

  • Infrastructure

    • CI improvements (#7649): Added pull request reviewer automation via gitar
    • PR template updates (#7596): Added comprehensive reviewers checklist to pull request description
    • Issue automation (#7615): GitHub action to standardize issue descriptions and labeling
    • Code ownership (#7659): Added natemort and c-warren to CODEOWNERS

  • Code Generation

    • Missing codegen (#7663): Fixed missing code generation that had snuck into master

Security & Authorization

  • ResetStickyTaskList auth (#7340): Added to non-domain auth API list
  • Auth README documentation (#7368): Comprehensive authentication documentation

Infrastructure

  • Docker compose for OpenSearch (#7510): Local development support for OpenSearch
  • Go toolchain upgrade (#7414): Updated Go toolchain and mockery
  • ETCD integration test fixes (#7502): Stabilized ETCD-based tests
  • Multi-cluster test scripts (#7327): Moved scripts to proper location

Migration Notes

Schema Upgrades Required

  • Visibility Schema v0.10 (Cassandra): Adds ExecutionStatus, CronSchedule, and ScheduledExecutionTime fields
  • Visibility Schema v0.8 (MySQL): Adds execution-related fields
  • Visibility Schema v0.9 (PostgreSQL): Adds execution-related fields
  • Visibility Schema v0.2 (SQLite): Adds execution-related fields
  • Elasticsearch Templates: Updated for v6, v7, and OpenSearch v2
  • PostgreSQL support for DomainAudit (#7665)
  • Cassandra schema updates synced with IDL changes (#7727, #7723)

Dynamic Config Changes

  • New config: persistence.rateLimiterBypassCallerTypes - List of caller types to bypass persistence rate limiting

Documentation

  • Auth system documentation (#7368)
  • Canary monitoring dashboard guide (#7464)
  • Domain update and failover documentation (#7382)
  • Multi-cluster setup improvements (#7282)
  • Contributor guide updates (#7337)

New Contributors

Resources


Release v1.3.6 published by neil-xie on 2025-10-15​

What's Changed

Added

Changed

Fixed

Removed

New Contributors

Full Changelog: https://github.com/cadence-workflow/cadence/compare/v1.3.5...v1.3.6


Release v1.3.5 published by jakobht on 2025-09-18​

New Features & Major Enhancements

  • Shard Management & Distribution:
    • We now assign shards to empty executors, so when a new executor joins it will have shards assigned to it (#7175) by @jakobht
    • Support for creating ephemeral shards (#7209) by @jakobht
    • Remove done ephemeral shards (#7212) by @jakobht
    • Integration of removal of ephemeral shards in the executor library (#7216) by @jakobht
    • Determinized the shard assignments (#7184) by @jakobht
    • Add multi-namespace support to shard distributor executor client (#7236) by @jakobht
  • Active-Active:
    • [active-active] Insert cluster selection policy row as part of continue-as-new (#7146) by @taylanisikdemir
    • Return ActiveClusterSelectionPolicy on DescribeWorkflow requests (#7181) by @c-warren
    • [active-active] Add simple lru cache for workflow cluster selection policies (#7174) by @taylanisikdemir
  • Authorization & Security:
    • Introduce a feature flag to enable admin authorization (#7168) by @gazizayestemir
    • [admin-auth]Make ListDomainEndpoint accessible for customers (#7187) by @gazizayestemir
  • History & Matching:
    • Define feature flags to support virtual queue split for history queue v2 (#7141) by @Shaddoll
    • Implement handler for queue pending task count alert (#7150) by @Shaddoll
    • Support dispatching activities with a specific TaskListKind (#7157) by @natemort
  • Workflow Diagnostics:
    • [Wf-Diagnostics] return details of currently running diagnostics on duplicate trigger (#7142) by @sankari165
    • [Wf-Diagnostics] Limit number of issues per type returned in diagnostics (#7189) by @sankari165
  • Serialization:
    • Add a feature flag for serialization encoding (#7148) by @tubignat

Bug Fixes

  • History / Matching:
    • Do not generate scheduled task with timestamp in the past (#7070) by @Shaddoll
    • Don't attempt to dispatch expired activities (#7204) by @natemort
    • Fix standby task processing for history queue v1 (#7144) by @Shaddoll
    • Fix bugs for history queue v2 (#7178) by @Shaddoll
    • Reset update queue state timer on failure (#7194) by @Shaddoll
    • [active-active] Fix lookup workflow by domain id (#7171) by @taylanisikdemir
  • Persistence & Visibility:
    • Remove version constraint for OpenSearch bulk delete requests (#7163) by @neil-xie
    • Remove version constraint for es v6 and v7 bulk delete requests (#7167) by @neil-xie
    • Fix write fallback logic when use advanced writing mode but only db is available (#7200) by @neil-xie
    • catch scroll node not found exception and start fresh scroll (#7155) by @anuragrai16
  • CLI & UI:
    • Fix the incorrect error message output of the command cadence workflow start/run (#7182) by @ins-tril
    • fix: update the incorrect UI URL (#7183) by @ins-tril
    • fixed typo in Search Attributes creation (#7208) by @dkrotx
  • Other Fixes:
    • [Wf-Diagnostics] fix marshalling error in failure rootcause (#7166) by @sankari165
    • Grafana: Client Concurrency Panel bugs fixes (#7198) by @Bueller87

Improvements & Refactoring

  • Core Internals & Performance:
    • Improve handling of unknown isolation groups (#7143) by @natemort
    • Validate workflow reset based on requested event type (#7116) by @fimanishi
    • Force creating new virtual slice every 5 mins (#7161) by @Shaddoll
    • Check peer existence before release peer (#7162) by @neil-xie
    • Set a limit on page size for history queue v2 (#7179) by @Shaddoll
    • Added shard not found error (#7186) by @jakobht
    • Bypass failover drills for active-active domains (#7191) by @c-warren
    • Retry standby tasks immediately on failover (#7199) by @Shaddoll
    • Improve domain failover for history queue v2 (#7205) by @Shaddoll
  • Code Quality & API:
    • exposes constructor for Cassandra DB instance tooling use (#7153) by @davidporter-id-au
    • makes the option to pass in a GoCQL connection public (#7154) by @davidporter-id-au
    • [Wf-Diagnostics] remove workflow retry as a potential diagnostic issue (#7156) by @sankari165
    • Fix Best Practice in Rule File (#7152) by @jayne-s
    • Refactered the shard reassignment method (#7165) by @jakobht
    • refactor unit test cases for ConstructStartWorkflowRequest to reduce code duplication (#7206) by @ansidev
    • Refactor data manager interface (#7229) by @ansidev
  • Observability (Metrics & Logging):
    • Introduce new tag for actor type (#7197) by @sankari165
    • Improve logging and metric for history queue v2 (#7221) by @Shaddoll
    • Add Domain Redirection Metrics for Active-Active (#7202) by @c-warren
    • Add complete history tasks metric (#7234) by @Shaddoll
  • Configuration & Defaults:
    • Update the default value of history.readNoSQLHistoryTaskFromDataBlob to true (#7173) by @Shaddoll

Tooling, CI, & Operations

  • CLI & Admin Tools:
    • Add MCP Tool for Cadence Workflow Commands (#7151) by @jayne-s
    • [cli]Make security token an optinal argument (#7192) by @gazizayestemir
  • Local Development & Docker:
    • Use a target-arch-compatible (and more up to date) version of dockerize (#7149) by @Groxx
    • Add direnv support for local development (AI generated/assisted) (#7188) by @jakobht
    • [docker] set matching.enableClientAutoConfig to true in dynamic config (#7193) by @shijiesheng
    • Changing to legacy docker image of bitnami (#7219) by @dkrotx
  • Observability:
    • Add client autoscaler grafana panels (#7177) by @Bueller87
  • Testing & CI:
    • Fix flaky test for virtual queue (#7145) by @Shaddoll
    • Test reapply events active-active logic (#7135) by @c-warren
    • Run replication simulation scenarios as part of CI (#7158) by @taylanisikdemir
    • Fix thrift decode test flakyness (#7160) by @taylanisikdemir
    • Test Mutable State Load() in Active-Active scenarios (#7159) by @c-warren
    • Test StartTransaction (#7169) by @c-warren
    • Test allocateTimerIDsLocked() (#7172) by @c-warren
    • Set up simulation test for history queue v2's pending task alert (#7180) by @Shaddoll
    • Set up integration test for pending task alert of history queue v2 (#7196) by @Shaddoll
    • Changing one of cli tests to be more explicit (#7211) by @dkrotx
    • Fixing flaky-tests in matching (#7213) by @dkrotx
    • Changing test to be compatible with urfave3 (#7217) by @dkrotx

Documentation

  • Add How-To for Async API setup (#7176) by @taylanisikdemir
  • Adding Directives + Using NewBatchFuture for Concurrency Example (#7170) by @jayne-s

New Contributors

Full Changelog: https://github.com/cadence-workflow/cadence/compare/v1.3.4...v1.3.5


Release v1.3.4 published by neil-xie on 2025-08-20​

What's Changed

Added

Changed

Fixed

Removed

Full Changelog: https://github.com/cadence-workflow/cadence/compare/v1.3.3...v1.3.4


Release v1.3.3 published by neil-xie on 2025-08-06​

What's Changed

Added

Changed

Fixed

Removed

New Contributors

Full Changelog: https://github.com/cadence-workflow/cadence/compare/v1.3.2...v1.3.3


For a comprehensive list of changes and detailed release notes, please visit the Cadence GitHub Releases page.