At Uber, we previously relied on a dynamic configuration service to manually control the number of partitions for scalable tasklists. This configuration approach introduced several operational challenges:

• Error-prone: Manual updates and deployments were required.
• Unresponsive: Adjustments were typically reactive, often triggered by customer reports or observed backlogs.
• Irreversible: Once increased, the number of partitions was rarely decreased due to the complexity of the two-phase process, especially when anticipating future traffic spikes.

To address these issues, we introduced a new component in the Cadence Matching service: Adaptive Tasklist Scaler. This component dynamically monitors tasklist traffic and adjusts partition counts automatically. Since its rollout, we've seen a significant reduction in incidents and operational overhead caused by misconfigured tasklists.

---

What is a Scalable Tasklist?​

A scalable tasklist is one that supports multiple partitions. Since Cadence’s Matching service is sharded by tasklist, all requests to a specific tasklist are routed to a single Matching host. To avoid bottlenecks and enhance scalability, tasklists can be partitioned so that multiple Matching hosts handle traffic concurrently.

These partitions are transparent to clients. When a request arrives at the Cadence server for a scalable tasklist, the server selects an appropriate partition. More details can be found in this document.

How Is the Number of Partitions Manually Configured?​

The number of partitions for a tasklist is controlled by two dynamic configuration properties:

1. matching.numTasklistReadPartitions: Specifies the number of read partitions.
2. matching.numTasklistWritePartitions: Specifies the number of write partitions.

To prevent misconfiguration, a guardrail is in place to ensure that the number of read partitions is never less than the number of write partitions.

When increasing the number of partitions, both properties are typically updated simultaneously. However, due to the guardrail, the order of updates doesn't matter—read and write partitions can be increased in any sequence.

In contrast, decreasing the number of partitions is more complex and requires a two-phase process:

1. First, reduce the number of write partitions.
2. Then, wait for any backlog in the decommissioned partitions to drain completely.
3. Finally, reduce the number of read partitions.

Because this process is tedious, error-prone, and backlog-sensitive, it is rarely performed in production environments.

---

How Does Adaptive Tasklist Scaler Work?​

The architecture of the adaptive tasklist scaler is shown below:

1. Migrating Configuration to the Database​

The first key change was migrating partition count configuration from dynamic config to the Cadence cluster’s database. This allows the configuration to be updated programmatically.

• The adaptive tasklist scaler runs in the root partition only.
• It reads and updates the partition count.
• Updates propagate to non-root partitions via a push model, and to pollers and producers via a pull model.
• A version number is associated with each config. The version only increments through scaler updates, ensuring monotonicity and consistency across components.

2. Monitoring Tasklist Traffic​

The scaler periodically monitors the write QPS of each tasklist.

• If QPS exceeds an upscale threshold for a sustained period, the number of read and write partitions is increased proportionally.
• If QPS falls below a downscale threshold, only the write partitions are reduced initially. The system then waits for drained partitions to clear before reducing the number of read partitions, ensuring backlog-free downscaling.

---

Enabling Adaptive Tasklist Scaler​

Prerequisites​

To use this feature, upgrade Cadence to v1.3.0 or later.

Also, migrate tasklist partition configurations to the database using this guide.

Configuration​

The scaler is governed by the following dynamic configuration parameters:

• matching.enableAdaptiveScaler: Enables the scaler at the tasklist level.
• matching.partitionUpscaleSustainedDuration: Duration that QPS must stay above threshold before triggering upscale.
• matching.partitionDownscaleSustainedDuration: Duration below threshold required before triggering downscale.
• matching.adaptiveScalerUpdateInterval: Frequency at which the scaler evaluates and updates partition counts.
• matching.partitionUpscaleRPS: QPS threshold per partition that triggers upscale.
• matching.partitionDownscaleFactor: Factor applied to introduce hysteresis, lowering the QPS threshold for downscaling to avoid oscillations.

---

Monitoring and Metrics​

Several metrics have been introduced to help monitor the scaler’s behavior:

QPS and Thresholds​

• estimated_add_task_qps_per_tl: Estimated QPS of task additions per tasklist.
• tasklist_partition_upscale_threshold: Upscale threshold for task additions.
• tasklist_partition_downscale_threshold: Downscale threshold for task additions.

The estimated_add_task_qps_per_tl value should remain between the upscale and downscale thresholds. If not, the scaler may not be functioning properly.

Partition Configurations​

• task_list_partition_config_num_read: Number of current read partitions.
• task_list_partition_config_num_write: Number of current write partitions.
• task_list_partition_config_version: Version of the current partition configuration.

These metrics are emitted by various components: root and non-root partitions, pollers, and producers. Their values should align under normal conditions, except immediately after updates.

---

Status at Uber​

We enabled adaptive tasklist scaler across all Uber clusters in March 2025. Since its deployment:

• Zero incidents have been reported due to misconfigured tasklists.
• Operational workload related to manual scaling has been eliminated.
• Scalability and resilience of Matching service have improved significantly.

We are excited to announce the release of cadence-web v4.0.0—a complete rewrite of the Cadence web app. Cadence has always been about empowering developers to manage complex workflows, and with this release, we not only modernize the web interface by embracing today’s cutting-edge technologies but also strengthen the open source community by aligning our tools with the broader trends seen across the industry.

What's new in cadence-web v4.0.0​

• Revamped UI & Experience – A fresh, modern interface designed for better usability and efficiency.
• Multi-Cluster Support – The UI can now connect to multiple Cadence clusters.
• Performance Improvements – Faster load times, optimised API calls, and a smoother experience.

We’re excited to announce that all Cadence GitHub repositories have been consolidated under the cadence-workflow organization! 🎉

Previously, Cadence repositories were distributed across multiple organizations at Uber: uber, uber-go, uber-common. To improve developer cohesiveness and simplify access, the Cadence Core team has migrated all open-source repositories to the cadence-workflow organization.

For example, our main repository has moved from:

👉 uber/cadence

To its new home:

👉 cadence-workflow/cadence

You can find the full list of Cadence repositories here 👉 orgs/cadence-workflow/repositories

At Uber, we want to achieve regional resilience such that losing a zone within a region can be tolerated without requiring a cross-region failover. We also want to make sure that losing a zone only affects a subset of workload, at most, rather than everything. However, in Cadence-based systems, the workload in a region is distributed randomly across all workers in the region at a “task-level granularity”, which means a workflow may be worked on by any worker in the region where the domain is active. To achieve this goal, we introduced Zonal Isolation for Cadence Workflows - a feature designed to pin workflows to the zone they are started in, so that zonal isolation can be achieved at a workflow-level.

What is Zonal Isolation for Cadence Workflows?​

At high-level, Zonal Isolation for Cadence Workflows can be thought in 2 levels:

1. Task-level isolation: All decision tasks and activity tasks of a workflow are only processed by workers from the zone where the workflow was started
2. Infrastructure-level isolation: Within a regional Cadence cluster, workflows are handled by server instances in the same zone where they were started, and the corresponding data is stored in that zone as well.

Infrastructure-level isolation is quite challenging to implement as it requires significant changes to the core design of the Cadence server. Due to the complexity involved, support for this feature is not planned for the foreseeable future.

As a result, the focus remains on achieving task-level zonal isolation outside the Cadence server, which offers a more practical and immediate way to improve system resilience. It provides the capability of ensuring that an unhealthy zone (i.e. bad deployment of workers) only affect a subset of workflows (started from a certain zone) rather than every workflow in a Cadence domain.

We’ve heard your feedback: deploying Cadence has been a challenge, especially with limited documentation on operational aspects. So far, we’ve only provided a few docker compose files to help you get started on a development machine. However, deploying and managing Cadence at scale requires a deep understanding of underlying services, configurations and their dependencies.

To address these challenges, we’re launching several initiatives to make it easier to deploy and operate Cadence clusters. These include deployment specs for common scenarios, monitoring dashboards, alerts, runbooks, and more comprehensive documentation.

At Uber, we run several big multitenant Cadence clusters with hundreds of domains in each. The clusters being multi-tenant means potential noisy neighbor effects between domains.

An essential aspect of avoiding this is managing how workflows interact with our infrastructure to prevent any single workflow from causing instability for the whole cluster. To this end, we are excited to introduce Workflow ID-based rate limits — a new feature designed to protect our clusters from problematic workflows and ensure stability across the board.

Why Workflow ID-based Rate Limits?​

We already have rate limits for how many requests can be sent to a domain. However, since Cadence is sharded on the workflow ID, a user-provided input, an overused workflow with a particular id might overwhelm a shard by making too many requests. There are two main ways this happens:

1. A user starts, or signals the same workflow ID too aggressively,
2. A workflow starts too many activities over a short period of time (e.g. thousands of activities in seconds).

Introduction​

If you haven’t heard about Cadence, this section is for you. In a short description, Cadence is a code-driven workflow orchestration engine. The definition itself may not tell enough, so it would help splitting it into three parts:

• What’s a workflow? (everyone has a different definition)
• Why does it matter to be code-driven?
• Benefits of Cadence

What is a Workflow?​

In the simplest definition, it is “a multi-step execution”. Step here represents individual operations that are a little heavier than small in-process function calls. Although they are not limited to those: it could be a separate service call, processing a large dataset, map-reduce, thread sleep, scheduling next run, waiting for an external input, starting a sub workflow etc. It’s anything a user thinks as a single unit of logic in their code. Those steps often have dependencies among themselves. Some steps, including the very first step, might require external triggers (e.g. button click) or schedules. In the more broader meaning, any multi-step function or service is a workflow in principle.

If I change code logic inside an Cadence activity (for example, my activity is calling database A but now I want it to call database B), will it trigger an non-deterministic error?​

NO. This change will not trigger non-deterministic error.

An Activity is the smallest unit of execution for Cadence and what happens inside activities are not recorded as historical events and therefore will not be replayed. In short, this change is deterministic and it is fine to modify logic inside activities.

Does changing the workflow definition trigger non-determinstic errors?​

YES. This is a very typical non-deterministic error.

When a new workflow code change is deployed, Cadence will find if it is compatible with Cadence history. Changes to workflow definition will fail the replay process of Cadence as it finds the new workflow definition imcompatible with previous historical events.

Here is a list of common workflow definition changes.

• Changing workflow parameter counts
• Changing workflow parameter types
• Changing workflow return types

The following changes are not categorized as definition changes and therefore will not trigger non-deterministic errors.

• Changes of workflow return values
• Changing workflow parameter names as they are just positional

Welcome to the latest of our regular monthly Community Spotlight updates that gives you news from in and around the Cadence community!

It's been a couple of months since our last update so we have a lot of updates to share with you.

Please see below for a roundup of the highlights:

Welcome to the latest of our regular monthly Community Spotlight updates that gives you news from in and around the Cadence community!

Please see below for a roundup of the highlights:

More Cadence How To's​

You might have noticed that we have had a few more contributions to our blog from Chris Qin. Chris has been busy sharing insights, and tips on a few important Cadence topics. The objective is to help the community with any potential problems.

Here are the latest topics:

• Bad Practices and Anti-Patterns with Cadence - Part 1

• Non-Determistic Errors, Replayers and Shadowers

Even if you have not encountered these use cases - it is good to be prepared and have a solution ready.Please take a look and let us have your feedback.

Chris is also going to take a look at the Cadence Samples to make sure they are all working and if not - he's going to re-write them so that they do!

Thanks very much Chris for all the work you are doing to help improve the project!