Eliminating Noisy Neighbors in Cadence Clusters
Cadence is a multi-tenant workflow orchestration platform, where a single cluster can serve many domains running different types of workflows. When a domain has a sudden traffic spike, it can consume a disproportionate share of resources used for history task processing and degrade performance for other domains in the same cluster, which is a noisy neighbor problem.
As the scale and diversity of workloads in a Cadence cluster grow, isolation between domains becomes increasingly important for maintaining predictable performance and reliability. To address this challenge, we redesigned two key components of the Cadence History service (the history task scheduler and history queue) to provide stronger domain-level isolation.




This is not a bug. It is how Cadence works by design, and it is the right default for most workloads. But three problems follow from it in production, and most teams hit at least one of them before they know the solution exists.
