Introduction to Shard Manager
Shard Manager is an open-source, general-purpose sharding service. It assigns the shards of any sharded application to its hosts, and keeps that assignment balanced by load as hosts come and go. It runs as its own service, backed by etcd, and applications integrate through its Go clients. Cadence Matching is one of the services that uses it.
You can also read our post about Shard Manager on the Uber Engineering blog, and find the source code on GitHub.
Why Shard Manager
A common way to assign shards to hosts is a consistent hash ring. Cadence, for example, has long used Ringpop. While mathematically and theoretically beautiful, consistent hashing gives a set of problems:
- Load Balancing - A ring balances by shard count, never by shard load. Moving a shard means every host changing its view of the ring at the same instant, and nothing coordinates that. Methods such as virtual nodes make it possible to shuffle the shards, but do not allow fine-grained control.
- Graceful Handovers - When a shard moves in a traditional consistent hash ring, requests will simultaneously go to both owners, causing availability drops. With a centralized Shard Manager we let the old owner drain before the new one takes over. It also opens the door to warming caches on the new owner before the handover, and truly zero-downtime transfers. Graceful handover is not implemented yet.
- Debuggability and Introspection - In a traditional consistent hash ring the state of the shard assignments is spread across all participants, and there is no authority on what the correct state is. A single misbehaving instance can therefore cause issues across the cluster, and shard assignment issues are extremely hard to debug. With central assignment the mapping is a record we can inspect with
smctl, the Shard Manager CLI. - Operability - A ring cannot be steered. There is no way to take a single hot shard off a host, or to empty a host ahead of a deploy, without changing the membership list itself.
Shard Manager solves all of these problems by centralizing the assignment decision: balancing by real load, introspection with smctl, and operability such as draining shards and hosts.
The diagram below shows how Shard Manager works, with executors processing the shards and spectators routing requests to them.
Running it
Shard Manager needs etcd. The repository ships a compose file:
docker compose -f docker/github_actions/docker-compose.yml up -d etcd
make bins
./shard-manager-server start --services shard-distributor
It reads config/development.yaml by default, which points at localhost:2379, defines a few development namespaces, and serves gRPC on port 7943.
To watch it do something, run the canary. It starts executors and a pinger against a fixed and an ephemeral namespace. It is also a good example of how integration with Shard Manager looks:
make start-shard-manager-canary
Where to go next
- Architecture. How the service, etcd, leader election, and rebalancing fit together.
- Service Integration. How to add Shard Manager to your own service.
- cadence-workflow/shard-manager. The source, the client libraries, and
smctl.