Storage scan
It is common to have large data sets partitioned across a large number of hosts or databases, or having billions of files in an Amazon S3 bucket. Cadence is an ideal solution for implementing the full scan of such data in a scalable and resilient way. The standard pattern is to run an activity (or multiple parallel activities for partitioned data sets) that performs the scan and heartbeats its progress back to Cadence. In the case of a host failure, the activity is retried on a different host and continues execution from the last reported progress.
Samples
Runnable samples covering the pieces of this pattern:
| Sample | Description | Code |
|---|---|---|
| Split and merge | Fans out parallel activities per partition and merges their results | Go |
| Bounded concurrency | Processes many items with a fixed number of parallel workers | Go |
| Heartbeat progress and resume | Activity reports progress via heartbeat and resumes from the last checkpoint after a failure | Go |
A real-world example:
- Cadence internal system workflow that performs periodic scan of all workflow_execution records