HARDDistributed Systems
Design a Distributed Task Scheduler
Design a distributed task scheduler like Apache Airflow or Temporal that can handle millions of scheduled jobs with dependencies.
Estimated Time: 45 minutes
#DAG#Task Queue#Scheduling#Fault Tolerance
Solution Overview
Model workflows as DAGs. Use consistent hashing for task distribution. Implement checkpointing for fault tolerance.
Solution Overview
Model workflows as DAGs. Use consistent hashing for task distribution. Implement checkpointing for fault tolerance.
Hints to Get Started
1
How to handle task dependencies?
2
Exactly-once execution guarantee
3
Dead letter queue for failed tasks
Components
- •Scheduler Service
- •Worker Pool
- •Task Queue
- •State Store
- •Monitoring
Scheduling Strategies
- •Cron-based
- •Event-driven
- •Dependency-based