HARDDistributed Systems
Design a Distributed Task Scheduler
Design a distributed task scheduler like Apache Airflow or Temporal that can handle millions of scheduled jobs with dependencies.
Estimated Time: 45 minutes
#DAG#Task Queue#Scheduling#Fault Tolerance
Solution Overview
Model workflows as DAGs. Use consistent hashing for task distribution. Implement checkpointing for fault tolerance.
Hints to Get Started
1
How to handle task dependencies?
2
Exactly-once execution guarantee
3
Dead letter queue for failed tasks
Components
- •Scheduler Service
- •Worker Pool
- •Task Queue
- •State Store
- •Monitoring
Scheduling Strategies
- •Cron-based
- •Event-driven
- •Dependency-based