Build a Secure Scheduler Control Plane
A timer is not an operations product. The moment several teams depend on scheduled work, people need to answer four questions safely: what is configured, what ran, who may change it, and what happens when coordination breaks? This lesson turns those questions into one integrated control plane.
You will build a Spring Boot service with a persisted task registry, ad-hoc triggering, searchable execution history, a professional web console, role-based access, database leases, leader election, integration tests, and bounded chaos controls. Work is real: the service executes live schedule and ad-hoc runs and records durable outcomes. No sample rows are inserted for the dashboard.
The important design split
The management plane accepts human intent: create, pause, edit, delete, inspect, and trigger. The execution plane turns persisted definitions into runs. Keeping that boundary visible matters. At large scale, management traffic may be tiny while execution traffic reaches millions per second.
The database is the source of truth. The browser never invents status; it refreshes task and execution records. An API response saying “accepted” means a durable execution exists, not that work has finished.
Three roles narrow the blast radius:
VIEWERcan inspect definitions, status, and history.OPERATORcan also trigger ad-hoc executions.ADMINcan mutate task definitions and run resilience experiments.
The demo uses HTTP Basic authentication because it is transparent for learning and scripts. Production deployments should terminate TLS, use an identity provider through OAuth 2.0/OIDC, rotate credentials, and record an audit event for every mutation.
Coordination without a magic “leader” flag
Leadership is a renewable database lease. Every heartbeat locks one lease row, checks its owner and expiry, then renews or declines. A standby may claim only an expired lease. The same primitive prevents overlapping runs of one task.
This gives an essential rule: ownership has a deadline. A boolean leader=true can survive forever after a network split; a lease cannot. The classroom lease is intentionally short. In production, choose a duration longer than normal database pauses but shorter than the recovery objective.
Only the current leader scans due definitions and submits executions. Scheduled requests carry deterministic idempotency keys. API callers may supply their own Idempotency-Key, so retrying a timed-out request returns the existing execution instead of creating another. A unique constraint on the key makes that guarantee durable.