Legacy batch jobs to SQL pipelines

Context

A General Ledger organization still depended on a large COBOL batch estate. Teams needed faster change cycles, better observability, and pipelines that could survive retries without corrupting accounting state.

Constraints

• Financial correctness mattered more than migration speed; partial automation could be worse than slower automation with evidence.
• Operations were still organized around batch windows, logs, and runbooks, so new tooling had to fit those workflows.
• Jobs still needed to show which environment and ledger slice they touched.

Approach

1. Inventory the real batch graph — which jobs are pure transform, which encode business rules, which are safe to parallelize.
2. Move selected jobs incrementally to SQL-based ETL — with explicit logging and idempotency at load boundaries, while leaving jobs that still needed the legacy path in place.
3. Instrument for operators — log parsing and replay-friendly steps so retries could be performed intentionally and traced afterward.
4. Add internal tooling — a full-stack, metadata-driven parameter dashboard with soft locks, versioning, audit logs, and approval workflows so report owners no longer had to request production edits by email.
5. Revamp the dashboard safely — clean architecture, atomic models, and decoupled metadata made the parameter workflow easier to scale and reuse.

Trade-offs

Not every job belonged in the first migration wave. Some stayed on legacy paths until the test harnesses and sign-off process caught up. The migration stayed incremental: jobs moved only after the required tests and sign-off process were ready.

Outcomes

• Internal project records measured one workflow at about two hours before the change and under five minutes on the modernized path.
• Internal project records indicate that most of the targeted legacy jobs moved to SQL-based ETL; the remaining jobs stayed on legacy paths pending tests and sign-off.
• Parameter and report management moved toward versioned, auditable workflows instead of ad hoc edits.
• A General Ledger journal-testing workflow that previously required several manual steps became a button-triggered run, removing the manual work required for that workflow.