01Context
The client's data pipeline was a large set of PowerShell scripts. Each one took raw data and turned it into a transformed, structured format in MySQL on Google Cloud. It worked, but it was slow, and scripts on their own don't give you the scheduling, retries and run history an orchestrator does.
Porting 1,200 jobs by hand would have taken months and produced 1,200 slightly different DAGs. So the migration itself became the thing to automate.
02The scale
03Decisions
D1Automate the conversion, don't hand-port
The automation turned the existing job logic into Airflow DAGs that all followed one pattern: pick up raw data, transform it, load it into structured tables.
Trade-offJobs that didn't fit the pattern still needed a person, and they took the longest.
D2One shape for every DAG
Same structure, same naming, same retry behaviour. Anyone on the team could debug any job without learning its quirks first.
Trade-offA few jobs were pushed into a shape that wasn't ideal for them.
04Results
05Under the hood
What I designed or built myself:
- rewrite PowerShell job logic rebuilt in Python as Airflow DAGs.
- load Raw data into transformed, structured tables in Cloud SQL for MySQL on GCP.
- pattern One DAG structure shared by all 1,200+ jobs.
- speed Processing 10x faster than the scripts it replaced.