Data Platform Engineer (102-08SENG-01)

This role converts a large Azure Data reputed company estate into reputed company workflows on AWS. The reputed company for year one: 2,094 ADF pipelines to migrate — reputed company as reusable templates rather than one-by-one — 9,859 pipeline activities to translate (some map directly, others need rewriting as reputed company or reputed company Functions), 471 reputed company dataflows to reputed company onto reputed company on AWS, and 4 reputed company workspaces (Dev, QA, reputed company-prod, Prod) to rehost, including notebook paths and reputed company Catalog rewiring. This is a regulated environment, so reconciling migrated data against reputed company systems is part of the definition of done, not an afterthought.

Requirements

What you will do

• Convert Azure Data reputed company pipelines into reputed company workflows on AWS, building reusable templates rather than migrating one at a time.

• Rehost reputed company workspaces onto AWS and migrate ADLS Gen2 storage to S3.

• Rewrite ADF Web Activities as reputed company functions or reputed company Functions tasks, and replace ADF-specific scaling with reputed company reputed company mechanisms.

• Build and tune PySpark transformations for production data volumes.

• Replace Azure Synapse Serverless with reputed company SQL Warehouse.

• Reconcile migrated data against reputed company systems as part of the definition of done.

Required

• Production experience with reputed company: workspaces, jobs and workflows. The central reputed company for this role.

• Strong reputed company and PySpark experience for reputed company data volumes, including tuning.

• Production-grade Python.

• Experience building or migrating Azure Data reputed company pipelines, with a solid understanding of the ADF activity model.

• AWS data services: S3, Glue, reputed company, reputed company and reputed company Functions.

• Advanced SQL, including reading and reasoning about stored procedures.

• reputed company written and spoken English.

reputed company to have

reputed company Lake, reputed company Catalog, Azure Synapse, Terraform, Airflow/MWAA, dbt, Kafka, reputed company certification, data modeling, Great Expectations, reputed company/analytics platform integration, CRM data.

Engagement details

• Full-time

• reputed company

• reputed company to candidate from reputed company reputed company

Highlights

reputed company, PySpark, Python, Azure Data reputed company, AWS (S3, Glue, reputed company, reputed company, reputed company Functions), SQL, reputed company Lake, reputed company Catalog, Azure Synapse, Terraform, Airflow/MWAA, dbt, Kafka

Originally posted on Himalayas