Senior Azure Data Engineer

Job Description

Job Description Senior Azure Data Engineer Remote | Enterprise Data Warehouse / ETL Modernization
Occasional travel to Springfield, IL, approximately 2–3 times per year Position Summary We are seeking a Senior Azure Data Engineer to help design, build, and operate a next-generation enterprise data platform on Microsoft Azure. This is a hands-on data engineering role with a strong focus on Azure Data Factory, Databricks, PySpark, Azure Synapse, modern Lakehouse architecture, advanced SQL, Python, and enterprise data warehouse modernization. The ideal candidate will have experience building scalable enterprise data pipelines and modernizing legacy ETL/data warehouse environments, including migrations from Informatica PowerCenter and on-premises platforms to Azure-based solutions. Key ResponsibilitiesData Pipeline Development

• Design and build robust, reusable, parameter-driven ingestion and transformation pipelines using Azure Data Factory, Synapse Pipelines, Databricks and/or Microsoft Fabric Data Factory.
• Develop and optimize PySpark notebooks and jobs using Azure Databricks or Synapse Spark.
• Build high-performing ELT workflows leveraging platforms such as Synapse Dedicated SQL Pool, Azure SQL and Teradata.
• Develop scalable data solutions supporting enterprise analytics, regulatory reporting, operational dashboards and emerging AI/ML use cases.

Lakehouse & Data Warehousing

• Implement Medallion Architecture using Bronze, Silver and Gold layers.
• Work with Azure Data Lake Storage Gen2, Delta Lake, Parquet and structured streaming.
• Design dimensional models using Kimball star and snowflake methodologies.
• Work with Data Vault modeling patterns.
• Implement Slowly Changing Dimensions, Change Data Capture and late-arriving data patterns.
• Optimize distributed SQL workloads, including distribution keys, partitioning and clustered columnstore indexes.

Azure Platform & DevOps

• Develop and support Azure Data Factory pipelines, datasets, linked services, triggers, parameterization and mapping data flows.
• Work with Azure, Self-hosted and SSIS Integration Runtimes.
• Implement CI/CD for data pipelines using Azure DevOps.
• Work with YAML pipelines and infrastructure-as-code technologies including ARM, Bicep and/or Terraform.
• Implement logging, auditing and monitoring using Azure Monitor, Log Analytics and KQL.
• Work with Azure Key Vault, Entra ID, managed identities, service principals and private networking.

Migration & Modernization

• Lead or contribute to legacy-to-cloud data modernization initiatives.
• Support Informatica PowerCenter to Azure Data Factory migrations.
• Help modernize on-premises Teradata, Oracle and SQL Server environments to Azure Synapse, Databricks, Fabric and other modern data platforms.
• Participate in workload assessment, capacity planning and cost modeling for target-state architectures.
• Support production incident response and troubleshooting for critical data pipelines.

Required Qualifications

• Strong hands-on expertise with Azure Data Factory, including:
  • Pipelines
  • Datasets
  • Linked services
  • Triggers
  • Parameterization
  • Mapping Data Flows
  • Azure, Self-hosted and SSIS Integration Runtimes
• Strong hands-on Databricks and PySpark experience.
• Production experience with one or more of:
  • Azure Synapse Analytics
  • Azure Databricks
  • Microsoft Fabric
• Strong Azure Data Lake Storage Gen2 experience.
• Strong Python data engineering experience, including pandas and PySpark.
• Advanced SQL experience, including:
  • Window functions
  • CTEs
  • Query optimization
  • Execution plan analysis
  • Performance tuning
• Experience with modern Lakehouse architecture, Delta Lake and Parquet.
• Experience with Azure Key Vault and Azure AD / Entra ID.
• Experience with Azure DevOps and CI/CD.
• Experience with Azure Monitor, Log Analytics and KQL.
• Proficiency with T-SQL.
• Familiarity with Spark SQL, KQL, PowerShell and Bash scripting.

Preferred Qualifications

• 5+ years of data warehouse development experience.
• 5+ years of data modeling experience using ERWIN or similar tools.
• 2+ years of experience with Azure Data Factory and Snowflake.
• Experience modernizing Informatica PowerCenter or other legacy ETL environments.
• Experience with Teradata, Oracle and/or SQL Server.
• Medicaid domain knowledge is a plus.

Strong Candidates Will Have

• Hands-on Azure Data Factory development.
• Strong Databricks and PySpark experience.
• Strong Python and advanced SQL skills.
• Experience building enterprise-scale ETL/ELT pipelines.
• ADLS Gen2 and Delta Lake experience.
• Lakehouse and Medallion Architecture experience.
• Azure Synapse and/or Microsoft Fabric experience.
• Azure DevOps and CI/CD experience.
• Legacy ETL modernization or Informatica-to-Azure migration experience.
• Enterprise data warehouse experience.