Job Description
Job Description
Senior Data Engineer – Python, SQL, Azure & Databricks
Job Overview
We are seeking an experienced Senior Data Engineer to design, build, and optimize scalable enterprise data solutions. This position is ideal for a highly technical professional with strong experience in Python, SQL, ETL/ELT development, Azure, and Databricks.
The Senior Data Engineer will be responsible for developing reliable data pipelines, designing data models, improving data infrastructure, and transforming complex datasets into secure, scalable, analysis-ready solutions.
You will collaborate with data scientists, analysts, technical leaders, and cloud/infrastructure teams to ensure data systems are reliable, maintainable, secure, and optimized for analytical workloads.
Key Responsibilities
- Design, develop, and maintain enterprise-scale ETL/ELT data pipelines using Python and SQL.
- Build and optimize data models, data warehouses, and modern lakehouse environments.
- Develop scalable solutions using technologies such as Azure Data Factory and Databricks.
- Ingest and transform large datasets from APIs, relational databases, flat files, and unstructured data sources.
- Develop processes for data cleansing, validation, transformation, and integration.
- Implement data quality, metadata management, and data governance standards.
- Collaborate with cloud and infrastructure teams to deploy and maintain data solutions within Microsoft Azure.
- Monitor and troubleshoot data pipelines and identify opportunities to improve performance, reliability, and scalability.
- Implement logging, monitoring, alerting, and CI/CD practices for production data pipelines.
- Design and maintain relational and NoSQL data structures, schemas, and data models.
- Apply appropriate data security, access control, governance, and compliance practices.
- Document data architecture, pipeline logic, technical processes, and operational procedures.
- Collaborate with data scientists and analysts to understand evolving data requirements and deliver analysis-ready datasets.
Required Qualifications
- 8+ years of professional experience in data engineering, software engineering, database architecture, or a closely related field.
- 6+ years of hands-on experience with Python and SQL, including development of scalable ETL/ELT pipelines.
- 2+ years of experience working with cloud-based data platforms and technologies such as Azure Data Factory.
- 1+ year of hands-on experience with modern data processing platforms such as Databricks.
- Strong experience integrating and transforming data from multiple sources, including APIs, relational databases, flat files, and other enterprise systems.
- Experience with relational and NoSQL databases.
- Strong understanding of data modeling, database design, and schema design.
- Knowledge of enterprise data governance, security, access control, and compliance principles.
- Experience developing and supporting reliable production data pipelines.
- Bachelor's degree in Computer Science, Engineering, Information Systems, Data Science, or another related quantitative or technical field.
- Ability to support on-site work.
- Must be a U.S. Citizen or Lawful Permanent Resident (Green Card holder).
Preferred Qualifications
- Previous experience supporting large-scale data engineering initiatives in a government, public-sector, healthcare, or highly regulated environment.
- Experience collaborating with data scientists, analysts, and business stakeholders to develop data models supporting evolving analytical requirements.
- Knowledge of IT infrastructure, enterprise data access methodologies, and cloud architecture.
- Experience implementing advanced data security, governance, and compliance controls.
- Experience with CI/CD and automated deployment practices for data engineering workloads.
- Microsoft Azure certifications, such as Azure Data Engineer Associate, or comparable cloud/data certifications.
Ideal Candidate
The ideal candidate is a hands-on Senior Data Engineer who can take ownership of complex data pipelines and enterprise data environments from design through production. You should be comfortable working with large and diverse datasets, solving data quality and performance challenges, and collaborating across technical teams.
Strong hands-on expertise with Python, SQL, Azure Data Factory, Databricks, ETL/ELT, data modeling, and enterprise data architecture will be important for success in this role.