Data Scientist

Job Description

Job DescriptionData Scientist
Indianapolis,IN
Hybrid

Key Responsibilities & DutiesData Strategy & Architecture

• Contribute to the enterprise data technology strategy, helping resolve complex data integration challenges (such as real-time event-based ingestion, data quality, and stewardship).

• Provide technical direction and oversight for designing and implementing integrated data solutions across secure systems.

• Assist with defining, implementing, and enforcing modern data management policies and engineering design standards.

Pipeline Development & Lakehouse Engineering

• Design, build, and maintain robust batch and streaming data pipelines in Azure Databricks utilizing PySpark, Spark SQL, and Delta Lake following a medallion (bronze/silver/gold) architecture.

• Implement incremental ingestion workflows via Auto Loader, Structured Streaming, or Delta Live Tables (DLT).

• Optimize cluster performance, right-size SQL warehouses, and troubleshoot partitioning, shuffle, skew, and caching issues using Spark UI.

Data Warehousing & Modeling

• Develop, load, and tune dimensional data models, star-schema facts, dimensions, stored procedures, and views (including SCD Type 1 and Type 2) in Azure Synapse Analytics (dedicated and serverless SQL pools).

• Query Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views.

• Utilize advanced loading features such as COPY INTO, PolyBase, and CETAS to build parameterized pipelines.

Governance, Security & Operations

• Apply rigorous data governance and security controls (Unity Catalog permissions, role-based access, row/column-level security, and data masking) to protect sensitive data in accordance with regulatory standards (e.g., HIPAA, PII/PHI guidelines).

• Orchestrate and monitor data workflows using Databricks Workflows/Jobs and Azure Data Factory (ADF) pipelines, managing scheduling, alerting, and failure recovery.

• Implement CI/CD best practices using Git, Azure DevOps, or GitHub for version control and artifact deployment.

Required Experience & Technical Skills

• Education & Experience:

  • Master’s degree in Data Science, Analytics, Informatics, Computer Science, Mathematics, Engineering, or a related field (with 4+ years of relevant experience); OR

  • Bachelor’s degree in a related field plus 2+ years of relevant experience; OR

  • 4+ years of hands-on experience in a dedicated data engineering role. (Note: Relevant certifications and coursework may be considered.)

• Azure Databricks: 4+ years developing with notebooks in PySpark, Spark SQL, and R (transformations, joins, aggregations, window functions); implementing ACID transactions, MERGE upserts, schema evolution, and table maintenance (OPTIMIZE, Z-order, VACUUM).

• Lakehouse & Architecture: 4+ years working with medallion architecture, Unity Catalog (workspaces, catalogs, schemas, lineage, external locations), and incremental data ingestion.

• Azure Synapse Analytics: 4+ years designing MPP warehouse tables (hash/round-robin/replicated distribution, columnstore indexes), workload management, result-set caching, and pool optimization.

• Orchestration & DevOps: Proven experience scheduling dependencies via ADF Pipelines, Databricks Workflows, and managing codebases with Git, Azure DevOps, or GitHub (branching, PRs, asset bundles).

• Security & Monitoring: Experience designing ADLS Gen2 folder/file-format structures, Entra ID access control, managed identities, Key Vault integration, and monitoring workloads using Azure Monitor/Log Analytics.

Preferred Qualifications & Certifications

• Databricks Certified Data Engineer Associate (Professional certification is a strong plus).

• Microsoft Certified: Azure Data Engineer Associate (DP-203).

• BI Integration: Experience connecting Power BI to Synapse (Import vs. DirectQuery) and optimizing model design for high-performance dashboarding.