Job Description
Databricks Technical Architect
Location: Remote
Job Type: Full-time
Role Overvie
wWe are seeking a hands-on Databricks Technical Architect to design, build, and scale our modern data platform. This is a builder's role: you will own the architecture of our Lakehouse while remaining deeply engaged in implementation — writing PySpark, tuning Delta tables, defining Unity Catalog policies, and reviewing production code
.You will partner with data engineering, analytics, governance, and business teams to deliver a secure, performant, and cost-efficient platform that serves both analytical and AI/ML workloads
.
Key Responsibiliti
- esArchitect end-to-end data solutions on Databricks, spanning ingestion, transformation, governance, and serving layers, using the medallion (Bronze/Silver/Gold) pattern on Delta Lak
- e.Lead hands-on implementation of production data pipelines in PySpark and SQL, with Scala where appropriate. Define and enforce engineering standards through code reviews, reference implementations, and reusable framework
- s.Own the Unity Catalog design — including catalog/schema structure, access controls, row- and column-level security, data classification, lineage, and audit. Partner with the data governance function to operationalize policies at the platform laye
- r.Define the serving layer strategy: Databricks SQL warehouses, materialized views, performance tuning for BI consumers (Power BI, Tableau), and semantic modeling patterns that bridge Gold tables to business consumptio
- n.Establish platform-as-code practices using Terraform and/or Databricks Asset Bundles. Design and operate CI/CD pipelines for notebooks, jobs, and infrastructur
- e.Own cost and performance governance: cluster policies, serverless vs. classic compute decisions, Photon usage, job vs. all-purpose cluster strategy, and DBU budgeting and chargebac
- k.Design orchestration using Databricks Workflows and Lakeflow Declarative Pipelines, integrating with Airflow where cross-platform orchestration is require
- d.Partner with ML and AI teams on integration patterns for MLflow, feature engineering on Delta, and emerging Databricks AI capabilities (vector search, Mosaic, Agent Bricks). This role does not own model development but ensures the platform supports i
- t.Provide technical leadership and mentorship to a team of [N] data engineers, and serve as the senior Databricks voice in cross-functional architecture forum
s.
Required Qualificati
- ons8–12+ years in data engineering, data architecture, or related roles, with 3+ years of production Databricks experien
- ce.Strong hands-on expertise in Apache Spark with PySpark and Spark SQL. Scala is a plus but not requir
- ed.Production experience with Unity Catalog, including governance design, access control, and linea
- ge.Deep experience designing Lakehouse architectures on Delta Lake using medallion patterns, with a strong grasp of partitioning, Z-ordering, liquid clustering, and Delta optimization techniqu
- es.Implementation experience on at least one major cloud platform: AWS, Azure, or G
- CP.Infrastructure-as-code experience with Terraform and/or Databricks Asset Bundles, plus CI/CD for data engineering workflows (Git-based, with environment promotio
- n).Strong understanding of data modeling (dimensional, Data Vault, or equivalent), ETL/ELT patterns, and distributed systems interna
- ls.Demonstrated experience with cost optimization and performance tuning on Databricks at sca
- le.Working knowledge of data governance, security, and compliance principles, with the ability to translate policy into platform contro
ls.
Preferred Qualificat
- ionsDatabricks certifications: Data Engineer Professional and/or Solution Archit
- ect.Experience with Structured Streaming and event-driven architectures (Kafka, Event Hubs, Kinesis, or Pub/S
- ub).Experience with Lakeflow Connect or equivalent ingestion framewo
- rks.Familiarity with MLflow, feature stores, and ML platform integration patte
- rns.Experience designing serving layers for Power BI or Tableau at scale, including semantic model
- ing.Exposure to open table format interoperability (Delta UniForm, Iceberg) and Delta Shar
- ing.Experience in a regulated industry (financial services, healthcare, public sector) with corresponding compliance framewo
rks.