Job Summary:
We are seeking an experienced and highly motivated Lead Data Engineer to design, develop, and implement scalable data products and solutions utilizing the Databricks Data Intelligence Platform and Microsoft Azure. This role is ideal for a hands-on technical individual who combines deep expertise in data engineering, dimensional modeling, platform development, and modern data engineering practices and agentic code generation with a passion for emerging capabilities within the Databricks ecosystem.
The successful candidate will be responsible for developing reusable engineering frameworks, implementing enterprise data solutions, mentoring engineering teams, and driving adoption of modern Databricks technologies including Azure Data Factory, Lakeflow, Lakebase, Agentbricks, Genie, Genie Spaces, Genie Code, Databricks Apps, and AI Agent Frameworks. Working within established enterprise architecture standards, this individual will play a critical role in delivering scalable Lakehouse solutions that support analytics, reporting, artificial intelligence, machine learning, and agentic applications across the organization. The Lead Data Engineer will collaborate closely with Product Managers, Project Managers, Data Engineers, Analytics teams, and business stakeholders to build modern, scalable, secure, and high-quality data products that drive business value.
Required Qualifications
- Bachelor’s degree in computer science, Information Systems, Engineering, Data Analytics, or a related field, or equivalent professional experience.
- 7+ years of experience in Data Engineering, Data Platform Engineering and Data Architecture.
- Demonstrated experience leading technical architecture and solution design initiatives.
- Expert-level knowledge of:
- Databricks
- Delta Lake
- Apache Spark
- Spark SQL
- SQL
- Python / PySpark
- Azure Data Factory
- Strong expertise in dimensional modeling, data warehouse design, and semantic modeling.
- Experience implementing enterprise Lakehouse and data platform architectures.
- Hands-on experience with:
- Unity Catalog
- Delta Live Tables (DLT)
- Auto Loader
- Databricks Workflows
- DBT
- Lakeflow Pipelines
- Experience implementing CI/CD pipelines and DevOps practices using Azure DevOps, Git, and Infrastructure as Code.
- Strong understanding of data quality, governance, observability, monitoring, and operational support.
- Excellent analytical, problem-solving, communication, and stakeholder management skills.
- Proven experience mentoring engineers and leading technical delivery teams.
Preferred Qualifications
- At least 7-10+ years related experience
- Experience implementing AI-driven and agentic solutions within Databricks.
- Experience with Databricks Mosaic AI, Vector Search, Agent Bricks, Genie, and Databricks Apps.
- Experience building Retrieval-Augmented Generation (RAG) and semantic retrieval solutions.
- Knowledge of Azure Synapse Analytics and modern cloud-native data architectures.
- Experience supporting Power BI semantic models and enterprise reporting environments.
- Azure and Databricks certifications a plus.
- Experience designing reusable engineering frameworks and platform accelerators.
- Experience developing AI-assisted development tooling, reusable skills, or code generation capabilities.
Success Profile
The ideal candidate is a influential technical contributor who can effectively balance architecture, engineering, mentorship, and architecture. They possess a deep understanding of enterprise data architecture and dimensional modeling while remaining actively engaged in solution development. This individual thrives in a fast-paced environment, drives engineering excellence, embraces innovation, and enables the organization to leverage modern Databricks, AI, and Lakehouse technologies to deliver scalable, governed, and business-focused data products