Senior Databricks Architect/ Databricks Engineer- Contract- Remote

Rate- £700/day Outside ir35

Work from Home

Duration- 12 months minimum (with possible extensions)

This is a Senior Databricks hands on Architect who is a Databrick engineer by trade. This is a long-term contract engagement to provide dedicated data engineering depth within the Databricks and Shared IEP Services data layer, owning delivery across these workstreams and ensuring the resulting pipelines and schemas are production-grade, governed, and maintainable by the permanent team. The contractor will operate primarily within the Databricks and Shared IEP Services data layer, owning data engineering delivery across three areas:

1)     Product data ingestion — Building and maintaining daily pipelines from content and assessment products into the shared back-office data layer.

2)     AI feature enablement — Designing and maintaining the schemas and data pipelines that power LLM-based features (GenAI Goals, Bulk Upload, Evaluation Document processing), including handling of parsed, normalized, and PII-sensitive document data.

3)     IEP & Medicaid integration — Building and maintaining pipelines for IEP, Medicaid, and related service data into Databricks and other IEP systems.

Essential Criteria

1. Medallion architecture depth — designing and maintaining Bronze, Silver, and Gold schemas and pipelines using Databricks Delta Lake, including schema evolution and enforcement.

2. Production pipeline engineering — orchestrating and operationally supporting complex pipelines; idempotency, robust error handling, and reconciliation.

3. Unity Catalog governance — fine-grained access control, data lineage, and row/column-level security.

4. Cross-system data integration — handling data cleanliness, integrity, normalization, backfilling, reconciliation, and other ETL challenges across disparate source systems

5. Data privacy & compliance — anonymizing, masking, or tokenizing PII; working understanding of FERPA, COPPA, or GDPR.

6. Performance & cost optimization — Z-ordering, liquid clustering, data skipping, and shuffle-partition management to control cloud spend.

7. Senior engineering experience — 8+ years overall software engineering, including 3+ years in a senior data engineering role.

Desirable Criteria

1.     Familiarity with Databricks AI development capabilities (e.g. vector search, model serving, LLM pipelines).

2.     EdTech engineering experience.

3.     Familiarity with EdTech specifications and standards such as 1EdTech, EdFi, CEDS, or OneRoster.