Databricks Platform Engineering Lead is required by a global IT software company to lead data engineering, platform optimization, and AI enablement across the data function.

You will be responsible for:

• Leading a team of data engineers building, optimizing, and maintaining scalable data pipelines and platform operations on Databricks.
• Driving the adoption of Databricks technologies across the company.
• leading Databricks operations ensuring reliability, scalability, and cost optimization.
• Redesigning existing data engineering pipelines to increase their performance and scalability.
• Introducing AI/MLOps practices to manage the lifecycle of data and AI-driven products.
• Establishing data engineering platform monitoring framework.

Required experience and skills:

• Deep expertise in Databricks including Delta Lake, Unity Catalog, MLflow, Databricks SQL, DBX tooling.
• Apache Spark.
• Python engineering and ability to perform unit testing, build CI/CD pipelines, and utilise AI-assisted development.
• MLOps lifecycle.
• Team leading and mentoring experience.
• Any experience in AI-assisted data engineering (prompt engineering, using LLMs for code generation & optimization, AI output monitoring etc..) will be beneficial.
• Streaming data platforms (Kinesis, Kafka or similar) will be an advantage.