Senior Data Engineer- SC Cleared

Rate- £500-600 per day Inside ir35 contract,

Work from home

Must be currently SC Cleared

Our client is a global software enterprise looking for a Senior Data Engineer (Databricks + Talend) for their Data Engineering team. Should have experience in designing, building, and maintaining data pipelines and infrastructure within the Databricks environment.

Responsibilities: -

• Work on standards principles and best practices- design, develop, and maintain scalable and robust data pipelines on Databricks.
• Improve performance, fix broken pipelines- Optimize and troubleshoot existing data pipelines for performance and reliability.
• Review the current Databricks architecture
• Redesign the Databricks architecture
• Collaborate with data scientists and analysts to understand data requirements and deliver solutions.
• Ensure data quality and integrity across various data sources.
• Implement data security and compliance best practices.
• Monitor data pipeline performance and conduct necessary maintenance and updates.
• Document data pipeline processes and technical specifications.

Required skillsets: -

• Proficiency with Databricks and Spark.
• Strong SQL skills and experience with relational databases.
• Data Modelling, Hive, SQL, Talend ETL
• Experience with big data technologies (e.g., Hadoop, Kafka).
• Knowledge of data warehousing concepts and ETL processes.
• Excellent problem-solving and analytical skills.
• Should have both Waterfall and Agile background.
• Experience in reviewing user guide, technical guide, ETL, and Data Model documents
• Define end-to-end workflow