Senior Data Engineer

Senior Data Engineer - Databricks - £700pd (Inside IR35) - London/Hybrid

We are working with a leading global commodities trading business that is looking for a Senior Data Engineer on a contract basis.

This is a hands-on role focused on building and expanding a global data and analytics platform used across trading. You’ll work closely with data scientists, ML engineers, quants and business stakeholders to design scalable data solutions and support analytics and ML/AI use cases.

You’ll be responsible for:

• Designing, building and maintaining scalable data pipelines and Delta Lake architectures in Databricks on AWS
• Developing and enhancing Front Office data warehouse solutions with a focus on performance, reliability and data quality
• Preparing ML-ready datasets and supporting production-grade ML/AI pipelines
• Implementing CI/CD pipelines, testing frameworks and observability for data engineering workflows
• Contributing to MLOps practices including model tracking, deployment and monitoring
• Working closely with data scientists, quants and engineering teams on data modelling and platform design
• Ensuring strong standards around data governance, security and performance

What We’re Looking For

• Expert-level experience with Databricks, Apache Spark and PySpark
• Strong knowledge of Delta Lake, Unity Catalog, MLflow and Databricks Workflows
• Very strong Python and SQL experience
• Strong AWS knowledge across services such as S3, Lambda, Glue, IAM, EC2 and EKS
• Experience using Terraform for Infrastructure as Code
• Strong understanding of CI/CD, testing, deployment automation and monitoring within data engineering environments

Previous experience in commodities, trading or financial markets is useful but not essential.

This is a hybrid role, with 3 days per week required in their London offices. It is an initial 6 month contract, with a high likelihood of multiple extensions.

If this sounds interesting, don't hesitate to apply with your latest CV attached for immediate consideration!