Overview
We are seeking a Lead Data Engineer to design, build and lead the delivery of robust, secure and scalable data platforms supporting analytics, reporting and regulatory needs. You will set engineering standards, mentor engineers, and partner with stakeholders to deliver trusted data products across the organisation.
Key Responsibilities
- Lead the end-to-end design and implementation of data pipelines (batch and streaming), data warehouses/lakes, and curated datasets.
- Define architecture patterns, coding standards, CI/CD practices and data engineering ways of working.
- Optimise performance, cost and reliability of data platforms; implement monitoring, alerting and SLAs.
- Champion data quality, lineage, cataloguing and documentation; embed automated testing and validation.
- Collaborate with product, analytics, risk and compliance teams to translate requirements into scalable solutions.
- Ensure solutions meet Financial Services controls including governance, auditability, retention and privacy.
- Lead incident response and root-cause analysis; drive continuous improvement and technical debt reduction.
Required Skills & Experience
- Extensive experience in data engineering, with proven leadership of teams and complex deliveries.
- Strong SQL and programming skills (Python and/or Scala/Java); clean, maintainable engineering practices.
- Hands-on expertise with cloud data services (AWS, Azure or GCP), including security and networking fundamentals.
- Experience with modern data platforms and tools such as Spark, Kafka/Kinesis/Event Hubs, dbt, Airflow, Databricks, Snowflake/BigQuery/Redshift/Synapse (or equivalent).
- Proficiency in data modelling (dimensional and/or Data Vault), partitioning strategies and performance tuning.
- CI/CD and Infrastructure as Code (e.g., Git, Terraform/CloudFormation), containerisation (Docker) and orchestration principles.
- Strong understanding of data governance, access controls, encryption, key management and secrets handling.
- Experience operating in regulated environments (e.g., FCA/PRA), including audit and controls evidence.
- Excellent stakeholder management and ability to communicate complex concepts clearly.
Nice to Have
- Experience with GDPR, data retention policies and privacy-by-design approaches.
- Knowledge of observability tooling and data reliability engineering practices.
- Experience enabling ML/AI workloads and feature stores.
#LI-VS1