Job Specification — Data Engineer x3 (Databricks/PySpark/Azure) - Ireland
Location: Limerick (preferred) OR UK based remote
Pay rate: €400-430 per day
6 month initial contract

Team: Data & Analytics — Supply Chain Reports to: Engineering Lead, within a SAFe Agile Release Train

1. About the Role
You will join a Data & Analytics team building a large-scale supply chain visibility and decision-intelligence platform. The platform ingests data from multiple enterprise source systems and curates it through a layered (medallion-style) data architecture to power interactive analytics for planners, supply chain leaders, and decision-makers.
The team delivers continuously on a 2-week sprint cadence with regular major release cycles, in a regulated environment where data accuracy and traceability are critical.
Data architecture (medallion-style): L0 (raw) ? L1 (standardized) ? L2 (curated)

2. Role Summary
We are looking for a Data Engineer to design, build, and maintain the data pipelines that feed the analytics layer. You will own data ingestion, transformation, and quality across the L0 ? L1 ? L2 layers, working in Databricks/PySpark and Azure Data Factory, ensuring curated, trustworthy data reaches the downstream applications.
You will work in an Agile squad alongside fellow data engineers and QA, contributing to a regulated environment where data accuracy and traceability are critical.

3. Key Responsibilities

Build & maintain data pipelines across the L0 ? L1 ? L2 layers using Databricks (PySpark) and Azure Data Factory (ADF).
Develop curated data models in the L2 layer that power supply chain KPIs (inventory health, demand/supply waterfalls, network health, plan attainment, risk logic, etc.).
Integrate new source systems into the platform, mapping raw feeds to standardized and curated schemas.
Implement data quality and validation controls — automated checks, reconciliation, and traceability from source to dashboard.
Write and maintain unit tests for your code (rather than relying on manual testing), contributing to automated test runs as part of the pipeline.
Optimize performance & cost of Spark jobs and data workflows — query tuning, indexing, and partitioning strategies; manage Unity Catalog governance.
Build and monitor ADF pipelines — orchestrate, schedule, and troubleshoot Azure Data Factory data flows.
Support releases on a regular major release cadence — participate in SIT/UAT, troubleshoot defects, and support hotfixes.
Collaborate in sprint ceremonies (planning, refinement, standups, retros) and document key design decisions.
Maintain CI/CD hygiene — pull requests, database migrations, code quality gates, and artifact management.

4. Required Skills & Experience
Core technical (must-have)

Area	Requirement
Data processing	Hands-on experience with Spark and Delta Lake for data processing and analysis (Databricks / PySpark)
Languages	Proficiency in Python (used for SQL-based transformations and pipeline logic)
SQL platforms	Experience with SQL Server and/or Synapse (formerly SQL Data Warehouse)
Data modeling	Solid understanding of data modeling principles and concepts
Performance tuning	Practical experience optimizing data processing, including indexing and partitioning for query performance
Orchestration	Comfortable building, monitoring, and managing Azure Data Factory (ADF) pipelines
Testing	Experience implementing unit testing practices for code (not relying solely on manual testing)
CI/CD	Understanding of CI/CD processes and their implementation
Technical depth	Strong technical depth across databases, testing, and performance tuning

Ways of working

Comfortable in Agile/Scrum (SAFe Release Train experience a plus).
Experience supporting frequent releases and hotfix cycles.
Strong collaboration with QA and analytics consumers.

5. Nice-to-Have / Differentiators

Familiarity with Common Data Layer (CDL) concepts.
Python-based Spark development and monitoring ADF pipelines at scale.
neo4j / graph databases and Redis caching.
Supply chain / manufacturing domain experience.
Experience with AI-assisted tooling (e.g., GitHub Copilot and similar tools across the SDLC).

6. Tech Stack You'll Work With

Layer	Technologies
Data / ETL	Databricks (Spark + Delta Lake, PySpark), Azure Data Factory, Azure Blob, Unity Catalog
Databases	SQL Server / Synapse, neo4j Graph DB, Redis
CI/CD & Quality	Jenkins, Bitbucket, Artifactory, Flyway, SonarQube
Testing	Unit testing, automated test runs in CI
Collaboration	Jira, Confluence

7. What Success looks like

First Sprint — Onboarded to the L0?L1?L2 data architecture; environment access set up (Databricks, source control, CI/CD, SQL Server); shadowing a squad through a sprint, first User Story delivered.
First Release — Independently delivering data-engineering stories within a sprint; writing unit tests for your pipelines; raising clean pull requests that pass code quality gates.
Success — Owning a data domain end-to-end; contributing to a major release; proposing pipeline or quality improvements.

Data Engineer (Databricks/PySpark/Azure) - Ireland

Job Description

Similar Jobs.