Data Engineer

Data Engineer

£600pd Inside IR35

2-3 days on site in London

Initial 6 month engagement

emagine is a high-end professional services consultancy and solutions firm specialising in providing business and technology services to the financial services sector, we power progress, solve challenges and deliver real results through tailored high-end consulting services and solutions.

We have created a culture of openness and integrity by building genuine and strong relationships and partnerships, enabling us to be uncompromising in our dedication in delivering the optimal service for our clients. Our commitment is not just towards our clients but we aim to foster a positive and equitable working environment with our consultants and colleagues which stems from our core values: Confident, Dedicated, Responsible, Genuine.

2 Databricks data Engineers: We are seeking 2 Senior Databricks Data Engineers to design, build, and scale a modern Lakehouse platform using Spec-Driven Development (SDD), Apache Airflow, and dbt. In this role, you will translate architectural requirements into reliable production solutions, ensuring data contracts, transformations, and orchestration processes are clearly defined, thoroughly tested, and fully automated.

Key Responsibilities

• Design and develop scalable Databricks data pipelines based on the Medallion Architecture (Bronze, Silver, and Gold layers) using Delta Lake.

• Lead the adoption of Spec-Driven Development (SDD) by creating machine-readable specifications and data contracts before implementation.

• Build, monitor, and optimize workflow orchestration pipelines using Apache Airflow.

• Develop, test, and maintain modular data transformation models with dbt, integrated with the Databricks SQL and Spark ecosystem.

• Optimize PySpark workloads, manage cluster configurations, and control compute and storage costs.

• Implement data governance, granular access controls, and data lineage tracking through Unity Catalog.

Qualifications

• 5+ years of hands-on data engineering experience, including extensive production experience with Databricks.

• Strong expertise in Apache Airflow for workflow scheduling, dependency management, and error handling.

• Advanced knowledge of dbt modeling, testing practices, and macro development.

• Practical experience with Spec-Driven Development (SDD) and metadata-driven data platform design.

• Expert-level proficiency in Python (PySpark) and SQL.

• Familiarity with Git, CI/CD processes, and infrastructure-as-code practices.

Multi-Cloud Experience (AWS, Azure, GCP)

• AWS: Configure and manage integrations with Amazon S3, IAM roles, and VPCs to support secure and scalable data platforms.

• Azure: Implement secure data storage using Azure Data Lake Storage Gen2 (ADLS Gen2), integrate with Microsoft Entra ID, and deliver data to analytics platforms such as Power BI.

• GCP: Build and operate data workflows using Google Cloud Storage (GCS) while enabling integration with BigQuery and Vertex AI for downstream analytics and AI use cases.

Our People

The ideal consultants will share our values and be aligned with our ways of working and as your career progresses, you can expect to work across all areas of the project lifecycle, from strategy to implementation. This will provide you with a broad base of experience from which to build an outstanding career.

The ideal consultants will share our values and be aligned with our ways of working and as your career progresses, you can expect to work across all areas of the project lifecycle.

We pride ourselves on;

• Providing our people with a supportive culture, rooted in our values and driven by our purpose.

• Promoting a culture of inclusion, collaboration, well-being, and learning and development.

• Providing increased agility and flexibility within our hybrid working model

• Investing in employees’ growth through ongoing training and development

• Autonomy to take ownership of projects, making decisions and demonstrating individual expertise

• Providing an transparent performance and career management experience.

Our consultants are integral to delivering successful consulting engagements, addressing our clients’ most pressing business challenges, and build lasting value in disciplines such as:

• Solve sophisticated, ambiguous business, change and technology problems, bringing structure and meticulous analysis and planning, acting, and taking decisions with little strategic direction

• Build, develop and sustain trusted senior client relationships in the C-suite by remaining highly attuned to client needs

• Drive, enable and support the business, partnering with our leaders, clients, and consultants across our practices to take the best of emagine to our clients through opportunity identification/qualification, solution development/presentation

Interested?

At emagine, we are committed to building an international and diverse team by embracing our different backgrounds.

If you are up to the challenge and would like to find out more, get in touch with us immediately, our internal recruitment team is always keen to hear from dynamic individuals that are looking to further their career and explore their full potential. 

“emagine is an equal opportunity employer, and employment practices are based strictly on merit. It is the policy of the Company to give equal opportunity in employment regardless of sex, sexual orientation, marital status, race, age, disability, gender reassignment, pregnancy and maternity, religion or ethnic origin”