Data Engineer

Department: Digital Factory

Role Overview

The Data Engineer is responsible for designing, building, and operating scalable, reliable, and secure data pipelines and data platforms that power digital products, analytics use cases, reporting, and data-driven services. Acting as a key contributor within cross-functional delivery teams, the role collaborates closely with Product, Backend, Frontend, Architecture, Data Analytics, and platform teams to deliver trusted data assets across ingestion, transformation, storage, and access layers.

At Supervising Associate level, the role leads complex data engineering delivery, sets technical direction and strengthens team capability across initiatives.

Key Responsibilities

Lead technical delivery across complex data workstreams:

• Make design decisions and resolve cross-platform dependencies, governance risks, and escalations.
• Mentor engineers, enforce engineering and data standards, and drive improvements in architecture, quality, lineage, security, observability, cost, and technical debt management.

Data Pipeline & Platform Delivery

• Design, implement, and maintain batch and streaming data pipelines using Python and modern data engineering frameworks.
• Build reliable ingestion and transformation processes from internal and external data sources, including APIs, databases, files, and events.
• Develop reusable data processing components and frameworks following software engineering best practices.

Data Quality, Reliability & Observability

• Implement data validation, quality checks, and monitoring to ensure accuracy, completeness, timeliness, and trustworthiness of data.
• Write clean, testable code with unit and integration tests for data pipelines and processing components.
• Instrument pipelines with logging, metrics, alerts, and operational monitoring.

Performance, Scalability & Architecture

• Optimize data processing performance, storage layouts, query efficiency, and platform resource usage.
• Design pipelines for scalability and resilience using retries, idempotency, checkpointing, and backpressure patterns.
• Apply partitioning, caching, and parallelization strategies where appropriate.

Security, Governance & Continuous Improvement

• Implement secure data handling practices, including access controls, encryption, secrets management, and compliant data processing.
• Ensure alignment with enterprise, regulatory, privacy, data classification, retention, and lineage requirements.
• Support audits and data governance processes where required.

What success in this role will look like

• Demonstrates successful leadership of complex data delivery, stronger engineering capability and consistent adoption of platform, quality and governance standards.
• Reliable delivery and operation of scalable data pipelines and data platform components.
• Trusted data assets with strong accuracy, completeness, timeliness, and observability controls.
• Clean, testable, documented, and maintainable Python and SQL-based data engineering solutions.
• Effective support for incidents, troubleshooting, root-cause analysis, and durable remediation.
• Provide mentorship and guidance to less experienced team members.

Qualifications & Experience

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, Data Engineering, Information Technology, or a related field.
• Professional experience building data pipelines and data platforms using Python.
• Strong experience with SQL and data modeling concepts, including dimensional, normalized, analytical, or lakehouse patterns.
• Hands-on experience with relational and analytical data stores, such as PostgreSQL, SQL Server, data warehouses, or data lakes.
• Experience with CI/CD, automated testing, and code reviews for data workloads.
• Expertise on cloud platforms, with Azure/Fabric, and relevant data services.
• Work effectively as part of a team, contributing to collective success while remaining open to feedback and coaching from more experienced colleagues.
• Strong understanding of data security, access control, observability, and production support practices.
• 7+ Years of Experience are required.

Preferred Experience

• Experience with modern data frameworks and tools such as Spark, Airflow, dbt, Azure Data Factory, Microsoft Fabric Databricks, or equivalent platforms.
• Hands-on experience with streaming and event-based data processing using Kafka, Azure Event Hubs, Service Bus, or similar technologies.
• Experience with lakehouse or warehouse platforms.