Senior Data Engineer – Scala / Spark / Hadoop (f/m)

We are looking for an experienced Data Engineer to join an international team working on a strategic regulatory initiative within the financial sector. You will develop and maintain a data platform and reporting processes operating in a complex Big Data environment.

Responsibilities:

• Design, develop and maintain Data Engineering solutions using Hadoop and Apache Spark.

• Build scalable solutions for processing large volumes of data.

• Analyse complex business requirements and translate them into effective technical solutions.

• Develop and optimise SQL queries and data-processing workflows.

• Build and enhance data pipelines and reporting processes.

• Create and maintain workflows using Apache Oozie and Apache Airflow.

• Develop data solutions using Scala.

• Work with event-driven architecture based on Apache Kafka.

• Maintain and improve CI/CD processes using Git, pull requests and Jenkins.

• Provide L3 support for production environments.

• Investigate production incidents and technical problems in cooperation with relevant operational teams.

• Collaborate with business stakeholders to define requirements and analyse issues.

• Ensure the quality, performance, security and reliability of data solutions.

Requirements:

• Strong hands-on experience with Scala.

• Commercial experience with Apache Hadoop.

• Practical knowledge of Apache Spark and large-scale data processing.

• Experience building and maintaining Data Engineering pipelines.

• Experience with Apache Kafka and event-driven architecture.

• Practical knowledge of CI/CD processes, particularly Jenkins.

• Very good knowledge of Git, code review and pull request processes.

• Strong SQL development and optimisation skills.

• Experience supporting production environments.

• Ability to independently investigate complex technical issues.

• English proficiency sufficient for daily work in an international team.

• Availability to work from the Warsaw or Gdańsk office twice per week.

• Ability to perform the work from Poland.

Nice to have:

• Kubernetes experience.

• Knowledge of Apache Flink.

• Experience with Apache Airflow or Apache Oozie.

• Previous involvement in regulatory or reporting projects.

• Experience in the financial sector.

What we offer:

• Participation in a strategic regulatory initiative with a tangible impact on the financial sector.

• Work within a large and advanced Big Data environment.

• A modern technology stack including Scala, Hadoop, Spark, Kafka, Airflow and Jenkins.

• A high level of independence and influence over technical solutions.

• Cooperation with an experienced international team.

• Stable, long-term cooperation.

• A professional environment focused on quality, security and expert development.

• An opportunity to gain experience in financial regulations and regulatory reporting.