Data Engineer

Our client is looking for 1-2 Senior and selfgoing Data Engineers to help them migrate their finance and HR data domains. The assignment can be done remote, but there is a office in Sweden/Stockholm if you would like to visit.

The assignment:

We have a set of ETL jobs loading and transforming financial data using ETL tool Matillion and Snowflake as the database. The source is in Azure data lake (backed by dynamics). The goal of the project is to understand how the current system is running, migrate it to a combination of Airflow+dbt+Snowflake jobs while using a data mesh/data products strategy. The migration will not be 1:1 but rather an improvement of the current implementation e.g., currently Row Level Security is used to manage access to sensitive data. The new implementation could possibly use role level access control instead of RLS.

Some of the task would be

• Understand and document current pipelines written in Matillion

• Write ingestion pipeline using Airflow dag

• Simplify and convert the mappings to dbt models (some parts may run in airflow if they have python code snippets)

• Write dbt docs to document all models

• Write unit tests in dbt for data tests

• Help with reconciliation process between the current reports and new reports from the newly developed system

In all of these tasks, the most complex part would be to understand and document the existing implementation as currently there is no documentation and we would need to read the pipelines manually.

Start: Asap
Workplace: Sweden or remote from Sweden.

Length: 3-6 months