Data Engineer

Project Description

The Data Engineer will join a data-infrastructure team responsible for a scalable, robust, self-service analytics platform.

Responsibilities:

• Maintaining an ingestion pipeline processing billions of events daily against a defined SLA

• Supporting teams that build and optimize complex data pipelines

• Identifying platform problems with other engineering teams

• Developing tools and frameworks for the data platform

• Supporting data scientists and analysts with production workloads

• Contributing to ML-platform and streaming initiatives

• Establishing software-engineering and data-development practices

Requirments:

Must have:

• 3+ years of experience in data engineering

• Experience working with an orchestration tool (e.g., Airflow)

• Experience working with a large data set

• Experience building complex ETL pipelines

• Experience working with cloud providers such as GCP, AWS, or Azure

• Strong programming skills in Spark with Scala and Python

• Experience working with CI/CD tools like Jenkins and Git

• Strong understanding of Software Engineering practices and principles

• Excellent problem-solving and communication skills

• Self-motivated and have a proven ability to take the initiative to own the problems that come up and solve them

Nice to have:

• Preferred experience with Kubernetes

• Kafka or another messaging system, streaming platforms

• BigQuery or Redshift data-warehouse administration.