Senior Data Engineer

Key Responsibilities: 

• Data Management: Extract, combine, transform, load, manipulate, and cleanse data from various sources to ensure high-quality datasets for analysis. This also includes data migration from legacy systems into next generation systems

• Collaboration: Work closely with data modelers and business analysts to understand the data model and source-to-staging data mapping. 

• Data Structures: Create and integrate data structures suitable for analysis, including developing DDLs to create various MS Cloud SQL Server tables and views in Pond (a subset of the Data Lake) DB.

• Databricks Development: Design and develop Databricks notebooks for extracting data from various input files, joining related entities to create Shared File Formats (SFFs). 

• Ingestion Framework: Configure the Ingestion Framework (IF) tables to utilize the existing framework for loading staging and Data Lake Core (DLC) tables. 

• Algorithm Development: Develop and deploy algorithms in performance-efficient code across analytical and data platforms. 

• Data Pipelines: Build large-scale batch and real-time data pipelines using data processing frameworks in cloud platforms or on-premises environments. 

• Continuous Integration: Leverage best practices in continuous integration and delivery to enhance data workflows. 

• Data Governance: Track data consumption patterns, collaborating with data governance and information stewards, and participate in vetting and promoting content for the curated data catalog. 

• Automation: Recommend and implement automation in existing and future integration flows to improve efficiency. 

• Data Quality Improvement: Drive advancements in data acquisition, testing, and tooling to enhance data quality. 

• Requirement Analysis: Play a key role in requirement analysis and project estimates to ensure alignment with business goals. 

• Performance Monitoring: Monitor performance and optimize SQL queries for maximum efficiency. 

• Documentation: Prepare detailed design documents with prototypes and screenshots based on Business Requirement Documents (BRD) and System Requirement Documents. 

• Cloud Tools Utilization: Work with various cloud data tools such as Azure SQL, Azure Data Factory, and Azure Databricks to ingest and prepare data for analysis as per business requirements. 

Qualifications: 

• Education: Bachelor’s degree in Computer Science, Engineering, or a related field; Master’s degree preferred. 

• Experience: 5+ years of experience in data engineering or a related field, with a strong focus on ETL processes and cloud data tools. 

• Certifications required:  Certified Azure Data Engineer, Databricks Certified Data Engineer 

  Technical Skills: 

• Proficiency in programming languages such as Python, Java, or Scala. 

• Experience with MS Cloud SQL Server, Azure Data Factory, and Azure Databricks, PySpark, Spark SQL, ADLS, SQL, PowerBI, SSIS, SSRS, Agile and Scrum 

• Strong knowledge of SQL and experience with relational databases. 

• Familiarity with big data technologies and data processing frameworks.