Senior Data Engineer

We are a team of tech enthusiasts on a mission to bring together the best minds in IT services and analytics. Our goal? To build cutting-edge IT and Analytical Hubs that empower our partners to become truly data-driven organizations.

For our client’s Data Management & Data Foundation area, we are looking for an exceptional Senior Data Engineer – a highly experienced, hands-on engineer who can independently take ownership of complex technical challenges and turn them into robust, scalable solutions.

This is not a maintenance-focused or business-as-usual role. You will join a high-seniority engineering group working on strategic and technically challenging initiatives, moving between projects depending on priorities and where the strongest engineering expertise is needed.

The wider Data Management organization consists of approximately 30 engineers and technology specialists, while this role will focus primarily on a selected set of high-impact initiatives.

Your role

As a Senior Data Engineer, you will:

• Design, develop, and optimize complex data solutions and data pipelines on Azure and Databricks.

• Take end-to-end technical ownership of assigned engineering problems – from understanding the challenge through solution design, implementation, testing, and production deployment.

• Write high-quality, production-grade code using Python, PySpark, and SQL.

• Work extensively with Azure Databricks and modern capabilities of the Databricks ecosystem.

• Explore and implement AI-enabled capabilities within the data platform, including solutions involving Databricks AI features, RAG-based approaches, data discovery, and conversational access to enterprise data.

• Contribute to the migration and modernization of existing data platforms and solutions into Databricks-based architectures.

• Work with Unity Catalog and contribute to scalable approaches to data discovery, governance, and access.

• Solve technically difficult or non-standard problems where there may not be an existing blueprint or ready-made solution.

• Proactively identify opportunities to improve architecture, performance, scalability, reliability, and maintainability.

• Challenge existing approaches when you see a better technical solution and actively contribute your own ideas.

• Collaborate with Data Engineers, Architects, Product Owners, DataOps, QA, DevOps, and other technology teams.

• Support architectural discussions and contribute to technical design decisions across the Data Management landscape.

• Move between strategic initiatives as priorities evolve, bringing strong engineering expertise wherever it is most needed.

This is a role for someone who enjoys building things, solving difficult engineering problems, and having a real technical impact rather than simply executing predefined tasks.

What we are looking for

Must-have skills

• 8+ years of professional experience in Data Engineering, Software Engineering, or a closely related field.

• Strong, hands-on commercial experience with Azure Databricks.

• Very good knowledge of the Azure cloud ecosystem and cloud-based data architectures.

• Excellent programming skills in Python – we are looking for a strong coder, not someone who only uses Python for basic notebook development.

• Advanced PySpark skills and experience working with large-scale data processing workloads.

• Advanced SQL, including complex transformations, optimization, and analytical/window functions.

• Strong understanding of data engineering principles, ETL/ELT patterns, data pipelines, distributed processing, and production data platforms.

• Experience designing and delivering solutions independently, with limited supervision.

• Ability to understand an unfamiliar technical problem, propose an approach, and drive it through to implementation.

• Strong troubleshooting and problem-solving skills.

• Experience working with production-grade engineering practices, including version control, testing, CI/CD, monitoring, and deployment processes.

• Strong understanding of software development principles and the ability to produce clean, maintainable, well-structured code.

• High level of ownership, independence, and proactivity.

• Excellent English communication skills, both written and spoken.

Particularly valuable

We will be especially interested in candidates who also bring:

• Hands-on experience with AI capabilities within Databricks and modern AI-enabled data solutions.

• Experience with RAG architectures, LLM-based applications, or conversational interfaces built on enterprise data.

• Experience with Databricks Genie or similar conversational data discovery capabilities.

• Strong knowledge of Unity Catalog.

• Experience migrating enterprise data platforms or data management solutions to Databricks.

• Understanding of data cataloguing, metadata management, lineage, governance, and data discovery.

• Strong awareness of solution and data architecture and the ability to contribute meaningfully to architectural decisions.

• Experience designing reusable frameworks, components, or engineering standards rather than only individual pipelines.

• Performance optimization experience in Databricks / Spark environments.

• Experience with Azure DevOps and mature CI/CD practices for data solutions.

Architectural experience is not a prerequisite, but a Senior Data Engineer who combines excellent coding skills with strong architectural awareness would be an especially good fit.

What will you work on?

You will contribute to several strategic projects within the Data Management domain, including:

Data platform migration to Databricks

Migration and modernization of existing data management capabilities into the Databricks ecosystem, including engineering support, platform integration, and implementation of new Databricks-based solutions.

AI-powered data discovery solution

Development of a new approach to enterprise data discovery.

The solution combines capabilities such as Databricks, Unity Catalog, AI-powered data discovery, RAG, and conversational interfaces to create a chat-based experience that helps business units discover and understand available data.

Instead of relying on numerous manual discovery sessions with individual operating companies, the aim is to enable users to explore datasets, metadata, and available data products through an intelligent conversational interface.

Data Foundation

Development and evolution of the organization’s Data Foundation in Databricks, creating scalable technical foundations that can be reused across teams, markets, and data products.

Who will succeed in this role?

You are likely to enjoy this position if you:

• are a strong engineer first and foremost;

• like coding and still want coding to be a significant part of your daily work;

• are comfortable being given a difficult problem rather than a detailed list of implementation steps;

• can work independently and make sound technical decisions;

• naturally look for improvements instead of accepting existing solutions as fixed;

• enjoy exploring new technologies and translating them into practical enterprise solutions;

• are comfortable moving between different initiatives and technical contexts;

• can discuss architecture with senior engineers and architects while remaining deeply hands-on;

• prefer an environment where technical competence, ownership, and initiative matter more than job titles.

What we offer

• Opportunity to work on high-impact, strategic data initiatives rather than repetitive business-as-usual development.

• Significant technical ownership and freedom in how problems are solved.

• Hands-on work with Azure, Databricks, Python, PySpark, SQL, and emerging AI capabilities.

• Exposure to modern applications of AI and RAG in enterprise data management and discovery.

• Opportunity to influence technical architecture and engineering practices.

• Collaboration with experienced engineers and architects across an international technology organization.

Recruitment process

The recruitment process is straightforward and focused on both mutual fit and technical depth:

• Initial HR interview – a conversation about your experience, motivation, expectations, and the role itself.

• 60-minute technical interview with two senior members of the engineering team.

• Depending on the situation, an additional conversation with a senior stakeholder may be included.

We are looking for someone who can operate at a genuinely senior level – a strong coder, an independent problem solver, and an engineer who brings ideas rather than waits for instructions.

If this sounds like the kind of engineering environment in which you do your best work, we’d like to hear from you.

Please add to your CV the following clause:

"I hereby agree to the processing of my personal data included in my job offer by hubQuest spółka z ograniczoną odpowiedzialnością located in Warsaw for the purpose of the current recruitment process.”

If you want to be considered in the future recruitment processes please add the following statement:

"I also agree to the processing of my personal data for the purpose of future recruitment processes.”