AI Quality Engineer

Technologies we use

Expected

• Python

Optional

• Git
• SQL
• JavaScript

Your responsibilities

• Translating business expectations, user outcomes and operational risks into measurable evaluation criteria.
• Defining what successful, partially successful and unsuccessful system behaviour looks like, including the gradient between clear success and clear failure.
• Designing evaluation approaches for conversational, multi-turn and non-deterministic AI systems.
• Building and maintaining representative evaluation datasets covering common, rare, boundary and high-risk scenarios.
• Assessing dataset quality, including coverage, class balance, duplicates, data leakage, label quality, synthetic-versus-production differences and dataset freshness.
• Selecting appropriate manual, deterministic, model-based and statistical evaluation methods.
• Analysing errors by category, user journey, risk level and likely root cause.
• Distinguishing model failures from failures in retrieval, chunking, ranking, prompts, tools, guardrails, integrations, source data and evaluators.
• Validating automated evaluators, including LLM-as-a-Judge, against human judgement.
• Detecting blind spots, noise, bias and misleading aggregation in evaluation results.
• Creating baselines, controlled comparisons and regression tests that allow meaningful comparison between system versions.
• Prioritising tests according to the probability and impact of failure rather than coverage alone.
• Communicating results, uncertainty, limitations and residual risks in a form that supports release and product decisions.
• Proactively identifying emerging failure modes based on system changes, production observations and research.

Our requirements

• Professional experience in quality engineering, software testing, test automation, data science, software engineering or a related field (we can consider different seniority levels).
• Experience or strong interest in working with AI-powered solutions, either through developing, testing, evaluating or supporting them.
• Good Python proficiency and experience writing clean, maintainable code for analysis, automation or testing purposes.
• Interest in understanding how AI systems behave, how reliable they are, and how their quality can be measured.
• Strong analytical and problem-solving skills, with the ability to investigate issues and identify root causes.
• Ability to translate business objectives, customer outcomes and risks into practical evaluation approaches.
• Strong communication skills and the ability to explain findings to both technical and non-technical stakeholders.
• A curious, proactive mindset and a genuine interest in AI, emerging technologies and system robustness.

Optional

• Experience evaluating, testing or monitoring AI-powered systems in production environments.
• Working with structured data, especially JSON, logs and tabular evaluation results.
• Understanding of LLM behaviour, tokenisation, context limits, parameters and non-determinism.
• Understanding of multi-turn context, RAG, grounding, faithfulness, hallucinations, guardrails, attacks (like prompt injection etc.) and tool/function calling.
• Software engineering practices: Git, modular code, configuration management and automated tests.

What we offer

• Contract of employment
• Performance Bonus
• Private medical care (ENEL-MED)
• Cafeteria system of benefits (e.g. MultiSport card)
• Additional parent privileges:
• baby leave – 6 weeks leave for second parent
• shortened working time after parental leave
• Employer-funded group insurance
• Employee Shares Plan
• Extra day off for birthday
• Paid time off to volunteer
• Additional leave acquired over the years in the company (even 5 days more in a year)
• Wellbeing Leave - additional two weeks paid leave after 10 years of service
• Psychological support consultations
• Access to the e-learning platforms
• Internal employee referral program
• Hybrid working model from Poland with a few days in Warsaw office per month
• Attractive office location in Warsaw - Gdański Business Center
• Additional entitlements for people with disabilities - Work Without Barriers
• Work environment that promotes diversity - Diversity Charter

Benefits

• sharing the costs of sports activities
• private medical care
• sharing the costs of professional training & courses
• life insurance
• remote work opportunities
• flexible working time
• saving & investment scheme
• birthday celebration
• employee referral program
• extra leave

Hi, we're glad you're here! We are hiring to our Aviva Services Excellence Centre! Take a look at our job description - maybe it suits you or one of your friends?

Aviva Services Excellence Centre

We help our 18 million customers to save for the future and manage the risks of everyday life. To give these customers the best possible products and service we know we must make Aviva the most attractive choice for talented, entrepreneurial people with diverse backgrounds and an evolving range of expertise and insight.

So, we’re passionate about helping our 22,000 people to do the best work of their lives, to enable them to make a positive difference to the lives of our customers.

This is how we work