QA Engineer (AI + backend)

Engagement: We are partnering with a US-based health-tech company on the takeover of a production AI-powered mobile coaching platform. Real users, real production, real releases — not a greenfield project.

Role summary: Own quality on the two hardest surfaces of a production AI coaching platform:

the LLM agents that answer differently every run, and the API layer everything else depends on. 

As AI & Backend QA Engineer you own the quality of what the model says and what the APIs return. On the AI side that means building the evaluation practice: golden sets, quality metrics, and regression detection that catch a degraded answer before a user does. On the backend side it means API test suites and consumer-driven contract testing with Pact — agreed with the client's US engineering team as a standing requirement for this role, not an optional technique.

The defining skill is localization: when an answer is wrong, you can tell whether the problem sits in the model, in the service code, or in the input data — and you bring the evidence, not a guess.

What you'll be testing:

• AI core — a Python service with LangGraph agents producing streaming, non-deterministic coaching responses; this

• role's primary surface

• Backend APIs — a NestJS REST gateway behind Auth0, plus the services behind it

• Mobile app — React Native, shipping to iOS and Android app stores; AI features surface directly in the app that

• members use

• Data layer — MongoDB, Redis, and Postgres behind the services

• Process — Agile/Scrum, Jira for work tracking and defects, TestRail for test management

• You join the Polish QA function alongside our Manual QA Engineer and QA Automation Engineer, and work in a shared workflow with the client's US-based engineering and QA teams.

What you'll do:

• Build and run LLM evaluation. Curate datasets and golden sets, define response quality metrics, and catch quality regressions between prompt and model versions before they reach production

• Define acceptance criteria for non-deterministic output. Assertions on properties and contracts — structure, constraints, invariants — rather than exact wording

• Treat prompts as tested artifacts. Change a prompt, measure the effect of that change; probe systematically for hallucinations and prompt injection

• Own API testing. REST endpoints, JSON payloads, auth token flows, error handling — driven through Postman and Swagger/OpenAPI specs

• Implement contract testing with Pact. Consumer-driven contracts between the mobile app, the gateway, and the backend services, verified on every change

• Test with security in mind. Authorisation boundaries, token misuse, malformed input — the API-level abuse cases nobody writes a happy-path test for

• Localise failures across the stack. Logs, traces, and database queries to pin a defect to the model, the service code, or the input data

• Help wire evaluations into CI/CD. Evaluation runs in the pipeline and quality gates on pull requests, so a bad prompt change cannot merge quietly

• Keep the record straight. Results mapped to test cases in TestRail, defects logged and driven to closure in Jira, findings written up so the team can act on them

Must-have skills:

• 3+ years in technical QA or test automation, including API testing on production systems

• Python — writing and maintaining test code, pytest, fixtures

• Strong HTTP and API fundamentals — REST, JSON, status codes, headers, auth tokens

• Contract testing with Pact — consumer-driven contracts in practice; this is a hard requirement for the role, agreed with the client's engineering team

• Security-aware API testing — authorisation boundaries, token handling, malformed and hostile input

• Comfort with non-determinism — defining acceptance criteria when there is no single correct output: assertions on properties and contracts rather than exact wording

• LLM evaluation — datasets and golden sets, response quality metrics, catching quality regressions between prompt and model versions

• Prompt engineering — the ability to change a prompt and measure the effect of that change; awareness of hallucinations and prompt injection

• Backend savvy — HTTP and REST, JSON, tokens, logs, database queries; you can tell whether a problem sits in the model, in the service code, or in the input data

• Postman and Swagger/OpenAPI as daily tools

• Git in a team workflow — branches, pull requests, reviews

• TestRail for test-case management and run tracking

• Mobile product context (Android/iOS) — comfortable testing features that ship in a mobile app, including apprelease timing and store builds

• Strong English — you work in a shared workflow with US colleagues

• Must work hours that overlap US Mountain Time afternoons (~14:00–18:00 PL time)

Nice-to-haves:

• Langfuse or equivalent (LangSmith, Phoenix) — working with traces and attaching evaluations to them

• LangGraph — testing an agent as a state graph, both a single node and the full flow

• CI/CD — running evaluations in a pipeline and quality gates on pull requests

• Python with Appium — mobile automation exposure alongside the API work

• Testing other AI or LLM-driven features in production

• Experience taking over quality ownership of someone else's product

Offer:

• Location: 100% remote

• Engagement: B2B contract

• Duration: Initial 3-month contract, expected to extend

• Rate: upt to 160 PLN/h

• Start: asap