QA Engineer (AI + backend)

Apreel is partnering with a US-based health-tech company to manage the quality of a production AI-powered mobile coaching platform. We are looking for a QA Engineer to take ownership of the most critical components of the system: the LLM agents and the backend API layer.

Key responsibilities

• Build and run LLM evaluation practices, including curating golden sets and defining quality metrics to catch regressions.
• Define acceptance criteria for non-deterministic outputs by focusing on properties and constraints rather than exact wording.
• Implement consumer-driven contract testing using Pact to ensure seamless communication between mobile and backend services.
• Perform security-aware API testing, focusing on authorization boundaries, token handling, and malformed input.
• Localize failures across the stack by analyzing logs, traces, and database queries to identify if issues stem from the model, service code, or input data.
• Integrate evaluation processes into CI/CD pipelines to enforce quality gates on pull requests.

Requirements

• 3+ years of experience in technical QA or test automation, with a strong focus on API testing in production environments.
• Proficiency in Python for writing and maintaining test code, including pytest and fixtures.
• Deep understanding of HTTP, REST, JSON, and authentication flows.
• Hands-on experience with contract testing using Pact.
• Experience with LLM evaluation, prompt engineering, and managing non-deterministic outputs.
• Ability to work during hours that overlap with US Mountain Time afternoons.

What we offer

• Fully remote work environment.
• Engagement via B2B contract.
• Opportunity to work on a live, production-grade AI platform with a global team.