Senior Automation Test Engineer – AI Systems (f/m/x)

Sii Sp. z o.o. is seeking a Senior Automation Test Engineer to join a team building an AI-powered operating system for healthcare marketing. In this role, you will go beyond traditional testing to own the quality, evaluation, and monitoring of complex AI systems, ensuring the stability of a full-stack architecture while scaling AI evaluation pipelines.

Key responsibilities

• Design and maintain evaluation frameworks for Large Language Models and custom agentic harnesses.
• Develop automated scoring mechanisms to measure agent accuracy, reasoning reliability, and hallucination rates.
• Collaborate with engineering teams to implement monitoring for data drift and model performance degradation.
• Own the end-to-end testing strategy across the Angular frontend, Node.js backend, and Python-based AI microservices.
• Integrate AI evaluation and testing harnesses into CI/CD pipelines to enforce continuous quality gates.

Requirements

• Minimum 5 years of experience in software testing, with at least 2 years focused on AI/ML applications or LLM workflows.
• Proficiency in Python for testing scripts and familiarity with JavaScript/TypeScript for Node.js and Angular ecosystems.
• Experience with evaluation frameworks such as LangSmith, Langfuse, Ragas, or MLflow.
• Strong system-level design skills and the ability to build production-grade test infrastructure from the ground up.
• Advanced level of English.

What we offer

• Employment stability within a leading technology consulting company with a strong market position.
• Access to a wide range of fascinating projects for prestigious global brands.
• Attractive benefits package including private healthcare and a benefits cafeteria platform.
• Investment in professional growth through meetups, webinars, and a dedicated training platform.
• A friendly work environment recognized multiple times as a Great Place to Work.