AI Engineer with Deep Data Science Understanding

emagine Polska is looking for an experienced AI Engineer to join a project within the Scandinavian banking sector. You will play a key role in developing the AI Skills Registry, focusing on building reliable components for AI solutions that operate effectively across diverse platforms.

Key responsibilities

• Build components for AgentCore and Amazon Quick while running evaluation frameworks for skills and tools, including rubric design and LLM-as-judge calibration.
• Ensure the reliability and accuracy of developed AI components through rigorous testing.
• Operationalize the evaluation and iteration phases to maintain robust practices across agent builds.
• Define and track quality and cost-per-outcome metrics for production agents using tools like Arize AX.
• Conduct structured error analysis on agent outputs to identify failure patterns and implement effective technical fixes.

Requirements

• Hands-on experience in evaluating machine learning or LLM-based systems within production environments.
• Working knowledge of evaluation methodology, including error analysis and rubric design.
• Strong proficiency in Python and advanced data analysis skills.
• Solid experience with cloud-native ML infrastructure, specifically AWS Bedrock.
• Ability to work independently and contribute effectively within a developing team environment.

What we offer

• A fully remote working environment allowing for flexibility and focus.
• Engagement in a long-term project within the stable and innovative Scandinavian banking industry.
• Opportunity to work with cutting-edge AI technologies and agent-based architectures.
• Professional development through collaboration with experienced teams on complex, high-impact solutions.