Senior AI QA Test Automation Engineer – CX Agent

About the role

We are looking for a Senior AI QA Test Automation Engineer to lead AI quality and test automation for our CX Agent Suite, a multi‑agent customer‑support system built on AWS Bedrock AgentCore. The role combines hands‑on engineering with strategic guidance, working closely with QA leadership, Data Science, and Engineering.

Key responsibilities

• Design and build scalable AI evaluation pipelines that measure accuracy, relevance, tone, hallucination rate, safety compliance, and task completion for multiple agents.
• Develop and maintain AI/ML frameworks, libraries, and tooling for the CX Agent Suite, extending the current DeepEval harness.
• Prototype and evaluate agent‑orchestration frameworks such as Strands Agents, influencing go/no‑go adoption decisions.
• Integrate AI‑driven testing into GitLab CI/CD pipelines and manage Terraform‑provisioned, EKS‑hosted services.
• Produce data‑driven quality analytics, root‑cause analysis, and intelligent test prioritization using OpenTelemetry, CloudWatch, X‑Ray, and LangFuse data.

Required profile

• BSc/MSc in Computer Science, AI, or a related discipline.
• 6+ years of hands‑on QA/Test Automation experience, with a strong background in building automation frameworks.
• 1+ years applying AI/ML to software testing, delivering production‑ready AI solutions.

Required skills

• Strong Python programming; experience with Java or TypeScript is a plus.
• Deep knowledge of AWS services, especially Bedrock, Terraform, EKS, and Docker/Kubernetes.
• Hands‑on experience with LLM evaluation techniques, DeepEval, and agent‑orchestration SDKs (e.g., Strands Agents).
• Proficiency integrating AI tooling into GitLab CI/CD pipelines and working with MLOps pipelines.
• Familiarity with observability tools such as OpenTelemetry, CloudWatch, X‑Ray, LangFuse, and vector databases.

What we offer

• Attractive remuneration package.
• Intellectually stimulating work environment.
• Continuous personal development and international training opportunities.