Job Overview:
Quality for agentic systems is the focus. Our Client builds products that combine web apps, backend services, retrieval, and AI-driven agents, and this role defines how quality engineering works across them. The work spans testing agents, APIs, RAG workflows, and tools, then using agents themselves to boost coverage and evidence. Expect to mix deterministic automation with agent exploration and human-reviewed acceptance to keep complex behavior safe and repeatable.
This is a senior seat on a collaborative quality effort that partners closely with product owners, engineers, and domain experts. You own strategy and get hands-on with automation and evaluation, working across TypeScript, JavaScript, Python, modern test frameworks, and cloud tooling. Location and work model were not provided and will be discussed during early conversations. If you enjoy turning ambiguous AI behavior into clear, testable expectations, this job is worth a look.
Job Responsibilities:
- Set the automation approach for testing web UIs, APIs, back-end components, agent flows, internal tools, and retrieval pipelines
- Apply agent platforms and practical AI capabilities to testing across the organization
- For agent-assisted workflows, navigate apps, exercise APIs, inspect results, flag unexpected behavior, and collect evidence
- Agents generate test scenarios and edge cases, analyze requirements, surface coverage gaps, and assist failure investigation
- Build shared test libraries and utilities that product and engineering teams can use across apps and services
- Mix exploratory runs by agents with strict assertions and acceptance checks reviewed by people
- Create deterministic test packs that verify agent choices such as routing, which tools they pick, auth checks, citations, grounding, and how errors are handled
- Produce synthetic data and fixtures for repeatable runs, covering models, mocked tool outputs, retrieval results, test identities, and planned failure cases
- RAG behaviors get tested for retrieval relevance, grounding, citation patterns, access control, and resistance to unsupported answers
- Test apps and agent flows across happy paths, negative and edge cases, plus security, accessibility, and adversarial scenarios
- Regression coverage that tracks changes to apps, prompts, models, tools, knowledge sources, schemas, and orchestration logic
- Streaming and asynchronous interactions are covered across agents, APIs, UIs, and event-driven services
- CI/CD integration of automated tests and quality evidence, with failures made easy to reproduce and diagnose
- Work early with engineers, product managers, and subject experts to nail down acceptance criteria and testability before any build starts
- Define guardrails for agent-assisted tests, covering credentials, sensitive information, destructive operations, and anything that touches production
- Evidence is clearly distinguished between local synthetic results and live-environment signals, authorization scope, model quality, and production readiness
Job Requirements:
- Significant background in QA engineering, software testing, and building test automation
- Able to craft production-grade automation in TypeScript, JavaScript, Python, or a similar language
- Hands-on testing across web apps, APIs, distributed services, asynchronous workflows, and event-driven systems
- Hands-on practice with unit, integration, contract, end-to-end, regression, and exploratory testing
- Strong test design skills, including risk-based approaches, fixtures and mocks, failure injection, and making results reproducible
- Experience wiring automated tests and quality reports into CI and CD pipelines
- Skilled at tracing failures across app code, infrastructure, data layers, and service interactions
- Real curiosity about applying agents and generative AI as everyday testing tools
- Good judgment on when to use agent assist, deterministic automation, or human review
- Clear communication that turns ambiguous behavior into explicit, testable expectations
- Background with AI-driven testing agents, browser-based agents, or autonomous testing workflows is a plus (Preferred)
- Background testing LLM apps, agents, RAG, semantic search, or conversational interfaces is a plus (Preferred)
- Bonus if you can assess grounding, hallucination risk, citations, prompt-injection exposure, tool misuse, and data leakage (Preferred)
- Familiarity with agent frameworks like Amazon Bedrock, Bedrock AgentCore, Strands Agents, or comparable tools is a plus (Preferred)
- Experience with AWS services, IAM, CloudWatch, and testing cloud-native applications is a plus (Preferred)
- Knowledge of vector databases, embeddings, document ingestion, and retrieval-quality evaluation is a plus (Preferred)
- Hands-on with Playwright, Vitest, Jest, Cypress, Pact, or k6 is a plus (Preferred)
- Exposure to MCP, tool-calling patterns, or building reusable agent capabilities is a plus (Preferred)
- Knowledge of model or prompt evaluation platforms, or building custom golden-dataset harnesses, is a plus (Preferred)
- Next.js and React testing experience is a plus (Preferred)
- Experience touching accessibility, performance, security, or resilience testing is a plus (Preferred)
- Past testing work in regulated settings, security-sensitive shops, or healthcare is a plus (Preferred)