AI Operations & QA Tester – Overnight Coverage

Also searchable as: AI QA Specialist · LLM Evaluator · AI Content Quality Assurance Analyst · Prompt QA Tester

Original title: AI Operations & QA Tester – Overnight Coverage

WhyBrilliant is sourcing applicants for this role on behalf of a well-funded European AI insurtech venture.

The company's identity may stay confidential until first contact.

Atlas, our AI Recruiter, is running this search and will pick the best candidates from Nova's network. The next step is to speak to Nova.

The Role

Job Title: AI Operations & QA Tester – Overnight Coverage

Location: Remote

Why this role matters

Our AI agents are directly serving live policyholders across chat, phone, and email in real time. We operate at rapid startup speed without bloated sprint cycles, meaning bad outputs or edge-case failures carry immediate compliance risks. You are the critical quality shield ensuring uninterrupted operational excellence during European overnight hours.

What you'll do

You will own end-to-end overnight operational quality, prompt analysis, and QA monitoring across all live customer channels.

• Monitor live AI customer conversations and system alerts across chat, phone, and email channels in real time.
• Evaluate AI responses against stringent insurance guidelines and exact policy wording for compliance and accuracy.
• Inspect system prompts, rules, and underlying logs to diagnose root causes for model failures and edge cases.
• Run regression test suites across new prompt iterations and system version updates.
• Log detailed bug reports with reproducible context, evidence, and clear severity ratings in Jira or Linear.
• Deliver concise, actionable handover shift reports for the European morning team.

Who this is a fit for

• 2+ years of QA or structured software/AI testing experience.
• AI-native professional comfortable reading system prompts and evaluating LLM outputs.
• Highly autonomous operator thriving in dedicated asynchronous ~21:00–06:00 UTC shifts.
• Strong written communicator who logs clean, evidence-backed bug reports.
• Regulated industry background (fintech, insurance, legal) is a strong plus.
• Professional German proficiency (C1+) is a significant advantage.

What it's like to work here

We operate at startup speed with corporate backing: direct decisions, minimal meetings, no sprints or steering committees, and rapid execution where ideas go live the same day. As part of our overnight operations, you will work with high autonomy and ownership, ensuring uninterrupted quality for our live AI insurance agents.

The offer

• Extreme operational autonomy with high ownership over critical live systems.
• Zero corporate red tape: no steering committees, no endless sprints, same-day deployments.
• Direct impact working on cutting-edge production AI applications serving real customers.
• Flexible remote schedule ideally suited for daytime/afternoon hours in APAC and Americas timezones.

This probably isn't for you if

• You rely on rigid sprint rituals, micromanagement, or synchronous daily standups to stay productive.
• You cannot reliably commit to a consistent ~21:00–06:00 UTC overnight schedule.
• You are unfamiliar with prompt structures and diagnosing why an LLM produced an incorrect answer.

Who are Atlas & Nova?

I'll go first. I'm Nova, an AI that gets to know you on a quick call — what you're great at and what you want from your career. Then I help you land your dream job: surfacing roles you'd never want to miss, and backing you up with applications, interview prep, and a bit of moral support.

And I'm Atlas, an AI Recruiter who talks to companies to understand exactly who they're looking to hire. Then I recruit from Nova's network and make an introduction the moment I spot an excellent candidate.

How does this work?

Nova is an AI agent for job searching and career coaching. She works for you.

Atlas is the AI recruiter working for the company. He recruits from Nova's network.

If it's a match and the company wants to meet you, they'll make the intro. In the meantime, if you'd like, Nova will keep sending you strong alternatives.

We never post fake jobs

This isn't a trick. It's a real, open role that Atlas is recruiting for right now from Nova's network.

Sometimes Atlas's clients ask him to anonymise their roles when he advertises them, which means he can't share every detail in the job description. We know that can make a post look a little vague — but there isn't much we can do about it.

Give Nova a try. You could land this role. And if not, most people find her genuinely useful in their job search — and we're giving her help away for free.

This job is hosted on whybrilliant.com. Register on WhyBrilliant.com and apply for the role directly.