🎁 Before you apply, rehearse this interview. Create your free WorkMundi account and get an Interview Training on HelpsYouSpeak — no cost, no card. I want my training →
At Collective Health, we’re transforming how employers and their people engage with their health benefits by seamlessly integrating cutting-edge technology, compassionate service, and world-class user experience design. This is not a traditional QA role . You will be the quality owner for an LLM-based multi-agent pipeline that autonomously adjudicates health insurance claims for self-funded plan sponsors. You are building a Three-Tier Evaluation Framework to ensure our Gemini-powered agents reason correctly, call tools accurately, and produce DOL-ready outcomes. You will work at the intersection of Vertex AI, healthcare compliance, and high-scale data engineering. Your work directly determines whether claims are paid correctly and whether the company can withstand a Department of Labor (DOL) or state DOI audit. The stakes are real, the domain is hard, and the problems are genuinely novel. What you'll do: Outcome Evaluation (The "What") Golden Set Governance: Build and maintain a versioned library of "Grounding Data" results by working with senior claims examiners to define "Ground Truth." Model-as-a-Judge Automation: Design automated "LLM-grading-LLM" workflows using custom rubrics to score factual grounding and policy compliance. Semantic Assertion Framework: Develop testing libraries that move beyond string matching to validate semantic equivalence and numerical accuracy in agent outputs. Trajectory Evaluation (The "How") Function-Call Auditing: Use Vertex AI traces to programmatically verify that mandatory tools (via MCP) were invoked with correct arguments. Orchestration Logic Validation: Assert that agents respect defined priorities across the four architectural layers: Data & Knowledge, Orchestration, Agentic Reasoning, and Tooling. Reasoning Trace Auditing: Ensure every autonomous decision is traceable to a specific SOP sentence and a live API data point. Continuous Automated Regression (The "Always") CI/CD Integration: Every prompt or model update in Vertex AI Prompt Management must trigger an automated regression run. Auto-SxS: Own the automated pairwise comparison process to detect logic drift between "New" and "Production" agent versions. Mocking & Resilience: Build a Vertex AI/ADK mocking layer to simulate model responses, allowing for thousands of logic tests in seconds with zero API costs. To be successful in this role, you'll need: Required Skills (The Core Bar) 5+ years of of experience in software testing and development Python SDET Expertise : Expert in Python and pytest , specifically building custom mocking frameworks for external APIs ( Vertex AI/ADK ). AI/LLM Observability : Hands-on experience with Vertex AI Experiments , Auto-SxS , and Cloud Logging for trace analysis. Data Literacy : Expert-level SQL (BigQuery) and Pandas skills to "diff" massive datasets and identify adjudication discrepancies. Prompt Engineering for QA : Ability to analyze "System Instructions" and refine prompts based on failed test cases to close logic gaps. Architectural Testing : Experience testing multi-layer systems involving RAG ( Vertex AI Search ), state management ( LangGraph ), and function calling. Preferred Skills (The "Nice-to-Haves") Healthcare/Claims Domain: Familiarity with claims adjudication concepts (pend reason codes, COB, eligibility, stop-loss). Compliance Knowledge: Understanding of HIPAA/PHI handling and writing test evidence for regulatory bodies (DOL/DOI). Human-in-the-Loop Testing: Experience in "Shadow Mode" monitoring—comparing agent decisions against human expert (MCA) baselines. Pay Transparency Statement This is a hybrid position based out of our Lehi or Plano offices , with the expectation of being in office at least two weekdays per week . #LI-hybrid The actual pay rate offered within the range will depend on factors including geographic location, qualifications, experience, and internal equity. In addition to the salary, you will be eligible for 115000 stock options and benefits like health insurance, 401k, and paid time off . Learn more about our benefits at https://jobs.collectivehealth.com/benefits/ . Lehi, UT Pay Range $99,200 — $124,000 USD Plano, TX Pay Range $109,120 — $136,400 USD Why Join Us? Mission-driven culture that values innovation, collaboration, and a commitment to excellence in healthcare Impactful projects that shape the future of our organization Opportunities for professional development through internal mobility opportunities, mentorship programs, and courses tailored to your interests Flexible work arrangements and a supportive work-life balance We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status. Collective Health is committed to providing support to candidates who require reasonable accommodation during the interview process. If you need assistance, please contact recruiting-accommodations@collectivehealth.com . Privacy Notice For more information about why we need your data and how we use it, please see our privacy policy: https://collectivehealth.com/privacy-policy/ .
Here's how to pick the right one and stand out in your application.
144.883Jobs
31.687IN
81%EN
That number is real. WorkMundi's database shows 144,883 open engineer roles across the world. India has the most with 31,687 jobs, followed by the United States with 30,084. If you just finished reading one job ad and felt paralyzed by choice, you're not alone—but this scale is actually an advantage. It means you can afford to be selective.
Start by geography and language. The majority of engineer ads—117,837 of them—have the job posting text written in English. Use that as one filter, but remember: the ad text language tells you nothing about whether the role actually requires you to speak English day-to-day. Read the job description carefully. Then check which countries have the volume you're targeting. Singapore, Poland, and Australia round out the top five after India and the US.
Next, learn who's hiring. Accenture has posted 2,801 engineer roles. andurilindustries, speechify, and jobgether are also actively recruiting. If you're applying to one of these names, research their hiring patterns and interview style before you apply. That homework pays off.
When you interview, expect the question every engineer hears: 'Tell me about a time you had to debug a problem that wasn't in your job description.' Have a specific story ready—not a general one. Name the tools, the deadline pressure, and what you learned. Hiring managers listen for whether you see problem-solving as part of the role itself, not a favour.