Test Automation Lead AI Testing

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless workbut its work worth doing. If youre driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us. Are you passionate about leveraging technology to automate and transform business processes Do you have a deep understanding of AI and test automation and a desire to drive innovation in enterprise systems If so, the Tech Enablement and Automation team at Tech@Lilly is looking for a Lead Consultant in Test Automation to help scale our automation capabilities and quality practices across the organization. As a Lead consultant in Test Automation, you will be responsible for driving the quality and reliability of automation solutions across the enterprise. In this role, you will work closely with Engineers and cross-functional teams throughout the full application lifecycleensuring that systems are thoroughly tested, resilient, and aligned with business needs from development through to production. Are you innovative and curious about new technology that can make business processes easy and more efficient Come join our team! What Youll Be Doing: As a Lead Consultant you will play a pivotal role in shaping and advancing the companys enterprise-wide quality assurance and engineering strategy for AI powered systems and solutions. You will leverage both your technical expertise and leadership capabilities to oversee the delivery of scalable, high-quality AI and automation solutions. Your responsibilities will include driving resolution of complex testing challenges, leading the implementation of critical enhancements, and spearheading modernization initiatives to ensure our AI and automation frameworks and practices are future-ready and aligned with business objectives. What You Will Bring: Deep expertise in test automation frameworks and strategies, with the ability to extend them to validate AI and ML models, data pipelines, and intelligent applications.Hands-on expertise with AI evaluation methodologies, model benchmarking, and quality metricsA strong understanding of AI testing methodologies such as model validation, bias detection, explainability testing, and performance benchmarking.Ability to design end-to-end automated testing solutions for AI-enabled systems involving APIs, UI, data, and model outputs.Proven experience collaborating with data scientists, ML engineers, and software developers to ensure test coverage and model reliability. How You Will Succeed: Lead with influence by promoting a culture of engineering excellence, responsible AI, and continuous improvement.Develop and maintain automation frameworks tailored for testing AI/ML workflows, including model training, inference, and integration.Create and execute automated regression, performance, and data validation tests to ensure the stability and quality of AI-driven systems.Implement AI-specific test approaches such as synthetic data generation, result explainability validation, and continuous model monitoring.Drive innovation by identifying opportunities to embed automation and intelligence into the testing lifecycle using GenAI tools and frameworks.Partner cross-functionally to define and track AI quality metrics (accuracy, precision, recall, fairness) and integrate them into CI/CD pipelines.What You Should Bring: Deep understanding of LLMs, Generative AI, RAG, and Agentic AI architecturesHands-on experience in AI evaluation frameworks (RAGAS, DeepEval, LLM-as-a-Judge, LangFuse)Expertise in Prompt Validation, Prompt Injection Testing and Adversarial Testing (Red Teaming)Knowledge of Data Validation, Data Quality, Data Drift, and Training/Inference Pipeline TestingDeep hands-on experience with a broad range of test automation tools and technologies,UI and functional testing: Playwright, CypressAPI testing: PostmanPerformance testing: k6, JMeterStrong understanding of supervised/unsupervised learning.Experience with ML platforms and tools and cloud AI services.Knowledge of DevOps practices and CI/CD tools like Jenkins, GitHub Actions, or Azure DevOps.Excellent analytical, debugging, and communication skills with a passion for quality and innovation.Define, implement, and own test strategies across enterprise-level projects and programs.Lead and mentor global and cross-functional QA teams, promoting best practices and continuous improvement.Collaborate with senior leadership, product owners, architects, and development teams to ensure quality is built into all stages of .