Prompt Smart Solutions - GenAI Automation Test Consultant

Job Title: Prompt Smart Solutions - GenAI Automation Test Consultant Company: Prompt Smart Solutions Experience: 10 - 12 Years Skills: Generative AI, Automation Testing, Testing, AI/ML Testing, STLC Process, Java, Selenium, Python, Functional Testing, Regression Testing, Integration Testing Description: Job Title : Gen AI Test ConsultantLocation : PuneExperience Required : 10+ YearsJob Summary : We are seeking an experienced and visionary Gen AI Test Consultant to drive the design, development, and implementation of advanced test automation frameworks for our GenAI-driven products and platforms. The ideal candidate will possess deep expertise in software testing, automation strategies, and AI/ML/GenAI technologies, with a proven track record of leading high-performing QA teams in complex, fast-paced environments. The candidate should understand the STLC landscape of the client and should be able to prescribe GenAI based solutions to fast track the clients test deliverables.Key Responsibilities : Leadership & Strategy : - Provide GenAI based solutions to the clients STLC issues.- Lead and mentor a team of GenAI test automation engineers, fostering a culture of innovation, collaboration, and continuous improvement.- Define and execute the overall AI/ML/GenAI test automation strategy for AI/ML/GenAI products, ensuring alignment with business objectives and engineering best practices.- Collaborate with client and account stakeholders, product managers, data scientists, developers, and DevOps teams to ensure high-quality releases.Test Automation Development : - Architecture, Design, develop, and maintain robust and reusable test automation frameworks in any of the following : Java-Selenium, Python-Selenium, Java-Appium, JavaScript -Cypress.- Develop, execute, and maintain automated test scripts for functional, regression, integration, and performance testing.- Integrate automated tests into CI/CD pipelines to support continuous testing and delivery.GenAI Testing : - Design GenAI/Agentic AI based solutions, can involve multiple tools, agents, MCPs.- Understand business requirements/user stories and create test deliverables using GenAI.- Create reusable prompt libraries utilizing different LLMs GPT, Mistral, Llama, Claude, Gemini.- Understanding RAG (Retrieval-Augmented Generation) based architecture and test GenAI based RAG applications. - Expertise on performing LLM validation using frameworks like EVALS, RAGAS for correctness, hallucination detection, reliability, and performance benchmarking.Quality Assurance & Process Improvement : - Analyze test results, identify defects, and collaborate with development teams for resolution.- Continuously improve test automation processes, tools, and methodologies to enhance test coverage and efficiency.- Contribute to the definition and tracking of key quality metrics and provide insights to stakeholders.Coding & Technology : - Should be able to script in Java/Selenium/Python/Javascript/C#.- Should be able to write code in Python and worked with Python libraries for AI/ML (pandas, numpy, scikitlearn, streamlit, etc) Documentation & Reporting : - Prepare and maintain detailed test plans, test cases, and test documentation.- Report test progress, results, and risks to project stakeholders in a clear and timely manner.Stakeholder Management : - Communicate test plans, progress, risks, and results to technical and non-technical stakeholders.- Provide technical guidance and training to QA and development teams on AI/ML/GenAI testing best practices.Good to Have : - Exposure to responsible AI / model safety practices.- Experience with data annotation or evaluation dataset creation.Required Skills & Qualifications : - Bachelors or Masters degree in Computer Science, Engineering, or a related field.- 10+ years of experience in software testing and automation, with at least 3 years in a leadership role.- Strong experience in designing and implementing test automation frameworks using tools such as Selenium, Appium, Cypress, TestNG, JUnit, or similar. Should have good experience on BDD, Cucumber.- Experience on GenAI in Cloud hyperscalers AWS Bedrock, Google Vertex AI, Azure OpenAI.- Experience in testing RAG based GenAI applications. Should have worked with any of the Vector Databases Pinecone, ChromaDB, FAISS, etc.- Proficiency in programming languages such as Python/Java/JavaScript/C#.- Good understanding of SDLC, STLC, agile methodologies, and DevOps practices.- Strong analytical, problem-solving, and communication skills.Preferred Qualifications : - Relevant certifications (ISTQB, AI/ML, cloud technologies).- Experience working in regulated industries (finance, healthcare, etc.).- Contributions to open-source test automation or AI/GenAI communities. .