New Emerging Trends in the Quality Engineering Space with AI | Vanya Seth | Testflix 2025
Retrieval-Augmented Generation can sound convincing while still being wrong. This session focuses on moving beyond surface-level metrics and turning stochastic AI outputs into evidence-backed, verifiable results. It explores how to test the entire RAG pipeline, from ingestion and indexing to retrieval, grounding, and answerability, ensuring every response is traceable to the right source, policy, and user context.
The talk introduces practical testing techniques such as rationale overlap, abstention SLOs, robustness checks for typos and paraphrases, counterfactual testing, and drift detection through embedding and index analysis. It also covers chaos testing, poisoning detection, and enforcing “smart silence” when answers lack support. The outcome is a CI-ready testing approach that transforms RAG from impressive demos into auditable, reliable systems, giving QA teams confidence, clarity, and control.
#AutomationTesting #TestAutomation #ITindustry #TheTestTribe #Leadership
This video is of one of the sessions presented at #Testflix - World’s Leading Virtual Software Testing Conference.
About the Speaker:
Vanya Seth, Head of Technology at ThoughtWorks
Vanya is an experienced technologist with a demonstrated history of working in the IT services industry, Vanya is a strategic consultant specializing in data mesh, platforms, delivery infrastructure, cloud native evolutionary architecture and AI first software delivery.
In her current role as Head of Technology for Thoughtworks in India, Vanya shapes new pursuits, strategically advises on client projects, and formulates meaningful and resilient technology strategies. She is also part of the Thoughtworks' Technology Advisory Board that puts together thoughts on emerging technology trends in the industry, resulting in the much sought after Technology Radar
Vanya is a passionate technologist with a knack for solving complex problems, at scale. She is also deeply involved with open source communities.
Connect with Vanya on LinkedIn - https://www.linkedin.com/in/vanyaseth1809/
Testflix 2025 Proud Sponsors:
Premier Sponsor – BrowserStack: https://www.browserstack.com/
Platinum Sponsor – AutoTester.ai: https://autotester.ai/
Platinum Sponsor – UiPath: https://www.uipath.com/
Gold Sponsor – ACCELQ: https://www.accelq.com/
Gold Sponsor – Sahi Pro: https://www.sahipro.com/
Silver Sponsor – Beeceptor: https://beeceptor.com/
Bronze Sponsor – Baseline11: https://baseline11.com/
Learn from industry experts with Thrive EdSchool courses, grow your network with software testers at The Test Tribe events, and join Asia's largest testing community on Discord.
Upskill yourself with Thrive EdSchool courses: https://bit.ly/thrivettt
Grow your network with software testers with the events at The Test Tribe: https://bit.ly/tttevents
Become a member of Asia's largest testing community: https://bit.ly/3FONxJP
About The Test Tribe:
The Test Tribe is the World’s Largest Software Testing Community turned EdTech Startup. Founded in 2018, our mission is to give the Testing Craft the glory it deserves while empowering testers globally through courses, memberships, cohorts, meetups, hackathons, and global conferences. With 800+ events and 165K+ members across 152+ countries, we’re helping testers collaborate, learn, and grow together.