Systems | Development | Analytics | API | Testing

You’ve probably encountered the online conversations about GitHub’s AI scaling challenges. A lot of the chatter is about how forgiving (or not) users should be to GitHub in light of inexorable traffic increases but also GitHub’s own decision-making. There is certainly irony in the fact that it was a swashbuckling GitHub, against the objections of many open source developers, that pressed Copilot on the industry.

In a release test I published in July, Katalon’s AI runner reported seven cases passed and four failed. I could open the steps, inspect their screenshots, and watch the browser session. The failed rows showed which expectations broke. The passed rows deserve the same scrutiny: what had the agent observed before it called each step passed? An AI test runner has to perform the action and judge the outcome. A false pass is a passed verdict when the expected behavior did not occur.

Your team has more changes to test, more environments to cover and less time to maintain regression suites. AI can help produce tests, but that introduces another workload: deciding whether those tests check the right behavior, whether a repair hides a defect and whether the evidence is enough to support a release. Agentic QA tools aim to take on more of that work, from planning and authoring to execution and failure investigation.

Enterprise software organizations allocate most of their total IT budgets to software validation and quality maintenance. According to the Consortium for Information and Software Quality (CISQ), poor software quality drains over $2.41 trillion annually from the US economy in operational failures, technical debt, and unmitigated production incidents.

Bots make up more than a third of internet traffic, and API endpoints attract more of it than web pages do. Scanners hunting for.env files, credential-stuffing scripts, LLM training crawlers, and the long tail of low-effort probes all hit your gateway before they ever reach a service. Every one of those requests costs you — in upstream load, in noise across your analytics, and in the bill.

“Customers can apply a discount code” looks ready for testing. Give that sentence to an AI model and it can produce cases for valid, invalid, expired, and reused codes in seconds. The trouble starts when someone asks what a valid code should do. Which basket is eligible? Does the discount change the displayed total before or after tax? Should the saved order carry the same amount?

In one release test, my AI agent triaged 43 backlog items into 134 executable cases in Katalon True Platform. The first pass at the steps caused trouble. We had bundled setup, navigation, action, and judgment into long instructions. The runner returned verdicts, but a failed step left me unsure which expectation it had checked. We split the cases into one action and one checkable expected result per step. The failures became specific enough to review against the screenshots and session recording.