How leading AI companies really build: lessons from 40+ engineering leaders

What does it actually take to ship Gen 2 AI experiences to real users at scale? Matthew O'Riordan, CEO of Ably, shares insights from conversations with 40+ engineering leaders — including at unicorns and public corporations — on where AI delivery breaks and what production teams are doing about it.

Topics covered:

• Gen 1 vs Gen 2 AI: why the shift to async, multimodal experiences changes the infrastructure problem
• Why there are no standards yet - and why every team is building custom agent-to-client orchestration
• Progress visibility, interruption, and concurrency: the UX patterns every production AI team ends up needing
• Human-in-the-loop: escalation patterns that don't require a separate chat product
• LLM failure rates and durable workflows: how teams manage crashes and error states
• Corporate network interference: why companies are switching from HTTP streaming to WebSockets
• Token costs and batching: how cost became a sensitive engineering line item
• The re-architecture phase: when cobbled-together AI stacks become the bottleneck
• Enterprise requirements: data residency, PII protection, and audit trails at B2B scale

Timestamps:

0:00 Introduction

0:27 The research

2:03 The 80/20 reality

3:38 Why everyone builds their own

5:35 The framework journey

9:27 The UX patterns that matter

12:53 Human-in-the-loop

15:26 Agent-to-agent communication

17:33 The technical surprises

22:00 Enterprise realities

23:00 What's coming

25:16 Cost controls

28:04 Conclusion

Links:
Ably AI Transport: https://ably.com/ai-transport
AI Transport docs: https://ably.com/docs/ai-transport
Ably GitHub: https://github.com/ably

Ably on LinkedIn: https://www.linkedin.com/company/ably-realtime
Ably on X: https://x.com/ablyrealtime
Ably on Facebook: https://www.facebook.com/ablyrealtime

About Ably:
Ably is an enterprise-grade realtime messaging platform. We help developers build, ship, and scale realtime features — from chat and notifications to AI agent streaming and live data sync. Over 2 billion devices served every month, with 100% uptime for 7+ years.

Learn more: https://ably.com

#AIAgents #RealtimeAI #WebSockets #DeveloperTools #Ably