This article was written by Verbit’s Chief Technology Officer Simon Rapoport as part of a new series featuring his leadership insights.
In March 2026, the House and Senate introduced the Research and Oversight of AI in Courts Act, a bill that would create a federal task force to study how AI speech-to-text and voice recognition should be allowed to operate in federal courtrooms. That’s a notable moment. Lawmakers aren’t just asking whether AI belongs in the courtroom anymore. They’re asking how it should behave once it’s there, especially when it’s operating live, in real time, in front of a judge, a jury, or a witness under oath.
That question sits right in the middle of what my team builds every day. So I want to use this space to unpack something that gets thrown around constantly in AI conversations but rarely gets explained properly: what “real time” actually means, why it’s harder than it sounds, and why chasing speed for its own sake can be the wrong goal entirely.
A quick refresher, for anyone newer to this
Most of the AI tools people interact with today – ChatGPT, Claude, and similar assistants – are built on large language models, or LLMs. An LLM doesn’t just retrieve a pre-written answer. It works through a response: drafting, checking itself, sometimes revising before it shows you anything. That process is usually invisible. You just experience it as a few seconds, or up to a minute, of waiting. Once you know that’s an LLM “thinking” behind the scenes, a simple tradeoff becomes obvious: the more thinking involved, the slower the answer.
If you already know all of that, bear with me, because it’s the foundation for what actually matters here.
Real-time isn't one thing
Here’s the part people get wrong most often: “real-time” is not a fixed, universal standard. It means something different depending entirely on what you’re building.
In defense robotics, real-time might mean response times measured in the tens to low hundreds of milliseconds. In consumer voice systems like phone-based voicebots, it might mean something closer to 300 milliseconds. At Verbit, in legal, we typically talk about near real-time, because our response times are measured in seconds, sometimes minutes, not milliseconds.
That’s not a limitation we’re apologizing for. It’s a deliberate reflection of the use case. If Legal Visor, our AI insights platform for attorneys, flags an inconsistency in a witness’ testimony and surfaces it to an attorney in five seconds instead of one, it makes no practical difference to that attorney. He’s not acting on it within a second either way. Chasing millisecond-level speed in the context of depositions and arbitrations wouldn’t make the product better. It would just mean we spent engineering effort optimizing for a number nobody actually needed.
This is also, I suspect, exactly the kind of nuance that Congress’s proposed task force will have to sort through. My honest prediction is one of three outcomes:
1) Regulators either decide the risk is too high and restrict where this kind of live AI can run at all
2) They allow it, but require some form of certification, similar to how court reporters are certified today
3) They focus specifically on where the processing happens, requiring on-device processing rather than sending live courtroom audio to the cloud, especially if privacy is the primary concern.
Which path they take will depend entirely on which risk they’re most worried about.
What's actually happening when an AI "thinks" about your case
Here’s where I want to go a layer deeper, because I think this is genuinely misunderstood, including by plenty of people building in this space.
When you ask an LLM a single question about a specific legal case, one backed by hundreds or thousands of pages of exhibits and case documents, a single prompt and a single model call isn’t enough to produce a genuinely useful answer. You need something considerably more complex: a system where multiple AI agents work together, an orchestrating agent directing the others, each one pulling from different data sources (vector databases, document management systems, and more), filtering and ranking what’s relevant, and feeding the best of it back into the models producing your answer. This is often called a multi-agent architecture combined with retrieval-augmented generation, or RAG for short, a technique for grounding an AI’s answer in your specific documents rather than relying only on what the model already “knows” in general.
In practice, that means when you ask one question, the system behind the scenes might be asking dozens or hundreds of smaller questions of its own: searching, retrieving, cross-checking, finding inconsistencies between sources, and going back to resolve them before giving you a single clean answer. That orchestration is the real “thinking” happening behind the scenes, and it’s a meaningfully harder engineering problem than people assume when they picture AI as a single model just typing out a response.
It’s also, frankly, why I’m skeptical of a lot of what gets marketed as AI innovation in legal tech right now. A large share of AI products today are what I’d call LLM wrappers: a clean interface sitting in front of a single model call, with little else underneath. That’s not what we’re building. And it’s part of why we don’t want to be thought of as a single product, whether that’s Legal Visor’s AI insights for attorneys or Legal Capture, our legal transcription solution for courtrooms and digital court reporters. We’re building toward something broader: a verbal intelligence platform, one where deep case understanding and multi-agent reasoning are the foundation, not a bolt-on feature.
Where I think this is all heading
One more thing worth sharing, something I’ve been thinking about a lot lately. There’s a useful way to think about how AI has evolved that I’d describe as a spiral rather than a straight line. Before LLMs, from roughly 2000 to 2020, the field trained narrow neural networks for specific tasks: one model for one job. Then LLMs arrived with a different bet entirely: build one enormous, general-purpose model that performs reasonably well at nearly anything. That’s the approach the industry has been running with for the last five years.
I think we’re about to spiral back, but at a higher level. As Fin (formerly Intercom) CEO and Founder Eoghan McCabe put it recently, we should expect more “speciation” in AI the way nature produces wildly different kinds of brains for different niches, rather than a single all-knowing oracle. Instead, you take a smaller model that still has real reasoning ability at its core, and you specialize it for a specific job.
This matters for a very practical reason: open-weight, open-source models are closing the performance gap with the biggest commercial frontier models faster than most people expected. That opens a real opportunity to build smaller, faster, cheaper, fine-tuned vertical models that outperform general-purpose models on specific tasks, like identifying inconsistencies in legal testimony or deeply understanding case-specific language. We’re not doing this today, but it’s a direction I believe strongly in, and one I think will define the next phase of legal AI.
It also raises a harder problem worth being honest about: this field changes fast, and “changing” doesn’t always mean “improving” for your specific use case. Foundation models today can reach end-of-life, in the sense of no longer being the right fit for your system, in six months or less. That means constant re-evaluation isn’t optional. We’re investing directly in two things to manage that: building guardrails and evaluation pipelines that continuously protect the accuracy of our results, and leaning into specialized vertical models to keep performance consistent over time, rather than hoping a general-purpose model stays reliable for a task it was never built specifically to do.
Real-time, in the end, isn’t really the goal. Reliable, defensible, deeply informed answers, delivered at whatever speed the situation actually demands, is the goal. That’s a less flashy story than “instant AI,” but it’s the one that actually holds up in a courtroom.
About our CTO Simon Rapoport
Simon brings with him over two decades of experience in technology leadership and innovation. Prior to joining Verbit, Simon served as CTO at AudioCodes Ltd., where he successfully led the company to the forefront of AI innovation by aligning technology initiatives with business goals and spearheading the development of GenAI architecture.
Before his tenure at AudioCodes, Simon was the CTO and VP of R&D at Callverso, where he developed a groundbreaking Conversational AI Platform and oversaw all technological aspects of the business. His strategic vision and expertise have consistently driven companies to maintain a competitive edge in rapidly evolving markets.
Simon’s extensive career also includes pivotal roles at MyHeritage Ltd., where he led the design and deployment of scalable architectures that significantly increased user growth, and at Poly Information Ltd., which he founded. At Poly Information, Simon managed strategic partnerships and R&D, contributing to the company’s public listing on the London Stock Exchange.
Simon holds a BSc in Computer Sciences and a degree in Management from Tel Aviv University and has authored several international patents in AI and Computational Linguistics.
