FEED Validator

for Atom and RSS and KML

Congratulations!

This is a valid RSS feed.

Recommendations

This feed is valid, but interoperability with the widest range of feed readers could be improved by implementing the following recommendations.

line 1, column 38: Use of unknown namespace: https://www.oreilly.com/rss/custom [help]

<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
                                      ^

line 521, column 0: content:encoded should not contain fetchpriority attribute [help]
line 521, column 0: content:encoded should not contain decoding attribute (10 occurrences) [help]
line 521, column 0: content:encoded should not contain sizes attribute (10 occurrences) [help]
line 611, column 0: content:encoded should not contain loading attribute (7 occurrences) [help]

Source: https://www.oreilly.com/radar/feed/index.xml

<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
xmlns:content="http://purl.org/rss/1.0/modules/content/"
xmlns:media="http://search.yahoo.com/mrss/"
xmlns:wfw="http://wellformedweb.org/CommentAPI/"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
xmlns:custom="https://www.oreilly.com/rss/custom"
>
<channel>
<title>Radar</title>
<atom:link href="https://www.oreilly.com/radar/feed/" rel="self" type="application/rss+xml" />
<link>https://www.oreilly.com/radar</link>
<description>Now, next, and beyond: Tracking need-to-know trends at the intersection of business and technology</description>
<lastBuildDate>Thu, 18 Sep 2025 10:12:31 +0000</lastBuildDate>
<language>en-US</language>
<sy:updatePeriod>
hourly </sy:updatePeriod>
<sy:updateFrequency>
1 </sy:updateFrequency>
<generator>https://wordpress.org/?v=6.8.2</generator>
<image>
<url>https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/04/cropped-favicon_512x512-160x160.png</url>
<title>Radar</title>
<link>https://www.oreilly.com/radar</link>
<width>32</width>
<height>32</height>
</image>
<item>
<title>Generative AI in the Real World: Faye Zhang on Using AI to Improve Discovery</title>
<link>https://www.oreilly.com/radar/podcast/generative-ai-in-the-real-world-faye-zhang-on-using-ai-to-improve-discovery/</link>
<comments>https://www.oreilly.com/radar/podcast/generative-ai-in-the-real-world-faye-zhang-on-using-ai-to-improve-discovery/#respond</comments>
<pubDate>Thu, 18 Sep 2025 10:12:22 +0000</pubDate>
<dc:creator><![CDATA[Ben Lorica and Faye Zhang]]></dc:creator>
<category><![CDATA[AI & ML]]></category>
<category><![CDATA[Generative AI in the Real World]]></category>
<category><![CDATA[Podcast]]></category>
<guid isPermaLink="false">https://www.oreilly.com/radar/?post_type=podcast&p=17467</guid>
<media:content
url="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2024/01/Podcast_Cover_GenAI_in_the_Real_World-scaled.png"
medium="image"
type="image/png"
/>
<description><![CDATA[In this episode, Ben Lorica and AI Engineer Faye Zhang talk about discoverability: how to use AI to build search and recommendation engines that actually find what you want. Listen in to learn how AI goes way beyond simple collaborative filtering—pulling in many different kinds of data and metadata, including images and voice, to get […]]]></description>
<content:encoded><![CDATA[
In this episode, Ben Lorica and AI Engineer Faye Zhang talk about discoverability: how to use AI to build search and recommendation engines that actually find what you want. Listen in to learn how AI goes way beyond simple collaborative filtering—pulling in many different kinds of data and metadata, including images and voice, to get a much better picture of what any object is and whether or not it’s something the user would want.
About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2025, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.
Check out <a href="https://learning.oreilly.com/playlists/42123a72-1108-40f1-91c0-adbfb9f4983b/?_gl=1*16z5k2y*_ga*MTE1NDE4NjYxMi4xNzI5NTkwODkx*_ga_092EL089CH*MTcyOTYxNDAyNC4zLjEuMTcyOTYxNDAyNi41OC4wLjA." target="_blank" rel="noreferrer noopener">other episodes</a> of this podcast on the O’Reilly learning platform.
<h2 class="wp-block-heading">Transcript</h2>
This transcript was created with the help of AI and has been lightly edited for clarity.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=0" target="_blank" rel="noreferrer noopener">0:00</a>: Today we have Faye Zhang of Pinterest, where she’s a staff AI engineer. And so with that, very welcome to the podcast.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=14" target="_blank" rel="noreferrer noopener">0:14</a>: Thanks, Ben. Huge fan of the work. I’ve been fortunate to attend both the Ray and NLP Summits. I know where you serve as chairs. I also love the O’Reilly AI podcast. The recent episode on A2A and the one with Raiza Martin on NotebookLM have been really inspirational. So, great to be here. 
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=33" target="_blank" rel="noreferrer noopener">0:33</a>: All right, so let’s jump right in. So one of the first things I really wanted to talk to you about is this work around <a href="https://arxiv.org/abs/2503.00619" target="_blank" rel="noreferrer noopener">PinLanding</a>. And you’ve published papers, but I guess at a high level, Faye, maybe describe for our listeners: What problem is PinLanding trying to address?
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=53" target="_blank" rel="noreferrer noopener">0:53</a>: Yeah, that’s a great question. I think, in short, trying to solve this trillion-dollar discovery crisis. We’re living through the greatest paradox of the digital economy. Essentially, there’s infinite inventory but very little discoverability. Picture one example: A bride-to-be asks ChatGPT, “Now, find me a wedding dress for an Italian summer vineyard ceremony,” and she gets great general advice. But meanwhile, somewhere in Nordstrom’s hundreds of catalogs, there sits the perfect terracotta Soul Committee dress, never to be found. And that’s a $1,000 sale that will never happen. And if you multiply this by a billion searches across Google, SearchGPT, and Perplexity, we’re talking about a $6.5 trillion market, according to Shopify’s projections, where every failed product discovery is money left on the table. So that’s what we’re trying to solve—essentially solve the semantic organization of all platforms versus user context or search.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=125" target="_blank" rel="noreferrer noopener">2:05</a>: So, before PinLanding was developed, and if you look across the industry and other companies, what would be the default—what would be the incumbent system? And what would be insufficient about this incumbent system?
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=142" target="_blank" rel="noreferrer noopener">2:22</a>: There have been researchers across the past decade working on this problem; we’re definitely not the first one. I think number one is to understand the catalog attribution. So, back in the day, there was multitask R-CNN generation, as we remember, [that could] identify fashion shopping attributes. So you would pass in-system an image. It would identify okay: This shirt is red and that material may be silk. And then, in recent years, because of the leverage of large scale VLM (vision language models), this problem has been much easier.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=183" target="_blank" rel="noreferrer noopener">3:03</a>: And then I think the second route that people come in is via the content organization itself. Back in the day, [there was] research on join graph modeling on shared similarity of attributes. And a lot of ecommerce stores also do, “Hey, if people like this, you might also like that,” and that relationship graph gets captured in their organization tree as well. We utilize a vision large language model and then the foundation model CLIP by OpenAI to easily recognize what this content or piece of clothing could be for. And then we connect that between LLMs to discover all possibilities—like scenarios, use case, price point—to connect two worlds together.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=235" target="_blank" rel="noreferrer noopener">3:55</a>: To me that implies you have some rigorous eval process or even a separate team doing eval. Can you describe to us at a high level what is eval like for a system like this? 
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=251" target="_blank" rel="noreferrer noopener">4:11</a>: Definitely. I think there are internal and external benchmarks. For the external ones, it’s the Fashion200K, which is a public benchmark anyone can download from Hugging Face, on a standard of how accurate your model is on predicting fashion items. So we measure the performance using the recall top-k metrics, which says whether the label appears among the top-end prediction attribute accurately, and as a result, we were able to see 99.7% recall for the top ten.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=287" target="_blank" rel="noreferrer noopener">4:47</a>: The other topic I wanted to talk to you about is recommendation systems. So obviously there’s now talk about, “Hey, maybe we can go beyond correlation and go towards reasoning.” Can you [tell] our audience, who may not be steeped in state-of-the-art recommendation systems, how you would describe the state of recommenders these days?
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=323" target="_blank" rel="noreferrer noopener">5:23</a>: For the past decade, [we’ve been] seeing tremendous movement from foundational shifts on how RecSys essentially operates. Just to call out a few big themes I’m seeing across the board: Number one, it’s kind of moving from correlation to causation. Back then it was, hey, a user who likes X might also like Y. But now we actually understand why contents are connected semantically. And our LLM AI models are able to reason about the user preferences and what they actually are.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=358" target="_blank" rel="noreferrer noopener">5:58</a>: The second big theme is probably the cold start problem, where companies leverage semantic IDs to solve the new item by encoding content, understanding the content directly. For example, if this is a dress, then you understand its color, style, theme, etc. 
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=377" target="_blank" rel="noreferrer noopener">6:17</a>: And I think of other bigger themes we’re seeing; for example, Netflix is merging from [an] isolated system into a unified intelligence. Just this past year, Netflix [updated] their multitask architecture where [they] shared representations, into one they called the UniCoRn system to enable company-wide improvement [and] optimizations.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=404" target="_blank" rel="noreferrer noopener">6:44</a>: And very lastly, I think on the frontier side—this is actually what I learned at the AI Engineer Summit from YouTube. It’s a DeepMind collaboration, where YouTube is now using a large recommendation model, essentially teaching Gemini to speak the language of YouTube: of, hey, a user watched this video, then what might [they] watch next? So a lot of very exciting capabilities happening across the board for sure.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=435" target="_blank" rel="noreferrer noopener">7:15</a>: Generally it sounds like the themes from years past still map over in the following sense, right? So there’s content—the difference being now you have these foundation models that can understand the content that you have more granularly. It can go deep into the videos and understand, hey, this video is similar to this video. And then the other source of signal is behavior. So those are still the two main buckets?
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=473" target="_blank" rel="noreferrer noopener">7:53</a>: Correct. Yes, I would say so. 
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=475" target="_blank" rel="noreferrer noopener">7:55</a>: And so the foundation models help you on the content side but not necessarily on the behavior side? <a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=483" target="_blank" rel="noreferrer noopener">8:03</a>: I think it depends on how you want to see it. For example, on the embedding side, which is a kind of representation of a user entity, there have been transformations [since] back in the day with the BERT Transformer. Now it’s got long context encapsulation. And those are all with the help of LLMS. And so we can better understand users, not to next or the last clicks, but to “hey, [in the] next 30 days, what might a user like?”
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=511" target="_blank" rel="noreferrer noopener">8:31</a>: I’m not sure this is happening, so correct me if I’m wrong. The other thing that I would imagine that the foundation models can help with is, I think for some of these systems—like YouTube, for example, or maybe Netflix is a better example—thumbnails are important, right? The fact now that you have these models that can generate multiple variants of a thumbnail on the fly means you can run more experiments to figure out user preferences and user tastes, correct?
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=545" target="_blank" rel="noreferrer noopener">9:05</a>: Yes. I would say so. I was lucky enough to be invited to one of the engineer network dinners, [and was] speaking with the engineer who actually works on the thumbnails. Apparently it was all personalized, and the approach you mentioned enabled their rapid iteration of experiments, and had definitely yielded very positive results for them.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=569" target="_blank" rel="noreferrer noopener">9:29</a>: For the listeners who don’t work on recommendation systems, what are some general lessons from recommendation systems that generally map to other forms of ML and AI applications? 
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=584" target="_blank" rel="noreferrer noopener">9:44</a>: Yeah, that’s a great question. A lot of the concepts still apply. For example, the knowledge distillation. I know Indeed was trying to tackle this. 
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=596" target="_blank" rel="noreferrer noopener">9:56</a>: Maybe Faye, first define what you mean by that, in case listeners don’t know what that is. 
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=602" target="_blank" rel="noreferrer noopener">10:02</a>: Yes. So knowledge distillation is essentially, from a model sense, learning from a parent model with larger, bigger parameters that has better world knowledge (and the same with ML systems)—to distill into smaller models that can operate much faster but still hopefully encapsulate the learning from the parent model.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=624" target="_blank" rel="noreferrer noopener">10:24</a>: So I think what Indeed back then faced was the classic precision versus recall in production ML. Their binary classifier needs to really filter out the batch job that you would recommend to the candidates. But this process is obviously very noisy, and sparse training data can cause latency and also constraints. So I think back in the work they published, they couldn’t really get effective separate résumé content from Mistral and maybe Llama 2. And then they were happy to learn [that] out-of-the-box GPT-4 achieved something like 90% precision and recall. But obviously GPT-4 is more expensive and has close to 30 seconds of inference time, which is much slower.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=681" target="_blank" rel="noreferrer noopener">11:21</a>: So I think what they do is use the distillation concept to fine-tune GPT 3.5 on labeled data, and then distill it into a lightweight BERT-based model using the temperature scale softmax, and they’re able to achieve millisecond latency and a comparable recall-precision trade-off. So I think that’s one of the learnings we see across the industry that the traditional ML techniques still work in the age of AI. And I think we’re going to see a lot more in the production work as well.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=717" target="_blank" rel="noreferrer noopener">11:57</a>: By the way, one of the underappreciated things in the recommendation system space is actually UX in some ways, right? Because basically good UX for delivering the recommendations actually can move the needle. How you actually present your recommendations might make a material difference.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=744" target="_blank" rel="noreferrer noopener">12:24</a>: I think that’s very much true. Although I can’t claim to be an expert on it because I know most recommendation systems deal with monetization, so it’s tricky to put, “Hey, what my user clicks on, like engage, send via social, versus what percentage of that…
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=762" target="_blank" rel="noreferrer noopener">12:42</a>: And it’s also very platform specific. So you can imagine TikTok as one single feed—the recommendation is just on the feed. But YouTube is, you know, the stuff on the side or whatever. And then Amazon is something else. Spotify and Apple [too]. Apple Podcast is something else. But in each case, I think those of us on the outside underappreciate how much these companies invest in the actual interface.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=798" target="_blank" rel="noreferrer noopener">13:18</a>: Yes. And I think there are multiple iterations happening on any day, [so] you might see a different interface than your friends or family because you’re actually being grouped into A/B tests. I think this is very much true of [how] the engagement and performance of the UX have an impact on a lot of the search/rec system as well, beyond the data we just talked about.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=821" target="_blank" rel="noreferrer noopener">13:41</a>: Which brings to mind another topic that is also something I’ve been interested in, over many, many years, which is this notion of experimentation. Many of the most successful companies in the space actually have invested in experimentation tools and experimentation platforms, where people can run experiments at scale. And those experiments can be done much more easily and can be monitored in a much more principled way so that any kind of things they do are backed by data. So I think that companies underappreciate the importance of investing in such a platform.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=868" target="_blank" rel="noreferrer noopener">14:28</a>: I think that’s very much true. A lot of larger companies actually build their own in-house A/B testing experiment or testing frameworks. Meta does; Google has their own and even within different cohorts of products, if you’re monetization, social. . . They have their own niche experimentation platform. So I think that thesis is very much true.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=891" target="_blank" rel="noreferrer noopener">14:51</a>: The last topic I wanted to talk to you about is context engineering. I’ve talked to numerous people about this. So every six months, the context window for these large language models expands. But obviously you can’t just stuff the context window full, because one, it’s inefficient. And two, actually, the LLM can still make mistakes because it’s not going to efficiently process that entire context window anyway. So talk to our listeners about this emerging area called context engineering. And how is that playing out in your own work?
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=938" target="_blank" rel="noreferrer noopener">15:38</a>: I think this is a fascinating topic, where you will hear people passionately say, “RAG is dead.” And it’s really, as you mentioned, [that] our context window gets much, much bigger. Like, for example, back in April, Llama 4 had this staggering 10 million token context window. So the logic behind this argument is quite simple. Like if the model can indeed handle millions of tokens, why not just dump everything instead of doing a retrieval?
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=968" target="_blank" rel="noreferrer noopener">16:08</a>: I think there are quite a few fundamental limitations towards this. I know folks from contextual AI are passionate about this. I think number one is scalability. A lot of times in production, at least, your knowledge base is measured in terabytes or petabytes. So not tokens. So something even larger. And number two I think would be accuracy.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=993" target="_blank" rel="noreferrer noopener">16:33</a>: The effective context windows are very different. Honestly, what we see and then what is advertised in product launches. We see performance degrade long before the model reaches its “official limits.” And then I think number three is probably the efficiency and that kind of aligns with, honestly, our human behavior as well. Like do you read an entire book every time you need to answer one simple question? So I think the context engineering [has] slowly evolved from a buzzword, a few years ago, to now an engineering discipline.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=1035" target="_blank" rel="noreferrer noopener">17:15</a>: I’m appreciative that the context windows are increasing. But at some level, I also acknowledge that to some extent, it’s also kind of a feel-good move on the part of the model builders. So it makes us feel good that we can put more things in there, but it may not actually help us answer the question precisely. Actually, a few years ago, I wrote kind of a tongue-and-cheek post called “<a href="https://gradientflow.substack.com/p/structure-is-all-you-need" target="_blank" rel="noreferrer noopener">Structure Is All You Need</a>.” So basically whatever structure you have, you should help the model, right? If it’s in a SQL database, then maybe you can expose the structure of the data. If it’s a knowledge graph, you leverage whatever structure you have to provide the model better context. So this whole notion of just stuffing the model with as much information, for all the reasons you gave, is valid. But also, philosophically, it doesn’t make any sense to do that anyway.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=1110" target="_blank" rel="noreferrer noopener">18:30</a>: What are the things that you are looking forward to, Faye, in terms of foundation models? What kinds of developments in the foundation model space are you hoping for? And are there any developments that you think are below the radar? 
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=1132" target="_blank" rel="noreferrer noopener">18:52</a>: I think, to better utilize the concept of “contextual engineering,” that they’re essentially two loops. There’s number one within the loop of what happened. Yes. Within the LLMs. And then there’s the outer loop. Like, what can you do as an engineer to optimize a given context window, etc., to get the best results out of the product within the context loop. There are multiple tricks we can do: For example, there’s the vector plus Excel or regex extraction. There’s the metadata fillers. And then for the outer loop—this is a very common practice—people are using LLMs as a reranker, sometimes across the encoder. So the thesis is, hey, why would you overburden an LLM with a 20,000 ranking when there are things you can do to reduce it to top hundred or so? So all of this—context assembly, deduplication, and diversification—would help our production [go] from a prototype to something [that’s] more real time, reliable, and able to scale more infinitely.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=1207" target="_blank" rel="noreferrer noopener">20:07</a>: One of the things I wish—and I don’t know, this is wishful thinking—is maybe if the models can be a little more predictable, that would be nice. By that, I mean, if I ask a question in two different ways, it’ll basically give me the same answer. The foundation model builders can somehow increase predictability and maybe provide us with a little more explanation for how they arrive at the answer. I understand they’re giving us the tokens, and maybe some of the, some of the reasoning models are a little more transparent, but give us an idea of how these things work, because it’ll impact what kinds of applications we’d be comfortable deploying these things in. For example, for agents. If I’m using an agent to use a bunch of tools, but I can’t really predict their behavior, that impacts the types of applications I’d be comfortable using a model for.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=1278" target="_blank" rel="noreferrer noopener">21:18</a>: Yeah, definitely. I very much resonate with this, especially now most engineers have, you know, AI empowered coding tools like Cursor and Windsurf—and as an individual, I very much appreciate the train of thought you mentioned: why an agent does certain things. Why is it navigating between repositories? What are you looking at while you’re doing this call? I think these are very much appreciated. I know there are other approaches—look at Devin, that’s the fully autonomous engineer peer. It just takes things, and you don’t know where it goes. But I think in the near future there will be a nice marriage between the two. Well, now since Windsurf is part of Devin’s parent company.
<a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=1325" target="_blank" rel="noreferrer noopener">22:05</a>: And with that, thank you, Faye. <a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Faye_Zhang_updated.mp3#t=1328" target="_blank" rel="noreferrer noopener">22:08</a>: Awesome. Thank you, Ben.
]]></content:encoded>
<wfw:commentRss>https://www.oreilly.com/radar/podcast/generative-ai-in-the-real-world-faye-zhang-on-using-ai-to-improve-discovery/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
<item>
<title>Prompt Engineering Is Requirements Engineering</title>
<link>https://www.oreilly.com/radar/prompt-engineering-is-requirements-engineering/</link>
<comments>https://www.oreilly.com/radar/prompt-engineering-is-requirements-engineering/#respond</comments>
<pubDate>Wed, 17 Sep 2025 10:27:38 +0000</pubDate>
<dc:creator><![CDATA[Andrew Stellman]]></dc:creator>
<category><![CDATA[AI & ML]]></category>
<category><![CDATA[Commentary]]></category>
<guid isPermaLink="false">https://www.oreilly.com/radar/?p=17463</guid>
<media:content
url="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2019/10/in-dis-big-bang-10a-1400x950.jpg"
medium="image"
type="image/jpeg"
/>
<custom:subtitle><![CDATA[We’ve Been Here Before]]></custom:subtitle>
<description><![CDATA[In the rush to get the most from AI tools, prompt engineering—the practice of writing clear, structured inputs that guide an AI tool’s output—has taken center stage. But for software engineers, the skill isn’t new. We’ve been doing a version of it for decades, just under a different name. The challenges we face when writing […]]]></description>
<content:encoded><![CDATA[
In the rush to get the most from AI tools, prompt engineering—the practice of writing clear, structured inputs that guide an AI tool’s output—has taken center stage. But for software engineers, the skill isn’t new. We’ve been doing a version of it for decades, just under a different name. The challenges we face when writing AI prompts are the same ones software teams have been grappling with for generations. Talking about prompt engineering today is really just continuing a much older conversation about how developers spell out what they need built, under what conditions, with what assumptions, and how to communicate that to the team.
The software crisis was the name given to this problem starting in the late 1960s, especially at the <a href="https://www.google.com/url?q=https://en.wikipedia.org/wiki/NATO_Software_Engineering_Conferences&sa=D&source=docs&ust=1758049259577140&usg=AOvVaw31-sMSlquvRBVJB-EQgmJ3" target="_blank" rel="noreferrer noopener">NATO Software Engineering Conference</a> in 1968, where the term “software engineering” was introduced. The crisis referred to the widespread industry experience that software projects were over budget and late, and often failed to deliver what users actually needed.
There was a common misconception that these failures were due to programmers lacking technical skill or teams who needed more technical training. But the panels at that conference focused on what they saw as the real root cause: Teams and their stakeholders had trouble understanding the problems they were solving and what they actually needed to build; communicating those needs and ideas clearly among themselves; and ensuring the delivered system matched that intent. It was fundamentally a human communication problem.
Participants at the conference captured this precisely. Dr. Edward E. David Jr. from Bell Labs noted there is often no way even to specify in a logically tight way what the software is supposed to do. Douglas Ross from MIT pointed out the pitfall where you can specify what you are going to do, and then do it as if that solved the problem. Prof. W.L. van der Poel summed up the challenge of incomplete specifications: Most problems simply aren’t defined well enough at the start, so you don’t have the information you need to build the right solution.
These are all problems that cause teams to misunderstand the software they’re creating before any code is written. And they should all sound familiar to developers today who work with AI to generate code.
Much of the problem boils down to what I’ve often called the classic “do what I meant, not what I said” problem. Machines are literal—and people on teams often are too. Our intentions are rarely fully spelled out, and getting everyone aligned on what the software is supposed to do has always required deliberate, often difficult work.
Fred Brooks wrote about this in his classic and widely influential “<a href="https://www.google.com/url?q=https://www.researchgate.net/publication/220477127_No_Silver_Bullet_Essence_and_Accidents_of_Software_Engineering&sa=D&source=docs&ust=1758049259576173&usg=AOvVaw3xG4vlWyKDq4_fbxDJWQ7r" target="_blank" rel="noreferrer noopener">No Silver Bullet</a>” essay. He argued there would never be a single magic process or tool that would make software development easy. Throughout the history of software engineering, teams have been tempted to look for that silver bullet that would make the hard parts of understanding and communication go away. It shouldn’t be surprising that we’d see the same problems that plagued software teams for years reappear when they started to use AI tools.
By the end of the 1970s, these problems were being reframed in terms of quality. Philip Crosby, Joseph M. Juran, and W. Edwards Deming, three people who had enormous influence on the field of quality engineering, each had influential takes on why so many products didn’t do the jobs they were supposed to do, and these ideas are especially true when it comes to software. Crosby argued quality was fundamentally conformance to requirements—if you couldn’t define what you needed clearly, you couldn’t ensure it would be delivered. Juran talked about fitness for use—software needed to solve the user’s real problem in its real context, not just pass some checklists. Deming pushed even further, emphasizing that defects weren’t just technical mistakes but symptoms of broken systems, and especially poor communication and lack of shared understanding. He focused on the human side of engineering: creating processes that help people learn, communicate, and improve together.
Through the 1980s, these insights from the quality movement were being applied to software development and started to crystallize into a distinct discipline called requirements engineering, focused on identifying, analyzing, documenting, and managing the needs of stakeholders for a product or system. It emerged as its own field, complete with conferences, methodologies, and professional practices. The IEEE Computer Society formalized this with its first International Symposium on Requirements Engineering in 1993, marking its recognition as a core area of software engineering.
The 1990s became a heyday for requirements work, with organizations investing heavily in formal processes and templates, believing that better documentation formats would ensure better software. Standards like IEEE 830 codified the structure of software requirements specifications, and process models such as the software development life cycle and CMM/CMMI emphasized rigorous documentation and repeatable practices. Many organizations invested heavily in designing detailed templates and forms, hoping that filling them out correctly would guarantee the right system. In practice, those templates were useful for consistency and compliance, but they didn’t eliminate the hard part: making sure what was in one person’s head matched what was in everyone else’s.
While the 1990s focused on formal documentation, the Agile movement of the 2000s shifted toward a more lightweight, conversational approach. User stories emerged as a deliberate counterpoint to heavyweight specifications—short, simple descriptions of functionality told from the user’s perspective, designed to be easy to write and easy to understand. Instead of trying to capture every detail upfront, user stories served as placeholders for conversations between developers and stakeholders. The practice was deliberately simple, based on the idea that shared understanding comes from dialogue, not documentation, and that requirements evolve through iteration and working software rather than being fixed at the project’s start.
All of this reinforced requirements engineering as a legitimate area of software engineering practice and a real career path with its own set of skills. There is now broad agreement that requirements engineering is a vital area of software engineering focused on surfacing assumptions, clarifying goals, and ensuring everyone involved has the same understanding of what needs to be built.
<h2 class="wp-block-heading">Prompt Engineering Is Requirements Engineering</h2>
Prompt engineering and requirements engineering are literally the same skill—using clarity, context, and intentionality to communicate your intent and ensure what gets built matches what you actually need.
User stories were an evolution from traditional formal specifications: a simpler, more flexible approach to requirements but with the same goal of making sure everyone understood the intent. They gained wide acceptance across the industry because they helped teams recognize that requirements are about creating a shared understanding of the project. User stories gave teams a lightweight way to capture intent and then refine it through conversation, iteration, and working software.
Prompt engineering plays the exact same role. The prompt is our lightweight placeholder for a conversation with the AI. We still refine it through iteration, adding context, clarifying intent, and checking the output against what we actually meant. But it’s the full conversation with the AI and its context that matters; the individual prompts are just a means to communicate the intent and context. Just like Agile shifted requirements from static specs to living conversations, prompt engineering shifts our interaction with AI from single-shot commands to an iterative refinement process—though one where we have to infer what’s missing from the output rather than having the AI ask us clarifying questions.
User stories intentionally focused the engineering work back on people and what’s in their heads. Whether it’s a requirements document in Word or a user story in Jira, the most important thing isn’t the piece of paper, ticket, or document we wrote. The most important thing is that what’s in my head matches what’s in your head and matches what’s in the heads of everyone else involved. The piece of paper is just a convenient way to help us figure out whether or not we agree.
Prompt engineering demands the same outcome. Instead of working with teammates to align mental models, we’re communicating to an AI, but the goal hasn’t changed: producing a high-quality product. The basic principles of quality engineering laid out by Deming, Juran, and Crosby have direct parallels in prompt engineering:
<ul class="wp-block-list">
<li>Deming’s focus on systems and communication: Prompting failures can be traced to problems with the process, not the people. They typically stem from poor context and communication, not from “bad AI.”</li>
<li>Juran’s focus on fitness for use: When he framed quality as “fitness for use,” Juran meant that what we produce has to meet real needs—not just look plausible. A prompt is useless if the output doesn’t solve the real problem, and failure to create a prompt that’s fit for use will result in hallucinations.</li>
<li>Crosby’s focus on conformance to requirements: Prompts must specify not just functional needs but also nonfunctional ones like maintainability and readability. If the context and framing aren’t clear, the AI will generate output that conforms to its training distribution rather than the real intent.</li>
</ul>
One of the clearest ways these quality principles show up in prompt engineering is through what’s now called context engineering—deciding what the model needs to see to generate something useful, which typically includes surrounding code, test inputs, expected outputs, design constraints, and other important project information. If you give the AI too little context, it fills in the blanks with what seems most likely based on its training data (which usually isn’t what you had in mind). If you give it too much, it can get buried in information and lose track of what you’re really asking for. That judgment call—what to include, what to leave out—has always been one of the deepest challenges at the heart of requirements work.
There’s another important parallel between requirements engineering and prompt engineering. Back in the 1990s, many organizations fell into what we might call the template trap—believing that the right standardized form or requirements template could guarantee a good outcome. Teams spent huge effort designing and filling out documents. But the real problem was never the format; it was whether the underlying intent was truly shared and understood.
Today, many companies fall into a similar trap with prompt libraries, or catalogs of prewritten prompts meant to standardize practice and remove the difficulty of writing prompts. Prompt libraries can be useful as references or starting points, but they don’t replace the core skill of framing the problem and ensuring shared understanding. Just like a perfect requirements template in the 1990s didn’t guarantee the right system, canned prompts today don’t guarantee the right code.
Decades later, the points Brooks made in his “No Silver Bullet” essay still hold. There’s no single template, library, or tool that can eliminate the essential complexity of understanding what needs to be built. Whether it’s requirements engineering in the 1990s or prompt engineering today, the hard part is always the same: building and maintaining a shared understanding of intent. Tools can help, but they don’t replace the discipline.
AI raises the stakes on this core communication problem. Unlike your teammates, the AI won’t push back or ask questions—it just generates something that looks plausible based on the prompt that it was given. That makes clear communication of requirements even more important.
The alignment of understanding that serves as the foundation of requirements engineering is even more important when we bring AI tools into the project, because AI doesn’t have judgment. It has a huge model, but it only works effectively when directed well. The AI needs the context that we provide in the form of code, documents, and other project information and artifacts, which means the only thing it knows about the project is what we tell it. That’s why it’s especially important to have ways to check and verify that what the AI “knows” really matches what we know.
The classic requirements engineering problems—especially the poor communication and lack of shared understanding that Deming warned about and that requirements engineers and Agile practitioners have spent decades trying to address—are compounded when we use AI. We’re still facing the same issues of communicating intent and specifying requirements clearly. But now those requirements aren’t just for the team to read; they’re used to establish the AI’s context. Small variations in problem framing can have a profound impact on what the AI produces. Using natural language to increasingly replace the structured, unambiguous syntax of code removes a critical guardrail that’s traditionally helped protect software from failed understanding.
The tools of requirements engineering help us make up for that missing guardrail. Agile’s iterative process of the developer understanding requirements, building working software, and continuously reviewing it with the product owner was a check that ensured misunderstandings were caught early. The more we eliminate that extra step of translation and understanding by having AI generate code directly from requirements, the more important it becomes for everyone involved—stakeholders and engineers alike—to have a truly shared understanding of what needs to be built.
When people on teams work together to build software, they spend a lot of time talking and asking questions to understand what they need to build. Working with an AI follows a different kind of feedback cycle—you don’t know it’s missing context until you see what it produces, and you often need to reverse engineer what it did to figure out what’s missing. But both types of interaction require the same fundamental skills around context and communication that requirements engineers have always practiced.
This shows up in practice in several ways:
<ul class="wp-block-list">
<li>Context and shared understanding are foundational. Good requirements help teams understand what behavior matters and how to know when it’s working—capturing both functional requirements (what to build) and nonfunctional requirements (how well it should work). The same distinction applies to prompting but with fewer chances to course-correct. If you leave out something critical, the AI doesn’t push back; it just responds with whatever seems plausible. Sometimes that output looks reasonable until you try to use it and realize the AI was solving a different problem.</li>
<li>Scoping takes real judgment. Developers who struggle to use AI for code typically fall into two extremes: providing too little context (a single sentence that produces something that looks right but fails in practice) or pasting in entire files expecting the model to zoom in on the right method. Unless you explicitly call out what’s important—both functional and nonfunctional requirements—it doesn’t know what matters.</li>
<li>Context drifts, and the model doesn’t know it’s drifted. With human teams, understanding shifts gradually through check-ins and conversations. With prompting, drift can happen in just a few exchanges. The model might still be generating fluent responses until it suggests a fix that makes no sense. That’s a signal that the context has drifted, and you need to reframe the conversation—perhaps by asking the model to explain the code or restate what it thinks it’s doing.</li>
</ul>
History keeps repeating itself: From binders full of scattered requirements to IEEE standards to user stories to today’s prompts, the discipline is the same. We succeed when we treat it as real engineering. Prompt engineering is the next step in the evolution of requirements engineering. It’s how we make sure we have a shared understanding between everyone on the project—including the AI—and it demands the same care, clarity, and deliberate communication we’ve always needed to avoid misunderstandings and build the right thing.
]]></content:encoded>
<wfw:commentRss>https://www.oreilly.com/radar/prompt-engineering-is-requirements-engineering/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
<item>
<title>MCP in Practice</title>
<link>https://www.oreilly.com/radar/mcp-in-practice/</link>
<comments>https://www.oreilly.com/radar/mcp-in-practice/#respond</comments>
<pubDate>Tue, 16 Sep 2025 11:22:59 +0000</pubDate>
<dc:creator><![CDATA[Ilan Strauss, Sruly Rosenblat, Isobel Moure and Tim O’Reilly]]></dc:creator>
<category><![CDATA[AI & ML]]></category>
<category><![CDATA[Research]]></category>
<guid isPermaLink="false">https://www.oreilly.com/radar/?p=17440</guid>
<media:content
url="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2019/12/in-dis-canyon-5b-1400x950.jpg"
medium="image"
type="image/jpeg"
/>
<custom:subtitle><![CDATA[Mapping Power, Concentration, and Usage in the Emerging AI Developer Ecosystem]]></custom:subtitle>
<description><![CDATA[The following was originally published in Asimov’s Addendum, September 11, 2025. Learn more about the AI Disclosures Project here. 1. The Rise and Rise of MCP Anthropic’s Model Context Protocol (MCP) was released in November 2024 as a way to make tools and platforms model-agnostic. MCP works by defining servers and clients. MCP servers are local or remote end […]]]></description>
<content:encoded><![CDATA[
The following was <a href="https://asimovaddendum.substack.com/p/read-write-act-inside-the-mcp-server" target="_blank" rel="noreferrer noopener">originally published in </a><a href="https://asimovaddendum.substack.com/p/read-write-act-inside-the-mcp-server" target="_blank" rel="noreferrer noopener">Asimov’s Addendum</a>, September 11, 2025. Learn more about the AI Disclosures Project <a href="https://www.ssrc.org/programs/ai-disclosures-project/" target="_blank" rel="noreferrer noopener">here</a>.
<h2 class="wp-block-heading">1. The Rise and Rise of MCP</h2>
Anthropic’s<a href="https://www.anthropic.com/news/model-context-protocol" target="_blank" rel="noreferrer noopener"> Model Context Protocol</a> (MCP) was released in November 2024 as a way to make tools and platforms model-agnostic. MCP works by defining servers and clients. MCP servers are local or remote end points where tools and resources are defined. For example, GitHub released an MCP server that allows LLMs to both read from and write to GitHub. MCP clients are the connection from an AI application to MCP servers—they allow an LLM to interact with context and tools from different servers. An example of an MCP client is Claude Desktop, which allows the Claude models to interact with thousands of MCP servers.
In a relatively short time, MCP has become the backbone of hundreds of AI pipelines and applications. Major players like Anthropic and OpenAI have built it into their products. Developer tools such as Cursor (a coding-focused text editor or IDE) and productivity apps like <a href="https://www.raycast.com/EvanZhouDev/mcp" target="_blank" rel="noreferrer noopener">Raycast</a> also use MCP. Additionally, thousands of <a href="https://arxiv.org/abs/2506.13538" target="_blank" rel="noreferrer noopener">developers</a> use it to integrate AI models and access external tools and data without having to build an entire ecosystem from scratch.
In previous work published with AI Frontiers, we argued that <a href="https://ai-frontiers.org/articles/open-protocols-prevent-ai-monopolies" target="_blank" rel="noreferrer noopener">MCP can act</a> as a great unbundler of “context”—the data that helps AI applications provide more relevant answers to consumers. In doing so, it can help decentralize AI markets. We argued that, for MCP to truly achieve its goals, it requires support from:
<ol class="wp-block-list">
<li>Open APIs: So that MCP applications can access third-party tools for agentic use (write actions) and context (read)</li>
<li>Fluid memory: Interoperable LLM memory standards, accessed via MCP-like open protocols, so that the memory context accrued at OpenAI and other leading developers does not get stuck there, preventing downstream innovation</li>
</ol>
We expand upon these two points in a <a href="https://ssrc-static.s3.us-east-1.amazonaws.com/Protocols-and-Power-Moure-OReilly-Strauss_SSRC_08272025.pdf" target="_blank" rel="noreferrer noopener">recent policy note</a>, for those looking to dig deeper.
More generally, we argue that protocols, like MCP, are actually <a href="https://asimovaddendum.substack.com/p/disclosures-i-do-not-think-that-word" target="_blank" rel="noreferrer noopener">foundational “rules of the road” for AI markets</a>, whereby open disclosure and communication standards are built into the network itself, rather than imposed after the fact by regulators. Protocols are fundamentally market-shaping devices, architecting markets through the permissions, rules, and interoperability of the network itself. They can have a big impact on how the commercial markets built on top of them function too.
<h3 class="wp-block-heading">1.1 But how is the MCP ecosystem evolving?</h3>
Yet we don’t have a clear idea of the shape of the MCP ecosystem today. What are the most common use cases of MCP? What sort of access is being given by MCP servers and used by MCP clients? Is the data accessed via MCP “read-only” for context, or does it allow agents to “write” and interact with it—for example, by editing files or sending emails?
To begin answering these questions, we look at the tools and context which AI agents use via MCP servers. This gives us a clue about what is being built and what is getting attention. In this article, we don’t analyze MCP clients—the applications that use MCP servers. We instead limit our analysis to what MCP servers are making available for building.
We assembled a large dataset of MCP servers (n = 2,874), scraped from <a href="https://www.pulsemcp.com/" target="_blank" rel="noreferrer noopener">Pulse</a>.1 We then enriched it with GitHub star-count data on each server. On GitHub, stars are similar to Facebook “likes,” and <a href="https://homepages.dcc.ufmg.br/~mtov/pub/2018-jss-github-stars.pdf" target="_blank" rel="noreferrer noopener">developers use them</a> to show appreciation, bookmark projects, or indicate usage.
In practice, while there were plenty of MCP servers, we found that the top few garnered most of the attention and, likely by extension, most of the use. Just the top 10 servers had nearly half of all GitHub stars given to MCP servers.
Some of our takeaways are:
<ol class="wp-block-list">
<li>MCP usage appears to be fairly concentrated. This means that, if left unchecked, a small number of servers and (by extension) APIs could have outsize control over the MCP ecosystem being created.</li>
<li>MCP use (tools and data being accessed) is dominated by just three categories: Database & Search (RAG), Computer & Web Automation, and Software Engineering. Together, they received nearly three-quarters (72.6%) of all stars on GitHub (which we proxy for usage).</li>
<li>Most MCP servers support both read (access context) and write (change context) operations, showing that developers want their agents to be able to act on context, not just consume it.</li>
</ol>
<h2 class="wp-block-heading">2. Findings</h2>
To start with, we analyzed the MCP ecosystem for concentration risk.
<h3 class="wp-block-heading">2.1 MCP server use is concentrated</h3>
We found that MCP usage is concentrated among several key MCP servers, judged by the number of GitHub stars each repo received.
Despite there being thousands of MCP servers, the top 10 servers make up nearly half (45.7%) of all GitHub stars given to MCP servers (pie chart below) and the top 10% of servers make up 88.3% of all GitHub stars (not shown).
<figure class="wp-block-image size-full"><img fetchpriority="high" decoding="async" width="1444" height="742" src="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-2.png" alt="The top 10 servers received 45.7% of all GitHub stars in our dataset of 2,874 servers." class="wp-image-17441" title="Chart" srcset="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-2.png 1444w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-2-300x154.png 300w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-2-768x395.png 768w" sizes="(max-width: 1444px) 100vw, 1444px" /><figcaption class="wp-element-caption">The top 10 servers received 45.7% of all GitHub stars in our dataset of 2,874 servers.</figcaption></figure>
This means that the majority of real-world MCP users are likely relying on the same few services made available via a handful of APIs. This concentration likely stems from network effects and practical utility: All developers gravitate toward servers that solve universal problems like web browsing, database access, and integration with widely used platforms like GitHub, Figma, and Blender. This concentration pattern seems typical of developer-tool ecosystems. A few well-executed, broadly applicable solutions tend to dominate. Meanwhile, more specialized tools occupy smaller niches.
<h3 class="wp-block-heading">2.2 The top 10 MCP servers really matter</h3>
Next, the top 10 MCP servers are shown in the table below, along with their star count and what they do.
Among the top 10 MCP servers, GitHub, Repomix, Context7, and Framelink are built to assist with software development: Context7 and Repomix by gathering context, GitHub by allowing agents to interact with projects, and Framelink by passing on the design specifications from Figma directly to the model. The Blender server allows agents to create 3D models of anything, using the popular open source Blender application. Finally, Activepieces and MindsDB connect the agent to multiple APIs with one standardized interface: in MindsDB’s case, primarily to read data from databases, and in Activepieces to automate services.
<figure class="wp-block-image size-full"><img decoding="async" width="1296" height="1260" src="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-3.png" alt="The top 10 MCP servers with short descriptions, design courtesy of Claude." class="wp-image-17442" srcset="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-3.png 1296w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-3-300x292.png 300w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-3-768x747.png 768w" sizes="(max-width: 1296px) 100vw, 1296px" /><figcaption class="wp-element-caption">The top 10 MCP servers with short descriptions, design courtesy of Claude.</figcaption></figure>
The dominance of agentic browsing, in the form of Browser Use (61,000 stars) and Playwright MCP (18,425 stars), stands out. This reflects the fundamental need for AI systems to interact with web content. These tools allow AI to navigate websites, click buttons, fill out forms, and extract data just like a human would. Agentic browsing has surged, even though it’s far less token-efficient than calling an API. Browsing agents often need to wade through multiple pages of boilerplate to extract slivers of data a single API request could return. Because many services lack usable APIs or tightly gate them, browser-based agents are often the simplest—sometimes the only—way to integrate, underscoring the limits of today’s APIs.
Some of the top servers are unofficial. Both the Framelink and Blender MCP are servers that interact with just a single application, but they are both “unofficial” products. This means that they are not officially endorsed by the developers of the application they are integrating with—those who own the underlying service or API (e.g., GitHub, Slack, Google). Instead, they are built by independent developers who create a bridge between an AI client and a service—often by reverse-engineering APIs, wrapping unofficial SDKs, or using browser automation to mimic user interactions.
It is healthy that third-party developers can build their own MCP servers, since this openness encourages innovation. But it also introduces an intermediary layer between the user and the API, which brings risks around trust, verification, and even potential abuse. With open source local servers, the code is transparent and can be vetted. By contrast, remote third-party servers are harder to audit, since users must trust code they can’t easily inspect.
At a deeper level, the repos that currently dominate MCP servers highlight three encouraging facts about the MCP ecosystem:
<ol class="wp-block-list">
<li>First, several prominent MCP servers support multiple third-party services for their functionality. MindsDB and Activepieces serve as gateways to multiple (often competing) service providers through a single server. MindsDB allows developers to query different databases like PostgreSQL, MongoDB, and MySQL through a single interface, while Taskmaster allows the agent to delegate tasks to a range of AI models from OpenAI, Anthropic, and Google, all without changing servers.</li>
<li>Second, agentic browsing MCP servers are being used to get around potentially restrictive APIs. As noted above, Browser Use and Playwright access internet services through a web browser, helping to bypass API restrictions, but they instead run up against anti-bot protections. This circumvents the limitations that APIs can impose on what developers are able to build.</li>
<li>Third, some MCP servers do their processing on the developer’s computer (locally), making them less dependent on a vendor maintaining API access. Some MCP servers examined here can run entirely on a local computer without sending data to the cloud—meaning that no gatekeeper has the power to cut you off. Of the 10 MCP servers examined above, only Framelink, Context7, and GitHub rely on just a single cloud-only API dependency that can’t be run locally end-to-end on your machine. Blender and Repomix are completely open source and don’t require any internet access to work, while MindsDB, Browser Use, and Activepieces have local open source implementations.</li>
</ol>
<h3 class="wp-block-heading">2.3 The three categories that dominate MCP use</h3>
Next, we grouped MCP servers into different categories based on their functionality.
When we analyzed what types of servers are most popular, we found that three dominated: Computer & Web Automation (24.8%), Software Engineering (24.7%), and Database & Search (23.1%).
<figure class="wp-block-image size-full"><img decoding="async" width="1456" height="683" src="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-4.png" alt="Software engineering, computer and web automation, and database and search received 72.6% of all stars given to MCP servers." class="wp-image-17443" title="Chart" srcset="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-4.png 1456w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-4-300x141.png 300w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-4-768x360.png 768w" sizes="(max-width: 1456px) 100vw, 1456px" /><figcaption class="wp-element-caption">Software Engineering, Computer & Web Automation, and Database & Search received 72.6% of all stars given to MCP servers.</figcaption></figure>
Widespread use of Software Engineering (24.7%) MCP servers aligns with <a href="https://arxiv.org/abs/2503.04761" target="_blank" rel="noreferrer noopener">Anthropic’s economic index</a>, which found that an outsize portion of AI interactions were related to software development.
The popularity of both Computer & Web Automation (24.8%) and Database & Search (23.1%) also makes sense. Before the advent of MCP, web scraping and database search were highly integrated applications across platforms like ChatGPT, Perplexity, and Gemini. With MCP, however, users can now access that same search functionality and connect their agents to any database with minimal effort. In other words, MCP’s <a href="https://ai-frontiers.org/articles/open-protocols-prevent-ai-monopolies" target="_blank" rel="noreferrer noopener">unbundling</a> effect is highly visible here.
<h3 class="wp-block-heading">2.4 Agents interact with their environments</h3>
Lastly, we analyzed the capabilities of these servers: Are they allowing AI applications just to access data and tools (read), or instead do agentic operations with them (write)?
Across all but two of the MCP server categories looked at, the most popular MCP servers supported both reading (access context) and writing (agentic) operations—shown in turquoise. The prevalence of servers with combined read and write access suggests that agents are not being built just to answer questions based on data but also to take action and interact with services on a user’s behalf.
<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="1456" height="974" src="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-5.png" alt="Showing MCP servers by category. Dotted red line at 10,000 stars (likes). The most popular servers support both read and write operations by agents. In contrast, almost no servers support just write operations." class="wp-image-17444" srcset="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-5.png 1456w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-5-300x201.png 300w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-5-768x514.png 768w" sizes="auto, (max-width: 1456px) 100vw, 1456px" /><figcaption class="wp-element-caption">Showing MCP servers by category. Dotted red line at 10,000 stars (likes). The most popular servers support both read and write operations by agents. In contrast, almost no servers support just write operations.</figcaption></figure>
The two exceptions are Database & Search (RAG) and Finance MCP servers, in which read-only access is a common permission given. This is likely because data integrity is critical to ensuring reliability.
<h2 class="wp-block-heading">3. The Importance of Multiple Access Points</h2>
A few implications of our analysis can be drawn out at this preliminary stage.
First, concentrated MCP server use compounds the risks of API access being restricted. As we discussed in “<a href="https://asimovaddendum.substack.com/p/protocols-and-power" target="_blank" rel="noreferrer noopener">Protocols and Power</a>,” MCP remains constrained by “what a particular service (such as GitHub or Slack) happens to expose through its API.” A few powerful digital service providers have the power to shut down access to their servers.
One important hedge against API gatekeeping is that many of the top servers try not to rely on a single provider. In addition, the following two safeguards are relevant:
<ul class="wp-block-list">
<li>They offer local processing of data on a user’s machine whenever possible, instead of sending the data for processing to a third-party server. Local processing ensures that functionality cannot be restricted.</li>
<li>If running a service locally is not possible (e.g., email or web search), the server should still support multiple avenues of getting at the needed context through competing APIs. For example, MindsDB functions as a gateway to multiple data sources, so instead of relying on just one database to read and write data, it goes to great lengths to support multiple databases in one unified interface, essentially making the backend tools interchangeable.</li>
</ul>
Second, our analysis points to the fact that current restrictive API access policies are not sustainable. Web scraping and bots, accessed via MCP servers, are probably being used (at least in part) to circumvent overly restrictive API access, complicating the <a href="https://slate.com/technology/2025/08/uk-online-safety-act-reddit-wikipedia-open-internet.html" target="_blank" rel="noreferrer noopener">increasingly common</a> practice of banning bots. Even OpenAI is coloring outside the API lines, using a third-party service to access Google Search’s results through web scraping, thereby <a href="https://www.theinformation.com/articles/openai-challenging-google-using-search-data" target="_blank" rel="noreferrer noopener">circumventing its restrictive API</a>.
Expanding structured API access in a meaningful way is vital. This ensures that legitimate AI automation runs through stable, documented end points. Otherwise, developers resort to brittle browser automation where privacy and authorization have not been properly addressed. Regulatory guidance <a href="https://ai-frontiers.org/articles/open-protocols-prevent-ai-monopolies" target="_blank" rel="noreferrer noopener">could push</a> the market in this direction, as with open banking in the US.
Finally, encouraging greater transparency and disclosure could help identify where the bottlenecks in the MCP ecosystem are.
<ul class="wp-block-list">
<li>Developers operating popular MCP servers (above a certain usage threshold) or providing APIs used by top servers should report usage statistics, access denials, and rate-limiting policies. This data would help regulators identify emerging bottlenecks before they become entrenched. GitHub might facilitate this by encouraging these disclosures, for example.</li>
<li>Additionally, MCP servers above certain usage thresholds should clearly list their dependencies on external APIs and what fallback options exist if the primary APIs become unavailable. This is not only helpful in determining the market structure, but also essential information for security and robustness for downstream applications.</li>
</ul>
The goal is not to eliminate all concentration in the network but to ensure that the MCP ecosystem remains contestable, with multiple viable paths for innovation and user choice. By addressing both technical architecture and market dynamics, these suggested tweaks could help MCP achieve its potential as a democratizing force in AI development, rather than merely shifting bottlenecks from one layer to another.
<hr class="wp-block-separator has-alpha-channel-opacity is-style-wide"/>
<h2 class="wp-block-heading">Footnotes</h2>
<ol class="wp-block-list">
<li>For this analysis, we categorized each repo into one of 15 categories using GPT-5 mini. We then human-reviewed and edited the top 50 servers that make up around 70% of the total star count in our dataset.</li>
</ol>
<hr class="wp-block-separator has-alpha-channel-opacity is-style-wide"/>
<h2 class="wp-block-heading">Appendix</h2>
<h3 class="wp-block-heading">Dataset</h3>
The full dataset, along with descriptions of the categories, can be found here (constructed by Sruly Rosenblat):
<a href="https://huggingface.co/datasets/sruly/MCP-In-Practice" target="_blank" rel="noreferrer noopener">https://huggingface.co/datasets/sruly/MCP-In-Practice</a>
<h3 class="wp-block-heading">Limitations</h3>
There are a few limitations to our preliminary research:
<ul class="wp-block-list">
<li>GitHub stars aren’t a measure of download counts or even necessarily a repo’s popularity.</li>
<li>Only the name and description were used when categorizing repos with the LLM.</li>
<li>Categorization was subject to both human and AI errors and many servers would likely fit into multiple categories.</li>
<li>We only used the Pulse list for our dataset; other lists had different servers (e.g., Browser Use isn’t on mcpmarket.com).</li>
<li>We excluded some repos from our analysis, such as those that had multiple servers and those we weren’t able to fetch the star count for. We may miss some popular servers by doing this.</li>
</ul>
<h3 class="wp-block-heading">MCP Server Use Over Time</h3>
<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="1456" height="916" src="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-6.png" alt="The growth of the top nine repos’ star count over time from MCP’s launch date on November 25, 2024, until September 2025. NOTE: We were only able to track the Browser-Use’s repo until 40,000 stars; hence the flat line for its graph. In reality, roughly 21,000 stars were added over the next few months (the other graphs in this blog are properly adjusted)." class="wp-image-17445" srcset="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-6.png 1456w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-6-300x189.png 300w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-6-768x483.png 768w" sizes="auto, (max-width: 1456px) 100vw, 1456px" /><figcaption class="wp-element-caption">The growth of the top nine repos’ star count over time from MCP’s launch date on November 25, 2024, until September 2025. Note: We were only able to track Browser Use’s repo until 40,000 stars; hence the flat line for its graph. In reality, roughly 21,000 stars were added over the next few months. (The other graphs in this post are properly adjusted.)</figcaption></figure>

]]></content:encoded>
<wfw:commentRss>https://www.oreilly.com/radar/mcp-in-practice/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
<item>
<title>When AI Writes Code, Who Secures It?</title>
<link>https://www.oreilly.com/radar/when-ai-writes-code-who-secures-it/</link>
<comments>https://www.oreilly.com/radar/when-ai-writes-code-who-secures-it/#respond</comments>
<pubDate>Mon, 15 Sep 2025 10:37:10 +0000</pubDate>
<dc:creator><![CDATA[Chloé Messdaghi]]></dc:creator>
<category><![CDATA[AI & ML]]></category>
<category><![CDATA[Security]]></category>
<category><![CDATA[Commentary]]></category>
<guid isPermaLink="false">https://www.oreilly.com/radar/?p=17436</guid>
<media:content
url="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2019/06/hacker-1944688_crop-b34a76e3cab9c07c5900b706c70a12c3-1.jpg"
medium="image"
type="image/jpeg"
/>
<description><![CDATA[In early 2024, a striking deepfake fraud case in Hong Kong brought the vulnerabilities of AI-driven deception into sharp relief. A finance employee was duped during a video call by what appeared to be the CFO—but was, in fact, a sophisticated AI-generated deepfake. Convinced of the call’s authenticity, the employee made 15 transfers totaling over […]]]></description>
<content:encoded><![CDATA[
In early 2024, a striking deepfake fraud case in Hong Kong brought the vulnerabilities of AI-driven deception into sharp relief. A finance employee was duped during a video call by what appeared to be the CFO—but was, in fact, a sophisticated AI-generated deepfake. Convinced of the call’s authenticity, the employee made <a href="https://www.ft.com/content/b977e8d4-664c-4ae4-8a8e-eb93bdf785ea?" target="_blank" rel="noreferrer noopener">15 transfers totaling over $25 million</a> to fraudulent bank accounts before realizing it was a scam.
This incident exemplifies more than just technological trickery—it signals how trust in what we see and hear can be weaponized, especially as AI becomes more deeply integrated into enterprise tools and workflows. From embedded LLMs in enterprise systems to autonomous agents diagnosing and even repairing issues in live environments, AI is transitioning from novelty to necessity. Yet as it evolves, so too do the gaps in our traditional security frameworks—designed for static, human-written code—revealing just how unprepared we are for systems that generate, adapt, and behave in unpredictable ways.
<h2 class="wp-block-heading">Beyond the CVE Mindset</h2>
Traditional secure coding practices revolve around known vulnerabilities and patch cycles. AI changes the equation. A line of code can be generated on the fly by a model, shaped by manipulated prompts or data—creating new, unpredictable categories of risk like prompt injection or emergent behavior outside traditional taxonomies.
A 2025 Veracode study found that <a href="https://www.techradar.com/pro/nearly-half-of-all-code-generated-by-ai-found-to-contain-security-flaws-even-big-llms-affected?" target="_blank" rel="noreferrer noopener">45% of all AI-generated code contained vulnerabilities</a>, with common flaws like weak defenses against XSS and log injection. (Some languages performed more poorly than others. Over 70% of AI-generated Java code had a security issue, for instance.) Another 2025 study showed that repeated refinement can make things worse: After just five iterations, critical vulnerabilities rose by <a href="https://arxiv.org/abs/2506.11022?" target="_blank" rel="noreferrer noopener">37.6%</a>.
To keep pace, frameworks like the <a href="https://owasp.org/www-project-top-10-for-large-language-model-applications/" target="_blank" rel="noreferrer noopener">OWASP Top 10 for LLMs</a> have emerged, cataloging AI-specific risks such as data leakage, model denial of service, and prompt injection. They highlight how current security taxonomies fall short—and why we need new approaches that model AI threat surfaces, share incidents, and iteratively refine risk frameworks to reflect how code is created and influenced by AI.
<h2 class="wp-block-heading">Easier for Adversaries</h2>
Perhaps the most alarming shift is how AI lowers the barrier to malicious activity. What once required deep technical expertise can now be done by anyone with a clever prompt: generating scripts, launching phishing campaigns, or manipulating models. AI doesn’t just broaden the attack surface; it makes it easier and cheaper for attackers to succeed without ever writing code.
In 2025, researchers unveiled PromptLocker, the first AI-powered ransomware. Though only a proof of concept, it showed how theft and encryption could be automated with a local LLM at remarkably low cost: about <a href="https://www.tomshardware.com/tech-industry/cyber-security/ai-powered-promptlocker-ransomware-is-just-an-nyu-research-project-the-code-worked-as-a-typical-ransomware-selecting-targets-exfiltrating-selected-data-and-encrypting-volumes" target="_blank" rel="noreferrer noopener">$0.70 per full attack using commercial APIs</a>—and essentially free with open source models. That kind of affordability could make ransomware cheaper, faster, and more scalable than ever.
This democratization of offense means defenders must prepare for attacks that are more frequent, more varied, and more creative. The <a href="https://github.com/mitre/advmlthreatmatrix" target="_blank" rel="noreferrer noopener">Adversarial ML Threat Matrix</a>, founded by Ram Shankar Siva Kumar during his time at Microsoft, helps by enumerating threats to machine learning and offering a structured way to anticipate these evolving risks. (He’ll be discussing the difficulty of securing AI systems from adversaries at <a href="https://learning.oreilly.com/live-events/security-superstream-secure-code-in-the-age-of-ai/0642572204099/" target="_blank" rel="noreferrer noopener">O’Reilly’s upcoming Security Superstream</a>.)
<h2 class="wp-block-heading">Silos and Skill Gaps</h2>
Developers, data scientists, and security teams still work in silos, each with different incentives. Business leaders push for rapid AI adoption to stay competitive, while security leaders warn that moving too fast risks catastrophic flaws in the code itself.
These tensions are amplified by a widening skills gap: Most developers lack training in AI security, and many security professionals don’t fully understand how LLMs work. As a result, the old patchwork fixes feel increasingly inadequate when the models are writing and running code on their own.
The rise of “vibe coding”—relying on LLM suggestions without review—captures this shift. It accelerates development but introduces hidden vulnerabilities, leaving both developers and defenders struggling to manage novel risks.
<h2 class="wp-block-heading">From Avoidance to Resilience</h2>
AI adoption won’t stop. The challenge is moving from avoidance to resilience. Frameworks like <a href="https://www.databricks.com/blog/announcing-databricks-ai-security-framework-20" target="_blank" rel="noreferrer noopener">Databricks’ AI Risk Framework (DASF)</a> and the <a href="https://www.nist.gov/itl/ai-risk-management-framework" target="_blank" rel="noreferrer noopener">NIST AI Risk Management Framework</a> provide practical guidance on embedding governance and security directly into AI pipelines, helping organizations move beyond ad hoc defenses toward systematic resilience. The goal isn’t to eliminate risk but to enable innovation while maintaining trust in the code AI helps produce.
<h2 class="wp-block-heading">Transparency and Accountability</h2>
Research shows AI-generated code is often simpler and more repetitive, <a href="https://arxiv.org/abs/2508.21634" target="_blank" rel="noreferrer noopener">but also more vulnerable</a>, with risks like hardcoded credentials and path traversal exploits. Without observability tools such as prompt logs, provenance tracking, and audit trails, developers can’t ensure reliability or accountability. In other words, AI-generated code is more likely to introduce high-risk security vulnerabilities.
AI’s opacity compounds the problem: A function may appear to “work” yet conceal vulnerabilities that are difficult to trace or explain. Without explainability and safeguards, autonomy quickly becomes a recipe for insecure systems. Tools like <a href="https://atlas.mitre.org/matrices/ATLAS" target="_blank" rel="noreferrer noopener">MITRE ATLAS</a> can help by mapping adversarial tactics against AI models, offering defenders a structured way to anticipate and counter threats.
<h2 class="wp-block-heading">Looking Ahead</h2>
Securing code in the age of AI requires more than patching—it means breaking silos, closing skill gaps, and embedding resilience into every stage of development. The risks may feel familiar, but AI scales them dramatically. Frameworks like Databricks’ AI Risk Framework (DASF) and the NIST AI Risk Management Framework provide structures for governance and transparency, while MITRE ATLAS maps adversarial tactics and real-world attack case studies, giving defenders a structured way to anticipate and mitigate threats to AI systems.
The choices we make now will determine whether AI becomes a trusted partner—or a shortcut that leaves us exposed.
<hr class="wp-block-separator has-alpha-channel-opacity is-style-wide"/>
Ensure your systems remain secure in an increasingly AI-driven world Join Chloé Messdaghi and a lineup of top security professionals and technologists for O’Reilly’s Security Superstream: Secure Code in the Age of AI. They’ll share practical insights, real-world experiences, and emerging trends that will help you code more securely, build and deploy secure models, and protect against AI-specific threats. It’s free for O’Reilly members. <a href="https://learning.oreilly.com/live-events/security-superstream-secure-code-in-the-age-of-ai/0642572204099/" target="_blank" rel="noreferrer noopener">Save your seat here</a>. Not a member? <a href="https://www.oreilly.com/start-trial/" target="_blank" rel="noreferrer noopener">Sign up for a free 10-day trial</a> to attend—and check out all the other great resources on O’Reilly.

]]></content:encoded>
<wfw:commentRss>https://www.oreilly.com/radar/when-ai-writes-code-who-secures-it/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
<item>
<title>Taming Chaos with Antifragile GenAI Architecture</title>
<link>https://www.oreilly.com/radar/taming-chaos-with-antifragile-genai-architecture/</link>
<comments>https://www.oreilly.com/radar/taming-chaos-with-antifragile-genai-architecture/#respond</comments>
<pubDate>Thu, 11 Sep 2025 10:49:38 +0000</pubDate>
<dc:creator><![CDATA[Shreshta Shyamsundar]]></dc:creator>
<category><![CDATA[AI & ML]]></category>
<category><![CDATA[Commentary]]></category>
<guid isPermaLink="false">https://www.oreilly.com/radar/?p=17429</guid>
<media:content
url="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/AI-in-Chaos-2.jpg"
medium="image"
type="image/jpeg"
/>
<custom:subtitle><![CDATA[Turn Volatility into Your Greatest Strategic Asset]]></custom:subtitle>
<description><![CDATA[What if uncertainty wasn’t something to simply endure but something to actively exploit? The convergence of Nassim Taleb’s antifragility principles with generative AI capabilities is creating a new paradigm for organizational design powered by generative AI—one where volatility becomes fuel for competitive advantage rather than a threat to be managed. The Antifragility Imperative Antifragility transcends […]]]></description>
<content:encoded><![CDATA[
What if uncertainty wasn’t something to simply endure but something to actively exploit? The convergence of <a href="https://en.wikipedia.org/wiki/Antifragile_(book)" target="_blank" rel="noreferrer noopener">Nassim Taleb’s antifragility principles</a> with generative AI capabilities is creating a new paradigm for organizational design powered by generative AI—one where volatility becomes fuel for competitive advantage rather than a threat to be managed.
<h2 class="wp-block-heading">The Antifragility Imperative</h2>
Antifragility transcends resilience. While resilient systems bounce back from stress and robust systems resist change, antifragile systems actively improve when exposed to volatility, randomness, and disorder. This isn’t just theoretical—it’s a mathematical property where systems exhibit positive convexity, gaining more from favorable variations than they lose from unfavorable ones.
To visualize the concept of positive convexity in antifragile systems, consider a graph where the x-axis represents stress or volatility and the y-axis represents the system’s response. In such systems, the curve is upward bending (convex), demonstrating that the system gains more from positive shocks than it loses from negative ones—by an accelerating margin.
The convex (upward-curving) line shows that small positive shocks yield increasingly larger gains, while equivalent negative shocks cause comparatively smaller losses.
For comparison, a straight line representing a fragile or linear system shows a proportional (linear) response, with gains and losses of equal magnitude on either side.
<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="1600" height="1066" src="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-1.png" alt="Graph illustrating positive convexity: Antifragile systems benefit disproportionately from positive variations compared to equivalent negative shocks." class="wp-image-17430" srcset="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-1.png 1600w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-1-300x200.png 300w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-1-768x512.png 768w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-1-1536x1023.png 1536w" sizes="auto, (max-width: 1600px) 100vw, 1600px" /><figcaption class="wp-element-caption">Graph illustrating positive convexity: Antifragile systems benefit disproportionately from positive variations compared to equivalent negative shocks.</figcaption></figure>
The concept emerged from Taleb’s observation that certain systems don’t just survive Black Swan events—they thrive because of them. Consider how Amazon’s supply chain AI during the 2020 pandemic demonstrated true antifragility. When lockdowns disrupted normal shipping patterns and consumer behavior shifted dramatically, Amazon’s demand forecasting systems didn’t just adapt; they used the chaos as training data. Every stockout, every demand spike for unexpected products like webcams and exercise equipment, every supply chain disruption became input for improving future predictions. The AI learned to identify early signals of changing consumer behavior and supply constraints, making the system more robust for future disruptions.
For technology organizations, this presents a fundamental question: How do we design systems that don’t just survive unexpected events but benefit from them? The answer lies in implementing specific generative AI architectures that can learn continuously from disorder.
<h2 class="wp-block-heading">Generative AI: Building Antifragile Capabilities</h2>
Certain generative AI implementations can exhibit antifragile characteristics when designed with continuous learning architectures. Unlike static models deployed once and forgotten, these systems incorporate feedback loops that allow real-time adaptation without full model retraining—a critical distinction given the resource-intensive nature of training large models.
Netflix’s recommendation system demonstrates this principle. Rather than retraining its entire foundation model, the company continuously updates personalization layers based on user interactions. When users reject recommendations or abandon content midstream, this negative feedback becomes valuable training data that refines future suggestions. The system doesn’t just learn what users like. It becomes expert at recognizing what they’ll hate, leading to higher overall satisfaction through accumulated negative knowledge.
The key insight is that these AI systems don’t just adapt to new conditions; they actively extract information from disorder. When market conditions shift, customer behavior changes, or systems encounter edge cases, properly designed generative AI can identify patterns in the chaos that human analysts might miss. They transform noise into signal, volatility into opportunity.
<h2 class="wp-block-heading">Error as Information: Learning from Failure</h2>
Traditional systems treat errors as failures to be minimized. Antifragile systems treat errors as information sources to be exploited. This shift becomes powerful when combined with generative AI’s ability to learn from mistakes and generate improved responses.
IBM Watson for Oncology’s failure has been attributed to synthetic data problems, but it highlights a critical distinction: Synthetic data isn’t inherently problematic—it’s essential in healthcare where patient privacy restrictions limit access to real data. The issue was that Watson was trained exclusively on synthetic, hypothetical cases created by Memorial Sloan Kettering physicians rather than being validated against diverse real-world outcomes. This created a dangerous feedback loop where the AI learned physician preferences rather than evidence-based medicine.
When deployed, Watson recommended potentially fatal treatments—such as prescribing bevacizumab to a 65-year-old lung cancer patient with severe bleeding, despite the drug’s known risk of causing “severe or fatal hemorrhage.” A truly antifragile system would have incorporated mechanisms to detect when its training data diverged from reality—for instance, by tracking recommendation acceptance rates and patient outcomes to identify systematic biases.
This challenge extends beyond healthcare. Consider AI diagnostic systems deployed across different hospitals. A model trained on high-end equipment at a research hospital performs poorly when deployed to field hospitals with older, poorly calibrated CT scanners. An antifragile AI system would treat these equipment variations not as problems to solve but as valuable training data. Each “failed” diagnosis on older equipment becomes information that improves the system’s robustness across diverse deployment environments.
<h2 class="wp-block-heading">Netflix: Mastering Organizational Antifragility</h2>
Netflix’s approach to chaos engineering exemplifies organizational antifragility in practice. The company’s famous “Chaos Monkey” randomly terminates services in production to ensure the system can handle failures gracefully. But more relevant to generative AI is its content recommendation system’s sophisticated approach to handling failures and edge cases.
When Netflix’s AI began recommending mature content to family accounts rather than simply adding filters, its team created systematic “chaos scenarios”—deliberately feeding the system contradictory user behavior data to stress-test its decision-making capabilities. They simulated situations where family members had vastly different viewing preferences on the same account or where content metadata was incomplete or incorrect.
The recovery protocols the team developed go beyond simple content filtering. Netflix created hierarchical safety nets: real-time content categorization, user context analysis, and human oversight triggers. Each “failure” in content recommendation becomes data that strengthens the entire system. The AI learns what content to recommend but also when to seek additional context, when to err on the side of caution, and how to gracefully handle ambiguous situations.
This demonstrates a key antifragile principle: The system doesn’t just prevent similar failures—it becomes more intelligent about handling edge cases it has never encountered before. Netflix’s recommendation accuracy improved precisely because the system learned to navigate the complexities of shared accounts, diverse family preferences, and content boundary cases.
<h2 class="wp-block-heading">Technical Architecture: The LOXM Case Study</h2>
JPMorgan’s LOXM (Learning Optimization eXecution Model) represents the most sophisticated example of antifragile AI in production. Developed by the global equities electronic trading team under Daniel Ciment, LOXM went live in 2017 after training on billions of historical transactions. While this predates the current era of transformer-based generative AI, LOXM was built using deep learning techniques that share fundamental principles with today’s generative models: the ability to learn complex patterns from data and adapt to new situations through continuous feedback.
Multi-agent architecture: LOXM uses a reinforcement learning system where specialized agents handle different aspects of trade execution.
<ul class="wp-block-list">
<li>Market microstructure analysis agents learn optimal timing patterns.</li>
<li>Liquidity assessment agents predict order book dynamics in real time.</li>
<li>Impact modeling agents minimize market disruption during large trades.</li>
<li>Risk management agents enforce position limits while maximizing execution quality.</li>
</ul>
Antifragile performance under stress: While traditional trading algorithms struggled with unprecedented conditions during the market volatility of March 2020, LOXM’s agents used the chaos as learning opportunities. Each failed trade execution, each unexpected market movement, each liquidity crisis became training data that improved future performance.
The measurable results were striking. LOXM improved execution quality by 50% during the most volatile trading days—exactly when traditional systems typically degrade. This isn’t just resilience; it’s mathematical proof of positive convexity where the system gains more from stressful conditions than it loses.
Technical innovation: LOXM prevents catastrophic forgetting through “experience replay” buffers that maintain diverse trading scenarios. When new market conditions arise, the system can reference similar historical patterns while adapting to novel situations. The feedback loop architecture uses streaming data pipelines to capture trade outcomes, model predictions, and market conditions in real time, updating model weights through online learning algorithms within milliseconds of trade completion.
<h2 class="wp-block-heading">The Information Hiding Principle</h2>
<a href="https://www.google.com/url?q=https://en.wikipedia.org/wiki/Information_hiding&sa=D&source=docs&ust=1757588683474584&usg=AOvVaw0GSAvIFboxsnVPkkNh7N8A" target="_blank" rel="noreferrer noopener">David Parnas’s information hiding principle</a> directly enables antifragility by ensuring that system components can adapt independently without cascading failures. In <a href="https://www.google.com/url?q=https://dl.acm.org/doi/10.1145/361598.361623&sa=D&source=docs&ust=1757588683474722&usg=AOvVaw3U2j64n0pCWduXPdTRr_Mj" target="_blank" rel="noreferrer noopener">his 1972 paper</a>, Parnas emphasized hiding “design decisions likely to change”—exactly what antifragile systems need.
When LOXM encounters market disruption, its modular design allows individual components to adapt their internal algorithms without affecting other modules. The “secret” of each module—its specific implementation—can evolve based on local feedback while maintaining stable interfaces with other components.
This architectural pattern prevents what Taleb calls “tight coupling”—where stress in one component propagates throughout the system. Instead, stress becomes localized learning opportunities that strengthen individual modules without destabilizing the whole system.
<h2 class="wp-block-heading">Via Negativa in Practice</h2>
Nassim Taleb’s concept of “via negativa”—defining systems by what they’re not rather than what they are—translates directly to building antifragile AI systems.
When Airbnb’s search algorithm was producing poor results, instead of adding more ranking factors (the typical approach), the company applied via negativa: It systematically removed listings that consistently received poor ratings, hosts who didn’t respond promptly, and properties with misleading photos. By eliminating negative elements, the remaining search results naturally improved.
Netflix’s recommendation system similarly applies via negativa by maintaining “negative preference profiles”—systematically identifying and avoiding content patterns that lead to user dissatisfaction. Rather than just learning what users like, the system becomes expert at recognizing what they’ll hate, leading to higher overall satisfaction through subtraction rather than addition.
In technical terms, via negativa means starting with maximum system flexibility and systematically removing constraints that don’t add value—allowing the system to adapt to unforeseen circumstances rather than being locked into rigid predetermined behaviors.
<h2 class="wp-block-heading">Implementing Continuous Feedback Loops</h2>
The feedback loop architecture requires three components: error detection, learning integration, and system adaptation. In LOXM’s implementation, market execution data flows back into the model within milliseconds of trade completion. The system uses streaming data pipelines to capture trade outcomes, model predictions, and market conditions in real time. Machine learning models continuously compare predicted execution quality to actual execution quality, updating model weights through online learning algorithms. This creates a continuous feedback loop where each trade makes the next trade execution more intelligent.
When a trade execution deviates from expected performance—whether due to market volatility, liquidity constraints, or timing issues—this immediately becomes training data. The system doesn’t wait for batch processing or scheduled retraining; it adapts in real time while maintaining stable performance for ongoing operations.
<h2 class="wp-block-heading">Organizational Learning Loop</h2>
Antifragile organizations must cultivate specific learning behaviors beyond just technical implementations. This requires moving beyond traditional risk management approaches toward Taleb’s “via negativa.”
The learning loop involves three phases: stress identification, system adaptation, and capability improvement. Teams regularly expose systems to controlled stress, observe how they respond, and then use generative AI to identify improvement opportunities. Each iteration strengthens the system’s ability to handle future challenges.
Netflix institutionalized this through monthly “chaos drills” where teams deliberately introduce failures—API timeouts, database connection losses, content metadata corruption—and observe how their AI systems respond. Each drill generates postmortems focused not on blame but on extracting learning from the failure scenarios.
<h2 class="wp-block-heading">Measurement and Validation</h2>
Antifragile systems require new metrics beyond traditional availability and performance measures. Key metrics include:
<ul class="wp-block-list">
<li>Adaptation speed: Time from anomaly detection to corrective action</li>
<li>Information extraction rate: Number of meaningful model updates per disruption event</li>
<li>Asymmetric performance factor: Ratio of system gains from positive shocks to losses from negative ones</li>
</ul>
LOXM tracks these metrics alongside financial outcomes, demonstrating quantifiable improvement in antifragile capabilities over time. During high-volatility periods, the system’s asymmetric performance factor consistently exceeds 2.0—meaning it gains twice as much from favorable market movements as it loses from adverse ones.
<h2 class="wp-block-heading">The Competitive Advantage</h2>
The goal isn’t just surviving disruption—it’s creating competitive advantage through chaos. When competitors struggle with market volatility, antifragile organizations extract value from the same conditions. They don’t just adapt to change; they actively seek out uncertainty as fuel for growth.
Netflix’s ability to recommend content accurately during the pandemic, when viewing patterns shifted dramatically, gave it a significant advantage over competitors whose recommendation systems struggled with the new normal. Similarly, LOXM’s superior performance during market stress periods has made it JPMorgan’s primary execution algorithm for institutional clients.
This creates sustainable competitive advantage because antifragile capabilities compound over time. Each disruption makes the system stronger, more adaptive, and better positioned for future challenges.
<h2 class="wp-block-heading">Beyond Resilience: The Antifragile Future</h2>
We’re witnessing the emergence of a new organizational paradigm. The convergence of antifragility principles with generative AI capabilities represents more than incremental improvement—it’s a fundamental shift in how organizations can thrive in uncertain environments.
The path forward requires commitment to experimentation, tolerance for controlled failure, and systematic investment in adaptive capabilities. Organizations must evolve from asking “How do we prevent disruption?” to “How do we benefit from disruption?”
The question isn’t whether your organization will face uncertainty and disruption—it’s whether you’ll be positioned to extract competitive advantage from chaos when it arrives. The integration of antifragility principles with generative AI provides the roadmap for that transformation, demonstrated by organizations like Netflix and JPMorgan that have already turned volatility into their greatest strategic asset.
]]></content:encoded>
<wfw:commentRss>https://www.oreilly.com/radar/taming-chaos-with-antifragile-genai-architecture/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
<item>
<title>Building AI-Resistant Technical Debt</title>
<link>https://www.oreilly.com/radar/building-ai-resistant-technical-debt/</link>
<comments>https://www.oreilly.com/radar/building-ai-resistant-technical-debt/#respond</comments>
<pubDate>Wed, 10 Sep 2025 10:03:48 +0000</pubDate>
<dc:creator><![CDATA[Andrew Stellman]]></dc:creator>
<category><![CDATA[AI & ML]]></category>
<category><![CDATA[Commentary]]></category>
<guid isPermaLink="false">https://www.oreilly.com/radar/?p=17422</guid>
<media:content
url="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/Abstract-colors-7.jpg"
medium="image"
type="image/jpeg"
/>
<custom:subtitle><![CDATA[When Speed Creates Long-Term Pain]]></custom:subtitle>
<description><![CDATA[Anyone who’s used AI to generate code has seen it make mistakes. But the real danger isn’t the occasional wrong answer; it’s in what happens when those errors pile up across a codebase. Issues that seem small at first can compound quickly, making code harder to understand, maintain, and evolve. To really see that danger, […]]]></description>
<content:encoded><![CDATA[
Anyone who’s used AI to generate code has seen it make mistakes. But the real danger isn’t the occasional wrong answer; it’s in what happens when those errors pile up across a codebase. Issues that seem small at first can compound quickly, making code harder to understand, maintain, and evolve. To really see that danger, you have to look at how AI is used in practice—which for many developers starts with vibe coding.
Vibe coding is an exploratory, prompt-first approach to software development where developers rapidly prompt, get code, and iterate. When the code seems close but not quite right, the developer describes what’s wrong and lets the AI try again. When it doesn’t compile or tests fail, they copy the error messages back to the AI. The cycle continues—prompt, run, error, paste, prompt again—often without reading or understanding the generated code. It feels productive because you’re making visible progress: errors disappear, tests start passing, features seem to work. You’re treating the AI like a coding partner who handles the implementation details while you steer at a high level.
Developers use vibe coding to explore and refine ideas and can generate large amounts of code quickly. It’s often the natural first step for most developers using AI tools, because it feels so intuitive and productive. Vibe coding offloads detail to the AI, making exploration and ideation fast and effective—which is exactly why it’s so popular.
The AI generates a lot of code, and it’s not practical to review every line every time it regenerates. Trying to read it all can lead to cognitive overload—mental exhaustion from wading through too much code—and makes it harder to throw away code that isn’t working just because you already invested time in reading it.
Vibe coding is a normal and useful way to explore with AI, but on its own it presents a significant risk. The models used by LLMs can hallucinate and produce made-up answers—for example, generating code that calls APIs or methods that don’t even exist. Preventing those AI-generated mistakes from compromising your codebase starts with understanding the capabilities and limitations of these tools, and taking an approach to AI-assisted development that takes those limitations into account.
Here’s a simple example of how these issues compound. When I ask AI to generate a class that handles user interaction, it often creates methods that directly read from and write to the console. When I then ask it to make the code more testable, if I don’t very specifically prompt for a simple fix like having methods take input as parameters and return output as values, the AI frequently suggests wrapping the entire I/O mechanism in an abstraction layer. Now I have an interface, an implementation, mock objects for testing, and dependency injection throughout. What started as a straightforward class has become a miniature framework. The AI isn’t wrong, exactly—the abstraction approach is a valid pattern—but it’s overengineered for the problem at hand. Each iteration adds more complexity, and if you’re not paying attention, you’ll end up with layers upon layers of unnecessary code. This is a good example of how vibe coding can balloon into unnecessary complexity if you don’t stop to verify what’s happening.
<h2 class="wp-block-heading">Novice Developers Face a New Kind of Technical Debt Challenge with AI</h2>
Three months after writing their first line of code, a Reddit user going by SpacetimeSorcerer posted a frustrated update: Their AI-assisted project had reached the point where making any change meant editing dozens of files. The design had hardened around early mistakes, and every change brought a wave of debugging. They’d hit the wall known in software design as “shotgun surgery,” where a single change ripples through so much code that it’s risky and slow to work on—a classic sign of technical debt, the hidden cost of early shortcuts that make future changes harder and more expensive.
<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="1600" height="547" src="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image.png" alt="I am giving up" class="wp-image-17423" title="I am giving up" srcset="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image.png 1600w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-300x103.png 300w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-768x263.png 768w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/image-1536x525.png 1536w" sizes="auto, (max-width: 1600px) 100vw, 1600px" /><figcaption class="wp-element-caption">A Reddit post describing the frustration of AI-accelerated technical debt (used with permission).</figcaption></figure>
AI didn’t cause the problem directly; the code worked (until it didn’t). But the speed of AI-assisted development let this new developer skip the design thinking that prevents these patterns from forming. The same thing happens to experienced developers when deadlines push delivery over maintainability. The difference is, an experienced developer often knows they’re taking on debt. They can spot antipatterns early because they’ve seen them repeatedly, and take steps to “pay off” the debt before it gets much more expensive to fix. Someone new to coding may not even realize it’s happening until it’s too late—and they haven’t yet built the tools or habits to prevent it.
Part of the reason new developers are especially vulnerable to this problem goes back to the Cognitive Shortcut Paradox.1 Without enough hands-on experience debugging, refactoring, and working through ambiguous requirements, they don’t have the instincts built up through experience to spot structural problems in AI-generated code. The AI can hand them a clean, working solution. But if they can’t see the design flaws hiding inside it, those flaws grow unchecked until they’re locked into the project, built into the foundations of the code so changing them requires extensive, frustrating work.
The signals of AI-accelerated technical debt show up quickly: highly coupled code where modules depend on each other’s internal details; “God objects” with too many responsibilities; overly structured solutions where a simple problem gets buried under extra layers. These are the same problems that typically reflect technical debt in human-built code; the reason they emerge so quickly in AI-generated code is because it can be generated much more quickly and without oversight or intentional design or architectural decisions being made. AI can generate these patterns convincingly, making them look deliberate even when they emerged by accident. Because the output compiles, passes tests, and works as expected, it’s easy to accept as “done” without thinking about how it will hold up when requirements change.
When adding or updating a unit test feels unreasonably difficult, that’s often the first sign the design is too rigid. The test is telling you something about the structure—maybe the code is too intertwined, maybe the boundaries are unclear. This feedback loop works whether the code was AI-generated or handwritten, but with AI the friction often shows up later, after the code has already been merged.
That’s where the “trust but verify” habit comes in. Trust the AI to give you a starting point, but verify that the design supports change, testability, and clarity. Ask yourself whether the code will still make sense to you—or anyone else—months from now. In practice, this can mean quick design reviews even for AI-generated code, refactoring when coupling or duplication starts to creep in, and taking a deliberate pass at naming so variables and functions read clearly. These aren’t optional touches; they’re what keep a codebase from locking in its worst early decisions.
AI can help with this too: It can suggest refactorings, point out duplicated logic, or help extract messy code into cleaner abstractions. But it’s up to you to direct it to make those changes, which means you have to spot them first—which is much easier for experienced developers who have seen these problems over the course of many projects.
Left to its defaults, AI-assisted development is biased toward adding new code, not revisiting old decisions. The discipline to avoid technical debt comes from building design checks into your workflow so AI’s speed works in service of maintainability instead of against it.
<hr class="wp-block-separator has-alpha-channel-opacity is-style-wide"/>
<h2 class="wp-block-heading">Footnote</h2>
<ol class="wp-block-list">
<li>I’ll discuss this in more detail in a forthcoming Radar article on October 8.</li>
</ol>

]]></content:encoded>
<wfw:commentRss>https://www.oreilly.com/radar/building-ai-resistant-technical-debt/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
<item>
<title>Megawatts and Gigawatts of AI</title>
<link>https://www.oreilly.com/radar/megawatts-and-gigawatts-of-ai/</link>
<comments>https://www.oreilly.com/radar/megawatts-and-gigawatts-of-ai/#respond</comments>
<pubDate>Tue, 09 Sep 2025 10:54:36 +0000</pubDate>
<dc:creator><![CDATA[Mike Loukides]]></dc:creator>
<category><![CDATA[AI & ML]]></category>
<category><![CDATA[Commentary]]></category>
<guid isPermaLink="false">https://www.oreilly.com/radar/?p=17410</guid>
<media:content
url="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/Smaller-Is-Power.jpg"
medium="image"
type="image/jpeg"
/>
<description><![CDATA[We can’t not talk about power these days. We’ve been talking about it ever since the Stargate project, with half a trillion dollars in data center investment, was floated early in the year. We’ve been talking about it ever since the now-classic “Stochastic Parrots” paper. And, as time goes on, it only becomes more of […]]]></description>
<content:encoded><![CDATA[
We can’t not talk about power these days. We’ve been talking about it ever since the <a href="https://en.wikipedia.org/wiki/Stargate_LLC" target="_blank" rel="noreferrer noopener">Stargate</a> project, with half a trillion dollars in data center investment, was floated early in the year. We’ve been talking about it ever since the now-classic “<a href="https://dl.acm.org/doi/10.1145/3442188.3445922" target="_blank" rel="noreferrer noopener">Stochastic Parrots</a>” paper. And, as time goes on, it only becomes more of an issue.
“Stochastic Parrots” deals with two issues: AI’s power consumption and the fundamental nature of generative AI; selecting sequences of words according to statistical patterns. I always wished those were two papers, because it would be easier to disagree about power and agree about parrots. For me, the power issue is something of a red herring—but increasingly, I see that it’s a red herring that isn’t going away because too many people with too much money want herrings; too many believe that a monopoly on power (or a monopoly on the ability to pay for power) is the route to dominance.
Why, in a better world than we currently live in, would the power issue be a red herring? There are several related reasons:
<ul class="wp-block-list">
<li>I have always assumed that the first generation language models would be highly inefficient, and that over time, we’d develop more efficient algorithms.</li>
<li>I have also assumed that the economics of language models would be similar to chip foundries or pharma factories: The first chip coming out of a foundry costs a few billion dollars, everything afterward is a penny apiece.</li>
<li>I believe (now more than ever) that, long-term, we will settle on small models (70B parameters or less) that can run locally rather than giant models with trillions of parameters running in the cloud.</li>
</ul>
And I still believe those points are largely true. But that’s not sufficient. Let’s go through them one by one, starting with efficiency.
<h2 class="wp-block-heading">Better Algorithms</h2>
A few years ago, I saw a fair number of papers about more efficient models. I remember a lot of articles about pruning neural networks (eliminating nodes that contribute little to the result) and other techniques. Papers that address efficiency are still being published—most notably, DeepMind’s recent “<a href="https://arxiv.org/abs/2507.10524" target="_blank" rel="noreferrer noopener">Mixture-of-Recursions</a>” paper—but they don’t seem to be as common. That’s just anecdata, and should perhaps be ignored. More to the point, DeepSeek shocked the world with their R1 model, which they claimed cost roughly 1/10 as much to train as the leading frontier models. A lot of commentary insisted that DeepSeek wasn’t being up front in their measurement of power consumption, but since then several other Chinese labs have released highly capable models, with no gigawatt data centers in sight. Even more recently, OpenAI has released gpt-oss in two sizes (120B and 30B), which were <a href="https://www.theinformation.com/articles/openai-says-gpt-5-one-size-fits-new-open-model-cheap-train?rc=7em78a" target="_blank" rel="noreferrer noopener">reportedly</a> much less expensive to train. It’s not the first time this has happened—I’ve been told that the Soviet Union developed amazingly efficient data compression algorithms because their computers were a decade behind ours. Better algorithms can trump larger power bills, better CPUs, and more GPUs, if we let them.
What’s wrong with this picture? The picture is good, but much of the narrative is US-centric, and that distorts it. First, it’s distorted by our belief that bigger is always better: Look at our cars, our SUVs, our houses. We’re conditioned to believe that a model with a trillion parameters has to be better than a model with a mere 70B, right? That a model that cost <a href="https://en.wikipedia.org/wiki/GPT-4#:~:text=Sam%20Altman%20stated%20that%20the,4%20had%201%20trillion%20parameters." target="_blank" rel="noreferrer noopener">a hundred million dollars</a> to train has to be better than one that can be trained economically? That myth is deeply embedded in our psyche. Second, it’s distorted by economics. Bigger is better is a myth that would-be monopolists play on when they talk about the need for ever bigger data centers, preferably funded with tax dollars. It’s a convenient myth, because convincing would-be competitors that they need to spend billions on data centers is an effective way to have no competitors.
One area that hasn’t been sufficiently explored is extremely small models developed for specialized tasks. Drew Breunig <a href="https://www.dbreunig.com/2025/08/01/does-the-bitter-lesson-have-limits.html" target="_blank" rel="noreferrer noopener">writes</a> about the tiny chess model in Stockfish, the world’s leading chess program: It’s small enough to run in an iPhone, and replaced a much larger general-purpose model. And it <a href="https://www.youtube.com/watch?v=yc0bFlW56tY&t=528s" target="_blank" rel="noreferrer noopener">soundly defeated</a> Claude Sonnet 3.5 and GPT-4o.1 He also writes about the 27 million parameter <a href="https://arxiv.org/pdf/2506.21734" target="_blank" rel="noreferrer noopener">Hierarchical Reasoning Model (HRM)</a> that has beaten models like Claude 3.7 on the ARC benchmark. Pete Warden’s Moonshine does real-time speech-to-text transcription in the browser—and is as good as any high-end model I’ve seen. None of these are general-purpose models. They won’t vibe code; they won’t write your blog posts. But they are extremely effective at what they do. And if AI is going to fulfill its destiny of “disappearing into the walls,” of becoming part of our everyday infrastructure, we will need very accurate, very specialized models. We will have to free ourselves of the myth that bigger is better.2
<h2 class="wp-block-heading">The Cost of Inference</h2>
The purpose of a model isn’t to be trained; it’s to do inference. This is a gross simplification, but part of training is doing inference trillions of times and adjusting the model’s billions of parameters to minimize error. A single request takes an extremely small fraction of the effort required to train a model. That fact leads directly to the economics of chip foundries: The ability to process the first prompt costs millions of dollars, but once they’re in production, <a href="https://andymasley.substack.com/p/a-cheat-sheet-for-conversations-about" target="_blank" rel="noreferrer noopener">processing a prompt costs fractions of a cent</a>. Google has <a href="https://services.google.com/fh/files/misc/measuring_the_environmental_impact_of_delivering_ai_at_google_scale.pdf" target="_blank" rel="noreferrer noopener">claimed</a> that processing a typical text prompt to Gemini takes 0.24 watt-hours, significantly less than it takes to heat water for a cup of coffee. They also claim that increases in software efficiency have led to a 33x reduction in energy consumption over the past year.
That’s obviously not the entire story: Millions of people prompting ChatGPT adds up, as does usage of newer “reasoning” modules that have an extended internal dialog before arriving at a result. Likewise, driving to work rather than biking raises the global temperature a nanofraction of a degree—but when you multiply the nanofraction by billions of commuters, it’s a different story. It’s fair to say that an individual who uses ChatGPT or Gemini isn’t a problem, but it’s also important to realize that millions of users pounding on an AI service can grow into a problem quite quickly. Unfortunately, it’s also true that increases in efficiency often don’t lead to reductions in energy use but to solving more complex problems within the same energy budget. We may be seeing that with reasoning models, image and video generation models, and other applications that are now becoming financially feasible. Does this problem require gigawatt data centers? No, not that, but it’s a problem that can justify the building of gigawatt data centers.
There is a solution, but it requires rethinking the problem. Telling people to use public transportation or bicycles for their commute is ineffective (in the US), as will be telling people not to use AI. The problem needs to be rethought: redesigning work to eliminate the commute (O’Reilly is 100% work from home), rethinking the way we use AI so that it doesn’t require cloud-hosted trillion parameter models. That brings us to using AI locally.
<h2 class="wp-block-heading">Staying Local</h2>
Almost everything we do with GPT-*, Claude-*, Gemini-*, and other frontier models could be done equally effectively on much smaller models running locally: in a small corporate machine room or even on a laptop. Running AI locally also shields you from problems with availability, bandwidth, limits on usage, and leaking private data. This is a story that would-be monopolists don’t want us to hear. Again, this is anecdata, but I’ve been very impressed by the results I get from running models in the 30 billion parameter range on my laptop. I do vibe coding and get mostly correct code that the model can (usually) fix for me; I ask for summaries of blogs and papers and get excellent results. Anthropic, Google, and OpenAI are competing for tenths of a percentage point on highly gamed benchmarks, but I doubt that those benchmark scores have much practical meaning. I would love to see a study on the difference between Qwen3-30B and GPT-5.
What does that mean for energy costs? It’s unclear. Gigawatt data centers for doing inference would go unneeded if people do inference locally, but what are the consequences of a billion users doing inference on high-end laptops? If I give my local AIs a difficult problem, my laptop heats up and runs its fans. It’s using more electricity. And laptops aren’t as efficient as data centers that have been designed to minimize electrical use. It’s all well and good to scoff at gigawatts, but when you’re using that much power, minimizing power consumption saves a lot of money. Economies of scale are real. Personally, I’d bet on the laptops: Computing with 30 billion parameters is undoubtedly going to be less energy-intensive than computing with 3 trillion parameters. But I won’t hold my breath waiting for someone to do this research.
There’s another side to this question, and that involves models that “reason.” So-called “reasoning models” have an internal conversation (not always visible to the user) in which the model “plans” the steps it will take to answer the prompt. A recent paper <a href="https://nousresearch.com/measuring-thinking-efficiency-in-reasoning-models-the-missing-benchmark/" target="_blank" rel="noreferrer noopener">claims</a> that smaller open source models tend to generate many more reasoning tokens than large models (3 to 10 times as many, depending on the models you’re comparing), and that the extensive reasoning process eats away at the economics of the smaller models. Reasoning tokens must be processed, the same as any user-generated tokens; this processing incurs charges (which the paper discusses), and charges presumably relate directly to power.
While it’s surprising that small models generate more reasoning tokens, it’s no surprise that reasoning is expensive, and we need to take that into account. Reasoning is a tool to be used; it tends to be particularly useful when a model is asked to solve a problem in mathematics. It’s much less useful when the task involves looking up facts, summarization, writing, or making recommendations. It can help in areas like software design but is likely to be a liability for generative coding. In these cases, the reasoning process can actually become misleading—in addition to burning tokens. Deciding how to use models effectively, whether you’re running them locally or in the cloud, is a task that falls to us.
Going to the giant reasoning models for the “best possible answer” is always a temptation, especially when you know you don’t need the best possible answer. It takes some discipline to commit to the smaller models—even though it’s difficult to argue that using the frontier models is less work. You still have to analyze their output and check their results. And I confess: As committed as I am to the smaller models, I tend to stick with models in the 30B range, and avoid the 1B–5B models (including the excellent Gemma 3N). Those models, I’m sure, would give good results, use even less power, and run even faster. But I’m still in the process of peeling myself away from my knee-jerk assumptions.
Bigger isn’t necessarily better; more power isn’t necessarily the route to AI dominance. We don’t yet know how this will play out, but I’d place my bets on smaller models running locally and trained with efficiency in mind. There will no doubt be some applications that require large frontier models—perhaps generating synthetic data for training the smaller models—but we really need to understand where frontier models are needed, and where they aren’t. My bet is that they’re rarely needed. And if we free ourselves from the desire to use the latest, largest frontier model just because it’s there—whether or not it serves your purpose any better than a 30B model—we won’t need most of those giant data centers. Don’t be seduced by the AI-industrial complex.
<hr class="wp-block-separator has-alpha-channel-opacity is-style-wide"/>
<h2 class="wp-block-heading">Footnotes</h2>
<ol class="wp-block-list">
<li>I’m not aware of games between Sockfish and more recent Claude 4, Claude 4.1, and GPT-5 models. There’s every reason to believe the results would be similar.</li>
<li>Kevlin Henney makes a related point in “<a href="https://www.oreilly.com/radar/scaling-false-peaks/" target="_blank" rel="noreferrer noopener">Scaling False Peaks</a>.”</li>
</ol>
]]></content:encoded>
<wfw:commentRss>https://www.oreilly.com/radar/megawatts-and-gigawatts-of-ai/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
<item>
<title>A “Beam Versus Dataflow” Conversation</title>
<link>https://www.oreilly.com/radar/a-beam-versus-dataflow-conversation/</link>
<comments>https://www.oreilly.com/radar/a-beam-versus-dataflow-conversation/#respond</comments>
<pubDate>Mon, 08 Sep 2025 10:28:48 +0000</pubDate>
<dc:creator><![CDATA[Aaron Black]]></dc:creator>
<category><![CDATA[AI & ML]]></category>
<category><![CDATA[Research]]></category>
<guid isPermaLink="false">https://www.oreilly.com/radar/?p=17406</guid>
<media:content
url="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/Beam-pipeline.jpg"
medium="image"
type="image/jpeg"
/>
<description><![CDATA[I’ve been in a few recent conversations about whether to use Apache Beam on its own or run it with Google Dataflow. On the surface, it’s a tooling decision. But it also reflects a broader conversation about how teams build systems. Beam offers a consistent programming model for unifying batch and streaming logic. It doesn’t […]]]></description>
<content:encoded><![CDATA[
I’ve been in a few recent conversations about whether to use Apache Beam on its own or run it with Google Dataflow. On the surface, it’s a tooling decision. But it also reflects a broader conversation about how teams build systems.
Beam offers a consistent programming model for unifying batch and streaming logic. It doesn’t dictate where that logic runs. You can deploy pipelines on Flink or Spark, or you can use a managed runner like Dataflow. Each option outfits the same Beam code with very different execution semantics.
What’s added urgency to this choice is the <a href="https://bostoninstituteofanalytics.org/blog/the-rise-of-real-time-data-science-in-2025-tools-trends-and-techniques/" target="_blank" rel="noreferrer noopener">growing pressure on data systems to support machine learning and AI workloads</a>. It’s no longer enough to transform, validate, and load. Teams also need to feed real-time inference, scale feature processing, and orchestrate retraining workflows as part of pipeline development. Beam and Dataflow are both increasingly positioned as infrastructure that supports not just analytics but active AI.
Choosing one path over the other means making decisions about flexibility, integration surface, runtime ownership, and operational scale. None of those are easy knobs to adjust after the fact.
The goal here is to unpack the trade-offs and help teams make deliberate calls about what kind of infrastructure they’ll want.
<h2 class="wp-block-heading">Apache Beam: A Common Language for Pipelines</h2>
Apache Beam provides a shared model for expressing data processing workflows. This includes the kinds of batch and streaming tasks most data teams are already familiar with, but it also now includes a growing set of patterns specific to AI and ML.
Developers write Beam pipelines using a single SDK that defines what the pipeline does, not how the underlying engine runs it. That logic can include parsing logs, transforming records, joining events across time windows, and applying trained models to incoming data using built-in inference transforms.
Support for AI-specific workflow steps is improving. Beam now offers the <a href="https://beam.apache.org/documentation/transforms/python/elementwise/runinference/" target="_blank" rel="noreferrer noopener">RunInference API</a>, along with <a href="https://beam.apache.org/documentation/transforms/python/elementwise/mltransform/" target="_blank" rel="noreferrer noopener">MLTransform</a> utilities, to help deploy models trained in frameworks like TensorFlow, PyTorch, and scikit-learn into Beam pipelines. These can be used in batch workflows for bulk scoring or in low-latency streaming pipelines where inference is applied to live events.
Crucially, this isn’t tied to one cloud. Beam lets you define the transformation once and pick the execution path later. You can run the exact same pipeline on Flink, Spark, or Dataflow. That level of portability doesn’t remove infrastructure concerns on its own, but it does allow you to focus your engineering effort on logic rather than rewrites.
Beam gives you a way to describe and maintain machine learning pipelines. What’s left is deciding how you want to operate them.
<h2 class="wp-block-heading">Running Beam: Self-Managed Versus Managed</h2>
If you’re running Beam on Flink, Spark, or some custom runner, you’re responsible for the full runtime environment. You handle provisioning, scaling, fault tolerance, tuning, and observability. Beam becomes another user of your platform. That degree of control can be useful, especially if model inference is only one part of a larger pipeline that already runs in your infrastructure. Custom logic, proprietary connectors, or non-standard state handling might push you toward keeping everything self-managed.
But building for inference at scale, especially in streaming, <a href="https://openproceedings.org/2024/conf/edbt/paper-156.pdf" target="_blank" rel="noreferrer noopener">introduces friction</a>. It means tracking model versions across pipeline jobs. It means watching watermarks and tuning triggers so inference happens precisely when it should. It means managing restart logic and making sure models fail gracefully when cloud resources or updatable weights are unavailable. If your team is already running distributed systems, that may be fine. But it isn’t free.
Running Beam on Dataflow simplifies much of this by taking infrastructure management out of your hands. You still build your pipeline the same way. But once deployed to Dataflow, scaling and resource provisioning are handled by the platform. Dataflow pipelines can stream through inference using native Beam transforms and benefit from newer features like automatic model refresh and tight integration with Google Cloud services.
This is particularly relevant when <a href="https://cloud.google.com/vertex-ai/docs/pipelines/dataflow-component" target="_blank" rel="noreferrer noopener">working with Vertex AI</a>, which allows hosted model deployment, feature store lookups, and GPU-accelerated inference to plug straight into your pipeline. Dataflow enables those connections with lower latency and minimal manual setup. For some teams, that makes it the better fit by default.
Of course, not every ML workload needs end-to-end cloud integration. And not every team wants to give up control of their pipeline execution. That’s why understanding what each option provides is necessary before making long-term infrastructure bets.
<h2 class="wp-block-heading">Choosing the Execution Model That Matches Your Team</h2>
Beam gives you the foundation for defining ML-aware data pipelines. Dataflow gives you a specific way to execute them, especially in production environments where responsiveness and scalability matter.
If you’re building systems that require operational control and that already assume deep platform ownership, managing your own Beam runner makes sense. It gives flexibility where rules are looser and lets teams hook directly into their own tools and systems.
If instead you need fast iteration with minimal overhead, or you’re running real-time inference against cloud-hosted models, then Dataflow offers clear benefits. You onboard your pipeline without worrying about the runtime layer and deliver predictions without gluing together your own serving infrastructure.
If inference becomes an everyday part of your pipeline logic, the balance between operational effort and platform constraints starts to shift. The best execution model depends on more than feature comparison.
A well-chosen execution model involves commitment to how your team builds, evolves, and operates intelligent data systems over time. Whether you prioritize fine-grained control or accelerated delivery, both Beam and Dataflow offer robust paths forward. The key is aligning that choice with your long-term goals: consistency across workloads, adaptability for future AI demands, and a developer experience that supports innovation without compromising stability. As inference becomes a core part of modern pipelines, choosing the right abstraction sets a foundation for future-proofing your data infrastructure.
]]></content:encoded>
<wfw:commentRss>https://www.oreilly.com/radar/a-beam-versus-dataflow-conversation/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
<item>
<title>Generative AI in the Real World: Luke Wroblewski on When Databases Talk Agent-Speak</title>
<link>https://www.oreilly.com/radar/podcast/generative-ai-in-the-real-world-luke-wroblewski-on-when-databases-talk-agent-speak/</link>
<comments>https://www.oreilly.com/radar/podcast/generative-ai-in-the-real-world-luke-wroblewski-on-when-databases-talk-agent-speak/#respond</comments>
<pubDate>Thu, 04 Sep 2025 16:01:45 +0000</pubDate>
<dc:creator><![CDATA[Ben Lorica and Luke Wroblewski]]></dc:creator>
<category><![CDATA[Generative AI in the Real World]]></category>
<category><![CDATA[Podcast]]></category>
<guid isPermaLink="false">https://www.oreilly.com/radar/?post_type=podcast&p=17394</guid>
<media:content
url="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2024/01/Podcast_Cover_GenAI_in_the_Real_World-scaled.png"
medium="image"
type="image/png"
/>
<description><![CDATA[Join Luke Wroblewski and Ben Lorica as they talk about the future of software development. What happens when we have databases that are designed to interact with agents and language models rather than humans? We’re starting to see what that world will look like. It’s an exciting time to be a software developer. About the […]]]></description>
<content:encoded><![CDATA[
Join Luke Wroblewski and Ben Lorica as they talk about the future of software development. What happens when we have databases that are designed to interact with agents and language models rather than humans? We’re starting to see what that world will look like. It’s an exciting time to be a software developer.
About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2025, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.
Check out <a href="https://learning.oreilly.com/playlists/42123a72-1108-40f1-91c0-adbfb9f4983b/?_gl=1*16z5k2y*_ga*MTE1NDE4NjYxMi4xNzI5NTkwODkx*_ga_092EL089CH*MTcyOTYxNDAyNC4zLjEuMTcyOTYxNDAyNi41OC4wLjA." target="_blank" rel="noreferrer noopener">other episodes</a> of this podcast on the O’Reilly learning platform.
<h2 class="wp-block-heading">Timestamps</h2>
<ul class="wp-block-list">
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=0" target="_blank" rel="noreferrer noopener">0:00</a>: Introduction to Luke Wroblewski of Sutter Hill Ventures. </li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=36" target="_blank" rel="noreferrer noopener">0:36</a>: You’ve talked about a paradigm shift in how we write applications. You’ve said that all we need is a URL and model, and that’s an app. Has anyone else made a similar observation? Have you noticed substantial apps that look like this?</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=68" target="_blank" rel="noreferrer noopener">1:08</a>: The future is here; it’s just not evenly distributed yet. That’s what everyone loves to say. The first websites looked nothing like robust web applications, and now we have a multimedia podcast studio running in the browser. We’re at the phase where some of these things look and feel less robust. And our ideas for what constitutes an application change in each of these phases. If I told you pre-Google Maps that we’d be running all of our web applications in a browser, you’d have laughed at me. </li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=133" target="_blank" rel="noreferrer noopener">2:13</a>: I think what you mean is an MCP server, and the model itself is the application, correct?</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=144" target="_blank" rel="noreferrer noopener">2:24</a>: Yes. The current definition of an application, in a simple form, is running code and a database. We’re at the stage where you have AI coding agents that can handle the coding part. But we haven’t really had databases that have been designed for the way those agents think about code and interacting with data.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=177" target="_blank" rel="noreferrer noopener">2:57</a>: Now that we have databases that work the way agents work, you can take out the running-code part almost. People go to Lovable or Cursor and they’re forced to look at code syntax. But if an AI model can just use a database effectively, it takes the role of the running code. And if it can manage data visualizations and UI, you don’t need to touch the code. You just need to point the AI at a data structure it can use effectively. <a href="https://mcpui.dev/" target="_blank" rel="noreferrer noopener">MCP UI</a> is a nice example of people pushing in this direction.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=252" target="_blank" rel="noreferrer noopener">4:12</a>: Which brings us to something you announced recently: AgentDB. You can find it at <a href="http://agentdb.dev" target="_blank" rel="noreferrer noopener">agentdb.dev</a>. What problem is AgentDB trying to solve?</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=274" target="_blank" rel="noreferrer noopener">4:34</a>: Related to what we were just talking about: How do we get AI agents to use databases effectively? Most things in the technology stack are made for humans and the scale at which humans operate.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=306" target="_blank" rel="noreferrer noopener">5:06</a>: They’re still designed for a DBA, but eliminating the command line, right? So you still have to have an understanding of DBA principles?</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=319" target="_blank" rel="noreferrer noopener">5:19</a>: How do you pick between the different compute options? How do you pick a region? What are the security options? And it’s not something you’re going to do thousands of times a day. Databricks just shared some stats where they said that thousands of databases per agent get made a day. They think 99% of databases being made are going to be made by agents. What is making all these databases? No longer humans. And the scale at which they make them—thousands is a lowball number. It will be way, way higher than that. How do we make a database system that works in that reality?</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=382" target="_blank" rel="noreferrer noopener">6:22</a>: So the high-level thesis here is that lots of people will be creating agents, and these agents will rely on something that looks like a database, and many of these people won’t be hardcore engineers. What else?</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=405" target="_blank" rel="noreferrer noopener">6:45</a>: It’s also agents creating agents, and agents creating applications, and agents deciding they need a database to complete a task. The explosion of these smart machine uses and workflows is well underway. But we don’t have an infrastructure that was made for that world. They were all designed to work with humans.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=451" target="_blank" rel="noreferrer noopener">7:31</a>: So in the classic database world, you’d consider AgentDB more like OLTP rather than analytics and OLAP.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=462" target="_blank" rel="noreferrer noopener">7:42</a>: Yeah, for analytics you’d probably stick your log somewhere else. The characteristics that make AgentDB really interesting for agents is, number 1: To create a database, all you really need is a unique ID. The creation of the ID manifests a database out of thin air. And we store it as a file, so you can scale like crazy. And all of these databases are fully isolated. They’re also downloadable, deletable, releasable—all the characteristics of a filesystem. We also have the concept of a template that comes along with the database. That gives the AI model or agent all the context it needs to start using the database immediately. If you just point Claude at a database, it will need to look at the structure (schema). It will build tokens and time trying to get the structure of the information. And every time it does this is an opportunity to make a mistake. With AgentDB, when an agent or an AI model is pointed at the database with a template, it can immediately write a query because we have in there a description of the database, the schema. So you save time, cut down errors, and don’t have to go through that learning step every time the model touches a database.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=622" target="_blank" rel="noreferrer noopener">10:22</a>: I assume this database will have some of the features you like, like ACID, vector search. So what kinds of applications have people built using AgentDB? </li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=653" target="_blank" rel="noreferrer noopener">10:53</a>: We put up a little demo page where we allow you to start the process with a CSV file. You upload it, and it will create the database and give you an MCP URL. So people are doing things like personal finance. People are uploading their credit card statements, their bank statements, because those applications are horrendous.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=699" target="_blank" rel="noreferrer noopener">11:39</a>: So it’s the actual statement; it parses it?</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=705" target="_blank" rel="noreferrer noopener">11:45</a>: Another example: Someone has a spreadsheet to track jobs. They can take that, upload it, it gives them a template and a database and an MCP URL. They can pop that job-tracking database into Claude and do all the things you can do with a chat app, like ask, “What did I look at most recently?”</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=755" target="_blank" rel="noreferrer noopener">12:35</a>: Do you envision it more like a DuckDB, more embedded, not really intended for really heavy transactional, high-throughput, more-than-one-table complicated schemas?</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=769" target="_blank" rel="noreferrer noopener">12:49</a>: We currently support DuckDB and SQLite. But there are a bunch of folks who have made multiple table apps and databases.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=789" target="_blank" rel="noreferrer noopener">13:09</a>: So it’s not meant for you to build your own CRM?</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=798" target="_blank" rel="noreferrer noopener">13:18</a>: Actually, one of our go-to-market guys had data of people visiting the website. He can dump that as a spreadsheet. He has data of people starring repos on GitHub. He has data of people who reached out through this form. He has all of these inbound signals of customers. So he took those, dropped them in as CSV files, put it in Claude, and then he can say, “Look at these, search the web for information about these, add it to the database, sort it by priority, assign it to different reps.” It’s CRM-ish already, but super-customized to his particular use case. </li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=867" target="_blank" rel="noreferrer noopener">14:27</a>: So you can create basically an agentic Airtable.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=878" target="_blank" rel="noreferrer noopener">14:38</a>: This means if you’re building AI applications or databases—traditionally that has been somewhat painful. This removes all that friction.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=900" target="_blank" rel="noreferrer noopener">15:00</a>: Yes, and it leads to a different way of making apps. You take that CSV file, you take that MCP URL, and you have a chat app.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=917" target="_blank" rel="noreferrer noopener">15:17</a>: Even though it’s accessible to regular users, it’s something developers should consider, right?</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=925" target="_blank" rel="noreferrer noopener">15:25</a>: We’re starting to see emergent end-user use cases, but what we put out there is for developers. </li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=938" target="_blank" rel="noreferrer noopener">15:38</a>: One of the other things you’ve talked about is the notion that software development has flipped. Can you explain that to our listeners?</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=956" target="_blank" rel="noreferrer noopener">15:56</a>: I spent eight and a half years at Google, four and a half at Yahoo, two and a half at ebay, and your traditional process of what we’re going to do next is up front: There’s a lot of drawing pictures and stuff. We had to scope engineering time. A lot of the stuff was front-loaded to figure out what we were going to build. Now with things like AI agents, you can build it and then start thinking about how it integrates inside the project. At a lot of our companies that are working with AI coding agents, I think this naturally starts to happen, that there’s a manifestation of the technology that helps you think through what the design should be, how do we integrate into the product, should we launch this? This is what I mean by “flipped.”</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=1061" target="_blank" rel="noreferrer noopener">17:41</a>: If I’m in a company like a big bank, does this mean that engineers are running ahead?</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=1075" target="_blank" rel="noreferrer noopener">17:55</a>: I don’t know if it’s happening in big banks yet, but it’s definitely happening in startup companies. And design teams have to think through “Here’s a bunch of stuff, let me do a wash across all that to fit in,” as opposed to spending time designing it earlier. There are pros and cons to both of these. The engineers were cleaning up the details in the previous world. Now the opposite is true: I’ve built it, now I need to design it.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=1135" target="_blank" rel="noreferrer noopener">18:55</a>: Does this imply a new role? There’s a new skill set that designers have to develop?</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=1147" target="_blank" rel="noreferrer noopener">19:07</a>: There’s been this debate about “Should designers code?” Over the years lots of things have reduced the barrier to entry, and now we have an even more dramatic reduction. I’ve always been of the mindset that if you understand the medium, you will make better things. Now there’s even less of a reason not to do it.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=1190" target="_blank" rel="noreferrer noopener">19:50</a>: Anecdotally, what I’m observing is that the people who come from product are able to build something, but I haven’t heard as many engineers thinking about design. What are the AI tools for doing that?</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=1219" target="_blank" rel="noreferrer noopener">20:19</a>: I hear the same thing. What I hope remains uncommoditized is taste. I’ve found that it’s very hard to teach taste to people. If I have a designer who is a good systems thinker but doesn’t have the gestalt of the visual design layer, I haven’t been able to teach that to them. But I have been able to find people with a clear sense of taste from diverse design backgrounds and get them on board with interaction design and systems thinking and applications.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=1262" target="_blank" rel="noreferrer noopener">21:02</a>: If you’re a young person and you’re skilled, you can go into either design or software engineering. Of course, now you’re reading articles saying “forget about software engineering.” I haven’t seen articles saying “forget about design.”</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=1291" target="_blank" rel="noreferrer noopener">21:31</a>: I disagree with the idea that it’s a bad time to be an engineer. It’s never been more exciting.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=1306" target="_blank" rel="noreferrer noopener">21:46</a>: But you have to be open to that. If you’re a curmudgeon, you’re going to be in trouble.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=1313" target="_blank" rel="noreferrer noopener">21:53</a>: This happens with every technical platform transition. I spent so many years during the smartphone boom hearing people say, “No one is ever going to watch TV and movies on mobile.” Is it an affinity to the past, or a sense of doubt about the future? Every time, it’s been the same thing.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=1357" target="_blank" rel="noreferrer noopener">22:37</a>: One way to think of AgentDB is like a wedge. It addresses one clear pain point in the stack that people have to grapple with. So what’s next? Is it Kubernetes?</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=1389" target="_blank" rel="noreferrer noopener">23:09</a>: I don’t want to go near that one! The broader context of how applications are changing—how do I create a coherent product that people understand how to use, that has aesthetics, that has a personality?—is a very wide-open question. There’s a bunch of other systems that have not been made for AI models. A simple example is search APIs. Search APIs are basically structured the same way as results pages. Here’s your 10 blue links. But an agentic model can suck up so much information. Not only should you be giving it the web page, you should be giving it the whole site. Those systems are not built for this world at all. You can go down the list of the things we use as core infrastructure and think about how they were made for a human, not the capabilities of an enormous large language model.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=1479" target="_blank" rel="noreferrer noopener">24:39</a>: Right now, I’m writing an article on enterprise search, and one of things people don’t realize is that it’s broken. In terms of AgentDB, do you worry about things like security, governance? There’s another place black hat attackers can go after.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=1520" target="_blank" rel="noreferrer noopener">25:20</a>: Absolutely. All new technologies have the light side and the dark side. It’s always been a codebreaker-codemaker stack. That doesn’t change. The attack vectors are different and, in the early stages, we don’t know what they are, so it is a cat and mouse game. There was an era when spam in email was terrible; your mailbox would be full of spam and you manually had to mark things as junk. Now you use gmail, and you don’t think about it. When was the last time you went into the junk mail tab? We built systems, we got smarter, and the average person doesn’t think about it.</li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=1591" target="_blank" rel="noreferrer noopener">26:31</a>: As you have more people building agents, and agents building agents, you have data governance, access control; suddenly you have AgentDB artifacts all over the place. </li>
<li><a href="https://cdn.oreillystatic.com/radar/generative-ai-real-world-podcast/GenAI_in_the_Real_World_with_Luke_Wroblewski.mp3#t=1626" target="_blank" rel="noreferrer noopener">27:06</a>: Two things here. This is an underappreciated part of this. Two years ago I launched my own personal chatbot that works off my writings. People ask me what model am I using, and how is it built? Those are partly interesting questions. But the real work in that system is constantly looking at the questions people are asking, and evaluating whether or not it responded well. I’m constantly course-correcting the system. That’s the work that a lot of people don’t do. But the thing I’m doing is applying taste, applying a perspective, defining what “good” is. For a lot of systems like enterprise search, it’s like, “We deployed the technology.” How do you know if it’s good or not? Is someone in there constantly tweaking and tuning? What makes Google Search so good? It’s constantly being re-evaluated. Or Google Translate—was this translation good or bad? Baked in early on.</li>
</ul>
]]></content:encoded>
<wfw:commentRss>https://www.oreilly.com/radar/podcast/generative-ai-in-the-real-world-luke-wroblewski-on-when-databases-talk-agent-speak/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
<item>
<title>AI Security Takes Center Stage at Black Hat USA 2025</title>
<link>https://www.oreilly.com/radar/ai-security-takes-center-stage-at-black-hat-usa-2025/</link>
<pubDate>Thu, 04 Sep 2025 09:52:40 +0000</pubDate>
<dc:creator><![CDATA[Simina Calin]]></dc:creator>
<category><![CDATA[AI & ML]]></category>
<category><![CDATA[Security]]></category>
<category><![CDATA[Commentary]]></category>
<guid isPermaLink="false">https://www.oreilly.com/radar/?p=17388</guid>
<media:content
url="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/Abstract-colors-9.jpg"
medium="image"
type="image/jpeg"
/>
<description><![CDATA[The security landscape is undergoing yet another major shift, and nowhere was this more evident than at Black Hat USA 2025. As artificial intelligence (especially the agentic variety) becomes deeply embedded in enterprise systems, it’s creating both security challenges and opportunities. Here’s what security professionals need to know about this rapidly evolving landscape. AI systems—and […]]]></description>
<content:encoded><![CDATA[
The security landscape is undergoing yet another major shift, and nowhere was this more evident than at <a href="https://www.blackhat.com/us-25/" target="_blank" rel="noreferrer noopener">Black Hat USA 2025</a>. As artificial intelligence (especially the agentic variety) becomes deeply embedded in enterprise systems, it’s creating both security challenges and opportunities. Here’s what security professionals need to know about this rapidly evolving landscape.
AI systems—and particularly the AI assistants that have become integral to enterprise workflows—are emerging as prime targets for attackers. In one of the most interesting and scariest presentations, Michael Bargury of Zenity demonstrated previously unknown <a href="https://www.blackhat.com/us-25/briefings/schedule/index.html#ai-enterprise-compromise---0click-exploit-methods-46442" target="_blank" rel="noreferrer noopener">“0click” exploit methods</a> affecting major AI platforms including ChatGPT, Gemini, and Microsoft Copilot. These findings underscore how AI assistants, despite their robust security measures, can become vectors for system compromise.
AI security presents a paradox: As organizations expand AI capabilities to enhance productivity, they must necessarily increase these tools’ access to sensitive data and systems. This expansion creates new attack surfaces and more complex supply chains to defend. NVIDIA’s AI red team highlighted this vulnerability, revealing how <a href="https://www.blackhat.com/us-25/briefings/schedule/#from-prompts-to-pwns-exploiting-and-securing-ai-agents-46681" target="_blank" rel="noreferrer noopener">large language models (LLMs) are uniquely susceptible to malicious inputs</a>, and demonstrated several novel exploit techniques that take advantage of these inherent weaknesses.
However, it’s not all new territory. Many traditional security principles remain relevant and are, in fact, more crucial than ever. Nathan Hamiel and Nils Amiet of Kudelski Security showed how AI-powered development tools are <a href="https://www.blackhat.com/us-25/briefings/schedule/index.html#hack-to-the-future-owning-ai-powered-tools-with-old-school-vulns-45871" target="_blank" rel="noreferrer noopener">inadvertently reintroducing well-known vulnerabilities into modern applications</a>. Their findings suggest that basic application security practices remain fundamental to AI security.
Looking forward, threat modeling becomes increasingly critical but also more complex. The security community is responding with new frameworks designed specifically for AI systems such as MAESTRO and NIST’s AI Risk Management Framework. The <a href="https://genai.owasp.org/resource/owasp-gen-ai-agentic-security-top-10-global-kickoff-presentation/" target="_blank" rel="noreferrer noopener">OWASP Agentic Security Top 10 project</a>, launched during this year’s conference, provides a structured approach to understanding and addressing AI-specific security risks.
For security professionals, the path forward requires a balanced approach: maintaining strong fundamentals while developing new expertise in AI-specific security challenges. Organizations must reassess their security posture through this new lens, considering both traditional vulnerabilities and emerging AI-specific threats.
The discussions at Black Hat USA 2025 made it clear that while AI presents new security challenges, it also offers opportunities for innovation in defense strategies. Mikko Hypponen’s opening keynote presented a <a href="https://www.blackhat.com/us-25/briefings/schedule/#keynote-three-decades-in-cybersecurity-lessons-learned-and-what-comes-next-48195" target="_blank" rel="noreferrer noopener">historical perspective on the last 30 years of cybersecurity advancements</a> and concluded that security is not only better than it’s ever been but poised to leverage a head start in AI usage. Black Hat has a way of underscoring the reasons for concern, but taken as a whole, this year’s presentations show us that there are also many reasons to be optimistic. Individual success will depend on how well security teams can adapt their existing practices while embracing new approaches specifically designed for AI systems.
]]></content:encoded>
</item>
<item>
<title>Looking Forward to AI Codecon</title>
<link>https://www.oreilly.com/radar/looking-forward-to-ai-codecon/</link>
<pubDate>Wed, 03 Sep 2025 17:25:12 +0000</pubDate>
<dc:creator><![CDATA[Tim O’Reilly]]></dc:creator>
<category><![CDATA[AI & ML]]></category>
<category><![CDATA[Events]]></category>
<guid isPermaLink="false">https://www.oreilly.com/radar/?p=17395</guid>
<media:content
url="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2019/06/abstract-2370445_1920_crop-72496aa8e2aa221169247ee80be31a15.jpg"
medium="image"
type="image/jpeg"
/>
<description><![CDATA[I’m really looking forward to our second O’Reilly AI Codecon, Coding for the Agentic World, which is happening on September 9, online from 8am to noon Pacific time, with a follow-on day of additional demos on September 16. But I’m also looking forward to how the AI market itself unfolds: the surprising twists and turns […]]]></description>
<content:encoded><![CDATA[
I’m really looking forward to our second O’Reilly AI Codecon, <a href="https://www.oreilly.com/AgenticWorld/" target="_blank" rel="noreferrer noopener">Coding for the Agentic World</a>, which is happening on September 9, online from 8am to noon Pacific time, with a follow-on <a href="https://learning.oreilly.com/live-events/oreilly-demo-day/0642572227968/" target="_blank" rel="noreferrer noopener">day of additional demos</a> on September 16. But I’m also looking forward to how the AI market itself unfolds: the surprising twists and turns ahead as users and developers apply AI to real-world problems.
The pages linked above give details on the program for the events. What I want to give here is a bit of the why behind the program, with a bit more detail on some of the fireside chats I will be leading.
<h2 class="wp-block-heading">From Invention to Application</h2>
There has been so much focus in the past on the big AI labs, the model developers, and their razzle-dazzle about AGI, or even ASI. That narrative implied that we were heading toward something unprecedented. But if this is a “<a href="https://www.oreilly.com/radar/is-ai-a-normal-technology/" target="_blank" rel="noreferrer noopener">normal technology</a>” (albeit one as transformational as electricity, the internal combustion engine, or the internet), we know that LLMs themselves are just the beginning of a long process of discovery, product invention, business adoption, and societal adaptation.
That process of collaborative discovery of the real uses for AI and reinvention of the businesses that use it is happening most clearly in the software industry. It is where AI is being pushed to the limits, where new products beyond the chatbot are being introduced, where new workflows are being developed, and where we understand what works and what doesn’t.
This work is often being pushed forward by individuals, who are “<a href="https://yalebooks.yale.edu/book/9780300195668/learning-by-doing/" target="_blank" rel="noreferrer noopener">learning by doing</a>.” Some of these individuals work for large companies, others for startups, others for enterprises, and others as independent hackers.
Our focus in these AI Codecon events is to smooth adoption of AI by helping our customers cut through the hype and understand what is working. O’Reilly’s mission has always been <a href="https://www.oreilly.com/about/" target="_blank" rel="noreferrer noopener">changing the world by sharing the knowledge of innovators</a>. In our events, we always look for people who are at the forefront of invention. As outlined in the call to action for the first event, I was concerned about the chatter that AI would make developers obsolete. I <a href="https://www.oreilly.com/radar/the-end-of-programming-as-we-know-it/" target="_blank" rel="noreferrer noopener">argued instead</a> that it would profoundly change the process of software development and the jobs that developers do, but that it would make them more important than ever.
It looks like I was right. There is a huge ferment, with so much new to learn and do that it’s a really exciting time to be a software developer. I’m really excited about the practicality of the conversation. We’re not just talking about the “what if.” We’re seeing new AI powered services meeting real business needs. We are witnessing the shift from human-centric workflows to agent-centric workflows, and it’s happening faster than you think.
We’re also seeing widespread adoption of the protocols that will power it all. If you’ve followed my work from open source to Web 2.0 to the present, you know that I believe strongly that the most dynamic systems have “an architecture of participation.” That is, they aren’t monolithic. The barriers to entry need to be low and business models fluid (at least in the early stages) for innovation to flourish.
When AI was framed as a race for superintelligence, there was a strong expectation that it would be winner takes all. The first company to get to ASI (or even just to AGI) would soon be so far ahead that it would inevitably become a dominant monopoly. Developers would all use its APIs, making it into the single dominant platform for AI development.
Protocols like MCP and A2A are instead enabling a decentralized AI future. The explosion of entrepreneurial activity around agentic AI reminds me of the best kind of open innovation, much like I saw in the early days of the personal computer and the internet.
I was going to use my opening remarks to sound that theme, and then I read Alex Komoroske’s marvelous essay, “<a href="https://www.techdirt.com/2025/06/16/why-centralized-ai-is-not-our-inevitable-future/" target="_blank" rel="noreferrer noopener">Why Centralized AI Is Not Our Inevitable Future</a>.” So I asked him to do it instead. He’s going to give an updated, developer-focused version of that as our kickoff talk.
Then we’re going into a section on agentic interfaces. We’ve lived for decades with the GUI (either on computers or mobile applications) and the web as the dominant ways we use computers. AI is changing all that.
It’s not just agentic interfaces, though. It’s really developing true AI-native products, searching out the possibilities of this new computing fabric.
<h2 class="wp-block-heading">The Great Interface Rethink</h2>
In the “normal technology” framing, a fundamental technology innovation is distinct from products based on it. Think of the invention of the LLM itself as electricity, and ChatGPT as the equivalent of Edison’s incandescent light bulb and the development of the distribution network to power it.
There’s a bit of a lesson in the fact that the telegraph was the first large-scale practical application of electricity, over 40 years before Edison’s lightbulb. The telephone was another killer app that used electricity to power it. But despite their scale, these were specialized devices. It was the infrastructure for incandescent lighting that turned electricity into a <a href="https://en.wikipedia.org/wiki/General-purpose_technology" target="_blank" rel="noreferrer noopener">general-purpose technology</a>.
The world soon saw electrical resistance products like irons and toasters, and electric motors powering not just factories but household appliances such as washing machines and eventually refrigerators and air conditioning. Many of these household products were plugged into light sockets, since the pronged plug as we know it today wasn’t introduced until 30 years after the first light bulb.
<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="1086" height="824" src="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/Looking-Forward-to-AI-Codecon.png" alt="" class="wp-image-17399" srcset="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/Looking-Forward-to-AI-Codecon.png 1086w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/Looking-Forward-to-AI-Codecon-300x228.png 300w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/09/Looking-Forward-to-AI-Codecon-768x583.png 768w" sizes="auto, (max-width: 1086px) 100vw, 1086px" /><figcaption class="wp-element-caption"><a href="https://www.facebook.com/photo/?fbid=10158281345021884&set=gm.3012577095721332" target="_blank" rel="noreferrer noopener">Found on Facebook</a>: “Any ideas what this would have been used for? I found it after pulling up carpet – it’s in the corner of a closet in my 1920s ‘fixer-upper’ that I’m slowly bringing back to life. It appears to be for a light bulb and the little flip top is just like floor outlets you see today, but can’t figure out why it would be directly on the floor.”</figcaption></figure>
The lesson is that at some point in the development of a general purpose technology, product innovation takes over from pure technology innovation. That’s the phase we’re entering now.
Look at the evolution of LLM-based products: GitHub Copilot embedded AI into Visual Studio Code; the interface was an extension to VS Code, a 10-year-old GUI-based program. Google’s AI efforts were tied into its web-based search products. ChatGPT broke the mold and introduced the first radically new interface since the web browser. Suddenly, chat was the preferred new interface for everything. But Claude took things further with Artifacts and then Claude Code, and once coding assistants gained more complex interfaces, that kicked off today’s fierce competition between coding tools. The next revolution is the construction of a new computing paradigm where software is composed of intelligent, autonomous agents.
I’m really looking forward to Rachel-Lee Nabors’s talk on how, with an agentic interface, we might transcend the traditional browser: AI agents can adapt content directly to users, offering privacy, accessibility, and flexibility that legacy web interfaces cannot match.
But it seems to me that there will be two kinds of agents, which I call “demand side” and “supply side” agents. What’s a “demand side” agent? Instead of navigating complex apps, you’ll simply state your goal. The agent will understand the context, access the necessary tools, and present you with the result. The vision is still science fiction. The reality is often a kludge powered by browser use or API calls, with MCP servers increasingly offering an AI-friendlier interface for those demand-side agents to interact with. But why should it stop there? MCP servers are static interfaces. What if there were agents on both sides of the conversation, in a dynamic negotiation? I suspect that while demand-side agents will be developed by venture funded startups, most server-side agents will be developed by enterprises as a kind of conversational interface for both humans and AI agents that want access to their complex workflows, data, and business models. And those enterprises will often be using agentic platforms tailored for their use. That’s part of the “supply side agent” vision of companies like Sierra. I’ll be talking with Sierra cofounder Clay Bavor about this next step in agentic development.
We’ve grown accustomed to thinking about agents as lonely consumers—“tell me the weather,” “scan my code,” “summarize my inbox.” But that’s only half the story. If we build supply-side agent infrastructure—autonomous, discoverable, governed, negotiated—we unlock agility, resilience, security, and collaboration.
My interest in product innovation, not just advances in the underlying technology, is also why I’m excited about my fireside chat with Josh Woodward, who co-led the team that developed NotebookLM at Google. I’m a huge fan of NotebookLM, which in many ways brought the power of RAG (retrieval-augmented generation) to end users, allowing them to collect a set of documents into a Google drive, and then use that collection to drive chat, audio overviews of documents, study guides, mind maps, and much more.
NotebookLM is also a lovely way to build on the deep collaborative infrastructure provided by Google Drive. We need to think more deeply about collaborative interfaces for AI. Right now, AI interaction is mostly a solitary sport. You can share the outputs with others, but not the generative process. I wrote about this recently in “<a href="https://www.oreilly.com/radar/people-work-in-teams-ai-assistants-in-silos/" target="_blank" rel="noreferrer noopener">People Work in Teams, AI Assistants in Silos</a>.” I think that’s a big miss, and I’m hoping to probe Josh about Google’s plans in this area, and eager to see other innovations in AI-mediated human collaboration.
GitHub is another existing tool for collaboration that has become central to the AI ecosystem. I’m really looking forward to talking with outgoing CEO Thomas Dohmke both about the ways that GitHub already provides a kind of exoskeleton for collaboration when using AI code-generation tools. It seems to me that one of the frontiers of AI-human interfaces will be those that enable not just small teams but eventually large groups to collaborate. I suspect that GitHub may have more to teach us about that future than we now suspect.
And finally, we are now learning that managing context is a critical part of designing effective AI applications. My cochair Addy Osmani will be talking about the emergence of context engineering as a real discipline, and its relevance to agentic AI development.
<h2 class="wp-block-heading">Tool-Chaining Agents and Real Workflows</h2>
Today’s AI tools are largely solo performers—a Copilot suggesting code or a ChatGPT answering a query. The next leap is from single agents to interconnected systems. The program is filled with sessions on “tool-to-tool workflows” and multi-agent systems.
Ken Kousen will showcase the new generation of coding agents, including Claude Code, Codex CLI, Gemini CLI, and Junie, that help developers navigate codebases, automate tasks, and even refactor intelligently. In her talk, Angie Jones takes it further: agents that go beyond code generation to manage PRs, write tests, and update documentation—stepping “out of the IDE” and into real-world workflows.
Even more exciting is the idea of agents collaborating with each other. The Demo Day will showcase a multi-agent coding system where agents share, correct, and evolve code together. This isn’t science fiction; Amit Rustagi’s talk on decentralized AI agent infrastructure using technologies like WebAssembly and IPFS provides a practical architectural framework for making these agent swarms a reality.
<h2 class="wp-block-heading">The Crucial Ingredient: Common Protocols</h2>
How do all these agents talk to each other? How do they discover new tools and use them safely? The answer that echoes throughout the agenda is the Model Context Protocol (MCP).
Much as the distribution network for electricity was the enabler for all of the product innovation of the electrical revolution, MCP is the foundational plumbing, the universal language that will allow this new ecosystem to flourish. Multiple sessions and an entire Demo Day are dedicated to it. We’ll see how Google is using it for agent-to-agent communication, how it can be used to control complex software like Blender with natural language, and even how it can power novel SaaS product demos.
The heavy focus on a standardized protocol signals that the industry is maturing past cool demos and is now building the robust, interoperable infrastructure needed for a true agentic economy.
If the development of the internet is any guide, though, MCP is a beginning, not the end. TCP/IP became the foundation of a layered protocol stack. It is likely that MCP will be followed by many more specialized protocols.
<h2 class="wp-block-heading">Why This Matters</h2>
<figure class="wp-block-table"><table class="has-fixed-layout"><thead><tr><th>Theme</th><th>Why It’s Thrilling</th></tr></thead><tbody><tr><td>Autonomous, Distributed AI</td><td>Agents that chain tasks and operate behind the scenes can unlock entirely new ways of building software.</td></tr><tr><td>Human Empowerment & Privacy</td><td>The push against centralized AI systems is a reminder that tools should serve users, not control them.</td></tr><tr><td>Context as Architecture</td><td>Elevating input design to first-class engineering—this will greatly improve reliability, trust, and AI behavior over time.</td></tr><tr><td>New Developer Roles</td><td>We’re seeing developers transition from writing code to orchestrating agents, designing workflows, and managing systems.</td></tr><tr><td>MCP & Network Effects</td><td>The idea of an “AI-native web,” where agents use standardized protocols to talk, is powerful, open-ended, and full of opportunity.</td></tr></tbody></table></figure>
I look forward to seeing you there!
<hr class="wp-block-separator has-alpha-channel-opacity"/>
We hope you’ll join us at AI Codecon: Coding for the Agentic World on September 9 to explore the tools, workflows, and architectures defining the next era of programming. It’s free to attend. <a href="https://www.oreilly.com/AgenticWorld/" target="_blank" rel="noreferrer noopener">Register now to save your seat</a>. And join us for <a href="https://learning.oreilly.com/live-events/oreilly-demo-day/0642572227968/" target="_blank" rel="noreferrer noopener">O’Reilly Demo Day</a> on September 16 to see how experts are shaping AI systems to work for them via MCP.

]]></content:encoded>
</item>
<item>
<title>Understanding the Rehash Loop</title>
<link>https://www.oreilly.com/radar/understanding-the-rehash-loop/</link>
<pubDate>Wed, 03 Sep 2025 10:21:28 +0000</pubDate>
<dc:creator><![CDATA[Andrew Stellman]]></dc:creator>
<category><![CDATA[Commentary]]></category>
<guid isPermaLink="false">https://www.oreilly.com/radar/?p=17392</guid>
<media:content
url="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2019/06/fractal-1832620_1920_crop-c59dcd2418c34f9dccc1469aba4b0ba1.jpg"
medium="image"
type="image/jpeg"
/>
<custom:subtitle><![CDATA[When AI Gets Stuck]]></custom:subtitle>
<description><![CDATA[This article is part of a series on the Sens-AI Framework—practical habits for learning and coding with AI. In “The Sens-AI Framework: Teaching Developers to Think with AI,” I introduced the concept of the rehash loop—that frustrating pattern where AI tools keep generating variations of the same wrong answer, no matter how you adjust your […]]]></description>
<content:encoded><![CDATA[
This article is part of a series on the <a href="https://www.oreilly.com/radar/the-sens-ai-framework/">Sens-AI Framework</a>—practical habits for learning and coding with AI.
In “<a href="https://www.oreilly.com/radar/the-sens-ai-framework/" target="_blank" rel="noreferrer noopener">The Sens-AI Framework: Teaching Developers to Think with AI</a>,” I introduced the concept of the rehash loop—that frustrating pattern where AI tools keep generating variations of the same wrong answer, no matter how you adjust your prompt. It’s one of the most common failure modes in AI-assisted development, and it deserves a deeper look.
Most developers who use AI in their coding work will recognize a rehash loop. The AI generates code that’s almost right—close enough that you think one more tweak will fix it. So you adjust your prompt, add more detail, explain the problem differently. But the response is essentially the same broken solution with cosmetic changes. Different variable names. Reordered operations. Maybe a comment or two. But fundamentally, it’s the same wrong answer.
<h2 class="wp-block-heading">Recognizing When You’re Stuck</h2>
Rehash loops are frustrating. The model seems so close to understanding what you need but just can’t get you there. Each iteration looks slightly different, which makes you think you’re making progress. Then you test the code and it fails in exactly the same way, or you get the same errors, or you just recognize that it’s a solution that you’ve already seen and dismissed multiple times.
Most developers try to escape through incremental changes—adding details, rewording instructions, nudging the AI toward a fix. These adjustments normally work during regular coding sessions, but in a rehash loop, they lead back to the same constrained set of answers. You can’t tell if there’s no real solution, if you’re asking the wrong question, or if the AI is hallucinating a partial answer and too confident that it works.
When you’re in a rehash loop, the AI isn’t broken. It’s doing exactly what it’s designed to do—generating the most statistically likely response it can, based on the tokens in your prompt and the limited view it has of the conversation. One source of the problem is the context window—an architectural limit on how many tokens the model can process at once. That includes your prompt, any shared code, and the rest of the conversation—usually a few thousand tokens total. The model uses this entire sequence to predict what comes next. Once it has sampled the patterns it finds there, it starts circling.
The variations you get—reordered statements, renamed variables, a tweak here or there—aren’t new ideas. They’re just the model nudging things around in the same narrow probability space.
So if you keep getting the same broken answer, the issue probably isn’t that the model doesn’t know how to help. It’s that you haven’t given it enough to work with.
<h2 class="wp-block-heading">When the Model Runs Out of Context</h2>
A rehash loop is a signal that the AI ran out of context. The model has exhausted the useful information in the context you’ve given it. When you’re stuck in a rehash loop, treat it as a signal instead of a problem. Figure out what context is missing and provide it.
Large language models don’t really understand code the way humans do. They generate suggestions by predicting what comes next in a sequence of text based on patterns they’ve seen in massive training datasets. When you prompt them, they analyze your input and predict likely continuations, but they have no real understanding of your design or requirements unless you explicitly provide that context.
The better context you provide, the more useful and accurate the AI’s answers will be. But when the context is incomplete or poorly framed, the AI’s suggestions can drift, repeat variations, or miss the real problem entirely.
<h2 class="wp-block-heading">Breaking Out of the Loop</h2>
Research becomes especially important when you hit a rehash loop. You need to learn more before reengaging—reading documentation, clarifying requirements with teammates, thinking through design implications, or even starting another session to ask research questions from a different angle. Starting a new chat with a different AI can help because your prompt might steer it toward a different region of its information space and surface new context.
A rehash loop tells you that the model is stuck trying to solve a puzzle without all the pieces. It keeps rearranging the ones it has, but it can’t reach the right solution until you give it the one piece it needs—that extra bit of context that points it to a different part of the model it wasn’t using. That missing piece might be a key constraint, an example, or a goal you haven’t spelled out yet. You typically don’t need to give it a lot of extra information to break out of the loop. The AI doesn’t need a full explanation; it needs just enough new context to steer it into a part of its training data it wasn’t using.
When you recognize you’re in a rehash loop, trying to nudge the AI and vibe-code your way out of it is usually ineffective—it just leads you in circles. (“Vibe coding” means relying on the AI to generate something that looks plausible and hoping it works, without really digesting the output.) Instead, start investigating what’s missing. Ask the AI to explain its thinking: “What assumptions are you making?” or “Why do you think this solves the problem?” That can reveal a mismatch—maybe it’s solving the wrong problem entirely, or it’s missing a constraint you forgot to mention. It’s often especially helpful to open a chat with a different AI, describe the rehash loop as clearly as you can, and ask what additional context might help.
This is where problem framing really starts to matter. If the model keeps circling the same broken pattern, it’s not just a prompt problem—it’s a signal that your framing needs to shift.
Problem framing helps you recognize that the model is stuck in the wrong solution space. Your framing gives the AI the clues it needs to assemble patterns from its training that actually match your intent. After researching the actual problem—not just tweaking prompts—you can transform vague requests into targeted questions that steer the AI away from default responses and toward something useful.
Good framing starts by getting clear about the nature of the problem you’re solving. What exactly are you asking the model to generate? What information does it need to do that? Are you solving the right problem in the first place? A lot of failed prompts come from a mismatch between the developer’s intent and what the model is actually being asked to do. Just like writing good code, good prompting depends on understanding the problem you’re solving and structuring your request accordingly.
<h2 class="wp-block-heading">Learning from the Signal</h2>
When AI keeps circling the same solution, it’s not a failure—it’s information. The rehash loop tells you something about either your understanding of the problem or how you’re communicating it. An incomplete response from the AI is often just a step toward getting the right answer. These moments aren’t failures. They’re signals to do the extra work—often just a small amount of targeted research—that gives the AI the information it needs to get to the right place in its massive information space.
AI doesn’t think for you. While it can make surprising connections by recombining patterns from its training, it can’t generate truly new insight on its own. It’s your context that helps it connect those patterns in useful ways. If you’re hitting rehash loops repeatedly, ask yourself: What does the AI need to know to do this well? What context or requirements might be missing?
Rehash loops are one of the clearest signals that it’s time to step back from rapid generation and engage your critical thinking. They’re frustrating, but they’re also valuable—they tell you exactly when the AI has exhausted its current context and needs your help to move forward.
]]></content:encoded>
</item>
<item>
<title>Radar Trends to Watch: September 2025</title>
<link>https://www.oreilly.com/radar/radar-trends-to-watch-september-2025/</link>
<pubDate>Tue, 02 Sep 2025 10:10:37 +0000</pubDate>
<dc:creator><![CDATA[Mike Loukides]]></dc:creator>
<category><![CDATA[Radar Trends]]></category>
<category><![CDATA[Commentary]]></category>
<guid isPermaLink="false">https://www.oreilly.com/radar/?p=17379</guid>
<media:content
url="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2023/06/radar-1400x950-4.png"
medium="image"
type="image/png"
/>
<custom:subtitle><![CDATA[Developments in AI, Security, Web, and More]]></custom:subtitle>
<description><![CDATA[For better or for worse, AI has colonized this list so thoroughly that AI itself is little more than a list of announcements about new or upgraded models. But there are other points of interest. Is it just a coincidence (possibly to do with BlackHat) that so much happened in security in the past month? […]]]></description>
<content:encoded><![CDATA[
For better or for worse, AI has colonized this list so thoroughly that AI itself is little more than a list of announcements about new or upgraded models. But there are other points of interest. Is it just a coincidence (possibly to do with BlackHat) that so much happened in security in the past month? We’re still seeing programming languages—even some new programming languages for writing AI prompts! If you’re into retrocomputing, the much-beloved Commodore 64 is back—with an upgraded audio chip, a new processor, much more RAM, and all your old ports. Heirloom peripherals should still work.
<h2 class="wp-block-heading">AI</h2>
<ul class="wp-block-list">
<li>OpenAI has released their <a href="https://platform.openai.com/docs/guides/realtime" target="_blank" rel="noreferrer noopener">Realtime APIs</a>. The model supports MCP servers, phone calls using the SIP protocol, and image inputs. The release includes <a href="https://openai.com/index/introducing-gpt-realtime/" target="_blank" rel="noreferrer noopener">gpt-realtime</a>, an advanced speech-to-speech model.</li>
<li>ChatGPT now supports <a href="https://help.openai.com/en/articles/6825453-chatgpt-release-notes#h_fb3ac52750" target="_blank" rel="noreferrer noopener">project-only memory</a>. Project memory, which can use previous conversations for additional context, can be limited to a specific project. Project-only memory gives more control over context and prevents one project’s context from contaminating another.</li>
<li><a href="https://arxiv.org/abs/2501.01665" target="_blank" rel="noreferrer noopener">FairSense</a> is a framework for <a href="https://techxplore.com/news/2025-08-fairness-tool-ai-bias-early.html" target="_blank" rel="noreferrer noopener">investigating whether AI systems are fair</a> early on. FairSense runs long-term simulations to detect whether a system will become unfair as it evolves over time.</li>
<li><a href="https://agents4science.stanford.edu/" target="_blank" rel="noreferrer noopener">Agents4Science</a> is a new academic conference in which all the submissions will be <a href="https://www.technologyreview.com/2025/08/22/1122304/ai-scientist-research-autonomous-agents/" target="_blank" rel="noreferrer noopener">researched, written, reviewed, and presented primarily by AI</a> (using text-to-speech for presentations).</li>
<li>Drew Breunig’s mix and match <a href="https://www.dbreunig.com/2025/08/21/a-guide-to-ai-titles.html" target="_blank" rel="noreferrer noopener">cheat sheet for AI job titles</a> is a classic. </li>
<li>Cohere’s <a href="https://cohere.com/blog/command-a-reasoning" target="_blank" rel="noreferrer noopener">Command A Reasoning</a> is another powerful, partially open reasoning model. It is available on <a href="https://huggingface.co/CohereLabs/command-a-reasoning-08-2025?ref=cohere-ai.ghost.io" target="_blank" rel="noreferrer noopener">Hugging Face</a>. It claims to outperform gpt-oss-120b and DeepSeek R1-0528.</li>
<li>DeepSeek has <a href="https://api-docs.deepseek.com/news/news250821" target="_blank" rel="noreferrer noopener">released</a> DeepSeekV3.1. This is a hybrid model that supports reasoning and nonreasoning use. It’s also faster than R1 and has been designed for agentic tasks. It uses reasoning tokens more economically, and it was much less expensive to train than GPT-5.</li>
<li>Anthropic has added the <a href="https://www.anthropic.com/research/end-subset-conversations" target="_blank" rel="noreferrer noopener">ability to terminate chats</a> to Claude Opus. Chats can be terminated if a user persists in making harmful requests. Terminated chats can’t be continued, although users can start a new chat. The feature is currently experimental.</li>
<li>Google has <a href="https://developers.googleblog.com/en/introducing-gemma-3-270m/" target="_blank" rel="noreferrer noopener">released</a> its <a href="https://feedly.com/i/entry/oWwSZ9Xu4Zg49bpJDa0MWTMCyM67pScdNU+gxmUAZuo=_198aa66551c:380d5:c853ad2e" target="_blank" rel="noreferrer noopener">smallest model yet</a>: Gemma 3 270M. This model is designed for fine-tuning and for deployment on small, limited hardware. Here’s a <a href="https://huggingface.co/spaces/webml-community/bedtime-story-generator" target="_blank" rel="noreferrer noopener">bedtime story generator</a> that runs in the browser, built with Gemma 3 270M. </li>
<li>ChatGPT has <a href="https://www.bleepingcomputer.com/news/artificial-intelligence/openai-rolls-out-gmail-calendar-and-contacts-integration-in-chatgpt/" target="_blank" rel="noreferrer noopener">added GMail, Google Calendar, and Google Contacts</a> to its group of connectors, which integrate ChatGPT with other applications. This information will be used to provide additional context—and presumably will be used for training or discovery in ongoing lawsuits. Fortunately, it’s (at this point) opt-in. </li>
<li>Anthropic has <a href="https://www.bleepingcomputer.com/news/artificial-intelligence/claude-gets-1m-tokens-support-via-api-to-take-on-gemini-25-pro/" target="_blank" rel="noreferrer noopener">upgraded</a> Claude Sonnet 4 with a <a href="https://www.anthropic.com/news/1m-context" target="_blank" rel="noreferrer noopener">1M token context window</a>. The larger context window is only available via the API.</li>
<li>OpenAI <a href="https://openai.com/index/introducing-gpt-5/" target="_blank" rel="noreferrer noopener">released</a> GPT-5. Simon Willison’s <a href="https://simonwillison.net/2025/Aug/7/gpt-5/" target="_blank" rel="noreferrer noopener">review</a> is excellent. It doesn’t feel like a breakthrough, but it is quietly better at delivering good results. It is claimed to be less prone to hallucination and incorrect answers. One quirk is that with ChatGPT, GPT-5 determines which model should respond to your prompt.</li>
<li>Anthropic is researching <a href="https://www.anthropic.com/research/persona-vectors" target="_blank" rel="noreferrer noopener">persona vectors</a> as a means of training a language model to behave correctly. Steering a model toward inappropriate behavior during training can be a kind of “vaccination” against that behavior when the model is deployed, without compromising other aspects of the model’s behavior.</li>
<li>The <a href="https://sakana.ai/dgm/" target="_blank" rel="noreferrer noopener">Darwin Gödel Machine</a> is an agent that can read and modify its own code to improve its performance on tasks. It can add tools, re-organize workflows, and evaluate whether these changes have improved its performance.</li>
<li>Grok is at it again: <a href="https://www.theverge.com/report/718975/xai-grok-imagine-taylor-swifty-deepfake-nudes" target="_blank" rel="noreferrer noopener">generating nude deepfakes of Taylor Swift</a> without being prompted to do so. I’m sure we’ll be told that this was the result of an unauthorized modification to the system prompt. In AI, some things are predictable.</li>
<li>Anthropic has <a href="https://www.anthropic.com/news/claude-opus-4-1" target="_blank" rel="noreferrer noopener">released</a> Claude Opus 4.1, an upgrade to its flagship model. We expect this to be the “gold standard” for generative coding.</li>
<li>OpenAI has <a href="https://openai.com/open-models/" target="_blank" rel="noreferrer noopener">released</a> two open-weight models, their first since GPT-2: <a href="https://huggingface.co/openai/gpt-oss-120b" target="_blank" rel="noreferrer noopener">gpt-oss-120b</a> and <a href="https://huggingface.co/openai/gpt-oss-20b" target="_blank" rel="noreferrer noopener">gpt-oss-20b</a>. They are reasoning models designed for use in agentic applications. Claimed <a href="https://openai.com/index/introducing-gpt-oss/" target="_blank" rel="noreferrer noopener">performance</a> is similar to OpenAI’s o3 and o4-mini.</li>
<li>OpenAI has also released a “response format” named <a href="https://cookbook.openai.com/articles/openai-harmony" target="_blank" rel="noreferrer noopener">Harmony</a>. It’s not quite a protocol, but it is a standard that specifies the format of conversations by defining roles (system, user, etc.) and channels (final, analysis, commentary) for a model’s output.</li>
<li>Can AIs <a href="https://techxplore.com/news/2025-07-ai-evolve-guilt-social-environments.html" target="_blank" rel="noreferrer noopener">evolve guilt</a>? Guilt is expressed in human language; it’s in the training data. The AI that deleted a production database because it “panicked” certainly <a href="https://www.pcgamer.com/software/ai/i-destroyed-months-of-your-work-in-seconds-says-ai-coding-tool-after-deleting-a-devs-entire-database-during-a-code-freeze-i-panicked-instead-of-thinking/" target="_blank" rel="noreferrer noopener">expressed guilt</a>. Whether an AI’s expressions of guilt are meaningful in any way is a different question.</li>
<li><a href="https://github.com/musistudio/claude-code-router" target="_blank" rel="noreferrer noopener">Claude Code Router</a> is a tool for routing Claude Code requests to different models. You can choose different models for different kinds of requests.</li>
<li>Qwen has released a thinking version of their flagship model, called <a href="https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507" target="_blank" rel="noreferrer noopener">Qwen3-235B-A22B-Thinking-2507</a>. Thinking cannot be switched on or off. The model was trained with a new reinforcement learning algorithm called <a href="https://www.arxiv.org/abs/2507.18071" target="_blank" rel="noreferrer noopener">Group Sequence Policy Optimization</a>. It burns a lot of tokens, and it’s not very good at <a href="https://simonwillison.net/2025/Jul/25/qwen3-235b-a22b-thinking-2507/#atom-everything" target="_blank" rel="noreferrer noopener">pelicans</a>.</li>
<li>ChatGPT is releasing “<a href="https://www.bleepingcomputer.com/news/artificial-intelligence/chatgpt-is-rolling-out-personality-toggles-to-become-your-assistant/" target="_blank" rel="noreferrer noopener">personalities</a>” that control how it formulates its responses. Users can select the personality they want to respond: robot, cynic, listener, sage, and presumably more. </li>
<li>DeepMind has created <a href="https://blog.google/technology/google-deepmind/aeneas/" target="_blank" rel="noreferrer noopener">Aeneas</a>, a new model designed to help scholars understand ancient fragments. In ancient text, large pieces are often missing. Can AI help place these fragments into contexts where they can be understood? Latin only, for now.</li>
</ul>
<h2 class="wp-block-heading">Security</h2>
<ul class="wp-block-list">
<li>The US Cybersecurity and Infrastructure Security Agency (CISA) has <a href="https://www.bleepingcomputer.com/news/security/cisa-warns-of-actively-exploited-git-code-execution-flaw/" target="_blank" rel="noreferrer noopener">warned</a> that a serious <a href="https://nvd.nist.gov/vuln/detail/cve-2025-48384" target="_blank" rel="noreferrer noopener">code execution vulnerability</a> in Git is currently being exploited in the wild.</li>
<li>Is it possible to build an agentic browser that is <a href="https://guard.io/labs/scamlexity-we-put-agentic-ai-browsers-to-the-test-they-clicked-they-paid-they-failed" target="_blank" rel="noreferrer noopener">safe</a> from prompt injection? <a href="https://brave.com/blog/comet-prompt-injection/" target="_blank" rel="noreferrer noopener">Probably</a> <a href="https://simonwillison.net/2025/Aug/25/agentic-browser-security/#atom-everything" target="_blank" rel="noreferrer noopener">not</a>. Separating user instructions from website content isn’t possible. If a browser can’t take direction from the content of a web page, how is it to act as an agent?</li>
<li>The solution to Part 4 of <a href="https://en.wikipedia.org/wiki/Kryptos" target="_blank" rel="noreferrer noopener">Kryptos</a>, the CIA’s decades-old cryptographic sculpture, is <a href="https://www.schneier.com/blog/archives/2025/08/jim-sanborn-is-auctioning-off-the-solution-to-part-four-of-the-kryptos-sculpture.html" target="_blank" rel="noreferrer noopener">for sale</a>! Jim Sanborn, the creator of Kryptos, is auctioning the solution. He hopes that the winner will preserve the secret and take over verifying people’s claims to have solved the puzzle. </li>
<li>Remember XZ, the supply-chain attack that granted backdoor access via a trojaned compression library? It <a href="https://www.binarly.io/blog/persistent-risk-xz-utils-backdoor-still-lurking-in-docker-images" target="_blank" rel="noreferrer noopener">never went away</a>. Although the affected libraries were quickly patched, it’s still active, and propagating, via Docker images that were built with unpatched libraries. Some gifts keep giving.</li>
<li>For August, <a href="https://embracethered.com/blog/" target="_blank" rel="noreferrer noopener">Embrace the Red</a> published <a href="https://embracethered.com/blog/posts/2025/announcement-the-month-of-ai-bugs/" target="_blank" rel="noreferrer noopener">The Month of AI Bugs</a>, a daily post about AI vulnerabilities (mostly various forms of prompt injection). This series is essential reading for AI developers and for security professionals.</li>
<li>NIST has finalized a <a href="https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-232.pdf" target="_blank" rel="noreferrer noopener">standard</a> for <a href="https://techxplore.com/news/2025-08-lightweight-cryptography-standard-small-devices.html" target="_blank" rel="noreferrer noopener">lightweight cryptography</a>. Lightweight cryptography is a cryptographic system designed for use by small devices. It is useful both for encrypting sensitive data and for authentication. </li>
<li>The <a href="https://darkpatternstipline.org/" target="_blank" rel="noreferrer noopener">Dark Patterns Tip Line</a> is a site for reporting dark patterns: design features in websites and applications that are designed to trick us into acting against our own interest.</li>
<li>OpenSSH supports <a href="https://www.openssh.com/pq.html" target="_blank" rel="noreferrer noopener">post-quantum key agreement</a>, and in versions 10.1 and later, will warn users when they select a non-post-quantum key agreement scheme.</li>
<li><a href="https://arstechnica.com/security/2025/08/adult-sites-use-malicious-svg-files-to-rack-up-likes-on-facebook/" target="_blank" rel="noreferrer noopener">SVG files can carry a malware payload</a>; pornographic SVGs include JavaScript payloads that automate clicking “like.” That’s a simple attack with few consequences, but much more is possible, including cross-site scripting, denial of service, and other exploits.</li>
<li>Google’s AI agent for discovering security flaws, <a href="https://cloud.google.com/blog/products/identity-security/cloud-ciso-perspectives-our-big-sleep-agent-makes-big-leap" target="_blank" rel="noreferrer noopener">Big Sleep</a>, has <a href="https://techcrunch.com/2025/08/04/google-says-its-ai-based-bug-hunter-found-20-security-vulnerabilities/" target="_blank" rel="noreferrer noopener">found 20 flaws</a> in popular software. DeepMind discovered and reproduced the flaws, which were then verified by human security experts and reported. Details won’t be provided until the flaws have been fixed.</li>
<li>The US CISA (Cybersecurity and Infrastructure Security Agency) has <a href="https://www.bleepingcomputer.com/news/security/cisa-open-sources-thorium-platform-for-malware-forensic-analysis/" target="_blank" rel="noreferrer noopener">open-sourced</a> <a href="https://www.cisa.gov/resources-tools/resources/thorium" target="_blank" rel="noreferrer noopener">Thorium</a>, a platform for malware and forensic analysis.</li>
<li>Prompt injection, again: A new prompt injection attack embeds <a href="https://bdtechtalks.com/2025/07/30/legalpwn-llm-prompt-injection/" target="_blank" rel="noreferrer noopener">instructions in language that appears to be copyright notices and other legal fine print</a>. To avoid litigation, many models are configured to prioritize legal instructions.</li>
<li>Light can be <a href="https://techxplore.com/news/2025-07-secret-codes-fake-videos.html" target="_blank" rel="noreferrer noopener">watermarked</a>; this may be useful as a technique for detecting fake or manipulated video.</li>
<li><a href="https://www.bleepingcomputer.com/news/security/ai-cuts-vciso-workload-by-68-percent-as-demand-skyrockets-new-report-finds/" target="_blank" rel="noreferrer noopener">vCISO (Virtual CISO) services are thriving</a>, particularly among small and mid-size businesses that can’t afford a full security team. The use of AI is cutting the vCISO workload. But who takes the blame when there’s an incident?</li>
<li>A <a href="https://www.bleepingcomputer.com/news/security/hackers-target-python-devs-in-phishing-attacks-using-fake-pypi-site/" target="_blank" rel="noreferrer noopener">phishing attack against PyPI users</a> directs them to a fake PyPI site that tells them to verify their login credentials. Stolen credentials could be used to plant malware in the genuine PyPI repository. Users of <a href="https://www.bleepingcomputer.com/news/security/mozilla-warns-of-phishing-attacks-targeting-add-on-developers/" target="_blank" rel="noreferrer noopener">Mozilla’s add-on repository</a> have also been targeted by phishing attacks.</li>
<li>A new ransomware group named <a href="https://arstechnica.com/security/2025/07/after-blacksuit-is-taken-down-new-ransomware-group-chaos-emerges/" target="_blank" rel="noreferrer noopener">Chaos</a> appears to be a rebranding of the BlackSuit group, which was taken down recently. BlackSuit itself is a rebranding of the Royal group, which in turn is a descendant of the Conti group. Whack-a-mole continues.</li>
<li>Google’s <a href="https://security.googleblog.com/2025/07/introducing-oss-rebuild-open-source.html" target="_blank" rel="noreferrer noopener">OSS Rebuild</a> project is an important step forward in supply chain security. Rebuild provides build definitions along with metadata that can confirm projects were built correctly. OSS Rebuild currently supports the NPM, PyPl, and Crates ecosystems.</li>
<li>The <a href="https://www.bleepingcomputer.com/news/security/npm-package-is-with-28m-weekly-downloads-infected-devs-with-malware/" target="_blank" rel="noreferrer noopener">JavaScript package “is,</a>” which does some simple type checking, has been infected with malware. Supply chain security is a huge issue—be careful what you install!</li>
</ul>
<h2 class="wp-block-heading">Programming</h2>
<ul class="wp-block-list">
<li><a href="https://github.com/automazeio/ccpm" target="_blank" rel="noreferrer noopener">Claude Code PM</a> is a workflow management system for programming with Claude. It manages PRDs, GitHub, and parallel execution of coding agents. It claims to facilitate collaboration between multiple Claude instances working on the same project. </li>
<li>Rust is increasingly used to <a href="https://thenewstack.io/rust-pythons-new-performance-engine/" target="_blank" rel="noreferrer noopener">implement performance-critical extensions</a> to Python, gradually displacing C. Polars, Pydantic, and FastAPI are three libraries that rely on Rust.</li>
<li>Microsoft’s <a href="https://microsoft.github.io/poml/latest/" target="_blank" rel="noreferrer noopener">Prompt Orchestration Markup Language</a> (<a href="https://medium.com/data-science-in-your-pocket/microsoft-poml-programming-language-for-prompting-adfc846387a4" target="_blank" rel="noreferrer noopener">POML</a>) is an HTML-like markup language for writing prompts. It is then compiled into the actual prompt. POML is good at templating and has tags for tabular and document data. Is this a step forward? You be the judge.</li>
<li><a href="https://claudiacode.com/" target="_blank" rel="noreferrer noopener">Claudia</a> is an “elegant desktop companion” for Claude Code; it turns terminal-based Claude Code into something more like an IDE, though it seems to focus more on the workflow than on coding.</li>
<li>Google’s <a href="https://developers.googleblog.com/en/introducing-langextract-a-gemini-powered-information-extraction-library/" target="_blank" rel="noreferrer noopener">LangExtract</a> is a simple but powerful Python library for extracting text from documents. It relies on examples, rather than regular expressions or other hacks, and shows the exact context in which the extracts occur. LangExtract is open source.</li>
<li>Microsoft appears to be <a href="https://www.theverge.com/news/757461/microsoft-github-thomas-dohmke-resignation-coreai-team-transition" target="_blank" rel="noreferrer noopener">integrating GitHub into its AI team</a> rather than running it as an independent organization. What this means for GitHub users is unclear. </li>
<li>Cursor now has a <a href="https://cursor.com/en/cli" target="_blank" rel="noreferrer noopener">command-line interface</a>, almost certainly a belated response to the success of Claude Code CLI and Gemini CLI. </li>
<li><a href="https://thenewstack.io/why-latency-is-quietly-breaking-enterprise-ai-at-scale/" target="_blank" rel="noreferrer noopener">Latency</a> is a problem for enterprise AI. And the root cause of latency in AI applications is usually the database.</li>
<li>The <a href="https://www.musicradar.com/music-tech/weve-been-sleeping-for-30-years-please-excuse-us-the-commodore-64-is-back-packed-with-extra-power-for-chiptune-music-makers" target="_blank" rel="noreferrer noopener">Commodore 64</a> is back. With several orders of magnitude more RAM. And all the original ports, plus HDMI. </li>
<li>Google has <a href="https://blog.google/technology/developers/introducing-gemini-cli-github-actions/" target="_blank" rel="noreferrer noopener">announced</a> Gemini CLI GitHub Actions, an addition to their agentic coder that allows it to work directly with GitHub repositories. </li>
<li><a href="https://www.infoworld.com/article/4029053/jetbrains-working-on-higher-abstraction-programming-language.html" target="_blank" rel="noreferrer noopener">JetBrains is developing a new programming language</a> for use when programming with LLMs. That language may be a dialect of English. (<a href="https://www.oreilly.com/radar/formal-informal-languages/" target="_blank" rel="noreferrer noopener">Formal informal languages</a>, anyone?) </li>
<li><a href="https://www.ponylang.io/discover/" target="_blank" rel="noreferrer noopener">Pony</a> is a new programming language that is type-safe, memory-safe, exception-safe, race-safe, and deadlock-safe. You can <a href="https://playground.ponylang.io/" target="_blank" rel="noreferrer noopener">try</a> it in a browser-based playground.</li>
</ul>
<h2 class="wp-block-heading">Web</h2>
<ul class="wp-block-list">
<li>The AT Protocol is the core of Bluesky. Here’s a <a href="https://mackuba.eu/2025/08/20/introduction-to-atproto/" target="_blank" rel="noreferrer noopener">tutorial</a>; use it to build your own Bluesky services, in turn making Bluesky truly federate. </li>
<li>Social media is broken, and <a href="https://arstechnica.com/science/2025/08/study-social-media-probably-cant-be-fixed/" target="_blank" rel="noreferrer noopener">probably can’t be fixed</a>. Now you know. The surprise is that the problem isn’t “algorithms” for maximizing engagement; take algorithms away and everything stays the same or gets worse. </li>
<li>The <a href="https://waxy.org/2025/08/vote-on-the-2025-tiny-awards-finalists/" target="_blank" rel="noreferrer noopener">Tiny Awards Finalists</a> show just how much is possible on the Web. They’re moving, creative, and playful. For example, the <a href="https://www.trafficcamphotobooth.com/about.html" target="_blank" rel="noreferrer noopener">Traffic Cam Photobooth</a> lets people use traffic cameras to take pictures of themselves, playing with ever-present automated surveillance.</li>
<li>A US federal court has <a href="https://storage.courtlistener.com/recap/gov.uscourts.cand.372884/gov.uscourts.cand.372884.756.0_2.pdf" target="_blank" rel="noreferrer noopener">found</a> that <a href="https://arstechnica.com/tech-policy/2025/08/jury-finds-meta-broke-wiretap-law-by-collecting-data-from-period-tracker-app/" target="_blank" rel="noreferrer noopener">Facebook illegally collected data</a> from the women’s health app Flo. </li>
<li>The <a href="https://www.htmlhobbyist.com/" target="_blank" rel="noreferrer noopener">HTML Hobbyist</a> is a great site for people who want to create their own presence on the web—outside of walled gardens, without mind-crushing frameworks. It’s not difficult, and it’s not expensive.</li>
</ul>
<h2 class="wp-block-heading">Biology and Quantum Computing</h2>
<ul class="wp-block-list">
<li>Scientists have created <a href="https://phys.org/news/2025-08-scientists-cells-biological-qubit-multidisciplinary.html" target="_blank" rel="noreferrer noopener">biological qubits</a>: quantum qubits built from proteins in living cells. These probably won’t be used to break cryptography, but they are likely to give us insight into how quantum processes work inside living things.</li>
</ul>
]]></content:encoded>
</item>
<item>
<title>Working with Contexts</title>
<link>https://www.oreilly.com/radar/working-with-contexts/</link>
<pubDate>Thu, 28 Aug 2025 10:02:43 +0000</pubDate>
<dc:creator><![CDATA[Drew Breunig]]></dc:creator>
<category><![CDATA[AI & ML]]></category>
<category><![CDATA[Deep Dive]]></category>
<guid isPermaLink="false">https://www.oreilly.com/radar/?p=17373</guid>
<media:content
url="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/Abstract-colors-3.jpg"
medium="image"
type="image/jpeg"
/>
<description><![CDATA[The following article comes from two blog posts by Drew Breunig: “How Long Contexts Fail” and “How to Fix Your Contexts.” Managing Your Context Is the Key to Successful Agents As frontier model context windows continue to grow,1 with many supporting up to 1 million tokens, I see many excited discussions about how long-context windows […]]]></description>
<content:encoded><![CDATA[
The following article comes from two blog posts by Drew Breunig: “<a href="https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html" target="_blank" rel="noreferrer noopener">How Long Contexts Fail</a>” and “<a href="https://www.dbreunig.com/2025/06/26/how-to-fix-your-context.html" target="_blank" rel="noreferrer noopener">How to Fix Your Contexts</a>.”
<h2 class="wp-block-heading">Managing Your Context Is the Key to Successful Agents</h2>
As frontier model context windows continue to grow,1 with many supporting up to 1 million tokens, I see many excited discussions about how long-context windows will unlock the agents of our dreams. After all, with a large enough window, you can simply throw everything into a prompt you might need—tools, documents, instructions, and more—and let the model take care of the rest.
Long contexts kneecapped RAG enthusiasm (no need to find the best doc when you can fit it all in the prompt!), enabled MCP hype (connect to every tool and models can do any job!), and fueled enthusiasm for agents.2
But in reality, longer contexts do not generate better responses. Overloading your context can cause your agents and applications to fail in surprising ways. Contexts can become poisoned, distracting, confusing, or conflicting. This is especially problematic for agents, which rely on context to gather information, synthesize findings, and coordinate actions.
Let’s run through the ways contexts can get out of hand, then review methods to mitigate or entirely avoid context fails.
<h3 class="wp-block-heading">Context Poisoning</h3>
Context poisoning is when a hallucination or other error makes it into the context, where it is repeatedly referenced.
The DeepMind team called out context poisoning in the <a href="https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf" target="_blank" rel="noreferrer noopener">Gemini 2.5 technical report</a>, which <a href="https://www.dbreunig.com/2025/06/17/an-agentic-case-study-playing-pok%C3%A9mon-with-gemini.html" target="_blank" rel="noreferrer noopener">we broke down previously</a>. When playing Pokémon, the Gemini agent would occasionally hallucinate, poisoning its context:
<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
An especially egregious form of this issue can take place with “context poisoning”—where many parts of the context (goals, summary) are “poisoned” with misinformation about the game state, which can often take a very long time to undo. As a result, the model can become fixated on achieving impossible or irrelevant goals.
</blockquote>
If the “goals” section of its context was poisoned, the agent would develop nonsensical strategies and repeat behaviors in pursuit of a goal that cannot be met.
<h3 class="wp-block-heading">Context Distraction</h3>
Context distraction is when a context grows so long that the model over-focuses on the context, neglecting what it learned during training.
As context grows during an agentic workflow—as the model gathers more information and builds up history—this accumulated context can become distracting rather than helpful. The Pokémon-playing Gemini agent demonstrated this problem clearly:
<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
While Gemini 2.5 Pro supports 1M+ token context, making effective use of it for agents presents a new research frontier. In this agentic setup, it was observed that as the context grew significantly beyond 100k tokens, the agent showed a tendency toward favoring repeating actions from its vast history rather than synthesizing novel plans. This phenomenon, albeit anecdotal, highlights an important distinction between long-context for retrieval and long-context for multistep, generative reasoning.
</blockquote>
Instead of using its training to develop new strategies, the agent became fixated on repeating past actions from its extensive context history.
For smaller models, the distraction ceiling is much lower. A <a href="https://www.databricks.com/blog/long-context-rag-performance-llms" target="_blank" rel="noreferrer noopener">Databricks study</a> found that model correctness began to fall around 32k for Llama 3.1-405b and earlier for smaller models.
If models start to misbehave long before their context windows are filled, what’s the point of super large context windows? In a nutshell: summarization3 and fact retrieval. If you’re not doing either of those, be wary of your chosen model’s distraction ceiling.
<h3 class="wp-block-heading">Context Confusion</h3>
Context confusion is when superfluous content in the context is used by the model to generate a low-quality response.
For a minute there, it really seemed like everyone was going to ship an <a href="https://www.dbreunig.com/2025/03/18/mcps-are-apis-for-llms.html" target="_blank" rel="noreferrer noopener">MCP</a>. The dream of a powerful model, connected to all your services and stuff, doing all your mundane tasks felt within reach. Just throw all the tool descriptions into the prompt and hit go. <a href="https://www.dbreunig.com/2025/05/07/claude-s-system-prompt-chatbots-are-more-than-just-models.html" target="_blank" rel="noreferrer noopener">Claude’s system prompt</a> showed us the way, as it’s mostly tool definitions or instructions for using tools.
But even if <a href="https://www.dbreunig.com/2025/06/16/drawbridges-go-up.html" target="_blank" rel="noreferrer noopener">consolidation and competition don’t slow MCPs</a>, context confusion will. It turns out there can be such a thing as too many tools.
The <a href="https://gorilla.cs.berkeley.edu/leaderboard.html" target="_blank" rel="noreferrer noopener">Berkeley Function-Calling Leaderboard</a> is a tool-use benchmark that evaluates the ability of models to effectively use tools to respond to prompts. Now on its third version, the leaderboard shows that every model performs worse when provided with more than one tool.4 Further, the Berkeley team, “designed scenarios where none of the provided functions are relevant…we expect the model’s output to be no function call.” Yet, all models will occasionally call tools that aren’t relevant.
Browsing the function-calling leaderboard, you can see the problem get worse as the models get smaller:
<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="866" height="358" src="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/Tool-Calling-Irrelevance-Score-for-Gemma-Models.png" alt="Tool-calling irrelevance score for Gemma models (chart from dbreunig.com, source: Berkeley Function-Calling Leaderboard; created with Datawrapper)" class="wp-image-17374" title="Tool-Calling Irrelevance Score for Gemma Models" srcset="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/Tool-Calling-Irrelevance-Score-for-Gemma-Models.png 866w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/Tool-Calling-Irrelevance-Score-for-Gemma-Models-300x124.png 300w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/Tool-Calling-Irrelevance-Score-for-Gemma-Models-768x317.png 768w" sizes="auto, (max-width: 866px) 100vw, 866px" /></figure>
A striking example of context confusion can be seen in a <a href="https://arxiv.org/pdf/2411.15399?" target="_blank" rel="noreferrer noopener">recent paper</a> that evaluated small model performance on the <a href="https://arxiv.org/abs/2404.15500" target="_blank" rel="noreferrer noopener">GeoEngine benchmark</a>, a trial that features 46 different tools. When the team gave a quantized (compressed) Llama 3.1 8b a query with all 46 tools, it failed, even though the context was well within the 16k context window. But when they only gave the model 19 tools, it succeeded.
The problem is, if you put something in the context, the model has to pay attention to it. It may be irrelevant information or needless tool definitions, but the model will take it into account. Large models, especially reasoning models, are getting better at ignoring or discarding superfluous context, but we continually see worthless information trip up agents. Longer contexts let us stuff in more info, but this ability comes with downsides.
<h3 class="wp-block-heading">Context Clash</h3>
Context clash is when you accrue new information and tools in your context that conflicts with other information in the context.
This is a more problematic version of context confusion. The bad context here isn’t irrelevant, it directly conflicts with other information in the prompt.
A Microsoft and Salesforce team documented this brilliantly in a <a href="https://arxiv.org/pdf/2505.06120" target="_blank" rel="noreferrer noopener">recent paper</a>. The team took prompts from multiple benchmarks and “sharded” their information across multiple prompts. Think of it this way: Sometimes, you might sit down and type paragraphs into ChatGPT or Claude before you hit enter, considering every necessary detail. Other times, you might start with a simple prompt, then add further details when the chatbot’s answer isn’t satisfactory. The Microsoft/Salesforce team modified benchmark prompts to look like these multistep exchanges:
<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="867" height="196" src="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/MicrosoftSalesforce-team-benchmark-prompts.png" alt="Microsoft/Salesforce team benchmark prompts" class="wp-image-17375" title="Microsoft/Salesforce team benchmark prompts" srcset="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/MicrosoftSalesforce-team-benchmark-prompts.png 867w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/MicrosoftSalesforce-team-benchmark-prompts-300x68.png 300w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/MicrosoftSalesforce-team-benchmark-prompts-768x174.png 768w" sizes="auto, (max-width: 867px) 100vw, 867px" /></figure>
All the information from the prompt on the left side is contained within the several messages on the right side, which would be played out in multiple chat rounds.
The sharded prompts yielded dramatically worse results, with an average drop of 39%. And the team tested a range of models—OpenAI’s vaunted o3’s score dropped from 98.1 to 64.1.
What’s going on? Why are models performing worse if information is gathered in stages rather than all at once?
The answer is context confusion: The assembled context, containing the entirety of the chat exchange, contains early attempts by the model to answer the challenge before it has all the information. These incorrect answers remain present in the context and influence the model when it generates its final answer. The team writes:
<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
We find that LLMs often make assumptions in early turns and prematurely attempt to generate final solutions, on which they overly rely. In simpler terms, we discover that when LLMs take a wrong turn in a conversation, they get lost and do not recover.
</blockquote>
This does not bode well for agent builders. Agents assemble context from documents, tool calls, and from other models tasked with subproblems. All of this context, pulled from diverse sources, has the potential to disagree with itself. Further, when you connect to MCP tools you didn’t create there’s a greater chance their descriptions and instructions clash with the rest of your prompt.
<h2 class="wp-block-heading">Learnings</h2>
The arrival of million-token context windows felt transformative. The ability to throw everything an agent might need into the prompt inspired visions of superintelligent assistants that could access any document, connect to every tool, and maintain perfect memory.
But, as we’ve seen, bigger contexts create new failure modes. Context poisoning embeds errors that compound over time. Context distraction causes agents to lean heavily on their context and repeat past actions rather than push forward. Context confusion leads to irrelevant tool or document usage. Context clash creates internal contradictions that derail reasoning.
These failures hit agents hardest because agents operate in exactly the scenarios where contexts balloon: gathering information from multiple sources, making sequential tool calls, engaging in multi-turn reasoning, and accumulating extensive histories.
Fortunately, there are solutions!
<h2 class="wp-block-heading">Mitigating and Avoiding Context Failures</h2>
Let’s run through the ways we can mitigate or avoid context failures entirely.
Everything is about information management. Everything in the context influences the response. We’re back to the old programming adage of “<a href="https://en.wikipedia.org/wiki/Garbage_in,_garbage_out" target="_blank" rel="noreferrer noopener">garbage in, garbage out</a>.” Thankfully, there’s plenty of options for dealing with the issues above.
<h3 class="wp-block-heading">RAG</h3>
Retrieval-augmented generation (RAG) is the act of selectively adding relevant information to help the LLM generate a better response.
Because so much has been written about RAG, we’re not going to cover it here beyond saying: It’s very much alive.
Every time a model ups the context window ante, a new “RAG is dead” debate is born. The last significant event was when Llama 4 Scout landed with a 10 million token window. At that size, it’s really tempting to think, “Screw it, throw it all in,” and call it a day.
But, as we’ve already covered, if you treat your context like a junk drawer, the junk will <a href="https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html#context-confusion" target="_blank" rel="noreferrer noopener">influence your response</a>. If you want to learn more, here’s a <a href="https://maven.com/p/569540/i-don-t-use-rag-i-just-retrieve-documents" target="_blank" rel="noreferrer noopener">new course that looks great</a>.
<h3 class="wp-block-heading">Tool Loadout</h3>
Tool loadout is the act of selecting only relevant tool definitions to add to your context.
The term “loadout” is a gaming term that refers to the specific combination of abilities, weapons, and equipment you select before a level, match, or round. Usually, your loadout is tailored to the context—the character, the level, the rest of your team’s makeup, and your own skill set. Here, we’re borrowing the term to describe selecting the most relevant tools for a given task.
Perhaps the simplest way to select tools is to apply RAG to your tool descriptions. This is exactly what Tiantian Gan and Qiyao Sun did, which they detail in their paper “<a href="https://arxiv.org/abs/2505.03275" target="_blank" rel="noreferrer noopener">RAG MCP</a>.” By storing their tool descriptions in a vector database, they’re able to select the most relevant tools given an input prompt.
When prompting DeepSeek-v3, the team found that selecting the right tools becomes critical when you have more than 30 tools. Above 30, the descriptions of the tools begin to overlap, creating confusion. Beyond 100 tools, the model was virtually guaranteed to fail their test. Using RAG techniques to select fewer than 30 tools yielded dramatically shorter prompts and resulted in as much as 3x better tool selection accuracy.
For smaller models, the problems begin long before we hit 30 tools. One paper we touched on previously, “<a href="https://arxiv.org/abs/2411.15399" target="_blank" rel="noreferrer noopener">Less is More</a>,” demonstrated that Llama 3.1 8b fails a benchmark when given 46 tools, but succeeds when given only 19 tools. The issue is context confusion, not context window limitations.
To address this issue, the team behind “Less is More” developed a way to dynamically select tools using an LLM-powered tool recommender. The LLM was prompted to reason about “number and type of tools it ‘believes’ it requires to answer the user’s query.” This output was then semantically searched (tool RAG, again) to determine the final loadout. They tested this method with the <a href="https://gorilla.cs.berkeley.edu/leaderboard.html" target="_blank" rel="noreferrer noopener">Berkeley Function-Calling Leaderboard</a>, finding Llama 3.1 8b performance improved by 44%.
The “Less is More” paper notes two other benefits to smaller contexts—reduced power consumption and speed—crucial metrics when operating at the edge (meaning, running an LLM on your phone or PC, not on a specialized server). Even when their dynamic tool selection method failed to improve a model’s result, the power savings and speed gains were worth the effort, yielding savings of 18% and 77%, respectively.
Thankfully, most agents have smaller surface areas that only require a few hand-curated tools. But if the breadth of functions or the amount of integrations needs to expand, always consider your loadout.
<h3 class="wp-block-heading">Context Quarantine</h3>
Context quarantine is the act of isolating contexts in their own dedicated threads, each used separately by one or more LLMs.
We see better results when our contexts aren’t too long and don’t sport irrelevant content. One way to achieve this is to break our tasks up into smaller, isolated jobs—each with its own context.
There are <a href="https://arxiv.org/abs/2402.14207" target="_blank" rel="noreferrer noopener">many</a> <a href="https://www.microsoft.com/en-us/research/articles/magentic-one-a-generalist-multi-agent-system-for-solving-complex-tasks/" target="_blank" rel="noreferrer noopener">examples</a> of this tactic, but an accessible write-up of this strategy is Anthropic’s <a href="https://www.anthropic.com/engineering/built-multi-agent-research-system" target="_blank" rel="noreferrer noopener">blog post detailing its multi-agent research system</a>. They write:
<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
The essence of search is compression: distilling insights from a vast corpus. Subagents facilitate compression by operating in parallel with their own context windows, exploring different aspects of the question simultaneously before condensing the most important tokens for the lead research agent. Each subagent also provides separation of concerns—distinct tools, prompts, and exploration trajectories—which reduces path dependency and enables thorough, independent investigations.
</blockquote>
Research lends itself to this design pattern. When given a question, multiple agents can identify and separately prompt several subquestions or areas of exploration. This not only speeds up the information gathering and distillation (if there’s compute available), but it keeps each context from accruing too much information or information not relevant to a given prompt, delivering higher quality results:
<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
Our internal evaluations show that multi-agent research systems excel especially for breadth-first queries that involve pursuing multiple independent directions simultaneously. We found that a multi-agent system with Claude Opus 4 as the lead agent and Claude Sonnet 4 subagents outperformed single-agent Claude Opus 4 by 90.2% on our internal research eval. For example, when asked to identify all the board members of the companies in the Information Technology S&P 500, the multi-agent system found the correct answers by decomposing this into tasks for subagents, while the single-agent system failed to find the answer with slow, sequential searches.
</blockquote>
This approach also helps with tool loadouts, as the agent designer can create several agent archetypes with their own dedicated loadout and instructions for how to utilize each tool.
The challenge for agent builders, then, is to find opportunities for isolated tasks to spin out onto separate threads. Problems that require context-sharing among multiple agents aren’t particularly suited to this tactic.
If your agent’s domain is at all suited to parallelization, be sure to <a href="https://www.anthropic.com/engineering/built-multi-agent-research-system" target="_blank" rel="noreferrer noopener">read the whole Anthropic write-up</a>. It’s excellent.
<h3 class="wp-block-heading">Context Pruning</h3>
Context pruning is the act of removing irrelevant or otherwise unneeded information from the context.
Agents accrue context as they fire off tools and assemble documents. At times, it’s worth pausing to assess what’s been assembled and remove the cruft. This could be something you task your main LLM with or you could design a separate LLM-powered tool to review and edit the context. Or you could choose something more tailored to the pruning task.
Context pruning has a (relatively) long history, as context lengths were a more problematic bottleneck in the natural language processing (NLP) field prior to ChatGPT. Building on this history, a current pruning method is <a href="https://arxiv.org/abs/2501.16214" target="_blank" rel="noreferrer noopener">Provence</a>, “an efficient and robust context pruner for question answering.”
Provence is fast, accurate, simple to use, and relatively small—only 1.75 GB. You can call it in a few lines, like so:
<pre class="wp-block-code"><code>from transformers import AutoModel
provence = AutoModel.from_pretrained("naver/provence-reranker-debertav3-v1", trust_remote_code=True)
# Read in a markdown version of the Wikipedia entry for Alameda, CA
with open('alameda_wiki.md', 'r', encoding='utf-8') as f:
alameda_wiki = f.read()
# Prune the article, given a question
question = 'What are my options for leaving Alameda?'
provence_output = provence.process(question, alameda_wiki)</code></pre>
Provence edited the article, cutting 95% of the content, leaving me with only <a href="https://gist.github.com/dbreunig/b3bdd9eb34bc264574954b2b954ebe83" target="_blank" rel="noreferrer noopener">this relevant subset</a>. It nailed it.
One could employ Provence or a similar function to cull documents or the entire context. Further, this pattern is a strong argument for maintaining a structured5 version of your context in a dictionary or other form, from which you assemble a compiled string prior to every LLM call. This structure would come in handy when pruning, allowing you to ensure the main instructions and goals are preserved while the document or history sections can be pruned or summarized.
<h3 class="wp-block-heading">Context Summarization</h3>
Context summarization is the act of boiling down an accrued context into a condensed summary.
Context summarization first appeared as a tool for dealing with smaller context windows. As your chat session came close to exceeding the maximum context length, a summary would be generated and a new thread would begin. Chatbot users did this manually in ChatGPT or Claude, asking the bot to generate a short recap that would then be pasted into a new session.
However, as context windows increased, agent builders discovered there are benefits to summarization besides staying within the total context limit. As we’ve seen, beyond 100,000 tokens the context becomes distracting and causes the agent to rely on its accumulated history rather than training. Summarization can help it “start over” and avoid repeating context-based actions.
Summarizing your context is easy to do, but hard to perfect for any given agent. Knowing what information should be preserved and detailing that to an LLM-powered compression step is critical for agent builders. It’s worth breaking out this function as its own LLM-powered stage or app, which allows you to collect evaluation data that can inform and optimize this task directly.
<h3 class="wp-block-heading">Context Offloading</h3>
Context offloading is the act of storing information outside the LLM’s context, usually via a tool that stores and manages the data.
This might be my favorite tactic, if only because it’s so simple you don’t believe it will work.
Again, <a href="https://www.anthropic.com/engineering/claude-think-tool" target="_blank" rel="noreferrer noopener">Anthropic has a good write-up of the technique</a>, which details their “think” tool, which is basically a scratchpad:
<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
With the “think” tool, we’re giving Claude the ability to include an additional thinking step—complete with its own designated space—as part of getting to its final answer… This is particularly helpful when performing long chains of tool calls or in long multi-step conversations with the user.
</blockquote>
I really appreciate the research and other writing Anthropic publishes, but I’m not a fan of this tool’s name. If this tool were called <code>scratchpad</code>, you’d know its function immediately. It’s a place for the model to write down notes that don’t cloud its context and are available for later reference. The name “think” clashes with “<a href="https://www.anthropic.com/news/visible-extended-thinking" target="_blank" rel="noreferrer noopener">extended thinking</a>” and needlessly anthropomorphizes the model… but I digress.
Having a space to log notes and progress works. Anthropic shows pairing the “think” tool with a domain-specific prompt (which you’d do anyway in an agent) yields significant gains: up to a 54% improvement against a benchmark for specialized agents.
Anthropic identified three scenarios where the context offloading pattern is useful:
<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<ol class="wp-block-list">
<li>Tool output analysis. When Claude needs to carefully process the output of previous tool calls before acting and might need to backtrack in its approach;</li>
<li>Policy-heavy environments. When Claude needs to follow detailed guidelines and verify compliance; and</li>
<li>Sequential decision making. When each action builds on previous ones and mistakes are costly (often found in multi-step domains).</li>
</ol>
</blockquote>
<h2 class="wp-block-heading">Takeaways</h2>
Context management is usually the hardest part of building an agent. Programming the LLM to, as Karpathy says, “<a href="https://x.com/karpathy/status/1937902205765607626" target="_blank" rel="noreferrer noopener">pack the context windows just right</a>,” smartly deploying tools, information, and regular context maintenance, is the job of the agent designer.
The key insight across all the above tactics is that context is not free. Every token in the context influences the model’s behavior, for better or worse. The massive context windows of modern LLMs are a powerful capability, but they’re not an excuse to be sloppy with information management.
As you build your next agent or optimize an existing one, ask yourself: Is everything in this context earning its keep? If not, you now have six ways to fix it.
<hr class="wp-block-separator has-alpha-channel-opacity is-style-wide"/>
<h2 class="wp-block-heading">Footnotes</h2>
<ol class="wp-block-list">
<li>Gemini 2.5 and GPT-4.1 have 1 million token context windows, large enough to throw <a href="https://en.wikipedia.org/wiki/Infinite_Jest" target="_blank" rel="noreferrer noopener">Infinite Jest</a> in there with plenty of room to spare.</li>
<li>The “<a href="https://ai.google.dev/gemini-api/docs/long-context#long-form-text" target="_blank" rel="noreferrer noopener">Long form text</a>” section in the Gemini docs sum up this optmism nicely.</li>
<li>In fact, in the Databricks study cited above, a frequent way models would fail when given long contexts is they’d return summarizations of the provided context while ignoring any instructions contained within the prompt.</li>
<li>If you’re on the leaderboard, pay attention to the “Live (AST)” columns. <a href="https://gorilla.cs.berkeley.edu/blogs/12_bfcl_v2_live.html" target="_blank" rel="noreferrer noopener">These metrics use real-world tool definitions contributed to the product by enterprise</a>, “avoiding the drawbacks of dataset contamination and biased benchmarks.”</li>
<li>Hell, this entire list of tactics is a strong argument for why <a href="https://www.dbreunig.com/2025/06/10/let-the-model-write-the-prompt.html" target="_blank" rel="noreferrer noopener">you should program your contexts</a>.</li>
</ol>
]]></content:encoded>
</item>
<item>
<title>MCP Introduces Deep Integration—and Serious Security Concerns</title>
<link>https://www.oreilly.com/radar/mcp-introduces-deep-integration-and-serious-security-concerns/</link>
<pubDate>Wed, 27 Aug 2025 09:52:30 +0000</pubDate>
<dc:creator><![CDATA[Andrew Stellman]]></dc:creator>
<category><![CDATA[AI & ML]]></category>
<category><![CDATA[Commentary]]></category>
<guid isPermaLink="false">https://www.oreilly.com/radar/?p=17350</guid>
<media:content
url="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/Abstract-color-four.jpg"
medium="image"
type="image/jpeg"
/>
<description><![CDATA[MCP—the Model Context Protocol introduced by Anthropic in November 2024—is an open standard for connecting AI assistants to data sources and development environments. It’s built for a future where every AI assistant is wired directly into your environment, where the model knows what files you have open, what text is selected, what you just typed, […]]]></description>
<content:encoded><![CDATA[
MCP—the Model Context Protocol introduced by Anthropic in November 2024—is an open standard for connecting AI assistants to data sources and development environments. It’s built for a future where every AI assistant is wired directly into your environment, where the model knows what files you have open, what text is selected, what you just typed, and what you’ve been working on.
And that’s where the security risks begin.
AI is driven by context, and that’s exactly what MCP provides. It gives AI assistants like GitHub Copilot everything they might need to help you: open files, code snippets, even what’s selected in the editor. When you use MCP-enabled tools that transmit data to remote servers, all of it gets sent over the wire. That might be fine for most developers. But if you work at a financial firm, hospital, or any organization with regulatory constraints where you need to be extremely careful about what leaves your network, MCP makes it really easy to lose control of a lot of things.
Let’s say you’re working in Visual Studio Code on a healthcare app, and you select a few lines of code to debug a query—a routine moment in your day. That snippet might include connection strings, test data with real patient info, and part of your schema. You ask Copilot to help and approve an MCP tool that connects to a remote server—and all of it gets sent to external servers. That’s not just risky. It could be a compliance violation under HIPAA, SOX, or PCI-DSS, depending on what gets transmitted.
These are the kinds of things developers accidentally send every day without realizing it:
<ul class="wp-block-list">
<li>Internal URLs and system identifiers</li>
<li>Passwords or tokens in local config files</li>
<li>Network details or VPN information</li>
<li>Local test data that includes real user info, SSNs, or other sensitive values</li>
</ul>
With MCP, devs on your team could be approving tools that send all of those things to servers outside of your network without realizing it, and there’s often no easy way to know what’s been sent.
But this isn’t just an MCP problem; it’s part of a larger shift where AI tools are becoming more context-aware across the board. Browser extensions that read your tabs, AI coding assistants that scan your entire codebase, productivity tools that analyze your documents—they’re all collecting more information to provide better assistance. With MCP, the stakes are just more visible because the data pipeline is formalized.
Many enterprises are now facing a choice between AI productivity gains and regulatory compliance. Some orgs are building air-gapped development environments for sensitive projects, though achieving true isolation with AI tools can be complex since many still require external connectivity. Others lean on network-level monitoring and data loss prevention solutions that can detect when code or configuration files are being transmitted externally. And a few are going deeper and building custom MCP implementations that sanitize data before transmission, stripping out anything that looks like credentials or sensitive identifiers.
One thing that can help is organizational controls in development tools like VS Code. Most security-conscious organizations can centrally disable MCP support or control which servers are available through group policies and GitHub Copilot enterprise settings. But that’s where it gets tricky, because MCP doesn’t just receive responses. It sends data upstream, potentially to a server outside of your organization, which means every request carries risk.
Security vendors are starting to catch up. Some are building MCP-aware monitoring tools that can flag potentially sensitive data before it leaves the network. Others are developing hybrid deployment models where the AI reasoning happens on-premises but can still access external knowledge when needed.
Our industry is going to have to come up with better enterprise solutions for securing MCP if we want to meet the needs of all organizations. The tension between AI capability and data security will likely drive innovation in privacy-preserving AI techniques, federated learning approaches, and hybrid deployment models that keep sensitive context local while still providing intelligent assistance.
Until then, deeply integrated AI assistants come with a cost: Sensitive context can slip through—and there’s no easy way to know it has happened.
]]></content:encoded>
</item>
</channel>
</rss>
<!--
Performance optimized by W3 Total Cache. Learn more: https://www.boldgrid.com/w3-total-cache/
Object Caching 247/249 objects using Memcached
Page Caching using Disk: Enhanced (Page is feed)
Minified using Memcached
Served from: www.oreilly.com @ 2025-09-18 18:11:38 by W3 Total Cache
-->

If you would like to create a banner that links to this page (i.e. this validation result), do the following:

Download the "valid RSS" banner.
Upload the image to your own server. (This step is important. Please do not link directly to the image on this server.)
Add this HTML to your page (change the image src attribute if necessary):

<a href="http://www.feedvalidator.org/check.cgi?url=https%3A//www.oreilly.com/radar/feed/index.xml"><img src="valid-rss-rogers.png" alt="[Valid RSS]" title="Validate my RSS feed" /></a>

If you would like to create a text link instead, here is the URL you can use:

http://www.feedvalidator.org/check.cgi?url=https%3A//www.oreilly.com/radar/feed/index.xml

Home · About · News · Docs · Terms