Tag: evals
-
Simon Willison’s Weblog: Anthropic: How we built our multi-agent research system
Source URL: https://simonwillison.net/2025/Jun/14/multi-agent-research-system/#atom-everything Source: Simon Willison’s Weblog Title: Anthropic: How we built our multi-agent research system Feedly Summary: Anthropic: How we built our multi-agent research system OK, I’m sold on multi-agent LLM systems now. I’ve been pretty skeptical of these until recently: why make your life more complicated by running multiple different prompts in parallel…
-
Enterprise AI Trends: Evals Startups Want Enterprise Money for Table-Stakes Features
Source URL: https://nextword.substack.com/p/evals-startups-want-enterprise-money Source: Enterprise AI Trends Title: Evals Startups Want Enterprise Money for Table-Stakes Features Feedly Summary: They want to be the next “Datadog" or "Snowflake", but can they fool everyone at the same time? AI Summary and Description: Yes **Summary:** The text provides a critical analysis of the emerging market for “evals” platforms…
-
Simon Willison’s Weblog: Shisa V2 405B: Japan’s Highest Performing LLM
Source URL: https://simonwillison.net/2025/Jun/3/shisa-v2/ Source: Simon Willison’s Weblog Title: Shisa V2 405B: Japan’s Highest Performing LLM Feedly Summary: Shisa V2 405B: Japan’s Highest Performing LLM Leonard Lin and Adam Lensenmayer have been working on Shisa for a while. They describe their latest release as “Japan’s Highest Performing LLM". Shisa V2 405B is the highest-performing LLM ever…
-
Simon Willison’s Weblog: How often do LLMs snitch? Recreating Theo’s SnitchBench with LLM
Source URL: https://simonwillison.net/2025/May/31/snitchbench-with-llm/#atom-everything Source: Simon Willison’s Weblog Title: How often do LLMs snitch? Recreating Theo’s SnitchBench with LLM Feedly Summary: A fun new benchmark just dropped! Inspired by the Claude 4 system card – which showed that Claude 4 might just rat you out to the authorities if you told it to “take initiative" in…
-
Hamel’s Blog: LLM Eval FAQ
Source URL: https://hamel.dev/blog/posts/evals-faq/ Source: Hamel’s Blog Title: LLM Eval FAQ Feedly Summary: Our Course On AI Evals I’m teaching a course on AI Evals with Shreya Shankar. Here are some of the most common questions we’ve been asked. We’ll be updating this list frequently. Q: Is RAG dead? Question: Should I avoid using RAG for…
-
Cloud Blog: Getting AI to write good SQL: Text-to-SQL techniques explained
Source URL: https://cloud.google.com/blog/products/databases/techniques-for-improving-text-to-sql/ Source: Cloud Blog Title: Getting AI to write good SQL: Text-to-SQL techniques explained Feedly Summary: Organizations depend on fast and accurate data-driven insights to make decisions, and SQL is at the core of how they access that data. With Gemini, Google can generate SQL directly from natural language — a.k.a. text-to-SQL. This…
-
Simon Willison’s Weblog: Building software on top of Large Language Models
Source URL: https://simonwillison.net/2025/May/15/building-on-llms/#atom-everything Source: Simon Willison’s Weblog Title: Building software on top of Large Language Models Feedly Summary: I presented a three hour workshop at PyCon US yesterday titled Building software on top of Large Language Models. The goal of the workshop was to give participants everything they needed to get started writing code that…
-
Simon Willison’s Weblog: Expanding on what we missed with sycophancy
Source URL: https://simonwillison.net/2025/May/2/what-we-missed-with-sycophancy/ Source: Simon Willison’s Weblog Title: Expanding on what we missed with sycophancy Feedly Summary: Expanding on what we missed with sycophancy I criticized OpenAI’s initial post about their recent ChatGPT sycophancy rollback as being “relatively thin" so I’m delighted that they have followed it with a much more in-depth explanation of what…
-
Simon Willison’s Weblog: Understanding the recent criticism of the Chatbot Arena
Source URL: https://simonwillison.net/2025/Apr/30/criticism-of-the-chatbot-arena/#atom-everything Source: Simon Willison’s Weblog Title: Understanding the recent criticism of the Chatbot Arena Feedly Summary: The Chatbot Arena has become the go-to place for vibes-based evaluation of LLMs over the past two years. The project, originating at UC Berkeley, is home to a large community of model enthusiasts who submit prompts to…
-
Simon Willison’s Weblog: Exploring Promptfoo via Dave Guarino’s SNAP evals
Source URL: https://simonwillison.net/2025/Apr/24/exploring-promptfoo/#atom-everything Source: Simon Willison’s Weblog Title: Exploring Promptfoo via Dave Guarino’s SNAP evals Feedly Summary: I used part three (here’s parts one and two) of Dave Guarino’s series on evaluating how well LLMs can answer questions about SNAP (aka food stamps) as an excuse to explore Promptfoo, an LLM eval tool. SNAP (Supplemental…