evals – Page 2 – Experimental News Clipping Site

Simon Willison’s Weblog: How often do LLMs snitch? Recreating Theo’s SnitchBench with LLM

May 31, 2025

—

by

Source URL: https://simonwillison.net/2025/May/31/snitchbench-with-llm/#atom-everything Source: Simon Willison’s Weblog Title: How often do LLMs snitch? Recreating Theo’s SnitchBench with LLM Feedly Summary: A fun new benchmark just dropped! Inspired by the Claude 4 system card – which showed that Claude 4 might just rat you out to the authorities if you told it to “take initiative" in…

Hamel’s Blog: LLM Eval FAQ

May 29, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://hamel.dev/blog/posts/evals-faq/ Source: Hamel’s Blog Title: LLM Eval FAQ Feedly Summary: Our Course On AI Evals I’m teaching a course on AI Evals with Shreya Shankar. Here are some of the most common questions we’ve been asked. We’ll be updating this list frequently. Q: Is RAG dead? Question: Should I avoid using RAG for…

Cloud Blog: Getting AI to write good SQL: Text-to-SQL techniques explained

May 16, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://cloud.google.com/blog/products/databases/techniques-for-improving-text-to-sql/ Source: Cloud Blog Title: Getting AI to write good SQL: Text-to-SQL techniques explained Feedly Summary: Organizations depend on fast and accurate data-driven insights to make decisions, and SQL is at the core of how they access that data. With Gemini, Google can generate SQL directly from natural language — a.k.a. text-to-SQL. This…

Simon Willison’s Weblog: Building software on top of Large Language Models

May 15, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://simonwillison.net/2025/May/15/building-on-llms/#atom-everything Source: Simon Willison’s Weblog Title: Building software on top of Large Language Models Feedly Summary: I presented a three hour workshop at PyCon US yesterday titled Building software on top of Large Language Models. The goal of the workshop was to give participants everything they needed to get started writing code that…

Simon Willison’s Weblog: Expanding on what we missed with sycophancy

May 2, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://simonwillison.net/2025/May/2/what-we-missed-with-sycophancy/ Source: Simon Willison’s Weblog Title: Expanding on what we missed with sycophancy Feedly Summary: Expanding on what we missed with sycophancy I criticized OpenAI’s initial post about their recent ChatGPT sycophancy rollback as being “relatively thin" so I’m delighted that they have followed it with a much more in-depth explanation of what…

Simon Willison’s Weblog: Understanding the recent criticism of the Chatbot Arena

Apr 30, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://simonwillison.net/2025/Apr/30/criticism-of-the-chatbot-arena/#atom-everything Source: Simon Willison’s Weblog Title: Understanding the recent criticism of the Chatbot Arena Feedly Summary: The Chatbot Arena has become the go-to place for vibes-based evaluation of LLMs over the past two years. The project, originating at UC Berkeley, is home to a large community of model enthusiasts who submit prompts to…

Simon Willison’s Weblog: Exploring Promptfoo via Dave Guarino’s SNAP evals

Apr 24, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://simonwillison.net/2025/Apr/24/exploring-promptfoo/#atom-everything Source: Simon Willison’s Weblog Title: Exploring Promptfoo via Dave Guarino’s SNAP evals Feedly Summary: I used part three (here’s parts one and two) of Dave Guarino’s series on evaluating how well LLMs can answer questions about SNAP (aka food stamps) as an excuse to explore Promptfoo, an LLM eval tool. SNAP (Supplemental…

Simon Willison’s Weblog: Quoting Andrew Ng

Apr 18, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://simonwillison.net/2025/Apr/18/andrew-ng/ Source: Simon Willison’s Weblog Title: Quoting Andrew Ng Feedly Summary: To me, a successful eval meets the following criteria. Say, we currently have system A, and we might tweak it to get a system B: If A works significantly better than B according to a skilled human judge, the eval should give…

Simon Willison’s Weblog: Quoting Ted Sanders, OpenAI

Apr 17, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://simonwillison.net/2025/Apr/17/ted-sanders/ Source: Simon Willison’s Weblog Title: Quoting Ted Sanders, OpenAI Feedly Summary: Our hypothesis is that o4-mini is a much better model, but we’ll wait to hear feedback from developers. Evals only tell part of the story, and we wouldn’t want to prematurely deprecate a model that developers continue to find value in.…

AWS News Blog: AWS Weekly Review: Amazon S3 Express One Zone price cuts, Pixtral Large on Amazon Bedrock, Amazon Nova Sonic, and more (April 14, 2025)

Apr 14, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://aws.amazon.com/blogs/aws/aws-weekly-review-amazon-s3-express-one-zone-price-cuts-pixtral-large-on-amazon-bedrock-amazon-nova-sonic-and-more-april-14-2025/ Source: AWS News Blog Title: AWS Weekly Review: Amazon S3 Express One Zone price cuts, Pixtral Large on Amazon Bedrock, Amazon Nova Sonic, and more (April 14, 2025) Feedly Summary: The Amazon Web Services (AWS) Summit 2025 season launched this week, starting with the Paris Summit. These free events bring together the…

Tag: evals