Tag: evals
-
Simon Willison’s Weblog: TIL: Running a gpt-oss eval suite against LM Studio on a Mac
Source URL: https://simonwillison.net/2025/Aug/17/gpt-oss-eval-suite/#atom-everything Source: Simon Willison’s Weblog Title: TIL: Running a gpt-oss eval suite against LM Studio on a Mac Feedly Summary: TIL: Running a gpt-oss eval suite against LM Studio on a Mac The other day I learned that OpenAI published a set of evals as part of their gpt-oss model release, described in…
-
Simon Willison’s Weblog: Claude Opus 4.1
Source URL: https://simonwillison.net/2025/Aug/5/claude-opus-41/ Source: Simon Willison’s Weblog Title: Claude Opus 4.1 Feedly Summary: Claude Opus 4.1 Surprise new model from Anthropic today – Claude Opus 4.1, which they describe as “a drop-in replacement for Opus 4". My favorite thing about this model is the version number – treating this as a .1 version increment looks…
-
Simon Willison’s Weblog: Using GitHub Spark to reverse engineer GitHub Spark
Source URL: https://simonwillison.net/2025/Jul/24/github-spark/ Source: Simon Willison’s Weblog Title: Using GitHub Spark to reverse engineer GitHub Spark Feedly Summary: GitHub Spark was released in public preview yesterday. It’s GitHub’s implementation of the prompt-to-app pattern also seen in products like Claude Artifacts, Lovable, Vercel v0, Val Town Townie and Fly.io’s Phoenix New. I wrote about Spark back…
-
Simon Willison’s Weblog: TimeScope: How Long Can Your Video Large Multimodal Model Go?
Source URL: https://simonwillison.net/2025/Jul/23/timescope/#atom-everything Source: Simon Willison’s Weblog Title: TimeScope: How Long Can Your Video Large Multimodal Model Go? Feedly Summary: TimeScope: How Long Can Your Video Large Multimodal Model Go? New open source benchmark for evaluating vision LLMs on how well they handle long videos: TimeScope probes the limits of long-video capabilities by inserting several…
-
Simon Willison’s Weblog: Frequently Asked Questions (And Answers) About AI Evals
Source URL: https://simonwillison.net/2025/Jul/3/faqs-about-ai-evals/#atom-everything Source: Simon Willison’s Weblog Title: Frequently Asked Questions (And Answers) About AI Evals Feedly Summary: Frequently Asked Questions (And Answers) About AI Evals Hamel Husain and Shreya Shankar have been running a paid, cohort-based course on AI Evals For Engineers & PMs over the past few months. Here Hamel collects answers to…
-
Simon Willison’s Weblog: microsoft/vscode-copilot-chat
Source URL: https://simonwillison.net/2025/Jun/30/vscode-copilot-chat/#atom-everything Source: Simon Willison’s Weblog Title: microsoft/vscode-copilot-chat Feedly Summary: microsoft/vscode-copilot-chat As promised at Build 2025 in May, Microsoft have released the GitHub Copilot Chat client for VS Code under an open source (MIT) license. So far this is just the extension that provides the chat component of Copilot, but the launch announcement promises…
-
Simon Willison’s Weblog: Trying out the new Gemini 2.5 model family
Source URL: https://simonwillison.net/2025/Jun/17/gemini-2-5/ Source: Simon Willison’s Weblog Title: Trying out the new Gemini 2.5 model family Feedly Summary: After many months of previews, Gemini 2.5 Pro and Flash have reached general availability with new, memorable model IDs: gemini-2.5-pro and gemini-2.5-flash. They are joined by a new preview model with an unmemorable name: gemini-2.5-flash-lite-preview-06-17 is a…
-
Simon Willison’s Weblog: Anthropic: How we built our multi-agent research system
Source URL: https://simonwillison.net/2025/Jun/14/multi-agent-research-system/#atom-everything Source: Simon Willison’s Weblog Title: Anthropic: How we built our multi-agent research system Feedly Summary: Anthropic: How we built our multi-agent research system OK, I’m sold on multi-agent LLM systems now. I’ve been pretty skeptical of these until recently: why make your life more complicated by running multiple different prompts in parallel…
-
Enterprise AI Trends: Evals Startups Want Enterprise Money for Table-Stakes Features
Source URL: https://nextword.substack.com/p/evals-startups-want-enterprise-money Source: Enterprise AI Trends Title: Evals Startups Want Enterprise Money for Table-Stakes Features Feedly Summary: They want to be the next “Datadog" or "Snowflake", but can they fool everyone at the same time? AI Summary and Description: Yes **Summary:** The text provides a critical analysis of the emerging market for “evals” platforms…
-
Simon Willison’s Weblog: Shisa V2 405B: Japan’s Highest Performing LLM
Source URL: https://simonwillison.net/2025/Jun/3/shisa-v2/ Source: Simon Willison’s Weblog Title: Shisa V2 405B: Japan’s Highest Performing LLM Feedly Summary: Shisa V2 405B: Japan’s Highest Performing LLM Leonard Lin and Adam Lensenmayer have been working on Shisa for a while. They describe their latest release as “Japan’s Highest Performing LLM". Shisa V2 405B is the highest-performing LLM ever…