Experimental News Clipping Site

Tag: evals

  • Simon Willison’s Weblog: Pydantic Evals

    —

    by

    Kurt Seifried
    in

    Source URL: https://simonwillison.net/2025/Apr/1/pydantic-evals/#atom-everything Source: Simon Willison’s Weblog Title: Pydantic Evals Feedly Summary: Pydantic Evals Brand new package from the Pydantic AI team which directly tackles what I consider to be the single hardest problem in AI engineering: building evals to determine if your LLM-based system is working correctly and getting better over time. The feature…

  • Simon Willison’s Weblog: Quoting Greg Kamradt

    —

    by

    Kurt Seifried
    in

    Source URL: https://simonwillison.net/2025/Mar/25/greg-kamradt/ Source: Simon Willison’s Weblog Title: Quoting Greg Kamradt Feedly Summary: Today we’re excited to launch ARC-AGI-2 to challenge the new frontier. ARC-AGI-2 is even harder for AI (in particular, AI reasoning systems), while maintaining the same relative ease for humans. Pure LLMs score 0% on ARC-AGI-2, and public AI reasoning systems achieve…

  • Simon Willison’s Weblog: What’s new in the world of LLMs, for NICAR 2025

    —

    by

    Kurt Seifried
    in

    Source URL: https://simonwillison.net/2025/Mar/8/nicar-llms/ Source: Simon Willison’s Weblog Title: What’s new in the world of LLMs, for NICAR 2025 Feedly Summary: I presented two sessions at the NICAR 2025 data journalism conference this year. The first was this one based on my review of LLMs in 2024, extended by several months to cover everything that’s happened…

  • Simon Willison’s Weblog: Leaked Windsurf prompt

    —

    by

    Kurt Seifried
    in

    Source URL: https://simonwillison.net/2025/Feb/25/leaked-windsurf-prompt/ Source: Simon Willison’s Weblog Title: Leaked Windsurf prompt Feedly Summary: Leaked Windsurf prompt The Windurf Editor is Codeium’s highly regarded entrant into the fork-of-VS-code AI-enhanced IDE model first pioneered by Cursor (and by VS Code itself). I heard online that it had a quirky system prompt, and was able to replicate that…

  • Simon Willison’s Weblog: Aider Polyglot leaderboard results for Claude 3.7 Sonnet

    —

    by

    Kurt Seifried
    in

    Source URL: https://simonwillison.net/2025/Feb/25/aider-polyglot-leaderboard/ Source: Simon Willison’s Weblog Title: Aider Polyglot leaderboard results for Claude 3.7 Sonnet Feedly Summary: Aider Polyglot leaderboard results for Claude 3.7 Sonnet Paul Gauthier’s Aider Polyglot benchmark is one of my favourite independent benchmarks for LLMs, partly because it focuses on code and partly because Paul is very responsive at evaluating…

  • Hacker News: Show HN: Mastra – Open-source TypeScript agent framework

    —

    by

    Kurt Seifried
    in

    Source URL: https://github.com/mastra-ai/mastra Source: Hacker News Title: Show HN: Mastra – Open-source TypeScript agent framework Feedly Summary: Comments AI Summary and Description: Yes Summary: The text introduces Mastra, a TypeScript framework designed to facilitate the rapid development of AI applications. It emphasizes key functionalities such as LLM model integration, agent systems, workflows, and retrieval-augmented generation…

  • Simon Willison’s Weblog: Building a SNAP LLM eval: part 1

    —

    by

    Kurt Seifried
    in

    Source URL: https://simonwillison.net/2025/Feb/12/building-a-snap-llm/#atom-everything Source: Simon Willison’s Weblog Title: Building a SNAP LLM eval: part 1 Feedly Summary: Building a SNAP LLM eval: part 1 Dave Guarino (previously) has been exploring using LLM-driven systems to help people apply for SNAP, the US Supplemental Nutrition Assistance Program (aka food stamps). This is a domain which existing models…

  • Simon Willison’s Weblog: OpenAI o3-mini, now available in LLM

    —

    by

    Kurt Seifried
    in

    Source URL: https://simonwillison.net/2025/Jan/31/o3-mini/#atom-everything Source: Simon Willison’s Weblog Title: OpenAI o3-mini, now available in LLM Feedly Summary: o3-mini is out today. As with other o-series models it’s a slightly difficult one to evaluate – we now need to decide if a prompt is best run using GPT-4o, o1, o3-mini or (if we have access) o1 Pro.…