Experimental News Clipping Site

Tag: llama.cpp

  • Simon Willison’s Weblog: llm-llama-server 0.2

    —

    by

    Kurt Seifried
    in

    Source URL: https://simonwillison.net/2025/May/28/llama-server-tools/ Source: Simon Willison’s Weblog Title: llm-llama-server 0.2 Feedly Summary: llm-llama-server 0.2 Here’s a second option for using LLM’s new tool support against local models (the first was via llm-ollama). It turns out the llama.cpp ecosystem has pretty robust OpenAI-compatible tool support already, so my llm-llama-server plugin only needed a quick upgrade to…

  • Simon Willison’s Weblog: Large Language Models can run tools in your terminal with LLM 0.26

    —

    by

    Kurt Seifried
    in

    Source URL: https://simonwillison.net/2025/May/27/llm-tools/ Source: Simon Willison’s Weblog Title: Large Language Models can run tools in your terminal with LLM 0.26 Feedly Summary: LLM 0.26 is out with the biggest new feature since I started the project: support for tools. You can now use the LLM CLI tool – and Python library – to grant LLMs…

  • Simon Willison’s Weblog: Trying out llama.cpp’s new vision support

    —

    by

    Kurt Seifried
    in

    Source URL: https://simonwillison.net/2025/May/10/llama-cpp-vision/#atom-everything Source: Simon Willison’s Weblog Title: Trying out llama.cpp’s new vision support Feedly Summary: This llama.cpp server vision support via libmtmd pull request – via Hacker News – was merged earlier today. The PR finally adds full support for vision models to the excellent llama.cpp project. It’s documented on this page, but the…

  • The Register: <em>El Reg’s</em> essential guide to deploying LLMs in production

    —

    by

    Kurt Seifried
    in

    Source URL: https://www.theregister.com/2025/04/22/llm_production_guide/ Source: The Register Title: <em>El Reg’s</em> essential guide to deploying LLMs in production Feedly Summary: Running GenAI models is easy. Scaling them to thousands of users, not so much Hands On You can spin up a chatbot with Llama.cpp or Ollama in minutes, but scaling large language models to handle real workloads…

  • Simon Willison’s Weblog: Gemma 3 QAT Models

    —

    by

    Kurt Seifried
    in

    Source URL: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Source: Simon Willison’s Weblog Title: Gemma 3 QAT Models Feedly Summary: Gemma 3 QAT Models Interesting release from Google, as a follow-up to Gemma 3 from last month: To make Gemma 3 even more accessible, we are announcing new versions optimized with Quantization-Aware Training (QAT) that dramatically reduces memory requirements while maintaining…

  • Docker: Run LLMs Locally with Docker: A Quickstart Guide to Model Runner

    —

    by

    Kurt Seifried
    in

    Source URL: https://www.docker.com/blog/run-llms-locally/ Source: Docker Title: Run LLMs Locally with Docker: A Quickstart Guide to Model Runner Feedly Summary: AI is quickly becoming a core part of modern applications, but running large language models (LLMs) locally can still be a pain. Between picking the right model, navigating hardware quirks, and optimizing for performance, it’s easy…

  • Hacker News: Heap-overflowing Llama.cpp to RCE

    —

    by

    Kurt Seifried
    in

    Source URL: https://retr0.blog/blog/llama-rpc-rce Source: Hacker News Title: Heap-overflowing Llama.cpp to RCE Feedly Summary: Comments AI Summary and Description: Yes Summary: The text provides a detailed, technical exploration of exploiting a remote code execution vulnerability within the Llama.cpp framework, specifically focusing on a heap-overflow issue and its associated mitigations. It offers insights into the unique memory…

  • Hacker News: A Practical Guide to Running Local LLMs

    —

    by

    Kurt Seifried
    in

    Source URL: https://spin.atomicobject.com/running-local-llms/ Source: Hacker News Title: A Practical Guide to Running Local LLMs Feedly Summary: Comments AI Summary and Description: Yes Summary: The text discusses the intricacies of running local large language models (LLMs), emphasizing their applications in privacy-critical situations and the potential benefits of various tools like Ollama and Llama.cpp. It provides insights…

  • Hacker News: Llama.cpp AI Performance with the GeForce RTX 5090 Review

    —

    by

    Kurt Seifried
    in

    Source URL: https://www.phoronix.com/review/nvidia-rtx5090-llama-cpp Source: Hacker News Title: Llama.cpp AI Performance with the GeForce RTX 5090 Review Feedly Summary: Comments AI Summary and Description: Yes Summary: The text discusses initial performance benchmarks of NVIDIA’s GeForce RTX 5090 graphics card specifically in relation to AI performance using the Llama.cpp framework. This relevance to AI performance makes it…