prefix caching – Experimental News Clipping Site

Cloud Blog: vLLM Performance Tuning: The Ultimate Guide to xPU Inference Configuration

Aug 25, 2025

—

by

Source URL: https://cloud.google.com/blog/topics/developers-practitioners/vllm-performance-tuning-the-ultimate-guide-to-xpu-inference-configuration/ Source: Cloud Blog Title: vLLM Performance Tuning: The Ultimate Guide to xPU Inference Configuration Feedly Summary: Additional contributors include Hossein Sarshar, Ashish Narasimham, and Chenyang Li. Large Language Models (LLMs) are revolutionizing how we interact with technology, but serving these powerful models efficiently can be a challenge. vLLM has rapidly become…

Cloud Blog: Rightsizing LLM Serving on vLLM for GPUs and TPUs

Aug 19, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://cloud.google.com/blog/topics/developers-practitioners/rightsizing-llm-serving-on-vllm-for-gpus-and-tpus/ Source: Cloud Blog Title: Rightsizing LLM Serving on vLLM for GPUs and TPUs Feedly Summary: Additional contributors include Hossein Sarshar and Ashish Narasimham. Large Language Models (LLMs) are revolutionizing how we interact with technology, but serving these powerful models efficiently can be a challenge. vLLM has rapidly become the primary choice for…

Cloud Blog: Scalable AI starts with storage: Guide to model artifact strategies

Aug 14, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://cloud.google.com/blog/topics/developers-practitioners/scalable-ai-starts-with-storage-guide-to-model-artifact-strategies/ Source: Cloud Blog Title: Scalable AI starts with storage: Guide to model artifact strategies Feedly Summary: Managing large model artifacts is a common bottleneck in MLOps. Baking models into container images leads to slow, monolithic deployments, and downloading them at startup introduces significant delays. This guide explores a better way: decoupling your…

Cloud Blog: Your guide to taking an open model from discovery to a production-ready endpoint on Vertex AI

Jul 25, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://cloud.google.com/blog/products/ai-machine-learning/take-an-open-model-from-discovery-to-endpoint-on-vertex-ai/ Source: Cloud Blog Title: Your guide to taking an open model from discovery to a production-ready endpoint on Vertex AI Feedly Summary: Developers building with gen AI are increasingly drawn to open models for their power and flexibility. But customizing and deploying them can be a huge challenge. You’re often left wrestling…

The Cloudflare Blog: Workers AI gets a speed boost, batch workload support, more LoRAs, new models, and a refreshed dashboard

Apr 11, 2025

—

by

Kurt Seifried

in Uncategorized

Source URL: https://blog.cloudflare.com/workers-ai-improvements/ Source: The Cloudflare Blog Title: Workers AI gets a speed boost, batch workload support, more LoRAs, new models, and a refreshed dashboard Feedly Summary: We just made Workers AI inference faster with speculative decoding & prefix caching. Use our new batch inference for handling large request volumes seamlessly. AI Summary and Description:…

Tag: prefix caching

Cloud Blog: vLLM Performance Tuning: The Ultimate Guide to xPU Inference Configuration

Cloud Blog: Rightsizing LLM Serving on vLLM for GPUs and TPUs

Cloud Blog: Scalable AI starts with storage: Guide to model artifact strategies

Cloud Blog: Your guide to taking an open model from discovery to a production-ready endpoint on Vertex AI

The Cloudflare Blog: Workers AI gets a speed boost, batch workload support, more LoRAs, new models, and a refreshed dashboard