Tag: optimized checkpointing
-
Cloud Blog: Save early and often with multi-tier checkpointing to optimize large AI training jobs
Source URL: https://cloud.google.com/blog/products/ai-machine-learning/using-multi-tier-checkpointing-for-large-ai-training-jobs/ Source: Cloud Blog Title: Save early and often with multi-tier checkpointing to optimize large AI training jobs Feedly Summary: As foundation model training infrastructure scales to tens of thousands of accelerators, efficient utilization of those high-value resources becomes paramount. In particular, as the cluster gets larger, hardware failures become more frequent (~…
-
Cloud Blog: Train AI for less: Improve ML Goodput with elastic training and optimized checkpointing
Source URL: https://cloud.google.com/blog/products/ai-machine-learning/elastic-training-and-optimized-checkpointing-improve-ml-goodput/ Source: Cloud Blog Title: Train AI for less: Improve ML Goodput with elastic training and optimized checkpointing Feedly Summary: Want to save some money on large AI training? For a typical PyTorch LLM training workload that spans thousands of accelerators for several weeks, a 1% improvement in ML Goodput can translate to…