Experimental News Clipping Site

Tag: Apache Airflow

  • Cloud Blog: What’s new with Google Data Cloud

    —

    by

    Kurt Seifried
    in

    Source URL: https://cloud.google.com/blog/products/data-analytics/whats-new-with-google-data-cloud/ Source: Cloud Blog Title: What’s new with Google Data Cloud Feedly Summary: June 9 – June 13 Introducing Pub/Sub Single Message Transforms (SMTs), to make it easy to perform simple data transformations such as validate, filter, enrich, and alter individual messages as they move in real time right within Pub/Sub. The first…

  • Hacker News: LLM Workflows then Agents: Getting Started with Apache Airflow

    —

    by

    Kurt Seifried
    in

    Source URL: https://github.com/astronomer/airflow-ai-sdk Source: Hacker News Title: LLM Workflows then Agents: Getting Started with Apache Airflow Feedly Summary: Comments AI Summary and Description: Yes **Summary:** The text presents an SDK for integrating large language models (LLMs) into Apache Airflow workflows. This novel approach enhances AI orchestration by providing refined task decorators that streamline calling LLMs,…

  • Cloud Blog: Cloud Composer 3: The next generation of data pipeline orchestration

    —

    by

    Kurt Seifried
    in

    Source URL: https://cloud.google.com/blog/products/data-analytics/cloud-composer-3-for-apache-airflow/ Source: Cloud Blog Title: Cloud Composer 3: The next generation of data pipeline orchestration Feedly Summary: As any data team can tell you, managing Apache Airflow is often a complex undertaking, with many hours spent wrestling with reliability, security, and efficient resource scaling. This valuable time could be better spent deriving insights…

  • Hacker News: Apache Airflow: Key Use Cases, Architectural Insights, and Pro Tips

    —

    by

    Kurt Seifried
    in

    Source URL: https://codingcops.com/apache-airflow/ Source: Hacker News Title: Apache Airflow: Key Use Cases, Architectural Insights, and Pro Tips Feedly Summary: Comments AI Summary and Description: Yes **Summary:** The text discusses Apache Airflow, an open-source tool designed for managing complex workflows and big data pipelines. It highlights Airflow’s capabilities in orchestrating ETL processes, automating machine learning workflows,…

  • The Cloudflare Blog: Training a million models per day to save customers of all sizes from DDoS attacks

    —

    by

    system automation
    in

    Source URL: https://blog.cloudflare.com/training-a-million-models-per-day-to-save-customers-of-all-sizes-from-ddos Source: The Cloudflare Blog Title: Training a million models per day to save customers of all sizes from DDoS attacks Feedly Summary: In this post we will describe how we use anomaly detection to watch for novel DDoS attacks. We’ll provide an overview of how we build models which flag unusual traffic…

  • Cloud Blog: Apache Airflow ETL in Google Cloud

    —

    by

    system automation
    in

    Source URL: https://cloud.google.com/blog/products/data-analytics/different-ways-to-run-apache-airflow-on-google-cloud/ Source: Cloud Blog Title: Apache Airflow ETL in Google Cloud Feedly Summary: Are you thinking about running Apache Airflow on Google Cloud? That’s a popular choice for running a complex set of tasks, such as Extract, Transform, and Load (ETL) or data analytics pipelines. Apache Airflow uses a Directed Acyclic Graph (DAG)…