Source URL: https://cloud.google.com/blog/products/data-analytics/announcing-intelligent-unified-governance-in-bigquery/
Source: Cloud Blog
Title: Introducing BigQuery unified governance: universal, intelligent, and open
Feedly Summary: Data is the critical foundation for AI, yet a vast amount of data’s potential remains untapped. Why? Data quality remains a top barrier. To use enterprise data to drive analytics-driven decisions and build differentiated AI, businesses need to be able to find, understand, and trust their data assets. This requires effective data governance encompassing discovery, cataloging, metadata management, quality assurance, sharing, and access control.
The stakes are high. According to Gartner, “through 2026, those organizations that don’t enable and support their AI use cases through an AI-ready data practice will see over 60% of AI projects fail to deliver on business SLAs and be abandoned.”
At Google Cloud Next 25, we’re announcing BigQuery unified governance, powerful data governance capabilities that help enterprises keep pace with governance complexities. Data silos, fragmented metadata, and ambiguous ownership create significant risks and impede innovation. BigQuery unified governance provides services and tools organizations need to simplify data management and unlock actionable insights.
BigQuery’s built-in, intelligent governance simplifies data and AI management, helping organizations discover, understand, and leverage their assets, transforming governance from a burden into a powerful tool for data activation. Central to BigQuery governance is BigQuery universal catalog, a unified, AI-powered data catalog that natively integrates Dataplex, BigQuery sharing, security and metastore capabilities, bringing together business, technical, and runtime metadata.
BigQuery’s unified governance capabilities are:
1. Unified: BigQuery brings governance directly into the heart of your data-to-AI lifecycle, enabling discovery, understanding, governance, and utilization of your data assets and AI models. This gives data administrators, stewards, and custodians robust tools for metadata management and policy enforcement, providing end-to-end data-to-AI lineage, data profiling, insights, and secure sharing. And with the new universal semantic search, finding the right data is as simple as asking a question in natural language.
2. Intelligent: New governance capabilities powered by gen AI stand to revolutionize data management. By harnessing the power of large language models (LLMs), BigQuery universal catalog can help you uncover hidden relationships between BigQuery data assets, enable automated metadata curation and intelligent query recommendations at scale, automate governance, and democratize data-driven insights across the organization.
3. Open: BigQuery universal catalog insulates you from change with support for open storage standards such as Apache Iceberg, and a unified runtime metastore across SQL, open-source engines, and AI/ML. The BigQuery metastore, which is included in the BigQuery universal catalog, is Iceberg-compliant, enabling a multi-engine, multi-vendor architecture for governance and use of fully managed Iceberg data.
ANZ Bank, a multinational banking and financial services provider, uses the BigQuery universal catalog for comprehensive data governance, discovery, and observability.
“With BigQuery universal catalog, ANZ has significantly improved the reliability and trustworthiness of our data. The centralized data quality monitoring and automated validation features are increasing confidence and efficiency in critical business outputs and decisions based on accurate and consistent information. BigQuery governance has become a cornerstone of our data governance strategy, ensuring our data is not just available, but dependable." Artur Kaluza, Head of Data Strategy and Transformation, Risk, ANZ
aside_block
Noteworthy features
The new unified governance experience in BigQuery provides a centralized interface within the BigQuery UI for managing, securing, and sharing data and AI assets. In addition, we are introducing a wide range of key new features and capabilities across governance, sharing, and security.
Governance
1. Full-catalog search with semantic understanding (preview): Users can now discover data and AI resources across projects and data silos within BigQuery using full-catalog semantic search. This feature introduces natural-language search capabilities, making it easier for both technical and non-technical users to search the catalog.
2. Automated metadata curation (preview): BigQuery universal catalog can now automatically generate metadata for BigQuery tables, including table and column descriptions, improving data discovery and support gen AI applications.
3. AI-powered knowledge engine (preview): Users can efficiently discover hidden relationships within a dataset with automated entity-relationship visualization. By leveraging inferred relationships, BigQuery universal catalog generates suggestions for cross-table queries and natural language questions, getting new data teams up to speed fast on unfamiliar data assets.
4. Data products (preview): BigQuery data products allow data owners to create, share, and govern collections of data assets by use case, packaging and sharing them within and across organizations in a way that’s consistent, governed, and that follows security best practices.
5. Business glossary (GA): The BigQuery business glossary provides organizations with a shared understanding of their data. Customers can define and administer company terms, identify data stewards for these terms, and attach them to data asset fields, improving context, collaboration, and search.
6. Automatic at-scale cataloging of BigLake and object tables (GA): BigQuery universal catalog harvests up-to-date metadata for structured and unstructured data from Cloud Storage, and uses it to automatically create query-ready BigLake tables at scale.
7. Automated anomaly detection (preview): BigQuery universal catalog automates data anomaly detection to help you identify data errors, inconsistencies, and outliers in your data, reducing the time you spend identifying and resolving data issues.
Full catalog search with semantic understanding
Automated metadata curation
Sharing
8. BigQuery sharing integration with Google Cloud Marketplace (preview): Data owners can monetize datasets in BigQuery sharing (formerly Analytics Hub) through Google Cloud Marketplace.
9. Stream sharing in BigQuery (GA): Curate and share valuable real-time streams with Pub/Sub topics in BigQuery sharing.
10. Stored procedure sharing in BigQuery (preview): Share SQL stored procedures and enable execution in the subscriber’s project without revealing the actual code.
11. Query template sharing in BigQuery (preview): Customize, reuse, and restrict SQL queries in a clean room through publisher-defined query templates.
Security
12. Data policies on columns (preview): Create raw access and data-masking policies associated directly to a column and that can be reused across columns and tables.
13. Subquery support with row-level security (GA): BigQuery universal catalog now supports SQL subqueries in security access policy definitions, enabling row filtering without changing existing data models.
These built-in governance advancements within the BigQuery platform help organizations unlock the full potential of their data and AI initiatives.
In addition to the innovation in BigQuery, we continue to partner with third-party catalog providers to complement their governance capabilities. For example, Collibra’s enterprise-wide governance for data and AI extends BigQuery universal catalog capabilities to provide end-to-end visibility, quality and stewardship across hybrid and multicloud environments. This partnership helps ensure more teams can discover and trust the data they need to do AI, no matter where it lives, accelerating and strengthening every use case.
By embedding governance into BigQuery and automating metadata management, BigQuery universal catalog is helping businesses move beyond the challenges of data silos and operational inefficiency, ultimately driving innovation and accelerating business impact. Ready to learn more? You can join several sessions covering the latest in BigQuery governance, sharing, and security featuring customer speakers:
1. What’s new in data and AI governance with Levi’s and Verizon | Apr 9, 4pm
2. Redefine data and AI governance in the unified BigQuery platform with Walmart and Box | Apr 11, 12:30pm
3. From chaos to confidence: Master governance in the age of AI with EDMC, ANZ Bank, Intesa Sanpaolo Group, and Google | Apr 9, 11am
4. Unlock the power of secure data sharing with BigQuery with Liveramp and Levi’s | Apr 9, 4pm
5. Data fabric on BigQuery: an architect’s perspective of why and how to do it with Virgin Media O2 | Apr 11, 8:30am
AI Summary and Description: Yes
Summary: The text discusses significant advancements in data governance capabilities embodied in Google Cloud’s BigQuery platform, aiming to enable enterprises to effectively manage their data assets and leverage AI. The innovations focus on comprehensive data governance, automated metadata management, and enhanced security mechanisms, emphasizing the importance of robust data practices to prevent AI project failures.
Detailed Description:
The provided text elaborates on the critical role of data quality and governance in successfully implementing AI initiatives within organizations. Notable points mentioned include:
– **Data Quality as a Barrier**: The text highlights that poor data quality is a significant hindrance for enterprises aiming to utilize their data effectively, leading to the potential failure of AI projects.
– **Gartner’s Insight**: A prediction from Gartner underscores the necessity for AI-enabled organizations to adopt effective data practices to avoid high failure rates in AI initiatives.
– **Introduction of BigQuery Unified Governance**:
– This platform offers a set of robust governance tools designed to simplify data management and enhance the discoverability of data assets.
– Addressing issues like data silos, fragmented metadata, and ownership ambiguities that can stifle innovation.
– **Key Features of BigQuery Unified Governance**:
1. **Unified Data Governance**: Integrates data management directly into the AI lifecycle, providing robust tools for metadata management, policy enforcement, and insights generation.
2. **AI-powered Insights**: Leverages generative AI to improve metadata curation and automate governance, thereby enabling organizations to democratize access to data-driven insights.
3. **Open Standards Support**: Incorporates open storage standards, allowing for a more adaptable governance architecture that can accommodate varied data sources and engines.
– **Practical Use Case**: The text cites ANZ Bank’s successful implementation of the BigQuery universal catalog to enhance data trustworthiness and decision-making processes, illustrating the practical benefits of robust data governance.
– **Innovative Governance Experience**: The new governance capabilities provide:
– Semantic search capabilities, enabling ease of data discovery through natural language queries.
– Automated metadata curation, bolstering support for generative AI applications.
– A business glossary for improved collaboration and context.
– **Security Enhancements**:
– Introduction of column-level data policies and subquery support for improved data access control.
– Overall emphasis on security best practices throughout the data lifecycle.
By innovating within the governance sector of BigQuery, Google Cloud aims to alleviate common pain points that organizations face in data management and compliance, facilitating more effective data utilization for AI projects. The text indicates a concerted effort to ensure that organizations can navigate the complexities of data governance, thus fostering innovation and minimizing operational inefficiencies.
In conclusion, the advancements introduced through BigQuery unified governance provide both a strategic framework and practical tools to address the multifaceted challenges related to data management, AI readiness, and compliance, ultimately supporting business growth and resilience.