Cloud Blog: How to use gen AI for better data schema handling, data quality, and data generation

Source URL: https://cloud.google.com/blog/products/data-analytics/how-gemini-in-bigquery-helps-with-data-engineering-tasks/
Source: Cloud Blog
Title: How to use gen AI for better data schema handling, data quality, and data generation

Feedly Summary: In the realm of data engineering, generative AI models are quietly revolutionizing how we handle, process, and ultimately utilize data. For example, large language models (LLMs) can help with data schema handling, data quality, and even data generation.
Building upon the recently released Gemini in BigQuery Data preparation capabilities, this blog showcases areas where gen AI models are making a significant impact in data engineering with automated solutions for schema management, data quality automation, and generation of synthetic and structured data from diverse sources, providing practical examples and code snippets.
1. Data schema handling: Integrating new datasets
Data movement and maintenance is an ongoing challenge across all data engineering teams. Whether it’s moving data between systems with different schemas or integrating new datasets into existing data products, the process can be complex and error-prone. This is often exacerbated when dealing with legacy systems; in fact, 32% of organizations cite migrating the data and the app as their biggest challenge, according to Flexera’s 2024 State of the Cloud Report.
Gen AI models offer a powerful solution by assisting in automating schema mapping and transformation on an ongoing basis. Imagine migrating customer data from a legacy CRM system to a new platform, and combining it with additional external datasets in BigQuery. The schemas likely differ significantly, requiring intricate mapping of fields and data types. Gemini, our most capable AI model family to date, can analyze both schemas and generate the necessary transformation logic, significantly reducing manual effort and potential errors.
A common approach to data schema handling that we’ve seen from data engineering teams involves creating a lightweight application that receives messages from Pub/Sub, retrieves relevant dataset information from BigQuery and Cloud Storage, and uses the Vertex AI Gemini API to map source fields to target fields and assign a confidence score. Here is example code showing a FunctionDeclaration to perform the mapping-confidence task:

code_block

1 2 2024 24 3 4 5 7 a accuracy Act ads advancement advancements AGI AI ai model AI models Alloy AlloyDB analysis analytics and Answer. anti API APIs Application applications Arch architecture Aria ARM art as Auto Automated Solutions automation awareness base64 Behavior Best best practices bias biases BigQuery branding building business business applications by C capabilities CERN challenges CI/CD CIA class CleaR client Cloud cloud architecture cloud data cloud storage code code examples Col complexity compliance concerns Configuration configurations consistency Console content Context context window control control systems core cost Costs cross Customer customer data D data data accuracy data analytics data engineering data formats data generation data migration data movement data pipeline data pipelines data preparation data processing data quality Data Scarcity data schema data schema handling data sources data types data validation data workflows data-driven DataOps dataset datasets de decision decision-making deduplication DeFi deployment depth detection driven driven insights dual e effective efficiency efficient election end engagement engineering engineering teams Entra environment error error detection errors evaluation evaluation service Excel exp expert External external datasets face fact feature features fine first for free full g Gemini Gemini model Gemini models Gemini Pro Gen GenAI generated generated data generation generative Generative AI generative AI models Go Google Google Cloud Gretel grounding gs hallucination hallucinations high Highlight HR http HTTPS image implementation implications in industry information infrastructure insights integration inter iOS IRS ite J json Just k Key l labels land language language model language models large large datasets large language model large language models Large Language Models (LLMs) learning led legacy systems liability lightweight limitations llm llms lm logic long low mac machine Machine Learning machine learning model machine learning models making man management max metrics migration Mila mini Mir mixed ML modal modalities model model family model performance model training models Monitor monitoring multi Multimodal multimodal capabilities nation no non NPU o o1 oE of off on one operation opt organization organizations ory out Outputs over patterns pdf performance personally identifiable information Personally Identifiable Information (PII) Pipeline pipelines platform play point porting post Postgre Postgres PostgreSQL potential Power practical example pre preparation preserving principles privacy privacy concerns privacy professionals process processes processing product production production environment products professionals project prompt Prompt Engine prompts Pub/Sub public Py question R rack rag rate Ray RCE real real-world data Real-World Scenarios reasoning red Regulation regulations release reliability report reporting reproducibility resource resources response return right Risk risks Ro Role s safe safety scalable schema search sec security Semantic Semantic Search sensitive information service services settings severity side Sig Sim Simple SoC source specific Speech sql SSE standardization start state storage structured structured data structured formats structured output structured outputs synthetic Synthetic Data synthetic data generation synthetic dataset system System Instructions systems T Task tasks Teams tech techniques test Testing text the to token tokens tool tools Tor TP training transformation transformations transformative trial trie UI unstructured data up US usage use use cases user V val Validation Valuation version version control Vertex Vertex AI video visibility VPP Wi Wind workflow workflows x XAI YouTube