Source URL: https://cloud.google.com/blog/products/data-analytics/bigquery-migration-services-innovations/
Source: Cloud Blog
Title: Simplify your data platform migration with AI-powered BigQuery Migration Services
Feedly Summary: Migrating data workloads to BigQuery, our unified Data to AI platform, just got significantly easier. You no longer have to choose between unlocking value from your data assets by migrating to a modern data platform, or mitigating risk by staying put. You can achieve both with BigQuery Migration Services, a collection of free-to-use, cloud-native services that enable large-scale transformations for data warehouses and data lakes by breaking down migrations into templated, iterative and manageable steps. They move data, code, and business logic from on-premises and cloud platforms to BigQuery, utilizing a “next-best action” approach that minimizes time-to-migrate and maximizes ROI for your business transformation.
At Google Cloud Next 25, we announced several new innovations in BigQuery Migration Services, including coverage for data science and expanding support for data engineering and data analytics workloads. New capabilities span across four stages of a data platform migration: 1) automated assessment and planning, 2) automatic code translation, 3) data migration, and 4) validation.
BigQuery Migration Services
Let’s look at the new innovations in BigQuery Migration Services.
1. Automated discovery and assessment with estimated total cost of ownership
Your data platform migration journey begins with automated discovery and assessment of the source environment. BigQuery Migration Services’ automated assessments provide details of the existing environment, create an insights-filled view of the workloads’ projected landed state on BigQuery (including performance and estimated total cost of ownership), and guide you on how to get to BigQuery (migration planning). You can run an assessment with the push of a button on the Google Cloud console, which delivers a detailed Looker-studio report and BigQuery datasets as output. Assessments are available for Teradata, Snowflake, and Redshift, and today, we also announced that assessments for Oracle/Exadata and Cloudera/Hive are available immediately, and that a Databricks assessment is coming soon.
To help with a structured and successful migration, we also announced a source lineage service in preview. This service automatically identifies and groups dependencies between workloads, creating an explicit ordering in which to move them, helping to minimize risk and disruption, and improving time-to-value.
aside_block
2. Automated code translations
Some of our heaviest investments in BigQuery Migration Services over the past years have been in our code translation services, which migrate code from 15+ sources. Today, we announced advancements in Gemini-enhanced code translation, which was previously only available in interactive mode, letting you translate code, like you would with, say, Google Translate.
Now, Gemini-enhanced code translations are also available in batch and API modes, helping you migrate at scale. Coupled with a new unified Translation API that backs all three modes, you can first translate bulk code using batch or API modes, and then fine-tune and debug it using interactive mode.
Gemini-enhanced translations
Now, you can also preprocess your code with Gemini, so you can migrate code that’s not just SQL, but also other kinds of code, e.g., an ETL job with SQL embedded inside XML. This means you don’t need to submit perfectly clean SQL, and can translate code from sources that don’t have full compiler coverage yet.
Finally, there’s an enhanced user-experience in the console to guide you at each step of the translation process, suggesting the next-best action to get you to the finish line.
Enhanced User Experience
These advancements dramatically reduce code conversion times while continuing to deliver over 95% accuracy, helping you tackle large migration jobs with greater efficiency.
3. Data, metadata and permissions migration
Historically, BigQuery Migration Services have supported large-scale data migrations from Teradata and Amazon Redshift. Today, BigQuery Migration Services support incremental updates from Teradata, batch and incremental file and permission migrations from Cloudera, and batch and incremental data migration from Snowflake, all in preview. All migrated data is automatically validated as part of the migration process.
4.Intelligent end-to-end validation
Each step of the migration process will soon include an intelligent validation mechanism that can incorporate schema and data-type updates, vs. static data checksum comparisons that exist today. You can combine validation with source lineage, making it easy to quickly identify discrepancies between source and target environments. This comprehensive code, data, and dependency validation helps ensure your business applications stay intact as you incrementally move them.
Together, these investments in each of the four stages of a data platform migration help automate your journey while containing risk, providing deterministic outcomes, and faster ROI.
Customer successes
Customers trust BigQuery Migration Services for migrating their mission-critical workloads. BigQuery Migration Services usage has grown 3x year over year, with thousands of customers using the services to migrate workloads to BigQuery.
“By migrating from Databricks to BigQuery and combining our own models with the models provided by Google Cloud, we’ve improved the performance and efficiency of our machine learning processes and better positioned ourselves for ongoing growth.” – Hamdi Amroun, Head of AI, Yassir
“BigQuery has unlocked unprecedented scalability and flexibility for VMO2, improving data platform availability and uptime, which ultimately enhances customer experience. By moving all key functions to Google Cloud, VMO2 has reduced its TCO for equivalent on-premises platforms by approximately 30%." – Vinay Pai, Head of Data Architecture, Virgin Media O2
Take the next steps
Ready to start migrating your data platform to BigQuery? We’re ready to help!
Learn more about BigQuery Migration Services and try it for free.
Connect with Google Cloud Consulting and select migration partners standardized on BigQuery Migration Services to achieve predictable and cost-effective outcomes.
Sign up today for the BigQuery migration incentives program for additional benefits such as Google Cloud credits, implementation services and cloud egress credits.
AI Summary and Description: Yes
Summary: The text discusses new advancements in Google Cloud’s BigQuery Migration Services, designed to simplify the migration of data workloads to a modern data platform. It emphasizes the importance of balancing value extraction from data migration with risk mitigation, providing automated tools and services for seamless transitions.
Detailed Description:
The provided text outlines several significant innovations and features within Google Cloud’s BigQuery Migration Services aimed at streamlining the transition of data workloads to the BigQuery platform. The text highlights that these services now allow businesses to migrate their data without sacrificing risk management, thereby enhancing operational efficiency and return on investment.
Key Points and Features:
– **Automated Discovery and Assessment**: Initiates the migration process through automated discovery and assessment of existing environments, providing insights into projected performance and total cost of ownership.
– **Automated Code Translation**: Enhancements in code translation services (now Gemini-enhanced) allow for seamless migration of code from over 15 sources, supporting batch and API modes, which facilitate large-scale migrations.
– **Data, Metadata, and Permissions Migration**: Expanded support for various data sources including Teradata, Redshift, Cloudera, and Snowflake, with support for incremental updates and automatic validation during the migration process.
– **Intelligent End-to-End Validation**: Each migration step includes a comprehensive validation mechanism to ensure data integrity and minimize discrepancies between source and target environments, enhancing reliability during migrations.
– **Customer Success Stories**: Testimonials showcase how BigQuery Migration Services have significantly improved operational performance for companies, highlighting successful transitions that have led to lower total cost of ownership (TCO) and enhanced scalability.
The text serves as a strategic overview for security and compliance professionals in understanding how these migration services not only facilitate data transitioning but also ensure that risk is managed throughout the process. It provides insights into the efficiency of the migration journey and the importance of automation in maintaining data integrity and compliance with regulatory requirements during such transitions.