Source URL: https://cloud.google.com/blog/products/management-tools/opentelemetry-now-in-google-cloud-observability/
Source: Cloud Blog
Title: OpenTelemetry Protocol comes to Google Cloud Observability
Feedly Summary: OpenTelemetry Protocol (OTLP) is a data exchange protocol designed to transport telemetry from a source to a destination in a vendor-agnostic fashion. Today, we’re pleased to announce that Cloud Trace, part of Google Cloud Observability, now supports users sending trace data using OTLP via telemetry.googleapis.com.
Fig 1: Both in-process and collector based configurations can use native OTLP exporters to transmit telemetry data
Using OTLP to send telemetry data to observability tooling with these benefits:
Vendor-agnostic telemetry pipelines: Use native OTLP exporters from in-process or collectors. This eliminates the need to use vendor-specific exporters in your telemetry pipelines.
Strong telemetry data integrity: Ensure your telemetry data preserves the OTel data model during transmission and storage and avoid transformations into proprietary formats.
Interoperability with your choice of observability tooling: Easily send telemetry to one or more observability backends that support native OTLP without any additional OTel exporters
Reduced client-side complexity and resource usage: Move your telemetry processing logic such as applying filters to the observability backend, reducing the need for custom rules and thus client-side processing overhead.
Let’s take a quick look at how to use OTLP from Cloud Trace.
Cloud Trace and OTLP in action
Sending trace data using OTLP via telemetry.googleapis.com is now the recommended best practice for both new and existing users — especially for those who expect to send high volumes of trace data.
Fig 2: Trace explore page in Cloud Trace highlighting fields that leverage OpenTelemetry semantic conventions
The Trace explorer page makes extensive use of OpenTelemetry conventions to offer a rich user experience when filtering and finding traces of interest. For example,
The OpenTelemetry convention service.name is used to indicate which services a span is originating from.
The status of the span is indicated by the OpenTelemetry’s span status.
Cloud Trace’s internal storage system now uses the OpenTelemetry data model natively for organizing and storing your trace data. The new storage system enables much higher limits when trace data is sent through telemetry.googleapis.com. Key changes include:
Attribute sizes: Attribute keys can now be up to 512 bytes (from 128 bytes), and values up to 64 KiB (from 256 bytes).
Span details: Span names can be up to 1024 bytes (from 128 bytes), and spans can have up to 1024 attributes (from 32).
Event and link counts: Events per span increase to 256 (from 128), and links per span are now 128.
We believe sending your trace data using OTLP will result in an better user experience in the trace explorer UI and Observability Analytics, along with the above storage limit increases.
Google Cloud’s vision for OTLP
Providing OTLP support for Cloud Trace is just the beginning. Our vision is to leverage OpenTelemetry to generate, collect, and access telemetry across Google Cloud. Our commitment to OpenTelemetry extends across all telemetry types — traces, metrics, and logs — and is a cornerstone of our strategy to simplify telemetry management and foster an open cloud environment.
We understand that in today’s complex cloud environments, managing telemetry data across disparate systems, inconsistent data formats, and vast volumes of information can lead to observability gaps and increased operational overhead. We are dedicated to streamlining your telemetry pipeline, starting with focusing on native OTLP ingestion for all telemetry types so you can seamlessly send your data to Google Cloud Observability. This will help foster true vendor neutrality and interoperability, eliminating the need for complex conversions or vendor-specific agents.
Beyond seamless ingestion, we’re also building capabilities for managed server-side processing, flexible routing to various destinations, and unified management and control over your telemetry across environments. This will further our observability experience with advanced processing and routing capabilities all in one place.
The introduction of OTLP trace ingestion with telemetry.googleapis.com is a significant first step in this journey. We’re continually working to expand our OpenTelemetry support across all telemetry types with additional processing and routing capabilities to provide you with a unified and streamlined observability experience on Google Cloud.
Get started today
We encourage you to begin using telemetry.googleapis.com for your trace data by following this migration guide. This new endpoint offers enhanced capabilities, including higher storage limits and an improved user experience within Cloud Trace Explorer and Observability Analytics.
AI Summary and Description: Yes
Summary: The text provides an overview of the newly implemented support for the OpenTelemetry Protocol (OTLP) within Google Cloud Trace, highlighting its benefits such as vendor-agnostic telemetry, increased data integrity, and improved user experience. This is significant for AI, cloud, and infrastructure security professionals aiming for streamlined observability in complex environments.
Detailed Description: The announcement regarding OpenTelemetry Protocol (OTLP) integration in Google Cloud offers several noteworthy implications for the security and operations domains.
– **Vendor-Agnostic Telemetry Pipelines:**
– The integration allows users to utilize native OTLP exporters, moving away from vendor-specific solutions.
– This enhances flexibility in telemetry management, reducing reliance on individual vendor tools.
– **Data Integrity Enhancements:**
– By preserving the OpenTelemetry data model, stakeholders can ensure that telemetry information remains consistent throughout transport and storage, which is critical for accurate monitoring and compliance.
– **Interoperability:**
– Users can seamlessly send traces to multiple observability backends that support OTLP, enhancing tool choice and integration without the hassle of additional exporters.
– **Reduction in Client-Side Complexity:**
– Offloading telemetry processing to the observability backend lessens client-side resource usage and simplifies operational overhead, allowing for more efficient data collection and management.
– **Higher Limits in Trace Data Storage:**
– Key storage enhancements include:
– Attribute key sizes increased to 512 bytes, while value sizes can reach 64 KiB.
– Span names expanded to 1024 bytes with the capacity to hold up to 1024 attributes.
– Increased event and link counts enabling more comprehensive tracing capabilities.
– **Future Directions:**
– Google Cloud aims to extend OTLP practices to metrics and logs, creating a comprehensive telemetry management framework that promotes open standards.
– The strategic focus is on reducing observability gaps and operational complexities inherent in multi-cloud environments, which has implications for compliance and data governance.
– **Call to Action:**
– Users are encouraged to migrate to the new telemetry processing endpoint, promising an improved experience in data handling and analytics.
Overall, the integration of OpenTelemetry in Google Cloud Trace not only simplifies telemetry management but also enhances the observability features necessary for security and compliance within cloud environments, making it a pivotal development for security professionals.