Source URL: https://cloud.google.com/blog/products/databases/ford-pro-intelligence-built-on-bigtable-nosql-database/
Source: Cloud Blog
Title: How Ford Pro uses Bigtable to harness connected vehicle telemetry data
Feedly Summary: Ford Pro Intelligence is a cloud-based platform that is used for managing and supporting fleet operations of its commercial customers. Ford commercial customers range from small businesses, large enterprises like United Postal Service and Pepsi where fleets can be thousands of vehicles, and government groups and municipalities like the City of Dallas. The Ford Pro Intelligence platform collects connected vehicle data from fleet vehicles to help fleet operators streamline operations, increase productivity, reduce cost of ownership, and improve their fleet’s performance and overall uptime through the alerts on vehicle health and maintenance.
Telemetry data from vehicles provides a wealth of opportunity, but it also presents a challenge: planning for the future as cars and services evolve. We needed a platform that could support the volume, variety and velocity of vehicle data as automotive innovations emerge, including new types of car sensors, more advanced vehicles, and increased integration with augmented data sources like driver information, local weather, road conditions, maps, and more.
In this blog, we’ll discuss our technical requirements, the decision-making process, and how building our platform with Bigtable, Google Cloud’s flexible NoSQL database for high throughput and low-latency applications at scale, unlocked powerful features for our customers like real-time vehicle health notifications, AI-powered predictive maintenance, and in-depth fleet monitoring dashboards.
aside_block
Scoping the problem
We wanted to set some goals for our platform based on our connected vehicle data. One of our primary goals is to provide real-time information for fleet managers. For example, we want to inform our fleet partners immediately if tire pressure is low, a vehicle requires brake maintenance, or there is an airbag activation, so they can take action.
Connected vehicle data can be extremely complex and variable. When Ford Pro set out to build its vehicle telemetry platform, we knew we needed a database that could handle some unique challenges. Here’s what we had to consider:
A diverse and growing vehicle ecosystem: We handle telemetry data from dozens of car and truck models, with new sensors added every year to support different requirements. Plus, we support non-Ford vehicles too!
Connectivity isn’t a guarantee: A “connected" car isn’t always connected. Vehicles go offline due to spotty service or even just driving through a tunnel. Our platform needs to handle unpredictable or duplicated streams of time-series data.
Vehicles are constantly evolving: Manufacturers frequently push over-the-air updates that change how vehicles operate and the telemetry data they generate. This means our data is highly dynamic, and our database needs to support a flexible, ever-evolving schema.
Security is paramount: At Ford, we are committed to our customer’s data privacy and security. It’s imperative to our technology. We serve customers around the world and must ensure we can easily incorporate privacy and security measures while maintaining regulatory compliance, such as GDPR, in every country we operate.
These challenges, along with the application feature requirements, we knew that we needed an operational data store that can support low-latency access for both real-time and historical data with a flexible schema.
Where we started
The Ford Pro Intelligence platform offers a diverse range of features and services that cater to the diverse needs of our customers. To ensure flexibility in data access, we prioritize real-time reporting of vehicle status, event-based notifications, location services, and historical journey reconstruction. These capabilities necessitate a variety of data access methods to support both real-time and historical data access — all while maintaining low latency and high throughput to meet the demands of Ford customers.
Our starting point was an Apache Druid-based data warehouse that contained valuable historical data. While Apache Druid could handle high-throughput write traffic and generate reports, it was not able to support our low-latency API requirements or high data volumes. As a result, we started working with Google Cloud to explore our options.
We began our search with BigQuery. We already used BigQuery for reporting, so this option would have given us a serverless, managed version of what we already had. While BigQuery was able to perform the queries we wanted, our API team raised concerns about latency and scale — we required single-digit millisecond latency with high throughput. We discussed putting a cache layer in front of BigQuery for faster service of the latest data but soon discovered that it wouldn’t scale for the volume and variety of requests we wanted to offer our customers.
From there, we considered several alternative options, including Memorystore and PostgreSQL. While each of these solutions offered certain advantages, they did not meet some of our specific requirements in several key areas. We prioritized low-latency performance to ensure real-time processing of data and seamless user experiences. Flexibility, in terms of schema design, to accommodate our evolving data structures and wide column requirements was also a must. Scalability was another crucial factor as we anticipated significant growth in data volume and traffic over time.
When we looked at Bigtable, its core features of scalable throughput and low latency made it a strong contender. NoSQL is an ideal option for creating a flexible schema, and Bigtable doesn’t store empty values, which is great for our sparse data and cost optimization. Time-series data is also inherent to Bigtable’s design; all data written is versioned with a timestamp, making it a naturally good fit for use cases with vehicle telemetry data. Bigtable also met our needs for an operational data store and analytics data source, allowing us to handle both of these workloads at scale on a single platform. In addition, Bigtable’s data lifecycle management features are specifically geared toward handling the time-oriented nature of vehicle telemetry data. The automated garbage collection policies use time and version as criteria for purging obsolete data effectively, enabling us to manage storage costs and reduce operational overhead.
In the end, the choice was obvious, and we decided to use Bigtable as our central vehicle telemetry data repository.
Ford Pro Telematics and Bigtable
We receive vehicle telemetry data as a protocol buffer to a passthrough service hosted on Compute Engine. We then push that data to Pub/Sub for Google-scale processing by a streaming Dataflow job that writes to Bigtable. Ford Pro customers can access data through our dashboard or an API for both historical lookback for things like journey construction and real-time access to see fleet status, position, and activity.
Figure 1: High-level architecture showing vehicle telemetry data capture
With Bigtable helping to power Ford Pro Telematics, we have been able to provide a number of benefits for our customers, including:
Enabling the API service to access telematics data
Improving data quality with Bigtable’s built-in time series data management features
Reducing operational overhead with a fully managed service
Delivering robust data regulation compliance tooling across regions
The platform provides interactive dashboards that display relevant information, such as real-time vehicle locations, trip history, detailed trip information, live map tracking and EV charging status. Customers can also set up real-time notifications about vehicle health and other important events, such accidents, delays, or EV charging faults. For example, a fleet manager can use the dashboard to track the location of a vehicle and dispatch assistance if an accident occurs.
Figure 2: Real-time dashboards show fleet status and location
We leverage BigQuery alongside Bigtable to generate reports. BigQuery is used for long-running reports and analysis, while Bigtable is used for real-time reporting, and direct access to vehicle telemetry. Regular reports are available for fleet managers, including vehicle consumption, driver reimbursement reports, and monthly trip wrap ups. Our customers can also leverage and integrate this data into their own tooling using our APIs, which enable them to query vehicle status and access up to one year of historical data.
Figure 3: Vehicle telemetry analysis
Looking to the future
The automotive industry is constantly evolving, and with the advent of connected vehicles, there are more opportunities than ever before to improve the Ford commercial customer experience. Adopting a fully managed service like Bigtable allows us to spend less time maintaining our own infrastructure and more time innovating and adding new features to our platform. Our company is excited to be at the forefront of this innovation, and we see many ways that we can use our platform to help our customers.
One of the most exciting possibilities is the use of machine learning to predict vehicle maintenance and create service schedules. By collecting data from vehicle diagnostics over time, we can feed this information into machine learning models that can identify patterns and trends. This will allow us to alert customers to potential problems before they even occur, and to schedule service appointments at the most convenient times.
Another area where we can help our customers is in improving efficiency. By providing insights about charging patterns, EV range, and fuel consumption, we can help fleet managers optimize their operations. For example, if a fleet manager knows that there are some shorter routes for their cars, they can let those cars hit the road without a full charge. This can save time and money, and it can also reduce emissions.
In addition to helping our customers save time and money, we are also committed to improving their safety and security. Our platform can provide alerts for warning lights, oil life, and model recalls. This information can help customers stay safe on the road, and it can also help them avoid costly repairs.
We are already getting great feedback from customers about our platform, and we are looking forward to further increasing their safety, security, and productivity. We believe that our platform has the potential to revolutionize the automotive industry, and we are excited to be a part of this journey.
Get started
Learn more about Bigtable and why it is a great solution for automotive telemetry and time series data.
Read more on how others like Palo Alto Networks, Flipkart, and Globo are reducing cloud spend while improving service performance, scalability and reliability by moving to Bigtable.
Ready to unlock the power of Bigtable? Start a free trial today!
AI Summary and Description: Yes
Summary: Ford Pro Intelligence is a cloud-based platform designed to manage fleet operations for commercial customers, utilizing connected vehicle data to enhance productivity and performance. The platform leverages Google Cloud’s Bigtable for handling diverse telemetry data efficiently, ensuring low latency and scalability while prioritizing security and compliance with regulations such as GDPR.
Detailed Description: The Ford Pro Intelligence platform addresses the needs of its diverse commercial customers, ranging from small businesses to large enterprises and municipalities. Key points include:
– **Data Collection and Purpose**:
– Collects real-time telemetry data from various vehicle models to monitor health and maintenance.
– Offers alerts for potential issues, contributing directly to improved fleet management and enhanced decision-making.
– **Technical Challenges**:
– The complexity of managing diverse vehicle data due to varying models and sensor technologies.
– Challenges with connectivity that can lead to inconsistent data streams, particularly in areas with poor service.
– **Flexibility and Scalability**:
– The need for a database solution that can accommodate evolving data structures and handle high volumes of real-time data efficiently.
– Bigtable was chosen for its NoSQL capabilities, such as flexible schema design and optimized management of time-series data, which is ideal for vehicle telemetry.
– **Security and Compliance**:
– Strong emphasis on data privacy and security to protect customer data across global operations.
– Assurance of compliance with regulations like GDPR, reflecting the commitment to operational integrity and protecting customer interests.
– **Features Enabled by Bigtable**:
– Real-time access to telemetry data through an API and interactive dashboards.
– Capabilities for predictive maintenance using machine learning by analyzing historical vehicle data patterns.
– Improved operational oversight with real-time notifications regarding vehicle conditions.
– **Future Prospects**:
– Commitment to continuous innovation through machine learning applications to optimize fleet operations and enhance customer safety.
– Focus on sustainability by helping fleet managers minimize emissions and manage resources effectively.
This platform serves as a model for integrating cloud solutions with operational data management in the automotive sector, presenting significant insights for professionals in AI, cloud computing, and infrastructure security.