What Is Big Data Analytics for IoT?

Big data analytics for IoT is the practice of turning continuous machine telemetry into decisions. Connected industrial equipment produces a constant stream of readings, and the value is not in storing them but in spotting the pattern that predicts a failure, a maintenance window, or a drop in efficiency before it costs anything.

What makes it different from conventional analytics is the shape of the data. It arrives continuously rather than in batches, in bursts that vary by orders of magnitude, from sources that rarely agree on format. A traditional data warehouse expects structured records on a predictable schedule. Neither assumption holds when the source is a fleet of machines.

That pushes the architecture in a specific direction. Ingest has to absorb variable load without dropping readings. Enrichment has to add the context that makes a reading meaningful, time zone, location, asset history, at the point of ingest rather than later. Quality checks have to run continuously, because a sensor that starts reporting nonsense will quietly poison every downstream model if nothing catches it.

Multi-tenancy adds a further constraint. When several manufacturers, sometimes direct competitors, run on the same platform, isolation is not a feature but a precondition for the platform existing at all.

Cost is the other constraint people underestimate. Processing continuous telemetry at scale is expensive if compute runs whether or not there is anything to process, which is why on-demand worker processes and tiered storage matter as much to the economics as the analytics do to the outcome.

The engagement below shows how those requirements shaped a working pipeline for GE Digital, from event ingestion through to the analytics and machine learning that sit on top of it.

The enabling technologies of IoT — connected devices and sensors with acceleration of analytics via the cloud — rely on an integrated data infrastructure. GE Digital is at the leading edge of the convergence in Industrial IoT, providing a blended data pipeline that integrates cutting-edge digital instrumentation with financial systems like pricing, billing, and subscription management. With faster access and insight across physical plant, customer, and platform partners, GE Digital provides unique leverage via advanced analytics and artificial intelligence, to achieve data-driven efficiency, productivity, and profitability.

The Challenge

GE Digital needed an agile data fabric suited both for real-time data ingestion and enrichment, in tandem with a broad spectrum of systems of record. Building a new agile data pipeline needed a more modular approach beyond conventional data lakes and traditional data warehouses. Moreover, the architecture needed to accommodate a scalable, secure approach to tenant isolation and service multi-tenancy, as often different competitors in a single industry needed the confidence that their data was private and secure. In addition to maintaining quality data streams at ingest, it needed to deliver variable capacity for continuous bursts to data flows. This adaptive approach is essential to unlocking the potential of machine learning and AI for smart, connected products. It can unleash untapped digital potential for asset-intensive industries.

The Solution

CloudGeometry provides GE Digital’s IIoT infrastructure with data pipeline logic and integration services. This blends event data into a flexible platform serving a range of downstream analytics cases, ranging from business intelligence dashboards to machine learning. The data ingestion process draws on asynchronous event payloads from both physical and digital devices. Different enrichment algorithms create extensions to even data, such as: time-and-date for calendar intervals (e.g., global time zones) of cycles (for subscription reporting); geo-tagging to add information on event locales; and the like. Data quality is continuously tracked across data cleansing, mapping, and quality processes.

Asynchronous processing of IIoT event streams is a key to cost-effective data integration in the context of true multi-tenancy. CloudGeometry built containerized event logic that runs worker processes on demand, spinning up and shutting down pipelines to optimize compute costs. The pipeline architecture also takes advantage of storage economies thru split caching of both short and long-term key-value stores, complemented by metadata stores, for configuring of both events and the range of devices.

The Benefits

The versatility of end-to-end data pipeline and data integration by CloudGeometry provides GE Digital a flexible, continuously adaptable data movement as IIoT adoption accelerates to its full potential.

<div class="case__txt--cols"><div><h4>Data Ingestion</h4><p>Use powerful, modern data connector framework to pull data from any source, stream or batch; add new data & transformation rules in minutes, not weeks.</p></div><div><h4>Agile Analytics</h4><p>Cut friction of transformation, aggregation, computation combining streaming (Kinesis), Data Warehouse (Redshift) and relational (Aurora) data stores.</p></div><div><h4>Elastic Microservices</h4><p>Easily configure and run Dockerized event-driven, pipeline-related data tasks with Kubernetes.</p></div></div>