Real-time pipeline monitoring for the energy sector

Pipeline systems and sensors generate masses of information. With so much at their fingertips, operators have more data than ever to help transform pipeline efficiency.

The question is, how does our intelligent pipeline solution, Digipipe, help operators squeeze maximum value out of their data? We leverage real-time stream processing for IoT workloads with InfinyOn, a real-time event streaming company. 

At the beginning of March 2023, we held a webinar with InfinyOn showcasing how we use Fluvio to collect, clean and transmit large amounts of system or sensor data through Digipipe. For an in-depth understanding and to see a demo of Digipipe in action, give the whole webinar a watch or catch up on the highlights below.

Klarian Data Ingestion

There is a lot of data out there, and Klarian can ingest it from many places:

• Our proprietary edge controller
• External APIs such as commodity pricing to enrich data
• Third-party data integrations such as existing SCADA, control room and incident management data

Our agnostic data ingestion gives Digipipe a distinct advantage: it captures data from nearly any source you can think of. The advantage is that the more data you can ingest, the greater the potential insight.

The Data Challenge

You must do more than ingest data to get insight. At Klarian, we want to help you squeeze all possible value from your data. Once we’ve ingested it, we store the data, stream it to frontends and perform actions on the data in transit.

When developing Digipipe, we faced various hurdles in delivering these capabilities. Some of our main challenges were:

• Many moving parts from protocols varying from OPC to Modbust to MQTT
• Lots of supporting tools are required, each requiring logging, monitoring and maintenance
• Third-party data integrations required a lot of cleaning
• Our data team spent 90% of their time cleaning data before they could work on it

Entering a strategic technology agreement with InfinyOn and deploying Fluvio within our tech stack helped us overcome these data challenges.

Simplified architecture for improved efficiency

Klarian uses Fluvio, InfinyOn’s open-source data streaming platform, to simplify Digipipe’s architecture. We use a series of connectors (e.g., MQTT, HTTP) managed inside the InfinyOn Cloud service. It allows us to write and configure data to be transformed and streamed in.

Integrating Fluvio has delivered significant benefits for Digipipe:

• Simplified architecture
• Efficiency improvement
• Extensible

Where Fluvio sits in our Digipipe platform

We use Fluvio throughout Digipipe, including for:

• Data ingestion
• Real-time data visualisation
• STL workflow
• Streaming AI

Digipipe and Fluvio in action

Digipipe offers many views, including a map view and various dashboard views.

Fluvio enables us to stream data straight to the map view. You can view tickets and incidents from the home screen and operations teams get immediate snapshots of performance.

We also feed data from the InfinyOn Cloud into our dashboards for a real-time feed. We combine real-time data with a mix of historical and contextual data to provide a powerful view. Digipipe is highly configurable, and users can modify dashboards by adding blocks made up of a query or data stream.

How we’ve integrated Fluvio into our platform also allows you to work with streams and data frames. You can build out Web Assembly modules making everything portable and easily sharable across your team. Digipipe enables stream processing, so your data teams don’t have to spend 70% of their time cleaning data.

This article offers a snapshot of the webinar. If anything piqued your interest, you should give the whole thing a watch for more detail and insight into how we deliver real-time pipeline monitoring for the energy sector:

If you want to learn more about our technology and stay up to date on future webinars, sign-up for our newsletter: See More with Klarian.