Gravio Blog
June 10, 2021

[Tutorial] Graphing IoT Data at the Edge with RRDTool and Gravio

This is a small tutorial how to create a time series graph in Excel using Gravio Edge IoT CO2 sensor data. While this tutorial refers to the IoT Platform Gravio, parts of it can be applied independently of the Iot Platform as long as you have your data in CSV format.
[Tutorial] Graphing IoT Data at the Edge with RRDTool and Gravio

Introduction

In this tutorial, we will take a look at how to use the IoT Edge Computing platform Gravio to collect data to be injected into and visualised by the RRDtool (Round-Robin Database Tool). 

Approach

In the below example, we use a Gravio Zigbee CO2 sensor and frequently add the incoming sensor data to the local RRDTool database using the “Execute” component. With this command you can execute a command line command.

Note: If you have a Gravio Basic subscription, you could use a Temperature sensor instead, as that will also provide Time Series data, though these readings are sent less frequently.

About RRD

Website: https://oss.oetiker.ch/rrdtool/ 

The RRDTool is a very popular open-source database created by Tobias Oetiker and aims to handle time series data such as network bandwidth, temperatures or CPU load. The data is stored in a circular buffer based database, hence the name “Round Robin”,  thus the system storage footprint remains constant over time. The RRDTool was published as open source software for the first time in 1999.

The RRDTool can be hosted on a variety of platforms, either locally or on a cloud server. The source code can be downloaded from the website and compiled. 

Pricing & Account Information

The RRDTool is open source under the terms of the GNU General Public License V2 or later. This means you can do most things you want with this software as long as you do not claim you created the software and don't sell it (or modified version of it) under a license other than the GNU GPL. The author Tobias Oetiker requests that publications related to RRDTool should credit and link back to his website using this banner:

Tutorial

In this tutorial we look into how to create time series data and add it to the RRDTool database. We use command line instructions triggered by the Gravio IoT Edge platform to add data points to the database.

First of all, let’s look into the basics of how to use RRD. You need mainly 3 commands to create an RRD graph:

  • A command to create the database (and its associated archives)
  • A command that’s used to add data points. This is used the most frequently.
  • A command to create the graph images

There are some good tutorials on the RRDTool website, but for the sake of simplicity we will share the whole process from A to Z here in this post, too.


Requirements

  • In this example, we use macOS, however it is also possible to run RRDTool as well as Gravio on Linux or Windows, where the commands may vary
  • You need to have basic knowledge of the command line
  • You need to install RRDTool. To achieve this, if you have “brew” you can use the command brew install rrdtool in accordance with this tutorial 
  • You need at least one input sensor from this. In this tutorial, we use the Gravio CO2 sensor, which comes with Gravio Standard. But you can also use a Gravio Temperature sensor in the Gravio Basic package, or, if you like to use the USB camera and the people counting AI model, you can use Gravio Free (note: USB cameras are supported on macOS and Linux Gravio HubKits)

Steps 

Part 1: Creation of the database

Step 1: 

RRDTool works mainly with Unix timestamps, the amount of seconds that have passed since the first of January 1970  0:00:00. (UTC)

First we create a database with the starting point. Open your terminal, go to your project folder, for example ~/rrd and type for example:

rrdtool create co2.rrd --start 1623723909 DS:co2:GAUGE:20:U:U RRA:AVERAGE:0.75:2:864000

This means we create a database called co2.rrd and have it start on 1623723909 (we suggest you choose a point in time as close as possible to your current time. We also have a data source (DS) called “co2” and the database works like a GAUGE (as opposed to a counter which is adding the numbers up). 

The 20 after the : means that whenever we get no data for more than 20 seconds, the value is unknown. Also we do not set an upper and lower limit (U = Unknown)

Step 2:

Then we set an RRA (Round Robin Archive). According to the official RRDTool documentation, the format is RRA:{AVERAGE | MIN | MAX | LAST}:xff:steps:rows

AVERAGE: the average of the data points is stored.

MIN: the smallest of the data points is stored.

MAX: the largest of the data points is stored.

LAST: the last data points is used.

xff: The xfiles factor defines what part of a consolidation interval may be made up from *UNKNOWN* data while the consolidated value is still regarded as known. It is given as the ratio of allowed *UNKNOWN* primary data points to the number of primary data points in the interval. Thus, it ranges from 0 to 1 (exclusive).

steps: defines how many of these primary data points are used to build a consolidated data point which then goes into the archive. See also "STEP, HEARTBEAT, and Rows As Durations".

Rows: defines how many generations of data values are kept in an RRA. Obviously, this has to be greater than zero.

So in our example we record the AVERAGE value for the archive. We also set the ratio of unknown to known consolidated data as 0.75. That means RRDTool can create a consolidated data point even if just 75% of the primary data is available. If xff was set to 0, it would mean RRDTool was not allowed to show a consolidated data point if not all of the primary data points are available. If xxf was set to 0.9999, it means hardly any primary data point is required to create a consolidated one.

The number 2 means that 2 primary data points will create one consolidated data point.

The 864000 means that we will keep 864000 generations of data to make graphs from. This is essentially how many rows the database has before it starts over.


Part 2: Adding Data to the Database

Step 3:

Once the database is created, we add the data to the database. For that, we need to send the command line command

/usr/local/bin/rrdtool update /path/to/database/co2.rrd unixtime:datavalue

To do this in Gravio, we create an exec component for each time sensor data comes from the sensor. This is how it looks like:



In our case, the command is 

'/usr/local/bin/rrdtool update /Users/christoph/Documents/Dev/RRDTool/co2.rrd '+ToInt(tv.Timestamp)+":"+tv.Data


This will add the tv.Data whenever it comes in (adding it to an Unix timestamp)

Note, for this to work, you need to give Gravio full disk access, otherwise MacOS will not allow Gravio to write outside its folder:

Step 4:

Once the Action is created, we need to create a trigger, to trigger it. This is is how the trigger looks like:

Make sure it’s assigned to your newly created Action in the “Action” tab



That’s it. Gravio will now record incoming data in your RRD database.

Part 3: Creating PNGs from the Database

Step 5:

As a next step, we want to create the actual PNG image file from the database. Be sure to wait a few minutes or hours until you get a meaningful graph. But once you have data, you can use this command to achieve create a graph:

rrdtool graph /path/to/output/file/co2.png DEF:co2values=/path/to/database/co2.rrd:co2:AVERAGE LINE1:co2values#FF0000


This means we are writing a file co2.png to the /path/to/output/file and we are using the /path/to/database/co2.rrd database for the data.



We create a variable co2values and define the path to the database. 


We tap into the co2 datasource and show the average.


We use line type LINE1 (1 pixel wide) from the co2values variable and set the color to red #FF0000

Step 6:

Here are some alternative outputs:

rrdtool graph /path/to/output/file/co2.png DEF:co2values=/path/to/database/co2.rrd:co2:AVERAGE LINE4:co2values#00FF00



You can of course have Gravio create the graphs for you in regular intervals, or in fact after every time a new data point is added.

This is it. It’s quite simple, isn’t it? If you have any questions or suggestions, please feel free to join our Slack channel and get in touch.

Summary

The RRDTool is an excellent tool to create local PNG files with data. It also helps create overviews without being faced with the challenge of ever increasing databases. To use it, we need to be familiar with the command line, which makes it exceptionally flexible while being “low-tech”. You can see how the above data has been gathered by downloading the package here. You can then upload the package again in your settings view:


RRD Snapshot

  • On-Prem / self hosted options: Yes
  • Cloud / hosted options: Yes
  • Commercial options: Yes
  • Open source options: Yes


Latest Posts
[Case Study] ASICS Corporation Footwear Production Control Department Custom Production Department
How Asics is using the Gravio Edge Computing Platform
Monday, July 26, 2021
Read More
[Tutorial] Graphing IoT Data at the Edge with RRDTool and Gravio
This is a small tutorial how to create a time series graph in Excel using Gravio Edge IoT CO2 sensor data. While this tutorial refers to the IoT Platform Gravio, parts of it can be applied independently of the Iot Platform as long as you have your data in CSV format.
Wednesday, July 7, 2021
Read More