InfluxDB IoT dev guide

Introduction

Notice

🚧 This guide is a work-in-progress.

This guide is for developers looking to write Internet-of-Things applications using the InfluxDB platform. InfluxDB provides an API for programmatically writing, querying, and managing time series data.

We’ll describe the concepts, architecture, functionality, needed to build an IoT times series application. Then we construct the code, piece by piece.

Who should read this guide

Anyone interested in building web applications for IoT and times series data collection.

This guide is targeted toward moderately experienced developers (Those who have a built a web application before, know some JavaScript, or have coded with APIs), although beginners should be able to learn from it as well!

How to use this guide

The guide is written to be read from beginning to end, but can be browsed as well. You’ll want to get the example application source code. The code for this application is meant to be read with this guide. We'll walk through writing the most important bits of the code, but some assembly is required. We provide the example application as a reference implementation.

Key concepts in InfluxDB

The following concepts are essential for working with InfluxDB. They are at the core of the way it writes, stores, and accesses time series data.

Read the docs 😎📘

Throughout this guide we will link to the InfluxDB documentation, which contains a wealth of useful information.

If you are new to InfluxDB, you might want to familiarize yourself with installing and writing data with InfluxDB first.

Line protocol

Line protocol is InfluxDB’s data input format.

Use line protocol format to write data into InfluxDB. Each line represents a data point. Data points contain tags and fields.

Example

Here is an example of line protocol:

environment,devId=b47f6944 Temp=21.00,Lat=50.087325,Lon=14.407154 1603091412

White space, commas, and position all have important meanings in line protocol. Let’s examine the parts of this line.

environment,devId=b47f6944 Temp=21.00,Lat=50.087325,Lon=14.407154 1603091412
|         | |            | |                                    | |        |
+---------+ +------------+ +------------------------------------+ +--------+
measurement tags           fields                                 timestamp

The measurement comes first and must be followed by a comma. After the comma is a tag set, then a space, followed by a field set. The timestamp is optional and comes after another space. If a timestamp is not provided, the current time is used.

InfluxDB resources

The code in the example application frequently uses the following four concepts.

User

InfluxDB users are granted permission to access the database. Users are added as a member of an organization and use authentication tokens to access resources.

Token

Tokens (or authentication tokens) verify user and organization permissions in InfluxDB 2. There are different types of athentication tokens:

admin tokens grant full read and write access to all resources in all organizations¹
all-access tokens grant full read and write access to all resources in an organization
read and write tokens grant read or write access to specific resources in an organization

Bucket

A bucket is a named location where time series data is stored. All buckets have a retention policy, a duration of time that each data point persists. All buckets belong to an organization.

Organization

An organization is a logical workspace for a group of users. Members, buckets, tasks, and dashboards (along with a number of other resources), belong to an organization.

Cardinality

Performance in accessing time series data can be affected by the cardinality of that data. In InfluxDB, the primary concern is series cardinality.

Flux

Flux is InfluxDB’s scripting language. Flux code can be sent by other code as strings to be executed by the InfluxDB API.

A simple query might look like this:

from(bucket:"example-bucket")
  |> range(start:-1h)
  |> filter(fn:(r) =>
    r._measurement == "my-measurement" and
    r.my-tag-key == "my-tag-value"
  )

Query data from a bucket with the from() function. The range() function filters records based on time bounds. The filter() function filters data based on conditions defined in a predicate function (fn).

You can use Flux for a variety of tasks including but not limited to:

data transformations
data analysis
writing custom tasks, checks, and notifications

InfluxDB Cloud does not support admin tokens.

Annotated CSV

An Annotated CSV is the output format of a Flux query with InfluxDB 2.0.

For example, imagine that we’re writing data about the number of “calico” and “tabby” cats, both “young” and “adult” cats, in two shelters, “A” and “B”. The data layout looks like this:

Bucket	Measurement
“cats-and-dogs”	“cats”

Tag Keys	Tag Values
“shelter”	“A”, “B”
“type”	“calico” , “tabby”

Field Keys
“young” , “adult”

If we query our data with:

from(bucket: "cats-and-dogs")
  |> range(start: 2020-05-15T00:00:00Z, stop: 2020-05-16T00:00:00Z)
  |> filter(fn: (r) => r["_measurement"] == "cats")
  |> filter(fn: (r) => r["_field"] == "adult")
  |> filter(fn: (r) => r["shelter"] == "A")
  |> filter(fn: (r) => r["type"] == "calico")
  |> limit(n:2)Copy

This is what our Annotated CSV result looks like:

#group,false,false,true,true,false,false,true,true,true,true
#datatype,string,long,dateTime:RFC3339,dateTime:RFC3339,dateTime:RFC3339,double,string,string,string,string
#default,_result,,,,,,,,,
,result,table,_start,_stop,_time,_value,_field,_measurement,shelter,type
,,0,2020-05-15T00:00:00Z,2020-05-16T00:00:00Z,2020-05-15T18:50:33.262484Z,8,adult,cats,A,calico
,,0,2020-05-15T00:00:00Z,2020-05-16T00:00:00Z,2020-05-15T18:51:48.852085Z,7,adult,cats,A,calico

The Annotated CSV has three Annotations. The Annotation rows describe column properties and start with a #.

#group: A boolean that indicates the column is part of the group key. A group key is a list of columns for which every row in the table has the same value. Let’s look at the difference between the true and false columns, or the columns that are and aren’t part of the group key, respectively.
- true columns: In our example query above, we’ve filtered by a single field type, adult, a single “shelter” tag, “A”, and a single “type” tag, “calico”. These values are constant across rows, so those columns are set to true. Also, filtering for a single value across tags and fields means that all of our tables will belong to the same table. Therefore the table column is also true for this query. The _start and _stop columns, defined by our range, are constants across the rows so these values are also true.
- false columns: The _time and _value columns have different values values across rows which is why they receive a false for the value of the #group Annotation.
#datatype: Describes the type of data or which line protocol element the column represents.
#default: The value to use for rows with an empty value. So for example, if we had assigned our query to the variable ourQuery this annotation would look like: #default,ourQuery,,,,,,,,,

IoT Center: introduction

(The IoT Center application)

The Iot Center application manages IoT devices that write data into InfluxDB. IoT Center shows connected devices and measured values from InfluxDB in custom dashboards. It is designed to demonstrate one possible application architecture for a web app using InfluxDB and IoT clients.

Each IoT device measures temperature. Depending on the connected sensors, it can provide additional measurements like humidity, pressure, and CO2 concentration. Each device can either provide static GPS coordinates or actual coordinates from a connected GPS module.

Reading the source code

The IoT Center is designed for demonstration purposes with this guide in mind. To improve readability of the source code, features are developed in separate files with minimum sharing among the components. While a production application would be structured differently, the IoT Center can be used to inspire further development.

The project also includes source code for IoT devices based on Arduino (ESP8266/32) and Raspberry Pi with Python. We will cover device code in later chapters.

Run the IoT Center

Before running the application (the server and web interface), install InfluxDB 2.0.0+. Download the version matching your OS and architecture from the InfluxData downloads page. Configure your username, organization, and token.

For detailed instructions, see Get started with InfluxDB.

Then, clone the IoT Center repository.

git clone https://github.com/bonitoo-io/iot-center-v2.git
cd iot-center-v2
git checkout d93d7d963f23059ce53ac4a3813a1f822f257ba5

Next, install the requirements:

InfluxDB must be running when the IoT Center starts. Start the InfluxDB server by running:

influxd

Configure your environment variables so that IoT Center can communicate with InfluxDB:

export INFLUX_TOKEN=<mysecrettoken>
export INFLUX_ORG=<myorg>
export INFLUX_BUCKET= iot_center # this is set by IoT center by default
export INFLUX_URL= http://localhost:8086 # this is set by IoT center by default

Finally, run the application:

cd iot-center-/app/
yarn install
yarn build
yarn start
open http://localhost:5000

For more, see the IoT Center README page.

Create the back end: A tale of two APIs

There will be two APIs at the core of the IoT center.

The InfluxDB API is served by the influxd process (at localhost:8086, by default). We need to write code to interact with it. The influxdb-client-js JavaScript library will handle much of the communication with the InfluxDB API.

The IoT Center API will provide functionality for managing devices. We will write the IoT Center API ourselves. To do this we'll use Node.js and Express.js to provide HTTP paths for registering devices and other functions.

In the following sections, we will create the server that interacts with these two back end APIs. In addition to helping understand the architecture of the IoT Center’s interactions with data and devices, coding these APIs will help us understood the UI, and how we might extend it.

Requirements

Before continuing, be sure to have the following installed:

Set up and configure InfluxDB

In order to communicate with InfluxDB, the IoT Center application needs some basic information. The application must then be able to enforce the state of the InfluxDB configuration so that devices can reliably send API requests.

The InfluxDB specific parts of the IoT Center’s own API are in app/server/influxdb/. These contain functions for the onboarding process, authorizations, and organizations. We’ll place InfluxDB API functionality in this influxdb/ directory as we go.

The code we’ll look at below contains helper functions for configuration that will be used in the Express routes, which serve and handle the paths of the HTTP API.

As we build the server, we’ll link to locations in final code so you can see how it all comes together.

Create the project

To begin, create a new folder in your home directory:

mkdir $HOME/iot-center-app/

Within iot-center-app/, create a directory for our back-end code:

cd iot-center-app/
mkdir server

Configure InfluxDB with environment variables

We configure the application’s communication with InfluxDB by environment variables.

Open a text editor and save the following code as env.js in the iot-center-app/app/server/ directory. When the server starts, the following code will run to retrieve environment variables:

const INFLUX_URL = process.env.INFLUX_URL || 'http://localhost:8086'
const INFLUX_TOKEN = process.env.INFLUX_TOKEN || 'my-token'
const INFLUX_ORG = process.env.INFLUX_ORG || 'my-org'
const INFLUX_BUCKET = 'iot_center'

// Defaults when on boarding a fresh new InfluxDB instance
const onboarding_username = 'my-user' // InfluxDB user
const onboarding_password = 'my-password' // InfluxDB password

// recommended interval for clients to refresh configuration in seconds
const configuration_refresh = 3600

Let’s take these environment variables in turn:

The application server will us the INFLUX_TOKEN value to authenticate its requests to the InfluxDB API. This should be an all-access token. This is the server’s “master token” (so to speak) and should only be used by the IoT Center server.
INFLUX_URL tells the application where the InfluxDB server is located. By default, the server will look for a local installation of InfluxDB at http://localhost:8086.¹
INFLUX_ORG is the name of the organization for the IoT Center to use.
INFLUX_BUCKET is the name of the bucket to which devices will write data.

With the configuration data now available, let’s add some basic logging to display them when the server starts:

// --snip--

function logEnvironment() {
  console.log(`INFLUX_URL=${INFLUX_URL}`)
  console.log(`INFLUX_TOKEN=${INFLUX_TOKEN ? '***' : ''}`)
  console.log(`INFLUX_ORG=${INFLUX_ORG}`)
  console.log(`INFLUX_BUCKET=${INFLUX_BUCKET}`)
}

Finally we need to export the variables so that they can be used elsewhere:

// --snip--

module.exports = {
  INFLUX_URL,
  INFLUX_TOKEN,
  INFLUX_ORG,
  onboarding_username,
  onboarding_password,
  configuration_refresh,
  INFLUX_BUCKET,
  logEnvironment,
}

Create server entrypoint file

From the server directory, let’s now create a file call index.js. This file will be responsible for starting the server.

We should now have a directory structure like this:

.
└── app
    └── server
        ├── env.js
        └── index.js

First, we will import INFLUX_URL from the environment variables we defined in env.js We will also import the Express web framework we'll be using to run the server. Paste the following into server/index.js:

const {logEnvironment, INFLUX_URL} = require('./env')
const express = require('express')
const proxy = require('express-http-proxy')

// --snip--

async function startApplication() {
  const app = express()

  // start proxy to InfluxDB to avoid CORS blocking with InfluXDB OSS v2 Beta
  app.use('/influx', proxy(INFLUX_URL))
  console.log(`Enable proxy from /influx/* to ${INFLUX_URL}/*`)

  // start HTTP server
  const port = process.env.PORT || 5000
  app.listen(port, process.env.HOSTNAME || '0.0.0.0')

  logEnvironment()
  console.log(`Listening on http://localhost:${port}`)
}

// --snip--

startApplication().catch((e) => {
  console.error('Failed to start', e)
})

Install Node dependencies and test run the server

Install the Node dependencies. From server/ directory, do:

yarn add express express-http-proxy

Let's test what we've written so far:

cd iot-center-app/app/server/
node index.js

You should see this output:

$ node index.js
Enable proxy from /influx/* to http://localhost:8086/*
INFLUX_URL=http://localhost:8086
INFLUX_TOKEN=***
INFLUX_ORG=my-org
INFLUX_BUCKET=iot_center
Listening on http://localhost:5000

Later, you might want to pass a URL for InfluxDB Cloud or another address on the local network.

InfluxDB API: Onboarding

We use the InfluxDB API via the InfluxDB Javascript client library.

Organize InfluxDB functionality in `app/server/influxdb/`

When our IoT Center server starts, it should try to connect to InfluxDB. We want to use the InfluxDB API to check that the database is connected, and that the target bucket is present and accessible. If IoT Center doesn’t find those things, it should set up InfluxDB and create the necessary buckets.

Create an onboarding.js file at server/influxdb/onboarding.js.

mkdir influxdb
touch influxdb/onboarding.js

We will place much more code in this directory going forward. For now, we focus on onboarding.

Onboard InfluxDB

First we import the InfluxDB setup API from the client library client. Then, import the configuration variables from env.js.

const {InfluxDB, HttpError} = require('@influxdata/influxdb-client')
const {SetupAPI} = require('@influxdata/influxdb-client-apis')
const {
  INFLUX_URL,
  INFLUX_TOKEN,
  INFLUX_ORG,
  onboarding_username,
  onboarding_password,
  INFLUX_BUCKET,
} = require('../env')

Create the onboardInfluxDB function in onboarding.js. Paste the following:

async function onboardInfluxDB() {
  const url = INFLUX_URL
  const setupApi = new SetupAPI(new InfluxDB({url}))
}

We need the URL of InfluxDB, so we can talk to its API server. In this line we create a SetupAPI object.

We use getSetup() to check whether InfluxDB has been onboarded, and the postSetup() method to do the setup (by POSTing a request to the InfluxDB API).

async function onboardInfluxDB() {
    const url = INFLUX_URL
    const setupApi = new SetupAPI(new InfluxDB({url}))
    const {allowed} = await setupApi.getSetup()
    if (allowed) {
        await setupApi.postSetup({
            body: {
                org: INFLUX_ORG,
                bucket: INFLUX_BUCKET,
                username: onboarding_username,
                password: onboarding_password,
                token: INFLUX_TOKEN,
            },
        })
        console.log(`InfluxDB has been onboarded.`)
    }
    console.log(`InfluxDB was not onboarded.`)
}

The complete onboarding function

Here is the full onboardInfluxDB function:

async function onboardInfluxDB() {
  const url = INFLUX_URL
  const setupApi = new SetupAPI(new InfluxDB({url}))
  try {
    const {allowed} = await setupApi.getSetup()
    if (allowed) {
      await setupApi.postSetup({
        body: {
          org: INFLUX_ORG,
          bucket: INFLUX_BUCKET,
          username: onboarding_username,
          password: onboarding_password,
          token: INFLUX_TOKEN,
        },
      })
      console.log(`InfluxDB has been onboarded.`)
    }
    let bucketNotFound = false
    try {
      await getBucket(INFLUX_BUCKET)
      console.log(`Bucket '${INFLUX_BUCKET}' exists.`)
    } catch (e) {
      if (e instanceof HttpError && e.statusCode === 401) {
        console.error(
          `Unauthorized to determine whether a bucket '${INFLUX_BUCKET}' exists.`
        )
      } else if (e instanceof HttpError && e.statusCode === 404) {
        // bucket not found
        bucketNotFound = true
      } else {
        console.error(
          `Unable to check whether a bucket '${INFLUX_BUCKET}' exists.`,
          e
        )
      }
    }
    if (bucketNotFound) {
      await createBucket(INFLUX_BUCKET)
      console.log(`Bucket ${INFLUX_BUCKET} created.`)
    }
  } catch (error) {
    console.error(
      `Unable to determine whether InfluxDB at ${INFLUX_URL} is onboarded.`,
      error
    )
  }
}

Bucket creation

This snippet creates a bucket if bucketNotFound is true:

if (bucketNotFound) {
  await createBucket(INFLUX_BUCKET)
  console.log(`Bucket ${INFLUX_BUCKET} created.`)
}

createBucket() is another function that we will define in the server/influxdb/ directory.

Conclusion

Next, we start creating our API for managing devices.

Register and authorize devices

Before the IoT Center allows a device to communicate with its server or InfluxDB, it must validate the device’s configuration and distribute credentials in the form of a token.

In the context of the IoT Center, a device being registered is meant to be coupled with the permissions model of InfluxDB. A device is registered if it is in the database and has been issued credentials.

InfluxDB 2 requires authentication by default. Each request to the API must provide a properly-scoped token. For instance, a write request must provide a token that has write permissions to the particular bucket.

The IoT Center automatically creates an authorization token for every device using the InfluxDB authorizations API. A device, identified by a unique deviceId, uses HTTP GET to receive a InfluxDB URL, organization, bucket, and token. The device uses this information to start writing data to InfluxDB.

The code used by IoT Center to interact with the InfluxDB authorization API is in app/server/influxdb/authorizations.js.

Importing the InfluxDB authorizations API

The file that defines the authorization-related parts of the IoT Center API begins with several import statements. The first import brings in the InfluxDB authorizations API.

const {AuthorizationsAPI} = require('@influxdata/influxdb-client-apis')
const influxdb = require('./influxdb')
const {getOrganization} = require('./organizations')
const {getBucket} = require('./buckets')
const {INFLUX_BUCKET} = require('../env')

const authorizationsAPI = new AuthorizationsAPI(influxdb)
const DESC_PREFIX = 'IoT Center: '
const CREATE_BUCKET_SPECIFIC_AUTHORIZATIONS = false