Adding data to a CouchDB from Arduino.

Sometimes when using the Arduino to collect data, I need to store this data somewhere. Usually I just send the data over the usb/serial link and then have a python script running on my computer that collects the data and stores it in a database.
But now that I have a ethernet shield for my arduino, I figured I would try to remove one step from that equation.

Some time ago, as part of my work, I looked at different NoSQL databases, and one database caught my eye. Even though it did not fit the problem I had then, CouchDB still intrigued me. I especially liked the cached views and build in map/reduce functionality.

Unlike most other databases, who uses proprietary binary protocols, CouchDB has a simple HTTP RESTful API. This makes it easy to talk to from the arduino.

Lets get started, first we need to initialize the ethernet board:

#include <WProgram.h>
#include <wiring.h>
#include <HardwareSerial.h>
#include <SPI.h>
#include <Ethernet.h>
#include <utility/socket.h>
#include <avr/eeprom.h>

static const uint8_t g_gateway[4] = {0, 0, 0, 0};
static const uint8_t g_subnet[4] = {255, 255, 255, 0};
static const uint8_t g_ip[4] = {192, 168, 1, 170}; // change me!
static const uint8_t g_mac[6] = {0x90, 0xa2, 0xda, 0x00, 0x25, 0x65};

void setup()
{
  Serial.begin(115200*8);
  Serial.println("GO");

  W5100.init();
  W5100.setMACAddress((uint8_t*)g_mac);
  W5100.setIPAddress((uint8_t*)g_ip);
  W5100.setGatewayIp((uint8_t*)g_gateway);
  W5100.setSubnetMask((uint8_t*)g_subnet);
}

I have chosen to stay away from the higher level ethernet api (Server/Client ...), and instead use the lower level W5100 and socket api. This is mainly to avoid the blocking nature of the Client class.


Unfortunately the WIZnet controller does not know its own MAC address, so we have to hardcode it into the source. You can use almost any made up MAC address, but if you have multiple ethernet shields, you must make sure all used MAC addresses are unique. If you look at the backside of the ethernet shield PCB, you will find that the Arduino people have been nice enough to print a unique MAC address on the board.

You will also need to change the ip address, at line 11, to one that matches your local network.

If you have multiple arduinos and ethernet shields, you might want to program a unique IP and MAC address into the arduinos eeprom. On my boards I use the bytes from 0x3f0. Here is a example how how to read the eeprom:

void setup()
{
  Serial.begin(115200*8);
  Serial.println("GO");

  uint8_t ip_mac[4+6];
  eeprom_read_block(ip_mac, (const void*)0x3f0, sizeof(ip_mac));
  if(ip_mac[0] == 255)
  {
    Serial.println("PANIC: missing IP address");
    for(;;) /**/ ;
  }

  W5100.init();
  W5100.setMACAddress(ip_mac+4);
  W5100.setIPAddress(ip_mac);
  W5100.setGatewayIp((uint8_t*)g_gateway);
  W5100.setSubnetMask((uint8_t*)g_subnet);
}

Next, we have the main loop. This example will read some analog and digital ports and then connect to the CouchDB to store the result in the database called "test1".

#define FD 0
#define DB_NAME "test1"
 
static const uint8_t g_couchAddr[4] = {192, 168, 1, 1};
static const uint16_t g_couchPort = 5984;
 
void loop()
{
  enum {STATE_IDLE, STATE_CONNECTING, STATE_CLOSE_WAIT} netstate = STATE_IDLE;
 
  uint32_t nextsampleat = millis();
  uint16_t lasta0;
  uint16_t lasta1;
  uint8_t lastd2;
  bool hassample = false;
 
  for(;;)
  {
    uint32_t now = millis();
    if(int32_t(now-nextsampleat)>=0)
    {
      Serial.println("sampling value");
      lasta0 = analogRead(0);
      lasta1 = analogRead(1);
      lastd2 = digitalRead(2);
      hassample = true;
      nextsampleat += 10*1000; // 10 secs;
    }
 
    uint8_t sockstatus = W5100.readSnSR(FD);
    switch(netstate)
    {
      case STATE_IDLE:
        if(hassample)
        {
          Serial.println("connecting");
          socket(FD, SnMR::TCP, 1100, 0);
          connect(FD, (uint8_t*)g_couchAddr, g_couchPort);
          netstate = STATE_CONNECTING;
        }
        break;
 
      case STATE_CONNECTING:
        if(sockstatus == SnSR::ESTABLISHED)
        {
          Serial.println("connected, sending doc");
 
          char doc[64];
          unsigned doclen = snprintf_P(doc, sizeof(doc),
            PSTR("{\"a0\":%u, \"a1\":%u, \"d2\":%u}"), lasta0, lasta1, lastd2);
 
          char header[64];
          unsigned headerlen = snprintf_P(header, sizeof(header),
            PSTR("POST /" DB_NAME "/ HTTP/1.0\r\n"));
          send(FD, (const uint8_t*)header, headerlen);
          headerlen = snprintf_P(header, sizeof(header),
            PSTR("Content-Type: application/json\r\n"));
          send(FD, (const uint8_t*)header, headerlen);
          headerlen = snprintf_P(header, sizeof(header),
            PSTR("Content-Length: %u\r\n\r\n"), doclen);
          send(FD, (const uint8_t*)header, headerlen);
 
          send(FD, (const uint8_t*)doc, doclen);
 
          netstate = STATE_CLOSE_WAIT;
        }
        else if(sockstatus == SnSR::CLOSED)
        {
          Serial.println("conection failed");
          hassample = false;
          netstate = STATE_IDLE;
        }
        break;
 
      case STATE_CLOSE_WAIT:
        if(sockstatus == SnSR::CLOSE_WAIT || sockstatus == SnSR::CLOSED)
        {
          Serial.println("conection closed");
          // ignoring http reply since we can't deal with db errors anyway.
          close(FD);
          hassample = false;
          netstate = STATE_IDLE;
        }
        break;
    }
  }
}

Here is what the code does:

• line 29+30: the address and port of the CouchDB. You most likely need to change this to match your computers address.
• line 44-53: every 10 seconds some hardware ports are read, and the result stored away to later transmission to the db.
• line 55-110: the async network state machine.
• line 59-65: when a new sample is ready, a connection to the database is opened.
• line 92-97: if we fail to connect to the database, the sample is thrown away, and the connection will be retried when a new sample is ready.
• line 73-75: the CouchDB document is created.
• line 77-86: creation and sending of the HTTP header.
• line 88: here the CouchDB document is sent to the server.
• line 92-109: the connection is closed, and we wait for acknowledgment.

When running the code, CouchDB will start to contain documents much like this:

{
  "_id": "135cfbc9bc4709ba24e4d84b5006ae91",
  "_rev": "1-43e825eaf711e9c7b4aaf4274813d677",
  "a0": 484,
  "a1": 1023,
  "d2": 0
}

Looks nice and all, but for my project I also need to know when each sample were made. One option is to add a DS1307 real time clock, but since I'm more of a software guy, I choose to let CouchDB add the timestamp.

To insert a timestamp into the document, we use a small javascript snippet called a update hander. A update handler can also add a new document to the database, so we can add and update the document in one database call.

This is the update handler used for my samples, it will create a new document, add a posix timestamp and the 3 samples.

{
  "updates": {
    "new": "function(doc, req) {
      return [{
        _id:req.uuid, 
        time:(new Date()).getTime()/1000.0, 
        a0:Number(req.form.a0), a1:Number(req.form.a1), d2:Number(req.form.d2)
      }, \"updated\"];
    }"
  }
}

To upload this design document to the database, save it as test1_design.js and use curl:

curl -X PUT http://duff:5984/test1/_design/test -d @test1_design.js

The arduino code must be change slightly to use this new update handler.

Since we are no longer adding the document directly to the database, but calling the update handler, the post URL must be changed to:

#define DB_NAME "/test1/_design/test/_update/new"

We also need to change the HTTP post data from a JSON document to HTTP post arguments:

unsigned doclen = snprintf_P(doc, sizeof(doc), 
  PSTR("a0=%u;a1=%u;d2=%u"), lasta0, lasta1, lastd2);

And at last the HTTP mime type needs to be changed:

headerlen = snprintf_P(header, sizeof(header), 
  PSTR("Content-Type: application/x-www-form-urlencoded\r\n"));

After running this new version, all new documents will have a timestamp:

{
  "_id": "135cfbc9bc4709ba24e4d84b500ac935",
  "_rev": "1-9d17bbf14ccaac2435deaba4bc1b9ead",
  "time": 1296423519.513,
  "a0": 549,
  "a1": 1023,
  "d2": 0
}

All we need now is a CouchApp to render graphs of the collected data, but that is beyond my current javascript capabilities :)