tag:blogger.com,1999:blog-45064087529134828012024-03-06T08:48:08.077+01:00BITheapAnonymoushttp://www.blogger.com/profile/14345112203509983206noreply@blogger.comBlogger13125tag:blogger.com,1999:blog-4506408752913482801.post-76645475193105554792012-05-19T21:21:00.000+02:002012-05-19T21:24:25.333+02:00Rant: #pragma(pack) evilnessWasting time on stuff like can ruin a otherwise great day:<br />
<br />
<pre class="brush:[cpp];">#define WIN32_LEAN_AND_MEAN
#include <winsock2.h>
#include <windows.h>
#include <stdio.h>
int main()
{
printf("size: %d\n", (int)sizeof(JOBOBJECT_BASIC_LIMIT_INFORMATION));
return 0;
}
</pre>
<i>cl test.cpp && test.exe</i><br />
<b>size: 44</b> (wrong)<br />
<pre class="brush:[cpp];">#define WIN32_LEAN_AND_MEAN
#include <windows.h>
#include <winsock2.h>
#include <stdio.h>
int main()
{
printf("size: %d\n", (int)sizeof(JOBOBJECT_BASIC_LIMIT_INFORMATION));
return 0;
}
</pre>
<i>cl test.cpp && test.exe</i><br />
<b>size: 48</b> (right)<br />
<br />
I have just spent a few hours trying to figure out why SetInformationJobObject() kept failing in one of my projects.<br />
And when it's because Microsoft can't decide if LARGE_INTEGER should be aligned to 4 or 8 bytes, I can't help but cursing their name.<br />
<br />
The problem apears to be that winsock2.h does a "pragma pack(4)" before including windows.h, this will mess up some of the types defined in windows.h, including JOBOBJECT_BASIC_LIMIT_INFORMATION.<br />
<br />
This was tested with VC2010 and the v7.0A sdk.<br />
<br />Anonymoushttp://www.blogger.com/profile/14345112203509983206noreply@blogger.com1tag:blogger.com,1999:blog-4506408752913482801.post-66099532269811925802012-03-04T22:51:00.000+01:002012-03-07T17:41:46.697+01:00FT232R BitBang mode is broken.<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaF3tj4ptB_R8tHZPOLX6XqcDdU2I97XY8_WcM2ZQ2GVrAJJU-AaFnCnYlil2dv1431xoFbO8WgVJ-rLSzFnuYrjSliQlCgB-CgLZ0g67VUYc6UHONTdVSor6-rkchDwO447ngrjW9E4Q/s1600/ft232rl.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><br /><img border="0" height="198" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaF3tj4ptB_R8tHZPOLX6XqcDdU2I97XY8_WcM2ZQ2GVrAJJU-AaFnCnYlil2dv1431xoFbO8WgVJ-rLSzFnuYrjSliQlCgB-CgLZ0g67VUYc6UHONTdVSor6-rkchDwO447ngrjW9E4Q/s320/ft232rl.jpg" width="320" /></a></div>
While trying to use a FTDI RT232RL in synchronous bitbang mode, I (like many others apparently) discovered that this feature is completely broken in most chips currently on the market.<br />
<br />
The problem is described in the <a href="http://www.ftdichip.com/Support/Documents/TechnicalNotes/TN_120_FT232R%20Errata%20Technical%20Note.pdf">errata</a> but not anywhere in the <a href="http://www.ftdichip.com/Support/Documents/DataSheets/ICs/DS_FT232R.pdf">datasheet</a> or <a href="http://www.ftdichip.com/Support/Documents/AppNotes/AN_232R-01_Bit_Bang_Mode_Available_For_FT232R_and_Ft245R.pdf">appnode</a> describing the bitbang modes.<br />
<br />
It annoys me to no end that the bug is not described in the bitbang appnote, it would have saved me hours of frustration, and could possible save other people quite a bit of wasted time too.<br />
<br />
When I was trying to find out what was wrong with my project, I ended up with this small repo case, that should have resulted in a 500 Hz square wave signal.<br />
<br />
<pre class="brush:[cpp];">#include <stdio.h>
#include <string.h>
#include <ftdi.h>
int main()
{
ftdi_context ftdic;
ftdi_init(&ftdic);
if(ftdi_usb_open_desc(&ftdic, 0x0403, 0x6001, NULL, NULL) < 0)
{
fprintf(stderr, "ftdi_usb_open_desc failed: %s\n",
ftdi_get_error_string(&ftdic));
exit(1);
}
if(ftdi_set_baudrate(&ftdic, 1000) < 0)
{
fprintf(stderr, "ftdi_set_baudrate failed: %s\n",
ftdi_get_error_string(&ftdic));
exit(1);
}
if(ftdi_set_bitmode(&ftdic, 0x01, BITMODE_SYNCBB) < 0)
{
fprintf(stderr, "ftdi_set_bitmode failed: %s\n",
ftdi_get_error_string(&ftdic));
exit(1);
}
uint8_t data[256];
for(int i=0; i<sizeof(data); i++)
data[i] = i&1;
for(;;)
{
ftdi_write_data(&ftdic, data, sizeof(data));
uint8_t data2[256];
ftdi_read_data(&ftdic, data2, sizeof(data2));
}
return 0;
}
</pre>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaphY2EVv4DGMUtc1eybOSlLA_VbEFCwIUflghKFSMR1oTlR6h6aGqxQyR4iu2DOtWm38AROCQbMlCROD2OkSIPaHce4SWpJD7XjeHqZJ8CSGg5lNDVBAN6Ql8Fk5P_41JSE6giZWhwbU/s1600/test2.gif" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaphY2EVv4DGMUtc1eybOSlLA_VbEFCwIUflghKFSMR1oTlR6h6aGqxQyR4iu2DOtWm38AROCQbMlCROD2OkSIPaHce4SWpJD7XjeHqZJ8CSGg5lNDVBAN6Ql8Fk5P_41JSE6giZWhwbU/s1600/test2.gif" /></a><br />
But when hooking up a scope to the tx line of the FT232, it's easy to see that the output is a complete mess. The pulse widths are all over the place, from 60<span style="background-color: white; font-family: sans-serif; font-size: 13px; line-height: 20px;">μ</span>s to 1ms.<br />
<div style="clear: both;">
<br />
In the errata, it's suggested that the fix is to run with a baudrate of 3000000, and just generate longer pulses to compensate. This might sound like a usable workaround, but FTDI neglected to tell that the chip is only running with 12.5MBit/s usb speed. Far from the needed ~65MBit/s needed for 3000000 baud (3000000baud * 10bits/byte (with bit stuffing) * 2 (we need to both send and recv) * 10% (usb header overhead).<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_HBInfhpKGFfr6YK6zMop5UPUn5qDWKWLMVf5XK_A9Y4UbKJo_V63r4l57jHCu0Ke0l6ON7JlAO3jPtnHpd5gfdgFH8Epr-BmaXXshohWUK6zbiEl4_Wc9S_T29gBc7IAj7EYrlvrd9Q/s1600/test2.gif" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_HBInfhpKGFfr6YK6zMop5UPUn5qDWKWLMVf5XK_A9Y4UbKJo_V63r4l57jHCu0Ke0l6ON7JlAO3jPtnHpd5gfdgFH8Epr-BmaXXshohWUK6zbiEl4_Wc9S_T29gBc7IAj7EYrlvrd9Q/s1600/test2.gif" /></a></div>
Running the chip at 3000000 baud (changing 1000 to 3000000 in the ftdi_set_baudrate call), does not work either.<br />
Not only do we not have enough usb bandwitdh (can be seen as the spratic blocks of data in the top part), but we only get 0.5<span style="background-color: white; font-family: sans-serif; font-size: 13px; line-height: 20px;">μ</span>s long pulses. Not 0.33<span style="background-color: white; font-family: sans-serif; font-size: 13px; line-height: 20px;">μ</span>s as expected.<br />
The pulse widths are better than at lower speeds, but still not stable.<br />
<br />
We can only hope that vendors will soon sell out of there devices using the A and B revisions of these chips, so that we can get some breakout boards with revision C, that supposedly will work as advertised.<br />
<br />
<br /></div>Anonymoushttp://www.blogger.com/profile/14345112203509983206noreply@blogger.com7tag:blogger.com,1999:blog-4506408752913482801.post-90320867328730505262011-10-30T22:42:00.000+01:002011-10-30T22:42:55.020+01:00Scary beautiful.<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5Jea-0lmJ6VPVvCWHDNYuu6OT1DTkpf6W7SBbLqXo9vTquswb5EFr2diDwcB1Z60iVbmyAfeSIuvROpPh8KNlhbHEHIYERj3Vl9BMZV1COUO_ObCiGryjZi3eKjfUm9Lb6I5uhn1To4Q/s1600/includes.old.pngcrush.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="242" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5Jea-0lmJ6VPVvCWHDNYuu6OT1DTkpf6W7SBbLqXo9vTquswb5EFr2diDwcB1Z60iVbmyAfeSIuvROpPh8KNlhbHEHIYERj3Vl9BMZV1COUO_ObCiGryjZi3eKjfUm9Lb6I5uhn1To4Q/s320/includes.old.pngcrush.png" width="320" /></a></div>
This is a graph of include statements in a project I'm currently working on.<br />
I generated this because I'm trying to reduce the time it takes to compile, and I like the look of the <a href="http://www.graphviz.org/">dot</a> produced graph, so I wanted to share it.<br />
<br />
The small squares are source files, and there brightness indicates how many headers they include.<br />
<br />
The circles are header files, and there brightness tells how often they are included during a clean build.<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkC7Z9RPejWuYZTchyphenhyphenCPetP4h_pR9QSVbz590L8wXZvtevkZJ7n7aGyqysVwjzHTR8x-eTY6brCpOe_PK7oBtMRsjatRgeqTOcOAgsl2PtUqZCHnQ2LwMgjBxIHvT4yeoCODbtYxcRKz8/s1600/includes.pngcrush.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="249" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkC7Z9RPejWuYZTchyphenhyphenCPetP4h_pR9QSVbz590L8wXZvtevkZJ7n7aGyqysVwjzHTR8x-eTY6brCpOe_PK7oBtMRsjatRgeqTOcOAgsl2PtUqZCHnQ2LwMgjBxIHvT4yeoCODbtYxcRKz8/s320/includes.pngcrush.png" width="320" /></a></div>
I'm currently playing with <a href="http://code.google.com/p/include-what-you-use/">include-what-you-use</a> to clean up the code a bit, and after running it on parts of the code tree, it reduced the build time with almost 30%. Not bad!Anonymoushttp://www.blogger.com/profile/14345112203509983206noreply@blogger.com0tag:blogger.com,1999:blog-4506408752913482801.post-68114806163340665172011-03-02T23:58:00.000+01:002012-03-03T13:52:27.011+01:00Norge NGE101 and Google PowerMeter scripts.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgE6v8Q5YsqlZEVKwl-6YOPXoTU4clzUhvJij5NJo_fhFnBi637ztWv54ivjJPe3-quroXSTSG_2g-jISjNV9zdfYW8oTax7x_z3tdc1VIAtVPhuOzkqJUYFyGxrvZI4YSgvGoAUsfgQDw/s1600/Google-PowerMeter.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgE6v8Q5YsqlZEVKwl-6YOPXoTU4clzUhvJij5NJo_fhFnBi637ztWv54ivjJPe3-quroXSTSG_2g-jISjNV9zdfYW8oTax7x_z3tdc1VIAtVPhuOzkqJUYFyGxrvZI4YSgvGoAUsfgQDw/s1600/Google-PowerMeter.jpg" /></a></div>
I just uploaded the first (rough) version of the arduino code and scripts that I use for tracking my personal power usage.<br />
<br />
The code can be found on <a href="https://github.com/jhansen/arduino_ask_decoders">github</a>.<br />
<br />
I don't expect anybody else to be able to use the code as is yet, but it should be a good starting point.<br />
<br />
The code has only been tested on Linux (Ubuntu), and it will most likely require some changes before it's possible to build and run on Windows.Anonymoushttp://www.blogger.com/profile/14345112203509983206noreply@blogger.com2tag:blogger.com,1999:blog-4506408752913482801.post-56816225390233886512011-02-14T00:48:00.000+01:002012-03-03T13:51:37.944+01:00Kenwood TH-D72<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWitA3AT_G7BisaHO3PwTUnj46kKp2TDccKYZ2kwaH2Z6DGz22ByUF9reYdGRvYjQl-lZLPwg4vz2Li_f0r6vtmHlZKt85Xrdfs7muaYxjp_DTu1jU3T99KwbZNkmWq3yOvcPt6a9em0M/s1600/kenwood_th-d72.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWitA3AT_G7BisaHO3PwTUnj46kKp2TDccKYZ2kwaH2Z6DGz22ByUF9reYdGRvYjQl-lZLPwg4vz2Li_f0r6vtmHlZKt85Xrdfs7muaYxjp_DTu1jU3T99KwbZNkmWq3yOvcPt6a9em0M/s400/kenwood_th-d72.jpg" width="207" /></a></div>
<br />
To celebrate that I got my HAM license back, I bought myself a new toy, a dual band 2m and 70cm Kenwood TH-D72.<br />
<br />
I have not been a active HAM operator for almost 20 years, but besides the complexity of the toys :) not much seem to have changed.<br />
<br />
Back then, I was playing around with ax.25 <a href="http://en.wikipedia.org/wiki/Packet_radio">packet radio</a> on 2m running at a whopping 1200 baud. Today, we are still using 1200 baud, but the Terminal Node Controller are now build into some high end handsets.<br />
<br />
Here in Denmark, there are not much old-school packet radio activity, but here in Copenhagen, I can hear lots of <a href="http://en.wikipedia.org/wiki/Automatic_Packet_Reporting_System">APRS</a> traffic. APRS is build on top of the old ax.25 protocol, and is mostly used to report information about HAM's and there gadgets. All the APRS traffic is bounced around the network a bit, and the packages that manage to hit a internet gateway is then collected for anybody to study. One popular site that visualizes this traffic is <a href="http://aprs.fi/?addr=vallensb%C3%A6k,%20denmark&mt=roadmap&z=11&timerange=86400">APRS.fi</a>.<br />
<br />
In an age where we all are carrying internet capable phones, it might seem strange to be fascinated by something as silly as 1200 baud data connections with limited range, but I'm enthralled but the openness of most of it.<br />
<br />
One thing that is not open, at all, is the radios we can buy, and even though we are allowed to build and use our own equipment, its hard to beat the size and features of the mass produces handsets that exists.<br />
<br />
This is also true for the D72, it has a USB interface that's used for firmware updates, GPS positions, packet radio TNC and configuring the radio, but unfortunately Kenwood have not documented most of the capabilities of the USB interface. My messing around a bit, I managed to find some commands, but I'm sure there are many more.<br />
<br />
I hope to find some time in the near future to see if I can't wrestle the firmware out of the D72 and take a closer look at what makes it tick, and discover some more commands, so that I can control the radio from linux.Anonymoushttp://www.blogger.com/profile/14345112203509983206noreply@blogger.com1tag:blogger.com,1999:blog-4506408752913482801.post-18872559453964563212011-01-31T00:13:00.000+01:002011-10-30T21:43:12.415+01:00Adding data to a CouchDB from Arduino.Sometimes when using the Arduino to collect data, I need to store this data somewhere. Usually I just send the data over the usb/serial link and then have a python script running on my computer that collects the data and stores it in a database.<br />
But now that I have a <a href="http://www.arduino.cc/en/Main/ArduinoEthernetShield">ethernet shield</a> for my arduino, I figured I would try to remove one step from that equation.<br />
<br />
Some time ago, as part of my work, I looked at different NoSQL databases, and one database caught my eye. Even though it did not fit the problem I had then, <a href="http://couchdb.apache.org/">CouchDB</a> still intrigued me. I especially liked the cached views and build in map/reduce functionality.<br />
<br />
Unlike most other databases, who uses proprietary binary protocols, CouchDB has a simple HTTP RESTful API. This makes it easy to talk to from the arduino.<br />
<br />
Lets get started, first we need to initialize the ethernet board:<br />
<br />
<pre class="brush:cpp">#include <WProgram.h>
#include <wiring.h>
#include <HardwareSerial.h>
#include <SPI.h>
#include <Ethernet.h>
#include <utility/socket.h>
#include <avr/eeprom.h>
static const uint8_t g_gateway[4] = {0, 0, 0, 0};
static const uint8_t g_subnet[4] = {255, 255, 255, 0};
static const uint8_t g_ip[4] = {192, 168, 1, 170}; // change me!
static const uint8_t g_mac[6] = {0x90, 0xa2, 0xda, 0x00, 0x25, 0x65};
void setup()
{
Serial.begin(115200*8);
Serial.println("GO");
W5100.init();
W5100.setMACAddress((uint8_t*)g_mac);
W5100.setIPAddress((uint8_t*)g_ip);
W5100.setGatewayIp((uint8_t*)g_gateway);
W5100.setSubnetMask((uint8_t*)g_subnet);
}
</pre>
<br />
I have chosen to stay away from the higher level ethernet api (Server/Client ...), and instead use the lower level W5100 and socket api. This is mainly to avoid the blocking nature of the <i>Client</i> class.<br />
<br />
<br />
Unfortunately the WIZnet controller does not know its own MAC address, so we have to hardcode it into the source. You can use almost any made up MAC address, but if you have multiple ethernet shields, you must make sure all used MAC addresses are unique. If you look at the backside of the ethernet shield PCB, you will find that the Arduino people have been nice enough to print a unique MAC address on the board.<br />
<br />
You will also need to change the ip address, at line 11, to one that matches your local network.<br />
<br />
If you have multiple arduinos and ethernet shields, you might want to program a unique IP and MAC address into the arduinos eeprom. On my boards I use the bytes from 0x3f0. Here is a example how how to read the eeprom:<br />
<pre class="brush:cpp;firstline:14">void setup()
{
Serial.begin(115200*8);
Serial.println("GO");
uint8_t ip_mac[4+6];
eeprom_read_block(ip_mac, (const void*)0x3f0, sizeof(ip_mac));
if(ip_mac[0] == 255)
{
Serial.println("PANIC: missing IP address");
for(;;) /**/ ;
}
W5100.init();
W5100.setMACAddress(ip_mac+4);
W5100.setIPAddress(ip_mac);
W5100.setGatewayIp((uint8_t*)g_gateway);
W5100.setSubnetMask((uint8_t*)g_subnet);
}
</pre>
<br />
<br />
Next, we have the main loop. This example will read some analog and digital ports and then connect to the CouchDB to store the result in the database called "test1".<br />
<br />
<pre class="brush:cpp;firstline:26">
#define FD 0
#define DB_NAME "test1"
static const uint8_t g_couchAddr[4] = {192, 168, 1, 1};
static const uint16_t g_couchPort = 5984;
void loop()
{
enum {STATE_IDLE, STATE_CONNECTING, STATE_CLOSE_WAIT} netstate = STATE_IDLE;
uint32_t nextsampleat = millis();
uint16_t lasta0;
uint16_t lasta1;
uint8_t lastd2;
bool hassample = false;
for(;;)
{
uint32_t now = millis();
if(int32_t(now-nextsampleat)>=0)
{
Serial.println("sampling value");
lasta0 = analogRead(0);
lasta1 = analogRead(1);
lastd2 = digitalRead(2);
hassample = true;
nextsampleat += 10*1000; // 10 secs;
}
uint8_t sockstatus = W5100.readSnSR(FD);
switch(netstate)
{
case STATE_IDLE:
if(hassample)
{
Serial.println("connecting");
socket(FD, SnMR::TCP, 1100, 0);
connect(FD, (uint8_t*)g_couchAddr, g_couchPort);
netstate = STATE_CONNECTING;
}
break;
case STATE_CONNECTING:
if(sockstatus == SnSR::ESTABLISHED)
{
Serial.println("connected, sending doc");
char doc[64];
unsigned doclen = snprintf_P(doc, sizeof(doc),
PSTR("{\"a0\":%u, \"a1\":%u, \"d2\":%u}"), lasta0, lasta1, lastd2);
char header[64];
unsigned headerlen = snprintf_P(header, sizeof(header),
PSTR("POST /" DB_NAME "/ HTTP/1.0\r\n"));
send(FD, (const uint8_t*)header, headerlen);
headerlen = snprintf_P(header, sizeof(header),
PSTR("Content-Type: application/json\r\n"));
send(FD, (const uint8_t*)header, headerlen);
headerlen = snprintf_P(header, sizeof(header),
PSTR("Content-Length: %u\r\n\r\n"), doclen);
send(FD, (const uint8_t*)header, headerlen);
send(FD, (const uint8_t*)doc, doclen);
netstate = STATE_CLOSE_WAIT;
}
else if(sockstatus == SnSR::CLOSED)
{
Serial.println("conection failed");
hassample = false;
netstate = STATE_IDLE;
}
break;
case STATE_CLOSE_WAIT:
if(sockstatus == SnSR::CLOSE_WAIT || sockstatus == SnSR::CLOSED)
{
Serial.println("conection closed");
// ignoring http reply since we can't deal with db errors anyway.
close(FD);
hassample = false;
netstate = STATE_IDLE;
}
break;
}
}
}
</pre>
<br />
Here is what the code does:<br />
<ul><br />
<li><i>line 29+30</i>: the address and port of the CouchDB. You most likely need to change this to match your computers address.</li>
<li><i>line 44-53</i>: every 10 seconds some hardware ports are read, and the result stored away to later transmission to the db.</li>
<li><i>line 55-110</i>: the async network state machine.</li>
<li><i>line 59-65</i>: when a new sample is ready, a connection to the database is opened.</li>
<li><i>line 92-97</i>: if we fail to connect to the database, the sample is thrown away, and the connection will be retried when a new sample is ready.</li>
<li><i>line 73-75</i>: the CouchDB document is created.</li>
<li><i>line 77-86</i>: creation and sending of the HTTP header.</li>
<li><i>line 88</i>: here the CouchDB document is sent to the server.</li>
<li><i>line 92-109</i>: the connection is closed, and we wait for acknowledgment.</li>
</ul>
<br />
<br />
When running the code, CouchDB will start to contain documents much like this:<br />
<pre class="brush:js">
{
"_id": "135cfbc9bc4709ba24e4d84b5006ae91",
"_rev": "1-43e825eaf711e9c7b4aaf4274813d677",
"a0": 484,
"a1": 1023,
"d2": 0
}
</pre>
<br />
Looks nice and all, but for my project I also need to know <i>when</i> each sample were made. One option is to add a <a href="http://www.maxim-ic.com/datasheet/index.mvp/id/2688">DS1307</a> real time clock, but since I'm more of a software guy, I choose to let CouchDB add the timestamp.<br />
<br />
To insert a timestamp into the document, we use a small javascript snippet called a <a href="http://wiki.apache.org/couchdb/Document_Update_Handlers">update hander</a>. A update handler can also add a new document to the database, so we can add and update the document in one database call.<br />
<br />
This is the update handler used for my samples, it will create a new document, add a posix timestamp and the 3 samples.<br />
<pre class="brush:js">
{
"updates": {
"new": "function(doc, req) {
return [{
_id:req.uuid,
time:(new Date()).getTime()/1000.0,
a0:Number(req.form.a0), a1:Number(req.form.a1), d2:Number(req.form.d2)
}, \"updated\"];
}"
}
}
</pre>
To upload this design document to the database, save it as test1_design.js and use curl:<br />
<pre>curl -X PUT http://duff:5984/test1/_design/test -d @test1_design.js</pre>
<br />
<br />
The arduino code must be change slightly to use this new update handler.<br />
<br />
Since we are no longer adding the document directly to the database, but calling the update handler, the post URL must be changed to:<br />
<pre class="brush:cpp;firstline:27">
#define DB_NAME "/test1/_design/test/_update/new"
</pre>
<br />
We also need to change the HTTP post data from a JSON document to HTTP post arguments:<br />
<pre class="brush:cpp;firstline:74">
unsigned doclen = snprintf_P(doc, sizeof(doc),
PSTR("a0=%u;a1=%u;d2=%u"), lasta0, lasta1, lastd2);
</pre><br />
<br />
And at last the HTTP mime type needs to be changed:<br />
<pre class="brush:cpp;firstline:81">
headerlen = snprintf_P(header, sizeof(header),
PSTR("Content-Type: application/x-www-form-urlencoded\r\n"));
</pre><br />
<br />
After running this new version, all new documents will have a timestamp:<br />
<pre class="brush:js">
{
"_id": "135cfbc9bc4709ba24e4d84b500ac935",
"_rev": "1-9d17bbf14ccaac2435deaba4bc1b9ead",
"time": 1296423519.513,
"a0": 549,
"a1": 1023,
"d2": 0
}
</pre>
<br />
<br />
All we need now is a <a href="http://couchapp.org/page/index">CouchApp</a> to render graphs of the collected data, but that is beyond my current javascript capabilities :)Anonymoushttp://www.blogger.com/profile/14345112203509983206noreply@blogger.com2tag:blogger.com,1999:blog-4506408752913482801.post-1742479757879988402011-01-25T23:18:00.000+01:002011-10-30T21:31:29.366+01:00NGE101 – Norgo wireless energy meter (part 6)In the previous posts I showed the bits in the captured frames in the order they were captured and decoded, ie. the first received bit were shown to the left.<br />
When I managed to decode the main payload in the frames, I learned that the frames are send with the least significant bit (lsb) first, so I was actually showing the captured frames in "reverse". This ended up being a bit confusing since binary number are normally shows with the msb at the left, and lsb at the right, so in this post I will show the frames reverse compared to the previous posts, so what looked like:<br />
<br />
<pre>preamble/header payload checksum
11111010100011001001 110010111110011111010000000000000000 110100111110010
11111010100011001001 000101111110011111010000000000000000 000111100001010
11111010000011001001 00100111111100000000 111000010110010
11111010000011001001 01100111111100000000 111001010010000</pre>
<br />
will now look like this:<br />
<br />
<pre>checksum payload header/preamble
010011111001011 000000000000000010111110011111010011 10010011000101011111
010100001111000 000000000000000010111110011111101000 10010011000101011111
010011010000111 00000000111111100100 10010011000001011111
000010010100111 00000000111111100110 10010011000001011111</pre>
<br />
The first thing I wanted to figure out, was how the checksum should be calculated, so I started by capturing and looking at thousands of frames to see if I could find a pattern.<br />
<br />
Here are a few frame pairs where just the lowest bit in the payload are different:<br />
<br />
<pre>11<strong><span style="color: red;">1</span></strong>010<strong><span style="color: red;">0</span></strong>010<strong><span style="color: red;">1</span></strong>0001 0000000011001000101<strong><span style="color: red;">0</span></strong> 10010011000001011111
11<strong><span style="color: red;">0</span></strong>010<strong><span style="color: red;">1</span></strong>010<strong><span style="color: red;">0</span></strong>0001 0000000011001000101<strong><span style="color: red;">1</span></strong> 10010011000001011111
11<strong><span style="color: red;">0</span></strong>000<strong><span style="color: red;">0</span></strong>110<strong><span style="color: red;">0</span></strong>1000 0000000011010001001<strong><span style="color: red;">0</span></strong> 10010011000001011111
11<strong><span style="color: red;">1</span></strong>000<strong><span style="color: red;">1</span></strong>110<strong><span style="color: red;">1</span></strong>1000 0000000011010001001<strong><span style="color: red;">1</span></strong> 10010011000001011111
01<strong><span style="color: red;">0</span></strong>010<strong><span style="color: red;">0</span></strong>100<strong><span style="color: red;">0</span></strong>1000 0000000011010001011<strong><span style="color: red;">0</span></strong> 10010011000001011111
01<strong><span style="color: red;">1</span></strong>010<strong><span style="color: red;">1</span></strong>100<strong><span style="color: red;">1</span></strong>1000 0000000011010001011<strong><span style="color: red;">1</span></strong> 10010011000001011111</pre>
<br />
If you look closely at the checksum, you might notice that the xor of the two checksums in each pair is the same value: <em>001000100010000</em>.<br />
<br />
More frames with just one flipped bit in the payload:<br />
<br />
<pre> 1<strong><span style="color: red;">1</span></strong>111<strong><span style="color: red;">1</span></strong>101<strong><span style="color: red;">0</span></strong>10010 000000001111101110<strong><span style="color: red;">0</span></strong>1 10010011000001011111
^ 1<strong><span style="color: red;">0</span></strong>111<strong><span style="color: red;">0</span></strong>101<strong><span style="color: red;">1</span></strong>10010 000000001111101110<strong><span style="color: red;">1</span></strong>1 10010011000001011111
= 010001000100000
1<strong><span style="color: red;">1</span></strong>010<strong><span style="color: red;">0</span></strong>001<strong><span style="color: red;">0</span></strong>00101 000000001111110010<strong><span style="color: red;">0</span></strong>0 10010011000001011111
^ 1<strong><span style="color: red;">0</span></strong>010<strong><span style="color: red;">1</span></strong>001<strong><span style="color: red;">1</span></strong>00101 000000001111110010<strong><span style="color: red;">1</span></strong>0 10010011000001011111
= 010001000100000</pre>
<br />
<pre> <strong><span style="color: red;">1</span></strong>000<strong><span style="color: red;">1</span></strong>000<strong><span style="color: red;">0</span></strong>111000 00000001000000000<strong><span style="color: red;">0</span></strong>00 10010011000001011111
^ <strong><span style="color: red;">0</span></strong>000<strong><span style="color: red;">0</span></strong>000<strong><span style="color: red;">1</span></strong>111000 00000001000000000<strong><span style="color: red;">1</span></strong>00 10010011000001011111
= 100010001000000
<strong><span style="color: red;">0</span></strong>001<strong><span style="color: red;">1</span></strong>011<strong><span style="color: red;">0</span></strong>111001 00000001000000011<strong><span style="color: red;">0</span></strong>00 10010011000001011111
^ <strong><span style="color: red;">1</span></strong>001<strong><span style="color: red;">0</span></strong>011<strong><span style="color: red;">1</span></strong>111001 00000001000000011<strong><span style="color: red;">1</span></strong>00 10010011000001011111
= 100010001000000</pre>
<br />
After some time I managed to find these patterns:<br />
<br />
<pre>111110000000001 - payload bit 12
011011010000000 - payload bit 11
001101101000000 - payload bit 10
110110100100000 - payload bit 9
001011000010000 - payload bit 8
100101100001000 - payload bit 7
100010100000100 - payload bit 6
000001000000010 - payload bit 5
100000100000001 - payload bit 4
000100010000000 - payload bit 3
100010001000000 - payload bit 2
010001000100000 - payload bit 1
001000100010000 - payload bit 0
110100000001000 - header bit n
111010000000100 - header bit n-1
011101000000010 - header bit n-2</pre>
<br />
To me, it looked very much like these xor patterns were generated by a "many-to-many" <a href="http://en.wikipedia.org/wiki/Linear_feedback_shift_register">linear feedback shift register</a> (lfsr), but no matter now much I tried, I could not figure it out.<br />
<br />
So to crack the lsfr, I wrote a c++ program that used a <a href="http://en.wikipedia.org/wiki/Genetic_algorithm">genetic algorithm</a> to find the lsfr tabs:<br />
<br />
<pre class="brush:cpp">#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <math.h>
#include <time.h>
#include <algorithm>
typedef unsigned int uint;
uint patterns[] = {
0b111110000000001, 0b011011010000000, 0b001101101000000, 0b110110100100000,
0b001011000010000, 0b100101100001000, 0b100010100000100, 0b000001000000010,
0b100000100000001, 0b000100010000000, 0b100010001000000, 0b010001000100000,
0b001000100010000, 0b110100000001000, 0b111010000000100, 0b011101000000010,
};
static const uint pattern_cnt = (sizeof(patterns)/sizeof(patterns[0]));
uint calcErrors(uint mask[15])
{
unsigned errors = 0;
for(unsigned i=0; i<pattern_cnt-1; i++)
{
unsigned ipattern = patterns[i];
unsigned opattern = patterns[i+1];
unsigned pattern = ipattern;
pattern = (pattern>>1);
for(int biti=0; biti<15; biti++)
{
if(ipattern&(1<<biti))
pattern ^= mask[biti];
}
errors += __builtin_popcount(pattern^opattern);
}
return errors;
}
struct genome
{
uint mask[15];
uint errors;
bool operator<(const genome &other) const { return errors<other.errors; }
};
static const uint POP_SIZE = 10000;
int main()
{
srand(time(NULL));
static genome population[POP_SIZE];
for(uint i=0; i<POP_SIZE; i++)
for(uint j=0; j<15; j++)
population[i].mask[j] = rand()&0x7fff;
for(uint generation=0; generation<1000000; generation++)
{
for(uint i=0; i<POP_SIZE; i++)
population[i].errors = calcErrors(population[i].mask);
// sort genomes, so that the most fit will be in the beginning of the array
std::sort(population, population+POP_SIZE);
if(population[0].errors == 0)
{
for(uint j=0; j<15; j++) printf("%04x ", population[0].mask[j]);
printf("\n");
break;
}
static genome newpopulation[POP_SIZE];
for(uint i=0; i<POP_SIZE; i++)
{
uint parent1 = pow(double(rand())/RAND_MAX, 3.0)*POP_SIZE;
uint parent2 = pow(double(rand())/RAND_MAX, 3.0)*POP_SIZE;
// produce offspring:
memcpy(&newpopulation[i], &population[parent1], sizeof(newpopulation[i]));
for(uint j=0; j<15; j++)
if(rand()<RAND_MAX/2)
newpopulation[i].mask[j] = population[parent2].mask[j];
// mutate offspring
if(rand()<RAND_MAX/10)
{
uint bit = rand()%15;
uint mask = rand();
for(int j=0; j<15; j++)
if((mask>>j)&1)
newpopulation[i].mask[j] ^= 1<<(bit%15);
}
}
memcpy(population, newpopulation, sizeof(population));
}
return 0;
}
</pre>
<br />
It only took the program around 15 seconds to find the correct "taps": <em>0x4880 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x2080 0x4000 0x4000 0x4000 0x4000 0x4000 0x4000</em>.<br />
<br />
To verify that I now had enough information to verify the checksum of captured frames, I wrote another small python script that calculates the checksum for a frame, and checks itagains the checksum in the captured frame:<br />
<pre class="brush:py">
</pre>
<pre class="brush:py">checksum_taps = (0x4880, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x2080, 0x4000, 0x4000, 0x4000, 0x4000, 0x4000, 0x4000)
def next_mask(mask):
next_mask = mask>>1
for i in range(15):
if mask&(1<<i):
next_mask ^= checksum_taps[i]
return next_mask
def verify(checksum, data, datalen):
mask = 0x0001
cchecksum = 0
for i in range(datalen-1, 7, -1):
mask = next_mask(mask)
if (data>>i)&1:
cchecksum ^= mask
assert checksum == cchecksum
# |checksum-----| |payload---------------------------||header----||preamb|
verify(0b010011111001011, 0b00000000000000001011111001111101001110010011000101011111, 36+20)
verify(0b010100001111000, 0b00000000000000001011111001111110100010010011000101011111, 36+20)
verify(0b010011010000111, 0b0000000011111110010010010011000001011111, 20+20)verify(0b000010010100111, 0b0000000011111110011010010011000001011111, 20+20
</pre>
<br />
By trial and error, I also managed to find out that the <a href="http://en.wikipedia.org/wiki/Preamble_(communication)">preamble</a> is 8 bits long, and not included in the checksum, so the packages now look like this:<br />
<pre>
</pre>
<pre>checksum payload header preamble
010011111001011 000000000000000010111110011111010011 100100110001 01011111
010100001111000 000000000000000010111110011111101000 100100110001 01011111
010011010000111 00000000111111100100 100100110000 01011111
000010010100111 00000000111111100110 100100110000 01011111</pre>
<br />
Now that I can generate frames with correct checksums, I can start sending my own frames to the NG101 receiver and see how it reacts when the different bits are set. This will make it easier to discover what all the bits the the header and payload are used for.<br />
But that will have to wait for another day :)Anonymoushttp://www.blogger.com/profile/14345112203509983206noreply@blogger.com0tag:blogger.com,1999:blog-4506408752913482801.post-7539456138261271352011-01-09T20:55:00.000+01:002011-10-30T21:23:13.654+01:00NGE101 – Norgo wireless energy meter (part 5)This time, I will take a quick look at the long frames captured from the NGE101. Just like the short frames, they consist of 3 parts; a 20 bit preamble/header, the payload and last, a 15 bit checksum.<br />
<br />
<pre>preamble/header payload checksum
11111010100011001001 110010111110011111010000000000000000 110100111110010
11111010100011001001 000101111110011111010000000000000000 000111100001010
11111010100011001001 001111111110011111010000000000000000 100111000101011
11111010100011001001 000010000001011111010000000000000000 111000001111000
11111010100011001001 010001000001011111010000000000000000 001001000101100
11111010100011001001 110011000001011111010000000000000000 101011000011000</pre>
<br />
The payload is an ever increasing value, and when converted to decimal, they are:<br />
<pre>
000101111110011111010000000000000000 = 780264
001111111110011111010000000000000000 = 780284
000010000001011111010000000000000000 = 780304
010001000001011111010000000000000000 = 780322
110011000001011111010000000000000000 = 780339</pre>
<br />
The values are the number of times the LED has flashed on my smart meter since I installed the NGE.<br />
Since these frames are send exactly 86 seconds apart, it's easy to calculate the average power usage between two of the samples:<br />
<pre>
((780284-780264 flashes) / 86.0 s * 3600 s/h)/(1000 flashes/kWh) = 0.837 kW</pre>
<br />
Here, where I live, we pay around 2 kr/kWh, so if I sustain this usage for a whole day, I would end up paying:<br />
<pre>
0.837 kW * 24 h * 2 kr/kWh = 40.19 kr.</pre>
<br />
Next time, I will take a look at the package headers of both the short and long frames.Anonymoushttp://www.blogger.com/profile/14345112203509983206noreply@blogger.com0tag:blogger.com,1999:blog-4506408752913482801.post-53726812774976499092011-01-09T15:44:00.000+01:002012-03-03T13:50:53.554+01:00NGE101 – Norgo wireless energy meter (part 4)Now that I have managed to capture some raw data from the NGE101, it's time to decode it.<br />
The NGE sends two types of packages, one that's 55 bits long and another that is 71 bits long.<br />
In this post I will concentrate on the shorted packages.<br />
<br />
Here are a few of the thousands of packages collected in the last few days:<br />
<br />
<pre>11111010000011001001 00100111111100000000 111000010110010
11111010000011001001 01100111111100000000 111001010010000
11111010000011001001 00110111111100000000 111000000111010
11111010000011001001 00000000000010000000 000111000010001
11111010000011001001 10010000000010000000 000101011011101</pre>
<br />
I have separated the package into 3 "chunks"<br />
<ol>
<li>the first 20 bits are always the same in every package, it's most likely a preamble, device id, channel and other static data.</li>
<br />
<li>the middle 10-15 bits are almost always different, but when looking at thousands of samples, its possible to see that it's a binary value.</li>
<br />
<li>the last 15 bits looks like random noise, so it it most likely a checksum.</li>
</ol>
<br />
The middle chunk of bits is the one we want, so I captured a few packages and wrote down the power usage reported by the NGE101 reciever. In the following samples I have removed everything but the middle bits:<br />
<pre></pre>
<pre>captured package | receiver shows
... 11000011100010000000 ... 0.81
... 01011011111100000000 ... 0.90
... 00000010111100000000 ... 0.94
... 01110000000100000000 ... 1.78
... 10101100001000000000 ... 3.42</pre>
<br />
<span id="_marker"> </span><br />
<br />
It looks like the the <a href="http://en.wikipedia.org/wiki/Least_significant_bit">least significant bit</a> is to the left, so when decoded we get:<br />
<pre></pre>
<pre>captured | receiver shows
4547 0.81
4058 0.90
3904 0.94
2062 1.78
1077 3.42</pre>
<br />
Since the captured value and the one shown on the receiver are inversely proportional, we find the factor by multiplying them: 1077 * 3.42 = 3683. This is very close to the number of seconds in an hour, 3600, but not quite.<br />
In a earlier post I speculated that the micro controller in the transmitter was clocked by a 32768 kHz crystal, and based on that speculation, it's likely that the timing code does not use base 10, but base 2. This also simplifies the hardware, since no divisions are necessary as shifts can be used instead.<br />
With that in mind, the 3600 would end up as 3600*1024/1000 = 3686 which is much closer to the derived factor.<br />
<br />
Since my smart meter blinks its LED once for every Wh used, it should now be apparent that the number captured from the transmitter is in fact the number of milliseconds (in base2) between two blinks. And to convert from the captured number to power usage we just use this formula:<br />
<pre></pre>
<pre>(3600*1024/1000) / captured_value</pre>
<br />
In the next post I will take a look at the longer, 71 bit, frames.Anonymoushttp://www.blogger.com/profile/14345112203509983206noreply@blogger.com2tag:blogger.com,1999:blog-4506408752913482801.post-65869106440246500772011-01-06T23:21:00.000+01:002012-03-03T13:49:40.378+01:00NGE101 – Norgo wireless energy meter (part 3)Before decoding the data from the NGE101, it was necessary to find out the duration of the pulses it sends. To do this I needed a small sketch that could output the exact duration of the received pulses.<br />
<br />
This is the code I came up with, it will print a stream of bits and there durations in microseconds.<br />
<br />
<pre class="brush:[cpp];">uint8_t lastinput = LOW;
uint32_t lastinputat;
void setup()
{
pinMode(2, INPUT);
Serial.begin(115200*8);
lastinputat = micros();
}
void loop()
{
uint8_t input = digitalRead(2);
if(input != lastinput)
{
uint32_t now = micros();
uint32_t dt = now - lastinputat;
lastinput = input;
lastinputat = now;
Serial.print(input, 10);
Serial.print(' ');
Serial.println(dt, 10);
}
}
</pre>
<br />
The output looks something like this:<br />
<pre>1 1068
0 472
1 488
0 416
1 568
...</pre>
<br />
The durations were not as consistent as I had hoped they would be, so to get a better idea of what I was dealing with, I wrote a python gui application to graph the durations.<br />
<br />
<pre class="brush:py;collapse:true">import wx
import re
import serial
import sys
import time
import math
WIN_HEIGHT = 128
WIN_WIDTH = 590 # pixels
DURATION_PER_PIXEL = 0.003 / (WIN_WIDTH-1)
class TestFrame(wx.Frame):
def __init__(self):
wx.Frame.__init__(self, None, title='test', size=(WIN_WIDTH, WIN_HEIGHT))
self.line_re = re.compile('([01]) *([0-9]+)')
self.ser = serial.Serial('/dev/ttyACM0', 115200*8, timeout=0.000)
self.last = None
self.recvqueue = ''
self.durations = [[0]*WIN_WIDTH, [0]*WIN_WIDTH]
self.panel = wx.Panel(self, size=(WIN_WIDTH, WIN_HEIGHT))
self.panel.Bind(wx.EVT_PAINT, self.on_paint)
self.Fit()
self.Show(True)
self.timer = wx.Timer(self.panel, 123)
wx.EVT_TIMER(self.panel, 123, self.on_timer)
self.timer.Start(10)
def on_paint(self, event):
dc = wx.PaintDC(self.panel)
dc.SetPen(wx.Pen('red', 1))
dc.DrawLine(0, WIN_HEIGHT/2, WIN_WIDTH, WIN_HEIGHT/2)
for i in range(0, WIN_WIDTH, 1):
t = i*DURATION_PER_PIXEL
if math.floor(t/0.001) != math.floor((t-DURATION_PER_PIXEL)/0.001):
dc.DrawLine(i, 0, i, WIN_HEIGHT)
elif math.floor(t/0.0001) != math.floor((t-DURATION_PER_PIXEL)/0.0001):
dc.DrawLine(i, WIN_HEIGHT/2-8, i, WIN_HEIGHT/2+8)
dc.SetPen(wx.Pen('black', 1))
for i in range(WIN_WIDTH):
dc.DrawLine(i, WIN_HEIGHT/2+1, i, WIN_HEIGHT/2+1+self.durations[0][i])
dc.DrawLine(i, WIN_HEIGHT/2-1, i, WIN_HEIGHT/2-1-self.durations[1][i])
def on_timer(self, event):
redraw = False
while True:
line = self.ser.readline()
if not line: break
m = self.line_re.match(line)
if not m: continue
bit = int(m.group(1))
duration = float(m.group(2))/1000000.0
print bit, duration
self.durations[bit][min(int(duration/DURATION_PER_PIXEL), WIN_WIDTH-1)] += 1
redraw = True
if redraw:
self.panel.Refresh()
app = wx.App(False)
frame = TestFrame()
app.MainLoop()</pre>
<br />
The result looked like this. The window covers a durations of 3 milliseconds, so there is a tall vertical line per millisecond.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiw4ohY_DMVmnFfPpZEnqQc701qgulAvWfj8Zu6KARmT22zo9cANH7W2j_5j_o7SiXuSl5Ieqw33TJtjPgWJUvc6xaVCjNwCeGEpD_sSCs30Zwe_rcteJWI6bu92JzOebJrPzt9kUb74fA/s1600/pulse_durations.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiw4ohY_DMVmnFfPpZEnqQc701qgulAvWfj8Zu6KARmT22zo9cANH7W2j_5j_o7SiXuSl5Ieqw33TJtjPgWJUvc6xaVCjNwCeGEpD_sSCs30Zwe_rcteJWI6bu92JzOebJrPzt9kUb74fA/s1600/pulse_durations.png" /></a></div>
<br />
It's now possible to see that the durations fall into 4 groups:<br />
<ul><br />
<li>short high pulses (logic level 1) in the range 400-600 milliseconds.</li>
<br />
<li>short low pulses (logic level 0) in the range 300-500 milliseconds.</li>
<br />
<li>long high pulses in the range 900-1100 milliseconds.</li>
<br />
<li>long low pulses in the range 800-1000 milliseconds.</li>
</ul>
<br />
I suspect the difference in timing from the high and low pulses is because the sender has an extremely slow cpu, and it spends a few more cycles when sending a 1 than sending a 0. A bit of goggling suggest that 32.768 kHz is very common speed for low powered embedded micro controllers. At that speed, the time difference (0.15 ms) between high and low pulses are just around 5 cycles.<br />
<br />
Visualizing the data as a pulse train, using __, _ for low pulses, and --, - for high pulses, a pattern start to revel itself. Here are 4 bursts of data:<br />
<pre>__--__--__-_--_-_-_-_-_-__--_-_-__-_-_--__--_-__-_--__--_-__-_-_--_-_-_-_-_-_-_-_-__--_-_-__-_--_-__--__--_-_-
__--__--__-_--_-_-_-_-_-__--_-_-__-_-_--_-_-__-_-_-_-_-_--__-_-_--_-_-_-_-_-_-_-_-_-_-__-_-_--_-__--_-__--__--
__--__--__-_--_-_-_-_-_-__--_-_-__-_-_--__-_-_--_-_-_-_-__--_-_-__-_-_-_-_-_-_-_-_-_-_--__-_-_--_-__--_-_-__-_
__--__--__-_--_-_-_-_-_-__--_-_-__-_-_--__-_--_-_-__-_-_--__-_-_--_-_-_-_-_-_-_-_-__-_--__-_--_-_-_-_-__-_--__</pre>
<br />
The script used to print these pulse trains:<br />
<br />
<pre class="brush:py">import serial
import sys
import time
serdev = '/dev/ttyACM0'
ser = serial.Serial(serdev, 115200*8)
def decode(bit, duration):
if bit==1 and duration>=400 and duration<=600: return '-'
if bit==1 and duration>=900 and duration<=1100: return '--'
if bit==0 and duration>=300 and duration<=500: return '_'
if bit==0 and duration>=800 and duration<=1000: return '__'
return None
while True:
line = ser.readline().strip()
pulse = decode(int(line[0]), int(line[2:]))
if pulse: sys.stdout.write(pulse)
else: sys.stdout.write('\n') sys.stdout.flush()</pre>
A examination of the pulse trains reveals that short pulses always come in pairs!
This suggest that the data is likely to be <a href="http://en.wikipedia.org/wiki/Manchester_encoding">Manchester encoded</a>, or more likely <a href="http://en.wikipedia.org/wiki/Differential_Manchester_encoding">Differential Manchester encoded</a>.
After applying a differential Manchester decoder to the data, we see some very promising patterns:
<br />
<pre>1111101000001100100111011011000010000000101000010000110
(100 ms delay)
1111101000001100100111011011000010000000101000010000110
(43 s delay)
11111010100011001001101000100100100111010000000000000000101011110111110
1111101000001100100111100011000010000000001000101011110
1111101000001100100110011101000010000000110001011010101
1111101000001100100110011101000010000000110001011010101
11111010100011001001010110100100100111010000000000000000001000001101111
1111101000001100100110000011000010000000001001001101101
1111101000001100100111000011000010000000001000001001111
1111101000001100100111000011000010000000001000001001111
11111010100011001001011101100100100111010000000000000000111000101001001
1111101000001100100111011101000010000000110000011110111</pre>
The data is being send it bursts with around 43 seconds between them. In each burst there are two frames, separated by around 100 ms.
As can be seen, every second burst have a longer frame than usual, the other bursts just appear to be repeating the same frame twice.
The differential Manchester decoder:
<br />
<pre class="brush: py;">lastshort = False
while True:
line = ser.readline().strip()
pulse = decode(int(line[0]), int(line[2:]))
if pulse in ('_', '-'):
if lastshort:
sys.stdout.write('0')
lastshort = not lastshort
elif pulse in ('__', '--'):
if lastshort:
print 'ERROR'
lastshort = False
else:
sys.stdout.write('1')
else:
lastshort = False
sys.stdout.write('\n') sys.stdout.flush()</pre>
That's it for now, next time I will try to make some sense of the newly discovered bits...Anonymoushttp://www.blogger.com/profile/14345112203509983206noreply@blogger.com1tag:blogger.com,1999:blog-4506408752913482801.post-83438199847975439142011-01-06T00:15:00.000+01:002012-03-03T13:48:16.167+01:00NGE101 - Norgo wireless energy meter (part 2)I want to capture the data from the NGE101 power meter transmitter so that I can later analyze the data, or maybe even send it to <a href="http://www.google.com/powermeter">Google Powermeter</a>. But first I need to find out how the data is transmitted and encoded.<br />
<br />
According to the manual the NGE101 is transmitting the data on 433 MHz, so I dug out an old wireless weather station receiver that I had lost the transmitter for.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjzxLUF0WJpUrlkhBWFC2crbmtauRDoeuopLDslQ4gtmb69Jlu39577_HLXCJr60Fn7jUepRB-q_xLy9WQqUFPQsmT6fr99rUNhDGMtP99G3i45R1AGQCMtfniNX-z506U9dxVRYFIAbo/s1600/IMG_6194.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjzxLUF0WJpUrlkhBWFC2crbmtauRDoeuopLDslQ4gtmb69Jlu39577_HLXCJr60Fn7jUepRB-q_xLy9WQqUFPQsmT6fr99rUNhDGMtP99G3i45R1AGQCMtfniNX-z506U9dxVRYFIAbo/s320/IMG_6194.jpg" width="320" /></a></div>
</td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span class="Apple-style-span" style="font-size: small;">Old weather station with 433 MHz receiver.</span></td></tr>
</tbody></table>
Even though the weather station was powered by 3V (2 AA batteries) I did not see any components on the receiver PCB that should not be able to handle the 5V from the arduino uno. Its impossible to see in the picture, but the IC on the PCB is a LMV358 op-amp, and its rated for 2.7-5.5V.<br />
<br />
The receiver only has three pins, and they are labeled V,R and G. V and G are used for Vcc and Gnd, so by elimination, the R pin must carry the received signal.<br />
<br />
When hooking up the receiver to the arduino, I just got a lot of random static, but with some occasional faint hints of something that sounded like bursts of old-school modem noise.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3cuagRXSLstXCb9Jic-WgH7h5xgMh4j76u5V9LxC7CewHGkguArUIqWs_8aXAyyjo2pti3ASzU5XCkG9xuHceRON5Aj7fC7Emiylmqs8emPjWjD5Ur3_YPbMsNuIdpT9sqh_b_02kpms/s1600/IMG_6169.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3cuagRXSLstXCb9Jic-WgH7h5xgMh4j76u5V9LxC7CewHGkguArUIqWs_8aXAyyjo2pti3ASzU5XCkG9xuHceRON5Aj7fC7Emiylmqs8emPjWjD5Ur3_YPbMsNuIdpT9sqh_b_02kpms/s320/IMG_6169.jpg" width="320" /></a></div>
</td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span class="Apple-style-span" style="font-size: small;">Arduino and the 433 MHz receiver.</span></td></tr>
</tbody></table>
Fearing that I might have killed the receiver, I quickly cut the power, and then did what I should have done first; I connected the receiver to the 3.3V pin on the arduino.<br />
Unfortunately the output voltage from the receiver was not high enough for the arduino to detect it, but that was easily fixed by using a 4050 buffer to pull the signal up a bit.<br />
<div style="clear: both;">
<a href="http://www.blogger.com/blogger.g?blogID=4506408752913482801"></a></div>
<br />
This worked beautifully, I started to get some nice clear data from the arduino. I'm not sure why I got so much noise when it was connected to 5V, maybe there is a lot of digital noise on the 5V power line, maybe its simply not meant to run of that high a voltage.<br />
<br />
I wrote a small arduino sketch to capture the signal and output the data as a .au sample. Here is one of the captured data frames:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdXFY1nk56bN2XoPFhyphenhyphenbZfmvqL8jUcfCxvqCQ6vOPBqGbooXUwWKCPSzCa0l7U9iFxQQsF4Yk-bpTGz_Xc9saM0WWoqzaB5Hp5mKnVTe7Q6n_-m3KEntCfQ8UhGFGGiRRXmLBmpRG8eGo/s1600/data_capture.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdXFY1nk56bN2XoPFhyphenhyphenbZfmvqL8jUcfCxvqCQ6vOPBqGbooXUwWKCPSzCa0l7U9iFxQQsF4Yk-bpTGz_Xc9saM0WWoqzaB5Hp5mKnVTe7Q6n_-m3KEntCfQ8UhGFGGiRRXmLBmpRG8eGo/s1600/data_capture.png" /></a></div>
<br />
And here is the source code:<br />
<br />
<pre class="brush:[cpp];">uint32_t nextsampleat = 0;
void setup()
{
pinMode(2, INPUT);
Serial.begin(115200*8);
Serial.write((const uint8_t*)".snd", 4); // magic
Serial.write((const uint8_t*)"\x00\x00\x00\x18", 4); // header size
Serial.write((const uint8_t*)"\xff\xff\xff\xff", 4); // data size (-1 = unknown)
Serial.write((const uint8_t*)"\x00\x00\x00\x02", 4); // coding (2 = 8-bit linear PCM)
Serial.write((const uint8_t*)"\x00\x00\x27\x10", 4); // rate (10kHz)
Serial.write((const uint8_t*)"\x00\x00\x00\x01", 4); // channels
nextsampleat = micros();
}
void loop()
{
uint32_t now = micros();
if(int32_t(now-nextsampleat) &gt;= 0)
{
uint8_t input = digitalRead(2):
Serial.write(input?64:-64);
nextsampleat += 100;
}
}</pre>
<br />
A small python script that connects to the arduino, and just dumps all incomming data into a file:<br />
<br />
<pre class="brush:[py];">import serial, struct
ser = serial.Serial('/dev/ttyACM0', 115200*8)
samplefile = open('sample.au', 'wb')
while True:
data = ser.read(1)
samplefile.write(data)
</pre>
Next time I will examine how the data is encoded.Anonymoushttp://www.blogger.com/profile/14345112203509983206noreply@blogger.com11tag:blogger.com,1999:blog-4506408752913482801.post-83468611197675424152011-01-05T00:19:00.000+01:002012-03-03T13:43:00.521+01:00NGE101 - Norgo wireless energy meter (part 1)<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9Ih-pQ_DAN73fST9AVFQo15wiGVcnWuFfnxrElHQnZGbd-emkQEhOyBqdF5U6aweZ-rw2AvquyyF2q7esgE6_JsxjoNGUGAHqv8KxPEX9XwDx4bD7MVVhIZFWtUyAj_bLXGOV6IuYPic/s1600/official_device_image.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9Ih-pQ_DAN73fST9AVFQo15wiGVcnWuFfnxrElHQnZGbd-emkQEhOyBqdF5U6aweZ-rw2AvquyyF2q7esgE6_JsxjoNGUGAHqv8KxPEX9XwDx4bD7MVVhIZFWtUyAj_bLXGOV6IuYPic/s1600/official_device_image.jpg" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Norgo NG101 Receiver and transmitter</td></tr>
</tbody></table>
<br />
About a month ago, I bought a NGE101, it's a cool gadget that records and displays how much power a household is using.<br />
<br />
It works by counting the light pulses emitted from digital <a href="http://en.wikipedia.org/wiki/Smart_meter">Smart meters</a> and then transmits that data back to the receiver. According to the manual, it does this once every 45 seconds.<br />
<br />
This also means that the device will only with with the new digital smart meters, and not the old "analog" type with a spinning wheel and mechanical counters.<br />
<br />
The smart meter that is installed in my apartment blinks a red LED light once for every 1 kW that is used, so by timing the duration between two blinks, it is possible to calculate how much power is being used at any moment.<br />
<br />
You can read more about the device <a href="http://www.norgo.dk/products/">here</a>.<br />
<br />
Its all nice and good that the NGE101 can show the real time usage, and also the totals for the last 5 hours, 5 weeks and 5 months, but I want more :)<br />
<br />
I would really like to collect the real time power usage numbers into a database so that I can graph them over time with a higher resolution that what the NGE101 can do itself.<br />
<br />
As a first step I took the receiver apart, mostly because I was curious, but besides a few testing pads and open jumpers, the only real interesting thing I spotted was the dangling ground wire coming out of the receiver PCB. Its not the antenna, as it's just connected to ground and the real antenna can be seen to the far right in the "Guts" image. I don't know a lot about antenna design, but it seem to be strangely placed if it is supposed to be the ground plane for the monopole antenna. Antenna design is apparently really complicated, so who knows :)<br />
<div>
<br /></div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsu0KPuL26cYUgvLfi7rpDTzoqPt_9dLneJ64kzIt1MzM8SNax9TNpMnwOGO2Ave4_3crvoqBVqMsKQCZ1sJtdrDx8ldV1hZqfrd0UusJj9kGdj4PGmnca_-ApBKJtZd8djcvP41DdEGY/s1600/IMG_6163.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="66" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsu0KPuL26cYUgvLfi7rpDTzoqPt_9dLneJ64kzIt1MzM8SNax9TNpMnwOGO2Ave4_3crvoqBVqMsKQCZ1sJtdrDx8ldV1hZqfrd0UusJj9kGdj4PGmnca_-ApBKJtZd8djcvP41DdEGY/s200/IMG_6163.jpg" width="100" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Guts</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPrDDuIFS_gA5Y3gb2S2_28x7J4cRlczfFV8VXgfBaeVEAhg9WNgVccz35GQDN4F3efYx7VszJAbJzz3tRCRMUkxXf8CVmF71CqwhO123VrDY998b-ubp5ZESPnHk5LRFEOj0VSSgO39Q/s1600/IMG_6155.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="66" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPrDDuIFS_gA5Y3gb2S2_28x7J4cRlczfFV8VXgfBaeVEAhg9WNgVccz35GQDN4F3efYx7VszJAbJzz3tRCRMUkxXf8CVmF71CqwhO123VrDY998b-ubp5ZESPnHk5LRFEOj0VSSgO39Q/s200/IMG_6155.jpg" width="100" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Main logic board</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjB15I801KnDRQnNmUF-8QS1Hk2_b2h9RZi9i1ROdBHsGwWk-odJGaEYwmbG9C6NClDewCuwcPQhlT-Z9a7aMT2FwLxm9r7nh-UppIzHtRHA1SKhs2NnUIrK7SINOwL6bfrQEFXKRo1-bM/s1600/IMG_6159.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="66" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjB15I801KnDRQnNmUF-8QS1Hk2_b2h9RZi9i1ROdBHsGwWk-odJGaEYwmbG9C6NClDewCuwcPQhlT-Z9a7aMT2FwLxm9r7nh-UppIzHtRHA1SKhs2NnUIrK7SINOwL6bfrQEFXKRo1-bM/s200/IMG_6159.jpg" width="100" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Transmitter circuit board</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwrVNDbfBVOZVRA6x2TFJcQavlMdSN5mcAHd5LL7DfVV-4B5UPKgIT5tfy1AIw-wHEHfUyE3hrqAOPelcQArBTexCM168H3kX-ggVWEDBkN23OHsa_SfUYOvhAydLCD8Z2XAMpBQ-QSAQ/s1600/IMG_6157.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="66" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwrVNDbfBVOZVRA6x2TFJcQavlMdSN5mcAHd5LL7DfVV-4B5UPKgIT5tfy1AIw-wHEHfUyE3hrqAOPelcQArBTexCM168H3kX-ggVWEDBkN23OHsa_SfUYOvhAydLCD8Z2XAMpBQ-QSAQ/s200/IMG_6157.jpg" width="100" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Keyboard circuit board</td></tr>
</tbody></table>
<div style="clear: both;">
</div>
<br />
<div>
I hope that the 433 MHz transmitter is using <a href="http://en.wikipedia.org/wiki/Amplitude-shift_keying">ASK modulation</a>, which is commonly used in wireless weather stations and other one-way cheap wireless devices. And I also happen to have a arduino microcontroller and a 433 MHz receiver module handy, so the next step is to see if I can receive any data from the NG101 transmitter.</div>
<div>
<br /></div>
<div>
But that will have to be another post, another day.</div>
<div>
<br /></div>Anonymoushttp://www.blogger.com/profile/14345112203509983206noreply@blogger.com0tag:blogger.com,1999:blog-4506408752913482801.post-52638112834261934442011-01-01T21:55:00.000+01:002012-03-03T10:20:58.572+01:00Happy new year!<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEoDccn08E1sOruPxXaW8utaN6x7X2_EC7Q1J9fYvhCiTozfJxl7jSfZs4xUgQGmW_ejFwZLVrUJFIkkQyEB3jEsPYFwiWv805pAMVfZBztmbNMiC5rH3bKY3jbcjkZgfOQpnbGoc_moo/s1600/IMG_6097_940_198.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="132" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEoDccn08E1sOruPxXaW8utaN6x7X2_EC7Q1J9fYvhCiTozfJxl7jSfZs4xUgQGmW_ejFwZLVrUJFIkkQyEB3jEsPYFwiWv805pAMVfZBztmbNMiC5rH3bKY3jbcjkZgfOQpnbGoc_moo/s640/IMG_6097_940_198.jpg" width="640" /></a></div>
<br />
<br />
My one and only new years resolution this year was to get started with blogging.<br />
<br />
I'm fairly bad at formulating myself in written form, and if there is one thing that I have learned from playing RPG games, it's that the only way to get better at something is to keep repeating it, again and again :)Anonymoushttp://www.blogger.com/profile/14345112203509983206noreply@blogger.com0