Filtering DHT temperature sensor

I have some temperature sensors hooked up to Pi’s in the chicken coop and the basement, but was getting some nasty readings. The sensor in the coop would give erratic values of 5% humidity a few times an hour and the sensor in the basement was continuously dropping to zero.

Before filtering, results from my coop

I’m assuming you have the sensors hooked up already, if not the posts I got my inspiration from are linked at the bottom and there is a how-to.

The Idea of filtering the results is explained in the links as well, but in short: By leaving out all results outside of the standard deviation and calculating the means you get more consistent results.

In my case I collect 60 values, calculate means and standard deviation and exclude all erratic values. I use the Adafruit_DHT library to read the sensor, math and numpy to do the calculations and urllib.request to send the values to Domoticz. You can install all of those with pip3 (sudo pip3 install).

Here is the code:

import math
import numpy
import threading
from time import sleep
from datetime import datetime
import sys
import Adafruit_DHT
import urllib.request

# parameters
DHT_type    = 22
OneWire_pin = 4
sensor_idx  = 93
url_json    = "http://192.168.1.104:8080/json.htm?type=command&param=udevice&idx="

filtered_temperature = [] # here we keep the temperature values after removing outliers
filtered_humidity = [] # here we keep the filtered humidity values after removing the outliers

lock = threading.Lock() # we are using locks so we don't have conflicts while accessing the shared variables
event = threading.Event() # we are using an event so we can close the thread as soon as KeyboardInterrupt is raised

# function which eliminates the noise
# by using a statistical model
# we determine the standard normal deviation and we exclude anything that goes beyond a threshold
# think of a probability distribution plot - we remove the extremes
# the greater the std_factor, the more "forgiving" is the algorithm with the extreme values
def eliminateNoise(values, std_factor = 2):
    mean = numpy.mean(values)
    standard_deviation = numpy.std(values)

    if standard_deviation == 0:
        return values

    final_values = [element for element in values if element > mean - std_factor * standard_deviation]
    final_values = [element for element in final_values if element < mean + std_factor * standard_deviation]

    return final_values

# function for processing the data
# filtering, periods of time, yada yada
def readingValues():
    seconds_window = 60 # after this many second we make a record
    values = []
    
    while not event.is_set():
        counter = 0
        while counter < seconds_window and not event.is_set():
            temp = None
            humidity = None
            try:
                humidity, temp = Adafruit_DHT.read_retry(DHT_type, OneWire_pin)

            except IOError:
                print("we've got IO error")

            if math.isnan(temp) == False and math.isnan(humidity) == False:
                values.append({"temp" : temp, "hum" : humidity})
                counter += 1

            sleep(1)

        lock.acquire()
        filtered_temperature.append(numpy.mean(eliminateNoise([x["temp"] for x in values])))
        filtered_humidity.append(numpy.mean(eliminateNoise([x["hum"] for x in values])))
        lock.release()

        values = []

def Main():
    # here we start the thread
    # we use a thread in order to gather/process the data separately from the printing proceess
    data_collector = threading.Thread(target = readingValues)
    data_collector.start()

    while not event.is_set():
        if len(filtered_temperature) > 0: # or we could have used filtered_humidity instead
            lock.acquire()

            # here you can do whatever you want with the variables: print them, file them out, anything
            temperature = filtered_temperature.pop()
            humidity = filtered_humidity.pop()
#            print('{},{:.01f},{:.01f}' .format(datetime.now().strftime("%Y-%m-%d %H:%M:%S"), temperature, humidity))

            cmd = url_json  + str(sensor_idx) + "&nvalue=0&svalue=" + str(temperature) + ";" + str(humidity) + ";0"
#            print (cmd)
            urllib.request.urlopen(cmd)

            lock.release()
        # wait a second before the next check
        sleep(1)

    # wait until the thread is finished
    data_collector.join()
    
if __name__ == "__main__":
    try:
        Main()

    except KeyboardInterrupt:
        event.set()

You can also download this from my github.
On your pi do ‘wget’ and paste the URL using CTRL=SHIFT+V

I run the script on startup and get results every minute and a bit.

After almost 24 hours, the graph in Domoticz now looks like this:

Graph after filtering.

I’m happy with the result and I hope you will enjoy a cleaner graph as well. Like I said, I only connected the dots between a tutorial to read the sensor and send data to domoticz and a how to to filter DHT results

Let me know how it worked for you!

OTGW on remote pi with ser2net

Now I am building my own domotica system with Domoticz, I bought and built a open therm gateway (OTGW) to intercept and modify the communications between my boiler and the Honeywell Round thermostat I have in the livingroom.

The situation is as follows, I have a rPi in the basement, connected to an RFLink and the smart meter. (The water meter is on the to-add list.) In the attic I have a rPi running some other software and a Synology NAS. The latter fails in running Domoticz and device sharing seems tedious with two different devices so I went with the pi.

Connected everything, setup device sharing per the manual and everything went fine. Or so I thought. Every time I changed the setpoint on the master Domoticz, a new device called ‘setpoint’ would appear on the slave and the setpoint would not be changed on the thermostat/boiler.
Googling this problem got me on two tracks: Write a script on the slave that fixes this or use the OTGW remotely via SER2NET. I chose the latter because I dind’t like the extra software on the pi in the attic anyway and this seemed less of a hassle.

Installing ser2net

SSH into your device and enter the following:

sudo apt update
sudo apt upgrade
sudo apt install ser2net -y

Find out where your OTGW lives and enter:

dmesg | grep tty

To be honest, I couldn’t make out which was the OTGW but since the other USB device is a hdd, I made an educated guess and it worked. I went with /dev/ttyUSB0
Next we will edit the ser2net config by opening the file.

sudo nano /etc/ser2net.config

In this file at the bottom, enter:
4000:raw:0:/dev/ttyUSB0:9600,NONE,1STOPBIT,8DATABITS
which breaks down into: ‘PORT:STATE:TIMEOUT:DEVICE:OPTIONS’ where 0 disables timeout and our options are the baudrate, parity, number of stop bits and number of data bits

Next, add a hardware device to your domoticz choosing ‘OpenTherm Gateway with LAN interface’ for type and connecting to the IP of your device and the port (4000 in our case*) you specified.
Allow new devices for 5 minutes and you’ll see the devices come to your Domoticz server!

* Don’t forget to open this port in your firewall if you have any. In case of UFW

sudo ufw allow from 192.168.1.0/24 to any port 4000 comment 'Allow OTGW ser2net on local lan'

Have fun!