Filtering DHT temperature sensor

I have some temperature sensors hooked up to Pi’s in the chicken coop and the basement, but was getting some nasty readings. The sensor in the coop would give erratic values of 5% humidity a few times an hour and the sensor in the basement was continuously dropping to zero.

I’m assuming you have the sensors hooked up already, if not the posts I got my inspiration from are linked at the bottom and there is a how-to.

The Idea of filtering the results is explained in the links as well, but in short: By leaving out all results outside of the standard deviation and calculating the means you get more consistent results.

In my case I collect 60 values, calculate means and standard deviation and exclude all erratic values. I use the Adafruit_DHT library to read the sensor, math and numpy to do the calculations and urllib.request to send the values to Domoticz. You can install all of those with pip3 (sudo pip3 install).

Here is the code:

import math
import numpy
import threading
from time import sleep
from datetime import datetime
import sys
import Adafruit_DHT
import urllib.request

# parameters
DHT_type    = 22
OneWire_pin = 4
sensor_idx  = 93
url_json    = "http://192.168.1.104:8080/json.htm?type=command&param=udevice&idx="

filtered_temperature = [] # here we keep the temperature values after removing outliers
filtered_humidity = [] # here we keep the filtered humidity values after removing the outliers

lock = threading.Lock() # we are using locks so we don't have conflicts while accessing the shared variables
event = threading.Event() # we are using an event so we can close the thread as soon as KeyboardInterrupt is raised

# function which eliminates the noise
# by using a statistical model
# we determine the standard normal deviation and we exclude anything that goes beyond a threshold
# think of a probability distribution plot - we remove the extremes
# the greater the std_factor, the more "forgiving" is the algorithm with the extreme values
def eliminateNoise(values, std_factor = 2):
    mean = numpy.mean(values)
    standard_deviation = numpy.std(values)

    if standard_deviation == 0:
        return values

    final_values = [element for element in values if element > mean - std_factor * standard_deviation]
    final_values = [element for element in final_values if element < mean + std_factor * standard_deviation]

    return final_values

# function for processing the data
# filtering, periods of time, yada yada
def readingValues():
    seconds_window = 60 # after this many second we make a record
    values = []
    
    while not event.is_set():
        counter = 0
        while counter < seconds_window and not event.is_set():
            temp = None
            humidity = None
            try:
                humidity, temp = Adafruit_DHT.read_retry(DHT_type, OneWire_pin)

            except IOError:
                print("we've got IO error")

            if math.isnan(temp) == False and math.isnan(humidity) == False:
                values.append({"temp" : temp, "hum" : humidity})
                counter += 1

            sleep(1)

        lock.acquire()
        filtered_temperature.append(numpy.mean(eliminateNoise([x["temp"] for x in values])))
        filtered_humidity.append(numpy.mean(eliminateNoise([x["hum"] for x in values])))
        lock.release()

        values = []

def Main():
    # here we start the thread
    # we use a thread in order to gather/process the data separately from the printing proceess
    data_collector = threading.Thread(target = readingValues)
    data_collector.start()

    while not event.is_set():
        if len(filtered_temperature) > 0: # or we could have used filtered_humidity instead
            lock.acquire()

            # here you can do whatever you want with the variables: print them, file them out, anything
            temperature = filtered_temperature.pop()
            humidity = filtered_humidity.pop()
#            print('{},{:.01f},{:.01f}' .format(datetime.now().strftime("%Y-%m-%d %H:%M:%S"), temperature, humidity))

            cmd = url_json  + str(sensor_idx) + "&nvalue=0&svalue=" + str(temperature) + ";" + str(humidity) + ";0"
#            print (cmd)
            urllib.request.urlopen(cmd)

            lock.release()
        # wait a second before the next check
        sleep(1)

    # wait until the thread is finished
    data_collector.join()
    
if __name__ == "__main__":
    try:
        Main()

    except KeyboardInterrupt:
        event.set()

You can also download this from my github.
On your pi do ‘wget’ and paste the URL using CTRL=SHIFT+V

I run the script on startup and get results every minute and a bit.

After almost 24 hours, the graph in Domoticz now looks like this:

I’m happy with the result and I hope you will enjoy a cleaner graph as well. Like I said, I only connected the dots between a tutorial to read the sensor and send data to domoticz and a how to to filter DHT results

Let me know how it worked for you!