Particulate measurements near bushfires

Back in the 2019 Tasmanian bushfires we were lucky enough to be the other side of the Huon River from the fireground,  however we had a LOT of smoke (compare the photo below to the street view image)

We were also lucky enough to take part in a Huon Valley trial to compare HEPA filter systems which used a pair of CSIRO Smoke Observation Gadgets (further details on these units in Sensors 2021, 21(21), 7206; https://doi.org/10.3390/s21217206) to log the air quality (pm2.5 and pm10 measurements) inside and out. 

It's now 2025, and while we've been very lucky compared to the US and mainland Australia, we now have a bushfire nearby (~6km from our house) that contains the following advice:

Residents should be aware that smoke and ash may be visible in the Channel area over the coming weeks, and anyone who is at higher risk from smoke should enact their personal health plans.

So, it's time to DIY some new smoke sensors! (I would just rely on the excellent BLANkET network already in place in Tasmania, but there aren't any downwind of the current fire)

Hardware

I already have a pair of PMS7003 sensors unused in a 'future projects' box, so we'll use them. They are supported by esphome, so using an esp32/esp8266 with them seems a simple choice

Mine (located in Kettering) will use a wired esp32 (because POE makes that simple), and I'll see if I can get one mounted at the Channel Men's Shed in Margate which will need to use wifi

I don't have a nice shiny 3d printed enclosure, so it'l use something like an old yoghurt tub to weatherproof them.

Build

.. To Be Continued at the weekend :-)

Garden Irrigation, Pt 1

Some more background to my 'home assistant powered irrigation system' -- I'll write up a couple more articles as build progresses - I'm still waiting on some critical parts to arrive, like PVC pipe fittings and an Arduino Mega.

The Grand Plan.
The 2 main garden bed areas need to be automatically watered. The small 'hexagon' area contains 6 raised beds and a small central area. The main veggie patch (still being built. slow progress...) is around 20 beds. Later automation will include more smarts (hooking in my rainfall sensor, some buried gypsum sensors and soil temperature monitoring), but for now, the priority is to save on the 30-60 mins with a hosepipe in the evenings. Oh, and we're on tank (rainwater collection) water, so wastage is bad mmmkay.

The irrigation units themselves won't be 'smart' - All they need to do is send valve (relay) status, and respond to controls. I have an internal MQTT message broker, so a basic arduino + ethernet shield + 8 channel relay board does the small bed, and an arduino mega + ethernet (or an ethermega) + 3 8 channel relay shields the main veggie patch.

Sample arduino code is available on GitHub, (functions? who needs functions...) and yeah, it's not elegant. While it follows the syntax for the home assistant MQTT switch device, some of the defaults aren't the same, so I need to explictitly set them for now in the hass config. such as

switch:
  - platform: mqtt
    name: Hexagon bed 1
    icon: mdi:water
    availability_topic: status/hexIrrigation
    command_topic: control/hexIrrigation/valve1
    payload_on: "1"
    payload_off: "0"
    state_topic: status/hexIrrigation/valve1
    state_on: "On"
    state_off: "Off"

You can see here that I use status/hexIrrigation as LWT (so I know if the controllers on/offline) as well as the the relay state. I'm only using retain on the LWT, not the valves themselves. Power cycling the unit will (should!) reset back to all solenoids off, but I don't yet check and publish this, so hass may get confused if it loses comms. "Fixed in the next release" apparently :-)

I also want to integrate a flow sensor into each unit if possible (suspect I'm running out of digital IO on the small unit) to get an overall volume of water that's gone into the garden - fed, as normal into grafana.

I'll throw in another couple of postings (with pics) as the build progresses. but today I had an audience while working on the fence... (it will be fully netted to keep out the possums, pademelons and our hens)

Measuring empy space

If you've been following my recent posts on twitter, you'll be aware that I've finally got round to hooking up the MaxBotix rangefinders into our water tanks. So far I've done the main house tank and have sketches and buried duct ready to do the ones that water the garden.

Some more technical details here than I can fit in tweets:
* The sensors themselves sit through the roof of the tank. This involves drilling a 3/4" hole in the top of the tank and inserting the sensor from below, so you won't want to put it more than an arms reach from the access hole. The sensors I purchased have NPR thereads not BSP, so you may want to pick up matching nuts at the same time depending on local availability.
* I mounted some 6-way header pins on the sensor, and soldered the female end onto the cable so that it didn't twist as I tightened it up. All mounted in a 20mm conduit box on the tank.

Inside the shed, I'm just using a clone arduino with a wiznet ethernet shield (ebay/aliexpress) and a very basic sketch using the pulse width measurement from the sensor. As normal, it's available on GitHub for anyone interested. It pushes to an on-site MQTT broker (mosquitto) that's used to gather various sensor metrics and fed into home-assistant, where it also gets archived into InfluxDB for longer term plots

status/housetanksensor online

sensors/housetanks/depthbelowsensor 419

sensors/housetanks/litres 20762

I use the status for LWT messages, and the sensors has the raw air-gap measurement (mm below sensor - range is 300 -> 5000) as well as a calculated volume (see the arduino code for constants and calculation)

Importing into hass is as simple as

  - platform: mqtt

    name: House water tanks volume

    icon: mdi:water-pump

    availability_topic: status/housetanksensor

    unit_of_measurement: l

    state_topic: sensors/housetanks/litres

into the configuration.yaml

Next up, getting the irrigation system working...

Overlaying SLURM job timings on Grafana plots

As you may have noticed, I'm quite fond of Grafana and use it at home and work. One of the dashboards I have at work is the general state of our lustre filesystems, showing IO and metadata traffic, collected by a custom python script (I'm working on converting this to a real collectd python plugin) which stores the data in an influxDB.

I've since written a small python script that talks to our SLURM accounting DB, so that given a jobID, we can get the start/end times and overlay those using the annotations API. One minor niggle in that the API expects epoch milliseconds, and seems to be tied to the TZ of the browser that generated the API key.

however...

~$ annotate_job 2924399

Found the following job:

  User: bskjerven (pawsey0001)

  Cluster: magnus, Partition: workq, QOS: normal

  Nodes: 768, CPUs: 36864

  Start: 2018-06-11 17:23:22, End: 2018-06-11 19:54:44

Got something back - Annotate? (y/n) y

200 - Annotation added




and lo - 

weewx to home-assistant

At home I have a Fine Offset (this one branded by Jaycar) weather station that publishes to weather.two-fifteen.com via weewx (this is much simpler now I don't have to have the thing solar mounted in a field using a 3G dongle - at least the NBN is useful for some things) but I'd like to be able to use some of the measurements in home-assistant.

There isn't yet a direct plugin (spare time project anyone?) that I can see, but because I'm pushing the metrics locally to influxdb (have I mentioned I like drawing graphs of things?) for grafana, it's possible to use this in home-assistant via weewx-influx

Weewx config:
[StdRESTful]
      [[Influx]]
        host = localhost
        database = weather
        unit_system = METRIC

and on the home-assistant server:
  - platform: influxdb
    host:    203.0.113.88
    queries:
    - name: Outside Temp
      database: weather
      measurement: 'record'
      field: 'outTemp_C'
      group_function: last
      where: 'time > now() - 5m'
      unit_of_measurement: °C
      value_template: '{{ value | round(1) }}'

and lo and behold, you should end up with something like this:

which when compared to our nearest BOM observation site up the road in Grove correlates nicely (Grove is in a valley, we're at the top of a hill)

Plotting Lustre MDS stats

At $dayjob we have several large filesystems - for example our /scratch system has 3.1 PB of space using over 1000 HDDs. Although each vendor offers their own dashboard for monitoring they're all a little bit crap and don't integrate with anything else.

Cue an afternoon setting up influxdb (trivial) and grafana (also trivial) on a spare VM and a simple python script run on the metadata servers:

[admin@snx11038n003 ~]$ cat push_mdt_stats.py
#!/usr/local/bin/python2.7
import urllib
import time

def grabbit(mds):
 post = ""
 with open(('/proc/fs/lustre/mdt/%s/md_stats' % mds), 'r') as f:
    for line in f:
        k,v,null = line.split(None,2)
        if k == "snapshot_time":
            ts=int(float(v)*1000000)
        else:
            post += 'metadata,fs={3} {0}={1} {2}\n'.format(k,v,ts,mds)
 with open(('/proc/fs/lustre/mdd/%s/changelog_users' % mds), 'r') as f:
    tmp = f.read().split()
    # we can cheat here as they have the same format - 3rd item in list is current changelog count, and then
    # from the 6th item on we get changelog id / position to pull into a dict
    head = int(tmp[2])
    clog = dict(zip(tmp[5:][0::2], tmp[5:][1::2]))
    post += 'changelog,fs={2} head={0} {1}\n'.format(head,ts,mds)
    for cl,count in clog.items():
        post += 'changelog,fs={3} {0}={1} {2}\n'.format(cl,count,ts,mds)

 post=post.encode('ascii')
 p = urllib.urlopen('http://influxbox:8086/write?db=lustre&precision=u',post)
 #print(p.getcode())

while True:
  try:
    grabbit('snx11038-MDT0000')
  except:
    sys.exit("Whoa, that went a bit Pete Tong!")
  time.sleep(10)

And a couple of clicks in Grafana can soon knock up a dashboard: