Measuring empy space

If you've been following my recent posts on twitter, you'll be aware that I've finally got round to hooking up the MaxBotix rangefinders into our water tanks. So far I've done the main house tank and have sketches and buried duct ready to do the ones that water the garden.

Some more technical details here than I can fit in tweets:
* The sensors themselves sit through the roof of the tank. This involves drilling a 3/4" hole in the top of the tank and inserting the sensor from below, so you won't want to put it more than an arms reach from the access hole. The sensors I purchased have NPR thereads not BSP, so you may want to pick up matching nuts at the same time depending on local availability.
* I mounted some 6-way header pins on the sensor, and soldered the female end onto the cable so that it didn't twist as I tightened it up. All mounted in a 20mm conduit box on the tank.

Inside the shed, I'm just using a clone arduino with a wiznet ethernet shield (ebay/aliexpress) and a very basic sketch using the pulse width measurement from the sensor. As normal, it's available on GitHub for anyone interested. It pushes to an on-site MQTT broker (mosquitto) that's used to gather various sensor metrics and fed into home-assistant, where it also gets archived into InfluxDB for longer term plots

status/housetanksensor online

sensors/housetanks/depthbelowsensor 419

sensors/housetanks/litres 20762

I use the status for LWT messages, and the sensors has the raw air-gap measurement (mm below sensor - range is 300 -> 5000) as well as a calculated volume (see the arduino code for constants and calculation)

Importing into hass is as simple as

  - platform: mqtt

    name: House water tanks volume

    icon: mdi:water-pump

    availability_topic: status/housetanksensor

    unit_of_measurement: l

    state_topic: sensors/housetanks/litres

into the configuration.yaml

Next up, getting the irrigation system working...

weewx to home-assistant

At home I have a Fine Offset (this one branded by Jaycar) weather station that publishes to weather.two-fifteen.com via weewx (this is much simpler now I don't have to have the thing solar mounted in a field using a 3G dongle - at least the NBN is useful for some things) but I'd like to be able to use some of the measurements in home-assistant.

There isn't yet a direct plugin (spare time project anyone?) that I can see, but because I'm pushing the metrics locally to influxdb (have I mentioned I like drawing graphs of things?) for grafana, it's possible to use this in home-assistant via weewx-influx

Weewx config:
[StdRESTful]
      [[Influx]]
        host = localhost
        database = weather
        unit_system = METRIC

and on the home-assistant server:
  - platform: influxdb
    host:    203.0.113.88
    queries:
    - name: Outside Temp
      database: weather
      measurement: 'record'
      field: 'outTemp_C'
      group_function: last
      where: 'time > now() - 5m'
      unit_of_measurement: °C
      value_template: '{{ value | round(1) }}'

and lo and behold, you should end up with something like this:

which when compared to our nearest BOM observation site up the road in Grove correlates nicely (Grove is in a valley, we're at the top of a hill)

Plotting Lustre MDS stats

At $dayjob we have several large filesystems - for example our /scratch system has 3.1 PB of space using over 1000 HDDs. Although each vendor offers their own dashboard for monitoring they're all a little bit crap and don't integrate with anything else.

Cue an afternoon setting up influxdb (trivial) and grafana (also trivial) on a spare VM and a simple python script run on the metadata servers:

[admin@snx11038n003 ~]$ cat push_mdt_stats.py
#!/usr/local/bin/python2.7
import urllib
import time

def grabbit(mds):
 post = ""
 with open(('/proc/fs/lustre/mdt/%s/md_stats' % mds), 'r') as f:
    for line in f:
        k,v,null = line.split(None,2)
        if k == "snapshot_time":
            ts=int(float(v)*1000000)
        else:
            post += 'metadata,fs={3} {0}={1} {2}\n'.format(k,v,ts,mds)
 with open(('/proc/fs/lustre/mdd/%s/changelog_users' % mds), 'r') as f:
    tmp = f.read().split()
    # we can cheat here as they have the same format - 3rd item in list is current changelog count, and then
    # from the 6th item on we get changelog id / position to pull into a dict
    head = int(tmp[2])
    clog = dict(zip(tmp[5:][0::2], tmp[5:][1::2]))
    post += 'changelog,fs={2} head={0} {1}\n'.format(head,ts,mds)
    for cl,count in clog.items():
        post += 'changelog,fs={3} {0}={1} {2}\n'.format(cl,count,ts,mds)

 post=post.encode('ascii')
 p = urllib.urlopen('http://influxbox:8086/write?db=lustre&precision=u',post)
 #print(p.getcode())

while True:
  try:
    grabbit('snx11038-MDT0000')
  except:
    sys.exit("Whoa, that went a bit Pete Tong!")
  time.sleep(10)

And a couple of clicks in Grafana can soon knock up a dashboard: