Wednesday 2 September 2020

Thoughts on Statuspage.io integrations

 $dayjob uses statuspage.io for our public status page. Although I've got some things automated, there's still a bunch of components that need manual updating when there's an issue (enough to keep me out of mischief working out how to automate them without false alerts anyway). However this post isn't for getting info _in_ to statuspage, but for how to get it _out_, and what I want.

We're predominantly a command line shop (no, we don't yet have a JupyterHub frontend), so users are presented with a MOTD on login. Now, wouldn't it be good if that could be updated automatically with details of upcoming planned outages as well as any recent (and current) incidents that affected the service you're currently logged into?

So - Armed with the API, it should be possible to get upcoming maintenance[✓] and the impacted components[✓], but where do I map the cluster name to the statuspage group_id? or, for that matter any of the autogenerated id strings. Hard coding them into scripts is out, a lookup makes sense but how many CMDBs out there come with that sorta functionality built in. and we're back to another 'where is my source of truth?' problem. Sure I can string match components->name and check that "group": true and then pull the ID, but... yeah faffy. 

Anyway, after much parsing (all hail requests) it's possible to get the various incidents/component states/planned maintenance out and the resulting text snippets touched to the correct updated_at timestamp whereupon they can be pulled into the final output whenever the files are regenerated by a Makefile build under Jenkins. The goal of DRY is starting to be achieved by updating one place (currently statuspage) and having that trigger a build via webhook which then distributes the info out to the clusters.

I think a former colleague summed it up fairly well tho.


Thursday 2 April 2020

Solar off-grid cabin

We have a small cabin on the site that used to have a 200w panel, 65Ah deep cycle lead acid battery, el-cheapo PWM charger to power some LED strip when we visited.

While this was OK for occasional use, there's no way it'd be any good for regular usage, let alone constant 12v fan for the composting loo, water pump, fridge, wifi etc. So, time for an upgrade.

First off, new panels. The roof faces pretty much north with not much shading. Jaycar sells a fairly decent range of rails and clamps, and I purchased 3*300w (seemed to be the sweet spot for size/cost/output) panels from ebay that were 800mm wide to fit on a 2560mm rail. 

Wired in series, they still fall under the wiring rules definition of 'Extra Low Voltage' (< 120v ripple free DC). Even on the sunniest of days I've only seen them putting out about 70v so far.

Batteries. The performance benefit of LiFePO4 is huge compared to lead-acid. I took a chance and purchased a set of 8*123 Ah cells from AliExpress ("BLS Official Store"). These came in at 778 USD delivered, nicely packaged and all 8 cells were within 0.02v of each other when I checked.
For protection / balancing I purchased a 'Daly' 8s BMS with a common port (again, AliExpress), and a 120A isolator from Jaycar. To keep cable runs as short as reasonably possible and not clutter up the limited space in the cabin, I decided to put the batteries in a lockable toolchest (bunnings) fixed to the supports for the bathroom shed. There's a small condensation drain in the corner, and the batteries themselves are raised from the base slightly with some strips of yellowtongue. My one gripe about this is the cables on the BMS are slightly too short to position a the BMS neatly.

Linking the two together is a fairly chunky charge controller - I went for a 50A EPEVER Tracer, mainly because they seemed to have a good reputation for the price, and published specs for the modbus-rs485 output. Once you go to the 50-100A level range of controllers, the load connections aren't on the charge controller, but need to be taken off the battery itself. There are a couple of switched outputs that can be used for LV disconnect and/or external charging relays. Oh, and it's a little larger than the old one we had.

Back to ebay for a switchboard and some DC rated MCBs (electric_super) to try and get a neat job. I've got plenty of space behind the melamine chipboard panel for the terminal posts (battery incomers), LV disconnect relay and a POE injector for the wifi access point

My plan is to have remote telemetry back to home-assistant / grafana long term, but for now I've put an 'MT50' meter into the cabin (with only moderate swearing for the cable run) 

So, what next?
I'd like to run an inverter occasionally, but am not convinced we have enough capacity in the batteries (it'd be used to run things like a vacuum cleaner). Today was heavily overcast and the bms dropped the batteries with LV protection, so perhaps an option to connect a charger (this would mean a lot of extension leads or a genset) - potentially one of the combined units? depends on wiring regs if its "fixed wiring"

Getting the telemetry off is becoming more important now we're approaching winter. I've discovered I can't use the wired tail on the UAP-AC-M when it's operating in 'wireless uplink' mode or it thinks its uplink is on the wired side and keeps disconnecting from its upstream partner. So, I'm looking for a low-power wifi SBC that'll easily let me script up modbus monitoring from the tracer, water usage from an impeller sensor, and the water tank volume from an ultrasonic sensor, and fling the whole lot at the on-site MQTT broker.

Inside? well I've added some upstairs lights, multiple DC-DC converters so there's places to charge a mobile phone, and some better kitchen lighting. As well as the fridge and waterpump (we now have both cold AND HOT running water)







Feeling Pumped!

Having just had a day without power, and then going round the site to check everything came back online correctly (including services such a...