Thoughts on Statuspage.io integrations

 $dayjob uses statuspage.io for our public status page. Although I've got some things automated, there's still a bunch of components that need manual updating when there's an issue (enough to keep me out of mischief working out how to automate them without false alerts anyway). However this post isn't for getting info _in_ to statuspage, but for how to get it _out_, and what I want.

We're predominantly a command line shop (no, we don't yet have a JupyterHub frontend), so users are presented with a MOTD on login. Now, wouldn't it be good if that could be updated automatically with details of upcoming planned outages as well as any recent (and current) incidents that affected the service you're currently logged into?

So - Armed with the API, it should be possible to get upcoming maintenance[✓] and the impacted components[✓], but where do I map the cluster name to the statuspage group_id? or, for that matter any of the autogenerated id strings. Hard coding them into scripts is out, a lookup makes sense but how many CMDBs out there come with that sorta functionality built in. and we're back to another 'where is my source of truth?' problem. Sure I can string match components->name and check that "group": true and then pull the ID, but... yeah faffy. 

Anyway, after much parsing (all hail requests) it's possible to get the various incidents/component states/planned maintenance out and the resulting text snippets touched to the correct updated_at timestamp whereupon they can be pulled into the final output whenever the files are regenerated by a Makefile build under Jenkins. The goal of DRY is starting to be achieved by updating one place (currently statuspage) and having that trigger a build via webhook which then distributes the info out to the clusters.

I think a former colleague summed it up fairly well tho.

The writing's on the wall

For the last couple of jobs I've had, having some sort of status display has proved itself really useful. Things like an overloaded nagios dashboard help to drill down to see what system issues you may have, but on large systems there'll always be some component that's not green (however your service should work around these transparently to the end user).  In a smaller team without 24/7 operations staff and shift handovers, how do you know things aren't on fire when you walk into the office? - I'll ignore the fact that you probably read your email over breakfast.

Concerto

At $job[-1] we put a spare monitor on the office wall and ran concerto on a PC feeding it. The backend at that time was php, and made assumptions such as assuming that short tags were OK - I hacked on a branch to make it more standard with the scientific linux systems we were using at the time. Given this was (is?) a student project out of the Rensselaer Institute ir's hardly suprising as young developers want LATEST SHINY. They then went through a second system effect, rewriting from scratch and completely missing the launch of the raspberry pi, which could have made a killer combination.

The problem is that many browser based clients need the overhead of X11 and all the various hacks to remove mouse cursors and make them more 'kiosky'

info-beamer

Fast forwards a few years and I have mini-magnus on my desk showing status and a set of 3 unused monitors on a wall looking to display some info. A quick bit of research flagged up info-beamer which has been used in production at the CCC events for several years. I've been playing with it for a couple of days now in the standalone pi and hosted variant.

hosted info-beamer

The install of this is very smoothly done. Small zip file download to populate an SD card with the raspberry pi bootloader files and a squashfs  and it self-installs the rest of the distribution from S3 and prompts you to register the node to your info-beamer account (in a similar way to a chromecast).

Florian has made some really nice touches to the setup - little things like setting cec_osd_name if your TV screen uses it, a custom kernel logo, and suppression of all but the player-related boot messages. In a public area, this makes it look a lot more professional than most of the other solutions which show the operating system before launching a player should they reboot. I've only played with a couple of the sample packages, but the install process is slick, if initially confusing terminology between packages, setups and playlists.

I'd love to see this integrated with indico - the meeting software used at CERN. Hint :-)

standalone info-beamer

The personal use player is distributed as a binary executable - I can understand the reasons for that, but it doesn't feel right. It doesn't (yet?) come as a debian package - It should be trivial for me to do myself according to the docs. When that's done, I'll install and run using systemd rather than the daemontools method (which is used by the various syncer scripts used in the hosted version). My concern is that the logging is presently noisy (good for debugging, bad for SD lifetime) 

I'm also planning on managing these R-Pi nodes using Ansible (Puppet is overkill for this as I want something that can bootstrap up a fresh SD card without needing extra daemons running) so I suspect there'll be future blogging on that. 

I've not yet investigated sending values directly to the info-beamer listening port (hoping I can do something with an MQTT subscriber) but pushing json files from various subsystems seems to work well. Obligatory screenshot:

Prototype display testing

Overall, I'm very impressed. Given I'd started learning lua for some nodemcu work I should be able to develop something functional. I've asked the work graphic designer to assist, so hopefully it won't end up "engineer style"

Yubico / Yubikeys

I'm impressed.

I have small keyring with a USB memory stick, 2 yubikeys and a cofee machine cashless RFID key on. Stupidly I left said key in the vending machine. The system operators at work collected it and set about finding the owner. 1st up, they discovered that they can't read any files off yubikeys (heh) but googled the image and found the manufacturers website and said they had found serial #.... and #.....

Having spend the last 2 days rummaging in car / home looking for it, I got an email from yubico saying that <email> had found my key, based on the serial no of the one I purchased. Work also got an email for the other serial no, which they traced to me.

I'm *very* impressed by this level of attention at yubico, and it means when distributing keys within the business it pays to keep track of the serial no (printed under the 2d barcode).

It does raise the question of how (if) one should notify yubico if they're passed on - especially if I overwrite the yubi profile (as I have done) to prove that I am the rightful owner of the key.

Things to consider with revocation / blocking, especially with a distributed architecture like Fedora....

puppet /. facter / v12n

As I'm starting to get into puppet, especially when used with the foreman for dashboard display, I've started noticing a few (fixable) oddities

1st up ios that not all RHEL clones are treated equally -- some patches are just adding detection and flaggin, others go the full hog and make sure its in all the "constrain:" sections in facter too. I guess I need to do a code review and pull in the extras. Cue lots of different VMs
(the joys of a weeks home leave from work..)

secondly, virtualisation detection is 'flakey' -- some (vmware) are really well detected, but new kernels give false positives (see issue #7723) so I'm going to set about tidying these up and adding hyper-V detection properly

Bash, PS1, PROMPT_COMMAND and other fun

I've just spent 30 mins trying to understand the flow between the various files that set a bash prompt on fedora to do the following: (assumption is here that you're in a colour xterm)

I want:
* my username to change colour depending if I still have a valid kerberos token
* the hostname of the machine I'm on to be in RED if I'm root
* displayed path to be as simple as possible
* $ if I'm a minion, # if root as mormal
* command line editing to work sensibly, no wierdisms on wrapping long lines

so - I used to have on my machine something like the following:

if [ "$PS1" != "" ] ; then
 klist -s 
 if [ $? -eq 0 ] ; then
 PS1='\[\033[32m\]\u@\h\[\033[0m\]:\w\$ '
 else
 PS1='\[\033[36m\]\u\[\033[32m\]@\h\[\033[0m\]:\w\$ '
 fi
fi

and something similar for root.

But on my laptop (F14) I wanted it system wide, so went down the approach of customising /etc/bashrc where you find calls to
PROMPT_COMMAND=/etc/sysconfig/bash-prompt-xterm

1) /etc/sysconfig/bash-prompt-* aren't included, so you're on your own
2) It must point to an executable script that is run every time before displaying the prompt
3) PS1 is still displayed *AND* if you use tab completion / ctrl-l, its *only* PS1 thats displayed on your screen, not the output from PROMPT_COMMAND

so: DON'T do the following:

cat /etc/sysconfig/bash-prompt-xterm 
#!/usr/bin/env bash 
# set green username if we have a valid kerberos token, else cyan
klist -s
if [ $? -eq 0 ] ; then
 K='[32m'
else
        K='[36m'
fi
# set hostname in red if we're root, green otherwise
if [ ${USER} = 'root' ] ; then
 U='[31m'
else
 U='[32m'
fi

printf "^[%s%s^[[37m@^[%s%s^[[0m:%s " ${K} ${USER} ${U} ${HOSTNAME%%.*} "${PWD/#$HOME/~}"
#echo -ne "\033]0;${USER}@${HOSTNAME%%.*}:${PWD/#$HOME/~}\007"

(where ^[ is 'ctrl-v, esc' in vim)
because you end up with stuff like

aelwell@pcitgtelwell:~[aelwell@pcitgtelwell ~]$

until you press ctrl-l and end up with just
[aelwell@pcitgtelwell ~]$ (ie, $PS1)


but *DO* make the call to see if you have a valid token and set the xterm titlebar in /etc/sysconfig/bash-prompt-xterm, but if you're altering PS1, then do so in the traditional places of /etc/bashrc and (as suggested in that file) a custom modification shell script in /etc/profile.d/ directory.

Ho Hum. Hope this clears up for anyone else trying to work out what the sysconfig/bash-prompt-* files do.

oh, and does anyone know a lighter call to see if a token is still valid than 'klist -s'?

Daily WTF Candiate

(Credit for spotting this one goes over to the fine folks at scotgrid who were bitten by the fallout.

Anyway, Useless Use of Cat^H^H^HMore award goes to http://bit.ly/dI2W3c

The saga of SAGA

for anyone who wishes to install a glite-WN without the overhead of SAGA, the magic voodoo on the command line is

yum --enablerepo dag groupinstall glite-WN --exclude \*saga\*

Makefile Faffage

I've inherited a piece of legacy code (an inirscript that needed bringing up to FHS and Fedora packaging guidelines) -- as part of the testing I realised I needed a noddy way to generate the tar.gz sources. There was already a Makefile in the package (for ETICS) but I hacked in the following to quickly build up a NVR-tarfile.

Not pretty, but if anyone fancies some Makefile golf, I'm open to suggestions:

dist:
echo ${package}-`sed \
-e '/^Version:/!d' \
-e 's/[^0-9.]*\([0-9.]*\).*/\1/' \
-e q \
${package}.spec` > .fname
-rm -rf `cat .fname`
cp -lvr src/ `cat .fname`
tar chzf `cat .fname`.tar.gz `cat .fname`
-rm -rf `cat .fname` .fname

and yes, it bears a striking resemblance to the GNU Complex Makefile Example

Design vs Reality

Decided to check that savannah was behaving as it should be. As with any good process we have a nice flowchart that depicts the allowed transition states.

ie:

however, thanks to a quick 'copy n paste' into graphviz it appears the reality is a little more complicated. Ho hum, cleanup time....




update: Just tidied up savannah and lo:

Studies on a hashtag

I'm at the EGEE09 conference this week, and they're also plugging use of tagging in social networking such as the #egee hashtag in twitter and the 'egee09' tag in flickr. As a small experiment I created an IRC channel on freenode (#EGEE09) and announced it with a hashtag on twitter. Within 3 minutes there were an additional 3 users, only one of whom was in my normal twitter cloud, ie the other 2 came in purely from the hashtag (and one of those was under a minute later)

so, twitter is certainly becoming a respectable way of broadcasting announcements to an interested group of people who are subscribing to a hashtag.

oh and as an aside, the IRC channel paid off -- I discovered that GridPP had some nice laptop bags via it and secured one for myself during coffee.

Anyway, back to more important work -- listening to how the EU proposes to carve up the funding for call7

BeautifulSoup + savannah

At work, we use savanne to power http://savannah.cern.ch where we keep track of all the glite-middleware (amongst other things)

However, one of the problems with savanne/savannah is that it doesn't have a nice machine readable API, but requires the user to click with a browser.

Cue Beautiful Soup and lo, the python SavannahCLI was born.

Well, in its early years it was unloved and handed round from developer to developer, until it arrived as a pedantic teenager to yours truly who decided to get rid of the horrible 'operation, item, name, value" syntax and replace with something more, um, fluid instead. (--patch 1234 --set foo=bar). It also had a huge set of hard coded lookup tables for converting select items to names and vice versa -- WHY?

thanks to some googling, I came across this which, with a little refinement gave me:

def enum_forms():
  # we search within the table tag so that we lose the LHS searchbox
  selects = soup.find('table').findAll('select')
  for sel in selects:
     sel_name = sel['name']
     print "SELECT NAME=%s" % sel_name 
     for o in sel.findAll('option'):
         print "  %s, %s" % (o['value'],o.renderContents())

hey, it's not finished yet, but sure saves a whole pile of hard-coded badness.