Thoughts on Statuspage.io integrations

 $dayjob uses statuspage.io for our public status page. Although I've got some things automated, there's still a bunch of components that need manual updating when there's an issue (enough to keep me out of mischief working out how to automate them without false alerts anyway). However this post isn't for getting info _in_ to statuspage, but for how to get it _out_, and what I want.

We're predominantly a command line shop (no, we don't yet have a JupyterHub frontend), so users are presented with a MOTD on login. Now, wouldn't it be good if that could be updated automatically with details of upcoming planned outages as well as any recent (and current) incidents that affected the service you're currently logged into?

So - Armed with the API, it should be possible to get upcoming maintenance[✓] and the impacted components[✓], but where do I map the cluster name to the statuspage group_id? or, for that matter any of the autogenerated id strings. Hard coding them into scripts is out, a lookup makes sense but how many CMDBs out there come with that sorta functionality built in. and we're back to another 'where is my source of truth?' problem. Sure I can string match components->name and check that "group": true and then pull the ID, but... yeah faffy. 

Anyway, after much parsing (all hail requests) it's possible to get the various incidents/component states/planned maintenance out and the resulting text snippets touched to the correct updated_at timestamp whereupon they can be pulled into the final output whenever the files are regenerated by a Makefile build under Jenkins. The goal of DRY is starting to be achieved by updating one place (currently statuspage) and having that trigger a build via webhook which then distributes the info out to the clusters.

I think a former colleague summed it up fairly well tho.

Fedora / Netatalk / OS X Lion / TimeMachine

Being the cheapskate that I am, I'm not buying a shiny apple-branded time-capsule for backups when I have a perfectly stylish NAS (since the pic was taken I'm using the via-eden board there's a 1.5TB HDD in the box)

I'd previously used netatalk with leopard under mythbuntu, but following a clean-up and migration to Fedora 16 (Verne) it needed reinstalling (esp as we upgraded to Lion on some of the macs)

So, a HOWTO if anyone is hunting for this and some notes

• You don't need avahi separately anymore - new netatalk includes it
• You need to allow tcp/548 in your iptables rules (I added to /etc/sysconfig/iptables)
  -A INPUT -p tcp -m state --state NEW -m tcp --dport 548 -j ACCEPT
• SELinux. Yeah. probably needs fixing but 'setenforce permissive' worked :-/
• I'm not convinced you need the 'defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1' anymore. 

Here's what I did

1) Create a separate disk partition for time machine and mount it (I'm using LVM and XFS)

$> grep time /etc/fstab
/dev/mapper/linuxvg-timemachinelv /export/timemachine xfs defaults 1 2
$> df -h /export/timemachine
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/linuxvg-timemachinelv  200G  6.0G  194G   3% /export/timemachine

2) create a separate user for the backups (not strictly needed but I chowned /export/rimemachine to that user to sort out permissions

#> useradd timelord
#> passwd timelord
#> chown timelord: /export/timemachine

3) Install and configure netatalk (although 2.2.0 is in Fedora 16, I decided to use the rawhide version of 2.2.2)

#> yum --enablerepo rawhide install netatalk

The config files are in /etc/netatalk and (see the gude at An Esurient Trollop ) you'll need to edit afpd.conf
(I added -mimicmodel Macmini and uams_guest.so to -uamlist: I have a ro media share)

- -tcp -noddp -mimicmodel Macmini -uamlist uams_guest.so,uams_dhx.so,uams_dhx2.so -nosavepassword

and AppleVolumes.default

# Share out the Videos
/export/media/Video Videos options:ro,noadouble
#Time Machine
/export/timemachine TimeMachine options:tm

Startup netatalk

#> systemctl start netatalk.service
#> systemctl enable netatalk.service

 So far so good -- assuming no failures you should see something like this in your logs

afpd[28742]: AFP/TCP started, advertising 192.168.1.251:548 (2.2.2)

Now, onto the mac -- I wasn't seeing the host automatically appear in the finder, but a simple command-k (connect to server) and typing the hostname worked (it expands to afp://hostname automatically), and was prompted for my (normal) username and password to connect to $HOME on the netatalk server.

In the time machine preferences I could then select the remote TimeMachine volume on the NAS, enter my 'timelord' username/password combo and it started to so a backup.

More news (and a rest restore) to follow...

Bash, PS1, PROMPT_COMMAND and other fun

I've just spent 30 mins trying to understand the flow between the various files that set a bash prompt on fedora to do the following: (assumption is here that you're in a colour xterm)

I want:
* my username to change colour depending if I still have a valid kerberos token
* the hostname of the machine I'm on to be in RED if I'm root
* displayed path to be as simple as possible
* $ if I'm a minion, # if root as mormal
* command line editing to work sensibly, no wierdisms on wrapping long lines

so - I used to have on my machine something like the following:

if [ "$PS1" != "" ] ; then
 klist -s 
 if [ $? -eq 0 ] ; then
 PS1='\[\033[32m\]\u@\h\[\033[0m\]:\w\$ '
 else
 PS1='\[\033[36m\]\u\[\033[32m\]@\h\[\033[0m\]:\w\$ '
 fi
fi

and something similar for root.

But on my laptop (F14) I wanted it system wide, so went down the approach of customising /etc/bashrc where you find calls to
PROMPT_COMMAND=/etc/sysconfig/bash-prompt-xterm

1) /etc/sysconfig/bash-prompt-* aren't included, so you're on your own
2) It must point to an executable script that is run every time before displaying the prompt
3) PS1 is still displayed *AND* if you use tab completion / ctrl-l, its *only* PS1 thats displayed on your screen, not the output from PROMPT_COMMAND

so: DON'T do the following:

cat /etc/sysconfig/bash-prompt-xterm 
#!/usr/bin/env bash 
# set green username if we have a valid kerberos token, else cyan
klist -s
if [ $? -eq 0 ] ; then
 K='[32m'
else
        K='[36m'
fi
# set hostname in red if we're root, green otherwise
if [ ${USER} = 'root' ] ; then
 U='[31m'
else
 U='[32m'
fi

printf "^[%s%s^[[37m@^[%s%s^[[0m:%s " ${K} ${USER} ${U} ${HOSTNAME%%.*} "${PWD/#$HOME/~}"
#echo -ne "\033]0;${USER}@${HOSTNAME%%.*}:${PWD/#$HOME/~}\007"

(where ^[ is 'ctrl-v, esc' in vim)
because you end up with stuff like

aelwell@pcitgtelwell:~[aelwell@pcitgtelwell ~]$

until you press ctrl-l and end up with just
[aelwell@pcitgtelwell ~]$ (ie, $PS1)


but *DO* make the call to see if you have a valid token and set the xterm titlebar in /etc/sysconfig/bash-prompt-xterm, but if you're altering PS1, then do so in the traditional places of /etc/bashrc and (as suggested in that file) a custom modification shell script in /etc/profile.d/ directory.

Ho Hum. Hope this clears up for anyone else trying to work out what the sysconfig/bash-prompt-* files do.

oh, and does anyone know a lighter call to see if a token is still valid than 'klist -s'?

ssh-copy-id

I normally have my ~/.ssh/config so that it only uses my key or gssapi to login to other sites, however this poses a problem when reinstalling a server and don't yet have my key on it

ssh -F /dev/null root@hostname

works, but this wasn't working with ssh-copy-id.

Turns out that the ssh-copy-id script is braindead and only expects one or two (with the 1st beginning with -i) arguments.

lo and behold, some single quotes did the trick:

aelwell@pcitgtelwell:~$ ssh-copy-id '-F /dev/null root@concerto'
Are you sure you want to continue connecting (yes/no)? yes
root@concerto's password:
Now try logging into the machine, with "ssh '-F /dev/null root@concerto'", and check in: ....

tada!