BaloneyGeek's Place

BaloneyGeek's Place

Operator! Give me the number for 911!

KDE Infrastructure on DigitalOcean

KDE's server inventory is a mixed bag. We have a few physical machines that were donated to us. There's some sponsored colocation. We also rent a couple of big machines from Hetzner, divvy them up into smaller containers with lxc and host services there. Today, I can announce that we're adding droplets from DigitalOcean to that bag.

Not only would a full-blown cloud infrastructure on something like AWS be prohibitively expensive, our situation doesn't merit such an infrastructure. Our more powerful servers are dedicated as build slaves for our CI system, and a server with 32GB of RAM and dual-redundant SSDs with ZFS on Linux based storage is currently used for the code repositories, and will soon be used to host Phabricator.

We could, however, do with something in between cloud "compute" resources and a physical server that we manually manage, and DigitalOcean's droplets fit the bill right there. DO's droplets are small - we can dedicate a 1GB droplet to hosting websites, which would allow us to isolate web hosting from other services while not wasting resources we'd never use. They're also standard KVM machines, which allows us the level of manual control we'd like.

There are some additional aspects of using DigitalOcean that I like:

  • All our existing servers are either in continental Europe or the United States. DigitalOcean has datacenters in Asia that I'm particularly looking forward to making use of, to service our contributors from Asia Pacific (particularly India) better.
  • Depending on demand, we could bring up new servers or shut down existing ones at short notice. While we don't do something along these lines now, once we have the capability I could see us pre-emptivly adding temporary server capacity to handle high-traffic events, like a new Krita release.

But this isn't the best part of this post.

Once we realised we'd have some use for DigitalOcean's offerings, we went and asked them if they'd be willing to sponsor us under their programme for supporting open-source software projects. To our utter delight, they were very enthusiastic about supporting us and set us up with an account and a lot of free credits to start us out.

So in the next few months, expect to have KDE's existing online services to get more reliable as we add failovers, and new services to spring up as we start putting plans for the additional capacity into action.

Till next time!

Message Passing - Our Telegram-IRC Relay Service

Yesterday, we launched a service to relay messages between IRC and Telegram, and we're syncing 4 channels for KDE and 5 more for Kubuntu at this moment. The sync is two-way, so whatever people say on Telegram appears on the IRC channel, and vice versa. Almost everything works - almost being everything except that files and stickers shared on Telegram don't appear on IRC.

So here's a blog post about how we did it. An exclusive behind the scenes look at how KDE Sysadmin conducts their business, if you will.

Server Set-Up

The server that runs IRC services (a bouncer, bots for Zabbix, Bugzilla etc.) is an LXC container running Ubuntu 14.04.4 LTS. We don't use docker or other application container services to virtualise apps - since we run such a diverse set of services written in different languages on the server, we just confine apps to their own users, and try to keep the dependencies confined to the user as far as possible.

To run the Telegram-IRC relay, we chose to use TeleIRC. TeleIRC fits our bill perfectly. It has all the features we want. It's also written in JavaScript, which means it comes from the same community that brought us examples of groundbreaking engineering such as left-pad and is-positive-integer. Isolating this service from the rest of the system is critical, at least from a security perspective.

Step 1: Node.js

Ubuntu 14.04.4 LTS carries Node.js version 0.10 in its repositories. It's too old and won't do, as we found out much after actually installing teleirc, because npm won't complain when you're installing but teleirc will refuse to start because os has no method homedir().

So we'll have to manually obtain a current version of Node.js. Thankfully, we can use the Node Version Manager to obtain a binary build of a current version of Node.js directly from the Node website.

Start with a fresh user - let's call it teletubby - and log in. Because it's Node, the "recommended" method for installing nvm is by curl-ing a script and piping it to bash. It's an idea that's brilliantly simple and has provably zero security flaws. Of course, because we're KDE and we like to do things the hard way, I decided to forego this easy install procedure and do things manually:

teletubby (~) $: mkdir TeleIRC
teletubby (~) $: cd TeleIRC
teletubby (~/TeleIRC) $: git clone https://github.com/creationix/nvm.git

At this point it'll clone the nvm repository to ~/TeleIRC/nvm. If you're feeling particularly adventurous, you can use this as-is (using the current master), but I like to stay on the stable branch. nvm's repo makes that easy - all you need to do is to checkout the latest stable tag:

teletubby (~/TeleIRC/nvm) $: git checkout `git describe --abbrev=0 --tags`

At this point, you'll need to add the nvm set-up script to your Bash profile, so add the following lines to ~/.profile:

export NVM_DIR="$HOME/TeleIRC/nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

Log out and log back in (or start a new login shell), and you should be able to use the nvm command right away.

You'll actually have to install a version of Node.js now. The current LTS branch of Node.js is the v4.4 branch (the current release as of writing is v4.4.4). You can do:

teletubby (~) $: nvm ls-remote

to see what versions of Node.js are available for you to install. To install the 4.4 branch, just do:

teletubby (~) $: nvm install v4.4

And when it's done installing, run node to make sure you get a Node.js prompt.

Step 2: Telegram Bot and IRC Account

To get the Telegram Bot account, you'll have to talk to @BotFather.

Start by asking for a new bot:

/newbot

BotFather is pretty conversational. It'll first ask you for the full name you want to give to the bot, and then an username. Our service is called KDE IRC Relay Service, and the username is IrcsomeBot. Note that your bot's username must end with either bot or _bot.

Once you give it a full name and an username, it'll give you an API key. Keep this key secure. If you lose it, you can generate a new one, but make sure it's never compromised.

To let your bot see every message that's said in the group (so that it can read the messages and relay them to IRC), you'll have to disable privacy for the bot:

<Me>: /setprivacy
<BotFather>: Choose a bot to change group messages settings.
<Me>: @IrcsomeBot
<BotFather>: 'Enable' - your bot will only receive messages that either start with the '/' symbol or mention the bot by username.
<BotFather>: 'Disable' - your bot will receive all messages that people send to groups.
<BotFather>: Current status is: ENABLED
<Me>: Disable
<BotFather>: Success! The new status is: DISABLED. /help

Note that to denote the channel now, we're using the notation @username, not just username. We'll prefix the username with the @ sign everywhere we want to refer to the bot from now on.

All the essential Bot account set-up is complete, so it's time to move on to the IRC account creation. This differs from IRC network to IRC network, but for Freenode it's pretty simple. Start by picking a nickname that your bot will use, logging on to chat.freenode.net with any IRC client as said nickname, and running the following command at the server window:

/msg NickServ REGISTER password email@example.com

You'll get an email from Freenode with another /msg command you'll need to type to confirm your registration. Do that, and you're done.

Step 3: TeleIRC

Once node is up and running, and you've got your accounts, it's time to install TeleIRC.

Installing TeleIRC is incredibly easy. Just run:

teletubby (~/TeleIRC) $: npm install teleirc

And check that ~/TeleIRC/node_modules/teleirc/bin/teleirc exists. At this point, you might want to add $HOME/TeleIRC/node_modules/.bin to your PATH environment variable, so that you can run the teleirc command without prefixing it with a path.

The first thing to do is to generate a configuration file, and edit it. Generate the sample config by running:

teletubby (~/TeleIRC) $: teleirc --genconfig

This will create a sample configuration file and drop it at ~/.teleirc/config.js. Edit it with your favourite text editor. It's well commented, so you shouldn't have any problem setting things up. The Telegram API Key goes into a variable defined near the top of the file, and the IRC settings go towards the bottom.

For Freenode, the default IRC config generated doesn't have spaces for a nick and a password, so refer to the snippet below for what to set:

config.ircOptions = {
    userName: 'FreenodeUserName',
    realName: 'The Real Name',
    nick: 'yournick',
    password: 'yourpassword',
    port: 7000,
    secure: true,
    sasl: true
};

Freenode allows SSL connections over port 7000, and you'll need to have SASL enabled because otherwise TeleIRC won't be able to authenticate your nick and password.

Of course, don't forget to actually set up the channel-group mappings.

Once you're done, start teleirc by running:

teletubby (~/TeleIRC) $: teleirc

Nothing should bomb, except a bunch of warning about chat_ids not being found. That's not an error. Now just go to every group where you want the bot to be present, add the bot's username and say something. The relay should just start working.

Step 4: Running Forever

It turns out you can actually do this without messing with your init scripts. All you need is another npm package called forever, and Cron. Yes, Cron.

Start by installing forever:

teletubby (~/TeleIRC) $: npm install forever

Check if it runs:

teletubby (~/TeleIRC) $: forever start `which teleirc`
teletubby (~/TeleIRC) $: forever list

You should see a table with an entry for teleirc, along with a path to the log file, which you can cat to read teleirc's output. The entry should also have a number associated with it (0, if it's the only service running). You can control it by running:

teletubby (~/TeleIRC) $: forever restart 0   # to restart the service
teletubby (~/TeleIRC) $: forever stop 0      # to stop the service

Once you've verified that things run, and you can control the process, it's time to make sure that the service runs at system start. Start by creating a script with the following content:

#!/bin/sh
forever start `which teleirc`

Then, run:

teletubby (~/TeleIRC) $: crontab -e

Your text editor should open with a crontab file. You'll need to add one line to the bottom:

@reboot /full/path/to/your/start/script.sh

And make sure that the cron service is enabled. That's it!

In Conclusion

So the service is up and running, and any KDE IRC channel that wants to be mirrored with a Telegram group just needs to file a Sysadmin Task on our Phabricator instance.

I hope this helped you out if you're looking for ways to bridge your IRC and Telegram channels together. Until next time!

The Sysadmin Squad

It's been close to a year since I started contributing to KDE long term, and since then a lot has happened.

Spectacle is out as part of the Applications suite as of KDE Applications 15.12, and not only do the users like it, other devs love it too. Maintaining Spectacle isn't as hectic as it used to be in the early days. I get a steady stream of mostly benign bug reports, which is nothing I can't handle in a few hours over the weekend. Wayland support still hasn't materialised though, and it looks like it'll be a while. Proper High-DPI (retina displays) support did materialise though, courtesy of Kai Uwe Broulik, and will land in 15.12.1.

Something I've come to realise, however, is that day-by-day I'm gravitating over to the Sysadmin side.

The Project Predicament

KDE's online infrastructure is gigantic, no small part of which is taken up by Git servers. I'm not sure of the exact numbers, but a ballpark figure for the number of repositories we host would be in the 2,000-2,500 region.

We're also in the middle of a migration. KDE's Project Management Website runs on ChilliProject. ChilliProject is slow and unwieldy, but that we could make do with. What we can't make do with is that ChilliProject is deprecated and is no longer developed. There are bugs and security holes and we've had to disable chunks of functionality to mitigate these.

So we're migrating to Phabricator. We already have a Phabricator instance running, and quite a few projects have already been migrated. So far our experience has been good, and the Phabricator documentation is pretty good. And pretty funny (seriously, you should see their website).

Let's Git Overhaulin'

Back in September, we set up a read-only mirror of KDE's public project repositories on GitHub. d_ed liaised with GitHub and got is the organization username, and then I got busy mucking around with the post-recieve hook in our git.kde.org repositories.

The scripting itself was pretty simple. bcooksley had already written a small python script that git push-ed only branches and tags to upstream servers, so it was just a matter of executing that with the GitHub remote URL.

Now KDE has its own infrastructure of read-only git mirrors which mirror every repository, not just the public project ones. They're the Anongit servers, a network of (currently) 6 such servers spread worldwide which developers can clone from. They're available at anongit.kde.org.

Since we already mirror code from git.kde.org, why couldn't we just add the GitHub remote URL to the list of servers to mirror to? Because we don't push to the Anongit servers, we tell them (via a HTTP hook) that git.kde.org has been updated, please pull changes.

Now there isn't anything inherently wrong with this approach; it's just that we have no control over what happens if one of the Anongit servers go down. We'd ideally want to log failed mirroring attempts, and to do that effectively we'd need to push to our mirrors and log failures (and maybe retry if a mirroring attempt failed).

Propagator

Sysadmins hold the keys to much of KDE's infrastructure - user accounts, passwords, SSH keys, you name it, we have it. Understandably, we don't just let anyone be a part of the Sysadmin team. But the truth is, KDE's Sysadmin team is severely understaffed and the list of tickets to attend to is miles long. We need all the help we can get.

That said, you don't need to be a part of the core Sysadmin team to be able to help them out. And Sysadmin work isn't just SSH-ing into servers and running commands - there's plenty of coding to be done, and that work doesn't need access to the keys.

So, Propagator. Over the winter holidays, I got to work on a small-ish daemon that would manage our entire distributed git infrastructure - and alleviate the Sysadmins' need to ever touch the Anongit servers themselves.

Propagator is written in Python 3, and uses a Celery task queue to mirror-push changes to our Anongit servers. We also wanted to converge code-paths for pushing to GitHub, so Propagator does that too. Propagator also creates, sets the descriptions on, moves and deletes repositories on the Anongit servers via a SSH API, and on GitHub via their REST API.

In fact, with Propagator, managing repositories is as easy as creating them inside Phabricator. Propagator will automatically create repositories on the upstream servers (both Anongit servers and GitHub) on first push if they don't exist, effectively making things zero-administration.

What about moves, renames, and the rest? Well, you log in to a shell on git.kde.org and run:

$: mirrorctl create AwesomeRepo.git "Awesome app to do something super cool"
$: mirrorctl rename AwesomeRepo.git AwesomeApp.git
$: mirrorctl delete AwesomeApp.git

...and so on. Propagator will not send out requests to every single configured upstream server and tell it to carry out these commands. Easy as pie.

Unfortunately, Propagator now lives inside the repo-management repo in KDE, but I do plan to separating it out into its own repository and distributing it as a standalone product in its own right as a KDE project.

Closing Notes

KDE is awesome. No, it really is.

We see a lot of newcomers pop up on KDE's IRC channels and mailing lists and ask how to contribute. For the record, we appreciate code, documentation, artwork, translations, evangelism and publicity (but do refrain from streaking across campus with a giant KDE tattoo... *wink wink*).

And we need Sysadmins. If you have a particular set of skills that make it a nightmare for people like us to have problems with our online infrastructure, we want you.