Brexit Is Not Happenning

David Cameron is an astounding genius. Of course, you’d expect nothing less from an alumni of Eton and Oxford, but to see his genius in action in such a grand public scale makes it no less incredible.

This meme was doing the rounds on Facebook yesterday. But I beg to differ - this wasn’t a rage-quit. This was a chess move so calculated and so devastating that the full extent of its consequences will take a long time to become apparent.

Biggest Rage Quit of 2016

With his resignation, David Cameron ensured that the person who succeeds him is going to commit political suicide. And someone will have to succeed him, because someone will have to become the next Prime Minister of the United Kingdom. And that person will have to die. Whether or not that person is Boris Johnson remains to be seen.

Here’s the deal. The referendum was set up to be advisory, not binding. The British Parliament is under no legal obligation to follow through on the results on the referendum, but if the government does not follow up, the entire premise of democracy falls apart. Therefore, Britain has to leave the EU, and to do that, someone will have to inform the European Commission by sending them a notice under Article 50 of the Lisbon Treaty. That someone was going to be David Cameron, until he decided to resign and let his successor do it.

On the face of it, that would be an honour for his pro-Leave successor. But of course, things aren’t so simple:

  • If the successor follows through and invokes Article 50, Scotland and Northern Ireland breaks away and joins the EU. The United Kingdom is no longer united. A mountain of laws and regulations need to be torn up and new ones written in its place. The English economy collapses. The public quickly loses patience, and the blood is now on the successor’s hands. His career is over, as is the United Kingdom as we know it.

  • If the successor does not follow through and fails to invoke Article 50, the premise of the democratic government falls apart. The governance of the UK is no longer democratic, and the entire establishment is a farce. The successor’s career is over, as is probably the entire government’s. It doesn’t end there, however. The next person on the chair also finds himself in the same conundrum, as the next. Until the will of the British people change and they decide to stay - and make it known in another referendum - this cycle continues.

It appears I’m not alone in this theory. Someone commented exactly along these lines on a Guardian article, which is what led me to think twice - hey, I thought this line of thought was an effect of too much House of Cards, but it appears I might not be completely crazy after all - and take this out of my mind and put it to paper. Well, the Internet.

The next few months are going to be very interesting.

Message Passing - Our Telegram-IRC Relay Service

Yesterday, we launched a service to relay messages between IRC and Telegram, and we’re syncing 4 channels for KDE and 5 more for Kubuntu at this moment. The sync is two-way, so whatever people say on Telegram appears on the IRC channel, and vice versa. Almost everything works - almost being everything except that files and stickers shared on Telegram don’t appear on IRC.

So here’s a blog post about how we did it. An exclusive behind the scenes look at how KDE Sysadmin conducts their business, if you will.

Server Set-Up

The server that runs IRC services (a bouncer, bots for Zabbix, Bugzilla etc.) is an LXC container running Ubuntu 14.04.4 LTS. We don’t use docker or other application container services to virtualise apps - since we run such a diverse set of services written in different languages on the server, we just confine apps to their own users, and try to keep the dependencies confined to the user as far as possible.

To run the Telegram-IRC relay, we chose to use TeleIRC. TeleIRC fits our bill perfectly. It has all the features we want. It’s also written in JavaScript, which means it comes from the same community that brought us examples of groundbreaking engineering such as left-pad and is-positive-integer. Isolating this service from the rest of the system is critical, at least from a security perspective.

Step 1: Node.js

Ubuntu 14.04.4 LTS carries Node.js version 0.10 in its repositories. It’s too old and won’t do, as we found out much after actually installing teleirc, because npm won’t complain when you’re installing but teleirc will refuse to start because os has no method homedir().

So we’ll have to manually obtain a current version of Node.js. Thankfully, we can use the Node Version Manager to obtain a binary build of a current version of Node.js directly from the Node website.

Start with a fresh user - let’s call it teletubby - and log in. Because it’s Node, the “recommended” method for installing nvm is by curl-ing a script and piping it to bash. It’s an idea that’s brilliantly simple and has provably zero security flaws. Of course, because we’re KDE and we like to do things the hard way, I decided to forego this easy install procedure and do things manually:

teletubby (~) $: mkdir TeleIRC
teletubby (~) $: cd TeleIRC
teletubby (~/TeleIRC) $: git clone https://github.com/creationix/nvm.git

At this point it’ll clone the nvm repository to ~/TeleIRC/nvm. If you’re feeling particularly adventurous, you can use this as-is (using the current master), but I like to stay on the stable branch. nvm‘s repo makes that easy - all you need to do is to checkout the latest stable tag:

teletubby (~/TeleIRC/nvm) $: git checkout `git describe --abbrev=0 --tags`

At this point, you’ll need to add the nvm set-up script to your Bash profile, so add the following lines to ~/.profile:

export NVM_DIR="$HOME/TeleIRC/nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

Log out and log back in (or start a new login shell), and you should be able to use the nvm command right away.

You’ll actually have to install a version of Node.js now. The current LTS branch of Node.js is the v4.4 branch (the current release as of writing is v4.4.4). You can do:

teletubby (~) $: nvm ls-remote

to see what versions of Node.js are available for you to install. To install the 4.4 branch, just do:

teletubby (~) $: nvm install v4.4

And when it’s done installing, run node to make sure you get a Node.js prompt.

Step 2: Telegram Bot and IRC Account

To get the Telegram Bot account, you’ll have to talk to @BotFather.

Start by asking for a new bot:

/newbot

BotFather is pretty conversational. It’ll first ask you for the full name you want to give to the bot, and then an username. Our service is called KDE IRC Relay Service, and the username is IrcsomeBot. Note that your bot’s username must end with either bot or _bot.

Once you give it a full name and an username, it’ll give you an API key. Keep this key secure. If you lose it, you can generate a new one, but make sure it’s never compromised.

To let your bot see every message that’s said in the group (so that it can read the messages and relay them to IRC), you’ll have to disable privacy for the bot:

<Me>: /setprivacy
<BotFather>: Choose a bot to change group messages settings.
<Me>: @IrcsomeBot
<BotFather>: 'Enable' - your bot will only receive messages that either start with the '/' symbol or mention the bot by username.
<BotFather>: 'Disable' - your bot will receive all messages that people send to groups.
<BotFather>: Current status is: ENABLED
<Me>: Disable
<BotFather>: Success! The new status is: DISABLED. /help

Note that to denote the channel now, we’re using the notation @username, not just username. We’ll prefix the username with the @ sign everywhere we want to refer to the bot from now on.

All the essential Bot account set-up is complete, so it’s time to move on to the IRC account creation. This differs from IRC network to IRC network, but for Freenode it’s pretty simple. Start by picking a nickname that your bot will use, logging on to chat.freenode.net with any IRC client as said nickname, and running the following command at the server window:

/msg NickServ REGISTER password [email protected]

You’ll get an email from Freenode with another /msg command you’ll need to type to confirm your registration. Do that, and you’re done.

Step 3: TeleIRC

Once node is up and running, and you’ve got your accounts, it’s time to install TeleIRC.

Installing TeleIRC is incredibly easy. Just run:

teletubby (~/TeleIRC) $: npm install teleirc

And check that ~/TeleIRC/node_modules/teleirc/bin/teleirc exists. At this point, you might want to add $HOME/TeleIRC/node_modules/.bin to your PATH environment variable, so that you can run the teleirc command without prefixing it with a path.

The first thing to do is to generate a configuration file, and edit it. Generate the sample config by running:

teletubby (~/TeleIRC) $: teleirc --genconfig

This will create a sample configuration file and drop it at ~/.teleirc/config.js. Edit it with your favourite text editor. It’s well commented, so you shouldn’t have any problem setting things up. The Telegram API Key goes into a variable defined near the top of the file, and the IRC settings go towards the bottom.

For Freenode, the default IRC config generated doesn’t have spaces for a nick and a password, so refer to the snippet below for what to set:

config.ircOptions = {
userName: 'FreenodeUserName',
realName: 'The Real Name',
nick: 'yournick',
password: 'yourpassword',
port: 7000,
secure: true,
sasl: true
};

Freenode allows SSL connections over port 7000, and you’ll need to have SASL enabled because otherwise TeleIRC won’t be able to authenticate your nick and password.

Of course, don’t forget to actually set up the channel-group mappings.

Once you’re done, start teleirc by running:

teletubby (~/TeleIRC) $: teleirc

Nothing should bomb, except a bunch of warning about chat_ids not being found. That’s not an error. Now just go to every group where you want the bot to be present, add the bot’s username and say something. The relay should just start working.

Step 4: Running Forever

It turns out you can actually do this without messing with your init scripts. All you need is another npm package called forever, and Cron. Yes, Cron.

Start by installing forever:

teletubby (~/TeleIRC) $: npm install forever

Check if it runs:

teletubby (~/TeleIRC) $: forever start `which teleirc`
teletubby (~/TeleIRC) $: forever list

You should see a table with an entry for teleirc, along with a path to the log file, which you can cat to read teleirc‘s output. The entry should also have a number associated with it (0, if it’s the only service running). You can control it by running:

teletubby (~/TeleIRC) $: forever restart 0   # to restart the service
teletubby (~/TeleIRC) $: forever stop 0 # to stop the service

Once you’ve verified that things run, and you can control the process, it’s time to make sure that the service runs at system start. Start by creating a script with the following content:

#!/bin/sh
forever start `which teleirc`

Then, run:

teletubby (~/TeleIRC) $: crontab -e

Your text editor should open with a crontab file. You’ll need to add one line to the bottom:

@reboot /full/path/to/your/start/script.sh

And make sure that the cron service is enabled. That’s it!

In Conclusion

So the service is up and running, and any KDE IRC channel that wants to be mirrored with a Telegram group just needs to file a Sysadmin Task on our Phabricator instance.

I hope this helped you out if you’re looking for ways to bridge your IRC and Telegram channels together. Until next time!

Summer Is Coming

It’s been a long time since I blogged - it’s already nearly 5 months into the year and I haven’t put any words to the keyboard yet - and I have a metric tonne of things to share.

I’ll just jump right in.

conf.kde.in

CKI was a hoot and a half this year. The Lakshmi Narayan Mittal Institute of Information Technology - in Jaipur - hosted India’s very own Akademy in the first week of March. Because I stay just two hours away from Jaipur, I just had to be there, and so proposed two talks. By February both had been accepted, and it suddenly dawned on me that a year after starting to contribute to KDE I’d finally meet some of the people - whom I only knew from their IRC nicks - in person.

CKI turned out to be a mixed bag for me. Both the talks went incredibly well. Or so I hear - I was drugged out of my mind because I fell ill the moment I reached the venue. My memory of the event is just limited to the talks themselves, a hurried photoshoot (because the sun was very very roasty), and a visit to the medical center on campus where they confirmed I had an elevated blood pressure, a temperature of 104.5 degrees Fahrenheit, and a racing pulse. Of the two and a half days I spent giving talks and meeting so many people, I seem to remember only 3 hours worth of events. Which is surprising, since I only slept for a total of 3 hours over the entire weekend. Yes, I had insomnia too, because the fever and the hypertension clearly weren’t enough.

Quick change of plans, an early morning double-decker train ride to Delhi, a last-minute first-class ticket on the Calcutta Rajdhani Express, and post-event I race home to get checked by specialist doctors. It turns out I have a pretty severe upper respiratory tract infection. I live to tell the tale.

Sagar, Shivam and co. put together an incredible event though, worthy of every single bit of praise you can throw at them. There are pictures. Lots of them. Here.

Server Love

I managed to get a ton of work done on the Sysadmin side of KDE.

First, we managed to kill of projects.kde.org for good. It used to run ChiliProject - which has been discontinued and no longer provides security updates - and used to be a constant source of headaches for the syasadmins, with the seemingly endless HTTP 500 ISEs we’d generate. After it went down in the middle of CKI - and resulted in a few embarrassing moments in the middle of talks - we decided it had to go.

One of projects.ko‘s more important features was that it would generate an XML file with metadata about all of KDE’s projects that could be used by multiple teams: the CI guys to do automated build testing, kdesrc-build to build KDE projects from source, the i18n guys to properly map translation branches and so on. So when I was at home recuperating from my infection I wrote a set of scripts that generated this data from another source - the sysadmin/repo-metadata.git repository - that we set up just for this purpose. The first weekend after I returned to college, Ben and I killed off projects and replaced it with our homebrew solution. It worked at the first try.

Of course, with my college blocking SSH I had to use scm as a jump host and that let to another set of funny things - but that’s another story.

Killing projects wasn’t enough though - kde_projects.xml is a gargantuan file that takes nearly 10 seconds to generate, and drains a lot of bandwidth. So I started working on an API service that should replace it, and I hope to get it finalised over the summer.

Finally, we finished the minimum required feature-set for Propagator (for KDE’s purposes), but we’re pushing to make it a fully featured product that other folk could use too. One of my CKI talks was centered around Propagator, and spoke of what’s next. And this brings me to the next topic.

Google Summer of Code

KDE, as usual, is participating as an organisation in Google Summer of Code this year. Sysadmin got lucky this year - of the 37 slots that we got from Google, we were able to devote 3 to Sysadmin.

One of those slots is going to be for a project which I’ll be mentoring. The project is a mish-mash of stuff to improve things on the Sysadmin side - but most of it is centered around making incremental improvements to Propagator. I’m excited to be able to work with Priya Satbhaya on this. Ubuntu 16.04 LTS is here, I have very few things left to take care of on the Ops bit of DevOps, and Priya will take care of the Dev angle. At any rate, we should be able to start dogfooding Propagator mid-July onwards.

I’m also mentoring a second project, and that project excites me for a different reason. The project is based around adding a staging area for doing file operations on discontinuous selections in Dolphin. I’m excited about this because it’ll be a terrific feature to have, and because I know nothing about Dolphin’s codebase myself. This is going to be very different mentoring experience for me, because with Arnav Dhamija I’m going to be learning from my student, not the other way round. Of course, I have Pinak to help me out, and the entire 3000+ strong Kommunity to help Arnav out with advice on code. I have my hopes set pretty high on this project too.

My Summer Plans

Finally, I have some summer plans of my own. I’ll be based out of Gurgaon during June and July, working as a DevOps Engineering Intern at 1mg (formerly HealthkartPlus), where I’ll be working on tests and deployment automation, and other challenges that crop up while I’m around.

And because I’ll be in the Delhi area and making some bucks, I hope to be able to take a few holidays. I really want to ride on the Gatimaan Express (India’s only 100mph train) and see Agra afresh. I may also probably pop over to Landour-Mussourie and try to meet Ruskin Bond. Much of my sanity in my misspent childhood was preserved only by reading his stories, and trying to write in his style. I want his autograph.

So I guess I’ve written enough now. It’s 5 AM IST, and I should get some sleep. Until next time!

The Sysadmin Squad

It’s been close to a year since I started contributing to KDE long term, and since then a lot has happened.

Spectacle is out as part of the Applications suite as of KDE Applications 15.12, and not only do the users like it, other devs love it too. Maintaining Spectacle isn’t as hectic as it used to be in the early days. I get a steady stream of mostly benign bug reports, which is nothing I can’t handle in a few hours over the weekend. Wayland support still hasn’t materialised though, and it looks like it’ll be a while. Proper High-DPI (retina displays) support did materialise though, courtesy of Kai Uwe Broulik, and will land in 15.12.1.

Something I’ve come to realise, however, is that day-by-day I’m gravitating over to the Sysadmin side.

The Project Predicament

KDE’s online infrastructure is gigantic, no small part of which is taken up by Git servers. I’m not sure of the exact numbers, but a ballpark figure for the number of repositories we host would be in the 2,000-2,500 region.

We’re also in the middle of a migration. KDE’s Project Management Website runs on ChilliProject. ChilliProject is slow and unwieldy, but that we could make do with. What we can’t make do with is that ChilliProject is deprecated and is no longer developed. There are bugs and security holes and we’ve had to disable chunks of functionality to mitigate these.

So we’re migrating to Phabricator. We already have a Phabricator instance running, and quite a few projects have already been migrated. So far our experience has been good, and the Phabricator documentation is pretty good. And pretty funny (seriously, you should see their website).

Let’s Git Overhaulin’

Back in September, we set up a read-only mirror of KDE’s public project repositories on GitHub. d_ed liaised with GitHub and got is the organization username, and then I got busy mucking around with the post-recieve hook in our git.kde.org repositories.

The scripting itself was pretty simple. bcooksley had already written a small python script that git push-ed only branches and tags to upstream servers, so it was just a matter of executing that with the GitHub remote URL.

Now KDE has its own infrastructure of read-only git mirrors which mirror every repository, not just the public project ones. They’re the Anongit servers, a network of (currently) 6 such servers spread worldwide which developers can clone from. They’re available at anongit.kde.org.

Since we already mirror code from git.kde.org, why couldn’t we just add the GitHub remote URL to the list of servers to mirror to? Because we don’t push to the Anongit servers, we tell them (via a HTTP hook) that git.kde.org has been updated, please pull changes.

Now there isn’t anything inherently wrong with this approach; it’s just that we have no control over what happens if one of the Anongit servers go down. We’d ideally want to log failed mirroring attempts, and to do that effectively we’d need to push to our mirrors and log failures (and maybe retry if a mirroring attempt failed).

Propagator

Sysadmins hold the keys to much of KDE’s infrastructure - user accounts, passwords, SSH keys, you name it, we have it. Understandably, we don’t just let anyone be a part of the Sysadmin team. But the truth is, KDE’s Sysadmin team is severely understaffed and the list of tickets to attend to is miles long. We need all the help we can get.

That said, you don’t need to be a part of the core Sysadmin team to be able to help them out. And Sysadmin work isn’t just SSH-ing into servers and running commands - there’s plenty of coding to be done, and that work doesn’t need access to the keys.

So, Propagator. Over the winter holidays, I got to work on a small-ish daemon that would manage our entire distributed git infrastructure - and alleviate the Sysadmins’ need to ever touch the Anongit servers themselves.

Propagator is written in Python 3, and uses a Celery task queue to mirror-push changes to our Anongit servers. We also wanted to converge code-paths for pushing to GitHub, so Propagator does that too. Propagator also creates, sets the descriptions on, moves and deletes repositories on the Anongit servers via a SSH API, and on GitHub via their REST API.

In fact, with Propagator, managing repositories is as easy as creating them inside Phabricator. Propagator will automatically create repositories on the upstream servers (both Anongit servers and GitHub) on first push if they don’t exist, effectively making things zero-administration.

What about moves, renames, and the rest? Well, you log in to a shell on git.kde.org and run:

$: mirrorctl create AwesomeRepo.git "Awesome app to do something super cool"
$: mirrorctl rename AwesomeRepo.git AwesomeApp.git
$: mirrorctl delete AwesomeApp.git

…and so on. Propagator will not send out requests to every single configured upstream server and tell it to carry out these commands. Easy as pie.

Unfortunately, Propagator now lives inside the repo-management repo in KDE, but I do plan to separating it out into its own repository and distributing it as a standalone product in its own right as a KDE project.

Closing Notes

KDE is awesome. No, it really is.

We see a lot of newcomers pop up on KDE’s IRC channels and mailing lists and ask how to contribute. For the record, we appreciate code, documentation, artwork, translations, evangelism and publicity (but do refrain from streaking across campus with a giant KDE tattoo… \wink wink**).

And we need Sysadmins. If you have a particular set of skills that make it a nightmare for people like us to have problems with our online infrastructure, we want you.

The Return Of The Prodigal Son

After some years of hopping domains, this blog is back where it started, at blog.BaloneyGeek.com.

The reason for this is half financial, and half aesthetic. I used to own a couple of domains (including BaloneyGeek.com, Boudhayan.in and BoudhayanGupta.me, the last of which I’d picked up for free for a year via the GitHub Student Developer Pack), but paying for the renewal of all of these domains every year is expensive, especially when I’m still a student without a steady source of income. Also, the main reason for keeping the additional domains around was to have a proper “professional” e-mail ID, which I now have courtesy of KDE. I can finally consolidate around one domain, BaloneyGeek.com.

So now that I’ve had the time to cook up a semi-decent set of templates and a design for the blog, I’m using Pelican, which is a static site generator written in Python, to generate this site. The templates are Jinja2, the CSS is hand-written (no LESS or SCSS foo), and that swanky little logo was cooked up with a photo of my fancy signature (you can’t use that with my bank, sorry) taken on my 2013 Moto G and hacked around with in GIMP. The CSS is mobile-first, so the site should still look semi-decent in IE6 as well as on phones. The font is Intel’s Clear Sans, which I love.

I’ve also finally gotten a writing workflow which I’m comfortable with. I use GitHub’s Atom to write, and while Atom is slow to start up, the writing environment and colours are very soothing. I’m very finicky about my text-editors, and I love Atom, even more than I love Sublime Text. With Pelican, you can use reStructuredText (or even AsciiDoc, if you prefer) to write your posts, but I like and write in Markdown. The entire site is a git repository, and it’s available on GitHub, if you want to take a look. Anyway, once I’ve written a post, git commit-ed and git push-ed, Travis-CI automatically pulls in my site, builds the static pages and pushes them to the gh-pages branch. Yep, this is totally geeked out. I’d gotten the idea from Torrie, who too uses Pelican and has her site deployed by CircleCI instead of Travis. I just found Travis easier to configure.

Now that I finally have a writing environment that pleases my OCD as well as integrates well with my command-line based workflow, I should finally start writing more regularly. Let’s see how this goes.

Hejdå!