BaloneyGeek's Place

BaloneyGeek's Place

Operator! Give me the number for 911!

Akademy

As I write this post, I'm peering out of the window of a long pressurised metal tube hurtling through the air 39,000 feet above sea level, watching a mesmerising sunset as the plane passes over Baku. The sky has turned from bright blue to deep red to deep purple to pitch black in a matter of ten minutes. It is yet another 4 hours before the plane lands in New Delhi and the officials stamp my little travel diary signalling the end of the events of the last 7 days.

I'm still trying to make sense of the last week.

Perhaps writing this post will help me figure out how I almost made it without crying for a decade only to have to let the floodgates open on the flight back.

On the 31st of August, I left India for the very first time in my life, for Berlin, to attend QtCon and Akademy, the annual world conference of the KDE Community. I had expectations from the trip. I expected to finally put a face and a voice to all the IRC nicks I geeked out with over the last 2 years. I expected to attend talks that would blow my mind. I expected to eat some really good food. I expected to see a few things around the city.

All of that happened. And then some more. A lot more. I fell in love. With so many things.

The city. The cars. The roads. The public transport. The complete strangers who would always make eye contact, smile and say hello. The one time I was travelling on the U-Bahn well past midnight and ended up singing the cup song with two people whom I had never seen before and never will see again. The architecture. The sleepless nights spent walking all around the city exploring because we'd have events during the daytime.

Spending a week four thousand miles away from your daily life does make you think.

Is it just the first Akademy that turns you upside down, or is it every Akademy? Does it make a bigger impact on me than on others because as she put it, very emphatically, "You're still such a child!"?

I wonder what it will be like 10 years later. I may have a completely different life, a completely different career. KDE may be a distant memory for me. I may not even remember QtCon 2016 as a whole. But the first KDE e.V. AGM, the first time I stepped into the BCC and spent the entire day in the back office trying to get video recording to work. The time we dressed up and walked in the freezing cold to Checkpoint Charlie just because we felt so crazy. The last night before I left when she wouldn't let me sleep - "It's your last night in Berlin! It needs to be special!" - and then we went down to the East Side Gallery and couldn't get back because it was well past 2 AM and neither the normal buses nor the U- and S-Bahns were running. These will be permanently etched in my memory.

Thank you everyone who made this week so memorable for me. All these go out to different people - thank you for taking us to the Russian restaurant, thank you for grabbing hold of me one night when I had lost my phone and making me justify to myself why I even am in KDE, thank you for making it financially feasible for me to be there, thank you for dragging me down to the hotel with the aquarium, thank you for giving me so many reasons to be so incredibly happy to be there. Thank you for being so gosh darn energetic, infecting me with said energy and making me get by with 3 hours of sleep a night.

And last but by far the most, this is for you. You, the eyelid-batting, compulsive hair-untangling, soft-talking little bundle of joy. I have so many things to express to and about you, but words fail me. Thank you for all that you did, and did not. Thank you for completely decimating my blocks and filters. Thank you for making the best part of this trip not the fact that I was in a different country for the first time in my life. Please don't get lost :-)

Auf wiedersehen Deutscheland, und danke für alles!

Shell One-liners to Generate Random Passwords

I've been on the lookout for ways to generate random passwords from the shell. One of the ways to do that, as I've learned from people, is to run the slappaswd utility a couple of times, get enough random output, and then just concatenate them together to form a long enough password. Crude, but suffices, until you realise that the slappaswd utility comes from OpenLDAP and you'd rather not pollute your computer with that filth. What next?

These passwords aren't for to be remembered by humans; I'd probably store these passwords in an encrypted text file and just copy-paste them wherever they're required. So they can be completly random. This makes our job so much easier.

Entropy

The first step to generating a random password is to get enough entropy. Fortunately, on Linux and most *NIX systems, you'll have either /dev/random and /dev/urandom to help you along. Just start by dumping enough bytes from it into stdout.

Something to take note of though - historically, /dev/random used to be a source of completely random bytes captured from the environment, electrical noise, system events, etc., and /dev/urandom would be a pseudorandom number generator that could either be cryptographically secure, or not. That difference has blurred somewhat these days, and you should consult the documentation for your operating system to see where those bytes are coming from.

You can use something like Haveged to inject more deterministically generated entropy (yes, I get the irony there) into the system.

First Shot

Once you've decided on a source of random bytes, the password generation is simple - just get enough bytes, and convert them into an ASCII string. The best way to do that is to simply base64-encode raw bytes, like so:

$: dd if=/dev/urandom bs=32 count=1 2>/dev/null | base64 -w0
4QK3nyD+ONf1ZnubDOYlZ9a4qI9CBYT5TWUrcc24YJY=

This is as simple as it can get - read the PRNG (/dev/urandom) with dd, get 1 block (count) of 32 bytes (bs), redirect stderr to /dev/null to suppress dd's status report, and base64-encode the resulting output. The -w0 switch is to prevent base64 from inserting newlines in the output after every few characters (the default is 76 characters).

Although having symbols in your password makes it more secure (more possible characters to choose from when brute-forcing the password), it's possible to remove the non-alphanumeric characters in the password. Just pipe the output to tr, like so:

$: dd if=/dev/urandom bs=32 count=1 2> /dev/null | base64 -w0 | tr -cd "[[:alnum:]]"
AB7qAV5bnDZb4Q0QydQFVFyuMmU7UsZDWn2dvwY0bKs

I really recommend you just use the first example, and increase the bs value given to dd to get longer passwords. However, you might want to get fancier, and because it's shell, there are ways to.

Entropy, Redux

You might want to have relatively short (30-60 character) passwords that are generated from massive amounts of entropy, say over 8 kilobytes of random data. While this is overkill, you could actually do it by throwing a hash function into the mix.

Here's what the pipe would have to do:

  1. Get a massive amount of random data and dump it into stdout, possibly with dd if=/dev/urandom bs=8192 count=1.
  2. Hash that data to reduce the number of bytes in the output. You shouldn't really think in the direction of MD5 at all - choose between sha{1,224,256,384,512}sum, and strip all extra data from the output
  3. The standard sha*sum commands output the hex digest of the hash bytes, so you'll have to convert them back into raw bytes.
  4. Now just run base64 to encode the data, and pull out the symbols if you want to.

Now I'm not sure about what the math will look like here, but something tells me this extra effort is pretty much useless. 32 bytes of purely random data will require the same brute-force effort to crack as 32 bytes of hash generated from 8 kilobytes of purely random data. In any case, I've outlined the pipe below.

Password Generation, Redux.

I'll explain the pipe in bits.

First, the hash. I'm going to use MD5 to demonstrate here because the hashes are nice and short, but you should never use MD5 in practice because the hashes are nice and short.

Here's how the ouput of the md5sum command (and all the sha*sum commands) look like:

$: echo hello | md5sum
b1946ac92492d2347c6235b4d2611184  -

You have the hash, followed by a couple of spaces, and then the filename. Since it's reading from stdin, the filename is simply -.

To get just the hash, pipe it to cut, like so:

$: echo hello | md5sum | cut -d " " -f 1
b1946ac92492d2347c6235b4d2611184

You could just base64-encode this string, like so:

$: echo hello | md5sum | cut -d " " -f 1 | base64 -w0
YjE5NDZhYzkyNDkyZDIzNDdjNjIzNWI0ZDI2MTExODQK

Notice that there are no symbols in this string? That's because you're encoding bytes that are already alphanumeric, and base64 encodes alphanumeric ASCII values to alphanumeric ASCII values.

Another way to do this would be to convert the hash into its raw bytes, and then base64-encode that. You'd get a shorter password (half as many characters; hex-encoding a byte takes two bytes), but you'd get symbols. You'll be trading password length for a bigger alphabet space.

To convert the hash to raw bytes, you can use Perl, Python, shell builtins, sed and what not, but the simplest way is to simply use xxd, a command that comes with vim. Here's how to base64-encode the raw bytes of the hash:

$: echo hello | md5sum | cut -d " " -f 1 | xxd -r -p | base64 -w0
sZRqySSS0jR8YjW00mERhA==

And that's basically it. Use a longer hash function and some actual random input, and you're done:

$: dd id=/dev/urandom bs=16384 count=1 2> /dev/null | sha384sum | cut -d " " -f 1 | xxd -r -p | base64 -w0
OLBgp1GsljhM2TJ+sbHjaiH9txEUvgdDTAzHv2P24donTt6/529l+9Ua0vFImLlb

Shell Shortcuts

To make this easier, you can just create a small function and put that into your ~/.zshrc or ~/.bashrc. This is what I've got:

function genpasswd () {
    local RPASSWD=$(dd if=/dev/urandom bs=$1 count=1 2> /dev/null | base64 -w0)
    if [[ $2 = "-n" ]]; then
        echo $RPASSWD | tr -cd "[[:alnum:]]"
    else
        echo $RPASSWD
    fi
}

This function takes one mandatory argument, and one optional one. The first argument is the number of random bytes you want, and the second is -n if you want symbols to be stripped. Works like so:

$: genpasswd 32
XAXVnxe6UoigD1oSyaGFPGGqhkaJdZtTcIWGW1kJQQU=

And if you want no symbols, then:

$: genpasswd 32 -n
uwhCmpnfG6ELBT7j8QOzk2wqFa8S3JUH8Fq2fIDCLY4

Fun?

Till next time!

KDE Infrastructure on DigitalOcean

KDE's server inventory is a mixed bag. We have a few physical machines that were donated to us. There's some sponsored colocation. We also rent a couple of big machines from Hetzner, divvy them up into smaller containers with lxc and host services there. Today, I can announce that we're adding droplets from DigitalOcean to that bag.

Not only would a full-blown cloud infrastructure on something like AWS be prohibitively expensive, our situation doesn't merit such an infrastructure. Our more powerful servers are dedicated as build slaves for our CI system, and a server with 32GB of RAM and dual-redundant SSDs with ZFS on Linux based storage is currently used for the code repositories, and will soon be used to host Phabricator.

We could, however, do with something in between cloud "compute" resources and a physical server that we manually manage, and DigitalOcean's droplets fit the bill right there. DO's droplets are small - we can dedicate a 1GB droplet to hosting websites, which would allow us to isolate web hosting from other services while not wasting resources we'd never use. They're also standard KVM machines, which allows us the level of manual control we'd like.

There are some additional aspects of using DigitalOcean that I like:

  • All our existing servers are either in continental Europe or the United States. DigitalOcean has datacenters in Asia that I'm particularly looking forward to making use of, to service our contributors from Asia Pacific (particularly India) better.
  • Depending on demand, we could bring up new servers or shut down existing ones at short notice. While we don't do something along these lines now, once we have the capability I could see us pre-emptivly adding temporary server capacity to handle high-traffic events, like a new Krita release.

But this isn't the best part of this post.

Once we realised we'd have some use for DigitalOcean's offerings, we went and asked them if they'd be willing to sponsor us under their programme for supporting open-source software projects. To our utter delight, they were very enthusiastic about supporting us and set us up with an account and a lot of free credits to start us out.

So in the next few months, expect to have KDE's existing online services to get more reliable as we add failovers, and new services to spring up as we start putting plans for the additional capacity into action.

Till next time!

Brexit Is Not Happenning

David Cameron is an astounding genius. Of course, you'd expect nothing less from an alumni of Eton and Oxford, but to see his genius in action in such a grand public scale makes it no less incredible.

This meme was doing the rounds on Facebook yesterday. But I beg to differ - this wasn't a rage-quit. This was a chess move so calculated and so devastating that the full extent of its consequences will take a long time to become apparent.

Biggest Rage Quit of 2016

With his resignation, David Cameron ensured that the person who succeeds him is going to commit political suicide. And someone will have to succeed him, because someone will have to become the next Prime Minister of the United Kingdom. And that person will have to die. Whether or not that person is Boris Johnson remains to be seen.

Here's the deal. The referendum was set up to be advisory, not binding. The British Parliament is under no legal obligation to follow through on the results on the referendum, but if the government does not follow up, the entire premise of democracy falls apart. Therefore, Britain has to leave the EU, and to do that, someone will have to inform the European Commission by sending them a notice under Article 50 of the Lisbon Treaty. That someone was going to be David Cameron, until he decided to resign and let his successor do it.

On the face of it, that would be an honour for his pro-Leave successor. But of course, things aren't so simple:

  • If the successor follows through and invokes Article 50, Scotland and Northern Ireland breaks away and joins the EU. The United Kingdom is no longer united. A mountain of laws and regulations need to be torn up and new ones written in its place. The English economy collapses. The public quickly loses patience, and the blood is now on the successor's hands. His career is over, as is the United Kingdom as we know it.

  • If the successor does not follow through and fails to invoke Article 50, the premise of the democratic government falls apart. The governance of the UK is no longer democratic, and the entire establishment is a farce. The successor's career is over, as is probably the entire government's. It doesn't end there, however. The next person on the chair also finds himself in the same conundrum, as the next. Until the will of the British people change and they decide to stay - and make it known in another referendum - this cycle continues.

It appears I'm not alone in this theory. Someone commented exactly along these lines on a Guardian article, which is what led me to think twice - hey, I thought this line of thought was an effect of too much House of Cards, but it appears I might not be completely crazy after all - and take this out of my mind and put it to paper. Well, the Internet.

The next few months are going to be very interesting.

Message Passing - Our Telegram-IRC Relay Service

Yesterday, we launched a service to relay messages between IRC and Telegram, and we're syncing 4 channels for KDE and 5 more for Kubuntu at this moment. The sync is two-way, so whatever people say on Telegram appears on the IRC channel, and vice versa. Almost everything works - almost being everything except that files and stickers shared on Telegram don't appear on IRC.

So here's a blog post about how we did it. An exclusive behind the scenes look at how KDE Sysadmin conducts their business, if you will.

Server Set-Up

The server that runs IRC services (a bouncer, bots for Zabbix, Bugzilla etc.) is an LXC container running Ubuntu 14.04.4 LTS. We don't use docker or other application container services to virtualise apps - since we run such a diverse set of services written in different languages on the server, we just confine apps to their own users, and try to keep the dependencies confined to the user as far as possible.

To run the Telegram-IRC relay, we chose to use TeleIRC. TeleIRC fits our bill perfectly. It has all the features we want. It's also written in JavaScript, which means it comes from the same community that brought us examples of groundbreaking engineering such as left-pad and is-positive-integer. Isolating this service from the rest of the system is critical, at least from a security perspective.

Step 1: Node.js

Ubuntu 14.04.4 LTS carries Node.js version 0.10 in its repositories. It's too old and won't do, as we found out much after actually installing teleirc, because npm won't complain when you're installing but teleirc will refuse to start because os has no method homedir().

So we'll have to manually obtain a current version of Node.js. Thankfully, we can use the Node Version Manager to obtain a binary build of a current version of Node.js directly from the Node website.

Start with a fresh user - let's call it teletubby - and log in. Because it's Node, the "recommended" method for installing nvm is by curl-ing a script and piping it to bash. It's an idea that's brilliantly simple and has provably zero security flaws. Of course, because we're KDE and we like to do things the hard way, I decided to forego this easy install procedure and do things manually:

teletubby (~) $: mkdir TeleIRC
teletubby (~) $: cd TeleIRC
teletubby (~/TeleIRC) $: git clone https://github.com/creationix/nvm.git

At this point it'll clone the nvm repository to ~/TeleIRC/nvm. If you're feeling particularly adventurous, you can use this as-is (using the current master), but I like to stay on the stable branch. nvm's repo makes that easy - all you need to do is to checkout the latest stable tag:

teletubby (~/TeleIRC/nvm) $: git checkout `git describe --abbrev=0 --tags`

At this point, you'll need to add the nvm set-up script to your Bash profile, so add the following lines to ~/.profile:

export NVM_DIR="$HOME/TeleIRC/nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

Log out and log back in (or start a new login shell), and you should be able to use the nvm command right away.

You'll actually have to install a version of Node.js now. The current LTS branch of Node.js is the v4.4 branch (the current release as of writing is v4.4.4). You can do:

teletubby (~) $: nvm ls-remote

to see what versions of Node.js are available for you to install. To install the 4.4 branch, just do:

teletubby (~) $: nvm install v4.4

And when it's done installing, run node to make sure you get a Node.js prompt.

Step 2: Telegram Bot and IRC Account

To get the Telegram Bot account, you'll have to talk to @BotFather.

Start by asking for a new bot:

/newbot

BotFather is pretty conversational. It'll first ask you for the full name you want to give to the bot, and then an username. Our service is called KDE IRC Relay Service, and the username is IrcsomeBot. Note that your bot's username must end with either bot or _bot.

Once you give it a full name and an username, it'll give you an API key. Keep this key secure. If you lose it, you can generate a new one, but make sure it's never compromised.

To let your bot see every message that's said in the group (so that it can read the messages and relay them to IRC), you'll have to disable privacy for the bot:

<Me>: /setprivacy
<BotFather>: Choose a bot to change group messages settings.
<Me>: @IrcsomeBot
<BotFather>: 'Enable' - your bot will only receive messages that either start with the '/' symbol or mention the bot by username.
<BotFather>: 'Disable' - your bot will receive all messages that people send to groups.
<BotFather>: Current status is: ENABLED
<Me>: Disable
<BotFather>: Success! The new status is: DISABLED. /help

Note that to denote the channel now, we're using the notation @username, not just username. We'll prefix the username with the @ sign everywhere we want to refer to the bot from now on.

All the essential Bot account set-up is complete, so it's time to move on to the IRC account creation. This differs from IRC network to IRC network, but for Freenode it's pretty simple. Start by picking a nickname that your bot will use, logging on to chat.freenode.net with any IRC client as said nickname, and running the following command at the server window:

/msg NickServ REGISTER password [email protected]

You'll get an email from Freenode with another /msg command you'll need to type to confirm your registration. Do that, and you're done.

Step 3: TeleIRC

Once node is up and running, and you've got your accounts, it's time to install TeleIRC.

Installing TeleIRC is incredibly easy. Just run:

teletubby (~/TeleIRC) $: npm install teleirc

And check that ~/TeleIRC/node_modules/teleirc/bin/teleirc exists. At this point, you might want to add $HOME/TeleIRC/node_modules/.bin to your PATH environment variable, so that you can run the teleirc command without prefixing it with a path.

The first thing to do is to generate a configuration file, and edit it. Generate the sample config by running:

teletubby (~/TeleIRC) $: teleirc --genconfig

This will create a sample configuration file and drop it at ~/.teleirc/config.js. Edit it with your favourite text editor. It's well commented, so you shouldn't have any problem setting things up. The Telegram API Key goes into a variable defined near the top of the file, and the IRC settings go towards the bottom.

For Freenode, the default IRC config generated doesn't have spaces for a nick and a password, so refer to the snippet below for what to set:

config.ircOptions = {
    userName: 'FreenodeUserName',
    realName: 'The Real Name',
    nick: 'yournick',
    password: 'yourpassword',
    port: 7000,
    secure: true,
    sasl: true
};

Freenode allows SSL connections over port 7000, and you'll need to have SASL enabled because otherwise TeleIRC won't be able to authenticate your nick and password.

Of course, don't forget to actually set up the channel-group mappings.

Once you're done, start teleirc by running:

teletubby (~/TeleIRC) $: teleirc

Nothing should bomb, except a bunch of warning about chat_ids not being found. That's not an error. Now just go to every group where you want the bot to be present, add the bot's username and say something. The relay should just start working.

Step 4: Running Forever

It turns out you can actually do this without messing with your init scripts. All you need is another npm package called forever, and Cron. Yes, Cron.

Start by installing forever:

teletubby (~/TeleIRC) $: npm install forever

Check if it runs:

teletubby (~/TeleIRC) $: forever start `which teleirc`
teletubby (~/TeleIRC) $: forever list

You should see a table with an entry for teleirc, along with a path to the log file, which you can cat to read teleirc's output. The entry should also have a number associated with it (0, if it's the only service running). You can control it by running:

teletubby (~/TeleIRC) $: forever restart 0   # to restart the service
teletubby (~/TeleIRC) $: forever stop 0      # to stop the service

Once you've verified that things run, and you can control the process, it's time to make sure that the service runs at system start. Start by creating a script with the following content:

#!/bin/sh
forever start `which teleirc`

Then, run:

teletubby (~/TeleIRC) $: crontab -e

Your text editor should open with a crontab file. You'll need to add one line to the bottom:

@reboot /full/path/to/your/start/script.sh

And make sure that the cron service is enabled. That's it!

In Conclusion

So the service is up and running, and any KDE IRC channel that wants to be mirrored with a Telegram group just needs to file a Sysadmin Task on our Phabricator instance.

I hope this helped you out if you're looking for ways to bridge your IRC and Telegram channels together. Until next time!