Heartbleed OpenSSL Vulnerability and Ubuntu 12.04 Precise Servers

Introduction

I'm trying not to be alarmist, but it seems we've got a "holy fucking shit" security bug that everyone who runs a server needs to be aware of. And that everyone who uses the Internet needs to be aware of. Note that this was written on April 7, 2014. Please check to make sure this information is still up-to-date if you're reading this much after that.

The information on this blog post focuses on what to do if you're running Ubuntu LTS 12.04 Precise. (Because that's what the servers I administer run. I haven't had time to research and write a general-purpose blog post.)

A good writeup is at heartbleed.com. A (very overwhelmed and slow) testing site is at filippo.io/Heartbleed.

TL;DR Me

If you're running Ubuntu 12.04 servers, run this command:

openssl version -a

If you see a "built" date of "Mon Apr 7 20:33:29 UTC 2014" or later, you have a fixed version. If you don't, you still have the unfixed version.

If you don't see the patched version, you need to upgrade immediately.

Whether or not you had the updated version you next need to revoke and reissue your SSL keys. And then you need to revoke and reissue any data that could have been in RAM on the server, because it could have been compromised.

Also, change your passwords on every sensitive on-line service you use.

What Does This Vulnerability Let Attackers Do?

They can read anything from the RAM on the attacked system. Like private SSL keys. Or anything else that might be in RAM.

Yeah. Holy Fucking Shit.

Here's the writeup from heartbleed.com:

What is being leaked?

Encryption is used to protect secrets that may harm your privacy or security if they leak. In order to coordinate recovery from this bug we have classified the compromised secrets to four categories: 1) primary key material, 2) secondary key material and 3) protected content and 4) collateral.

What is leaked primary key material and how to recover?

These are the crown jewels, the encryption keys themselves. Leaked secret keys allows the attacker to decrypt any past and future traffic to the protected services and to impersonate the service at will. Any protection given by the encryption and the signatures in the X.509 certificates can be bypassed. Recovery from this leak requires patching the vulnerability, revocation of the compromised keys and reissuing and redistributing new keys. Even doing all this will still leave any traffic intercepted by the attacker in the past still vulnerable to decryption. All this has to be done by the owners of the services.

What is leaked secondary key material and how to recover?

These are for example the user credentials (user names and passwords) used in the vulnerable services. Recovery from this leaks requires owners of the service first to restore trust to the service according to steps described above. After this users can start changing their passwords and possible encryption keys according to the instructions from the owners of the services that have been compromised. All session keys and session cookies should be invalided and considered compromised.

What is leaked protected content and how to recover?

This is the actual content handled by the vulnerable services. It may be personal or financial details, private communication such as emails or instant messages, documents or anything seen worth protecting by encryption. Only owners of the services will be able to estimate the likelihood what has been leaked and they should notify their users accordingly. Most important thing is to restore trust to the primary and secondary key material as described above. Only this enables safe use of the compromised services in the future.

How to Upgrade & Verify You Have Upgraded

To verify whether or not you have the problem, do:

openssl version -a

If you see a "built on" date of "Mon Apr 7 20:33:29 UTC 2014" then you have the upgraded version. If not, you need to do the following steps:

apt-get update
apt-get install -y openssl libssl1.0.0
openssl version -a 
# (You should now see the April 7 date.)
lsof -n | grep ssl | grep DEL
# If you see results here, you have processes still using the old OpenSSL that to be restarted.

What Else do you Have to do Besides Update?

Once you're patched your servers, you should definitely generate new SSL keys and CSRs, and then put new SSL certificates in place. This must be done AFTER you patch.

After that, you've got to go through the soul-searching and unpleasant process of figuring out what other passwords / certificates / secret information / whatever could have been in RAM on your server, and reissue/revoke those. Can't help you here, as it's going to be different depending on what you're doing with your server.

For instance, it will be fairly common for services to need to revoke all user passwords and have the users create new passwords.

For End Users (Not Server Admins)

For you personally, do you use the same password on more than one site? Change it immediately so it's different on each site. Password storage utilities such as 1Password or LastPass will be helpful here.

Once services have patched their OpenSSL implementation, you'll want to change your password on EVERY SERVICE. Yeah, I know. That sucks. But that's how big a problem this is.

You can check whether or not your bank (for instance) has fixed this by using the filippo.io/Heartbleed tool I mentioned above.