On Monday, April 7th, a major vulnerability named Heartbleed was discovered in OpenSSL, the library that secures many websites, mail servers, and VPNs on the internet. It is estimated that two-thirds of secure websites on the internet use OpenSSL, and most of those sites used the vulnerable version. Belly was one of them.
We have no evidence that the Heartbleed vulnerability was used to access any of Belly’s services or data. Belly was made aware of the vulnerability at 5:00pm (Central Daylight Time), and we were completely patched by 6:30pm. As an extra precaution, we also changed our security certificates by 10:00pm. However, we do recommend you change the password on your Belly account. Everyone who works at Belly has.
Because this vulnerability was so widespread, we also recommend changing your password on most websites. Mashable has a list of what major websites were affected. While you are doing this, it would be a perfect time to start using a password manager like 1password or Lastpass. These tools make it very easy to use a unique password for every website you log in to, which is a great idea.
At 5:00pm (all times Central Daylight Time), we were made aware of the Heartbleed vulnerability. Around 6:00pm, a patch for Ubuntu Linux 12.04 (our operating system of choice) was made available. By 6:30pm, this patch was deployed to our load balancers, which do all of the SSL termination for all domains under bellycard.com and bel.ly. By 10:00pm, we had revoked the SSL certificates for those domains and replaced them with new ones.
The Heartbleed vulnerability can also affect clients running OpenSSL, not just servers. Because of this, we also patched all of our servers. We use Chef on all of our servers, so we were able to push the patch out very quickly. This was complete by 1:00am on Tuesday. Here’s the Chef recipe we used to patch our services. Feel free to use, modify, or contribute to this recipe.
We currently utilize Amazon’s Elastic Load Balancer (ELB) for domains under getbelly.com. Sometime Tuesday morning, our ELB instance was patched for the vulnerability. By 1:30pm on Tuesday, we had revoked and replaced the certificate for getbelly.com.
A Note on Certificate Revocation
Due to this vulnerability, we revoked all of our SSL certificates, which means the old certificates are no longer valid. However, most browsers do not actually check for certificate revocation, as it can slow down browsing and, in some cases, can be bypassed. Most SSL libraries on mobile devices do not check this either. This means that any website that was at any point vulnerable, will likely remain vulnerable until their current certificate expires.
To prevent this from happening in the future, Belly is taking an extra step. In new versions of our mobile apps and in-store app, we will ensure that communication with our revoked certificates is not possible. We recommend others do the same, and we will provide code samples of this in the near future.
Protecting our Members and Merchants is our daily objective. By reacting quickly to resolve this issue, we’re confident that your information is once again secure. Because let’s be real, “Belly”, “Heart”, and “Bleed” should never be used in the same sentence.