Heartbleed was a security bug in the OpenSSLcryptography library, which is a widely used implementation of the Transport Layer Security (TLS) protocol. It was introduced into the software in 2012 and publicly disclosed in April 2014. Heartbleed could be exploited regardless of whether the vulnerable OpenSSL instance is running as a TLS server or client. It resulted from improper input validation (due to a missing bounds check) in the implementation of the TLS heartbeat extension. Thus, the bug's name derived from heartbeat. The vulnerability was classified as a buffer over-read, a situation where more data can be read than should be allowed.
System administrators were frequently slow to patch their systems. As of 20 May 2014[update], 1.5% of the 800,000 most popular TLS-enabled websites were still vulnerable to Heartbleed. As of 21 June 2014[update], 309,197 public web servers remained vulnerable. As of 23 January 2017[update], according to a report from Shodan, nearly 180,000 internet-connected devices were still vulnerable. As of 6 July 2017[update], the number had dropped to 144,000, according to a search on shodan.io for "vuln:cve-2014-0160". As of 11 July 2019[update], Shodan reported that 91,063 devices were vulnerable. The U.S. was first with 21,258 (23%), the top 10 countries had 56,537 (62%), and the remaining countries had 34,526 (38%). The report also broke the devices down by 10 other categories such as organization (the top 3 were wireless companies), product (Apache httpd, nginx), or service (https, 81%).
The Heartbeat Extension for the Transport Layer Security (TLS) and Datagram Transport Layer Security (DTLS) protocols was proposed as a standard in February 2012 by RFC6520. It provides a way to test and keep alive secure communication links without the need to renegotiate the connection each time. In 2011, one of the RFC's authors, Robin Seggelmann, then a Ph.D. student at the Fachhochschule Münster, implemented the Heartbeat Extension for OpenSSL. Following Seggelmann's request to put the result of his work into OpenSSL, his change was reviewed by Stephen N. Henson, one of OpenSSL's four core developers. Henson failed to notice a bug in Seggelmann's implementation, and introduced the flawed code into OpenSSL's source code repository on 31 December 2011. The defect spread with the release of OpenSSL version 1.0.1 on 14 March 2012. Heartbeat support was enabled by default, causing affected versions to be vulnerable.
According to Mark J. Cox of OpenSSL, Neel Mehta of Google's security team privately reported Heartbleed to the OpenSSL team on 1 April 2014 11:09 UTC.
The bug was named by an engineer at Synopsys Software Integrity Group, a Finnish cyber security company that also created the bleeding heart logo and launched the domain heartbleed.com to explain the bug to the public. While Google's security team reported Heartbleed to OpenSSL first, both Google and Codenomicon discovered it independently at approximately the same time. Codenomicon reports 3 April 2014 as their date of discovery and their date of notification of NCSC for vulnerability coordination.
If you need strong anonymity or privacy on the Internet, you might want to stay away from the Internet entirely for the next few days while things settle.
The Sydney Morning Herald published a timeline of the discovery on 15 April 2014, showing that some organizations had been able to patch the bug before its public disclosure. In some cases, it is not clear how they found out.
Bodo Möller and Adam Langley of Google prepared the fix for Heartbleed. The resulting patch was added to Red Hat's issue tracker on 21 March 2014. Stephen N. Henson applied the fix to OpenSSL's version control system on 7 April. The first fixed version, 1.0.1g, was released on the same day. As of 21 June 2014[update], 309,197 public web servers remained vulnerable. As of 23 January 2017[update], according to a report from Shodan, nearly 180,000 internet-connected devices were still vulnerable. The number had dropped to 144,000 as of 6 July 2017[update], according to a search on shodan.io for "vuln:cve-2014-0160".
The Canada Revenue Agency reported a theft of Social Insurance Numbers belonging to 900 taxpayers, and said that they were accessed through an exploit of the bug during a 6-hour period on 8 April 2014. After the discovery of the attack, the agency shut down its website and extended the taxpayer filing deadline from 30 April to 5 May. The agency said it would provide credit protection services at no cost to anyone affected. On 16 April, the RCMP announced they had charged a computer science student in relation to the theft with unauthorized use of a computer and mischief in relation to data.
The UK parenting site Mumsnet had several user accounts hijacked, and its CEO was impersonated. The site later published an explanation of the incident saying it was due to Heartbleed and the technical staff patched it promptly.
Anti-malware researchers also exploited Heartbleed to their own advantage in order to access secret forums used by cybercriminals. Studies were also conducted by deliberately setting up vulnerable machines. For example, on 12 April 2014, at least two independent researchers were able to steal private keys from an experimental server intentionally set up for that purpose by CloudFlare. Also, on 15 April 2014, J. Alex Halderman, a professor at University of Michigan, reported that his honeypot server, an intentionally vulnerable server designed to attract attacks in order to study them, had received numerous attacks originating from China. Halderman concluded that because it was a fairly obscure server, these attacks were probably sweeping attacks affecting large areas of the Internet.
In August 2014, it was made public that the Heartbleed vulnerability enabled hackers to steal security keys from Community Health Systems, the second-biggest for-profit U.S. hospital chain in the United States, compromising the confidentiality of 4.5 million patient records. The breach happened a week after Heartbleed was first made public.
Many major web sites patched the bug or disabled the Heartbeat Extension within days of its announcement, but it is unclear whether potential attackers were aware of it earlier and to what extent it was exploited.
Based on examinations of audit logs by researchers, it has been reported that some attackers may have exploited the flaw for at least five months before discovery and announcement. Errata Security pointed out that a widely used non-malicious program called Masscan, introduced six months before Heartbleed's disclosure, abruptly terminates the connection in the middle of handshaking in the same way as Heartbleed, generating the same server log messages, adding "Two new things producing the same error messages might seem like the two are correlated, but of course, they aren't."
According to Bloomberg News, two unnamed insider sources informed it that the United States' National Security Agency had been aware of the flaw since shortly after its appearance but—instead of reporting it—kept it secret among other unreported zero-day vulnerabilities in order to exploit it for the NSA's own purposes. The NSA has denied this claim, as has Richard A. Clarke, a member of the National Intelligence Review Group on Intelligence and Communications Technologies that reviewed the United States' electronic surveillance policy; he told Reuters on 11 April 2014 that the NSA had not known of Heartbleed. The allegation prompted the American government to make, for the first time, a public statement on its zero-day vulnerabilities policy, accepting the recommendation of the review group's 2013 report that had asserted "in almost all instances, for widely used code, it is in the national interest to eliminate software vulnerabilities rather than to use them for US intelligence collection", and saying that the decision to withhold should move from the NSA to the White House.
The RFC 6520 Heartbeat Extension tests TLS/DTLS secure communication links by allowing a computer at one end of a connection to send a Heartbeat Request message, consisting of a payload, typically a text string, along with the payload's length as a 16-bit integer. The receiving computer then must send exactly the same payload back to the sender.
The affected versions of OpenSSL allocate a memory buffer for the message to be returned based on the length field in the requesting message, without regard to the actual size of that message's payload. Because of this failure to do proper bounds checking, the message returned consists of the payload, possibly followed by whatever else happened to be in the allocated memory buffer.
Heartbleed is therefore exploited by sending a malformed heartbeat request with a small payload and large length field to the vulnerable party (usually a server) in order to elicit the victim's response, permitting attackers to read up to 64 kilobytes of the victim's memory that was likely to have been used previously by OpenSSL. Where a Heartbeat Request might ask a party to "send back the four-letter word 'bird'", resulting in a response of "bird", a "Heartbleed Request" (a malicious heartbeat request) of "send back the 500-letter word 'bird'" would cause the victim to return "bird" followed by whatever 496 subsequent characters the victim happened to have in active memory. Attackers in this way could receive sensitive data, compromising the confidentiality of the victim's communications. Although an attacker has some control over the disclosed memory block's size, it has no control over its location, and therefore cannot choose what content is revealed.
The affected versions of OpenSSL are OpenSSL 1.0.1 through 1.0.1f (inclusive). Subsequent versions (1.0.1g and later) and previous versions (1.0.0 branch and older) are not vulnerable. Installations of the affected versions are vulnerable unless OpenSSL was compiled with -DOPENSSL_NO_HEARTBEATS.
The problem can be fixed by ignoring Heartbeat Request messages that ask for more data than their payload need.
Version 1.0.1g of OpenSSL adds some bounds checks to prevent the buffer over-read. For example, the following test was introduced to determine whether a heartbeat request would trigger Heartbleed; it silently discards malicious requests.
if(1+2+payload+16>s->s3->rrec.length)return0;/* silently discard per RFC 6520 sec. 4 */
The OpenSSL version control system contains a complete list of changes.
The data obtained by a Heartbleed attack may include unencrypted exchanges between TLS parties likely to be confidential, including any form post data in users' requests. Moreover, the confidential data exposed could include authentication secrets such as session cookies and passwords, which might allow attackers to impersonate a user of the service.
An attacker having gained authentication material may impersonate the material's owner after the victim has patched Heartbleed, as long as the material is accepted (for example, until the password is changed or the private key revoked). Heartbleed therefore constitutes a critical threat to confidentiality. However, an attacker impersonating a victim may also alter data. Indirectly, Heartbleed's consequences may thus go far beyond a confidentiality breach for many systems.
A survey of American adults conducted in April 2014 showed that 60 percent had heard about Heartbleed. Among those using the Internet, 39 percent had protected their online accounts, for example by changing passwords or canceling accounts; 29 percent believed their personal information was put at risk because of the Heartbleed bug; and 6 percent believed their personal information had been stolen.
Although the bug received more attention due to the threat it represents for servers, TLS clients using affected OpenSSL instances are also vulnerable. In what The Guardian therefore dubbed Reverse Heartbleed, malicious servers are able to exploit Heartbleed to read data from a vulnerable client's memory. Security researcher Steve Gibson said of Heartbleed that:
It's not just a server-side vulnerability, it's also a client-side vulnerability because the server, or whomever you connect to, is as able to ask you for a heartbeat back as you are to ask them.
An analysis posted on GitHub of the most visited websites on 8 April 2014 revealed vulnerabilities in sites including Yahoo!, Imgur, Stack Overflow, Slate, and DuckDuckGo. The following sites have services affected or made announcements recommending that users update passwords in response to the bug:
The Canadian federal government temporarily shut online services of the Canada Revenue Agency (CRA) and several government departments over Heartbleed bug security concerns. Before the CRA online services were shut down, a hacker obtained approximately 900 social insurance numbers. Another Canadian Government agency, Statistics Canada, had its servers compromised due to the bug and also temporarily took its services offline.
Platform maintainers like the Ethereum Foundation advised their users to change passwords.
The servers of LastPass were vulnerable, but due to additional encryption and forward secrecy, potential attacks were not able to exploit this bug. However, LastPass recommended that its users change passwords for vulnerable websites.
The Tor Project recommended that Tor relay operators and hidden service operators revoke and generate fresh keys after patching OpenSSL, but noted that Tor relays use two sets of keys and that Tor's multi-hop design minimizes the impact of exploiting a single relay. 586 relays later found to be susceptible to the Heartbleed bug were taken off-line as a precautionary measure.
Android 4.1.1, used in various portable devices. Chris Smith writes in Boy Genius Report that just this one version of Android is affected but that it is a popular version of Android (Chitika claim 4.1.1 is on 50 million devices; Google describe it as less than 10% of activated Android devices). Other Android versions are not vulnerable as they either have heartbeats disabled or use an unaffected version of OpenSSL.
Several services have been made available to test whether Heartbleed affects a given site. However, many services have been claimed to be ineffective for detecting the bug. The available tools include:
CrowdStrike Heartbleed Scanner – Scans routers, printers and other devices connected inside a network including intranet web sites.
Netcraft Site Report – indicates whether a website's confidentiality could be jeopardized due to a past exploitation of Heartbleed by checking data from Netcraft's SSL Survey to determine whether a site offered the heartbeat TLS Extension prior to the Heartbleed disclosure. The Netcraft Extensions for Chrome, Firefox and Opera also perform this check, whilst looking for potentially compromised certificates.
Other security tools have added support for finding this bug. For example, Tenable Network Security wrote a plugin for its Nessus vulnerability scanner that can scan for this fault. The Nmap security scanner includes a Heartbleed detection script from version 6.45.
Sourcefire has released Snort rules to detect Heartbleed attack traffic and possible Heartbleed response traffic. Open source packet analysis software such as Wireshark and tcpdump can identify Heartbleed packets using specific BPF packet filters that can be used on stored packet captures or live traffic.
Vulnerability to Heartbleed is resolved by updating OpenSSL to a patched version (1.0.1g or later). OpenSSL can be used either as a standalone program, a dynamic shared object, or a statically-linked library; therefore, the updating process can require restarting processes loaded with a vulnerable version of OpenSSL as well as re-linking programs and libraries that linked it statically. In practice this means updating packages that link OpenSSL statically, and restarting running programs to remove the in-memory copy of the old, vulnerable OpenSSL code.
After the vulnerability is patched, server administrators must address the potential breach of confidentiality. Because Heartbleed allowed attackers to disclose private keys, they must be treated as compromised; key pairs must be regenerated, and certificates that use them must be reissued; the old certificates must be revoked. Heartbleed also had the potential to allow disclosure of other in-memory secrets; therefore, other authentication material (such as passwords) should also be regenerated. It is rarely possible to confirm that a system which was affected has not been compromised, or to determine whether a specific piece of information was leaked.
Since it is difficult or impossible to determine when a credential might have been compromised and how it might have been used by an attacker, certain systems may warrant additional remediation work even after patching the vulnerability and replacing credentials. For example, signatures made by keys that were in use with a vulnerable OpenSSL version might well have been made by an attacker; this raises the possibility integrity has been violated, and opens signatures to repudiation. Validation of signatures and the legitimacy of other authentications made with a potentially compromised key (such as client certificate use) must be done with regard to the specific system involved.
Since Heartbleed threatened the privacy of private keys, users of a website which was compromised could continue to suffer from Heartbleed's effects until their browser is made aware of the certificate revocation or the compromised certificate expires. For this reason, remediation also depends on users making use of browsers that have up-to-date certificate revocation lists (or OCSP support) and honour certificate revocations.
Root causes, possible lessons, and reactions
Although evaluating the total cost of Heartbleed is difficult, eWEEK estimated US$500 million as a starting point.
David A. Wheeler's paper How to Prevent the next Heartbleed analyzes why Heartbleed wasn't discovered earlier, and suggests several techniques which could have led to a faster identification, as well as techniques which could have reduced its impact. According to Wheeler, the most efficient technique which could have prevented Heartbleed is a test suite thoroughly performing robustness testing, i.e. testing that invalid inputs cause failures rather than successes. Wheeler highlights that a single general-purpose test suite could serve as a base for all TLS implementations.
According to an article on The Conversation written by Robert Merkel, Heartbleed revealed a massive failure of risk analysis. Merkel thinks OpenSSL gives more importance to performance than to security, which no longer makes sense in his opinion. But Merkel considers that OpenSSL should not be blamed as much as OpenSSL users, who chose to use OpenSSL, without funding better auditing and testing. Merkel explains that two aspects determine the risk that more similar bugs will cause vulnerabilities. One, the library's source code influences the risk of writing bugs with such an impact. Secondly, OpenSSL's processes affect the chances of catching bugs quickly. On the first aspect, Merkel mentions the use of the C programming language as one risk factor which favored Heartbleed's appearance, echoing Wheeler's analysis.
On the same aspect, Theo de Raadt, founder and leader of the OpenBSD and OpenSSH projects, has criticized the OpenSSL developers for writing their own memory management routines and thereby, he claims, circumventing OpenBSD C standard library exploit countermeasures, saying "OpenSSL is not developed by a responsible team." Following Heartbleed's disclosure, members of the OpenBSD project forked OpenSSL into LibreSSL.
The author of the change which introduced Heartbleed, Robin Seggelmann, stated that he missed validating a variable containing a length and denied any intention to submit a flawed implementation. Following Heartbleed's disclosure, Seggelmann suggested focusing on the second aspect, stating that OpenSSL is not reviewed by enough people. Although Seggelmann's work was reviewed by an OpenSSL core developer, the review was also intended to verify functional improvements, a situation making vulnerabilities much easier to miss.
OpenSSL core developer Ben Laurie claimed that a security audit of OpenSSL would have caught Heartbleed. Software engineer John Walsh commented:
Think about it, OpenSSL only has two [fulltime] people to write, maintain, test, and review 500,000 lines of business critical code.
The OpenSSL foundation's president, Steve Marquess, said "The mystery is not that a few overworked volunteers missed this bug; the mystery is why it hasn't happened more often." David A. Wheeler described audits as an excellent way to find vulnerabilities in typical cases, but noted that "OpenSSL uses unnecessarily complex structures, which makes it harder to both humans and machines to review." He wrote:
There should be a continuous effort to simplify the code, because otherwise just adding capabilities will slowly increase the software complexity. The code should be refactored over time to make it simple and clear, not just constantly add new features. The goal should be code that is "obviously right", as opposed to code that is so complicated that "I can't see any problems".
LibreSSL made a big code cleanup, removing more than 90,000 lines of C code just in its first week.
According to security researcher Dan Kaminsky, Heartbleed is sign of an economic problem which needs to be fixed. Seeing the time taken to catch this simple error in a simple feature from a "critical" dependency, Kaminsky fears numerous future vulnerabilities if nothing is done. When Heartbleed was discovered, OpenSSL was maintained by a handful of volunteers, only one of whom worked full-time. Yearly donations to the OpenSSL project were about US$2,000. The Heartbleed website from Codenomicon advised money donations to the OpenSSL project. After learning about donations for the 2 or 3 days following Heartbleed's disclosure totaling US$841, Kaminsky commented "We are building the most important technologies for the global economy on shockingly underfunded infrastructure." Core developer Ben Laurie has qualified the project as "completely unfunded". Although the OpenSSL Software Foundation has no bug bounty program, the Internet Bug Bounty initiative awarded US$15,000 to Google's Neel Mehta, who discovered Heartbleed, for his responsible disclosure.
Paul Chiusano suggested Heartbleed may have resulted from failed software economics.
The industry's collective response to the crisis was the Core Infrastructure Initiative, a multimillion-dollar project announced by the Linux Foundation on 24 April 2014 to provide funds to critical elements of the global information infrastructure. The initiative intends to allow lead developers to work full-time on their projects and to pay for security audits, hardware and software infrastructure, travel, and other expenses. OpenSSL is a candidate to become the first recipient of the initiative's funding.
After the discovery Google established Project Zero which is tasked with finding zero-day vulnerabilities to help secure the Web and society.
^Lee, Ariana (13 April 2014). "How Codenomicon Found The Heartbleed Bug Now Plaguing The Internet". ReadWrite. Archived from the original on 5 September 2017. Retrieved 4 December 2017. Discovered independently by Google engineer Neel Mehta and the Finnish security firm Codenomicon, Heartbleed has been called "one of the most serious security problems to ever affect the modern web."
^Smith, Gerry (10 April 2014). "How The Internet's Worst Nightmare Could Have Been Avoided". The Huffington Post. Archived from the original on 19 July 2017. Retrieved 16 April 2020. The bug revealed this week was buried inside 10 lines of code and would have been spotted in an audit, according to Laurie, who works on the security team at Google.