Heartbleed: A great time to think about incident response

by Isaac Potoczny-Jones

Heartbleed is the nickname of a dangerous OpenSSL vulnerability that was just announced. A security update was already available before the announcement, and this is definitely a vulnerability where quickly patching makes a big difference. A fast response matters here because malware wasn't in the wild yet, so many sites likely can prevent any negative consequences with quick action.

The necessity for rapid response to vulnerabilities illustrates why you should have an incident response procedure in place. An incident response procedure allows for a measured, planned response to a security incident like this one. In this blog post, we'll walk you through the basics of putting together an incident response plan, mostly based on NIST's incident response process.

What is a security incident and what should I do to plan for it?

A security incident is any "violation or imminent threat of violation of computer security policies, acceptable use policies, or standard security practices." This might include attacks from the outside or inappropriate use internally. Every organization should have a definition of what types of events they consider to be security incidents and how they respond to and report different kinds of incidents. Heartbleed should be considered an incident because it's an imminent threat, although not necessarily a violation yet.

If you have an incident response procedure in place, you probably already know if you are affected by Heartbleed and have possibly addressed the incident. Without a procedure in place, you might just be reading news reports and wondering if you should care. The difference in these two responses will have a huge impact on whether sensitive information gets leaked and how long it will take to address Heartbleed.

More generally, without an incident response procedure, administrators make preventable mistakes in handling or reporting an incident. The development of an incident response procedure is a security best practice according to both SANS' top 20 list of recommended security controls, and NIST 800-53, which recommends incident response (IR-4) for even "low impact" security environments.

Here's the basic approach: preparation, detection and analysis, containment, eradication, recovery, and recording lessons learned. SANS suggests that there is a set of quick wins that will help immensely: develop a set of written procedures, including personnel roles; who should report incidents; when they should report them; what information they should provide; and how users are to be trained. Communication with the media can be crucial for high-profile incidents.

  • Preparation: Establish the incident response capability in the first place, open the lines of communication to the relevant people inside and outside your organization, and have the correct tools in place, e.g., forensic hardware and software. Most importantly, know who is accountable for watching for incidents and responding to them. With Heartbleed, many administrators were able to patch it before most people had even heard about it.

  • Detection and analysis: This step involves discovering the type of incident (in this case, a serious vulnerability in critical software). Are you vulnerable? Do you operate an OpenSSL server? An Intrusion Detection System can often help, but in this case it's all about monitoring vulnerability announcements and knowing your infrastructure. Logging is also important for analysis, so maintaining good logs is important. This step includes documenting the incident as you analyze it, and notifying the appropriate parties. Notification is extremely important since other organizations that you collaborate with might also need to handle the incident in their environments. For instance, if your users' passwords or clients' private data were disclosed, what steps should they take?

  • Containment, eradication, and recovery: Containment is about stopping the spread of the incident, whether through privilege escalation or the spread of a virus. For Heartbleed, containment is everything. The longer your systems remain vulnerable, the more likely it is that someone will get around to extracting sensitive information from them. This is a case where, after patching the server, you should go back to the "analysis" step. If an attacker did get into the system, what might they have gotten? If they extracted the server's private key, what could they do with it? Is there any evidence that someone used privileges from this system to get into another system? This step often includes forensics, i.e., gathering information without damaging evidence. Eradication and recovery sometimes involve restoring systems from a known, good backup.

  • Post-incident activity: This step involves analyzing lessons learned and cause analysis. Was your team's response effective? Could it have been better? What new processes should be put in place? Retaining evidence for potential legal activity is also important. Recording the process you followed also helps when your users, clients, or the media have questions later about how you handled the incident.

Benefits of having an incident response plan

Again, the detailed documentation from NIST helps to clarify why incident response is important: quicker containment and recovery from attacks, better preparation for future attacks, and dealing with legal issues that can arise. Besides the positive benefits, it can be required by law in some circumstances.

  • It helps to make sure the appropriate steps are taken - e.g., patching quickly, making sure that the attacker doesn't gain access to other parts of the system, making sure critical servers don't get taken offline. Without a plan in place, administrators might fail to contain the damage, fail to collect forensic information, and even fail to stop the attack.

  • It helps to recover quickly so that disruption and loss is minimized. For Internet retailers, every hour of downtime costs money. For some organizations, every hour lost can damage their mission.

  • It helps in learning; incidents should be examined and procedures modified so that the handling of future incidents is improved.

  • It helps to deal with the legal issues that arise; it provides steps for collecting appropriate information before the logs are destroyed.

Team considerations

People are everything, and each organization is different. Think through these questions for your organization: How centralized should your incident response team be? How should the team communicate with other parties? Should it be made up of employees or include external parties? Is 24/7 response necessary? NIST also emphasizes providing the team with learning and growth opportunity through training.


Heartbleed is a very serious security incident, and it's likely that your team needs to handle it immediately. Once the dust has settled, take some time to put a security incident response plan in place. It's a stitch in time that will save nine. Contact us if you'd like some help.

About Galois

Galois' mission is to ensure trustworthiness in critical systems. Security incident response helps address today's problem, risk management helps address tomorrow's problems. Our goal is that some day, critical systems can be built with poweful tools to completely eliminate the root causes of many vulnerabilities.


Tech talk: A short examination on the intersection of security and usability (or How usable security could save us all)

Galois is pleased to host the following tech talk. These talks are open to the interested public--please join us! (There is no need to pre-register for the talk.)

A short examination on the intersection of security and usability (or How usable security could save us all)
Morgan Miller
Tuesday, 8 April 2014, 11am
Galois Inc.
421 SW 6th Ave. Suite 300,
Portland, OR, USA
(3rd floor of the Commonwealth building)

abstract: Cryptographic tools have become more powerful in the last three decades. With that power has come complexity. To use or even understand most security tools you need a thorough understanding of mathematics which makes them inaccessible to the general public. The discipline of usability has been growing as well in the past three decades. There have been few but promising overlaps in usability and security which may provide vital tools for managing our digital selves, upholding the principal of privacy, and preserving freedom of speech.

bio: Morgan Miller studied elliptic curve encryption at Reed college and bilinear pairing based digital signatures at the university of Lugano. Since finishing her masters in 2010, she has worked as a usability analyst at Experience Lab. She maintains a deep and foundational interest in civil liberties and the changing landscape of our increasingly wired world.


Tech Talk: Practical Challenges to Secure Computation

Galois is pleased to host the following tech talk. These talks are open to the interested public--please join us! (There is no need to pre-register for the talk.)

Practical Challenges to Secure Computation
John Launchbury
Tuesday, 1 April 2014, 11am
Galois Inc.
421 SW 6th Ave. Suite 300,
Portland, OR, USA
(3rd floor of the Commonwealth building)

abstract: In secure computation, one or more parties collaborate to compute a result while keeping all the inputs private. That is, no-one can gain knowledge about the inputs from the other parties, except what can be determined from the output of the computation. Methods of secure computation include fully homomorphic encryption (where one party owns the input data and the other party performs the whole computation), and secure multiparty computation (where multiple parties collaborate in the computation itself). The underlying methods are still exceedingly costly in time, space, and communication requirements, but there are also many other practical problems to be solved before secure computation can be usable. For programmers, the algorithm construction is often nonintuitive; for compiler writers, the machine assumptions are very different from usual; and for application designers, the application information flow has to match the security architecture. In this talk we will highlight these challenges, and indicate promising research directions.

bio: Dr. John Launchbury is Chief Scientist of Galois, Inc. Prior to founding Galois, John was a full professor in Computer Science and Engineering at the Oregon Graduate Institute School of Science and Engineering at OHSU. His instruction style earned him several awards for outstanding teaching, and he is internationally recognized for his work on the analysis and semantics of programming languages, and on the Haskell programming language in particular. John received First Class Honors in Mathematics from Oxford University in 1985. He holds a Ph.D. in Computing Science from University of Glasgow and won the British Computer Society's distinguished dissertation prize. In 2010, John was inducted as a Fellow of the Association for Computing Machinery (ACM).


Joe Kiniry Joins Galois as Principal Investigator

Joe Kiniry has recently joined Galois as a Principal Investigator. He was previously an academic in Europe for twelve years, most recently a Full Professor at the Technical University of Denmark. He has extensive experience in formal methods, high-assurance software engineering, foundations of computer science and mathematics, and information security. Specific areas that he has worked in include software verification foundations and tools, digital election systems and democracies, smart-cards, smart-phones, critical systems for nation states, and CAD systems for asynchronous hardware.

Joe is really excited to be working at Galois. He has been a fan of Galois, as it is a darling of the formal methods community, for many years. "Galois is full of super-sharp researchers and engineers doing really interesting work on hard problems. I look forward to furthering the Galois mission of applying formal methods for good."

Read more about Joe here.


Galois and voidALPHA Produce Free Game for DARPA to Help Crowdsource Software Security

As part of DARPA’s Crowd Sourced Formal Verification (CSFV) program, Galois has partnered with game design and development experts voidALPHA to produce a free online formal verification game, StormBound.

Formal verification is the most rigorous way to thwart attacks against IT systems and applications upon which military, government, and commercial organizations rely. Traditional formal methods, however, require specially trained engineers to manually scour software—a process that up to now has been too slow and costly to apply beyond small software components. In contrast, CSFV games such as StormBound translate players’ actions into program annotations and generate mathematical proofs to help verify the absence of important classes of flaws in software written in the C programming language. We have developed an automated process that creates new puzzles for each verification problem in need of a solution. With StormBound, we aim to investigate whether large numbers of non-experts playing the game can perform formal verification faster and more cost-effectively than conventional processes.

For more information, contact Aaron Tomb or Jef Bell.

Approved for Public Release, Distribution Unlimited