Last night I watched as the driver of a rental moving truck took the top of the truck clear off as he drove under an overpass that was too low for clearance. The top scraped off a bit like the top of a sardine can; it peeled back and bits of curly-cued steal flew across Storrow Drive, one of the main crosstown parkways in Boston, MA. As the top separated from the rest of the structure, the sides were dislodged, and the result was four side panels swaying side to side, threatening to fall of the base of the truck completely and allow its contents to stand out in the open, unprotected (kind of like some companies’ data…).

My first thought was, “Poor guy. Probably just graduated.” Then it occurred to me that this man’s first big step on his own, without the safety net of school or parents, was moving himself out of a college dorm and into the next part of his life, and he was not at all prepared, at least for this particular step. Then I started thinking about security incidents and how they can be just like the top of the truck, peeling off slowly at first, then spreading damage until the contents—data—are exposed or lost.  

You can call it another lonely day

Before you think, “Gee, it was only a driving accident,” let’s consider a few other things and how they relate to incident preparedness: In all likelihood, this young man had been living in Boston for at least a little while, which meant he would have known about Storrow Drive, experienced its traffic, and also been aware of the “low clearance signs” posted on every bridge and overpass. Storrow is, in fact, restricted to cars; trucks and busses are not permitted due to the clearance issues. Even if the driver was not aware of the restriction and had failed to see the bright yellow signs, he did know before he started driving the truck that he was going to need to move his belongings out of his old dwelling and transport them by roadway to his next destination. This should indicate a need for preparation, maybe a little research, and definitely mapping out his route.

All of this is akin to preparing for a security incident. In the “It’s not if, but when” era, every security team should have its security plan mapped out and practiced well ahead of time, yet every day there is news of companies that don’t know what to do when they discover an incident, much less have a defined process to follow when handling a breach.

While incident response planning is a lengthy process that can’t be described in one short article, there are some basic steps every organization can take towards building or refining an incident response preparedness program. Even organizations with a well-defined program need to review and update the plan regularly to ensure it reflects current risks, threats, and organizational requirements and capabilities.

According to NIST, there are four major categories for handling an incident:

  1. Preparation
  2. Detection and Analysis
  3. Containment, Eradication, and Recovery
  4. Post-Incident Activity

Some other organizations categorize these steps a little differently but the gist remains the same.

Preparation

Establishing an incident response capability also means that some action toward deterring incidents will naturally occur, though network security and the tools and techniques of incident handling are complex and outside the scope of this post.

In the preparation phase, security and IT teams, alongside the business, should think through the resources that must be in place when an incident strikes. Resources include a cross-functional team to manage the incident, policy documents for incident handling, timelines, organizational reporting structures, up-to-date contact lists (including outside resources like local FBI agents, forensic investigators, or legal teams), documentation guidelines, and a list of the tools and technologies that can help stop the incident before it further spreads and mitigate any damage.

While preparation doesn’t traditionally include risk assessments or user training and awareness, linking current findings and occurrences to the plan may prove beneficial as a baseline during an incident.

Detection and Analysis

Incident identification requires that teams have already established baseline activities and can see deviations from the norm which indicate a true incident. Firewalls, intrusion detection/prevention systems, various log files, file integrity monitoring, alerts, and more will serve as technical indicators which require further analysis by security and IT teams before an incident is declared.

Once is it determined that an incident has been detected, incident handlers must be brought in to start analyzing and documenting findings. Reporting and notification throughout the organization begins at this phase as well.

NIST warns that analyzing an incident is not always straight forward and that the best incident response plan includes “highly experienced and proficient staff members who can analyze the precursors and indicators effectively and efficiently and take appropriate actions.” The guidance continues to say that organizations lacking the right skill set (either internally or contracted) could result in inefficient and costly mistakes during detection and analysis.

Containment, Eradication, and Recovery

NIST groups these three efforts into one phase while SANS breaks them into three separate phases. Whichever approach you choose for your organization, the first step is to limit any damage that has occurred and prevent further damage from happening. The key to containment is understanding the organization’s risk appetite, upon which decisions about containment can be made. This is where it becomes extremely important to have completed the pre-work involved in incident handling. If the organization’s risk strategy hasn’t been defined or communicated, incident handlers have nothing upon which to base decisions and ensuing actions.

Once the risk strategy is known, the handlers can proceed to containment, which my mean shutting down systems or taking them offline, sandboxing, wiping data, etc. The way in which an incident is contained will depend on the type of incident it is. As a part of the planning process, organizations should map out major types of incidents and understand the best mitigation and containment strategies for each category. Of course, even identical categories of incidents will look and act differently inside different organizations, but having a general baseline will speed this process considerably and inform decisions.

Following containment, incident handlers will need to eliminate the problem (e.g., malware removal, changing compromised credentials) and recover affected systems. Before this phase begins, teams should create backups and carefully document processes, in case anything goes wrong during a system wipe or reimage, for example. All affected systems must be tested and monitored to ensure reinfections cannot reoccur after the team has restored and brought systems back into normal operation. Because attackers will often return to the weakest point and reinitiate an attack, it is especially important to validate that restoration did not include the same vulnerabilities present in the system prior to restoration.

Post-Incident Activity

Once the incident has been handled and systems and operations have resumed normalcy, many teams also return to their regular routines. While dwelling on past problems isn’t the solution to preventing future ones, the business as a whole can learn a great deal about the organization and improving security so that future incidents can be prevented or discovered more quickly and efficiently.

All organizations should conduct post-incident “lessons learned” meetings to understand exactly what happened, how it happened, what processes were taken during the incident, what went wrong, what went right, and how things can be improved in preparation for and during the next incident.

A post-incident activity checklist can be created to help the organization through this phase and document any findings that can be used to improve incident response planning.

Tell me why everything turned around

Even the best laid plans often go awry, but without any advanced planning, IT and security teams are significantly more likely to crash and burn than those organizations who’ve taken the time to read the signs, map out their journey, and avoid the most obvious vulnerabilities along the way.