The 3 hidden costs of incident response

Even for well-run security organizations, justifying expenditures can be difficult.

Sometimes it takes a significant event – the proverbial learning moment – before security teams see a needed increase in budget for staff, training and tools. This happens because it’s straightforward to analyze the costs to a business stemming from a breach that causes an outage, loss of data, or even adversely impacts a stock price.

However, there are many hidden costs to cybersecurity. Sometimes these are overlooked because they are harder to quantify but illuminating these costs can go a long way to helping justify security budgets. In the process, we hope we avoid a disastrous incident and the high cost of a breach altogether.

Here’s a look at the three hidden costs of incident response.

1. The cost of false positives

Most security organizations rely on security scanning tools to generate alerts, which requires investigation by the incident response team. Historically signatures were the primary means of detection and were reliable and accurate.

That’s clearly changed, as threats have become more sophisticated, they are capable of morphing and slipping past signature defenses. As a result, detection techniques evolved and now include the added protection of behavior-based detection. 

Behavior-based detection alone isn’t a panacea because it’s prone false-positives – alerts for behaviors that are suspicious, but not necessarily malicious. Naturally, most security organizations would prefer this sensitivity to suspicious behavior, versus permitting the occasional threat to pass.

Still, the activity of sorting through suspicious alerts takes up time and effort – and may distract from examining the truly malicious alerts. This drags on security efficiency and inhibits optimal organizational performance, which bears a cost.

2. The cost of trivial-true positives

Like false positives, a trivial-true positive is an alert that is technically correct but largely irrelevant. For example, a detection system may trigger an alert over an email attachment that contains a 10-year-old virus. It is technically correct. However, if your system is updated to Windows 10, and has an even moderately updated virus scanner, the chances of this becoming a serious problem is low.

Unfortunately, trivial-true positives can sometimes be more disruptive than false positives. This is because determining the context to properly triage these is time-consuming. In my experience, a trivial true positive take upwards of two or three times longer to triage than a false positive.

3. The costs of discovery dwell time

Dwell time is the period between the time of the attack, breach, or compromise – and the time of detection. Each minute that occurs from that moment of the attack provides the adversary the opportunity to:

• Perform reconnaissance;
• Spread laterally;
• Establish a wider foothold to prevent eradication;
• Encrypt data;
• Exfiltrate data; and
• Vandalize the network and potentially cause costly damage.

This is a problem because, as a recent study indicated, attacks often exist for upwards of 90 days or longer before they are detected. As an adversary on your network, there is little that they couldn’t do in that time frame.

As a result, dwell time is inextricably linked to false positives and trivial-true positives. Every moment wasted investigating erroneous alerts contributes to the length of time it takes to discover actual hidden threats on your network. Time wasted on these pursuits is time that could be spent reacting to actual threats, proactively scoping threats or even hunting down hidden threats.

Strategies to overcome these hidden costs

An analyst survey last year helps to place a price tag of some of these hidden costs context. It found that 60% of financial services organizations receive 100,000 alerts per day and about half of respondents said just one in five alerts are related to a unique security event.

If we conservatively estimate it takes 15 minutes to investigate an alert, assuming the standard 40-hour week, the math works out to hundreds of weeks of investigatory work that are generated every day. The analyst that conducted the survey called this “unsustainable” and while the number of erroneous alerts will vary by vertical market, suffice to say the volume is uniformly high.

Eliminating false positives entirely is probably not realistic, so I recommend focusing on reducing the false-positive-to-detection and trivial-true-positive-to-detection ratio. The baseline ratio I’d suggest is 10:1 which means for every 10 alerts identified only one should be trivial-true positive or false-positive.

Improving that ratio – and reducing the hidden costs – requires an integrated effort across people, process and technology. Here are several tips for achieving that goal:

• Audit and discard systems that generate too much distracting noise. Your team can’t afford to waste their time with them;
• Collect relevant contextual information and have it on hand to better triage and safely ignore irrelevant alerts;
• Develop methods and train your staff to quickly evaluate events so they can focus on what is important, and ignore what isn’t;
• Effectively scope actual events to avoid addressing initial targets but miss potential lateral spread of threats that came along for the ride; and
• Assume that your detection systems have missed something and devote some amount of time to threat hunting; this can double as a professional development exercise.

As a community, security tends to think of our challenges in terms of threats, but costs and budgets merit a place on the list of top challenges. Every business function seeks to apply finite resources to maximum benefit, and to do that effectively in security, like threats, requires a keen understanding of those costs that are known and those that are hiding.