Red Sky Weekly (1-11-14): 'IR'nomics 101 (Incident Response Economics 101)

I met with a venture capital company yesterday. We hadn't really thought about meeting with funding sources until this week when one of our incoming members asked if we'd like of have a conversation. Why not? You just never know where new members or referrals, or possible research/analysis for the lab might come from.

I prepared five slides. On the first slide, we described the problem that we solve. The extemporaneous version goes something like this:

Companies everywhere are having their computers broken into.

They lose credit cards, business information, privacy data and intellectual property… all at very high costs in terms of money, reputation, and business operations. In fact, in 2012 we ran what my former boss would call a 'gin and tonic' survey. I asked the question of several dozen CISOs "what did the last targeted or APT attack cost you to clean up?" The smallest number was $1.9 million. The highest was $10 million. Ponema Institute last year reported an average of $1.4 million.

The VC didn't believe the numbers. He couldn't understand how response might cost so much. I don't think he thought I was making it up, but he just had no idea why. 

So let's try this.. for the money guys, business guys, or for you CISOs out there that have to communicate this to your CIO or C-suite, I'm caling this post 'IR'nomics 101.

First let's level-set the field. You need to understand a few variables. In every case, dozens of variables go into even the most basic detection and response. Here are just a few:

• Heterogeneity - Every environment, even the small ones, are going to be heterogeneous. Chances are you'll have mobile, cloud, connections to sales staff, possibly manufacturing systems, BYOD.. and if you've acquired another company to allow growth, you've acquired their heterogeneity as well, leaving you with a heterogeneous system of interconnected heterogeneous systems. And worse, you don't acquire companies for their impeccable network hygiene, you buy them to make money. And when they stop, you divest. 
• Complexities in layers (of heterogeneous defense in depth): So now that you understand the heterogeneity in your environment, consider the infosec posture that you've either built, haven't built, or inherited through acquisition. If you've not been through the forklift security upgrade following your first oh sh*t moment, I'm betting your security posture wasn't built purposefully, it was built on the fly to accommodate growth... if at all.
• Autonomy of businesses: The terms 'division', 'sector', or 'business unit'. All mean the same thing.. autonomous units of business operations. And do you know where the Presidents or VPs of these business operations report? Not to the CISO! They get graded on revenues and margins, not on their impeccable network hygiene. And you know what? There's a good chance your security team (if you have one) doesn't have eyes on all of these autonomous businesses. In fact, I can guarantee it. 
• Geolocation and connectivity: Even with a system in a building down the road, geolocation adds cost. Administration, monitoring, security and response all require travel, or, having local help desks, administration, and likely at least one local security person. 

Without considering maturity of the team, skill levels, situational awareness, and many others, you understand a small sample of the variables associated with 'IR'nomics lesson...

Let's use 1000 computers for our example. I've operated in the 100,000+ computer space, but those numbers are staggering and my VC friend will absolutely not believe those numbers.. so let's keep it smaller for now.

In our example, the CIO (there may not be a CISO yet) gets a call from the FBI (our call came from NCIS), telling us that there's a problem. So you download a host based tool to check your systems --CarbonBlack, the Maddrix tools, Mandiant (FireEye?), or one of the others. And on your first run, what do you find? You're gonna want a drink. Your stomach will hurt and you, as the CISO will fear for your job. You're going to have at least 10% (this is being REALLY conservative) of your computers being reported compromised.  

So let's assume 100 computers are now being reported compromised. What next? Here's the typical work flow:

1. Locate the machine: Typically the security team will want a copy of one of the machines, so they'll run it down. This almost always takes time. The scanners don't necessarily give you the location of the computer, but you'll probably look in a global directory of some sort, or possibly call HR. However this happens, it's probably going to take a few hours to locate the first offending device.  
2. Pull it off line: This isn't as simple as walking into an office and unplugging a machine. In larger companies you may have to call a help desk or a desktop team, to make sure that first employee is taken offline. Maybe this is another couple of hours required by either your own, or another department. Regardless, it costs money. 
3. Bring it back to the office to tear it apart and figure out what's going on: Here's where the fun starts. Unless you're planning on burning down the machine and rebuilding (as many do), you're probably going to want to know what caused the scanner to flag. Is this real? False positive? How bad is it? How do we keep it from happening again? The first machine gets a day or so of attention. In my last job, the average seemed to be about three days of intrusion analysis in total. This number will drop with experience, but three work days is probably about right.
4. Moving forward: Now that you know what cause the problem, you've got to come up with a strategy to fix it. In most cases, this will absolutely be a team sport. You still have 99 other machines that you've not looked at, in various parts of the company. 100 machines offline is going to really hurt. Maybe you take a weekend for the clean up. You'll have your entire IT and Infosec teams on board. You'll probably burn and reload all 100. You'll generate rules for your IPS, add a tool or two to your network; maybe reconfigure some security controls. Depending on the response, this can get really expensive --especially when companies don't bring in consultants who've been through this before --and usually they don't. 

Bottom line.. one of our members (who tends to measure everything) says that his average cost to clean up a desktop is about $10,000. The average server cleanup cost is about $40,000. So even in this very simple example, using even basic numbers, the cleanup of this 100 computers, assuming a mix of desktop and servers, $1 million in response time is quickly realized. Now add in strategy, communications, network changes, responses added to intrusion prevention systems, HIPS, antivirus, etc... and we've not even considered losses, fines, or financial remediation for losses of privacy information, credit card data, intellectual property or long term competitiveness. $1.4 million (per Ponema) is an easy number to swallow.

Now consider this. Even the most sophisticated companies will face at least one of these breaches per month. Most say they have at least one every week. And if you've not been through it before, you're more likely to deal with as many as three to five every day. 

The sky is not falling.

Every company goes through a maturation process. It's probably better described as a growth spurt. (baptism by fire?). All companies start out as consumers of intelligence. Their security team will go to the Internet and start digging for places to get help. Or maybe (the smart companies) will hire a consultant who'll tell them where to get data (usually indicators of compromise). You'll consume as many IOCs as you can get. You won't care about the story behind them. You'll implement them without thinking. And after a while, you'll start producing your own. You'll want to know who's doing it to you. You'll start digging for more information, people to talk to, and you'll share war stories over beers. You'll build an informal network of co-miserates. Comiseration will quickly turn to sharing intelligence and tips. And you'll get better at detection and response. And soon, these events will be your new normal. Those targeted events will become routine. You just do it. 

The idea behind threat intelligence sources is that you can significantly reduce the cycle times discussed above by comparing notes with others. Some folks don't mind doing it on open forums, Google groups, etc. These are usually free sources of good raw, tactical information, and the conversations can often times tip you off to the latest trending attacks. Others, maybe those with regulatory concerns, concerns for intellectual property or just those who don't want to show their cards on the Internet want to get their intelligence in more private locations. Bankers don't like to talk openly to other bankers about a cyber breach if they think there's a government regulator in the room. Healthcare, Energy, Defense, and many others have similar concerns. So they come into Red Sky. They ask questions, compare notes, and share information. And in those conversations, help each other diagnose happenings on their networks. 

The rest of my elevator pitch?

...And there’s a seemingly endless supply of places you can buy or download ‘cyber indicators’ – pieces information that can help you know if you’ve been broken into or protect yourself from future break-ins. But how do you know which of those you should use? Which ones are any good? Which ones are used to protect your type of business? to use to protect yourself from the ones most likely to strike today? Tomorrow? …or the ones most likely to do the most damage to your business?

That requires context. Context comes from intelligence and analysis.

Red Sky and Wapack Labs offer that contextual information that can help the security team decide what to protect against today, then tomorrow, then next week.

Until next time.. 

Have a great weekend!
Jeff