House is on Fire. Call 911? Do I have a choice?

If you were awakened in the middle of the night by the smell of smoke, would you call 911? Do you have a choice?

I’ve been talking and working with several organizations lately who for whatever reason, chose not to call 911. Worse, some (most all) either don’t have smoke detectors or the batteries have died, they don’t get tested annually, and aren’t even wired to a place that will let them be heard when they go off in the middle of the night.

So, what happens when there’s a fire and the owner is awakened by the smell of smoke? Maybe he’ll grab a fire extinguisher or buckets of water. Within a short time, the fire grows. Grab the garden hose!.... the whole time, as the neighborhood gathers to watch his house burn to the ground, little by little, he forbids the neighbors from calling the fire department.

Get the picture? We’re not talking about smoke. We’re not talking about fire at all. We’re talking about the stubbornness of IT directors and CIOs with emotional connections to the idea that whatever happens in the networks that they built; whatever happens, they can fix it. Let’s think...

• The fire smoldering deep in their networks is largely undetectable by their current smoke detectors. Those things were installed years ago, and even though they auto-update, they might detect the old stuff, but can’t detect the new.
• The team is all fairly new, and while they know tools exist on the network, they have no idea how to use them.
• “The IT guy has been with the us for years. He’s never let us down before. We’ll cut him some slack.. just a few more months.”

Sound familiar?

So here’s the thing. In the last 30 days I’ve talked with at least three companies in this exact situation. One has started submitting information to Red Sky without actually joining. Another has a CSO and an IT director, but the IT guy doesn’t trust the CSO and thinks he can do it on his own. The third isn’t a corporation... but I could write a series of posts on government information security!

So let me pose a couple of thought questions...

Should IT security fall under the CSO when IT has no security organization? What responsibility is held by the CSO when no Infosec organization exists? If not the CSO, then who? In many of the companies I talk to, they have a CSO who’s responsible for physical security. The CSOs usually have no IT experience, but is the only security guy in the many of the companies. So what is their responsibility? If not the CSO, then who?

At what point do you call for help? Who do you call? FBI? Police? Consultant? When IT spends months playing ‘whack-a-mole’, when should IT be required to get outside assistance? How much of the budget should be allowed to be spent before IT is required to blow the whistle? When that occurs, who should they call?

Last, what role does the board play? When IT is unable to stop the intrusions, how much time/money should be spent before senior management reports to shareholders?  When a company refuses to ask for help, how much time (money) should be spent before liabilities fall to the board and senior management for not acting sooner? Is there a liability to the board for not notifying shareholders and requiring management to seek assistance?

BREAK BREAK

I haven’t done a Red Sky update in a couple of weeks. We have a lot going on...

• This week we’re gearing up for our 5th quarterly threat day (in Tampa). We are really looking forward to a first time face-to-face with several members and to further building out the trust relationship which is so important in our space.
• Two new Fusion Reports were released to our community. The latest introduced a new threat group to our list of tracked adversaries and provided detailed analysis on the leveraged protocol as well as mitigation recommendations. The second report provided additional analysis and attribution on a recent highly-publicized compromise.
• We’ve added a new member to Beadwindow. Our newest member is a state level organization for higher education. We really like working with the city, state and local governments!
• Last, we’ve just taken possession of the space in the Manchester mills. The new company (a Red Sky Alliance company; this one incorporated in NH) is called Wapack Labs, and we’re bootstrapping this one with contract security intelligence work, a bit of R&D, and some research.

I’m looking forward to talking with more of you in the future. We’re giving two more threat briefs, I’ll be presenting heading for Dallas this week, speaking at a McKinsey event in New Jersey and then headed for New York for another panel discussion with the financial community. We’re busy and doing great!

Enough for now.
Have a great week!
Jeff

MAGIC BOXES and the RSA CONFERENCE

What an awesome week for me and for Red Sky at the RSA conference!  It was a privilege to be able to speak to some of the smartest people in the business and equally as flattering when they would see my shirt and say, “You’re with Red Sky?  I’ve heard of you guys.”  What an easy way to strike up a conversation and I’m leaving here with a renewed confidence that we are getting it right when it comes to the challenges we all face with APT and our adversaries.

If you’ve never been to the RSA conference the best way to describe it would be a 10,000 square foot shopping mall called the “Expo” shoehorned between overcrowded classrooms.   A place where almost everyone dooms their inboxes with eternal spam in exchange for a $20 remote control helicopter – if you’re lucky enough to win it!  No thanks.  A place where the line to have your picture taken with Darth Vader is fifty-feet long and the line to have your picture taken with the 49’rs cheerleaders is non-existent.  And no, I’m not making this up!

And all the while I was walking through the Expo, I kept listening to the vendor sales pitches and I got to thinking; all this technology being sold exists for one reason – to prevent or limit the damage to humans caused by humans.  No wonder we can’t secure our networks. We’re looking in the wrong place!  We’re taking devices that are programmed to act rationally and asking them to protect us from irrational human behavior.  Stop me if you've heard this before "Hmm….I think I’ll disable this anti-virus software because it’s making my streaming video slow!"

There is not a single device, yet, that can predict WHY someone will act in the manner they do.  And until one is programmed to understand the stresses of losing a job, or a client, greed, or the want to be famous or notorious, the concept of dropping a device in your network and thinking you’re protected is a losing proposition.  I’m not suggesting we don’t need firewalls, IDS, IPS, and DLP systems but what I am saying is simply this;  in all the hype about the next magic box that will save us from ourselves, the real force multiplier in solving this problem is often forgotten – people.  Simple right?  Let me give you an example.

At the end of a long day at the conference, I struck up a conversation with the IT manager of a mortgage clearing house with several billions, with a “B”, dollars on the balance sheet.  After a while, and when he felt comfortable talking, he shared with me that his network had been targeted, breached, and was most likely still leaking information.  His purpose for being at the RSA conference was to find a solution, a “magic box”, to make his troubles go away.  To be fair, not HIS troubles, but his boss’s troubles!  No wonder we’re losing the cyber war; unfortunately, this story is all too common.

Now, I could have sold him on what we do at Red Sky and gone into my elevator pitch etc. but what he really needed in that moment was the comfort in knowing that he isn’t alone, a sympathetic ear to listen to his problem and tell him that there are others in the same place he finds himself.  I told him very simply, “Take a deep breath. Break the problem down into small pieces. And put your plan to paper. And act.”  I handed him my card and told him if he ever had any questions to call.  Will he, maybe, not sure, but he wasn’t looking to buy anything and I wasn't selling.  He was looking for someone who would listen and who he could trust.  Besides, Red Sky doesn’t sell new bosses but we can make the smarter!

When I say people are the only way we’re going to solve this problem, this is exactly what I’m talking about.  Sometimes, you have to look beyond the sale by listening to the problem and offering your advice.  That’s not to say selling isn’t important, but at Red Sky, we believe people come first.  You build trust through communication, integrity, and genuine care for others. Do this first and the sales will take care of themselves.  How many vendors do this, listen to their customers as humans with real problems that need to be solved?  Many say they do, but think for a minute.  How do you build trust through the persistent and overwhelming noise on the floor of the RSA Expo?  Simple answer – you can’t.  This is why I can predict with overwhelming confidence that you’ll never see a Red Sky booth at a trade show!

Which leads me to a few closing thoughts. 

The leading principal we at Red Sky live by is first and foremost, we are a community. When people ask what Red Sky is all about, I always start by saying, “Red Sky is a community of really smart people with diverse backgrounds, talents, and expertises, helping one another solve the APT problem.” 

If we all know we can’t solve our problems with a magic box, isn’t it equally true that we cannot solve our problems on our own?  Sure, you can go it alone, but the point is, you don’t have to.  However, if you do and you solve the APT problem?  Well, now that’s a sales pitch I would pay to hear! 

If you’re interested in being part of the community or if you demand photographic proof that the 49’rs cheerleaders were being neglected, please feel free to email me directly: rgamache@redskyalliance.org

APT is hard, but not impossible

When Jeff asked me to write this week’s blog, I jumped at the opportunity.  What an incredibly busy week not only for Red Sky but for the security world as a whole!  As many of us were getting prepared and turning our eyes to San Francisco and the RSA conference, on Tuesday Mandiant shook things up and released their controversial “APT1” report!  The conference will be all abuzz!  More on Mandiant's report in a bit.

Living in Northern New England, I often talk to organizations, banks, and companies on the small side.  Interestingly, one bank CISO described his bank as one such “small” bank with nearly a billion dollars in assets!  To be fair, relatively speaking, that is probably a small bank, but who wouldn’t want 1% of what is considered “small”?! I digress…  And like Northern New England, there is a sense of security that comes with living here.  The pace is slow and crime is low and all too often this tranquility results in what I call “cyber complacency” or the “I’m too small to own” syndrome.  Unfortunately, cyber criminals are not bound by the same societal values of the communities where their targets reside. 

I’ve had many conversations with good security people and CISO’s that do not see them as ever being the target of APT because simply put, and quoting, “We’re too small. There are bigger fish to fry before they ever get to us.”  Oh, really?  I can’t entirely blame some people for holding this attitude, APT is hard, not only understand for many of the decision makers but also extremely hard to defend against when you’re outgunned and understaffed.

These conversations generally lead me into a story I often tell about a small defense contractor working on a very niche project for the defense department.  When asking what measures they were taking against the APT threat, the response was, “APT is too hard to deal with. Besides, were too small. No one cares what we’re doing.” Unfortunately, someone did care and this small company was gutted of its intellectual property.  Result: Aside from the hundreds of thousands of dollars worth of intellectual property lost, the company lost its competitive advantage in the market space and we, as a nation, may have lost our competitive advantage on the battle field.

When I tell this story the climate in the room often dramatically changes from “We’re too small to get owned” to “We know we are exposed but we’re spending a ton on security already and we don’t even know where to start with APT.”  Again, APT is hard, but can you afford to ignore it?  The adversary knows this and those that wish to steal from you are not doing it alone. They have teams of people targeting you, which brings me back to Mandiant. 

Mandiant’s release of the APT1 report has been met with both strong applause and strong criticism. In my opinion, I think there are merits in the arguments on both sides.  Whether you agree with Mandiant’s decision or not, the release of the report pushes the APT problem and “APT1”, lurking in the shadowy corners of cyberspace into the light for everyone to see.  Mandiant has thrust the conversation about the APT problem, its tenacity, and its effects, light years forward and I myself can only see the positives in that.

To me, there are two take aways from the Mandiant report that should raise the hairs on any CISO as well as anyone in the C-Suite.  One is something we all know – Once you’re the target, they’re coming in whether you like it or not.  They will outspend you in both time and money, and when they do get in, and they will, they’re there to stay!   The second take away and a more subtle one – The adversary is working in teams. Not only in teams of highly trained people in the technical trades but people trained in linguistics, cultural attitudes, human intelligence, and economics.  Can you afford a team equal in size and expertise?  Probably not.

APT is hard.  Red Sky members know this very well.  Red Sky is made up of multiple mature incident response teams from some of the largest Fortune 500 companies sharing information, assisting one another, and working together to solve the complex APT problem.  Red Sky members form a team of very smart analysts and technical experts from a widely diverse number of industries and disciplines.  

As a Red Sky member, these groups of professionals, facing the same threats as you, become a part of your team and you become part of theirs.  The point is your adversaries number in the hundreds if not thousands.  You can’t ignore that and you’re going to eventually have to start somewhere – Mandiant has made that abundantly clear. You can go about it alone but you don’t have to – ask for help and join the conversation!

For all of you traveling to the RSA conference, I wish you very safe travels.  If you’re like me, you’re leaving early to avoid the storm working its way eastward!  If you’re interested in speaking with me about Red Sky and how our members can help you, please feel free to reach out to me at rgamache@redskyalliance.org.

It’s going to be a great event and I’m looking forward to the presentations and the good people who are working in the trenches every day.

See you in San Francisco!

Rick

The costs of cleanup, and two new analysis reports

I’ve been sick as heck after coming back from Colorado last week. I’m guessing the altitude really messed with me. And this bug, I’m assuming a bit more than a cold, but less than a flu is in full bloom right now (and the last few days). Regardless, I drafted this blog earlier in the week. It’s a bit rough, but I’m going to use it anyway. I hope you enjoy it...

BT BT 
 
840,000 users
1.9 million devices
$40 million dollars per year in security clean-up costs

This is what General Shelton (Air Force Space Command) quoted during the AFCEA conference last week.

While these numbers aren’t as big as I thought they might be, the amount of money spent per user -- $47.60, for cleanup alone, bothers me a little, especially given the tight networks run by the military -disk images are standardized, every point of presence is protected; monitored by a computer network defense security provider (CND-SP), manned by scores of highly trained, highly skilled active duty, civilian, and contract technical, intelligence and law enforcement personnel.

This is the AIR FORCE we’re talking about. One of the most technologically sophisticated fighting forces in the world.  These guys manage thousands of satellites in global orbit, can drop a needle sized bomb on a desired impact site the size of a gnat’s ass from 200 million miles away (I’m guessing), while dog fighting at Mach 2. Whats more, the AF is probably close to (or is) the earliest DoD leader in the field of information security.  These guys are good.

So why is the Air Force still spending $47.60 per user on cleanup?

Because these aren’t your father’s hackers. And the technology we’ve got today doesn’t stop them. For example, last month, another VERY mature security team blogged that they'd discovered that their systems had been hit in a sophisticated attack. This attack occurred when a handful of employees visited a mobile developer website that was compromised. The compromised website hosted an exploit which then allowed malware to be installed on these employee laptops (this is called a watering hole attack). The laptops were fully-patched and running up-to-date anti-virus software.

Back to the AF.. One of the other comments I thought was incredibly telling: “Not all actors are caught at the gateways. Many are caught internally by professional airman. 60% of all DoD security rules are created by AF airman at 24th AF.” I understood this to say, not technology, but airman... human.

I’m an old Navy guy (and Coastie), and prefer going to sea over flying in airplanes any day, but what this tells me is that even one of the most technologically sophisticated forces in the world still needs to apply massive amounts of human brainpower to defending their networks. What’s more, the AF is targeted for operations, not long term R&D, and they certainly aren’t considered supply chain, so think about this...

Like the Air Force, you too need a steady stream of information security professionals. The problem is, there aren’t enough to go around, and many in the field today are burning out. The Air Force has a lot of people (840,000!), has dedicated training pipelines and the best technology, yet they still spend $47.60 per users cleaning up after security incidents. Even this savvy, highly educated group spends $40 million per year on incident response, and remember, this doesn’t include the ongoing operating expenses of actually managing information security, running the 24x7 SOC, etc. It’s cleanup.

I have more stats for you.. from a commercial company, given to me by an enterprise level Director of Incident Response. These guys keep pretty good numbers:

• This company owns roughly 135,000 computers
• They experience hundreds of thousands of attempted targeted attacks every week, with 3-5 successful every day
• They collect, and analyze (as best they can) over 1 Pb of log data per year
• They spend roughly $10,000 per desktop to cleanup, and $40,000 per server

The company suffers 3-5 attacks every day, penetrating ~2% of their network every week.

So in the Air Force, with all of those sophisticated controls, training, etc., $47.60 per user is pretty low, compared to this commercial company, where it costs $10,000 per user laptop and $40,000 per server to clean up. The AF is doing pretty well. but this company is very mature in dealing with APT and has a highly sophisticated information security team. Yet they pay nearly 210 times the cost of the AF in cleanup!

Even more shocking is that an adversary can build or buy a piece of malware for almost nothing compared to the cost of the data they acquire or destroy, and the damage they cause--and that attack cost is declining rapidly as more and more malware and new vulnerabilities are monetized.

What does it cost you? Where will you get the help you need to understand how hackers are changing, targeting, and stealing information and money from your company?

BT BT

This week was crazy busy in the portal. We analyzed a recent 0-day and provided relevant indicators to our membership. Two reports also went out the door. The first, our second Intel (non technical) Analysis Report (IAR) for 2013 which provided high-level analysis on a new targeting TTP involving a prolific actor. The second report was our fourth fusion (technical) report for this year and included in-depth analysis on a frequently leveraged downloader program. The report provided detailed protocol analysis as well as a custom decoder and tailored signatures for detection.

Don’t be a wallflower. Ask someone in Red Sky Alliance or Beadwindow. Call today for an introduction to our community. With every Red Sky demo, we’ll give you our latest white paper “How Great Companies Fight Targeted Attacks and APT”.  This paper outlines a roadmap, at an executive level, in less than 10 pages, seven items companies who’ve dealt with, survived, and thrived in the face of Targeted attacks and APT have done effectively to defend themselves against targeted and advanced persistent threats.


Until next time,
Have a great week!
Jeff

"Attackers collaborate, defenders are drowning in a sea of data."

I’m a little behind on getting my weekly blog posted this week, relaxing after a week in altitude in Colorado Springs at the AFCEA Cyber Symposium. My body just isn’t used to spending that much time at 6000+ feet! It was an ok conference, but a couple of speakers struck chords. On a day one panel were Brett Hartman, now with Cisco, but also the former CTO at RSA (during their much publicized breach), and Dr. Phillis Schneck from McAfee. Two quotes from the panel that I found especially on target:

Attackers collaborate, defenders are drowning in a sea of data.

-Brett Harttman, Cisco Systems

It’s pretty safe to believe that everything is owned.”
“Can’t take the humans out of the loop.”  Human and machine.

-Dr. Phillis Schneck (McAfee)

These are key points. Here’s why.

This week we analyzed malware that was believed to be one piece of code was used in the NY Times attacks. It’s not automated analysis, it’s automated assisted analysis. Code is pulled from a machine after enterprise-wide searches for unique ‘indicators of compromise’. This is done in one of a few ways, but the most common is through software that examines each computer for file names, sizes, MD5 hash matches, or other pieces of metadata about the files on the system. If there’s a match, a file, piece of running memory, and/or a forensic image of the computer is pulled --either locally, or remotely.

Here’s the problem. In any given enterprise, when you run those host based tools to look for these indicators of compromise (IOCs), any given company is going to be inundated with results --and most will not be false positives.

So here’s what happens...

1. You load a system that can inventory computers, their file systems, and the infrastructure. There are some really nice tools out there that do this work for us --Mandiant’s MIR, Carbon Black, RSA ECAT, and others. (Note that I didn’t call out anti-virus. AV alone is simply not in the same class as these other tools. The use of signature-based anti-virus alone as a defense is no longer enough in a world attacks and threats are changing daily.)
1. Every file on every device is identified, and the meta-data about those files are collected and sent to a centralized analysis device for aggregation and correlation. (Not all require movement of data to the centralized analysis machine. Some perform the work locally on the host, others send data back.)
2. At the same time, a good team will have some form of network analysis device running on their Internet Point of Presence (iPoP). Whether a netflow analysis tool (i.e.: Arbor, NetWitness, etc.), or full packet capture... thereby aggregating even more data (that must later be analyzed).
3. Now comes the hard part... every piece of that data must be examined, separating the known good from the suspected bad --our indicators of compromise. Done manually, correlations of this data could take months. Thankfully we don’t have to do this manually.
1. The results? You’re not going to like this... expect big numbers. In a small or medium sized company, without a formalized information security structure, expect every computer to be reported as compromised. In a larger enterprises where formal information security processes exist, expect a large percentage of your systems to be compromised at first, and then as you work the issues, the numbers will level out. And, in enterprises where formal infosec processes exist, even those with great process, training, etc., it is not uncommon to experience a 2-5% daily detection (probable compromise) rate. Many companies (companies with GREAT infosec team) have stated that they are attacked hundreds of thousands of times per week. These numbers, and a 2-5% targeted compromise rate is not uncommon. In fact, it’s probably a pretty good number. So imagine this.. your company owns 1,000 computers. Giving you the benefit of the doubt, Let’s say you probably have a great, highly trained, very mature infosec team. 20-50 compromised computers per day (7 days/week) could (should) be expected.
4. Assuming you have a great team (I’ll assume that you do!), and you have 20 compromised computers every day (140 per week), and each computer is a 100 Gb hard drive (yes, I know this is small), not including shares, SAN devices, network attached storage, cloud, etc., You need to be prepared to perform incident response on 140 systems and 1.4 Tb of host based data per week, or roughly 6 Tb per month, plus whatever you’ve collected at the iPoP. That’s like examining more than six Libraries of Congress every month!  Now, how might you pull out the good (bad) stuff? Automation (and hopefully on systems that are not compromised!) You’re going to want to run tools against those systems to help you assess which ones are really compromised, or at a minimum, prioritize your work. How do you do that?

So, both Brett and Dr. Schneck are absolutely correct.

Defenders are drowning in data --both enterprise data that must be analyzed, and potential sources of good intelligence and indicators. Defenders MUST gather information from as many good sources as possible. You need sources of known goods, sources of high confidence bads; and, you must share information between your peers, other industries. Look for sources of information that will give you high quality indicators that can be placed in your networks that can proactively block, drop, and stop those attacks before they successfully penetrate your environment.  An interesting observation - smaller infosec shops generally tend to try and save money by seeking out open source lists of indicators. Their teams are often times smart technically, and they choose to spend time reading the open source lists of bugs. While they can obtain a lot of good information, to find those nuggets requires a lot of time reading, analyzing, and evaluating the information before actual implementation. This is a counterproductive. Research times can be reduced significantly, by purchasing a membership in an information sharing group like Red Sky (or others), where many members are reading those same lists, and talking about them in a private environment. This is a game changer. In fact, a July 2012 McKinsey Report states the average worker spends 28 hours per week reading email. The report suggests that knowledge workers (I’d call an infosec pros knowledge workers) using a social environment (like Red Sky?) to exchange knowledge can double the benefits received, and increase productivity by 25%!

This goes to the heart of why Red Sky. How long before your incident responders burn out? What if you could reduce their workload by 25% by participating in our social environment? What if they could be twice as productive by reducing cycle times needed to research cause and effect?  You can.

In conclusion.

• Defenders are drowning in data, and losing the fight. (Harttman)
• Humans are required in the loop to understand the nuances of daily changing threats and attacks. (Schneck)
• Current thinking on how to capture intelligence and information isn’t working. Red Sky Alliance and its public sector portal, Beadwindow, are working. (Stutzman)

Call today for an introduction to our community. With every Red Sky demo, we’ll give you our latest white paper “How Great Companies Fight Targeted Attacks and APT”.  This paper outlines a roadmap, at an executive level, in less than 10 pages, seven items companies who’ve dealt with, survived, and thrived in the face of Targeted attacks and APT have done effectively to defend themselves against targeted and advanced persistent threats.

Until next time,

Have a great week!
Jeff