Saturday, December 21, 2013

Red Sky Weekly (12/21/13): Been there, done that, got the t-shirt!

Been there, done that, got the t-shirt is a saying that ran like water flowing across the bow of any of the many ships I spent time on during my early career. It means what is means. Been there, done that, and when we finished, we passed out t-shirts with the campaign, operation, or team logo on it. Sometimes the t-shirts are made from pride, sometimes their made to help heal. Sometimes their made to show unity.

Target earned their t-shirt this week. Sorry guys. I actually do know what it feels like to work the better part of the 168 available hours in a week fighting the networks. Thankfully, I was never in the global press because of it! Hang in there. And let me know when the shirts come out. I'd like to buy one! Neither Red Sky or the lab are first line incident responders, but we are tracking this closely. While it's not apparent (yet) how this all came to be, it is widely known that starting in 2009 Target went through a massive transformation where iron was replaced by hypervisors, and the companies in the know [1] published case studies (we have approximately a dozen more) discussing Target, their circumstances.

...until September 2009, Target’s POS systems and asset-protection ran on physical servers. By the second quarter of 2012, the company deployed 15,000 virtual guests running on more than 3,600 Hyper-V hosts across the entire store network. This includes 300,000 endpoints for servers, virtual machines, mobile devices, PCs, and POS registers.” This also includes an asset-protection solution. The list of technologies has had more than 25 CVE-rated vulnerabilities posted in only the last two months.

My point is this...

Networks are complex. Complexity causes pain... not sometimes; every time. Sadly, complexity is a necessary evil... and it's getting worse. 

And it's getting worse fast... far faster than builders and defenders can operate. 300,000 heterogeneous endpoints in 1700+ retail locations with 15,000 virtual machines running more than 3600 Hyper-V hosts. Add to this the "cloud" (I really hate that word!) that is the internal Target WAN connecting all the pieces, the external clouds used by the third party IT providers, the payment processors that connect (presumably centrally somewhere), and all of the other variables that go along with such a massive, geographically diverse, a non-IT oriented retail focused company. Add to that the fact that the third parties who run IT don't hold stock in the company and probably have a slightly less vested interest in their fiduciary requirement for managing the networks than they do in generating revenue from their customer... not a poke.. it's a fact of life.

Sorry Target. My best to you guys.  I'm certain there'll be some good lessons learned coming from this.

And "BZ!" to Krebs. Well done! Nice reporting sir!

BT BT

Next week will be the last blog of 2013. It's been a hell of a year.


  • 37 blue chip companies represented in Red Sky Alliance, with another dozen or so in Beadwindow. We wanted to keep it a small, trusted group. So far, so good. 
  • Thousands of running threads produced more than 40,000 high quality, properly primary sourced, non-watered down APT and targeted event IOCS in nearly 200 analysis products published detailing full context of the incidents; plus over 300,000 products collected from open sources, used for pivoting off the 40,000 analyzed by Red Sky and its members.
  • Wapack Labs opened to handle some of the non-information sharing requests. As an example, we recently delivered a country study that will be used by a governmental organization overseas to help them secure their small nation... good stuff, but not necessarily information sharing related.
It's been a hell of a year indeed. 

Ok, until next week, I wish you the very best holiday season possible. Next week will be our 2014 predictions post, so hang in there.. one more to go and it's on to the new year!

Merry Christmas, Happy Holidays!
Jeff






[1] http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?CaseStudyID=4000009407