In Part 1 of “Tales of the Black Hat NOC: The Stages of Security Adolescence,” I discussed the maturation process of the Black Hat NOC, and security strategies in general. In the blog post below – you can see the adjustments we made and additional steps we took towards optimizing our NOC at Black Hat.
Emotional Development (Day 5-6)
*If teenagers can be said to have a reason for being (besides sleeping in on weekends and cleaning out the refrigerator), it would have to be asserting their independence.
Monitored Classrooms (continued)
Provided classroom specific reports identifying abnormal behavior
Met with NOC networking resources to map network and understand RSA Netwitness tapping points/visibility
Suspicious binaries by classroom
Small sample from Malware Analysis – Obvious that hunting down the “bad” will not scale.
Tuned alerting, customized NOC dashboards,
Even with tuning, still received flood of alerts – it is the Black Hat network after all.
At this point, we had to make a choice:
We could continue to hunt down every squirrel, likely having loads of fun and feeling productive with all the malware we would find.
Black Hat analyst Sconzo, proud of the recent malware find
We could take a step forward in our emotional development and assert a bit of security “independence”. It would require leaving the flashy malware samples and ICMP tunneling behind in favor of a quieter, more targeted approach: using Business-Driven Security to focus on the assets most critical to the Black Hat NOC.
While there was still some squirrel chasing, we began to narrow our focus. And RSA Netwitness, a solution that provides complete visibility into the conference network traffic, was identified as our first critical asset.
RSA Netwitness Authentication Dashboard. Each analyst’s device was tagged within Netwitness. Unknown events become identifiable and actionable
Regression (Day 7)
Like a teenager coming home with a fresh tattoo, not all changes are positive. On day seven, we stumbled on our path of security maturity. Classes ended and the Black Hat network was completely overhauled. The conference wifi spun up overnight, which destroyed our classroom context, and provided us with a brand new, large, and flat network.
Tune, Tune, Tune, Goose
Emotional Development Continued (Day 8)
While the network change set us back in regards to familiarity, the communication and process already put in place, allowed us to catch up and adjust to the new network. Similar to days five and six, we had additional meetings with Black Hat NOC network resources (beginnings of a security program/process taking hold?). We were able to move forward and identify additional, security sensitive points in the network.
Identified/defined critical network segments
With a baseline established, inbound network activity to critical management networks can be monitored for unknowns or anomalies.
Identified Registration system as an un-monitored, critical asset
Day 8… better late than never?
Social Development (Day 9)
*Not all teenagers (security programs) enter and exit adolescence at the same age or display these same behaviors. What’s more, throughout much of adolescence, a youngster (analyst) can be farther along in some areas of development than in others.
As much as I want to describe the uber mature end state that we achieved, there is only so much progress we could make in a week. While this development stage seems to be the perfect tie in to threat intelligence and information sharing, we are already approaching our attention deficit limit so it will have to wait for another time.
Day 9 Tasks: Tear Down
Too long in the NOC or too long in Vegas?
My own quick summary of the growth within the Black Hat NOC security program…
Progress, certainly not perfection.
We ran into issues that many organizations face when trying to implement a Business-Driven Security strategy. One of those issues being the existence of silos and difficulty communicating priorities across those silos. While significant obstacles during a time-constrained conference, they are issues that can be solved with time and effort. For the most part, the other vendor silos within the Black Hat NOC were extremely cooperative. Our struggles occurred when other team’s priorities did not line up with security monitoring priorities – i.e. fighting to keep the network up and running, patching a vulnerability that was announced DURING Black Hat, or generally getting caught up in the whirlwind associated with running a network for a week-long hacker conference. To identify and monitor more of Black Hat’s critical assets going forward, information sharing is needed early and often (before fire alarms have a chance to go off for any team required in the information sharing).
These are issues that every SOC faces on a daily basis and something we can always improve on moving forward.
And that is what’s so great about security – we learn, we mature, we continue to improve. If we are lucky enough to work Black Hat again, I look forward to what that summary report will look like…I can tell you one thing, it will contain a lot fewer malware samples and a lot more risk based analysis.
