Add Post   Gallery
This is a community portal. Sign up on the left and start posting about analytics and visualization of security data.



Log Visualization in the Cloud - Webinar

On August 19th, at 10am PST I will be giving a Webinar on the topic of visualization. You can register and watch the Webinar right here:

A BrightTALK Channel

Cloud-based Log Analysis and Visualization

I was giving a talk at RMLL 2010, a french free software conference. The title, Cloud-based Log Analysis and Visualization, already gives the content away. But in case, here is the abstract for the talk:

Cloud computing has changed the way businesses operate, the way businesses make money, and the way business have to protect their assets and information. More and more software applications are moving into the cloud. People are running their proxies in the cloud and soon you will be collecting your logs in the cloud. You shouldn't have to deal with log collection and log management. You should be able to focus your time on getting value out of the logs; to do log analysis and visualization.

In this presentation we will explore how we can leverage the cloud to build security visualization tools. We will discuss some common visualization libraries and have a look at how they can be deployed to solve security problems. We will see how easy it is to quickly stand up such an application. To close the presentation, we will look at a number of security visualization examples that show how security data benefits from visual representations. For example, how can network traffic, firewall data, or IDS data be visualized effectively?

"Trojan Pong" and other malware data visualization ideas

"Trojan Pong" and other malware data visualization ideas

This small experimental project was done for the Shadowserver Foundation. They are a volunteer, Not for Profit organization who deal in the capture, analysis and dissemination of data and intelligence relating to nefarious activity on the internet. Shadowserver provided us with one day worth of data (which was several gigabytes) for us to apply some known techniques, and experiment with some new ones.

The idea of this project was simply to provide some ideas as to ways to represent their massive datasets visually. There's lot of work to go, however here are few early ideas. My favourite is a light-hearted time series visualization in the theme of an old favourite arcade game originally released in 1972 "Pong".

See all of the samples at

SSHD brute force attempts - userids and IPs

SSHD brute force attempts - userids and IPs

One of many tests with Afterglow, visualizing SSHD brute force logins (yellow) vs source IP addresses (green).

This one shows quickly the IPs that have the most activity (one IP has the most: the yellow explosion in the middle), along with popularly attempted userids, and the IPs which have been attempting the same userids.

Monitoring / Visualisation Stations, & relevance of layer 4 traffic

Opinions sought from those working in the relevant areas - handed this document in as part of a degree project in security visualisation & monitoring, and the feedback was that the network and monitoring station/s are not realistic, and that I should have focused on port 80 and layer 7 traffic only, as layer 4 is not relevant any longer. The link provided below is only part of the document, I presume it's the part they had issues with. I wasn't actually intending to focus on web traffic, which was made clear in the document anyway (tho I did indicate to them that with the likes of Rumints packet contents visualiser, it is certainly viable to utilise that to match up with malware signature databases - but that aspect wasn't the focus of the project).
I don't expect it says anything that people working in those areas will be unaware of, and the general intention was to address what would be required for a monitoring station / network, which includes visualisation software, that would work in real-time as well as offline analysis and traffic capture.
The grouping into 'objectives' is just part of how the work has to be presented to comply with guidelines. Cheers for input, I know you're probably busy.

nb - the last part is probably wrong about ad-hoc IPs; I can't remember exactly right now how they are handed out; they probably aren't always dynamic esp. now it's more common to get fixed-IP SIMs.

Spam - A 2 day comparison with afterglow.

Spam - A 2 day comparison with afterglow.

I finally got my spam stats up and running. The results are amazing.
Lightyellow = Subject || Red = Sender || Black = Recipient

It is pretty easy to find the one user that appears to get a significant amount of Spam :). If I had to guess, I would say the single subject, large source and large destination likely originate from Botnets?

The results are from Wed and Thurs of last week.

Libemu sctest' output, created from PDF shellcodes

Libemu sctest' output, created from PDF shellcodes

I extracted this image using PDF malware that I got for analysis purpose. By using perl script I filter out the unneeded content and later put it in sctest(libemu tool). The graph created using dot command in Graphviz package

EDV - Event Data Visualization

Afterglow has been on my list of 'neat tools' for quite some time. Thankfully, last month I finally had a bit of spare time to really play with it.

The result was EDV:

See the page for more info. Keep in mind, this is BETA!

It currently supports Snort (Sguil DB format). However, even the untrained eye can easily modify it for straight Snort
or anything else you can MySQL query. Once you have your sources defined it will take care of the rest.

The tool is static (controlled by configs and cron) for now but I do plan on adding a query tab to the web page so that you can do on the fly queries. Low priority for now. I have been focusing on 2 parsers that log directly to MySQL. One parses Syslog output from a Barracuda spam firewall and the other URL info captured by URLSnarf. These will be my next additions.

Comments and suggestions welcome.


Zombie network activity representation by Dorothy

Zombie network activity representation by Dorothy

This graph is automatically generated by the Dorothy framework anytime a new malware is analyzed.
It aggregates three different kind of information : 1) the network activity 2) the dns host resolutions 3) the GET / POST resquest
In this way, we can be able to easily define certain activity related to botnet communications.
A quick legend :
Colors :
Green = Services / hostnames
Red = General target
Purple Red = Known C&C ( in this example there isn't any)
Purple = C&C Web target
Light blue = private network host

Circle = Target
Triangle = Source

The shape's dimension represent the network activity related to that node.

FDP visualization for Nepenthes using Afterglow and python-geoip

FDP visualization for Nepenthes using Afterglow and python-geoip

I created the image by using Nepenthes' log, later put the country info by using python-geoip. Finally, use Afterglow and graphviz to illustrate them.