Add Post   Gallery
This is a community portal. Sign up on the left and start posting about analytics and visualization of security data.

 


 

Display Time in Link Graphs

I just wrote a blog entry about some ideas of displaying time in link graphs. This is a problem that has bugged me for a while and I still don't have a good solution. The blog entry outlines some ideas and alternatives. Maybe you have a better way to visualize relationships and time in the same graph?

Housekeeping - Comments to entries

I have made a minor change with regards to letting people post comments to discussion entries. It used to be the case that anyone was able to post comments on the site. Unfortunately that meant that I got spammed quite badly. I realized that I had a huge approval queue for comments. I went through some of them and published them. Sorry if I deleted a comment of yours. Please repost if your comment got lost.

From now on, new comments can only be posted when logged in. Sorry for the inconvenience, but this should help a lot to make discussions more interactive through the comments.

Thanks for everybody that commented on broken links and such. I hope I fixed everything at this point. As always, if you have any input for the site, please let me know. Either by sending me an email or posting something here. Thx!

Picviz: Let's see uncommon URL (part 2/?)

Picviz: Let's see uncommon URL (part 2/?)

Today, I would like to see if the urls that are not common in the previous graph, In this graph, heatline rendering plugin is used to check with line coloration if an event is regular. In the fourth axis, you can see lines going at the bottom and red lines go there. So let's forget about this and filter to only display lines that appear above 50% of this axis.

The filter is between single quotes, just like what you'd do with tcpdump ( I actually took their code to handle this ;-) ).

This line was typed to get the graph you can see here:
pcv -Tpngcairo -Rheatline -Avirus access-wallinfire.net.pcv 'show plot > 50% on axis 4' -ra > picviz-uncommonurls.png

If we take a random IP, such as the one we clearly see on the second axis, 213.192.60.19, and googling about it, we find that this was an infected machine. The url here tells more about it.

As a conclusion for this graph, you can see that among all those lines of log, with a very empiric approach, we really discovered something. Not a very innovative attack I admit, but enough to keep searching (I will post ongoing researches here, keep following!).

Ah, and by the way Raffy, since you asked to only display lines every few times, I added the -L option, taking a number (N) as argument meaning every N lines you display the text.

Picviz graphing apache logs

Picviz graphing apache logs

This parallel coordinates graph shows 412429 lines of one of my wallinfire.net access log with generated with Picviz svn. This is the first of a set of graphs which will derivate from this one. The most complete one.

To generate such a graph, simply use the apache-access2picviz Perl script available from trunk/tools. Then, use the heatline plugin to see line frequencies: the more green the line is, the lowest it appears. When a line is in red, it means it comes often. This way you can easily see if an event is regular or not. To generate this image, you can type: pcv -Tpngcairo access.pcv -Rheatline -Avirus -rra > accesslogs.png

First axis = Time (24 hour) with 00:00 at the bottom and 23:59 on the very top.

Second axis = Source IP with 0.0.0.0 at the bottom and 255.255.255.255 on the very top.

Third axis = HTTP request type.

Fourth = Request

Tomorrow, I will post a filtered graph, on the request axis to see what are the IP addresses that are doing abnormal requests.

Picviz is available as free software on http://www.wallinfire.net/picviz

SecViz now has a Twitter feed

Follow SecViz on the brand new twitter feed: @SecViz.

 

Security Visualization Workshop in Hong Kong

As part of the ISSummit in Hong Kong, I will be teaching a one day workshop on security visualization. The following is the abstract of the training:

As networks become ever more complex, securing them becomes more and more difficult. The solution is visualization. Using today's state-of-the-art data visualization techniques, you can gain a far deeper understanding of what's happening on your network right now. You can uncover hidden patterns of data, identify emerging vulnerabilities and attacks, and respond decisively with countermeasures that are far more likely to succeed than conventional methods. The attendees will get an overview of visualization, data sources for IT security, and learn how to generate visual representations of IT data. The training is filled with hands-on exercises.

The talk is going over the following individual topics:


  1. Section 1:Visualization
    Visualization is the core topic of this training. The first section introduces some basic visualization concepts and graph design principles that help generate visually effective graphs.

  2. Section 2:Data Sources
    Visualization cannot exist without data. This section discusses a variety of data sources relevant to computer security. I show what type of data the various devices generate, show how to parse the data, and then discuss some of the problems associated with each of the data sources.

  3. Section 3:Visually Representing Data
    Data can be visualized in many different ways. This section takes a closer look at various forms of visualizations. It first discusses generic graph properties and how they can help encode information. It then delves into a discussion of specific visualizations, such as charts, box plots, parallel coordinates, links graphs, and treemaps. The section ends with a discussion of how to choose the right graph for the data visualization problem at hand.

  4. Section 4: Data Visualization Tools
    After a short introduction to different data formats used by visualization tools, this section then discusses visualization to
    ols and libraries. Based on the Data Visualization and Analysis UNIX (DAVIX) distribution I show how simple it is to generate
    visual representations of IT data.

  5. Section 5: Perimeter Threat
    This section is a collection of use-cases. It starts out with a discussion of use-cases involving traffic-flow analysis. Everything from detecting worms to isolating denial-of-service attacks and monitoring traffic-based policies is covered. The use-cases are then extended to firewall logs, where a large firewall ruleset is analyzed first. In a second part, firewall logs are used to assess the ruleset to find potential misconfigurations or security holes. Intrusion detection signature tuning is the next two use-case. The remainder of the section looks at application layer data. Email server logs are analyzed to find open relays and identify email-based attacks. The section closes with a discussion of visualizing vulnerability scan data.


Should you be in Hong Kong on November 20th, come check out the training. Should you miss it, I will be teaching a two day workshop at SourceBoston, Boston in March 2009.

GEO Tagging Attacks

GEO Tagging Attacks

I am working on a Parser that transforms any kind of ASCII input log file into KML files used in Google Earth/Maps.

What does it do?

1. Reads the log
1a. Extracts an IP (column or Regex based)
1b. Extracts a node name (column or Regex based)
1c. Extracts a description (column or Regex based)
2. Retrieves a longitude and latitude to the given IP address
3. Writes a KML file for Google Earth

Why?

Well I was recently analyzing a big set of IDS events with abut 99% of false positives. Most of them originated from partners of the company located in England and France. Only a few of them - the real positives - originated from countries like Iran, Pakistan, Brazil, Russia and China.
That way it was really smooth to determine the real attacks and leave the false positives aside.

The picture is an animated GIF. Hope this format comes through. :)
It shows an analysis of a mod_security log file.

I'll provide a download link for the script soon. (the sticking point is the GeoIP database; most of them are commercial ones; currently I use TOR network and a GEO IP web service to determine the LAT and LON parameters. That should not be the final solution, right?!)

---- Update -----
I use a local database now which makes it very - Very - fast. It parses 8500 mod_security events in 10 secs. The resulting KML file has 5 MB. I generated a new picture set showing all events of 9 month placed in a world map. And I have to say ... "bad - bad bad Italy!". I am quite sure that this was the spreading of the MPack Malware Kit this spring.

--- Update 2 ---
I advanced the script with an option to define a source and target IP address to draw lines in the world map. Nice, isn't it? Now it is possible to parse log files with source and target to determine the connections between the nodes. Targets are marked with different icons.

Video Interview for Applied Security Visualization

I recorded a short, 10 minute video where I am interviewed by Johnvey Hwang about the Applied Security Visualization book. We are talking about why I wrote the book, what the book is about, and also quickly talk about DAVIX. Tune in.

 
 
 
 

Skyrails 3D OpenGL visualisation

Skyrails is a social network (or any graph really) visualization system. It has a built in programming language for processing (as far as visualisation attributes goes) the graph and its attributes. The system is not only aimed at expert users though, because through the scripting languages menus can be built and the system can be used by any users.

The main distinguishing point of the system comes from the built in scripting language, the added flexibility of how to represent attributes (nodes can be binded to planes and spheres based on their attributes) and the scriptability of the user interface system. This makes skyrails ideal for creating presentations targeted at the average users.

http://cgi.cse.unsw.edu.au/~wyos/skyrails/

skyrails in action:
http://www.youtube.com/watch?v=I2d312_dXEs