Benefits of Visualization

So what are the benefits of visualization over other techniques? My favorite answer is this:

  • "Visualization not only helps you answer questions that you have, but it elicits questions that you did not even think of before. So for some things you can come up with algorithms to solve your problems, but for others, you don't even know your problem upfront!"

There are many more benefits to visualization. Here are just a few:

  • The bandwidth of data you can transfer in a picture is much bigger than having a human look at log files or textual data.

  • Relationships become very apparent. Sometimes they are completely hidden without visualization.

  • Interactive visualizations benefit from dynamic queries which are an incredible tool to explore data.

  • Visualization inspires. You look at a picture or a graph and suddenly you realize what is really going on.

  • It's a great tool to communicate information in a very compact and often easy to understand way.

  • It definitely reduces analysis and response times. Sifting through thousands of line of logs is definitely slower than looking at a few graphs of the same data.

I am curious what other's think. Let's add to the list!

NSF - Science and Engineering Visualization Challenge

The National Science Foundation (NSF) has a challenge for science and engineering visualizations published. I am not sure if I have some visualizations that would qualify for the challenge. But maybe some of you have security data that could make the bar. I think it would be great to draw attention to visualization in the security space. So if you have something. Submit it!

Iptables config

Iptables config

This graph shows the IPTables output graphically.
Blue is for UDP and Yellow for TCP.

Generated by Ruined (http://ruined.sf.net)

Spider attack on a web server

Spider attack on a web server

The red pillar in the image shows the barrage of HTTP requests over the whole content space (Z axis, vertical) from a single IP address (Y axis, horizontal). The red color is due to 5xx status code of the response. My article
A New, Improved Visualization for Web Server Logs
has more details. Raju Varghese (raju -at- intellisoft.ch).

Problems with Visualization?

On his blog, Anton started an entry about logging and gets into the topic of too many logs. I was suggesting visualization to analyze the vast amounts of logs in order to get a better handle/understanding of them. Anton countered with this:

Is this really the place to start a visualization fight? :-)

You know what my issue with visualization are:
- tools need really skilled analysts
- often the resulting picture is no more insightful than the original
 log pile
- I kinda prefer an analytic system which is smart to a visualization 
system which is... not so smart.

Let's move this discussion to secviz :-)
Here ya go. To answer Anton's objections:
  • You need skilled analysts to read log file in the first place! So no excuse. I would even argue that visualization makes it easier on the analyst! I agree that we need better tools nevertheless!
  • I agree, _sometimes_ the pictures are not more insightful. But in general they are. I think what is missing are good guidelines on what graphs help with what situations. I am working on that.
  • Visualization has the benefit that it not only helps you answer questions that you have, but it elicits questions that you did not even think of before. So for some things you can come up with algorithms to solve your problems, but for others, you don't even know your problem upfront!
  • I am curious to hear what others think.

    Graph Visualization Survey

    Another excellent paper or in this case a survey. The authors do a great job of surveying the space of structured data visualization. They explain very well what graph layouting is, what the different algorithms are, where the problems are hidden, what the solutions are, how interaction plays into all of this, and also discuss three-dimensional views and what their benefit is. Awesome survey, really worth reading if you are interested in graph layouts.

    Perception in Visualization

    I read a fantastic paper on visual perception. A must read for everyone designing visual systems. The paper is called Perception in Visualization, written by Christopher G. Healey. The paper is very very practical. It presents the theory behind perception very well and always gives examples. Some of the topics covered are:


    • preattentive perception

    • Feature Integration Theory

    • Texton Theory

    • Similarity Theory

    • Postattentive Vision

    • Change Blindness


    Again, a very well written and very educational paper!

    Visualization Trends in Security Products

    It is fairly interesting to see how security prodcuts are maturing. In the last couple of years I have seen quite some progress in products using visualization. Let's look back a few years. Network-based IDSs, for example, logged events in a log file; text [and some still do!]. Over time, reporting was added; a way to summarize historical data. Drop a pie-chart on the report and you have something that you can hand to your collegues. Shortly after that dashboards came about. Finally we had something to show to our managers, not just our peers. Most products have a dashboard today. Not all of them are very useful, but at least they have one ;) The next evolutionary step was to link the dashboards with the data itself. Drill-down was added.

    And this is where we are today. Most products are at this stage. Only a few products took this a steps further. They added for example dashboards that link to other dashboards, which show more specific information. Some products even offer customizable dashboards (not all do!). You have the capabilities to either build your own or change predefined ones.

    There are only a handful of products in the security space which take visualization a bit more serious. Thos products offer visual interfaces which support dynamic queries [basically the capability to let you change/interact with the graphs on the fly.]. This is clearly how it should be. It gives the user the tools he needs to interact with the data.

    I am very convinced that dynamic, interactive, visual interfaces are going to be added to more and more products. They are incredibly powerful and invalueable for data anlysis and representation!

    Visualization Features in Security Products

    I was attending the RSA Conference all week long. During one day my mission was to find out what the state of visualization in security products is. Here is what I found:
    - Most products have reporting features
    - A lot of products use dashboards which let you interact and drill-down into the details. This generally means clicking on one of the bars in a bar chart to get to the underlying textual representation of the events.
    - Some products use drilldowns to get from one dashboard to another (nice!)
    - Some proudcts let you customize the dashboards or change the visualization parameters interactively. Keyword: Dynamic Queries (very nice!)
    - Only one company that I talked to uses a visual interface (a treemap) as their main way of interacting with the product. They even let you change the parameters on the fly! (very very nice!l!)

    My whish list:

    - More visual interfaces.
    - More interactive dashboards. Being able to drill-down from one dashboard into another to get more information.
    - More meaningful dashboards. Tell me why a certain graph is important in the dashboard. What's the use-case for showing it?
    - More products using better visualization (have you heard of treemaps?)
    - Interactive visuals. Let me choose how I want my data represented. Make it configurable. But don't overload the interface with features. Make sure there are valid use-cases and make them obvious to me! Wizzards?

    Visualization Programming Language

    I am pretty amazed with the Processing project. It's a full-blown, java-based programming language which has added commands to generate 3D graphs. I played around with it and pretty quickly built a tool which plots 3D coordinates which are stored in a file, onto the screen. It's fully animated, interactive, etc. The real killer is that the tool will generate a JAR with the entire code executable on Linux or Windows OR as an applet. Really worth having a look at!