During the last security incident that I worked on, I needed to grind through 20gb of log files looking for any odd log lines that would indicate the point where the bad guys got in. If I had done it manually, I would still be looking at log data. Instead, I built a tool that converted logs into pattern templates and looked for templates that I had never seen before. This allowed me to zero in on just a few hundred log lines out of all the data.
Templater is a small and fast log processor that provides simple artificial ignorance capabilities. You use the tool to process past log data and store templates that represent normal log line structures. You then run the tool against current or target logs and all normal patterns are automatically ignored. The parser is fast and capable of processing millions of lines per minute. For weblogs and firewall logs, I average 9M lines per minute on a 2GHz x86 machine running *NIX. The template strategy was originally proposed by a friend of mine in 2003 who later built a tool called never before seen (NBS) which also provides artificial ignorance for arbitrary text data as well as text structures.