PREVIOUSLY: Introduction to logging issues
Whichever type of product you choose, reporting capabilities are critical. Tracy Hulver, Vice President of marketing and product development of netForensics says most customers want many different items in their reports, so his company provides some templates, but also makes it easy for clients to create their own.
Customised reports are crucial for compliance purposes, but some aspects of the reporting process are also needed for real-time threat analysis. "We put all the information online so the security professional can solve problems immediately," says Jim Pflaging, president and CEO of SenSage. "You can also set rules that look into the past, and can do threshold and violation alerts based on long-term trend analysis. That also has advantages for compliance."
How a product handles ordinary network traffic flow is also important. Some products, such as netForensics' nFX Log One and RSA's enVision, don't offer the ability to import this information, making it more difficult to correlate particular network events with security breaches. "It is on our 2008 product road map," says Hulver.
In addition, IT staffs need to work with logging application program interfaces that can import other kinds of logs--such as those from custom applications--into a central repository, and query mechanisms to search through this archive to get the right kinds of information out of it. "Many times there are custom ERP applications that make use of .NET or J2EE that require their logs to be aggregated, so it's a must to have an extensible API to do that," says Chris Pick, Vice President of products and marketing for NetIQ.
Another factor in a purchasing decision is how scalable and extensible a solution will be, and what happens to the existing repository when new logs are imported or added. Vendors have adopted their approaches to address this issue. "We can easily extend our data schemas without having to change the underlying database or the collection process. We have built a relational approach without all the overhead associated with the relational database," says Pflaging.
"We developed our own purpose-built, object-oriented database that allows us to scale to billions of daily events," says Matt Stevens, CTO of the information and event management group at RSA. "We can easily generate metadata when new data sources are added to it."
To syslog or not to syslog
Standby in the logging world is syslog, which provides a framework for collecting and storing log data but has well-known performance issues and can drop some data during periods of high network use. Some vendors also support a more recent version called syslog-ng (for next generation) that includes delivery using TCP instead of UDP.
"Syslog-ng tries to solve that problem with guaranteed delivery, but that can slow down the collection process," says ArcSight's Njemanze. The trade-off is having a high-performance collector that misses log events but keeps up with real-time traffic analysis for threat mitigation, versus having something more complete but lags behind in real-time collection.
"When you are capturing all this log data you shouldn't be forced to filter or normalise any of it, because that slows things down," says Stevens.
As a result, LogLogic offers two different log management product lines. One stores its logs in a SQL database, while the other uses raw files. "It is important to do both," says Anton Chuvakin, director of product management of LogLogic. "Some users of log data want the flexibility to do visualisation and compliance reports, while others want to be able to do full text searches."
"In practice, most of our customers tend to go with traditional syslog because they want to see current messages, even if this means that they lose a few in the collection process. Whichever method you employ, make sure that the system you use to capture logs has the capacity to keep up with the message traffic," says Njemanze.
"Syslog is pretty bad and has all sorts of issues, but it's also really common, and there are millions of devices that write to its format," says Chuvakin. "Sometimes convenience can override security concerns."
Without a doubt, log management is a tough task to tackle, but the security and compliance benefits it can provide have become essential. And while the market of available tools that can help ease the process is rather convoluted, it may become clearer as vendors hone their products. Both log managers and SIMs will continue to converge as vendors add features to complement and extend their product lines. For the next few years, however, it is likely that IT and security managers will need both kinds of products to satisfy multiple needs.