Enterprises are swimming in a sea of logs. The deluge includes logs from servers, security systems such as firewalls and IDSes, events from network infrastructure devices such as routers and access gateways, and from various software and hosted services. Making it even more overwhelming is that the information isn't necessarily collected in a way to resolve security incidents in real time, or to troubleshoot situations that involve multiple segments of the enterprise network infrastructure.
Increasingly, however, IT administrators are under pressure to get a handle on their logging practices and manage log data. Various regulatory frameworks require some type of audit trail, making log management critical for demonstrating compliance, while the Payment Card Industry (PCI) Data Security Standard specifically calls out the need for log review. Logs can also be useful when the lawyers ask for evidence of who did what and when.
"We have seen a shift in the market toward regulatory and government-based standards to drive purchases of log management systems," says Chris Pick, Vice President of products and marketing for NetIQ.
Part of the challenge is the need to look at logging from an enterprise perspective, and move toward having a common and centralized repository for all logging data.
The ultimate goal is to have this single repository used for a variety of purposes, from satisfying auditors and responding to e-discovery requests in the event of a lawsuit, to managing real-time security threat analysis and network and applications troubleshooting.
"Logging standards and practices typically do not exist across an organization, and they are difficult to enforce even if they do exist," says Jay Leek, manager of corporate IT security services at Nokia.
Leek suggests enlisting the aid of the internal legal staff to help bring about this unification; there are numerous regulations that require logs to be kept for varying time periods, and in some cases disposed of after established deadlines for privacy reasons. For example, HIPAA mandates a seven-year retention period, while PCI requires one year. It gets more complex for global corporations, where European and Asian laws come into play. And logs need to be intact if they are going to be used as evidence in civil or criminal proceedings, placing other requirements on their use. Given the contradictory legal requirements, having lawyers as partners becomes essential.
"We need to bring together IT and legal departments to help put in place overall enterprise IT logging standards," adds Leek. "Lawyers tend to not be very technical people. If you can make it simpler for them, they will make things simpler for you. But don't let your legal department run a logging project; instead, incorporate their advice, and try to speak the same language."
A primary objective should be to prevent departments from setting up their own log management tools, creating multiple places where logs live, says Matt Stevens, CTO of the information and event management group at RSA, the security division of EMC. "You need analysis across the enterprise and to make it accessible for all users, and log management needs to be an element of the overall management infrastructure," he says.
Log management is also now part of the overall network security infrastructure. As blended threats become more frequent and more corporate applications make deeper use of the Internet for connectivity, having a unified logging repository becomes another tool in the security chest to protect the enterprise.
"It is not your father's security landscape anymore," says Robert Whiteley, an analyst with Forrester Research. "Nowadays, threat mitigation is deeply embedded into the overall network infrastructure. But how well you maintain your environment is critical, and there is a huge range in terms of how data can be exposed for analysis and manipulated."
Log managers versus SIMs
In some cases, a log management tool is seen as the first step in the analytical chain. But the market for such tools has become confusing, with dual product lines from log management vendors that have branched out into the security information management (SIM) space, and SIM vendors that offer other versions of their products for log management.
"Increasingly our midsized customers want to just get their auditors off their back and don't necessarily want to implement a full-blown SIM," says Tracy Hulver, Vice President of marketing and product development of netForensics.
The issue is that the two approaches--SIMs and log management tools--serve different masters and are often two different products, with SIMs focused on correlation and real-time alerts, and log managers emphasizing long-term archiving and evidence preservation. "Log management is more useful for ad hoc, after-the-fact investigation, while SIM is more concerned about codifying the business rules and notifying the security team to respond to a problem," says Hugh Njemanze, CTO and executive vice president of engineering of ArcSight.
As a result, some of the SIM and log management product lines have been developed independently, even those sold by the same vendor. For example, netForensics and NetIQ don't offer a common repository for their two product lines, although the former is working toward this goal and hopes to have a unified repository by the end of this year. "In the past we had a single repository for our SIM and log management product lines but there were some issues," says NetIQ's Pick. "Now we have a SQL database on the real-time side for the SIM, and have flat files that are indexed for the log archive server."
Another difference between log management tools and SIMs is how they analyze data. Most log managers do "free text" searching, which is useful for finding particular records that can be used for legal evidence. SIMs tend to normalize and analyze network events and can correlate different conversations between computers or IP addresses, which is useful in resolving incidents or tracking down exploits.
Choosing between a SIM and a log management tool depends on your organization's log management goals. If compliance and auditing requirements are your pressing issues, then start with a traditional log management tool. If you're more worried about breaches, start with a SIM.
In the end, you may decide you need both.