In the past few years, companies have spent billions of dollars to update their IT infrastructures to meet requirements from various European and US government regulations.

One of the more noticeable and most important recommendations of these regulations is record-keeping. For example, Sarbanes-Oxley recommends that all companies "maintain financial records for seven years." In order to ensure the accuracy of corporate financial and business information, this recommendation also pertains to records that are used to "audit unauthorised access, misuse and fraud." Other regulations such as HIPAA also recommend keeping records for up to six years.

Altered log data prohibits court admissibility
The integrity of information is crucial when submitting evidence to the court. Just like crime-scene evidence, which prosecutors must prove hasn't been tampered with, electronic data submitted to the court must adhere to the same stringent requirements. As such, log data generated by the IT infrastructure also has to be archived in its original and unaltered format.

Reports generated from the logs are usually insufficient to convince the other side (defence or prosecution) that they haven't been tampered with. Lawyers from either side may question the accuracy of the reports and will want to perform their own analyses. For example, if you claim that someone has sent out data from the financial servers, how do you substantiate that claim? Tampered data can't be used as evidence to prove your claim. In these scenarios, unaltered logs have to be provided.

In addition to the unaltered logs, evidence may be needed to prove that the logs weren't tampered with. Some companies have chosen to digitally sign the log files collected and then keep the digital signatures at a location separate from the logs. Others have chosen to store logs on WORM (write once, read many) drives such as CD-R/DVD-R or storage devices such as EMC's Centera. Both processes ensure that tampering of logs can be detected or prevented.

Documented collection processes enable trust
But why would the court or the auditors trust the archived unaltered logs? Auditors are always looking to see whether the log data can be tampered with or modified at any point during the collection process. Was the transport encrypted over the WAN to ensure confidentiality? Were the logs signed during transmission to ensure integrity? What programs or processes handled these logs during the collection process? Are these programs or processes clearly documented to ensure that no fake data was injected into the stream? Were any users involved during this collection process?

This is where clear and detailed documentation on the collection process is required. The process of how logs are handled from the point where logs are generated to where logs are archived - and everything in between - must be clearly documented to prove that the log collection process is reliable and secure and that no data was tampered with. The documentation process should include details such as the encryption or digital signature algorithms used during transmission, the likelihood of data loss during the collection process, any manual process that required human intervention and users who touched any of the logs,

Long retention periods allow timely investigation
Even though there's no explicit regulatory requirement that companies must keep all log data for the full recommended time period, many experts agree that for Sarbanes-Oxley or HIPAA compliance, unaltered logs should be kept online for at least 12 months. However, your auditors or your corporate policy may require a longer retention period. If you don't already have a corporate information retention policy, create one now.

Having an online archive of log data allows timely investigations and also provides long-term reporting for the auditors. Many security investigations, especially those involving security policy or acceptable-use violations, may require mining of logs as far back as 12 months to ensure that no details are missing. Without the log data online, the investigations will take much longer, since the IT administrators would have to restore logs from off-line backup, such as tapes.

Regarding financial auditing, the auditors may also want to go back several quarters to look at the financial results. In order to prove the integrity of the financial data, related log data might be required to prove that there was no unauthorised or inappropriate access during those periods.

The importance of keeping unaltered logs for evidence, whether for the court, the auditors or human resources, can't be underestimated. It should be one of the most critical requirements when building your compliance infrastructure.

Jian Zhen CISM CISSP is the director of product management at LogLogic, a Californian log management vendor. He has been in the information security industry for nine years and also has a blog, Operational Intelligence.