There is no doubt that one of the major problems facing organisations of all sizes is the retention of data. Historically, the problem was answered by simply throwing more disk space at the issue and hoping that the reason for storage would never really come back to haunt the organisation. It is this reason for storage that is now the driving force behind new solution requirements. Storage for storage’s sake is no longer the issue. Data exists for a purpose, and while transactional data has found a home in relational and flat-file systems, from where it can be recovered easily, there is a large volume of data that is stored for no other reason than there is a legal requirement to do so. Business event data is that class of data that is read-only, and therefore unchanging. It forms the record of an event happening – such as a telephone call. Where to store this data is the problem.

Keeping such records is only one part of the problem. If there was no requirement to ever access the data, then storage style is not an issue. However, if there is a requirement for retrieval then it is best stored in a relational system. The idea of storing such huge volumes of data in expensive relational systems is a non-starter. Therefore organisations are faced with a clear dichotomy. They can store data to answer legal requirements, but in order to retrieve that data it has to be stored in a ‘searchable’ system; which is hugely expensive. We have reached a stage where organisations are paying lip service to storage legislation, and paying the financial penalty when someone comes along and asks for some specific data, which is impossible to retrieve. The fact that these penalties have, in the past, been in the order of several millions of pounds, amply demonstrates the scale of the problem. It is cheaper to take the risk and pay the penalty than implement a retrieval system.

This is not a situation that can continue. Regulatory authorities are not going to carry on handing out fines, they are going to take more stringent measures to ensure full compliance. Thus, organisations are facing a problem of immense proportions. The solution to which has been addressed by CopperEye with its Greenwich product.

CopperEye has seen the problem and answered it by allowing organisations to store this business event data in cheap repositories, but applying indexing that makes the data responsive to SQL statements.

CopperEye Greenwich is non-intrusive and does not alter, move, or delete the data files – but simply discovers, parses, and presents their contents through relational views available for SQL query over ODBC from either a relational database management system (such as Oracle) and/or business intelligence tool (such as Business Objects).

Greenwich operates as a seamless component of an enterprise architecture, behaving as a remote database server that connects to other relational databases and business intelligence tools. It autonomously discovers structured flat files on the underlying file system and parses and indexes their contents to present them as views over ODBC. Thus Greenwich does not load or move the data it finds, but provides direct access to it via standard SQL.

Greenwich provides access to enormous volumes of unchanging business event data without the time delays associated with traditional database solutions. In many instances unchanging business event data can be accessed through Greenwich without ever needing to be loaded into a database – significantly reducing the hardware required for production applications, enhancing both the application’s performance for end users while dramatically expanding the length of historical business event data available from within the application.

Greenwich provides rapid, highly selective query access to precisely the business event records of interest, even when they are buried within literally billions of other records, as far as querying goes it has the same functionality as a relational implementation, but with much higher throughput and without the associated management costs.

Implementation time is also rapid, with terabytes of data under management within a matter of weeks, and can be extended to manage tens- or hundreds-of-terabytes shortly thereafter. This performance is achieved using CopperEye’s patented indexing technology and is accomplished without the need for costly and time-consuming tuning to deliver the extreme performance promised from the solution. Greenwich can be easily demonstrated at enterprise data volumes in a matter of ten working days.

Finally, Greenwich has already demonstrated a total cost of ownership that is 85%-90% less than implementing a commercial database solution of comparable performance. As a result, Greenwich virtually eliminates cost as a barrier to storing as much data as is required, keeping it indefinitely, while having immediate access when needed.

For several years now, CopperEye has used its indexing technology within specific solutions, but mainly concentrating on increasing performance metrics for relational databases. With the entry into the business event data market, it has found a home for its strong technology that will, in our opinion, make it a well-known name in a very short space of time.