Weta Digital's near-Petabyte storage infrastructure has several interesting aspects but is it relevant to the generality of businesses, small, medium, or large?
I'd argue not.
1. Weta only deals with files. There is no need for sub-file level access by applications. There is nothing like a database with records and fields needing to be accessed. That provided the gateway to a SAN-free environment.
2. Weta has no legacy computing baggage. It only came into existence a scant thirteen years ago. There are no generations of software, borne on mainframes and minicomputers and then migrated and modified onto Unix servers and Windows servers and three-tier architecture and object programming and Java and ....
Weta has a very large but very sparse environment with relatively few applications that run and have run on first Irix and then Linux servers. These servers are Intel x86 architecture blades. Compared to the average data centre and its meandering and contorted history this is a lean, frightningly mean, and exceedingly large infrastructure that is light years away from an enterprise data centre. It is more akin to an academic high-performance computing environment but without that environment's software legacy.
Weta has had all the advantages of starting its computing environment on a green field.
3. Weta has no general business apps running on its renderwall and other supercomputers. It only has a few apps, such as the aptly-named MASSIVE. These apps are all part of the same application set geared to producing films combining live and artificial, SFX-ridden, sequences. A slimmed down application set, compared to the average enterprise mix of applications, has been paired with a slimmed down IT hardware infrastructure.
Whatever general apps Weta runs, such as quarterly financials and HR payrolls, run somewhere else. They may run on the supercomputer clusters but if so run on it as quasi-guests and not as prime drivers of IT purchase and connectivity.
4. Weta has, largely speaking, a uniform operating system environment - Linux and some Irix. That marks it out as completely different from most enterprises.
5. Weta has a need for totally massive file movements almost from end-to-end of its back end storage<->workstation infrastructure. The infrastructure is geared to providing for these massive information flows and protecting the data that is stored and created. Most enterprises have a national if not international flow of data of all sorts, from mail to spreadsheets to inter-app messages to web browser requests.
Again Weta has an essentially local environment - in a single data centre - with no wide area network carrying huge amounts of highly differentiated data traffic.
6. Weta has no need for any information lifecycle management. Again it has only had a very short life and its data is homogenous - movie content bits and pieces.
7. Weta has no need for any storage virtualisation. It needs an immense silo to hold its data and this silo is three-tier: online; nearline; and offline. It's creating files at such a rate that disks just fill and it gets more and they fill. It doesn't need to migrate data between arrays much or to improve disk utilisation - the two main reasons for getting virtualised disk storage.
On these seven counts - and no doubt there would be more if we looked more closely - Weta is so far beyond the average enterprise experience that it can only be used for not very granular conclusions, such as:-
- NAS can be used at an enterprise level
- LTO is preferred to DLT
- Gigabit Ethernet can sustain humungous network traffic
- Tape has a role as the ultimate repository of data for long-term storage
None of these four items are news really. So Weta is impressive but not in our everyday universe. Like the films it renders it is a special effect and not real, enterprise, life.
Find your next job with techworld jobs