Storage for Editing - Build it and they will FILL it!

Big rooms for data storage are not new!

Since I first bumped into non linear editing in the early 1990's the capacity and size of digital storage has long been a point of great debate. So I thought I'd have a look back and perhaps a look forward to see where we can all end up

History

The 1990's
I worked for a UK Broadcaster in the 90's and it was here that I bumped into Avid and Lightworks, at the time, the two most dominant NLE systems about. Still operating as
offline systems, I fast became involved in helping manage the systems (Avid in particular) and their storage woes. SCSI drives were the order of the day and we were forever worrying about the length of cables, the number of drives in the 'stack', SCSI ID's and drive order (wide before narrow, Ultra before Fast). Let's also not forget the endless 'de-fragmentation' process that was needed here too! Phone conversations with editors to help address storage issues (or lack of them) required an endless stream of questions and trust that the person on the other end of the phone, was actually following your requests. a typical 1997 phone conversation would be...

Editor: "All my media is Offline. I can't see the drives on my desktop!?"
Engineer: "Can you see the actual drives? Are they physically turned on?"
Editor: "Yes, it's turned on."
Engineer: "So the drives are all on? Not just the computer." 
Editor:  "The drives? Er... what do they look like?"
Engineer: "They'll be close to the computer on the floor and maybe cream or black in colour, they'll have a light on the front if they're turned on."
Editor: "No they're definitely on."
Engineer: "Can you just check the cable is plugged in between the disks and the computer?"
Editor: "Hold on." Sound of fumbling under the desk "Ah no they're turned off. Should I turn them on?"

You get the idea.

Looking back its hard to imagine that 3 or 4 programs would share an edit suite and their video content (offline resolution obviously) co-exist on one or two 2GB of SCSI storage.

Above: Bottom drive is a 5 1/4" Avid original drive, more then likely 'Narrow' and the top one 
is the 'Barn Door' update, that could be up to 23GB in size and 'fast' and 'wide.'

The 90's saw NLE systems progress from offline systems with two or three 1GB or 2GB External "Classic" SCSI drives into online suites with broadcast quality Standard Definition pictures being delivered from Ultra Fast and Wide SCSI drives stacked to knee high! Systems such as the MC1000 and MC4000 were the Avid online systems of the day and Lightworks also rolled some heavy guns out with the HeavyWorks which offered multi-cam editing and broadcast quality pictures.

Disks were now commonly 73GB in size and it was not uncommon to find between 4 and 8 drives on a single system (device ID hell). People were becoming hungry for storage as picture quality improved and system's stability grew. 

The 2000's
This saw Avid release Unity MediaNetwork and change away from SCSI drives to shared storage (for those with big budgets) While the rest of the world moved on with easily attached external eSATA drives in FireWire and USB enclosures. Final Cut Pro started to emerge as the new kid on the block and the Mini-DV content explosion became a drive management nightmare and Avid also enter the fray with their first software only public offering with Avid Xpress DV. 

I used to get cold sweats whenever I saw these

Drive sizes and performance jumped again and quickly drives were 500GB and 1TB drives were looming large. Storage wasn't the issue it was managing it and stopping needless duplication onto endless pocket drives. More isn't always better...

"Have you got the drive with the footage of the clowns on it? I thought I had it, but I might have taken it home on one of the FireWire drives. It has to on one of these drives somewhere!"

Workstations were commonly becoming replaced with laptops for 'on the go' editing, complimented with external pocket drives brimming with DV content. Avid Unity was now complimented by its new big brother the Avid ISIS 7000 (2007) and started to be measured in hundreds of Terra-bytes as broadcasters ingested news content for fast turn around editing with Avid NewsCutter and the likes.

BBC NEwsroom - MediaCity UK

High Definition editing also starts to chip in, demanding higher sustained data-rates from Video Arrays and greater and greater capacity. News rooms such as the BBC were also quick to take up HD for ENG gathering and distribution further pushing the need for Peta Bytes (PB) of storage.

NOW
With regards to local disks we aven't moved on all that much other then in capacity. I now have an old HP Workstation (XW8600) under my bench which has a couple of of 7200RPM 1TB eSATA drives inside it, and with this I can happily produce HD and SD content. Even 4K content. However, even my poultry needs are getting close to the limit of the 2TB of storage I have.

I've swapped out my computers boot disk for a Solid State Drive (SSD) for a quantum leap in performance and speed (eSATA 3.2), allowing my workstation to boot in a minute rather than 10. But the expense and current size of the SSD's is still holding me back from using these as media drives for my edit suite.

The bigger change however, has come in the form of 'remote' editing and using cloud based services for storing and editing your project. But this still has issues especially when your internet connection speed is variable to say the least.

The Future
This is where it's becoming a little fuzzy. Will we ever get to the stage where we'll hold ALL our RAW rushes in the cloud and edit in proxy form until we're ready to produce our final output file?

Well, that's possible already, with the likes of Avid MediaCentral as well as other platforms for larger facilities and broadcasters. For smaller productions however the security of having your own rushes, no matter what your internet connection speed, is a necessity and I wouldn't expect it to change any time soon.

The only thing we can say is, we'll need more storage to sure. 100TB SSD will fast become a necessity. But to accommodate this sort of capacity, will require a more accurate and intelligent file systems to sort vast quantities. So we're now in the hands of the OS developers as well as the hard disk manufacturers.

That said, the one thing you can say about this industry, you never know what around the corner!

Lets see what NAB 2017 brings us.



.