We've recently experienced a minor hardware issue with our data storage (spoiler: everything is fine and intact), and wanted to share our experience in redundant back ups (or as our manager would put it, 'backing up the backup').
Throughout the years of filming and producing surgical videos, we have accumulated a sizeable cache of data, be it in the form of raw unedited video, or uncompressed 3D renders. A rough estimate of the total disk space we need to store the files in a backup drive comes to about ~27TB (27,000 GB), way beyond the capacity of typical hard drives.
Our solution was to store these files in RAID drives, or Redundant Arrays of Independent Drives. Essentially, this mode of storage utilizes multiple hard drives to distribute data, in configurations that allow for redundancy. In other words, if one of these drives fail, the data can be rebuilt based on information stored in the other (healthy) drives; hence effectively providing a backup for the backup.
A real life example: A few weeks ago we noticed one of our RAID drives had very slow data transfer speeds; upon an on-site inspection we confirmed one of the drives became faulty (with the red LED indicator as shown below). Thankfully, because of the RAID configuration, the data was still intact but now “vulnerable”. This means if another drive fails, we will lose the data completely.
A quick call to a local computer supply store allowed us to replace the faulty drive with an identical model. The remaining drives essentially rebuild the redundancy from scratch, which will protect us from a future drive failure.
The RAID configuration does, however, come with a cost of storage capacity. Depending on the configuration, a RAID drive may only have 50~75% available for actual storage of the files - the remainder of space is utilized for redundancy.
We base our RAID drive on a balance of cost and data safety, and elected to use a RAID 5 configuration, where 75% of capacity is used, and 1 drive failure can be tolerated - a fair cost to pay for peace of mind.
As always, stay safe (both physically and data-wise), and stay tuned for more updates.
-The TVASurg team