I would like to get an advice on how to store archive video and open to discuss what I have setup.
Our production team produce 160G data every week or may be even more if there is special event in that week. This team wants to have single folder that have all the archive and raw video files that they can pull as they need. I would say this is more of warm and cold storage type per Jesses’ note (thanks to Jesse).Data Storage / Policies for Video?
With limited budget, I found most cost effective way to do with Synology RS2416+. It cost me $3800s to start and add hard drive as the team need. Currently, this unit has 12 x 8TB with raid 6 and 70TB for one single volume. It has been working good. No issue other than two hard drives went bad within two years and I had to replace it. What I like this is the management of the device such as update, notification of hardware failure. Now it is full, there is 4TB left and no additional bay available.
Rs2416+ can expand with RX1217RP to add another 12 hard drives. That is another 70TB if I add 12 x 8TB hard drives. I trying to figure out with Synology Tech Support about adding 12TB hard drive instead of 8TB if that is allow.
I have few concern to ask.
Can 140TB single volume can be safe?
What could be the issue in long run with this setup?
Should I store this files in different media? Any different idea?
I wouldn’t consider total storage to be the main metric when deciding how “safe” a volume is. It comes down to number of spindles, the quality of the drives and the amount of recoverability built-in.
In your example, a 12 drive array of 8TB drives probably takes about 26 hours to rebuild after a failure so you’re reasonably safe as long as you’re quick about replacing failed drives. At 24-drives you’re pushing your luck a bit, so I’d recommend doing 2 RAID 6 arrays instead of trying to keep it a single volume. Also, expanding an array is pretty risky since it requires forcing a rebuild for each drive you add (and doing that 12x is a lot of extra load while your array is degraded).
Also, remember, RAID is not backup. If you Synology dies or your RAID filesystem/raidset gets corrupt, you’ll need to reference your backups.
Synology’s software isn’t really designed for extremely large arrays. Performance will slow down and some functions will become extremely slow. This is in-common with all “traditional” NAS appliances, and will generally end up requiring either a Quantum/EMC/NetApp type product, scaling out (adding more NAS units) or building something based on a more scalable filesystem like ZFS or ReFS.
Tape is a lot more reliable for archive storage. It’s also a pretty cost-effective backup option, and there exist plenty of great systems for managing the storage. At the end of the day, it ultimately comes down to “how much is it worth to you.” Yes, there are better options, but they aren’t cheap.
That was never a decision that fell to me, but I provided my video team with enough high-speed storage they could edit from for 4 weeks (to deal with vacations and special events) and enough slower storage for reference material for a year. While it’s possible to combine this into a single volume using storage tiering or other technology, all it takes is someone reviewing old footage to blow out this week’s material from the cache and everything slows to a crawl again.
I know about limited budgets. However for the cost of the new synology equipment you could get a basic LTO tape changer and something like Archiware P5 archiving module which has been built with video in mind. The archiware web interface allows any authorized user to send or retrieve files from archive, With the capacity and speed of current tape technology, the only time you’d need to wait fro files if is the tape is not in the drive.
If you are looking for a storage system that is easy to increase size, check into Drobo. You can just add hard drives as needed. Some of them even have Flash built in to help speed up frequently accessed files.
For anyone who happens to stumble across this topic.
You do not need to buy new hardware. You can do very well from refurb servers off ebay and storage.
Question 4. Write down how much it would cost to keep storing and guarding this data per year, over 5 years. Then let someone in leadership decide if they want to keep every precious moment of something OR if they want to review what is or is not needed and discard.
Synology NAS with CAT6A (10GbE) is good enough for video editing needs and if you need to cold storage some video files, tape storage will bring a lot more value; you can look at using Backup Exec.