We’re looking to consolidate and organize our video. At any given time we have several terabytes of video. We record our Sunday services, but also have other videos that we create. I’m looking for hardware options to consolidate our video. Any suggestions? We don’t currently a file server.
Here is a recommendation (Cheapest to Greatest):
Cold Storage Options:
Tape LTO Library that can store up to 24TB each tape for LTO-9 (expected to be out later this year; LTO-8 having patent issues I heard)
Cold Storage Cloud - Microsoft Archive Blob Storage or Amazon Glacier (extracting costs are high)
Video Edit off NAS:
Synology Rackmount NAS - https://www.synology.com/en-global/products?chassis=Rackmount&lan=10gbe_built_in (video editing directly off Synology NAS)
Storinator - https://www.45drives.com/products/storage/ (if you need lots of space)
Dell EMC - Isilon https://www.dellemc.com/en-sg/storage/isilon/index.htm#collapse (what the Media Industry use)
Right or wrong, we purchased external USB drives fore each of our two videographers. At around $100 for one or two TB, it was affordable. They keep jobs in motion on their workstations and jobs they’ve put to bed on the external drives. Non-profits get 1 TB for free and then buying extra space is pretty affordable. I plan to use our TB of SharePoint storage for our image library (340k files in 4,4457 folders) though so we’re sticking with the USB drives for now.
External USB drives work great and it is definitely a great idea to use the free 1TB Sharepoint or OneDrive storage for your image library.
The pivoting points will be when the team requires a storage option that is faster for edits, allows better team collaboration, better reliability of storage and greater need for large storage capacities.
I would recommend a combination of a NAS (e.g. Synology) as your main mass storage device on LAN. Make sure you use at least 1Gbit/s Ethernet connections to the workstations and don’t rely on WiFi. Also if you use Macs, use an external 1TB drive as local working backup for your editing suite for work in progress. If you are using PCs to edit, then a good USB3 drive will help.
Once you have the edited material, transfer to NAS for managing your library. Synology’s Video Stationeries is good for this.
When looking at material for online use, render your video master in the library down to 720p as well as lower resolutions for web use as well as audio-only for podcasts. Bear in mind that not everyone has unlimited bandwidth, especially on mobile devices, so provide options for them.
Host the public video material on somewhere like YouTube or other public hosting service and line to your main web site for access. You can usually embed the link to the video within your web site HTML without incurring the bandwidth hit if you do it right.
It would be best to install CAT6A cables, patch panels and keystone connectors when you are planning for a new venue or upgrade. This will ensure you have the ability to go up to 10GbE copper when you require in the near future. When video editing off a NAS, using 10GbE will be best. The new iMacs come with 10GbE ports already and many newer Synology NAS models have 10GbE options.
CAT6 = 55m for 10GbE
CAT6A = 100m for 10GbE (make sure the contractor knows how to properly ground the connectors and wires)
You should store your files on multiple redundant servers in geographically different locations. One backup drive next to your computer is not worth much if you have a flood or fire.
Multiple backups will keep your image files safe as long as you are able to take care of them. If you want them to last for future generations you need to endow your IT department in perpetuity—at least that is what the video archive experts recommended.
I’m actually revisiting video storage options as well right now, and backup/archive is my biggest issue. We save just about everything (well… compressed versions of everything), so we have over 200TB of data and only about 10 TB of backup storage.
And yes, we routinely use 5 year old content as we produce a few TV shows from our sermons, and we offer our entire 30 year library online (the first 10 years or so being audio only).
I’m seriously considering a Tape Drive (not an autoloader) for archiving, since we’d want to spin off the data to a Tape then put it in a climate controlled room somewhere off-site, then possibly use SATA drives on-site for the primary storage (internal drives stored in anti-static cases, and then use a drive adapter when needed).
This really only solves our legacy data issue and doesn’t help with our current rolling 2 years of media that we’d want on-hand, which I’m thinking a large NAS should work for.
Has anyone else used a desktop LTO 8 drive for purposes like this? (it looks like LTO 9 may take a minute to come out, although some companies like Quantum have a “buy now and upgrade later for free” program)
I found this recently. Sony Optical Archiving. Designed for 100 years of storage on optical with 5.5TB of capacity. Exploring this to see if it is a better solution than LTO.
LTOs unfortunately have a shelf life of about 2 LTO generations (about 7 years I think) and you would need to move out all the content from the older tapes to the newer tapes every 2 future generations (in my case LTO5 tapes can’t be read on LTO8 drives). Each LTO tape can last up to 30 years if stored correctly provided you still have a LTO drive that can read them
I’ve been working in the high-capacity storage sector for the last 10 years or so. Tape is really only used these days for very long-term archive storage, e.g. back-up and forget until a major disaster happens. Optical is tending to creep in as an alternative and is gradually replacing tape as a more reliable solution. However, most data centres are now using standard hard drives as backup for <5 year recovery. This is because Flash drives have taken over as the main disk technology for processing and hard drives have massive capacity and density, with technology to put them to sleep/off (to save power) for years without issue, but with instant wake and access capability that neither tape nor optical tech can match.
Given the capacity you are looking for, I would suggest you consider a top-end Synology Rack Station. It will have more capacity than you need, with expansion modules for more drive space, automated backup/mirroring capability, etc. They are also extremely easy to install and manage and have designed their platform to sit very well within the video storage market.
Synology Enterprise models (those with 5 years warranty) are pretty reliable. Been using various models for about 8 years now.
My personal view is to use the normal RAID (prefer RAID 6) instead of Synology Hybrid RAID and ensure your volume resides within the same chassis for performance reasons. Also to max your RAM at the start if you expect to get additional expansion chassis units. Also prefer HGST Helium drives and add the 10 GbE option for future proofing.
I think my reason to lean towards tape is I can get about 3x the capacity in tape + autoloader than I can for a NAS of the same price.
We currently have well over 200 TB of media, so a good Synology with 10GbE would be nearly $20k for a good reliable solution (still more expensive than 360 TB of uncompressed LTO 8 with a 24-tape autoloader).
LTO 9 is supposed to handle 24 TB per tape (uncompressed) which is insane.
My thought is to archive everything older than 3 years to LTO 8, then make another set of archives to SATA HDD, and store the HDD’s on-site and tapes off-site in climate controlled storage. That way, a single Synology NAS should be able to handle a rolling 2 or 3 years of media for our staff, freeing up about 90% of our current storage, which would then make backup storage needs even lower.
If executive wants an ideal scenario, we could even upload quite a bit of it to cold storage in Azure, but even having it in 2 places would make me feel better than where we are now. I’ve also had 3 storage arrays fail in the past 5 years, and one of them we had to have recovered which cost more than our proposed tape autoloader with 360 TB of tape.
I would still prefer having Synology enterprise arrays, but I’d want to at least double it for backup needs, which makes tape sound good to me even as we look at getting a Synology array… I just hate having an immediate need months before LTO 9 is released. Maybe I should just rent a drive…
Also, that Sony optical media looks amazing, it is nearly twice the price of LTO 8, but I do trust archival optical more than I trust magnetic tape…but LTO has been around for so long that the odds of finding an LTO drive are likely higher than finding a proprietary optical archival drive (then again, who knows what will be more common 10 years from now).
Sorry for rambling, I’m just trying to get all my thoughts out there before deciding on a direction, because I really need to not only get a backup of all of our files soon, I also need to offload a lot of data from our storage arrays so we can downsize to a more reasonable amount of network available storage.
Sat through a presentation by Sony Optical Disc Archival. Here are some main points:
- Currently 11 discs inside a cartridge with total of 5.5TB, next Gen4 will be 11TB
- 50% of Japan TV Stations moved from LTO to Sony Optical Disc Archival (ODA), including top China and Taiwan TV Stations
- HBO and FOX using
- Tested against water damage, from tsunami, earthquake damage and even boiling in hot water.
- Store at room temperature and can handle extreme temp ranges short term; for long term storage 10-30C
- Disc designed for 100 years
- Write once Read Many
- USB-C Drive 3 Gbps (Read) 1.5 Gbps (Write)
- Fibre drive same as USB-C but longer wire lengths
- 7U-42U rackmount for their PetaSite Scalable Library System
- 165TB (30 cartridges) to 2.9PB (535 cartridges)