We have a new Synology NAS (roughly 120TB usable) that I hope will replace about 3 separate media servers.
We have the following media-rich teams:
Video (unsurprisingly huge storage needs)
Web (surprisingly small storage needs)
Art/Design (medium needs, often sharing content with everyone else)
Radio (lots of audio, around 1TB so far)
I’ve gone back and forth on how to carve it up, but I know ultimately, I want to have the following conceptual shares:
Shared Folder Concepts:
Active Working storage (current projects for up-coming events/delivery dates)
Archive Storage (already produced and delivered, completed projects, preferably keeping items for 15 months before putting in “cold storage”)
Library storage (common media re-used throughout the years, purchased transitions, audio packs, logos, etc.)
One non-shared volume:
Cold Storage (likely one copy on Tape, and another copy in Azure Archival storage, only for critical files we never want to lose but don’t need on a regular basis, like teaching library masters)
The big concept I’m battling with now, is should I create 3 file shares and give all media teams access to all of them? Should I create a file share for each media team and then make folders within each of those for these 3 purposes?
Some things to consider:
During active work is when media teams need to share the most.
If I don’t set limits, media will just get stored in whatever volume has the most space.
If I don’t make Archive or Library Storage read-only, it will be used as working folders.
Here’s what I think my next steps are based on what I’ve written above:
Create a single 20TB “Work” share for current projects, and create sub-folders for each team allowing them to manage their working files in their preferred manner. All media staff would have read/write access to all folders in the Work share (I’m not messing with non-inherited permissions)
Create a single 80TB “Archive” share for archives that we want access to reference throughout the year (this is that rolling 15 months), making it read-only for everyone and only allowing one “archiver” per team (to discourage it becoming a working folder)
Create a single 20TB “Library” share for library files that is read-only for everyone and allowing only one “librarian” per team.
Work with each team to make sure they understand what goes where and establish a schedule for archiving.
General Concepts:
Share sizes are not going to be carved into dedicated volumes (we just have one massive RAID 6 volume), they will just have a quota.
Over the years we will add additional shelves of storage, possibly adding faster storage for editing directly from the storage array.
Working folders will have nightly incremental backups with a rolling 14 days of retention to the older server this is replacing.
Archival folders will have incremental backups taken nightly to tape with unlimited retention, and we’ll just cycle tapes every 6 months.
Library folders will have incremental backups taken nightly to tape with unlimited retention, and we’ll cycle those tapes every 6 months as well, but I will also have them backed up nightly with only 7 days of retention somewhere local.
Questions:
Any thoughts on this plan? I know every team is different, but I also know there are some common pitfalls the first time you do something like this that some of you may have run in to.
You’ve definitely thought this through. We also have a Synology with much smaller needs than you have and I believe you are thinking well about this. We have done something similar with having a share that is an “active” share where active projects are stored. We then also back this up via R-Sync to another Synology NAS that is off-site nightly. We then have an “inactive” share where they put projects once they are done with them which we back up less frequently and do some to our smaller off-site storage and some to what you call “cold storage”.
The biggest issue we have is that these teams don’t do well identifying which projects are active and which they don’t need any more. They would love a storage space that is infinitely large and easy to access as their desktop. Unfortunately for them, that is just not feasible for us.
The only other thought is that when multiple teams are using the same share it is harder for them to feel ownership over the space, but that is more of a cultural concern than a technical one.
Thanks for sharing your thoughts with the community!
One aspect I included in my presentation to the media teams is that the archivist for each team will need to work with their budget maintainer to decide how much data they want to keep long-term.
I may also include quotes for additional shelves/redundancy so the media teams using the storage have more of a say in the purchasing and upgrades of the Synology.
I unfortunately only have a single large Synology so I can’t replicate, but once I free up our old storage I’ll at least replicate to that until it dies. I think nightly off-site R-Sync sounds great!
Use BTRFS with RAID 6 but I would split the volume in drives of 6 HDDs. Cos rebuild time will be extremely long if you use all 12 HDDs or more in a volume. And try not to span your volume across expansion units.
In case the chassis fails, easier to move the sets of 6 to other Synology models. I tend to buy 6 first then wait for the bigger HDD capacities in the future and prices to drop to buy subsequent HDDs when the first volume is filled up to 50%.
Consider the Sony Optical Drive Archival solution. We are moving away from LTO.
I read that Synology DSM 7.0 comes with faster RAID 6 rebuild times but enterprise models yet to receive the update.
I would max out the RAM at the beginning in anticipation of expansion units in the future.
Use 10 GbE LAN ports and switches for full HD editing. Consider Link Aggregation. Netgear switches work well. CAT 6 cables can handle 10 GbE for up to 55m.
For Synology cloud backup, do consider Backblaze.com. One of the cheaper but reliable options.
I’m a little curious about the Sony Optical over LTO. Is there something that sold you on Optical over tape? I think the cost per TB to archive isn’t bad, considering it is a one-time fee per cartridge, around $35/TB, but since LTO is closer to $7/TB uncompressed I’m still tempted to get an LTO8 desktop drive and just store the tapes at our co-lo facility where the temperature remains constant.
We have 64GB of RAM so far for our 120TB usable array, but I’m not sure what would be considered ideal. At this time we aren’t editing live off of it, but we do have dual 10GbE uplinks to our core. If we add another shelf, it will likely be a dedicated volume for our teaching library and possibly a smaller but faster SSD array for editing live off the unit so we can finally get rid of desktop Thunderbolt drives that keep failing.
Here were the considerations of moving from LTO to Sony Optical
LTO generations can only last for 2, e.g. LTO8 can be read by LTO10 but not LTO11 so every 5-7 years we need to migrate all the data from the one generation to the next.
Every 5-7 years we need to invest in a new LTO solution which can be expensive; especially if we are looking at a LTO library
LTO risk of data loss due to dirty drive heads or mold is low but a concern
Retrieval of data can be slow
Sony ODA points:
Cartridges are designed to last 100 years, tsunami and earthquake tested and can store at warmer temperatures
Cartridges can be read by all future generations of ODA drives
The files stored can be accessed by both Mac and PC using USB. Open UDF format.
Current Gen3 Read speed is 3 Gbps, Write is 1.5 Gbps
Next Gen4 cartridge will be raised from 5.5TB to 11TB
50% of Japan TV stations moved from LTO to Sony ODA
Biggest China TV stations using Sony ODA
Fox and HBO are customers as well
Your Synology Enterprise NAS I assume allows 2 expansion units? If so, I think 64GB Ram is the max and would be more than enough. My video editing team connects to our Synology using 10GbE LAN via a 10GbE Netgear switch and can live edit with Adobe Premiere for HD files.
Hey, @OneSeventeen I am interested in what you finally came up with for this project. I am currently in the same boat and love what you have going on here.
I wound up just making shared volumes for each team, plus a “Library” file share for all of them.
I let each team manage their own files, but I did encourage them strongly to use a “Current”, “Recent” and “Archive” folder so I could use different backup jobs on each folder.
I am very ashamed to admit we still haven’t finished migrating things over and setting up backup jobs yet… “We’re experiencing a high call volume at this time…”
