Our IT and Production/Media teams are working on getting friendlier with each other (aka working together on solutions rather that just being silos). One thing I’m working on with them is the network on which all of their stuff talks with each other. They use a few “self-contained” solutions (Dante, Art-Net, and a 10G switch for video-editing), but some of their equipment (e.g. live streaming stuff, devices that need to connect with our normal network and those self-contained networks, etc.) connect with our “normal” network via switches that are also hosting regular network devices such as APs, wall ports, etc. While nothing is explicitly/obviously going wrong as a result of these devices being on the same switch, they occasionally experience connection glitches that they can find no explanation for outside of the potential of there being a bandwidth bottleneck somewhere (or some other unknown issue).
That said, I am wanting to make sure that our equipment is set up in a best practice format, so that I can say without a doubt (inasmuch as that is possible in our field!) that our equipment wouldn’t be bottlenecking their stuff. Currently they have a Cisco SG500-52P Gigabit switch in their booth which is the device handling anything of theirs that’s going to be on our normal network (along with some APs, wall ports, etc.). They also have a Cisco SX350X-08 8-Port 10G switch used internally for video-editing stuff (not normal network connected). The SG500 is the device in question for whether or not something is getting bottle-necked somewhere. They and I had a few thoughts on how to improve performance/separation of media stuff, if it’s even necessary:
First of all, is it possible I can put this concern to rest if the SG-500 is a non-blocking switch (so every port gets the full 1G bandwidth)? Are most modern, especially Cisco, managed switches going to be non-blocking?
If that’s not a good answer, here were some of potential solutions. Are any of them “best” practice for a situation like this?
Get a second switch for their booth and have “main” network devices (APs, wall ports, etc.) be on one and media stuff the another, while still allowing internet traffic to both.
VLAN out Production stuff and prioritize its traffic.
Get a 10G switch with more ports and put everything in the Production room on it. (So essentially upgrade the 1G switch.)
Double NIC devices so that they can talk to each other both on their own local switch and the main network switch.
Or is there potentially something else I’m missing? Thanks in advance!
Our setup is option 5. Don’t know if it is best practice, but since we didn’t install or desgin their network, I try to stay hands off. It lets them keep using a big flat network that I do want to fix, but I don’t have to own if I leave it alone. The more the setup allows them to fix things on their own if they want, the better. I can help if they get stuck, but not being the only number on the list is nice.
That being said, I am feeling the same pain trying to keep up as more of the AV equipment is running over IP networks and they want to use the existing WiFi/network infrastructure. I plan to connect their workstations back to the core over 10Gb copper/fiber and share that switch, but everything else I try to keep on its own hardware. Reality isn’t quite that clean, but I am pushing towards a clear distinction between the two.
First thing to remember is that all audio and video content is very ‘latency-sensitive’ and will therefore be badly affected by network congestion if you do not take control of the situation. Also, be very wary of mixing any networking kit with other proprietary systems such as Dante, which are highly optimised for AV in a manner that your Cisco kit is not!
The best way to do this is to separate out all your AV kit onto separate Cat5e or better cabling infrastructure as much as you can. This will allow you to optimise the AV specific kit to get the best out of it. Likewise run all your non-AV stuff on its own separate cable infrastructure also.
Where the two must of necessity meet, e.g. streaming output, WiFi connectivity for your iPad or Android control surfaces, route these through your IP switches and routers on a dedicated VLAN. You will also need to turn on whatever packet QoS controls you have to prioritise AV traffic above everything else. (Note: if you are running VoIP anywhere, this should also be run on separate independent infrastructure and VLANS).
Don’t look to run completely separate switches as any modern IP switch these days has adequate QoS, VLAN and other controls to do what you need to do. If you add in additional switches, it does not necessarily help and will likely just complicate the number of devices to be managed.
Also, if you double-NIC the devices, you are likewise also asking for more complications more cost, more complexity to manage, etc.
Sit down and plan out layer, by layer through the OSI or TCP/IP stacks what actually needs to connect to what and why. That will help you adequately separate out where needed and identify where QoS controls, VLANs or separated kit is absolutely essential and where it is not.
We have started to combine our IT and AV network devices over the past 18 months. So far everything has worked but, it takes a little while to fix all the issues that pop-up. I am thankful that our AV team has been patient. We keep offering them features like “you can have access to this network anywhere on campus not just in this room” and “your iPad can connect to the internet and your lighting controller at the same time.” We are aiming for redundant switches with redundant paths for Sunday morning AV traffic. Adding ping monitors to let them know when AV equipment goes down midweek has also been helpful. Whenever we run into a problem, we solve it with logs and visibility. We use Cisco switches in the 4948 family and graph as much as we can using SNMP data through Cacti. We log to local and a syslog server so we can chase issues that aren’t happening right now. Graphs help us see how much traffic is actually moving instead of guessing. Now we know what normal looks like so atypical traffic stands out. Here is what we have learned:
Lots of VLANs. We have VLANs for lighting controllers, audio controllers, and video streaming. Some of the AV equipment is sensitive to multicast traffic so we create separate networks to make sure it won’t be affected.
Get the IGMP Querier close to the multicast traffic. This setup prevents the multicast traffic from going all over the building just to get back to where it started. Our on-stage switches source the audio traffic that is processed in the tech booth. We have redundant IGMP Queriers in the TB so that traffic doesn’t have to go to our core switches as well. Our normal Dante audio multicast traffic is about 110 Mbps.
10 Gbps uplinks. We realized about 6 months ago that we need 10 Gbps uplinks to avoid issues. We use low latency video streaming at one of our campuses. Each video stream is 450-500 Mbps. We can’t do 2 sources on a 1 Gbps link cleanly. 10 Gbps links fix this issue.
We haven’t had the need to configure QoS for the Dante traffic yet. I used traffic policing at a previous job and it always caused it own set of issues so, I will only add if we need it.