AV/Production Department & IT Best Practices

Technology General
1

Hi everyone, we have a multi-campus church with its own AV-Production department which handles music & services, including sound & lighting during the week for classrooms, etc. There is an ongoing discussion in which this department wants to have full control over any computer in use in at least the sound booths & video/audio production rooms, laptops, etc., with a specific desire to remove antivirus and to have full local admin privileges so that making a sudden, on-the-fly change at any time is possible. It’s been stated that the direction they feel they should go is to get all of the computer management for these machines completely out of IT.

From an AV-Production & musician’s perspective, these aren’t computers, but dedicated pieces of hardware dedicated to their one purpose and nothing else. On a more specific note, as a whole, these computers are primarily geared toward Waves Central, Dante Virtual Soundcard, Dante Controller, ProPresenter, Blackmagic, QLab, Ableton Live, and so on, each with their own combination of these functions. There is also some Yamaha-specific chatter to soundboards via Yamaha CL Editor, StageMix, and MonitorMix. A small handful of people have VPN access, which is primarily used for connecting to ProPresenter machines.

From an IT perspective, I’d say that there is a bit of a misconception whenever anybody says that something is isolated and cannot impact other devices around it; these “isolated” computers are on the network and can be controlled remotely, so therefore they are note truly isolated. There is also a misconception around just how dedicated a computer is; while it’s true any given computer might be put in place with the intention of serving a particular purpose, each machine does have additional capabilities. It’s much less like a musical instrument (an analogy which has often been used) and more like the computer it actually is.

We do have a desire to support everyone in these areas as fully as we possibly can, within reason, but we are in disagreement as to what “best practices” are in this case. When someone tells me that a “best practice” includes removing antivirus, that directly conflicts with IT best practices. How do others of you handle these sorts of requests in your respective environments? Are there anything specific practices you already have in place regarding computers and/or network infrastructure arrangements?

2

Questions for you

  • If you have an AV department, is there also a formal IT department?
  • Who reports to whom and does AV dept have any formal authority over IT?
  • Who sets IT-related policies?

My first thought is security. Removing antivirus and VPN access w/o MFA (not sure that’s your status) seem like risks that are unacceptable nowadays. Giving AV department local admin seems like a risk that’s mitigatable (especially if you can implement MFA for local admin).

This does sound like an opportunity for education. Removing antivirus is a terrible idea if the computer is connected to any network at all. But enforcement of these IT-best practices sounds like it needs to come from a policy update if a basic conversation between departments doesn’t yield anything.

3

I would also consider putting them on an Internet only subnet that has no access to any internal subnets. When a computer is compromised, it won’t impact any other subnets.

You’ll need to do engineering around DHCP and DNS if this is provided by internal servers.

4

I am probably a little simplistic compared to you all, but I am the only AV production manager and also the only IT person on staff at our medium size church. I am just wondering whether the issue the AV folks are having with Antivirus is the slow down that updates to those programs often intrude on the machine resources with and how that can add slowed down responses to functionality when during services quick response is so crucial. In which case it seems to me a compromise could be to turn off auto updates and update virus definitions and run scans manually outside of production timeslots.

5

Your idea seems to be on the right track. If you can disable software updates, signature/definition updates, and scans during critical time frames (I’m assuming Sunday mornings, but whenever you need them to not run) that would be a great first step.

Which antivirus software are you running?

6

We only use Windows Defender and everything automatic is disabled and only
runs when I initiate it. I like being in charge lol!

7

I managed to reply to you, Carrie, as if you were Al. My bad. Sorry everyone.

Carrie - being in control is nice. Automation is also nice. :slight_smile:

Al - Carrie’s idea is good, but still doesn’t solve the issue of AV wanting all IT removed from the computers. A dedicated VLAN is a good option, but direct-to-internet with no antivirus sounds risky. If these computers are as important as the AV department says, then it is wise to protect them properly.

8

You’re on the right track…

This conversation comes up all the time about computers in the booth, but we don’t often talk about the other devices there like Audio Consoles, Lighting Consoles, Video Switchers, etc… These are often just Linux computers, or maybe even Windows Embedded! There is nothing wrong with separating your mindset about computers in the AVL production world from traditional computers, however that is ONLY reasonable if you’ve also put in place the controls to make this viable. Doing this well is going to involve a lot of thing that, quite frankly, most churches are unprepared and unwilling to do.

  • There must be a clear written policy defining controls and requirements for security in the org
  • There must be a clearly defined exception to the above policy along with how you will mitigate the risk associated with that exception

Yeah, that’s already pushing it for most churches since the AVL team just wants to get their job done without IT stepping on them!

Ultimately, a well designed network with multiple VLANs and properly configured firewall rules is the answer. If you live in the world of one single flat network for AVL that can talk to everything, you really shouldn’t be making exceptions for any gear.

Chris Green

For the record, all my AVL systems have my security stack on them and get updates. :wink:

1 Like
9

Andy’s questions at the top are excellent starting points, and their answers would be critical to making any meaningful contribution.

I also agree, strongly, with the thrust of Chris’s contribution directly above me. Namely that it is a fine thing to do, but there must be controls and policies.

Namely - and this is what most church AVL departments hate to hear - if we are going to remove all management and security from a desktop, that desktop must and can not have internet access. period.

Your dedicated devices that can’t be automatically patched or are otherwise out of support - be they audio surfaces, lighting systems, video wall controllers, vision mixers, etc - your dedicated devices should already be on isolated vlans with their only way to outside systems via jump boxes and drop points.

I have no problem, at all, with treating AVL machines like dedicated equipment - free from IT management and security packages. But to do so means treating them like the dedicated equipment they have elected to become - and deeply bury them behind non-routed networks.

You can use dual-homed (and auto-updating…) NAS boxes to provide file access to the outside world, or bridging to dropbox/onedrive/etc.

TL;DR: The request isn’t bad, but the controls must be in place to fulfill the request.

-Karl P

10

Thanks everyone. This is the sort of conversation that tends to pop up about once every other year, and it’s good to have recent input from others for reference. Some of the particular subtopics have been points of contention in prior meetings, but are worth revisiting from a purely objective perspective. Having that area of the network segmented off is one such topic. Prior crews have had some mutually exclusive requests in this regard - i.e. “keep everything completely separate” was right there with “full access to everything.” While we’re readdressing similar topics with a largely new crew that has fresh eyes, it’s a good idea to cover this ground again. In the meantime, they have inherited the existing environment.

As to the specifics, the computers in questions are nearly all Macs. We are running Sentinel One for antivirus. We were running Trend Micro in the past. We do have them on a separate set of VLANs, but there is also a very large subnetwork that got plugged in almost as if we were their ISP. This was initially done by a contracted company who did not collaborate with IT ahead of time, and actually had several sets of VLANs which overlap with our own. We had to react to this project in real time, and have had to hammer things into place over the course of time, occasionally punching specific holes against our better judgment in order to make amends for the way it was put in, in many cases finding the lesser of two evils so that they can function within the larger environment while still staying mostly separate. It’s messy, but my hope is that we can work together toward a cleaner environment going forward.

11

As someone who has been in AVL for over 30 years and in IT for over 25 years, I fall in the separation camp. The few times that my AVL networks have been subordinate to IT (when I wasnt over both), I had AV failures caused by the IT department Every. Single. Time.

A major difference between the AVL world and the IT world is the sense of urgency. When things go down in the IT world, it is generally accepted that things will fail and will take time to restore. The Mean Time to Restore is measured in hours and sometimes days. Most times, a failure doesn’t stop the entire business. At most, you are affecting a few hundred people if the entire network goes down.

In the AVL world, if there is an equipment failure, the Mean Time to Restore is typically measured in seconds, not even minutes. It is quite common for an AVL failure to impact thousands of “customers” (congregants, audience members, etc.) on top of the people who are on-stage. There is no going home and starting again tomorrow.

In the church world, this is compounded by the fact that IT is typically a 9am - 5pm, Monday through Friday job. The “prime time” for AVL however, is typically outaide of those hours.

Another consideration is that AVL equipment is such a niche project in the IT world that little to no testing is performed prior to updates. As an IT manager, imagine you have a Monday Morning All-Staff and Board video conference scheduled for 8:30am. You walk into the office at 8am (to give time to get your coffee and boot your computer) to find that an un-tested firmware update waa automatically applied to your edge router over the weekend without your knowledge. This firmware update happened to overwrite your config file. And your backup is unusable because the manufacturer decided to restructure the database. In addition, this update took out all 27 of your VPNs, meaning that all remote campuses were down, too. Now, in a panic, you try to log-in to see what happened, but you cant get in because your MSP and the manufacturer says they will return your call tomorrow when they return to the office. Now, multiply this by every device you use on a daily basis and you can see why AVL departments have some trepidation when dealing with IT.

Now, I did say that I’m also an IT Director (of a network with 27 sites and a couple thousand endpoints) in a HIPAA compliant network. So I do understand the need for security and the like. In my experience, a separated AVL and IT network is the best solution. In a perfect world, the AVL network would sit behind its own router with its own ISP and not touch the corporate network. Unfortunately realtity is not perfect. To reflect the world of compromose that we live in, I prefer to havea close relationship between the IT and AVL departments. I set up separate VLANs for each area of AVL (Audio 2 VLANs - 1 for carrying things like Dante and 1 for control systems, Lighting gets a VLAN, Video gets 2 also -1 for control and file transfer and 1 for NDI). I then have a computer that has multiple NICs that is on all VLANs plus one back to the corporate network so I can manage all of them from a central location. This computer is kept secure with an Anti-Virus/Malware package that the AVL department can pause when needed for updates. I do the same thing with Anti-virus and separate NICs for any computer thats need production and internet access. I do turn off all auto-updating, but then handle them manually once the updates have been tested with the necessary AVL software. This keeps the IT department from having to test the multitide of packages that are only run in the AVL department.

TL:DR Separation of AVL and IT networks is a must, but it requires close collaboration between the two departments.