Building a robust solution

Our stake suffered a very significant interruption of service during building-to building broadcast of stake conference from the Stake Center to one outlying building and some shut-ins this last week. This was our third or fourth Stake Conference broadcast and we have also been the center of some multi-stake broadcasts for area training meetings. One would have thought that we had this mastered. We suffered the obvious failure in interruption of service but we still don't know why. The error was compounded by communications issues between operators at both buildings. Two key contributers were remote due to illness and travel requirements. There may have been problems with testing etc. There was certainly a lack of robustness in the solution. Even before the significant interruption, the sound quality was borderline unacceptable. Clearly we have a ways to go before its a flip on the switch and forget it solution. We need a solution that is so capable that only lightly trained operators can handle it on the day of service as though they were just turning it on.

It is time to go back to the drawing board and I am wondering if there are any templates for planning building to building communications that can help us at a "meta" level to properly assess our issues, plan robust solutions, effectively test the solutions under a variety of realistic conditions and set expectations for all involved parties. We have people who are perfectly capable of "wiring" the solution and have done a stellar job within the constraints that they have faced. The template document I am looking for isn't necessarily for their use but rather for the Stake Presidency and PEC's use. i.e. a step up document for the guidance of the PEC.

Thanks for any guidance.

Just curious if you were using the Webcast Communicator box or a PC with the Meetinghouse Webcast software?

The first question is what was the source of the interruption? The system is only as good or reliable as your Internet provider(s).

We don't know. Perhaps we need to find another provider. We can run off and find one solution to one problem and watch it fail on another problem next time. This works for home solutions but it is discouraging and "costly" when 400 people are denied a stake conference experience. We are looking to attack this more holistically and systematically.

One tip is that you need to turn off/disconnect all the wireless in the chapel. Otherwise the various tablets/smartphones, etc can suck up the bandwidth and even the available IP addresses.

I asked my question above, because I've discovered a design flaw in the Webcast Communicator box that causes it to reset after about 30-90 minutes of continuous use. The problem is, the interior of the box gets overly warm and besides a few vent slots, there is nothing to move the warm air out. A quick fix is to simply take the cover off. In testing, I ran it this way for two separate 8 hour stretches without a single hiccup.

I don't know if you are using the Webcast Communicator or not. I would just hate to see you redesign your whole system, change ISP's, etc., with what could be a quick fix.

Mikerowaved wrote:Just curious if you were using the Webcast Communicator box or a PC with the Meetinghouse Webcast software?

I don't know. I am not a technician. I am looking for a project planning and execution roadmap that will help us understand the whole problem and come up with a very robust solution. Yes this is the touchy-feely side of IT but we too frequently are constrained by circumstances and budgets to hack together a technical solution that "works" but isn't robust, redundant, scalable etc. We are looking to take a step back and see how we can empower our technical people with a roadmap to achieve a turn it on, it always works solution.

Mikerowaved: Thanks for this information. I will pass it on

corwinslack wrote:Mikerowaved: Thanks for this information. I will pass it on

I like all the suggestions, but I would make sure that you have adequate bandwidth at the stake center. This isn't the download speed, but the upload speed. I agree with the fact that you may want to look at something different than the Webcast Communicator as a test.

One thing that I did during our first stake conference, was have my laptop connected to the stream, so that I could see exactly what everyone else in the stake was seeing. This way, I knew if they were having an issue.

Just thinking, I would say take time to write down each problem and see if you can find a common theme. Then you can see what the common theme is.

I'm sorry I can't find the reference, but I recall seeing on this forum, the wiki, or the church facilities pages that there are future buildings being planned or piloted with webcasting/videoconferencing capabilities built in. It will be interesting to see how those setups compare to the retrofits and temporary system we cobble together now in terms of webcast resiliency.

I think what you're really wanting is a solution like chapel audio. In my years as a church member, I've only had 1 sunday where our chapel audio didn't work as designed. Most chapels don't even have a backup system because the regular system is so reliable. The system auto-tunes itself to the number of microphones present, the volume of the speaker, and other factors.

Unfortunately, Webcasting simply isn't mature enough (in church use) to have that kind of reliability. To do this REALLY well with the current technology, it can get very expensive. So the church has (smartly) opened up a very flexible solution so that Stakes can get as fancy or as simple as they feel comfortable with and can afford.

Some day, as technology matures, we may get to something that is embedded, reliable, and consistent across buildings. For now, I think we're still in tinker stage, finding out what is best and allowing the technology to continue to evolve and get less expensive to allow for greater adoption.

For now though, if you take a step back, you can do a couple things in your planning:

1. Simply the system. This includes people involved and technology involved. More points of failure = more possibility of failure. As much as everyone wants to replicate the Conference Center with its multiple camera positions, remote operators, and advanced audio - it makes your setup very complex, difficult to setup, difficult to run, and especially difficult to troubleshoot in case of a problem.

2. Identify the critical path. For most setups the critical path is the camera/microphone, the webcast computer/communicator, the building wiring to the ISP, the ISP itself, the receiving building's ISP, the receiving building's wiring, the receiving laptop, and the receiving projector and audio. In your planning, have completely independent backup solution for if the system becomes dysfunctional. (My suggestion here is that most stake centers have separate telephone systems from Internet, usually with a line near the satellite/audio cabinet from the days when that was the backup system for General Conference. Revive that system and find a free telephone conferencing service you can use if needed in a pinch.) If you have more time or resources, you can have a backup plan for each leg of the critical path and practice failover when there is an issue.

3. Gather tools and have them ready for troubleshooting the critical path.

4. Test, Test, Test.

There are probably more great ideas out there, but hopefully these help.