Stake Conference Broadcasts - Alternative to Church Servers

We're using Meetinghouse Webcast to broadcast stake conference from our stake center to 3 other chapels (our stake is rural and spread out).

We've put a lot of time, effort, and money into our equipment and internet setup, and it is quite reliable. We do lots of testing before-hand.

However, having a successful test during the week or on Saturday has nothing to do with how things will work on Sunday morning, when the Church servers are getting hammered with dozens of other stakes at the same time.

Yesterday we had our stake conference, and after having our test broadcasts go for 4 hours without a hiccup, we couldn't stay connected to the Church servers for more than 30 seconds. With 30 minutes to go we prayed about it and then had everyone come to the stake center, scrambling to setup every chair in the building and deal with parking, etc.

To complicate matters, for the first time I wasn't able to reach anyone at Church HQ, just getting a recorded message that ran me in circles.

We've had this problem in the past, but not to this extent. Usually dropping from high to medium quality kept us connected. This time even low quality dropped after 30 seconds.

Our internet is 8M / 1M at the chapels, and 16M / 2M at the stake center (turned up for the week of stake conference). Bandwidth speed tests showed all was working fine on our end.

What are others doing to deal with this?

Options include:

1. Setting up our own streaming server (would work fine for the other 3 buildings, but there would be no bandwidth left for shut-ins).

2. Working with our ISP to setup a streaming server for us and other stakes in the area, if they're willing.

3. Use a company that provides this service (there are many out there, and multiple companies could be worked with to eliminate the server as a single point of failure).

4. Perhaps the Church could setup regional servers (for instance we're in southern Utah, and there are probably enough stakes to justify a server being placed at an ISP down here, with a fall back option to servers at Church HQ).

5. Upgrades and additional reliability for the servers at Church HQ.

6. Give up on broadcasting altogether and cram people into every nook and cranny of the stake center each time.

I'm not sure what we'll end up doing, but we won't be rolling the dice anymore on the day of stake conference without some sort of fall back (and we've never gotten good enough audio quality using the phone backup to rely on that).

The disruptions and quality problems are significant enough to affect the attitude of the members about stake conference, so the status quo can't continue.

I really appreciate everything the Church has done to get us this far, and embracing technology is absolutely the way to go, so I applaud the effort and direction this is going.

Thanks,
Brian

Are you sure it's the church servers and not the ISP? You might have hit a limit with them. I wouldn't go to a lot of effort until you've definitively determined where the problem lies.

RussellHltn wrote:Are you sure it's the church servers and not the ISP? You might have hit a limit with them. I wouldn't go to a lot of effort until you've definitively determined where the problem lies.

I have to wonder if there wasn't some very large issue at CHQ this weekend. We have had reported now a pretty irate PVC user, and webcast failure, and we know this board was off and on all weekend. These three things may not correlate to the same time frame. Certainly the pre-recorded message that went round and round could have been the Global Service Center not having hours during the time frame of most Stake Conference Meetings (Ours would be 10am to 12pm Central, nobody would pick up the phone if we called having issues). But it could be that there were problems

Anyone have a good answer about it? Is Kurt Olsen out there, or is he recovering from a very busy weekend?

I would say as a follow-up. I've successfully completed webcasts for stake conference over the last 3 years at least 5 times. Successfully webcast a meeting involving the General YW, RS, and Primary Presidents, and at least 3 webcasts to 12-stake groups for events leading up to the KC Temple dedication. I've been overwhelmingly successful each time, and that is truly a miracle. Only 2 of my 9 buildings have better than a 1.5D/.5U speeds in our buildings. It is the lowest quality broadcast that can be delivered, but it certainly saves the saints time and money!

Edited for update, we also webcast the KC Temple groundbreaking, we used an aircard in a router with the webcaster (this was before the software version) running on generator power.

JohnShaw wrote:I have to wonder if there wasn't some very large issue at CHQ this weekend.

It will be interesting to see what the findings are. It's possible that the church might want to put the streaming servers on a different network connection so that any problems with servers/etc don't affect it.

RussellHltn wrote:It will be interesting to see what the findings are. It's possible that the church might want to put the streaming servers on a different network connection so that any problems with servers/etc don't affect it.

I also think it would be interesting... I know of at least one global-enterprise data center architect who lives and breathes high availability that would love to provide an eagle-eye observation to some of the designs.

JohnShaw wrote:I also think it would be interesting... I know of at least one global-enterprise data center architect who lives and breathes high availability that would love to provide an eagle-eye observation to some of the designs.

Is he a member?

RussellHltn wrote:Are you sure it's the church servers and not the ISP? You might have hit a limit with them. I wouldn't go to a lot of effort until you've definitively determined where the problem lies.

Unless someone from CHQ provides more details I have no way to be sure, but I can say from experience that performance is consistently worse on Sunday mornings than it is on Saturday evenings or testing during the week (this is my 7th stake conference broadcast, plus 5 or 6 other smaller broadcasts).

The only "evidence" I can go on is anecdotal, but all of my own experience points to overload at CHQ. This is the first time I haven't been able to talk to a human (other than the operator) at CHQ, and it happened to come at a time when other network resources at CHQ were also offline.

In any event, I haven't found a predictable, consistent solution and I'm still looking for one. It would cost about $100 per stake conference to use a 3rd party server, so I'll likely do some testing on that (particularly since they likely wouldn't have a Sunday morning rush).

BrianTAllen wrote:Unless someone from CHQ provides more details I have no way to be sure, but I can say from experience that performance is consistently worse on Sunday mornings than it is on Saturday evenings or testing during the week (this is my 7th stake conference broadcast, plus 5 or 6 other smaller broadcasts).

You may want to try testing on Saturday morning. Seems to me that from the perspective of ISP load, it would be more like Sunday morning then the other times mentioned.

RussellHltn wrote:You may want to try testing on Saturday morning.

We are always told to test on the previous Sunday at the same time that the conference is scheduled. We have more trouble on Sundays that at anytime on Saturday.

But even Sundays are inconsistent. I suspect it may depend on how much stuff Hq is supporting on a given Sunday.

RussellHltn wrote:You may want to try testing on Saturday morning. Seems to me that from the perspective of ISP load, it would be more like Sunday morning then the other times mentioned.

Yeah, we test a lot on Saturday morning, too. Worked great. I just mentioned Saturday night because we broadcast the adult session to 2 of the 3 other chapels and the broadcast went for almost 4 hours (an extra hour before and after) without a hiccup.

Same with Sunday morning. The priesthood leadership session broadcast (starting at 7 AM Sunday morning) went for just under 2 hours, then died. Once it died we never could reconnect to CHQ for longer than 30 seconds or so.

lajackson wrote:We are always told to test on the previous Sunday at the same time that the conference is scheduled. We have more trouble on Sundays that at anytime on Saturday.

But even Sundays are inconsistent. I suspect it may depend on how much stuff Hq is supporting on a given Sunday.

I've always been hesitant to test on Sundays because I don't want my test to interrupt / compete with someone else's live broadcast. If a bunch of people are testing on Sunday morning, you could easily add 50% - 100% more traffic to already strained servers.

I'm alive...

I don't necessarily want to jump to the conclusion that this issue is due to an overload on the Church webcast servers. The reality is, we just completed a major upgrade to our server infrastructure. New outbound reflector servers were installed, and we upgraded the redundancy/failover system. This was completed several weeks ago, and from what we've observed so far, it's had a very positive effect on system reliability.

We also implemented a new webcast monitoring system that goes far beyond what we've ever had. Every 5 minutes, our system creates a new webcast in the same way that a user does - it creates a new stream, and we monitor every step of the process to make sure the stream is created properly, gets routed to the outbound reflectors, and streams out properly. If anything along the way fails, our engineers get notified.

That being said, I have our engineers checking the logs from this past weekend to see what the system load was and see if there were any errors. I will report back the results.

In the meantime, I would suggest there is a strong possibility that you were the victim of perhaps our most common webcasting issue. Users test during the week, and everything works great. Then they come to the actual meeting, and the webcast falls apart. The reason: meeting attendees come to the meeting packing smartphones, iPads, etc., and connect (either knowingly or unknowingly) to the building's wi-fi, and consume the bandwidth, which kills the webcast.

The only way to prevent that is to completely disable all other access to the building Internet - wired and wireless. Announcing from the pulpit and asking people nicely to shut off their phones just doesn't cut it.

Brian - you may or may not have disabled other Internet access; I didn't see that in your description. But it has been a common problem.

ksolsen wrote:The only way to prevent that is to completely disable all other access to the building Internet - wired and wireless. Announcing from the pulpit and asking people nicely to shut off their phones just doesn't cut it.

The problem with the suggestion to disable wireless access to the building Internet is that the Church-managed Cisco 881W offers no reasonable way to turn off its wireless. We can disconnect any access points easily enough, but the wireless capability of the firewall itself is not under our control.

Kurt, I know it's not your area of responsibility, but since this very common issue has such an impact on webcasts, perhaps you could plead with your counterparts in the Meetinghouse Internet area to consider providing some way for us to disable the 881W's wireless for such events.

aebrown wrote:The problem with the suggestion to disable wireless access to the building Internet is that the Church-managed Cisco 881W offers no reasonable way to turn off its wireless. We can disconnect any access points easily enough, but the wireless capability of the firewall itself is not under our control.

Kurt, I know it's not your area of responsibility, but since this very common issue has such an impact on webcasts, perhaps you could plead with your counterparts in the Meetinghouse Internet area to consider providing some way for us to disable the 881W's wireless for such events.

In the, LDSTech Broadcast Aug 3, 2012 (~35 mins into it), Kurt said they were working on an official solution that would bypass or otherwise disable the firewall for a webcast, so maybe it's not too far off. In the mean time, I think a reasonable alternative (as he suggested) would be to allow the firewall to be bypassed for the express purpose of webcasting a stake conference.

Mikerowaved wrote:In the mean time, I think a reasonable alternative (as he suggested) would be to allow the firewall to be bypassed for the express purpose of webcasting a stake conference.

However, the firewall is doing two things: Protecting the webcast server from the Internet, and as a router allowing more than one computer to be connected. You could substitute a home type router, but that's additional equipment.

I believe the suggestion has been to remove the antennas from the 881W. That limits the usable range to a very small value. Probably small enough to prevent problems.

Mikerowaved wrote:In the, LDSTech Broadcast Aug 3, 2012 (~35 mins into it), Kurt said they were working on an official solution that would bypass or otherwise disable the firewall for a webcast, so maybe it's not too far off. In the mean time, I think a reasonable alternative (as he suggested) would be to allow the firewall to be bypassed for the express purpose of webcasting a stake conference.

Actually, with my new role, the product we call Technology Manager is slowly migrating over to my responsibility. Technology Manager offers the ability to manage Internet access in the meetinghouses. We are working in the requirement to restrict Internet access so that you could turn off wireless and dedicate it solely to a webcast. I don't have an exact ETA on that feature yet, but we are driving toward it.

In the meantime, although Church IT security doesn't love this recommendation, I wouldn’t blame you if you physically bypassed the router only for the duration of the webcast. Then immediately reconnect it afterward.

Disconnecting the antennas can help, but it is not a full solution - it decreases signal strength, but the signal can still be available even with them disconnected.

Kurt, sounds great. I would like to add one thing and that is to use the PVC for broadcasting. We successfully in Jun 2012 broadcasted our stake conference that way. We had no drops. Also, we could cut down the bandwidth by limiting the number of people we saw on the screen. There are still a few bugs to work out, but overall I found it to be great, more reliable and a much better option if we lose the Internet connection at the stake center so that we can broadcast it via phone.

So just some food for thought.

ksolsen wrote:Brian - you may or may not have disabled other Internet access; I didn't see that in your description. But it has been a common problem.

This was actually the first broadcast where we DID turn off the wireless (actually we changed the password so no one could connect).

But bandwidth speed tests from the broadcast machine still showed 15M down and 2 M up, so nothing was consuming the bandwidth. Also, the problems started before the crowds really showed up, and the only thing that changed (as far as I can tell), was that once we got disconnected we could no longer reconnect for more than 30 seconds.

It's great that the infrastructure was just upgraded (I'm sure more and more of the stakes are using this, all on Sunday morning, which poses a real challenge for your IT team). I'd be interested to know if the logs show anything.

Thanks!
Brian

Absolutely. PVC can be a very good alternative to use for stake conferences. With its H.264 SVC technology, it is more tolerant of varying Internet conditions.

BrianTAllen wrote:This was actually the first broadcast where we DID turn off the wireless (actually we changed the password so no one could connect).

But bandwidth speed tests from the broadcast machine still showed 15M down and 2 M up, so nothing was consuming the bandwidth. Also, the problems started before the crowds really showed up, and the only thing that changed (as far as I can tell), was that once we got disconnected we could no longer reconnect for more than 30 seconds.

It's great that the infrastructure was just upgraded (I'm sure more and more of the stakes are using this, all on Sunday morning, which poses a real challenge for your IT team). I'd be interested to know if the logs show anything.

Thanks!
Brian

Brian - we're doing a deep dive on the logs tomorrow. There may have been a specific issue. Can you send me your unit number used in the webcast? That way I can track your specific streams.

ksolsen wrote:Brian - we're doing a deep dive on the logs tomorrow. There may have been a specific issue. Can you send me your unit number used in the webcast? That way I can track your specific streams.

Kurt, I PM'd the unit number to you. Sorry, I just now saw your post.

Thanks,
Brian

Brian & others tracking this post,

We have spent a number of hours over the past few days looking at the log files for webcast events on August 26. Based on your unit number, Brian, we were able to see the starts and stops you experienced.

We also looked at what was happening on all webcast-related servers (inbound reflectors, coordinator, web servers, and outbound reflectors). I can tell you that the issue was not server capacity. There were a fair number of webcast events happening, but we did not exceed server capacity. The issue seems to be network-related on the HQ side. I'll post additional details as we get them.

Kurt,

Thank you so much for the update and the serious consideration you're giving this. I really appreciate it.

It's great to know the capacity is there and this was just a one-off problem that's being looked into.

Thanks,
Brian

We are having a Stake conference broadcast this month and due to the size of our stake, we are gathering in three buildings. The problem for me is that over the last two weeks, our new Cisco firewalls have crashed several times and two of the three buildings (one of which is the stake center). Physical Facilities has been called, but I am nervous about our internet outages.

Is the September conference somehow available in advance? Having the broadcast on DVD at each building would be tremendously reassuring.

kwinpete wrote:our new Cisco firewalls have crashed several times and two of the three buildings (one of which is the stake center).

I'd have a hard look at the environment they've been placed in. Cisco is a high-grade device. While I'm sure a bad unit gets out now and then, to have more the one suggests it's not being installed in a compatible environment - be it bad power, high heat or lack of ventilation.

I have been working to get a broadcast I felt was reliable for two years now... I have found a service that works perfectly but it is not provided by the church. The church has three systems. I have noticed the older two systems seem to work better on Sunday but not what I would call adequate. The Newest Webcast system works great on any day but Sunday. This seems to be an issue with the internet traffic in my area on Sunday. As an additional option I have started using a third party video conferencing service. The service I used for my most recent conference was Zoom.us. This service uses significantly less bandwidth, and this seems to be the key to resolving the issues I am having. When I perform my tests on Sunday using the Zoom.us service, I don't have to worry about disabling the wireless in the building in order to get enough band with to broadcast or receive. This service just works. I have the church systems setup as an alternative and continue to test them under the same circumstances but the only broadcast that works well is the Zoom.us system. I want to use the free church systems but due to their performance I have moved to third party options. The Zoom.com service has some cost to it. There is a monthly charge to use their service and this allows a video conference system to be setup. The Basic Charge is $15 a month. This allows unlimited meetings but to do just a broadcast there is another option. The Webinar function of this service costs $55 a month for 100 attendees. This option seems to be the best fit, but is more costly. I would like to resolve the ongoing issues this stake has with the provided Church Systems, but I have been unsuccessful thus far.