VidiU webcasting, but no QOS? Am I reading this right?

I'm really excited about the prospect of broadcasting stake conference to other buildings in the stake. We have a few towns in out stake, and it would be really nice to cut down on travel (especially for our older members). I'm guessing it would serve 30-40% of the stake at their local buildings. Stake leaders have been whispering about it for some time, and now this announcement will probably push us in that direction.

I am very concerned, however, with the proliferation of LDSAccess devices in our building (phones, tablets, laptops, etc). It would not take much to overwhelm our Internet connection. I can imagine someone clueless streaming video to their smartphone from the mother's lounge while they listen to the meeting. The VidiU device should be considered mission critical, high priority on the network (above and beyond all other network devices here locally). Similar priority should be given at the receiving sites. Why haven't we received word on setting up QOS (etc) for webcasting?

In short, naive members should not even be capable of DDOSing our upload. How can we address this? Is there anything I can do myself? Am I overlooking something?

jasondivy wrote:Am I overlooking something?

Yes. TM (Technology Manager) has the ability to shut off all the wireless in the building. The standing advice is that all wireless is to be disabled for the webcast. Obviously the webcast equipment is to be hardwired, but that should be done anyway.

No offense, but that sounds pretty dumb. I would be very surprised if the 881 wasn't fully capable of QOS, or some other sort of traffic shaping. It's not an inexpensive piece of equipment. The hardware is already in place, it just needs to be properly configured.

For that matter I could even put a DD-WRT box between the ISP and the 881. I could hang the VidiU off the Cheap-o router and set up QOS on my own. (assuming the stream does not validate somehow by going through the VPN; unlikely) But I'd rather not. I'd rather do things by the book if I can.

Everyone is going to be expecting to have Internet access for their phones, not the least of which will be the high council. Take that away from them, and we'll hear no end of grumbling and "why not"s I'd rather not have to tell them that it's technically possible, but out of our hands.

So how does QOS work? And can it work in this situation? Keep in mind the bottleneck is the ISP link. If two streams of video are coming down, how can QOS at the chapel decide what's prioritized for the ISP bottleneck? Wouldn't the QOS have to be set at the ISP's routers to prioritize which packets get sent in the first place?

jasondivy wrote:No offense, but that sounds pretty dumb. I would be very surprised if the 881 wasn't fully capable of QOS, or some other sort of traffic shaping. It's not an inexpensive piece of equipment. The hardware is already in place, it just needs to be properly configured.

Whether or not is possible CHQ has not configured the 881's for that and there is no option for any local configuration. All configuration is handled by CHQ. I don't know why they haven't done that but I'd guess that either it hasn't been a high enough priority to do yet or they feel that supporting it for all units would be too much of a burden on LUS. There are far fewer possible technical problems with just shutting off wifi.

jasondivy wrote:For that matter I could even put a DD-WRT box between the ISP and the 881. I could hang the VidiU off the Cheap-o router and set up QOS on my own. (assuming the stream does not validate somehow by going through the VPN; unlikely) But I'd rather not. I'd rather do things by the book if I can.

I suppose you could do that (although bypassing the firewall is probably against policy). But another consideration is that the next person with your calling may not know how to do all that and LUS will not provide any help with it.

jasondivy wrote:Everyone is going to be expecting to have Internet access for their phones, not the least of which will be the high council. Take that away from them, and we'll hear no end of grumbling and "why not"s I'd rather not have to tell them that it's technically possible, but out of our hands.

Well don't tell them that. Just tell them that it is the only option endorsed and supported by the church. If they really need internet access they can use their cellular network.

Wouldn't the QOS have to be set at the ISP's routers to prioritize which packets get sent in the first place?

No, actually. All data is bundled up and sent through the VPN. I don't know what type of facility it goes to, but if any additional routers needed to be involved, the church already controls them.

In theory: Out-bound data is easy. Queue or drop anything that doesn't have priority. Inbound is harder, but still manageable. Remember, TCP behavior on a dropped packet is to assume that the route has been saturated, to back off on bandwidth, and resend. In this way, even download speeds can be managed. It would take either a deliberate DDOS or a state-less protocol (UDP, etc) to saturate the inbound line if the router employed selective packet dropping. (I'm not sure what method Cisco uses. They're bound to be at least that smart.)

People use DD-WRT (etc) at home with QOS all the time to limit download speeds of errant applications (or users) without having any control over their ISPs routers. This isn't out of the ordinary, it's just not something most home users ever encounter.

CHQ has not configured the 881's for that and there is no option for any local configuration

I've noticed.

There are far fewer possible technical problems with just shutting off wifi.

I beg to differ. It's a technical problem that exists today. I only found the recommendation to turn wifi off after starting this post. (I was actively hunting for the answer.) Most STS won't find it at all. I wonder how many stake conferences are going to have bizarre, intermittent problems before they discover that little nugget...

The biggest benefit of a centrally managed network is that you can fix each problem once, and push that one solution out to the entire network. It would make more sense to actually fix the issue once for ALL the 881 boxes out there in one fell swoop than it would to simply pray that every STS has patience to read every bit of text on the site. "Oh, I made a note in the documentation that it is someone else's problem. Job done."

Now if they're just busy fixing other issue, that would be different. Maybe they just haven't gotten around to this one yet. (I doubt it, but it is possible.)

But another consideration is that the next person with your calling may not know how to do all that...

Not a problem (but see below). (1) If I left it there, it would continue silently working without anyone knowing. I wouldn't, of course, but I could. (2) If I pulled it out after every use then the next guy is stuck with exactly the same problem. It doesn't create any new problems for him, it just unsolves one.

... and LUS will not provide any help with it.

This is a half-valid a concern. It isn't that it would be hard to do (or test), but if something else went wrong it could really confuse LUS and detract from solving the real problem (whatever that might be). As I said, I'd prefer to do this by the book. This is a big reason why.

Well don't tell them that.

I'm not going to lie, even by omission. If that's the route we go, then they deserve to know.

If they really need internet access they can use their cellular network.

Not everyone has a data plan, especially for laptops and tablets. Yes, people use those.

jasondivy wrote:Not everyone has a data plan, especially for laptops and tablets. Yes, people use those.

Do you mean to tell me that they might actually have to listen to the conference?

Do you mean to tell me that they might actually have to listen to the conference?

Who's to say that they're not?

There are any number of reasons why someone may want to be online during Stake Conference. (Some reasonable, others less so; I'm not judging.)

jasondivy wrote:There are any number of reasons why someone may want to be online during Stake Conference. (Some reasonable, others less so; I'm not judging.)

I agree that there are some valid reasons, and I'm not judging either. But, since we started streaming stake conferences several years ago, the standard (and first) procedure for a successful broadcast has always been to disable the WiFi. Because the Church does not choose to expend tithing funds on industrial strength bandwidth, what we have becomes a precious resource that we reserve solely for the broadcast if we wish to get the signal through to our outlying units.

jasondivy wrote:All data is bundled up and sent through the VPN.

I don't think so. The VPN is only used for some data. The whole internet doesn't go through there. And the webcast solution is based on Azure. So it's not inside the VPN.

The other problem is that the church needs a configuration that works with all ISPs.

And you've already pointed out that UDP isn't very manageable. So bottom line, the best method to protect the video stream is to turn off general access.

jasondivy wrote:

Well don't tell them that.

I'm not going to lie, even by omission. If that's the route we go, then they deserve to know.

What lie? Saying that it's the only option that is endorsed and supported by the church implicitly indicates that there may be other options but it doesn't emphasize it. Saying "It's technically possible, but..." makes it sound like it should work but somebody is incompetent or otherwise isn't doing what they could do. It has negative connotations towards the church's technical team and/or decision makers. I don't think that's called for and would keep my explanation to the main point.

I don't think so. The VPN is only used for some data. The whole internet doesn't go through there.

Then why is it so much faster to run a speed test when you bypass the 881? And why does the speed test tell me that my signal is originating in the wrong city? (It gives me the correct city when I bypass.) I conclude that every speed test I've run has gone across the VPN. I know you can't get internet when that light is off (but that's not conclusive).

And the webcast solution is based on Azure. So it's not inside the VPN.

Irrelevant. Don't get hung up on the idea that QOS is entirely for intranet service. QOS is regularly used for webcasts, video conferencing, VOIP, and other mission critical high-bandwidth or low-latency applications -- even if the other end is across the Internet.

The other problem is that the church needs a configuration that works with all ISPs.

And why wouldn't it? I've outlined several reasons why it would. Your turn.

And you've already pointed out that UDP isn't very manageable.

How many people are using UDP to surf the web? We could simply block UDP. If someone felt like being fancy, we could have a switch on tm.lds.org to turn UDP on and off. I included it for completeness, not because it was an actual problem. A deliberate DDOS is always a problem, unless the church wants to have dedicated lines. In other words, even if you turn wireless off you still have this problem.

Because the Church does not choose to expend tithing funds on industrial strength bandwidth, what we have becomes a precious resource that we reserve solely for the broadcast if we wish to get the signal through to our outlying units.

That's exactly what I'm talking about, but without the unnecessary inconvenience of shutting out stake members. I'm not suggesting we run enterprise grade fiber to every building. I'm suggesting we make sane use of the DSL and cable connections that we have.

Saying "It's technically possible, but..." makes it sound like it should work but somebody is incompetent or otherwise isn't doing what they could do. It has negative connotations towards the church's technical team and/or decision makers.

Are you really going make me spell it out for you? No, I don't think church leaders are being incompetent. I do think someone has really dropped the ball on this. (someone on the technical staff)

Do you think we are automatically perfect because we belong to the church? Does being part of the technical staff of the church make you immune to mistakes? Of course not. The Lord even chewed out Joseph from time to time. ALL of us err. You and me included. When I err, I get called on it, especially when other people are relying on me.

They didn't give us the ability to manage the 881. Fine. I get it. I wouldn't either. But that means that it needs to be properly set up to begin with. I don't think it is.

To this point, nobody has enunciated a single good reason why they haven't dealt with this. Maybe a reason exists. At this point, I conclude it is likely that there is no good reason. Either someone overlooked it, or someone doesn't want to be bothered.

jasondivy wrote:To this point, nobody has enunciated a single good reason why they haven't dealt with this. Maybe a reason exists. At this point, I conclude it is likely that there is no good reason. Either someone overlooked it, or someone doesn't want to be bothered.

And I conclude there is a good reason, but we haven't been told about it. I base that on past issues that seemed to have no good reason, only to learn there is one. Sometimes it's just priorities or amount of effort required didn't justify the benefit. Sometimes there really is a technical issue. When you do get a chance to talk to the folks at CHQ, you find they're not a clueless as it might seem.

russellhltn wrote:TM (Technology Manager) has the ability to shut off all the wireless in the building.

The first time I tested this feature of TM, I found that it shut off the 881W wireless and the 1041n WAPs, but it did not shut off the legacy Cisco Aeronet 1200 in the stake center. I had to manually turn off that WAP.

drepouille wrote:

russellhltn wrote:TM (Technology Manager) has the ability to shut off all the wireless in the building.

The first time I tested this feature of TM, I found that it shut off the 881W wireless and the 1041n WAPs, but it did not shut off the legacy Cisco Aeronet 1200 in the stake center. I had to manually turn off that WAP.

Ah, good point. It's also not likely to run off any legacy APs that were installed back when it was all installed on the stake's dime.

jasondivy wrote:I only found the recommendation to turn wifi off after starting this post. (I was actively hunting for the answer.) Most STS won't find it at all. I wonder how many stake conferences are going to have bizarre, intermittent problems before they discover that little nugget...

The checklist for managing a webcast event https://www.lds.org/help/support/manage-webcast-event includes "Disable Meetinghouse Internet Wireless Access to preserve Internet bandwidth at each location."

This message is displayed prominently on the TM home page: "Wireless access may now be disabled/enabled through Technology Manager. For meetinghouses the Stake Technology Specialist or Stake/District/Ward/Branch presidencies may do so under the direction of the meetinghouse Stake/District President. The disable feature prevents devices from using LDSAccess."

Here is a case where QOS doesn't mean a hill-o-beans when it comes to a webcast. I was an STS in a stake with 9 meetinghouses, 7 of which had no better than 1.54Mbps down speeds, the stake center we did our webcasts from also was in this speed range, which meant that the upload was barely 300K.

This solution is designed to be successful from a site with MAX 300Kbps upload speeds and still be successful. I don't disagree that if I had a 50Mbps pipe that some kind of QOS could be put in place to allow the wireless and those at the meetinghouse usage while the webcast is going on. But, it isn't applicable in all cases, and, frankly, could become a worse situation, for those not very technical, to understand at what point it would be ok to tell members they can use their devices, and when they can't. The Wireless ON/Wireless OFF works no matter what. I think the KISS principle applies.

I work in and environment where QOS is used heavily internally between our data centers, and externally on circuits between our clients and our data centers, I've seen it blow up, I've seen it react differently in versions of code on the routers, I've seen a need to tweak ongoing as bandwidth is increased or decreased or new solutions come into play. Using a personal vpn will bypass it, too many complications IMHO.

jasondivy wrote:Everyone is going to be expecting to have Internet access for their phones, not the least of which will be the high council. Take that away from them, and we'll hear no end of grumbling and "why not"s I'd rather not have to tell them that it's technically possible, but out of our hands.

Prior to our recent multi-stake conference, I sent out multiple e-mails to all leaders, asking them to tell everyone attending the conference that the wireless in the stake center would be disabled. They were asked to bring scriptures, either on paper or on a cell data plan. As the attendees arrived, we could see a few with puzzled looks on their faces as they tried to use their devices, but they survived.

We had a multi-stake broadcast recently and unfortunately the satellite feed went down, so the backup was brought into play at very short notice. A PC using an Internet feed. To make sure we didn't have buffering problems, I shut down the wifi access to the building, 15 minutes before the start of the broadcast. I made a quick announcement to the audience to that effect. No one complained, or even commented about the lack of wifi access. Even the late comers, who hadn't heard the announcement!!

So, I am a STS and have implemented my own version of QOS. During Stake Conference broadcasts to our three other buildings and shut-ins I go through several steps to create an equivalent of QOS for our broadcast. I remotely shutdown the wireless from my broadcast sites and receiving sites. I really love this ability. I then manually go in and pull the ethernet cables from every port on the 881 except my connections for the VidiU device and the system controlling the VidiU device. I had to do this because I have had the occasion where some clerk or other individual has decided to access the internet through a clerks hardwired access DURING conference no less. I had notified the Stake Presidency and High Council of such action. The Stake Presidency have given me permission to take whatever measures necessary to provide the best possible streaming experience within the confines of explicit or implicit church policy. (Explicit and implicit as per my interpretation so do not start a thread on these definitions please.)

In my stake, we had WIFI on during one conference and the reception was very poor. We had them pull the plug on the WAPS and the broadcast improved greatly. There have been a few members who have ragged me about it, but they understand that I need to shut off the WIFI in the buildings in order to maintain a good quality conference. People shouldn't be on their phones during conference and if they are then they should use their data plans, that is what they are for. My stake president has said do what is necessary because he wants to ensure that the broadcast is the best it can be.

As for QoS, it is used only when the pipe gets saturated. Remember the Internet has NO QoS, in fact it doesn't even guarantee that the packet will get across there. So on the sending side, you could use QoS to prioritize the traffic leaving the site, but once it hits the Internet, the QoS marking is ignored. So when the packet get to the receiving site, I'm pretty sure that the QoS tag that was set from the sending side has been stripped and remember coming from the Internet the QoS tagging is being ignored and gets thrown into the default class with all other traffic coming down.

Now about the path and the VPN. The VPN is only used for communication with HQ and to manage the routers. The Family History is monitored by their group through the same VPN, but as for the rest of the traffic it goes straight to the Internet. If you want to confirm this, do a tracert -d to www.lds.org or any other site on the Internet. I can pretty much tell you that you will not see a 10.x.x.x address anywhere in the trace route and if you do it is because the modem that connects to the ISP has a 10.x.x.x address and I can pretty surely say that it won't be the IP address for the VPN. So no internet traffic goes over the VPN and you might want to check to see where the DNS servers are pointing because where ever they are located can tell it where it thinks the system is.

I use OPENDNS and when I VPN into my office, I usually connect through a gateway in the west or south and I'm on the east coast. So, in a way, I believe that the DNS server plays a roll in where speedtest.net thinks you are.

Just my 2 cents worth from a network engineer by profession.
Terry