plott wrote: Sun Apr 10, 2022 3:42 pm
You say it has nothing to do with the servers - but when people ask you to improve availability they don't always just mean the physical hardware. There is a lot you can do to improve availability. 1) increase bandwidth 2) redundant and distributed hardware and processes to handle requests. 3). Properly sized, benchmarked and stress tested databases and webserver utility(full stack). 4) Networking - this includes being able to handle and address issues such a denial of service attacks.
Just saying you can't increase availability because you can't add more hardware is a basic cop-out
Based upon the symptoms, it definitely seems that you are having issues addressing the # of web requests on Sundays. Something seems to be crashing early in the morning when usage increases and then when you do finally get it back up the requests are fulfilled so slowly it seems like you have some serious bottlenecks in the full stack, storage, or networking.
There is a reason websites that experience heavy traffic don't always go down. Proper testing and preparation makes a lot of difference. Most often these issues arise with new websites because the weren't designed and tested to handle the realistic web traffic expected. When engineering these systems there is also a necessity in ensuring a safety factor to handle spikes.
There have been plenty of suggestions in the forums over the past couple of weeks that the Church needs to add more hardware or increase bandwith or <insert helpful suggestion here>. You have even joined in with your suggestions: "if the down time is due to overuse - you can fire up more servers to handle it in real-time."
Saying it isn't a matter of just adding hardware isn't a copout -- it is an acknowledgement that there are some complex engineering problems that may involve changes to the full stack from the underlying network all the way up to application performance issues. The suggestions that come in to add more hardware underestimate the complexity of the problem and you are making the point that it isn't just a "throw more resources at it" problem.
I'm not sure what your point is with your message. Are you suggesting that you don't think anyone is doing work like this and you think they should? Are you suggesting that the engineers involved don't know what to do next so you're telling them how you think they should solve the problem? Do you think that there aren't several engineering teams working on this from multiple angles to try and address the issues?
