Google Calendar Sync Issue (Robots.txt)
Re: Google Calendar Sync Issue
You can use the URL only once every nine minutes. If you try to refresh/reread it more often then that, you will get an error. Creating a new URL does NOT reset the timer. Note that this includes all devices that use the URL, so if you've just added a device or service to sync to, that could be the problem.
So we can better help you, please edit your Profile to include your general location.
Re: Google Calendar Sync Issue
Re: Google Calendar Sync Issue
So we can better help you, please edit your Profile to include your general location.
Re: Google Calendar Sync Issue
I wonder if there is a setting in Google that tells robots.txt to allow all? Remember the error message is:
Could not fetch the url because robots.txt prevents us from crawling the url.
When I google this issue, here's what I get:
URLs restricted by robots.txt errors
Google was unable to crawl the URL due to a robots.txt restriction. This can happen for a number of reasons. For instance, your robots.txt file might prohibit the Googlebot entirely; it might prohibit access to the directory in which this URL is located; or it might prohibit access to the URL specifically. Often, this is not an error. You may have specifically set up a robots.txt file to prevent us from crawling this URL. If that is the case, there's no need to fix this; we will continue to respect robots.txt for this file.
If a URL redirects to a URL that is blocked by a robots.txt file, the first URL will be reported as being blocked by robots.txt (even if the URL is listed as Allowed in the robots.txt analysis tool).
Re: Google Calendar Sync Issue
The fact that you could download the iCAL file shows that the LDS.org calendar sync service is working properly.azwheels wrote:I put the url into the Google Chrome web browser. It downloaded a file, which I opened in Notepad. Looks like it contains info about each calendar event.
I wonder if there is a setting in Google that tells robots.txt to allow all?
I don't understand why Google Calendar would even care about a robots.txt file. After all, the only operation that is needed is to download a single file from the URL that you supply (the sync URL). There should be no web crawling involved.
I personally sync my LDS.org calendar to Google Calendar, and I've never seen that error message.
Re: Google Calendar Sync Issue
Re: Google Calendar Sync Issue
I would think that is quite unlikely. The message does seem like something Google could be reporting, and that report would be based on information that Google sees on the lds.org server as it tries to download the file. I wouldn't think that malware would be likely at all to interfere with that part of the process.azwheels wrote:OK. Do you think maybe some malware or virus could cause this?
I'm not sure what to suggest. You might try generating a new sync URL on the LDS.org calendar and see if that helps.
Re: Google Calendar Sync Issue
Re: Google Calendar Sync Issue
You might re-check your procedure for entering the URL. I don't think "robots.txt" comes into play unless you're asking google to index the link to make it available for a web search.azwheels wrote:Could not fetch the url because robots.txt prevents us from crawling the url.
So we can better help you, please edit your Profile to include your general location.
Re: Google Calendar Sync Issue
Re: Google Calendar Sync Issue
Re: Google Calendar Sync Issue
Re: Google Calendar Sync Issue
Re: Google Calendar Sync Issue
Re: Google Calendar Sync Issue
Re: Google Calendar Sync Issue
So we can better help you, please edit your Profile to include your general location.
Re: Google Calendar Sync Issue
Re: Google Calendar Sync Issue
So we can better help you, please edit your Profile to include your general location.
Re: Google Calendar Sync Issue
Re: Google Calendar Sync Issue
CK Stratford
Feel free to follow me @ https://twitter.com/CKStratford
Feel free to follow me @ https://www.youtube.com/user/BomWarriorVlog
Re: Google Calendar Sync Issue
In the meantime, I suppose the workaround is to manually check the lds.org/church-calendar ? Has anyone else found a reasonable work around?
PS: I ran into this issue because I deleted my church calendar from my google calendar settings, generated a new sync URL, and then tried to add my church calendar back to google calendar.
Code:
Could not fetch the url because robots.txt prevents us from crawling the url.Re: Google Calendar Sync Issue
Thanks
Re: Google Calendar Sync Issue
So we can better help you, please edit your Profile to include your general location.
Re: Google Calendar Sync Issue
Re: Google Calendar Sync Issue
Re: Google Calendar Sync Issue
Re: Google Calendar Sync Issue
I wonder if it's necessary to delete the calendar. Have you tried re-importing into the existing calendar?ramonc wrote:Come Saturday night I will delete my google calendar SLC-Manual, Create it again, download from lds.org again and import it again.
My primary use is to get the sync onto my Android device. Unlike iOS, Android does not have a way to do it natively. But there are apps that can be installed. One possible advantage - you can sync as fits your schedule rather than wait until Google gets around to it.
So we can better help you, please edit your Profile to include your general location.
It may be an issue of interpreting standards. Robots.txt is a file used by the site to direct web crawlers what is and is not to be indexed. (See Robots exclusion standard) The question is, should Google pay attention to Robots.txt when being directed by a specific user to retrieve an ICS file from a specific location for their own use? It's understandable that the church doesn't want Google to index everyone's calendar into their search engine, but there's no intent to block what Google does in behalf of a specific request from a user.brucejay wrote:This is a problem on Google's end. I have had this problem before, and have it now. Google is not able to communicate with the server. However, a browser can communicate with the server and download the calendar to an ics file that can be imported into the Google calendar. The questions and comments need to be directed to Google as to the sync process.
Web browsers ignore robots.txt as they are not "robots" but work as directed by the user.
So we can better help you, please edit your Profile to include your general location.
To the person who recommended to delete the "robot.txt" from the HTML. Can you be more specific about how this is done?
Whoever made that suggestion is a bit confused. The file robots.txt (it has an s on that "robots", by the way) is not in the HTML. It is a separate file. And since it is one file for LDS.org, you can't just remove it from the Calendar sync service without affecting more than just sync URLs.rbraithwaite1995 wrote:So basically nothing has been resolved on this issue?
To the person who recommended to delete the "robot.txt" from the HTML. Can you be more specific about how this is done?
Search engines gather the information for their search indexes by "crawling" the web using applications known as "web crawlers" or "spiders" or "robots" to find pages of content, then they put that information in indexes so that they can search for that content. The robots.txt file allows sites to tell search engines to ignore anything they find in certain parts of their site.
The robots.txt file for LDS.org can be found at https://www.lds.org/robots.txt . If you examine the file, you'll see a line that says "Disallow: /church-calendar". That means that web crawlers are supposed to ignore everything found in lds.org/church-calendar and below. So if we were only talking about web crawling, then Google would be correct in ignoring sync URLs, since they are found below lds.org/church-calendar. But reading a sync URL has nothing to do with crawling the web, so Google's Calendar application is misinterpreting the purpose of robots.txt if they read that and assume that they can't open a file from there.
I'm not so sure. When I look up the purpose of Robots.txt, it seems to be aimed at automated processes, not just search engine web crawlers.aebrown wrote:Google's Calendar application is misinterpreting the purpose of robots.txt if they read that and assume that they can't open a file from there.
When I look for complaints about the robots.txt for Google calendar - I can find them, but they are from years ago. And thanks to the Wayback Machine we can see that the there was a change to the church's robots.txt done on Aug 13. So while we can argue standards, it was the church that made the change that broke things. The fact it hasn't been changed back makes me think there's a security issue involved.
So we can better help you, please edit your Profile to include your general location.
Another issue (should probably start a new thread for this, but I'm not doing it now) is that for few montyhs already I have not received any email notifications from lds calendar even though based on my settings and events added I should have.
With this I mean the link for embedded calendar.
How long did you wait? There's a 9 minute throttle on the calendar. If you try to use it within the 9 minutes, you'll get a http error (although many things won't report that to you.) In most browsers, you'll end up with nothing. Creating a new URL won't reset the timer - it's tied to your account.diidu wrote:After that I did not see the error anymore, but no sync either. Copying the link to browser window displays an empty calendar in Firefox.
It's addressed in the LDS.org calendar not sending email updates thread.diidu wrote:Another issue (should probably start a new thread for this, but I'm not doing it now) is that for few montyhs already I have not received any email notifications from lds calendar even though based on my settings and events added I should have.
So we can better help you, please edit your Profile to include your general location.
What's your source? Everything I read talks about web crawlers. See here, or here.russellhltn wrote:I'm not so sure. When I look up the purpose of Robots.txt, it seems to be aimed at automated processes, not just search engine web crawlers.
It's true that the Church made a change, and it was an important change, but I don't think it's fair to say that it was that change "that broke things." That implies that the Church did something wrong in their change. The change they made on August 13 was to change the "disallow" directory from "/calendar" to "/church-calendar". I don't know why they ever had "/calendar" in the disallow category, since the calendar stuff has been under "/church-calendar" for years. I would think it's more accurate to say that the Church finally corrected their mistake.russellhltn wrote:When I look for complaints about the robots.txt for Google calendar - I can find them, but they are from years ago. And thanks to the Wayback Machine we can see that the there was a change to the church's robots.txt done on Aug 13. So while we can argue standards, it was the church that made the change that broke things.
It may or may not be a security issue. If the web crawlers for search engines somehow became aware of Sync URLs and started crawling those URLs (perhaps because they come across them in some other source where someone makes them available, such as sometimes happens here on LDSTech where someone posts their personal sync URL), then they would be affecting the throttle mechanism. It could be that the exclusion is simply to avoid such a problem.russellhltn wrote:The fact it hasn't been changed back makes me think there's a security issue involved.
That second link says "and other web robots". It should be noted that the file is "Robots" not "Crawler" or "Spider".
The bottom line is that Google thinks their calendar import bot should obey Robots.txt. It's made worse that it appears Googlebot is the name of both the web indexing function and the calendar import function. (Some other Google indexing function have different names so they can be specifically controlled by the text file.)
We can debate fault, but it's that change that caused things stop working. Not a change implemented on Google's side. Had this text file been that way from the beginning, it would appear that Google Calendar sync would have never worked and perhaps never been offered.aebrown wrote:It's true that the Church made a change, and it was an important change, but I don't think it's fair to say that it was that change "that broke things."
So where do we go from here? Unless the church changes the text file, perhaps allowing just Googlebot to get files, this isn't going to get fixed anytime soon.
So we can better help you, please edit your Profile to include your general location.
And what is a robot? You seem to be assuming that it might mean something other than a crawler or spider. But the definition given at http://www.robotstxt.org/ is:russellhltn wrote:That second link says "and other web robots". It should be noted that the file is "Robots" not "Crawler" or "Spider".
The key attribute is that robots "traverse the Web automatically." That is definitely NOT what is happening when Google Calendar accesses a Sync URL -- that is hitting a specific URL provided by a user.Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses.
You're correct that Google seems to think that. But Google is wrong.russellhltn wrote:The bottom line is that Google thinks their calendar import bot should obey Robots.txt.
That's rather speculative. You're assuming that Google's implementation now is the same that it was back when Calendar Sync was implemented. I personally have no idea whether robots.txt was considered in Google Calendar's sync at that point, or if it was changed in the meantime. I don't think we have enough information to draw a conclusion one way or the other.russellhltn wrote:We can debate fault, but it's that change that caused things stop working. Not a change implemented on Google's side. Had this text file been that way from the beginning, it would appear that Google Calendar sync would have never worked and perhaps never been offered.aebrown wrote:It's true that the Church made a change, and it was an important change, but I don't think it's fair to say that it was that change "that broke things."
Changing the robots.txt file to allow only Googlebot (or removing that disallow line entirely) would fix the calendar sync, but the Church might not think it would be acceptable. That would almost certainly subvert the reasons why the Church specifically chose to disallow the /church-calendar directory in robots.txt.russellhltn wrote:So where do we go from here? Unless the church changes the text file, perhaps allowing just Googlebot to get files, this isn't going to get fixed anytime soon.
I certainly don't know why /church-calendar is in robots.txt. Much of what is presented on web pages under that path is protected by login. The main Calendar view, and all its related pages under Settings all appear under that path, but none of those would be available to a web crawler, so that's no reason to disallow that path via robots.txt. The only URLs I know of under that path that are publicly visible are for sync URLs, which contain a very long identifier as part of their path. A web crawler would have no way to visit such URLs unless a link to that URL were posted on some other publicly available page. One possible motive for disallowing that path in robots.txt is to avoid the effects of the throttle mechanism being activated as web crawlers follow such links that they found on a public page. If that's the main reason, I personally think the Church could remove their exclusion. Anyone who posts their Sync URL publicly deserves whatever pain comes from the throttle mechanism being activated by a web crawler, since they were specifically told "This URL is provided for your own personal use. It should not be shared with anyone else."
Another possibility is for the Church to try to get Google to change their implementation. That certainly seems like a long shot, but Google has on occasion been known to make changes based on user feedback.
From what I've read of this issue (people's comments who appear to be far more technologically "in the know" than am I), it seems like the Church needs to "suck it up," play nice with Google and let us sync up. Because in the real world, people use their GOOGLE calendars, and rarely look at the Church's proprietary online calendar, unless they are scheduling a building. And it sure would be nice to know what's going on in my ward and stake by looking at my phone, where (like most people today) I live and die.
Ultimately, that's really up to a standards body.aebrown wrote:You're correct that Google seems to think that. But Google is wrong.russellhltn wrote:The bottom line is that Google thinks their calendar import bot should obey Robots.txt.
This post from February and this post from 2013 shows it's been that way at least that long. Someone even started a service to deal with the problem for iCloud users.aebrown wrote:That's rather speculative. You're assuming that Google's implementation now is the same that it was back when Calendar Sync was implemented. I personally have no idea whether robots.txt was considered in Google Calendar's sync at that point, or if it was changed in the meantime. I don't think we have enough information to draw a conclusion one way or the other.
The good news is that things are moving behind the scenes on the church's side. We can expect a solution soon. Maybe even today.
So we can better help you, please edit your Profile to include your general location.
Unfortunately, there's no standards body for robots.txt -- it's simply a de facto standard.russellhltn wrote:Ultimately, that's really up to a standards body.aebrown wrote:You're correct that Google seems to think that. But Google is wrong.russellhltn wrote:The bottom line is that Google thinks their calendar import bot should obey Robots.txt.
The robots.txt file for LDS.org no longer has a "Disallow" line for /church-calendar, which is good news.russellhltn wrote:The good news is that things are moving behind the scenes on the church's side. We can expect a solution soon. Maybe even today.
However, when I try to add my calendar sync URL to my Google Calendar, I still get the error message about robots.txt. Perhaps Google is doing some caching, which will take some time to expire.
I'm seeing the same thing. Your guess sounds as good as any.aebrown wrote:The robots.txt file for LDS.org no longer has a "Disallow" line for /church-calendar, which is good news.
However, when I try to add my calendar sync URL to my Google Calendar, I still get the error message about robots.txt. Perhaps Google is doing some caching, which will take some time to expire.
So we can better help you, please edit your Profile to include your general location.
I found this, where Google says: "First, the cache of the robots.txt file must be refreshed (we generally cache the contents for up to one day)."aebrown wrote:However, when I try to add my calendar sync URL to my Google Calendar, I still get the error message about robots.txt. Perhaps Google is doing some caching, which will take some time to expire.
So it may still take another 20 hours or so until Google's cache expires. Hopefully the sync process will start working once their cache of robots.txt is refreshed.
However, nothing actually gets imported. The church calendar appears on my list of Other Calendars, but there are no events. And previously, I at least saw a "Loading..." popup right after I selected "Add by URL", which remained in place for a few seconds until the robots.txt error message was displayed. Now I don't see anything happen, except that the calendar appears in the list of Other Calendars. There are no events. And I'm making sure that I wait far more than 9 minutes between my attempts, so there should be no throttle issues.
An update to my update: Hallelujah!aebrown wrote:An update:
I decided to generate a new Sync URL. When I used the new Sync URL in "Add by URL" in Google Calendar, I got the "Importing calendar by URL..." popup. So that was comforting.
Then that message went away, and no events appeared at first. But I went back and checked a few minutes later, and all the events from my stake calendar are truly there.
I have no way of knowing now whether generating a new Sync URL was an essential step; it may be that caches finally got completely cleared out, and so it would have worked now even with my old Sync URL. Perhaps someone else can try it with an existing Sync URL and report what happens.
In any case, it seems to work correctly now for an initial import. I'll edit an event on the stake calendar and see if and when that change comes across to my Google Calendar.
I think the new sync URL allows you to side-step Google's cache and speed things up. I'll just leave my main account alone and see when it reflects last night's change.aebrown wrote:I have no way of knowing now whether generating a new Sync URL was an essential step; it may be that caches finally got completely cleared out, and so it would have worked now even with my old Sync URL. Perhaps someone else can try it with an existing Sync URL and report what happens.
But at this point, I think things are back to normal - but our testing is showing some of the lag in using Google calendar instead of a native app on a device.
So we can better help you, please edit your Profile to include your general location.
I waited for few minutes, more minutes and now I've waited for two days. Anyway I don't remember exactly what I did. Today I tried the direct sync link again in two different browsers and ascii text content was received (readable in notepad). The content seemed like it would be usable for syncing. However I still don't see my lds calendar in my google calendar. If the content received is error free, I assume the problem is at google's end. Any idea what I can do to fix it?russellhltn wrote: How long did you wait? There's a 9 minute throttle on the calendar.
Thank's for the link. I'll check that.russellhltn wrote:It's addressed in the LDS.org calendar not sending email updates thread.
We know that the Church changed robots.txt on LDS.org sometime yesterday afternoon. Sometime this morning it appears that Google's cache of that file finally expired, opening the way for Google Calendar to properly process synced calendars.diidu wrote:Today I tried the direct sync link again in two different browsers and ascii text content was received (readable in notepad). The content seemed like it would be usable for syncing. However I still don't see my lds calendar in my google calendar. If the content received is error free, I assume the problem is at google's end. Any idea what I can do to fix it?
So the remaining delay for existing calendars should be related to however often Google Calendar checks for updates to the content on a Sync URL. No one seems to know that time for certain, but there have been reports that it is around 24 hours. So an automatic update might not happen until sometime tomorrow morning.
If you don't want to wait, you can certainly generate a new Sync URL (on the Sync page of the LDS.org Calendar) and use that in your Google Calendar. That should let you see your LDS.org calendar on Google Calendar within a few minutes.
My Google calendar has updated by itself with no intervention on my part. I'm sure in a little bit that my phone will update to reflect the calendar. So at this point It think it's safe to say everything is back to normal.russellhltn wrote:I'll just leave my main account alone and see when it reflects last night's change.
So we can better help you, please edit your Profile to include your general location.
It finally did. I was disappointed in the lag between the Google calendar and Android update. Apparently a update coming from a URL doesn't do a push update to devices.russellhltn wrote:I'm sure in a little bit that my phone will update to reflect the calendar.
Still, even if the update takes 48 hours, that's probably adequate for most usage. I'm assuming that leaders are not in the habit of making last-minute changes and when they do they send a email. But it would suggest that those who really want up-to-date syncing should look into installing an app that can sync from the URL directly.
So we can better help you, please edit your Profile to include your general location.
Good question. I don't have a answer. Based on my experience, it could be a day. I think things manually entered into Google gets pushed out faster, but somehow Google doesn't "trigger" on changes coming in from a URL.rbraithwaite1995 wrote:how long will I need to wait for google calendar to be pushed to ical on my mac or iphone calendar??
So we can better help you, please edit your Profile to include your general location.
Google doesn't push to your iPhone, anymore than LDS.org pushes to Google Calendar. Both are pull operations. So the question is how long it takes your iCal to pull new data from Google. I don't know those apps, but you might be able to do something in those apps to request an update if you want it to happen faster.rbraithwaite1995 wrote:ok, so I now also have success with importing LDS Calendar to google calendar. Now Im just waiting for it to update on my iphone calendar!! nothing yet...it's only been about 15 mins...would think this would be quicker...how long will I need to wait for google calendar to be pushed to ical on my mac or iphone calendar??
Thanks.
Yes, that's my experience as well.rbraithwaite1995 wrote:usually my iphone calendar will update in real time when anything is added to my google calendar
I agree. But that's something that is outside of church control. It's either an iCal or Google issue. Most likely it's Google as I had a similar problem with my Android.rbraithwaite1995 wrote:Im not trying to get new calendar items from lds.org to google calendar and then to ical. I just want to get all the lds.org events that are already there that are currently in my google calendars to go to my ical. I just dont think it should take more than 12 hours for this to happen.
Unless there's a need to have it come from your Google calendar, you might want to consider entering the sync directly into your phone. I understand iOS devices can sync directly while Android needs an add-on app to do it. That way sync is under your control.
So we can better help you, please edit your Profile to include your general location.