Tools for Making Ward Directories and Maps

Hello,

I wanted to let folks know about some resources for making ward directories and maps.

One of the challenges is ways of converting information into a usable format and keeping it tight. I try to distribute our ward directory on a single page 8.5x11 sheet on one side and other information, such as leadership, building numbers, schedules on the other.

I've released a command-line tool for converting MLS Membership exports into various directory formats.

http://ldsadmin.org

The resulting information can easily be put into word processor or spreadsheet documents for formatting and printing.

Also, using the above tool to pre-process the data (or not), you can easily put together a Google Earth map utilizing a method I detailed on the excellent LDS Clerks wiki. Google Earth is free software for easily visualizing geographic information in 3D.

Take care,

David Bailey

davebailey wrote:
Also, using the above tool to pre-process the data (or not), you can easily put together a Google Earth map utilizing a method I detailed on the excellent LDS Clerks wiki. Google Earth is free software for easily visualizing geographic information in 3D.

Thanks, David. I have used the method you describe -- having batchgeocode.com geocode the records and create a KML file for Google Earth. It works, but with one large caveat: it does not handle geocoding errors.

The batchgeocode.com web site internally uses the geocoder at Yahoo, submitting each address and getting the result as an XML transaction. But batchgeocode.com discards the errors and warnings that Yahoo sends back. Rather, it just takes Yahoo's best guess and plots it. Usually Yahoo guesses right in parsing and mathcing an address, but sometimes it guesses quite wrong. The batchgeocode.com site will plot those errors -- families mapped in grossly wrong locations, such as Zip code centroids or incorrect streets -- without so much as a warning to the user.

Well, that's true in the import/export process, but you can fix points in Google Earth if you are aware of errors before distributing the KML file. Also, I have found the process to be very accurate, typically within a block of its actual location, at least where I am in the Southeastern United States.

If you know of an error free geocoding method, I'm all ears!

davebailey wrote:Well, that's true in the import/export process, but you can fix points in Google Earth if you are aware of errors before distributing the KML file. Also, I have found the process to be very accurate, typically within a block of its actual location, at least where I am in the Southeastern United States.

If you know of an error free geocoding method, I'm all ears!

Actually, I don't know of an error-free geogcoded method, because geocoding is inherently error-prone. The best that can be done is to trap the flagged errors and precision qualifiers and deal with those as special cases. That is why quality geocoders -- even Yahoo's free API -- flag such errors and warnings. But users ignore those specific warnings at their own peril.

Unfortunately there does not seem to be a batch-geocoding method that reports the error flags and is also legal and free for general use. It is even problematical to license commercial solutions on a scale useful to local units. (Search terms such as "geocode" on this forum and you will find lots of discussion about that problem.) The best hope for a general solution for local units is that Church developers are known to be working on a central tool that we hope will be released sooner rather than later. Such a tool, presumably, will engage local clerks to correct errors.

Your suggested method for correcting the errors in the KML file assumes that the user can detect the errors somehow. That can happen if addresses are miscoded so far away to be obvious outliers, but finding other errors witihout benefit of the underlying geocoder's precision flags is not easy. It's a matter of not knowing what you don't know.

In the case of batchgeocoding,com, Yahoo's geocoder does flag precision warnings, but batchgeocode.com ignores the flags. (I wrote some geocoding scripts that accessed Yahoo's API directly, and retrieved the results along with the precision fields, and created KML for Google Earth after a process of manual review by clerks. But I withdrew those scriptswhen I discovered this method seems to violate Yahoo's terms of use.)

There are other third-party tools, such as this one from GPSVisualizer, that do show the warnings about precision from Yahoo or Google -- but only one address at a time. The online providers seem to tolerate this use of their geocoding services -- even though the third-party sites such as batchgeocode.com and gpsvisualiser.com may violate the terms of use of theiir APIs -- so long as batch geocoding suppresses the precision flags. Presumably that is to protect the intellectual property and markets of the original providers such as Tele Atlas, which market professional-level geocoding. Much of the pro stuff categorizes the precision of its output with even more granularity. But at a minimum, we need to see the precision categories reported by free online geocoders such as those at Yahoo or Google.

Considering that I have never heard about flagging precision warnings (I'm assuming you mean when it can't find the street or the address on that street) I bow to your superior knowledge on the subject.

I'll think about ways to resolve this, but I think you're right. When the church offers a Google Map mashup for clerk use that can be edited and corrected is when we'll have a good answer for this.

davebailey wrote:I'll think about ways to resolve this, but I think you're right. When the church offers a Google Map mashup for clerk use that can be edited and corrected is when we'll have a good answer for this.

I have high hopes that the Church will develop a robust geocoding tool that will engage clerks to fix the inevitable errors. Since most of those errors are ultimately caused by dirty data in the MLS address fields, I hope that tool also includes help with validation and standardization of those addresses and capturing the corrected addresses in MLS. It is a classic GIGO (garbage in, garbage out) problem.

As far as the precision flags go, there are many possible categories of granularity that different professional geocoding engines might use. (I assume that the Church is able to license such enterprise solutions.) But it seems that most applications, including this one, ultimately have to employ those categories to define breakpoints for three major quality groupings:

1. Geocoded addresses that are automatically accepted because they meet a certain standard. In the case of Yahoo's output, this would be data that was classified as geocoded at the "address" level without warnings. (Actually, I think the Church likely will even adopt a higher standard, with the objective of capturing so-called "rooftop" or parcel coordinates without the range-interpolation errors present in Yahoo and similar tools. That might entail a combination of premium commercial geocoding and manual correction by clerks.)
2. Geocoded addresses that are automatically rejected because they flunk the quality standard, or addresses that could not be geocoded at all. That would include addresses where the street number is invalid, or the street could not be found at all, etc. What Yahoo typically does in such cases is fall back to centroid coordinates of a whole street, Zip code or even city -- which typically will be far away from the real location -- and flag that in a field that describes the precision.
3. Geocoded addresses that are ambiguous, and need human verification. This would include addresses where the geocoder cannot find a sure match for the street address, but finds something that is close in spelling, etc.. Or the geocoder might not find "Montana Street" but substitutes "Montana Ave," or assumes "123 Elm St" when the input data is just "123 Elm." Geocoding software is usually right about such guesses, but sometimes it is wrong. Good software will provide the warnings along with the corrected address, so the user can verify it or override it manually.

In both of the latter two cases, clerks need the ability to fix underlying address problems and geocode again, or manually supply coordinates by some other means. Those means might include a visual tool such as maps.lds.org now employs to correct meetinghouse locations. They should also provide latitude and longitude fields that could be manually entered into a screen form. Some small percentage of addresses simply will not geocode automatically, and that percentage grows to be large in rural areas. In developing countries, manual geocoding may be the only way to capture the data.

boomerbubba wrote:three major quality groupings

Wouldn't there be another where the address is accepted without problems, but the software isn't sure exactly where 123 is, but is plotting a position between the 100 and 200 block?

RussellHltn wrote:Wouldn't there be another where the address is accepted without problems, but the software isn't sure exactly where 123 is, but is plotting a position between the 100 and 200 block?

That is actually what Yahoo's geocoding engine does as its best case, in my experience. Those records geocoded at its "address" level are just interpolated within such a range, and Yahoo's API has no higher category of precision. Until recently that has been state-of-the-art in the geocoding industry.

But higher-end providers now are folding in parcel-level data gleaned from local property tax records. Somehwat misleadingly, this is often called "rooftop level" geocoding, although "parcel level" would be a more accurate label. Even within the United States, in high-end geocoding services, the coverage of this level of precision is far from complete. However, it is increasingly possible to achieve this precision by filling in the gaps by manual intervention.

I rather think the Church will be aiming for this level of precision somehow. Not only is it more convenient for general purposes such as viewing my home-teaching route, but precise geocoding is very important for emergency-response purposes. Since emergency preparedness is one of the many uses for geocoding in the Church, and we do possess an unparalleled reservoir of free labor, I would expect the Church to aim for that high standard in capturing the data.

After much trial and error, I finally succeeded in uploading our membership onto Google Earth. I am still working out all the possibilities but have one problem. When I first managed to load it, I was able to also see the locations on Google Maps. But now when I go to "Show in Google Maps" i get only a map of the area with none of the locations noted. Google Earth is wonderful, but I can't figure out how to show the street names that you get with Maps. I am very new at this type of thing, so I have probably missed something very simple...

zaneclark wrote:After much trial and error, I finally succeeded in uploading our membership onto Google Earth. I am still working out all the possibilities but have one problem. When I first managed to load it, I was able to also see the locations on Google Maps. But now when I go to "Show in Google Maps" i get only a map of the area with none of the locations noted. Google Earth is wonderful, but I can't figure out how to show the street names that you get with Maps. I am very new at this type of thing, so I have probably missed something very simple...

This is a Google shortcoming. The "View in Google Maps" function in Google Earth is a misleading label. The function does not actually export to the Google Maps web site the placemarks and data you created and displayed in your local Google Earth application. It just opens the web site and moves the standard map to the same location.

(That's a good thing for us, since uploading both names and addresses to the Google Maps website would probably put you in violation of Church policy guidance. It may be possible to upload the same KML file to Google Maps by other means, but I do not recommend it for that reason. In our ward, when we upload addresses to Google Maps to get a better printable rendering, we omit the names. And we do not rely on the batchgeocode.com geocoding.)

To see streets and street names within Google Earth, in the "Layers" sidebar on the left of the screen, make sure the item labeled "Roads" is checked. Then when you zoom in, the roads and their names should become visible.

OK, I was able to display the streets finally but now I have another problem. I want to share this with some of the ward leaders but when I tested this by using the email icon on the Google Earth page, it will not open in the receiving computer. I know they have to have GE downloaded so I used my daughter as a test run and made sure she had GE downloaded but the data on our ward locations will still not show. There are 3 choices for emailing, graphic, view, or image. I have tried all 3 so which is the one that should be working?

zaneclark wrote:OK, I was able to display the streets finally but now I have another problem. I want to share this with some of the ward leaders but when I tested this by using the email icon on the Google Earth page, it will not open in the receiving computer. I know they have to have GE downloaded so I used my daughter as a test run and made sure she had GE downloaded but the data on our ward locations will still not show. There are 3 choices for emailing, graphic, view, or image. I have tried all 3 so which is the one that should be working?

I have never used the email icon from inside Google Earth. If you want to share the KML or KMZ file, do it outside of Google Earth. Just attach the file to your email like any other attachment. That's what we do in our ward. (Our KML files are produced by a completely different and more reliable method than the one described above, but that should make no difference as far as this particular problem with Google Earth goes. If a file will open in your copy of Google Earth, it should also open in someone else's copy of Google Earth. Of course, everyone should make sure they have the current version installed.)

Thank you David! Using your very clear instructions, I have created a geo map for our ward. I am still trying to figure all the uses for this, other than to amaze and astound some of the leaders. It is a constant fight to keep leaders interested in the possibilities of the ward website and although this isn't directly related to the site, at least they know the web administrator is doing something!

The best way that I have found to share it is just as Boomerbubba says, attach the KML or KMZ file to an email or transmit the file via a USB drive.

As far as how to stoke the imagination for the ward, get a slice of a PEC meeting when you either have your local area cached on your laptop, or better yet do the meeting where you can get Internet access and open the ward map file in Google Earth.

Explain how the special information can be used to make home teaching or visiting teaching routes better, but more importantly, it's great for doing disaster planning and things like setting up block captains. Usually, I find the Elders Quorum President or ward emergency specialist, if not the bishop, picks up on the value of this information.

http://www.mormongps.com

I created a program that takes the members from the users ward website and creates a CSV or TomTom specific file. You can then use that file on http://garmin.gps-data-team.com/poi_manager.php and convert it to a KML, etc file if needed.

I need to put together a bunch of maps with members houses marked on them for our fast offering routes. I've looked into using Google Earth for this because I'd be able to automate that process and update it regularly. However, the images you get from Google Earth are terrible for printing maps. Is there anyway to get a Google Maps type view from Google Earth? Is there some better way of doing this I should consider?

JPark wrote:I need to put together a bunch of maps with members houses marked on them for our fast offering routes. I've looked into using Google Earth for this because I'd be able to automate that process and update it regularly. However, the images you get from Google Earth are terrible for printing maps. Is there anyway to get a Google Maps type view from Google Earth? Is there some better way of doing this I should consider?

The short answer about Google Earth is no. It is a great medium for on-screen browsing, but a lousy medium for printing.

After much effort and research, in our ward we ended up usings the My Maps feature of the Google Maps web site instead, along with a lot of custom scripting. The principal reason for this is to avoid violating church guidelines, which prohibit uploading names and addresses exported from MLS to such a third-part web site. (We store that data locally and print it via the scripts.) Also, we have our own local solution to the basic problem of geocoding all the addresses.

Our solution is described in this forum post. (A footnote: The linked post says we needed special permission from Google to do this, which was technically true at the time but not longer is.)

I had a go at making a mapping tool a year or two ago as well (html&javascript based, using google for geocoding, but output to kml, so still have to use google earth.)

But I'd now recommend this newer Adobe Air based app:

http://hpaulsen.com/hpaulsen.com/WardMap.html

The printing side of things still isn't great (though better than google earth), but its got a lot of features that make it the best for on-screen use at least.

I've been working on a solution for this, also. It's still rough, but I think it shows promise. It will print out maps, though I'm looking at improving the output. It will also allow you to quickly change from viewing everybody to viewing only Elders, Young Men, Young Women, recent converts, etc. It also includes the ability to draw geographical regions (corresponding to GEO codes) and find out who lives in each region (once the GEO codes in MLS are updated to match).

http://hpaulsen.com/hpaulsen.com/WardMap.html

-Hyrum

HPaulsen wrote:I've been working on a solution for this, also. It's still rough, but I think it shows promise. It will print out maps, though I'm looking at improving the output. It will also allow you to quickly change from viewing everybody to viewing only Elders, Young Men, Young Women, recent converts, etc. It also includes the ability to draw geographical regions (corresponding to GEO codes) and find out who lives in each region (once the GEO codes in MLS are updated to match).

http://hpaulsen.com/hpaulsen.com/WardMap.html

-Hyrum

Although I haven't tried your program yet, I have browsed your web site and read through the description. In that outline you mention that "much of the sensitive" information is removed. Can you be more specific? I assume you mean Birth, and ordinance information. Since the software is creating a map you would still need to maintain name, address, city, etc. as well as a priesthood office or class to be able to sort by Elders, YM, YW, etc.

The only fields that I don't use right now from Membership.csv are "MRN", "Endowed", "Rec Exp", "Mission", "Spouse Member", "Sealed to Spouse", and "Sealed to Prior" - basically, the temple and recommend information.

The age is stored as a floating point value, so the birthdate could conceivably be extracted (though it would take a lot of work to do so). I originally stored the age as only an integer, but doing so was inadequate for the nursery/primary divide.

I have considered adding a filter, "Likely Active", which would include people who:

• Have a current recommend
• Have had an ordinance recorded within the last two years (excluding baptism but including confirmation)
• Have recently returned from a mission
• Have a family member who matches any of the above

I can see this being a big help to (e.g.) newly called EQ presidents, who need to create home teaching routes but don't know who they can call. Unfortunately, there are risks associated with this if people take it too seriously. It would be far from infallible - just a nice starting point. Anyway, I've tested it and like the results, but I'm not sure I'm comfortable releasing it....

HPaulsen wrote:





I have considered adding a filter, "Likely Active", which would include people who:

• Have a current recommend
• Have had an ordinance recorded within the last two years (excluding baptism but including confirmation)
• Have recently returned from a mission
• Have a family member who matches any of the above

In our ward, for leadership use, we sometimes find it useful to flag as a proxy for "active" any family with a member who has a calling or serves as a home- or visiting-teacher. (The obvious caveats apply about innacuracies, assumptions and timeliness.) I have made maps that color-code families that way for use by the leadership and FT missionaries. To avoid offending anyone, we are careful not to label the codes as "active" or not.

boomerbubba wrote:In our ward, for leadership use, we sometimes find it useful to flag as a proxy for "active" any family with a member who has a calling or serves as a home- or visiting-teacher. (The obvious caveats apply about innacuracies, assumptions and timeliness.) I have made maps that color-code families that way for use by the leadership and FT missionaries. To avoid offending anyone, we are careful not to label the codes as "active" or not.

Oops - yes, home teachers and people with callings should also be "active". If I were to add this filter, is there a better label than "likely active" that would still be clear as to what it means but which would be less likely to cause offense?

I originally had three activity categories: likely active, unknown, and likely inactive (for people who were behind in ordinances or with family behind in ordinances). I coded them with suns (likely active), moons (unknown), and stars (likely inactive). Families with both likely active and likely inactive members were going to be coded differently, but I didn't get that far. In my excitement, I showed my wife, and she (rightly) vetoed it. I guess I got a bit carried away....

I still think it's useful information, but even being conscious of the dangers of stereotyping, I fell into the same trap.

HPaulsen wrote:If I were to add this filter, is there a better label than "likely active" that would still be clear as to what it means but which would be less likely to cause offense?

In applications for which I have used my indicator, I have not been so constrained for space that I needed to use a terse label. I deliberately avoided that. On some limited-distribution maps I just use color codes, and a verbose description or legend that says something like "Serves as HT/VT or in a calling." That is factual and non-judgmental. I just explain to the users verbally that it is a rough indicator of activity.

Even though my applications have been primarily for leadership use -- as Google Earth files, printed maps or just spreadsheets -- I did not want to publish any label that would offend anyone if they happened to see it applied to themselves (or not). For example, I make large-scale wall maps of all ward families for our FT missionaries' apartment, where it might be seen by some visitors.

Also, the composition of this indicator uses no data that is regarded as highly confidential, such as temple recommend or ordinance data. Including such elements would mean the file should not be shared outside the bishopric and perhaps other key leaders. (For that matter, sharing any map files derived from MLS data should be done according to the bishop's direction.)

Looking if someone knows of a local desktop process or application that will change a LDSMaps download file (.csv) to a format that can be used for the following devices:

A) TomTom: convert from LDSMaps .csv ward download file to a format for TomTom XL device

B) Garmin: convert from LDSMaps .csv ward download file to a format for Garmin Nuvi device

Notes:
1) The LDSMaps download file (.csv) has Longitude, Latitude and if the address has been verified http://beta-maps.lds.org
2) User trhodes provides an application, but it uses the Local Unit Web Site (LUWS) membership for input and then it creates files for GPS devices
http://tech.lds.org/forum/showthread.ph ... ght=tomtom
http://www.mormongps.com

I looked into this when I first got my Garmin for Christmas.
From what I recall, it would be straightforward to edit a .csv file in Excel and then run the utility that can be downloaded from the Garmin website. The Excel process should be able to be automated with a macro.

I planned to create the Excel file to do this, but since our family will soon be moving out of the stake, I just haven't yet taken the time.

pete_arnett wrote:Looking if someone knows of a local desktop process or application that will change a LDSMaps download file (.csv) to a format that can be used for the following devices:

A) TomTom: convert from LDSMaps .csv ward download file to a format for TomTom XL device

I did this exactly with my Tom Tom... unfortunately I don't remember exactly how, but let me look it up, re-teach myself, then post it.

For now, if you want to research yourself, you can google search "Tom Tom create custom POI list"

(Essentially, you create a custom Point of Interest (POI) list from the coordinates downloaded from LDSMaps download... I'll post more when I find all the instructions).

Thanks!

pete_arnett wrote:Looking if someone knows of a local desktop process or application that will change a LDSMaps download file (.csv) to a format that can be used for the following devices:

A) TomTom: convert from LDSMaps .csv ward download file to a format for TomTom XL device

B) Garmin: convert from LDSMaps .csv ward download file to a format for Garmin Nuvi device

Okay, I found it:

For TomTom, you have two options:

Option 1: Follow instructions in TomTom documentation for create your own content - (click Points of Interest > Web Developers Guide, follow instructions on the web developers guide page... in fact, there is a lot of good information in there )

Option 2: Install POIEdit

For Garmin, it looks like you can just install POIEdit and export to the right format (as long as it is a Garmin Nuvi or en zumo, see supported devices )