SuperPAF/HyperPAF Idea

SuperPAF/HyperPAF Idea

I attended the recent TechTalk in Salt Lake City (Jan. 18) and found it very helpful. I want to take this opportunity to put in my two cents worth of opinion. I am happy to see the Church starting a new set of genealogy data projects.

A new worldwide missionary/genealogy system is needed
My main personal interest is in seeing Church genealogy systems include a major strategic element that is aimed at serving the entire world and drawing hundreds of millions of people into using it. Our existing database systems, containing many Americans and Europeans who made up so much of our early membership, could be viewed as an extended “beta test” we are still working on before rolling it out to the world.

The overriding goal of the coming worldwide system should be to entice billions of people who have never heard of the Church to ask themselves why the Church even cares about genealogies, even as those people are actively and happily using the system to enter their own genealogy for preservation purposes. Their questions will eventually lead them to learn about our Plan of Salvation. Every person on the planet cares about their ancestors, and many include that concern in their religions.

In summary, I believe we need a truly generalized worldwide “genealogy research results storage” system that encourages and supports extensive cooperation, something that is quite a bit more than the specialized temple work system we have now. It is possible now to go far beyond the Ancestral File, Pedigree Resource File, and similar commercial efforts of an earlier day.

This system is intended to sit on top of all the large “raw genealogy data” databases in the world, acting as the lineage-linked summary and index to it all.

The elusive “One World Family Tree”
As with our past systems, it should do at least as much missionary work as genealogy work, but on a much bigger scale.

Success with this grand idea has eluded many a serious attempt in the past, but it is now perfectly feasible to do a really fine job of it, if the genealogy community is willing to study and accept a slightly different paradigm, a new way of thinking about the old goals and methods. This year seems like the perfect time to explore it, with all the new resources going into genealogy systems. I could describe the suggested system to any desired level of detail, but this is probably not the right time and place to try to do that. (I do have a running prototype at http://www.genreg.com – which needs more introductory verbiage and pictures. There is lots of theory (ten years worth) under the hood that is hard to spell out quickly).

At the risk of causing instant misunderstanding, skepticism, and controversy, I will briefly describe three key features. The most notable internal difference comes in storing the “finished” data in descendant format because that discipline almost automatically eliminates 95% of the duplication problems and questions we see today. The most notable external feature is that people could, theoretically, dispense with all their PC-based software and enter and modify all their data online, having near-instant access to everyone else’s best work as well, making ease of cooperation a major benefit. There are three software layers, 1) one that allows an individual to enter and modify his or her own data, as with today’s PAF, 2) a higher level “SuperPAF” that allows many members of an extended family (or workgroup) to enter updates simultaneously (with appropriate controls), and 3) the highest level “HyperPAF” which allows anyone in the world to view lineage-linked data that has been designated as “public,” and to enter competing, but nondestructive overlays of data for particular people or families. A third crucial feature provides for the explicit linking of name records to source records and images or other documents or multimedia objects, by actual URL or by more generic library call number references. No longer need these important links be buried in someone’s obscure notes, but can become part of the active database. Perfecting this new system using the American/European data “beta test” should allow it to be rolled out to the world with high credibility for all to use.

(Notes: 1. Purely practical and performance reasons would cause PAF-like software to be used for a long time, even if only as an assembler of data segments to be added to the larger collection. 2. This system could be viewed as a very specialized “wiki” linked to other less specialized wikis.)

This relatively inexpensive system (low storage requirements) would allow a person newly interested in genealogy research, anywhere in the world, Church member or not, to find out in a few minutes all the genealogy research that had been done in the world that is linked to him or her (or a near relative). (Their looking has nothing to do with temple work at this point). They could then quickly determine what new non-duplicative genealogy research they might profitably contribute to the whole.

This new system would be powerful for LDS use as well, because long before any questions about submitting temple work came up, all relevant completed genealogies (and related history, stories, photos, etc.) could be explored. This would almost completely avoid the current problem where people, especially new members, may work for years to gather family names for temple work, only to find at the end that 98% have had their work done multiple times before, probably because those names came out of Church systems in the first place. That is quite shocking and discouraging to many neophyte genealogists.

Incidental operational benefits
Strangely enough, implementing the new system, where “finished” data is stored, should eventually cut down drastically on the high volume of FamilySearch hits and other expected future open-ended searches of a similar nature. Besides its social and missionary contributions, it should more than pay for itself by perhaps cutting in half the number of servers the Church would otherwise need to deploy to support its genealogy systems.

Having the new system in place should offer a huge incentive for people to participate in the Online Indexing (microfilm data transcription) process related to the Granite Mountain Vault project, especially if people are able to choose the microfilm they wish to transcribe, film that has a high likelihood of relating to their family. Having a reliable central place to store some of the results of their transcription, especially the parts relating to their family name identification and lineage-linking, should give them a real sense of accomplishment, a job done well, once and for all.

In fact, if the database system I propose were done early, and the vault images were gradually put online (or copies of those same images offered by other online data suppliers were used), it would be theoretically possible to skip much of the (centrally driven) Online Indexing process as now envisioned for those vault images. People using the online images (or even the traditional microfilm copies), and linking them to their family members in the new database, using actual record keys or library-style references, would in the process be gradually creating an index to those images as a byproduct of their efforts. If desired, at some point a final visual comparison of the images and the partially completed index could finish off the indexing/transcription process for the most often-used segments of the image database. (The links of names to source images could be turned around, inverted. This view would begin with source images and show the names linked-to from each image. The results could be completed and corrected, if necessary, much like the current plan for Online Indexing)


Kent Huff

----------------------------------------
801-798-8441, cell 801-787-3729
1748 West 900 South
Spanish Fork, Utah 84660
huffkw@juno.com, http://www.GenReg.com
computer consultant, attorney, author
================================================

Less-abstract genealogy idea

I see that several have clicked on my post about SuperPAF and HyperPAF, but no one has ventured a comment so far. I will assume for the moment that what I said was too abstract, or otherwise missed the mark, and try a different approach.

I will try to put the idea a little more concretely by expressing it in terms of changes to what I think is already on the drawing boards.
------------------------------------------
My suggestions for the next 4 upgrades to planned genealogy systems, as I understand those systems.

1. Add an online (originally empty) PAF-like database structure, separate from the New Family Search database(s), plus appropriate update and report screens or pages.

--In general, this feature should avoid many of the difficulties and limitations of multiple kinds of updates made against the New Family Search (NFS) databases(s) in place. For example, a user could choose one version of a name from NFS, move a copy of it to the PAF-like online database, and forget about any other duplicates that might exist. S(he) would be making no changes to the main databases. Any enhancements made to that name – relationship links, source document links, more complete data, etc., would be made in this separate, personal online space, for which only that person is responsible.

--It appears that updates to the NFS multiple databases may come from many different, uncoordinated sources, some inside the Church and some outside the Church, many potentially changing the structure of the data itself, changing the number of occurrences of certain names, etc. The suggested change would leave only Church-initiated updates to the NFS data, and should greatly lower the update programming logic complexity levels and chances for strange errors to occur. Avoiding data corruption in the big databases seems like a potential worry that would be removed by the suggested separation between source and target databases. Otherwise, the Church system may have to keep track of many versions of the databases, instead of just letting the users do all that in their new private space. The plan as I have heard it, essentially has source and target databases overlying each other. That seems unsustainable in the long run.

--Users can request private storage space, with password protection. Let a user put any genealogy-related data (s)he wants there.

--Anyone in the world (including “future members”) could use this space to store and update their genealogy data for free. This service would invite the world to learn something about the Church and it web offerings. These people could find this database useful, even if they have no interest at all in our temple doctrines or practices, and even if they can expect to find no data of interest to them from our existing databases.

--This private space could replace some or all of an individual’s usage of PC-based PAF-like programs.

--Data might be kept in the private space indefinitely, or compilations of data assembled there might be downloaded to a PC-based database.

--Complex Access Control specifications would not need to be applied to the private storage space, only to the NFS databases.


2. Add the ability to enter live links from private space records to specific NFS or Granite Mountain Vault (GMV) source record images.

3. Add features that would allow multiple family members to update the private space simultaneously.

4. Add features that would give the ability to anyone in the world to read data for deceased individuals in any of the private areas marked as “public access allowed.”

There are several more layers of upgrades I might suggest beyond these four, but these are plenty for the moment.

Since you are looking for some feedback to your ideas, I'll try to respond to get the ball rolling...

huffkw wrote:Less-abstract genealogy idea
1. Add an online (originally empty) PAF-like database structure, separate from the New Family Search database(s), plus appropriate update and report screens or pages.

The value that I see the new FamilySearch offering is to jump start your pedigree so that you don't have to start with an empty database structure. Even though this data probably needs to be tweaked to be sure that it has the correct information and duplicates are handled correctly, this seems like a step in the right direction for many inexperienced genealogists who don't want to re-enter genealogical data which they know is already available.

huffkw wrote: --Users can request private storage space, with password protection. Let a user put any genealogy-related data (s)he wants there.

There are already online services that offer this service very well (e.g., rootsweb), but for which the gedcom files are completely independent of each other. Is your suggestion different from this?

huffkw wrote: 2. Add the ability to enter live links from private space records to specific NFS or Granite Mountain Vault (GMV) source record images.

Yes, to make the system extensible, it needs to be able to point to other sources of data, including multimedia files, online documents, or references on other sites. But in doing so, this opens the issues of how to manage these external sources when they become unavailable either temporarily or permanently.

huffkw wrote: 3. Add features that would allow multiple family members to update the private space simultaneously.
4. Add features that would give the ability to anyone in the world to read data for deceased individuals in any of the private areas marked as “public access allowed.”

The new FamilySearch will do this, if I recall the LDS Tech Talk presentation correctly. A few other current open source apps also do this (e.g., http://www.phpgedview.net/)

I believe what you are saying is that a bunch of seperate databases should be created for each individual user and that users can import into their databases information from the New Family Search (NSF). And that they can choose to import the NSF data into their own. They can also chose to upload their info to the NSF.

Well first off everyone can do that already, (except uploading to FS), with 3rd party. They only thing that you changed is that the church will host everyone's personal database, instead of having the database on their computer. From what is public and searchable, with the new system they just have to upload it, while all their information that is private will stay with them.

I might of misunderstood and if so I'd like some clarification, thanks.

I see two things already that I had heard something about.

1. I had heard somewhere that they were going to have a way to point to an image that you find an ancestor in from the indexing that is going on now. So if you find an ancestor, you can find the image, and create a link in your tree to that image so that others can see what you have. Never heard anything more about that.

2. One of the reasons for on-site storage vs. home PC is that in some parts of the world, Internet cafes and similar sites are where people go to connect to the net. I'm not sure if thumb-drives will be the solution for carrying around large amounts of genealogical data, but I hear that that is the way many do it, they use the PCs USB port, work with their things, then take the drive with the updated material with them. It could work that way but how woulod such a scenario play out with genealogical data as right now most of these USB thumb drives only have around 5 megs although I could be wrong about larger ones.

JamesAnderson wrote:I see two things already that I had heard something about.

1. I had heard somewhere that they were going to have a way to point to an image that you find an ancestor in from the indexing that is going on now. So if you find an ancestor, you can find the image, and create a link in your tree to that image so that others can see what you have. Never heard anything more about that.

Understandable, luckily we have some others that have posted their notes, but in summary there will be a simple PAF like program which users can run and that will interface in the the NFS. This program most likely will be based on Java so it can go cross platform, (Maybe even have an applet form for those that can only use the internet) Although I think you have to place the picture on the internet before you can link to it. I might be wrong though.

JamesAnderson wrote: 2. One of the reasons for on-site storage vs. home PC is that in some parts of the world, Internet cafes and similar sites are where people go to connect to the net. I'm not sure if thumb-drives will be the solution for carrying around large amounts of genealogical data, but I hear that that is the way many do it, they use the PCs USB port, work with their things, then take the drive with the updated material with them. It could work that way but how woulod such a scenario play out with genealogical data as right now most of these USB thumb drives only have around 5 megs although I could be wrong about larger ones.

The only problem is storage. The church is already using pentabytes of storage for FS and if everyone else stored a seperate database that copies the FS system I can imagine that we couldn't get enough storage. Small USB drives (64 MB) are not that costly now in the USA (I don't know about other countries but I supose that the prices for a small capacity USB drive are about the same as a floppy disk) And can easily hold the information needed. My 8 full generation and 23 extended generation database fits under 20 MB with some images. The only problem is when you get a lot of media.

Thanks for jumping in, everybody. You already know I have an idea of what I would like to see the Church genealogy systems evolve into.
But getting your reactions, and trying to respond to them, can tell me if I am all wet, or if it really could be done as I hope, and whether other people would agree and value it.

I will jump around a bit:
-----------------------------------------------
thedqs: “the only problem is storage”

As I estimate it, the Church only has about 40 to 50 million unique names in all its active genealogy and temple ordinance files at this point. All the rest are duplicates of one kind or another, even if that totals 1 billion records. (At this point, it would be hard to get the real number of unique names).

Allowing 2,000 characters storage for each of 50 million names only comes to 100 GB total. Pretty small. All the 300 million deceased Americans would only take up about 600GB, still a small amount. Maybe spend $500 for a 1Terabyte drive.

Putting the whole “bit mountain” on disk in image and text format will truly take petabytes, with megabytes per image, but I am assuming that the file I am suggesting will have no more than short URLs or library-style general references to these huge image files, so it will not expand much.

A little extra overhead from some minimal duplication will also not hurt much.
-------------------------------------------
JamesAnderson: “Internet cafes . . . thumb drives”

I would like to see it be perfectly easy for people all over the world to go to a genealogy center, an institute building, a chapel, or a friend’s home and do this work. No need to carry anything around at all except a logon. No expensive hardware to break, lose, or have stolen. This matters a lot in most Third World countries.
--------------------------------------------
thedqs: “problem . . a lot of media.”

I assume there will not be extra media added to the file I suggest, but one could put photos on a wiki somewhere and store a URL to it in the central database.
----------------------------------
JamesAnderson: “point [a name] to an image”
I hope that is what will become of the whole “bit mountain” project. Most of the project’s value would be lost it that were not true.
---------------------------------
thedqs: “a bunch of separate databases . . . everyone can already do that . .. . 3rd party . . . upload public data.”

My hope is that by doing it the way I suggest, the central database will contain almost no duplicates, making it a real treasure. As it is, we have on our PCs hundreds and thousands of copies of the same names. Putting them up for public use in that form is only helpful to the most patient among us who will sift through it all for that little bit of gold dust. (I could suggest a new utility to help with avoiding duplication, but it could be done efficiently enough manually once there is a decision to put it on the central database.)
--------------------------------------
haledn: “(Originally empty) . . . .jump start your pedigree”

My assumption is that some group like the Hale family could add their large GEDCOM to the central file, and in the process, any names that were among the descendents of the original Hale ancestor would be numbered in a way to keep them differentiated from the rest, and that the deceased among that number could be made publicly viewable.

“already online services that offer this service”

It is not just the free space and indexing features I am interested in.
I am looking for a few important new features here that I don’t believe exist elsewhere.
1. The owner/user could update their GEDCOM data in place, not merely reload it periodically, and could add live links from names to other Church records such as images from the vault. References to source records move out of notes and into data elements/rows.
And the GEDCOM data would have been renumbered to allow for separate identification of the most valuable descendent-form data, all those with the Hale surname, thus avoiding much of the confusion caused by seeing data on hundreds of other offshoot surnames.

“manage these external sources when they become unavailable”

The work of creating a near-duplicate-free central database pays off handsomely in many ways, but one of them is the ability to create a unique number for each name in the central database. Once that is done, any outside database can then use that number to link its additional data, of any type, to the name in the central database – obituaries, photos, stories, source images, etc. This means that there really are no “broken links” if things move around. The central database need not even keep track of these outside databases, except perhaps to list their main domain name so that other programs can do the “distributed database” processing needed to marry up the all the data found in various places on one person.

---Thanks again for your thoughts. I hope these comments help.

===============================================

Thank you for clarifying a bit and I think that this is the general direction of the NFS. Though I think it goes more along the World Family Tree project where everyone uploads their pedegree and then can do work on it colaberatively, (although I believe living people would remain hidden except to their close relatives). Though how this would avoid duplicates any better then what the system tries to do now I need a little more explination. From what I understand the new NFS would show you any close matches and you get to decide if it is a duplicate or not, which from what I gather is the same basis of your suggestion.

Thanks for hanging in there.

I hate to be a negative spoil sport, and maybe we will have to wait until this summer to see what it is really all about, but I think people are expecting too many new features from what is planned for NFS. I hope I am wrong.

My mental picture so far is that nothing conceptually will change except that several more databases can be searched in parallel (yielding more hits), and the cycle of adding new ordinances to the IGI or membership records will be accelerated, so that new temple work or living member ordinances will be recorded every week instead of every 2 or 3 years as it was in the days of the Ancestral File. The more prompt updating is good, and this will help avoid some ordinance duplication for a few really intense genealogists that check names every week, but will not be a big factor for the many genealogists who only use the Temple Ready check as the last step after a large amount of research effort. We can assume, almost by definition, that if a name is in the NFS, their temple work has been done, so we are not likely to be covering much new ground there. There might be a lot of new ground in the “bit mountain” project.

I was told the IGI part of the NFS is 80% made up of single names receiving single ordinances, as it has always been. There may be huge numbers of duplicate records (for example 200) for some ancestors. There will be a few records with two names linked as in marriages or three names linked as in sealing of children to parents, as has always been. In other words, there is not much pedigree data here. Too bad the original family group sheets, from which some of this data came, which linked three generations, were not kept intact and linked together, but that is where we are, with only fragments in the IGI. The Pedigree Resource File, with its many GEDCOMs will be included in the search, and many duplicates will be found there, as has always been. In other words, very little will change from the current situation as far as solving the duplication problem.

The new twist
So here is where I would introduce a slightly radical idea to knock out most duplication in one swoop and hopefully end it once for all in the new database I am suggesting. Here is a simplified view: when a GEDCOM goes in, it is run through a utility program (I have tested the code) and only the largest single-surname descendent structure in that whole GEDCOM is kept. The rest disappears for public purposes. If I have 100,000 names, but only 3,000 are in a single-surname descent structure (like the Huffs starting in 1630 with Engelbert, or the Hales or the Leavitts) only the 3,000 is kept. Whole GEDCOMs will be dropped from public view if no descendant groups reach 1,000 in size. The idea is that the Huffs might be expected to do a good job on the Huff-surname descendants of a Huff, but shouldn’t be trusted or assigned to do any other surnames. If the Huffs married into the Thomases, let the Thomases do the Thomases, and no one else. Instantly, you have no duplicates unless there happen to be two large single-surname descendant structures with the same surname that made it through the utility. That is unlikely, but if it happens, then just keep the largest and drop the other. No dups. Then the data owners can continue to add to that one chosen structure, hopefully teaming up with all others interested in that descendant structure. The non-dup database will need links from wives in one structure to their place as a daughter in their maiden name single-surname descendant structure. Then the job is finished of putting links between structures.

Is anyone ready to do something radical to kill dups worldwide? I think many are.

I am guessing the final result of processing all the GEDCOMs from every source should yield something in the range of 20-40 million non-duplicated names, but there is no way to know until we do it. Then the fragments of families skipped over by the utility can be found and added in, with new features aiding that process.

Expanding on some points
After some thought, I would like to expand on a few points brought up earlier.
-------------------------------------------
thedqs: “the only problem is storage”

Actually, I fear we could be missing a great promotional opportunity here.

Note that if each user were to put in 5,000 names and each took 2,000 bytes of space, or 10 megabytes in all, that would cost only about one cent apiece for storage space. (At $1 per gigabyte, 10 MB costs about one penny).

Perhaps we should be giving away free genealogy workspace like we give away free copies of the Bible and Book of Mormon. At a penny a person for genealogy data storage space, that is far cheaper than giving away books. Yahoo and Google give away much more (2GB+) for free (to get people to use their site and view their ads), so we ought to be able to do something similar. Some nice ads about Book of Mormon topics could get them reading the book online. And who is going to write “The Great (Early) American Novel” on a Book of Mormon topic, or write the blockbuster game on Nephite battle tactics? If Mel Gibson can make money on a Mayan movie theme, why can’t we do something like that? We could advertise many such things on this site (assuming someone does the creative work).

Some calculations: If 10 million people used the free site, that would be a one-time cost of $100,000 for storage. Spread over a 5-year hardware life, that is $20,000 a year. That equals about 4 full-time missionaries for a year. Seems like a good cost/benefit trade for drawing 10 million people into closer contact with the Church. Eventually supporting a staggering (upper limit of) 50 million engaged users worldwide would cost about $100,000 a year, the same as 20 missionaries, but the positive public relations impact would be far more than 5 times larger than the 10 million participants.
--------------------------------------
Database value
What is the social value to the world of one 40 million-name, dynamic, integrated, genealogy database free of duplicates, and designed to grow endlessly in size, richness, and provable accuracy, while always staying free of duplicates? That value seems pretty high to me. The world would really clamor to use it, and would want to add its data to such a treasure.

It would look like one big integrated database to the outsider, even though it is actually cooperatively maintained in many smaller pieces inside. (Assume only deceased names can be seen without special permission from data owners.)

There is a large chance that many people would get on board and gradually grow the database to include 600 million Americans, living and dead, and 600 Europeans, living and dead. Then the Church would have created a new wonder of the world. (And these millions of non-Church participants would be doing much of the Church members’ genealogy research for them for free, since we have many of the same relatives.)
If someone wants to present a name or a family to the world, inviting anyone who has data on that person or family to add it (in one of several ways) to what is known already, this is the place to put it.

In pure monetary terms that sophisticated database would be worth many billions of dollars, so we can be sure it would be noticed and used. The Church might even figure out a way to recoup its operating costs.
--------------------------------------
haledn: “how to manage external sources when they become unavailable”

Three kinds of links needed
The central database needs to support three kinds of links.
1. Library references that are not hotlinks, but still tell you where the documents can be found, and are in standardized electronic database form so they can be examined for such things as percentage of a public record set that has been used/referenced, etc.
2. Hotlinks to Church source records. Those records should not move around much, so there is little likelihood the links will become broken.
3. The unique person number assigned within the database will allow anyone, anywhere to tie data to that person, with no updates required to the central database.
-------------------------------------
A simple client-side program to integrate all data on a person
If there are multiple sites on the web that contain data about one person (obituaries, photos, stories, videos, source images, etc.), a fairly simple client-side program (to begin with) could take the current list of such registered add-on data sites and pull out and assemble whatever they have to offer. These peer-to-peer operations would keep most of this kind of high volume data assembly activity out of the central site.

Note that for some simple HTML web sites, by using the unique person number, the indexing to that site could be done by Google.
--------------------------------------
Private firm involvement
I just realized that I have not suggested how private firms might get involved in processes related to this new database. I will have to write up something on that.

Suggestions on private firm involvement in genealogy program development

Here is what I hope to see happen.

1. The Church sponsors the development and operation of this whole cooperative central “finished genealogy” database concept. That way, it can be sure the underlying basic software is appropriate and reliable. I see this database as the summary and index for the whole genealogy world, and therefore needs to be done well.

Trying to chop that single development project into internal Church-created pieces, and external private company-created pieces, while maintaining consistency of concept and quality of code, sounds like a project manager’s coordination nightmare to me. It would probably be better to have a nice conceptual split between the two groups.

2. With the basic database in operation, supplying unique person numbers, the main focus of new private company work would be outside that database, exploiting all the new possibilities for storing genealogy-related media on the web, and then finding, collecting, and displaying that data, publishing it in one-of-a-kind books, formatting and editing it for use on family websites and wikis, storing copies of the collected data on genealogists’ PCs, etc.

All this web-stored data could be tied together by the central database person number. That is the logical breakthrough that can gradually overcome the chaos of today’s genealogy web research and data storage. Having a reliable person number available is probably far more valuable to private companies profitability than helping develop new ways to access the Church’s ordinance files, membership files, etc.

We might view the new person number as a development like the GPS satellites that allowed a huge flurry of commercial and consumer gadgets to be developed and sold.

The private companies would need to create the peer-to-peer client-based programs (simple at first, and then increasing more complex) that would do the “loosely coupled distributed database” processing to help find and assemble media from anywhere on the web, usually tied to individuals identified in the central database. (PAF-like products might find it useful to add this new data element of “central database person number” along with its internal Record Id, Ancestral File ID number, etc., and add in some of the code to do the consolidating of web data.)

At a genealogy conference I attended about two years ago (was it at the BYU Family History Technology conference, noon session?) a librarian from the Midwest forcefully made the point that the world is producing mountains of new data in numerous new media formats, but there were few good places to store this data as it relates to genealogy and specific people. Since then we have seen YouTube.com, MySpace.com, photo storage sites, etc., but has anyone seen such data storage options for more serious genealogy research storage purposes? I don’t know of any. This sounds like a rich mine for private companies to dig in.

3. The traditional PAF-like programs would continue to play an important role for a long time, although that role might gradually change a bit. Right now, I believe nearly all the good data in the world is stored on someone’s PC-PAF-like data file. Normally, if it is in a central file somewhere, it has been degraded or trimmed in various ways on the way in. What is needed is a central file (system) that stores data of as high a quality or higher than found in PC holdings. That is what I am suggesting. But at least for Americans, the PC-held data will probably still be the trusted gold standard for a long time.

Although much new data will probably be keyed directly into the central database, especially links between families in separate submission spaces, large amounts of researched names will probably continue to be assembled and verified in PC-PAF-like programs and then uploaded and linked into the central file. But these uploads will likely be much smaller than the initial uploads. (Periodic downloads containing all updates, will be available for backups, for analysis, and for family purposes no one has thought of yet.)

Great ideas, I've had similar ideas for 15 years and the technology is now becoming available
that makes it possible to start the process of implementing this vision. Let me add some more
ideas to this.

You mention that it should be possible to link between source records and people's
genealogies using URL's of source records. I think it needs to be even more detailed
than that. First one should be able to make references to part of a page, essentially
drawing a rectangle around something in the source record and create a link record
from this rectangle. Even more it should be possible to also link this record to other
things such as transscribed records of the source. After some time there will be
many links, both links of high quality and links of low quality so here it should be
possible to use both manual and automatic reference techniques to measure the
quality of links.

The base of those ideas I have I wrote down in Ph.D thesis:
http://citeseer.ist.psu.edu/161921.html
I presented some of those ideas at the first workshop on Family History Technology at BYU,
2001:
http://www.fht.byu.edu/prev_workshops/w ... nstrom.pdf
And I made a presentation last year at Google where those ideas were part of a presentation
around a future open source storage system I am thinking of.
http://video.google.com/videoplay?docid ... 2299655271

Rgrds Mikael Ronstrom

huffkw wrote:SuperPAF/HyperPAF Idea

I attended the recent TechTalk in Salt Lake City (Jan. 18) and found it very helpful. I want to take this opportunity to put in my two cents worth of opinion. I am happy to see the Church starting a new set of genealogy data projects.

A new worldwide missionary/genealogy system is needed
My main personal interest is in seeing Church genealogy systems include a major strategic element that is aimed at serving the entire world and drawing hundreds of millions of people into using it. Our existing database systems, containing many Americans and Europeans who made up so much of our early membership, could be viewed as an extended “beta test” we are still working on before rolling it out to the world.

The overriding goal of the coming worldwide system should be to entice billions of people who have never heard of the Church to ask themselves why the Church even cares about genealogies, even as those people are actively and happily using the system to enter their own genealogy for preservation purposes. Their questions will eventually lead them to learn about our Plan of Salvation. Every person on the planet cares about their ancestors, and many include that concern in their religions.

In summary, I believe we need a truly generalized worldwide “genealogy research results storage” system that encourages and supports extensive cooperation, something that is quite a bit more than the specialized temple work system we have now. It is possible now to go far beyond the Ancestral File, Pedigree Resource File, and similar commercial efforts of an earlier day.

This system is intended to sit on top of all the large “raw genealogy data” databases in the world, acting as the lineage-linked summary and index to it all.

The elusive “One World Family Tree”
As with our past systems, it should do at least as much missionary work as genealogy work, but on a much bigger scale.

Success with this grand idea has eluded many a serious attempt in the past, but it is now perfectly feasible to do a really fine job of it, if the genealogy community is willing to study and accept a slightly different paradigm, a new way of thinking about the old goals and methods. This year seems like the perfect time to explore it, with all the new resources going into genealogy systems. I could describe the suggested system to any desired level of detail, but this is probably not the right time and place to try to do that. (I do have a running prototype at http://www.genreg.com – which needs more introductory verbiage and pictures. There is lots of theory (ten years worth) under the hood that is hard to spell out quickly).

At the risk of causing instant misunderstanding, skepticism, and controversy, I will briefly describe three key features. The most notable internal difference comes in storing the “finished” data in descendant format because that discipline almost automatically eliminates 95% of the duplication problems and questions we see today. The most notable external feature is that people could, theoretically, dispense with all their PC-based software and enter and modify all their data online, having near-instant access to everyone else’s best work as well, making ease of cooperation a major benefit. There are three software layers, 1) one that allows an individual to enter and modify his or her own data, as with today’s PAF, 2) a higher level “SuperPAF” that allows many members of an extended family (or workgroup) to enter updates simultaneously (with appropriate controls), and 3) the highest level “HyperPAF” which allows anyone in the world to view lineage-linked data that has been designated as “public,” and to enter competing, but nondestructive overlays of data for particular people or families. A third crucial feature provides for the explicit linking of name records to source records and images or other documents or multimedia objects, by actual URL or by more generic library call number references. No longer need these important links be buried in someone’s obscure notes, but can become part of the active database. Perfecting this new system using the American/European data “beta test” should allow it to be rolled out to the world with high credibility for all to use.

(Notes: 1. Purely practical and performance reasons would cause PAF-like software to be used for a long time, even if only as an assembler of data segments to be added to the larger collection. 2. This system could be viewed as a very specialized “wiki” linked to other less specialized wikis.)

This relatively inexpensive system (low storage requirements) would allow a person newly interested in genealogy research, anywhere in the world, Church member or not, to find out in a few minutes all the genealogy research that had been done in the world that is linked to him or her (or a near relative). (Their looking has nothing to do with temple work at this point). They could then quickly determine what new non-duplicative genealogy research they might profitably contribute to the whole.

This new system would be powerful for LDS use as well, because long before any questions about submitting temple work came up, all relevant completed genealogies (and related history, stories, photos, etc.) could be explored. This would almost completely avoid the current problem where people, especially new members, may work for years to gather family names for temple work, only to find at the end that 98% have had their work done multiple times before, probably because those names came out of Church systems in the first place. That is quite shocking and discouraging to many neophyte genealogists.

Incidental operational benefits
Strangely enough, implementing the new system, where “finished” data is stored, should eventually cut down drastically on the high volume of FamilySearch hits and other expected future open-ended searches of a similar nature. Besides its social and missionary contributions, it should more than pay for itself by perhaps cutting in half the number of servers the Church would otherwise need to deploy to support its genealogy systems.

Having the new system in place should offer a huge incentive for people to participate in the Online Indexing (microfilm data transcription) process related to the Granite Mountain Vault project, especially if people are able to choose the microfilm they wish to transcribe, film that has a high likelihood of relating to their family. Having a reliable central place to store some of the results of their transcription, especially the parts relating to their family name identification and lineage-linking, should give them a real sense of accomplishment, a job done well, once and for all.

In fact, if the database system I propose were done early, and the vault images were gradually put online (or copies of those same images offered by other online data suppliers were used), it would be theoretically possible to skip much of the (centrally driven) Online Indexing process as now envisioned for those vault images. People using the online images (or even the traditional microfilm copies), and linking them to their family members in the new database, using actual record keys or library-style references, would in the process be gradually creating an index to those images as a byproduct of their efforts. If desired, at some point a final visual comparison of the images and the partially completed index could finish off the indexing/transcription process for the most often-used segments of the image database. (The links of names to source images could be turned around, inverted. This view would begin with source images and show the names linked-to from each image. The results could be completed and corrected, if necessary, much like the current plan for Online Indexing)


Kent Huff

----------------------------------------
801-798-8441, cell 801-787-3729
1748 West 900 South
Spanish Fork, Utah 84660
huffkw@juno.com, http://www.GenReg.com
computer consultant, attorney, author
================================================

mikron:
Thanks for letting us know about your interest and expertise.
I want to read all your stuff, and watch the Google video, and react. But that will take a little time.

So I thought a quick reply might be in order, for those who read the thread this far.

One of my design preferences is that the most central database be pretty compact and mostly act as an index to everything else -- no huge photos or images, etc., and just enough notes to make clear the identification of individuals. For example, if eventually there are 1.2 billion non-duplicated American and European names in the database, at 2,000 bytes each, that is about 2.4 terabytes. It would cost maybe $2,000 to store. That is a very small space compared to the space needed for all the source document images that are planned.

The database would be intensely used and soak up a lot of CPU cycles, but would otherwise be quite manageable in size. To start with, a mere 200 GB drive would do the job, up to 100 million names. One could even cache multiple copies to improve performance. That is probably not something you would try with giant image databases.

Unless there is a third-party site out there that allows the archival of geneological media, you could have the church allow the storage of media on a seperate server so that it wouldn't interfere with basic storage of geneological information.

thedqs:
You are right.
I hope that is how it turns out.

I think many missing the point, here, though. Storage for name hierarchy is nice, but not sufficient. People will use the website when it works like a Wiki. In other words, not only do we want a hierarchy of all 20 billion people who've inhabited the earth in the past 6k years, we want a nice write-up on each with uploaded pictures, journals, ordinances for all religions, etc. We also want nice tools to search the hierarchy for relations, browse it for fun, and read it for education. Suppose you start out with 100MB per entry. Multiply that by 20bil and you need 2 exabytes of storage. It sounds huge and expensive, but in reality it is easily feasible over the next 20 years. We already have databases that can support 20bil entries over a distributed system. That part is already done.

And all the talk about thumb drives... Those are for places without internet connections. And all the talk about client software; what's it for? Web applications these days are plenty advanced for nice data entry, file format import, data browsing, etc.

huffkw wrote: thedqs: “the only problem is storage”

As I estimate it, the Church only has about 40 to 50 million unique names in all its active genealogy and temple ordinance files at this point. All the rest are duplicates of one kind or another, even if that totals 1 billion records. (At this point, it would be hard to get the real number of unique names).

Allowing 2,000 characters storage for each of 50 million names only comes to 100 GB total. Pretty small. All the 300 million deceased Americans would only take up about 600GB, still a small amount. Maybe spend $500 for a 1Terabyte drive.

Putting the whole “bit mountain” on disk in image and text format will truly take petabytes, with megabytes per image, but I am assuming that the file I am suggesting will have no more than short URLs or library-style general references to these huge image files, so it will not expand much.
===============================================

haledn wrote: The value that I see the new FamilySearch offering is to jump start your pedigree so that you don't have to start with an empty database structure. Even though this data probably needs to be tweaked to be sure that it has the correct information and duplicates are handled correctly, this seems like a step in the right direction for many inexperienced genealogists who don't want to re-enter genealogical data which they know is already available.

I think you're missing the point here. Get over personal genealogy. We don't want personal genealogy. We want a record of the people of the planet that has no duplication.

brannon01 wrote: In other words, not only do we want a hierarchy of all 20 billion people who've inhabited the earth in the past 6k years, we want a nice write-up on each with uploaded pictures, journals, ordinances for all religions, etc.
. . . .
Web applications these days are plenty advanced for nice data entry, file format import, data browsing, etc.

I certainly agree with your sweeping view of what this should all be about, and the product we should be after.

I have just been trying to make it as manageable as possible by suggesting how it could be subdivided for actual construction.

mikron:
I like your more detailed link idea. I had not thought any more about it than just giving a line number as one might do with a book, page, and line. But it would be pretty cool to have a document image page come up with a highlighted rectangle around the data of interest. How would you describe the rectangle size and location in some generic way?
----------------------------
I read your 2001 FHT paper. I think it is great, especially your ability to estimate the number of servers needed to process the six object classes you describe: Application Objects, Catalog Objects, Link Objects, Scanned Objects, Translation Objects, and Name Translation Objects. I have wondered if it would be feasible to load the entire set of Application Objects and Link Objects (and perhaps other small objects) into server main memory, and it appears it is. That should deliver an enormous performance boost.

On your link object type that could specify which part of an image page is referenced, I wonder if that would need to be one step more generalized by using percentages of distances between the four corners of the image, since there could be many sizes and compressions used in moving the image around and displaying it, and any absolute size or pixel count could be inappropriate in some cases.

I see most of my ideas as relating to yours by specifying a further set of rules for the formation and interaction of objects, mostly the Application Objects and Link Objects. I am happy to leave the underlying industrial strength memory and storage systems to somebody else.
---------------------------
Your telecom paper sounds like slightly familiar territory.
I should mention that I worked for a time (among a staff of 900 programmers) on the gigantic Verizon telecom Event Processing System covering most of the US east coast. We were testing for about 200 million transactions a day in 1999, but I assume it has gotten much bigger since then.
I have read only the first chapter of your paper, but I was fascinated by the idea of using a second main memory as a backup or commit space, plus all the bottleneck studies and solutions.
---------------------------
The Google video mentioned your major PhD work adopted for MySQL clusters. That was pretty interesting. It makes me wonder whether the Church guys have used any of your open source work on very large data storage systems. Seems like it would be a good thing for them to check out, if they haven’t already.
---------------------------

I heard talk that the records stored in the granite vaults are being recorded electronically. Do you think that PAF could connect to that database, So when you enter someone in that PAF recognizes, It brings up a tree that is already existing and asks if it is your family. If it is, then it imports.

With this ides making one big tree connecting all of us to Adam could be a little easier.

WaddleDee:
I have been told that the process of putting the 2.5 million roles of microfilm in the vault into computer image form will take about 7 years. And the human readable parts of those pages will gradually be put into text form. Other documents such as temple records are already in text form and indexed.

I think nearly everyone wants to have a genealogy computer system that will do exactly as you suggest. That is my goal. We have all the computer technology we need to do it, so we just need to get busy. You should be able to put your grandparents’ names in and get back all the names that are connected to them, going back for centuries, without any duplicate names to confuse things.

It would be nice if you could put your own name in, and get the connected names, but the names of the living may be considered too sensitive, so they will have to be password protected, and you would have to contact someone to get permission to start a pedigree with your young self. But hopefully you know all your living relatives anyway.

If this project is not all done by the time you get out of college or computer school, then you can make sure it gets done yourself!

Sounds cool!

But I do agree with you about the "sensitive" information. If you want to add someone as young to your pedigree, then obviously you have to know the parents.
But then again, seeing that you just happened to be related closer to your neighboor then you thought, could be interesting to know...

THe records in the granite vault are being ditalized under the project of Internet Indexing, which is still in beta being tested by a select number of Stakes. This used to be the old Extraction Program where you'd get the microfilm or jpeg of a microfilm, type it up and then sumbit the results to the church through your director. By making use of the internet any member should be able to do this work and they have perdicted that in 7 years the entire Mountian will the digitalized. The records will be avaliable on the new Family Search and through the new client that the church most likely will replace PAF with.

Since the forum began, and there seemed to be some interest, I have added three short articles to my website describing the ideas I would like to see implemented. The current list of articles are as follows:

The Intended Revolution in Genealogy Research
A Tour of the Main Menu
The "Impossible Dream" of More Efficient Genealogy Research
System Feature and Operation Overview
System Logic Overview, May 2005 (PDF, 24 pages, 262K).
PowerPoint for April 3, 2003 BYU Family History Technology conference (PPT, 40 slides, 600K).

See them all at:
http://www.genreg.com/study/welcome3VC.html

Or go to http://www.genreg.com to see the current home page, then see the prototype main menu at http://www.genreg.com/famil05c.asp

This forum discussion thread seems to have nearly run its course. It has been worthwhile. I am glad that several people have participated with their comments, and a larger number have at least taken a quick look at the goings on here.

I guess I have a question mostly for the moderators. Government regulatory agencies hold hearings, and politicians run polls to better quantify public reactions to new ideas. Is there such a mechanism at the Church that relates to the many new genealogy computer developments in progress? At a recent genealogy conference keynote meeting held at the Salt Lake Radisson Hotel, Bill Mangum and Rich ? described a group that was doing research into how genealogists now do research and how new computer systems might assist them. Is this exploratory work and related design work all internal, or is there an appropriate place for more detailed discussions involving outsiders? I have heard that some of the private vendors of such things as genealogy PC software have been consulted, but I wonder if they will have covered all the possible topics and options. There may be many diverse opinions among genealogists, but perhaps there is a way to categorize and consolidate them.

I am sorry, but I can't answer that. (As in, "I don't know")

But if you PM tomw, I am sure he will be able to answer that.

WaddleDee wrote:I am sorry, but I can't answer that. (As in, "I don't know")

But if you PM tomw, I am sure he will be able to answer that.

I actually sent an email over to the Family History dept. for someone to chime in on your questions as I also don't know the answers to these questions.

Tom

Maybe these two guys can help us. They both work for the LDS Church Family History Department and seem to be in the know about things. They even have blogs. Here are their names and a link to their blogs. I have made comments on their lastest blog entries inviting them to monitor the forum and make comments. What do you all think - good idea?

1.) Grant Skousen
http://familysearchlabs.blogspot.com/20 ... -tree.html

2.) Dan Lawyer
http://eatslikeahuman.blogspot.com/

dlongmore:

Good job on your sleuthing. These guys are definitely knowledgeable and deeply involved in genealogy software development. It would be great to grasp what they are up to and add a few more ideas to their already bursting mix. One mentioned that he missed the TechTalk but has nonetheless been getting emails on the topic.
I sent an email to those two guys and to the Geni.com guys who are trying to change the genealogy website world.

It is my understanding that when you log onto NewFamilySearch with your church password you will see yourself and your personal Church records and all your ancestors who were members while living and their personal church records all tied into the Family Histroy pedigrees. The system will only display personal Church info of your own ancestors. In the same way that personal info is tied to a persons ID, I would like to see your own preferences tied to your ID. Example: If there are two sets of parents listed for an ancestor you could pick the parents you believe are correct and those would be shown in your pedigrees when you log on. When there are duplicate children entries that you do not have authorization to remove you could pick your choice and the others would not be displayed in your view of a Family Group Record. etc... This way everyone can personalize their Family History VIew and not have to duplicate everything in a private Pedigree taking up extra space. These personal preference markers would only take up bytes of space not megabites required for everyone to make their own copies of everything.

It is my understanding that links to original documents and multimedia items will already be included in NewFamilySearch and many automaticaly linked as the images are indexed and brought online.

The new scanning and indexing project of the info in the Vaults has already been going for a while. You can participate now by going to: www.familysearchindexing.org

A joint project from FamilySearchIndexing and the State of Utah was just completed and posted in the Utah State Arcives allowing you to look up images of Utah Death Certificates online for the years 1905-1954 go to: http://historyresearch.utah.gov/indexes/index.html

And regarding the above USB Flash Drive discussion, Flash Drives are available up to 8 Gigabites. It is hard to even find them smaller than 256 MG anymore. I have seen sales where you could get 1 GB Flash Drives for $14.

Just to put a plug for the family search indexing, it is a great way to relax for 10 mins a day. You do not need to complete an entire packet in one day but you can easily get 10-20 names done in 10 mins.

You just sign up to be a volunteer. It takes about 2-3 days before you get your username and password but after that, 10 mins a day gets a LOT done.

Future Features (that are not movies)

Garys Turn:
“It is my understanding that when you log onto NewFamilySearch with your church password you will see yourself and your personal Church records and all your ancestors who were members while living and their personal church records all tied into the Family Histroy pedigrees. . . . . .

It is my understanding that links to original documents and multimedia items will already be included in NewFamilySearch and many automaticaly linked as the images are indexed and brought online.”

I hope your description of the data-linking features of the NewFamilySearch is correct. You have identified a need and an expectation that I suspect and hope that many people have. We really ought to have that feature, it seems to me.

I have been suggesting something similar. I expect that it would be far simpler and more efficient for all concerned, and save some mind-boggling (and error-prone?) programming work, if the personalized data you describe were actually kept in a separate database structure, with links to other databases as needed.

As you suggest, the personalized portion needs to be able to select and consolidate data from the many separate NewFamilySearch databases, and well as from the even newer, and eventually much larger, Granite Mountain digitization project.

If the personal data feature is embedded in the NewFamilySearch system, it will probably tend to be limited to that data (50 million unique names in 1 billion records?). I expect that design choice would also make it many times harder to include the much larger (and less plowed) Granite Mountain material (1 billion unique names in 20 billion name entries?) as part of the personalized data, and so it might not happen at all. That would be a great loss.

The extra storage space needed would still be very tiny, certainly less than 1% of the current and planned size of all databases.

huffkw wrote:mikron:
I like your more detailed link idea. I had not thought any more about it than just giving a line number as one might do with a book, page, and line. But it would be pretty cool to have a document image page come up with a highlighted rectangle around the data of interest. How would you describe the rectangle size and location in some generic way?

My thinking is that it requires the following data:
1) Page ID
2) X and Y position of upper left corner and length in X and Y directions.

Given this information it should be feasible to index on Page ID such that when you click
on a certain X and Y position on the page it should be possible to quickly deliver all link objects
covering that position.

I'm assuming that one should have quality ratings of links as well as what the links refer to. Thus after
clicking on a link one gets a pop-up window which provides links to:
1) Transcriptions of the page part
2) Links to family trees referring to the position
3) Links to other Link Objects (e.g when there are several recordings of the same basic information,
e.g. in Sweden we have church records of births and for the period 1860 and forward we have
transcriptions of these books that have also been microfilmed).

----------------------------
I read your 2001 FHT paper. I think it is great, especially your ability to estimate the number of servers needed to process the six object classes you describe: Application Objects, Catalog Objects, Link Objects, Scanned Objects, Translation Objects, and Name Translation Objects. I have wondered if it would be feasible to load the entire set of Application Objects and Link Objects (and perhaps other small objects) into server main memory, and it appears it is. That should deliver an enormous performance boost.


Yep, it's definitely getting there, I'm currently at a benchmark center and they have clusters with
2.4 TBytes of memory so it's definitely feasible to put all the family trees and transcriptions and
indexing information in memory.

On your link object type that could specify which part of an image page is referenced, I wonder if that would need to be one step more generalized by using percentages of distances between the four corners of the image, since there could be many sizes and compressions used in moving the image around and displaying it, and any absolute size or pixel count could be inappropriate in some cases.


I guess one has to specify the position in relative terms in percent of page size rather than in pixels.
Then the graphical software that translates the mouse clicks into a database query has to translate
the pixel position into a relative position used by the database.

I see most of my ideas as relating to yours by specifying a further set of rules for the formation and interaction of objects, mostly the Application Objects and Link Objects. I am happy to leave the underlying industrial strength memory and storage systems to somebody else.
---------------------------
Your telecom paper sounds like slightly familiar territory.
I should mention that I worked for a time (among a staff of 900 programmers) on the gigantic Verizon telecom Event Processing System covering most of the US east coast. We were testing for about 200 million transactions a day in 1999, but I assume it has gotten much bigger since then.
I have read only the first chapter of your paper, but I was fascinated by the idea of using a second main memory as a backup or commit space, plus all the bottleneck studies and solutions.
---------------------------
The Google video mentioned your major PhD work adopted for MySQL clusters. That was pretty interesting. It makes me wonder whether the Church guys have used any of your open source work on very large data storage systems. Seems like it would be a good thing for them to check out, if they haven’t already.

I'll be presenting some of this at the Family History Technology Workshop at BYU in 2 weeks from now
so will be interesting to also hear where the church is on this right now.

---------------------------

Text entered to overcome bug in Web software

WaddleDee wrote:I heard talk that the records stored in the granite vaults are being recorded electronically. Do you think that PAF could connect to that database, So when you enter someone in that PAF recognizes, It brings up a tree that is already existing and asks if it is your family. If it is, then it imports.

With this ides making one big tree connecting all of us to Adam could be a little easier.

I think what we want is something pretty similar to source code version systems. There is the
major copy which is central and then each person has his own tree which contains parts
downloaded from the central database but also parts he is working on.

I think it is also import to be able to insert research ideas, so if a person has been identified
as a possible father to a person it should be possible to insert him into the database but where
the researcher can specify that the result is so far uncertain. Given indexing to real images
it should be possible for other researchers to assist then in giving more solid evidences of
the family tree connections.

I see where you may be going with this.

So you are saying that when we have a tree up, and we know certain sources, and those sources have been indexed and scanned and are thus online, that we should be able to link to them?

If so that would be an extraordinary thing, to see exactly where the data in the tree was found and so forth.

Eventually, there will be other sources, including material the person who has the data has found, such as a picture of a headstone, and I think there ought to be a way one could attach things like that to the record to indicate a source record for a burial (the headstone photo), for example. Maybe not initially, but down the road hopefully sooner than later.

One question was asked about electronic images, and they are taking digital pictures now, 100 cameras are all-digital now, 15 megapixel cameras are being used. No retakes are ever needed (compared to the problems routinely cropping up when doing microfilm shoots) in virtually every case, a DVD or two is burned onsite, given to the party or archive owner at the location they just shot the images in, and the images are sent off via the net to the vault, where they are stored for later use at what to me presently is an undetermined point in the rollout of new FamilySearch.

Long thread so I didn't read all of it; I hope I can add my two cents in adequately and that it doesn't duplicate too much. I come at this topic using the following logic from my perspective as an amateur genealogist: I exist therefore my ancestors exist! My job is to prove they existed to the best of my ability and to provide source documentation. Now I may not be able to find source documentation - it may not exist - but how am I to know unless I have access to all the databases of all source records?

Although perfection may not be obtained, I think a "smart database" is definitely something that should be considered. Let me explain my non-technical view of it:

1. Digitizing a database and allowing it to be searched is great, BUT what if the database searched itself and linked data that was "similar" - uniquely identifies individuals and links the digital source record/image to that individual - that would be even better!

EX: Each database type, i.e. Census is digitized and then grouped into Surname sub-categories, since Census is taken sequentially every 10 years THEORETICALLY one could start at ANY Census year and begin linking individuals in previous/subsequential Census years. Of course I realize Census is an imperfect database - it is an incomplete database - not everyone was counted, the Census taker recorded information incorrectly, people didn't go by the same name - or with the same birthdate - HOWEVER a "smart database" can check for inconsistencies and cross-check against the family group.

2. So if a "smart database" works on itself and also takes end-user data and compares an accurate database MAY result with the end goal that a user inputs one ancestor/family group and not only finds their ancestors but decendants as well. So not only does the database search for and find individuals but it links family groups generationally and displays original source images.

I believe the Census database would be a perfect test case/base for implementation of such a project because if its sequential nature. Once the algorithm was perfected then other database types could be added - of course a perfected Census database would depend on linking maiden names requiring birth/marriage/death database "smartness" as well.

Wouldn't this result be great? The end-user inputs one individual/family group and ends up with not only the lineage-linked information they may be searching for but the original source documents!

JAOCON wrote:"smart database"

I think my biggest fear would be the smart database would make some errors and match people who are similar but are not really the same person together. You could see that happening with common names. Then people would accept it as true without looking at the connections in depth.

An automated system that would help find and link families will/would be great. I have wondered about a system that could use birth, marriage, and death information to (semi-) automatically build and link a pedigree. Census and other records could certainly figure into something like that.

However, I suspect it will be difficult, especially with census records, to automatically deal with the large number of inconsistencies in the raw data. I hope those working on such systems don't underestimate the level of inconsistencies in the data, as I think it would be very easy to do that.

For example, my wife found what appears quite strongly to be the same family in two adjacent decades (1920 and 1930, if I remember correctly). The earlier has a female child Harriet of age 3. The later one has a male child named Harry of age 13. She guesses Harriet used "Harry" as a nickname and the census taker got the gender wrong. She is exploring other resources to try to confirm (or refute) that guess.

newFamilySearch (NFS) is a smart database in some respects. It searches the databases in FamilySearch and provides them as possible links, all someone has to do is look at the possible links and decide if the records NFS has found should be linked. Check out some of the Screenshots that are linked to on this site. As more databases are added to NFS they will also be searched for possible links, this will include census and other records.