gb308 wrote:
I don't understand technology that well....but, it seems to me that avoiding duplications should be a simple database, computer issue.
You are right, but no one seems to know it yet. Here is a small part of my proposal to kill the duplication beast once and for all.
(I would prefer to send this lengthy response as an email to the person who started this thread and raised the question of why so many duplicates, but email does not seem to be an option, so I will try it here.)
The topic of duplicate names has been a very frustrating one for me, as apparently it has been for you. The fact that a simple, complete, efficient, straightforward solution to the problem is possible makes it even more frustrating.
You have inspired me to write A PAINFULLY DETAILED VERSION of the basic concept needed to solve the problem, in hopes that someone will actually read it and grasp my point.
I have been trying for years to introduce a new computer concept to put a complete end to this problem of duplication, but without any success. I cannot say that a single person, programmer or layman, has grasped the simple concept I offer. Maybe you will be the first. If I had the money, I would have done it all myself, so that I would not need to explain it in advance to anyone; I could just SHOW them how it works. But I don’t personally have the $1 to $2 million it would take to do it right.
Here is this brain-bending stumper. Turn up your thinking cap to full power or you will miss it. Suppose that we had a computer system that contained all past and current completed genealogy research online, in a format so that you could almost instantly determine what research work had already been done that relates to you, BEFORE YOU DID ANY RESEARCH OF YOUR OWN. This could be done on the first day a person felt an interest in doing genealogy research and temple work. It might take that person an hour or two to learn about everything of importance that had been done before that was related to that person, so that person need not repeat ANY RESEARCH in those areas. If you wished, you could also spend a few moments longer and examine some of the areas that had NOT been researched, so that you could choose a good place to add something new which might be of value to yourself and others.
Now, let me make it clear that I am not talking only about temple work that has been done. The research in this database would be ALL the research, regardless of whether it had been submitted for temple work or not. This would allow people outside of the Church to use the system just as well as any member, so that members need not duplicate the work of other US genealogists.
One more time: The names in the temple can only get there if people do research work and submit any names found. BUT, if you can tell, at step 1, what research has already been done, YOU NEED NEVER GET TO THE POINT OF HAVING TO CHECK TO SEE IF ANY NEW NAMES PRODUCED BY YOUR RESEARCH HAVE ALREADY HAD TEMPLE WORK DONE FOR THEM. You just avoid the whole question, and you can do it because you can check centrally all completed genealogy RESEARCH, not merely all completed TEMPLE WORK. There would be no need for a Temple Ready system, or anything like it, because no one would be doing any duplicate research which needed to be checked for duplicate names.
Now to try some contrast: In today’s (old) system concept, which goes back at least 60 years, long before computers or the Internet were even thought of, we imagine, and remain convinced, that the only thing we can possibly do to stop duplicates from going to the temple and then into the Church ordinance records, is to expect people to do all their research, taking however many years it may take, and only then, with all that work done and names identified, do we try to see if those names have been processed before. That is a terribly wasteful use of member research effort, with 95% of the effort wasted. That was indeed all one could expect to be able to do using a manual check in a manual system, 60 years ago, or even 20 years ago. There never was the slightest chance, back then, that the many researchers, working independently, outside of a big, well-managed genealogy research firm, could coordinate their efforts to avoid duplicate research effort in identifying their ancestors. But today, such full, advance coordination of work could be a very simple matter. Perhaps all that is required is that the mere possibility of it be recognized by those with the power to implement it.
===================================================
Old Versus New System Sequence -- diagram
We need to reverse the sequence and change the method of checking for duplicate work
Old System sequence:
Phase 1 -- Research -- may last many years.
--only minimal and unpredictable coordination of effort is possible during research.
--95% of the effort is normally wasted in duplicate research.
Phase 2 – Name check phase
--lasts a few hours or days.
--95% of work is typically found to be duplication, causing disappointment of researchers. After doing all that otherwise wasted work, they tend to send the names in anyway, mostly defeating the purpose of the Temple Ready and similar systems.
Phases:
|--------------------------------------------1----------------------------------------------------|-2-|
New System sequence:
Phase 1 -- Check status of all research related to you.
--Spend a few hours on this task. A non-duplicating research plan can be formulated.
Phase 2 -- Research -- may last many years, but all of it is productive.
--Do only research that has not been done before. 98% to 99% will be new data.
--There is no name check at the end. There is no need to check for duplicate names submitted for temple work, because very few, if any, will slip through or arrive at that point. A last check might be done just to make sure the system is working correctly, but the duplication rate should be in the range of an insignificant 1% to 2%.
Phases:
|-1-|--------------------------------------------------- 2 -------------------------------------------->
===================================================
Now I have to assume that you still do not get it, because it seems every Church member’s brain is so conditioned by 150 years of manual, paper-based system thinking that those brains simply cannot form the new concept, regardless of the number of times the concept is repeated in words. But maybe you can break out of that mold and grasp it. If so, I do hope you will let me know how you cracked the code, and that you will pass your insight along to others.
Why can no one grasp this simple new idea? I don’t know, but I have to assume that there are very few who have any real idea of what the Internet makes possible. And of those few people who do understand the mathematical and cooperative power of the Internet, there are few or none who have made any deep study of genealogy processes and possibilities. In any event, so far the concept has never popped up where there might be a person who has the power to act on it.
I have to assume for most people there is an instant wall of complete denial that any such “magical” database as I have described could possibly exist in the real world. Surely it could be no more that a foolish flight of fancy, they might say, since nothing like that now exists.
But let me begin to explain what this new database might look like. In one good beginning example, it might consist of a central collection of all the current contents of everyone’s PAF files on their home computers. (That is where all the best data is kept. That data is usually degraded as it goes into current public systems. This needs to be fixed so that data going into central systems goes UP in quality and usability.) With all serious PAF files online, everyone could see what everyone else had done and was working on.
But wouldn’t there still be massive duplication and confusion in those combined PAF files, even more than in the temple ordinance files? Isn’t the Pedigree Resource File exactly that -- a copy of everyone’s PAF file? Perhaps several copies of PAF files were submitted to PRF at different stages of development? Yes, to all those questions. That is what makes using the PRF a confusing task.
So here is where a tiny amount of new logic can make a huge difference. There is no need to put up with this confusing duplication, and no need to try to “merge” the duplication out of existence. Some simple rules for entering data, and for showing whether data is complete enough, possibly including source references, is all that is needed to stop essentially all duplication. I won’t try to get more detailed than that here, but it is not very tricky, mostly just requiring a change in attitude and sequence. But, of course, changing attitudes is itself a very big challenge.
In the past, the standard “solution” for duplication, going back 30 to 40 years to the Four Generation project and the resulting Ancestral File, was to try to take that mountain of heavily-duplicated data and mash it all together. People have invented innumerable machine algorithms to try to do this massive merge automatically. The results are usually pretty bad. This can become the classic Garbage In/Garbage Out situation, including a new Garbage-Squared multiplier. The New Family Search system used a machine algorithm to more carefully merge a portion of the data, but then has assigned Church members to do the many tens of millions of hours of manual merging it will take to finish the job of reducing the 1.5 billion names down to perhaps 100 million unique names.
But when all that huge amount of NFS merging work is done, I believe there will still be no efficient way to prevent discouraging and wasteful duplicate RESEARCH effort done before ever getting to the NFS duplicate check stage. There will still only be some restraint on duplicate names, produced by unnecessary duplicate research, passing into the temple name processing channels. (The NFS system might conceivably be modified to let people put in all their new research-in-progress, far ahead of, and on top of, and eventually many times larger than, the temple work actually completed. But this is mixing apples and oranges, and would likely cause a whole host of problems, for programmers and for users. It would be better to keep the different fruits in two different baskets – two databases.)
In my view, it is hard to say whether the New Family Search system will be better or worse than today’s Temple Ready system in stopping duplicate names. If it uses only the merged names, the “Summary View,” it will logically catch less duplicates than the current Temple Ready system. If it uses ALL names, merged or not, it will hardly be any different from today’s TR system.
To summarize, I argue that the concept underlying the Temple Ready process, and its replacement, the New Family Search system, is now technologically obsolete. The basic concept is at least 50 years old. I consider the methods used to be about 20 years out of date because they have not fully embraced the new opportunities made possible by the Internet. (This is true even though the NFS uses the Internet intensively.)
Consequences
I predict that, with the NFS system in place, the number of duplicate entries that get into the temple ordinance files each year will likely be higher than ever before. Our directly attacking the SYMPTOM even harder (of duplicate names being sent to the temples), rather than recognizing the real PROBLEM -- the unnecessarily duplicated research effort -- will actually make things worse.
There will likely be less really new research (and thus fewer new, unduplicated temple names), and even more reliance on existing names in Church systems. (Nearly all names in Church ordinance files are only there because their temple work has been done. So they would automatically be duplicates if they were reintroduced into Church temple work systems).
The reduction in new research, caused by the large new NFS member workload, means that even more duplicates will be resubmitted than in the past. The temples need new names, and duplicates are the easiest to find, as any local genealogy librarian can attest. This unintended consequence of more duplicates will probably be deplored, but also mostly ignored, because everyone assumes there can be no better solution.
As you saw in an earlier post on this thread, when things began 150 years ago, it was solely the member’s duty to avoid duplicates. Apparently, that eventually proved to be ineffective as the volume of names grew, so it was accepted as a corporate responsibility to try to minimize the flow of duplicate names into the temples and the temple ordinance files. I believe the Temple Index Bureau was the name of that original central processing group. Apparently, that TIB method also proved to be ineffective, presumably because the number of members, temples, and submitted names mushroomed further.
So then the responsibility was put back on the members. The Temple Ready system automated the prior centralized manual process and was offered as a helpful tool. But in my opinion, that tool was not sufficient. Today’s 10 or 15 times average duplication rate in ordinance files shows the problem was only partially solved. The New Family Search system is an attempt to improve upon the TR system, but I believe it still falls short of what is needed. At least conceptually, it puts the burden on members to correct 150 years of duplication of name submissions before they can move on to new areas of research. And even then, the NFS system will probably do little to curtail the continued duplication of the research itself.
Apparently, the old ways of thinking are hard to shake, even as the Internet offers new options. A better central solution holds out the possibility of a 20-times increase in productivity and accuracy. Until that opportunity is grasped, I expect the behavior of genealogists will remain unchanged, and the duplicates will keep rolling in.
