Possible nFS Combining and Uncombining Solutions

I started this thread to continue the discussion from another thread.

One of the biggest concerns with nFS is the combining and uncombining of records. There are problems with some folders containing to many records and this slows things to a crawl. There are problems with folders where combining errors create folders which have multiple different individuals in them and we have the task of seperating out the different individuals.

What are your ideas on how you feel these processes could be improved so they function more efficiently?

We hear a lot of complaining about the problems, lets discuss your ideas for possible solutions.

I'd like to see a block on combining records when it would result in a impossible combination - such as a son who is his own father. Or two individuals who are simply too far apart in time or space to possibly be the same. The error would have to be resolved or bad data disputed before the combining could take place.

At least that keeps a bad problem from getting much worse before it gets sorted out. Also it requires more mouse clicks to mess things up.

Russell's corollary #2 to Murphy's law: If it's possible to mess things up with a singe mouse click, some one will click it. :rolleyes:

I wonder if something like how programmers use 'deprecated' could be used. If you could mark a record as 'bad' or 'really this other person', then you could lock the bad records without actually merging.

Specifically, if I had 3 J. Smiths, with no other information, It seems silly to 'merge' those records, when all that is different is an IGI # or other identifier. Why not archive / delete / make static 2 of the 3, and mark them appropriately. Marked records would be read only, and maybe not even stored in the same way as active records. They could be flattened to raw text and stored / displayed as a note, or attached file or somesuch.

This tagging would also help with erroneous information, as you could note the reason for the bad record. You might have a record like 'J. Smith, Birthdate incorrect on marriage record, correct birthday in record 384883 from family bible'.

Marking records in this way is MUCH more intrusive than merging. I am not sure if it would be possible to 'unmark' the records easily, especially if they were flattened to text as I recommend above.

That being the case, I would probably allow users to vote for the tag, but only allow moderators to actually flatten the record.

This might be very useful for well know / vetted lines like royalty and church historical figures.

A bit rambling, but there it is:
The Earl

There are several example combining problems I've come across, and each has a different solution which comes to mind. These are based on my limited understanding of how the NFS system works, and my background in software engineering.

For some of instances where I'd like to separate individuals, I can see a single field where the difference lies (for example, name of spouse). In these cases, just being able to sort the merged records by a certain field would speed up splitting. This would be equivalent to adding an ORDER BY statement onto a SQL query (or whatever is being used to retrieve the records from storage).

In other instances, the problem is hugely compounded by overzealous combining (fathers combined with sons combined with grandparents and other unrelated people, etc.). These are the ones which seem to be causing the biggest problem. My solution for this last one would be to change from the current "View Combined Records" approach which displays only the last 5 individuals to a tree view of the combine events. Rather than picking off individual records one at a time, what is usually wanted is the ability to find a single combine point in the tree and undo only that one. Thus, a merge which combined a person with 300 records with a (wrong) person of 100 records doesn't require working through a stack of 400 records, pulling out 100 people one at a time. The only problem to that is whether or not the system is currently storing the merge events (so as to be able to construct the tree). Whether or not the data is historically available, it could be added and used from here on.

The third combine error I've come across consist of single records which list their point of origin as Ancestral File, and are clearly containing information about two distinct individuals. These usually cause the system to recommend records from both individuals as being matches (which makes sense). However, there doesn't seem to be any way to unlink the problem record. These are the times when all out destruction of the bad record would be nice.

I also agree with the need for having a block preventing separated individuals from being combined. There is such a block in place for individuals with Church Membership records.

And with all of these, it would be very valuable to have the ability to flag individuals such that I either have a readily available report which shows when any are combined or split by other people, or have a notification (email or NFS internal notice) when such an event occurs.

Another variant on this would be to be able to work on groups of r ecords where you know there are a couple hundred separate entries for that name but you know how many can be in one folder.

I heard there is a knowledge document out there that says the .93 release has the limit at 80 records per folder, so if say there were 200 entries (I have one possibly myself in one of my lines) then there should be a way to have multiple folders to bring together like entries as well, where it is known that everything is very nearly the same but the problem is there are just so darn many entries! And a way to see that you have multiple folders on the individual so you know what you are working on.

The last thing is that when you find temple work on one entry anywhere in any of the folders you have open on an individual should this be a viable workaround for the IOUS problem pending further developments on this, it should also be able to carry over to the multiple folders, so that someone doesn't just come along and say 'there's temple work here', and redo the work. In most cases in combining the duplicates a duplicate set of ordinances will do it but there's no guarantee that will work every last time either.

I like the idea Earl gave of Archiving some of the records. I think that once a folder contains completed ordinances and someone has selected the correct entries in the Summary area. All other records which are not used should be archived and not loaded each time the folder is loaded. The archived records could be loaded if someone clicks on the Details tab or the combined records view, but it would sure make things run eaiser if only the summary information and ordinance records were loaded unless someone requests to see the other combined records. Then when we are just moving through the pedigrees we wouldn't have to wait while all the duplicates are loaded for each folder.

I know the Church will not want to allow users to delete or change someone elses submissions but if we could also identify bad records and mark them somehow it would help. (until the owner claims them and corrects them) Then those bad records would not be used in decisions to merge.

Some other options might be:
Archive unclaimed records until someone claims them
Allow users to mark records to be archived
Allow several users working together to mark a record as bad and archive it

Like Rick I would like to have the ability to uncombine blocks of similar records. If the list of duplicates were displayed in spreedsheet format with the headings at the top instead of along the side and we had the ability to click on a heading and have all the records be sorted by an event, like a birth date. Then all the records with the incorrect birth date would be together in a block, then if they would allow us to select a block of records to uncombine and put them all into one folder then we wouldn't have to go through the combining process to get them all back in the right folders.

Example: I have an ancestors folder with over 900 names in it. There are at least 3 different individuals (father, son, and grandson) that I have identified in that folder with 100's of records for each of the 3 individuals. If we could sort by birthdate then mark a block of records to be split off into a new folder if would be so much better than separating them one at a time then combining them one at a time back into the correct folder.

When I originally brought this issue up in a different thread I had in mind the idea of separating blocks of records. Most often I see two or three distinct individuals (and their respective combined records) being combined as one individual. To sort them out into the two or three individuals as a block of records would be ideal. I also liked Rick's idea of being able to locate the point at which the mistaken combine point was made and just undoing that action. These things could save us all a lot of time. Time that I feel would be better spent on researching and finding new individuals and going to the temple rather than cleaning up mistakes in the NFS database.

I agree with some of Rick Parker's comments. I spent more than a week trying to make sense out of my PA Quaker Moores and Millers which were all scrambled because of loops within loops. Before the 80 record limitation, I separated records from 5 different generations until I identified the problem. Someone identified as bbull2810432 had submitted information to Pedigree Resource File which contained multiple parents. He had one person connected to a spouse 3 or 4 generations away, and another married to his grandmother as well as his wife, as well as others linking mistakes. I know of al least 5 linking errors in his submission, but feedback resulted only in my being told that we have to wait until that person gets access to correct his own errors. I separated out about 5 generations of his submission in order to get the family to appear correctly in nFS...but someone has gone back in and recombined some of the names again. I wish there were a way to quarantine a specific record that is known to have erroneous links within it, if it can't be deleted. When I consider how many hours I spent working on this problem, I think the developers should consider our worthy efforts to create a correct database, and stop protecting those who have made a mess of it.

GarysTurn wrote:I think that once a folder contains completed ordinances and someone has selected the correct entries in the Summary area. All other records which are not used should be archived and not loaded each time the folder is loaded.

I need to submit this formally through the nFS feedback system, but there also needs to be a way to select which marriage information to display.

Additionally, when there is more than one ordinance date for a given ordinance, we need either the ability to select which date to display (a la "Summary"), or the system should automatically display the earliest date for a given ordinance.

I have an example in my line of proxy initiatory occurring several years AFTER proxy endowment which is itself several decades AFTER live sealing to spouse (for some reason the "live" endowment was never recorded on the membership record, or the temple records in question have not been extracted that far back or are otherwise not correctly linked to this individual).

Selecting the "correct" information or the "preferred" spouse or parents needs to be visible to the person choosing it. One person's preferences need not change another person's selected settings.

jbh001 wrote:
Additionally, when there is more than one ordinance date for a given ordinance, we need either the ability to select which date to display (a la "Summary"), or the system should automatically display the earliest date for a given ordinance.

The oldest ordinance date in the folder is already the default.

I agree we should be able to select the correct marriage date.

We do already have the ability to select the preferred parents, we need that same option for the preferred spouse. This selection of the preferred parents can be different for each user.

GarysTurn wrote:The oldest ordinance date in the folder is already the default.

Then there are bugs in the system because "old" Family Search for this individual has IGI entries that have BOTH a 1997 date (Boise Temple) and a 1992 date (Logan Temple) for proxy endowment. Apparently nFS is pulling one IGI date for the initiatory and the other date for the endowment. The logic nFS uses to determine which of these dates to use for "initiatory" escapes me since IGI labels both dates as "endowment."

Regardless, I would think that by now the temple records would have been extracted to IGI so that the original 1919 live endowment date (FHL Film 177977) was available somewhere to link to this individual. I guess I'll just have to wait for the digitized versions of the old temple records are made and made available for online searching.

jbh001, that is my experience, too.

Unfortunately, the feedback system on nFS all seems to be filtered through nFS missionaries, rather than an option to give feedback directly to IT or whatever they call the tech dept.

I did provide feedback on the same problem that you noted, but the rambling responses from the volunteers did not address the issue and closed the case. Why provide feedback via nFS if it never gets to the IT folks working on the software because the volunteer answered his/her own question & then closed the case?

A some of us think the OLDEST ordinance date should be the valid one. The other dates, like all duplicate information (I have one relative endowed 15 times) should be sent to a archived or referenced but not listed. If any "new" information was entered it would be a dispute and could be dealth with in the new system.

There is some information that is obviously wrong (a person born in 1480 with a recorded baptism/christening in 1940 comes to mind) where I am trying to contact the submitter comes to mind.

Alan

jbh001 wrote:I need to submit this formally through the nFS feedback system, but there also needs to be a way to select which marriage information to display.

Additionally, when there is more than one ordinance date for a given ordinance, we need either the ability to select which date to display (a la "Summary"), or the system should automatically display the earliest date for a given ordinance.

I have an example in my line of proxy initiatory occurring several years AFTER proxy endowment which is itself several decades AFTER live sealing to spouse (for some reason the "live" endowment was never recorded on the membership record, or the temple records in question have not been extracted that far back or are otherwise not correctly linked to this individual).

I think one question we need to ask with separating records is do we want to create orphan quarantined records that just sit out there. Or would it better to add a summery view on the relationship screen much like there is one for individual information.(Like whatjbh001 said)

This is a big issue with me especially with my tree containing a smith line. I don't necessarily see a problem with individual information. I more see a problem with relationship information. Like someone’s who record should be restricted being married to at least 7 wifes but he only(To my recollection) had one and the sealing were done incorrectly by members.

I also agree with the need for having a block preventing (or at least making it more difficult) separated individuals (I will call them quarantined record to distinct them from not this individual separations, the current do not combine notice does not work.) from being combined. There is such a block in place for individuals with Church Membership records.

I recently separated an entire AF submission (5 generations) to where the children were born when one of the parents was dead. So I separated the entire AF submission out of the tree. I then added a DO NOT COMBINE NOTICE. Did not dispute the records but did follow current policy on that. I was a bit frustrated to find out that they were later combined. I came to the conclusion that someone using Family Insight combined the record without seeing the note. It would be nice to add an extra warning to a flagged record asking are you really sure this information is correct and though that error out not only to people using NFS but people combining though the API of NFS.

In regards to separating records when I separate a record I want the relationships separated out of the record view (Currently when you separate a record with relationships it keeps it in the tree as a Alternate relationship. It is fine if it stays there but I would like it to go out of view. Or a copy sent to the record I am separating so I can then FLAG the record as incorrect(this needs improvement too see paragraph above.)

Comments on what has been said so far.

RickParker wrote:The third combine error I've come across consist of single records which list their point of origin as Ancestral File, and are clearly containing information about two distinct individuals. These usually cause the system to recommend records from both individuals as being matches (which makes sense). However, there doesn't seem to be any way to unlink the problem record. These are the times when all out destruction of the bad record would be nice.

Due to it seams to be necessary to preserve information I don't think destruction is the answer. I think for display issues "filtering" incorrect relationships, or completely separating adding a flag would be an answer.

RussellHltn wrote:I'd like to see a block on combining records when it would result in a impossible combination - such as a son who is his own father. Or two individuals who are simply too far apart in time or space to possibly be the same. The error would have to be resolved or bad data disputed before the combining could take place.

I really like this idea but be a little leant on this. Some women have had children past child bearing years.

GarysTurn wrote:I like the idea Earl gave of archiving some of the records. I think that once a folder contains completed ordinances and someone has selected the correct entries in the Summary area. All other records which are not used should be archived and not loaded each time the folder is loaded.

I think this is a good idea but only when primary sources have been attached to the events displayed in the summery screen verifying all information is correct.

GarysTurn wrote:Some other options might be:
Archive unclaimed records until someone claims them

This would be nice but may leave out valuable data.

GarysTurn wrote:Allow users to mark records to be archived

I think this is a good idea depending on what it would prevent while searching for duplicates.

GarysTurn wrote:Allow several users working together to mark a record as bad and archive it

This is a very good idea and could be attached to the dispute system.(which also could have some improvements currently the dispute system only displays information and makes records look bad and people are disputing incorrectly due to the usability issue with disputes.) Set a threshold on votes and not only archive it but take the record out of view.

whitcomb wrote:A some of us think the OLDEST ordinance date should be the valid one. The other dates, like all duplicate information (I have one relative endowed 15 times) should be sent to a archived or referenced but not listed. If any "new" information was entered it would be a dispute and could be death with in the new system.

I have a relative in my direct line what was baptized twice when he was alive once at 6 ½ and the other at the age of 8. I still think that selection should be restricted and the proper procedure is due to it is LDS Ordnance data submitting a membership correction on the record.