Search Errors

Where is the correct place to report errors in the online scriptures? I once found a broken link in one of the footnotes (I can't remember the details but I think it was supposed to point to somewhere in JSH). Today I noticed an error on http://scriptures.lds.org/en/harmony/harmony5
On the line "Jesus returns to Galilee and preaches" it links to Luke 4:14-14. This should be 14-15.

This forum will do fine. Thanks for pointing it out. I'll forward over to the team responsible for the online scriptures.

Tom

tomw wrote:This forum will do fine. Thanks for pointing it out. I'll forward over to the team responsible for the online scriptures.

Tom

English scriptures: I searched for the word "carnal" and 27 references were returned, but 2 Ne. 9:39 was not in the list. This scripture refers to "carnally-minded". AFAIK the same 27 references are returned when you search for "carnally." If you search for "carnally minded" 2 Ne. 9:39 will be returned. I'm not sure if this is an error. It may be that the word is not a form of carnal because it is hyphenated. I had not even noticed that the word was hyphenated in the text.

Thanks,
Dennis

dmaynes wrote:English scriptures: I searched for the word "carnal" and 27 references were returned, but 2 Ne. 9:39 was not in the list. This scripture refers to "carnally-minded". AFAIK the same 27 references are returned when you search for "carnally." If you search for "carnally minded" 2 Ne. 9:39 will be returned. I'm not sure if this is an error. It may be that the word is not a form of carnal because it is hyphenated. I had not even noticed that the word was hyphenated in the text.

Thanks,
Dennis

Have you tried using any wildcard characters, such as the asterisk ('*')? It would be interesting to know how that affects your search results (i.e., "carnal*" or "carnally*").

It would be nice though if the scripture search feature had Stemming (if it doesn't already).

mkmurray wrote:Have you tried using any wildcard characters, such as the asterisk ('*')? It would be interesting to know how that affects your search results (i.e., "carnal*" or "carnally*").

"carnal*" returns 28 references including 2 Ne 9:39.
"carnally-" returns no references.

"carnally-*" returns 5 references including 2 Ne. 9:39. "carnally" returns 4 references (when Search all word forms is not checked). It seems strange that "carnally-" returns no references, but "carnally-*" returns all four references that have the word "carnally" by itself.

"*minded" returns 21 references including 2 Ne. 9:39.
"minded' returns 12 references (when Search all word forms is not checked). 2 Ne. 9:39 is not listed.
"minded" returns 189 references (when Search all word forms is checked). 2 Ne. 9:39 is not listed.

Off-topic: Is there a way to see just the references without the text context in the returned search results?

tomw wrote:This forum will do fine. Thanks for pointing it out. I'll forward over to the team responsible for the online scriptures.

Tom

The help under search states:

'Use quotation marks ( " ) around a series of words to limit the search to an exact phrase. Try entering “let there be light”.'

The search doesn't always appear to return only scriptures that have the exact phrase. For example, if you enter "be one" some of the scriptures that are returned do not contain the exact phrase "be one." http://scriptures.lds.org/en/search?search=%22be+one%22&do=Search

2 Ne. 9:12 is an example of a scripture that is returned with this search but does not contain the exact phrase, "be one." http://scriptures.lds.org/en/2_ne/9/12#12

Is this working the way that it is intended to work? The scriptures that are returned will have the words "be" and "one" but sometimes there are intervening words.

Thanks,
Dennis

dmaynes wrote: Is this working the way that it is intended to work? The scriptures that are returned will have the words "be" and "one" but sometimes there are intervening words.

Thanks,
Dennis

Yes, that is how it is designed.

Tom

tomw wrote:Yes, that is how it is designed.

There appear to be inconsistencies with the design implementation. In all of the tests below, the "Search all word forms" is turned off.

I did the following searches:

"be" -- (quotes) 1454 entries returned
"one" -- (quotes) 946 entries returned

be one -- (No quotes) 946 entries returned
--> This is the same as "one" (the word 'be' must have been discarded)
--> There was no message indicating that the word "be" was discarded.
--> There is a message that the word "be" is discarded when it is used stand alone without quotes.

be & one -- (no quotes) 383 entries returned
"be" "one" -- (quotes used, ampersand was not used) 383 entries returned
--> If this is correct, the number of scriptures with "be" but not having "one" is 1071
--> Similarly, the number of scriptures with "one" but not having ""be" is 563.
--> The total number of scriptures with "be" or "one" (inclusive or) is 2017.

be | one -- (No quotes) 1503 entries returned
"be" | "one" -- (quotes) 1503 entries returned
--> This is a contradiction. It should have been 2017.

"one be" -- (quotes) returned 24 scriptures.
"be one" -- (quotes) returned 65 scriptures.
--> The total of these two searches should have been at least 383, and maybe larger because if both words are together then one must appear before the other.

"one be" & "be one" -- (quotes) returned 1 scripture.

Following the inclusion/exclusion computations, the number of scriptures where "be" precedes "one," but "one" does not follow "be" is 64. With the same computation, the number of scriptures where "one" precedes "be," but "be" does not follow "one" is 23.

"one be" | "be one" -- (quotes) returned 86 scriptures. This value should have been 88.

--> According to the the "be" & "one" search, the value of "one be" | "be one" should have been 383, because if both words are in the same scripture either "be' follows "one" or vice-versa.

Please e-mail me if you want to talk.

Thanks,
Dennis

dmaynes wrote:There appear to be inconsistencies with the design implementation. In all of the tests below, the "Search all word forms" is turned off.
...
be & one -- (no quotes) 383 entries returned
"be" "one" -- (quotes used, ampersand was not used) 383 entries returned
--> If this is correct, the number of scriptures with "be" but not having "one" is 1071
--> Similarly, the number of scriptures with "one" but not having ""be" is 563.
--> The total number of scriptures with "be" or "one" (inclusive or) is 2017.

The issues above may be partially due to counting variations. It is possible for a scripture or reference to be double counted because of the way the search results are grouped.

For example, if you search for "horse", "rider" and "horse rider;" you will have 31, 5 and 28 results returned. The five scriptures with rider all have horse preceding. So, you might expect to have 26 scriptures (not 28) with just horse, but not rider. Two additional scriptures are counted because the reference groups Ex. 15:1, 19, 21 and Job:38:18-19 are split and reported in each query.

This could very well be happening with the "be" and "one" examples in the preceding post. The grouping/splitting of the references means that the counts cannot be trusted.

Thanks,
Dennis

dmaynes wrote:This could very well be happening with the "be" and "one" examples in the preceding post. The grouping/splitting of the references means that the counts cannot be trusted.

I wanted to verify if I was "crying wolf" so I found an easier example.

Turn off word variations.
Search for ("chain") You will have 13 returned results.
Search for ("chain" & "neck"). You will have 4 returned results. You can verify that these are correct.
Search for ("chain neck"). You will have no (0) returned results. I expected four results returned because neck always follows chain in this example.
Search for ("chain" & !"neck"). You will have no (0) returned results. I expected to see nine or more (because of the counting variations).

Thanks,
Dennis

dmaynes wrote:Is this working the way that it is intended to work? The scriptures that are returned will have the words "be" and "one" but sometimes there are intervening words.

tomw wrote:Yes, that is how it is designed.

With apologies to Tom, I must disagree. If the search were designed this way, it would be a horrible design (unlike any other phrase search I've ever seen) and completely inconsistent with the documentation regarding phrase searches. I'm quite confident that the scripture search is not designed this way. In most cases I have tried, a search for a quoted phrase will only find the phrase.

I think some incorrect conclusions were drawn because of the results of the "be one" search. I would speculate that "be one" allows for intervening words because "be" is a very common word and is excluded from the normal search if you search for it by itself. So I could see how spurious results might be returned based on this unindexed word (not that it should do this, but it might be a less tested code path, or a compromise based on index size or performance). But if you search for "chain neck" you will indeed find no matches, because that phrase does not exist in the scriptures. That is the correct behavior; it is the "be one" search that has questionable results.

I also think there is some mishandling of AND with NOT. Continuing with the chain/neck example, if we search with "search all word forms" turned off, we have:
chain = 13
chain neck = 4
chain !neck = 9

This is exactly what one would expect. But with AND, we have:
chain = 13
chain & neck = 4
chain & !neck = 0

This last result is unexpected and seems to be a bug.

Alan_Brown wrote:With apologies to Tom, I must disagree. If the search were designed this way, it would be a horrible design (unlike any other phrase search I've ever seen) and completely inconsistent with the documentation regarding phrase searches. I'm quite confident that the scripture search is not designed this way. In most cases I have tried, a search for a quoted phrase will only find the phrase.

I think some incorrect conclusions were drawn because of the results of the "be one" search. I would speculate that "be one" allows for intervening words because "be" is a very common word and is excluded from the normal search if you search for it by itself. So I could see how spurious results might be returned based on this unindexed word (not that it should do this, but it might be a less tested code path, or a compromise based on index size or performance).

I agree that the apparent forgiveness of intervening words in the design can be confusing. It appears that in actuality the design is either 0 (zero) or 1 (one) intervening words. As an example, consider "tempt the Lord thy God."

"tempt God" returns 5 results. Three of those have the exact phrase "tempt God" and the other two have one intervening word. But, "tempt the Lord thy God" is not returned with this search. (3 intervening words).

"tempt Lord God" returns 4 results. All of these are variants of "tempt the Lord thy God".

The engine doesn't seem to distinguish on what the intervening word is. If we return to the example of "be one" and 2 Ne. 9:12 the intervening word is "restored."

This is exactly what one would expect. But with AND, we have:
chain = 13
chain & neck = 4
chain & !neck = 0

This last result is unexpected and seems to be a bug.

It looks like a bug to me also.

dmaynes wrote:I agree that the apparent forgiveness of intervening words in the design can be confusing. It appears that in actuality the design is either 0 (zero) or 1 (one) intervening words. As an example, consider "tempt the Lord thy God."

The original statement was that "intervening words" would still allow a match, and that would not be acceptable in my opinion.

I guess it's a bit more reasonable to have a single intervening word, but this still seems odd, particularly if the intervening word is a significant word. I can see where it is reasonable that "forth fruit" would match "bring forth her fruit", since "her" is insignificant. But I have a hard time with a design that allows "forth fruit" to match "forth good fruit" "forth tame fruit" and "forth evil fruit". It allows for too many false positives, which may require you to sift through a lot of incorrect matches to find what you are looking for.

When I search for <<honor father mother>> with "search all word forms" checked, two references are returned: 1Ne. 17:55 and Mosiah 13:20. The Old Testament and New Testament references are not returned.

The reason they are not returned is because the Old and New Testament use the King's English and the word "honor" is spelled "honour" in the King James Version of the Bible.

What does it take to bind these two words together?

I would believe that spelling variations of the same word are definitely the same word form. There are a lot of these variations in the scriptures and it would be helpful if you did not need to know whether English or American spellings were being used.

As another example, consider "plow" and "plough." I wanted to find http://scriptures.lds.org/en/luke/9/62#62 Luke 9:62 -- "And Jesus said unto him, No man, having put his hand to the aplough, and blooking back, is fit for the kingdom of God." I couldn't remember the reference and I couldn't spell plough.

Thanks,
Dennis

Alan_Brown wrote:I also think there is some mishandling of AND with NOT. Continuing with the chain/neck example, if we search with "search all word forms" turned off, we have:
chain = 13
chain neck = 4
chain !neck = 9

This is exactly what one would expect. But with AND, we have:
chain = 13
chain & neck = 4
chain & !neck = 0

This last result is unexpected and seems to be a bug.

There is some sort of error with the NOT function and it does not necessarily require the AND (&) to be explicitly listed. As an example with "search all word forms" turned on (I don't think it matters for this example), search::

horse rider = 9 results, 10 references
horse rider heels = 1 result, 1 reference
horse rider !heels = 4 results, 5 references

This last result is also a bug, because 9 references should be returned.

dmaynes wrote:There is some sort of error with the NOT function and it does not necessarily require the AND (&) to be explicitly listed. .

Order of the search terms also matters with the NOT function.

myrrh returns 13 results (with 2 multi-verse results)
frankincense returns 12 results (with 3 multi-verse results)
myrrh frankincense returns returns 3 results (with 1 multi-verse result)
frankincense !myrrh returns 9 results (with 2 multi-verse results)
(!myrrh) frankincense returns 0 results --- This is an ERROR
frankincense myrrh returns 3 results (with 1 multi-verse result)
!frankincense myrrh returns 0 results -- This is an ERROR
myrrh !frankincense returns 10 results (with 1 multi-verse result)

Except for the errors when the NOT function is at the beginning, these results seem reasonable.

dmaynes wrote:There is some sort of error with the NOT function and it does not necessarily require the AND (&) to be explicitly listed. As an example with "search all word forms" turned on (I don't think it matters for this example), search::

horse rider = 9 results, 10 references
horse rider heels = 1 result, 1 reference
horse rider !heels = 4 results, 5 references

This last result is also a bug, because 9 references should be returned.

Actually, this is working properly. Your assumption that "search all word forms" wouldn't matter is incorrect. If you do this with "search all word forms" turned off, you get the obviously correct results:

horse rider = 5
horse rider heels = 1
horse rider !heels = 4

The help page for searching tells you that if you want to do any kind of advanced search, that the "search all word forms" setting is ignored. If you want to include word forms, you need to follow the search term with a tilde. Thus:

horse~ rider~ = 9
horse~ rider~ heels = 1
horse~ rider~ !heels = 8

These results all match the documentation and are internally consistent.

dmaynes wrote:Order of the search terms also matters with the NOT function.

myrrh returns 13 results (with 2 multi-verse results)
frankincense returns 12 results (with 3 multi-verse results)
myrrh frankincense returns returns 3 results (with 1 multi-verse result)
frankincense !myrrh returns 9 results (with 2 multi-verse results)
(!myrrh) frankincense returns 0 results --- This is an ERROR
frankincense myrrh returns 3 results (with 1 multi-verse result)
!frankincense myrrh returns 0 results -- This is an ERROR
myrrh !frankincense returns 10 results (with 1 multi-verse result)

Except for the errors when the NOT function is at the beginning, these results seem reasonable.

I'm not so sure this is a problem, either. The documentation says:

Use the symbols “ & ” (and), “ | ” (or), or “ ! ” (not) between words for Boolean searches (the words themselves cannot be used to designate Boolean searching).

Note that it says that these special symbols are to be used between words. By putting the ! (NOT) operator at the beginning of the search string you are breaking that rule. In my testing, the order doesn't matter at all, as long as you always put the NOT operator between words.

Alan_Brown wrote:Actually, this is working properly. Your assumption that "search all word forms" wouldn't matter is incorrect. ... The help page for searching tells you that if you want to do any kind of advanced search, that the "search all word forms" setting is ignored.

Ah, now I see. It is confusing that the default search mechanism of "search all word forms" is used when multiple words are provided as long as none of the special search symbols are used. But, when any of the special search symbols are present, the "search all word forms" is no longer allowed.

I'm not sure how to eliminate the confusion. The current search results indicate that "all word forms" or "all occurrences" were returned. This is a very subtle distinction and not likely to be noticed.

I appreciate your time. I assume if I have had trouble like this using the scripture search that others will have or have had trouble, also. I suspect that several of my posts have been confused because of this issue. I would suggest that if a user enters a query with multiple words that the search engine append the "`" special character to all the words if the "search all word forms" box is checked. In other words, you were able to get the expected behavior by explicitly using the special tilde character.

Alan_Brown wrote:Note that it says that these special symbols are to be used between words. By putting the ! (NOT) operator at the beginning of the search string you are breaking that rule. In my testing, the order doesn't matter at all, as long as you always put the NOT operator between words.

I'm wondering if the ! (NOT) operator is being parsed and handled as a binary logical operator. This would explain why forms like <<rider & !horse>> or <<!horse & rider>> fail. The only way that the ! (NOT) operator appears to work is if it is placed between the words, exactly as stated in the documentation. Any other placement appears to result in a search that fails. For example, "(Nephi | Lehi) & Jerusalem" succeeds, but "(Nephi | Lehi) & !Jerusalem" fails.

dmaynes wrote:I'm wondering if the ! (NOT) operator is being parsed and handled as a binary logical operator.

If you read the NOT (!) operator as "AND NOT" or "BUT NOT" you will get the correct search.

For example, if you want to find all the scriptures containing LAMAN but not LEMUEL, you would enter <<laman ! lemuel>>. If you want to find all the scriptures containing NEPHI and LEHI but not JERUSALEM, you would enter <<nephi & lehi ! jerusalem>>.

It makes no sense to say "but not JERUSALEM and NEPHI and LEHI." Therefore, this syntax is invalid.

Because the ! operator is read "BUT NOT," it makes no sense to combine "AND BUT NOT". It is already a conjunctive operator. Being a conjunctive operator, it is commutative with AND (as long as it is not first in the logical phrase). It is not commutative with OR. If you need to use an OR operator, you should use parenthesis.

So, <<pillar & cloud & ! fire>> is invalid because you cannot read find "pillar AND cloud AND BUT NOT fire."Search priority or precedence appears to follow a strict left-to-right ordering. If you want the ordering with precedence on the right, you need to use parenthesis. The search <<pillar ! fire | cloud>> is read "pillar BUT NOT fire OR cloud." It is performed strictly left-to-right. The verses are found (1) those containing pillar, then those with fire are removed and finally any additional verses with cloud are added. It cannot be reordered with parenthesis. The search <<pillar | cloud ! fire>> has two possible orderings: <<(pillar | cloud) ! fire>> (this is the default ordering) and <<pillar | (cloud ! fire)>>. This second ordering is the same as <<cloud ! fire | pillar>>, and is read "cloud BUT NOT fire OR pillar.

A mathematician would view these operators as set operations. While there exist equivalences between set and logical operators, the exact operation is important. In terms of sets the AND (&) operation is equivalent to the intersection operator, the OR (|) operation is equivalent to the union operator, and the BUT NOT (!) operation is equivalent to the set subtraction operator.

Thanks,
Dennis

Alan_Brown wrote:With apologies to Tom, I must disagree.

No need to apologize as I may have misread or misunderstood the question. I'll forward this thread on to the developers, however.

Tom

dmaynes wrote:If you read the NOT (!) operator as "AND NOT" or "BUT NOT" you will get the correct search.

For example, if you want to find all the scriptures containing LAMAN but not LEMUEL, you would enter <<laman ! lemuel>>. If you want to find all the scriptures containing NEPHI and LEHI but not JERUSALEM, you would enter <<nephi & lehi ! jerusalem>>.

It makes no sense to say "but not JERUSALEM and NEPHI and LEHI." Therefore, this syntax is invalid.

Because the ! operator is read "BUT NOT," it makes no sense to combine "AND BUT NOT". It is already a conjunctive operator. Being a conjunctive operator, it is commutative with AND (as long as it is not first in the logical phrase). It is not commutative with OR. If you need to use an OR operator, you should use parenthesis.

So, <<pillar & cloud & ! fire>> is invalid because you cannot read find "pillar AND cloud AND BUT NOT fire."Search priority or precedence appears to follow a strict left-to-right ordering. If you want the ordering with precedence on the right, you need to use parenthesis. The search <<pillar ! fire | cloud>> is read "pillar BUT NOT fire OR cloud." It is performed strictly left-to-right. The verses are found (1) those containing pillar, then those with fire are removed and finally any additional verses with cloud are added. It cannot be reordered with parenthesis. The search <<pillar | cloud ! fire>> has two possible orderings: <<(pillar | cloud) ! fire>> (this is the default ordering) and <<pillar | (cloud ! fire)>>. This second ordering is the same as <<cloud ! fire | pillar>>, and is read "cloud BUT NOT fire OR pillar.

A mathematician would view these operators as set operations. While there exist equivalences between set and logical operators, the exact operation is important. In terms of sets the AND (&) operation is equivalent to the intersection operator, the OR (|) operation is equivalent to the union operator, and the BUT NOT (!) operation is equivalent to the set subtraction operator.

Thanks,
Dennis

So by using Computer Science jargon in an effort to make this explanation quite a bit shorter, the NOT operator (!) is implemented as a binary logical operator and not as a unary logical operator.

Of course, this is what is trying to be explained to non-technical folks by saying you must put the operator between words, as found in the help documentaton.

tomw wrote:This forum will do fine. Thanks for pointing it out. I'll forward over to the team responsible for the online scriptures.

Here's the search URL: http://scriptures.lds.org/en/search?typ ... ecked&bw=1

This search for <<mary virgin>> should return two scriptures (Luke 1:27 and Alma 7:10). It doesn't always work. I printed the search results page as a PDF file and I am attaching it.

I have gotten the search to fail by toggling the "Sort by relevance" box.

Here's a test session:
1- Start at http://scriptures.lds.org
2- Enter <<virgin mary>> in the search box and press Enter
3- Failed
4- Toggle "sort by relevance" Failed
5- Toggle "sort by relevance" Succeeded
6- Toggle "sort by relevance" Succeeded
7- Toggle "sort by relevance" Failed
8- Toggle "sort by relevance" 4 times, Succeeded each time
9- Toggle "sort by relevance" Failed

I don't know why this is failing.

Could this be related to my browser? Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5

Thanks,
Dennis

dmaynes wrote:This search for <<mary virgin>> should return two scriptures (Luke 1:27 and Alma 7:10). It doesn't always work. I printed the search results page as a PDF file and I am attaching it.

Could this be related to my browser? Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5

I can confirm this erratic behavior for both Firefox 3 and IE7.

dmaynes wrote:This search for <<mary virgin>> should return two scriptures (Luke 1:27 and Alma 7:10). It doesn't always work. I printed the search results page as a PDF file and I am attaching it.

Could this be related to my browser? Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5

I can confirm this erratic behavior for both Firefox 3 and IE7.

Alan_Brown wrote:I can confirm this erratic behavior for both Firefox 3 and IE7.

Thanks - I wasn't sure if it would be reproduced. The error is specific to the word "mary." It does not reproduce with other words that I tried.

Failure counts (20 toggles of "sort by relevance")

<<mary>> 5
<<mary virgin>> 15
<<virgin mary>> 2
<<mary born>> 5
<<born mary>> 7
<<virgin>> 0
<<born>> 0
<<virgin born>> 0
<<born joseph>> 0
<<joseph mary>> 0 (upon repeat, I did see failures)
<<mary jesus>> 2
<<jesus mary>> 13

1. The problem is associated with the word "mary"
2. "Sort by relevance" does not matter
3. Because of the variability and runs of failures, the trials are not independent (i.e., it's not stateless; there appears to be a service state that causes this to happen; this is probably related to a caching mechanism). I did a first-order Markov analysis on 90 trials and the test of independence failed with a chi-square value of 18 and 1 d.f. (probability < .0005)
4. All of the above tests were performed with "search all word forms" checked.
5. The problem reproduces with "search all word forms" not checked.
6. I believe that time between searches interacts with the runs of failures and successes. The longer the time interval between searches, the failures and successes appear to have greater independence. This makes sense if the failures are stored in the service cache.

Thanks,
Dennis

Nice work! I've alerted the developers.

Tom

tomw wrote:Nice work! I've alerted the developers.

It didn't seem reasonable that "mary" was the only word where the search would have trouble.

I found several additional words. Please note that they do not all appear to fail with the same frequency. I had to click "Search" for merit a lot of times before the failure reproduced. Only the word "merit" fails (with 0 results) or succeeds (with 8 results) like "mary."

I compiled this list by looking at words in the Topical Guide under "M" and searching them. I eventually started looking at only "short" words because those appeared to be the only ones that failed.

magic - inconsistent results: magics Morm. 1:19 is not always displayed. - returns 1 or 2 results.
"Search all word forms" is checked.
Toggle "Sort by relevance" twenty times.

maid - returns 24 or 31 results.
"Search all word forms" is checked.
"Sort by relevance" is not checked.
Click "search" twenty times.

man - returns 1071 or 1070 results.
"Search all word forms" is checked.
"Sort by relevance" is not checked.
Click "search" twenty times.

meat - returns 158 or 165 results.
"Search all word forms" is checked.
"Sort by relevance" is not checked.
Click "search" twenty times.

melt - returns 29 or 31 results.
"Search all word forms" is checked.
"Sort by relevance" is not checked.
Click "search" twenty times.

merit - returns 0 or 8 results.
"Search all word forms" is checked.
"Sort by relevance" is not checked.
Click "search" twenty times.

Maybe the examples where the number of results changes will help identify the problem, if it is the same problem.

Thanks,
Dennis

tomw wrote:This forum will do fine. Thanks for pointing it out. I'll forward over to the team responsible for the online scriptures.

Tom

Interesting inconsistencies exist because there are two different search engines for searching the scriptures:

1-- Search from the Scriptures home page (http://scriptures.lds.org)
2-- Search from the main website page (http://www.lds.org), with a "filter" on scriptures

I haven't explored this thoroughly, but one such inconsistency is in the way stemming works.

Scripture search for <<treasure heart>> from the scripture search engine and 10 references are returned.

Scripture filter for <<treasure heart>> from the lds.org search engine and 11 references are returned. (You have to filter by the New Testament, Book of Mormon, and Doctrine and Covenants. Filter by All Scriptures returns Topical Guide hits, also.)

The extra reference from the lds.org search engine is Romans 2:5 -- "But after thy hardness and impenitent heart treasurest up unto thyself wrath against the day of wrath and revelation of the righteous judgment of God;"

The word "TREASUREST" is returned in the lds.org search engine, it is not found using the scripture search engine.

If you search for <<TREASUREST>> (no quotes) using the scripture search engine, Romans 2:5 is the only reference returned. This is because the word "treasurest" is not recognized to have the same word form as "treasure."

Is the development team interested in knowing about these stemming inconsistencies? Is there an appropriate way to report them?

Thanks,
Dennis

dmaynes wrote:It didn't seem reasonable that "mary" was the only word where the search would have trouble.

I found several additional words. Please note that they do not all appear to fail with the same frequency. I had to click "Search" for merit a lot of times before the failure reproduced. Only the word "merit" fails (with 0 results) or succeeds (with 8 results) like "mary."

This problem now appears to be resolved. It is not reproducing for me. I had the problem with the word "hen" about a week ago, so the fix seems to be relatively recent. Good job!

Thanks,
Dennis

tomw wrote:This forum will do fine. Thanks for pointing it out. I'll forward over to the team responsible for the online scriptures.

Tom

I admit to being somewhat contrary at times. I wanted to see if the exact phrase search would span references. It does not. However, I did find two problems.

If you search for the words <<what desired thou father>> with search all word forms turned off, 4 references will be returned.

Here's the URL http://scriptures.lds.org/en/search?type=words&last=what+desired+thou+father&help=&wo=checked&search=what+desired+thou+father&iw=scriptures&tx=checked&hw=checked&sw=checked&bw=1

The problem is that the words <<thou>> and <<what>> appear to be ignored. The search should return no references. Is it desirable to ignore some words in the search? If words are ignored, it seems like the proper feedback to the user is that the words <<thou>> and <<what>> are frequent words and were ignored in the search. I could make an argument for words like <<and>>, <<or>>, and <<the>>, but I don't think these two words should be ignored.

By the way, the search from http://www.lds.org (restricted to The Old Testament) returns Daniel 2:23 with this search. The other references are not returned. The three words <<desired>>, <<fathers>>, and <<thou>> are highlighted. The word <<what>> does not appear in the reference. It is not highlighted. The http://www.lds.org search engine also appears to be ignoring this word.

The next problem appears to be related to the stemming (Search all words forms) function. If you search for the words <<desirest father>> with search all word forms turned on, no references will be returned.

http://scriptures.lds.org/en/search?search=desirest+father&do=Search

However, Daniel 2:23 should be returned because <<desirest>> is a word form of <<desire>>. The scripture search function from http://www.lds.org finds Daniel 2:23 because it places <<desired>> and <<desirest>> in the same word form group.

Thanks,
Dennis

dmaynes wrote:I admit to being somewhat contrary at times.

Here's more contrariness. If you search for the words <<what did our father do when we were upon it>> you will have 759 hits returned.

http://scriptures.lds.org/en/search?type=words&last=desired+father+things+it&help=&wo=checked&search=what+did+our+father+do+when+we+were+upon+it&do=Search&iw=scriptures&tx=checked&af=checked&hw=checked&sw=checked&bw=1

The problem is that a lot of little words are being ignored in the searches. It seems problematic that some of these words are ignored. Here's a partial list that I was able to discover. I'm sure there are more words.

who
thou
what
when
why
did
but
our
we
were
are
be
for
upon
on
is
was
it

If you include any of these words in your search, they are currently ignored.

Thanks,
Dennis

dmaynes wrote:The problem is that a lot of little words are being ignored in the searches. It seems problematic that some of these words are ignored. Here's a partial list that I was able to discover. I'm sure there are more words.

The search I have been using is the three words <<Jesus wept STOP-WORD>>. I am assuming the mishandling of these little words is due to their presence on the stop-word list of the search engine. I don't have a great recommendation on how to handle these stop words, but I find it strange that the search for the words <<Jesus wept when you were there>> would return John 11:35 "Jesus wept." Here is a more complete list of words where this behavior is demonstrated:

a
about
all
also
an
and
any
are
as
at
be
but
by
can
did
for
from
get
has
have
he
her
hers
him
his
how
if
in
is
it
its
may
me
my
not
of
on
or
our
out
shall
she
so
that
the
thee
them
their
there
they
this
thou
thy
upon
us
was
we
were
what
when
who
why
with
ye
you
your

It seems that the specific algorithm for dealing with the stop words needs to be carefully evaluated. There seems like there are three ways to deal with this problem: (1) Post-process all returned lists after the stop words are eliminated, (2) Treat stop words like any other word, or (3) Implement an alternate indexing probe function for the stop word list.

I do not believe that the behavior of ignoring stop words in the query without any form of user feedback is good. It causes a lot of false positives to be returned, or some confusion on the part of the user.

Another possibility is to follow the current procedure. If a group of stop words only are provided to the search (such as <<out there>>) the returned page states:

The words OUT and THERE are not significant words and were not used in the search.

If words are going to be ignored, then the user should be told which words are being ignored.

Thanks,
Dennis

tomw wrote:This forum will do fine. Thanks for pointing it out. I'll forward over to the team responsible for the online scriptures.

Tom

There is something strange with the search engine's interpretation of searching for <<can not>>.

If I search for <<Jesus wept can not>> I get a list of 264 hits. http://scriptures.lds.org/en/search?typ ... ecked&bw=1

If I search for <<Jesus wept>> I get a list of 3 hits. http://scriptures.lds.org/en/search?typ ... ecked&bw=1

Why would I get more hits when I add the words <<can not>>?

If I search for <<Jesus~ wept~ can~ not~>> I get 0 hits. http://scriptures.lds.org/en/search?type=words&last=jesus+wept&help=&wo=checked&search=jesus~+wept~+can~+not~&do=Search&iw=scriptures&tx=checked&af=checked&hw=checked&sw=checked&bw=1

And, that is correct.

If I search for <<Jesus wept cannot>> I get 0 hits. http://scriptures.lds.org/en/search?type=words&last=jesus~+wept&help=&wo=checked&search=jesus+wept+cannot&do=Search&iw=scriptures&tx=checked&af=checked&hw=checked&sw=checked&bw=1

And, again this is correct.

The problem seems to be with the "stop words" of "can" and "not," some automatic combining of "can" and "not", which results in the discard of "Jesus" and "wept" from the search.

I don't think I have done anything wrong, but it sure seems strange.

Thanks,
Dennis

Perhaps it detects "not" as a keyword and tries to implement some additional logic.

mkmurray wrote:Perhaps it detects "not" as a keyword and tries to implement some additional logic.

I don't believe that it is an error introduced by trying to implement search logic.

Instead, it appears to be some mishandling in the processing of the stop words.

Here are additional observations.

If you are searching for "all word forms" the words "can" and "not" get combined together as the word "cannot" and other words appear to be ignored.

Example: <<fountain>> returns 38 hits: http://scriptures.lds.org/en/search?type=words&last=not+can+fountain~&help=&wo=checked&search=fountain&do=Search&iw=scriptures&tx=checked&hw=checked&sw=checked

<<fountain cannot>> returns 1 hit: http://scriptures.lds.org/en/search?typ ... ecked&bw=1

<<fountain not can>> returns 38 hits: http://scriptures.lds.org/en/search?typ ... ecked&bw=1 (because "not" and "can" are ignored in the search - I reported this bug earlier in the thread)

<<fountain can not>> returns 297 hits: http://scriptures.lds.org/en/search?typ ... ecked&bw=1 (the word "fountain" appears to be ignored)

<<fountain~ can not>> returns 0 hits: http://scriptures.lds.org/en/search?type=words&last=fountain+can+not&help=&wo=checked&search=fountain~+can+not&do=Search&iw=scriptures&tx=checked&hw=checked&sw=checked&bw=1 (this is correct -- note the tilde affixed to the word "fountain" says "search all word forms" but it disables some sort of parsing function that is involved with the stop words. It is this parsing function that appears to have the problem.)

Thanks,
Dennis

The online scriptures have the same error as my hard copy version, which has been corrected in more recent print editions.

D&C 124:99b links to Isaiah 40:3, but should link to Isaiah 40:31