Statistically Linked Verses / Passages

I think it would be great to use statistics and/or machine learning to automatically create verse links with associated probabilities.

There are a lot of way this could be done, from simple to complex. One simple way would just be using a naive Bayes or "Bag of Words" approach: I click on a word, and then the program finds other verses that have that same word in it, but also have similar surrounding words.

These aren't my ideas alone, I give credit to Harold Stuart, the author of HandyScriptures for first suggesting linking scriptures by their base languages (Hebrew / Greek / etc), and to another friend of mine I won't mention without his permission who also suggested we could detect an infinite number of scripture topics using text analytics combined with some non-parametric Bayesian modeling.

In short, the latter approach would automatically lead to topics growing out of the text. The number of topics and size can be determined automatically.

Anyways, I just think it would be really nice, and the simpler approaches are not hard if you have an index.

I'm intrigued--can you provide some use cases for what you're suggesting? BTW, you might be interested to know that the new search on LDS.org is driven by a commercial Bayesian engine.

The high level goal is that you click on a word in a verse and get returned to you a list of verses sorted by relevance. Relevance in this case is a measure of how much those other verses use that word in a similar context.

One simple example could be clicking on "commandments" in 1 Nephi 3:7 brings up a verse like 1 Nephi 17:3 first instead of "TG Commandments of God".

It detects the context of "commandments" in this case refers to the Lord's promises to help you fulfil them, and first returns verses using commandments in that context.


A higher end use case would be to return an entire hierarchial set up topics automatically. As a simple example, lets say I model the scriptures using a hierarchical latent dirichlet process. It returns to me first the words "faith, repentance, baptism"

I click on faith and it returns to me sub-topics under faith. Those sub-topics have other sub-topics, etc.

The cool part is that the topics are all auto-matically created using a state-of-the-art statistical modeling procedure.

Do you think this would be a substantial improvement over the Topical Guide? Granted, it would be more exhaustive, but would it uncover relationships that are not well-understood already? One problem that we've found with bayesian inference is that the vocabulary of Church content is relatively narrow, i.e., the ratio of rare (information-carrying) words is low compared to common words. Getting a meaningful taxonomy from such a limited vocabulary using statistical/probability approaches is challenging. Still, I would be interested in seeing a prototype of what you're suggesting.

In the end, of course, an examination of the scriptures should be more a spiritual journey than a mathematical one.

shellinese wrote:Do you think this would be a substantial improvement over the Topical Guide? Granted, it would be more exhaustive, but would it uncover relationships that are not well-understood already?

I personally think it would uncover "additional witnesses" that would help people understand concepts a little more. Of course I don't think it would replace inspiration.

The idea is just to make it so when you click a word, you get a more relevant set of results.

shellinese wrote: One problem that we've found with bayesian inference is that the vocabulary of Church content is relatively narrow, i.e., the ratio of rare (information-carrying) words is low compared to common words. Getting a meaningful taxonomy from such a limited vocabulary using statistical/probability approaches is challenging. Still, I would be interested in seeing a prototype of what you're suggesting.

I think we could get around this. One way is to have a predefined set of words that we "care" about, the rest we ignore. This predefined vocabulary could be generated as simply as just using those words that have topics in the Topical Guide, Bible Dictionary, or Index.

Here's a simple, brute force approach:

1. Take a complete verse-based index of the standard works, cull out all but the "interesting vocabulary" and save that.

2. When someone clicks on "faith", grab all of the verses from my index that have "faith" in them. For each of those verses, count the number of "interesting words" that occur in both the original verse and the verse being examined. Then, sort descending on that number and return the list.

The hardest parts here are:
1. Obtaining the "list of interesting words" (Topical Guide / Bible Dictionary / Index)
2. Obtaining a verse-based index of the standard works.

Again, this is the simplest approach I can think of, there are far more fancy ones (latent dirichlet allocation, etc) that we could get into down the road.


As for a prototype, I don't know when I'll have time to do one, especially working alone. I want to though.

If anyone else here is willing to help, this could be fun.

Josh,

I only took the AI class at BYU, not the Machine Learning. However, I'd love to help how I can. Perhaps I can get Logan interested in this too. This could be part of his interview process with you guys. It'll be like those little problems Microsoft gives you when you interveiw!

OK, I wrote a prototype, but it runs from the console.

I want to make a little web app out of it eventually, but for those of you who want to try it sooner:

Give me a verse and a phrase or word from that verse, and I will paste the search results here.



Verse anyone?

josh wrote:OK, I wrote a prototype, but it runs from the console.

I want to make a little web app out of it eventually, but for those of you who want to try it sooner:

Give me a verse and a phrase or word from that verse, and I will paste the search results here.



Verse anyone?

2 Ne. 31:19
"mighty to save"

Deut. 6:13
"fear the Lord"
...or should it be "fear the LORD" because of the small caps?

Here's one from a friend of mine:

Alma 32:21 on "faith"

Ether 12:6 Scores: 6.58823669484
And now, I, Moroni, would speak somewhat concerning these things; I would show unto the world that faith is things which are hoped for and not seen; wherefore, dispute not because ye see not, for ye receive no witness until after the trial of your faith.

Alma 7:17 Score: 6.47403262293
And now my beloved brethren, do you believe these things? Behold, I say unto you, yea, I know that ye believe them; and the way that I know that ye believe them is by the manifestation of the Spirit which is in me. And now because your faith is strong concerning that, yea, concerning the things which I have spoken, great is my joy.

Jacob 7:5 Score: 5.75066079295
And he had hope to shake me from the faith, notwithstanding the many revelations and the many things which I had seen concerning these things; for I truly had seen angels, and they had ministered unto me. And also, I had heard the voice of the Lord speaking unto me in very word, from time to time; wherefore, I could not be shaken.

mkmurray wrote:2 Ne. 31: 19

"mighty to save"

Here are the top 3 with their "scores"

Alma 7:14 score: 10.0421240624
Now I say unto you that ye must repent, and be born again; for the Spirit saith if ye are not born again ye cannot inherit the kingdom of heaven; therefore come and be baptized unto repentance, that ye may be washed from your sins, that ye may have faith on the Lamb of God, who taketh away the sins of the world, who is mighty to save and to cleanse from all unrighteousness.

Doctrine and Covenants 133:47 score:3.87831884748
And he shall say: I am he who spake in righteousness, mighty to save.

Alma 34:18 score: 3.55735206572
Yea, cry unto him for mercy; for he is mighty to save.

Another one mkmurray gave me over IM:

http://scriptures.lds.org/en/deut/6/13#13

"fear the Lord"

Deuteronomy 10:20 Score: 7.48286700798
Thou shalt fear the LORD thy God; him shalt thou serve, and to him shalt thou cleave, and swear by his name.

By the way, on that last one (Deut 6:13), that was the top score out of 74.

Just for fun, here is Alma 32:21 with "faith" again, but this time using "bigrams" or in other words, considering the words that come before or after each keyword. Notice the verses are a little different.

Ether 12:6.txt 10.5882366948
And now, I, Moroni, would speak somewhat concerning these things; I would show unto the world that faith is things which are hoped for and not seen; wherefore, dispute not because ye see not, for ye receive no witness until after the trial of your faith.

Moroni 10:23.txt 10.4507441362
And Christ truly said unto our fathers: If ye have faith ye can do all things which are expedient unto me.

Alma 32:26.txt 9.85455411358
Now, as I said concerning faith--that it was not a perfect knowledge--even so it is with my words. Ye cannot know of their surety at first, unto perfection, any more than faith is a perfect knowledge.

Prototype available here:

http://menkefamily.no-ip.org

Not very fancy though. You have to know the verse ahead of time, no fancy ajax throwing it up for you as you type yet.

Help me understand how this is qualitatively superior to the search on LDS.org, which returns the following three scriptures when given the term "faith" (after filtering out the non-scripture results):

Alma 44:4
Now ye see that this is the true faith of God; yea, ye see that God will support, and keep, and preserve us, so long as we are faithful unto him, and unto our faith, and our religion; and never will the Lord suffer that we shall be destroyed except we should fall into transgression and deny our faith.

Romans 1:17
For therein is the righteousness of God revealed from faith to faith: as it is written, The just shall live by faith.

James 2:18
Yea, a man may say, Thou hast faith, and I have works: shew me thy faith without thy works, and I will shew thee my faith by my works.

First, the usage is different.

When you type "faith" into lds.org's search, you want general results on faith. Perhaps the most "relevant" or common results from a "PageRank" point of view.

Here, the usage is instead you reading the scriptures or a verse like Alma 32:21 and wanting to see other verses that have "faith" in them that are similar to Alma 32:21. That's why the first verse returned (Ether 12:6) is more "similar" to Alma 32:21 than Alma 44:4 is.

It's more like an automatic cross-reference. It tries to maintain the same context a the verse you started in.

Second, qualitative results are hard to judge. If you wanted a verse more like Alma 32:21, then the search I posted seems more "relevant" to me. If you want general scriptures on faith, then it would be hard to argue either way.

So in summary for usage:

1. General topic search: lds.org is great.
2. Word or phrase search based on a given verse: Nothing exists for this yet except my prototype (that I know of).

Follow over to [thread=208]this thread[/thread] for the prototype.