Supporting Managed Content

In a release after 1.0 we want to be able to support all the managed content that the church has or will have available. This will probably mean using the Web Service (https://tech.lds.org/wiki/index.php/Gospel_Library_Web_Service)

However, we will probably take advantage of the ideas that the iPhone team came up with for solving this. Steve Liddle and I had a short discussion on this at the LDSTech Conference. I believe the iPhone team uses what they call zbooks. It is basically a SQLite database that has been zipped. They then download it, unzip, open the downloaded book db and import the records into the main db.

What other ideas do you have?

The problem with inserting everything into the main db is that sqlite on Android is very prohibitively slow inserting/deleting (or at least it used to be). I would prefer to keep each book in a separate DB and use the the uri prefix as a key to which database to look in.

I agree that there would definitely be some benefit to keeping the books in separate files.

We need to look at how that would affect search. Perhaps once we can safely move to Android 1.6+ (or if we can selectively turn on and off features depending on whether they are supported in the version of Android running on the device) we could harness the native search capabilities in order to give us more flexibility in the storage of the books.

I haven't looked closely at how the lucene index is currently tied back to the DB (lucene doesn't care, you can store any key that makes sense for the application), but I'm guessing it is by the URI, in which case selecting an individual db by URI prefix would just work. And lucene separate lucene indexes are easily "joined" at search time.

The Lucene index is indeed isolated from the database by URI. We can separate the two entirely if we'd like. Also, Lucene has a capability called a "multi-index searcher" that lets us search multiple separate indexes in a single query. I don't know how well that will scale under Android. If it scales, it would be easiest to create one Lucene index for each document database we create. I suspect that any more than a few Lucene indexes will start to slow down fairly quickly.

I'd prefer to group the document databases into major categories that we download as a bundle (e.g., Sunday School, Young Women, ...). If it really is a requirement that we download individual documents, I'm guessing we'll need to combine the Lucene index files for performance reasons anyway. But that's an empirical question -- we'll need to try it and measure performance.

I think the DB insert on Android issue is related to building indexes. I'll bet it's significantly faster if we disable indexes until after new rows are inserted. But I agree that it should be fine to store documents in their own databases according to URI prefix. We'll just need to store sufficient table-of-contents information in the main database so we can navigate to all the installed components.

Another question is whether we even want to support comprehensive full-text search on the device. We could limit search to the scriptures, or we could allow search only via the web using Stewart's online search service. It's cool to be able to do local search, but if it ends up to be a big hassle, maybe our time is better spent on other things first.

I like the idea of comprehensive search but it may indeed be best to use the web service search. However, the web service has the downside that it would be unavailable where there is no connection to the web service. That would happen under the following scenarios:

• Airplane mode
• Internet tablets (i.e. wifi-only devices) away from wifi
• Phones in rural areas or other areas with spotty coverage
• "Third-world" areas

Could we possibly use the web service to capture search index information and use it in a cached mode to allow us to search using local indexes?

Probably would be much easier to continue down the Lucene path rather than cache material coming from the web service. The issue with Lucene is the amount of storage required for its indexes. Personally, I have plenty of space I'd be happy to allocate to those indexes (and I'd much rather have the local/offline capability). What I wonder about is whether typical users would rather save the space and use the online search, or consume the storage space and have offline search. We could add analytics capabilities to the GL app to see what features the folks actually use.

swliddle wrote:What I wonder about is whether typical users would rather save the space and use the online search, or consume the storage space and have offline search.

Could you make that a user preference? That way for those who have the space the offline advantages could be used, but for those who are tight on space they would still be able to use the app relatively well.