tianon wrote:I realize this is a total necro-post (is it really 2011 already?), and I apologize, but I really just had to laugh here. A few hundred or more? We're talking about powers of two. Every time I go back one more generation in my personal tree, the number of people added is double the previous generation (two parents each), not to mention children, and I'd be extremely interested in caching at least 10 generations worth. Ideally, the entire tree could be cached, but that's not necessarily feasible.
Since we're talking about basically plain-text data here, it shouldn't be any kind of an issue to save 20, 30, or even 50 generations worth and still not have any sizable amount of data (well under a gigabyte).
(Some of the moderators frown on resurrecting old threads, but this one's on topic, so I don't see a problem in this case.)
In the question of how much data it would make sense to cache, issues come up concerning how long a time you want to cache the data, the level of likelihood you are willing to accept that the data may become stale, and how much resource it takes to fetch the data.
I'm pretty confident it would not be practical for the server to track which clients have which data cached, so there's no practical way to do server-push coherency. That leaves it up to the client to re-fetch the data to check for staleness.
The likelihood of any person's cached record being stale increases with time since the data was fetched. The likelihood of there being stale data in a cache increases with the number of records in the cache. On modest time and size scales, I would think both effects would be roughly linear, so the likelihood of a cache having stale data would grow with the product of the time since fetch and the number of person records held.
It takes a lot of resource to fetch data from a large store. One observation is that a program called GetMyAncestors takes several minutes to download summary data on four generations, 400-450 people. Once you go back far enough, the tree becomes a DAG once you become your own distant cousin (same ancestor on different branches), but that effect is still rather small at 10 generations back, so 10 generations would be around 25k-29k people. If fetch time scales linearly, that would take several hours.
Let's say, hypothetically, that nFS store was optimized for fetching summary data (which would be very inadequate for serious work) rather than detailed data. I don't know what database technology nFS is using (other than it has been stated they use Linux as the OS and Java as the programming language), so I'll extrapolate from what I have seen with a hash-based key-value store on a (relatively small) six-drive RAID6. With rotating magnetic disks, the main factor governing fetch speed is disk seeks. I have seen a disk array that can handle 400MB/s sequential access drop to around 2-4MB/s when reading widely scattered small blocks. With the K-V store I worked with, once the index outgrew RAM, lookups dropped to around a thousand lookups per second. Even if you're caching only summary data, and even if we assume there's a couple thousand dollars worth of data center resources per simultaneously active user, it would take a long time to fetch 1GB of data--and that's if the client had enough downstream ISP bandwidth. SSDs will eliminate essentially all the seek time, but SSDs are still rather expensive per TB, and the client ISP bandwidth would still be a bottleneck.
I still think caching a few hundred, or at most a thousand or two, people is much more practical than caching 10 generations, let alone 20-50 generations. Also, I think that if advances in speed would make it practical to cache 10 generations or more, the same advances in speed would make real-time fetch at least fast enough, which would cancel the need for caching.
