-
@ Mike Dilger
2025-04-03 20:58:58
When you want to find a subset of events in a large database of events, but you know that subset is very large and may cause a memory problem, there are a number of strategies to deal with this situation:
1) Iterator: Redesign your fetch function as an iterator, to remember where it left off and keep giving the next Event. Then you only have one event in memory at a time. This is very trad.
2) Screen: Add a screening function to your fetch function. You usually don't actually need all those events, and the screen function can reduce the result set by a lot.
3) Reference: If those events are mmapped, you can just pass a reference to each one where it sits, with no copys, and the OS will page in the data as accessed.
Gossip uses method (2) and I'm toying around with method (1). Pocket (and therefore Chorus) use method (3).