-
@ mleku
2025-04-27 11:24:43
#realy #devstr #progressreport
yesterday i spent a lot of time writing a novel (i think - it's the reverse of regular variable length integer encodings in that it uses the 8th bit as a terminal marker instead of one on each byte to indicate "there's more") variable integer encoder to write a binary encoding for use in the database that is written using the #golang io.Writer/io.Reader interface (so it can be used to stream the binary data over wire or to disk)
i figure the way the varint encoder works it's just a little faster because it only adds the 8th bit to the last byte so that's potentially as much as 1/7th of this operation, not huge but ok, anyway it was required for the streaming read/decode anyway, to make a new one.
it's not on by default, you have to set BINARY in the environment to make it do this, and there is no handling for this - if binary false, it tries to decode json, and if true, binary, so to change over, you need to export the events, nuke or just delete the database files ( ~/.local/share/realy ) and flip it to true and then import the events back (and if you deleted, you want to re-set your configuration, you really should keep a copy of the configuration json that gets stored in the database for reasons like this).
it's definitely noticably faster, the binary encoder i am quite sure will prove to be the fastest nostr database binary encoder, and it's also extremely simple, and the results are about a 30% reduction in event storage size, and maybe 1 microsecond to perform the encoding.