-
@ Counter Narrative
2023-10-28 02:28:11
As I continue to work on the concept of storage relays (see my last post about this on my profile) I have come to think about some of the strengths and weaknesses of current storage solutions both on Nostr and in general and how my humble project seeks to draw from the parts I think are useful.
## Some of the issues with decentralized data storage
A lot of people have spoken about solutions to decentralized storage in terms of using the following:
* IPFS
* Torrents
* Multiple file hosts
* Nostr relays
* P2P solutions
### IPFS
I never really got IPFS working and Fiatjaf has a great series of posts outlining his experiences with IPFS not functioning. It seems that unless you pin your data or have it up for a long time, you’re not going to be able to share much to many people. Of course, others seem to have better luck hosting and retrieving data and maybe I am not using it correctly.
#### Upside of IPFS
I think the idea of doing away with the concept of URLs (mostly) and locating data based on some inherent feature of the data might be a better way to interface with data on the Internet. This is especially true for a system that works to be decentralized.
Users don’t REALLY care about what directory structure a file lives in on another computer but rather just want to retrieve that file.
This is where I think IPFS’s concept makes sense.
#### Downside of IPFS
There seems to be a lot of overhead needed to interface with the IPFS network, something that I doubt Nostr clients or webpages would be able to incorporate. A workaround is using proxy servers that address the data in legacy URL structure and bring back the centralized point of failure.
I also personally never retrieve something I want over the network unless it’s something that has been around for ages.
### Torrents
Torrents always seem to be brought up around the discussion of decentralized storage. Torrents work great for popular data and for large data sets. It also has a bad reputation (because a lot of that popular data are movies and music).
#### Upside of Torrents
The idea of magnet links and descriptor files to piece together data also moves away from the URL structure of pulling down files over the Internet. Being able to pull in chunks of data from multiple sources that individually may be fleeting and unreliable on the network makes torrents work well.
#### Downside of Torrents
If you ever tried to download an obscure movie or ebook from a torrent you’ll see that torrents work great with popular data and not so well with lesser consumed things. It also doesn’t scale down very well if you want to post a small file or video.
The overhead for torrenting also requires implementing a torrent client which may not scale down for social media and other casual data transfers.
### Multiple file hosting services
Particularly on Nostr there are a few great people providing image and data hosting services with freedom technology in mind. This has helped make Nostr user friendly with media as many clients integrate image and video uploading with these services right in their client.
Additionally having these files uploaded in multiple places and potentially having metadata seek the file from either one of these sources can help with redundancy and central points of failure issues.
#### Upside of file hosting services
These services are stable and easily integrated into the web and other Nostr clients. The people who run these services have privacy and freedom in mind when running their services.
#### Downside of file hosting services
These services are individually centralized and also do not have a common API across each of them to generically interface with. There are also failure points both from technology and legal standpoints where a service could be shut down.
There’s also a trust issue broadly that the file you upload will be the one that users will access without tampering. This is true whether there is a redundancy setup across multiple hosts or not.
### Nostr relays
Nostr Relays are a thought that many, including myself have thought about for hosting data beyond text. Perhaps having a special Kind for just data or embedding the data within a note. Having notes that are cryptographically signed and stored provides a level of attestation to the data someone would post and make it tamper resistant.
#### Upside of Nostr
The simple concept of Nostr and notes being addressable in multiple ways (nevent, note, naddr, etc) makes things both consistent and tamper resistent. With the number of relay operators out there, the network effects of both users and operators would make it a strong choice.
#### Downside of Nostr
Nostr searches give you everything, really EVERYTHING. Embedded data being pulled in on a search would put considerable strain on both the client and the relay operator.
Subjectively I also don’t think that relay operators who already house a lot of nostr events want to also arbitrarily handle medium to large data by default.
Custom relays seem counter to the protocol specifications and also seem a little hacked together. Nostr may not need to be for everything, including non-text data storage.
### P2P solutions (broadly)
Peer-to-Peer solutions have worked well for many use cases of sharing data. Software like Keet allowed me to share a large video file with friends in very little time.
#### Upside of P2P
Being able to get data to and from someone directly cuts out censorship. In many cases things can be fast and generally reliable.
On larger P2P networks, there may be tradeoffs on speed and you can use the network effects of a pool of peers to get what you want.
#### Downside of P2P
Like with torrents, there is a lot of overhead to start with for sharing data in certain use cases. Posting something on Nostr and having each client peer with you or someone else to get the data will be unreliable at best. At worst you may end up with an IPFS problem.
## Plugging my storage relay idea
So all of the above is to show my journey in thinking about other technologies and how they could be used with Nostr and the internet in general. For the use cases I have in mind with social media and file sharing access there are strengths I find very useful and tradeoffs that may not fit.
Decentralized storage is a hard problem (I didn’t even mention the moderation aspect for some solutions).
The software I am currently working on borrows heavily from Nostr and torrents.
It is designed to be simple with the concept of pointer addressable content.
It is designed to have dumb servers and smart clients. The client dictates the context of the file, the server\relay stores the data.
The pointers and content are designed to be cryptographically tamper resistant.
It is designed not to be built on Nostr but rather integrated with it, using Nostr as an option for posting data descriptors.
There are tradeoffs and issues that do need to be dealt with with a storage relay method such as spam, objectionable content, reliability of the software, and building up a network of enough hosts that makes it truly decentralized.