tomo el fuego

see off system rebooted.

if nodes exist, we'll seek interop. if not, we may want to explore bit torrent integration.

Store Procedure

In order to store a file into the local OFF storage, resp. block cache, choose the tuple size t (default 3), split the source file s into blocks s(i) of size 128 KiB (pad with random data to fit) and for each, do the following:

• Select t-1 blocks for use as randomizer blocks, or for short, randomizers, from the existing OFF cache, which have not been used in previous iterations. If not enough randomizers exist in the cache, generate them using a random number generator.
• Calculate o(i) = s(i) XOR r(1) XOR r(2) XOR ... XOR r(t-1) and store the resulting block o(i) in the cache.
• Update the descriptor list, which contains the information on how to restore each source block s(i), with a new entry, which is a set of size t: {o(i), r(1), r(2) ... r(t-1)}.
• Finally, store the descriptor list in its own block (or blocks, if the list is larger than 128 KiB) and insert these blocks d(i) into the block cache and generate an OFF URL for referencing the source file and output it to the user or into the local OFF URL database.

Retrieve Procedure

To retrieve, obtain the descriptor block or blocks and for each contained set of size {displaystyle t}t, do the following:

• Obtain the listed blocks b(1), b(2) ... b(t). Though they have no identity any more at this point, they could be called o(i), r(1), r(2) ... r(t-1).
• Perform s(i) = b(1) XOR b(2) XOR ... XOR b(t) and output the resulting source data block s(i) to a viewer program or to storage.

OFF System 2: Electric Boogaloo in a word. building up from the content-addressed storage strategy employed by plan9. there is some existing software needs to be updated to use sha256 due to many known vulnerabilities and limitations.

• vacinit
• vacsrv
• ventisrv

plan9ANTs ventured into "score replication" as extensions to 9front versions of these file servers:

• 200110.fossil.9front
• 180114.ventiprog.rewrite
• 170122.ventiprog.mirroring

this wouldn't be a heropunch project without an added layer of ambition. at this point, we've only modernized the content-addressed storage backend. which isn't nearly enough to really make the point. there are two parts to this, the first is pretty direct: s3 compatible storage. the second takes us on a really fun journey into some novel ideas about effective shared storage (owner free filesystem):

• offs
• bit torrent

this is where things really start to get interesting. the main use of the torrent code here is to provide a krpc client that we can use to communicate with other off system nodes. we love bencode.

Fast memcpy, A System Design

ideally there will be a replication process or module that can handle score replication for vac, offs, and any other content-addressed storage system that we devise.

i tend to pattern my programs around erlang/otp which isn't directly supported in limbo (obviously), but i would like to start introducing otp-inspired patterns especially when it comes to these types of networking and communication programs. tomo, crucially, needs to evolve away from the unix model that it refined and towards the lisp machine model that it has accidentally emulated. this isn't meant to take over the focus of this ticket, but gives a sense of the direction that we are trying to take.

we are not likely to have a working JIT during the development process for this feature, so i'm making some notes for getting as much performance out of the interpreted code as possible. erlang famously doesn't actually benefit that much from JIT, but it'll be interesting to see what kind of performance changes we see when its possible to set -c5 in production.