Posts with tag "datashards"

Some updates: CapTP in progress, Datashards, chiptune experiments, etc

By Christopher Lemmer Webber on Tue 30 June 2020

(Originally written as a post for Patreon donors.)

Hello... just figured I'd give a fairly brief update. Since I wrote my last post I've been working hard towards the distributed programming stuff in Goblins.

In general, this involves implementing a protocol called CapTP, which is fairly obscure... the idea is generally to apply the same "object capability security" concept that Goblins already follows but on a networked protocol level. Probably the most prominent other implementation of CapTP right now is being done by the Agoric folks, captp.js. I've been in communication with them... could we achieve interoperability between our implementations? It could be cool, but it's too early to tell. Anyway it's one of those technical areas that's so obscure that I decided to document my progress on the cap-talk mailing list, but that's becoming the length of a small novel... so I guess, beware before you try to read that whole thing. I'm far enough along where the main things work, but not quite everything (CapTP supports such wild things as distributed garbage collection...!!!!)

Anyway, in general I don't think that people get too excited by hearing "backend progress is happening"; I believe that implementing CapTP is even more important than standardizing ActivityPub was in the long run of my life work, but I also am well aware that in general people (including myself!) understand best by seeing an interesting demonstration. So, I do plan another networked demo, akin to the time-travel Terminal Phase demo, but I'm not sure just how fancy it will be (yet). I think I'll have more things to show on that front in 1-2 months.

(Speaking of Goblins and games, I'm putting together a little library called Game Goblin to make making games on top of Goblins a bit easier; it isn't quite ready yet but thought I'd mention it. It's currently going through some "user testing".)

More work is happening on the Datashards front; Serge Wroclawski (project leader for Datashards; I guess you could say I'm "technical engineer") and I have started assembling more documentation and have put together some proto-standards documents. (Warning: WIP WIP WIP!!!) We are exploring with a standards group whether or not Datashards would be a good fit there, but it's too early to talk about that since the standards group is still figuring it out themselves. Anyway, it's taken up a good chunk of time so I figured it was worth mentioning.

So, more to come, and hopefully demos not too far ahead.

But let's end on a fun note. In-between all that (and various things at home, of course), I have taken a bit of what might resemble "downtime" and I'm learning how to make ~chiptunes / "tracker music" with Milkytracker, which is just a lovely piece of software. (I've also been learning more about sound theory and have been figuring out how to compose some of my own samples/"instruments" from code.) Let me be clear, I'm not very good at it, but it's fun to learn a new thing. Here's a dollhouse piano thing (XM file), the start of a haunted video game level (XM file), a sound experiment representing someone interacting with a computer (XM file), and the mandatory demonstration that I've figured out how to do C64-like phase modulation and arpeggios (XM file). Is any of that stuff... "good"? Not really, all pretty amateurish, but maybe in a few months of off-hour experiments it won't be... so maybe some of my future demos / games won't be quite as quiet! ;)

Hope everyone's doing ok out there...

Updates: ActivityPub Conference, and more

By Christopher Lemmer Webber on Tue 01 October 2019

NOTE: This update also appeared on my Patreon account. If you're reading the below and it sounds like I'm doing a lot of useful work, consider becoming a patron!

Hello all! It's been a couple of months since I've gotten out an update; much has happened.

First of all, ActivityPub Conf happened and was a big success! The video team got things recorded an uploaded so you can watch talks from the event, including my keynote (the audio quality is a bit messed up on this one, the others are better) and Mark Miller's keynote. The other talks were all also very excellent; I'm not going to iterate them all here because you can already go watch them! I think you will find there are many thematic threads between the videos.

We had about 40 people at the event; the first day was spent on talks and the second day was an "unconference" where groups self-organized to discuss various topics of mutual interest. One common thread was about the kinds of directions I've been pushing for in Spritely: distributed encrypted storage (Datashards, with Serge Wroclawski leading the conversation on that), object capabilities (OcapPub), stamps, etc. It was interesting to watch from the start to the end of the unconference day; particularly, the Pleroma folks were there and gave a lot of feedback. Towards the start of the day I think there was much more skepticism, but towards the end we were hearing belief and interest that these kinds of things could and should be implemented and would be of real use to fediverse participants. Lain of Pleroma in particular expressed that it helped to realize that even though I'm presenting all these ideas, they don't need to be implemented all at once; we can take them on piecemeal, and incrementalism is a perfectly valid approach. (Also "OcapPub" sounds like a new protocol, whereas it's really just a way-to-use ActivityPub mostly as it already exists. Maybe time for a new name for that?)

Anyway, ActivityPub Conf was a massive success; thank you everyone who came and participated. It's clear after APConf to me just how much of a difference getting folks together can make. For those who couldn't make it, let's thank the video team (DeeAnn Little, Sebastian Lasse, Markus Feilner) for getting those videos up!

On the topic of Datashards, we have a website and a nice logo now (courtesy of mray, who also made the ActivityPub logo). Serge Wroclawski (co-host with myself of Libre Lounge) has been increasingly helping with the project; before ActivityPub Conference and Rebooting Web of Trust we worked to make sure both of our implementations could talk to each other (Serge's Python implementation and my Racket implementation). At RWoT we showed a demo where I "beamed" the death star plans to Serge's computer. (We used the same content storage server, I uploaded the death star plans, rendered the QR code on my laptop, Serge scanned the QR code from his laptop, downloaded the file and showed off the plans from his computer... with the storage server having no idea about the contents of the data we were storing there!) People really liked that demo; we have had conversations about whether Datashards may serve as a foundational system for some other tools being made in that space; more later. In the meanwhile, I'm happy we have two applications in two different languages successfully being able to read and write each others' immutable datashards updates; the next step is making sure that mutability works the same.

Rebooting Web of Trust was also a very interesting event; the highlight being that I am now collaborating with some great folks on a secure user interfaces paper. We are taking the existing Mastodon web user interface and retooling it to reduce risks such as phishing, and open the path for more peer to peer systems (very timely, since that's the direction we want to take things). Unfortunately the amount of work to do on the paper is rather huge; it may take a while until the paper is complete. In the meanwhile, the Petnames paper has turned out to be a pre-requisite for the secure UIs one; that paper has been nearly complete for some time so I guess I have to finish the work. I recently added some new UI mockups to it, but there is still more to do.

Now that these conferences are over, I am putting time towards Spritely's core vision again: distributed virtual worlds. The foundational layer for that is Spritely Goblins, an ocap-secure distributed programming environment on top of Racket. I really enjoy hacking on this and I am happy to get time back to working on it. A topic of discussion came up between myself and Mark Miller at Rebooting Web of Trust though; am I unnecessarily duplicating effort between myself and the Agoric folks? In particular, they are building something equivalent (and arguably more featureful) to Spritely Goblins named SwingSet. This would run on top of Javascript rather than Racket/Lisp/Scheme. I have found that in the past I have been not very happy when working with Javascript, but Mark suggested I take a look at Agoric's Jessie subset of Javascript, which Mark described as "closer to Scheme" (not syntactically, but in terms of language-cleanliness). It does seem nicer than Javascript; when I admitted to Mark that I am addicted to parenthetical syntax, Mark posed the question about whether building a parenthetical version of Jessie would be less work than reproducing all the other things that Agoric is doing. It's a good point; I don't know. I'm unhappy with the idea of pivoting, but I do feel like it's probably true that due diligence suggests I should consider it carefully. It is true at least that I would probably reach a broader userbase more quickly with the option of Javascript syntax; it's hard for me to deny that. I will probably explore it with some smaller tests of Agoric's stuff. But in the meanwhile, I currently plan to release a very small version of the game demo using the toolkit I already am building while testing Agoric's infrastructure in parallel. I suspect we'll see the first user-visible outputs of this in early 2020.

There have been four new Libre Lounge episodes since my last update. That's still quite a few episodes to listen to, but slower than we previously were updating; all the travel is to blame. However that is settling down and I think we'll be updating more frequently soon. Even so, we have been updating!

In addition to all this, I suspect there will be at least two major announcements in the coming months; stay tuned. Work has already occured on both, but I can only say so much right now.

Thanks to everyone who has supported my work. I work much more than full time in the cause of advancing user freedom; it's not easy to fund this work. I appreciate all of you who are giving what you can.

Now, back to work!

How do Spritely's actor and storage layers tie together?

By Christopher Lemmer Webber on Thu 27 June 2019

I've been hacking away at Spritely (see previously). Recently I've been making progress on both the actor model (goblins and its rewrite goblinoid) as well as the storage layers (currently called Magenc and Crystal, but we are talking about probably renaming the both of them into a suite called "datashards"... yeah, everything is moving and changing fast right now.)

In the #spritely channel on freenode a friend asked, what is the big picture idea here? Both the actor model layer and the storage layer describe themselves as using "capabilities" (or more precisely "object capabilities" or "ocaps") but they seem to be implemented differently. How does it all tie together?

A great question! I think the first point of confusion is that while both follow the ocap paradigm (which is to say, reference/possession-based authority... possessing the capability gives you access, and it does not matter what your identity is for the most part for access control), they are implemented very differently because they are solving different problems. The storage system is based on encrypted, persistent data, with its ideas drawn from Tahoe-LAFS and Freenet, and the way that capabilities work is based on possession of cryptographic keys (which are themselves embedded/referenced in the URIs). The actor model, on the other hand, is based on holding onto a reference to a unique, unguessable URL (well, that's a bit of an intentional oversimplification for the sake of this explaination but we'll run with it) where the actor at that URL is "live" and communicated with via message passing. (Most of the ideas from this come from E and Waterken.) Actors are connected to each other over secure channels to prevent eavesdropping or leakage of the capabilities.

So yeah, how do these two seemingly very different layers tie together? As usual, I find that I most easily explain things via narrative, so let's imagine the following game scenario: Alice is in a room with a goblin. First Alice sees the goblin, then Alice attacks the goblin, then the goblin and Alice realize that they are not so different and become best friends.

The goblin and Alice both manifest in this universe as live actors. When Alice walks into the room (itself an actor), the room gives Alice a reference to the goblin actor. To "see" the goblin, Alice sends a message to it asking for its description. It replies with its datashards storage URI with its 3d model and associated textures. Alice can now query the storage system to reconstruct these models and textures from the datashards storage systems she uses. (The datashards storage systems themselves can't actually see the contents if they don't have the capability itself; this is much safer for peers to help the network share data because they can help route things through the network without personally knowing or being responsible for what the contents of those messages are. It could also be possible for the goblin to provide Alice with a direct channel to a storage system to retrieve its assets from.) Horray, Alice got the 3d model and images! Now she can see the goblin.

Assuming that the goblin is an enemy, Alice attacks! Attacking is common in this game universe, and there is no reason necessarily to keep around attack messages, so sending a message to the goblin is just a one-off transient message... there's no need to persist it in the storage system.

The attack misses! The goblin shouts, "Wait!" and makes its case, that both of them are just adventurers in this room, and shouldn't they both be friends? Alice is touched and halts her attack. These messages are also sent transiently; while either party could log them, they are closer to an instant messenger or IRC conversation rather than something intended to be persisted long-term.

They exchange their mailbox addresses and begin sending each other letters. These, however, are intended to be persisted; when Alice receives a message from the goblin in her mailbox (or vice versa), the message received contains the datashards URI to the letter, which Alice can then retrieve from the appropriate store. She can then always refer to this message, and she can choose whether or not to persist it locally or elsewhere. Since the letter has its own storage URI, when Alice constructs a reply, she can clearly mark that it was in reference to the previous letter. Even if Alice or the goblin's servers go down, either can continue to refer to these letters. Alice and the goblin have the freedom to choose what storage systems they wish, whether targeted/direct/local or via a public peer to peer routing system, with reasonable assumptions (given the continued strength of the underlying cryptographic algorithms used) that the particular entities storing or forwarding their data cannot read its content.

And so it is: live references of actors are able to send live, transient messages, but can only be sent to other actors whose (unguessable/unforgeable) address you have. This allows for highly dynamic and expressive interactions while retaining security. Datashards URIs allow for the storage and retrieval of content which can continue to be persisted by interested parties, even if the originating host goes down.

There are some things I glossed over in this writeup. The particular ways that the actors' addresses and references work is one thing (unguessable http based capability URLs on their own have leakage problems due to the way various web technologies are implemented, and not even every actor reference needs to be a long-lived URI; see CapTP for more details), how to establish connections between actor processes/servers (we can reuse TLS, or even better, something like tor's onion services), so are how interactions such as fighting can be scoped to a room (this paper explains how), as well as how we can map human meaningful names onto unguessable identifiers (the answer there is petnames). But I have plans for this and increasing confidence that it will come together... I think we're already on track.

Hopefully this writeup brings some clarity on how some of the components will work together, though!