Archives

Tags

Posts with tag "spritely"

How do Spritely's actor and storage layers tie together?

By Christopher Lemmer Webber on Thu 27 June 2019

I've been hacking away at Spritely (see previously). Recently I've been making progress on both the actor model (goblins and its rewrite goblinoid) as well as the storage layers (currently called Magenc and Crystal, but we are talking about probably renaming the both of them into a suite called "datashards"... yeah, everything is moving and changing fast right now.)

In the #spritely channel on freenode a friend asked, what is the big picture idea here? Both the actor model layer and the storage layer describe themselves as using "capabilities" (or more precisely "object capabilities" or "ocaps") but they seem to be implemented differently. How does it all tie together?

A great question! I think the first point of confusion is that while both follow the ocap paradigm (which is to say, reference/possession-based authority... possessing the capability gives you access, and it does not matter what your identity is for the most part for access control), they are implemented very differently because they are solving different problems. The storage system is based on encrypted, persistent data, with its ideas drawn from Tahoe-LAFS and Freenet, and the way that capabilities work is based on possession of cryptographic keys (which are themselves embedded/referenced in the URIs). The actor model, on the other hand, is based on holding onto a reference to a unique, unguessable URL (well, that's a bit of an intentional oversimplification for the sake of this explaination but we'll run with it) where the actor at that URL is "live" and communicated with via message passing. (Most of the ideas from this come from E and Waterken.) Actors are connected to each other over secure channels to prevent eavesdropping or leakage of the capabilities.

So yeah, how do these two seemingly very different layers tie together? As usual, I find that I most easily explain things via narrative, so let's imagine the following game scenario: Alice is in a room with a goblin. First Alice sees the goblin, then Alice attacks the goblin, then the goblin and Alice realize that they are not so different and become best friends.

The goblin and Alice both manifest in this universe as live actors. When Alice walks into the room (itself an actor), the room gives Alice a reference to the goblin actor. To "see" the goblin, Alice sends a message to it asking for its description. It replies with its datashards storage URI with its 3d model and associated textures. Alice can now query the storage system to reconstruct these models and textures from the datashards storage systems she uses. (The datashards storage systems themselves can't actually see the contents if they don't have the capability itself; this is much safer for peers to help the network share data because they can help route things through the network without personally knowing or being responsible for what the contents of those messages are. It could also be possible for the goblin to provide Alice with a direct channel to a storage system to retrieve its assets from.) Horray, Alice got the 3d model and images! Now she can see the goblin.

Assuming that the goblin is an enemy, Alice attacks! Attacking is common in this game universe, and there is no reason necessarily to keep around attack messages, so sending a message to the goblin is just a one-off transient message... there's no need to persist it in the storage system.

The attack misses! The goblin shouts, "Wait!" and makes its case, that both of them are just adventurers in this room, and shouldn't they both be friends? Alice is touched and halts her attack. These messages are also sent transiently; while either party could log them, they are closer to an instant messenger or IRC conversation rather than something intended to be persisted long-term.

They exchange their mailbox addresses and begin sending each other letters. These, however, are intended to be persisted; when Alice receives a message from the goblin in her mailbox (or vice versa), the message received contains the datashards URI to the letter, which Alice can then retrieve from the appropriate store. She can then always refer to this message, and she can choose whether or not to persist it locally or elsewhere. Since the letter has its own storage URI, when Alice constructs a reply, she can clearly mark that it was in reference to the previous letter. Even if Alice or the goblin's servers go down, either can continue to refer to these letters. Alice and the goblin have the freedom to choose what storage systems they wish, whether targeted/direct/local or via a public peer to peer routing system, with reasonable assumptions (given the continued strength of the underlying cryptographic algorithms used) that the particular entities storing or forwarding their data cannot read its content.

And so it is: live references of actors are able to send live, transient messages, but can only be sent to other actors whose (unguessable/unforgeable) address you have. This allows for highly dynamic and expressive interactions while retaining security. Datashards URIs allow for the storage and retrieval of content which can continue to be persisted by interested parties, even if the originating host goes down.

There are some things I glossed over in this writeup. The particular ways that the actors' addresses and references work is one thing (unguessable http based capability URLs on their own have leakage problems due to the way various web technologies are implemented, and not even every actor reference needs to be a long-lived URI; see CapTP for more details), how to establish connections between actor processes/servers (we can reuse TLS, or even better, something like tor's onion services), so are how interactions such as fighting can be scoped to a room (this paper explains how), as well as how we can map human meaningful names onto unguessable identifiers (the answer there is petnames). But I have plans for this and increasing confidence that it will come together... I think we're already on track.

Hopefully this writeup brings some clarity on how some of the components will work together, though!

I've been awarded the Samsung Stack Zero Grant

By Christopher Lemmer Webber on Thu 31 January 2019

Good news everyone! I've been awarded the Samsung Stack Zero Grant. But why not quote them?

Christopher Lemmer Webber is the co-editor and co-author of the now-ubiquitous ActivityPub protocol. While it provides a great framework for creating, updating, and deleting content across applications, it doesn’t provide any standardised mechanism for secure authorisation. With Spritely, Webber will work on extending the protocol in a backward-compatible manner, while at the same time building tools and applications that showcase its use. This will enable developers to build applications that enable richer interactions through a federated standard.

This should fund my next couple of years of work on full time advancement of the fediverse.

You may remember that I've talked about Spritely before. In fact I am finally in launch-mode... I am currently sitting in a wizard's tower at a hackathon, getting out the first release of Golem, a Spritely artifact.

Anyway, I'll be at FOSDEM 2019 giving a talk and on a panel. And after that I'll be speaking at CopyleftConf. Maybe I'll see you?

More news soon...

Announcing Goblins (the actor model foundation of Spritely)

By Christopher Lemmer Webber on Wed 31 October 2018

In my last post I talked about launching the Spritely project in all its ambitiousness. As I said, I plan on releasing it as a series of "demos". What I mean by "demo" is fairly loose: things you can try and see, artifacts to indicate progress as I go. Convergence slowly towards a goal.

Over the last week I released the first version of the foundation of this work, which I'm calling Goblins. v0.1, a pre-pre-alpha release, is now out the door. Goblins is an actor model implementation for the Racket programming language. I think if you know some Racket, then the Goblins documentation should be fairly approachable, and I'd love to hear feedback. Not everything is documented, but it should give you a nice taste of what the core ideas are. (Astute readers may observe that Goblins' name shares striking resemblance to another project I have worked on; this is no coincidence and is indeed a hint as to what I think the future direction of that area of work is.)

Most live distributed systems are in some way a language-level actor model implementation, but knowing that you're an actor model implementation may change how you do things. (ActivityPub is itself an actor model protocol, though I think not all implementors realize that.) Goblins is the third generation of an actor model I've written, the previous ones being XUDD and 8sync. (Maybe if you count the big rewrite I did of 8sync to run it on top of fibers, it could be the fourth generation.)

If you read the Spritely blogpost you'll know that distributed games are part of the long con here. This isn't the first time I've written a game system on top of one of these actor model implementations; in fact if you scroll down on 8sync website you'll (currently) see a video of me presenting on Mudsync, a multi-user dungeon written in 8sync, where in the talk instead of slides I had rooms, and the audience was moving around in the world and I was changing it in response to their suggestions.

But the drive of the actor model connected to social distributed games goes back further for me. I initially got interested in an actor model the first time I tried to work on the whole multiplayer game front about a decade ago. A friend and I spent a couple hours experimenting with threads and locks and after the first demo it was obvious that was never going to work. I remembered I had been told by a former coworker that they had used a system that had an actor model + coroutines as its basis; I didn't know what either of those things were yet, but as I looked into them it became clear to me that this was the right route. But one piece since then until recently remained a mystery to me, which was how to manage the complex web of authorization between the objects.

Last year at TPAC, the World Wide Web consortium's conference, I was staying with some people who asked me what I would do if I had unlimited time to work on whatever I wanted. I sheepishly admitted to my dream of a federated social web as a distributed game, and expected to be laughed out of the room. I was surprised to find out that nearly everyone I was working with had some background in this same work and dream. This gave me some hope that I wasn't completely foolish, and the observation that even failure in persuing this ambitious dream would probably result in useful artifacts.

And last year at Rebooting Web of Trust, I hit a big breakthrough. I had the chance to meet Mark S. Miller in person, someone who's work I read but barely understood at the time, and this meeting in some ways changed the direction of my career. For one thing, it lead to us collaborating on the foundation for the ocap-ld specification, which came out of a paper we collaborated on from that event (also available in nicely formatted PDF form). But what I also discovered was that Mark Miller and company had actually built the distributed and secure social network dream in the late 1990s and called it Electric Communities (rare video of it being demonstrated, and a rare video of some of the out-there ideas). What I found out was that they had solved most of the remaining questions about authorization and security through object capabilities, and that was through a pile of object capability patterns built on top of the actor model. While Electric Communities did not survive, its foundation of the the "E" programming language lived on, pulling in many of its good ideas with it. I proceeded to read as much documentation about E and surrounding capability patterns as I could, and this has majorly influenced Goblins' design. (Don't underestimate the depth of ideas on erights.org just because it looks like an ancient Geocities page... it did come from that era after all...)

Another key insight came from discovering the Object-Capability Security in Virtual Environments paper. I won't go into details on this except to say that it's a must-read for this problem domain, with the caveat that it pulls in a lot of terminology baggage from the object capability (aka ocap) community. The ocap community is one of those groups that has so many of the right ideas but which unfortunately is also drowning in its own verbiage, despite true efforts to make the ideas more accessible. We (and I guess I'm part of this group now, so "we" is appropriate) need to do better, but it isn't easy.

Goblins is still in its very early days. Its api is unstable and it surely has bugs (this one in particular I need to deal with, and which even exposed an underlying bug in Racket, which did get fixed quickly because the Racket community is amazing). Please don't build anything production-oriented on top of it yet. That said, I'd love users to try things. Check out the docs, see if they're up your alley.

If you like this kind of work and want to see more of it, consider donating; I don't have a corporate sponsor, so progress is funded by people like you. And keep watching here; more Spritely demos/artifacts/releases on the horizon.

Spritely: towards secure social spaces as virtual worlds

By Christopher Lemmer Webber on Sun 14 October 2018

If you follow me on the fediverse, maybe you already know. I've sent an announcement to my work that I am switching to doing a project named Spritely on my own full time. (Actually I'm still going to be doing some contracting with my old job, so I'll still have some income, but I'll be putting a full 40 hours a week into Spritely.)

tl;dr: I'm working on building the next generation of the fediverse as a distributed game. You can support this work if you so wish.

What on earth is Spritely?

"Well, vaporware currently", has been my joke since announcing it, but the plans, and even some core components, are starting to congeal, and I have decided it's time to throw myself fully into it.

But I still haven't answered the question, so I'll try to do so in bullet points. Spritely:

  • Aims to bring stronger user security, better anti-abuse tooling, stronger resistance against censorship, and more interesting interactions to users of the fediverse.
  • Is based on the massively popular ActivityPub standard (which I co-authored, so I do know a thing or two about this).
  • Aims to transform distributed social networks into distributed social games / virtual worlds. The dreams of the 90s are alive in Spritely.
  • Recognizes that ActivityPub is based on the actor model, and a pure version of the actor model is itself already a secure object capability system, so we don't have to break the spec to gain those powers... just change the discipline of how we use it.
  • Will be written in Racket.
  • Is an umbrella project for a number of modular tools necessary to get to this goal. The first, an object capability actor model system for Racket named Goblins, should see its first public release in the next week or two.
  • And of course it will be 100% free/libre/open source software.

That's a lot to unpack, and it also may sound overly ambitious. The game part in particular may sound strange, but I'll defend it on three fronts. First, not too many people run federated social web servers, but a lot of people run Minecraft servers... lots of teenagers run Minecraft servers... and it's not because Minecraft has the best graphics or the best fighting (it certainly doesn't), it's because Minecraft allows you to build a world together with your friends. Second, players of old MUDs, MOOs, MUSHes and etc from the 90s may recognize that modern social networks are structurally degenerate forms of the kinds of environments that existed then, but contemporary social networks lack the concept of a sense of place and interaction. Third, many interesting projects (Python's Twisted library, Flickr, much of object capability security patterns) have come out of trying to build such massively multiplayer world systems. Because of this last one in particular, I think that shooting for the stars means that if we don't make it we're likely to at least make the moon, so failure is okay if it means other things come out of it. (Also, four: it's a fun and motivating use case for me which I have explored before.)

To keep Spritely from being total vaporware, the way I will approach the project is by regularly releasing a series of "demos", some of which may be disjoint, but will hopefully increasingly converge on the vision. Consider Spritely a skunkworks-in-the-public-interest for the federated social web.

But why?

Standardizing ActivityPub was a much more difficult effort than anticipated, but equally or more so more successful than I expected (partly due to Mastodon's adoption launching it past the sound barrier). In that sense this is great news. We now have dozens of projects adopting it, and the network has (at last I looked) over 1.5 million registered users (which isn't the same as active users).

So, mission accomplished, right? Well, there are a few things that bother me.

  • The kind of rich interactions one can do are limited by a lack of authorization policy. Again, I believe object capabilities provide this, but it's not well explained to the public how to use it. (By contrast, Access Control Lists and friends are absolutely the wrong approach.)
  • Users are currently insufficiently protected from spam, abuse, and harassment while at the same time administrators are overwhelmed. This is leading a number of servers to move to a whitelisting of servers, which both re-centralizes the system and prioritizes big instances over smaller instances (it shouldn't matter what instance size you're on; arguably we should be encouraging smaller ones even). There are some paths forward, and I will hint at just one: what would happen if instead of one inbox, we had multiple inboxes? If I don't know you, you can access me via my public inbox, but maybe that's heavily moderated or you have to pay "postage". If I do know you, you might have an address with more direct access to me.
  • Relatedly, contemporary fediverse interfaces borrow from surveillance-capitalism based popular social networks by focusing on breadth of relationships rather than depth. Ever notice how the first thing Twitter shows you when you hover over a person's face is how many followers they have? I don't know about you, but I immediately compare that to my own follower count, and I don't even want to. This encourages high school popularity contest type bullshit, and it's by design. What if instead of focusing on how many people we can connect to we instead focused on the depth of our relationships? Much of the fediverse has imported "what works" directly from Facebook and Twitter, but I'd argue there's a lot we can do if we drop the assumption that this is the ideal starting base.
  • The contemporary view in the fediverse is that social scoping is like Python scoping: locals (instance) and globals (federation). Instance administrators are even encouraged to set up to run communities based on a specific niche, which is a nice reason to motivate administrators but it causes problems: even small differences between servers' expected policies often result in servers banning each other entirely. (Sometimes this is warranted, and I'm not opposed to moderation but rather looking for more effective forms of it.) Yet most of us are one person but part of many different communities with different needs. For instance, Alice may be a computer programmer, a tabletop game enthusiast, a fanfiction author, and a member of her family. In each of those settings she may present herself differently and also have different expectations of what is acceptable behavior. Alice should not need multiple accounts for this on different servers, so it would seem the right answer for community gathering is closer to something like mailing lists. What is acceptable at the gaming table may not be acceptable at work, and what happens on the fanfiction community perhaps does not need to be shared with one's family, and each community should be empowered to moderate appropriately.
  • I'd like to bridge the gap between peer to peer and federated systems. One hint as to how to do this: what happens when you run ActivityPub servers over Tor onion services or I2P? What if instead of our messages living at http addresses that could down, they could be securely addressed by their encrypted contents?
  • Finally, I will admit the most urgent reason for these concerns... I'm very concerned politically about the state of the world and what I see as increasing authoritarianism and flagrant violations of human rights. I have a lot of worry that if we don't normalize use of decentralized and secure private systems, we will lose the ability to host them, though we've never needed them more urgently.

There are a lot of opportunities, and a lot of things I am excited about, but I am also afraid of inaction and how many regrets I will have if I don't try. I have the knowledge, the privilege, and the experience to at least attempt to make a dent in some of these things. I might not succeed. But I should try.

Who's going to pay for all this?

I don't really have a funding plan, so I guess this is kind of a non-answer. However, I do have a Patreon account you could donate to.

But should you donate? Well, I dunno, I feel like that's your call. Certainly many people are in worse positions than I am; I have a buffer and I still am doing some contracting to keep myself going for a while. Maybe you know people who need the money more than I do, or maybe you need it yourself. If this is the case, don't hesitate: take care of yourself and your loved ones first.

That said, FOSS in general has the property of being a public good but tends to have a free rider problem. While we did some fundraising for some of this stuff a few years ago, I gave the majority of the money to other people. Since then I've been mostly funding work on the federated social web myself in one way or another, usually by contracting on unrelated or quasi-related things to keep myself above the burn rate. I have the privilege and ability to do it, and I believe it's critical work. But I'd love to be able to work on this with focus, and maybe get things to the point to pull in and pay other people to help again. Perhaps if we reach that point I'll look at putting this work under a nonprofit. I do know I'm unwilling to break my FOSS principles to make it happen.

Anyway... you may even still be skeptical after reading all this about whether or not I can do it. I don't blame you... even I'm skeptical. But I'll try to convince you the way I'm going to convince myself: by pushing out demos until we reach something real.

Onwards and upwards!