Vats and Propagators: towards a global brain

By Christine Lemmer-Webber on Sun 16 February 2020

(This is a writeup for future exploration; I will be exploring a small amount of this soon as a side effect of some UI building I am doing, but not a full system. A full system will come later, maybe even by years. Consider this a desiderata document. Also a forewarning that this document was originally written for an ocap-oriented audience, and some terms are left unexpanded; for instance, "vat" really just means a one-turn-at-a-time single-threaded event loop that a bunch of actors live in.)

We have been living the last couple of decades with networks that are capable of communicating ideas. However, by and large it is left to the humans to reason about these ideas that are propagated. Most machines that operate on the network merely execute the will of humans that have carefully constructed them. Recently neural network based machine learning has gotten much better, but merely resembles intuition, not reasoning. (The human brain succeeds by combining both, and a successful system likely will too.) Could we ever achieve a network that itself reasons? And can it be secure enough not to tear itself apart?

Near-term background

In working towards building out a demonstration of petname systems in action in a social network, I ran into the issue of changes to a petname database automatically being reflected through the UI. This lead me back down a rabbit hole of exploring reactive UI patterns, and also lead me back to exploring that section, and the following propagator section, of SICP again. This also lead me to rewatch one of my favorite talks: We Don't Really Know How to Compute! by Gerald Sussman.

At 24:54 Sussman sets up an example problem: specifically, an expert in electrical systems having a sense of how to be able to handle and solve an electrical wiring diagram. (The kind of steps explained are not dissimilar to the kind of steps that programmers go through while reasoning about debugging a coding problem.) Sussman then launches into an exploration of propagators, and how they can solve the problem. Sussman's explanation is better than mine would be, so I'll leave you to watch the video to see how it's used to solve various problems.

Okay, a short explanation of propagators

Well, I guess I'll give a little introduction to propagators and why I think they're interesting.

Propagators have gone through some revisions since the SICP days; relevant reading are the Revised Report on the Propagator Model, The Art of the Propagator, and to really get into depth with the ideas, Propagation networks: a flexible and expressive substrate for computation (Radul's PhD thesis).

In summary, a propagator model has the following properties:

  • There are cells with accumulate information about a value. Note! This is a big change from previous propagator versions! In the modern version of a propagator model, a cell doesn't hold a value, it accrues information about a value which must be non-contradictory.
  • Such cell information may be complete (the number 42 is all there is to know), whereas some other information may be a range of possibilities (hm, could be anywhere between -5 to 45...). As more information is made available, we can "narrow down" what we know.
  • Cells are connected together with propagators.
  • Information is (usually) bidirectional. For example, with the slope formula of y = (m * x) + b, we don't need to just solve for y... we could solve for m, x, or b given the other information. Similarly, partial information can propagate.
  • Contradictions are not allowed. Attempting to introduce contradictory information into the network will throw an exception.
  • We can "play with" different ideas via a Truth Maintenance System. What do we believe? Changes in our beliefs can result in changes to the generated topology of the network.
  • Debugging is quite possible. One of the goals of propagator networks is that you should be able to investigate and determine blame for a result. Relationships are clear and well defined. As Sussman says (roughly paraphrased), "if an autonomous car drives off the car of the road, I could sue the car manufacturer, but I'd rather sue the car... I want to hold it accountable for its decision making". The ability to hold accountability and determine blame stands in contrast to squishier systems like neural nets, genetic programs, etc (which are still useful, but not as easy to interrogate).

There are a lot of things that can be built with propagators as the general case of constraint solving and reasoning; functional reactive UIs, type checkers, etc etc.

Bridging vats and propagators

The prototype implementations are written in Scheme. The good news is, this means we could implement propagators on top of something like Spritely Goblins.

However (and, granted, I haven't completed it) I think there is one thing that is inaccurately described in Radul's thesis and Sussman's explanations, but which I think actually is no problem at all if we apply the vat model of computation (as in E, Agoric, Goblins): how distributed can these cells and propagators be? Section 2.1 of Radul's thesis explains propagators as asynchronous and completely autonomous, as if cells and their propagators could live anywhere on the computer network with no change in effectiveness. I think this is only partially true. The reference implementation actually does not fully explore this because it uses a single-threaded event loop that processes events until there are no more to process, during which it may encounter a contradiction and raise it. However I believe that the ability to "stop the presses" as it were is one of the nicest features of propagators and actually should not be lost... if we introduced asynchronous events coming in, there may be multiple events that come in at the same time and which try making changes to the propagator network in parallel. Thankfully a nice answer comes in form of a the vat model: it should be possible to have a propagator network within a single vat. Spritely Goblins' implementation of the vat model is transactional, so this means that if we try to introduce a contradiction, we could roll back immediately. This is the right behavior. As it turns out, this is very close to the propagator system in the way it's implemented in the reference implementation... I think the reference implementation did something more or less right while trying to do the simplest thing. Combined with a proper ocap vat model this should work great.

Thus, I believe that a propagator system (here I mean a propagator network, meaning a network of propagator-connected cells) should actually be vat-local. But wait, we talked about network (as in internet) based reasoning, and here I am advocating locality! What gives?

The right answer seems to me that propagator networks should be able to be hooked together, but a change to a vat-contained propagator system can trigger message passing to another vat-contained propagator system, which can even happen over a computer network such as the internet. We will have to treat propagator systems and changes to them as vat-local, but they can still communicate with other propagator systems. (This is a good idea anyway; if you communicate an idea with me and it's inconsistent with my worldview, it should be important for me to be able to realize that and use that as an opportunity to correct our misunderstandings between each other.)

However, cells are still objects with classic object references. This means it is possible to hold onto one and use it as either a local or networked capability. Attenuation also composes nicely; it should be possible to produce a facet of a cell that only allows read access or only allows adding information. It's clear and easily demonstrated that ocaps can be the right security model for the propagator model simply by realizing that both the propagator prototype system is written in scheme, and so is Jonathan Rees' W7 security kernel.

This is all to say, if we built the propagator model on top of an ocap-powered vat model, we'd already have a good network communication model, a good security model, and a transactional model. Sounds great to me.

Best of all, a propagator system can live alongside normal actors. We don't have to choose one or the other... a multi-paradigm approach can work great.

Speaking the same language

One of the most important things in a system that communicates is that ideas should be able to be expressed and considered in such a way that both parties understand. Of course, humans do this, and we call it "language".

Certain primitives exist in our system already; for optimization reasons, we are unlikely to want to build numbers out of mere tallying of numbers (such as in Peano arithmetic); we instead build in primitives for integers and a means of combination for them. So we will of course want to have several primitive data types.

But at some point we will want to talk about concepts that are not encoded in the system. If I would like to tell you about a beautiful red bird I saw, where would I even begin? Well obviously at minimum, we will have to have ways of communicating ideas such as "red" and "bird". We will have to build a vocabulary together.

Natural language vocabulary has a way of becoming ambiguous fast. A "note" passed in class versus a "note" in a musical score versus that I would like to "note" a topic of interest to you are all different things.

Linked data (formerly "semantic web") folks have tried to use full URIs as a way to get around this problem. For instance, two ActivityPub servers which are communicating are very likely speaking about the same thing if they both use "", which is to say they are talking about some written note-like message (probably a (micro)blog post). This is not a guarantee; vocabulary drift is still possible, but it is much less likely.

Unfortunately, http(s) based URIs are a poor choice for hosting vocabulary. Domains expire, websites go down, and choosing whether to extend a vocabulary in some namespace is (in the author's experience) a governance nightmare. A better option is "content-addressed vocabulary"; instead of "" we could instead simply take the text from the standard:

"Represents a short written work typically less than a single paragraph in length."

Hash that and you get "urn:sha256:54c14cbd844dc9ae3fa5f5f7b8c1255ee32f55b8afaba88ce983a489155ac398". No governance or liveness issues required. (Hashing mechanism upgrades, however, do pose some challenge; mapping old hashes to new ones for equivalence can be a partial solution.)

This seems sufficient to me; groups can collaborate somewhere to hammer out the definition of some term, simply hash the definition of it, and use that as the terminology URI. This also avoids hazards from choosing a different edge of Zooko's Triangle for vocabulary.

Now that we have this, we can express advanced new ideas across the network and experiment with new terms. Better yet, we might be even able to use our propagator networks to associate ideas with them. I think in many systems, content-addressed-vocabulary could be a good way to describe beliefs that could be considered, accepted, rejected in truth maintenance systems.

Cerealize me, cap'n!

One observation from Agoric is that it is possible to treat systems that do not resemble traditional live actor'y vats still as vats (and "machines") and develop semantics for message passing between them (and performing promise resolution) nonetheless, for instance blockchains.

Similarly, above we have observed that propagator systems can be built on top of actors; I believe it is also possible to describe propagator networks in terms of pure data. It should be possible to describe changes to a propagator network as a standard serialized ledger that can be transferred from place to place or reproduced.

However, the fact that interoperability with actors is possible is good, desirable, and thankfully a nice transitional place for experimentation (porting propagator model semantics to Spritely Goblins should not be hard).

Where to from here?

That's a lot of ideas above, but how likely is any of this stuff to be usable soon? I'm not anticipating dropping any current work to try to make this happen, but I probably will be experimenting in my upcoming UI work to try to have the UI powered by a propagator system (possibly even a stripped down version) so that the experimental seeds are in place to see if such a system can be grown. But I am not anticipating that we'll see anything like a fully distributed propagator system doing something interesting from my own network soon... but sometimes I end up surprised.

Closing the loop

I mentioned before that human brains are a combination of faster intuitive methods (resembling current work on neural nets) and slower, more calculating reasoning systems (resembling propagators or some logic programming languages). That's also to say nothing about the giant emotional soup that a mind/body tends to live in.

Realistically the emergence of a fully sapient system won't involve any of these systems independently, but rather a networked interconnection of many of them. I think the vat model of execution is a nice glue system for it; pulling propagators into the system could bring us one step closer, maybe.

Or maybe it's all just fantastical dreaming! Who knows. But it could be interesting to play and find out at some point... perhaps some day we can indeed get a proper brain into a vat.

State of Spritely for February 2020

By Christine Lemmer-Webber on Mon 10 February 2020

We are now approximately 50% of the way through the Samsung Stack Zero grant for Spritely, and only a few months more since I announced the Spritely project at all. I thought this would be a good opportunity to review what has happened so far and what's on the way.

In my view, quite a lot has happened over the course of the last year:

  • Datashards grew out of two Spritely projects, Magenc and Crystal. This provides the "secure storage layer" for the system, and by moving into Datashards has even become its own project (now mostly under the maintainership of Serge Wroclawski, who as it turns out is also co-host with me of Libre Lounge. There's external interest in this from the rest of the federated social web, and it was a topic of discussion in the last meeting of the SocialCG. While not as publicly visible recently, the project is indeed active; I am currently helping advise and assist Serge with some of the ongoing work on optimizations for smaller files, fixing the manifest format to permit larger files, and a more robust HTTP API for stores/registries. (Thank you Serge also for taking on a large portion of this work and responsibility!)

  • Spritely Goblins, the actor model layer of Spritely, continues its development. We are now up to release v0.5. I don't consider the API to be stable, but it is stabilizing. In particular, the object/update model, the synchronous communication layer, and the transactional update support are all very close to stable. Asynchronous programming mostly works but has a few bugs I need to work out, and the distributed programming environment design is coming together enough where I expect to be able to demo it soon.

  • In addition, I have finally started to write docs for Spritely Goblins. I think the tutorial above is fairly nice, and I've had a good amount of review from various parties, and those who have tried it seem to think it is fairly nice. (Please be advised that it requires working with the dev branch of Goblins at the time of writing.) v0.6 should the first release to have documentation after the major overhaul I did last summer (effectively an entire rewrite of the system, including many changes to the design after doing research into ocap practices). I cannot recommend that anyone else write production-level code using the system yet, but I hope that by the summer things will have congealed enough that this will change.

  • I have made a couple of publicly visible demos of Goblins' design. Weirdly enough all of these have involved ascii art.

    • The proto-version was the Let's Just Be Weird Together demo. Actually it's a bit strange to say this because the LJBWT demo didn't use Goblins, it used a library called DOS/HURD. However, writing this library (and adapting it from DOS/Win) directly informed the rewrite of Goblins, Goblinoid which eventually became Goblins itself, replacing all the old code. This is why I advocate demo-driven-development: the right design of an architecture flows out of a demo of it. (Oh yeah, and uh, it also allowed me to make a present for my 10th wedding anniversary, too.)

    • Continuing in a similar vein, I made the "Season's Greetings" postcard, which Software Freedom Conservancy actually used in their funding campaign this year. This snowy scene used the new rewrite of Goblins and allowed me to try to push the new "become" feature of Goblins to its limit (the third principle of actor model semantics, taken very literally). It wasn't really obvious to anyone else that this was using Goblins in any interesting way, but I'll say that writing this really allowed me to congeal many things about the update layer and it also lead to uncovering a performance problem, leading to a 10x speedup. Having written this demo, I was starting to get the hang of things in the Goblins synchronous layer.

    • Finally there was the Terminal Phase demo. (See the prototype announcement blogpost and the 1.0 announcement.) This was originally designed as a reward for donors for hitting $500/mo on my Patreon account (you can still show up in the credits by donating!), though once 1.0 made it out the door it seems like it raised considerable excitement on the r/linux subreddit and on Hacker News, which was nice to see. Terminal Phase helped me finish testing and gaining confidence in the transactional object-update and synchronous call semantics of Spritely Goblins, and I now have no doubt that this layer has a good design. But I think Terminal Phase was the first time that other people could see why Spritely Goblins was exciting, especially once I showed off the time travel debugging in Terminal Phase demo. That last post lead people to finally start pinging me asking "when can I use Spritely Goblins"? That's good... I'm glad it's obvious now that Goblins is doing something interesting (though the most interesting things are yet to be demo'ed).

  • I participated in, keynoted, and drummed up enthusiasm for ActivityPub Conference 2019. (I didn't organize though, that was Morgan Lemmer-Webber's doing, alongside Sebastian Lasse and with DeeAnn Little organizing the video recording.) We had a great speaker list and even got Mark S. Miller to keynote. Videos of the event are also available. While that event was obviously much bigger than Spritely, the engagement of the ActivityPub community is obviously important for its success.

  • Relatedly, I continue to co-chair the SocialCG but Nightpool has joined as co-chair which should relieve some pressure there, as I was a bit too overloaded to be able to handle this all on my own. The addition of the SocialHub community forum has also allowed the ActivityPub community to be able to coordinate in a way that does not rely on me being a blocker. Again, not Spritely related directly, but the health of the ActivityPub community is important to Spritely's success.

  • At Rebooting Web of Trust I coordinated with a number of contributors (including Mark Miller) on sketching out plans for secure UI designs. Sadly the paper is incomplete but has given me the framework for understanding the necessary UI components for when we get to the social network layer of Spritely.

  • Further along the lines of sketching out the desiderata of federated social networks, I have written a nearly-complete OcapPub: towards networks of consent. However, there are still some details to be figured out; I have been hammering them out on the cap-talk mailing list (see this post laying out a very ocappub-like design with some known problems, and then this analysis). The ocap community has thankfully been very willing to participate in working with me to hammer out the right security foundations, and I think we're close to the right design details. Of course, the proof of the pudding is in the demo, which has yet to be written.

Okay, so I hope I've convinced you that a lot has happened, and hopefully you feel that I am using my time reasonably well. But there is much, much, much ahead for Spritely to succeed in its goals. So, what's next?

  • I need to finish cleaning up the Goblins documentation and do a v0.6 release with it included. At that point I can start recommending some brave souls to use it for some simple applications.

  • A demo of Spritely Goblins working in a primarily asynchronous environment. This might simply be a port of mudsync as a first step. (Recorded demo of mudsync from a few years ago.) I'm not actually sure. The goal of this isn't to be the "right" social network design (not full OcapPub), just to test the async behaviors of Spritely Goblins. Like the synchronous demos that have already been done, the purpose of this is to congeal and ensure the quality of the async primitives. I expect this and the previous bullet point to be done within the next couple of months, so hopefully by the end of April.

  • Distributed networked programming in Goblins, and associated demo. May expand on the previous demo. Probably will come out about two months later, so end of June.

  • Prototype of the secure UI concepts from the forementioned secure UIs paper. I expect/hope this to be usable by end of third quarter 2020.

  • Somewhere in-between all this, I'd like to add a demo of being able to securely run untrusted code from third parties, maybe in the MUD demo. Not sure when yet.

  • All along, I continue to expect to push out new updates to Terminal Phase with more fun enemies and powerups to continue to reward donors to the Patreon campaign.

This will probably take most of this year. What you will notice is that this does not explicitly state a tie-in with the ActivityPub network. This is intentional, because the main goal of all the above demos are to prove more foundational concepts before they are all fully integrated. I think we'll see the full integration and it coming together with the existing fediverse beginning in early 2021.

Anyway, that's a lot of stuff ahead. I haven't even mentioned my involvement in Libre Lounge, which I've been on hiatus from due to a health issue that has made recording difficult, and from being busy trying to deliver on these foundations, but I expect to be coming back to LL shortly.

I hope I have instilled you with some confidence that I am moving steadily along the abstract Spritely roadmap. (Gosh, I ought to finally put together a website for Spritely, huh?) Things are happening, and interesting ones I think.

But how do you think things are going? Maybe you would like to leave me feedback. If so, feel free to reach out.

Until next time...

Time travel debugging in Spritely Goblins, previewed through Terminal Phase

By Christine Lemmer-Webber on Thu 23 January 2020

Time travel in Spritely Goblins shown through Terminal Phase

Okay, by now pretty much everyone is probably sick of hearing about Terminal Phase. Terminal Phase this, and Terminal Phase that! Weren't you getting back to other hacking on Spritely Goblins, Chris? And in fact I am, I just decided it was a good idea to demo one of the things that makes Goblins interesting.

What you're seeing above is from the experimental tt-debugger branch of Terminal Phase (not committed yet because it's a proof-of-concept, and not as clean as I'd like it to be, and also you need the "dev" branch of Goblins currently). When the user presses the "t" key, they are presented with a menu by which they can travel backwards and forwards in time. The player can select a previous state of the game from every two seconds and switch to that.

Here's the cool part: I didn't change a single line of game code to make this occur. I just added some code around the game loop that snapshotted the state as it currently existed and exposed it to the programmer.

What kind of time sorcery is this?

Dr. Who/Dr. Sussman fez comparison

Well, we're less the time-lord kind, more the functional programmer kind. Except, quasi-functional.

If you watched the part of the recent Terminal Phase video I made that shows off Goblins you'll remember that the way that objects work is that a reference to a Goblins object/actor is actually a reference that indirectly refers to a procedure for handling immediate calls and asynchronous messages. Relative to themselves (and in true actor fashion), objects specify first their initial version of themselves, and later can use a special "become" capability to specify a future version of themselves. From the perspective of the actor, this looks very functional. But from the perspective of one object/actor performing a call against another object/actor, it appears that things change.

Here is the simplest example of such an object, a cell that holds a single value:

;; Constructor for a cell.  Takes an optional initial value, defaults
;; to false.
(define (^cell bcom [val #f])
    ;; Called with no arguments; return the current value
    [() val]
    ;; Called with one argument, we become a version of ourselves
    ;; with this new value
     (bcom (^cell bcom new-val))]))

If you can't read Racket/Scheme, not a big deal; I'll just tell you that this cell can be called with no arguments to get the current value, and with one argument to set a value. But you'll see that in the former case, the value we would like to return to the caller is returned; in the latter case, we return the handler we would like to be for handling future messages (wrapped up in that bcom capability). In both cases, we aren't performing side effects, just returning something.. but in the latter case the kernel observes this and updates the current transaction's delta reflecting that this is the "new us". (Not shown here but supported: both becoming a new handler and returning a value.)

Without going into details, this makes it extremely easy to accomplish several things in Goblins:

  • Transactionality: Each "turn" of an event loop in Goblins is transactional. Rather than being applied immediately, a transaction is returned. Whether we choose to commit this or not is up to us; we will probably not, for instance, if an exception occurs, but we can record the exception (a default event loop is provided that does the default right-thing for you).
  • Snapshotting time: We can, as shown above, snapshot history and actually run code against previous state (assuming, again, that state is updated through the usual Goblins actor "become" means).
  • Time-travel debugging: Yeah, not just for Elm! I haven't built a nice interface for it in the demo above, but it's absolutely possible to expose a REPL at each snapshot in time in the game to "play around with" what's happening to debug difficult problems.

This is only a small portion of what makes Spritely Goblins interesting. The really cool stuff will come soon in the distributed programming stuff. But I realized that this is one of the more obviously cool aspects of Spritely Goblins, and before I start showing off a bunch of other interesting new things, I should show off a cool feature that exists in the code we already have!

Anyway, that's it... I hope I gave you a good sense that I'm up to interesting things. If you're excited by this stuff and you aren't already, consider donating to keep this work advancing.

Whew! I guess it's time I start writing some docs for Goblins, eh?

Terminal Phase 1.0

By Christine Lemmer-Webber on Sun 19 January 2020

Testing Terminal Phase image

I'm pleased to announce that Terminal Phase, a space shooter game you can play in your terminal, has achieved version 1.0. The game is completely playable and is a fun game (well, at least a number of playtesters told me they thought it was fun). It includes two levels (one of which is more balanced than the other), and more content is on its way (1.0 isn't the end!). You can see it being played above in cool-retro-term but it works in all sorts of terminals, including gnome-terminal and etc.

I also released a video recently ( mirror) of me doing a live playtest of the game and also showing off how to make new levels and program new enemies (which serves as kind of an introduction, but probably not the best one, to Spritely Goblins).

Terminal Phase was actually a reward for hitting the $500/mo milestone on my Patreon account, which we achieved a little over a week ago. I aimed to get 1.0 out the door by midnight on Wednesday but I actually released it a couple of hours later, closer to 2:30am, because I was trying to make the credits look cool:

Terminal Phase Credits

I think I succeeded, right? Maybe you would like your name in there; you can still do so by selecting a tier on my Patreon account. I released the game as FOSS, so whether you donate or not, you can still reap the benefits. But I figure making the credits look cool and putting peoples' names in there would be a good way of making people feel motivated. And there are more releases on the way; I'll be adding to this now and then and releasing more stuff occasionally. In fact you may notice the cool parallax scrolling starfield in the gif at the top of this post; I added that after 1.0. I guess it's a bit sneaky to put that on top of a post labeled 1.0, but the good news is that this means that 1.1 is not far away, which will include some new enemies (maybe a boss?), new levels, and yes, parallax starfields (and maybe your name in the credits if it isn't already).

Anyway, enough self-shilling; let's talk more about the game itself. Terminal Phase really had a number of goals:

  • Fun. Games are fun, and making them is (well, mostly) fun and interesting. And I feel like the FOSS world could use more fun.
  • Fundraising. I do a lot of work to enrich the commons; funding that stuff can be brutally hard, and obviously this was a fundraising tactic.
  • A litmus test. I wanted to see, "Do people care about funding FOSS games, in particular? Does this matter to people?" My suspicion is that there is an interest, even if niche, and that seems to have been validated. Great.
  • Pushing the medium of interactive terminal-based / ascii art content. Probably because it's a retro area, it's not really one where we see a lot of new content. We see a lot more terminal-based turn-based games, most notably roguelikes; why not more live action stuff? (Note that I have done two other projects I did this year in this same vein.)
  • Thanking donors. I had this Patreon account and that's great that people were being generous, but I felt like it would be nice to have something that felt quasi-tactile, like you got something visible back from it. I hope people feel like that succeeded.
  • But most importantly, advancing Spritely Goblins. Terminal Phase is a program to demonstrate and test how well about half of what Goblins does works well, namely transactional object interactions.

I feel like all of those were a success, but I really want to write more about that last one. Except, well, I already have in some detail, and maybe I'd repeat myself. But I'll say that developing Terminal Phase has made me dramatically more confident that the core parts of Spritely Goblins work well and make sense. That's good, and I can say that without a bunch of hand-waving; I built something that feels nice to use and to program.

That lets me move forward with solidifying and documenting what I have and focusing on the next layer: the asynchronous programming and distributed networked objects layers. The former of those two exists, the latter of those needs work, but both will be tested in a similar way soon; I plan on building some highly interactive demos to show off their ideas.

Anyway, I hope you enjoy the game, and thank you to everyone who donated and made it possible! Again, I plan to release more soon, including new levels, new enemies, boss battles, and yes, even some powerups. And if you like the game, consider becoming a supporter if you aren't already!

Now back to working on Spritely Goblins itself...

201X in review

By Christine Lemmer-Webber on Tue 31 December 2019

Well, this has been a big decade for me. At the close of 200X I was still very young as a programmer, had just gotten married to Morgan, had just started my job at Creative Commons, and was pretty sure everyone would figure out I was a fraud and that it would all come crashing down around me when everyone found out. (Okay, that last part is still true, but now I have more evidence I've gotten things done despite apparently pulling the wool over everyone's eyes.)

At work my boss left and I temporarily became tech lead, and we did some interesting things like kick off CC BY-SA and GPL compatibility work (which made it into the 4.0 license suite) and ran Liberated Pixel Cup (itself an interesting form of success, but I would like to spend more time talking about what the important lessons of it were... another time).

In 2011 I started MediaGoblin as a side project, but felt like I didn't really know what I was doing, yet people kept showing up and we were pushing out releases. Some people were even running the thing, and it felt fairly amazing. I left my job at Creative Commons in late 2012 and decided to try to make working on network freedom issues my main thing. It's been my main thing since, and I'm glad I've stayed focused in that way.

What I didn't expect was that the highlight of my work in the decade wasn't MediaGoblin itself but the standard we started participating in, which became ActivityPub. The work on ActivityPub arguably caused MediaGoblin to languish, but on the other hand ActivityPub was successfully ratified by the W3C as a standard and now has over 3.5 million registered users on the network and is used by dozens (at least 50) pieces of (mostly) interoperable software. That's a big success for all of it that worked on it (and there were quite a few), and in many ways I think is the actual legacy of MediaGoblin.

After ActivityPub becoming a W3C Recommendation, I took a look around and realized that other projects were using ActivityPub to accomplish the work of MediaGoblin maybe even better than MediaGoblin. The speed at which this decade passed made me conscious of how short time is and made me wonder how I should best budget it. After all, the most successful thing I worked on turned out to not be the networked software itself but the infrastructure for building networks. That lead me to reconsider whether my role was more importantly as trying to advance the state of the art, which has lead me to more recently start work on the federation laboratory called Spritely, of which I've written a bit about here.

My friend Serge Wroclawski and I also launched a podcast in the last year, Libre Lounge. I've been very proud of it; we have a lot of great episodes, so check the archive.

Keeping this work funded has turned out to be tough. In MediaGoblin land, we ran two crowdfunding campaigns, the first of which paid for my work, the second of which paid for Jessica Tallon's work on federation. The first campaign got poured entirely into MediaGoblin, the second one surprisingly resulted in making space so that we could do ActivityPub's work. (I hope people feel happy with the work we did, I do think ActivityPub wouldn't have happened without MediaGoblin's donors' support. That seems worth celebrating and a good outcome to me personally, at least.) I also was fortunate enough to get accepted into Stripe's Open Source Retreat and more recently my work on Spritely has been funded by the Samsung Stack Zero grant. Recently, people have been helping by donating on Patreon and both my increase in prominence from ActivityPub and Libre Lounge have helped grow that. That probably sounds like a lot of funding and success, but still most of this work has had to be quite... lean. You stretch that stuff out over nearly a decade and it doesn't account for nearly enough. To be honest, I've also had to pay for a lot of it myself too, especially by frequently contracting with other organizations (such as Open Tech Strategies and Digital Bazaar, great folks). But I've also had to rely on help from family resources at times. I'm much more privileged than other people, and I can do the work, and I think the work is necessary, so I've devoted myself to it. Sometimes I get emails asking how to be completely dedicated to FOSS without lucking out at a dedicated organization and I feel extra imposter-y in responding because I mean, I don't know, everything still feels very hand-to-mouth. A friend pointed to a blogpost from Fred Hicks at Evil Hat about how behind the scenes, things don't feel as sustainable sometimes, and that struck a chord with me (it was especially surprising to me, because Evil Hat is one of the most prominent tabletop gaming companies.) Nonetheless, I'm still privileged enough that I've been able to keep it working and stay dedicated, and I've received a lot of great support from all the resources mentioned above, and I'm happy about all that. I just wish I could give better advice on how to "make it work"... I'm in search of a good answer for that myself.

In more personal reflections of this decade, Morgan and I went through a number of difficult moves and some very difficult family situations, but I think our relationship is very strong, and some of the hardest stuff strengthened our resolve as a team. We've finally started to settle down, having bought a house and all that. Morgan completed one graduate program and is on the verge of completing her next one. A decade into our marriage (and 1.5 decades into our relationship), things are still wonderfully weird.

I'm halfway through my 30s now. This decade made it clearer to me how fast time goes. In the book A Deepness in the Sky, a space-trading civilization is described that measures time in seconds, kiloseconds, megaseconds, gigaseconds, etc. Increasingly I feel like the number of seconds ahead in life are always shorter than we feel like they will be; time is a truly precious resource. How much more time do I have to do interesting and useful things, to live a nice life? How much more time do we have left to get our shit together as a human species? (We seem to be doing an awful lot to shorten our timespan.)

I will say that I am kicking of 202X by doing something to hopefully contribute to lengthening both the timespan of myself and (more marginally individually, more significantly if done collectively) human society: 2020 will be the "Year of No Travel" for me. I hate traveling; it's bad for myself, bad for the environment. It's seemed most importantly to be the main thing that continues to throw my health out of whack, over and over again.

But speaking of time and its resource usage, a friend once told me that I had a habit in talks to "say the perfect thing, then ruin it by saying one more thing". I probably did something similar above (well, not claiming anything I write is perfect), but I'll preserve it anyway.

Everything, especially this blog, is imperfect anyway. Hopefully this next decade is imperfect and weird in a way we can, for the most part, enjoy.

Goodbye 201X, hello 202X.

Noncommercial Doesn't Compose (and it never will)

By Christine Lemmer-Webber on Fri 27 December 2019

NOTE: I actually posted this some time ago on license-discuss and some people suggested that I blog it. A good idea, which I never did, until now. The topic has't become any less relevant, so...

It's sad to see history repeat itself, but that's what history does, and it seems like we're in an awfully echo'y period of time. Given the volume of submissions in favor of some sort of noncommercial style license, I feel I must weigh in on the issue in general. Most of my thoughts on this developed when I worked at Creative Commons (which famously includes a "noncommercial" clause that can be mixed into a few licenses), and it took me a while to sort out why there was so much conflict and unhappiness over that clause. What was clear was that Non-Commercial and No-Derivatives were both not considered "free culture" licenses, and I was told this was drawn from the lessons of the free software world, but here we are hashing it out again so anyway...

(I'm not suggesting this is a CC position; Creative Commons hasn't to my knowledge taken an official stance on whether NonCommercial is right, and not everyone internally agreed, and also I don't work there anymore anyhow.)

I thank Rhea Myers for a lot of clarity here, who used to joke that NC (the shorthand name for Non-Commercial) really stood for "No Community". I think that's true, but I'll argue that even more so it stands for "No Composition", which is just as much or more of a threat, as I hope to explain below.

As a side note, I am of course highly empathetic to the motivations of trying to insert a noncommercial clause; I've worn many hats, and funding the software I've worked on has by far been the hardest. At first glance, an NC approach appears to be a way to solve the problem. Unfortunately, it doesn't work.

The first problem with noncommercial is that nobody really knows for sure what it means. Creative Commons made a great effort to gather community consensus. But my read from going through that is that it's still very "gut feel" for a lot of people, and while there's some level of closeness the results of the report result in nothing crisp enough to be considered "defined" in my view. Personally I think that nothing will ever hit that point. For instance, which of these is commercial, and which is noncommercial?

  • Playing music at home
  • Playing music overhead, in a coffee shop
  • A song I produced being embedded in a fundraising video by the Red Cross
  • Embedding my photo in a New York Times article
  • Embedding my photo in a Mother Jones article
  • Embedding my photo on Wikipedia (if you think this is a clear and easy example btw, perhaps you'd like to take a selfie with this monkey?)

But this actually isn't the most significant part of why noncommercial fails, has always failed, and will always fail in the scope of FOSS: it simply doesn't compose.

Using the (A)GPL as the approximate maxima (and not saying it's the only possible maxima) of license restrictions, we still have full composition from top to bottom. Lax and copyleft code can be combined and reused with all participants intending to participate in growing the same commons, and with all participants on equal footing.

Unfortunately, NC destroys the stack. NC has the kind of appeal that a lottery does: it's very fun to think about participating when you imagine yourself as the recipient. The moment you have to deal with it underneath, it becomes a huge headache.

I had an argument about this with someone I tend to work closely with, they began arguing for the need to insert NC style clauses into code, because developers gotta eat, which is at any rate a point I don't disagree with. But recently several of the formerly FOSS infrastructure switched to using an NC license, and they began to express that this opened up a minefield underneath them. If it felt like a minefield with just one or two libraries or utilities following the NC path, what will happen once it's the whole system?

What would using Debian be like if 1/4 of the packages were under NC licenses? Would you deploy it on your home machine? Would you deploy it on your personal VPS? Would you deploy it at your corporate datacenter? Even if I am a "noncommercial" user, if my VPS is at Linode, would Linode have to pay? What about Amazon? Worse yet... what if some of the package authors were dead or defunct?

To me it's no coincidence that we're seeing an interest in NC right at exactly the same time that faith in proprietary relicensing of copyleft code as a business strategy has begun to wane. If you were at my talk at CopyleftConf, you may have heard me talk about this in some detail (and some other things that aren't relevant to this right now). You can see the original table on slide 8 from my presentation, but here it is reproduced:

               Libre Commoner           Proprietary Relicensor

Motivation Protect the commons Develop income

Mitigating Tragedy of the commons Free rider problem

Wants Compliance Non-compliance

(By "tragedy of the commons", here I mean "prevent the commons from being eaten away".)

The difference in the "wants" field is extremely telling: the person I will call the "libre commoner" wants everyone to be able to abide by the terms of the license. The "propretary relicensor" actually hopes and dreams that some people will not comply with the license at all, because their business strategy depends on it. And in this part, I agree with Rob's "No-Community" pun.

Let me be clear: I'm not arguing with the desire to pay developers in this system, I'm arguing that this is a non-solution. To recap, here are the problems with noncommercial:

  • What commercial/noncommercial are is hard to define
  • NC doesn't compose; a tower of noncommercial-licensed tools is a truly brittle one to audit and resolve
  • The appeal of NC is in non-compliance

Noncommercial fails in its goals and it fails the community. It sounds nice when you think you'll be the only one on top, but it doesn't work, and it never will.

telnet 2333

By Christine Lemmer-Webber on Thu 19 December 2019


Try running "telnet 2333" in your terminal... you won't regret it, I promise! You should get to see a fun "season's greetings postcard" animated live in your terminal. My friends at Software Freedom Conservancy are running a supporters drive and I was happy to do this to help (if you were a supporter in the past, you should be getting a matching postcard in the mail!)

As you probably guessed, this is built on top of the same tech stack as Terminal Phase. Speaking of, at the time of writing we're getting quite close to that... only $40/month to go! Maybe we'll make it by the end of the year?

In the meanwhile, if you want to show your support for Conservancy and convince your friends to join, there are some matching banners you can put up on your website! And if you want to run the animated postcard yourself or generate any of its associated images, well... it's free software!

Terminal Phase: building a space shooter that runs in your terminal

By Christine Lemmer-Webber on Thu 07 November 2019

Terminal Phase runningincool-retro-term.{.align-center}

Yeah you read (and saw, via the above gif) that right! A space shooter! That runs in your terminal! Pew pew pew!

Well it's most of one, anyway. It's a prototype that I built as a test program for Spritely Goblins.

I've satisfied the technical needs I had in building the program; I might still finish it as a game, and it's close enough where making a satisfying game rather than just a short demo is super feasible, but I've decided to see whether or not there's actually enough interest in that at all by leaving that as a milestone on my Patreon. (We're actually getting quite close to meeting it... would be cool if it happened!)

But what am I, a person who is mostly known for work on a federated social web protocol, doing making a game demo, especially for a singleplayer game? Was it just for fun? It turns out it has more to do with my long term plans for the federated social web than it may appear.

And while it would be cool to get something out there that I would be proud of for its entertainment value, in the meanwhile the most interesting aspects of this demo to me are actually the technical ones. I thought I'd walk through what those are in this post, because in a sense it's a preview of some of the stuff ahead in Spritely. (Now that I've written most of this post, I have to add the forewarning that this blogpost wanders a lot, but I hope all the paths it goes down are sufficiently interesting.)

Racket and game development

Before we get to the Spritely stuff, I want to talk about why I think Racket is an underappreciated environment for making games. Well, in a certain sense Racket been advertised for its association with game-making, but in some more obscure ways. Let's review those:

  • Racket has historically been built as an educational teaching-programming environment, for two audiences: middle schoolers, and college freshmen. Both of them to some degree involve using the big-bang "toy" game engine. It is, by default, functional in its design, though you can mix it with imperative constructs. And yet maybe to some that makes it sound like it would be more complicated, but it's very easy to pick up. (DrRacket, Racket's bundled editor, also makes this a lot easier.) If middle schoolers can learn it, so can you.
  • Along those lines, there's a whole book called Realm of Racket that's all about learning to program by making games. It's pretty good!
  • Game studio Naughty Dog has used Racket to build custom in-house tools for their games.
  • Pioneering game developer John Carmack built prototypes for the Oculus Rift using Racket. (It didn't become production code, though John's assessments of Racket's strengths appear to align with my own largely; I was really pleased to see him repost on twitter my blogpost about Racket being an acceptable Python. He did give a caveat, however.)

So, maybe you've already heard about Racket used in a game development context, but despite that I think most people don't really know how to start using Racket as a game development environment. It doesn't help that the toy game engine, big-bang, is slowed down by a lot of dynamic safety checks that are there to help newcomers learn about common errors.

It turns out there are a bunch of really delightful game development tools for Racket, but they aren't very well advertised and don't really have a nice comprehensive tutorial that explains how to get started with them. (I've written up one hastily for a friend; maybe I should polish it up and release it as its own document.) Most of these are written by Racket developer Jay McCarthy, whom I consider to be one of those kind of "mad genius hacker" types. They're fairly lego-like, so here's a portrait of what I consider to be the "Jay McCarthy game development stack":

  • Lux, a functional game engine loop. Really this is basically "big-bang for grown-ups"; the idea is very similar but the implementation is much faster. (It has some weird naming conventions in it though; turns out there's a funny reason for that...)
  • By default Lux ships with a minimal rendering engine based on the Racket drawing toolkit. Combined with (either of) Racket's functional picture combinator libraries (pict or the 2htdp/image library), this can be used to make game prototypes very quickly, but they're still likely to be quite slow. Fortunately, Lux has a clever design by which you can swap out different rendering engines (and input mechanisms) which can compose with it.
  • One of these engines is mode-lambda which has the hilarious tagline of "the best 2d graphics of the 90s, today!" If you're making 2d games, this library is very fast. What's interesting about this library is that it's more or less designed off of the ideas from the Super Nintendo graphics engine (including support for Mode-7 style graphics, the graphic hack that powered games like the Super Nintendo version of Mario Kart). Jay talks about this in his "Get Bonus!" talk from a few years ago. (As a side note, for a while I was confused about Get Bonus's relationship to the rest of the Jay McCarthy stack; at 2018's RacketCon I queried Jay about this and he explained to me that that's more or less the experimental testing grounds for the rest of his libraries, and when the ideas congeal they get factored out. That makes a lot of sense!)
  • Another rendering engine (and useful library in general) is Jay's raart library. This is a functional picture library like pict, but instead of building up raster/vector graphic images, you're building up ascii art. (It turns out that ascii art is a topic of interest for me so I really like raart. Unsurprisingly, this is what's powering Terminal Phase's graphics.)

As I said before, the main problem with these is knowing how to get started. While the reference documentation for each library is quite good, none of them really have a tutorial that show off the core ideas. Fortunately each project does ship with examples in its git repository; I recommend looking at each one. What I did was simply split my editor and type in each example line by line so I could think about what it was doing. But yeah, we really could use a real tutorial for this stuff.

Okay, so 2d graphical and terminal programs are both covered. What about 3d? Well, there's really two options so far:

  • There's a very promising library called Pict3d that does functional 3d combinators. The library is so very cool and I highly, highly recommend you watch the Pict3d RacketCon talk which is hands down one of the coolest videos I've ever seen. If you use DrRacket, you can compose together shapes and not only will they display at the REPL, you can rotate around and view them from different angles interactively. Unfortunately it hasn't seen much love the last few years and it also mostly only provides tools out of the box for very basic geometric primitives. Most people developing 3d games would want to import their meshes and also their skeletal animations and etc etc and Pict3d doesn't really provide any of that yet. I don't think it even has support for textures. It could maybe have those things, but probably needs development help. At the moment it's really optimized for building something with very abstract geometry; a 3d space shooter that resembles the original Star Fox game could be a good fit. (Interestingly my understanding is that Pict3d doesn't really store things as meshes; I think it may store shapes in their abstract math'y form and raytrace on the GPU? I could be wrong about this.)
  • There's also an OpenGL library for Racket but it's very, very low level. (There's also this other OpenGL library but I haven't really looked at it.) However raw OpenGL is so extremely low level that you'll need to build a lot of abstractions on top of it to make it usable for anything.

So that covers the game loop, input, display.

That leaves us with audio and game behavior. I'll admit that I haven't researched audio options sufficiently; rsound seems like a good fit though. (I really need a proper Guix package of this for it to work on my machine for FFI-finding-the-library reasons...)

As for game behavior, that kind of depends on what your game is. When I wrote Racktris I really only had a couple of pieces of state to manage (the grid, where the falling piece was, etc) and it was fairly easy to do it using very basic functional programming constructs (really you're just creating a new version of these objects every time that a tick or input happens). As soon as I tried moving to a game with many independently operating objects doing their own thing, that approach fell apart.

The next thing I tried using was Racket's classes and object system. That was... okay, but it also felt like I was losing out on a lot. Previously I had the pleasant experience that "Woo, yeah! It's all functional!" A functional game engine is nice because returning to a prior snapshot in time is always possible (you can trivially make a game engine where you can rewind time, for instance; great for debugging) and it's much easier to poke and prod at a structure because you aren't changing it, you just are getting it back. Now I lost that feature.

In that conversation I had with Jay at RacketCon 2018, he suggested that I look at his DOS library for game behavior. I won't as quickly recommend this one; it's good for some game designs but the library (particularly the DOS/Win part) is fairly opinionated against objects ("processes") communicating with each other on the same tick; the idea is that each processes only can read what each other wrote on the previous tick. The goal here as I understand it is to eliminate an entire class of bugs and race conditions, but I quickly found that trying to work around the restrictions lead me to creating terrible hacks that were themselves very buggy.

This became really clear to me when I tried to implement a very simple wild west "quick draw" game a-la the Kirby quick draw and Samurai Kirby type games. (All this happened months back, when I was doing the anniversary animation I made earlier this year.) These are very simple games where two players wait for "Draw!" to appear on the screen before they press the button to "fire their gun". Fire first, you win. Fire second, you lose. Fire too early, you lose. Both fire at the same time, you draw. This is a very simple game, but trying to build it on top of DOS/Win (or my DOS/Hurd variant) was extremely hard to do while splitting the judge and players into separate objects. I ended up writing very contorted code that ultimately did communicate on the same tick, but via a layered approach that ended up taking me an hour to track down all the bugs in. I can't imagine scaling it up further.

But DOS had some good ideas, and I got to thinking about how to extend the system to allow for immediate calls, what would it look like? That's when I hit a series of epiphanies which resulted in a big rewrite of the Spritely Goblins codebase (which turned out to make it more useful for programming game behavior in a way that even fits very nicely into the Lux game loop). But I suppose I should really explain the what and why of Spritely Goblins, and how it fits into the larger goals of Spritely.

Terminal Phase and Spritely Goblins and stuff

Spritely Goblins is part of the larger Spritely project. Given Spritely's ambitious goal of "leveling up" the fediverse by extending it into the realm of rich and secure virtual worlds, we have to support distributed programming in a way that assumes a mutually suspicious network. (To get in the right mindset for this, maybe both watch my keynote and Mark Miller's keynote from the ActivityPub conference.) We really want to bake that in at the foundation of our design to build this right.

Thus Spritely Goblins is an asynchronous actor-ish distributed programming system on top of Racket. Kind of like Erlang, but with a focus on object capability security. Most of the good ideas have been taken from the E programming language ("the most interesting programming language you've never heard of"). The only programming environments I would consider viable to build Spritely on top of are ones that have been heavily informed by E, the best other candidate being the stuff Agoric is building on top of Javascript, such as their SwingSet architecture and Jessie (big surprise, since the folks behind E are mostly the folks behind Agoric), or some other more obscure language environments like Monte or, yes, Goblins. (Though, currently despite hanging out in Racket-land, which drinks deeply into the dream of everyone building their own languages, Goblins is just a library. If you want to run code you don't trust though, you'll have to wait until I release Spritely Dungeon, which will be a secure module / language restriction system for Racket. All in due time.)

Spritely Goblins already has some interesting properties:

  • All objects/actors are actually just procedures, waiting to be invoked! All "state" is merely the lexical scope of the enclosed procedure. Upon being invoked, a procedure can both return a value to its invoker (or in asynchronous programming, that fulfills the promise it is listening to) as well as specify what the next version of itself should be (ie, what procedure should be called the next time it handles a message).
  • Objects can only invoke other objects they have a reference to. This, surprisingly, is a sufficient security model as the foundation for everything we need (well, plus sealers/unsealers but I won't get into those here). This is the core observation from Jonathan Rees's A Security Kernel Based on the Lambda Calculus; object capability security is really just everyday programming via argument passing, which pretty much all programmers know how to do. (This applies to modules too, but more on that in a future post.)
  • In most cases, objects live in a "vat". This strange term from the object capability literature really means an event loop. Objects/actors can send messages to other objects in other vats; for the most part it doesn't matter where (on what machine, in what OS process, etc) other objects are when it comes to asynchronous message passing.
  • When asynchronous message passing, information is eventually resolved via promises. (Initially I had promises hidden behind coroutines in the core of the system, but it turns out that opens you to re-entrancy attacks if you aren't very careful. That may come back eventually, but with great care.)
  • While any object can communicate with any other object on any vat via message passing, objects on the same vat can do something that objects on separate vats can't: they can perform immediate calls (ie, something that looks like normal straight-ahead programming code, no coroutines required: you invoke the other object like a procedure, and it returns with a value). It turns out this is needed if you want to implement many interesting transactional things like financial instruments built on top of pure object capabilities. This also is nice for something like a game like Terminal Phase, where we really aren't doing anything asynchronous, are running on a fixed frame rate, and want to be deterministic. But a user should remember that (for important reasons I won't get into in this post) that immediate calls are strictly less universal than asynchronous message passing, since those can only be done between objects in the same vat. It's pleasant that Goblins can support both methods of development, including in an intermixed environment.
  • There is actually a lower level of abstraction than a vat, it turns out! This is something that is different than both E and Agoric's SwingSet I think and maybe even mildly novel; all the core operations (receiving a message, spawning an actor, etc) to operate on the actormap datastructure are exposed to the user. Furthermore, all of these operations are transactional! When using the lower-level actormap, the user receives a new actormap (a "transactormap") which is a delta to the parent actormap (either another transactormap or the root protected weak-hashtable actormap, a "whactormap").
  • This transactionality is really exciting. It means that if something bad happens, we can always roll back to a safe state (or rather, never commit the unsafe state at all). In the default vat, if a message is received and an uncaught exception occurs, the promise is broken, but all the effects caused by interactions from unhandling the message are as if they never occured. (Well that is, as long as we use the "become this procedure" mechanism in Goblins to manage state! If you mutate a variable, you're on your own. A Racket #lang could prevent your users from doing such naughty things if you so care.)
  • It also means that snapshotting an actormap is really easy. Elm used to advertise having a "time traveling debugger" where they showed off Mario running around, and you could reverse time to a previous state. Apparently this was removed but maybe is coming back. Anyway it's trivial to do such a thing with Goblins' actormap, and I built such a (unpublished due to being unpolished) demo.
  • Most users won't work with the actormap though, they'll work with the builtin vat that takes care of all this stuff for them. You can build your own vat, or vat-like tools, though.

Anyway, all the above works and exists. Actors can even speak to each other across vats... though, what's missing so far is the ability to talk to other objects/vats on other machines. That's basically what's next on my agenda, and I know how to do it... it's just a matter of getting the stuff done.

Well, the other thing that's missing is documentation. That's competing for next thing on the agenda.

But why a synchronous game right now?

If the really exciting stuff is the distributed secure programming stuff, why did I stop to do a synchronous non-distributed game on top of Spritely Goblins? Before I plowed ahead, given that the non-distributed aspects still rest on the distributed aspects, I wanted to make sure that the fundamentals of Spritely Goblins were good.

A space shooter is simple enough to implement and using ascii art in a terminal meant I didn't need to spend too much time thinking about graphics (plus it's an interesting area that's under-explored... most terminal-based games are roguelikes or other turn-based games, not real time). Implementing it allowed me to find many areas that could be improved usability-wise in Goblins (indeed, it's been a very active month of development for Goblins). You really know what things are and aren't nice designs by using them.

It's also a great way to identify performance bottlenecks. I calculated that roughly 1 million actor invocations could happen per second on my cheapo laptop... not bad. But that was when the actors didn't update themselves; when it came to the transactional updates, I could only seem to achieve about 65k updates per second. I figured this must be the transactionality, but it turns out it wasn't; the transactionality feature is very cheap. Can you believe that I got a jump from 65k updates per second to 680k updates per second just by switching from a Racket contract to a manual predicate check? (I expected a mild performance hit for using a contract over a manual predicate, but 10x...?) (I also added a feature so you can "recklessly" commit directly to the actormap without transactions... I don't recommend this for all applications, but if you do that you can get up to about 790k updates per second... which means that transactionality adds only about a 17% overhead, which isn't even close to the 10x gap I was seeing.) Anyway, the thing that lead me to looking into that in the first place was doing an experiment where I decided I wanted to see how many objects I could have updating at once. I might not have caught it otherwise. So making a game demo is useful for that kind of thing.

I feel now that I've gotten most of the churn out of that layer of the design out of the way so that I can move forward with the design on the distributed side of things next. That allows me to have tighter focus of things in layers, and I'm happy about that.

What's next?

So with that out of the way, the next task is to work on both the mutually suspicious distributed programming environment and the documentation. I'm not sure in which order, but I guess we'll find out.

I'll do something similar with the distributed programming environment as well... I plan to write something basic which resembles a networked game at this stage to help me ensure that the components work nicely together.

In the meanwhile, Terminal Phase is very close to being a nice game to play, but I'm deciding to leave that as a funding milestone on my Patreon. This is because, as far as my technical roadmap has gone, Terminal Phase has performed the role it needs to play. But it would be fun to have, and I'm sure other people would like to play it as a finished game (heck, I would like to play it as a finished game), but I'd like to know... do people actually care enough about free software games? About this direction of work? Am I on the right track? Not to mention that funding this work is also simply damn hard.

But, at the time of writing we're fairly close, (about 85% of the way there), so maybe it will happen. If it sounds fun to you, maybe pitch in.

But one way or another, I'll have interesting things to announce ahead. Stay tuned here, or follow me on the fediverse or on Twitter if you so prefer.

Onwards and upwards!

Updates: ActivityPub Conference, and more

By Christine Lemmer-Webber on Tue 01 October 2019

NOTE: This update also appeared on my Patreon account. If you're reading the below and it sounds like I'm doing a lot of useful work, consider becoming a patron!

Hello all! It's been a couple of months since I've gotten out an update; much has happened.

First of all, ActivityPub Conf happened and was a big success! The video team got things recorded an uploaded so you can watch talks from the event, including my keynote (the audio quality is a bit messed up on this one, the others are better) and Mark Miller's keynote. The other talks were all also very excellent; I'm not going to iterate them all here because you can already go watch them! I think you will find there are many thematic threads between the videos.

We had about 40 people at the event; the first day was spent on talks and the second day was an "unconference" where groups self-organized to discuss various topics of mutual interest. One common thread was about the kinds of directions I've been pushing for in Spritely: distributed encrypted storage (Datashards, with Serge Wroclawski leading the conversation on that), object capabilities (OcapPub), stamps, etc. It was interesting to watch from the start to the end of the unconference day; particularly, the Pleroma folks were there and gave a lot of feedback. Towards the start of the day I think there was much more skepticism, but towards the end we were hearing belief and interest that these kinds of things could and should be implemented and would be of real use to fediverse participants. Lain of Pleroma in particular expressed that it helped to realize that even though I'm presenting all these ideas, they don't need to be implemented all at once; we can take them on piecemeal, and incrementalism is a perfectly valid approach. (Also "OcapPub" sounds like a new protocol, whereas it's really just a way-to-use ActivityPub mostly as it already exists. Maybe time for a new name for that?)

Anyway, ActivityPub Conf was a massive success; thank you everyone who came and participated. It's clear after APConf to me just how much of a difference getting folks together can make. For those who couldn't make it, let's thank the video team (DeeAnn Little, Sebastian Lasse, Markus Feilner) for getting those videos up!

On the topic of Datashards, we have a website and a nice logo now (courtesy of mray, who also made the ActivityPub logo). Serge Wroclawski (co-host with myself of Libre Lounge) has been increasingly helping with the project; before ActivityPub Conference and Rebooting Web of Trust we worked to make sure both of our implementations could talk to each other (Serge's Python implementation and my Racket implementation). At RWoT we showed a demo where I "beamed" the death star plans to Serge's computer. (We used the same content storage server, I uploaded the death star plans, rendered the QR code on my laptop, Serge scanned the QR code from his laptop, downloaded the file and showed off the plans from his computer... with the storage server having no idea about the contents of the data we were storing there!) People really liked that demo; we have had conversations about whether Datashards may serve as a foundational system for some other tools being made in that space; more later. In the meanwhile, I'm happy we have two applications in two different languages successfully being able to read and write each others' immutable datashards updates; the next step is making sure that mutability works the same.

Rebooting Web of Trust was also a very interesting event; the highlight being that I am now collaborating with some great folks on a secure user interfaces paper. We are taking the existing Mastodon web user interface and retooling it to reduce risks such as phishing, and open the path for more peer to peer systems (very timely, since that's the direction we want to take things). Unfortunately the amount of work to do on the paper is rather huge; it may take a while until the paper is complete. In the meanwhile, the Petnames paper has turned out to be a pre-requisite for the secure UIs one; that paper has been nearly complete for some time so I guess I have to finish the work. I recently added some new UI mockups to it, but there is still more to do.

Now that these conferences are over, I am putting time towards Spritely's core vision again: distributed virtual worlds. The foundational layer for that is Spritely Goblins, an ocap-secure distributed programming environment on top of Racket. I really enjoy hacking on this and I am happy to get time back to working on it. A topic of discussion came up between myself and Mark Miller at Rebooting Web of Trust though; am I unnecessarily duplicating effort between myself and the Agoric folks? In particular, they are building something equivalent (and arguably more featureful) to Spritely Goblins named SwingSet. This would run on top of Javascript rather than Racket/Lisp/Scheme. I have found that in the past I have been not very happy when working with Javascript, but Mark suggested I take a look at Agoric's Jessie subset of Javascript, which Mark described as "closer to Scheme" (not syntactically, but in terms of language-cleanliness). It does seem nicer than Javascript; when I admitted to Mark that I am addicted to parenthetical syntax, Mark posed the question about whether building a parenthetical version of Jessie would be less work than reproducing all the other things that Agoric is doing. It's a good point; I don't know. I'm unhappy with the idea of pivoting, but I do feel like it's probably true that due diligence suggests I should consider it carefully. It is true at least that I would probably reach a broader userbase more quickly with the option of Javascript syntax; it's hard for me to deny that. I will probably explore it with some smaller tests of Agoric's stuff. But in the meanwhile, I currently plan to release a very small version of the game demo using the toolkit I already am building while testing Agoric's infrastructure in parallel. I suspect we'll see the first user-visible outputs of this in early 2020.

There have been four new Libre Lounge episodes since my last update. That's still quite a few episodes to listen to, but slower than we previously were updating; all the travel is to blame. However that is settling down and I think we'll be updating more frequently soon. Even so, we have been updating!

In addition to all this, I suspect there will be at least two major announcements in the coming months; stay tuned. Work has already occured on both, but I can only say so much right now.

Thanks to everyone who has supported my work. I work much more than full time in the cause of advancing user freedom; it's not easy to fund this work. I appreciate all of you who are giving what you can.

Now, back to work!

ActivityPub Conf 2019 Speakers

By Christine Lemmer-Webber on Wed 07 August 2019

Good news everyone! The speaker list for ActivityPub Conf 2019 is here! (In this document, below, but also in ODT and PDF formats.)

(Bad news everyone: registration is closed! We're now at 40 people registered to attend. However, we do aim to be posting recordings of the event afterwards if you couldn't register in time.)

But, just in case you'd rather see the list of speakers on a webpage rather than download a document, here you go:

Keynote: Mark Miller, “Architectures of Robust Openness”

Description coming soon! But we're very excited about Mark Miller keynoting.

Keynote: Chris Lemmer-Webber, “ActivityPub: past, present, future”

This talk gives an overview of ActivityPub: how did we get to this point? Where are we now? Where do we need to go? We'll paint a chart from past to a hopeful future with better privacy, richer interactions, and more security and control for our users.

Matt Baer, “Federated Blogging with WriteFreely”

We're building out one idea of what federated blogging could look like with separate ActivityPub-powered platforms, WriteFreely and -- one for writing and and one for reading. Beyond the software, we're also offering hosting services and helping new instances spring up to make community-building more accessible, and get ActivityPub-powered software into more hands. In this talk I'll go over our approach so far and where we're headed next.

Caleb James DeLisle, “The case for the unattributed message”

Despite it's significant contribution to internet culture, the archetype of the anonymous image board has been largely ignored by protocol designers. Perhaps the reason for this is because it's all too easy to conflate unattributed speech with unmoderated speech, which has shown itself to be a dead end. But as we've seen from Twitter and Facebook, putting a name on everything hasn't actually worked that well at improving the quality of discourse, but what it does do is put already marginalized people at greater risk.

What I credit as one of the biggest breakthroughs of the fediverse has been the loose federation which allows a person to choose their moderator, completely side stepping the question of undemocratic censorship vs. toxic free speech. Now I want to start a conversation about how we might marry this powerful moderation system to a forum which divorces the expression of thought from all forms of identity.

Cristina DeLisle, “OSS compliance with privacy by default and design”

Privacy is becoming more and more central in shaping the future of tech and the data protection legislation has contributed significantly to making this happen. Privacy by default and design are core principles that are fundamental to how software should be envisioned. The GDPR that came into the spotlight has a strong case to become a standard even outside European borders, influencing the way we protect personal data. However its impact might be, its implementation is still in its infancy. OSS has found itself facing the situation and one aspect which is particularly interesting on the tech side is how to incorporate the principles of privacy by default and design into the software that we build.

This talk is going to be an overview of how the GDPR has impacted FOSS communities, what do we mean by privacy by default and by design, how could we envision them applied in our OSS. It will bring examples from which we might find something interesting to learn from, regardless if we are looking at them as mistakes, best practices or just ways of doing things.

Michael Demetriou, “I don't know what I'm talking about: a newbie's introduction to ActivityPub”

I have just started my development journey in ActivityPubLand and I hope to have a first small application ready before ActivityPubConf. I was thinking that since I have close to zero experience with ActivityPub development, I could document my first month of experience, describe the onboarding process and point out useful resources and common pitfalls. In the end I can showcase what I've done during this period.

Luc Didry, “Advice to new fediverse administrators and developers”

Hosting an ActivityPub service is not like hosting another service… and it's the same for developing ActivityPub software. Here is some advice based on Framasoft's experience (we host a Mastodon instance and develop two ActivityPub software: PeerTube and Mobilizon – the last one is not yet out), errors and observations.

Maloki, “Is ActivityPub paving the way to web 3.0?”

A talk about how we're walking away from Web 2.0, and paving the way to Web 3.0 with ActivityPub development. We'll discuss what this could mean for the future of the web, we'll look at some of the history of the web, and also consider the social implications moving forward.

Pukkamustard, “The Semantic Social Network”

ActivityPub uses JSON-LD as serialization. This means [@context]{.title-ref} field all over the place. But really there is more behind this: ActivityPub speaks Linked Data. In this talk we would like to show what this means and how this can be used to do cool things. We might even convince you that the Fediverse is a huge distributed graph that could be queried in very interesting ways - that the Fediverse is a Semantic Social Network.

Schmittlauch, “Decentralised Hashtag Search and Subscription in Federated Social Networks”

Hashtags have become an important tool for organising topic-related posts in all major social networks, even having managed to spark social movements like #MeToo. In federated social networks, unfortunately so far the view on all posts of a hashtag is fragmented between instances.

For a student research paper I came up with an architecture for search and subscription of hashtag-posts in federated social networks. This additional backend for instances augments the Fediverse with a little bit of P2P technology.

As this architecture is still at a conceptual stage, after presenting my work I'd like to gather ideas and feedback from various Fediverse stakeholders: What do global hashtags mean for marginalised people and moderation, are they more a tool of empowerment or of harassment? How can this concept be represented in the ActivityPub protocol? And what stories do server devs have to tell about common attack scenarios?

Serge Wroclawski, “Keeping Unwanted Messages off the Fediverse”

Spam, scams and harassment pose a threat to all social networks, including the Fediverse. In this talk, we discuss a multilayered approach to mitigating these threats. We explore spam mitigation techniques of the past as well as new techniques such as OcapPub and Postage.