Well, I'm late in putting this one out there, but it's still worth putting on the record! About a month ago, I was fortunate enough to receive the O'Reilly Open Source Award. In fact, here's a picture of me receiving it!

Photo taken by Brandin Grams, CC BY 4.0, originally microblogged by Karen Sandler

(... there's a video too!)

So, getting the award was exciting and unexpected. Exciting, because just look at the list of people who have received this award previously!

Even to just share the stage with friends Marina Zhurakhinskaya of Outreachy, Stefano Zacchiroli of Debian, and Sarah Mei was a huge honor (okay, Sarah Mei is not someone I know, but I know very well of Railsbridge and the many ways it has paved forward diversity initiatives in free software). Plus, even though I don't know much about Hadoop, I know others who do, and the looming head of Doug Cutting behind us in videorecording excited them! And looking at former recipients, nearly everyone on this list is a person who has made humongous strides in shaping the world of free and open source software, and let's face it, almost everyone on this list has done more in that capacity than I have.

Which leads me to the surprise part... I was really not expecting this award, so when I saw the email informing me I was being awarded it I began to flag it as spam, assuming it was something that snuck past spamassassin. But wait, I actually know this award! I then preceded to try to get Morgan's attention (she was on the phone) by exuberantly waving my arms, which lead our dog to jump up and also wave her forelegs in a very similar fashion.

So that is all to say I'm very honored to have received the thing!

So, then I knew I received the award, but didn't know for what text I was nominated until the ceremony, but I was really happy with it when I heard it said: for my free software advocacy (and with those words) and my work on GNU MediaGoblin (GNU was pronounced with each letter as if a university chant... G! N! U!). I suspected as much and I'm glad I received the award under that description.

And that is also to say I am really glad for the recognition!

The recognition does help, in some way, as a counterbalance to other feelings. In another sense, I also look at the list of other people who have received the award and I feel a bit embarrassed because I think maybe I don't deserve it. I had confided in this to some friends (I guess it's no longer confiding now that I'm writing in a blogpost), and the response has mostly been something along the lines of "of course you do, just feel good about the thing", and of course that's also the kind of thing you want to hear from your friends, and it would be pretty embarrassing if they didn't say those kinds of things in response, and you begin to wonder that given that if you're playing some sort of scripted roles, and typing this I even wonder if this "I don't deserve it" text is doing that too, and maybe it is, maybe everything is... but I still think sometimes I am fooling everyone and when I am given recognition that I am some kind of tricky person, tricky enough to trick people into giving me an award for a pile of incomplete things.

I think there are a variety of reasons for this, one of the more obvious being that MediaGoblin is moving along but it isn't there yet, something I think nobody is more deeply aware of than myself. And people are still locked down by proprietary network services, we still don't have a generally agreed upon federation standard, and it is still really hard to run your own server. These aren't my battles alone and a good number of us are working on them, (and I even have interesting things to report in these areas that I have not yet blogged about) but when I think about things not being done I feel personally responsible for it. Part of the challenge also is that I do not generally look at my life as in terms of things I have accomplished, but in terms of the things I haven't, and obviously that is not a great strategy for feeling good about the things you've done.

But let me turn this blogpost back around again, before I look like some kind of jerk who gets an award and then mopes about it!

In this same sense the recognition has helped a lot. Not everything that I want to see done is done yet, but the O'Reilly Open Source Award selection process happens through previous winners, so clearly people who know better than I do have recognized that the things we've done and the direction we're heading are the right ones.

You may notice that I'm switching between "I" and "we", so a bit more on that... it's worth noting that any accomplishments I have are connected to some significant free software community, so they're hardly just "my" accomplishments... that statement of indicating that the roads we're heading down are the right ones applies to all the people in the communities of which I've become some apparent type of minor figurehead. (Be wary of pedestals, with coordination like mine, a great way to dash oneself against the floor...)

So! That's a lot of words there, but the crux it is that I'm still excited to have gotten the award, and as much of a worrier as I am, the recognition is especially nice as a kind of reassurance. In the meanwhile, I have a fancy new bookshelf and a nice little display for this hunk of glass:

Pretty cool hunk of glass, right? I think so!

PS: John Sullivan snapped this image of Stefano Zacchiroli and I right after the ceremony, on a trip to Powell's books.

"Award Winners" photo taken by John Sullivan CC BY-SA 4.0

Last night at OSCON I attended the lightning talks (here called "Ignite Talks"). Most of them were pretty good (I especially loved Emily Dunham's "First Impressions (the value of the 'noob')" talk), but the last talk of the night was titled "Why I don’t use the GPL" by Shane Curcuru, "VP of Brand Management at the Apache Software Foundation" (the association of which he invoked during his talk last night, which made me wonder if he was speaking on behalf of the ASF, which seemed surprising). (Edit: this was confirmed to not have been intended to be speaking on behalf of the ASF, which is good to hear. I don't have a recording so I'm not sure if Shane invoked his association or if the person doing the introducing did.)

It was a harsh talk. It was also the last talk of the night, and there was really no venue to respond to it (I looked to see if there would be future lightning talk slots at this conference, but there aren't). Though the only noise from the audience was applause, I know that doesn't mean everyone was happy, just polite... a number of my friends got up and left in the middle of the talk. But it needs a response... even if the only venue I have at the moment is my blog. That'll do.

So let me say it up front: my name is Chris Lemmer-Webber, and I am pro-GPL and pro-copyleft. Furthermore, I'm even pro-permissive (or "lax") licensing; I see no reason our sides should be fighting, and I think we can work together. This is one reason why this talk was so disappointing to me.

There's one particular part of the talk that really got to me though: at one point Shane said something along the lines of "I don't use copyleft because I don't care about the source code, I care about the users." My jaw dropped open at that point... wait a minute... that's our narrative. I've written on this before (indeed, at the time I thought that was all I had to say on this subject, but it turns out that's not true), but the most common version of anti-copyleft arguments are a "freedom to lock down" position (see how this is a freedom to remove freedoms position?), and the most common form of pro-copyleft arguments are a "freedom for the end-user" position.

Now there is an anti-copyleft position which does take a stance that copyleft buys into a nonfree system -- you might see this from the old school BSD camps especially -- a position that copyright itself is an unjust system, and to use copyright at all, even to turn the mechanisms of an evil machine against itself as copyleft does, is to support this unjust system. I can respect this position, though I don't agree with it (I think copyleft is a convenient tactical move to keep software and other works free). One difficulty with this position though is to really stay true to it, you logically are against proprietary software far more than you are against copyleft, and so you had better be against all those companies who are taking permissively licensed software and locking it down. This is decidedly not the position that Shane took last night: he explicitly referenced that the main reason you want to use lax licensing and avoid copyleft is it means that businesses are more willing to participate. Now, there are a good number of businesses which do work with copyleft, but I agree that anti-copyleft sentiments are being pushed from the business world. So let me parse that phrasing for you: copyleft means that everyone has to give back the changes that build upon your work, and not all businesses want to do this. The "businesses are more willing to participate" means that businesses can use your project as a stepping stone for something they can lock down. Some businesses are looking for a "proprietary differentiation point" to lock down software and distinguish themselves from their competitors.

As I said, I am not only pro-copyleft, I am also pro-permissive licensing. The difference between these is tactics: the first tactic is towards guaranteeing user freedom, the second tactic is toward pushing adoption. I am generally pro-freedom, but sometimes pushing adoption is important, especially if you're pushing standards and the like.

But let's step back for a moment. One thing that's true is that over the last many years we've seen an explosion of free and open source software... at the same time that computers have become more locked down than ever before! How can this be? It seems like a paradox; we know that free and open source software is supposed to free users, right? So why do users have less freedoms than ever? Mobile computing, the rise of the executable web, all of this has FOSS at its core, and developers seem to enjoy a lot of maneuverability, but computers seem to be telling us more what we can and can't do these days than we tell them. And notice... the rise of the arguments for permissive/lax licensing have grown simultaneously with this trend.

Free Speech Zone by Mustafa and Aziza, CC BY-SA 2.0

This is no coincidence. The fastest way to develop software which locks down users for maximum monetary extraction is to use free software as a base. And this is where the anti-copyleft argument comes in, because copyleft may effectively force an entity to give back at this stage... and they might not want to.

In Shane's talk last night, he argued against copyleft because software licenses should have "no strings attached". But the very strategy that is advocated above is all about attaching strings! Copyleft's strings say "you can use my stuff, as long as you give back what you make from it". But the proprietary differentiation strategy's strings say "I will use your stuff, and then add terms which forbid you to ever share or modify the things I build on top of it." Don't be fooled: both attach strings. But which strings are worse?

To return to the arguments made last night, though copyleft defends source, in my view this is merely a strategy towards defending users. And indeed, as in terms of where freedoms lie between those who make use of the source and code side of things vs the end-user-application side of things, one might notice a trend: there are very few permissively licensed projects which aim at end users. Most of them are stepping stones towards further software development. And this is great! I am glad that we have so many development tools available, and it seems that permissive/lax licensing is an excellent strategy here. But when I think of projects I use every day which are programs I actually run (for example, as an artist I use Blender, Gimp and Inkscape regularly), most of these are under the GPL. How many truly major end-user-facing software applications can you think of that are under permissive licenses? I can think of many under copyleft, and very few under permissive licenses. This is no coincidence. Assuming you wish to fight for freedom of the end user, and ensure that your software remains free for that end user, copyleft is an excellent strategy.

I have heard a mantra many times over the last number of years to "give away everything but your secret sauce" when it comes to software development. But I say to you, if you really care about user freedom: give away your secret sauce. And the very same secret sauce that others wish to lock down, that's the kind of software I tend to release under a copyleft license.

There is no reason to pit permissive and copyleft licensing against each other. Anyone doing so is doing a great disservice to user freedom.

My name is Chris Lemmer-Webber. I fight for the users, and I'm standing up for the GPL.

Addendum: Simon Phipps points out that all free licenses are "permissive" in a sense. I agree that "permissive" is a problematic term, though it is the most popular term of the field (hence my inclusion also of the term "lax" for non-copyleft licenses). If you are writing about non-copyleft licenses, it is probably best to use the term "lax" licenses rather than "permissive".

The state of packaging of libre web applications is, let's face it, a sad one. I say this as one of the lead authors of one of such a libre web application myself. It's just one component of why deploying libre web applications is also such a sad state of affairs (hence userops). It doesn't help that, for a long time, the status quo in all free software web applications (and indeed all web applications) was to check javascript and similar served-to-client web assets straight into your repository. This is as bad as it sounds, and leads to an even further disconnect (one of many) between the packages that a truly free distro might include (and have to manually link in after the fact) and those of your own package. Your package is likely to become stuck on a totally old version of things, and that's no good.

So, in an effort to improve things, MediaGoblin and many other projects have kicked the bad habit of including such assets directly in our repository. Unfortunately, the route we are taking to do this in the next release is to make use of npm and bower. I really did not want to do this... our docs already include instructions to use Python's packaging ecosystem and virtualenv, which is fine for development, but since we don't have proper system packaging, this means that this is the route users go for deployment as well. Which I guess would be fine, except that my experience is that language package managers break all the time, and when they break, they generally require an expert in that language to get you out of whatever mess you're in. So we added more language package management features... not so great. Now users are even more likely to hit language package management problems, now also ones that our community are less of experts in helping debug.

But what can we do? I originally thought of home-rolling our own solution, but as others rightly pointed out, this would be inventing our own package manager. So, we're sucking it up and going the npm/bower route.

But wait! There may be a way out... recently I've been playing with Guix quite a bit, and I came to realize that, at least for myself in development, it could be nice to have all the advantages of transactional rollbacks and etc. There is a really nice feature in Guix called guix environment which is akin to a "universal virtualenv" (also similar to JHBuild in Gnome land, but not tied to Gnome specifically)... it can give you an isolated environment for hacking, except not just restricted to Python or Javascript or Ruby or C... great! (Nix has something similar called nix-shell.) I know that I can't expect that Guix is usable for everyone right now, but for many, maybe this could be a nice replacement for Virtualenv + Bower, something I wrote to the mailing list about.

(As an aside, the challenge wasn't the "virtualenv" type side of things (pulling in all the server-side dependencies)... that's easy. The challenge is replacing the Bower part: how to link in all the statically-served assets from the Guix store right into the package? It's kind of a dynamic linking problem, but for various reasons, linking things into the package you're working on is not really easy to do in a functional packaging environment. But thanks to Ludo's advice and thanks to g-expressions, things are working!)

I'm happy to say that today, thanks to the help from the list, I came up with such a Virtualenv + Bower replacement prototype using "guix environment". And of course I wanted to test this on MediaGoblin. So here I thought, well, how about just for tonight I test on something simple. How about jQuery? How hard could that be? I mean, it just compiles down to one file, jquery.js. (Well, two... there's also jquery.js.min...)

Luckily, Guix has Node, so it has npm. Okay, the docs say to do the following:

# Enter the jquery directory and run the build script:
cd jquery && npm run build

Okay, it takes a while... but it worked! That seemed surprisingly easy. Hm, maybe too easy. Remember that I'm building a package for a purely functional distribution: we can't have any side effects like fetching packages from the web, every package used has to be an input and also packaged for Guix. We need dependencies all the way up the tree. So let's see, are there any dependencies? There seems to be a [node_modules]{.title-ref} directory... let's check that:

cwebber@earlgrey:~/programs/jquery$ ls node_modules/
commitplease          grunt-contrib-uglify  grunt-npmcopy        npm                   sinon
grunt                 grunt-contrib-watch   gzip-js              promises-aplus-tests  sizzle
grunt-cli             grunt-git-authors     jsdom                q                     testswarm
grunt-compare-size    grunt-jscs-checker    load-grunt-tasks     qunitjs               win-spawn
grunt-contrib-jshint  grunt-jsonlint        native-promise-only  requirejs

Yikes. Okay, that's 24 dependencies... that'll be a long night, but we can do it.

Except, wait... I mean, there's nothing so crazy here as in dependencies having dependencies, is there? Let's check:

cwebber@earlgrey:~/programs/jquery$ ls node_modules/grunt/node_modules/
async          eventemitter2  glob               iconv-lite  nopt
coffee-script  exit           grunt-legacy-log   js-yaml     rimraf
colors         findup-sync    grunt-legacy-util  lodash      underscore.string
dateformat     getobject      hooker             minimatch   which

Oh hell no. Okay, jeez, just how many of these node_modules directories are there? Luckily, it's not so hard to check (apologies for the hacky bash pipes which are to follow):

cwebber@earlgrey:~/programs/jquery$ find node_modules -name "node_modules" | wc -l
158

Okay, yikes. There are 158 dependency directories that were pulled down recursively. Wha?? To look at the list is to look at madness. Okay, how many unique packages are in there? Let's see:

cwebber@earlgrey:~/programs/jquery$ find node_modules -name "node_modules" -exec ls -l {} \; | grep -v total | awk '{print $9}' | sort | uniq | wc -l
265

No. Way. 265 unique packages (the list in its full glory), all to build jquery! But wait... there were 158 [node_modules]{.title-ref} directories... each one of these could have its own repeat of say, the minimatch package. How many non-unique copies are there? Again, easy to check:

cwebber@earlgrey:~/programs/jquery$ find node_modules -name "node_modules" -exec ls -l {} \; | grep -v total | awk '{print $9}' | wc -l
493

So, there's about double-duplication of all these packages here. Hrm... (Update: I have been told that there is an npm dedupe feature. I don't think this reduces the onerousness of packaging outside of npm, but I'm glad to hear it has this feature!)

Well, there is no way I am compiling jQuery and all its dependencies in this state any time soon. Which makes me wonder, how does Debian do it? The answer seems to be, currently just ship a really old version from back in the day before npm, when you could just use a simple Makefile.

Well for that matter then, how does Nix do it? They're also a functional package management system, and perhaps Guix can take inspiration there as Guix has in so many other places. Unfortunately, Nix just downloads the prebuilt binary and installs that, which in the world of functional package management is kind of like saying "fuck it, I'm out."

And let's face it, "fuck it, I'm out" seems to be the mantra of web application packaging these days. Our deployment and build setups have gotten so complicated that I doubt anyone really has a decent understanding of what is going on, really. Who is to blame? Is it conventional distributions, for being so behind the times and for not providing nice per-user packaging environments for development? Is it web developers, for going their own direction, for not learning from the past, for just checking things in and getting going because the boss is leaning over your shoulder and oh god virtualenv is breaking again on the server since we did an upgrade and I just have to make this work this time? Whose fault is it? Maybe pinning blame is not really the best anyway, but I feel that these are conversations that we should have been having, for distributions and web applications to work together, at least a decade ago. And it's not just Javascript; we're hardly better in the Python world. But it's no wonder that the most popular direction of deployment is the equivalent of rolling a whole distro up into a static binary, and I don't have to tell you what a sad state that is.

For me, at the moment, I'd like to be more conscious of what it takes to build software, not less. Reproducibility is key to long-term software freedom, else how can we be sure that the software we're running is really the software we say it is? But given all the above, it's hard to not have empathy for those who instead decide to toss in that towel and take a "fuck it, I'm out" approach to deployment.

But I hope we can do better. In the meanwhile, ensuring that users can actually build and package from top to bottom the software I'm encouraging them to use is becoming more of a priority for me, not less.

And I guess that may mean, if it isn't really feasible to reproduce your software, I can't depend on it in my own.

NOTE: This is a shameless cross-post of something I originally sent to the userops list, where we discuss deployment things.

Hello all,

For a while I've been considering, why is it so harder for me to migrate from server to server than it is for me to migrate from desktop to desktop? For years, ever since I discovered rsync, migrating between machines has not been hard. I simply rsync my home directory over to the new machine (or maybe even just keep the old /home/ directory's partition where it is!) and bam, I am done. Backing this up is easy; it's just another rsync away. (I use dirvish as a simple wrapper around rsync so it can manage incremental backups.)

If I set up a new machine, it is no worry. Even if my current machine dies, it is mostly no worry. Rsync back my home directory, and done. I will spend a week or so discovering that certain programs I rely on are not there, and I'll install them one by one. In a way it's refreshing: I can install the programs I need, and the old cruft is gone!

This is not true for servers. At the back of my mind I realized this, but until the end of Stefano Zacchiroli's excellent LibrePlanet talk when I posed a question surrounding this situation, I hadn't totally congealed in my head: why is it so much harder for me to move from server to server? Assume I even have the old server around and I want to move. It isn't easy!

So here are some thoughts that come out of this:

• For my user on my workstation, configuration and data are in the same place: /home/ (including lots of little dotfiles for the configs, and the rest is mostly data). Sure, there's some configuration stuff in /etc/ and data in /var/ but it mostly doesn't really matter, and copying that between machines is not hard.
• Similarly, for my user on workstation experience, it is very little stress if I set up a machine and am missing some common packages. I can just install them again as I find them missing.
• Neither of these are true for my server! In addition to caring about /home/, and even more importantly, I have to worry about configuration in /etc/ and data in /var/. These are both pains in the butt for me for different reasons.
• Lots of stuff in /etc/ is configuration that interacts with the rest of the system in specific ways. I could rsync it to a new machine, but I feel like that's just blindly copying over stuff where I really need to know how it works and how it was set up with the rest of the machine in the first place.
• This is compounded by the fact that people rarely set up one machine these days; usually they have to set up several machines for several users. Remembering how all that stuff worked is hard. The only solution seems to be to have some sort of reproducible configuration system. Hence the rise of salt, ansible, etc. But these aren't really "userops" systems, they're "devops"... developer focused. Not only do you need to know how they work, you need to know how the rest of the system works. And it's not easy to share that knowledge.
• /var/ is another matter. Theoretically, most of my program data goes there (unless, of course, it went to /srv/, god help us). But I can't just rsync it! There are some processes that are very persnickety about the stuff there. I have to dump my databases and etc before I can move them or back up. Nothing sets up an automatic cronjob for me on these, I have to know to dump postgres. Hopefully I set up a cronjob!
• While I as a workstation user don't stress too much if I'm missing some packages (just install them as I go), that is NOT true of my servers. If my mail servers aren't running, if jabber isn't on, (if SSH isn't running!!!), there are other servers expecting to communicate with my machine, and if I don't set them up, I miss out.
• Not only this, assuming I have moved between servers correctly, even once I have set up my machine and it has become a perfectly okay running special snowflake, there are certain routine tasks that require a lot of manual intervention, and I have only picked up the right steps by knowing the right friends, having run across the right tutorials which hopefully have shown me the right setup, etc. SSL configuration, I'm looking at you; the only savior that I have is that I have written myself my own little orgmode notes on what to do the next time my certs expire.
• My servers do become special snowflakes, and that is very stressful to me. I will, in the future, need to set up one more server, and remembering what I did in the past will be very hard.
• Assuming I use all the mainstream tools, not talking about "upcoming" ones, a better configuration management solution is probably the answer, right? That's a lot to ask users though: it's not a solution to existing deployment, because it doesn't remove the need to know about all the layers underneath, it just adds a new layer to understand.

Those are all headaches, and they are not the only headaches. But here are some thoughts on things that can help:

• If I recognize which parts of my system are "immutable" and which parts are "mutable", it's easier to frame how my system works.

  • /var/ is mutable, it's data. There's no making this "reproducible" really: it needs to be backed up and moved around.

  • My packages and system are immutable, or mostly should be. Even if not using a perfectly immutable system like guix/nix, it's helpful to act like this part of the system is pseudo-immutable, and simply derived from some listing of things I said I wanted installed. (Nix/Guix probably do this the most nicely though.)

  • /etc/ is similarly "immutable but derived" in the best case. I should be able to give the same system configuration inputs and always get the same system of packages and configuration files.

• I like Guix/Nix, but my usage of Debian and Fedora and friends is not going away anytime soon. Nonetheless, configuration management systems like puppet/ansible/salt help give the illusion of an immutable system derived from a set of inputs, even though they are working within a mutable one.

• Language packaging for deployment needs to die. Yes, I say this as a project that advocates that very route. We're doing it wrong, and I want to change it. (Language packaging for development though: that's great!)

• Asking people to use systems like ansible/salt/puppet is asking users too much. You're just asking them to learn one more layer on top of knowing how the whole system works. Sharing common code is mostly copy and paste. There are some layers built on top of here to mitigate this but afaict they aren't really good, not good enough. (I am working on something to solve this...)

• Pre-built containers are not the solution. Sorry container people! Containers can be really useful but only if they are built in some reproducible way. But very few people using Docker and etc seem to be doing this. But here's another thing: Docker and friends contain their own deployment domain specific languages, which is dumb. If a reproducible configuration system is good enough, it should be good enough for a VM or a container or a vanilla server or a desktop. So maybe we can use containers as lightweight and even sandboxed VMs, but we shouldn't be installing prebuilt containers on our servers alone as a system.

  Otherwise else you're running 80 heavy and expensive Docker images that slowly go out of date... now you're not maintaining 1 distribution install, you're maintaining 81 of them. Yikes! Good luck with the next Shellshock!

• Before Asheesh jumps in here: yes I will say that Sandstorm is taking maybe the best route as in terms of a system that uses containers heavily (and unlike Docker, they seem actually sandboxed) in that it seems to have a separation between mutable parts and immutable parts: the container is more or less an immutable machine from what I can tell that has /var/ mounted into it, which is a pretty good route.

  In this sense I think Sandstorm has a good picture of things. There are other things that I am still very unsure about, and Asheesh knows because I have expressed them to him (I sure hope that iframe thing goes away, and that daemons like Celery can run, and etc!) but at least in this sense, Sandstorm's container story is more sane.

So there are some reflections in case you are planning on debugging why these things are hard.

-- Chris

PS: If you haven't gotten the sense, the direction I'm thinking of is more along the lines of Guix becoming our Glorious Future (TM) assuming something like GuixOps can happen (go Dave Thompson, go Guix crew!) and a web UI can be built on top of it with some sort of common recipe system.

But I don't think our imperative systems like Debian are going away anytime soon; I certainly don't intend to move all my stuff over to Guix at this time. For that reason, I think there needs to be another program to fit the middle ground: something like salt/ansible/puppet, but with less insane one-off domain specific languages, with a sharable recipe system, and scalable both from developer-oriented scripts but also having a user-friendly web interface. I've begun working on this tool, and it's called Opstimal. Expect to hear more about it soon.

I was interviewed on Ryno the Bearded in an episode with the curious title "My Origin Story". I'm not sure whose that refers to, maybe both of us, because we both talked about our backstories. (Though, I think if I was going to lay out my "free software origin story", it would probably include some other things... but maybe it would get to be fairly rambly. I've thought about trying to write up what that is before, but I guess there were a lot of "moments" for my free software origins, not any one moment like "I was bitten by a libre radioactive spider".)

I really enjoyed doing this one, and maybe you'll enjoy listening to it. It's kind of rambly and conversational and we came in with very little as in terms of questions, but I think I tend to do fairly well in that format.

I did cut off the end of the interview by saying I had to go to the bathroom though. Not really the most dignified of exits on my part. Oh well.

I've been sick the last two days, and for some reason, I've been dreaming in lisp. It's not the first time I've dreamt in code, but all times have been in lisp. Maybe lisp is not as readable as many would like, but it seems dreamable.

Just got up from another nap, another lisp-based fever dream. This one about a cryptocurrencty on an actor model system, built for a MUD/AR type environment... no blockchain required (note: may be inspired by reading Rainbows End between naps):

• Various actors represent "reserves", like the Federal Reserve can print their own money, as much as they like, but there may be various community enforcements of this
• You might have different central banks / reserves on different servers
• The "value" or exchange rate determined by the market, like international currency.
• Currency is non-divisible (you have 100 rupees, but not .1 rupees)
• Bank has a private key, so does each actor.
• Basically more or less passing along capabilities (I think???) but maybe even the central bank signing whoever has "posession". The bank does not know who things are transferred to, but does verify that the transferring owner has the right capability currently belonging to that unit's ID, and does issue a new capability to the new owner.

Does this make sense? I'm too sick to know for now. But transactions flying everywhere, wrapped in parentheses, in my fevered mind.

If I have more crazy lisp fever dreams today I will record them here. No guarantee of sanity.

Great trip, spent 2 weeks doing the following:

• Attending W3C Social Working Group. Good things have happened; felt much more clear by the second day than by the first day. Lessons learned in that raising issues (normally I'm reasonably quiet, a certain amount of imposter syndrome while participating in the group) was helpful, some people said they were thinking about the same things. Anyway, much work to do ahead.
• Hanging out at Deb Nicholson's place with friends
• Gave a talk at Libreplanet which I'm happy with
• Had an awesome night of vegan salted caramel root beer floats (the best thing ever it turns out) and Hanabi with Aeva Palecek, Amy Guy, and Jessica Tallon; all friends of federation, but it was nice getting to decompress without worrying about that stuff for a bit. Way better than a trip to the bar!
• So many great people at Libreplanet. I'm kind of sad that I didn't get to spend a lot of time with everyone, but the up side is that I got to spend a lot of time with some particularly great people, and the main problem was there was just too many good people at once. A good problem.
• Mako and Karen both gave good keynotes, and I liked a lot of talks
• Lots of Veggie Galaxy, eaten
• Then took a trip to hang out with @Bassam Kurdali, Fateh, Libby Reinish, Tristen, at their respective places. It was nice to do this with @Tsyesika / Jessica Tallon joining me.
• Much of the week spent from the "Nerdodrome", Hampshire College's animation studio, where Bassam works. I spent most of the week working on a deployment system I'm specc'ing out. If it turns out to be something I use, I'll update with more info.
• Got to spend some time exploring Western Mass, which was great.

The trip was really worth it. I spent a lot of time with people I care about, and it felt very productive.

I'm not planning another conference for the remainder of 2015 though. Amongst other things, my back and wrists are killing me right now, and doing this morning's stretches was a painful wake-up. And I have a lot of difficult personal things in the year ahead. Oh, and let's not forget about how much work there is to do on MediaGoblin that I haven't gotten to.

But! This trip really helped set some directions for me for the year ahead, and I'm really grateful for that.

Looking forward to LibrePlanet 2016!

So off and on, I've been trying to understand monads. It turns out I have a use case: making web applications in the style I'd like but have them be asynchronous leads to trouble because you need a non-global-variable way of passing along context. I've tried thinking of some solutions, but a friend of mine convinced me that monads, if I could wrap my head around them, would solve the problem for me.

With that in mind, I did some reading... there are plenty of resources out there, some of them with nice pictures, and at one point I tried to understand them by just reading about them and not writing any code relevant to them. This lead to me guessing how they might work by trying to contextualize them to the problems I wanted to solve. So along that path, I had a misunderstanding, a mistaken vision of how monads might work, but while I was wrong, it turned out that this vision of ~monads is kind of fun and had some interesting properties, so I decided to code it up into a paradigm I'm jokingly calling "fauxnads".

Brace yourself, we're about to get into code, pseudo- and real, and it's going to be in Guile. (All code in this blogpost, GPLv3+ or LGPLv3+, as published by the FSF!)

So here's the rundown of fauxnads:

• They still pass along a context, and under the hood, they do pass it in still as the first argument to a function!
• The context gets passed up (and optionally, down... more on that in a few) in some kind of associative array... but we don't want to accidentally change the context that we passed to other functions already, so we'll use something immutable for that.
• The user doesn't really access the context directly. They specify what variables they want out of it, and the fauxnad macro extracts it for them.
• Fauxnads can add properties to the context that they'll call subroutines with so that subsequent fauxnad calls can have access to those.
• Calling child fauxnads happens via invoking a function (=>) exposed to the rest of the fauxnad via some lexical scope hacks.

So when sketching this out, I tried to boil down the idea to a quick demo:

;; fleshed out version of what a fauxnad should approx expand to
(define (context-test1 context arg1 arg2)
  (letrec ((new-context context)
           ;; Define function for manipulating the context
           (context-assoc
            (lambda (key value)
              (set! new-context
                    (vhash-cons key value new-context))))
           ;; a kind of apply function
           (=>
            (lambda (func . args)
              (apply func (cons new-context args))))) ;; should be gensym'ed

    ;; This part would be filled in by the macro.
    ;; The user would set which variables they want from the context
    ;; as well as possibly the default values
    (let ((a (or (vhash-assoc 'a context) "default-value for a"))
          (b (or (vhash-assoc 'b context) "default-value for b"))
          (c (or (vhash-assoc 'c context) "default-value for c")))
      (values
       (begin
         (context-assoc 'lol "cats")
         (=> context-test2 "sup cat")
         (context-assoc 'a "new a")
         (=> context-test2 "sup cat")
         (format #t "a is ~s, b is ~s, and c is ~s\n"
                 a b c)
         (string-append arg1 " " arg2))
       new-context))))

;; intentionally simpler, not a "real fauxnad", to demo
;; the fauxnad concept at its most minimal
(define (context-test2 context arg1)
  (begin
    (format #t "Got ~s from the context, and ~s from args, mwahaha!\n"
            (vhash-assoc 'a context)
            arg1))
  (values
   "yeahhh"
   context))

Then calling in the console:

scheme@(guile-user)> (context-test1 (alist->vhash '((a . 1) (b . 2))) "passed arg1" "passed arg2")
Got (a . 1) from the context, and "sup cat" from args, mwahaha!
Got (a . "new a") from the context, and "sup cat" from args, mwahaha!
a is (a . 1), b is (b . 2), and c is "default-value for c"
$79 = "passed arg1 passed arg2"
$80 = #<vhash 2205920 4 pairs>

Okay, whaaa? Let's look at the requirement again. We'll be passing in a function to the start of the function, and then having some other args. We'll then pass that along to subsequent functions. So more or less, we know that looks like this. (I know, not the most useful or pretty or functional code, but it's just a demo of the problem!)

(define (main-request-handler context request)
  ;; print out hello world in the right language
  (display (translate-this (context-get context 'lang) "Hello, world"))
  (newline)

  ;; now call another function
  (next-function new-context (smorgify arg1)))

(define (next-function context what-to-smorgify)
  (write-to-db
   ;; Lots of functions need access to the
   ;; current database connection, so we keep it in the context...
   (context-get context 'db-conn)
   (smorgify-this what-to-smorgify)))

But wouldn't it be cool if we didn't have to pass around the context? And what if we just said, "we want this and this from the context", then forgot about the rest of the context? We'd never need to call context-get again! It would also be cool to have a way to set things in the context for subsequent calls. Ooh, and if we coud avoid having to type "context" over and over again when passing it into functions, that would also be awesome.

So how about a syntax like this:

(define-fauxnad (our-special-function arg1)
  ((lang "en"))  ;; we want the language variable, but if not set,
                 ;; default to "en" or english
  ;; (body is below:)
  ;; print out hello world in the right language
  (display (translate-this lang "Hello, world"))
  (newline)
  ;; now call another function
  (=> next-function (smorgify arg1)))

We also know we want to use some sort of immutable hashmap. Guile provides vhashes which provide "typically constant-time" data access, and while there are some caveats (a new vhash returned by appending a key/value pair where that key already existed in the vhash will just keep the old pair around... but on our pseudo-stack that shouldn't happen very often, so vhashes should be fine), they work for our purposes.

Okay, cool. So what would that look like, expanded? Something along the lines of:

(define (our-special-function context arg1)
  (letrec ((new-context context)
           ;; Define function for manipulating the context
           (context-assoc
            (lambda (key value)
              (set! new-context
                    (vhash-cons key value new-context))))
           ;; a kind of apply function
           (=>
            (lambda (func . args)
              (apply func (cons new-context args))))) ;; should be gensym'ed

    ;; This part would be filled in by the macro.
    ;; The user would set which variables they want from the context
    ;; as well as possibly the default values
    (let ((lang (or (vhash-assoc 'lang context) "en")))
      (values
       (begin
         ;; print out hello world in the right language
         (display (translate-this (context-get context 'lang) "Hello, world"))
         (newline)

         ;; now call another function
         (next-function new-context (smorgify arg1)))
       new-context))))

As a bonus, we've taken advantage of Guile's multi-value return support, so any parent function which cares can get back the new context we defined for subsequent calls, in case we want to merge contexts or something. But functions not aware of this will simply ignore the second returned parameter. (I'm not sure this is a useful feature or not, but it's nice that Guile makes it easy to implement!)

That's clearly quite a complicated thing to implement manually though... so it's time to write some code to write code. That's right, it's macro time! Guile has some pretty cool hygienic macro support that uses "syntax tranformation"... a bit nicer than common lisp's defmacro, but also less low-level. Anwyay, if you're not familiar with that syntax, trust me that this does the right thing I guess:

(define-syntax define-fauxnad
  (lambda (x)
    (syntax-case x ()
      ((_ (func-name . args)
          ((context-key context-default) ...)
          body ...)
       (with-syntax ((=> (datum->syntax x '=>))
                     (context-assoc (datum->syntax x 'context-assoc)))
         #'(define (func-name context . args)
             (letrec ((new-context context)
                      ;; Define function for manipulating the context
                      (context-assoc
                       (lambda (key value)
                         (set! new-context
                               (vhash-cons key value new-context))))
                      ;; a kind of apply function
                      (=>
                       (lambda (func . func-args)
                         (apply func (cons new-context func-args))))) ;; should be gensym'ed

               ;; This part would be filled in by the macro.
               ;; The user would set which variables they want from the context
               ;; as well as possibly the default values
               (let ((context-key (or (vhash-assoc (quote context-key) context)
                                      context-default))
                     ...)
                 (values
                  (begin
                    body ...)
                  new-context)))))))))

Nice, now writing our fauxnads is dead-simple:

(define-fauxnad (our-special-function arg1)
  ((lang "en"))  ;; we want the language variable, but if not set,
  ;; default to "en" or english
  ;; (body is below:)
  ;; print out hello world in the right language
  (display (translate-this lang "Hello, world"))
  (newline)
  ;; now call another function
  (=> next-function (smorgify arg1)))

(define-fauxnad (next-function context what-to-smorgify)
  ((db-conn #nil))
  (write-to-db
   ;; Lots of functions need access to the
   ;; current database connection, so we keep it in the context...
   (context-get context 'db-conn)
   (smorgify-this what-to-smorgify)))

Okay, again, my demos don't make this look very appealing I suppose. We can now transform the original demos I sketched up into fauxnads though:

(define-fauxnad (context-test1 arg1 arg2)
  ((a "default value for a")
   (b "default value for b")
   (c "default value for c"))
  (context-assoc 'lol "cats")
  (=> context-test2 "sup cat")
  (context-assoc 'a "new a")
  (=> context-test2 "sup cat")
  (format #t "a is ~s, b is ~s, and c is ~s\n"
          a b c)
  (string-append arg1 " " arg2))

(define-fauxnad (context-test2 arg1)
  ((a #nil))
  (format #t "Got ~s from the context, and ~s from args, mwahaha!\n"
          a arg1)
  "yeahhh")

And calling it:

scheme@(guile-user)> (context-test1 (alist->vhash '((a . 1) (b . 2))) "passed arg1" "passed arg2")
Got (a . 1) from the context, and "sup cat" from args, mwahaha!
Got (a . "new a") from the context, and "sup cat" from args, mwahaha!
a is (a . 1), b is (b . 2), and c is "default value for c"
$81 = "passed arg1 passed arg2"
$82 = #<vhash 2090ae0 4 pairs>

Okay, so what's the point? I doubt this blogpost really would sell anyone on fauxnads, and maybe why would you use fauxnads when you can use real monads? But here's some interesting properties:

• fauxnads are still super simple functions that you can call manually: just pass in the context (a vlist) as the first parameter.
• the "binding" function for calling sub-fauxnads is sugar, but hidden (and otherwise inaccessible, because the function hygeine keeps you from accessing "new-context") from the user.
• I still like that you can get back the new context via multiple value return, but totally ignore it if you don't care about it.
• I understand how they work.

And on that last note, I still don't understand monads, but I feel like I'm getting closer to it. It was fun to document, and put to code, a misunderstanding though!

Update: Well, you post a thing, and sometimes that's enough for people to come and help you realize how wrong you are. Which is good! There are a number of ways forward (some obvious in retrospect). For one, pyrsistent does exist and looks nice and... well it's even actively developed. But even aside from that, there are several clean solutions: wrapper objects which "lock" the child object with getters but no setters, or even just using alist style tuples of tuples for a fake hashmap. Options are indeed abound.

And the exception thing? Well, that wasn't listed as a permanent problem below, but the solution is even easier! It would be simple to have a MessageError("some_identifier") which has a minimalist identifier which can be passed across the wire, and the directive of this error can be a special case.

Anyway, you can read the original post below. But it's good to be wrong. XUDD no longer has reason to be dead. Long live XUDD!

tl;dr: Kind of along post, but basically the lack of good functional data structures in Python has kinda killed the project.

One of my favorite projects ever has been to work on XUDD, an asynchronous actor model system for Python. Originally born out of a quest to build a MUD (hence the name), but eventually became the focus of being an actor model itself, it was a really interesting exploration for me.

There's a lot of things I like about the actor model... for one thing, functional programming is all the rage, right? But not all systems are easy to express in a purely functional style, and done right an actor model can be fairly object oriented'y, but done right, you can have your mutable cake and eat it too, safely! Your actor can mutate some of its own variables, but when it communicates across the wire, since it's a "shared nothing" environment you can get even better scale-to-the-moon type functionality than in many functional language systems: it's trivial to write code where actors communicate across multiple processes or machines in just the same way as if they were all on the same machine in the same process and thread.

I put a lot of thought into XUDD, and I've looked into some of the alternatives like Pykka, and I still think XUDD has some ideas that kicks butt on other systems. I still think the use of coroutines feels very clean and easy to read, the "hive" model is pretty nice, and the way it's built on top of the awesome asyncio system for Python are all things I'm happy with.

So, every once in a while I get an email from someone who reads the XUDD documentation and also gets excited and asks me what's going on.

The sad reality is: I'm stuck. I'm stuck on two fronts, and one I can figure my way out of, but the other one doesn't seem easy to deal with in Python as-is.

The first issue is of error propagation. This is solvable, but when an exception is raised, it would be nice to propagate this back to the original actor. There are some side issues I'm not sure about: in an inter-hive-communication (read: multiple machines or processes) type scenario, should we use standard exceptions and try to import and reproduce the same exception that was raised elsewhere? That seems like it could be... gnarly to do. Raising the error inside the original routine is also a bit tricky, but not too hard; python's coroutines can support it and I just need to think about it. So exceptions are annoying but solvable.

But the other issue... I'm not sure what to do about it. Basically, it's an issue of a lack of immutable types, or we might even say "purely functional datastructures" that are robust enough to continue with. Why does that matter? Messages sent between actors shouldn't have any mutable data. It's fine and well and even a nice feature for actors to be able to have mutable data within themselves, and actually even provide a nice way to pull of things that are just damned hard in purely functional systems, but between actors, mutable data is a no-no.

It's easy to see why: say we have a function that has a list in it, say of a number of children in my classroom, and I send this list over to an actor that controls some sort of database, right? I'm doing things in a nice, fancy coroutine'ish type way, which means my function can just suspend mid-execution while it waits for that database actor to generate some sort of reply and send it back to me. What happens if that other function pops one of the items off the list, or appends to it, or in some way actually mutates the list? Now, when my function continues, it'll be operating on a differently formed list than the one it thinks it has. I might have a reference to the third item in the list, but it turns out that there isn't a third item in the list anymore, because the other function popped it off. This can introduce all sorts of subtle bugs, and it's bumming me out that I don't have a good solution to them.

Now, there's a way around this: you can serialize each and every message completely before sending it to another actor. And of course, if actors are on different threads, processes, or even entirely different machines, of course we'd do this. But XUDD has the concept of actors being on the same hive, and there are a number of reasons for this, but one of them is that for local message passing, packing and unpacking data in some sort of serialized format for every call slows things down by a lot. When I originally began designing XUDD, the plan was for games that might need to shard out to a number of different servers but have players that can traverse different parts of the system and communicate with other shards (without knowing or the code mostly knowing that it's communicating with other actors that are technically remote). I want to be able to pass many messages at once to actors that are on the same hive, while still having a totally safe time of doing so. But there's no way to do so without a nice set of immutable / "purely functional" types, and Python just doesn't have this right now. None of the third party libraries I've found seem well maintained (am I missing something?), and the standard library is fairly deficient here. Why? I'm not really sure. I guess Python's history is just synchronously imperative enough that it just hasn't mattered.

I'd like to continue research into the actor model... I have some projects I'd like to work on where the actor model seems perfectly tuned to those tasks. What to do?

Well, I'm not really sure... I guess I could just serialize everything all the time, but it's kind of a bummer to me that so many cycles would be wasted for local computation. Maybe it's a dumb reason to feel exhausted with things, but that's the state of it. I'm not enough of a datastructure wizard to implement these things myself, but they exist. I've thought about giving up on XUDD being a Python project and to move over to something else... Guile has a cooperative REPL which would be great for debugging, and I really like the community there, so maybe that would be a nice place to go. Not really sure there's anything else I'm interested enough in at the moment. I think I'd miss Python. Or maybe I'm over-thinking everything in the first place? (Wouldn't be the first time.)

Maybe there's another way out. If you have any ideas, contact me.

While working on MediaGoblin's upcoming release, and (dare I say) feeling a bit bored and anxious and frustrated about the whole process, I decided that maybe it was time to look into some sort of entertainment. But for the most part, I feel like I've played all the interesting free software games out there. I've mostly beaten even the games of my old proprietary console using past. What to do?

I wondered what the state of good ol fashioned MUD systems are. I've had dreams for years of building such a system from scratch (hence my interest in the actor model). Answer: they feel like they're in the same place they were around 2000. But maybe this isn't bad? I tried playing some dungeon crawling'y MUDs but found myself a bit bored. But having remembered that Rob Myers has talked plenty about LambdaMOO, so that seems like good enough reason to connect and look around. (Note, connecting is not hard! I am using Mudlet at the moment.)

So, in-between actual coding and fretting about this and that, I've been getting in some downtime, poking around the LambdaMOO environment. And at first I was fairly unimpressed... this thing is what, 20 rooms big? What can I do here? But I've been finding some interesting things:

• That LambdaMOO is both the software and one particular server is fairly confusing, but no matter: both are of interesting construction. They're both also very old, 25 years old at this point (egads), but particularly as to the world, that leads to much depth.
• The main rooms of the house (see [help geography]{.title-ref}) are not as interesting as the places they lead to. You can enter a model railroad and actually walk around that imaginary city and end up in some interesting places, reading graffiti off of nightclub walls, etc. Find an old teletype and type on it, or play one of the functioning (!) board games sitting around. A dusty hallway leads to a full hack and slash style game engine, coded live inside the system itself.
• Most of the world is not really a dungeon crawler, though there is one hidden in it. Most of walking around LambdaMOO is exploring the many curiosities that users have provided there.
• Speaking of coding live inside the system, what's really interesting about LambdaMOO is that you're really manipulating a big database, and that many features are live coded from within the game itself. Everyone can experiment with adding rooms, though not everyone can program, though you may be able to get help from a certain buffer in the smoking room, if you can find it. (I just got access... I'll have to try things.)
• LambdaMOO is best known for a famous essay, but I think that essay is maybe not as interesting as it was in the 90s when online experiences were becoming a new thing? Still, the effect of the essay and the event it chronicles is still clear, including the way the world is introduced to you.
• Interesting to think of LambdaMOO in case of copyright and licensing, of which... well there seems to be no clear answer what is going on, but sharing seems not impeded by it.
• Being from the early 90s, it's amazing how much is here to explore, and how everything still lives. Since most of the world is pretty "low burn", that's nice for me, because I can just type in a command to look around somewhere every once in a rare while and leave LambdaMOO open in the background.
• I'm enjoying exploring the world, but I sense that LambdaMOO had a heyday, and while there are users connected, the sheer size of it vs the volume of connected users gives me the sense that today is not it. There's a certain amount of feeling of exploring a secret city, which is wonderful and sad. (It would be nice to have some friends who were interested in exploring with me, maybe I should cajole some into joining me.) It's an interesting piece of internet history, still here to explore, but since it's in a "living" database, what will happen when it shuts down? There won't be any Internet Archive to preserve the LambdaMOO memories, I fear. But maybe nothing is forever, anyway.

I have more, maybe crazier, thoughts on MUDs and maybe a way they could be more useful in this modern world. May I'll explore soon, but I think that the popularity of MUDs has diminished with World of Warcraft and Skyrim and the like, but maybe they have a place still if we can do some new things with them? And those are fairly hack-and-slash type systems, maybe MOO/MUD/MUSH systems have more to offer than popular MMO systems provide? Roguelikes seem more popular than ever, if anything... things can come back... maybe new interfaces are needed?...

In the meanwhile, if you drop by LambdaMOO, send me a message! I'm "paroneayea" there.