contributor drawing configuration autonomous interview email php tabletop sqlite deployment robots macros advocacy politics high priority openmoko foss fsf python life update creative-commons money campaign conference libreplanet tech privilege hatecrime personal blog blender freespeech wedding frostcast shared hosting javascript xmpp diet anniversary firefox cabbage zine flourish gimp dustycloud xudd code backups xkcd release art free-culture lisp meme by-sa identica pcf biking firebug gnome rsi emacs base64 music nouveau sfxr patents vegan recipe chipy pelican food freesw phone robogoblins speech recognition libre-graphics-meeting kinda-mediagoblin cooking gpl orgmode mediagoblin podcast economics security cookies art talk researchcation n900 beyond-javascript hy guile family uuid cc0 pycon miro licensing boingboing

Unofficial candidate list for high priority free software projects

By Christopher Allan Webber on Sat 20 December 2014

I'm happy to see the FSF has revitalized the high priority free software projects list.

The FSF is calling for input, so consider the following my "high priority wishlist": these are all things I wish I had nicer representations in free software, and the lack thereof is hindering user freedom in some urgent way.

Server deployability tools

Deployment needs to become easier if we expect people to have any amount of digital autonomy. Processes need to become generalized, abstracted. I have written some ideas on this. I am happy to talk to anyone interested in improving this space... :)

Federation tools

Obviously somewhat self-serving, though my very reason for working on federation tools is due to my belief that a lack of federation tooling leads to many problems. Those arguments already well understood by anyone reading this site, so I won't go into details.

Implementation of the Social WG standards would be a good direction to take. Making this implementation reusable would be helpful.

"Unity competitor" level game engine

I've also written before on why I believe that games matter to free software and free culture.

But I'll particularly emphasize here: 1) proprietary games are bringing DRM to otherwise free software operating systems via steam, 2) games are currently the world's most popular medium, so if expression here is particularly hard, this is not good and 3) games are a great way to recruit new hackers, as this is the dream of many.

Assuming you buy those arguments, we are missing out on many opportunities of expression by the sheer amount of work required to get the basics of most games running. In the proprietary world, there is a game engine called Unity that seems popular not just for developers but also for artists. What is free software's answer?

Though it will surely come up, I'm not confident in the Blender Game Engine. Godot looks promising.

(As a side note, both of the following systems are fairly low-level at the moment and thus do not fulfill the above "full featured and easy to use", but some interesting things are happening in the lands of both Sly and m.grl (both by friends of mine)... I hope more happens with these.)

Password management tooling

Managing passwords is hard but important, and the current state of affairs in free software is a sorry one. Many users are turning to "LastPass", which astounds me in a post-Snowden, post-Lavabit world. How to provide a way to store passwords safely and easy to use way?

KeePassX and others which require manually copying around passwords I think are simply not usable enough given the massive volume of times users need to enter passwords.

I currently use assword, as "assword gui", when bound to a key in your window manager, gives a simple-ish prompt, with passwords dumped into whatever textbox previously had focus in X11. It also pleasantly stores passwords in a simple format: gpg encrypted json. Not bad, but there's no nice "management" interface, and there's no "mobile" (read: Android/Replicant) client.

Easier encryption

It's great that we have GnuPG, but usage is hard. How to make easer? Maybe something like monkeysign could help web of trust things, but not only web of trust things are hard. SSL is also a great way to lose hair.

Relatedly, GPG could use your help, funding-wise! Consider making a donation.

Free phones and mobile devices

Is this software? Sort of, it also ties in with hardware.

Replicant is the best thing we have for phones. (I would love to see a resurrection of Maemo 5, but I realize I am in the minority here.)

Wearable computing

Where is free software's competitor to Google Glass? Even if it isn't cool right now, some competitor will make pervasive computing popular. Do we really want to wait for the iEye before we start worrying that we don't have a solution? This, like free phones, requires both hardware and software.

Circuit printing pipeline

Libre hardware which can print circuits could possibly get us well on our way towards building hardware that's safe, reproducible, verifiable. It won't help with chips, still will require lots of soldering, but maybe it could help.

Easy, peer to peer filesharing between friends

The return of peer to peer? Yes indeed. I see this one is already on the list though. :)

Activist/political orgs: stop using stupid email subjects

By Christopher Allan Webber on Sat 06 December 2014

I love nonprofits and activist orgs, and I think political campaigns should send emails to their base asking for support.

That said, for the last few years, I've gotten angry at nearly every email subject I've gotten from activist organizations. You've probably gotten the same, but I've collected some samples:

  • re: The site crashed!
  • Um...
  • re: Un-freaking-believable
  • Argh!
  • Please please please
  • Forward this!
  • Forward this →
  • This is dangerous:
  • Holy moly!!!
  • re: Corrupt?
  • re: Comcast (like a thousand of these)
  • Pancakes and syrup (yes, a real subject sent for non-breakfast-invitation reasons)

And the list goes on, and on. My "Political Campaigns" folder is a joke.

Apparently this started in Obama's re-election campaign and was found to be successful in terms of getting both eyeballs and dollars.

Is it still successful? If it was a novel trick, it isn't anymore. What's worse, these subject lines are awfully similar to two groups of emails I hate to get: the classic and loathed "ha ha forward this to friends" email from relatives, and spam. Which means many of these I automatically mark as read or accidentally misfile as spam.

But I'm tired of getting these emails. If you're an activist org, and you send an email to me like this, I'm unsubscribing. And as immature as it might be to say, I'm unlikely to donate to your campaign again, assuming I ever did.

Political/activist orgs: don't treat me like an idiot. Prove my brain energy is worth your time.

Donate to Software Freedom Conservancy

By Christopher Allan Webber on Wed 03 December 2014

Hello everyone! It's that time of the year again, and many people are considering: who they are going to donate to this year?

If you are a supporter of computing freedom, there are a lot of great places to put your money. The FSF and EFF are great places, and probably the most commonly recommended (also worth your money!), though this year I'm going to suggest you also consider the Software Freedom Conservancy. They just launched a membership program today, and I recommend you join. I just did!

Become a Conservancy Supporter!

Whether you realize it or not, you probably depend day to day on some software that Conservancy helps support... which it turns out is a lot: Git, Inkscape, Mercurial, Wine, BusyBox... these are just a few of the many projects that are members of Conservancy. And Conservancy does a lot of stuff for their member projects! (Not to mention a number of things that benefit free software that don't even hit that list, including Conservancy employees spreading information on best practices and participating in efforts to protect the legal freedoms of all free software... I recommend subscribing to Bradley Kuhn and Karen Sandler's podcast Free as in Freedom... I've learned a lot from there, and am learning more all the time!) What I'm trying to say here is: donating to Conservancy is a real bang for your buck. :)

So this holiday season, join me in helping make Software Freedom Conservancy's membership program a success! Your user freedoms will thank you for it!

How Hy backported "yield from" to Python 2

By Christopher Allan Webber on Thu 20 November 2014

Hello everyone! Time for a bit of a diversion towards one of my favorite projects, Hy! (I'm an occasional committer, but the main mastermind behind the project is my good friend Paul Tagliamonte.) For those of you who don't know, Hy is a Lisp that transforms into the Python AST. Even more fun: you can import .hy files in .py files and .py files in .hy files! Crazy!

Now, when many people hear that, they say, "Huh what, why on earth would you do such a thing?" The usual response is something like, "Because it's fun!" But today, dear readers, I am going to show you a real... dare I say practical reason for using Hy. Because a cool feature just landed in Hy: a backport of "yield from" to Python 2.

Let's back up a bit. First, you might not know what "yield from" is or why it's cool. Well, Python has this thing called coroutines which allow you to do cool things, including suspending and resuming functions, which it turns out is really great for writing asynchronous code (sure, just wake me back up when we get the next network stanza, eh buddy?). Once you provide the ability to nest together coroutines by "delegating to subgenerators" (what "yield from" does that "yield" does not), this stuff starts to get really powerful. This feature is so useful that it's the basis of maybe the world's coolest asynchronous programming environment, asyncio. Only one problem: "yield from" didn't exist until Python 3.3... which means you can't use it with Python 2. Bummer!

Or can you? Time for our second bit of context. Ever hear of a Lisp programmer talk about something called a "macro"? No? Okay, think harder. Maybe it was in one of those conversations where you were talking about your favorite new your-pet-language feature, and the Lisp hacker was like, "Oh that's cute... yeah Lisp had that decades ago." And then you got really mad and brought up a bunch more features, and the Lisp hacker kept saying that Lisp had them before you were born, and "Oh yeah, and whatever features Lisp doesn't have, you can add really fast because Lisp has macros. You can basically program any feature with macros." Maybe you asked them, what the heck is a macro, and they said something inane like "it's a feature where you can program other features, or write code that writes code", but you barely remember, because at that point you just wanted to punch them in their smug little face. (And besides, you wondered, if you have higher order functions, isn't that enough?)

Well friends, today it is my face that you will want to do the punching to, because I'm about to show you how cool macros are and why having them makes Hy so awesome. But, after the face punching thing, you'll also thank me. (Also, please don't punch me in the face, this blogpost is not consent for face-punching.)

Enough with the talk. Time for examples! Let's look at some code. Say you have this Python 3.3+ code:

from awesomelib import IrcBot, bake_cookie, async

def irc_to_cookies(**connection_stuff):
    our_bot = IrcBot(**connection_stuff)
    yield from our_bot.open_connection()

    while True:
        message = yield from our_bot.get_next_message()

        if message.command == "bake_cookie":
            yield from async(bake_cookie())

Groovy. In our example above, we built a cookie baker that can be plugged into our awesomelib asynchronous network library and cookie baking pipeline system. (And of course, we wrote it in Hy, because we love Hy, and you can still run Hy code in vanilla Python.) We're feeling pretty good about this. We kind of wish we could run it in Python 2.X still, but Python 3.3+ is the future anyway, and no use worrying about the past really... right?

Our Hy example looks pretty similar:

(defn irc-to-cookies [&kwargs connection-stuff]
  (setv our-bot (apply IrcBot [] connection-stuff))
  (yield-from (.open-connection our-bot))

  (while True
    (setv message (yield-from (.get-next-message our-bot)))

    ;; If an irc user gives the "bake_cookie" command,
    ;; put a cookie in our ez_bake oven
    (if (= message.command "bake_cookie")
      (yield-from (async (bake-cookie))))))

But there's something magical... this code makes use of yield-from, which in Python 3.3+ Hy just uses the actual real built in "yield from" (or more accurately, ast.YieldFrom). But what about Python 2? Mere higher ordered function magic can't save us here. We need a way to implement a new feature.

Except oh right, this is a lisp, and we have macros! So why not write a macro for yield-from?

And it turns out that's exactly what paultag did:

  (defmacro/g! yield-from [expr]
    `(do (import types)
         (setv ~g!iter (iter ~expr))
         (setv ~g!return nil)
         (setv ~g!message nil)
         (while true
           (try (if (isinstance ~g!iter types.GeneratorType)
                  (setv ~g!message (yield (.send ~g!iter ~g!message)))
                  (setv ~g!message (yield (next ~g!iter))))
           (catch [~g!e StopIteration]
             (do (setv ~g!return (if (hasattr ~g!e "value")
                                     (. ~g!e value)

This simple macro above is an implementation of yield-from which works in Python 2. The macro is more or less a function that writes new code to be expanded in place... allowing us to use basic building blocks of the language to build more complex features. Since in Lisp, code is a very simple, manipulatable data structure (lists!), we can literally write out code that writes code without too much trouble. (There's some magic going on with the ` character above, called backquoting... but it's best to read a tutorial on macros if it's not clear to you how the backquote is building the list of code there.) Hey look... we just brought a feature back to the future... as long as we're writing code in Hy, we can do subgenerator delegation with yield-from. Cool! That sure makes coroutines a lot more useful to those of us living in the past.

So wait, does this mean you can now use asyncio with Python 2? Well, not quite... asyncio is written in normal Python syntax, which means that it's using "yield from", not our more versatile "yield-from", and the library itself isn't written to support Python 2.7. So, no. (But, if asyncio was written in Hy, we could, even though Python 2.7 doesn't have "yield from"!)

The real goal of this article isn't to convince you to start backporting features to Python 2.X via Hy, though. Really, Python 3 is the future, write Python 3 code! But the point here is to get you thinking about how having macros allows you to implement new features now! Why wait for the features of Python 4.X? In Hy you can have them now! (Or even start prototyping them today!)

And that's worth getting excited about. And once you realize that, it's a bit easier to understand where lispers are coming from when they nerd out about how cool macros are. (Even if you still want to punch us in the face.)

(PS: think this is pretty cool? Hy is a really welcoming community, and there's a lot of fun stuff to do! Learn about language implementation and learn about lisp in a paradoxically fun and pythonic environment! We'd love to have you join in hacking with us!)

Javascript beyond Javascript (with Guile?)

By Christopher Allan Webber on Fri 15 August 2014

I'm learning more in my spare time / off normal hours about compilers and graph theory and reading various books on lisp. I have an agenda here, no idea if it'll happen; at the very worst I put a lot of tools in my toolkit that I should have had. But there is another reason... Well, this one's easiest to lay out point for point, so here goes:

  • Python is still my favorite language to write in day to day, but I guess that I keep feeling that any language that doesn't have a way to transcompile nicely to Javascript (or isn't Javascript itself) is unideal for writing really great web applications, for a simple reason: modern web applications are highly interactive, and I feel that a really stellar web application needs to share some code between the backend and the frontend. This becomes more obvious when writing dynamic templates or forms.

    While Python probably has hands-down the nicest asynchronous library in current development with asyncio, the above problem makes it feel like you can only do so much in the web world with it. (The Javascript Problem is a good page on the Haskell wiki which puts this all pretty nicely.)

    There's a lot lost by not being able to interweave applications between the frontend and the backend. Sadly, the best option in Python right now is to just give up and write entirely separate codebases in javascript that interact with your main codebase. But clients on the web are becoming thicker and thicker these days (even more so with technology like websockets and webrtc becoming prevalent), so this feels both weak and redundant.

  • You could say "just write javascript!" then, but let's face it, I don't really like Javascript. The language is getting better by leaps and bounds as time goes on, but a lot of the enthusiasm for the language still feels like Stockholm syndrome to me. But we're still left with Javascript. So what, then?

  • Transcompiling actually is becoming an increasingly appealing target. With asm.js this might not only be possible but actually very optimal. One only need look at the Unreal demo to see just how feasible transcompilation has become.

  • So okay, let's say we're doing some kind of transcompiling. We have two options, "transcompile wholesale", or "transcompile a subset that's acceptable for javascript and local evaluation". Obviously the former is desirable if possible, though the latter is a lot easier.

  • Indeed, transcompiling a subset has already been done, and well, by Clojurescript. I'm not really excited by anything too tied to the JVM, but still, pretty cool. Maybe good enough to get me into a hybrid Clojure/Clojurescript solution. But still... there's no asm.js target (maybe in the future?), and it feels like you're tossing out so much with Clojurescript, it really isn't the same language too much, just a very similarly overlapping language. I haven't tried it though... this is just from reading docs and watching talks :)

  • We could say maybe we could transcompile something like Python, but assuming we really want to have something like a template engine, we probably want to be writing in a full fledged version of the language, not a restricted subset. Python feels way too huge to expect for people to load in their browsers. So, what's more lightweight?

  • Let's continue in the lisp direction. Is there a way we can get a complete implementation in the browser, but of a language that's a bit more lightweight? How about Scheme? Scheme is notoriously simple to implement, maybe even too simple... and Javascript shares a lot of overlap with Scheme in design...

  • But which Scheme? How about Guile? Yes, Guile, GNU's extension language based on scheme.

Okay, wait, digression before we continue... I want to talk a bit more about Guile, because up until a few months ago my impression of it was still the somewhat-dismissive impression I had a few years ago. But I've come back to looking at Guile again. Largely this is thanks to Dave Thompson's Sly project. There is some seriously cool stuff going on with Sly, and that made me want to look into what's happening with Guile.

What I found is that Guile isn't the same thing it was a few years ago. For evidence of all the cool things that are happening under the hood, I point you to wingolog (warning: danger of being a real time sink; linking to Andy Wingo's blog has been described as "wingorolling" by a friend of mine). Guile isn't just an interpreter anymore, it has a sophisticated "compiler tower" since Guile 2.0. With the levels of abstraction that exist in Guile, could compiling to javascript/asm.js be a target? (What makes this feel extra appealing/feasible: Andy Wingo also works at his day job on improving javascript virtual machines, so maybe...?)

Well, better to ask Guile hackers themselves if it's possible... and I've pestered a number of them. The responses seem to be (and I may be representing incorrectly):

  • Getting the core language of Guile (which is of course scheme) ported over is not the hard part.
  • Making a subset of the language that works in Javascript is not too hard, but isn't too interesting. It would be much better to have the whole language work.
  • Overall, the most tedious part of porting guile to target any non-C target will be all of the library procedures currently implemented in C... however, gradually the Guile team is reducing the amount of C code in Guile and replacing it with Scheme code.
  • The problem is the runtime; you probably would want to re-use some higher level things like the allocator and garbage collector. Emscripten might do garbage collection correctly (guile relies on Boehm GC to do garbage collection); emscripten seems to provide a similar garbage collector? Tail calls are also a problem until at least es6 is common, and even then until they are optimized.
  • Maybe someone could try transcompiling Guile with emscripten as a test, but that probably wouldn't provide the right integration, and regardless, this would probably be a bit uncomfortable because emscripten relies on llvm, not gcc.
  • Targeting asm.js brings things more low-level, but the asm.js typedarray heap may not be appropriate; the guile-on-js heap and the javascript heap should really be the same.

Okay, so, that's a good number of problems that need to be overcome. Certainly it makes it feel like things are a ways off, but then again, it seems like there's strong interest. And it feels feasible... as an outsider to development. :)

But even assuming all the above are resolved, there are some other problems that I think scheme will always face adoption-wise:

  • I like lisp, and I like parentheses. I have emacs set up with rainbow-delimiters, smartparens, and highlight-parentheses (set up just to highlight the background, not the foreground, so it doesn't clobber rainbow-delimiters). Lisp's greatest enemy has never been its functional capability, but a general parenthesitis in the general population. This the big one, so let's come back to it in a moment.
  • Lots of scheme (lisp generally, but especially scheme) feels like it just is too in love with ideas that make it hard to approach. "car" and "cdr" are not good names but form the very basis of the language for historical reasons where even prominent lispers have had to backtrack to remember why. Linked lists as the core of lisp is great, but overuse of them is encouraged when other data types are much more appropriate: see teaching associative lists before hashmaps. While iteration tools are provided, there's simply too much emphasis on recursion. Recursion is powerful and awesome and necessary to solving many important computer science problems, but rarely needed by say, a web developer, and significantly harder to read and wrap one's mind around than iteration.
  • That said, these could be overcome with better introductory material. There seems to be interest in the guile world for this, thankfully!
  • Module names often look like robot serial numbers ("srfi-13") than anything human-readable and feel like they really need aliases.
  • The Guile community is very small, and does not seem to be very diverse. Outreach is greatly needed.

Returning to "parenthesitis", while I love lisp and all its parentheses, admittedly it's not the most readable language ever. I can wake up groggily at 5AM, be tossed a block of Python code, and even through blurry eyes, I can get a feel for what's happening in the code just by its structure. I can't say the same of any lisp.

But there's a way out of that situation! Guile's compiler stack is nicely set up so that another syntax can be laid on top of that. Guile's VM has semi-complete language implementations of ecmascript and some other languages on top of it. Recently Arne Babenhauserheide has written a whitespace to lisp preprocessor named wisp; I don't feel it really solves the problem by making things so much more readable, but it's a nice demonstration.

The most promising resolution I think would be to implement a syntax akin to Julia on top of Guile. If you haven't looked at Julia, it's a cool project: it has a python-like syntax with lisp-like power, and many other interesting features, including macros(!) in a language that is certified safe for those with parenthesitis. In fact, the language is largely built on top of scheme! So it might be nice to have a "sugarcoat" language that sits on top of Guile.

If all the above were achieved and a well developed web framework were written on top of Guile, it could be well positioned for writing web applications that are a joy to write, both on the backend and frontend.

One more thing: the complaint about "don't use emscripten" because it's built on top of llvm is an indication of how sorely needed a working javascript/asm.js compiler target is in GCC. ARM, X86, SPARC... these are all important compiler targets to have, but to advance user freedom where the user is today, the browser is the most important target of all.