Posts with tag "mediagoblin"

In which I receive the O'Reilly Open Source Award

By Chris Lemmer-Webber on Tue 01 September 2015

Well, I'm late in putting this one out there, but it's still worth putting on the record! About a month ago, I was fortunate enough to receive the O'Reilly Open Source Award. In fact, here's a picture of me receiving it!

receiving the award
Photo taken by Brandin Grams, CC BY 4.0, originally microblogged by Karen Sandler

(... there's a video too!)

So, getting the award was exciting and unexpected. Exciting, because just look at the list of people who have received this award previously!

Even to just share the stage with friends Marina Zhurakhinskaya of Outreachy, Stefano Zacchiroli of Debian, and Sarah Mei was a huge honor (okay, Sarah Mei is not someone I know, but I know very well of Railsbridge and the many ways it has paved forward diversity initiatives in free software). Plus, even though I don't know much about Hadoop, I know others who do, and the looming head of Doug Cutting behind us in videorecording excited them! And looking at former recipients, nearly everyone on this list is a person who has made humongous strides in shaping the world of free and open source software, and let's face it, almost everyone on this list has done more in that capacity than I have.

Which leads me to the surprise part... I was really not expecting this award, so when I saw the email informing me I was being awarded it I began to flag it as spam, assuming it was something that snuck past spamassassin. But wait, I actually know this award! I then preceded to try to get Morgan's attention (she was on the phone) by exuberantly waving my arms, which lead our dog to jump up and also wave her forelegs in a very similar fashion.

So that is all to say I'm very honored to have received the thing!

So, then I knew I received the award, but didn't know for what text I was nominated until the ceremony, but I was really happy with it when I heard it said: for my free software advocacy (and with those words) and my work on GNU MediaGoblin (GNU was pronounced with each letter as if a university chant... G! N! U!). I suspected as much and I'm glad I received the award under that description.

And that is also to say I am really glad for the recognition!

The recognition does help, in some way, as a counterbalance to other feelings. In another sense, I also look at the list of other people who have received the award and I feel a bit embarrassed because I think maybe I don't deserve it. I had confided in this to some friends (I guess it's no longer confiding now that I'm writing in a blogpost), and the response has mostly been something along the lines of "of course you do, just feel good about the thing", and of course that's also the kind of thing you want to hear from your friends, and it would be pretty embarrassing if they didn't say those kinds of things in response, and you begin to wonder that given that if you're playing some sort of scripted roles, and typing this I even wonder if this "I don't deserve it" text is doing that too, and maybe it is, maybe everything is... but I still think sometimes I am fooling everyone and when I am given recognition that I am some kind of tricky person, tricky enough to trick people into giving me an award for a pile of incomplete things.

I think there are a variety of reasons for this, one of the more obvious being that MediaGoblin is moving along but it isn't there yet, something I think nobody is more deeply aware of than myself. And people are still locked down by proprietary network services, we still don't have a generally agreed upon federation standard, and it is still really hard to run your own server. These aren't my battles alone and a good number of us are working on them, (and I even have interesting things to report in these areas that I have not yet blogged about) but when I think about things not being done I feel personally responsible for it. Part of the challenge also is that I do not generally look at my life as in terms of things I have accomplished, but in terms of the things I haven't, and obviously that is not a great strategy for feeling good about the things you've done.

But let me turn this blogpost back around again, before I look like some kind of jerk who gets an award and then mopes about it!

In this same sense the recognition has helped a lot. Not everything that I want to see done is done yet, but the O'Reilly Open Source Award selection process happens through previous winners, so clearly people who know better than I do have recognized that the things we've done and the direction we're heading are the right ones.

You may notice that I'm switching between "I" and "we", so a bit more on that... it's worth noting that any accomplishments I have are connected to some significant free software community, so they're hardly just "my" accomplishments... that statement of indicating that the roads we're heading down are the right ones applies to all the people in the communities of which I've become some apparent type of minor figurehead. (Be wary of pedestals, with coordination like mine, a great way to dash oneself against the floor...)

So! That's a lot of words there, but the crux it is that I'm still excited to have gotten the award, and as much of a worrier as I am, the recognition is especially nice as a kind of reassurance. In the meanwhile, I have a fancy new bookshelf and a nice little display for this hunk of glass:

On the bookshelf

On the bookshelf, zoomed in a bit

O'Reilly award, on display

Pretty cool hunk of glass, right? I think so!

PS: John Sullivan snapped this image of Stefano Zacchiroli and I right after the ceremony, on a trip to Powell's books.

Award Winners at Pow-ells
"Award Winners" photo taken by John Sullivan CC BY-SA 4.0

Interviewed on Ryno the Bearded

By Chris Lemmer-Webber on Tue 07 April 2015

I was interviewed on Ryno the Bearded in an episode with the curious title "My Origin Story". I'm not sure whose that refers to, maybe both of us, because we both talked about our backstories. (Though, I think if I was going to lay out my "free software origin story", it would probably include some other things... but maybe it would get to be fairly rambly. I've thought about trying to write up what that is before, but I guess there were a lot of "moments" for my free software origins, not any one moment like "I was bitten by a libre radioactive spider".)

I really enjoyed doing this one, and maybe you'll enjoy listening to it. It's kind of rambly and conversational and we came in with very little as in terms of questions, but I think I tend to do fairly well in that format.

I did cut off the end of the interview by saying I had to go to the bathroom though. Not really the most dignified of exits on my part. Oh well.

Researchcation, concluded

By Chris Lemmer-Webber on Tue 03 June 2014

I just got done with a three week thing I've dubbed "researchcation". It's exactly what it sounds like, research + vacation.

It's hard for me to take time away from MediaGoblin right now and have it still meet its goals as a project. On the other hand, there's a lot that we have planned for the year ahead, but some of it I'm not really prepared enough for to make optimal decisions on. In addition, the last year and a half really have just not given me much of a break at all, and running a crowdfunding campaign (not to mention two over two years) is really exhausting. (Not that I'm complaining about success!)

I was feeling pretty close to burnout, but given how much there is to get done, I decided to take a compromise on this break... instead of taking a full fledged vacation, I'd take a "researchcation": three weeks to recharge my batteries and step away from the day to day of the project. In the meanwhile of that, I'd work on some projects to prepare me for the year ahead. A number of good things came out of it, though not exactly the same things I expected coming in. But I think it was worth the time invested.

My original plan going in was that I would work on two things: something related to the Pump API and federation, and something related to deployment. It turns out I didn't get around to the deployment part, but working on the federation part was insightful, though not in all the ways I anticipated. Though I've read the Pump API document and helped advise a bit on the design of PyPump (not to take credit for that, clearly credit belongs to Jessica Tallon generally, not me), there's nothing really like having a solid project to toss you into things, and I wanted to take a non-MediaGoblin-codebase approach to playing around with the Pump API.

I started out by hacking on a small project called PumpBus, which was going to be a daemon which wrapped pypump and exposed a d-bus API. I figured this would make writing clients easier (and even make it possible to write an emacs client... yeah I know). I got far enough to where I was able to post a message from emacs lisp, then decided that what I was working on just wasn't that interesting and wasn't teaching me much more than I already knew.

Given that there was both the "research" but also the "-cation" component to this, I figured the risks of failure were low, so I'd up the challenge of what I was working on a bit. I instead started working on something I've dubbed Pydraulics: a python-powered implementation of the Pump API. Worst came to worst I'd learn a few things.

I decided from the outset to keep a few assumptions related to pydraulics:

  • The end goal would be to have something that provided interfaces for object storage and retreival... not to wrap the database itself, but hopefully there would not be too many views, and maybe this could happen on a per-view basis. This way you could easily wrap Pydraulics around whatever application and use the storage/database it's already using. That's the end goal. (I didn't get there ;))
  • I'd keep things simple database-wise: assuming you're not providing your own interface, the default interface provided is postgres + sqlalchemy only. There's some new JSON-related features in Postgresql that are pretty exciting and would be appropriate here.
  • I'd use this as an oportunity to think about MediaGoblin's codebase. I decided I'd see how easy or hard it would be to split out components from MediaGoblin as I needed them into something I dubbed "libgoblin". For now, I'd allow this to be messy, but hopefully would give me a chance to think about what libgoblin should be.
  • I'd also use to think about where MediaGoblin fits as in terms of recent developments in asynchronous Python coding.

So, what came out of it?

  • Turns out SQLAlchemy does a nice job of making use of Postgres' built-in JSON support. Early tests seemed to indicate that this choice would pay off well. (Left me wondering: how hard would it be for someone to write a python API-compatible implementation of pymongo or something?)
  • I ended up spending a lot more time on the libgoblin side of things than I expected. I didn't realize that MediaGoblin had become such a self-contained microframework until this point. I wanted to port the MediaGoblin OAuth views over to Pydraulics to save time, but it turned out this required porting over a significant amount of MediaGoblin code over to libgoblin. I did get the oauth views working though!
  • Asynchronous stuff turned out to be interesting to explore, and I'll expand on what I've been thinking below.
  • I did end up getting a much, much stronger sense of the Pump API, which of course was the main goal, though the implementation of that is not yet complete.

Pondering asynchronous coding developments and MediaGoblin/libgoblin/pydraulics turned out to be fruitful. Mostly I have been looking at "what would it take for libgoblin to be usefully integrated into asyncio?"

This turns out to be a bit more challenging than it appears at the outset for one reason: mg_globals. mg_globals is a pretty sad design in MediaGoblin that I'd like to get rid of; basically it makes it easy to write functions that don't have to have the database session and friends, template environment and etc passed into them, because those are set on a global variable level. That works (but is nasty) as long as you're not in a multithreaded environment, but breaks as soon as you are. I recently created a ticket reflecting such, suggesting switching over to werkzeug context locals (Flask makes heavy use of this). Werkzeug's hack is clever, using thread locals so that even in a multi-threaded environment, the objects you're accessing are still globals, but they're the right globals.

But Werkzeug's solution is not good enough for integration with asyncio, where you might be doing "asynchronous" behavior in the same thread, suspending and resuming coroutines or coming back to tasks or etc. As such, it's almost guaranteed in this system that you'll be clobbering the variables another task needs.

What to do? I did research to see if anyone had ideas. It looks like you could do such a thing with Task.current_task() in asyncio, and that would be fairly equivalent. I think you'd need careful implementation though... if you're not paying close attention you might not attach the right things to the right subtask, and that whole thing just seems... fragile. But it still is a neat idea to play around with.

But here's some ideas that I think are neat all combined, related to this problem:

  • The idea that the request or asyncio task is the main object that you attach useful variables to, and you just pass that thing around as a "universal context" like crazy. (The downside: what happens when you aren't using asyncio, or don't need an http request, like in a migration script?)
  • I like the idea of the application being multi-instance'able, and then having requests and a local context as a layer on top of that. So you've got the "instantiated application" layer, and on top of that the "current request" layer, and at both levels you can have variables attached (like the database engine on the application level, but the database session on the request level). That's an awesome distinction.

But you don't really know whether or not some bit of code is using an asyncio task, a web request, or whatever to pass around. Here's the thing though: it doesn't really matter most of the time. With rare exceptions, you're just looking for $OBJECT.db or $OBJECT.templates or something. You just need some kind of object you can tack attributes on to.

So that's my idea in libgoblin/pydraulics: you have an application and you want to do something with it (handle a request, execute a task, etc), you can tack stuff onto that object. So either create a fresh context object to tack stuff onto or just start tacking things onto an object you have!

Currently, this looks like:

# Pydraulics -- Easy Pump API integration into your software.
# Copyright (C) 2014 Pydraulics contributors.  See AUTHORS.
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program.  If not, see <>.
# Large parts of this heavily borrowed from MediaGoblin.
# Copyright (C) 2011-2014 MediaGoblin contributors, see MEDIAGOBLIN-AUTHORS

class PydraulicsApp(object):
    Pydraulics "basic" WSGI application

    # ...

    def gen_context(self, request=None):
        Generate or apply a context.

        If we have a request, use that as the context instead.
        if request is not None:
            c = request
   = self
            c = Context(self)

        c.template_env = jinja2.Environment(
            loader=self.template_loader, autoescape=True,

        # Set up database
        c.db = self.Session()

        if request is not None:
            c.url_map = url_map.bind_to_environ(request.environ, server_name=self.netloc)
            c.url_map = url_map.bind(
                # Maybe support this later

Anyway, simple enough. Then you have request.db made, or if you've just got a command line script and you need the equivalent of that and you already have your instantiated application, just run application.gen_context(). Thus, for utilities that are working with this application and need a variety of instantiated things (the database and the template engine and so on and so on) it's easy enough to just accept "context" as the first argument of the function, then use context.db and etc. (I've considered using just "c" or "ctx" instead of "context" as the variable name since it seems so common and since it conflicts a bit with template context and friends, though that's not very explicit.) So, this seems good.

At one point I got frustrated with the massive amount of porting to libgoblin I was having to do and thought "I really should probably just use Django or Flask itself." However, I found that neither framework really addresses the asyncio stuff I was dealing with above, and once I got enough ported of libgoblin over, libgoblin-based development is very fast and comfortable.

That said, it took up enough time working through those things where I didn't complete the Pump implementation. That's okay, I've got enough to do what's required on my end from MediaGoblin (and we've got good direction and help on the federation end this upcoming year where the most important thing is that I have a good understanding). I still think pydraulics is a pretty neat idea, and I may finish it, though it'll be back-burner'ed for now.

However, libgoblin is something I'm likely to extract. I'm convinced that MediaGoblin is at a point where it's stable enough to know what works and doesn't about the technical design, so that gives me a good basis to know what to build from here. There are other applications I'd like to build which should mesh nicely with MediaGoblin but which really don't belong as part of MediaGoblin itself, and would be kind of hacky add-ons. Clearly this is not the most important development, but towards the end of the summer as we hopefully get the Python 3 branch merged, I will be looking towards this.

Aside from this, on the "-cation" end of things, I took some time to relax and also reapproach my health. I may have a separate post on that soon.

So, that's that. Overall it was productive, but again, not quite in the ways I was expecting. I feel okay about that though... I wanted to do some hacking and not feel deeply pressured or stressed about it... if that wasn't true, I think the "-cation" part wouldn't have held up. So I feel okay that I wandered a bit, and the other things I worked on / found I think are important anyhow, and have me much better prepared for the year ahead. Not to mention the most important part: I feel pretty refreshed and capable of taking it on!

What's next for the coming week? Well now that this is all over, I'm organizing plans so we can get rewards out the door and do project planning for the year ahead. We've got a lot of promises to fulfill. Better get to it!

Empathy for PHP + Shared Hosting (Which is Living in the Past, Dude)

By Chris Lemmer-Webber on Sun 30 March 2014

After I wrote my blogpost yesterday about deployment it generated quite a bit of discussion on the pumpiverse. Mike Linksvayer pointed out (and correctly) that "anti-PHP hate" is a poor excuse for why the rest of us are doing so bad, so I edited that bit out of my text.

After this though, maiki made a great series of posts, first asking "Should a homeless person be able to 'host' MediaGoblin?" and then talking about their own experiences. Go read it and then come back. It's well written and there's lots to think about. (Read the whole thread, in fact!) The sum of it though is that there's a large amount of tech privilege involved in installing a lot of modern web applications, but maiki posts their own experiences about why having access to free software with a lower barrier to entry was key to them making changes in their life, and ends with the phrase "aim lower". (By the way, maiki is actually a MediaGoblin community member and for a long time ran an instance.)

So, let's start out with the following set of assertions, of which I think maiki and I both agree:

  • Tech privilege is a big issue, and that lowering the barrier to entry is critical.
  • PHP + shared hosting is probably the lowest barrier to entry we have, assuming your application falls within certain constraints. This is something PHP does right! (Hence the "empathy for PHP" above.)
  • "Modern" web applications written in Python, Ruby, Node, etc, all require a too much tech privilege to run and maintain, and this is a problem.

So given all that, and given that I "fixed up" my previous post by removing the anti-PHP language, the title I chose for this blogpost probably seems pretty strange, or like it's undoing all that work. And it probably seems strange that given the above, I'll still argue that the choices around MediaGoblin were actively chosen to tackle tech privilege, and that tackling these issues head-on is critical, or free software network services will actually be in a worse place, especially in a tech privilege sense.

That's a lot to unpack, so let's step back.

I think there's an element of my discussion about web technology and even PHP that hasn't been well articulated, and that fault is my own... but it's hard to explain without going into detail. So first of all, apologies; I have been antagonistic towards PHP, and that's unfair to the language that currently powers some of the most important software on earth. That's lame of me, and I apologize.

So that's the empathy part of this title. Then, why would I include that line from my slides, that "PHP is Living in the past, Dude", in this blogpost? It seems to undo everything I'm writing. Well, I want to explain what I meant about the above language. It's not about "PHP sucks". And it does relate to free software's future, and also 5factors into conversations about tech privilege. (It also misleading in that I do not mean that modern web applications can't be written in PHP, or that their communities will be bad for such a choice, but that PHP + shared hosting as a deployment solution assumes constraints insufficient for the network freedom future I think we want.)

Consider the move to GNOME 3, the subject of Bradley's "living in the past" blogpost: during the move to GNOME 3, there were really two tech privilege issues at stake. One is that actually you're requiring newer technology with OpenGL support, and that's a tech privilege issue for people who can't afford that newer technology. (If you volunteered at a FreeGeek center, you'd probably hear this complaint, for example.) But the other one is that GNOME 3 was also trying to make the desktop easier for people, and in a direction of usability that people expect these days. That's also a tech privilege issue, and actually closer to the one we're discussing now: if the barrier to entry is that things are too technical and too foreign to what users know and expect, you're still building a privilege divide. I think GNOME made the right decision on addressing privilege, and I think it was a forward-facing one.

Thus, let me come back around to why, knowing that Python and friends are much harder, I decided to write MediaGoblin in Python anyway.

The first one is functionality. MediaGoblin probably could never be a good video hosting platform on shared hosting + PHP only; the celery component, though it makes it harder to deploy, is the whole reason MediaGoblin can process media in the background without timing out. So in MediaGoblin's case (where media types like video were always viewed as a critical part of the project), Celery does matter. More and more modern web applications are being written in ways that PHP + Shared Hosting just can't provide: they need websockets, they need external daemons which process things, and so on.

And let's not forget that web applications are not the only thing. PHP + shared hosting does not solve the email configuration problem, for example. More and more people are moving to GMail and friends; this is a huge problem for user freedom on the net. And as someone who maintains their own email server, I don't blame them. Configuring and running this stuff is just too hard. And it's not like it's a new technology... email is the oldest stable federated technology we have.

Not to mention that I've argued previously that shared hosting is not user freedom friendly. That's almost a separate conversation, though.

I also disagree that things like encryption certificates, which are also hard, don't matter. I think peoples' privacy does matter immensely, and I think we've only seen more and more reason to believe that this is an area we must work on over the last few years. (You might say that "SSL is doing it wrong" anyway, and I agree, though that's a separate conversation. Proably something that does things right will be just as hard to set up signing-wise if it's actually secure, though.)

Let's also come back to me being a Python programmer. Even given all the above, there are a lot of people out there like me who are just not interested in programming in PHP. This doesn't mean there aren't good PHP communities, clearly there are. But I do think more and more web applications are being written in non-PHP languages, and there's good reason for that. But yes, that means that these web applications are hard to deploy.

What's the answer to that? Assuming that lots of people want to write things in non-PHP languages, and that PHP + shared hosting is insufficient for a growing number of needs anyway, what do we do?

For the most of the non-PHP network services world, it has felt like the answer is to not worry about the end user side of things. Why bother, when you aren't releasing your end web application anyway? And so we've seen the rise of devops coincide with the rise of "release everything but your secret sauce" (and, whether you like it or not, with the decline of PHP + shared hosting).

I was fully aware of all of this when I decided MediaGoblin would be written in Python. Part of it is because I like Python, and well, I'm the one starting the project! But part of it is because the patterns I described above are not going away. In order for us to engage the future of the web, I think we need to tackle this direction head-on.

In the meanwhile, it's hard. It's hard in the way that installing and maintaining a free software desktop was super hard for me back in 2001, when I became involved in free software for the first time. But installers have gotten better, and the desktop has gotten better. The need for the installfest has gone away. I think that we are in a similar state with free network services, but I believe things can be improved. And that's why I wrote that piece yesterday about deployment, because I am trying to think about how to make things better. And I believe we need to, to build web applications that meet the needs of what people expect, to make free network services comparable to the devops-backed modern architected proprietary network services of today.

So, despite what it might appear at the moment, tech privilege has always been on my mind, but it's something that's forward-looking. That's hard to explain though when you're stuck in the present. I hope this blogpost helps.

MediaGoblin's campaign for federation and privacy

By Chris Lemmer-Webber on Wed 12 March 2014

The MediaGoblin campaign is live! Well okay... it's been live for a couple of weeks now. I think the video above explains everything we're trying to do pretty well, so maybe you should watch that first. (Better yet, watch it on the campaign page, and hopefully donate while you're at it!)

So our website and campaign page and video and etc try to explain why we think you should donate to the campaign. But I thought I'd write here, also... there's something different, I think, about a personal blog post... some things are more easily said. So let me ramble on a bit.

I guess the easiest thing to open with is the most obvious... it's been an interesting year as in terms of making it clear *why* MediaGoblin matters. The danger of Snowden revelations have made it obvious that a highly centralized internet is a problem.

But awareness alone won't fix the problem, we need to really build solutions. I think just how true this is became obvious to me earlier in the year, when I spoke to another prominent internet activist (I won't name names) who said to me more or less: "The centralized internet is a problem, but we don't actually think we can get people to change their habits, that's too hard. So instead we're focused on talking about the problem and writing up what rights users should have."

I've thought about this line of reasoning a lot. I agree that getting people to change their habits is really hard. And raising awareness and talking about rights are super important. And pushing for governmental reforms are important. But let's face it, the NSA snooping was already breaking laws and violating our rights, and there isn't any evidence that those programs are ending any time soon, especially when it's so easy to keep them going in the present technological environment. We need to build something better. We need to actually build tools and make them usable and even enjoyable so that people can switch away.

To put it another way: when even the most prominent internet activist campaigns are using Facebook, YouTube, and Twitter to complain about the centralization effects those very services help to perpetuate, it shows you how much we need ways to communicate that aren't part of the problem. And that's exactly what we're working on with MediaGoblin.

It's true that these are hard things to do. They take resources, they take time. But we have to do them. And we can do them.

unlock characters

We have an opportunity here with MediaGoblin. If we can hit 1.0 and get federation support into MediaGoblin (which is mostly the first goal of the fundraising campaign), that alone would be huge. But we'd like to do more than that... we'd like to invest resources into making adding federation support easier to python web applications generally, add privacy features, and a bunch more. We've laid out what those goals are specifically in the "Unlock More Features!" section of the campaign page.

MediaGoblin is more than that, too. MediaGoblin is also a vision for what we think the future of free software could be. We work on network freedom issues because in a networked age, without network freedom, there is no user freedom. We work on making the software beautiful because we believe beautiful free software web applications are the only way that free software can be adopted by the world. We support diversity initiatives because we think diversity is important on its own, and because we believe that a diverse project is a better project. We work on messaging and making messaging that tries to be as accessible to everyone as it can, both because free software is something that everyone should enjoy, and without clear explainations of why these things matter, free software will remain a privilege for a technical elite. We believe user freedom belongs to everyone.

If that resonates with you, I encourage you to support our campaign. And consider spreading the word. Anything you do really does make a huge difference.

Thanks, internet. We do it for you.