Archives

Tags

advocacy researchcation wedding music mediagoblin biking money free-culture php cc0 blender shared hosting libreplanet emacs cooking pycon freespeech miro sfxr campaign flourish art politics firebug cabbage xmpp talk food fsf blog chipy diet anniversary kinda-mediagoblin phone frostcast speech recognition orgmode release art by-sa tabletop openmoko gimp pelican gpl hatecrime foss base64 dustycloud economics personal security configuration recipe boingboing sqlite xkcd robogoblins contributor drawing uuid deployment autonomous firefox backups nouveau python n900 conference libre-graphics-meeting life update code patents cookies identica pcf robots rsi meme interview podcast xudd licensing tech privilege creative-commons zine vegan family gnome

Researchcation, concluded

By Christopher Allan Webber on Tue 03 June 2014

I just got done with a three week thing I've dubbed "researchcation". It's exactly what it sounds like, research + vacation.

It's hard for me to take time away from MediaGoblin right now and have it still meet its goals as a project. On the other hand, there's a lot that we have planned for the year ahead, but some of it I'm not really prepared enough for to make optimal decisions on. In addition, the last year and a half really have just not given me much of a break at all, and running a crowdfunding campaign (not to mention two over two years) is really exhausting. (Not that I'm complaining about success!)

I was feeling pretty close to burnout, but given how much there is to get done, I decided to take a compromise on this break... instead of taking a full fledged vacation, I'd take a "researchcation": three weeks to recharge my batteries and step away from the day to day of the project. In the meanwhile of that, I'd work on some projects to prepare me for the year ahead. A number of good things came out of it, though not exactly the same things I expected coming in. But I think it was worth the time invested.

My original plan going in was that I would work on two things: something related to the Pump API and federation, and something related to deployment. It turns out I didn't get around to the deployment part, but working on the federation part was insightful, though not in all the ways I anticipated. Though I've read the Pump API document and helped advise a bit on the design of PyPump (not to take credit for that, clearly credit belongs to Jessica Tallon generally, not me), there's nothing really like having a solid project to toss you into things, and I wanted to take a non-MediaGoblin-codebase approach to playing around with the Pump API.

I started out by hacking on a small project called PumpBus, which was going to be a daemon which wrapped pypump and exposed a d-bus API. I figured this would make writing clients easier (and even make it possible to write an emacs client... yeah I know). I got far enough to where I was able to post a message from emacs lisp, then decided that what I was working on just wasn't that interesting and wasn't teaching me much more than I already knew.

Given that there was both the "research" but also the "-cation" component to this, I figured the risks of failure were low, so I'd up the challenge of what I was working on a bit. I instead started working on something I've dubbed Pydraulics: a python-powered implementation of the Pump API. Worst came to worst I'd learn a few things.

I decided from the outset to keep a few assumptions related to pydraulics:

  • The end goal would be to have something that provided interfaces for object storage and retreival... not to wrap the database itself, but hopefully there would not be too many views, and maybe this could happen on a per-view basis. This way you could easily wrap Pydraulics around whatever application and use the storage/database it's already using. That's the end goal. (I didn't get there ;))
  • I'd keep things simple database-wise: assuming you're not providing your own interface, the default interface provided is postgres + sqlalchemy only. There's some new JSON-related features in Postgresql that are pretty exciting and would be appropriate here.
  • I'd use this as an oportunity to think about MediaGoblin's codebase. I decided I'd see how easy or hard it would be to split out components from MediaGoblin as I needed them into something I dubbed "libgoblin". For now, I'd allow this to be messy, but hopefully would give me a chance to think about what libgoblin should be.
  • I'd also use to think about where MediaGoblin fits as in terms of recent developments in asynchronous Python coding.

So, what came out of it?

  • Turns out SQLAlchemy does a nice job of making use of Postgres' built-in JSON support. Early tests seemed to indicate that this choice would pay off well. (Left me wondering: how hard would it be for someone to write a python API-compatible implementation of pymongo or something?)
  • I ended up spending a lot more time on the libgoblin side of things than I expected. I didn't realize that MediaGoblin had become such a self-contained microframework until this point. I wanted to port the MediaGoblin OAuth views over to Pydraulics to save time, but it turned out this required porting over a significant amount of MediaGoblin code over to libgoblin. I did get the oauth views working though!
  • Asynchronous stuff turned out to be interesting to explore, and I'll expand on what I've been thinking below.
  • I did end up getting a much, much stronger sense of the Pump API, which of course was the main goal, though the implementation of that is not yet complete.

Pondering asynchronous coding developments and MediaGoblin/libgoblin/pydraulics turned out to be fruitful. Mostly I have been looking at "what would it take for libgoblin to be usefully integrated into asyncio?"

This turns out to be a bit more challenging than it appears at the outset for one reason: mg_globals. mg_globals is a pretty sad design in MediaGoblin that I'd like to get rid of; basically it makes it easy to write functions that don't have to have the database session and friends, template environment and etc passed into them, because those are set on a global variable level. That works (but is nasty) as long as you're not in a multithreaded environment, but breaks as soon as you are. I recently created a ticket reflecting such, suggesting switching over to werkzeug context locals (Flask makes heavy use of this). Werkzeug's hack is clever, using thread locals so that even in a multi-threaded environment, the objects you're accessing are still globals, but they're the right globals.

But Werkzeug's solution is not good enough for integration with asyncio, where you might be doing "asynchronous" behavior in the same thread, suspending and resuming coroutines or coming back to tasks or etc. As such, it's almost guaranteed in this system that you'll be clobbering the variables another task needs.

What to do? I did research to see if anyone had ideas. It looks like you could do such a thing with Task.current_task() in asyncio, and that would be fairly equivalent. I think you'd need careful implementation though... if you're not paying close attention you might not attach the right things to the right subtask, and that whole thing just seems... fragile. But it still is a neat idea to play around with.

But here's some ideas that I think are neat all combined, related to this problem:

  • The idea that the request or asyncio task is the main object that you attach useful variables to, and you just pass that thing around as a "universal context" like crazy. (The downside: what happens when you aren't using asyncio, or don't need an http request, like in a migration script?)
  • I like the idea of the application being multi-instance'able, and then having requests and a local context as a layer on top of that. So you've got the "instantiated application" layer, and on top of that the "current request" layer, and at both levels you can have variables attached (like the database engine on the application level, but the database session on the request level). That's an awesome distinction.

But you don't really know whether or not some bit of code is using an asyncio task, a web request, or whatever to pass around. Here's the thing though: it doesn't really matter most of the time. With rare exceptions, you're just looking for $OBJECT.db or $OBJECT.templates or something. You just need some kind of object you can tack attributes on to.

So that's my idea in libgoblin/pydraulics: you have an application and you want to do something with it (handle a request, execute a task, etc), you can tack stuff onto that object. So either create a fresh context object to tack stuff onto or just start tacking things onto an object you have!

Currently, this looks like:

# Pydraulics -- Easy Pump API integration into your software.
#
# Copyright (C) 2014 Pydraulics contributors.  See AUTHORS.
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
#
# Large parts of this heavily borrowed from MediaGoblin.
# Copyright (C) 2011-2014 MediaGoblin contributors, see MEDIAGOBLIN-AUTHORS

class PydraulicsApp(object):
    """
    Pydraulics "basic" WSGI application
    """

    # ...

    def gen_context(self, request=None):
        """
        Generate or apply a context.

        If we have a request, use that as the context instead.
        """
        if request is not None:
            c = request
            c.app = self
        else:
            c = Context(self)

        c.template_env = jinja2.Environment(
            loader=self.template_loader, autoescape=True,
            undefined=jinja2.StrictUndefined,
            extensions=[
                'jinja2.ext.autoescape'])

        # Set up database
        c.db = self.Session()

        if request is not None:
            c.url_map = url_map.bind_to_environ(request.environ, server_name=self.netloc)
        else:
            c.url_map = url_map.bind(
                self.netloc,
                # Maybe support this later
                script_name=None)

Anyway, simple enough. Then you have request.db made, or if you've just got a command line script and you need the equivalent of that and you already have your instantiated application, just run application.gen_context(). Thus, for utilities that are working with this application and need a variety of instantiated things (the database and the template engine and so on and so on) it's easy enough to just accept "context" as the first argument of the function, then use context.db and etc. (I've considered using just "c" or "ctx" instead of "context" as the variable name since it seems so common and since it conflicts a bit with template context and friends, though that's not very explicit.) So, this seems good.

At one point I got frustrated with the massive amount of porting to libgoblin I was having to do and thought "I really should probably just use Django or Flask itself." However, I found that neither framework really addresses the asyncio stuff I was dealing with above, and once I got enough ported of libgoblin over, libgoblin-based development is very fast and comfortable.

That said, it took up enough time working through those things where I didn't complete the Pump implementation. That's okay, I've got enough to do what's required on my end from MediaGoblin (and we've got good direction and help on the federation end this upcoming year where the most important thing is that I have a good understanding). I still think pydraulics is a pretty neat idea, and I may finish it, though it'll be back-burner'ed for now.

However, libgoblin is something I'm likely to extract. I'm convinced that MediaGoblin is at a point where it's stable enough to know what works and doesn't about the technical design, so that gives me a good basis to know what to build from here. There are other applications I'd like to build which should mesh nicely with MediaGoblin but which really don't belong as part of MediaGoblin itself, and would be kind of hacky add-ons. Clearly this is not the most important development, but towards the end of the summer as we hopefully get the Python 3 branch merged, I will be looking towards this.

Aside from this, on the "-cation" end of things, I took some time to relax and also reapproach my health. I may have a separate post on that soon.

So, that's that. Overall it was productive, but again, not quite in the ways I was expecting. I feel okay about that though... I wanted to do some hacking and not feel deeply pressured or stressed about it... if that wasn't true, I think the "-cation" part wouldn't have held up. So I feel okay that I wandered a bit, and the other things I worked on / found I think are important anyhow, and have me much better prepared for the year ahead. Not to mention the most important part: I feel pretty refreshed and capable of taking it on!

What's next for the coming week? Well now that this is all over, I'm organizing plans so we can get rewards out the door and do project planning for the year ahead. We've got a lot of promises to fulfill. Better get to it!

Fifth wedding anniversary, the RPG

By Christopher Allan Webber on Fri 30 May 2014

We had an awesome wedding anniversary today... 5 years! Morgan didn't know I had planned something ahead of time. I'm not good at surprises, but this time I managed to pull off a surprise, and I think with massive style.

So, some context. First of all, a number of my friends and I, and I guess me especially, have kind of become tabletop role playing game nerds over the last year. We've been playing a wide variety of games, especially in Fate (the basis of which is under a free culture license!), from space adventures to crime dramas to German-esque fairytale stories to some rather silly holiday adventures, to large scale inter-kingdom dramas. Though we have played some games in the style of $STANDARD_FANTASY games (what most people think of when they think of tabletop RPGs), most of the stuff we've been playing has been more narrative even then, and less about straight up monster-killing dungeon crawling.

We have played some games in the category of $STANDARD_FANTASY though, and one in particular has a character that my spouse Morgan plays in, a sort of "professor of magic history" (yeah, people who know Morgan may notice some overlaps) who uses jewelry to actually power up her abilities, but isn't particularly innately magically inclined. Morgan is not really one for fancy jewelry, but she does like the engagement ring that I got her, which is a bit fancy. When she plays as this character, she actually dresses up for it, and wears some matching earrings as well.

So anyway, today we played a game in that universe, and I GM'ed (my brother and his girlfriend also participated, they were awesome). Morgan was arriving in a somewhat swashbuckling'y type city, so I compelled her that she probably would have to remove her ring (the source of half her magic) until she figured out a way to get some sort of protection from thievery in the city. I figured this would piss off Morgan... I didn't realize how much. "I'm not playing a game with you if you take away my jewelry." She eventually accepted the compel after encouragement from other players, but she was pretty fuming'ly mad about it. "You're making me take off my engagement ring on our anniversary?" (Morgan had thought I had accidentally planned this game on our anniversary without realizing it. She didn't realize that for once in our lives, I was being smooth about things.)

It was a good game generally. There was an artifact delivered, etc etc. By the end of the game, an NPC said "Oh, thanks, I think I have a jewel that matches yours... I can't really use it because it's for fire magic users only. Now where did I put that?"

In real life I shuffled through some drawers looking and pulled out a small beat up looking urn. "Careful, this thing is hot. Only fire magic users seem to be able to open it with no problem."

Of course her character did, and of course she was able to bypass the "smoke" (white tissue paper) no problem. And what was inside? A pendant that perfectly matches her ring. ("Conflict-free" in real life, supposedly, to the extent those things are true. Also, aside from her engagement ring, this is the only piece of jewelry I have ever bought Morgan. We're not really that much of jewelry-presents type people, but I think an exception here was successful.) Oh yeah, and in-game the pendant gave her a fate-point-activated fire shield that can give her an offensive-defense against people trying to pickpocket her or doing melee attacks.

It was a stunt for sure, but it worked. It extra worked because usually I am such a doofus, and I did piss off Morgan by compelling her earlier, and it looked like there were a bunch of bits that turned out to be just me making stupid mistakes that then turned around and made the end reveal awesome.

Somehow I pulled off that stunt, all the players seemed to have a good time generally, I didn't give it away beforehand (usually I'm not so successfully sneaky), and yes, my intentional-though-appearing-accidental pissing-off maneuvers really did make the final result more fun for everyone. Success!

Anyway, so that's that. Morgan told me I'm the best, and nerdiest, husband ever, so I'll gladly take all of that.

Happy fifth anniversary, Morgan. Here's to many more.

A sweet and savory cabbage recipe

By Christopher Allan Webber on Sat 24 May 2014

There's plenty of interesting things to talk about lately, and I'll get to them soon. I'm on something I've titled "research-cation" where I'm still kind of working, but it's also kind of like vacation, but really I'm mostly working on doing research for MediaGoblin's future.

In the meanwhile, I'm back in diet mode, basically because the MediaGoblin campaign was hard on my health. But also, the tooling I had in orgmode was never that great, so I've revamped org-diet. I might write a separate post on this... there's a lot of reasons why I did the revamp (it's not in master yet, but in the date-tree branch). I'm now doing daily uploads of my current health status which you can view here (yes, org-diet now is super flexible about generating reports).

I'm not going into details on that in this post, but I did recently just re-make one of my favorite recipes of all time with a number of adjustments. I forgot just how good it is. Anyway, here it is:

Ingredients Calories Quantity Total
head cabbage 290 1 290
tbsp olive oil 119 1 119
Westsoy baked tofu square 90 4 360
medium onion 44 1 44
can kidney beans 385 1 385
apple 71 2 142
tbsp nutritional yeast 25 2 50
tbsp vegetarian bullion 0 1.5 0
tbsp tamari / braggs liquid aminos 0 2 0
tbsp cornstarch 30 1/3 10
clove garlic 4 4 16
Total   8 177

This recipe is cheap, healthy, and most importantly, delicious. It has very few calories (a mere 177 calories... that's nothing!) but tastes pretty amazing. I usually start some rice in the rice cooker before I kick this off… put in two cups and that's a mere 120 calories on top of this. Only 297 calories! Despite that, it's quite filling. (Tasty, too!)

The nutritional yeast is optional, but I like it. You can use whatever bullion you like, but I like the Frontier Natural Products beef-ish tasting vegetarian bullion. Alternately, adding brewers yeast and a bit more salt is great.

You also don't have to use the westsoy baked tofu. You could use any other protein here. A lot of other kinds you have to fry up in advance though, and the westsoy stuff is already done and tastes great and I'm lazy. If you don't have tamari or liquid aminos, just up the bullion.

This makes 8 servings! It usually takes me about 50 minutes to make but I'm slow.

Okay, so! Here's my recipe. You're going to need a large pot, a large cutting board, and a large mixing bowl.

  • Get out a large cutting board and chop up cabbage. You want it in pieces probably, though if you prefer tiny ribbons that's fine. Set aside in mixing bowl.
  • Chop onions and, if you like, a couple cloves of garlic. Chop up the apples into chunks or wedges. Personally I like wedges.
  • Take your large pot, add the olive oil (or whatever oil really) and onion (also garlic if using). Saute that for a few minutes, until it starts to brown. Add apple and saute a bit longer, until the apple starts to brown a little.
  • Add a cup of water and stir around the ingredients for about two minutes.
  • Dump in the cabbage into the pot. It'll seem like a lot and like you'll never be able to stir this thing. (I told you to get a big pot!) Don't worry, it cooks down.
  • Add salt and pepper. Add some more water… I think I usually add about 1 more cup at this point. Stir around the cabbage in and out of the water as best you can for a minute or so. Then cover the pot and let it cook down for five minutes.
  • While the cabbage is cooking, chop the baked tofu into cubes, or tear it apart with your fingers if you get grossed out by cubes of tofu.
  • Return to the pot. The cabbage should be a bit more cooked down now, but not quite there. Add bullion, nutritional yeast, and tamari/liquid aminos. Open the kidney beans and pour the excess liquid right into the pot. Add more salt and pepper if you like. Now stir that stuff. Get that cabbage in and out of the broth!
  • At this point you need to let the cabbage cook down. I usually let it cook down a little bit less than half way. Stir it occasionally.
  • Stir together the cornstarch in 1/4 cup of water. You're trying to make a small slurry that'll thicken the broth into a kind of gravy.
  • Open the pot and pour the cornstarch in and add the tofu. Time to start stirring again. Stir stir stir!
  • Give it a few minutes and the sauce should thicken. The cabbage should become tender but not totally mushy. When you hit that point, stop cooking.
  • Serve with rice or some kind of grain. I usually put a tablespoon of cheap parmesan cheese on top too, but whatever you like, go for it.

Enjoy!

Empathy for PHP + Shared Hosting (Which is Living in the Past, Dude)

By Christopher Allan Webber on Sun 30 March 2014

After I wrote my blogpost yesterday about deployment it generated quite a bit of discussion on the pumpiverse. Mike Linksvayer pointed out (and correctly) that "anti-PHP hate" is a poor excuse for why the rest of us are doing so bad, so I edited that bit out of my text.

After this though, maiki made a great series of posts, first asking "Should a homeless person be able to 'host' MediaGoblin?" and then talking about their own experiences. Go read it and then come back. It's well written and there's lots to think about. (Read the whole thread, in fact!) The sum of it though is that there's a large amount of tech privilege involved in installing a lot of modern web applications, but maiki posts their own experiences about why having access to free software with a lower barrier to entry was key to them making changes in their life, and ends with the phrase "aim lower". (By the way, maiki is actually a MediaGoblin community member and for a long time ran an instance.)

So, let's start out with the following set of assertions, of which I think maiki and I both agree:

  • Tech privilege is a big issue, and that lowering the barrier to entry is critical.
  • PHP + shared hosting is probably the lowest barrier to entry we have, assuming your application falls within certain constraints. This is something PHP does right! (Hence the "empathy for PHP" above.)
  • "Modern" web applications written in Python, Ruby, Node, etc, all require a too much tech privilege to run and maintain, and this is a problem.

So given all that, and given that I "fixed up" my previous post by removing the anti-PHP language, the title I chose for this blogpost probably seems pretty strange, or like it's undoing all that work. And it probably seems strange that given the above, I'll still argue that the choices around MediaGoblin were actively chosen to tackle tech privilege, and that tackling these issues head-on is critical, or free software network services will actually be in a worse place, especially in a tech privilege sense.

That's a lot to unpack, so let's step back.

I think there's an element of my discussion about web technology and even PHP that hasn't been well articulated, and that fault is my own... but it's hard to explain without going into detail. So first of all, apologies; I have been antagonistic towards PHP, and that's unfair to the language that currently powers some of the most important software on earth. That's lame of me, and I apologize.

So that's the empathy part of this title. Then, why would I include that line from my slides, that "PHP is Living in the past, Dude", in this blogpost? It seems to undo everything I'm writing. Well, I want to explain what I meant about the above language. It's not about "PHP sucks". And it does relate to free software's future, and also 5factors into conversations about tech privilege. (It also misleading in that I do not mean that modern web applications can't be written in PHP, or that their communities will be bad for such a choice, but that PHP + shared hosting as a deployment solution assumes constraints insufficient for the network freedom future I think we want.)

Consider the move to GNOME 3, the subject of Bradley's "living in the past" blogpost: during the move to GNOME 3, there were really two tech privilege issues at stake. One is that actually you're requiring newer technology with OpenGL support, and that's a tech privilege issue for people who can't afford that newer technology. (If you volunteered at a FreeGeek center, you'd probably hear this complaint, for example.) But the other one is that GNOME 3 was also trying to make the desktop easier for people, and in a direction of usability that people expect these days. That's also a tech privilege issue, and actually closer to the one we're discussing now: if the barrier to entry is that things are too technical and too foreign to what users know and expect, you're still building a privilege divide. I think GNOME made the right decision on addressing privilege, and I think it was a forward-facing one.

Thus, let me come back around to why, knowing that Python and friends are much harder, I decided to write MediaGoblin in Python anyway.

The first one is functionality. MediaGoblin probably could never be a good video hosting platform on shared hosting + PHP only; the celery component, though it makes it harder to deploy, is the whole reason MediaGoblin can process media in the background without timing out. So in MediaGoblin's case (where media types like video were always viewed as a critical part of the project), Celery does matter. More and more modern web applications are being written in ways that PHP + Shared Hosting just can't provide: they need websockets, they need external daemons which process things, and so on.

And let's not forget that web applications are not the only thing. PHP + shared hosting does not solve the email configuration problem, for example. More and more people are moving to GMail and friends; this is a huge problem for user freedom on the net. And as someone who maintains their own email server, I don't blame them. Configuring and running this stuff is just too hard. And it's not like it's a new technology... email is the oldest stable federated technology we have.

Not to mention that I've argued previously that shared hosting is not user freedom friendly. That's almost a separate conversation, though.

I also disagree that things like encryption certificates, which are also hard, don't matter. I think peoples' privacy does matter immensely, and I think we've only seen more and more reason to believe that this is an area we must work on over the last few years. (You might say that "SSL is doing it wrong" anyway, and I agree, though that's a separate conversation. Proably something that does things right will be just as hard to set up signing-wise if it's actually secure, though.)

Let's also come back to me being a Python programmer. Even given all the above, there are a lot of people out there like me who are just not interested in programming in PHP. This doesn't mean there aren't good PHP communities, clearly there are. But I do think more and more web applications are being written in non-PHP languages, and there's good reason for that. But yes, that means that these web applications are hard to deploy.

What's the answer to that? Assuming that lots of people want to write things in non-PHP languages, and that PHP + shared hosting is insufficient for a growing number of needs anyway, what do we do?

For the most of the non-PHP network services world, it has felt like the answer is to not worry about the end user side of things. Why bother, when you aren't releasing your end web application anyway? And so we've seen the rise of devops coincide with the rise of "release everything but your secret sauce" (and, whether you like it or not, with the decline of PHP + shared hosting).

I was fully aware of all of this when I decided MediaGoblin would be written in Python. Part of it is because I like Python, and well, I'm the one starting the project! But part of it is because the patterns I described above are not going away. In order for us to engage the future of the web, I think we need to tackle this direction head-on.

In the meanwhile, it's hard. It's hard in the way that installing and maintaining a free software desktop was super hard for me back in 2001, when I became involved in free software for the first time. But installers have gotten better, and the desktop has gotten better. The need for the installfest has gone away. I think that we are in a similar state with free network services, but I believe things can be improved. And that's why I wrote that piece yesterday about deployment, because I am trying to think about how to make things better. And I believe we need to, to build web applications that meet the needs of what people expect, to make free network services comparable to the devops-backed modern architected proprietary network services of today.

So, despite what it might appear at the moment, tech privilege has always been on my mind, but it's something that's forward-looking. That's hard to explain though when you're stuck in the present. I hope this blogpost helps.

Configuration Management for the People

By Christopher Allan Webber on Sat 29 March 2014

One of the things I've talked about in the talks I've been giving about MediaGoblin lately is the issue of deployment. Basically it boils down to the following points:

  • MediaGoblin is harder than we'd like to deploy, but it's probably easier than most other Python web applications. All modern web applications are currently hard to maintain and deploy.
  • We've worked to try to make it so you don't have to know how to be a $LANGUAGE developer (in this case, Python) to deploy MediaGoblin, but once things break you do have to be a $LANGUAGE developer, or know one. (We spend a lot of time answering these things in #mediagoblin. This can be improved if we get proper system packaging though (which is currently a work in progress, at least for Debian. I hope to see it soon...))
  • Even if you get system packaging, that's not enough. You need to do a mess of other things to set up web applications: configure the web server (Apache, Nginx), configure sending mail, configure a bunch of things. And let's not even talk about getting SSL set up right. Oy.
  • The reason people don't see that modern web applications are hard to deploy is because even though they are, there's a team of devops behind the scenes handling it for them.

We can do as good as we can to try to make MediaGoblin's docs easy to understand, but I'm convinced the solution needs to be a layer higher than MediaGoblin. That's probably one of two things:

  • A Platform as a Service solution like OpenShift. (Note, there are other proprietary ones out there, but if it's not free software, what's the point?) These solutions are kind of heavy, but they seem like a step in the right direction.
  • A configuration and deployment abstraction system, like Puppet/Chef/SaltStack/Ansible, etc.

The latter seems like the right solution to me, but it's not enough as-is. The configuration and deployment systems we presently have are too devops-focused. We can't free society by expecting everyone to join the world of devops... for one thing, these tools are way too hard to learn at present, and for second, a world where only the technically elite are free is a pretty nonfree world, really.

(I don't think things like Docker or virtual machine images are the answer. Aside from being pretty heavy, they don't solve the configuration issue.)

But these systems are pretty close. Close enough even! I think we can pull this off. Let's see what we need and what's missing:

  • Needs to have share'able recipes. (Yes yes, people currently do sad, sad things like dumping some recipes to code hosting platforms. That's not the same thing. I want something like an apt repository, but for recipes. (Note, juju might have this... I don't know much about it))
  • Recipes should be pretty simple... based on variables set by the user, or variables interpolated by user-specific settings (all these config management systems handle this, I think)... mostly I like what I've seen of how Salt handles this.
  • There should be an expectation that you should be able to mix and match recipes somewhat. This means some agreements higher up the chain on how we expect a mail server configuration is going to be described, etc. I'm not sure how the standards/governance around this could be best handled.
  • I think a layer on top of all this that's really needed is some kind of web UI for application install and configuration. If I install the MediaGoblin recipe, it may be that a lot of defaults can be set and guessed. But what if I want to turn on the video plugin, and I want to change my authentication system to using LDAP because that's what my company already uses, etc? I think this can be pretty minimal, we can have a specification that both describes config options as well as how to represent them in the web UI.
  • It shouldn't be tied to any one specific platform. Not to a wall wart, not to a VPS, not to a Raspberry Pi. It should be generic enough to work on all of these. Again, I see no reason this can't be pretty minimal.

I've been fielding this by people for a while, trying to quiz all the smarter-than-me-people I know about what they think, but I keep coming back to this. At a dinner at LibrePlanet, all the usual suspects were raised as possible solutions, and none of them seemed to fit. I was happy to hear Joey Hess say, "I think that Chris is right, we need the 'layer above apt' configuration solution." (If that wasn't some reassurance that I'm on the right track, I don't know what would be... if anyone would know it would be Joey...)

(Note: GNUToo suggested at FOSDEM that I should look at LuCI... LuCI always felt a bit clumsier than I'd like when I used it but maybe it does do these things. I don't know if it handling recipes like I said, but maybe? Worth looking into.)

Here's a rough plan on how I'd go about doing this, if I had time to do this (which I don't think I do, though I can help someone kick off the project, and I can make some contributions):

  • Start by investigating Salt (or Ansible?). It's a little bit heavy but not too heavy (the current python package appears to be 2.8mb...). Plan on using the master-less setup. Salt provides a lot of abstractions around installihng packages and a lot of these other things, so that may be a good start.
  • It might be that things need to be paired down to something even simpler, or that Salt is just too hard to build on top of. If so, start simple on recipes. I'd save a dependency graph system as something optional as a "milestone 2" type thing, personally.
  • You need some way to bundle a description of the variables to be provided but also how to represent them configuration-wise to the user. I think a Hy DSL might be the fastest way to start writing up variable descriptions including web represenation, but also embed code to retreive results in case a default is not provided.
  • At this point you should have a command-line tool that you can run, reads your currently installed recipes and your current settings, and executes them.
  • Build a web UI.
  • Figure out how to publish up a public repository of recipes. (May need to be distro specific.)

That's it I think. I think if we had something like this, it would simplify deployment greatly.

Are you interested in taking this on? We should talk! ;)

PS: We'd like to do further research and work into making MediaGoblin easier to deploy over the next year. Our capacity to do that largely depends on how much we're able to raise... consider donating to our campaign!

Edit: Mike Linksvayer makes strong points about previous language around PHP; removed. So, reverting language that makes PHP sounds like a problem, but I'll still argue that it's not actually a full solution (there are configuration issues not resolved by language choice)

Edit 2: I guess I didn't say this, so it's worth saying... a lot of difficulties in modern deployment are because people aren't using system packaging (this includes MediaGoblin's docs, which suggest the breaks-all-the-time-even-though-I-understand-why-but-our-users-don't world of Python packaging... we're waiting on Debian packaging. Real Soon Now, I hope!). Using system packaging certainly solves a lot of these headaches, but it doesn't solve nearly all of them. There's still the issues of configuring things, and there's really a lot... too much!... to configure: mail transfer agents, hooking your application up to the web server, SSL, so on and so on. That's something that hasn't been solved, especially not for non-devops people.

I've started cobbling together something that might solve things for non-devops people, and even devops people both! Maybe it'll see the light of day sometime. In the meanwhile, I'm really interested in other peoples' solutions to the problems described above.