I just got done with a three week thing I've dubbed "researchcation". It's exactly what it sounds like, research + vacation.

It's hard for me to take time away from MediaGoblin right now and have it still meet its goals as a project. On the other hand, there's a lot that we have planned for the year ahead, but some of it I'm not really prepared enough for to make optimal decisions on. In addition, the last year and a half really have just not given me much of a break at all, and running a crowdfunding campaign (not to mention two over two years) is really exhausting. (Not that I'm complaining about success!)

I was feeling pretty close to burnout, but given how much there is to get done, I decided to take a compromise on this break... instead of taking a full fledged vacation, I'd take a "researchcation": three weeks to recharge my batteries and step away from the day to day of the project. In the meanwhile of that, I'd work on some projects to prepare me for the year ahead. A number of good things came out of it, though not exactly the same things I expected coming in. But I think it was worth the time invested.

My original plan going in was that I would work on two things: something related to the Pump API and federation, and something related to deployment. It turns out I didn't get around to the deployment part, but working on the federation part was insightful, though not in all the ways I anticipated. Though I've read the Pump API document and helped advise a bit on the design of PyPump (not to take credit for that, clearly credit belongs to Jessica Tallon generally, not me), there's nothing really like having a solid project to toss you into things, and I wanted to take a non-MediaGoblin-codebase approach to playing around with the Pump API.

I started out by hacking on a small project called PumpBus, which was going to be a daemon which wrapped pypump and exposed a d-bus API. I figured this would make writing clients easier (and even make it possible to write an emacs client... yeah I know). I got far enough to where I was able to post a message from emacs lisp, then decided that what I was working on just wasn't that interesting and wasn't teaching me much more than I already knew.

Given that there was both the "research" but also the "-cation" component to this, I figured the risks of failure were low, so I'd up the challenge of what I was working on a bit. I instead started working on something I've dubbed Pydraulics: a python-powered implementation of the Pump API. Worst came to worst I'd learn a few things.

I decided from the outset to keep a few assumptions related to pydraulics:

• The end goal would be to have something that provided interfaces for object storage and retreival... not to wrap the database itself, but hopefully there would not be too many views, and maybe this could happen on a per-view basis. This way you could easily wrap Pydraulics around whatever application and use the storage/database it's already using. That's the end goal. (I didn't get there ;))
• I'd keep things simple database-wise: assuming you're not providing your own interface, the default interface provided is postgres + sqlalchemy only. There's some new JSON-related features in Postgresql that are pretty exciting and would be appropriate here.
• I'd use this as an oportunity to think about MediaGoblin's codebase. I decided I'd see how easy or hard it would be to split out components from MediaGoblin as I needed them into something I dubbed "libgoblin". For now, I'd allow this to be messy, but hopefully would give me a chance to think about what libgoblin should be.
• I'd also use to think about where MediaGoblin fits as in terms of recent developments in asynchronous Python coding.

So, what came out of it?

• Turns out SQLAlchemy does a nice job of making use of Postgres' built-in JSON support. Early tests seemed to indicate that this choice would pay off well. (Left me wondering: how hard would it be for someone to write a python API-compatible implementation of pymongo or something?)
• I ended up spending a lot more time on the libgoblin side of things than I expected. I didn't realize that MediaGoblin had become such a self-contained microframework until this point. I wanted to port the MediaGoblin OAuth views over to Pydraulics to save time, but it turned out this required porting over a significant amount of MediaGoblin code over to libgoblin. I did get the oauth views working though!
• Asynchronous stuff turned out to be interesting to explore, and I'll expand on what I've been thinking below.
• I did end up getting a much, much stronger sense of the Pump API, which of course was the main goal, though the implementation of that is not yet complete.

Pondering asynchronous coding developments and MediaGoblin/libgoblin/pydraulics turned out to be fruitful. Mostly I have been looking at "what would it take for libgoblin to be usefully integrated into asyncio?"

This turns out to be a bit more challenging than it appears at the outset for one reason: mg_globals. mg_globals is a pretty sad design in MediaGoblin that I'd like to get rid of; basically it makes it easy to write functions that don't have to have the database session and friends, template environment and etc passed into them, because those are set on a global variable level. That works (but is nasty) as long as you're not in a multithreaded environment, but breaks as soon as you are. I recently created a ticket reflecting such, suggesting switching over to werkzeug context locals (Flask makes heavy use of this). Werkzeug's hack is clever, using thread locals so that even in a multi-threaded environment, the objects you're accessing are still globals, but they're the right globals.

But Werkzeug's solution is not good enough for integration with asyncio, where you might be doing "asynchronous" behavior in the same thread, suspending and resuming coroutines or coming back to tasks or etc. As such, it's almost guaranteed in this system that you'll be clobbering the variables another task needs.

What to do? I did research to see if anyone had ideas. It looks like you could do such a thing with Task.current_task() in asyncio, and that would be fairly equivalent. I think you'd need careful implementation though... if you're not paying close attention you might not attach the right things to the right subtask, and that whole thing just seems... fragile. But it still is a neat idea to play around with.

But here's some ideas that I think are neat all combined, related to this problem:

• The idea that the request or asyncio task is the main object that you attach useful variables to, and you just pass that thing around as a "universal context" like crazy. (The downside: what happens when you aren't using asyncio, or don't need an http request, like in a migration script?)
• I like the idea of the application being multi-instance'able, and then having requests and a local context as a layer on top of that. So you've got the "instantiated application" layer, and on top of that the "current request" layer, and at both levels you can have variables attached (like the database engine on the application level, but the database session on the request level). That's an awesome distinction.

But you don't really know whether or not some bit of code is using an asyncio task, a web request, or whatever to pass around. Here's the thing though: it doesn't really matter most of the time. With rare exceptions, you're just looking for $OBJECT.db or $OBJECT.templates or something. You just need some kind of object you can tack attributes on to.

So that's my idea in libgoblin/pydraulics: you have an application and you want to do something with it (handle a request, execute a task, etc), you can tack stuff onto that object. So either create a fresh context object to tack stuff onto or just start tacking things onto an object you have!

Currently, this looks like:

# Pydraulics -- Easy Pump API integration into your software.
#
# Copyright (C) 2014 Pydraulics contributors.  See AUTHORS.
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
#
# Large parts of this heavily borrowed from MediaGoblin.
# Copyright (C) 2011-2014 MediaGoblin contributors, see MEDIAGOBLIN-AUTHORS

class PydraulicsApp(object):
    """
    Pydraulics "basic" WSGI application
    """

    # ...

    def gen_context(self, request=None):
        """
        Generate or apply a context.

        If we have a request, use that as the context instead.
        """
        if request is not None:
            c = request
            c.app = self
        else:
            c = Context(self)

        c.template_env = jinja2.Environment(
            loader=self.template_loader, autoescape=True,
            undefined=jinja2.StrictUndefined,
            extensions=[
                'jinja2.ext.autoescape'])

        # Set up database
        c.db = self.Session()

        if request is not None:
            c.url_map = url_map.bind_to_environ(request.environ, server_name=self.netloc)
        else:
            c.url_map = url_map.bind(
                self.netloc,
                # Maybe support this later
                script_name=None)

Anyway, simple enough. Then you have request.db made, or if you've just got a command line script and you need the equivalent of that and you already have your instantiated application, just run application.gen_context(). Thus, for utilities that are working with this application and need a variety of instantiated things (the database and the template engine and so on and so on) it's easy enough to just accept "context" as the first argument of the function, then use context.db and etc. (I've considered using just "c" or "ctx" instead of "context" as the variable name since it seems so common and since it conflicts a bit with template context and friends, though that's not very explicit.) So, this seems good.

At one point I got frustrated with the massive amount of porting to libgoblin I was having to do and thought "I really should probably just use Django or Flask itself." However, I found that neither framework really addresses the asyncio stuff I was dealing with above, and once I got enough ported of libgoblin over, libgoblin-based development is very fast and comfortable.

That said, it took up enough time working through those things where I didn't complete the Pump implementation. That's okay, I've got enough to do what's required on my end from MediaGoblin (and we've got good direction and help on the federation end this upcoming year where the most important thing is that I have a good understanding). I still think pydraulics is a pretty neat idea, and I may finish it, though it'll be back-burner'ed for now.

However, libgoblin is something I'm likely to extract. I'm convinced that MediaGoblin is at a point where it's stable enough to know what works and doesn't about the technical design, so that gives me a good basis to know what to build from here. There are other applications I'd like to build which should mesh nicely with MediaGoblin but which really don't belong as part of MediaGoblin itself, and would be kind of hacky add-ons. Clearly this is not the most important development, but towards the end of the summer as we hopefully get the Python 3 branch merged, I will be looking towards this.

Aside from this, on the "-cation" end of things, I took some time to relax and also reapproach my health. I may have a separate post on that soon.

So, that's that. Overall it was productive, but again, not quite in the ways I was expecting. I feel okay about that though... I wanted to do some hacking and not feel deeply pressured or stressed about it... if that wasn't true, I think the "-cation" part wouldn't have held up. So I feel okay that I wandered a bit, and the other things I worked on / found I think are important anyhow, and have me much better prepared for the year ahead. Not to mention the most important part: I feel pretty refreshed and capable of taking it on!

What's next for the coming week? Well now that this is all over, I'm organizing plans so we can get rewards out the door and do project planning for the year ahead. We've got a lot of promises to fulfill. Better get to it!