Personal python egg baskets / package repositories

At Creative Commons sometimes we have python packages that we use a lot but which aren't generally useful enough to put on PyPi (such as cc.license and cc.licenserdf). Luckily, as long as you and your organization don't have a problem with having your packages available publicly, it's fairly easy to put up a public egg basket, though not everyone seems to know how to do it. And doing so has several advantages:

• You can avoid cluttering up PyPi with packages that have a very marginal or internal audience. cc.licenserdf matters a lot for our site, but probably is worthless to anyone searching for "RDF" tools on pypi.
• Putting a package up on PyPi suggests a certain amount of responsibility to keep that updated and keep its API stable. If it's for internal use, maybe that responsibility is unwanted or unneeded.
• Using proper python packaging means you can take full advantage of python's ecosystem of installation tools, including useful dependency resolution.
• Making packaged releases encourages a certain amount of responsibility toward your internal dependencies, reducing the "cowboy coding" factor (fun, but often not good in an environment that requires stability).

So, how to do it? The first step is to make a directory that's static served by Apache. Ours is at http://code.creativecommons.org/basket/. The directory has the sticky bit set so that everyone on the server can write to it without clobbering each other, kind of like in /tmp/ on most GNU/Linux installs. Everyone puts their tarballs up here. As for you or your organization's "personal" packages, I think it's fine to put all your eggs in just one basket like this. :)

But what should those packages look like?

Generally at the toplevel of your package you have something that looks like:

from setuptools import setup, find_packages
import sys, os

setup(
    name='licenseclarity',
    version='0.1',
    packages=['licenseclarity'],
    description="Tools to clarify any licensing issue you have ever had",

    author='John Doe(ig)',
    author_email='johndoe-ig@example.org',
    license='MIT',

    # ... etc ...

    # Put your dependencies here...
    install_requires=[
        'setuptools',
        'dependency1',
        'dependency2',
        ],

    # And a link to your basket here
    dependency_links=[
        'http://code.creativecommons.org/basket/',
        ],
    )

Obviously, replace values with ones that make sense for your package. The main things we're talking about here are install_requires and dependency_links:

# Put your dependencies here...
install_requires=[
    'setuptools',
    'dependency1',
    'dependency2',
    ],

# And a link to your basket here
dependency_links=[
    'http://code.creativecommons.org/basket/',
    ],

Replace install_requires with whatever list of dependencies your package requires, and put your personal egg basket in the dependency_links list. After that, I guess set up whatever other large number of attributes you can apply to the setup function that make sense to your package (entry_points maybe? zip_safe=False is a nice one).

Lastly, maybe now that you have an egg basket, you'd like to know how to make some eggs to put in that basket. Pretty easy once you have setup.py! All you really need to do is this:

python setup.py sdist

Assuming things run well, do an `ls` into your dist/ directory. Hey, what do you know! There's a tarball there. cp or scp or (heaven forbid) ftp it to your egg basket, and there you go. You just made a release! Maybe the next time you do it you'll want to increment the version number.

Lastly I'd like to make a comment on only using buildout's config file or pip's requirements file to declare requirements: it's completely crazy, don't do it! Well okay, it's not completely crazy, but you really should fill out the install_requires section in setup.py. There are a couple of reasons for this:

• Recursive dependencies: maybe someone will use your module as a library, and then they'll have to declare your dependencies all over again if you don't put them in install_requires. Let python's package management tools handle the dependency graph logic, that's where it belongs.
• What happens if you build your tool in buildout and someone wants to use virtualenv/pip or vice versa? Both of these tools will check the install_requires section anyway, so be nice and fill it out!

Of course there are some awesome things that buildout and the pip requirements_file can do to; for example they can install dependencies from VCS and etc which is useful in particular cases, particularly for certain in-development packages. By all means, use these tools to do that (and lots of other cool things, because both buildout and virtualenv/pip are pretty awesome). Just be a good packager and also fill out install_requires.

Now you have your own egg basket (which is completely vegan), you know how to make packages to put in it, and everyone is happy. Horray! Today we learned things (maybe).