Posts with tag "guix"

Chicago GNU/Linux talk on Guix retrospective

By Christopher Lemmer Webber on Thu 01 October 2015

GuixSD logo

Friends... friends! I gave a talk on Guix last night in Chicago, and it went amazingly well. That feels like an undersell actually; it went remarkably well. There were 25 people, and apparently there was quite the waitlist, but I was really happy with the set of people who were in the room. I haven't talked about Guix in front of an audience before and I was afraid it would be a dud, but it's hard to explain the reaction I got. It felt like there was a general consensus in the room: Guix is taking the right approach to things.

I didn't expect this! I know some extremely solid people who were in the audience, and some of them are working with other deployment technologies, so I expected at some point to be told "you are wrong", but that moment didn't come. Instead, I met a large amount of enthusiasm for the subject, a lot of curious questions, and well... there was some criticism of the talk, though it mostly came to presentation style and approach. Also, I had promised to give two talks, both about federation and about Guix, but instead I just gave the latter and splatted over the latter's time. Though people seemed to enjoy themselves enough that I was asked to come back again and give the federation talk as well.

Before coming to this talk, I had wondered whether I had gone totally off the yaks in heading in this direction, but giving this talk was worth it in at least that the community reaction has been a huge confidence booster. It's worth persuing!

So, here are some things that came out of the talk for me:

  • The talk was clear, and generally people said that though I went into the subject to quite some depth, things were well understood and unambiguous to them.
  • This is important, because it means that once people understood the details of what I was saying, it gave them a better opportunity to evaluate for whether it was true or not... and so the general sense of the room that this is the right approach was reassuring.
  • A two tier strategy for pushing early adoption with "practical developers" probably makes sense:
    • Developers seem really excited about the "universal virtualenv" aspect of Guix (using "guix environment") and this is probably a good feature to start gaining adoption.
    • Getting GuixOps working and solidly demonstrable
  • The talk was too long. I think everything I said was useful, but I literally filled two talk slots. There are some obvious things that can be cut or reduced from the talk.
  • In a certain sense, this is also because the talk was not one, but multiple talks. Each of these items could be cut to a brief slide or two and then expanded into its own talk:
    • An intro to functional programming. I'm glad to see this this intro was very clear, and though concise, could be reduced within the scope of this talk to two quick slides rather than four with code examples.
    • An "Intro to Guile"
    • Lisp history, community, and its need for outreach and diversity
    • "Getting over your parenthesis phobia"
  • I simply unfolded an orgmode tree while presenting the talk, and while this made things easy on me, it's not very interesting for most audience members (though my friend Aeva clearly enjoyed it)

Additionally, upon hearing my talk, my friend Karl Fogel seemed quite convinced about Guix's direction (and there's few people whose analysis I'd rate higher). He observed that Guix's fundamentals seem solid, but that what it probably needs is institutional adoption at this point to bring it to the next level, and he's probably right. He also pointed out that it's not too much for an organization to invest themselves in Guix at this point, considering that developers are using way less stable software than Guix to do deployments. He suggested I try to give this talk at various companies, which could be interesting... well, maybe you'll hear more about this in the future. Maybe as a trial run I should submit some podcast episodes to Hacker Public Radio or something!

Anyway, starting next week I'm putting my words to action and working on doing actual deployments using Guix. Now that'll be interesting to write about! So stay tuned, I guess!

PS: You can download the orgmode file of the talk or peruse the html rendered version or even better check out my talks repo!

Let's Package jQuery: A Javascript Packaging Dystopian Novella

By Christopher Lemmer Webber on Fri 01 May 2015

The state of packaging of libre web applications is, let's face it, a sad one. I say this as one of the lead authors of one of such a libre web application myself. It's just one component of why deploying libre web applications is also such a sad state of affairs (hence userops). It doesn't help that, for a long time, the status quo in all free software web applications (and indeed all web applications) was to check javascript and similar served-to-client web assets straight into your repository. This is as bad as it sounds, and leads to an even further disconnect (one of many) between the packages that a truly free distro might include (and have to manually link in after the fact) and those of your own package. Your package is likely to become stuck on a totally old version of things, and that's no good.

So, in an effort to improve things, MediaGoblin and many other projects have kicked the bad habit of including such assets directly in our repository. Unfortunately, the route we are taking to do this in the next release is to make use of npm and bower. I really did not want to do this... our docs already include instructions to use Python's packaging ecosystem and virtualenv, which is fine for development, but since we don't have proper system packaging, this means that this is the route users go for deployment as well. Which I guess would be fine, except that my experience is that language package managers break all the time, and when they break, they generally require an expert in that language to get you out of whatever mess you're in. So we added more language package management features... not so great. Now users are even more likely to hit language package management problems, now also ones that our community are less of experts in helping debug.

But what can we do? I originally thought of home-rolling our own solution, but as others rightly pointed out, this would be inventing our own package manager. So, we're sucking it up and going the npm/bower route.

But wait! There may be a way out... recently I've been playing with Guix quite a bit, and I came to realize that, at least for myself in development, it could be nice to have all the advantages of transactional rollbacks and etc. There is a really nice feature in Guix called guix environment which is akin to a "universal virtualenv" (also similar to JHBuild in Gnome land, but not tied to Gnome specifically)... it can give you an isolated environment for hacking, except not just restricted to Python or Javascript or Ruby or C... great! (Nix has something similar called nix-shell.) I know that I can't expect that Guix is usable for everyone right now, but for many, maybe this could be a nice replacement for Virtualenv + Bower, something I wrote to the mailing list about.

(As an aside, the challenge wasn't the "virtualenv" type side of things (pulling in all the server-side dependencies)... that's easy. The challenge is replacing the Bower part: how to link in all the statically-served assets from the Guix store right into the package? It's kind of a dynamic linking problem, but for various reasons, linking things into the package you're working on is not really easy to do in a functional packaging environment. But thanks to Ludo's advice and thanks to g-expressions, things are working!)

I'm happy to say that today, thanks to the help from the list, I came up with such a Virtualenv + Bower replacement prototype using "guix environment". And of course I wanted to test this on MediaGoblin. So here I thought, well, how about just for tonight I test on something simple. How about jQuery? How hard could that be? I mean, it just compiles down to one file, jquery.js. (Well, two... there's also jquery.js.min...)

Luckily, Guix has Node, so it has npm. Okay, the docs say to do the following:

# Enter the jquery directory and run the build script:
cd jquery && npm run build

Okay, it takes a while... but it worked! That seemed surprisingly easy. Hm, maybe too easy. Remember that I'm building a package for a purely functional distribution: we can't have any side effects like fetching packages from the web, every package used has to be an input and also packaged for Guix. We need dependencies all the way up the tree. So let's see, are there any dependencies? There seems to be a node_modules directory... let's check that:

cwebber@earlgrey:~/programs/jquery$ ls node_modules/
commitplease          grunt-contrib-uglify  grunt-npmcopy        npm                   sinon
grunt                 grunt-contrib-watch   gzip-js              promises-aplus-tests  sizzle
grunt-cli             grunt-git-authors     jsdom                q                     testswarm
grunt-compare-size    grunt-jscs-checker    load-grunt-tasks     qunitjs               win-spawn
grunt-contrib-jshint  grunt-jsonlint        native-promise-only  requirejs

Yikes. Okay, that's 24 dependencies... that'll be a long night, but we can do it.

Except, wait... I mean, there's nothing so crazy here as in dependencies having dependencies, is there? Let's check:

cwebber@earlgrey:~/programs/jquery$ ls node_modules/grunt/node_modules/
async          eventemitter2  glob               iconv-lite  nopt
coffee-script  exit           grunt-legacy-log   js-yaml     rimraf
colors         findup-sync    grunt-legacy-util  lodash      underscore.string
dateformat     getobject      hooker             minimatch   which

Oh hell no. Okay, jeez, just how many of these node_modules directories are there? Luckily, it's not so hard to check (apologies for the hacky bash pipes which are to follow):

cwebber@earlgrey:~/programs/jquery$ find node_modules -name "node_modules" | wc -l

Okay, yikes. There are 158 dependency directories that were pulled down recursively. Wha?? To look at the list is to look at madness. Okay, how many unique packages are in there? Let's see:

cwebber@earlgrey:~/programs/jquery$ find node_modules -name "node_modules" -exec ls -l {} \; | grep -v total | awk '{print $9}' | sort | uniq | wc -l

No. Way. 265 unique packages (the list in its full glory), all to build jquery! But wait... there were 158 node_modules directories... each one of these could have its own repeat of say, the minimatch package. How many non-unique copies are there? Again, easy to check:

cwebber@earlgrey:~/programs/jquery$ find node_modules -name "node_modules" -exec ls -l {} \; | grep -v total | awk '{print $9}' | wc -l

So, there's about double-duplication of all these packages here. Hrm... (Update: I have been told that there is an npm dedupe feature. I don't think this reduces the onerousness of packaging outside of npm, but I'm glad to hear it has this feature!)

Well, there is no way I am compiling jQuery and all its dependencies in this state any time soon. Which makes me wonder, how does Debian do it? The answer seems to be, currently just ship a really old version from back in the day before npm, when you could just use a simple Makefile.

Well for that matter then, how does Nix do it? They're also a functional package management system, and perhaps Guix can take inspiration there as Guix has in so many other places. Unfortunately, Nix just downloads the prebuilt binary and installs that, which in the world of functional package management is kind of like saying "fuck it, I'm out."

And let's face it, "fuck it, I'm out" seems to be the mantra of web application packaging these days. Our deployment and build setups have gotten so complicated that I doubt anyone really has a decent understanding of what is going on, really. Who is to blame? Is it conventional distributions, for being so behind the times and for not providing nice per-user packaging environments for development? Is it web developers, for going their own direction, for not learning from the past, for just checking things in and getting going because the boss is leaning over your shoulder and oh god virtualenv is breaking again on the server since we did an upgrade and I just have to make this work this time? Whose fault is it? Maybe pinning blame is not really the best anyway, but I feel that these are conversations that we should have been having, for distributions and web applications to work together, at least a decade ago. And it's not just Javascript; we're hardly better in the Python world. But it's no wonder that the most popular direction of deployment is the equivalent of rolling a whole distro up into a static binary, and I don't have to tell you what a sad state that is.

For me, at the moment, I'd like to be more conscious of what it takes to build software, not less. Reproducibility is key to long-term software freedom, else how can we be sure that the software we're running is really the software we say it is? But given all the above, it's hard to not have empathy for those who instead decide to toss in that towel and take a "fuck it, I'm out" approach to deployment.

But I hope we can do better. In the meanwhile, ensuring that users can actually build and package from top to bottom the software I'm encouraging them to use is becoming more of a priority for me, not less.

And I guess that may mean, if it isn't really feasible to reproduce your software, I can't depend on it in my own.

Why is it hard to move from one machine to another? An analysis. [x-post from Userops]

By Christopher Lemmer Webber on Wed 08 April 2015

NOTE: This is a shameless cross-post of something I originally sent to the userops list, where we discuss deployment things.

Hello all,

For a while I've been considering, why is it so harder for me to migrate from server to server than it is for me to migrate from desktop to desktop? For years, ever since I discovered rsync, migrating between machines has not been hard. I simply rsync my home directory over to the new machine (or maybe even just keep the old /home/ directory's partition where it is!) and bam, I am done. Backing this up is easy; it's just another rsync away. (I use dirvish as a simple wrapper around rsync so it can manage incremental backups.)

If I set up a new machine, it is no worry. Even if my current machine dies, it is mostly no worry. Rsync back my home directory, and done. I will spend a week or so discovering that certain programs I rely on are not there, and I'll install them one by one. In a way it's refreshing: I can install the programs I need, and the old cruft is gone!

This is not true for servers. At the back of my mind I realized this, but until the end of Stefano Zacchiroli's excellent LibrePlanet talk when I posed a question surrounding this situation, I hadn't totally congealed in my head: why is it so much harder for me to move from server to server? Assume I even have the old server around and I want to move. It isn't easy!

So here are some thoughts that come out of this:

  • For my user on my workstation, configuration and data are in the same place: /home/ (including lots of little dotfiles for the configs, and the rest is mostly data). Sure, there's some configuration stuff in /etc/ and data in /var/ but it mostly doesn't really matter, and copying that between machines is not hard.
  • Similarly, for my user on workstation experience, it is very little stress if I set up a machine and am missing some common packages. I can just install them again as I find them missing.
  • Neither of these are true for my server! In addition to caring about /home/, and even more importantly, I have to worry about configuration in /etc/ and data in /var/. These are both pains in the butt for me for different reasons.
  • Lots of stuff in /etc/ is configuration that interacts with the rest of the system in specific ways. I could rsync it to a new machine, but I feel like that's just blindly copying over stuff where I really need to know how it works and how it was set up with the rest of the machine in the first place.
  • This is compounded by the fact that people rarely set up one machine these days; usually they have to set up several machines for several users. Remembering how all that stuff worked is hard. The only solution seems to be to have some sort of reproducible configuration system. Hence the rise of salt, ansible, etc. But these aren't really "userops" systems, they're "devops"... developer focused. Not only do you need to know how they work, you need to know how the rest of the system works. And it's not easy to share that knowledge.
  • /var/ is another matter. Theoretically, most of my program data goes there (unless, of course, it went to /srv/, god help us). But I can't just rsync it! There are some processes that are very persnickety about the stuff there. I have to dump my databases and etc before I can move them or back up. Nothing sets up an automatic cronjob for me on these, I have to know to dump postgres. Hopefully I set up a cronjob!
  • While I as a workstation user don't stress too much if I'm missing some packages (just install them as I go), that is NOT true of my servers. If my mail servers aren't running, if jabber isn't on, (if SSH isn't running!!!), there are other servers expecting to communicate with my machine, and if I don't set them up, I miss out.
  • Not only this, assuming I have moved between servers correctly, even once I have set up my machine and it has become a perfectly okay running special snowflake, there are certain routine tasks that require a lot of manual intervention, and I have only picked up the right steps by knowing the right friends, having run across the right tutorials which hopefully have shown me the right setup, etc. SSL configuration, I'm looking at you; the only savior that I have is that I have written myself my own little orgmode notes on what to do the next time my certs expire.
  • My servers do become special snowflakes, and that is very stressful to me. I will, in the future, need to set up one more server, and remembering what I did in the past will be very hard.
  • Assuming I use all the mainstream tools, not talking about "upcoming" ones, a better configuration management solution is probably the answer, right? That's a lot to ask users though: it's not a solution to existing deployment, because it doesn't remove the need to know about all the layers underneath, it just adds a new layer to understand.

Those are all headaches, and they are not the only headaches. But here are some thoughts on things that can help:

  • If I recognize which parts of my system are "immutable" and which parts are "mutable", it's easier to frame how my system works.

    • /var/ is mutable, it's data. There's no making this "reproducible" really: it needs to be backed up and moved around.
    • My packages and system are immutable, or mostly should be. Even if not using a perfectly immutable system like guix/nix, it's helpful to act like this part of the system is pseudo-immutable, and simply derived from some listing of things I said I wanted installed. (Nix/Guix probably do this the most nicely though.)
    • /etc/ is similarly "immutable but derived" in the best case. I should be able to give the same system configuration inputs and always get the same system of packages and configuration files.
  • I like Guix/Nix, but my usage of Debian and Fedora and friends is not going away anytime soon. Nonetheless, configuration management systems like puppet/ansible/salt help give the illusion of an immutable system derived from a set of inputs, even though they are working within a mutable one.

  • Language packaging for deployment needs to die. Yes, I say this as a project that advocates that very route. We're doing it wrong, and I want to change it. (Language packaging for development though: that's great!)

  • Asking people to use systems like ansible/salt/puppet is asking users too much. You're just asking them to learn one more layer on top of knowing how the whole system works. Sharing common code is mostly copy and paste. There are some layers built on top of here to mitigate this but afaict they aren't really good, not good enough. (I am working on something to solve this...)

  • Pre-built containers are not the solution. Sorry container people! Containers can be really useful but only if they are built in some reproducible way. But very few people using Docker and etc seem to be doing this. But here's another thing: Docker and friends contain their own deployment domain specific languages, which is dumb. If a reproducible configuration system is good enough, it should be good enough for a VM or a container or a vanilla server or a desktop. So maybe we can use containers as lightweight and even sandboxed VMs, but we shouldn't be installing prebuilt containers on our servers alone as a system.

    Otherwise else you're running 80 heavy and expensive Docker images that slowly go out of date... now you're not maintaining 1 distribution install, you're maintaining 81 of them. Yikes! Good luck with the next Shellshock!

  • Before Asheesh jumps in here: yes I will say that Sandstorm is taking maybe the best route as in terms of a system that uses containers heavily (and unlike Docker, they seem actually sandboxed) in that it seems to have a separation between mutable parts and immutable parts: the container is more or less an immutable machine from what I can tell that has /var/ mounted into it, which is a pretty good route.

    In this sense I think Sandstorm has a good picture of things. There are other things that I am still very unsure about, and Asheesh knows because I have expressed them to him (I sure hope that iframe thing goes away, and that daemons like Celery can run, and etc!) but at least in this sense, Sandstorm's container story is more sane.

So there are some reflections in case you are planning on debugging why these things are hard.

-- Chris

PS: If you haven't gotten the sense, the direction I'm thinking of is more along the lines of Guix becoming our Glorious Future (TM) assuming something like GuixOps can happen (go Dave Thompson, go Guix crew!) and a web UI can be built on top of it with some sort of common recipe system.

But I don't think our imperative systems like Debian are going away anytime soon; I certainly don't intend to move all my stuff over to Guix at this time. For that reason, I think there needs to be another program to fit the middle ground: something like salt/ansible/puppet, but with less insane one-off domain specific languages, with a sharable recipe system, and scalable both from developer-oriented scripts but also having a user-friendly web interface. I've begun working on this tool, and it's called Opstimal. Expect to hear more about it soon.

Guix package manager without "make install"

By Christopher Lemmer Webber on Sun 08 February 2015

I've been interested in Guix, the GNU functional package manager based on Guile and drawing much from Nix (though, I think, as a complete system, Guix is much more interesting... maybe a topic of its own post). After a great FOSDEM 2015, where amongst other things I got to speak to and hear from the super nice Guix developers (and even communicated much about the challenges with deploying libre webapps, a topic of one of the presentations I gave), I decided to finally install the thing.

Trouble is, if I'm mostly interested in playing with the package manager (there's now a complete Guix distro called the "Guix System Distribution", but it's a bit too alpha yet, and I'd like to play around in the system first anyhow), and I run Debian... but Guix isn't yet packaged for Debian (though I would like it to be!) But I hate... hate hate hate... doing a make install of a package to my system. make uninstall is so unreliable, and I like to keep my system as a combination of system packages (so, stuff through apt/dpkg) and stuff I've built but keep in my user's home directory. (Though, adding Guix to the mix now adds a third middle ground!) Plus, what if I want to hack on Guix, it seems like being able to have Guix use the code straight from my dev checkout is best, right?

Dave Thompson suggested that I do what he do: run Guix on top of Debian without a make install, straight from the git repo. This was indeed what I wanted to do, but it took me a while to figure out the exact process to get that to work (especially in getting the emacs mode to work with this), so I'm documenting that here... maybe it can be helpful to someone else as well?

So we're mostly going to follow the install docs up until the point where we do a make install, where we obviously won't. So:

# git clone the repo and cd into it
$ git clone git://
$ cd guix

# Install appropriate dependencies... insert your distro package manager here
# Not actually sure if -dev needed on libgcrypt20 or not :)
$ sudo apt-get install guile-2.0 libgcrypt20 libgcrypt20-dev build-essential \
    guile-gnutls guile-2.0-dev libsqlite3-dev pkg-config

# Build the package
$ ./bootstrap && ./configure && make

# Make the "worker users" and their group... this allows the daemon
# to offload package building while keeping things nicely contained
$ sudo groupadd guix-builder
$ for i in `seq 1 10`; do
    sudo useradd -g guix-builder -G guix-builder           \
                 -d /var/empty -s `sudo which nologin`          \
                 -c "Guix build user $i" --system          \

# Make the /gnu/store directory, where packages are kept/built
$ sudo mkdir -p /gnu/store
$ sudo chgrp guix-builder /gnu/store
$ sudo chmod 1775 /gnu/store

(Supposedly there's a way to run things without the extra users and /gnu/store, but you lose a lot of benefits... personally it seems to me that having the store and the daemon is worth it.)

Okay, you're done! Now you should be able to run guix straight from your checkout. But you can't run it directly, you need to use the ./pre-inst-env script which has many environment variables set... after all, we aren't doing a make install, but guix needs to know where all our stuff is! First let's start the daemon... it runs as root, but don't worry, all package building happens as the guix-builder users we set up above:

$ sudo ./pre-inst-env guix-daemon --build-users-group=guix-builder

So, maybe leave that running in a screen session, or daemonize it, or something, but leave it running. For now we're just testing, so whatever.

This is optional, but you may want to "authorize" the Guix substitutes repository. (Thanks to mthl in #guix on freenode for pointing this out!) Otherwise, Guix will spend a lot of time rebuilding things itself that have already been built elsewhere:

$ sudo ./pre-inst-env guix archive --authorize <

Now we can actually run guix! (Maybe from another terminal?):

# a bunch of stuff should run by... but the package will install
$ ./pre-inst-env guix package -i hello

$ ~/.guix-profile/bin/hello
Hello, world!

Great! It ran. But okay... all three of the last commands were annoying. We probably want a way to launch these much more easily, so let's add aliases for guix-daemon and guix, as well as putting our user's guix profile on our PATH. (Yes, that's right... different users can have different packages installed with guix pretty easily.) So, let's edit our ~/.bashrc and add the following lines:

function guix-enable() {
    # Guix stuff

    alias guix="~/devel/guix/pre-inst-env guix"
    alias guix-daemon="sudo ~/devel/guix/pre-inst-env guix-daemon --build-users-group=guix-builder"

    # add guix's bin to the path
    export PATH=$HOME/.guix-profile/bin:$PATH
    # and others
    export PYTHONPATH="$HOME/.guix-profile/lib/python3.4/site-packages"
    export GUILE_LOAD_PATH="$GUILE_LOAD_PATH:$HOME/.guix-profile/share/guile/site/2.0/"
    export GUILE_LOAD_COMPILED_PATH="$GUILE_LOAD_PATH:$HOME/.guix-profile/share/guile/site/2.0/"

You can source ~/.bashrc if you have existing terminals open, or start new terminals / bash sessions... whatever. :) Anyway, it should now be much easier to run things:

$ guix-enable   # run this whenever you want to play with guix things
$ guix package -i xeyes
$ xeyes

Okay, great! So, now we have a reasonable hacking setup, right?

Well, assuming you want to be using Guix as the emacs of distros, you probably want to take advantage of Guix's great emacs integration.

Unfortunately, I found the docs on this subject frustratingly did not work with the method used above. Luckily, I got some help on IRC, and with the following additions to my ~/.emacs, things work:

(add-to-list 'load-path "/home/cwebber/devel/guix/emacs")

(setq guix-guile-program '("/home/cwebber/devel/guix/pre-inst-env" "guile"))

(setq guix-load-path "/home/cwebber/devel/guix/emacs")

(require 'guix-init)

Obviously, change directory names as appropriate.

Now M-x guix-all-available-packages should work!

Hopefully I'll have more to say on why I think Guix is pretty interesting, but at the very least, it might be compelling to hear that Guix can be used as a sort of language-agnostic virtualenv. Pretty cool!

Hopefully that helps someone else out there!