#+TITLE: Guix(SD) #+AUTHOR: Christopher Allan Webber #+EMAIL: cwebber@dustycloud.org #+DATE: 2015-09-30 Wed #+OPTIONS: toc:nil ^:nil [[file:./static/guixsd-logo-lighttext.png]] * Introduction ** What's Guix? Not just "Yet Another Package Manager!" https://www.gnu.org/software/guix/ *** Some disclaimers - "technically" alpha, with caveats :) - I'm not a traditional distro / packager hater! (I love Debian; also thanks for the space, Red Hat!) *** A functional package manager "wtf does that mean?" you say! In brief: packages are based "purely" on their dependencies. And same with the dependencies of dependencies! We'll come back to this! *** Based on Nix [[file:./static/nixos-logo.png]] - Nix pioneered most of the ideas of functional package management! - Most of Guix's design follows Nix - But not the same codebase - Only Guix's daemon comes from Nix - They both use "Hydra", a continuous integration system, to build binary packages though - And not the same language: - Nix uses a mishmash of languages, including a special DSL for packaging - Guix uses Guile... for nearly everything *** Free as in Freedom - Default packages are 100% libre. - This is another difference from Nix... - You *could* taint things with nonfree software, but the base will always be pure. Better foundations: easier to taint something than to remove impurities later. *** (Other) features - Atomic: Either everything succeeds, or nothing does. Never have a system broken mid-install again. - Time travel: roll back and forward without pain. - Multiple profiles: - Your company's application A is on Django 2.X whereas application B is stuck on Django 1.9? No problem. - Each user can have their own profiles, separate from the system! You need GCC 5, but your sysadmin only installed GCC 4? No problem. - Can serve as a universal package manager - Can provide an isolated "environment" for hacking... Like a universal "virtualenv", for any language *** "The git of package management"???? This is kind of a stretch! But git might feel familiar, so.. : cwebber@earlgrey:~/devel/sly$ git cat-file -p 6573261753243257c7b168e174d45c9cd40061ee : : tree 3bcdcbeac414887a356b375a4b230f80159efe2f : parent c23f81b59db98d7af4fd91a41c9b366823662083 : author Christopher Allan Webber 1434053151 -0500 : committer David Thompson 1434416170 -0400 : : examples: Add Conway's Game of Life. : : * examples/life.scm: New file. : * examples/Makefile.am (nobase_dist_examples_DATA): Add it. - We refer to commits by their hashes... we'll see more hashes in Guix for similar-ish reasons... - A commit's parents get hashed in as a kind of depenency ... See how git's history is /immutably/ tied to its parents? - But feature-wise, making a new commit doesn't "destroy" the old version. (This metaphor only goes so far, for example, you can easily drop old history and reclaim that space in Guix...) ** What's GuixSD? - Now take Guix, and make it a distro! - Stands for Guix System Distribution ** Motivations *** Has this ever happened to you? - My system failed mid-upgrade and now it's BORKED - Oh no I thought this upgrade was okay but now it isn't - My server is a special snowflake and I have no idea how to reproduce or replace it - This deployment system has a turing tarpit of custom config languages (I've fallen and I can't get out!) *** Personal motivations But maybe most critically for me: - I work on a sizable network freedom project, GNU MediaGoblin - Network freedom is near-impossible current state of deployability - Running MediaGoblin has given me plenty of experience here - See many users start "self hosting", but give up because maintainance burden / anxiety too high - Tired of my project working or not depending on "phase of the moon" - Dependency based failures suddenly not working kills uptake and new contributor enthusiasm - Once had a hackathon, prepped all previous day for it, had users show up, and our dependencies were broken I don't want to live on that kind of planet anymore. So... how can Guix help? * Prereq: Understanding "functional" Or: "I'm not trying to trick you into learning monads, I promise!" ** Understanding pure functions *** A definition of pure functions : =================================================================== : A pure function: : : 1. Given the same arguments, must always produce the same output. : (Can't rely on any data that might change!) : 2. Doesn't cause any observable "side effects". : (Can't change anything that isn't its result!) : =================================================================== Simplified from: https://en.wikipedia.org/wiki/Pure_function (Yes, that's all functional programming means! But what is extrapolated from those requirements is what makes things interesting...) *** Your prescription: no side effects! "Side effects" are not allowed: - It can't do any IO - It can't read or write from the filesystem - No network access - No changing global variables or relying on changing variables - No GUIs, no widgets - Definitely no "random" procedure - No printing or beeping - No mutating data that is accessed elsewhere Coding with side effects is called "imperative programming" "impure" functions are more accurately "procedures" *** Functions vs procedures #+BEGIN_SRC python # A function def greet(name): return "Hello, " + name # A procedure (NOT a function) def greet_printer(name): print("Hello, " + name) #+END_SRC Printing to the screen is a "side effect" *** This is as far from a pure function as you can get #+BEGIN_SRC python def write_site_status_to_file(our_file, config): # mutation! url = config.pop("url") # time-dependent variables! current_time = datetime.datetime.now().isoformat() # File IO! our_file.write("\n--Site status at %s--\n" % current_time) # Network IO! site_status = urlopen(url).read() # More file IO! our_file.write(site_status) #+END_SRC ** A purely functional program can't do anything #+BEGIN_QUOTE ... Functional programs can't really do anything, since they can't have /side effects/. As Simon Peyton Jones, a well-known functional programmer, likes to say, "All you can do without side effects is push a button and watch the box get hot for a while." (Which isn't technically true, since even the box getting hot is a side effect.) -- Land of Lisp, p. 300 #+END_QUOTE ** So... what does it give you? That sounds like a lot of "can't", but functional programming gives you some amazing properties: - GUARANTEED reproducibility. By definition! - Concurrent programming is easy. No mutation means no locking! - Most of the hard bugs vanish (Ever heard of a heisenbug?) - You can "scrub" back and forth in time! Debug game by jumping to 20 seconds ago! (Elm and Sly) - Memoization means caching with /no/ cache invalidation challenges - Optimized "immutable" datastructures are cooooooooool - And much, much more Unfortunately, this isn't a functional programming talk! (I recommend reading Land of Lisp for a great explaination of where functional programming is awesome, applied to game AI!) ** Functional programming strategy *** The main thing: you gotta keep 'em separated : =================================================================== : : Break your program into two parts: : : - The clean part: purely functional! : Make as much of your program into this clean part as possible. : - The dirty part: do side effects here. : Make this as small and "quarantined" as possible. : =================================================================== *** The next step : =================================================================== : : Use the dirty part as scaffolding for the clean part, : this way state is pushed to the fringes of the program : : =================================================================== Think about it! The dirty system state flows into the functional system as an argument, and evalues to a result out back the other end to flow back into the imperative system. * Guix: a functional package manager [[file:./static/guix-logo.png]] ** Understanding pure functional packaging *** The general picture **** What does a pure function look like? [[file:./static/function.png]] **** So what does a pure package function look like? A lot like: [[file:./static/package_function.png]] *** "Pure function" package requirements - Packages are built out of a package definition (function) and take *very precise inputs* - Side effects during package building are forbidden! - No network IO! - No reading or writing outside of package building! - To enforce this, we'll build packages in a chroot that has very few privileges and no network access But wait... isn't package building kinda inherently imperative? Compiling files means reading from files, writing out new files, etc! - Correct observation! However, we can uphold the "pure function" contract as long as imperative code is "confined" to the chroot - Think of the chroot as the memory space of the function. It'll be discarded after the function "evaluates" The result of evaluating this "pure function" is called a "derivation" *** "Pure function" package benefits! - This means a binary version is a "memoized" version - The same inputs on the same architecture to the same package will always yield the same output! - Reproducibility! We can be sure that once something builds, it will always build. - No clobbering previous packages... if we can access their former inputs, we can always reproduce and return to their outputs. ** Exploring a little install *** What does an install look like? #+BEGIN_SRC sh guix package -i extremetuxracer #+END_SRC Now let's run it! #+BEGIN_SRC sh etr #+END_SRC *** Where'd that package go? What does this show us? : which etr Where does that point us to? - What's this =~/.guix-profile/= directory? - What's this =/gnu/store/= directory stuff in there points to? **** The profile A profile is a convenient set of packages / files a user installed, with the files they actually want to use. Let's peek around: : ls ~/.guix-profile/ However we notice that all the profile stuff points to this /gnu/store/ place! **** The store : ls /gnu/store/ This is where things actually are. These are the "outputs" of packages (the derivations!) Some files in there: : f1q8bwa1w41h4p5z2fm2mnm0zl2pmw7x-windowmaker-0.95.6 : f1ww63vzah59mr8v9fh2x0j8vfclhagn-extremetuxracer-0.6.0.drv : f1yi58m83wkpwk6fzlc5pr35ld40n4fi-gnu-ghostscript-9.14.0.tar.xz-builder : f207zb79b7zyh69i85dgfgfvr6c6df1p-autoconf-wrapper-2.69.drv : f20hsaiyhmx006l33h9dmfglfha1fs3h-slim-1.3.6.tar.xz-builder : f251icz2j7qfvyamkvrhd422jib70aq5-bash43-016.drv : f2786zjss1ja3prmqryqyn8bxy18pbvl-guile-wisp-0.8.6-guile-builder : f29hxsj7ry4zakx73abc7qp7b0640sqm-Python-3.4.3.tar.xz.drv : f29wav20kvz9f609vjp4da5ri6cd3sd3-xf86driproto-2.1.1.drv If we hash together the function and all its inputs, we can get a precise identifier. An extremely technically correct version of things: the "store" is "memoized storage" of your packages! (But you could always rebuild them!) ** How deep does it go? But wait, if a function is based off of its inputs... *** A package and its inputs Let's get a graph of coreutils: : guix graph coreutils | dot -Tpng > /tmp/coreutils-dag.png How about Tuxracer? :) : guix graph extremetuxracer | dot -Tpng > /tmp/extremetuxracer-dag.png How about GNU Hello? : guix graph hello | dot -Tpng > /tmp/hello-dag.png Oh wait... that looks really simple... it's just hello! How can that be? *** Implicit inputs The above commands ommitted "implicit inputs"... dependencies that are needed to build, but aren't needed to run. I've already generated the graph for coreutils: : guix graph coreutils --type=bag-emerged | dot -Tpng > ./static/graphs/coreutils-dag-emerged.png A much bigger graph can be generated if you include bootstrapping requirements too :) *** Generations : M-x guix-generations - These are previous instances of your profile - You can roll back to older versions if you need be - Once you don't need generations, delete them! - Reclaim old space with "guix gc" (yup, garbage collection) *** Takeaways - If something low down the chain gets a new verison, all of its dependencies will be rebuilt (inputs are verrrry precise) - This is made manageable: - You can drop old generations - And then garbage collect them! ** So what does a Guix package look like? How about a nice game of [[file:~/devel/guix/gnu/packages/games.scm::(define-public%20chess][chess]]? ** Guix as a universal virtualenv : guix environment ... a super cool command! Ship a guix environment file with your application! Great replacement for virtualenv / gems / jhbuild etc! ** GUI alternatives? *** You meant emacs right? *** guix-web *** .~~@YOUR GUI HERE!@~~. * GuixSD: a functional system distribution [[file:./static/guixsd-logo-lighttext-smaller.png]] Take Guix, and now distro'ify it! ** Note: You don't have to run GuixSD to use Guix, but it's cool! I run Guix on top of Debian! But if you do run GuixSD, there are some cool properties... ** Describe your system and go Declarative systems! Here's a minimalist version. [[file:~/sandbox/guix-vm-minimal-demo.scm]] ** Run it in a VM! On a container! I've got this demo file: [[file:~/sandbox/guix-vm-demo.scm]] Let's boot it up! : $(guix system vm ~/sandbox/guix-vm-demo.scm) ** Services *** Services and DMD Services part of system config defines: - Daemon management - Config management Uses a system called "dmd" to do this, also written in Guile This does mean that Guix does not use Systemd... but not for systemd-hater reasons! There is a big benefit to being able to "compose" a system in the same language as the rest of Guix... (can even programmatically generate a system definition based on other requirements) *** Coming soon: DRY (Don't Repeat Yourself) configuration What happens if you need to deploy MediaGoblin, and.. - Its configuration needs information so it can connect to the Mail Transfer Agent - It needs to provide configuration to nginx or apache? How can you do this without duplicating config information all over? Imagine a web or desktop UI where a user just fills in a few widgets with config stuff, and this can be used to compose the system definition! ** Deployment Coming soon, GuixOps! Guix makes these features easy to build: - Test-before-you-deploy - Rollbacks in case something goes wrong - Copying a "closure" of the system over the wire once it's ready Guix being written in Guile makes this much easier... ** GuixSD and the Filesystem Hierarchy Standard GuixSD obeys the Filesystem Heirarchy Standard, but "not too much" ... most of it goes in a profile, either the user's, or the system profile! Wait, whaaaat? Isn't the FHS holy writ? Consider the advantage: being able to roll your whole system back and forth... including your nginx/apache config and friends * Guile as a killer feature This is where Guix diverges from Nix ** What's Guile? [[file:./static/guile-logo.png]] See: file:./static/new_guile_website.png - Guile is most well known for its Scheme (a type of lisp) implementation - Less known: a full fledged VM (even preliminary Javascript support) - Guix is written in Guile Scheme. Why is this special? - Guix uses a fully fledged language, top to bottom - Scheme has long history and solid language design - Scheme's s-expression syntax is "homoiconic"... you can write "code that writes code" ** How much of Guix is written in Guile? Nearly all of it! - Guix internals - Guix package recipes - Your own system configuration This means things are highly scriptable... ** What can we do here? Since Guix, its packages, and its configuration are all in the same full-fledged language, no limit on what you can write: - A Web UI, why not? - A GNOME UI, why not? - GuixOps: coming soon! - >>Your need here!<< ** Getting over your parenthesis phobia *** Are parentheses ugly? S-expressions bring lots of power But does Lisp stand for: - Lots of Irritating Superfluous Parentheses? - or, Lisp Is Syntactically Beautiful? *** Lisp, minus the parentheses Let's look at an alternative syntax to s-expressions, with Wisp: file:./static/grep.w Now let's look at the same file, with s-expressions: file:./static/grep.scm Does it look so scary now? *** Your editor should help - rainbow delimiters - syntax navigation - smartparens / paredit ** G-expressions are awesome [[info:guix.info#G-Expressions]] Not going to into detail on this, but "write code that writes code" is maximized here Exercise for the audience: how can Guix do code generation of scripts cleanly via this method? ** Do we have to write scheme? No! read/write aspect of scheme means you can have a GUI construct system description and simply serialize it to disk "Code that writes code" rules the school. ** How does this compare against Nix's special language? #+BEGIN_SRC nix {stdenv, fetchurl}: stdenv.mkDerivation { name = "hello-2.6"; src = fetchurl { url = ftp://ftp.gnu.org/gnu/hello/hello-2.6.tar.gz; sha256 = "1h6fjkkwr7kxv0rl5l61ya0b49imzfaspy7jk9jas1fil31sjykl"; }; meta = { homepage = http://www.gnu.org/software/hello/manual/; license = "GPLv3+"; }; } #+END_SRC ** A word against "turing tarpits" Insert rant against yaml, then jinja2 + yaml, etc * Wrapping up ** Comparison to Docker, etc? *** beware of distro-sized binary black boxes - containers are great (and Guix supports them) - distro-sized statically compiled systems are not great *** davexunit's tenth rule With apologies to Greenspun: #+BEGIN_QUOTE All package management/deployment tools contain an ad hoc, informally-specified, bug-ridden, slow implementation of half of Guix. #+END_QUOTE *** blah blah http://sandervanderburg.blogspot.com/2015/04/an-evaluation-and-comparison-of-snappy.html ** Can Guix succeed? *** Guix is "The Right Thing" - Guix has solid fundamentals - A likely foundation for solving the deployment crisis *** ... but "right thing" != success **** "Worse is Better"? - The Lisp world got so much right we're still struggling to extract good ideas from that mine - Why was so much lost for so long? - "Worse is Better"? A mostly technical analysis... https://www.jwz.org/doc/worse-is-better.html **** But maybe it's cultural [[http://this-plt-life.tumblr.com/post/66298485729/when-im-being-propagandized-by-an-fp-weenie][Smugly being right]] is not a path to success! **** Worse is More Accessible? - We need to make it easy to get started - We need to lower the intimidation factor - We need to be active on outreach **** The community we could have ***** A tower for wizards? [[file:./static/sicp_wizard.jpg]] # PS: (I love wizards and SICP!) ***** Or a playground for everyone? [[file:./static/new_guile_website-crop.png]] ** Image attribution - Guix logo by - Thanks to Ricardo Wurmus for the excellent function / package images! ** Thanks for coming! If you're excited, we could use your help! - Join #guix on irc.freenode.net - Become a user! - Become a contributor!