Hello everyone! Time for a bit of a diversion towards one of my favorite projects, Hy! (I'm an occasional committer, but the main mastermind behind the project is my good friend Paul Tagliamonte.) For those of you who don't know, Hy is a Lisp that transforms into the Python AST. Even more fun: you can import .hy files in .py files and .py files in .hy files! Crazy!

Now, when many people hear that, they say, "Huh what, why on earth would you do such a thing?" The usual response is something like, "Because it's fun!" But today, dear readers, I am going to show you a real... dare I say practical reason for using Hy. Because a cool feature just landed in Hy: a backport of "yield from" to Python 2.

Let's back up a bit. First, you might not know what "yield from" is or why it's cool. Well, Python has this thing called coroutines which allow you to do cool things, including suspending and resuming functions, which it turns out is really great for writing asynchronous code (sure, just wake me back up when we get the next network stanza, eh buddy?). Once you provide the ability to nest together coroutines by "delegating to subgenerators" (what "yield from" does that "yield" does not), this stuff starts to get really powerful. This feature is so useful that it's the basis of maybe the world's coolest asynchronous programming environment, asyncio. Only one problem: "yield from" didn't exist until Python 3.3... which means you can't use it with Python 2. Bummer!

Or can you? Time for our second bit of context. Ever hear of a Lisp programmer talk about something called a "macro"? No? Okay, think harder. Maybe it was in one of those conversations where you were talking about your favorite new your-pet-language feature, and the Lisp hacker was like, "Oh that's cute... yeah Lisp had that decades ago." And then you got really mad and brought up a bunch more features, and the Lisp hacker kept saying that Lisp had them before you were born, and "Oh yeah, and whatever features Lisp doesn't have, you can add really fast because Lisp has macros. You can basically program any feature with macros." Maybe you asked them, what the heck is a macro, and they said something inane like "it's a feature where you can program other features, or write code that writes code", but you barely remember, because at that point you just wanted to punch them in their smug little face. (And besides, you wondered, if you have higher order functions, isn't that enough?)

Well friends, today it is my face that you will want to do the punching to, because I'm about to show you how cool macros are and why having them makes Hy so awesome. But, after the face punching thing, you'll also thank me. (Also, please don't punch me in the face, this blogpost is not consent for face-punching.)

Enough with the talk. Time for examples! Let's look at some code. Say you have this Python 3.3+ code:

from awesomelib import IrcBot, bake_cookie, async

def irc_to_cookies(**connection_stuff):
    our_bot = IrcBot(**connection_stuff)
    yield from our_bot.open_connection()

    while True:
        message = yield from our_bot.get_next_message()

        if message.command == "bake_cookie":
            yield from async(bake_cookie())

Groovy. In our example above, we built a cookie baker that can be plugged into our awesomelib asynchronous network library and cookie baking pipeline system. (And of course, we wrote it in Hy, because we love Hy, and you can still run Hy code in vanilla Python.) We're feeling pretty good about this. We kind of wish we could run it in Python 2.X still, but Python 3.3+ is the future anyway, and no use worrying about the past really... right?

Our Hy example looks pretty similar:

(defn irc-to-cookies [&kwargs connection-stuff]
  (setv our-bot (apply IrcBot [] connection-stuff))
  (yield-from (.open-connection our-bot))

  (while True
    (setv message (yield-from (.get-next-message our-bot)))

    ;; If an irc user gives the "bake_cookie" command,
    ;; put a cookie in our ez_bake oven
    (if (= message.command "bake_cookie")
      (yield-from (async (bake-cookie))))))

But there's something magical... this code makes use of yield-from, which in Python 3.3+ Hy just uses the actual real built in "yield from" (or more accurately, ast.YieldFrom). But what about Python 2? Mere higher ordered function magic can't save us here. We need a way to implement a new feature.

Except oh right, this is a lisp, and we have macros! So why not write a macro for yield-from?

And it turns out that's exactly what paultag did:

(if-python2
  (defmacro/g! yield-from [expr]
    `(do (import types)
         (setv ~g!iter (iter ~expr))
         (setv ~g!return nil)
         (setv ~g!message nil)
         (while true
           (try (if (isinstance ~g!iter types.GeneratorType)
                  (setv ~g!message (yield (.send ~g!iter ~g!message)))
                  (setv ~g!message (yield (next ~g!iter))))
           (catch [~g!e StopIteration]
             (do (setv ~g!return (if (hasattr ~g!e "value")
                                     (. ~g!e value)
                                     nil))
               (break)))))
           ~g!return))
  nil)

This simple macro above is an implementation of yield-from which works in Python 2. The macro is more or less a function that writes new code to be expanded in place... allowing us to use basic building blocks of the language to build more complex features. Since in Lisp, code is a very simple, manipulatable data structure (lists!), we can literally write out code that writes code without too much trouble. (There's some magic going on with the ` character above, called backquoting... but it's best to read a tutorial on macros if it's not clear to you how the backquote is building the list of code there.) Hey look... we just brought a feature back to the future... as long as we're writing code in Hy, we can do subgenerator delegation with yield-from. Cool! That sure makes coroutines a lot more useful to those of us living in the past.

So wait, does this mean you can now use asyncio with Python 2? Well, not quite... asyncio is written in normal Python syntax, which means that it's using "yield from", not our more versatile "yield-from", and the library itself isn't written to support Python 2.7. So, no. (But, if asyncio was written in Hy, we could, even though Python 2.7 doesn't have "yield from"!)

The real goal of this article isn't to convince you to start backporting features to Python 2.X via Hy, though. Really, Python 3 is the future, write Python 3 code! But the point here is to get you thinking about how having macros allows you to implement new features now! Why wait for the features of Python 4.X? In Hy you can have them now! (Or even start prototyping them today!)

And that's worth getting excited about. And once you realize that, it's a bit easier to understand where lispers are coming from when they nerd out about how cool macros are. (Even if you still want to punch us in the face.)

(PS: think this is pretty cool? Hy is a really welcoming community, and there's a lot of fun stuff to do! Learn about language implementation and learn about lisp in a paradoxically fun and pythonic environment! We'd love to have you join in hacking with us!)