Last summer, I went on a beach vacation with my family, and I needed something to do when we weren’t doing family things. My primary software skills are in the Python and Django world, and I decided that I want to something very different to stretch my brain in new ways. On a whim, I thought it might be fun to make a web framework in Lua.
Lua is a very compact, interpretted language that originates from a group of academics at a university in Brazil. Lua itself is wildly popular as an embedded language and can be found in all sorts of games and other tools. Lua also has a reputation for being an extremely fast interpretted language.
I played with Lua a few years ago and produce a Pong clone using the LÖVE 2D game engine. With my experience building that game, I recall that my tests finished in a blindingly fast amount of time. This memory is probably what led to my “random” choice when picking to build a Lua web framework.
My thinking for this project is something like this:
- Learning to make all the pieces of a web framework could be a lot of fun.
- If something fast and useful comes out the other side, great!
My first foray into this project started with a template rendering engine. Rather than go with regex pattern matching, I decided to go into hard mode and pick a PEG parser. I spent my beach vacation reading white papers and twisting my mind to understand how this kind of parser works.
After a few months of dabbling with the rendering engine, I felt like I was really limiting what I was able to see. With the engine, I reached a level where I will have to implement much of the expression grammar needed by Lua in order to parse a template properly and produce a render function that will perform its task properly. That part of the project quickly got into the weeds, so I made the call to pull back and think on different areas of the project.
Eventually, I remembered The Pragmatic Programmer and the discussion of tracer bullets. I needed something that can serve a small slice of functionality that can go all the way through the system. Building up this minimal working web framework should allow me to adapt to the direction I want to go. This is the idea behind a tracer bullet. Tracer bullets are used to find a target quickly by revealing where a weapon is aiming in a visible way and making adjustments after that to reach the target.
If I can build a crude web framework that reaches the target somewhat, i.e. responding to a browser request, then I can fill in the details after that minimal point.
In researching the Lua ecosystem, I found nothing like the WSGI/ASGI approach of the Python web world. There is no “standard” app server interface. Because of this, there is no common web server that can handle a common web app format. In other words, I could build a web framework, but I had nothing standard to plug it into.
Thus, my project grew in scope. In addition to building a web framework, I’ve decided to try my hand a building a web server too. Here is where choice starts to come into the picture.
- Do I build a web server that uses a threaded model where a main process farms out connections to a set of worker processes?
- Do I build an asynchronous web server that uses an event loop and runs with cooperative scheduling?
I guess the title of this article is a spoiler. I went with the async event loop pattern.
The best known aysnc programming model
in the web software world is undoubtedly Node.js.
After some research,
it became clear to me
that the underlying event loop
for Node.js call
libuv is an equally popular choice
when considering this async style
The popular Python async web server,
takes this exact approach
of implementing a web server
on top of
libuv as my choice,
but then I needed a way to include it
into my Lua project.
there exists a Lua web framework named
that followed a similar path.
libuv is a C library,
the Luvit developers built a binding libary
libuv as a Lua library.
I’m immensely grateful
that I don’t have to write that binding layer myself
(my C skills are very rusty
and I was never fantastic at C
to begin with).
One big challenge with
that it operates with a callback model.
In order to enable cooperative scheduling,
any call that would be blocking
like network I/O
expects to receive a callback.
I would like users (if ever there are any)
of my web framework
to be able to avoid callbacks.
The Uvicorn project is able to get around these callbacks
thanks to the amazing work
Here’s where I’m starting to run into challenges.
Lua’s standard library is very small.
There is no concept like
There are also no Lua language constructs
Lua has coroutines as its mechanism
for handling cooperative scheduling.
I’ve basically spent my entire weekend researching
to find good ways of using coroutines
to hide callbacks from calling code.
I founds some similar patterns in Luvit
that I’m hoping to replicate
to a degree.
My goal now to make a web server
and make an API
that feels like ASGI.
Since I don’t have the native
by ASGI in Python,
maybe I’ll spawn a variant of Lua ASGI.
Perhaps I should call it LASGI.
Where I’m at is building this bridge from callbacks to coroutines. Once I have that bridge, I’ll probably focus on HTTP 1.1 parsing to get the rough support that is required by the ASGI interface.
In the spirit of having a tracer bullet, when I have the most crude thing possible, I’ll switch back to the framework side and start handling ASGI messages. That will likely mean building a request router and some kind of request controller or handler.
I don’t know how often I’ll write this kinds of entries. I thought it might be useful to expose others to the messy process of building a project. If you’re looking for the code and all my notes, you can find my project on GitHub.