A Tour of Tornado (Part 1)

I stumbled across this blog post entitled ‘My Experience In Production with: Flask, Bottle, Tornado and Twisted', by 'The HFT Guy' (whoever that may be!?) and it made for a fascinating read. As someone who has come from the WSGI world (Django, Flask, etc), it was somewhat surprising to see a blog about Python web frameworks which declares that Tornado is ‘the only reasonable choice for web development’. Tornado represented a bit of an unknown for me. If you were to search for Django/Flask, or even FastAPI jobs in your local area, (ok nearest ‘tech hub’) there will undoubtedly be a lot of results. If you refine that same search to Tornado, you’ll be lucky to find a handful.

As developers we always want to think we have the best tools for the jobs, so this entertaining and opinionated blog post peaked my interest enough to give Tornado a little whirl.

What is Tornado?

Tornado describes itself as a ‘…web framework and asynchronous networking library…’, so from the outset it's clear that Tornado is not simply there to only build traditional CRUD type apps. While this tutorial will focus on the Web Application Framework part of Tornado, its docs give equal precedent to this, along with its HTTP Server & Client, networking tools and asynchronous programming utilities. Crucially, the framework is not a WSGI framework, so there is no requirement to run something like Gunicorn et al. between the app and the web server.

The official 'Hello World' app
 

import tornado.ioloop
import tornado.web

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        self.write("Hello, world")

def make_app():
    return tornado.web.Application([
        (r"/", MainHandler),
    ])

if __name__ == "__main__":
    app = make_app()
    app.listen(8888)
    tornado.ioloop.IOLoop.current().start()

For 'controllers' (or 'view' as they are more commonly called in Python), the convention in Tornado is to sub-class the web framework’s Request Handler and then override it’s various methods. Here, a ‘get’ method will respond to get requests. The request handler’s write method, ‘Writes the given chunk to the output buffer’. So far, so good…

The make_app() function returns a Web Application object, initialised with a Tuple consisting of a raw string (to accommodate regular expressions) and the handler Object. Obviously this is a mapping: requests whose path matches the regex will be passed to the correct handler.

In the main program, the make_app() method is called, it is set to listen on port 8888, (which starts an HTTP Server) and then finally, the Tornado IOLoop is started, which will run forever, (until stopped).

Its worth re-iterating at this point that Tornado is not a WSGI framework, so doesn’t require a WSGI or Dev server in-front of it. The way that you are running the Python process here, is analogous to how it would get run on a production server, (albeit with a Web-server/ reverse proxy)… I’ll cover this in more detail in a part 2 blog post.

Here is another trivial example which shows off a few more of Tornado’s features:

import tornado.ioloop
import tornado.options
import tornado.web

from tornado.options import define, options

define("port", default=8000, type=int)

class MainHandler(tornado.web.RequestHandler):
    def get(self,word):
        resp = {'The word is': word}
        self.write(resp)

if __name__ == "__main__":
    tornado.options.parse_command_line()
    app = tornado.web.Application(handlers=[(r"/print/(\w+)", MainHandler)])
    app.listen(options.port)
    tornado.ioloop.IOLoop.instance().start()

The first thing different here is that we are importing from Tornado’s options module, which is used for Command line parsing. 

from tornado.options import define, options

‘’…Options must be defined with tornado.options.define before use, generally at the top level of a module. The options are then accessible as attributes of tornado.options.options:...''

...

define("port", default=8000, type=int)

tornado.options.parse_command_line()

...

app.listen(options.port)

These lines of code ensure that when the app starts, it’s HTTPServer will listen on any port as defined in the command line, falling back to a default of 8000 if no arguments are supplied. If we wish, the program could now be run on a different port like so:

python app.py —port=5000 

This is important because when we want to come to deploy the application, we will likely run multiple instances each listening on different ports and then use ‘something’ in front to distribute load across the different processes.

The next most interesting thing about this enhanced script, is that we are coming good on the regex matching. ‘(\w+)’ will capture any word after /print/, and then that pattern match is presumed to be a parameter we can then access as the second argument in the Handler’s get method.

If we examine the applications response in Postman, we can see its ‘magically’ a JSON response. This is because for the write method, ‘If the given chunk is a dictionary, we write it as JSON and set the Content-Type of the response to be application/json’.


That’s pretty handy! With this nifty feature, it should be clear how quickly one could get some JSON API endpoints up and running with Tornado.

Templating

So what if you wanted to create a full stack web app? How do we get Tornado to give us a ‘template’ response? Here is an example:

import tornado.ioloop
import tornado.options
import tornado.web

from tornado.options import define, options

define("port", default=8000, type=int)

class TemplateHandler(tornado.web.RequestHandler):
    def get(self):
        self.render("example.html")

if __name__ == "__main__":
    tornado.options.parse_command_line()
    settings = {"template_path": "templates"}
    app = tornado.web.Application(handlers=[(r"/", TemplateHandler)],**settings)
    app.listen(options.port)
    tornado.ioloop.IOLoop.instance().start()

So the main difference here is that we are using the RequestHandler’s ‘render’ method rather than write, which takes an html file’s name as an arg. According to the docs, render, ‘Renders the template with the given arguments as the response.’ Which takes us nicely onto the Tornado templating language. We can give keyword arguments to the render method and then manipulate them via Tornado’s templating language.

import tornado.ioloop
import tornado.options
import tornado.web
import requests

from tornado.options import define, options

define("port", default=8000, type=int)

class DomainsHandler(tornado.web.RequestHandler):
    def get(self,word):
        response = requests.get(f"https://api.domainsdb.info/v1/domains/search?page=1&limit=10&domain={word}")
        self.render("domains.html",search_word=word
,**response.json())

if __name__ == "__main__":
    tornado.options.parse_command_line()
    settings = {"template_path": "templates"}
    app = tornado.web.Application(handlers=[(r"/domains/(\w+)", DomainsHandler)], **settings)
    app.listen(options.port)
    tornado.ioloop.IOLoop.instance().start()

If we examine this new app’s handlers, we can see that we are capturing any word after ‘/domains/‘ in the path. I’ve installed the trusty requests library and am making a call to a service I found called https://domainsdb.info/, which allows you to search for information about domain names, (documentation here). The path parameter gets searched against the domains API and its response is unpacked as keyword arguments passed to the render method.

A snippet of the API JSON response looks like this: 

{
  "domains": [
    {
      "domain": "peter-ammon.ch",
      "create_date": "2022-05-05T05:58:35.510068",
      "update_date": "2022-05-06T06:06:38.292103",
      "country": "US",
      "isDead": "False",
      "A": [
        "185.230.63.107",
        "185.230.63.186",
        "185.230.63.171"
      ],
      "NS": [
        "ns12.wixdns.net",
        "ns13.wixdns.net"
      ],
      "CNAME": null,
      "MX": [
        {
          "exchange": "mx2.mail.hostpoint.ch",
          "priority": 10
        }
      ],
      "TXT": null
    },

...

And the HTML template looks like: 

<html>
   <head>
      <title>Tornado serving a Template</title>
   </head>
   <body>
   <h1>Domain Search</h1>
   <p>The word was {{ search_word }}</p>
   <hr/>
   {% for dom in domains %}
         <h3>{{ dom['domain'] }}</h3>
        <table>
            <tr>
                <th>Created Date</th>
                <th>Expiry Date</th>
            </tr>
            <tr>
                <td>
                    {% set created = datetime.datetime.fromisoformat(dom['create_date']) %}
                    {{ created.strftime('%d-%m-%Y') }}
                </td>
                <td>
                    {% set updated = datetime.datetime.fromisoformat(dom['update_date']) %}
                    {{ updated.strftime('%d-%m-%Y') }}
                </td>
            </tr>
        </table>
    {% end %}
   </body>
 </html>

The results of the two together: 

More so than the Django template language, or Jinja2 (widely used in the world of Python), it seems to me that the Tornado Templating language makes more of a virtue of it being ‘Python in an HTML’ template. The Tornado docs state ’A Tornado template is just HTML (or any other text-based format) with Python control sequences and expressions embedded within the markup’. Contrast that with the Django docs which state:

‘ … you’ll want to bear in mind that the Django template system is not simply Python embedded into HTML. This is by design: the template system is meant to express presentation, not program logic.

The Django template system provides tags which function similarly to some programming constructs – an if tag for boolean tests, a for tag for looping, etc. – but these are not simply executed as the corresponding Python code, and the template system will not execute arbitrary Python expressions. Only the tags, filters and syntax listed below are supported by default (although you can add your own extensions to the template language as needed).’

So whereas in Django or Jinja2, you’ll see nested objects accessed with dot notation, in Tornado you can see the Python dictionary ‘key-name-square-brackets’  way of accessing the values:

…

{% for dom in domains %}

      <h3>{{ dom['domain'] }}</h3>

…

If its wrapped in double curly brackets: its an expression. If its wrapped in curly brackets and percentages, its a control statement.

Some Tornado modules and even Python standard library modules are available in the Tornado Templating Language. You can see we make use of datetime to convert a string to a datetime object (‘set' is required to declare a variable), so that we can then reformat it.

…
<td>
    {% set created = datetime.datetime.fromisoformat(dom['create_date']) %}
    {{ created.strftime('%d-%m-%Y') }}
</td>
… 

How did Tornado know where to look for the template? Take a look at these bits of the code from the app.py file:

...

settings = {"template_path": "templates"}

…

app = tornado.web.Application(handlers=[(r"/domains/(\w+)", DomainsHandler)], **settings)

'Settings' are just a dictionary which can be unpacked into the Application, when initialising.The same can be done for static resources:

settings = {
    "template_path": "templates",
    "static_path": "static",
}
app.py
| - static
    | - domain.css
| - templates
    | - domain.html

Once the files are arranged like above, we can call the static_url helper method in our template, which will return the URL for the resource, relative to the file path defined in settings.

<link rel="stylesheet" href="{{ static_url(“domain.css") }}">

 

Native Http Client

We're using the requests library but, as has already been hinted at, Tornado has its own Http Client. Can we use this instead? Yes, but we need to refactor the code slightly so that it uses the async/await syntax. This isnt a Python & concurrency tutorial, so I wont go into that here, but I got the below working, after reading this stackoverflow post along with generous timeout parameters.  

import tornado.ioloop
import tornado.options
import tornado.web
import json
from tornado.httpclient import AsyncHTTPClient

from tornado.options import define, options

define("port", default=8000, type=int)

class DomainsHandler(tornado.web.RequestHandler):
    async def get(self,word):
        http_client = AsyncHTTPClient()
        response = await http_client.fetch(f"https://api.domainsdb.info/v1/domains/search?page=1&limit=10&domain={word}")
        http_client.close()
        response_dict = json.loads(response.body.decode('utf-8'))
        await self.render("domains.html",search_word=word,**response_dict)

if __name__ == "__main__":
    tornado.options.parse_command_line()
    settings = {"template_path": "templates"}
    app = tornado.web.Application(handlers=[(r"/domains/(\w+)", DomainsHandler)], **settings)
    app.listen(options.port)
    tornado.ioloop.IOLoop.instance().start()

UI Modules.

Whilst the Tornado Templating Language does support template inheritance (like Django and Jinja2), it has its own cool ‘UI Modules’ concept, which allows for the creation of reusable components.

Take the above example, whereby we loop through all the domains and show a table for each one. We could break up the template and consider each ‘item’ in that loop as a separate UI Module, moving some of the logic out of the template, and back into a class in our pure Python code, (there is a school of thought which believes that presentational logic and business logic shouldn’t mix, of course, but that's another conversation!)

...

class DomainModule(tornado.web.UIModule): 
    def render(self, dom):
        name = dom['domain']
        created = datetime.datetime.fromisoformat(dom['create_date'])
        updated = datetime.datetime.fromisoformat(dom['update_date'])
        return self.render_string(‘modules/domain.html', name=name,created=created,updated=updated)

 

We’ve created a class which extends Tornado Web’s UIModule. Its render method accepts a ‘dom’, i.e the information for a single domain and we are grabbing its name, created and updated dates and performing the necessary transforms. The settings are then updated like so, giving the UIModule an alias.

settings = {"template_path": "templates", "static_path":"static", "ui_modules": {"Domain": DomainModule}}

And the loop in domains.html now does this:

{% for dom in domains %}
     {% module Domain(dom) %}
 {% end %}

As you would expect we now have a new html file for the UIModule,(as referenced in the render_string call), which resembles the body of the loop previously in domains.html:

<h3>{{ name }}</h3>
<table id="domains">
    <tr>
        <th>Created Date</th>
        <th>Expiry Date</th>
    </tr>
    <tr>
        <td>
            {{ created.strftime('%d-%m-%Y') }}
        </td>
        <td>
            {{ updated.strftime('%d-%m-%Y') }}
        </td>
    </tr>
</table>

We’ve taken out a lot of the datetime faffing-around and put that in the UIModule’s Python code, but this is still Python, so the values passed in as ‘created’ and ‘updated’ and datetime objects and we can call strftime() on them. I guess this is arguably presentational, so the aforementioned purists will be happy!

You can even break up your static assets with the javascript_files, or css_files methods, into more manageable chunks. If I rename my css file ‘main.css’ and move all the stuff to alter the appearance of the table into ‘domain-detail.css’, I can alter the UIModule class like so:

class DomainModule(tornado.web.UIModule):
    def render(self, dom):
        name = dom['domain']
        created = datetime.datetime.fromisoformat(dom['create_date'])
        updated = datetime.datetime.fromisoformat(dom['update_date'])
        return self.render_string('modules/domain.html', name=name,created=created,updated=updated)

    def css_files(self):
        return ‘domain-detail.css'

With the sprinkling in of some (very bad) CSS, the App is currently looking like this: 



There is even the option to inject javascript and css code directly into the template rendered, from the UI Module, with the embedded_css or embedded_javascript methods. 

Just a trivial example to demonstrate JS, If I take this out of the template:

<p>The word was {{ search_word }}</p>

I might decide that I really want to insist the user of this terrible app is confronted with the word that they have searched, via a popup.

...

class SubHeadingModule(tornado.web.UIModule):
    def render(self, word):
        self.word = word

    def embedded_javascript(self):
        return f'window.alert("The word searched was: " + "{self.word}");'

...

settings = {
        "template_path": "templates",
        "static_path": "static",
        "ui_modules": {"Domain": DomainModule, "SubHeading":SubHeadingModule},
    }

...
<html>
   <head>
    <link rel="stylesheet" href="{{ static_url("main.css") }}">
    <title>Tornado serving a Template</title>
   </head>
   <body>
   <div id="header">
       <h1>Domain Search</h1>
   </div>
   <div id="subheading" hidden>
      {% module SubHeading(search_word) %}
   </div>
   <hr/>
   {% for dom in domains %}
        {% module Domain(dom) %}
    {% end %}
    </body>
</html>



Now when the page loads, we get a popup to click on. All that behaviour is controlled by JavaScript embedded via a UI module in our Python code. Writing out JS as strings would probably get a bit tedious for more complex apps, it seems far more likely to me that the _files methods would be useful. Having said that, the trend in a lot of web development at the moment it to opt for ‘sprinklings’ of JavaScript, rather than a fully blown decoupled front-end, so who knows? At any rate, I think the Tornado Templating language and its UI Module concept is very cool and allows for a great flexibility for Python developers to fold in JS, CSS, HTML via files or otherwise!
 

CONCLUDING THOUGHTS

I do actually get it: the HFT Guy is right! The way Tornado encourages you to structure your projects feels addictive and intuitive, and (to me) it makes a nice contrast to the dominant patterns you see in a lot of other Python web frameworks. I’ll undoubtedly try and incorporate it into a non-trivial project.

I intend to write a part 2 of this tutorial relating to deployment. Watch this space!
 

Github Repository

You may also like: