Twitterpillar – An example app using shared templates

June 20, 2012 – 7:30 pm by Dave Elkan

I made the Twitterpillar app as a part of my recent SydJS presentation on sharing templates between the server and the browser.

The app will allow you to enter a twitter screen name and see what that user’s followers have tweeted recently. It’s made using the method of sharing templates between the server and browser which I described in the presentation.

In short, the templates are compiled on the server and used there to entirely render all pages requested. Also, the compiled templates are sent in the form of Javascript functions to the browser so it can update itself when the user performs an action rather than re-requesting the entire HTML page. Sent along with the HTML is the details of the user’s followers in json format. This data is requested by the app to produce the left-nav and follower details panel. It may as well be re-used by the browser so it is sent along at the bottom of the HTML. In this version, that data is not updated unless you refresh the entire page.

The History API is used to change the URL when a follower is selected. The urls the app switches to are normal URLs (i.e. with no #! ‘hashbang’). If a user of the app were to type in and request or refresh the app with one of these  proper URLs the HTML for the entire page will be returned from the server with that user selected and their tweets present. This is facilitated by the use of the shared templates.

Using twitterpillar

The twitterpillar app is a node.js app and was developed with node 0.6.

$ git clone git://github.com/dave-elkan/twitterpillar.git
$ cd twitterpillar
$ npm install -d
$ node app.js

Presentation

After running the app, the presentation can be found at http://localhost:3000/static/presso/ or alternatively you can view an online copy here: http://www.edave.net/template-presentation-062012

Introducing the Node.js Hogan Template Compiler module

May 13, 2012 – 2:44 pm by Dave Elkan

In my previous post I described why you should pre-compile your templates on the server.

I’ve just created the Hogan Template Compiler module which makes it easier to implement the entire process. It’s a simple framework-agnostic npm module which:

  • Pre-Compiles mustache templates and partials on the server.
  • Stringifies partials for use in the browser.

How the module works

The diagram below describes how you can compile your templates once on the server and then re-use them in the browser. This means that the browser never has to compile the templates.

  1. The Hogan Template Compiler  reads the templates and partials
  2. The browser requests the complete HTML page (including server-rendered partials).
  3. The browser loads the pre-compiled partials (in the form of a simple JavaScript file).
  4. The browser renders additional pre-compiled partials when needed.

Why pre-compiling and sharing templates is a good idea

There are heaps of benefits to pre-compiling and sharing templates! The biggest is performance.

Performance

Ajax onDomReady is suboptimal. It’s faster for the browser to display your web page if your server renders the entirety of every HTML response. This frees the browser from doing any additional ajax requesting or template compiling and rendering.

As an example, as it stands currently, if you go to someone’s twitter page then the static HTML file you’re served from the server is your own tweet stream, not of the user you’ve requested. The other user’s tweets are fetched via ajax and rendered over the top. This results in the pop you see when you first load the page.

Now, I’m not second-guessing Twitter’s choice in this matter as obviously there’s heaps of technical limitations running at their scale. However, if they could return the static HTML of the page you’ve requested, then your browsing experience would be better.

The Hogan Template Compiler allows you to render the entirety of your templates on the server making it possible to totally deliver every page of your site as quickly as possible. Subsequent rendering can be done in the browser with the shared pre-compiled partials which are created for you. They’re simply loaded as a JavaScript file.

SEO

Another benefit of having the entire page rendered server-side is that Google and other search engines can easily index all of your content.

One Template To Rule Them All

Using the very same templates for rendering on the server and in the browser means that there’s one source of truth for how your pages are rendered. Some of the sites I’ve worked on have had fragmented templating systems meaning I’d be forced to update the server-side and client-side templates separately.

Caching

Since the pre-compiled partials are delivered as a static JavaScript file, the browser will cache this for as long as you want it to.

How to use it

$npm install hogan-template-compiler

Load the module and point it to your partials directory:

var HoganTemplateCompiler = require('hogan-template-compiler'),

    templateDirectory = __dirname + "/views",
    hoganTemplateCompiler = HoganTemplateCompiler({
        partialsDirectory: templateDirectory + "/partials"
    });

Compile a template manually:

var compiledTemplate = hoganTemplateCompiler.compileTemplateFile(templateSourcePath);

Setup a route (in Express) to send the pre-compiled templates to the browser:

app.get("/templates.js",  function(req, res) {
    res.contentType(".js");
    res.send(hoganTemplateCompiler.getSharedTemplates());
});

For a working example with Express; try the following:

$ git clone https://dave-elkan@github.com/dave-elkan/pre-compiled-hogan-templates.git
$ cd pre-compiled-hogan-templates
$ npm install -d
$ node app.js

Get the Source

Grab yourself a copy at https://github.com/dave-elkan/hogan-template-compiler

Sharing pre-compiled templates between server and client with Hogan.js

February 25, 2012 – 1:37 pm by Dave Elkan

At work we use Closure Templates to render HTML on the client. Closure Templates boast the lovely feature of being able to compile templates down to simple javascript functions on the server for use on the client. This is good because:

  1. it relieves the browser of having to compile your templates
  2. you don’t have to clutter your markup with template source and
  3. you can render on the client and server with exactly the same template without sending HTML down the wire.

The last point offers a significant SEO benefit: By combining server rendered markup for the first page a user visits and client-rendered pushState delivered markup thereafter, you satisfy both your own desire to have a dynamic web app and also Google’s desire to read static HTML.

I was surprised there wasn’t any immediately obvious equivalent to Closure Templates for node.js. I went as far as trying (in vain) to get ejs to export stringified functions. I quickly threw out that pursuit when I found Hogan.js.

Hogan.js, from Twitter, is an implementation of the mustache templating language for use in the browser and on the server with node.js.

An example of creating and using shared templates with Hogan.js

This example will render a simple article page using a mustache template which can be used by the server and the client. All of the snippets below are taken from a working Express app available on github.

Loading and compiling the shared templates

The following function loads templates to share between the server and client from a specified directory and compiles them to a stringified javascript function.

/**
 * Reads and compiles hogan templates from the shared template
 * directory to stringified javascript functions.
 */
function readSharedTemplates() {
    var sharedTemplateFiles = fs.readdirSync(sharedTemplateDirectory);

    // Here we'll stash away the shared templates compiled script (as a string) and the name of the template.
    app.sharedTemplates = [];

    // Hogan like it's partials as template contents rather than a path to the template file
    // so we'll stash each template in a partials object so they're available for use
    // in other templates.
    app.sharedPartials = {};

    // Iterate over each sharedTemplate file and compile it down to a javascript function which can be
    // used on the client
    sharedTemplateFiles.forEach(function(template, i) {
        var functionName = template.substr(0, template.lastIndexOf(".")),
            fileContents = removeByteOrderMark(fs.readFileSync(sharedTemplateDirectory + template, "utf8"));

        // Stash the partial reference.
        app.sharedPartials[functionName] = fileContents;
        // Stash the compiled template reference.
        app.sharedTemplates.push({
            id: functionName,
            script: hogan.compile(fileContents, {asString: true}),
            // Since mustache doesn't boast an 'isLast' function we need to do that here instead.
            last: i === sharedTemplateFiles.length - 1
        });
    });
}

Delivering the pre-rendered templates

The result of the hogan.compile function when you pass {asString: true} (highlighted above) will return a stringified javascript function which can be sent to the browser. The name of the template file is used as the id of the template.

The output of readSharedTemplates is sent to the following mustache template which renders them as a simple javascript object.

var templates = {
{{#templates}}
    "{{id}}": new Hogan.Template({{{script}}}){{^last}},{{/last}}
{{/templates}}
};

Note the triple mustaches around the script variable. This prevents the javascript being HTML escaped.

The output of this template looks something like this:

var templates = {
    "article": new Hogan.Template(function(c,p,i){i = i || "";var b = i + "";var _ = this;b += "<h2>";b += (_.v(_.f("headline",c,p,0)));b += "</h2>";b += "\n" + i;b += "<p>";b += (_.v(_.f("bodyText",c,p,0)));b += "</p>";b += "\n";return b;;})
};

Handling the requests

The following request handler responds with the template javascript file. You don’t have to do it this way. You could easily add the template javascript as an inline script tag in the HTML response.

/**
 * Request handler for pre-compiled hogan.js templates.
 *
 * This function uses a hogan template of it's own which renders
 * calls to Hogan.Tempate. See views/sharedTemplates.mustache.
 */
app.get("/templates.js", readSharedTemplatesMiddleware, function(req, res, next) {
    var content = sharedTemplateTemplate.render({
        templates: app.sharedTemplates
    });
    res.contentType("application/javascript");
    res.send(content);
});

For each request to this handler the readSharedTemplatesMiddleware function is call which reloads the shared templates in development mode.

The other request handler renders the HTML for the initial page. It uses the contents of the shared article.mustache file as a partial which is in turn loaded by layout.mustache (highlighted below).

/**
 * Request handler for the homepage.
 *
 * Renders a hogan template on the server side which contains a form
 * which will update the article section on the client using the
 * pre-compiled template.
 */
app.get("/", function(req, res, next) {
    res.render("layout.mustache", {
        context: {
            headline: "This is a server-side rendered headline",
            bodyText: "This is some bodytext"
        },
        partials: {
            "article": app.sharedPartials["article"]
        }
    });
});

Rendering the shared template in the browser

On the client we can refer to the pre-rendered template function by calling the templates.article function. i.e.

templates.article({
    headline: "This is a client-generated headline",
    bodyText: "This is a client-generated body"
});

will generate:

<h1>This is a client-generated headline</h1>
<p>This is a client-generated body</p>

Running the example

To illustrate the use of a pre-compiled shared template on the client I’ve put a form in the output of layout.mustache which will update the server-rendered article using the client-side pre-compiled template.

Give it a run for yourself!

$ git clone git://github.com/dave-elkan/pre-compiled-hogan-templates.git
$ cd pre-compiled-hogan-templates
$ npm install -d
$ node app.js

Layers – Layered Architecture for node.js made easy

June 28, 2011 – 6:37 pm by Dave Elkan

EDIT – This guide is kinda out of date and probably not that applicable anymore. Try using a supported library like Flatiron or just plain old Express but by no means let me stop you if you want to take Layers and run with it.

In my previous post A Layered Node.js Architecture using Express I wrote on the benefits of using a layered architecture and how to implement one in a node.js web app. This post goes one step further by introducing Layers, a module which will help automatically load and neatly setup the layers (and routes) of your web app.

Layers currently only supports Express, but adding support for other frameworks is simple. Feel free to submit a pull request!

The following is a description of how Layers works. For the impatient or code-hungry there is the working Layered Express example from which the following snippets are taken.

Layers performs two basic operations:

  1. Loads all layer javascript files.
  2. Loosely couples the layers and adds routes.

Installing

$ npm install layers

Example app

Simply require it and instantiate with your app, the path to your layers, a function which returns your wiring (more on this in the next section) and, optionally, an options object. i.e.

var Layers = require('layers').Express,
    wiring = require('./layers/wiring');
new Layers(app, __dirname + '/layers', wiring);

File Layout

In this example the layout of the directories is:

__dirname
   - layers
       - controllers
       - services
       - views

Loading Layers

Each directory nested immediately within the layers directory defines a layer.

Each of these layer directories are recursively scanned for javascript files which are imported using require. The result of the require is inspected for two possibilities:

  1. An object
  2. A function which returns an object.

The object loaded or returned from the function is appended to the app object under the layer’s namespace.

e.g.

/path/to/app/layers/controllers/BookController.js becomes app.controllers.bookController
/path/to/app/layers/services/AuthorService.js becomes app.services.authorService

One exception is made when loading the layer files: Any file whose name begins with “Base” is ignored.
This name can be overriden by specifying the `excludePrefix` option.

Wiring

To wire the layers together you need to create a ‘wiring’ function. This function takes the layer populated app object as it’s only parameter and returns a hash of arrays indexed by the route they service. i.e.

module.exports = function(app) {
    var controllers = app.controllers,
        views = app.views;

    return {
        
        "/": [{
                action: controllers.homeController.getAuthorAndBookList,
                views: {
                    html: views.homeView
                }
            }
        ],
        
        "/books": [{
                action: controllers.bookController.getBookIndex,
                views: {
                    html: views.bookIndexView,
                    json: views.jsonView
                }
            }
        ],
        
        "/book/:key": [{
                action: controllers.bookController.getBookByKey,
                views: {
                    html: views.bookView,
                    json: views.jsonView
                }
            }
        ],
        
        "/authors": [{
                action: controllers.authorController.displayAuthorList,
                views: {
                    html: views.authorIndexView,
                    json: views.jsonView
                }
            }
        ],
        
        "/authors/:key": [{
                action: controllers.authorController.displayAuthorByKey,
                views: {
                    html: views.authorView
                }
            }
        ]
    };
};

The format of the wiring object is important. Each route supports many handlers which all have to define action and view properties.

Actions

A handler’s Action is simply a reference to a function which accepts the request and response objects as well as the callback and next functions. They are called in the context of the app object which allows for easy access to all of the other layers.

Action functions generally belong to the Controller layer. They are responsible for pulling variables from the request, their validation and sending them to the Service layer. This contrived example does not allow any ’1s’ in the Author’s key which is silly, but stop and imagine authentication at this point.

You will also notice that this controller simply delegates to the service layer, but at the same time it shields it from the request and response objects.

The callback variable which is owned and created by Layers is passed straight through to the service layer.

module.exports = {
    
    displayAuthorList: function(req, res, callback) {
        this.services.authorService.getList(callback);
    },

    displayAuthorByKey: function(req, res, callback, next) {
        var authorKey = req.params.key;
        if (authorKey.indexOf("1") > -1) {
            callback(new Error("No ones allowed!"));
        } else {
            this.services.authorService.getAuthorAndTheirBooks(authorKey, callback);            
        }
    }
};

Views

Views are wrappers around your favourite templating system. They also accept the request and response objects but also a third object which is the result of the action. This object is the payload sent to the template.

BaseExpressView can be used as a basis for your own Express (jade) Views. You only need to specify the “getTemplate” function.

Conclusion

Layers is a simple way to get your app up and running quickly in something which resembles a scalable way. It allows you to specify all of your routes in one place and to specify an ordered list of actions for each of them. It will load and run your layer code and namespace it on your app object for easy access.

Love to hear some feedback if you have any.

EDIT: Updated to replace gists with local code snippets. Gist wasn’t always loading.

A Layered Node.js Architecture using Express

March 22, 2011 – 9:49 pm by Dave Elkan

Using layers in your app is a good way to ensure separation of concerns. This post is not an in-depth description to this architecture pattern but more a quick description of each layer’s purpose and how I’ve implemented them in node using express. For the impatient, you can peruse the code of the working example here.

Software used

View Layer

  1. Accept some data
  2. Apply any required formatting
  3. Render

View instances wrap a template and supplies additional functions to help take the burdon off the templating system, whichever one you use. For example, text transformations, regular expressions, etc are best done outside of the template in a simple testable function. In the following AuthorView example, whilst being contrived, the format function sets the Authors name to upper case.

Controllers

  1. Map a URI
  2. Extract some parameters
  3. Kick off some work (it’s not concerned what nor how it gets done)
  4. Send the result to a View for rendering

This example illustrates the use of Controller.setupGetRoute, a convenience function which takes a route’s path, an action and a hash of Views mapped to the Accept header they respond to. The action property is a function in the current Controller instance which extracts the required parameters from the request and forwards them onto the Service Layer. Note the lack of error handling in the following AuthorController example. All errors resulting from the Service layer are passed from Controller.setupGetRoute directly to the callback, preventing clutter in the Controller instances.

For each request, the the Accept header is used to map the request to a View. If a match is made, the corresponding View’s render function is called with the request and response objects along with the result of the call to the action.

I accept that express middleware can be used instead of this call to the Service layer, however if you do you have to attach the result of the action to the request object. That feels a bit muddled to me.

Service Layer

  1. Do something for the Controller
  2. Return the result to the Controller

Try to think of the Service layer as the “do anything the Controller wants” layer. It consists of specific use-case functions, as in the following AuthorService gist. This function pull the author requested from the datastore and then grabs their books for good measure and returns the them both (or an error). The most important thing about the Service layer is that it separates the Controllers from the datastore or other sources of complexity. This ensures that writing (and testing) Controllers remains a simple task.

Services are inter-dependent singletons and are attached to the app. i.e. In the example below, the AuthorService uses the BookService to get the Author’s Books.

Data Access Layer

  1. Put, Get, Update and Delete datastore entries.

The Data Access Layer is our last layer. It is hard-tied to the datastore of your choice and manages every interaction with it. Once again, encapsulation is the key. Having all of your database code decoupled from your Controllers and Services makes them easier to test. It also makes moving from one datastore to another a much less painful process.

In the example code, each dao is tied to its own collection. In so, calling getList from AuthorDao will return a list of authors, from BookDao, books and so on. For now the example only handles Put and Get operations to keep the example from growing too large. Feel free to send a pull request!

Working Example

Pull a clone of the working example here: https://github.com/dave-elkan/layered-express

Notes on Models

Mongoose is making great strides to becoming the ORM for node. It does a wonderful job of taking care of the heavy lifting of MongoDB abstraction, validation, defaults, etc. However I don’t use it in this example as one of my main goals with using a layering technique is that sources of Error are centralised and handled in as few places as possible. Mongoose uses an “enriched model” pattern (where every model object has a save, update and delete function). Whilst this is very convenient, I wanted to enforce my database transactions to take place from one central point.