April 13, 2012

Learning How to Serve Up Static Files with Node

So I consider myself a Level 2 node.js/CoffeeScript beginner. I'm learning out in the open. Some of the things I come across are interesting from the perspective of someone climbing the learning curve. To gain experience, I wanted to share some things I've learned about one of the most basic functions of any web server: Serving static files.

Serving static files is something you might take for granted if you come from a background where the "website" is hosted on top of IIS or Apache. I never really gave it a lot of attention, but whatever framework I was using was silently, faithfully, serving up all my images, icons, style sheets, and scripts.

Node gives you pretty low-level access to the HTTP server. So when you make an HTTP application in Node -- you're technically writing the server. So this means you have to be responsible for handling all aspects of the web request and every web request including all those cool static files.

Writing it All from Scratch

So you might start the Node server up in the typical fashion.

http = require 'http'


server = http.createServer serve_file
port = 3000
server.listen port
console.log "The static file server is on port #{port}"

The function serve_file is the callback that will handle the request.

serve_file = (request, response) ->
  console.log "Serving #{request.url}"
  context = { request, response }
  fs.readFile 'public' + request.url, after_file_is_read(context)

The first thing the static file server has to do is actually read the file. We use the asynchronous version of the standard readFile method. I'm bundling up the request and response objects into something I'm calling context. You'll see how I'm handling this object (notice I'm intentionally not using the word class) in subsequent method calls.

after_file_is_read = (context) ->
  (error, data) ->
    console.log "The requested file has been read"
    context =  context extends {error, data}
    respond_with_file(context) or respond_with_error(context)

The method after_file_is_read isn't the actual callback. It's a method that is invoked immediately passing the context and returns a function (the real callback) and captures the current context in a closure. This helps prevent the Christmas tree effect with nested anonymous callbacks. Essentially the sequence can be followed by looking at functions defined on the left side of the code.

I'm probably being a little too clever (and maybe not in the good way), but something that is just too cool to pass up (during my experimentation phase) is to use the extends keyword to add the additional values of error and data into the "context". Essentially, extends adds two additional properties (error and data) onto the exsisting object that already has a request and response properties.

Now here's where it get's really clever. I call into the respond_with_file function. But what if there's an error? Shouldn't I be protecting the call to respond_with_file with an if statement and calling respond_with_error in an else block?

respond_with_file = (context) ->
  if context.error then return undefined
  console.log "Responding with the file"
  {response, data} = context
  response.writeHead 200, {"Content-Type":"text/html"}
  response.end data

I could, but instead, I'm turning the more conventional approach inside out. I'm making the absence of an error a pre-condition for executing the main body of the respond_with_file method. The function returns undefined if the precondition is not met. This let's me compose the functions using the or operator as I've done.

Notice the "Content-Type" header. There's a bug (a.k.a. an unimplemented enhancement) here.

The respond_with_error function does what you might expect. Based on the error received from readFile, it returns either a 404 or 500 response with the error message.

Now start the application up and browse on over to http://localhost:3000.

Ugh error! Sorry no default handling for index.html. Ok, browse over to http://localhost:3000/index.html.

Success!! We're serving up a static HTML page!

With the source I included an image. Browse on over to http://localhost/bunny.png

Ummm...that doesn't look anything like a bunny. Remember the "Content-Type". Well, you have to deliver the right one based on the media type you're serving. Otherwise, the browser has every right to be confused.

Now the Easier, More Powerful, More Complete, More Extensible Way

Wouldn't it be nice if there was a library already built that let us do all that web server stuff? But it can't get in the way like some BAFx would. Well there is of course. ConnectJS.

All that code turns into this:

connect = require 'connect'

server = connect()
server.use connect.static('public')

port = 3001
server.listen port

console.log "Connect is listening at #{port}"

The primary difference being that now it's a much more complete and robust solution. Browse over to http://localhost:3001/bunny.png

It works by adding a concept Connect calls middleware. I think of middleware as a plugin. Essentially, all the middleware objects that are attached to the connect() object are run, and when one of them handles the request it stops. There is existing middleware for a lot of things including authentication, logging, and many many more. In the code above you can see we're attaching the static middleware that serves up static content in the 'public' directory.

What's more, middleware in the context of Connect isn't some heavyweight component technology. It's simply a function and a callback. Very elegant. Very powerful. Very simple. Writing a connect compliant middleware component is for another time (but it won't be a long post).

The source for these examples is on GitHub and the annotated source is the GitHub page.

What's Next

I got plenty of XP from this exercise, but not quite enough to level up. Still need to tackle:

  • Talking to a persistence store
  • Handling authentication of users
  • Logging

Two out of three are handled by standard Connect middleware.

Then I might be ready for level 3. Just enough to start fighting some of the bigger monsters, and getting into bigger trouble.

April 3, 2012

The Node Beginner's Example in CoffeeScript

I started programming in the era when to play a game, you had to type it in by hand. To this day, I still learn best by following along with an extended tutorial; getting it to work initially and then playing around with it to see what happens.

I was delighted to find Manuel Kiessling's The Node Beginner's Book. What a little gem of a tutorial. It starts from the obligatory node http application, and then extends it to deal with real world problems like asynchronous handlers, listening to events, organizing the code, and more; without going too deep. As he states, it's designed to be the tutorial he wishes he had when he first started, and you'll go from beginner to advanced beginner.

I decided to follow along in CoffeeScript. The tutorial provided a great little workbench to try out all sorts of things from code organization to putting the project up on GitHub to trying out code documentation in docco.

I'll walk you through some of the code, but check out the docco documentation for the full details. Like the author did, there's more code, and more abstraction than is required. The point of the walk through is to have a place to try some things out that might come in a "real" application.

Here is the basic specification from The Node Beginner Book:

  1. Users can access the application from a web browser
  2. The user should see a welcome page when requesting http://domain/start which displays a file upload form
  3. By choosing an image file to up to upload and submitting the form, this image should be uploaded to http://domain/upload, where it is displayed once the upload is finished

Here's what you get if you type:

coffee begin.coffee

...and browse to http://localhost:8989.

Beauty, right? Hey, it's not a design project :)

A convention in node is to name the entry point of the application index. I'm not a fan of that convention. Index to me is a view name. My personal convention is to start the application in a module with the same name as the application itself. So the entry point to my application is called begin.

# The entry point to the application.

server = require './server'

server.start 8989

The server module is my code. It simply starts the canonical node HTTP server and starts listening on the passed port. It provides a handler that parses the incoming request URL, bundles them up into an object, and passes them into a router function.

The router is a simplistic route map. The little action function serves as a macro to return the actual function (functions are first-class citizens remember) that is exported inside the module referenced by name. This is a little trick of mine to put all the "actions" in their own folder. It works as long as the action exports its namesake. This shortens the map (routes). The single exported route function, looks up the passed URL path in the routes object literal (or sets the next action to the not_found handler) and processes the request.

# Router determines what to do with the requested path

action = (name) ->

routes = 
  "/": action('start')
  "/start": action('start')
  "/show/uploaded": action('show')
  "/show/sad_panda": action('show')
  "/upload": action('upload')

exports.route = (http) ->
  console.log "Routing #{http.path}"
  process = routes[http.path] ? action('not_found')
  process http

Again for the purpose of code organization I separated out the "views" into their own folder. Some of the simpler actions like start just write an HTML response.

# The start action

html = require '../lib/html'
view = require '../views/start_view'

exports.start = (http) ->
  console.log 'Start action invoked'
  http.respond = html.write_html
  http.respond view.render()

Notice how on line 8 (above), I attach the html.write_html to the http object. This is me noodling with the this property in JavaScript/CoffeeScript. It's different than the meaning of this in C#. The reference to this refers to the context of the function when it is called. It has nothing to do (necessarily) with an object instance of a class. So when you set the respond property to the function html.write_html you essentially spot weld a method on the instance that is held by the http parameter. Invoking it on line 9 (above) with http.render makes the @response properties of http available as on lines 8 and 9 below. Keep in mind that http is just an ordinary object (not a type from a class). The @response was attached to the object way back in the server.

exports.write_html = (html = '', status = 200) ->
  console.log 'writing html'
  headers = 
    "Content-Type" : "text/html"
    "Content-Length" : html.length

  @response.writeHead status, headers
  @response.end html

All that was me trying (probably to an extreme) trying to escape from the Kingdom of Nouns as referenced in the book.

One other discovery worth mentioning is handling asynchronous programming (without an async or await keyword). Take the show action; responsible for determining what image file to write.

# The show action displays an impage

html = require '../lib/html'
view = require '../views/show_error_view'
fs = require 'fs'

exports.show = (http) ->
  match = http.path.match /\/(\w+)$/
  http.image_name = if match then match[1] else 'sad_panda'
  console.log "Display the image #{http.image_name}"
  fs.readFile "./images/#{http.image_name}.png", 'binary', file_loaded(http)

file_loaded = (http) ->
  (error, file) ->
    if not error
      http.respond = html.write_png
      http.respond file
      http.respond = html.write_html
      http.respond view.render({image_name:http.image_name, error}), 500

The first lines just determine what image to display, in a rudimentary way. The last line of the show method calls node's readFile method. The last parameter is the callback to invoke when readFile is ready. I could have defined the function anonymously, and in this particular case it might not matter. Instead, something I really like is invoking a function that returns the actual callback.

If you look closely file_loaded is actually being invoked immediately. However, the result of the function is another function (it returns a function) that will serve as real callback later. The function file_loaded is also saving the state of the http object in a closure. This pattern also has the effect of avoiding deeper and deeper indentation across callbacks. The functions and the "steps" are all along the left side of the code file (but still very much asynchronous). I thought this was an easy way to understand what has happening.

Please checkout the GitHub page for a line-by-line account of the source. Please mess around with the code on your own, and check out The Node Beginner's Book.

What's Next

Now that I've leveled up to (Level 2) as a Node advanced beginner I realize just how far I have to go.

  • Exploring some of the bigger frameworks like Express/Connect and Zappa.
  • Doing some of the more mundane things
    • Authorization
    • Logging
    • Persistence

Node + CoffeeScript is Love.

March 22, 2012

Try and Tell Them That Today, and They Won't Believe You

I would have put long odds that JavaScript would someday rule the world. I remember it as an awkward and terribely mal-supported child in the early days; back when AOL was still around, and the primary purpose of JavaScript was to create a DHTML spinning clock that would follow your mouse.

Silently, over the years, and largely because every browser had to support it for the web to be useful, the JS runtime became the most ubiquitous runtime on the planet. Innovation happened in the background (my background); Gmail, CommonJS, jQuery, Node, etc... I mean, holy cow, the way to develop apps for Windows 8 will be HTML5 and JavaScript. Who would of thought?

But, it's still JavaScript. It's still indisputably an ugly, warty, and hastily thrown together language. Even The Good Parts can't really come up with that many good parts. JavaScript, as a language, is still unappealing to me which is simply a matter of taste. Obviously, it's won and people are doing great things with it.

"Every now and then I feel a temptation to design a programming language but then I just lie down until it goes away." — L. Peter Deutsch

Enter CoffeeScript. A language that compiles to JavaScript. It's not perfect, but it's appealing to me. It still has some of the warts left over from JavaScript, and a few new of its own. Attempting to use it as a shield from learning JavaScript is a mistake -- you need to know JavaScript. Beware. CoffeeScript is still JavaScript. I find it's syntax pleasant and it's features helpful.

There are several debates as to whether or not it's useful (Rob Conery summarizes and links to some of the more prominent talking points in his own struggle). Personally, I don't care that there is a debate at all. I will use CoffeeScript when I can because it's appealing to me. There's room for different tastes, at least in my world.

December 7, 2010

Creating a NuGet Package

NuGet is a package manager for .NET that was recently released by Microsoft as a CTP. This library is similar to gems or cpan or similar libraries in other languages. I decided to try my hand at creating a HelloWorld package.

First I needed a package. My original idea was to create something useful enough to contribute to the public feed of NuGet packages. I started by creating the Shelf library, which is a small set of extension methods. Notably, I created the Each<T> method that extends IEnumerable<T>. It takes an Action<T>, invoking the action for every item in the sequence. It's a trivial and small library at the moment, but imagine it is something useful and complicated. The library itself isn’t the point here.

Create the package “manifest” (my term).

    <description>Shelf is a library of common extension methods</description>
    <file src="Shelf\bin\Release\*.dll" target="lib" />
    <file src="Shelf\bin\Release\*.pdb" target="lib" />

The documentation says the files element is optional. It seems I didn’t discover the “convention” they were speaking about in the documentation. I needed it.

I ran the command-line tool nuget.exe passing it the pack command and the package manifest. It spat out the packaged file (BTW, you can use just about any zip tool to browse the contents of the package).

There are couple of different ways to deploy the package. Submitting your package for inclusion in the public feed is one option. You can put the file on any accessible URL. The NuGet source includes a “Server” utility. Until my library grows to a point where it’s not laughably trivial, I opted for just putting it in my file system and pointing the package manager to folder. Phil Haack has a great post explaining the deployment options.

So with my package (shelf.2010.1203.2330.42313.nupkg) located in my local packages folder, anytime I want to make use of the shelf library, I simply go to the package manager console and type

Install-Package shelf
NuGet goes off, makes sure it has the latest package, adds the package folder to my source if it doesn't already exist, downloads (copies) the assembly, and adds the reference to the project.

Here are some of the references, I’ve found useful:

Conclusion: Piece of cake and terribly useful for consumers of your library.

August 9, 2010

Mercurial .hgignore File for .NET Projects

I’m currently using Mercurial, and find myself starting out with this as a base .hgignore file.


May 7, 2010

Configuring SpecFlow to Work with MSTest

Add a reference to TechTalk.SpecFlow and then put this in an App.config file of the MSTest project:


    <unitTestProvider name="MsTest" />

May 4, 2010

Visual Studio 2010 Styles

If you haven’t checked out Studio Styles yet, please go do so. An improved version of the old Is Your IDE Hot or Not (the old site seems to have fallen into disrepair). Studio Styles is one of those well-executed, brilliant ideas, that has you saying, “Why didn’t I think of that?”

I spent some time creating a version of my favorite color scheme. It’s called Distant Stormy Shore.

CropperCapture[1] It’s based on the work by Thomas Restrepo on the Distant Shores color scheme.

Please download and give me a “Hot” rating if you like it.