Writing a URL Shortening Service in Erlang, Part 3

In this installment, we'll turn the core sherl_db module into a proper OTP application by creating a sherl gen_server, a supervisor, and a an application resource file. While the sherl_db module can already be used as a server (the tests in part 2 show that it can be called concurrently), setting up a gen_server will provide a standardized framework for packaging the code as well as a nice abstraction layer should we decide to replace the mnesia-based sherl_db module with something else.

First we will tackle the gen_server part of the show. The erlang-mode for Emacs provides a very useful template for creating new gen_servers; even if you don't use Emacs, you might want to steal the template idea as it proves quite useful. The gen_server will live in a module called simply sherl and in addition to the required gen_server behavior callbacks, exports the following API:

-export([start_link/0, stop/0, encode/1, decode/1]).

A client will call sherl:encode to turn a URL into a code that we will use as a short URL. The code we will use is the base 62 encoding of an integer that the sherl_db module assigns to the given URL. We choose base 62 as it allows us to represent short URL codes using A-Z, a-z, and 0-9 which will keep our short URLs very clean and easy to parse -- and in particular, will reserve "." for use as a format extension so that web users can append ".json" or ".xml" to retrieve the short URL meta data in a desired format. Below is the definition for the main sherl public API:

start_link() ->

stop() ->
  gen_server:call(?SERVER, stop).

encode(Url) ->
    gen_server:call(?SERVER, {encode, Url}).

decode(Code) ->
    gen_server:call(?SERVER, {decode, Code}).

While you don't have to define a wrapper API for a gen_server, it makes using it that much more convenient. With the above API, client code can simply call sherl:encode(Url) instead of gen_server:call(sherl, {encode, Url}).

As you can see, each API call maps to a message sent as a tuple with first element describing the desired action. The server will read the message and dispatch to the appropriate function to respond to the client's request.

Before going further with the API, a short diversion to implement the init callback. Here it is:

init([]) ->
    {ok, #state{}}.

Next up is implementing the handle_call callback. The is where the real work of the server gets done. In implementing the callback code, one realizes that what is really going on is message passing. The client sends the sherl gen_server a message and the sherl server examines the message to determine how to respond. Here's what we need to support the API defined above:

handle_call({decode, Code}, _From, State) ->
    case sherl_db:get_url(base62:decode(Code)) of
        undefined ->
            {reply, {not_found, Code}, State};
        UrlRec ->
            {reply, {ok, UrlRec#url.url}, State}
handle_call({encode, Url}, _From, State) ->
    Rec = sherl_db:get_code(Url),
    Code = base62:encode(Rec#url.code),
    {reply, {ok, Code}, State};
handle_call(stop, _From, State) ->
    {stop, normal, stopped, State};
handle_call(_Request, _From, State) ->
    {reply, ignored, State}.

Our server doesn't maintain any state between calls, but if it did, we would thread the state data through all the callbacks using the State variable that you see passed in as an argument and returned by all of the callbacks. Ok, takes care of the sherl gen_server. Now we need a supervisor and an application module. As these follow fairly mechanically from their required callbacks, I will simply list them below.

-define(SERVER, ?MODULE).

start_link() ->

init([]) ->
    Sherl = {sherl_main, {sherl, start_link, []},
              permanent, 2000, worker,[sherl]},
    {ok, {{one_for_one, 20, 60}, [Sherl]}}.

start(_Type, _StartArgs) ->
    case sherl_sup:start_link() of
        {ok, Pid} -> 
            {ok, Pid};
        Error ->

stop(_State) ->

Finally, we need an application resource file (sherl.app) and that goes like this:

%% -*- mode: erlang -*-
{application, sherl, 
 [{description, "sherl URL shortening core service"}, 
  {vsn, "0.1"}, 
  {modules, [base62, sherl, sherl_app, sherl_db, sherl_sup]}, 
  {registered, [sherl, sherl_sup]}, 
  {applications, [kernel, stdlib, mnesia]}, 
  {mod, {sherl_app, []}}, 
  {start_phases, []} 

I put sherl.app in the src directory along with the *.erl files and have the build process copy it to ebin/sherl.app since application resource files need to be on your code path for the application module to find them.

We now have an OTP application. Here's some example usage from the Erlang shell.

Eshell V5.7.1  (abort with ^G)
1> application:start(mnesia).
2> application:start(sherl).
3> sherl:encode("/posts/2009/06/20/writing-a-url-shortening-service-in-erlang-part-3/").
4> sherl:decode("6").
5> sherl:decode("abc").
6> gen_server:call(sherl, blah).
7> application:stop(sherl).

=INFO REPORT==== 20-Jun-2009::22:19:02 ===
    application: sherl
    exited: stopped
    type: temporary

In the sherl.app resource file, we listed mnesia as an application that must be running for sherl to start. The application framework will verify that mnesia is running and error out before starting sherl if mnesia is not running. It will not, however, start mnesia for you which is why, in the demo above, I had to explicitly start mnesia before starting sherl -- even though the sherl_db start function will start mnesia if needed.

So far we've written the core database module, sherl_db, tested it using Common Test, and packaged it into an OTP application. In the next installment, we'll add the HTTP interface for the sherl URL shortening service using the webmachine framework.

UPDATE I realized that the initialization of sherl isn't quite right when starting for the first time. After posting a question to the Erlang questions mailing list, I came up with the following adjustment to sherl_db.erl based on the helpful replies I received:

start(Nodes) ->
    case is_fresh_startup() of
        true ->
            case mnesia:system_info(is_running) of
                yes ->
                    error_logger:info_report("stopping mnesia"),
                _ -> pass
            error_logger:info_report("mnesia schema created"),
            error_logger:info_report("starting mnesia"),
            mnesia:create_table(url, [{disc_copies, Nodes},
                                      {attributes, record_info(fields, url)},
                                      {index, [url]}]),
            mnesia:create_table(counter, [{disc_copies, Nodes},
                                           record_info(fields, counter)}]),
            error_logger:info_report("mnesia tables created");
        {exists, Tables} ->
            ok = mnesia:wait_for_tables(Tables, 20000)

%% @spec is_fresh_startup() -> true | false
%% @doc Returns true if mnesia has not been initialized with
%% the sherl schema.
%% Thanks to Dale Harvey for this function posted to
%% the erlang questions mailing list.
is_fresh_startup() ->
    Node = node(),
    case mnesia:system_info(tables) of
        [schema] -> true;
        Tbls ->
            case mnesia:table_info(schema, cookie) of
                {_, Node} -> {exists, Tbls};
                _                 -> true

The issue was that since mnesia is listed as a required application in sherl.app, it needs to be started before the sherl application can start. But once mnesia is started, you cannot (easily) create a schema. The approach taken above is to detect this fresh start situation and stop mnesia in order to initialize the desired schema. Other approaches, which perhaps are more appropriate for a production environment, that were suggested on the list include using the builder module which will create a load script that could be customized or to make use of mnesia:change_config (although I believe this requires a separate running mnesia).

archived on 2009-06-20 in , , , , ,

blog comments powered by Disqus