Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Comment on


A project I'm working on now is a system for data transformation pipelines. Kind of like a makefile for data. So your pipeline might be something like:

  • Output the following tabular data as Excel:
    • Run the following query on the following database, resulting in tabular data.
      • SELECT, ... FROM ...
      • Create a temporary database with these tables in it:
        • Create a table named "person" and load it with tabular data:
          • Parse this data as CSV:
            • Download data from
        • Create a table named "companies" and load it with tabular data:
          • Parse this data as tab-delimited data:
            • Download data from

So you'd load up the pipeline, and tell the pipeline that you want the Excel file. And it thinks: OK, so I need to query this database, but first I need to build the database, how do I do that... I need to create it and load these tables... how do I do that... etc. You get the picture.

And in fact, some details of the pipeline are inferred during processing. We often don't want to hard-code file formats like CSV into the pipeline, but instead infer them from HTTP content-type headers and so forth.

It's still at a fairly early stage, but it's progressing quite nicely, and Moose roles are an important part of the implementation.

The reason being that at many stages in the process you need mix-and-match transformations. You need an object which accepts CSV and outputs Excel, or accepts tab-delimited data and outputs Excel, or accepts CSV and outputs an SQL table, or accepts an SQL table and outputs an HTML table.

OK, so each of these could be a class, but it's far easier to do as a role. When you come to a stage in processing, you instantiate a very generic object (which just has a few debugging methods, a unique identifier string, a link back to the pipeline object, etc) and compose it with the roles it needs to be able to handle (read CSV, write Excel).

The technique which you showed in your example is to make a subclass of Math::BigInt, and stuff the new methods into that. It's worth noting that actually, this is exactly how Moose role composition works internally - it creates a new class for you, and reblesses the object into that class. But it's all done on the fly at run-time, so you don't need to create modules for each class, choose names for them, etc. All that is abstracted away.


In reply to Re^3: Altering the inheritance path of an object by tobyink
in thread Altering the inheritance path of an object by citromatik

and:  <code> code here </code>

  • Please read these before you post! —
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Log In?

    What's my password?
    Create A New User
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others cooling their heels in the Monastery: (14)
    As of 2015-08-28 13:06 GMT
    Find Nodes?
      Voting Booth?


      Results (336 votes), past polls