Personal tools

HXT/Conversion of Haskell data from/to XML

From HaskellWiki

Revision as of 17:03, 19 April 2008 by UweSchmidt (Talk | contribs)

Jump to: navigation, search

1 Serializing and deserializing Haskell data to/from XML

With so called pickler functions and arrows, it becomes rather easy and straight forward to convert native Haskell values to XML and vice versa. The module Text.XML.HXT.Arrow.Pickle and submodules contain a set of picklers (conversion functions) for simple data types and pickler combinators for complex types.


2 The idea: XML pickler

For conversion of native Haskell data from and to external representations, there are two functions necessary, one for generating the external representation and one for reading/parsing the representation. The read/show pair often form such a pair of functions.

A so called pickler is a data value with two such conversion functions, but because it's necessary to apply a whole sequence of conversion functions, there is a state that has to be updated during encoding and decoding the external representation. So the simplest form of a pickler converting between a type a and a sequence of Chars looks like this.

type St    = [Char]
		, appUnPickle :: St -> (a, St)


The HXT picklers are an adaptation of these pickler combinators. The difference to Andrew Kennedys approach is, that the target is not a list of Chars but a list of XmlTrees. The basic picklers will convert data into XML text nodes. New are the picklers for creating elements and attributes.

The HXT pickler type is defined as follows

data St		= St { attributes :: [XmlTree]
		     , contents   :: [XmlTree]
		     , theSchema   :: Schema

In XML there are two places for storing informations, the attributes and the contents. Furthermore the pickler contains a third component for type information. This enables the derivation of a DTD from a set of picklers.

But we will see, that with the predefined picklers and the combinators we don't have to look very much into these internals. Let's start with an example.

3 Example: Processing football league data

3.1 The XML data structure

From the set of HXT/Practical example we'll take the data structure from HXT/Practical/Simple2 dealing with football league data. First let's have an idea about the structure of the XML data. Here is a part of the example XML data

<SEASON YEAR="1998">
  <LEAGUE NAME="National League">
    <DIVISION NAME="East">
        <PLAYER GIVEN_NAME="Marty" SURNAME="Malloy"
            POSITION="Second Base" GAMES="11"
        <PLAYER GIVEN_NAME="Ozzie" SURNAME="Guillen"
            POSITION="Shortstop" GAMES="83"
        <PLAYER GIVEN_NAME="Gerald" ...>
  <LEAGUE NAME="American League">
    <DIVISION NAME="East">
    <DIVISION NAME="Central">

3.2 The Haskell data model

Let's first analyze the underlying data model and then define an appropriate set of Haskell data type for internal representation.

  • The Divisions are also ideitfied by a String and consist of a list of Teams, so it's also a Map
  • A Player has a lot of attributes, we will simplify the internal model a bit, we will just include six fields, the firstName, the lastName, the position, atBats, hits and era. All others will be ignored.

So the Haskell data model looks like this

import Data.Map (Map, fromList, toList)
data Season = Season
    { sYear    :: Int
    , sLeagues :: Leagues
	      deriving (Show, Eq)
type Leagues   = Map String Divisions
type Divisions = Map String [Team]
data Team = Team
    { teamName :: String
    , city     :: String
    , players  :: [Player]
	    deriving (Show, Eq)
data Player = Player
    { firstName :: String
    , lastName  :: String
    , position  :: String
    , atBats    :: Maybe Int
    , hits      :: Maybe Int
    , era       :: Maybe Float
	      deriving (Show, Eq)

3.3 The predefined picklers

In HXT here is a class XmlPickler defining a single function xpickle for overloading the xpickle name.

class XmlPickler a where
    xpickle :: PU a


instance XmlPickler Int where
    xpickle = xpPrim
instance XmlPickler Integer where
    xpickle = xpPrim


    xpickle = xpPair xpickle xpickle
instance XmlPickler a => XmlPickler [a] where
    xpickle = xpList xpickle
instance XmlPickler a => XmlPickler (Maybe a) where
    xpickle = xpOption xpickle

Furthermore we need pickler for generating/reading element and attribute nodes

  • xpElem generates/parses an XML element node
  • xpAttr generates/parses an attribute node


3.4 Constructing the example picklers

For every Haskell type we will define a pickler.

For the own data types we will declare instances of XmlPickler

instance XmlPickler Season where
    xpickle = xpSeason
instance XmlPickler Team where
    xpickle = xpTeam
instance XmlPickler Player where
    xpickle = xpPlayer

Then the picklers are developed top down starting with xpSeason.

xpSeason	:: PU Season
    = xpElem "SEASON" $
      xpWrap ( uncurry Season
      xpPair (xpAttr "YEAR" xpickle) xpLeagues


xpLeagues	:: PU Leagues
    = xpWrap ( fromList
	     , toList ) $
      xpList $
      xpElem "LEAGUE" $
      xpPair (xpAttr "NAME" xpText) xpDivisions


xpDivisions	:: PU Divisions
    = xpWrap ( fromList
	     , toList
	     ) $
      xpList $
      xpElem "DIVISION" $
      xpPair (xpAttr "NAME" xpText) xpickle

The divisions are pickled by the same pattern as the leagues.

xpTeam	:: PU Team
    = xpElem "TEAM" $
      xpWrap ( uncurry3 Team


xpPlayer	:: PU Player
    = xpElem "PLAYER" $
      xpPair (xpTriple (xpAttr "GIVEN_NAME" xpText)
	               (xpAttr "SURNAME"    xpText)
	               (xpAttr "POSITION"   xpText))
             (xpTriple (xpOption (xpAttr "AT_BATS" xpickle))
	               (xpOption (xpAttr "HITS"    xpickle))
	               (xpOption (xpAttr "ERA"     xpPrim )))

The Player pickler looks a bit clumsy. A Player is mapped to an element PLAYER. But because of the many components, six in this case, we wrap a Player value in a pair of triples to use the predefined picklers xpPair and xpTriple. When needing picklers for more than five components, it's straight forward to derive e.g. an 'xp10Tuple`` from the sources of xpTriple and others.



3.5 A simple application

import Text.XML.HXT.Arrow
-- ...
main	:: IO ()
    = do
      runX ( xunpickleDocument xpSeason [ (a_validate,v_0)
					, (a_trace, v_1)
					, (a_remove_whitespace,v_1)
					, (a_preserve_comment, v_0)
					] "simple2.xml"
	     xpickleDocument xpSeason [ (a_indent, v_1)
				      ] "new-simple2.xml"
      return ()
-- the dummy for processing the unpickled data
processSeason	:: IOSArrow Season Season


The unpickled value, when formated a bit, looks like this

      { sYear = 1998
      , sLeagues = fromList
	[ ( "American League"
	  , fromList
	    [ ( "Central"
	      , [ Team { teamName = "White Sox"
		       , city = "Chicago"
		       , players = []}
		, ...
	    , ( "East"
	      , [ Team { teamName = "Orioles"
		       , city = "Baltimore"
		       , players = []}
		, ...
	    , ( "West"
	      , [ Team { teamName = "Angels"
		       , city = "Anaheim"
		       , players = []}
		, ...
	, ( "National League"
	  , fromList
	    [ ( "Central"
	      , [ Team { teamName = "Cubs"
		       , city = "Chicago"
		       , players = []}
		, ...
	    , ( "East"
	      , [ Team { teamName = "Braves"
		       , city = "Atlanta"
		       , players =
			 [ Player { firstName = "Marty"
				  , lastName = "Malloy"
				  , position = "Second Base"
				  , atBats = Just 28
				  , hits = Just 5
				  , era = Nothing}
			 , Player { firstName = "Ozzie"
				  , lastName = "Guillen"
				  , position = "Shortstop"
				  , atBats = Just 264
				  , hits = Just 73
				  , era = Nothing}
			 , ...
		, ...
	    , ( "West"
	      , [ Team { teamName = "Diamondbacks"
		       , city = "Arizona"
		       , players = []}
		, ...

4 Example: A toy programming language


The complete source is part of the HXT distribution.

4.1 The abstract syntax for the toy programming language

type Program	= Stmt
type StmtList	= [Stmt]
data Stmt
    = Assign  Ident  Expr
    | Stmts   StmtList 
    | If      Expr  Stmt (Maybe Stmt)
    | While   Expr  Stmt
      deriving (Eq, Show)
type Ident	= String
data Expr
    = IntConst	Int
    | BoolConst Bool
    | Var       Ident
    | UnExpr	UnOp  Expr
    | BinExpr	Op    Expr  Expr
      deriving (Eq, Show)
data Op
      deriving (Eq, Ord, Enum, Show)
data UnOp
    = UPlus | UMinus | Neg
      deriving (Eq, Ord, Read, Show)


For developing the picklers, there are two new aspects. This example contains data types with variants and it's a recursive structure.

4.2 The pickler definitions

xpProgram :: PU Program
xpProgram = xpElem "program" $
	    xpAddFixedAttr "xmlns" "program42" $
instance XmlPickler UnOp where
    xpickle = xpPrim
instance XmlPickler Op where
    xpickle = xpWrap (toEnum, fromEnum) xpPrim
instance XmlPickler Expr where
    xpickle = xpAlt tag ps
	tag (IntConst _    ) = 0
	tag (BoolConst _   ) = 1
	tag (Var _         ) = 2
	tag (UnExpr _ _    ) = 3
	tag (BinExpr _ _ _ ) = 4
	ps = [ xpWrap ( IntConst
					         xpAttr "value" $
					         xpickle )
	     , xpWrap ( BoolConst
						 xpAttr "value" $
						 xpWrap (toEnum, fromEnum) xpickle )
	     , xpWrap ( Var
						 xpAttr "name"  $
						 xpText )
	     , xpWrap ( uncurry UnExpr
                                               ( xpElem "unex" $
						 xpPair (xpAttr "op" xpickle) xpickle )
	     , xpWrap ( uncurry3 $ BinExpr
                                               ( xpElem "binex" $
						 xpTriple (xpAttr "op" xpickle) xpickle xpickle )
instance XmlPickler Stmt where
    xpickle = xpAlt tag ps
	tag ( Assign _ _ ) = 0
	tag ( Stmts _ )    = 1
	tag ( If _ _ _ )   = 2
	tag ( While _ _ )  = 3
	ps = [ xpWrap ( uncurry Assign
                                               ( xpElem "assign" $
						 xpPair (xpAttr "name" xpText) xpickle )
	     , xpWrap ( Stmts
						 xpList xpickle )
	     , xpWrap ( uncurry3 If
                                               ( xpElem "if" $
						 xpTriple xpickle xpickle xpickle )
	     , xpWrap ( uncurry While
                                               ( xpElem "while" $
						 xpPair xpickle xpickle )




The complete pickler definitions consist of about 60 lines of code.

4.3 A simple program as Haskell value

p2 :: Program
p2 = Stmts		
     [ Assign x (IntConst 6)
     , Assign y (IntConst 7)
     , Assign p (IntConst 0)
     , While
       ( BinExpr Neq (Var x) (IntConst 0) )
	    ( Stmts
	    ( Just ( Stmts
    x = "x"
    y = "y"
    p = "p"

An example program with all variants of statements and expressions.

4.4 The serialized program as XML

<program xmlns="program42">
    <assign name="x">
      <int value="6"/>
    <assign name="y">
      <int value="7"/>
    <assign name="p">
      <int value="0"/>
      <binex op="6">
        <var name="x"/>
        <int value="0"/>
        <binex op="6">
          <binex op="4">
            <var name="x"/>
            <int value="2"/>
          <int value="0"/>
          <assign name="x">
            <binex op="1">
              <var name="x"/>
              <int value="1"/>
          <assign name="p">
            <binex op="0">
              <var name="p"/>
              <var name="y"/>
          <assign name="x">
            <binex op="3">
              <var name="x"/>
              <int value="2"/>
          <assign name="y">
            <binex op="2">
              <var name="y"/>
              <int value="2"/>

This document is generated by executing the following piece of code

storeProgram :: IO ()
  = do
    runX ( constA p2
	   xpickleDocument xpProgram
             [ (a_indent, v_1)
             ] "pickle.xml"
    return ()

It's loaded from a file with

loadProgram :: IO Program
  = do
    [p2] <- runX ( xunpickleDocument xpProgram
                     [ (a_remove_whitespace, v_1)
                     , (a_validate, v_0)
                     ] "pickle.xml"
    return p2


5 A few words of advice

These picklers are a powerful tool for de-/serializing from/to XML. Only a few lines of code are needed for serializing as well as for deserializing. But they are absolutely intolerant when dealing with none valid XML. They are intended to read machine generated XML, ideally generated by the same pickler. When unpickling hand written or by foreign tools generated XML, please validate the XML before reading, preferably with RelaxNG or XML Schema, because of the more powerful type system than those with DTDs.







6.1 Serializing to Xml


import IO
import Char
import Text.XML.HXT.Arrow
import Data.Generics
-- our data class we'll convert into xml
data Config = 
   Config { username :: String,
            logNumDays :: Int,
            oleDbString :: String }
   deriving (Show, Typeable,Data)
-- helper function adapted from
-- (gshow replaced by gshow')
introspectData a = zip fields (gmapQ gshow' a)
    where fields = constrFields $ toConstr a
gshow' :: Data a => a -> String
-- create a config object to serialize:
-- test function, using our Config data type
testConversion = createHxtArrow( createConfig ) ()

-- hughperkins

6.2 Deserializing from Xml

Here's a solution to deserialize a simple haskell data type containing Strings and Ints.

It's not really pretty, but it works.




module ParseXml
import IO
import Char
import List
import Maybe
import Data.Generics hiding (Unit)
import Text.XML.HXT.Arrow hiding (when)
--data Config = Config{ age :: Int } 
createConfig = Config "qsdfqsdf" 3
--createConfig = Config 3
gshow' :: Data a => a -> String
-- helper function from
introspectData a = zip fields (gmapQ gshow' a)
    where fields = constrFields $ toConstr a
-- returns a string
                                | otherwise = Nothing
-- parse templateobject to get list of field names
-- apply these to xml to get list of values
-- return (fieldnames list, value list)
xmlToGShowFormat xml templateobject = 
   where mainconstructorname = (showConstr $ toConstr templateobject)
         fields = constrFields $ toConstr templateobject
         datatypes = gmapQ (dataTypeOf) templateobject
         constrs = gmapQ (toConstr) templateobject
         datatypereps = gmapQ (dataTypeRep . dataTypeOf) templateobject
         fieldtogshowformat (value,datatyperep) = case datatyperep of
            _ -> show(fromJust value)
         formattedfieldlist = map (fieldtogshowformat) (zip values datatypereps)
dotest = xmlDeserialize (xmlSerialize createConfig) createConfig :: Config

-- Uwe Schmidt: This code moved from main HXT