{"action":"created","issue":{"url":"https://api.github.com/repos/martinblech/xmltodict/issues/104","labels_url":"https://api.github.com/repos/martinblech/xmltodict/issues/104/labels{/name}","comments_url":"https://api.github.com/repos/martinblech/xmltodict/issues/104/comments","events_url":"https://api.github.com/repos/martinblech/xmltodict/issues/104/events","html_url":"https://github.com/martinblech/xmltodict/pull/104","id":78708068,"number":104,"title":"Several Enhancements","user":{"login":"jonlooney","id":12263474,"avatar_url":"https://avatars.githubusercontent.com/u/12263474?v=3","gravatar_id":"","url":"https://api.github.com/users/jonlooney","html_url":"https://github.com/jonlooney","followers_url":"https://api.github.com/users/jonlooney/followers","following_url":"https://api.github.com/users/jonlooney/following{/other_user}","gists_url":"https://api.github.com/users/jonlooney/gists{/gist_id}","starred_url":"https://api.github.com/users/jonlooney/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonlooney/subscriptions","organizations_url":"https://api.github.com/users/jonlooney/orgs","repos_url":"https://api.github.com/users/jonlooney/repos","events_url":"https://api.github.com/users/jonlooney/events{/privacy}","received_events_url":"https://api.github.com/users/jonlooney/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-05-20T21:37:52Z","updated_at":"2015-05-23T19:28:46Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/martinblech/xmltodict/pulls/104","html_url":"https://github.com/martinblech/xmltodict/pull/104","diff_url":"https://github.com/martinblech/xmltodict/pull/104.diff","patch_url":"https://github.com/martinblech/xmltodict/pull/104.patch"},"body":"I implemented several enhancements:\r\n* The ability to create dictionaries of elements based on known index-key elements.\r\n* The ability to force lists for certain tags.\r\n* The ability to strip namespaces from results.\r\n* The ability to receive data in new data structures that separate the XML attributes from the data.\r\n* The ability to parse ElementTree data into a dictionary.\r\n* The ability to use new parsing classes that keep track of default options.\r\n* The ability to use an iterator in streaming mode.\r\n* Whitespace stripping now applies to data in streaming mode.\r\n\r\nWhile implementing the enhancements, I took care not to disturb the default behavior. Therefore, all the changes should not impact existing users.\r\n\r\nAlso, all the changes have unit tests that cover them. (The code coverage is > 90%. Most of the misses are in lines of code that are meant to handle variations on the Element/ElementTree objects, depending on which library created them.)\r\n\r\n# Index Keys\r\nImagine you have this input:\r\n```xml\r\n<servers>\r\n    <server>\r\n        <name>server1</name>\r\n        <os>Linux</os>\r\n    </server>\r\n    <server>\r\n        <name>server2</name>\r\n        <os>Windows</os>\r\n    </server>\r\n</servers>\r\n```\r\nIn this case, it might be helpful to have the 'servers' dictionary keyed off of the server name. You can now do this using the index_keys option. With this option, the named tags will be \"promoted\" to be the key for their subtree.\r\n\r\nSo, for example, the index_keys=('name',) option will produce this data structure:\r\n```python\r\n{u'servers': {u'server1': {u'name': u'server1',\r\n                           u'os': u'Linux'},\r\n              u'server2': {u'name': u'server2',\r\n                           u'os': u'Windows'}}}\r\n```\r\nBut, what if you need the \"server\" tag because it is intermixed with other tags? In that case, you can turn off the \"index_keys_compress\" option.\r\n\r\nFor example:\r\n```python\r\n>>> xmltodict.parse(\"\"\"\r\n... <devices>\r\n...     <server>\r\n...         <name>server1</name>\r\n...         <os>Linux</os>\r\n...     </server>\r\n...     <server>\r\n...         <name>server2</name>\r\n...         <os>Windows</os>\r\n...     </server>\r\n...     <workstation>\r\n...         <name>host1</name>\r\n...         <os>Linux</os>\r\n...     </workstation>\r\n...     <workstation>\r\n...         <name>host2</name>\r\n...         <os>Windows</os>\r\n...     </workstation>\r\n... </devices>\r\n... \"\"\", new_style=True, index_keys=('name',), index_keys_compress=False).prettyprint(width=2)\r\n{u'devices': {u'server': {u'server1': {u'name': u'server1',\r\n                                       u'os': u'Linux'},\r\n                          u'server2': {u'name': u'server2',\r\n                                       u'os': u'Windows'}},\r\n              u'workstation': {u'host1': {u'name': u'host1',\r\n                                          u'os': u'Linux'},\r\n                               u'host2': {u'name': u'host2',\r\n                                          u'os': u'Windows'}}}}\r\n```\r\n# Force Lists\r\nSometimes, you have a node that may have one or more items. Rather than testing for both a list and single item, you can simplify your code by having the xmltodict parser always create a list for you.\r\n\r\nFor example, compare these outputs:\r\n```python\r\n>>> xmltodict.parse(\"\"\"\r\n... <servers>\r\n...     <server>\r\n...         <name>server1</name>\r\n...         <os>Linux</os>\r\n...     </server>\r\n... </servers>\r\n... \"\"\", new_style=True).prettyprint(width=2) \r\n{u'servers': {u'server': {u'name': u'server1',\r\n                          u'os': u'Linux'}}}\r\n>>> xmltodict.parse(\"\"\"\r\n... <servers>\r\n...     <server>\r\n...         <name>server1</name>\r\n...         <os>Linux</os>\r\n...     </server>\r\n...     <server>\r\n...         <name>server2</name>\r\n...         <os>Windows</os>\r\n...     </server>\r\n... </servers>\r\n... \"\"\", new_style=True).prettyprint(width=2) \r\n{u'servers': {u'server': [{u'name': u'server1',\r\n                           u'os': u'Linux'},\r\n                          {u'name': u'server2',\r\n                           u'os': u'Windows'}]}}\r\n```\r\nIn the first case rv['servers']['server'] points to a single item. In the second case, rv['servers']['server'] points to a list.\r\n\r\nYou can force this to always be a list by setting the \"force_list\" parameter:\r\n```python\r\n>>> xmltodict.parse(\"\"\"\r\n... <servers>\r\n...     <server>\r\n...         <name>server1</name>\r\n...         <os>Linux</os>\r\n...     </server>\r\n... </servers>\r\n... \"\"\", new_style=True, force_list=('server',)).prettyprint(width=2)\r\n{u'servers': {u'server': [{u'name': u'server1',\r\n                           u'os': u'Linux'}]}}\r\n```\r\n\r\n# Strip Namespaces\r\nLet me start by granting the truth that XML namespaces are an essential part of node and attribute names. Now, having said that, there are times when the namespaces are already well-known and are merely extra information that a user can (and will try to) safely ignore. In these cases, you can set the \"strip_namespace\" option to strip namespaces.\r\n\r\nFor example:\r\n```python\r\n>>> xmltodict.parse(\"\"\"\r\n... <servers xmlns=\"http://a.com/\" xmlns:b=\"http://b.com/\">\r\n...     <b:server>\r\n...         <name>test</name>\r\n...     </b:server>\r\n... </servers>\r\n... \"\"\", new_style=True, strip_namespace=True).prettyprint(width=2)\r\n{u'servers': {u'server': {u'name': u'test'}}}\r\n```\r\n\r\n# New Classes\r\nOne of the difficulties in dealing with XML data in Python is representing the richness of the XML data (including, especially, the dual layers of attributes and data) while creating the simplest data structure possible. I'm sure many people have tried. I tried again.\r\n\r\nThe fundamental premise here is this: if a user cares about an XML attribute, he/she knows to go looking for it. So, it is most important to present the main data in a simple format, and it is sufficient to provide one or more methods for users to find XML attributes.\r\n\r\nThe three data structures are:\r\n* XMLCDATANode: This \"quacks\" like a string/unicode. (For example, XMLCDATANode(\"a\") == \"a\" will evaluate to true.)\r\n* XMLDictNode: This \"quacks\" like a dict or OrderedDict.\r\n* XMLListNode: This \"quacks\" like a list.\r\n\r\nThese data structures have some extra methods to deal with attributes:\r\n* hasXMLattributes(): Returns True if there are XML attributes; False otherwise.\r\n* getXMLattribute(name[, default]): Returns the value of the XML attribute if it exists. Otherwise, it will return the default, if given, or raise a KeyError.\r\n* setXMLattribute(name, value): Sets the value of the XML attribute.\r\n* delXMLattribute(name): Delete the XML attribute. Raises a KeyError if the XML attribute does not exists.\r\n* getXMLattributes(): Returns the dictionary of XML attributes.\r\n\r\nThese data structures also implement a prettyprint() method which takes the same options as pprint() (except, of course, for the object to be printed). The prettyprint() method prints out the __data__ only and does not show the XML attributes. This decision was made for readability purposes. The __repr__() method shows both.\r\n\r\nI've already shown some examples of the new classes above. Here's another example:\r\n```python\r\n>>> rv = xmltodict.parse(\"\"\"\r\n... <servers>\r\n...     <server coolness=\"high\">\r\n...         <name>server1</name>\r\n...     </server>\r\n... </servers>\r\n... \"\"\", new_style=True)\r\n>>> repr(rv)\r\n\"XMLDictNode(XMLattrs=OrderedDict(), value=OrderedDict([(u'servers', XMLDictNode(XMLattrs=OrderedDict(), value=OrderedDict([(u'server', XMLDictNode(XMLattrs=OrderedDict([(u'coolness', u'high')]), value=OrderedDict([(u'name', XMLCDATANode(XMLattrs=OrderedDict(), value=u'server1'))])))])))]))\"\r\n>>> rv.prettyprint(width=2)\r\n{u'servers': {u'server': {u'name': u'server1'}}}\r\n>>> rv.hasXMLattributes()\r\nFalse\r\n>>> rv['servers'].hasXMLattributes()\r\nFalse\r\n>>> rv['servers']['server'].hasXMLattributes()\r\nTrue\r\n>>> rv['servers']['server'].getXMLattributes()\r\nOrderedDict([(u'coolness', u'high')])\r\n>>> rv['servers']['server'].getXMLattribute('coolness')\r\nu'high'\r\n>>> rv['servers']['server'].getXMLattribute('darkness', '@@NOTFOUND@@')\r\n'@@NOTFOUND@@'\r\n>>> rv['servers']['server']['name']\r\nXMLCDATANode(XMLattrs=OrderedDict(), value=u'server1')\r\n>>> rv['servers']['server']['name'] == 'server1'\r\nTrue\r\n>>> rv['servers']['server'].setXMLattribute('coolness', 'low')\r\n>>> rv['servers'].setXMLattribute('length', '1')\r\n>>> rv['servers'].setXMLattribute('delete_me', True)\r\n>>> rv['servers'].delXMLattribute('delete_me')\r\n>>> xmltodict.unparse(rv)\r\nu'<?xml version=\"1.0\" encoding=\"utf-8\"?>\\n<servers length=\"1\"><server coolness=\"low\"><name>server1</name></server></servers>'\r\n```\r\n\r\n# Parsing ElementTree Data\r\nSometimes, a user may use a library that returns an Element or ElementTree. In those cases, it would be useful to be able to convert it into an easy to use dictionary without having to first convert it to text. In those cases, the user can use the parse_lxml() method. (This was originally intended for lxml; hence, the name. However, it should work with ElementTree, as well. Indeed, I did much of my testing with cElementTree.)\r\n\r\nThe parse_lxml() method should take the same options as the parse() method.\r\n\r\nExample:\r\n```python\r\n>>> xml = etree.XML(\"<a><b>data</b></a>\")\r\n>>> xmltodict.parse_lxml(xml, new_style=True).prettyprint()\r\n{'a': {'b': u'data'}}\r\n```\r\n\r\n# Parsing Classes\r\nTwo new classes hold parsing defaults. They can be overridden on each invocation.\r\n\r\n* The Parser() class is used for parsing XML text.\r\n* The LXMLParser() class is used for parsing ElementTree objects.\r\n\r\nExample:\r\n```python\r\n>>> parser = xmltodict.Parser(new_style=True, index_keys=('name',))\r\n>>> parser(\"<a><b><name>item1</name></b></a>\").prettyprint()\r\n{u'a': {u'item1': {u'name': u'item1'}}}\r\n>>> parser(\"<a><b><name>item1</name></b></a>\", index_keys=()).prettyprint()\r\n{u'a': {u'b': {u'name': u'item1'}}}\r\n>>> parser(\"<a><b><name>item1</name></b></a>\", index_keys_compress=False).prettyprint()\r\n{u'a': {u'b': {u'item1': {u'name': u'item1'}}}}\r\n```\r\n# Iterators/Generators\r\nIn streaming mode, you can now use an iterator/generator to loop through the list of matching items. In all by Jython, this will be done with incremental parsing of the input file. The input file is processed 1KB at a time and then each matching node is returned on a subsequent iteration. Once all the matching nodes from the first 1KB are returned, the next 1KB is read (and so on).\r\n\r\n(Note: I'm not sure why, but the Travis CI test shows that Jython is failing the unit test that checks to make sure that the parsing really is done incrementally.)\r\n\r\nIf the `generator` argument evaluates to True and `item_depth` is non-zero, the parser will return an iterator. On each iteration, the code will return the next (path, item) tuple at the `item_depth` level. These are the same items (in the same format) that would be passed to the callback function; however, they are returned at each iteration.\r\n\r\nTwo corner cases: If `generator` is True and `item_depth` is zero, the code will return a single-item list with an empty path and the full document. If `generator` is True and `item_callback` is also set, the `item_callback` will be executed for each iteration prior to the iterator's return.\r\n\r\nExample:\r\n```python\r\n>>> xml = \"\"\"\\\r\n... <a prop=\"x\">\r\n...   <b>1</b>\r\n...   <b>2</b>\r\n... </a>\"\"\"\r\n>>> for (path, item) in xmltodict.parse(xml, generator=True, item_depth=2):\r\n...     print 'path:%s item:%s' % (path, item)\r\n...\r\npath:[(u'a', {u'prop': u'x'}), (u'b', None)] item:1\r\npath:[(u'a', {u'prop': u'x'}), (u'b', None)] item:2\r\n```\r\n\r\n# Whitespace Stripping Enhancement\r\nIn streaming mode, whitespace stripping now applies to streaming mode. Previously, if the item at the `item_depth` was a CDATA node, whitespace stripping was not applied prior to the item being sent to the callback function. Now, whitespace stripping takes effect prior to the call to the callback function (or return of a value from the iterator).\r\n\r\n(Whitespace stripping is still controlled by the `strip_whitespace` argument.)\r\n\r\nExample of the previous behavior:\r\n```python\r\n>>> print xml2\r\n<a>\r\n  <b>\r\n    <c>data1</c>\r\n    <d>data2</d>\r\n    <c>data3</c>\r\n  </b>\r\n  <b>\r\n    <c>data4</c>\r\n    <d>data5</d>\r\n    <c>data6</c>\r\n  </b>\r\n</a>\r\n\r\n>>> for (stack, value) in xmltodict.parse(xml2, generator=True, item_depth=3, item_callback=cb):\r\n...     print \"\\tValue: %r\" % value\r\n... \r\n\tValue: u'\\n  \\n    data1'\r\n\tValue: u'\\n    data2'\r\n\tValue: u'\\n    data3'\r\n\tValue: u'\\n  \\n    data4'\r\n\tValue: u'\\n    data5'\r\n\tValue: u'\\n    data6'\r\n```\r\nExample of the new behavior:\r\n```python\r\n>>> for (stack, value) in xmltodict.parse(xml2, generator=True, item_depth=3):\r\n...     print \"\\tValue: %r\" % value\r\n... \r\n\tValue: u'data1'\r\n\tValue: u'data2'\r\n\tValue: u'data3'\r\n\tValue: u'data4'\r\n\tValue: u'data5'\r\n\tValue: u'data6'\r\n```"},"comment":{"url":"https://api.github.com/repos/martinblech/xmltodict/issues/comments/104937704","html_url":"https://github.com/martinblech/xmltodict/pull/104#issuecomment-104937704","issue_url":"https://api.github.com/repos/martinblech/xmltodict/issues/104","id":104937704,"user":{"login":"jonlooney","id":12263474,"avatar_url":"https://avatars.githubusercontent.com/u/12263474?v=3","gravatar_id":"","url":"https://api.github.com/users/jonlooney","html_url":"https://github.com/jonlooney","followers_url":"https://api.github.com/users/jonlooney/followers","following_url":"https://api.github.com/users/jonlooney/following{/other_user}","gists_url":"https://api.github.com/users/jonlooney/gists{/gist_id}","starred_url":"https://api.github.com/users/jonlooney/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonlooney/subscriptions","organizations_url":"https://api.github.com/users/jonlooney/orgs","repos_url":"https://api.github.com/users/jonlooney/repos","events_url":"https://api.github.com/users/jonlooney/events{/privacy}","received_events_url":"https://api.github.com/users/jonlooney/received_events","type":"User","site_admin":false},"created_at":"2015-05-23T19:28:46Z","updated_at":"2015-05-23T19:28:46Z","body":"I added support for an iterator/generator in streaming mode. This seems to fulfill the request from #88. I updated my original comment to explain the new feature."}}