{"action":"created","issue":{"url":"https://api.github.com/repos/matthewhorridge/binaryowl/issues/6","labels_url":"https://api.github.com/repos/matthewhorridge/binaryowl/issues/6/labels{/name}","comments_url":"https://api.github.com/repos/matthewhorridge/binaryowl/issues/6/comments","events_url":"https://api.github.com/repos/matthewhorridge/binaryowl/issues/6/events","html_url":"https://github.com/matthewhorridge/binaryowl/issues/6","id":99954586,"number":6,"title":"Binary OWL vs. compression: some analysis","user":{"login":"sesuncedu","id":411675,"avatar_url":"https://avatars.githubusercontent.com/u/411675?v=3","gravatar_id":"","url":"https://api.github.com/users/sesuncedu","html_url":"https://github.com/sesuncedu","followers_url":"https://api.github.com/users/sesuncedu/followers","following_url":"https://api.github.com/users/sesuncedu/following{/other_user}","gists_url":"https://api.github.com/users/sesuncedu/gists{/gist_id}","starred_url":"https://api.github.com/users/sesuncedu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sesuncedu/subscriptions","organizations_url":"https://api.github.com/users/sesuncedu/orgs","repos_url":"https://api.github.com/users/sesuncedu/repos","events_url":"https://api.github.com/users/sesuncedu/events{/privacy}","received_events_url":"https://api.github.com/users/sesuncedu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-08-10T02:16:43Z","updated_at":"2015-08-13T22:13:00Z","closed_at":null,"body":"Files encoded using Binary OWL show really poor compression ratios with gzip and xz.  \r\n\r\nFor example, for the Gene Ontology, \r\nuncompressed  FSS uses *58M*\r\nuncompressed binowl uses  **28M**\r\n\r\nBut: \r\n\r\ngzipped FSS uses *5.3M*; xz uses *3.1M*. \r\ngzipped binowl uses **8.8M**;  xz uses **4.4M** \r\n\r\nSince the xz compressed FSS shrinks to *2.7M*  is the file is lexically sorted, it seems possible that changing the order in which axioms are written by binary owl may have a salutary effect on compression ratios.  \r\n\r\nThis seems to be the case:\r\n1. Sorting and renumbering the IRI table prior to output reduces the xz file to **4.3M**. \r\n2. Sorting axioms within each axiom type grouping  reduces the gz size to **4.8M**, and xz size to **2.8M**. \r\n\r\nBetter results may be possible if the previous approach of literal interning is revived; this would separate strings from the axiom structure, allowing for both to be optimized. \r\n\r\nThere may be better ordering than the standard owlapi sorts ; e.g. axioms associated with sibling classes may have similar patterns. \r\n \r\nThe encoding format may benefit from using variable length integer encodings, and entity/literal references may benefit from delta-encoding.  These changes can benefit uncompressed size and performance.   \r\n\r\nInstead of just using the most recent iri or literal id, delta coding could be made relative to a small number of most recent ids (e.g. four most recent ids). This may create longer matches.   \r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/matthewhorridge/binaryowl/issues/comments/130863684","html_url":"https://github.com/matthewhorridge/binaryowl/issues/6#issuecomment-130863684","issue_url":"https://api.github.com/repos/matthewhorridge/binaryowl/issues/6","id":130863684,"user":{"login":"sesuncedu","id":411675,"avatar_url":"https://avatars.githubusercontent.com/u/411675?v=3","gravatar_id":"","url":"https://api.github.com/users/sesuncedu","html_url":"https://github.com/sesuncedu","followers_url":"https://api.github.com/users/sesuncedu/followers","following_url":"https://api.github.com/users/sesuncedu/following{/other_user}","gists_url":"https://api.github.com/users/sesuncedu/gists{/gist_id}","starred_url":"https://api.github.com/users/sesuncedu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sesuncedu/subscriptions","organizations_url":"https://api.github.com/users/sesuncedu/orgs","repos_url":"https://api.github.com/users/sesuncedu/repos","events_url":"https://api.github.com/users/sesuncedu/events{/privacy}","received_events_url":"https://api.github.com/users/sesuncedu/received_events","type":"User","site_admin":false},"created_at":"2015-08-13T22:13:00Z","updated_at":"2015-08-13T22:13:00Z","body":"A few  more notes:\r\n\r\n Sorting subclassof axioms by superclass then subclass saves, and sorting annotation assertion axioms by literal-id, then property, then subject, saves about 3% off compressed size. \r\n\r\nWriting coded IRI deltas  as variable length integers, with delta base in the lower bits, and no explicit length word, reduces the uncompressed size by about 5%, but increases the compressed size slightly. This is probably due to alignment issues. "}}