{"action":"created","issue":{"url":"https://api.github.com/repos/google/cayley/issues/212","labels_url":"https://api.github.com/repos/google/cayley/issues/212/labels{/name}","comments_url":"https://api.github.com/repos/google/cayley/issues/212/comments","events_url":"https://api.github.com/repos/google/cayley/issues/212/events","html_url":"https://github.com/google/cayley/pull/212","id":57405673,"number":212,"title":"Add support for string collation","user":{"login":"LAlbertalli","id":3062008,"avatar_url":"https://avatars.githubusercontent.com/u/3062008?v=3","gravatar_id":"","url":"https://api.github.com/users/LAlbertalli","html_url":"https://github.com/LAlbertalli","followers_url":"https://api.github.com/users/LAlbertalli/followers","following_url":"https://api.github.com/users/LAlbertalli/following{/other_user}","gists_url":"https://api.github.com/users/LAlbertalli/gists{/gist_id}","starred_url":"https://api.github.com/users/LAlbertalli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LAlbertalli/subscriptions","organizations_url":"https://api.github.com/users/LAlbertalli/orgs","repos_url":"https://api.github.com/users/LAlbertalli/repos","events_url":"https://api.github.com/users/LAlbertalli/events{/privacy}","received_events_url":"https://api.github.com/users/LAlbertalli/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-02-12T01:43:20Z","updated_at":"2015-02-21T04:36:59Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/google/cayley/pulls/212","html_url":"https://github.com/google/cayley/pull/212","diff_url":"https://github.com/google/cayley/pull/212.diff","patch_url":"https://github.com/google/cayley/pull/212.patch"},"body":"This is just an initial pull requests to implement the support for string collation (somewhat similar to what exists in SQL Databases) that address the issue #200. It is just a starting point, I have some doubts that should be addressed.\r\nThe collation is supported only on BoltDB for now but the implementation on other storage engine should be easy.\r\n\r\n## Basic idea\r\nRight now the indexes to search in the database use the hash of the string. In order to support the string collation, before hashing the stream, the string is converted to its comparison key using the used collator. If no collation is specified the storage engine works as before.\r\n\r\n## Configuration\r\nIn order to configure the collation the following options have been added to the configuration file:\r\n#### **`collation_type`**\r\n\r\n  * Type: String\r\n  * Default: \"\"\r\n\r\n  The collation to be used for string comparison by the database. The string should be a valid BCP 47 language tag. If empty no collation is used.\r\n\r\n#### **`collation_options`**\r\n\r\n  * Type: List of String\r\n  * Default: []\r\n\r\n  A list of option tag to be applied to the selected collation. Valid options are: \"IgnoreCase\", \"IgnoreDiacritics\", \"IgnoreWidth\", \"Loose\", \"Force\", \"Numeric\"\r\n\r\nThe same options are available on the command line with `-collation_type` and `-collation_options`. The later is a comma separated string.\r\n\r\n## Open questions\r\n\r\n* Right now every string is collated. A proper implementation should implement a different behavior based on the type of the object as defined by RDF standard (http://www.w3.org/TR/n-quads/#n-quads-language):\r\n  * IRI should not be collated\r\n  * string literals should be collated according to the specified language or the default if it is not defined.\r\n  * string that represents a different datatype should not be collated\r\n\r\n  I'm not happy with this level of complication because it would complicate the query. A differentiation between IRIs (that should NOT be collated) and string (to be collated) could be interesting even if I think that right now the parser drops the the angular bracket so more work is required.\r\n* If the db is opened with a different collation there's no error but the behavior could be unexpected. Probably the collation should be stored in the db and checked open start.\r\n* The Collator is now implemented in the graph package but should be probably moved in a different package.\r\n* Only one collation is supported right now. Multiple collation support (e.g: specifying it at connection level) could be used but it should be properly designed and I don't see a use case."},"comment":{"url":"https://api.github.com/repos/google/cayley/issues/comments/75355494","html_url":"https://github.com/google/cayley/pull/212#issuecomment-75355494","issue_url":"https://api.github.com/repos/google/cayley/issues/212","id":75355494,"user":{"login":"barakmich","id":98356,"avatar_url":"https://avatars.githubusercontent.com/u/98356?v=3","gravatar_id":"","url":"https://api.github.com/users/barakmich","html_url":"https://github.com/barakmich","followers_url":"https://api.github.com/users/barakmich/followers","following_url":"https://api.github.com/users/barakmich/following{/other_user}","gists_url":"https://api.github.com/users/barakmich/gists{/gist_id}","starred_url":"https://api.github.com/users/barakmich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/barakmich/subscriptions","organizations_url":"https://api.github.com/users/barakmich/orgs","repos_url":"https://api.github.com/users/barakmich/repos","events_url":"https://api.github.com/users/barakmich/events{/privacy}","received_events_url":"https://api.github.com/users/barakmich/received_events","type":"User","site_admin":false},"created_at":"2015-02-21T04:36:59Z","updated_at":"2015-02-21T04:36:59Z","body":"I talked this PR over with some friends of mine from the Freebase world. (I do take it very seriously)\n\nWhile I still think that a textual index would be cool (and worth the growth in size) there's another place you could hook into the Extract, Transform, and Load.... In the Transform, instead of the Load.\n\nRight now we parse quads either strictly or non-strictly, but at the end of the day, these are functions from file to list of quads. \n\nYou could get equivalent behavior to this PR, without intentional loss of data or strange indexing, and without an extra index. Essentially, you could imagine functions which take sets of quads, and transform them into other sets of quads. Think  `io.Reader` wrapped in `bufio`\n\nCreating a Transform interface (name of your choice) which takes a list of quads and returns a list of quads -- before they get written -- would be kind of nice. In this case, the transform is simple; lowercase/canonicalize the strings (here you could also do appropriate things for RDF types, as you mention). More advanced versions of this could reconcile data, pull related data, deduplicate data... or any number of things.\n\nYou could also chain these, which simplifies some of the fanout of options you have.\n\nIndexing progresses as normal.\n\nWhat do you think?"}}