{"action":"created","issue":{"url":"https://api.github.com/repos/inveniosoftware/invenio/issues/2573","labels_url":"https://api.github.com/repos/inveniosoftware/invenio/issues/2573/labels{/name}","comments_url":"https://api.github.com/repos/inveniosoftware/invenio/issues/2573/comments","events_url":"https://api.github.com/repos/inveniosoftware/invenio/issues/2573/events","html_url":"https://github.com/inveniosoftware/invenio/issues/2573","id":50039516,"number":2573,"title":"RFC Centralizing record linking","user":{"login":"kaplun","id":119349,"avatar_url":"https://avatars.githubusercontent.com/u/119349?v=3","gravatar_id":"","url":"https://api.github.com/users/kaplun","html_url":"https://github.com/kaplun","followers_url":"https://api.github.com/users/kaplun/followers","following_url":"https://api.github.com/users/kaplun/following{/other_user}","gists_url":"https://api.github.com/users/kaplun/gists{/gist_id}","starred_url":"https://api.github.com/users/kaplun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kaplun/subscriptions","organizations_url":"https://api.github.com/users/kaplun/orgs","repos_url":"https://api.github.com/users/kaplun/repos","events_url":"https://api.github.com/users/kaplun/events{/privacy}","received_events_url":"https://api.github.com/users/kaplun/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/c_BibAuthorID","name":"c_BibAuthorID","color":"ededed"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/c_BibAuthority","name":"c_BibAuthority","color":"ededed"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/c_BibCheck","name":"c_BibCheck","color":"ededed"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/c_BibEdit","name":"c_BibEdit","color":"ededed"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/c_BibField","name":"c_BibField","color":"ededed"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/c_BibFormat","name":"c_BibFormat","color":"ededed"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/c_BibIndex","name":"c_BibIndex","color":"ededed"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/c_BibMerge","name":"c_BibMerge","color":"ededed"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/c_BibRank","name":"c_BibRank","color":"ededed"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/c_BibRecord","name":"c_BibRecord","color":"ededed"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/c_BibUpload","name":"c_BibUpload","color":"ededed"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/c_formatter","name":"c_formatter","color":"ededed"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/c_WebSearch","name":"c_WebSearch","color":"ededed"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/next","name":"next","color":"d4c5f9"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/p_major","name":"p_major","color":"e11d21"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/RFC","name":"RFC","color":"009800"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/s_CDS","name":"s_CDS","color":"4D94CE"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/s_INSPIRE","name":"s_INSPIRE","color":"207de5"},{"url":"https://api.github.com/repos/inveniosoftware/invenio/labels/t_enhancement","name":"t_enhancement","color":"eb6420"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":23,"created_at":"2014-11-25T15:18:28Z","updated_at":"2015-02-04T17:00:05Z","closed_at":null,"body":"## Current model in Invenio master\r\nLinking among records exist by virtue of dedicated MARC subfields. \r\n\r\n### Citations\r\nE.g. in Invenio master out of the box, this is a reference (as generated by refextract):\r\n```\r\n001329723 999C5 $$0647131$$hC. Quesne and V. M. Tkachuk$$o2$$sJ.Phys.,A37,4267$$y2004\r\n```\r\n`$$0` contains the official citation from record `1329723` to record `647131`. \r\n\r\nThis has the following characteristics:\r\n* This implementation exposes the internal record IDs.\r\n* In case a there is no `$0` the other subfields are passed to search engine (by `BibRank` and by `BibFormat`) in order to try to find at least one matching record (which in case would be stored subsequently in `$0`)\r\n* In order to know which records are cited by the current one, it's possible to use the search engine with `999C50:1329723` or use the citation dicts wich are cached by BibRank.\r\n* When a new record is added in the system which happens to be the one referred by a given citation through the other subfields, the new record has to be searched in `999C5s` (using its `773`) in order to see if some potential citation has to updated, in this case:\r\n```\r\n000647131 773__ $$c4267-4281$$pJ.Phys.$$vA37$$y2004\r\n```\r\n\r\n### Authors\r\nAuthors in `100__` and `700__` have their exact name used to identify the specific MARC field, which is then connected in the `aidPERSONIDPAPERS` table to an author profile.\r\n\r\nThese author profiles are connected to authority MARC records via e.g. an internal ID such as ORCID.\r\n\r\n### Affiliation\r\nIn the INSPIRE use case affiliations, next to authors in a paper, are literally connected with a corresponding institute record (via exact string matching).\r\n\r\n### Photos/Album\r\nIn CDS photos are connected to the parent album in a way that photos metadata are agumented with album metadata at indexing time.\r\n\r\n### Other links\r\nStill from the INSPIRE example `773__0` is used to link a book chapter record to the corresponding book record.\r\n`78502w` and `78708w` are used to explain  the superseed/superseeded relation.\r\n\r\n## Issues with the current model\r\n* There is not 1 model but several. Each type of link had to implement a dedicated daemon to enforce consistency, each type of link is manipulated in a dedicated way and indexed/cached in a dedicated way\r\n* By explicitly writing in the metadata the recid of the linked record, in case the latter is merged into another one, the latter has to be updated\r\n* It's not possible to express linking with a given level of confidence (e.g. imagine when a link is not made in an authoritative way but only guessed).\r\n\r\n# Proposal\r\nCentralize the control of links by having a new table (or SQLAlchemy Model) with the form:\r\n\r\nuuid | from | link type | to | confidence\r\n------|------|------|-----|-----\r\n1234-5678-1234-5678 | 123 | refersto | 456 | 0.9\r\n2345-1234-1243-1245 | 234 | authoredby | 567 | 1.0\r\n2345-1234-1243-1245 | 234 | affiliatedat | 678 | 0.5\r\n\r\n* The `uuid` would be associated with a given `recjson` field upon ingestion by the uploader workflow, for all those fields that have an outgoing link (for example each author field within a record would receive its own uuid, the same would happen for each reference).\r\n* Ideally, only relations from records representing static entities (e.g. papers) to dynamic one (e.g. people or institutions) should be stored, while the opposite direction should be calculated (and cached) at runtime. In this way we would avoid inconsistencies and continuous swarm of updates.\r\n* The confidence level could be used to store an authoritative link (1.0) or a weaker one (e.g. when it was guessed by a machine learning algorithm or by a fuzzy search). Ideally storing 0 would also have the value of ground truth.\r\n* This table could be used to populate *calculated fields* in recjson, where the recid is stored at runtime when the record is sent to be indexed to ElasticSearch.\r\n* When a relation is coming at input time (e.g. when an incoming paper, states that the author *foo* is really the one of record 123) this relation will be interpreted by a given recjson imported and written into the above relation table (a `uuid` is generated on the fly).\r\n* When a record is modified the `uuid` should be preserved, when the modification is not about changing a given link.\r\n* A new generic widget could be used that curators directly edit specific subset of this table (e.g. all the relations of type `refersto` for record 123. Such a widget could let the curator assign external records for not resolved relations by allowing for autocompletion etc.\r\n* The new bibrank for citation would simply look at this table\r\n* The new bibauthorid would manipulate this table\r\n* The new indexer based on elastic search would consume this table trough recjson\r\n* The new formatter would use this table to create records link\r\n\r\nThis table might grow over time but, thanks to SQLAlchemy transparent sharding it could be sharded WRT the `link_type` property, thus mapping to several specific tables.\r\n\r\n### Side note\r\nConnected with this RFC is the question whether we should still identify records by integers or rather switch to use more generic `uuid` and namespace them e.g. to use the same tool to handle not only bibliographic records but also annotations, comments, documents, library items... This should be the subject of a new RFC."},"comment":{"url":"https://api.github.com/repos/inveniosoftware/invenio/issues/comments/72893056","html_url":"https://github.com/inveniosoftware/invenio/issues/2573#issuecomment-72893056","issue_url":"https://api.github.com/repos/inveniosoftware/invenio/issues/2573","id":72893056,"user":{"login":"tiborsimko","id":517546,"avatar_url":"https://avatars.githubusercontent.com/u/517546?v=3","gravatar_id":"","url":"https://api.github.com/users/tiborsimko","html_url":"https://github.com/tiborsimko","followers_url":"https://api.github.com/users/tiborsimko/followers","following_url":"https://api.github.com/users/tiborsimko/following{/other_user}","gists_url":"https://api.github.com/users/tiborsimko/gists{/gist_id}","starred_url":"https://api.github.com/users/tiborsimko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tiborsimko/subscriptions","organizations_url":"https://api.github.com/users/tiborsimko/orgs","repos_url":"https://api.github.com/users/tiborsimko/repos","events_url":"https://api.github.com/users/tiborsimko/events{/privacy}","received_events_url":"https://api.github.com/users/tiborsimko/received_events","type":"User","site_admin":false},"created_at":"2015-02-04T17:00:05Z","updated_at":"2015-02-04T17:00:05Z","body":"Here is promised summary of my comments, most of which we discussed\r\nIRL on Monday.\r\n\r\n**WRT** use cases, there are some more, e.g. the linking in Periodicals or\r\namong Institutes, e.g. several other kinds of inter-record linking in\r\nCERN Open Data portal, such as \"constituent unit entry\", tag 774.  See\r\n[open data JSON MARC configuration](https://github.com/cernopendata/opendata.cern.ch/blob/master/invenio_opendata/base/recordext/fields/opendata.cfg).\r\n\r\n**WRT** proposed table, we may need an extra column storing properties of\r\nthe relation.  Say record R1 is a photo that was used as B/W on page 7\r\nof record R2, and as RGB on page 2 of record R3.  Where to store this\r\ninformation?  It logically belongs with the relation, not with the\r\nrecords themselves.  Hence perhaps a new PostgreSQL JSON column\r\nrepresenting abstract key-value properties store, where people can\r\nstore anything they need to describe the properties of the\r\nrelationship.  (Kind of like \"more info\" store in invenio/master.)\r\n\r\n(**WRT** various types of relationships, looking at all the various\r\nsubfields existing in the MARC relational tags shows that this\r\nproperty store can become rather big.  Hence perhaps a question\r\nwhether an abstract column would be enough or whether we need some\r\nfurther structure?  I guess a generic JSON column might be enough,\r\nsince [it is usually fast](http://simko.home.cern.ch/simko/postgresql-mongodb-json-select-speed.html) to search there, however the plethora of \r\nrelationship properties may require some deeper thoughts.)\r\n\r\n**WRT** records vs fields as first-class linkable citizens, wouldn't it be\r\nnice to use UUIDs for almost everything, and allow linking on any part\r\nof JSON nested level.  (See @aw-bib's example of inter-field linking\r\ninside the same MARC record.)  Kind of decorate parts of JSON in a LD\r\nstyle, meaning \"factor this tree out for me into a new record in\r\nanother JSON store and link them together\".  This would be great for\r\nlow-level system users.\r\n\r\n(If we make also fields linkable, then this mechanism could be used\r\nfor poor-man's authority record control, e.g. field representing\r\n\"arXiv subject category\" has only ~10 values (hep-th, hep-ph, etc) and\r\nso could just as well live as a link to this \"JSON based knowledge\r\nbase\", so to speak.  When do we use \"by-reference linking\" and where\r\ndo we stop and copy values and use \"multi-edit\" to manage the linking\r\ninformation?)\r\n\r\n**WRT** possibly representing relation information in record's \"master\r\nformat\" data, we have several use cases: (1) developers or system\r\nlibrarians are working with JSON all the way as much as possible; (2)\r\ncataloguers and curators might use tools the developers write, or\r\nmight prefer to work with \"native aster format\" such as MARC or\r\nUNIMARC or EAD that is in use on the given concrete Invenio\r\ninstallation; (3) general public will be served any kind of\r\n\"serialised version\" of the record, in read-only manner, so this is\r\nless critical.  My concern is about the second category, where we may\r\nneed to offer read-write like access to non-JSON formats.\r\n\r\nE.g. imagine a site receiving MARC authority records from a third\r\nparty; if they want to enrich the information, and send it back, and\r\nif they keep receiving and sending updates to these records, both from\r\nthird party and from authors, say, then do we want to represent the\r\nrelationship information only in the JSON or also in this \"editable\r\nnative format\" that cataloguers are working with?  If we speak about\r\nan Invenio installation that runs with maybe half an IT person and\r\nfive cataloguer persons, using some MS Windows EAD editor or\r\nsomething, and constantly receiving MARC authority records from the\r\nNational Library, the answer would be yes.  Hence, if we represent\r\nlinked information in these \"native editable formats\", we should make\r\nthings really lossless, which can go wild.  Would cataloguers agree to\r\nsee and edit UUIDs?  Would we hide them semi-transparently somehow?\r\n\r\nIOW, would we aim at lossless representation of the relationship\r\ninformation also in the \"native editable format\" the concrete site may\r\nuse (or not), which can get quite complex just looking at generating\r\nMARC out of the relationship property store?  And if we do, would we\r\nwant to represent it in a nice editable manner, not exposing internal\r\nUUIDs too much?\r\n\r\n**WRT** using a graph database, this should be definitely seriously\r\nconsidered...\r\n\r\n[raw brain dump]"}}