{"action":"created","issue":{"url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/101","labels_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/101/labels{/name}","comments_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/101/comments","events_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/101/events","html_url":"https://github.com/infolab-csail/WikipediaBase/issues/101","id":120070760,"number":101,"title":"support three levels of symbol-synonym priority (\"well known\" / sort-symbols-named)","user":{"login":"sfelshin","id":245779,"avatar_url":"https://avatars.githubusercontent.com/u/245779?v=3","gravatar_id":"","url":"https://api.github.com/users/sfelshin","html_url":"https://github.com/sfelshin","followers_url":"https://api.github.com/users/sfelshin/followers","following_url":"https://api.github.com/users/sfelshin/following{/other_user}","gists_url":"https://api.github.com/users/sfelshin/gists{/gist_id}","starred_url":"https://api.github.com/users/sfelshin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sfelshin/subscriptions","organizations_url":"https://api.github.com/users/sfelshin/orgs","repos_url":"https://api.github.com/users/sfelshin/repos","events_url":"https://api.github.com/users/sfelshin/events{/privacy}","received_events_url":"https://api.github.com/users/sfelshin/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/infolab-csail/WikipediaBase/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-12-03T00:58:09Z","updated_at":"2015-12-06T19:43:57Z","closed_at":null,"body":"START and Omnibase allow symbol-synonym pairs to be labeled with a priority, which states whether the pair is well-known (priority = `1`) or not (`0`).  See https://groups.csail.mit.edu/infolab/internal/srcdocs/omnibase/db-editors-guide.html#well-known for documentation and for an example of how START uses this information.  WikipediaBase sorts symbols into any number of subsets using the `sort-symbols-named` command.  Right now, START only supports two levels of priority: all symbols sorted into the first subset are considered well known and all others are not.\r\n\r\nIt's problematical to have only two levels.  We should have three: well known, neutral, and not well known.  (Maybe we should have even finer distinctions someday, but never mind about that for now.) Consider the query \"show me a picture of a tiger\".  START answers:\r\n\r\n> Tiger:\r\n> [picture from the https://en.wikipedia.org/wiki/Tiger article]\r\n> I know about 34 more terms called \"tiger\". You don't really want me to list them all, do you?\r\n> Source: Wikipedia \r\n\r\nIf you insist on listing all 34 other tigers, you see:\r\n\r\n> The Tiger (2005 film)\r\n> Tiger (1979 film)\r\n> [snip]\r\n> Tiger Shroff\r\n> Tiger Woods\r\n> ABBA\r\n> [snip]\r\n> Grumman F-11 Tiger\r\n> Harimau! Harimau!\r\n> Northrop F-5\r\n> Smilodon\r\n>  [snip]\r\n> Toyota Hilux\r\n\r\nSome of these symbols are pretty good tigers.  E.g., \"The Tiger (2005 film)\" and \"Tiger Shroff\" and \"Grumman F-11 Tiger\" because the symbol name contains \"tiger\", and \"Smilodon\" because a smilodon is quite well known by its popular name \"saber-toothed tiger\".  But others aren't obviously tigers at all, e.g., one of several models of the Toyota Hilux was known as the Hilux Tiger.\r\n\r\nSTART should be able to distinguish three levels of priority for various reasons:\r\n\r\n1) To give a clearer answer.  Either the initial answer could be\r\n\r\n> Tiger:\r\n> [picture]\r\n> I know about 18 more terms called \"tiger\" and 16 other terms associated with \"tiger\". ...\r\n> Source: Wikipedia \r\n\r\nor START could give the same initial answer, but when you \"list them all\", START could list them in two sets.  Or better yet, only the first set would be listed and there would be a second \"list them all\" for the not-well-known symbols; see the next item for why this would be better.\r\n\r\n2) For efficiency.  START fetches the value of an attribute before offering to answer about it (not always, but in many cases including for Wikipedia images).  This saves START from offering to give answers it can't give.  E.g., it looks stupid and annoys the user if we offer to show pictures of 30 items but when the user actually clicks on one or another of them, START says no picture is available.  But because there are 35 tigers, it takes a long time to fetch all their image attribute values, and START is in danger of timing out.  (See also https://github.com/infolab-csail/WikipediaBase/issues/100 regarding fetching images.)  START counts how many non-best-priority symbols there are, and if there are few enough, it lists them in the initial answer.  To avoid timeouts, it could stop counting after the threshold and answer\r\n\r\n> Tiger:\r\n> [picture]\r\n> I know about lots more terms called \"tiger\". You don't really want me to list them all, do you?\r\n> Source: Wikipedia \r\n\r\nbut that looks pretty cheesy.  If START could count just the high and neutral priority ones and answer something like this:\r\n\r\n> Tiger:\r\n> [picture]\r\n> I know about 18 more terms called \"tiger\" and several terms associated with \"tiger\". ...\r\n> Source: Wikipedia \r\n\r\nthat would be a nice compromise.\r\n\r\nAs for how we would distinguish the neutral from low priority symbol-synonym pairs, we could use any of several ways, or combine several ways:\r\n\r\n* Does the synonym that the user used appear anywhere in the symbol name?  Use case-insensitive match, and probably ignore punctuation, whitespace, and accents.  E.g., \"tiger\" matches \"Grumman F-11 Tiger\".\r\n\r\n* Is there a small edit distance between the user's synonym and the symbol name?  What would \"small\" mean?\r\n\r\n* Does the user's synonym appear frequently in the article text?  What would \"frequently\" mean?\r\n\r\n* etc."},"comment":{"url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/comments/162340501","html_url":"https://github.com/infolab-csail/WikipediaBase/issues/101#issuecomment-162340501","issue_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/101","id":162340501,"user":{"login":"fakedrake","id":408231,"avatar_url":"https://avatars.githubusercontent.com/u/408231?v=3","gravatar_id":"","url":"https://api.github.com/users/fakedrake","html_url":"https://github.com/fakedrake","followers_url":"https://api.github.com/users/fakedrake/followers","following_url":"https://api.github.com/users/fakedrake/following{/other_user}","gists_url":"https://api.github.com/users/fakedrake/gists{/gist_id}","starred_url":"https://api.github.com/users/fakedrake/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fakedrake/subscriptions","organizations_url":"https://api.github.com/users/fakedrake/orgs","repos_url":"https://api.github.com/users/fakedrake/repos","events_url":"https://api.github.com/users/fakedrake/events{/privacy}","received_events_url":"https://api.github.com/users/fakedrake/received_events","type":"User","site_admin":false},"created_at":"2015-12-06T19:43:57Z","updated_at":"2015-12-06T19:43:57Z","body":"Hello,\r\n\r\nDo you think we could also use the centrality of a symbol in the wikipedia page link graph to determine the priority (or even tricks from PageRank, although I haven't read up on any of that research)? That information is probably easy to obtain from [the dump](http://dumps.wikimedia.org/enwiki/20151201/) (search for `pagelinks`).\r\n\r\nTo also comment on your suggestions on ways we could combine to determine the priority of a page, maybe if we are to use the corresponding symbols purely as strings, we might want to also take into account their synonyms.\r\n\r\nPS. At the risk of taking your example about images a bit too literally, using the dumps it is probably easy for WikipediaBase to know in advance what images it can reliably fetch in one operation using again [the dumps](http://dumps.wikimedia.org/enwiki/20151201/) (search for `imagelinks`). There is some stuff to be said about the `imagelinks` table, like the fact that\r\n\r\n    select *  from imagelinks where il_from=20078709;\r\n\r\nwhere 200078709 being the id of tha article `Presidential_transition_of_Barack_Obama` will yield\r\n\r\n    [...]\r\n    | 20078709 | Wikibooks-logo.svg                                                       |                 0 |\r\n    | 20078709 | Wikinews-logo.svg                                                        |                 0 |\r\n    | 20078709 | Wikiquote-logo.svg                                                        |                 0 |\r\n    | 20078709 | Wikisource-logo.svg                                                      |                 0 |\r\n    [...]\r\n\r\nWhich are common for all pages. But these arguments are probably beside the point if the images-specific part of the issue you raise is just an example of the wider issue."}}