{"action":"created","issue":{"url":"https://api.github.com/repos/DOAJ/doaj/issues/877","labels_url":"https://api.github.com/repos/DOAJ/doaj/issues/877/labels{/name}","comments_url":"https://api.github.com/repos/DOAJ/doaj/issues/877/comments","events_url":"https://api.github.com/repos/DOAJ/doaj/issues/877/events","html_url":"https://github.com/DOAJ/doaj/issues/877","id":108592019,"number":877,"title":"Have we lost page number ordering on ToCs?","user":{"login":"dommitchell","id":5955396,"avatar_url":"https://avatars.githubusercontent.com/u/5955396?v=3","gravatar_id":"","url":"https://api.github.com/users/dommitchell","html_url":"https://github.com/dommitchell","followers_url":"https://api.github.com/users/dommitchell/followers","following_url":"https://api.github.com/users/dommitchell/following{/other_user}","gists_url":"https://api.github.com/users/dommitchell/gists{/gist_id}","starred_url":"https://api.github.com/users/dommitchell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dommitchell/subscriptions","organizations_url":"https://api.github.com/users/dommitchell/orgs","repos_url":"https://api.github.com/users/dommitchell/repos","events_url":"https://api.github.com/users/dommitchell/events{/privacy}","received_events_url":"https://api.github.com/users/dommitchell/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/DOAJ/doaj/labels/bug","name":"bug","color":"eb6420"}],"state":"open","locked":false,"assignee":{"login":"Steven-Eardley","id":1446355,"avatar_url":"https://avatars.githubusercontent.com/u/1446355?v=3","gravatar_id":"","url":"https://api.github.com/users/Steven-Eardley","html_url":"https://github.com/Steven-Eardley","followers_url":"https://api.github.com/users/Steven-Eardley/followers","following_url":"https://api.github.com/users/Steven-Eardley/following{/other_user}","gists_url":"https://api.github.com/users/Steven-Eardley/gists{/gist_id}","starred_url":"https://api.github.com/users/Steven-Eardley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Steven-Eardley/subscriptions","organizations_url":"https://api.github.com/users/Steven-Eardley/orgs","repos_url":"https://api.github.com/users/Steven-Eardley/repos","events_url":"https://api.github.com/users/Steven-Eardley/events{/privacy}","received_events_url":"https://api.github.com/users/Steven-Eardley/received_events","type":"User","site_admin":false},"milestone":null,"comments":3,"created_at":"2015-09-28T05:34:21Z","updated_at":"2015-10-03T21:47:03Z","closed_at":null,"body":"https://doaj.org/toc/1823-4690/10/8\r\n\r\nStart page: 1009. 1035, 1054 etc, 958, 972"},"comment":{"url":"https://api.github.com/repos/DOAJ/doaj/issues/comments/145293171","html_url":"https://github.com/DOAJ/doaj/issues/877#issuecomment-145293171","issue_url":"https://api.github.com/repos/DOAJ/doaj/issues/877","id":145293171,"user":{"login":"Steven-Eardley","id":1446355,"avatar_url":"https://avatars.githubusercontent.com/u/1446355?v=3","gravatar_id":"","url":"https://api.github.com/users/Steven-Eardley","html_url":"https://github.com/Steven-Eardley","followers_url":"https://api.github.com/users/Steven-Eardley/followers","following_url":"https://api.github.com/users/Steven-Eardley/following{/other_user}","gists_url":"https://api.github.com/users/Steven-Eardley/gists{/gist_id}","starred_url":"https://api.github.com/users/Steven-Eardley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Steven-Eardley/subscriptions","organizations_url":"https://api.github.com/users/Steven-Eardley/orgs","repos_url":"https://api.github.com/users/Steven-Eardley/repos","events_url":"https://api.github.com/users/Steven-Eardley/events{/privacy}","received_events_url":"https://api.github.com/users/Steven-Eardley/received_events","type":"User","site_admin":false},"created_at":"2015-10-03T21:47:03Z","updated_at":"2015-10-03T21:47:03Z","body":"This one was an excellent brain-teaser. \r\n\r\nThe crux of the problem was ``` _sort_articles()``` in ```portality.models.article.py``` was always getting ```None``` for the start_page, then everything was considered unsorted:\r\n\r\n```python\r\nfor art in articles:\r\n    sp = art.get(\"bibjson\",{}).get(\"start_page\", None)\r\n\r\n    # can't sort anything that doesn't have a start page\r\n    if sp is None:\r\n        unsorted.append(art)\r\n        continue\r\n    ...\r\n```\r\n\r\nWe were trying to get the start_page from the bibjson but it wasn't structured as this function expects - the query providing the data is one that pulls out individual fields, so we have to access the start_pages field like this instead:\r\n\r\n```python\r\nsp = art.get(\"bibjson.start_page\", [None])[0]\r\n```\r\n\r\nBecause the results we are reading from look like this:\r\n```json\r\n\"hits\": {\r\n    \"total\": 3,\r\n    \"max_score\": null,\r\n    \"hits\": [\r\n      {\r\n        \"_index\": \"doaj\",\r\n        \"_type\": \"article\",\r\n        \"_id\": \"\",\r\n        \"_score\": null,\r\n        \"fields\": {\r\n          \"bibjson.journal.number\": [\r\n            \"Issue 1\"\r\n          ],\r\n          \"bibjson.abstract\": [\"Abstract\"],\r\n          \"bibjson.title\": [\r\n            \"The Title\"\r\n          ],\r\n          \"bibjson.journal.volume\": [\r\n            \"2014\"\r\n          ],\r\n          \"bibjson.end_page\": [\r\n            \"5102\"\r\n          ],\r\n          \"bibjson.start_page\": [\r\n            \"5093\"\r\n          ],...\r\n```\r\n\r\nThe reason we were getting semi-sensible results all this time was that the ElasticSearch query includes  ```\"sort\": \"bibjson.start_page\"```, so it was pre-sorting the articles, but not doing a complete job of it. We were being a little unfair to ES, really, because we do store the start_pages as strings in case it's something like 'ix'."}}