{"action":"created","issue":{"url":"https://api.github.com/repos/GSA/DAP-Gov-wide-GA-Code/issues/10","labels_url":"https://api.github.com/repos/GSA/DAP-Gov-wide-GA-Code/issues/10/labels{/name}","comments_url":"https://api.github.com/repos/GSA/DAP-Gov-wide-GA-Code/issues/10/comments","events_url":"https://api.github.com/repos/GSA/DAP-Gov-wide-GA-Code/issues/10/events","html_url":"https://github.com/GSA/DAP-Gov-wide-GA-Code/issues/10","id":60292965,"number":10,"title":"Remove lowercase filter and www filter on URLs","user":{"login":"konklone","id":4592,"avatar_url":"https://avatars.githubusercontent.com/u/4592?v=3","gravatar_id":"","url":"https://api.github.com/users/konklone","html_url":"https://github.com/konklone","followers_url":"https://api.github.com/users/konklone/followers","following_url":"https://api.github.com/users/konklone/following{/other_user}","gists_url":"https://api.github.com/users/konklone/gists{/gist_id}","starred_url":"https://api.github.com/users/konklone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/konklone/subscriptions","organizations_url":"https://api.github.com/users/konklone/orgs","repos_url":"https://api.github.com/users/konklone/repos","events_url":"https://api.github.com/users/konklone/events{/privacy}","received_events_url":"https://api.github.com/users/konklone/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-03-09T03:07:44Z","updated_at":"2015-03-25T16:19:44Z","closed_at":null,"body":"The DAP currently includes a filter that lower-cases all URIs:\r\n\r\n![screenshot from 2015-03-08 22 59 32](https://cloud.githubusercontent.com/assets/4592/6549290/dc216cc6-c5e6-11e4-90dc-c607451f0c6f.png)\r\n\r\nAnd a filter that removes the `www` from domains:\r\n\r\n![screenshot from 2015-03-08 23 04 28](https://cloud.githubusercontent.com/assets/4592/6549319/846c518e-c5e7-11e4-948f-36f7a06838e1.png)\r\n\r\nI had previously assumed these were issues intrinsic to Google Analytics, and not specific to the DAP, which is why I am only raising this issue now.\r\n\r\nThese filters cause issues with the WH Petition site like #9 and https://github.com/GSA/analytics.usa.gov/issues/138, caused some of the hardcoded exceptions in https://github.com/GSA/analytics.usa.gov/pull/129, and are the cause of the nasa.gov problem in https://github.com/GSA/analytics.usa.gov/issues/140.\r\n\r\nI understand that the lowercase filter is there to unify websites which use inconsistent case to refer to case-insensitive URLs. However, [URLs are allowed to be case-sensitive](http://www.w3.org/TR/WD-html40-970708/htmlweb.html#h-4.1) (even if [domain names are not](http://tools.ietf.org/html/rfc4343)), and this is causing broken links to appear.\r\n\r\nSimilarly, though I would greatly prefer that domains handle a missing `www` gracefully, neither the internet not the government requires them to do so at this time.\r\n\r\nIf there are specific problem domains whose reporting is significantly improved by these filters, let's make the filters only apply to those whitelisted domains, and not to every URL that comes into the DAP."},"comment":{"url":"https://api.github.com/repos/GSA/DAP-Gov-wide-GA-Code/issues/comments/86102441","html_url":"https://github.com/GSA/DAP-Gov-wide-GA-Code/issues/10#issuecomment-86102441","issue_url":"https://api.github.com/repos/GSA/DAP-Gov-wide-GA-Code/issues/10","id":86102441,"user":{"login":"smarina04","id":3287495,"avatar_url":"https://avatars.githubusercontent.com/u/3287495?v=3","gravatar_id":"","url":"https://api.github.com/users/smarina04","html_url":"https://github.com/smarina04","followers_url":"https://api.github.com/users/smarina04/followers","following_url":"https://api.github.com/users/smarina04/following{/other_user}","gists_url":"https://api.github.com/users/smarina04/gists{/gist_id}","starred_url":"https://api.github.com/users/smarina04/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarina04/subscriptions","organizations_url":"https://api.github.com/users/smarina04/orgs","repos_url":"https://api.github.com/users/smarina04/repos","events_url":"https://api.github.com/users/smarina04/events{/privacy}","received_events_url":"https://api.github.com/users/smarina04/received_events","type":"User","site_admin":false},"created_at":"2015-03-25T16:19:44Z","updated_at":"2015-03-25T16:19:44Z","body":"Forced lowercase filter was applied on purpose, otherwise, mixed case in URLs is all over the map and it treats the same URL but with different cases as different URLs. When everything is forced to lowercase and makes URLs consistent, then the same URLs but with mixed cases get forced to be unified. We got a lot of those cases in the beginning of DAP and had to apply the lowercase filter to address it.\r\n\r\nRemoving www was also done on purpose for uniformity reasons.. otherwise, we got WWW, www, or simply a domain w/out \"www\" and all three would be treated as unique URLs. \r\n\r\nAppending the domain name (otherwise, in GA, everything would appear as part of a page without the domain name vs. full URL) We knew that by appending (faking it basically) the domain name to the pages would cause the links to not work when clicking on them. However, the appearance of full URLs was much more important because we handle thousands of websites vs. one, and agencies would have to guess what domain each page belonged to. So we decided that appending the domain names and thus, breaking the direct links was much more important than being able to click on the links. \r\n\r\nAll these and other data standardization techniques were performed because of the messy data we were getting in (remember, it's thousands of websites), which made analysis that much more complicated; e.g. having to merge/sum some of the data, or in cases or metrics (e.g. visits/sessions) the data could not be summed up at all. \r\n"}}