{"action":"created","issue":{"url":"https://api.github.com/repos/pi-engine/pi/issues/1324","labels_url":"https://api.github.com/repos/pi-engine/pi/issues/1324/labels{/name}","comments_url":"https://api.github.com/repos/pi-engine/pi/issues/1324/comments","events_url":"https://api.github.com/repos/pi-engine/pi/issues/1324/events","html_url":"https://github.com/pi-engine/pi/issues/1324","id":55463638,"number":1324,"title":"Browser perf & SEO > Robotx.txt and JS/CSS crawler access","user":{"login":"Marc-pi","id":919332,"avatar_url":"https://avatars.githubusercontent.com/u/919332?v=3","gravatar_id":"","url":"https://api.github.com/users/Marc-pi","html_url":"https://github.com/Marc-pi","followers_url":"https://api.github.com/users/Marc-pi/followers","following_url":"https://api.github.com/users/Marc-pi/following{/other_user}","gists_url":"https://api.github.com/users/Marc-pi/gists{/gist_id}","starred_url":"https://api.github.com/users/Marc-pi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Marc-pi/subscriptions","organizations_url":"https://api.github.com/users/Marc-pi/orgs","repos_url":"https://api.github.com/users/Marc-pi/repos","events_url":"https://api.github.com/users/Marc-pi/events{/privacy}","received_events_url":"https://api.github.com/users/Marc-pi/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/pi-engine/pi/labels/Architecture","name":"Architecture","color":"009800"},{"url":"https://api.github.com/repos/pi-engine/pi/labels/Critical","name":"Critical","color":"e11d21"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/pi-engine/pi/milestones/1","labels_url":"https://api.github.com/repos/pi-engine/pi/milestones/1/labels","id":570080,"number":1,"title":"Pi 2.5.0","description":"SEO Compat\r\nBrowser Cache\r\nThird Part lib update\r\nImprove CKEditor integration\r\nImage Service improvement\r\npi::url\r\nSocial buttons, OG and Twittter tags, microformats, \r\nVideo player\r\nFix bugs\r\nUpdate GA code\r\nMobile login session\r\nCore Refactoring\r\nWidget module refactoring\r\nLoad JS\r\n","creator":{"login":"taiwen","id":606523,"avatar_url":"https://avatars.githubusercontent.com/u/606523?v=3","gravatar_id":"","url":"https://api.github.com/users/taiwen","html_url":"https://github.com/taiwen","followers_url":"https://api.github.com/users/taiwen/followers","following_url":"https://api.github.com/users/taiwen/following{/other_user}","gists_url":"https://api.github.com/users/taiwen/gists{/gist_id}","starred_url":"https://api.github.com/users/taiwen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/taiwen/subscriptions","organizations_url":"https://api.github.com/users/taiwen/orgs","repos_url":"https://api.github.com/users/taiwen/repos","events_url":"https://api.github.com/users/taiwen/events{/privacy}","received_events_url":"https://api.github.com/users/taiwen/received_events","type":"User","site_admin":false},"open_issues":14,"closed_issues":31,"state":"open","created_at":"2014-02-15T02:37:58Z","updated_at":"2015-02-11T17:08:24Z","due_on":"2015-02-27T23:00:00Z","closed_at":null},"comments":14,"created_at":"2015-01-26T09:53:30Z","updated_at":"2015-02-11T17:08:24Z","closed_at":"2015-02-11T17:08:24Z","body":"Browser and crawlers are using robots.txt informations.\r\nLatests years, those improved the way they need some information to be accessible : \r\n\r\n>  Bing/Yahoo and Google need a proper access to JS and CSS ressources, to see your pages like users see. \r\n\r\nThis impacts : \r\n* browser display (over the fold display time and sometime some resources display like FA icons)\r\n* crawlers speed\r\n* SEO rankings\r\n\r\nReferences : \r\n* http://googlewebmastercentral.blogspot.fr/2014/10/updating-our-technical-webmaster.html\r\n* i guess it is the same for baidu since it is also a modern search engine\r\n* http://searchenginewatch.com/sew/how-to/2356447/managing-your-robotstxt-file-effectively\r\n* http://www.microsoft.com/web/seo/ => Bing/Yahoo \r\n\r\n==> if you use this MS latest tool, you get those Violation warnings\r\n\r\n````\r\n/public/vendor/\r\n/asset/~>yourthemecss&js&images\r\n/asset/~>yourmodulecss&js\r\n/script (captcha)\r\n/static/vendor/jquery\r\n/static/avatar/~>image\r\n/upload~>modules/images directories\r\n````\r\n\r\nOur actual default Robots.txt file\r\n\r\n````\r\nUser-agent: *\r\nDisallow: /cgi-bin/\r\nDisallow: /tmp/\r\nDisallow: /asset/\r\nDisallow: /module/\r\nDisallow: /public/\r\nDisallow: /script/\r\nDisallow: /setup/\r\nDisallow: /static/\r\nDisallow: /upload/\r\n\r\nUser-agent: Mediapartners-Google\r\nAllow: /\r\n \r\nUser-agent: Adsbot-Google\r\nAllow: /\r\n \r\nUser-agent: Googlebot-Image\r\nAllow: /\r\nAllow: /upload/\r\n \r\nUser-agent: Googlebot-Mobile\r\nAllow: /\r\n````\r\n\r\n* Images are accessible to google-bot image but not the search engine => i guess we can open /upload to crawlers, no?\r\n* what about those bots ? `Googlebot-Video`, `Googlebot-News`, `Mediapartners` (# `Mediapartners-Google`) => those are missing \r\n* we are too restrictive at blocking indexation for Bing/Yahoo/Baidu , and more opened to Google => need to improve\r\n\r\n> This has a huge impact on SEO rankings and pagespeed rankings & some browser display impact\r\n\r\n1/ We have for sure at least to update the default robots Disallow list for CSS/JS and perhaps images and some crawlers. A good robotx.txt file must be also light \r\n(you can read also http://www.elegantthemes.com/blog/tips-tricks/how-to-create-and-configure-your-robots-txt-file)\r\ni guess it will fix also th FA icons not being displayed by some users\r\n@taiwen @voltan => your thoughts about the dir to open ?\r\n\r\n2/ But i wonder if we have a more structural changes to make since those CSS/JSS dir resources are located in different directories @taiwen  : ...without having regression on running Pi on several servers/instances/sass mode (those dir must be crawlers accessible but also protected by index.html file)\r\n"},"comment":{"url":"https://api.github.com/repos/pi-engine/pi/issues/comments/73920093","html_url":"https://github.com/pi-engine/pi/issues/1324#issuecomment-73920093","issue_url":"https://api.github.com/repos/pi-engine/pi/issues/1324","id":73920093,"user":{"login":"voltan","id":562083,"avatar_url":"https://avatars.githubusercontent.com/u/562083?v=3","gravatar_id":"","url":"https://api.github.com/users/voltan","html_url":"https://github.com/voltan","followers_url":"https://api.github.com/users/voltan/followers","following_url":"https://api.github.com/users/voltan/following{/other_user}","gists_url":"https://api.github.com/users/voltan/gists{/gist_id}","starred_url":"https://api.github.com/users/voltan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/voltan/subscriptions","organizations_url":"https://api.github.com/users/voltan/orgs","repos_url":"https://api.github.com/users/voltan/repos","events_url":"https://api.github.com/users/voltan/events{/privacy}","received_events_url":"https://api.github.com/users/voltan/received_events","type":"User","site_admin":false},"created_at":"2015-02-11T17:08:24Z","updated_at":"2015-02-11T17:08:24Z","body":"For default installer it good. we can add others as custom edit on websites. I think we can close it"}}