{"action":"created","comment":{"url":"https://api.github.com/repos/ckan/ckan/pulls/comments/23870967","id":23870967,"diff_hunk":"@@ -119,6 +127,34 @@ def munge_filename(filename):\n     return filename\n \n \n+def munge_filename(filename):\n+    ''' Tidies a filename\n+\n+    Keeps the filename extension.\n+    Strips off any path on the front.\n+    It can be useful to keep the extension (e.g. .csv) in a resource when the\n+    format field set.\n+    '''\n+\n+    # just get the filename ignore the path\n+    path, filename = os.path.split(filename)\n+    # clean up\n+    filename = substitute_ascii_equivalents(filename)\n+    filename = filename.lower().strip()\n+    filename = re.sub(r'[^a-zA-Z0-9. -]', '', filename).replace(' ', '-')\n+    # resize if needed but keep extension\n+    name, ext = os.path.splitext(filename)\n+    # limit overly long extensions\n+    if len(ext) > 21:\n+        ext = ext[:21]\n+    # max/min size\n+    ext_length = len(ext)\n+    name = _munge_to_length(name, 3 - ext_length, 100 - ext_length)","path":"ckan/lib/munge.py","position":null,"original_position":50,"commit_id":"3e51f4519d1dc148adf8b44a644e06b01c935d62","original_commit_id":"40079f2196c21121e2afa0463facaf6d8df43cef","user":{"login":"davidread","id":307612,"avatar_url":"https://avatars.githubusercontent.com/u/307612?v=3","gravatar_id":"","url":"https://api.github.com/users/davidread","html_url":"https://github.com/davidread","followers_url":"https://api.github.com/users/davidread/followers","following_url":"https://api.github.com/users/davidread/following{/other_user}","gists_url":"https://api.github.com/users/davidread/gists{/gist_id}","starred_url":"https://api.github.com/users/davidread/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidread/subscriptions","organizations_url":"https://api.github.com/users/davidread/orgs","repos_url":"https://api.github.com/users/davidread/repos","events_url":"https://api.github.com/users/davidread/events{/privacy}","received_events_url":"https://api.github.com/users/davidread/received_events","type":"User","site_admin":false},"body":"Good spot on the negatives. It happens to work ok, but just to be sure I've added a min value of 1.\r\n\r\nNot sure I can do much about '.ext' without getting rid of splitext:\r\n```\r\n>>> os.path.splitext('.ext')\r\n('.ext', '')\r\n```\r\nAnd I'm not convinced it is an edge case to necessarily worry about.","created_at":"2015-01-30T20:43:41Z","updated_at":"2015-01-30T20:43:41Z","html_url":"https://github.com/ckan/ckan/pull/2257#discussion_r23870967","pull_request_url":"https://api.github.com/repos/ckan/ckan/pulls/2257","_links":{"self":{"href":"https://api.github.com/repos/ckan/ckan/pulls/comments/23870967"},"html":{"href":"https://github.com/ckan/ckan/pull/2257#discussion_r23870967"},"pull_request":{"href":"https://api.github.com/repos/ckan/ckan/pulls/2257"}}},"pull_request":{"url":"https://api.github.com/repos/ckan/ckan/pulls/2257","id":28319094,"html_url":"https://github.com/ckan/ckan/pull/2257","diff_url":"https://github.com/ckan/ckan/pull/2257.diff","patch_url":"https://github.com/ckan/ckan/pull/2257.patch","issue_url":"https://api.github.com/repos/ckan/ckan/issues/2257","number":2257,"state":"open","locked":false,"title":"Tobes 2057 fix filename munging","user":{"login":"davidread","id":307612,"avatar_url":"https://avatars.githubusercontent.com/u/307612?v=3","gravatar_id":"","url":"https://api.github.com/users/davidread","html_url":"https://github.com/davidread","followers_url":"https://api.github.com/users/davidread/followers","following_url":"https://api.github.com/users/davidread/following{/other_user}","gists_url":"https://api.github.com/users/davidread/gists{/gist_id}","starred_url":"https://api.github.com/users/davidread/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidread/subscriptions","organizations_url":"https://api.github.com/users/davidread/orgs","repos_url":"https://api.github.com/users/davidread/repos","events_url":"https://api.github.com/users/davidread/events{/privacy}","received_events_url":"https://api.github.com/users/davidread/received_events","type":"User","site_admin":false},"body":"Issue #2057\r\n\r\nThis PR is an improved version of Tobe's PR:\r\nhttps://github.com/ckan/ckan/pull/2149/files\r\n\r\nIf we upload a file with a huge name then the extension gets eaten and we end up with the datastore not being able to import the file correctly if the format has not been set for the resource\r\n\r\nAlso it seems browsers on windows 7 seem to pass the whole path along with the file name so this creates problems for these users.\r\n\r\nSo we remove the path and try to keep the extension - really long extensions (over 20 chars) get truncated but these are probably erroneous extensions anyway.","created_at":"2015-01-29T22:03:54Z","updated_at":"2015-01-30T20:43:41Z","closed_at":null,"merged_at":null,"merge_commit_sha":"8dfc3c93fba44786082f9f79223d6d1a62bb1784","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/ckan/ckan/pulls/2257/commits","review_comments_url":"https://api.github.com/repos/ckan/ckan/pulls/2257/comments","review_comment_url":"https://api.github.com/repos/ckan/ckan/pulls/comments/{number}","comments_url":"https://api.github.com/repos/ckan/ckan/issues/2257/comments","statuses_url":"https://api.github.com/repos/ckan/ckan/statuses/3e51f4519d1dc148adf8b44a644e06b01c935d62","head":{"label":"ckan:tobes-2057-fix-filename-munging","ref":"tobes-2057-fix-filename-munging","sha":"3e51f4519d1dc148adf8b44a644e06b01c935d62","user":{"login":"ckan","id":1630326,"avatar_url":"https://avatars.githubusercontent.com/u/1630326?v=3","gravatar_id":"","url":"https://api.github.com/users/ckan","html_url":"https://github.com/ckan","followers_url":"https://api.github.com/users/ckan/followers","following_url":"https://api.github.com/users/ckan/following{/other_user}","gists_url":"https://api.github.com/users/ckan/gists{/gist_id}","starred_url":"https://api.github.com/users/ckan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ckan/subscriptions","organizations_url":"https://api.github.com/users/ckan/orgs","repos_url":"https://api.github.com/users/ckan/repos","events_url":"https://api.github.com/users/ckan/events{/privacy}","received_events_url":"https://api.github.com/users/ckan/received_events","type":"Organization","site_admin":false},"repo":{"id":2750721,"name":"ckan","full_name":"ckan/ckan","owner":{"login":"ckan","id":1630326,"avatar_url":"https://avatars.githubusercontent.com/u/1630326?v=3","gravatar_id":"","url":"https://api.github.com/users/ckan","html_url":"https://github.com/ckan","followers_url":"https://api.github.com/users/ckan/followers","following_url":"https://api.github.com/users/ckan/following{/other_user}","gists_url":"https://api.github.com/users/ckan/gists{/gist_id}","starred_url":"https://api.github.com/users/ckan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ckan/subscriptions","organizations_url":"https://api.github.com/users/ckan/orgs","repos_url":"https://api.github.com/users/ckan/repos","events_url":"https://api.github.com/users/ckan/events{/privacy}","received_events_url":"https://api.github.com/users/ckan/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ckan/ckan","description":"CKAN is an open-source DMS (data management system) for powering data hubs and data portals. CKAN makes it easy to publish, share and use data. It powers datahub.io, catalog.data.gov and data.gov.uk among many other sites.","fork":false,"url":"https://api.github.com/repos/ckan/ckan","forks_url":"https://api.github.com/repos/ckan/ckan/forks","keys_url":"https://api.github.com/repos/ckan/ckan/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ckan/ckan/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ckan/ckan/teams","hooks_url":"https://api.github.com/repos/ckan/ckan/hooks","issue_events_url":"https://api.github.com/repos/ckan/ckan/issues/events{/number}","events_url":"https://api.github.com/repos/ckan/ckan/events","assignees_url":"https://api.github.com/repos/ckan/ckan/assignees{/user}","branches_url":"https://api.github.com/repos/ckan/ckan/branches{/branch}","tags_url":"https://api.github.com/repos/ckan/ckan/tags","blobs_url":"https://api.github.com/repos/ckan/ckan/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ckan/ckan/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ckan/ckan/git/refs{/sha}","trees_url":"https://api.github.com/repos/ckan/ckan/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ckan/ckan/statuses/{sha}","languages_url":"https://api.github.com/repos/ckan/ckan/languages","stargazers_url":"https://api.github.com/repos/ckan/ckan/stargazers","contributors_url":"https://api.github.com/repos/ckan/ckan/contributors","subscribers_url":"https://api.github.com/repos/ckan/ckan/subscribers","subscription_url":"https://api.github.com/repos/ckan/ckan/subscription","commits_url":"https://api.github.com/repos/ckan/ckan/commits{/sha}","git_commits_url":"https://api.github.com/repos/ckan/ckan/git/commits{/sha}","comments_url":"https://api.github.com/repos/ckan/ckan/comments{/number}","issue_comment_url":"https://api.github.com/repos/ckan/ckan/issues/comments/{number}","contents_url":"https://api.github.com/repos/ckan/ckan/contents/{+path}","compare_url":"https://api.github.com/repos/ckan/ckan/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ckan/ckan/merges","archive_url":"https://api.github.com/repos/ckan/ckan/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ckan/ckan/downloads","issues_url":"https://api.github.com/repos/ckan/ckan/issues{/number}","pulls_url":"https://api.github.com/repos/ckan/ckan/pulls{/number}","milestones_url":"https://api.github.com/repos/ckan/ckan/milestones{/number}","notifications_url":"https://api.github.com/repos/ckan/ckan/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ckan/ckan/labels{/name}","releases_url":"https://api.github.com/repos/ckan/ckan/releases{/id}","created_at":"2011-11-10T18:42:17Z","updated_at":"2015-01-29T21:34:10Z","pushed_at":"2015-01-30T20:38:00Z","git_url":"git://github.com/ckan/ckan.git","ssh_url":"git@github.com:ckan/ckan.git","clone_url":"https://github.com/ckan/ckan.git","svn_url":"https://github.com/ckan/ckan","homepage":"http://ckan.org/","size":169695,"stargazers_count":789,"watchers_count":789,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":442,"mirror_url":null,"open_issues_count":282,"forks":442,"open_issues":282,"watchers":789,"default_branch":"master"}},"base":{"label":"ckan:master","ref":"master","sha":"8f298cd9e658d788b031a4059271210aff6a3d94","user":{"login":"ckan","id":1630326,"avatar_url":"https://avatars.githubusercontent.com/u/1630326?v=3","gravatar_id":"","url":"https://api.github.com/users/ckan","html_url":"https://github.com/ckan","followers_url":"https://api.github.com/users/ckan/followers","following_url":"https://api.github.com/users/ckan/following{/other_user}","gists_url":"https://api.github.com/users/ckan/gists{/gist_id}","starred_url":"https://api.github.com/users/ckan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ckan/subscriptions","organizations_url":"https://api.github.com/users/ckan/orgs","repos_url":"https://api.github.com/users/ckan/repos","events_url":"https://api.github.com/users/ckan/events{/privacy}","received_events_url":"https://api.github.com/users/ckan/received_events","type":"Organization","site_admin":false},"repo":{"id":2750721,"name":"ckan","full_name":"ckan/ckan","owner":{"login":"ckan","id":1630326,"avatar_url":"https://avatars.githubusercontent.com/u/1630326?v=3","gravatar_id":"","url":"https://api.github.com/users/ckan","html_url":"https://github.com/ckan","followers_url":"https://api.github.com/users/ckan/followers","following_url":"https://api.github.com/users/ckan/following{/other_user}","gists_url":"https://api.github.com/users/ckan/gists{/gist_id}","starred_url":"https://api.github.com/users/ckan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ckan/subscriptions","organizations_url":"https://api.github.com/users/ckan/orgs","repos_url":"https://api.github.com/users/ckan/repos","events_url":"https://api.github.com/users/ckan/events{/privacy}","received_events_url":"https://api.github.com/users/ckan/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ckan/ckan","description":"CKAN is an open-source DMS (data management system) for powering data hubs and data portals. CKAN makes it easy to publish, share and use data. It powers datahub.io, catalog.data.gov and data.gov.uk among many other sites.","fork":false,"url":"https://api.github.com/repos/ckan/ckan","forks_url":"https://api.github.com/repos/ckan/ckan/forks","keys_url":"https://api.github.com/repos/ckan/ckan/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ckan/ckan/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ckan/ckan/teams","hooks_url":"https://api.github.com/repos/ckan/ckan/hooks","issue_events_url":"https://api.github.com/repos/ckan/ckan/issues/events{/number}","events_url":"https://api.github.com/repos/ckan/ckan/events","assignees_url":"https://api.github.com/repos/ckan/ckan/assignees{/user}","branches_url":"https://api.github.com/repos/ckan/ckan/branches{/branch}","tags_url":"https://api.github.com/repos/ckan/ckan/tags","blobs_url":"https://api.github.com/repos/ckan/ckan/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ckan/ckan/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ckan/ckan/git/refs{/sha}","trees_url":"https://api.github.com/repos/ckan/ckan/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ckan/ckan/statuses/{sha}","languages_url":"https://api.github.com/repos/ckan/ckan/languages","stargazers_url":"https://api.github.com/repos/ckan/ckan/stargazers","contributors_url":"https://api.github.com/repos/ckan/ckan/contributors","subscribers_url":"https://api.github.com/repos/ckan/ckan/subscribers","subscription_url":"https://api.github.com/repos/ckan/ckan/subscription","commits_url":"https://api.github.com/repos/ckan/ckan/commits{/sha}","git_commits_url":"https://api.github.com/repos/ckan/ckan/git/commits{/sha}","comments_url":"https://api.github.com/repos/ckan/ckan/comments{/number}","issue_comment_url":"https://api.github.com/repos/ckan/ckan/issues/comments/{number}","contents_url":"https://api.github.com/repos/ckan/ckan/contents/{+path}","compare_url":"https://api.github.com/repos/ckan/ckan/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ckan/ckan/merges","archive_url":"https://api.github.com/repos/ckan/ckan/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ckan/ckan/downloads","issues_url":"https://api.github.com/repos/ckan/ckan/issues{/number}","pulls_url":"https://api.github.com/repos/ckan/ckan/pulls{/number}","milestones_url":"https://api.github.com/repos/ckan/ckan/milestones{/number}","notifications_url":"https://api.github.com/repos/ckan/ckan/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ckan/ckan/labels{/name}","releases_url":"https://api.github.com/repos/ckan/ckan/releases{/id}","created_at":"2011-11-10T18:42:17Z","updated_at":"2015-01-29T21:34:10Z","pushed_at":"2015-01-30T20:38:00Z","git_url":"git://github.com/ckan/ckan.git","ssh_url":"git@github.com:ckan/ckan.git","clone_url":"https://github.com/ckan/ckan.git","svn_url":"https://github.com/ckan/ckan","homepage":"http://ckan.org/","size":169695,"stargazers_count":789,"watchers_count":789,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":442,"mirror_url":null,"open_issues_count":282,"forks":442,"open_issues":282,"watchers":789,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/ckan/ckan/pulls/2257"},"html":{"href":"https://github.com/ckan/ckan/pull/2257"},"issue":{"href":"https://api.github.com/repos/ckan/ckan/issues/2257"},"comments":{"href":"https://api.github.com/repos/ckan/ckan/issues/2257/comments"},"review_comments":{"href":"https://api.github.com/repos/ckan/ckan/pulls/2257/comments"},"review_comment":{"href":"https://api.github.com/repos/ckan/ckan/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/ckan/ckan/pulls/2257/commits"},"statuses":{"href":"https://api.github.com/repos/ckan/ckan/statuses/3e51f4519d1dc148adf8b44a644e06b01c935d62"}}}}