{"action":"opened","number":1557,"pull_request":{"url":"https://api.github.com/repos/dotnet/corefx/pulls/1557","id":34277105,"html_url":"https://github.com/dotnet/corefx/pull/1557","diff_url":"https://github.com/dotnet/corefx/pull/1557.diff","patch_url":"https://github.com/dotnet/corefx/pull/1557.patch","issue_url":"https://api.github.com/repos/dotnet/corefx/issues/1557","number":1557,"state":"open","locked":false,"title":"[Metadata] Support for very large String and Blob heaps","user":{"login":"tmat","id":41759,"avatar_url":"https://avatars.githubusercontent.com/u/41759?v=3","gravatar_id":"","url":"https://api.github.com/users/tmat","html_url":"https://github.com/tmat","followers_url":"https://api.github.com/users/tmat/followers","following_url":"https://api.github.com/users/tmat/following{/other_user}","gists_url":"https://api.github.com/users/tmat/gists{/gist_id}","starred_url":"https://api.github.com/users/tmat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tmat/subscriptions","organizations_url":"https://api.github.com/users/tmat/orgs","repos_url":"https://api.github.com/users/tmat/repos","events_url":"https://api.github.com/users/tmat/events{/privacy}","received_events_url":"https://api.github.com/users/tmat/received_events","type":"User","site_admin":false},"body":"Fixes bug https://github.com/dotnet/corefx/issues/1238.\r\n\r\nWe have previously assumed that the size of a metadata heap is less than 16MB (i.e. each heap offset fits into 24bits). This assumption allowed us to represent all handles in Handle “supertype” with 1B used to store the type of the handle. This assumption is however not correct. There are assemblies out there whose #String heap is larger than 16MB. In bug #1238 such an assembly is generated by C++/CLI compiler. \r\n\r\nThe ECMA335 spec isn’t clear on the upper limit of a metadata heap, other than it has to fit into 32 bits. \r\n\r\nWe propose that the limit on #String and #Blob heap size is 229 (0.5 GB), that is any index to these heaps fits into 29 bits.\r\nThere are a few reasons why I think this limitation is reasonable:\r\n1)\t229 is the maximum value representable by a compressed integer as defined elsewhere in the spec. Currently the metadata don’t encode heap indices anywhere using compressed integers. However in future we’d like to encode debugging information in ECMA format and we use compressed integers for efficient encoding of heap indices (see the spec draft in progress here: https://github.com/dotnet/roslyn/blob/portable-pdb/docs/specs/PortablePdb-Metadata.md). We could extend the definition of compressed integer to cover all 32 bit integers, but it would be simpler if we could leave it as is.\r\n2)\t0.5 GB is a very large heap. Having such a big PE file seems unreasonable and very rare scenario (if it exists at all).\r\n3)\tHaving 3 spare bits available is very beneficial for the implementation. It allows us to implement various WinRT projection tricks, namespaces, etc in very efficient way in the reader. If we had to represent heap indices with all 32 bits it would bloat various structures and waste memory. PE files over 0.5 GB of size are very very rare, but this wasted memory would affect all compilers and tools working with the metadata reader.\r\n\r\nThis change is assuming the heap size limit of 229 but structures public API so that we can increase the limit in future if necessary, without breaking changes.\r\n\r\nThe change introduces EntityHandle to represent any handle that references a table row. Logically EntityHandle is a “subtype” of Handle and each specific entity handle (TypeDefinitionHandle, TypeReferenceHandle, etc.) is a “subtype” of EntityHandle. On the other hand, StringHandle, NamespaceDefinitionHandle, BlobHandle, GuidHandle, and GuidHandle are not “subtypes” of EntityHandle. \r\n\r\nHandle is stored in 40 bits, EntityHandle is be stored in 32 bit, heap handles are stored in 32 bit as well. Whenever the consumers of the metadata reader need to store handles of different kinds in the same location they should use EntityHandles if all these handles represent entities and Handles if they don’t (which is rare).\r\n\r\nIn addition this change modifies the public API of TypeDefinition:\r\n-\tIt changes the return type of Namespace property to StringHandle (returns the full namespace string)\r\n-\tIt adds a property NamespaceDefinition returning NamespaceDefinitionHandle\r\n\r\nThis change also cleans up unnecessary internal casting from and to uint and fixes a possible overflow in TypeDefinition.GetTypeLayout().\r\n\r\nFixes internal bug 1048345.\r\n","created_at":"2015-04-28T15:11:04Z","updated_at":"2015-04-28T15:11:04Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/dotnet/corefx/pulls/1557/commits","review_comments_url":"https://api.github.com/repos/dotnet/corefx/pulls/1557/comments","review_comment_url":"https://api.github.com/repos/dotnet/corefx/pulls/comments{/number}","comments_url":"https://api.github.com/repos/dotnet/corefx/issues/1557/comments","statuses_url":"https://api.github.com/repos/dotnet/corefx/statuses/c61ee8a89babfad1c4c4a8a678f5cff3ae03e868","head":{"label":"tmat:EntityHandle","ref":"EntityHandle","sha":"c61ee8a89babfad1c4c4a8a678f5cff3ae03e868","user":{"login":"tmat","id":41759,"avatar_url":"https://avatars.githubusercontent.com/u/41759?v=3","gravatar_id":"","url":"https://api.github.com/users/tmat","html_url":"https://github.com/tmat","followers_url":"https://api.github.com/users/tmat/followers","following_url":"https://api.github.com/users/tmat/following{/other_user}","gists_url":"https://api.github.com/users/tmat/gists{/gist_id}","starred_url":"https://api.github.com/users/tmat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tmat/subscriptions","organizations_url":"https://api.github.com/users/tmat/orgs","repos_url":"https://api.github.com/users/tmat/repos","events_url":"https://api.github.com/users/tmat/events{/privacy}","received_events_url":"https://api.github.com/users/tmat/received_events","type":"User","site_admin":false},"repo":{"id":33101159,"name":"corefx","full_name":"tmat/corefx","owner":{"login":"tmat","id":41759,"avatar_url":"https://avatars.githubusercontent.com/u/41759?v=3","gravatar_id":"","url":"https://api.github.com/users/tmat","html_url":"https://github.com/tmat","followers_url":"https://api.github.com/users/tmat/followers","following_url":"https://api.github.com/users/tmat/following{/other_user}","gists_url":"https://api.github.com/users/tmat/gists{/gist_id}","starred_url":"https://api.github.com/users/tmat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tmat/subscriptions","organizations_url":"https://api.github.com/users/tmat/orgs","repos_url":"https://api.github.com/users/tmat/repos","events_url":"https://api.github.com/users/tmat/events{/privacy}","received_events_url":"https://api.github.com/users/tmat/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/tmat/corefx","description":"This repo contains the .NET Core foundational libraries, called CoreFX. It includes classes for collections, file systems, console, XML, async and many others.","fork":true,"url":"https://api.github.com/repos/tmat/corefx","forks_url":"https://api.github.com/repos/tmat/corefx/forks","keys_url":"https://api.github.com/repos/tmat/corefx/keys{/key_id}","collaborators_url":"https://api.github.com/repos/tmat/corefx/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/tmat/corefx/teams","hooks_url":"https://api.github.com/repos/tmat/corefx/hooks","issue_events_url":"https://api.github.com/repos/tmat/corefx/issues/events{/number}","events_url":"https://api.github.com/repos/tmat/corefx/events","assignees_url":"https://api.github.com/repos/tmat/corefx/assignees{/user}","branches_url":"https://api.github.com/repos/tmat/corefx/branches{/branch}","tags_url":"https://api.github.com/repos/tmat/corefx/tags","blobs_url":"https://api.github.com/repos/tmat/corefx/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/tmat/corefx/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/tmat/corefx/git/refs{/sha}","trees_url":"https://api.github.com/repos/tmat/corefx/git/trees{/sha}","statuses_url":"https://api.github.com/repos/tmat/corefx/statuses/{sha}","languages_url":"https://api.github.com/repos/tmat/corefx/languages","stargazers_url":"https://api.github.com/repos/tmat/corefx/stargazers","contributors_url":"https://api.github.com/repos/tmat/corefx/contributors","subscribers_url":"https://api.github.com/repos/tmat/corefx/subscribers","subscription_url":"https://api.github.com/repos/tmat/corefx/subscription","commits_url":"https://api.github.com/repos/tmat/corefx/commits{/sha}","git_commits_url":"https://api.github.com/repos/tmat/corefx/git/commits{/sha}","comments_url":"https://api.github.com/repos/tmat/corefx/comments{/number}","issue_comment_url":"https://api.github.com/repos/tmat/corefx/issues/comments{/number}","contents_url":"https://api.github.com/repos/tmat/corefx/contents/{+path}","compare_url":"https://api.github.com/repos/tmat/corefx/compare/{base}...{head}","merges_url":"https://api.github.com/repos/tmat/corefx/merges","archive_url":"https://api.github.com/repos/tmat/corefx/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/tmat/corefx/downloads","issues_url":"https://api.github.com/repos/tmat/corefx/issues{/number}","pulls_url":"https://api.github.com/repos/tmat/corefx/pulls{/number}","milestones_url":"https://api.github.com/repos/tmat/corefx/milestones{/number}","notifications_url":"https://api.github.com/repos/tmat/corefx/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/tmat/corefx/labels{/name}","releases_url":"https://api.github.com/repos/tmat/corefx/releases{/id}","created_at":"2015-03-30T03:23:20Z","updated_at":"2015-04-24T21:48:46Z","pushed_at":"2015-04-28T04:29:48Z","git_url":"git://github.com/tmat/corefx.git","ssh_url":"git@github.com:tmat/corefx.git","clone_url":"https://github.com/tmat/corefx.git","svn_url":"https://github.com/tmat/corefx","homepage":"","size":17453,"stargazers_count":0,"watchers_count":0,"language":"C#","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"dotnet:master","ref":"master","sha":"077407087801f795430d9abbfff8134218665fe0","user":{"login":"dotnet","id":9141961,"avatar_url":"https://avatars.githubusercontent.com/u/9141961?v=3","gravatar_id":"","url":"https://api.github.com/users/dotnet","html_url":"https://github.com/dotnet","followers_url":"https://api.github.com/users/dotnet/followers","following_url":"https://api.github.com/users/dotnet/following{/other_user}","gists_url":"https://api.github.com/users/dotnet/gists{/gist_id}","starred_url":"https://api.github.com/users/dotnet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dotnet/subscriptions","organizations_url":"https://api.github.com/users/dotnet/orgs","repos_url":"https://api.github.com/users/dotnet/repos","events_url":"https://api.github.com/users/dotnet/events{/privacy}","received_events_url":"https://api.github.com/users/dotnet/received_events","type":"Organization","site_admin":false},"repo":{"id":26295345,"name":"corefx","full_name":"dotnet/corefx","owner":{"login":"dotnet","id":9141961,"avatar_url":"https://avatars.githubusercontent.com/u/9141961?v=3","gravatar_id":"","url":"https://api.github.com/users/dotnet","html_url":"https://github.com/dotnet","followers_url":"https://api.github.com/users/dotnet/followers","following_url":"https://api.github.com/users/dotnet/following{/other_user}","gists_url":"https://api.github.com/users/dotnet/gists{/gist_id}","starred_url":"https://api.github.com/users/dotnet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dotnet/subscriptions","organizations_url":"https://api.github.com/users/dotnet/orgs","repos_url":"https://api.github.com/users/dotnet/repos","events_url":"https://api.github.com/users/dotnet/events{/privacy}","received_events_url":"https://api.github.com/users/dotnet/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/dotnet/corefx","description":"This repo contains the .NET Core foundational libraries, called CoreFX. It includes classes for collections, file systems, console, XML, async and many others.","fork":false,"url":"https://api.github.com/repos/dotnet/corefx","forks_url":"https://api.github.com/repos/dotnet/corefx/forks","keys_url":"https://api.github.com/repos/dotnet/corefx/keys{/key_id}","collaborators_url":"https://api.github.com/repos/dotnet/corefx/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/dotnet/corefx/teams","hooks_url":"https://api.github.com/repos/dotnet/corefx/hooks","issue_events_url":"https://api.github.com/repos/dotnet/corefx/issues/events{/number}","events_url":"https://api.github.com/repos/dotnet/corefx/events","assignees_url":"https://api.github.com/repos/dotnet/corefx/assignees{/user}","branches_url":"https://api.github.com/repos/dotnet/corefx/branches{/branch}","tags_url":"https://api.github.com/repos/dotnet/corefx/tags","blobs_url":"https://api.github.com/repos/dotnet/corefx/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/dotnet/corefx/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/dotnet/corefx/git/refs{/sha}","trees_url":"https://api.github.com/repos/dotnet/corefx/git/trees{/sha}","statuses_url":"https://api.github.com/repos/dotnet/corefx/statuses/{sha}","languages_url":"https://api.github.com/repos/dotnet/corefx/languages","stargazers_url":"https://api.github.com/repos/dotnet/corefx/stargazers","contributors_url":"https://api.github.com/repos/dotnet/corefx/contributors","subscribers_url":"https://api.github.com/repos/dotnet/corefx/subscribers","subscription_url":"https://api.github.com/repos/dotnet/corefx/subscription","commits_url":"https://api.github.com/repos/dotnet/corefx/commits{/sha}","git_commits_url":"https://api.github.com/repos/dotnet/corefx/git/commits{/sha}","comments_url":"https://api.github.com/repos/dotnet/corefx/comments{/number}","issue_comment_url":"https://api.github.com/repos/dotnet/corefx/issues/comments{/number}","contents_url":"https://api.github.com/repos/dotnet/corefx/contents/{+path}","compare_url":"https://api.github.com/repos/dotnet/corefx/compare/{base}...{head}","merges_url":"https://api.github.com/repos/dotnet/corefx/merges","archive_url":"https://api.github.com/repos/dotnet/corefx/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/dotnet/corefx/downloads","issues_url":"https://api.github.com/repos/dotnet/corefx/issues{/number}","pulls_url":"https://api.github.com/repos/dotnet/corefx/pulls{/number}","milestones_url":"https://api.github.com/repos/dotnet/corefx/milestones{/number}","notifications_url":"https://api.github.com/repos/dotnet/corefx/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/dotnet/corefx/labels{/name}","releases_url":"https://api.github.com/repos/dotnet/corefx/releases{/id}","created_at":"2014-11-06T23:42:48Z","updated_at":"2015-04-28T11:31:30Z","pushed_at":"2015-04-28T13:56:13Z","git_url":"git://github.com/dotnet/corefx.git","ssh_url":"git@github.com:dotnet/corefx.git","clone_url":"https://github.com/dotnet/corefx.git","svn_url":"https://github.com/dotnet/corefx","homepage":"","size":38407,"stargazers_count":7513,"watchers_count":7513,"language":"C#","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1493,"mirror_url":null,"open_issues_count":347,"forks":1493,"open_issues":347,"watchers":7513,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/dotnet/corefx/pulls/1557"},"html":{"href":"https://github.com/dotnet/corefx/pull/1557"},"issue":{"href":"https://api.github.com/repos/dotnet/corefx/issues/1557"},"comments":{"href":"https://api.github.com/repos/dotnet/corefx/issues/1557/comments"},"review_comments":{"href":"https://api.github.com/repos/dotnet/corefx/pulls/1557/comments"},"review_comment":{"href":"https://api.github.com/repos/dotnet/corefx/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/dotnet/corefx/pulls/1557/commits"},"statuses":{"href":"https://api.github.com/repos/dotnet/corefx/statuses/c61ee8a89babfad1c4c4a8a678f5cff3ae03e868"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":1,"additions":2890,"deletions":1553,"changed_files":68}}