{"action":"created","issue":{"url":"https://api.github.com/repos/nessos/FsPickler/issues/38","labels_url":"https://api.github.com/repos/nessos/FsPickler/issues/38/labels{/name}","comments_url":"https://api.github.com/repos/nessos/FsPickler/issues/38/comments","events_url":"https://api.github.com/repos/nessos/FsPickler/issues/38/events","html_url":"https://github.com/nessos/FsPickler/issues/38","id":56575860,"number":38,"title":"Support for serializing large datasets","user":{"login":"Rickasaurus","id":1429,"avatar_url":"https://avatars.githubusercontent.com/u/1429?v=3","gravatar_id":"","url":"https://api.github.com/users/Rickasaurus","html_url":"https://github.com/Rickasaurus","followers_url":"https://api.github.com/users/Rickasaurus/followers","following_url":"https://api.github.com/users/Rickasaurus/following{/other_user}","gists_url":"https://api.github.com/users/Rickasaurus/gists{/gist_id}","starred_url":"https://api.github.com/users/Rickasaurus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Rickasaurus/subscriptions","organizations_url":"https://api.github.com/users/Rickasaurus/orgs","repos_url":"https://api.github.com/users/Rickasaurus/repos","events_url":"https://api.github.com/users/Rickasaurus/events{/privacy}","received_events_url":"https://api.github.com/users/Rickasaurus/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2015-02-04T19:34:47Z","updated_at":"2015-02-06T00:28:23Z","closed_at":null,"body":"I was trying to use FsPickler to pull a largish (~27 million records) to disk and found that about 30 minutes in it failed with:\r\n\r\nNessos.FsPicker.FsPicklerException: Error serializing instance of type System.String[] --->  System.Runtime.Serialization.SerializationException: The internal array cannot expand to greater than Int32.MaxValue elements.\r\n\r\nThis occurs in a call to ObjectIDGenerator.Rehash() from ObjectIDGenerator.GetID(Object obj, Boolean& firstTime) which is called in FsPickler in CompositePickler`1.Write(WriteState state, String tag, T value) in CompositePickler.fs on line 189\r\n\r\nIt turns out this is a common problem in .NET, due to the internal use of ObjectIDGenerator to look for cycles. However, as this is a straight pull of a SQL database I can guarantee there's no cycles in this case.  \r\n\r\nOn a side note, the behavior of ObjectIDGenerator is poor in the failure case. It does not release the memory from its table thus leaving a big chunk of ram in use."},"comment":{"url":"https://api.github.com/repos/nessos/FsPickler/issues/comments/73159399","html_url":"https://github.com/nessos/FsPickler/issues/38#issuecomment-73159399","issue_url":"https://api.github.com/repos/nessos/FsPickler/issues/38","id":73159399,"user":{"login":"Rickasaurus","id":1429,"avatar_url":"https://avatars.githubusercontent.com/u/1429?v=3","gravatar_id":"","url":"https://api.github.com/users/Rickasaurus","html_url":"https://github.com/Rickasaurus","followers_url":"https://api.github.com/users/Rickasaurus/followers","following_url":"https://api.github.com/users/Rickasaurus/following{/other_user}","gists_url":"https://api.github.com/users/Rickasaurus/gists{/gist_id}","starred_url":"https://api.github.com/users/Rickasaurus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Rickasaurus/subscriptions","organizations_url":"https://api.github.com/users/Rickasaurus/orgs","repos_url":"https://api.github.com/users/Rickasaurus/repos","events_url":"https://api.github.com/users/Rickasaurus/events{/privacy}","received_events_url":"https://api.github.com/users/Rickasaurus/received_events","type":"User","site_admin":false},"created_at":"2015-02-06T00:28:23Z","updated_at":"2015-02-06T00:28:23Z","body":"Ahh, it all makes sense now. Certainly it would be ideal to have something improved, but I understand that can be a lot of work. I'd be pretty happy with just a way to bypass it. "}}