{"action":"created","issue":{"url":"https://api.github.com/repos/thebuzzmedia/universal-binary-json/issues/60","labels_url":"https://api.github.com/repos/thebuzzmedia/universal-binary-json/issues/60/labels{/name}","comments_url":"https://api.github.com/repos/thebuzzmedia/universal-binary-json/issues/60/comments","events_url":"https://api.github.com/repos/thebuzzmedia/universal-binary-json/issues/60/events","html_url":"https://github.com/thebuzzmedia/universal-binary-json/issues/60","id":51493736,"number":60,"title":"DISCUSSION - Reconciling optimized headers, flexible STC, n-dimensional arrays and schemas","user":{"login":"thebuzzmedia","id":533480,"avatar_url":"https://avatars.githubusercontent.com/u/533480?v=3","gravatar_id":"","url":"https://api.github.com/users/thebuzzmedia","html_url":"https://github.com/thebuzzmedia","followers_url":"https://api.github.com/users/thebuzzmedia/followers","following_url":"https://api.github.com/users/thebuzzmedia/following{/other_user}","gists_url":"https://api.github.com/users/thebuzzmedia/gists{/gist_id}","starred_url":"https://api.github.com/users/thebuzzmedia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thebuzzmedia/subscriptions","organizations_url":"https://api.github.com/users/thebuzzmedia/orgs","repos_url":"https://api.github.com/users/thebuzzmedia/repos","events_url":"https://api.github.com/users/thebuzzmedia/events{/privacy}","received_events_url":"https://api.github.com/users/thebuzzmedia/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/thebuzzmedia/universal-binary-json/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2014-12-09T22:51:48Z","updated_at":"2015-02-28T16:40:59Z","closed_at":null,"body":"This needs to be a bigger discussion, so filing here for thoughts...\r\n\r\n## Big ideas in-flight...\r\n1. #48 - Allow STC headers to repeating inside a container to more closely match JSON container design.\r\n2. #51 - Redefine container headers to have a single, consistent and concise format (#<TYPE><SIZE>)\r\n3. #43 [detail in comment](https://github.com/thebuzzmedia/universal-binary-json/issues/43#issuecomment-48524976) - Proposal to allow pre-defining of container 'structure' upfront - potentially massive space savings in highly optimized cases (currently a case UBJSON is not great at)\r\n4. #50 Alex's `TypeSpec` proposal; basically schemas. I think somewhat ties into the conversation around Point 3 above - basically allowing you to upfront define 'structure' of a container completely upfront - then the entire payload is just hard data.\r\n\r\n## My question\r\nWe are getting close to finalizing Point 1 above and allowing the headers (#51) to repeat during a container... that is great, but it doesn't address the potentially HUGE space savings discussed in Points 3 and 4 above and I wanted to make sure we weren't moving forward with a change that was either pointless (to be undone later) or conflicting with potentially addressing 3 and 4 later.\r\n\r\nThis issue is a forum for us to discuss where all these concerns are heading towards."},"comment":{"url":"https://api.github.com/repos/thebuzzmedia/universal-binary-json/issues/comments/76533536","html_url":"https://github.com/thebuzzmedia/universal-binary-json/issues/60#issuecomment-76533536","issue_url":"https://api.github.com/repos/thebuzzmedia/universal-binary-json/issues/60","id":76533536,"user":{"login":"Steve132","id":877616,"avatar_url":"https://avatars.githubusercontent.com/u/877616?v=3","gravatar_id":"","url":"https://api.github.com/users/Steve132","html_url":"https://github.com/Steve132","followers_url":"https://api.github.com/users/Steve132/followers","following_url":"https://api.github.com/users/Steve132/following{/other_user}","gists_url":"https://api.github.com/users/Steve132/gists{/gist_id}","starred_url":"https://api.github.com/users/Steve132/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Steve132/subscriptions","organizations_url":"https://api.github.com/users/Steve132/orgs","repos_url":"https://api.github.com/users/Steve132/repos","events_url":"https://api.github.com/users/Steve132/events{/privacy}","received_events_url":"https://api.github.com/users/Steve132/received_events","type":"User","site_admin":false},"created_at":"2015-02-28T16:40:59Z","updated_at":"2015-02-28T16:40:59Z","body":"So I know the benevolent dictator decided not to pursue the ND-array thing for now, so this doesn't matter, but I wanted to respond to this anyway.  My response isn't intended as a way to rekindle that debate.\r\n\r\n>the main thing JSON is missing as a transport format in practice is the moral equivalent of a DIFF. And that what I think UBJSON is interested in becoming, to actually get used and provide something valuable, is a useable transport format, not a binary JSON document translator. In that context, chunked transfers, and more, are extremely valuable.\r\n\r\nI'm sorry, but \"not a binary json format\" makes no sense to me, it's literally in the name.  \"Universal *binary json*\"\r\n\r\nI don't agree with you at all that the transport protocol layer and the data format layer are at all the same.  It's not the job of say, xml, or html, to handle HTTP/1.0 chunked transfers.  \r\n\r\nUBJSON may be useful in some contexts as a way to communicate over the web, but that is absolutely not what it's about.  It's a data format, that's *IT*.\r\n\r\nAs a matter of fact, I can't think of ANY data format that includes mechanisms for the underlying transfer protocol.  Separation of concerns is a hugely important thing.\r\n\r\nIf you want to diff a file, thats what diff format is.  UBJSON and JSON and data formats are NOT diff formats.  \r\n\r\nI respectfully just don't understand your philosophy at all.  \r\n\r\n>enabling header information to describe the 1D array data as either row based or column based is also important. My experience says you've got a 50/50 shot of getting either type of in-memory representation depending on the creators preference or processing algorithms. \r\n\r\nThere's no such thing as a column based or row based in-memory representation.  There is simply a linear data.  Column-based or row-based is simply a way of defining how to interpret the dimension array.  When computing the linear index, is the smallest stride dimension at the front of the dimension array or the back.   My UBJSON spec says the smallest stride dimension is at the front, which makes it column major. \r\n\r\nThe reason it is not necessary to allow the user to specify this in UBJSON N-D arrays is that the user can load the data from memory using either convention.  If their application requires accessing the data in a row-major way the user can simply reverse the dimension array before and and after.\r\n\r\nIntegrating that choice into the data format standard is completely un-necessary and complicated.\r\n\r\n>If you're going to take on optimizing matrix transfers, then consideration must also be given to sparse matrices. That's often the main thing missing from easily using JSON in practice, I have to send the sparse matrix as an object, and then recreate it as a sparse matrix on the other side. It'd be nice if that was more transparent and yes I think the spec ought to support the idea directly if it wants to transport matrices efficiently.\r\n\r\nNo, because the reason to transport matrices efficiently is not to create a general-purpose matrix format.  UBJSON is not matrix-math specific.  \r\n\r\nThe right way to think about UBJSON is that there are abstraction layers.    There's the application layer, which is the layer at which users specify their format and data using UBJSON primitives.  Their application is responsible for interpreting and validating those primitives as higher-level structures.  Good examples of this might be specifying a User type as an object, or maybe some kind of RPC mechansm that uses an object `{'method':'atan2','args':[2.0,2.3]}`  Or something else. \r\n\r\nUBJSON can't be concerned about this.\r\n\r\nThen the next layer is the data format layer.  This is the layer that UBJSON is supposed to help.  It provides efficient serialization of the set of primitives necessary to represent the application-level objects efficiently.\r\n\r\nThe next layer is the transport layer.  This is how the binary data is literally stored on disk, or encrypted, or the protocols defining how it is transported over the web, or diffed, or versioned.  \r\n\r\nUBJSON cannot be concerned about this either.\r\n\r\nIt seems to me that a lot of our disagreement seems to be that we have different philosophies about the different 'layers'.   I believe very strongly in keeping UBJSON limited to it's seperation of concerns whereas you are arguing to make it bleed into the layers above and below, which I disagree with strongly.\r\n\r\nAnyway, the reason why the ND-array proposal is important is not that 'numerical problems involving matrices are cool and we should make UBJSON into matlab'.  It's that it looks at an INCREDIBLY common Application-level structure that is repeated across dozens of applications (I would argue its present in literally every application that stands to benefit from UBJSON, and applications that don't have it wouldn't use UBJSON anyway).   It says \"This application-level structure is nearly ubiquitous across all our target applications, and there's no way to do it natively in UBJSON. Is there real benefit of some kind to making it into a primitive?  Is the benefit worth the cost?\"  I made a case already that in my opinion the answer is yes.\r\n\r\nHowever, since it's not just about 'lets have matrix math' then we have to address the same questions for the idea of making sparse arrays native in UBJSON.  In my opinion the application-level structure of a sparse array already has an equivalent UBJSON-level primitive that is efficient enough for that use (an object), and I've seen sparse arrays used so rarely in real-world data that I don't believe the costs of 'upgrading' that application-level structure to primitive level is at all worth it.  Most importantly, simply having an nd-array header doesn't imply we need to support sparse arrays natively just because they are both matrices, any more than having arrays implies we need to support binary trees natively because they are both sorted lists.\r\n\r\nHowever, interestingly, the ND-array proposal is *just that awesome* because even if I agreed with you, (which I don't), it becomes possible to add sparse matrices to the ND-array proposal by adding the following lines to the spec (which was one of my 3 suggestions on ways to handle objects in the proposal)\r\n\r\n    When an object contains a '@' header, the number of dimensions d is read.  Then, each object has d strings (instead of one string) as preceeding the value.\r\n\r\nI don't believe that's the best way to handle the '@' case with objects, but if I did then it has the benefits of 1) consistency in the header with array, 2) automatic native support for representing sparse ND-arrays efficiently as a primitive, just like you want.\r\n\r\n>What's plainly obvious is that JSON, as defined, will never, ever, get used in any sanely competent practice using ND arrays. The sane way to deal with transmitting this kind of data really efficiently is using 1D arrays and adding header information to describe it. So I recommend making exactly that kind of an \r\n\r\nYou and I agree on this.\r\n\r\n>Now whatever is deemed proper can be put into the header object to describe the efficient transfer for many kinds of arrays, including sparse descriptors. The specified [length] is the number of elements in the transferred 1D array and has nothing to do with the array size that is actually defined in the header (unless the header is empty in which case [length] is the size of the 1D array).\r\n\r\nI challenge you to come up with a truly useful piece of data that would be found in this object that is not already semantically contained in the type of the array and the dimensions.   \r\n\r\nSeriously, like I already said, you can add sparsity by adding the consistent header to objects like I said before, and the column-major order or row-major order is not a relevant detail for the primitive level construct (at the application level, maybe, but its not a relevant detail) and I can't currently think of ANYTHING else that would even *potentially* be useful to describe a layout.  Maybe stride or something but that makes no sense in a serialization context because there's no gaps in the data.\r\n\r\nAdding a 'parse an object in the header' also really breaks consistency with the way UBJSON parses currently and adds (as you mentioned) some extra [{][}] bytes at least."}}