{"action":"created","issue":{"url":"https://api.github.com/repos/Unidata/netcdf-c/issues/166","labels_url":"https://api.github.com/repos/Unidata/netcdf-c/issues/166/labels{/name}","comments_url":"https://api.github.com/repos/Unidata/netcdf-c/issues/166/comments","events_url":"https://api.github.com/repos/Unidata/netcdf-c/issues/166/events","html_url":"https://github.com/Unidata/netcdf-c/pull/166","id":119563196,"number":166,"title":"Max var dim","user":{"login":"gsjaardema","id":897294,"avatar_url":"https://avatars.githubusercontent.com/u/897294?v=3","gravatar_id":"","url":"https://api.github.com/users/gsjaardema","html_url":"https://github.com/gsjaardema","followers_url":"https://api.github.com/users/gsjaardema/followers","following_url":"https://api.github.com/users/gsjaardema/following{/other_user}","gists_url":"https://api.github.com/users/gsjaardema/gists{/gist_id}","starred_url":"https://api.github.com/users/gsjaardema/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gsjaardema/subscriptions","organizations_url":"https://api.github.com/users/gsjaardema/orgs","repos_url":"https://api.github.com/users/gsjaardema/repos","events_url":"https://api.github.com/users/gsjaardema/events{/privacy}","received_events_url":"https://api.github.com/users/gsjaardema/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-11-30T19:31:50Z","updated_at":"2015-12-09T14:29:42Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/Unidata/netcdf-c/pulls/166","html_url":"https://github.com/Unidata/netcdf-c/pull/166","diff_url":"https://github.com/Unidata/netcdf-c/pull/166.diff","patch_url":"https://github.com/Unidata/netcdf-c/pull/166.patch"},"body":"Related to the discussion in #148.\r\n\r\nAdds two defines \"NC_IGNORE_MAX_DIMS\" and \"NC_IGNORE_MAX_VARS\" which can be passed in the cmode argument of the nc_open call.  If present, then the application is verifying that it knows that it is creating a non-standard netcdf classic file which may not work correctly with some tools.  If one or both of the defines are present, then the checks when adding a new dimension or variable will not abort if the dimension count exceeds NC_MAX_DIMS or the variable count exceeds NC_MAX_VARS.\r\n\r\nNote that any tools that cannot handle a classic file with dimensions or variables exceeding the NC_MAX_VARS or NC_MAX_DIMS values will likely also fail if given a non-classic file with the same dimension and/or variable count, so I don't think that this patch is as bad as it may seem initially.\r\n\r\nThe alternative of just creating non-classic netcdf-4 based files (using hdf5) is not always desireable due to file sizes, output speed, need to use parallel-netcdf, or that fact that some of our clients want just the netcdf library without the issues involved in also building hdf5.  (This is not a complaint about netcdf-4 based on hdf5; we use it a lot and like the added functionality it gives us)."},"comment":{"url":"https://api.github.com/repos/Unidata/netcdf-c/issues/comments/163259531","html_url":"https://github.com/Unidata/netcdf-c/pull/166#issuecomment-163259531","issue_url":"https://api.github.com/repos/Unidata/netcdf-c/issues/166","id":163259531,"user":{"login":"gdsjaar","id":12431173,"avatar_url":"https://avatars.githubusercontent.com/u/12431173?v=3","gravatar_id":"","url":"https://api.github.com/users/gdsjaar","html_url":"https://github.com/gdsjaar","followers_url":"https://api.github.com/users/gdsjaar/followers","following_url":"https://api.github.com/users/gdsjaar/following{/other_user}","gists_url":"https://api.github.com/users/gdsjaar/gists{/gist_id}","starred_url":"https://api.github.com/users/gdsjaar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gdsjaar/subscriptions","organizations_url":"https://api.github.com/users/gdsjaar/orgs","repos_url":"https://api.github.com/users/gdsjaar/repos","events_url":"https://api.github.com/users/gdsjaar/events{/privacy}","received_events_url":"https://api.github.com/users/gdsjaar/received_events","type":"User","site_admin":false},"created_at":"2015-12-09T14:29:42Z","updated_at":"2015-12-09T14:29:42Z","body":"I agree that there should be some indication that a file has a variable or dimension count which exceeds the values of NC_MAX_VARS or NC_MAX_DIMS.  My thoughts were that the values in the *READING* library would be used for that warning.  In that case, the file itself would have no indication, but when it was opened for reading it would check the variable and dimension count; compare those against the NC_MAX_VARS and NC_MAX_DIMS values and if either was exceeded, a warning or error could/would be returned from the ex_open command.\n\nThis isn't optimal though since a reading application could easily be written to handle this case, so it shouldn't error out just because of the counts being exceeded.  A warning being printed would probably be better.  An application that didn't want the warning to be output could pass the NC_IGNORE_MAX_DIMS and NC_IGNORE_MAX_VARS flags to the ex_open call similarly to the ex_create calls.\n\nI haven't looked at the computational cost of counting the dims and vars, but if it is relatively cheap, it could be done during ex_open to provide this check.\n\nI'm not sure I understand the paragraph about the maximum values being limited by machine architecture.  As I see it, the only limit would theoretically be the maximum value of an integer which is used to return the counts.  There are some potential memory issues, but I can currently generate a netcdf file that is too large to be read on certain machines even if I don't exceed the dimension and variable counts.  If you could explain further what you mean in that paragraph, it would be helpful.\n\nThe other benefit of the counts being compared at read time is that this would open the possibility of changing the values of NC_MAX_VARS and NC_MAX_DIMS in the future; it also fits in better with the value of NC_MAX_VARS being a *suggested* maximum instead of an absolute maximum as it states in the documentation and the source code.\n\nIf you want me to pursue a pull request that prints the warning on read, let me know.\n..Greg\n\n\n--\n\"A supercomputer is a device for turning compute-bound problems into I/O-bound problems\"\n\nFrom: Ward Fisher <notifications@github.com<mailto:notifications@github.com>>\nReply-To: Unidata/netcdf-c <reply@reply.github.com<mailto:reply@reply.github.com>>\nDate: Tuesday, December 8, 2015 at 11:24 PM\nTo: Unidata/netcdf-c <netcdf-c@noreply.github.com<mailto:netcdf-c@noreply.github.com>>\nSubject: [EXTERNAL] Re: [netcdf-c] Max var dim (#166)\n\n\nIf I were convinced this was the path we wanted to take, I'd still have concerns about accepting this pull request. It doesn't capture the state of whether or not the defines were ignored in the metadata of the resultant files. This could result in netcdf being unable to read certain netcdf files, with no indication as to why the file is apparently corrupt. For this reason, I'd think we'd need to capture this information as a global attribute for diagnostic/forensic issues that might arise.\n\nIgnoring global attributes and leaving these maximum values to be limited by the machine architecture, with no information captured regarding what those maximums might be, could also lead to files which are now machine/platform/architecture dependent. Going back to our previous conversation, this isn't something we can do.\n\nIt seems like we would need to define a new data type, NC_NOLIMITS for example, that we could safely use for this purpose. It would be similar to the existing file type, except it wasn't defined by these limits (whether they are enforced or not).\n\n-\nReply to this email directly or view it on GitHub<https://github.com/Unidata/netcdf-c/pull/166#issuecomment-163105011>.\n"}}