{"action":"created","issue":{"url":"https://api.github.com/repos/stan-dev/stan/issues/1321","labels_url":"https://api.github.com/repos/stan-dev/stan/issues/1321/labels{/name}","comments_url":"https://api.github.com/repos/stan-dev/stan/issues/1321/comments","events_url":"https://api.github.com/repos/stan-dev/stan/issues/1321/events","html_url":"https://github.com/stan-dev/stan/issues/1321","id":59417476,"number":1321,"title":"Tweak format of written .csv files","user":{"login":"bgoodri","id":3386282,"avatar_url":"https://avatars.githubusercontent.com/u/3386282?v=3","gravatar_id":"","url":"https://api.github.com/users/bgoodri","html_url":"https://github.com/bgoodri","followers_url":"https://api.github.com/users/bgoodri/followers","following_url":"https://api.github.com/users/bgoodri/following{/other_user}","gists_url":"https://api.github.com/users/bgoodri/gists{/gist_id}","starred_url":"https://api.github.com/users/bgoodri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgoodri/subscriptions","organizations_url":"https://api.github.com/users/bgoodri/orgs","repos_url":"https://api.github.com/users/bgoodri/repos","events_url":"https://api.github.com/users/bgoodri/events{/privacy}","received_events_url":"https://api.github.com/users/bgoodri/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/stan-dev/stan/milestones/2","html_url":"https://github.com/stan-dev/stan/milestones/Future","labels_url":"https://api.github.com/repos/stan-dev/stan/milestones/2/labels","id":341383,"number":2,"title":"Future","description":"For tasks and features not scheduled for the next release.","creator":{"login":"bob-carpenter","id":3383807,"avatar_url":"https://avatars.githubusercontent.com/u/3383807?v=3","gravatar_id":"","url":"https://api.github.com/users/bob-carpenter","html_url":"https://github.com/bob-carpenter","followers_url":"https://api.github.com/users/bob-carpenter/followers","following_url":"https://api.github.com/users/bob-carpenter/following{/other_user}","gists_url":"https://api.github.com/users/bob-carpenter/gists{/gist_id}","starred_url":"https://api.github.com/users/bob-carpenter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bob-carpenter/subscriptions","organizations_url":"https://api.github.com/users/bob-carpenter/orgs","repos_url":"https://api.github.com/users/bob-carpenter/repos","events_url":"https://api.github.com/users/bob-carpenter/events{/privacy}","received_events_url":"https://api.github.com/users/bob-carpenter/received_events","type":"User","site_admin":false},"open_issues":154,"closed_issues":0,"state":"open","created_at":"2013-05-24T19:02:14Z","updated_at":"2015-07-22T18:01:02Z","due_on":null,"closed_at":null},"comments":15,"created_at":"2015-03-01T21:07:15Z","updated_at":"2015-08-18T17:57:00Z","closed_at":null,"body":"The current output of the .csv files when you include the warmup period looks like\r\n```\r\n# lines\r\n# of\r\n...\r\n# metadata\r\nlp__,accept_stat__,...\r\nnumber,number,...\r\nnumber,number,...\r\n...\r\nnumber,number,...\r\n# Adaptation terminated\r\n# Step size = number\r\n# Diagonal elements of inverse mass matrix:\r\n# number, number, ...\r\nnumber,number,...\r\nnumber,number,...\r\n...\r\nnumber,number,...\r\n# Elapsed Time: number seconds (Warm-up)\r\n#               number seconds (Sampling)\r\n#               number seconds (Total)\r\n```\r\nThis format is inefficient in time and RAM to read (e.g. into R) because it has to scan each line as text for the leading pound sign. Also, it does not play nicely with the optimized `fread` function in the data.table package that was apparently used by \r\nhttps://groups.google.com/d/msg/stan-users/urZeEMjjPHM/R0S41-hQbPMJ\r\n\r\nI think it would be better to \r\n\r\n1. Put a blank line in the .csv file after the last iteration\r\n2. Put the four lines of adaptation info after that instead of in the middle of the file\r\n\r\nThus, the format would look like\r\n```\r\n# lines\r\n# of\r\n...\r\n# metadata\r\nlp__,accept_stat__,...\r\nnumber,number,...\r\nnumber,number,...\r\n...\r\nnumber,number,...\r\n\r\n# Adaptation terminated\r\n# Step size = number\r\n# Diagonal elements of inverse mass matrix:\r\n# number, number, ...\r\n# Elapsed Time: number seconds (Warm-up)\r\n#               number seconds (Sampling)\r\n#               number seconds (Total)\r\n```\r\nDoes anything depend on the absence of blank lines or the presence of the adaptation info being in between the warmup iterations and the sampled iterations? "},"comment":{"url":"https://api.github.com/repos/stan-dev/stan/issues/comments/132296641","html_url":"https://github.com/stan-dev/stan/issues/1321#issuecomment-132296641","issue_url":"https://api.github.com/repos/stan-dev/stan/issues/1321","id":132296641,"user":{"login":"bob-carpenter","id":3383807,"avatar_url":"https://avatars.githubusercontent.com/u/3383807?v=3","gravatar_id":"","url":"https://api.github.com/users/bob-carpenter","html_url":"https://github.com/bob-carpenter","followers_url":"https://api.github.com/users/bob-carpenter/followers","following_url":"https://api.github.com/users/bob-carpenter/following{/other_user}","gists_url":"https://api.github.com/users/bob-carpenter/gists{/gist_id}","starred_url":"https://api.github.com/users/bob-carpenter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bob-carpenter/subscriptions","organizations_url":"https://api.github.com/users/bob-carpenter/orgs","repos_url":"https://api.github.com/users/bob-carpenter/repos","events_url":"https://api.github.com/users/bob-carpenter/events{/privacy}","received_events_url":"https://api.github.com/users/bob-carpenter/received_events","type":"User","site_admin":false},"created_at":"2015-08-18T17:57:00Z","updated_at":"2015-08-18T17:57:00Z","body":"I think th elonger term plan is to put the adaptation info\nin a separate file altogether.  I don't at all understand why\nit's a problem where it goes, but I don't like it as a comment\nmixed into an otherwise coherent CSV file.\n\n\n> On Aug 18, 2015, at 7:13 AM, bgoodri <notifications@github.com> wrote:\n> \n> Fast loading of CSVs is needed for a shinystan app that uploads a .csv file and creates an object that it can launch on while the meter is running.\n> Moreover, extremely large .csv files are effectively unloadable even locally because it can take hours and thereby force users to do things like https://groups.google.com/d/msg/stan-users/urZeEMjjPHM/R0S41-hQbPMJ .\n> Not-too-large .csv files can be effectively unloadable on a Network File System.\n> \n> It is only your preference for a spec that follows the sequential order of the sampler that is preventing us today from patching src/cmdstan/command.hpp from\n> \n> @@ -722,7 +722,6 @@ namespace stan {\n>          if (adapt_engaged) {\n>            dynamic_cast<stan::mcmc::base_adapter*>(sampler_ptr)\n>              ->disengage_adaptation();\n> -          writer.write_adapt_finish(sampler_ptr);\n>          }\n> \n> to\n> \n> @@ -740,6 +739,7 @@ namespace stan {\n>          sampleDeltaT = static_cast<double>(end - start) / CLOCKS_PER_SEC;\n> \n>          writer.write_timing(warmDeltaT, sampleDeltaT);\n> +        writer.write_adapt_finish(sampler_ptr);\n> \n> so that the resulting .csv file looks like\n> \n> # stan_version_major = 2\n> # stan_version_minor = 7\n> # stan_version_patch = 0\n> # model = bernoulli_model\n> # method = sample (Default)\n> #   sample\n> #     num_samples = 1000 (Default)\n> #     num_warmup = 1000 (Default)\n> #     save_warmup = 1\n> #     thin = 1 (Default)\n> #     adapt\n> #       engaged = 1 (Default)\n> #       gamma = 0.050000000000000003 (Default)\n> #       delta = 0.80000000000000004 (Default)\n> #       kappa = 0.75 (Default)\n> #       t0 = 10 (Default)\n> #       init_buffer = 75 (Default)\n> #       term_buffer = 50 (Default)\n> #       window = 25 (Default)\n> #     algorithm = hmc (Default)\n> #       hmc\n> #         engine = nuts (Default)\n> #           nuts\n> #             max_depth = 10 (Default)\n> #         metric = diag_e (Default)\n> #         stepsize = 1 (Default)\n> #         stepsize_jitter = 0 (Default)\n> # id = 0 (Default)\n> # data\n> #   file = bernoulli.data.R\n> # init = 2 (Default)\n> # random\n> #   seed = 1198819904\n> # output\n> #   file = output.csv (Default)\n> #   diagnostic_file =  (Default)\n> #   refresh = 100 (Default)\n> lp__,accept_stat__,stepsize__,treedepth__,n_leapfrog__,n_divergent__,theta\n> -16.649,5.72792e-17,2,2,3,0,0.83286\n> ... 1998 consecutive lines of comma-delimited numbers like ADVI does ...\n> -7.024,0.955167,0.956729,1,1,0,0.349523\n> \n> #  Elapsed Time: 0.022471 seconds (Warm-up)\n> #                0.022418 seconds (Sampling)\n> #                0.044889 seconds (Total)\n> \n> # Adaptation terminated\n> # Step size = 0.956729\n> # Diagonal elements of inverse mass matrix:\n> # 0.409482\n> \n> And that is not even that asequential because the (currently irrelevant, except as a 3rd-order diagnostic) stepsize and diagonal elements of the mass matrix are the same at the end of the sampling as they are at the end of the warmup. And that does not prevent progress toward the future where to become more than a 3rd-order diagonstic the step size and diagonal elements are presumably going to have to be written to a file in the same format as the data file or the init file so that it can be passed back. And it does not fail any unit test in cmdstan, although maybe some interface is currently expecting the adaptation info to be placed in the middle of the file. And it is a net decrease in hackishness. And it fixes an actual problem.\n> \n> —\n> Reply to this email directly or view it on GitHub.\n> \n\n"}}