{"action":"created","issue":{"url":"https://api.github.com/repos/OpenCPN/OpenCPN/issues/315","labels_url":"https://api.github.com/repos/OpenCPN/OpenCPN/issues/315/labels{/name}","comments_url":"https://api.github.com/repos/OpenCPN/OpenCPN/issues/315/comments","events_url":"https://api.github.com/repos/OpenCPN/OpenCPN/issues/315/events","html_url":"https://github.com/OpenCPN/OpenCPN/pull/315","id":93107354,"number":315,"title":"Optimized chart parsing","user":{"login":"seandepagnier","id":2342579,"avatar_url":"https://avatars.githubusercontent.com/u/2342579?v=3","gravatar_id":"","url":"https://api.github.com/users/seandepagnier","html_url":"https://github.com/seandepagnier","followers_url":"https://api.github.com/users/seandepagnier/followers","following_url":"https://api.github.com/users/seandepagnier/following{/other_user}","gists_url":"https://api.github.com/users/seandepagnier/gists{/gist_id}","starred_url":"https://api.github.com/users/seandepagnier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/seandepagnier/subscriptions","organizations_url":"https://api.github.com/users/seandepagnier/orgs","repos_url":"https://api.github.com/users/seandepagnier/repos","events_url":"https://api.github.com/users/seandepagnier/events{/privacy}","received_events_url":"https://api.github.com/users/seandepagnier/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-07-05T12:21:53Z","updated_at":"2015-07-08T06:52:47Z","closed_at":"2015-07-07T02:14:30Z","pull_request":{"url":"https://api.github.com/repos/OpenCPN/OpenCPN/pulls/315","html_url":"https://github.com/OpenCPN/OpenCPN/pull/315","diff_url":"https://github.com/OpenCPN/OpenCPN/pull/315.diff","patch_url":"https://github.com/OpenCPN/OpenCPN/pull/315.patch"},"body":"This is just scratching the surface, but it is now significantly faster (7 seconds instead of 60 seconds) to update the chart database with thousands of raster charts, and about twice as fast to build the SENC files (which even at this speed is still incredibly annoying when panning for the first time) \r\n"},"comment":{"url":"https://api.github.com/repos/OpenCPN/OpenCPN/issues/comments/119460047","html_url":"https://github.com/OpenCPN/OpenCPN/pull/315#issuecomment-119460047","issue_url":"https://api.github.com/repos/OpenCPN/OpenCPN/issues/315","id":119460047,"user":{"login":"seandepagnier","id":2342579,"avatar_url":"https://avatars.githubusercontent.com/u/2342579?v=3","gravatar_id":"","url":"https://api.github.com/users/seandepagnier","html_url":"https://github.com/seandepagnier","followers_url":"https://api.github.com/users/seandepagnier/followers","following_url":"https://api.github.com/users/seandepagnier/following{/other_user}","gists_url":"https://api.github.com/users/seandepagnier/gists{/gist_id}","starred_url":"https://api.github.com/users/seandepagnier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/seandepagnier/subscriptions","organizations_url":"https://api.github.com/users/seandepagnier/orgs","repos_url":"https://api.github.com/users/seandepagnier/repos","events_url":"https://api.github.com/users/seandepagnier/events{/privacy}","received_events_url":"https://api.github.com/users/seandepagnier/received_events","type":"User","site_admin":false},"created_at":"2015-07-08T06:52:47Z","updated_at":"2015-07-08T06:52:47Z","body":"I wanted to continue the discussion on parsing SENC.\r\n\r\nI found the speed to parse (and later load the stored SENC) can be considerably improved if we omit the items not selected for display.  For example, the M_QUAL field is practically never used but frequency contains complicated polygons which take a very long time first to retrieve from the chart file then to convert to triangles with the relatively slow douglas peucker algorithm and the glu tesselator, and the resulting data is never even used!  When I omitted just this one field I noticed 50% speedup in generating the S57 for many charts.\r\n\r\nWhat I propose is:\r\n1.  We omit the fields not selected by the user.  When the user changes the selections, the S57 is rebuilt (or better yet, appended)\r\n2.  Once read, the data isn't again written to disk as S57 then read again as it is now, but instead read directly in to be used, and it can be written to disk in a background thread.\r\n3.  We should completely change the format of the S57 file.  I have no idea if there is some standard for this or not, if so we should make our own standard and use that because the current format is incredibly inefficient for several reasons:  \r\n   A)  There is no table, so the whole file must be read even if you only care about certain fields\r\n   B)  It requires lots of string comparisons which are slow\r\n   C)  It just doesn't map into opengl efficiently.  The whole vector chart rendering is a relic of the dc-mode\r\n         rendering pass and not designed for opengl.  The S57 cache file should have binary VBO data that\r\n         gets directly uploaded to the gpu.  This would mean the fewest possible number of arrays and the least\r\n         possible number of gl calls which we aren't even anywhere close to at the moment.\r\n\r\n4.  All of the work should be done in a background thread, so the program doesn't get stuck, instead it might display a grey area, or possibly even a progress indicator on the chart itself (or on the chart bar) Also more than one chart can be parsed in parallel (case of quilt) and then the thumbnail can also be done as a separate lower priority job.\r\n5.  The S57 is compressed using lz4 to reduce disk space, currently the S57 generated is larger than the original chart in many cases!  it would also improve loading time because the data on disk is less especially for systems using slower storage.\r\n\r\nAny comments or suggestions appreciated, basically some of these are easy to accomplish, but some of it really not.\r\nI wanted to continue the discussion on parsing SENC.\r\n\r\nI found the speed to parse (and later load the stored SENC) can be considerably improved if we omit the items not selected for display.  For example, the M_QUAL field is practically never used but frequency contains complicated polygons which take a very long time first to retrieve from the chart file then to convert to triangles with the relatively slow douglas peucker algorithm and the glu tesselator, and the resulting data is never even used!  When I omitted just this one field I noticed 50% speedup in generating the S57 for many charts.\r\n\r\nWhat I propose is:\r\n1.  We omit the fields not selected by the user.  When the user changes the selections, the S57 is rebuilt (or better yet, appended)\r\n2.  Once read, the data isn't again written to disk as S57 then read again as it is now, but instead read directly in to be used, and it can be written to disk in a background thread.\r\n3.  We should completely change the format of the S57 file.  I have no idea if there is some standard for this or not, if so we should make our own standard and use that because the current format is incredibly inefficient for several reasons:  \r\n   A)  There is no table, so the whole file must be read even if you only care about certain fields\r\n   B)  It requires lots of string comparisons which are slow\r\n   C)  It just doesn't map into opengl efficiently.  The whole vector chart rendering is a relic of the dc-mode\r\n         rendering pass and not designed for opengl.  The S57 cache file should have binary VBO data that\r\n         gets directly uploaded to the gpu.  This would mean the fewest possible number of arrays and the least\r\n         possible number of gl calls which we aren't even anywhere close to at the moment.\r\n\r\n4.  All of the work should be done in a background thread, so the program doesn't get stuck, instead it might display a grey area, or possibly even a progress indicator on the chart itself (or on the chart bar) Also more than one chart can be parsed in parallel (case of quilt) and then the thumbnail can also be done as a separate lower priority job.\r\n5.  The S57 is compressed using lz4 (or possibly another) to reduce disk space, currently the S57 generated is larger than the original chart in many cases!  it would also improve loading time because the data on disk is less especially for systems using slower storage.\r\n\r\nAny comments or suggestions appreciated, basically some of these are easy to accomplish, but some of it really not."}}