{"action":"created","issue":{"url":"https://api.github.com/repos/winder/Universal-G-Code-Sender/issues/302","labels_url":"https://api.github.com/repos/winder/Universal-G-Code-Sender/issues/302/labels{/name}","comments_url":"https://api.github.com/repos/winder/Universal-G-Code-Sender/issues/302/comments","events_url":"https://api.github.com/repos/winder/Universal-G-Code-Sender/issues/302/events","html_url":"https://github.com/winder/Universal-G-Code-Sender/issues/302","id":111402980,"number":302,"title":"Huge file support - UGS to use constant memory!","user":{"login":"winder","id":125509,"avatar_url":"https://avatars.githubusercontent.com/u/125509?v=3","gravatar_id":"","url":"https://api.github.com/users/winder","html_url":"https://github.com/winder","followers_url":"https://api.github.com/users/winder/followers","following_url":"https://api.github.com/users/winder/following{/other_user}","gists_url":"https://api.github.com/users/winder/gists{/gist_id}","starred_url":"https://api.github.com/users/winder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/winder/subscriptions","organizations_url":"https://api.github.com/users/winder/orgs","repos_url":"https://api.github.com/users/winder/repos","events_url":"https://api.github.com/users/winder/events{/privacy}","received_events_url":"https://api.github.com/users/winder/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/winder/Universal-G-Code-Sender/labels/planned-enhancement","name":"planned-enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-10-14T13:52:39Z","updated_at":"2015-12-26T13:58:23Z","closed_at":null,"body":"Many people have noticed that UGS doesn't support very large files. This is due to UGS storing several copies of the entire file in memory for the duration of an operation. Specifically there are 3 places that are storing the entire file in memory (with additional storage for metadata!):\r\n\r\n1) Each line is stored in a GcodeCommand object, which has some additional metadata.\r\n2) If the visualizer is being used, each vertex is stored in memory - for arcs this vertex data can be even larger than a single command..\r\n3) The GUI even keeps its own copy as part of a Swing JTable component (GcodeTable.java), which stores even more metadata!!\r\n4) (sigh) The GUI console may also keep a copy of this data, which would continue to grow as the operation is executed causing the application performance to degrade even further.\r\n\r\nBasically a solution would need to address each of the above items to work in constant memory:\r\n\r\n1) When commands are preprocessed they are built up into a collection of GcodeCommand objects. This could just as easily be piped into a new gcode file with the preprocessed data.\r\n\r\n2) The visualizer is extremely naive in its implementation. I'm not sure how it could work in constant memory, but the memory usage could be optimized to do things like merge together tiny arc segments or line segments.\r\n\r\n3) The data used by the JTable could be written to disk and looked up as needed. I'm sure there are a lot of options for this. One [possible resource](http://stackoverflow.com/questions/934927/efficient-way-to-implement-paged-sql-result-table-in-a-java-desktop-application), and [another](http://www.coderanch.com/t/345383/GUI/java/JTable-Paging).\r\n\r\n4) The console output can be logged to a file, at which point there is probably a way to make the JTextArea discard old messages. A quick google turned up [this](http://compgroups.net/comp.lang.java.gui/buffered-jtextarea-or-equivalent/332460) which might be a solution."},"comment":{"url":"https://api.github.com/repos/winder/Universal-G-Code-Sender/issues/comments/167326880","html_url":"https://github.com/winder/Universal-G-Code-Sender/issues/302#issuecomment-167326880","issue_url":"https://api.github.com/repos/winder/Universal-G-Code-Sender/issues/302","id":167326880,"user":{"login":"winder","id":125509,"avatar_url":"https://avatars.githubusercontent.com/u/125509?v=3","gravatar_id":"","url":"https://api.github.com/users/winder","html_url":"https://github.com/winder","followers_url":"https://api.github.com/users/winder/followers","following_url":"https://api.github.com/users/winder/following{/other_user}","gists_url":"https://api.github.com/users/winder/gists{/gist_id}","starred_url":"https://api.github.com/users/winder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/winder/subscriptions","organizations_url":"https://api.github.com/users/winder/orgs","repos_url":"https://api.github.com/users/winder/repos","events_url":"https://api.github.com/users/winder/events{/privacy}","received_events_url":"https://api.github.com/users/winder/received_events","type":"User","site_admin":false},"created_at":"2015-12-26T13:58:23Z","updated_at":"2015-12-26T13:58:23Z","body":"The one nice thing you get from the jtable is that it matches up the\nresponse with the command. But still I was thinking the same thing.\n\nFor the visualizer a lot of people use it as a preview tool. Maybe I'll\njust add a warning when visualizing large files.\n\nOn Saturday, December 26, 2015, rugbymauri <notifications@github.com> wrote:\n\n> Is there really a need for the JTable?\n> logging to the console, would be enough.\n>\n> Or even the visualizer? I never need the visulizer.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/winder/Universal-G-Code-Sender/issues/302#issuecomment-167323127>\n> .\n>\n"}}