{"action":"created","issue":{"url":"https://api.github.com/repos/nohal/chartdldr_pi/issues/62","labels_url":"https://api.github.com/repos/nohal/chartdldr_pi/issues/62/labels{/name}","comments_url":"https://api.github.com/repos/nohal/chartdldr_pi/issues/62/comments","events_url":"https://api.github.com/repos/nohal/chartdldr_pi/issues/62/events","html_url":"https://github.com/nohal/chartdldr_pi/issues/62","id":61811900,"number":62,"title":"Updating the list of charts is too slow with huge chart catalogs","user":{"login":"transmitterdan","id":10525699,"avatar_url":"https://avatars.githubusercontent.com/u/10525699?v=3","gravatar_id":"","url":"https://api.github.com/users/transmitterdan","html_url":"https://github.com/transmitterdan","followers_url":"https://api.github.com/users/transmitterdan/followers","following_url":"https://api.github.com/users/transmitterdan/following{/other_user}","gists_url":"https://api.github.com/users/transmitterdan/gists{/gist_id}","starred_url":"https://api.github.com/users/transmitterdan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/transmitterdan/subscriptions","organizations_url":"https://api.github.com/users/transmitterdan/orgs","repos_url":"https://api.github.com/users/transmitterdan/repos","events_url":"https://api.github.com/users/transmitterdan/events{/privacy}","received_events_url":"https://api.github.com/users/transmitterdan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-03-15T13:25:51Z","updated_at":"2015-03-16T01:37:11Z","closed_at":null,"body":"ChartSource::IsNewerThanLocal() and ChartSource::ExistsLocaly() are called many times when checking for new/updated charts. The US ENC catalog contains over 1000 charts and over 8000 files. These two functions are usually called once each for every chart. They both do a blind linear search through the list of local files. The use of wxFileName date checking and parsing in these functions really slows them down. What I would suggest is that when ChartSource::GetLocalFiles() runs we store a list of shortened lower case file names. In addition we create an additional container to store the local date of each file. I don't know if we would still need a container for the entire path to local files or not.\r\n\r\nThen the functions IsNewerThanLocal() and ExistsLocaly() would not have to use wxFileName to check the date or whether the file exists locally. It could just do a linear search on the already shortened lower case list of names and then check the date saved by GetLocalFiles() directly. This would reduce the interaction with the file system by at least a factor of 100 and probably speed up populating the list of charts by an order of magnitude or more.\r\n\r\nA further optimization could be that the function ExistsLocaly() could return the index of the name so IsNewerThanLocal() does not have to do a linear search to find the local file date. We only call IsNewerThanLocal if we already know the file exists. Searching the list again is redundant I suspect."},"comment":{"url":"https://api.github.com/repos/nohal/chartdldr_pi/issues/comments/81338018","html_url":"https://github.com/nohal/chartdldr_pi/issues/62#issuecomment-81338018","issue_url":"https://api.github.com/repos/nohal/chartdldr_pi/issues/62","id":81338018,"user":{"login":"transmitterdan","id":10525699,"avatar_url":"https://avatars.githubusercontent.com/u/10525699?v=3","gravatar_id":"","url":"https://api.github.com/users/transmitterdan","html_url":"https://github.com/transmitterdan","followers_url":"https://api.github.com/users/transmitterdan/followers","following_url":"https://api.github.com/users/transmitterdan/following{/other_user}","gists_url":"https://api.github.com/users/transmitterdan/gists{/gist_id}","starred_url":"https://api.github.com/users/transmitterdan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/transmitterdan/subscriptions","organizations_url":"https://api.github.com/users/transmitterdan/orgs","repos_url":"https://api.github.com/users/transmitterdan/repos","events_url":"https://api.github.com/users/transmitterdan/events{/privacy}","received_events_url":"https://api.github.com/users/transmitterdan/received_events","type":"User","site_admin":false},"created_at":"2015-03-16T01:37:11Z","updated_at":"2015-03-16T01:37:11Z","body":"Pavel,\r\n\r\nThis is a \"patch\" that should speed up the checking of large chart catalogs against local files and dates.\r\n\r\n```\r\ndiff --git a/src/chartdldr_pi.cpp b/src/chartdldr_pi.cpp\r\nindex cfcfba9..e42f6bd 100644\r\n--- a/src/chartdldr_pi.cpp\r\n+++ b/src/chartdldr_pi.cpp\r\n@@ -51,6 +51,9 @@\r\n \r\n #include <wx/arrimpl.cpp>\r\n     WX_DEFINE_OBJARRAY(wxArrayOfChartSources);\r\n+\r\n+#include <wx/arrimpl.cpp>\r\n+    WX_DEFINE_OBJARRAY(wxArrayOfDateTime);\r\n     \r\n #ifdef __WXMAC__\r\n #define CATALOGS_NAME_WIDTH 300\r\n@@ -503,8 +506,7 @@ bool ChartSource::ExistsLocaly(wxString filename)\r\n       wxString file = tk.GetNextToken().MakeLower();\r\n       for (size_t i = 0; i < m_localfiles.Count(); i++)\r\n       {\r\n-            wxFileName fn(m_localfiles.Item(i));\r\n-            if(fn.GetName().MakeLower() == file)\r\n+            if (m_localfiles.Item(i) == file)\r\n                   return true;\r\n       }\r\n       return false;\r\n@@ -517,12 +519,9 @@ bool ChartSource::IsNewerThanLocal(wxString filename, wxDateTime validDate)\r\n       wxString file = tk.GetNextToken().MakeLower();\r\n       for (size_t i = 0; i < m_localfiles.Count(); i++)\r\n       {\r\n-            wxFileName fn(m_localfiles.Item(i));\r\n-            if(fn.GetName().MakeLower() == file)\r\n+            if(m_localfiles.Item(i) == file)\r\n             {\r\n-                  wxDateTime ct, mt, at;\r\n-                  fn.GetTimes(&at, &mt, &ct);\r\n-                  if (validDate.IsLaterThan(mt))\r\n+                  if (validDate.IsLaterThan(m_localdt.Item(i)))\r\n                   {\r\n                     update_candidate = true;\r\n                   }\r\n@@ -615,7 +614,7 @@ void ChartDldrPanelImpl::UpdateAllCharts( wxCommandEvent& event )\r\n     wxLogMessage( wxString::Format(_T(\"chartdldr_pi::UpdateAllCharts() downloaded %d out of %d charts.\"), attempted_to_update - failed_to_update, attempted_to_update) );\r\n        if ( failed_to_update > 0 )\r\n                 wxMessageBox( wxString::Format( _(\"%d out of %d charts failed to download.\\nCheck the list, verify there is a working Internet connection and repeat the operation if needed.\"),\r\n-                                             failed_to_update, attempted_to_update ), _(\"Chart Downloader\"), wxOK | wxICON_ERROR );\r\n+                                       failed_to_update, attempted_to_update ), _(\"Chart Downloader\"), wxOK | wxICON_ERROR );\r\n     if ( attempted_to_update > failed_to_update )\r\n             wxMessageBox( _(\"You have added/updated some of your charts.\\nTo make sure OpenCPN knows about them, go to the 'Chart Files' tab and select the 'Scan Charts and Update Database' option.\"), \r\n                     _(\"Chart Downloader\"), wxOK | wxICON_INFORMATION );\r\n@@ -695,14 +694,22 @@ void ChartDldrPanelImpl::UpdateChartList( wxCommandEvent& event )\r\n     wxRemoveFile ( tfn.GetFullPath() );\r\n }\r\n \r\n-wxArrayString ChartSource::GetLocalFiles()\r\n+void ChartSource::GetLocalFiles()\r\n {\r\n-      wxArrayString *ret = new wxArrayString();\r\n+      wxArrayString allFiles;\r\n       if( wxDirExists(GetDir()) )\r\n-        wxDir::GetAllFiles(GetDir(), ret);\r\n-      wxArrayString r(*ret);\r\n-      wxDELETE(ret);\r\n-      return r;\r\n+        wxDir::GetAllFiles(GetDir(), &allFiles);\r\n+      m_localdt.Clear();\r\n+      m_localfiles.Clear();\r\n+      for (size_t i = 0; i < allFiles.Count(); i++)\r\n+      {\r\n+            wxFileName fn(allFiles.Item(i));\r\n+            wxDateTime ct, mt, at;\r\n+            fn.GetTimes(&at, &mt, &ct);\r\n+            m_localdt.Add(new wxDateTime(mt));\r\n+            m_localfiles.Add(fn.GetName().Lower());\r\n+      }\r\n+      allFiles.Clear();\r\n }\r\n \r\n bool ChartDldrPanelImpl::DownloadChart(wxString url, wxString file, wxString title)\r\n@@ -799,7 +806,7 @@ After downloading the charts, please extract them to %s\"), pPlugIn->m_pChartCata\r\n                       }\r\n                   }\r\n             }\r\n-               if (cancelled)\r\n+                       if (cancelled)\r\n                 break;\r\n       }\r\n       SetSource(GetSelectedCatalog());\r\n@@ -899,6 +906,7 @@ void ChartDldrPanelImpl::AddSource( wxCommandEvent& event )\r\n             wxMessageBox( wxString::Format(_(\"Path %s seems not to be covered by your configured Chart Directories.\\nTo see the charts you have to adjust the configuration on the 'Chart Files' tab.\"), cs->GetDir().c_str()),\r\n                          _(\"Chart Downloader\") );\r\n \r\n+        SelectCatalog(m_lbChartSources->GetItemCount()-1);\r\n         pPlugIn->SaveConfig();\r\n     }\r\n     dialog->Close();\r\ndiff --git a/src/chartdldr_pi.h b/src/chartdldr_pi.h\r\nindex c220958..20dd407 100644\r\n--- a/src/chartdldr_pi.h\r\n+++ b/src/chartdldr_pi.h\r\n@@ -59,6 +59,7 @@ class ChartDldrPanelImpl;\r\n class ChartDldrGuiAddSourceDlg;\r\n \r\n WX_DECLARE_OBJARRAY(ChartSource *, wxArrayOfChartSources);\r\n+WX_DECLARE_OBJARRAY(wxDateTime, wxArrayOfDateTime);\r\n \r\n //----------------------------------------------------------------------------------------------------------\r\n //    The PlugIn Class Definition\r\n@@ -132,10 +133,11 @@ public:\r\n       void SetUrl(wxString url) { m_url = url; }\r\n       bool ExistsLocaly(wxString filename);\r\n       bool IsNewerThanLocal(wxString filename, wxDateTime validDate);\r\n-      void UpdateLocalFiles() { m_localfiles = GetLocalFiles(); }\r\n+      void UpdateLocalFiles() { GetLocalFiles(); }\r\n private:\r\n       wxArrayString m_localfiles;\r\n-      wxArrayString GetLocalFiles();\r\n+      void GetLocalFiles();\r\n+      wxArrayOfDateTime m_localdt;\r\n       wxString m_name;\r\n       wxString m_url;\r\n       wxString m_dir;\r\n```"}}