{"action":"created","issue":{"url":"https://api.github.com/repos/StanfordLegion/legion/issues/42","labels_url":"https://api.github.com/repos/StanfordLegion/legion/issues/42/labels{/name}","comments_url":"https://api.github.com/repos/StanfordLegion/legion/issues/42/comments","events_url":"https://api.github.com/repos/StanfordLegion/legion/issues/42/events","html_url":"https://github.com/StanfordLegion/legion/issues/42","id":101471774,"number":42,"title":"dma: infinite loop scaling df_attach_subregion_parallel (with partial fix)","user":{"login":"noahdesu","id":242417,"avatar_url":"https://avatars.githubusercontent.com/u/242417?v=3","gravatar_id":"","url":"https://api.github.com/users/noahdesu","html_url":"https://github.com/noahdesu","followers_url":"https://api.github.com/users/noahdesu/followers","following_url":"https://api.github.com/users/noahdesu/following{/other_user}","gists_url":"https://api.github.com/users/noahdesu/gists{/gist_id}","starred_url":"https://api.github.com/users/noahdesu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/noahdesu/subscriptions","organizations_url":"https://api.github.com/users/noahdesu/orgs","repos_url":"https://api.github.com/users/noahdesu/repos","events_url":"https://api.github.com/users/noahdesu/events{/privacy}","received_events_url":"https://api.github.com/users/noahdesu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-08-17T17:40:12Z","updated_at":"2015-08-17T17:40:24Z","closed_at":null,"body":"I'm working on scaling the `hdf_attach_subregion_parallel` example. It works fine up until 2^19 elements. When moving to 2^20 elements the dma_thread sits in a CPU bound loop. The following patch fixes the problem, but it seems this solution is only a band aid for scaling. Perhaps the completed requests aren't being reclaimed?\r\n\r\n```\r\ndiff --git a/runtime/lowlevel_dma.cc b/runtime/lowlevel_dma.cc\r\nindex 4645c7e..4888ec4 100644\r\n--- a/runtime/lowlevel_dma.cc\r\n+++ b/runtime/lowlevel_dma.cc\r\n@@ -2759,7 +2759,7 @@ namespace LegionRuntime {\r\n             log_dma.info(\"create mem->hdf xferdes\\n\");\r\n             XferDes* xd = new HDFXferDes<DIM>(channel_manager->get_hdf_write_channel(), false,\r\n                                               src_buf, dst_buf, src_mem_base, hdf_metadata,\r\n-                                              domain, oasvec, 100/*max_nr*/,\r\n+                                              domain, oasvec, 500/*max_nr*/,\r\n                                               XferOrder::DST_FIFO, XferDes::XFER_HDF_WRITE);\r\n             path.push_back(xd);\r\n             break;\r\n@@ -2970,7 +2970,7 @@ namespace LegionRuntime {\r\n             XferDes* xd = new HDFXferDes<DIM>(channel_manager->get_hdf_read_channel(), false,\r\n                                               src_buf, dst_buf, dst_mem_base, src_hdf_metadata,\r\n                                               domain, oasvec,\r\n-                                              100/*max_nr*/, Layouts::XferOrder::SRC_FIFO, XferDes::XFER_HDF_READ);\r\n+                                              500/*max_nr*/, Layouts::XferOrder::SRC_FIFO, XferDes::XFER_HDF_READ);\r\n             path.push_back(xd);\r\n             break;\r\n           }\r\ndiff --git a/runtime/realm/runtime_impl.cc b/runtime/realm/runtime_impl.cc\r\nindex ffc4fe0..68eff83 100644\r\n--- a/runtime/realm/runtime_impl.cc\r\n+++ b/runtime/realm/runtime_impl.cc\r\n@@ -989,7 +989,7 @@ namespace Realm {\r\n \r\n       // start dma system at the very ending of initialization\r\n       // since we need list of local gpus to create channels\r\n-      LegionRuntime::LowLevel::start_dma_system(dma_worker_threads, 100\r\n+      LegionRuntime::LowLevel::start_dma_system(dma_worker_threads, 500\r\n #ifdef USE_CUDA\r\n                        ,local_gpus\r\n #endif\r\n```"},"comment":{"url":"https://api.github.com/repos/StanfordLegion/legion/issues/comments/131902547","html_url":"https://github.com/StanfordLegion/legion/issues/42#issuecomment-131902547","issue_url":"https://api.github.com/repos/StanfordLegion/legion/issues/42","id":131902547,"user":{"login":"noahdesu","id":242417,"avatar_url":"https://avatars.githubusercontent.com/u/242417?v=3","gravatar_id":"","url":"https://api.github.com/users/noahdesu","html_url":"https://github.com/noahdesu","followers_url":"https://api.github.com/users/noahdesu/followers","following_url":"https://api.github.com/users/noahdesu/following{/other_user}","gists_url":"https://api.github.com/users/noahdesu/gists{/gist_id}","starred_url":"https://api.github.com/users/noahdesu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/noahdesu/subscriptions","organizations_url":"https://api.github.com/users/noahdesu/orgs","repos_url":"https://api.github.com/users/noahdesu/repos","events_url":"https://api.github.com/users/noahdesu/events{/privacy}","received_events_url":"https://api.github.com/users/noahdesu/received_events","type":"User","site_admin":false},"created_at":"2015-08-17T17:40:24Z","updated_at":"2015-08-17T17:40:24Z","body":"ping @jiazhihao "}}