{"action":"created","issue":{"url":"https://api.github.com/repos/geodynamics/vq/issues/59","labels_url":"https://api.github.com/repos/geodynamics/vq/issues/59/labels{/name}","comments_url":"https://api.github.com/repos/geodynamics/vq/issues/59/comments","events_url":"https://api.github.com/repos/geodynamics/vq/issues/59/events","html_url":"https://github.com/geodynamics/vq/issues/59","id":58134112,"number":59,"title":"Heisenbug","user":{"login":"kwschultz","id":5606202,"avatar_url":"https://avatars.githubusercontent.com/u/5606202?v=3","gravatar_id":"","url":"https://api.github.com/users/kwschultz","html_url":"https://github.com/kwschultz","followers_url":"https://api.github.com/users/kwschultz/followers","following_url":"https://api.github.com/users/kwschultz/following{/other_user}","gists_url":"https://api.github.com/users/kwschultz/gists{/gist_id}","starred_url":"https://api.github.com/users/kwschultz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kwschultz/subscriptions","organizations_url":"https://api.github.com/users/kwschultz/orgs","repos_url":"https://api.github.com/users/kwschultz/repos","events_url":"https://api.github.com/users/kwschultz/events{/privacy}","received_events_url":"https://api.github.com/users/kwschultz/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/geodynamics/vq/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/geodynamics/vq/labels/difficulty:%20hard","name":"difficulty: hard","color":"f7c6c7"},{"url":"https://api.github.com/repos/geodynamics/vq/labels/help%20wanted","name":"help wanted","color":"159818"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":25,"created_at":"2015-02-18T21:52:57Z","updated_at":"2015-05-19T16:18:36Z","closed_at":null,"body":"Error in running VQ on multiple processors with a model more complex than a few faults and a few thousand elements. Error is that the simulation gets stuck and cannot continue. Error is randomly reproducible, occurring at different points in multiple runs of the same simulation. \r\n\r\nWe must fix this bug before we can reliably run simulations on multiple processors. Right now we are effectively dead in the water with respect to a full CA simulation or even including aftershocks on smaller simulations.\r\n\r\nEric's summary of the bug when it occurred on a 3 processor run of a 6 fault subset from the UCERF2 California model:\r\n\"So, here’s my Sherlock Holmes take:\r\nFrom the backtrace, we know process 0 and 1 are stuck in distributeUpdateField(), while process 2 is in MPI_Recv() in processBlocksSecondaryFailures()\r\nSince the processes are in order, this means the MPI_Recv that is stuck must correspond to the solution of Ax=b being sent back from the root (process 0) to process 2\r\nThe only way this could happen is if the number of MPI_Send() calls from root does not match the number of MPI_Recv() calls in the other processes\r\nThe only way this mismatch could happen is if the total number of entries in global_id_list is not equal to the sum of the number of entries in local_id_list for each process\r\nor if processes have different understandings of the assignment of blocks to each process\r\nSince my laptop run is already at 4300 events with no problems, it seems more likely this is a bug caused by bad memory writing\r\nSuch that one of theses structures is being corrupted by something overwriting the existing data\r\nSo the question is how do we check whether this corruption is happening\""},"comment":{"url":"https://api.github.com/repos/geodynamics/vq/issues/comments/103573664","html_url":"https://github.com/geodynamics/vq/issues/59#issuecomment-103573664","issue_url":"https://api.github.com/repos/geodynamics/vq/issues/59","id":103573664,"user":{"login":"markyoder","id":7614835,"avatar_url":"https://avatars.githubusercontent.com/u/7614835?v=3","gravatar_id":"","url":"https://api.github.com/users/markyoder","html_url":"https://github.com/markyoder","followers_url":"https://api.github.com/users/markyoder/followers","following_url":"https://api.github.com/users/markyoder/following{/other_user}","gists_url":"https://api.github.com/users/markyoder/gists{/gist_id}","starred_url":"https://api.github.com/users/markyoder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markyoder/subscriptions","organizations_url":"https://api.github.com/users/markyoder/orgs","repos_url":"https://api.github.com/users/markyoder/repos","events_url":"https://api.github.com/users/markyoder/events{/privacy}","received_events_url":"https://api.github.com/users/markyoder/received_events","type":"User","site_admin":false},"created_at":"2015-05-19T16:18:36Z","updated_at":"2015-05-19T16:18:36Z","body":"still hanging on a child node (i think) MPI_Recv() call. the solution may be as simple as using MPI_Ssend() instead of MPI_Send(), the former being the \"synchronous\" version of the latter. see:\r\nhttp://stackoverflow.com/questions/17582900/difference-between-mpi-send-and-mpi-ssend\r\n\r\nthis article describes cases where MPI_Send() might be happy to move on when the corresponding MPI_Recv() has not actually done whatever the hell it's supposed to do."}}