{"action":"created","issue":{"url":"https://api.github.com/repos/casacore/casacore/issues/217","labels_url":"https://api.github.com/repos/casacore/casacore/issues/217/labels{/name}","comments_url":"https://api.github.com/repos/casacore/casacore/issues/217/comments","events_url":"https://api.github.com/repos/casacore/casacore/issues/217/events","html_url":"https://github.com/casacore/casacore/issues/217","id":101663401,"number":217,"title":"imager.clean crashes with \"pure virtual method called\"","user":{"login":"david-terrett","id":7045424,"avatar_url":"https://avatars.githubusercontent.com/u/7045424?v=3","gravatar_id":"","url":"https://api.github.com/users/david-terrett","html_url":"https://github.com/david-terrett","followers_url":"https://api.github.com/users/david-terrett/followers","following_url":"https://api.github.com/users/david-terrett/following{/other_user}","gists_url":"https://api.github.com/users/david-terrett/gists{/gist_id}","starred_url":"https://api.github.com/users/david-terrett/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/david-terrett/subscriptions","organizations_url":"https://api.github.com/users/david-terrett/orgs","repos_url":"https://api.github.com/users/david-terrett/repos","events_url":"https://api.github.com/users/david-terrett/events{/privacy}","received_events_url":"https://api.github.com/users/david-terrett/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-08-18T14:03:34Z","updated_at":"2015-08-21T11:48:09Z","closed_at":null,"body":"When run on a heavily loaded multi-core system, imager.clean() crashes with the message \"pure virtual method called\". A stack dump shows that the origin of the fault is Table::reliquishAutoLocks() and it appears that this method is calling the hasLock() method of a PlainTable that has been, or is in the process of being deleted, in another thread. The likely result is a call to a virtual function in the base class (see http://www.artima.com/cppsource/pure_virtual.html).\r\n\r\nThe basic problem is that although the methods that access the table cache use a mutex to protect the table while it is being accessed, they return a raw pointer to a PlainTableand that table may be deleted by another thread while the PlainTable is being accessed through the pointer. (there is another race condition, but not one we have seen in practice, in that the code fetches the number of entries in the table cache and then iterates over the table using an index. If an entry in the table is deleted by another thread, it could attempt to fetch an entry beyond the end of the table and get a null pointer returned).\r\n\r\nThe problem can be fixed by moving the code of Table::relinquishAutoLocks into TableCache and making the calls to the Table methods while holding the TableCache mutex which blocks the deletion of the tables in other threads. This solution has been implemented and tested and does fix the problem.\r\n\r\nI can supply a patch file (I don't seem to be able to attach it to the issue)"},"comment":{"url":"https://api.github.com/repos/casacore/casacore/issues/comments/133386322","html_url":"https://github.com/casacore/casacore/issues/217#issuecomment-133386322","issue_url":"https://api.github.com/repos/casacore/casacore/issues/217","id":133386322,"user":{"login":"david-terrett","id":7045424,"avatar_url":"https://avatars.githubusercontent.com/u/7045424?v=3","gravatar_id":"","url":"https://api.github.com/users/david-terrett","html_url":"https://github.com/david-terrett","followers_url":"https://api.github.com/users/david-terrett/followers","following_url":"https://api.github.com/users/david-terrett/following{/other_user}","gists_url":"https://api.github.com/users/david-terrett/gists{/gist_id}","starred_url":"https://api.github.com/users/david-terrett/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/david-terrett/subscriptions","organizations_url":"https://api.github.com/users/david-terrett/orgs","repos_url":"https://api.github.com/users/david-terrett/repos","events_url":"https://api.github.com/users/david-terrett/events{/privacy}","received_events_url":"https://api.github.com/users/david-terrett/received_events","type":"User","site_admin":false},"created_at":"2015-08-21T11:48:09Z","updated_at":"2015-08-21T11:48:09Z","body":"Ger,\r\n\r\nIt looks as if the “table in use” problem is caused by the file locking support in Lustre (maybe for other file systems too) being asynchronous to some degree.  Replacing:\r\n\r\n            if (isMultiUsed(False)) {\r\n                        unmarkForDelete (False, \"\");\r\n                        throw (TableError (\"Table \" + name_p + \" cannot be deleted;\"\r\n                                                           \" the table or a subtable is still used\"\r\n                                                           \" in another process\"));\r\n          }\r\n\r\nwith:\r\n\r\n      int nTrys = 5;\r\n      while (isMultiUsed(False)) {\r\n            if ( nTrys == 0 ) {\r\n                        unmarkForDelete (False, \"\");\r\n                        throw (TableError (\"Table \" + name_p + \" cannot be deleted;\"\r\n                                                           \" the table or a subtable is still used\"\r\n                                                           \" in another process\"));\r\n            }\r\n            sleep(1);\r\n            --nTrys;\r\n    }\r\n\r\nIn PlainTable::closeObject seems to fix it for us.\r\n\r\nDavid\r\n\r\nFrom: Ger van Diepen [mailto:notifications@github.com]\r\nSent: 19 August 2015 12:33\r\nTo: casacore/casacore\r\nCc: Terrett, David (STFC,RAL,RALSP)\r\nSubject: Re: [casacore] imager.clean crashes with \"pure virtual method called\" (#217)\r\n\r\nThanks for the patch.\r\n\r\nThe \"in use\" error message is another issue.\r\nWhen the Table system deletes a table on disk, it first checks if it is in\r\nuse in another process. Looking at your message it looks as if that is the\r\ncase. It simply means another process is still accessing that table.\r\nYou can use showtablelock to see which process is using a table.\r\n\r\nMaybe in your application each thread opens the same table and Linux treats\r\nthreads more or less as processes. So maybe the problem arises from that\r\nsituation.\r\nIf that is done, I think it is bad to delete the table in a particular\r\nthread. Deleting the table is basically a barrier, so all threads should\r\nsync before deletion.\r\n\r\nNote that casacore has the same philosophy as the standard library. It only\r\nuses mutexes for static objects; access to other objects is not locked to\r\navoid overhead. When operating yourself on a static object, you have to use\r\nlocks yourself.\r\nThe autorelinquish patch is a good one though. Casacore code using the\r\nTableCache, should be locked.\r\n\r\n\r\n\r\n.\r\n"}}