{"action":"closed","issue":{"url":"https://api.github.com/repos/rundeck/rundeck/issues/1064","labels_url":"https://api.github.com/repos/rundeck/rundeck/issues/1064/labels{/name}","comments_url":"https://api.github.com/repos/rundeck/rundeck/issues/1064/comments","events_url":"https://api.github.com/repos/rundeck/rundeck/issues/1064/events","html_url":"https://github.com/rundeck/rundeck/issues/1064","id":53660877,"number":1064,"title":"ERROR FrameworkProject - Cannot get nodes from [DirectoryResourceModelSource","user":{"login":"smithtimamy","id":2131926,"avatar_url":"https://avatars.githubusercontent.com/u/2131926?v=3","gravatar_id":"","url":"https://api.github.com/users/smithtimamy","html_url":"https://github.com/smithtimamy","followers_url":"https://api.github.com/users/smithtimamy/followers","following_url":"https://api.github.com/users/smithtimamy/following{/other_user}","gists_url":"https://api.github.com/users/smithtimamy/gists{/gist_id}","starred_url":"https://api.github.com/users/smithtimamy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smithtimamy/subscriptions","organizations_url":"https://api.github.com/users/smithtimamy/orgs","repos_url":"https://api.github.com/users/smithtimamy/repos","events_url":"https://api.github.com/users/smithtimamy/events{/privacy}","received_events_url":"https://api.github.com/users/smithtimamy/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rundeck/rundeck/labels/bug","name":"bug","color":"e10c02"},{"url":"https://api.github.com/repos/rundeck/rundeck/labels/needs-review","name":"needs-review","color":"0b02e1"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/rundeck/rundeck/milestones/30","labels_url":"https://api.github.com/repos/rundeck/rundeck/milestones/30/labels","id":909180,"number":30,"title":"2.4.1","description":"Next patch release","creator":{"login":"gschueler","id":55603,"avatar_url":"https://avatars.githubusercontent.com/u/55603?v=3","gravatar_id":"","url":"https://api.github.com/users/gschueler","html_url":"https://github.com/gschueler","followers_url":"https://api.github.com/users/gschueler/followers","following_url":"https://api.github.com/users/gschueler/following{/other_user}","gists_url":"https://api.github.com/users/gschueler/gists{/gist_id}","starred_url":"https://api.github.com/users/gschueler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gschueler/subscriptions","organizations_url":"https://api.github.com/users/gschueler/orgs","repos_url":"https://api.github.com/users/gschueler/repos","events_url":"https://api.github.com/users/gschueler/events{/privacy}","received_events_url":"https://api.github.com/users/gschueler/received_events","type":"User","site_admin":false},"open_issues":1,"closed_issues":16,"state":"open","created_at":"2014-12-18T17:57:35Z","updated_at":"2015-01-30T01:04:20Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2015-01-07T17:54:42Z","updated_at":"2015-01-30T01:04:20Z","closed_at":"2015-01-30T01:04:20Z","body":"We are running Rundeck 2.4.0-1 on Joyent SmartOS via the launcher. With this setup, we are using the \"Directory Resource Model Source\" method of defining nodes. On occasion, we get the below stack and job failure on some executions. This issue is intermittent and I have not been able to recreate it on my own.\r\n\r\n```\r\n2015-01-07 11:20:01,468 ERROR FrameworkProject - Cannot get nodes from [DirectoryResourceModelSource{directory=/path/to/resource/files/}]: null\r\njava.util.ConcurrentModificationException\r\n        at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)\r\n        at java.util.ArrayList$Itr.next(ArrayList.java:831)\r\n        at com.dtolabs.rundeck.core.resources.DirectoryResourceModelSource.loadNodeSets(DirectoryResourceModelSource.java:136)\r\n        at com.dtolabs.rundeck.core.resources.DirectoryResourceModelSource.getNodes(DirectoryResourceModelSource.java:131)\r\n        at com.dtolabs.rundeck.core.common.FrameworkProject.getNodeSet(FrameworkProject.java:525)\r\n        at com.dtolabs.rundeck.core.common.Framework.filterNodeSet(Framework.java:523)\r\n        at com.dtolabs.rundeck.core.common.Framework$filterNodeSet.call(Unknown Source)\r\n        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)\r\n        at com.dtolabs.rundeck.core.common.Framework$filterNodeSet.call(Unknown Source)\r\n        at rundeck.services.FrameworkService$_filterNodeSet_closure4.doCall(FrameworkService.groovy:275)\r\n        at rundeck.services.FrameworkService$_filterNodeSet_closure4.doCall(FrameworkService.groovy)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:606)\r\n        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)\r\n        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)\r\n        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1082)\r\n        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)\r\n        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)\r\n        at groovy.lang.Closure.call(Closure.java:412)\r\n        at groovy.lang.Closure.call(Closure.java:406)\r\n        at com.codahale.metrics.Timer.time(Timer.java:99)\r\n        at com.codahale.metrics.Timer$time$0.call(Unknown Source)\r\n        at MetricswebGrailsPlugin$_addDynamicMetricMethods_closure26.doCall(MetricswebGrailsPlugin.groovy:174)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:606)\r\n        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)\r\n        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)\r\n        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1082)\r\n        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)\r\n        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)\r\n        at groovy.lang.Closure.call(Closure.java:412)\r\n        at org.codehaus.groovy.runtime.metaclass.ClosureStaticMetaMethod.invoke(ClosureStaticMetaMethod.java:59)\r\n        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoMetaMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:313)\r\n        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:64)\r\n        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)\r\n        at rundeck.services.FrameworkService.filterNodeSet(FrameworkService.groovy:274)\r\n        at rundeck.services.FrameworkService$filterNodeSet.call(Unknown Source)\r\n        at rundeck.services.ExecutionService.createContext(ExecutionService.groovy:1010)\r\n        at rundeck.services.ExecutionService$createContext$3.callCurrent(Unknown Source)\r\n        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)\r\n        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)\r\n        at rundeck.services.ExecutionService.executeAsyncBegin(ExecutionService.groovy:743)\r\n        at rundeck.services.ExecutionService.executeAsyncBegin(ExecutionService.groovy)\r\n        at rundeck.services.ExecutionService$$FastClassByCGLIB$$ffb73873.invoke(<generated>)\r\n        at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)\r\n        at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)\r\n        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)\r\n        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)\r\n        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)\r\n        at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622)\r\n        at rundeck.services.ExecutionService$$EnhancerByCGLIB$$3e62918d.executeAsyncBegin(<generated>)\r\n        at rundeck.services.ExecutionService$executeAsyncBegin.call(Unknown Source)\r\n        at rundeck.quartzjobs.ExecutionJob.executeCommand(ExecutionJob.groovy:256)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:606)\r\n        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrap.invoke(PogoMetaMethodSite.java:252)\r\n        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:52)\r\n        at rundeck.quartzjobs.ExecutionJob.execute_internal(ExecutionJob.groovy:113)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:606)\r\n        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)\r\n        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)\r\n        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1082)\r\n        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)\r\n        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)\r\n        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1027)\r\n        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)\r\n        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)\r\n        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)\r\n        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)\r\n        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)\r\n        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)\r\n        at rundeck.quartzjobs.ExecutionJob$_execute_closure1.doCall(ExecutionJob.groovy:69)\r\n        at rundeck.quartzjobs.ExecutionJob$_execute_closure1.doCall(ExecutionJob.groovy)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:606)\r\n        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)\r\n        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)\r\n        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1082)\r\n        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)\r\n        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)\r\n        at groovy.lang.Closure.call(Closure.java:412)\r\n        at groovy.lang.Closure.call(Closure.java:406)\r\n        at com.codahale.metrics.Timer.time(Timer.java:99)\r\n        at com.codahale.metrics.Timer$time$1.call(Unknown Source)\r\n        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)\r\n        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)\r\n        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)\r\n        at rundeck.quartzjobs.ExecutionJob.execute(ExecutionJob.groovy:68)\r\n        at org.quartz.core.JobRunShell.run(JobRunShell.java:199)\r\n        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)\r\n2015-01-07 11:20:02,200 ERROR ExecutionUtilService - Execution failed: 41978: No matched nodes: NodeSet{includes={name=cmh-shortngxpd-001, dominant=false, }}\r\n```\r\n\r\nThings we have tried thus far to eliminate this issue that have not worked:\r\n* Move the directory from an NFS mount to the local filesystem - has not helped\r\n* Upgrade to version 2.4.0-1 from 2.3.2 - has not helped\r\n\r\nI have tried looking for similar issues on here but have not found any. Please let me know what additional information you might need "}}