{"action":"created","issue":{"url":"https://api.github.com/repos/uwsampa/grappa/issues/210","labels_url":"https://api.github.com/repos/uwsampa/grappa/issues/210/labels{/name}","comments_url":"https://api.github.com/repos/uwsampa/grappa/issues/210/comments","events_url":"https://api.github.com/repos/uwsampa/grappa/issues/210/events","html_url":"https://github.com/uwsampa/grappa/issues/210","id":75406965,"number":210,"title":"Segmentation Fault When Using NaiveGraphlab Engine.","user":{"login":"zmahdavi","id":5846018,"avatar_url":"https://avatars.githubusercontent.com/u/5846018?v=3","gravatar_id":"","url":"https://api.github.com/users/zmahdavi","html_url":"https://github.com/zmahdavi","followers_url":"https://api.github.com/users/zmahdavi/followers","following_url":"https://api.github.com/users/zmahdavi/following{/other_user}","gists_url":"https://api.github.com/users/zmahdavi/gists{/gist_id}","starred_url":"https://api.github.com/users/zmahdavi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zmahdavi/subscriptions","organizations_url":"https://api.github.com/users/zmahdavi/orgs","repos_url":"https://api.github.com/users/zmahdavi/repos","events_url":"https://api.github.com/users/zmahdavi/events{/privacy}","received_events_url":"https://api.github.com/users/zmahdavi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-05-12T00:34:27Z","updated_at":"2015-05-13T21:08:06Z","closed_at":null,"body":"Hi,\r\n\r\nI am trying to implement label propagation using the naive Graphlab engine. It works fine when I run it on one core and one process. However, i get segmentation fault when i try to run it on more than one core.\r\n\r\nMy first attempt was this:\r\n\r\n``` C++\r\nstruct label_counter {\r\n  // A map that counts the number of time each label is used.\r\n  std::map<std::string, int> label_count;\r\n\r\n  explicit label_counter(std::string label) {\r\n    if (label != Unknown)\r\n      label_count[label] = 1;\r\n  }\r\n\r\n  explicit label_counter() {}\r\n\r\n  label_counter& operator+=(const label_counter& other) { \r\n    for ( std::map<std::string, int>::const_iterator iter = other.label_count.begin();\r\n              iter != other.label_count.end(); ++iter ) {\r\n            label_count[iter->first] += iter->second;\r\n    }\r\n\r\n    return *this; \r\n  }\r\n};\r\n\r\nusing G = Graph<CCData,Empty>;\r\n\r\nstruct LabelPropagation : public GraphlabVertexProgram<G, label_counter> {\r\n  bool do_scatter;\r\n  std::string changelabel;\r\n\r\n  LabelPropagation(Vertex& v) {\r\n    do_scatter = false;\r\n  }\r\n\r\n  bool gather_edges(const Vertex& v) const { return true; }\r\n\r\n  Gather gather(const Vertex& v, Edge& e) const { \r\n    return label_counter(v->label); \r\n  }\r\n\r\n  void apply(Vertex& v, const Gather& total) {\r\n    int maxCount = 0;\r\n    std::string maxLabel = v->label;\r\n\r\n    // Figure out which label of the vertex's neighbors' labels is most common\r\n    for ( std::map<std::string, int>::const_iterator iter = total.label_count.begin();\r\n          iter != total.label_count.end(); ++iter ) {\r\n        if (iter->second > maxCount) {\r\n          maxCount = iter->second;\r\n          maxLabel = iter->first;\r\n        }\r\n    }\r\n\r\n    // All of neighbours are unknown, or it doesn't have nieghbour\r\n    if (maxLabel == Unknown) {\r\n      //LOG(INFO) << \"vertex id:  unknown \";// << v->id << \" Label is unknown.\";\r\n      do_scatter = false;\r\n      v->activate();\r\n    } else if (maxLabel.compare(v->label) != 0) {\r\n        v->label = maxLabel;\r\n      //  LOG(INFO) << \"vertex id: found\"; // << v->id << \" Label is: .\" << v->label;\r\n        changelabel = v->label;\r\n        do_scatter = true;\r\n    } else {\r\n      // LOG(INFO) << \"vertex id: \"; // << v->id << \" Label is: .\" << v->label << \" lebel didn't change\";\r\n      do_scatter = false;\r\n    }\r\n  }\r\n  \r\n  bool scatter_edges(const Vertex& v) const { return do_scatter; }\r\n  \r\n  Gather scatter(const Edge& e, Vertex& target) const {\r\n    // I think this is not needed.\r\n    return label_counter(changelabel);\r\n\r\n  }\r\n};\r\n```\r\n\r\nWhich causes segmentation fault when used on more than one node. I used freeze option and attached gdb, I see the below stack:\r\n\r\n```\r\n#9  0x00000000005b933e in _ZNSt8_Rb_treeIiSt4pairIKiiESt10_Select1stIS2_ESt4lessIiESaIS2_EE14_M_create_nodeIJRKS2_EEEPSt13_Rb_tree_nodeIS2_EDpOT_ ()\r\n#10 0x00000000005b57a1 in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_clone_node(std::_Rb_tree_node<std::pair<int const, int> > const*) ()\r\n#11 0x00000000005b0480 in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_copy(std::_Rb_tree_node<std::pair<int const, int> > const*, std::_Rb_tree_node<std::pair<int const, int> >*) ()\r\n#12 0x00000000005b04bf in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_copy(std::_Rb_tree_node<std::pair<int const, int> > const*, std::_Rb_tree_node<std::pair<int const, int> >*) ()\r\n#13 0x00000000005a9c78 in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_Rb_tree(std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > > const&) ()\r\n#14 0x00000000005a3a55 in std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > >::map(std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > const&) ()\r\n#15 0x000000000059d2e7 in label_counter::label_counter(label_counter const&) ()\r\n#16 0x000000000059d30e in void NaiveGraphlabEngine<Grappa::Graph<CCData, Grappa::Empty>, LabelPropagation>::run_sync<CCData, Grappa::Empty>(GlobalAddress<Grappa::Graph<CCData, Grappa::Empty> >)::{lambda(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&)#3}::operator()(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&) const::{lambda(Grappa::Graph<CCData, Grappa::Empty>::Edge&)#1}::operator()(Grappa::Graph<CCData, Grappa::Empty>::Edge) const::{lambda(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&)#1}::operator()(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&) const ()\r\n#17 0x00000000005a9d2d in decltype ({parm#2}(*(({parm#1}.pointer)()))) Grappa::impl::call<(Grappa::SyncMode)1, &(Grappa::impl::local_gce), Grappa::impl::Vertex<CCData, Grappa::Empty, false>, void, void NaiveGraphlabEngine<Grappa::Graph<CCData, Grappa::Empty>, LabelPropagation>::run_sync<CCData, Grappa::Empty>(GlobalAddress<Grappa::Graph<CCData, Grappa::Empty> >)::{lambda(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&)#3}::operator()(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&) const::{lambda(Grappa::Graph<CCData, Grappa::Empty>::Edge&)#1}::operator()(Grappa::Graph<CCData, Grappa::Empty>::Edge) const::{lambda(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&)#1}>(GlobalAddress<Grappa::impl::Vertex<CCData, Grappa::Empty, false> >, void NaiveGraphlabEngine<Grappa::Graph<CCData, Grappa::Empty>, LabelPropagation>::run_sync<CCData, Grappa::Empty>(GlobalAddress<Grappa::Graph<CCData, Grappa::Empty> >)::{lambda(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&)#3}::operator()(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&) const::{lambda(Grappa::Graph<CCData, Grappa::Empty>::Edge&)#1}::operator()(Grappa::Graph<CCData, Grappa::Empty>::Edge) const::{lambda(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&)#1}, void (Grappa::impl::Vertex<CCData, Grappa::Empty, false>::*)({lambda(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&)#1}&) const)::{lambda()#1}::operator()() const ()\r\n#18 0x00000000005b5827 in _ZZN6Grappa4impl11SpecializerILNS_8SyncModeE1EXadL_ZNS0_9local_gceEEEZNS0_4callILS2_1EXadL_ZNS0_9local_gceEEENS0_6VertexI6CCDataNS_5EmptyELb0EEEvZZZN19NaiveGraphlabEngineINS_5GraphIS5_S6_EE16LabelPropagationE8run_syncIS5_S6_EEv13GlobalAddressINS9_IT_T0_EEEENKUlRS7_E1_clESJ_ENKUlRNSA_4EdgeEE_clESM_EUlSJ_E_EEDTclfp0_decldtfp_7pointerEEESE_IT1_ET3_MSS_KFT2_RSQ_EEUlvE_E4callEsSX_MSX_KFvvEENKUlvE_clEv ()\r\n#19 0x00000000005eba59 in _ZN6Grappa7MessageIZNS_4impl11SpecializerILNS_8SyncModeE1EXadL_ZNS1_9local_gceEEEZNS1_4callILS3_1EXadL_ZNS1_9local_gceEEENS1_6VertexI6CCDa---Type <return> to continue, or q <return> to quit---bt\r\ntaNS_5EmptyELb0EEEvZZZN19NaiveGraphlabEngineINS_5GraphIS6_S7_EE16LabelPropagationE8run_syncIS6_S7_EEv13GlobalAddressINSA_IT_T0_EEEENKUlRS8_E1_clESK_ENKUlRNSB_4EdgeEE_clESN_EUlSK_E_EEDTclfp0_decldtfp_7pointerEEESF_IT1_ET3_MST_KFT2_RSR_EEUlvE_E4callEsSY_MSY_KFvvEEUlvE_E20deserialize_and_callEPc ()\r\n#20 0x000000000061bb9b in Grappa::impl::MessageBase::deserialize_and_call(char*) ()\r\n#21 0x00000000006151df in Grappa::impl::RDMAAggregator::deaggregate_buffer(char*, unsigned long) ()\r\n#22 0x0000000000616cda in Grappa::impl::RDMAAggregator::receive_buffer(Grappa::impl::RDMABuffer*) ()\r\n#23 0x0000000000615e73 in Grappa::impl::RDMAAggregator::receive_worker() ()\r\n#24 0x0000000000613697 in Grappa::impl::RDMAAggregator::activate()::{lambda()#2}::operator()() const ()\r\n#25 0x000000000061a618 in void Grappa::impl::worker_heapfunctor_proxy<Grappa::impl::RDMAAggregator::activate()::{lambda()#2}>(Grappa::Worker*,\r\n ```\r\n\r\nThen, I thought maybe that is because I am using map and allocate the memory locally, I changed the code to:\r\n\r\n``` C++\r\nstruct label_counter {\r\n  // A map that counts the number of time each label is used.\r\n  GlobalAddress<int> label_count;\r\n\r\n  explicit label_counter(int label) {\r\n    label_count = global_alloc<int>(totalGroup);\r\n    // Init the array\r\n    for (int i =0 ; i < totalGroup; i++) {\r\n       delegate::write((label_count + i ), 0);\r\n    }\r\n\r\n    if (label != Unknown)\r\n      delegate::write(label_count + label,  1);\r\n  }\r\n\r\n  explicit label_counter() {}\r\n\r\n  label_counter& operator+=(const label_counter& other) { \r\n    for (int i = 0; i < totalGroup; i++) {\r\n      int currentValue = delegate::read(other.label_count + i);\r\n      delegate::write((label_count + i), currentValue + 1);\r\n    }\r\n\r\n    return *this; \r\n  }\r\n};\r\n\r\nusing G = Graph<CCData,Empty>;\r\n\r\nstruct LabelPropagation : public GraphlabVertexProgram<G, label_counter> {\r\n  bool do_scatter;\r\n  int changelabel;\r\n\r\n  LabelPropagation(Vertex& v) {\r\n    do_scatter = false;\r\n  }\r\n\r\n  bool gather_edges(const Vertex& v) const { return true; }\r\n\r\n  Gather gather(const Vertex& v, Edge& e) const { \r\n    return label_counter(v->label); \r\n  }\r\n\r\n  void apply(Vertex& v, const Gather& total) {\r\n    int maxCount = 0;\r\n    int maxLabel = v->label;\r\n\r\n    for (int i = 0; i < totalGroup; i++) { \r\n        int labelValue = delegate::read(total.label_count + i);\r\n        // LOG(INFO) << \"label value: is \" << labelValue;\r\n        if (labelValue > maxCount) {\r\n          maxCount = labelValue;\r\n          maxLabel = i;\r\n        }\r\n    }\r\n\r\n    // All of neighbours are unknown, or it doesn't have nieghbour\r\n    if (maxLabel == Unknown) {\r\n      //LOG(INFO) << \"vertex id:  unknown \";// << v->id << \" Label is unknown.\";\r\n      do_scatter = false;\r\n      v->activate();\r\n    } else if (maxLabel != v->label) {\r\n        v->label = maxLabel;\r\n        //LOG(INFO) << \"vertex id: found\"; // << v->id << \" Label is: .\" << v->label;\r\n        changelabel = v->label;\r\n        do_scatter = true;\r\n    } else {\r\n       //LOG(INFO) << \"vertex id: \"; // << v->id << \" Label is: .\" << v->label << \" lebel didn't change\";\r\n      do_scatter = false;\r\n    }\r\n  }\r\n  \r\n  bool scatter_edges(const Vertex& v) const { return do_scatter; }\r\n  \r\n  Gather scatter(const Edge& e, Vertex& target) const {\r\n    // I think this is not needed.\r\n    return label_counter(changelabel);\r\n\r\n  }\r\n};\r\n\r\n```\r\nWhich still works fine on one node, but now I am getting this error when I run it on more than one node:\r\n\r\n```\r\n\r\n0: I0511 17:11:36.921659 25537 lp1.cpp:207] Finished loading graph\r\n0: I0511 17:11:36.921665 25537 lp1.cpp:209] \"construction_time\": 19.1613\r\n0: I0511 17:11:36.921675 25537 lp1.cpp:214] start laoding seed file\r\n0: I0511 17:11:40.701381 25537 lp1.cpp:166] Loaded Seed file. Count: 1005593\r\n0: I0511 17:12:12.902834 25537 lp1.cpp:224] Init the labels\r\n0: I0511 17:12:14.383291 25537 lp1.cpp:244] Init is complete\r\n2: F0511 17:12:14.533294 25282 TaskingScheduler.hpp:154] Check failed: in_no_switch_region_ == false (1 vs. 0) Trying to context switch in no-switch region\r\n1: F0511 17:12:14.534068 25538 TaskingScheduler.hpp:154] Check failed: in_no_switch_region_ == false (1 vs. 0) Trying to context switch in no-switch region\r\n3: F0511 17:12:14.533303 25283 TaskingScheduler.hpp:154] Check failed: in_no_switch_region_ == false (1 vs. 0) Trying to context switch in no-switch region\r\n1: *** Check failure stack trace: ***\r\n1: *** Aborted at 1431389534 (unix time) try \"date -d @1431389534\" if you are using GNU date ***\r\n\r\n```\r\n\r\nI feel there is something big in the design of this that I am missing. Appreciate your help. "},"comment":{"url":"https://api.github.com/repos/uwsampa/grappa/issues/comments/101816847","html_url":"https://github.com/uwsampa/grappa/issues/210#issuecomment-101816847","issue_url":"https://api.github.com/repos/uwsampa/grappa/issues/210","id":101816847,"user":{"login":"bholt","id":398411,"avatar_url":"https://avatars.githubusercontent.com/u/398411?v=3","gravatar_id":"","url":"https://api.github.com/users/bholt","html_url":"https://github.com/bholt","followers_url":"https://api.github.com/users/bholt/followers","following_url":"https://api.github.com/users/bholt/following{/other_user}","gists_url":"https://api.github.com/users/bholt/gists{/gist_id}","starred_url":"https://api.github.com/users/bholt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bholt/subscriptions","organizations_url":"https://api.github.com/users/bholt/orgs","repos_url":"https://api.github.com/users/bholt/repos","events_url":"https://api.github.com/users/bholt/events{/privacy}","received_events_url":"https://api.github.com/users/bholt/received_events","type":"User","site_admin":false},"created_at":"2015-05-13T21:08:06Z","updated_at":"2015-05-13T21:08:06Z","body":"right, all of the scatters of all the vertices will be in parallel (by design! that's the whole point of the graphlab gather-apply-scatter model). The only potential gotcha is if you're reading and updating the same state on a vertex in the scatter phase. In this case you're only reading `changelabel` and updating the delta cache. All the delta cache updates are still atomic so there aren't any lost updates. Then we wait for all the scatters to finish before moving on to the next iteration of the while loop and next `apply` stage."}}