{"action":"created","issue":{"url":"https://api.github.com/repos/grpc/grpc/issues/2423","labels_url":"https://api.github.com/repos/grpc/grpc/issues/2423/labels{/name}","comments_url":"https://api.github.com/repos/grpc/grpc/issues/2423/comments","events_url":"https://api.github.com/repos/grpc/grpc/issues/2423/events","html_url":"https://github.com/grpc/grpc/issues/2423","id":94951565,"number":2423,"title":"GRPC c++ stuck -- likely a (critical) deadlock","user":{"login":"kianooshm","id":12394155,"avatar_url":"https://avatars.githubusercontent.com/u/12394155?v=3","gravatar_id":"","url":"https://api.github.com/users/kianooshm","html_url":"https://github.com/kianooshm","followers_url":"https://api.github.com/users/kianooshm/followers","following_url":"https://api.github.com/users/kianooshm/following{/other_user}","gists_url":"https://api.github.com/users/kianooshm/gists{/gist_id}","starred_url":"https://api.github.com/users/kianooshm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kianooshm/subscriptions","organizations_url":"https://api.github.com/users/kianooshm/orgs","repos_url":"https://api.github.com/users/kianooshm/repos","events_url":"https://api.github.com/users/kianooshm/events{/privacy}","received_events_url":"https://api.github.com/users/kianooshm/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"vjpai","id":8864912,"avatar_url":"https://avatars.githubusercontent.com/u/8864912?v=3","gravatar_id":"","url":"https://api.github.com/users/vjpai","html_url":"https://github.com/vjpai","followers_url":"https://api.github.com/users/vjpai/followers","following_url":"https://api.github.com/users/vjpai/following{/other_user}","gists_url":"https://api.github.com/users/vjpai/gists{/gist_id}","starred_url":"https://api.github.com/users/vjpai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vjpai/subscriptions","organizations_url":"https://api.github.com/users/vjpai/orgs","repos_url":"https://api.github.com/users/vjpai/repos","events_url":"https://api.github.com/users/vjpai/events{/privacy}","received_events_url":"https://api.github.com/users/vjpai/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/grpc/grpc/milestones/25","html_url":"https://github.com/grpc/grpc/milestones/Beta","labels_url":"https://api.github.com/repos/grpc/grpc/milestones/25/labels","id":1134869,"number":25,"title":"Beta","description":"","creator":{"login":"ctiller","id":10120821,"avatar_url":"https://avatars.githubusercontent.com/u/10120821?v=3","gravatar_id":"","url":"https://api.github.com/users/ctiller","html_url":"https://github.com/ctiller","followers_url":"https://api.github.com/users/ctiller/followers","following_url":"https://api.github.com/users/ctiller/following{/other_user}","gists_url":"https://api.github.com/users/ctiller/gists{/gist_id}","starred_url":"https://api.github.com/users/ctiller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ctiller/subscriptions","organizations_url":"https://api.github.com/users/ctiller/orgs","repos_url":"https://api.github.com/users/ctiller/repos","events_url":"https://api.github.com/users/ctiller/events{/privacy}","received_events_url":"https://api.github.com/users/ctiller/received_events","type":"User","site_admin":false},"open_issues":214,"closed_issues":68,"state":"open","created_at":"2015-05-26T22:29:20Z","updated_at":"2015-07-15T15:04:20Z","due_on":null,"closed_at":null},"comments":5,"created_at":"2015-07-14T14:00:15Z","updated_at":"2015-07-15T15:24:47Z","closed_at":null,"body":"Took lots of investigation time. Reduced to the minimal example copied below. Blocking our project right now.\r\n\r\nSuppose a server and a number of concurrent clients, each creating their own channel to the server and sending a stream of requests (the *response* direction is not used). No shared data or locking. When there are more than just 6 client threads doing this, GRPC gets stuck. Fewer than 6 is fine -- running on a 4-core machine. Project is to run 100s of threads on a 48-core server.\r\n\r\n**Proto**:\r\n```\r\nsyntax = \"proto3\";\r\n\r\nservice MyService {\r\n    rpc Process(stream BatchRequest) returns (stream BatchResponse);\r\n}\r\n\r\nmessage Request {\r\n    uint64 x = 1;\r\n}\r\n\r\nmessage BatchRequest {\r\n    repeated Request requests = 1;\r\n}\r\n\r\nmessage BatchResponse {}\r\n```\r\n\r\n**Code**:\r\n```c++\r\n#include <grpc/grpc.h>\r\n#include <grpc++/channel_arguments.h>\r\n#include <grpc++/create_channel.h>\r\n#include <grpc++/server.h>\r\n#include <grpc++/server_builder.h>\r\n#include <grpc++/server_context.h>\r\n#include <grpc++/server_credentials.h>\r\n#include <grpc++/status.h>\r\n#include <grpc++/stream.h>\r\n#include <unistd.h>\r\n#include <cassert>\r\n#include <iostream>\r\n#include <memory>\r\n#include <string>\r\n#include <thread>\r\n#include <vector>\r\n\r\n#include \"myservice.pb.h\"\r\n#include \"myservice.grpc.pb.h\"\r\n\r\nusing grpc::ChannelArguments;\r\nusing grpc::ChannelInterface;\r\nusing grpc::ClientContext;\r\nusing grpc::ClientReaderWriter;\r\nusing grpc::CreateChannel;\r\nusing grpc::InsecureCredentials;\r\nusing grpc::InsecureServerCredentials;\r\nusing grpc::ServerBuilder;\r\nusing grpc::ServerContext;\r\nusing grpc::ServerReaderWriter;\r\nusing grpc::Status;\r\nusing std::cerr;\r\nusing std::endl;\r\nusing std::string;\r\nusing std::vector;\r\n\r\nstatic const string g_service_addr = \"0.0.0.0:7000\";\r\nstatic const int g_num_client_threads = 8;\r\nstatic const int g_batch_size = 10000;\r\nstatic const int g_num_requests = 1000000;\r\n\r\nclass MyServiceImpl : public MyService::Service {\r\n  public:\r\n    MyServiceImpl() {}\r\n    ~MyServiceImpl() {}\r\n\r\n    Status Process(ServerContext* context,\r\n                   ServerReaderWriter<BatchResponse, BatchRequest>* stream)\r\n                   override {\r\n        static int __thread_id = 0;\r\n        int thread_id = __thread_id++;  // Not really thread safe, but ok\r\n        cerr << \"Service instance <\" << thread_id << \"> started\\n\";\r\n        while (1) {\r\n            // Read the next request and discard. Nothing to write to the\r\n            // \"response\" direction of the stream; neither the client is\r\n            // expecting anything.\r\n            BatchRequest batch_request;\r\n            cerr << \"Service instance <\" << thread_id << \"> before Read()\\n\";\r\n            if (!stream->Read(&batch_request))\r\n                break;\r\n            cerr << \"Service instance <\" << thread_id << \"> after Read()\\n\";\r\n        }\r\n        cerr << \"Service instance <\" << thread_id << \">: end of stream\\n\";\r\n        return Status::OK;\r\n    }\r\n};\r\n\r\nclass ClientThread {\r\n  public:\r\n    explicit ClientThread(int thread_id) : thread_id_(thread_id) {\r\n        channel_ = CreateChannel(g_service_addr,\r\n                                 InsecureCredentials(), ChannelArguments());\r\n        stub_ = MyService::NewStub(channel_);\r\n        stream_ = stub_->Process(&client_context_);\r\n        cerr << \"Created GRPC channel to \" << g_service_addr << endl;\r\n    }\r\n\r\n    ~ClientThread() {\r\n        assert(stream_->WritesDone());\r\n        stream_->Finish().error_message();\r\n    }\r\n\r\n    void HandleRequest(const Request& request) {\r\n        Request* r = batch_request_.add_requests();\r\n        *r = request;\r\n        if (batch_request_.requests().size() >= g_batch_size) {\r\n            cerr << \"ClientThread<\" << thread_id_ << \"> before Write()\\n\";\r\n            assert(stream_->Write(batch_request_));\r\n            cerr << \"ClientThread<\" << thread_id_ << \"> after Write()\\n\";\r\n            batch_request_.Clear();\r\n        }\r\n    }\r\n\r\n  private:\r\n    const int thread_id_;\r\n    std::shared_ptr<ChannelInterface> channel_;\r\n    std::unique_ptr<MyService::Stub> stub_;\r\n    std::unique_ptr<ClientReaderWriter<BatchRequest, BatchResponse>> stream_;\r\n    ClientContext client_context_;\r\n    BatchRequest batch_request_;  // Buffer for batching\r\n};\r\n\r\nint main(int argc, char* argv[]) {\r\n    grpc_init();\r\n\r\n    ServerBuilder builder;\r\n    builder.AddListeningPort(g_service_addr, InsecureServerCredentials());\r\n    MyServiceImpl service;\r\n    builder.RegisterService(&service);\r\n    std::unique_ptr<grpc::Server> server(builder.BuildAndStart());\r\n    cerr << \"Server up and listening on \" << g_service_addr << endl;\r\n    std::thread server_thread{[&] {\r\n        server->Wait();\r\n    }};\r\n\r\n    vector<std::unique_ptr<ClientThread>> client_threads_;\r\n    for (int i = 0; i < g_num_client_threads; ++i) {\r\n        client_threads_.emplace_back(new ClientThread(i));\r\n    }\r\n\r\n    for (int i = 1; i < g_num_requests; ++i) {\r\n        Request request;\r\n        request.set_x(i);\r\n        client_threads_[i % client_threads_.size()]->HandleRequest(request);\r\n    }\r\n\r\n    client_threads_.clear();  // Close grpc streams\r\n    server->Shutdown();\r\n    server_thread.join();\r\n    grpc_shutdown();\r\n    return 0;\r\n}\r\n```\r\n\r\n**Makefile**:\r\n```\r\nLFLAGS := -std=c++11 -O2 -Wall -D_GLIBCXX_PERMIT_BACKWARD_HASH\r\nRFLAGS := -lgrpc++_unsecure -lgrpc -lgpr -lprotobuf -lpthread\r\n\r\ndebug_grpc: myservice.pb.cc myservice.grpc.pb.cc debug_grpc.cc\r\n\tg++ $(LFLAGS) -o debug_grpc debug_grpc.cc myservice.pb.cc myservice.grpc.pb.cc $(RFLAGS)\r\n\r\n%.grpc.pb.cc: %.proto\r\n\tprotoc -I . --grpc_out=. --plugin=protoc-gen-grpc=`which grpc_cpp_plugin` $<\r\n\r\n%.pb.cc: %.proto\r\n\tprotoc -I . --cpp_out=. $<\r\n\r\nclean:\r\n\trm -f *.pb.cc *.pb.h\r\n```\r\n\r\n**Output**: (i've reordered some lines for clarity)\r\n```\r\nServer up and listening on 0.0.0.0:7000\r\nCreated GRPC channel to 0.0.0.0:7000\r\nCreated GRPC channel to 0.0.0.0:7000\r\nCreated GRPC channel to 0.0.0.0:7000\r\nCreated GRPC channel to 0.0.0.0:7000\r\nCreated GRPC channel to 0.0.0.0:7000\r\nCreated GRPC channel to 0.0.0.0:7000\r\nCreated GRPC channel to 0.0.0.0:7000\r\nCreated GRPC channel to 0.0.0.0:7000 <-- so far, all 8 channels created successfully\r\n\r\nService instance <0> started\r\nService instance <1> started\r\nService instance <2> started\r\nService instance <3> started\r\nService instance <4> started\r\nService instance <5> started <-- problem: the last 2 threads successfully made a\r\n                                 connection but server's Process() is not called\r\n\r\nClientThread<0> before Write()\r\nClientThread<0> after Write()\r\nClientThread<1> before Write()\r\nClientThread<1> after Write()\r\nClientThread<1> before Write()\r\nClientThread<1> after Write()\r\nClientThread<2> before Write()\r\nClientThread<2> after Write()\r\nClientThread<2> before Write()\r\nClientThread<2> after Write()\r\nClientThread<3> before Write()\r\nClientThread<3> after Write()\r\nClientThread<3> before Write()\r\nClientThread<3> after Write()\r\nClientThread<4> before Write()\r\nClientThread<4> after Write()\r\nClientThread<4> before Write()\r\nClientThread<4> after Write()\r\nClientThread<5> before Write()\r\nClientThread<5> after Write()\r\nClientThread<5> before Write()\r\nClientThread<5> after Write()\r\nClientThread<6> before Write()\r\nClientThread<6> after Write()\r\nClientThread<6> before Write() <-- problem: this Write() never goes through\r\nClientThread<7> before Write()\r\nClientThread<7> after Write()\r\n\r\nService instance <0> before Read()\r\nService instance <0> after Read()\r\nService instance <0> before Read() <-- waiting for requests\r\nService instance <1> before Read()\r\nService instance <1> after Read()\r\nService instance <1> before Read()\r\nService instance <1> after Read()\r\nService instance <1> before Read() <-- waiting for requests\r\nService instance <2> before Read()\r\nService instance <2> after Read()\r\nService instance <2> before Read()\r\nService instance <2> after Read()\r\nService instance <2> before Read() <-- waiting for requests\r\nService instance <3> before Read()\r\nService instance <3> after Read()\r\nService instance <3> before Read()\r\nService instance <3> after Read()\r\nService instance <3> before Read() <-- waiting for requests\r\nService instance <4> before Read()\r\nService instance <4> after Read()\r\nService instance <4> before Read()\r\nService instance <4> after Read()\r\nService instance <4> before Read() <-- waiting for requests\r\nService instance <5> before Read()\r\nService instance <5> after Read()\r\nService instance <5> before Read()\r\nService instance <5> after Read()\r\nService instance <5> before Read() <-- waiting for requests\r\n```\r\nOf course the thread/instance IDs above at client and server side do not correspond, but that doesn't matter. The point is, Write() gets stuck as shown ---although the corresponding server thread is Read()-ing. Also, though the last two channels are created, the corresponding Process() method is not called at the server.\r\n\r\nOnce again, with fewer than 6 threads everything is fine and program runs to completion.\r\n\r\nPlease let me know if you need any more information to resolve this."},"comment":{"url":"https://api.github.com/repos/grpc/grpc/issues/comments/121651231","html_url":"https://github.com/grpc/grpc/issues/2423#issuecomment-121651231","issue_url":"https://api.github.com/repos/grpc/grpc/issues/2423","id":121651231,"user":{"login":"larsonmpdx","id":12436438,"avatar_url":"https://avatars.githubusercontent.com/u/12436438?v=3","gravatar_id":"","url":"https://api.github.com/users/larsonmpdx","html_url":"https://github.com/larsonmpdx","followers_url":"https://api.github.com/users/larsonmpdx/followers","following_url":"https://api.github.com/users/larsonmpdx/following{/other_user}","gists_url":"https://api.github.com/users/larsonmpdx/gists{/gist_id}","starred_url":"https://api.github.com/users/larsonmpdx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/larsonmpdx/subscriptions","organizations_url":"https://api.github.com/users/larsonmpdx/orgs","repos_url":"https://api.github.com/users/larsonmpdx/repos","events_url":"https://api.github.com/users/larsonmpdx/events{/privacy}","received_events_url":"https://api.github.com/users/larsonmpdx/received_events","type":"User","site_admin":false},"created_at":"2015-07-15T15:24:47Z","updated_at":"2015-07-15T15:24:47Z","body":"#include \"../../grpc/src/cpp/server/thread_pool.h\""}}