Google Container Engine (Kubernetes): Websocket (Socket.io) not working on multiple replicas
<p>I am new to Google Container Engine (GKE). When run on <code>localhost</code> it's working fine but when I deploy to production with GKE I got websocket error.</p>

<p>My node app is develop with <code>Hapi.js</code> and <code>Socket.io</code> and my structure is shown in image below.</p>

<p><a href="https://i.stack.imgur.com/4cUsG.png" rel="noreferrer">Application Architecture</a></p>

<p>I'm using Glue to compose Hapi server. Below is my <code>manifest.json</code></p>

<pre><code>{
...
"connections": [
    {
      "host": "app",
      "address": "0.0.0.0",
      "port": 8000,
      "labels": ["api"],
      "routes": {
        "cors": false,
        "security": {
          "hsts": false,
          "xframe": true,
          "xss": true,
          "noOpen": true,
          "noSniff": true
        }
      },
      "router": {
        "stripTrailingSlash": true
      },
      "load": {
        "maxHeapUsedBytes": 1073741824,
        "maxRssBytes": 1610612736,
        "maxEventLoopDelay": 5000
      }
    },
    {
      "host": "app",
      "address": "0.0.0.0",
      "port": 8099,
      "labels": ["web"],
      "routes": {
        "cors": true,
        "security": {
          "hsts": false,
          "xframe": true,
          "xss": true,
          "noOpen": true,
          "noSniff": true
        }
      },
      "router": {
        "stripTrailingSlash": true
      },
      "load": {
        "maxHeapUsedBytes": 1073741824,
        "maxRssBytes": 1610612736,
        "maxEventLoopDelay": 5000
      }
    },
    {
      "host": "app",
      "address": "0.0.0.0",
      "port": 8999,
      "labels": ["admin"],
      "routes": {
        "cors": true,
        "security": {
          "hsts": false,
          "xframe": true,
          "xss": true,
          "noOpen": true,
          "noSniff": true
        }
      },
      "router": {
        "stripTrailingSlash": true
      },
      "load": {
        "maxHeapUsedBytes": 1073741824,
        "maxRssBytes": 1610612736,
        "maxEventLoopDelay": 5000
      },
      "state": {
        "ttl": null,
        "isSecure": false,
        "isHttpOnly": true,
        "path": null,
        "domain": null,
        "encoding": "none",
        "clearInvalid": false,
        "strictHeader": true
      }
    }
  ],
...
}
</code></pre>

<p>And my <code>nginx.conf</code></p>

<pre><code>worker_processes                5; ## Default: 1
worker_rlimit_nofile            8192;
error_log                       /dev/stdout info;

events {
  worker_connections            4096; ## Default: 1024
}

http {
    access_log                  /dev/stdout;

    server {
        listen                  80          default_server;
        listen                  [::]:80     default_server;

        # Redirect all HTTP requests to HTTPS with a 301 Moved Permanently response.
        return                  301         https://$host$request_uri;
    }

    server {
        listen                  443         ssl default_server;
        listen                  [::]:443    ssl default_server;
        server_name             _;

        # Configure ssl
        ssl_certificate         /etc/secret/ssl/myapp.com.csr;
        ssl_certificate_key     /etc/secret/ssl/myapp.com.key;
        include                 /etc/nginx/ssl-params.conf;
    }

    server {
        listen                  443         ssl;
        listen                  [::]:443    ssl;
        server_name             api.myapp.com;

        location / {
            proxy_pass          http://api_app/;
            proxy_set_header    Host                $http_host;
            proxy_set_header    X-Real-IP           $remote_addr;
            proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;

            # Handle Web Socket connections
            proxy_http_version  1.1;
            proxy_set_header    Upgrade     $http_upgrade;
            proxy_set_header    Connection  "upgrade";
        }
    }

    server {
        listen                  443         ssl;
        listen                  [::]:443    ssl;
        server_name             myapp.com;

        location / {
            proxy_pass          http://web_app/;
            proxy_set_header    Host                $http_host;
            proxy_set_header    X-Real-IP           $remote_addr;
            proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;

            # Handle Web Socket connections
            proxy_http_version  1.1;
            proxy_set_header    Upgrade     $http_upgrade;
            proxy_set_header    Connection  "upgrade";
        }
    }

    server {
        listen                  443         ssl;
        listen                  [::]:443    ssl;
        server_name             admin.myapp.com;

        location / {
            proxy_pass          http://admin_app/;
            proxy_set_header    Host                $http_host;
            proxy_set_header    X-Real-IP           $remote_addr;
            proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;

            # Handle Web Socket connections
            proxy_http_version  1.1;
            proxy_set_header    Upgrade     $http_upgrade;
            proxy_set_header    Connection  "upgrade";
        }
    }

    # Define your "upstream" servers - the
    # servers request will be sent to
    upstream api_app {
        server                  localhost:8000;
    }

    upstream web_app {
        server                  localhost:8099;
    }

    upstream admin_app {
        server                  localhost:8999;
    }
}
</code></pre>

<p>Kubernetes service <code>app-service.yaml</code></p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: app-nginx
  labels:
    app: app-nginx
spec:
  type: LoadBalancer
  ports:
    # The port that this service should serve on.
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
    - port: 443
      targetPort: 443
      protocol: TCP
      name: https
  # Label keys and values that must match in order to receive traffic for this service.
  selector:
    app: app-nginx
</code></pre>

<p>Kubernetes Deployment <code>app-deployment.yaml</code></p>

<pre><code>apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: app-nginx
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: app-nginx
    spec:
      containers:
        - name: nginx
          image: us.gcr.io/myproject/nginx
          ports:
            - containerPort: 80
              name: http
            - containerPort: 443
              name: https
          volumeMounts:
              # This name must match the volumes.name below.
            - name: ssl-secret
              readOnly: true
              mountPath: /etc/secret/ssl
        - name: app
          image: us.gcr.io/myproject/bts-server
          ports:
            - containerPort: 8000
              name: api
            - containerPort: 8099
              name: web
            - containerPort: 8999
              name: admin
          volumeMounts:
              # This name must match the volumes.name below.
            - name: client-secret
              readOnly: true
              mountPath: /etc/secret/client
            - name: admin-secret
              readOnly: true
              mountPath: /etc/secret/admin
      volumes:
        - name: ssl-secret
          secret:
            secretName: ssl-key-secret
        - name: client-secret
          secret:
            secretName: client-key-secret
        - name: admin-secret
          secret:
            secretName: admin-key-secret
</code></pre>

<p>And I'm using <code>Cloudflare SSL full strict</code>.</p>

<p>Error get from Browser console:</p>

<pre><code>WebSocket connection to 'wss://api.myapp.com/socket.io/?EIO=3&amp;transport=websocket&amp;sid=4Ky-y9K7J0XotrBFAAAQ' failed: WebSocket is closed before the connection is established.
https://api.myapp.com/socket.io/?EIO=3&amp;transport=polling&amp;t=LYByND2&amp;sid=4Ky-y9K7J0XotrBFAAAQ Failed to load resource: the server responded with a status of 400 ()
VM50:35 WebSocket connection to 'wss://api.myapp.com/socket.io/?EIO=3&amp;transport=websocket&amp;sid=FsCGx-UE7ohrsSSqAAAT' failed: Error during WebSocket handshake: Unexpected response code: 502WrappedWebSocket @ VM50:35WS.doOpen @ socket.io.js:6605Transport.open @ socket.io.js:4695Socket.probe @ socket.io.js:3465Socket.onOpen @ socket.io.js:3486Socket.onHandshake @ socket.io.js:3546Socket.onPacket @ socket.io.js:3508(anonymous function) @ socket.io.js:3341Emitter.emit @ socket.io.js:6102Transport.onPacket @ socket.io.js:4760callback @ socket.io.js:4510(anonymous function) @ socket.io.js:5385exports.decodePayloadAsBinary @ socket.io.js:5384exports.decodePayload @ socket.io.js:5152Polling.onData @ socket.io.js:4514(anonymous function) @ socket.io.js:4070Emitter.emit @ socket.io.js:6102Request.onData @ socket.io.js:4231Request.onLoad @ socket.io.js:4312xhr.onreadystatechange @ socket.io.js:4184
socket.io.js:4196 GET https://api.myapp.com/socket.io/?EIO=3&amp;transport=polling&amp;t=LYByNpy&amp;sid=FsCGx-UE7ohrsSSqAAAT 400 ()
</code></pre>

<p>And here is Nginx's logs:</p>

<pre><code>[22/Nov/2016:12:10:19 +0000] "GET /socket.io/?EIO=3&amp;transport=websocket&amp;sid=MGc--oncQbQI6NOZAAAX HTTP/1.1" 101 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"
10.8.0.1 - - [22/Nov/2016:12:10:19 +0000] "POST /socket.io/?EIO=3&amp;transport=polling&amp;t=LYByQBw&amp;sid=MGc--oncQbQI6NOZAAAX HTTP/1.1" 200 2 "https://myapp.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"
10.128.0.2 - - [22/Nov/2016:12:10:20 +0000] "GET /socket.io/?EIO=3&amp;transport=polling&amp;t=LYByQKp HTTP/1.1" 200 101 "https://myapp.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"
10.8.0.1 - - [22/Nov/2016:12:10:21 +0000] "GET /socket.io/?EIO=3&amp;transport=polling&amp;t=LYByQWo&amp;sid=c5nkusT9fEPRsu2rAAAY HTTP/1.1" 200 24 "https://myapp.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"
2016/11/22 12:10:21 [error] 6#6: *157 connect() failed (111: Connection refused) while connecting to upstream, client: 10.8.0.1, server: api.myapp.com, request: "GET /socket.io/?EIO=3&amp;transport=polling&amp;t=LYByQaN&amp;sid=c5nkusT9fEPRsu2rAAAY HTTP/1.1", upstream: "http://[::1]:8000/socket.io/?EIO=3&amp;transport=polling&amp;t=LYByQaN&amp;sid=c5nkusT9fEPRsu2rAAAY", host: "api.myapp.com", referrer: "https://myapp.com/"
2016/11/22 12:10:21 [warn] 6#6: *157 upstream server temporarily disabled while connecting to upstream, client: 10.8.0.1, server: api.myapp.com, request: "GET /socket.io/?EIO=3&amp;transport=polling&amp;t=LYByQaN&amp;sid=c5nkusT9fEPRsu2rAAAY HTTP/1.1", upstream: "http://[::1]:8000/socket.io/?EIO=3&amp;transport=polling&amp;t=LYByQaN&amp;sid=c5nkusT9fEPRsu2rAAAY", host: "api.myapp.com", referrer: "https://myapp.com/"
10.8.0.1 - - [22/Nov/2016:12:10:22 +0000] "GET /socket.io/?EIO=3&amp;transport=polling&amp;t=LYByQaN&amp;sid=c5nkusT9fEPRsu2rAAAY HTTP/1.1" 200 4 "https://myapp.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"
</code></pre>

<p><strong>UPDATE</strong></p>

<p>When I change <code>replicas</code> to <code>1</code> in <code>app-deployment.yaml</code> it's work. But I think it's not a good solution. I need <strong>3 replicas</strong>. </p>

<pre><code>apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: app-nginx
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            app: app-nginx
</code></pre>

<p><strong>How to make it work with 3 replicas?</strong></p>