{"action":"created","issue":{"url":"https://api.github.com/repos/weaveworks/weave/issues/1107","labels_url":"https://api.github.com/repos/weaveworks/weave/issues/1107/labels{/name}","comments_url":"https://api.github.com/repos/weaveworks/weave/issues/1107/comments","events_url":"https://api.github.com/repos/weaveworks/weave/issues/1107/events","html_url":"https://github.com/weaveworks/weave/issues/1107","id":94025534,"number":1107,"title":"Weave can negotiate MTU sizes that are too small for Linux","user":{"login":"bboreham","id":8125524,"avatar_url":"https://avatars.githubusercontent.com/u/8125524?v=3","gravatar_id":"","url":"https://api.github.com/users/bboreham","html_url":"https://github.com/bboreham","followers_url":"https://api.github.com/users/bboreham/followers","following_url":"https://api.github.com/users/bboreham/following{/other_user}","gists_url":"https://api.github.com/users/bboreham/gists{/gist_id}","starred_url":"https://api.github.com/users/bboreham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bboreham/subscriptions","organizations_url":"https://api.github.com/users/bboreham/orgs","repos_url":"https://api.github.com/users/bboreham/repos","events_url":"https://api.github.com/users/bboreham/events{/privacy}","received_events_url":"https://api.github.com/users/bboreham/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/weaveworks/weave/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/weaveworks/weave/labels/[component/router]","name":"[component/router]","color":"f7c6c7"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-07-09T11:43:09Z","updated_at":"2015-07-09T15:02:49Z","closed_at":null,"body":"Evidence:\r\n\r\n````\r\nweave 2015/07/09 08:07:02.804486 EMSGSIZE on send, expecting PMTU update (IP packet was 60064 bytes, payload was 60056 bytes)\r\nweave 2015/07/09 08:07:02.804552 ->[80.152.241.31:25623|da:08:ed:d4:49:31(boot2docker)]: Effective PMTU set to 1438\r\nweave 2015/07/09 08:07:02.814692 EMSGSIZE on send, expecting PMTU update (IP packet was 1480 bytes, payload was 1472 bytes)\r\nweave 2015/07/09 08:07:02.814715 ->[80.152.241.31:25623|da:08:ed:d4:49:31(boot2docker)]: Effective PMTU set to 1426\r\nweave 2015/07/09 08:07:02.885650 EMSGSIZE on send, expecting PMTU update (IP packet was 1468 bytes, payload was 1460 bytes)\r\nweave 2015/07/09 08:07:02.885688 ->[80.152.241.31:25623|da:08:ed:d4:49:31(boot2docker)]: Effective PMTU set to 1422\r\nweave 2015/07/09 08:07:12.812633 Peer da:08:ed:d4:49:31(boot2docker) moved from true to 80.152.241.31:23844\r\nweave 2015/07/09 08:07:22.804784 Peer da:08:ed:d4:49:31(boot2docker) moved from true to 80.152.241.31:58782\r\nweave 2015/07/09 08:07:24.064417 ->[80.152.241.31:25623|da:08:ed:d4:49:31(boot2docker)]: Effective PMTU set to 183\r\nweave 2015/07/09 08:07:24.064443 ->[80.152.241.31:25623|da:08:ed:d4:49:31(boot2docker)]: Effective PMTU verified at 183\r\nweave 2015/07/09 08:16:49.405847 Discovered remote MAC 8e:26:b1:f7:96:88 at da:08:ed:d4:49:31(boot2docker)\r\nweave 2015/07/09 08:16:49.544373 Sending ICMP 3,4 (10.64.0.3 -> 10.64.0.2): PMTU= 183\r\nweave 2015/07/09 08:16:49.544460 Sending ICMP 3,4 (10.64.0.3 -> 10.64.0.2): PMTU= 183\r\nweave 2015/07/09 08:16:49.544491 Sending ICMP 3,4 (10.64.0.3 -> 10.64.0.2): PMTU= 183\r\nweave 2015/07/09 08:16:49.544520 Sending ICMP 3,4 (10.64.0.3 -> 10.64.0.2): PMTU= 183\r\nweave 2015/07/09 08:16:49.544559 Sending ICMP 3,4 (10.64.0.3 -> 10.64.0.2): PMTU= 183\r\nweave 2015/07/09 08:16:49.544592 Sending ICMP 3,4 (10.64.0.3 -> 10.64.0.2): PMTU= 183\r\n````\r\n\r\nLooking at packets using `tcpdump`, we see Linux is ignoring our 183-byte limit and sending packets of 552 bytes, which is 512 plus the 40-byte TCP header:\r\n\r\n````\r\n    pandora.weave.local.9042 > cli-test.weave.local.34517: Flags [.], cksum 0x1204 (correct), seq 196:696, ack 327, win 342, options [nop,nop,TS val 863080443 ecr 3493719], length 500\r\n10:03:04.561944 IP (tos 0x0, ttl 64, id 59620, offset 0, flags [DF], proto TCP (6), length 552)\r\n    pandora.weave.local.9042 > cli-test.weave.local.34517: Flags [.], cksum 0x4540 (correct), seq 696:1196, ack 327, win 342, options [nop,nop,TS val 863080443 ecr 3493719], length 500\r\n10:03:04.561967 IP (tos 0x0, ttl 64, id 59621, offset 0, flags [DF], proto TCP (6), length 552)\r\n    pandora.weave.local.9042 > cli-test.weave.local.34517: Flags [.], cksum 0x3a07 (correct), seq 1196:1696, ack 327, win 342, options [nop,nop,TS val 863080443 ecr 3493719], length 500\r\n10:03:04.561972 IP (tos 0x0, ttl 64, id 59622, offset 0, flags [DF], proto TCP (6), length 552)\r\n    pandora.weave.local.9042 > cli-test.weave.local.34517: Flags [.], cksum 0x11f5 (correct), seq 1696:2196, ack 327, win 342, options [nop,nop,TS val 863080443 ecr 3493719], length 500\r\n````\r\n\r\nBackground reading: http://cschramm.blogspot.co.uk/2012/12/why-does-linux-enforce-minimum-mtu-of.html\r\n\r\nI suggest that, if weave finds itself with an MTU lower than 552, it should drop the connection."},"comment":{"url":"https://api.github.com/repos/weaveworks/weave/issues/comments/120017860","html_url":"https://github.com/weaveworks/weave/issues/1107#issuecomment-120017860","issue_url":"https://api.github.com/repos/weaveworks/weave/issues/1107","id":120017860,"user":{"login":"dpw","id":15074,"avatar_url":"https://avatars.githubusercontent.com/u/15074?v=3","gravatar_id":"","url":"https://api.github.com/users/dpw","html_url":"https://github.com/dpw","followers_url":"https://api.github.com/users/dpw/followers","following_url":"https://api.github.com/users/dpw/following{/other_user}","gists_url":"https://api.github.com/users/dpw/gists{/gist_id}","starred_url":"https://api.github.com/users/dpw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dpw/subscriptions","organizations_url":"https://api.github.com/users/dpw/orgs","repos_url":"https://api.github.com/users/dpw/repos","events_url":"https://api.github.com/users/dpw/events{/privacy}","received_events_url":"https://api.github.com/users/dpw/received_events","type":"User","site_admin":false},"created_at":"2015-07-09T15:02:49Z","updated_at":"2015-07-09T15:02:49Z","body":"The connection is practically useless in any case, no?  When the kernel drops the MTU on the underlying network to 552, weave will come up with an effective MTU for the overlay network of less than 552, and report this back to the kernel via frag-needed responses, but they get ignored because they are less than 552.  What can we do except wait for the MTU issue to resolve itself?\r\n\r\n(I am going on my memories of #469 here.  I haven't looked at your user's situation carefully enough to be confident it corresponds. I'm not sure where 183 comes from.)\r\n\r\nHow to surface such conditions to the user? Log them! Avoiding all the noise about reconnections will make it easier to notice the log entries which say what is actually going on.\r\n\r\nIt would also be nice to show more connection state info in `weave status` and other UIs, but that's a separate issue."}}