{"action":"created","issue":{"url":"https://api.github.com/repos/strands-project/aaf_deployment/issues/263","labels_url":"https://api.github.com/repos/strands-project/aaf_deployment/issues/263/labels{/name}","comments_url":"https://api.github.com/repos/strands-project/aaf_deployment/issues/263/comments","events_url":"https://api.github.com/repos/strands-project/aaf_deployment/issues/263/events","html_url":"https://github.com/strands-project/aaf_deployment/issues/263","id":81342037,"number":263,"title":"[werner] side PCs becoming unresponsive","user":{"login":"marc-hanheide","id":1153084,"avatar_url":"https://avatars.githubusercontent.com/u/1153084?v=3","gravatar_id":"","url":"https://api.github.com/users/marc-hanheide","html_url":"https://github.com/marc-hanheide","followers_url":"https://api.github.com/users/marc-hanheide/followers","following_url":"https://api.github.com/users/marc-hanheide/following{/other_user}","gists_url":"https://api.github.com/users/marc-hanheide/gists{/gist_id}","starred_url":"https://api.github.com/users/marc-hanheide/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marc-hanheide/subscriptions","organizations_url":"https://api.github.com/users/marc-hanheide/orgs","repos_url":"https://api.github.com/users/marc-hanheide/repos","events_url":"https://api.github.com/users/marc-hanheide/events{/privacy}","received_events_url":"https://api.github.com/users/marc-hanheide/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/strands-project/aaf_deployment/labels/deployment_report","name":"deployment_report","color":"5319e7"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-05-27T08:17:13Z","updated_at":"2015-05-27T13:10:56Z","closed_at":null,"body":"We first had this after the walking group on Thu 21/5/15. As left-cortex wouldn't reboot, we had to open Werner and fix it. @lobinho did a great job and said:\r\n> apparently the ssd connection was loose. I'm not sure which one exactly was the cause, as I could not see anything obvious. After disconnecting and reconnecting them and also doing this for power supply it got on. However the fan was running before, so power did not seem to be the problem. I fixated the ssd with tape, just to make sure.\r\n> \r\n> Then he was claiming that no CPU fan was connected. Indeed, it was connected to the chassis fan, while the labeling is a bit ambiguous. I don't know why this did not bother him before. Same connection at the left-pc without problems. Anyhow, I switched to CPU-fan for both side pcs and now it is displayed correctly in the BIOS. That was it.\r\n\r\nThen on 27/5/15 again, both side PCs were dead in the morning. They had to be rebooted by @TobKoer but came up again afterwards. Post-Mortem analysis carried out leading to the following observations:\r\n\r\n* `werner-left-cortex` seems to have died after 7:17:\r\n\r\n  ```\r\nMay 27 07:05:01 werner-left-cortex CRON[29281]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)\r\nMay 27 07:15:01 werner-left-cortex CRON[4176]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)\r\nMay 27 07:17:01 werner-left-cortex CRON[5590]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)\r\nMay 27 09:48:06 werner-left-cortex rsyslogd: [origin software=\"rsyslogd\" swVersion=\"7.4.4\" x-pid=\"646\" x-info=\"http://www.rsyslog.com\"] start\r\nMay 27 09:48:06 werner-left-cortex rsyslogd: rsyslogd's groupid changed to 104\r\nMay 27 09:48:06 werner-left-cortex rsyslogd: rsyslogd's userid changed to 101\r\n  ```\r\n  No further useful logs could be obtained. Maybe the BIOS shut it down due to a temperature problem.\r\n* `werner-right-cortex` was still running. The logs around the time of `werner-left-cortex` dying look like this:\r\n\r\n  ```\r\nMay 27 07:15:01 werner-right-cortex CRON[29280]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)\r\nMay 27 07:17:01 werner-right-cortex CRON[29618]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397786] ------------[ cut here ]------------\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397797] WARNING: CPU: 0 PID: 0 at /build/buildd/linux-3.13.0/net/sched/sch_generic.c:264 dev_watchdog+0x276/0x280()\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397800] NETDEV WATCHDOG: eth0 (e1000e): transmit queue 0 timed out\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397801] Modules linked in: btrfs raid6_pq xor ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c rpcsec_gss_krb5 joydev hid_generic uvcvideo videobuf2_v\r\nmalloc videobuf2_memops usbhid videobuf2_core snd_usb_audio hid videodev snd_usbmidi_lib bnep rfcomm bluetooth nfsv4 snd_hda_codec_hdmi nfsd auth_rpcgss nfs_acl binfmt_misc nfs lockd sunrpc fscache eeepc_wmi a\r\nsus_wmi sparse_keymap snd_hda_codec_realtek x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_intel kvm snd_hda_codec snd_hwdep crct10dif_pclmul crc32_pclmul snd_pcm ghash_clmulni_intel snd_page_alloc aes\r\nni_intel snd_seq_midi aes_x86_64 lrw snd_seq_midi_event gf128mul glue_helper ablk_helper snd_rawmidi cryptd snd_seq nvidia(POX) serio_raw snd_seq_device snd_timer lpc_ich mei_me snd mei soundcore drm parport_p\r\nc video ppdev shpchp mac_hid wmi lp parport e1000e psmouse ahci ptp libahci pps_core\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397886] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P           OX 3.13.0-49-generic #83-Ubuntu\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397893] Hardware name: ASUS All Series/H87I-PLUS, BIOS 0904 10/30/2013\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397896]  0000000000000009 ffff88041ec03d98 ffffffff81721b36 ffff88041ec03de0\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397902]  ffff88041ec03dd0 ffffffff810677dd 0000000000000000 ffff880403fb0000\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397907]  ffff880407b42280 0000000000000001 0000000000000000 ffff88041ec03e30\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397912] Call Trace:\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397915]  <IRQ>  [<ffffffff81721b36>] dump_stack+0x45/0x56\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397926]  [<ffffffff810677dd>] warn_slowpath_common+0x7d/0xa0\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397930]  [<ffffffff8106784c>] warn_slowpath_fmt+0x4c/0x50\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397937]  [<ffffffff81646566>] dev_watchdog+0x276/0x280\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397941]  [<ffffffff816462f0>] ? dev_graft_qdisc+0x80/0x80\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397946]  [<ffffffff810743b6>] call_timer_fn+0x36/0x100\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397950]  [<ffffffff816462f0>] ? dev_graft_qdisc+0x80/0x80\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397954]  [<ffffffff8107534f>] run_timer_softirq+0x1ef/0x2f0\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397960]  [<ffffffff8106cc4c>] __do_softirq+0xec/0x2c0\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397965]  [<ffffffff8106d195>] irq_exit+0x105/0x110\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397970]  [<ffffffff81734a15>] smp_apic_timer_interrupt+0x45/0x60\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397975]  [<ffffffff8173339d>] apic_timer_interrupt+0x6d/0x80\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397977]  <EOI>  [<ffffffff815d4362>] ? cpuidle_enter_state+0x52/0xc0\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397986]  [<ffffffff815d4489>] cpuidle_idle_call+0xb9/0x1f0\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397992]  [<ffffffff8101d35e>] arch_cpu_idle+0xe/0x30\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.397996]  [<ffffffff810befa5>] cpu_startup_entry+0xc5/0x290\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.398002]  [<ffffffff81710037>] rest_init+0x77/0x80\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.398007]  [<ffffffff81d35f70>] start_kernel+0x438/0x443\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.398011]  [<ffffffff81d35941>] ? repair_env_string+0x5c/0x5c\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.398016]  [<ffffffff81d35120>] ? early_idt_handlers+0x120/0x120\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.398027]  [<ffffffff81d355ee>] x86_64_start_reservations+0x2a/0x2c\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.398031]  [<ffffffff81d35733>] x86_64_start_kernel+0x143/0x152\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.398033] ---[ end trace 5060a87a20a9f2f9 ]---\r\nMay 27 07:23:04 werner-right-cortex kernel: [418489.398053] e1000e 0000:00:19.0 eth0: Reset adapter unexpectedly\r\nMay 27 07:23:04 werner-right-cortex NetworkManager[862]: <info> (eth0): carrier now OFF (device state 100, deferring action for 4 seconds)\r\nMay 27 07:23:08 werner-right-cortex NetworkManager[862]: <info> (eth0): carrier now ON (device state 100)\r\nMay 27 07:23:08 werner-right-cortex kernel: [418493.496031] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx\r\nMay 27 07:25:01 werner-right-cortex CRON[30892]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)\r\nMay 27 07:27:33 werner-right-cortex kernel: [418758.924164] nfs: server werner not responding, still trying\r\nMay 27 07:30:01 werner-right-cortex CRON[31859]: (root) CMD (start -q anacron || :)\r\nMay 27 07:30:01 werner-right-cortex anacron[31862]: Anacron 2.3 started on 2015-05-27\r\nMay 27 07:30:01 werner-right-cortex anacron[31862]: Will run job `cron.daily' in 5 min.\r\nMay 27 07:30:01 werner-right-cortex anacron[31862]: Jobs will be executed sequentially\r\nMay 27 07:33:21 werner-right-cortex chronyd[1146]: Can't synchronise: no reachable sources\r\n  ```\r\n  Could be that this is just the network being taken down together with `werner-left-cortex`. @ToMadoRe or @mzillich, what is the wiring on werner? Is the hub powered via `werner-left-cortex`? The logs above also indicate that `werner-right-cortex` suffered a kernel error on the network device, possibly because it got disconnected/powered off. That's all we know at the minute. "},"comment":{"url":"https://api.github.com/repos/strands-project/aaf_deployment/issues/comments/105900999","html_url":"https://github.com/strands-project/aaf_deployment/issues/263#issuecomment-105900999","issue_url":"https://api.github.com/repos/strands-project/aaf_deployment/issues/263","id":105900999,"user":{"login":"mzillich","id":4312487,"avatar_url":"https://avatars.githubusercontent.com/u/4312487?v=3","gravatar_id":"","url":"https://api.github.com/users/mzillich","html_url":"https://github.com/mzillich","followers_url":"https://api.github.com/users/mzillich/followers","following_url":"https://api.github.com/users/mzillich/following{/other_user}","gists_url":"https://api.github.com/users/mzillich/gists{/gist_id}","starred_url":"https://api.github.com/users/mzillich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mzillich/subscriptions","organizations_url":"https://api.github.com/users/mzillich/orgs","repos_url":"https://api.github.com/users/mzillich/repos","events_url":"https://api.github.com/users/mzillich/events{/privacy}","received_events_url":"https://api.github.com/users/mzillich/received_events","type":"User","site_admin":false},"created_at":"2015-05-27T13:10:56Z","updated_at":"2015-05-27T13:10:56Z","body":"I opened Werner now.\r\nwerner-left-cortex is still powered by the old power supply. that might well be a reason it died.\r\nthe network switch has its own power connector, leading to the 12 V from the EBC"}}