{"action":"created","issue":{"url":"https://api.github.com/repos/NixOS/nixpkgs/issues/11276","labels_url":"https://api.github.com/repos/NixOS/nixpkgs/issues/11276/labels{/name}","comments_url":"https://api.github.com/repos/NixOS/nixpkgs/issues/11276/comments","events_url":"https://api.github.com/repos/NixOS/nixpkgs/issues/11276/events","html_url":"https://github.com/NixOS/nixpkgs/issues/11276","id":119012842,"number":11276,"title":"File System / SSD Failures on Lenovo X250 (not kernel, not hardware)","user":{"login":"aaronlevin","id":1580060,"avatar_url":"https://avatars.githubusercontent.com/u/1580060?v=3","gravatar_id":"","url":"https://api.github.com/users/aaronlevin","html_url":"https://github.com/aaronlevin","followers_url":"https://api.github.com/users/aaronlevin/followers","following_url":"https://api.github.com/users/aaronlevin/following{/other_user}","gists_url":"https://api.github.com/users/aaronlevin/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronlevin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronlevin/subscriptions","organizations_url":"https://api.github.com/users/aaronlevin/orgs","repos_url":"https://api.github.com/users/aaronlevin/repos","events_url":"https://api.github.com/users/aaronlevin/events{/privacy}","received_events_url":"https://api.github.com/users/aaronlevin/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/NixOS/nixpkgs/labels/0.kind:%20bug","name":"0.kind: bug","color":"84b6eb"},{"url":"https://api.github.com/repos/NixOS/nixpkgs/labels/6.topic:%20nixos","name":"6.topic: nixos","color":"fef2c0"},{"url":"https://api.github.com/repos/NixOS/nixpkgs/labels/9.needs:%20feedback","name":"9.needs: feedback","color":"eb6420"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":30,"created_at":"2015-11-26T10:17:33Z","updated_at":"2015-12-01T20:58:46Z","closed_at":null,"body":"I've been experiencing very frequent file system / ssd failures on my Lenovo X250. I've experienced this problem:\r\n\r\n- on two separate, brand new Lenovo x250s\r\n- had a Lenovo x250 working with Debian (Jesse) with no issues for months (same ssd, same firmware)\r\n- across several kernels: `3.18`, `4.0`, `4.1`, `4.2`, and `4.3`\r\n- with or without `TRIM` enabled (and all the `fstab` incanations: `noatime,nodirtime,discards`, etc.\r\n\r\nI am fairly certain this is a configuration issue. Specifically, NixOS likely isn't installing the right modules or setting the right configuration or having an option to grab the right firmware?\r\n\r\nI'm really desperate to fix this problem so I can use NixOS at work.\r\n\r\n## The Issue\r\n\r\nIt is hard to re-create and hard to debug. There is a `kernel` failure, the file system will hang, and get remounted in `read-only` mode (so I can't grab logs). The error happens intermittently. The longest period I've gone without experiencing the issue is a day and a half, but I've experienced it immediately on reboot a few times. It seems to happen more frequently after suspend, but not exclusively (I hit it one right after a reboot - I also hit it once *during* `systemctl rescue`.\r\n\r\nI have taken photos of the failure on my cell phone (sometimes I can catch logs while tailing `dmesg` when the error happens).\r\n\r\nfrom `dmesg` (I'm writing out what I see on my photo):\r\n\r\n```\r\nata1.00: exception Emask 0x0 SAct 0x1e0000 SErr 0x50000 action 0x6 frozen\r\nata1: SError: { PHYRdyChg CommWake}\r\nata1.00: failed command: WRITE FPDMA QUEUED\r\nata1.00: cmd 61/08:00:18:ee:7a/00:00:02:00:00/40 tag 0 ncq .... (and on)\r\nata1.00: status: { DRDY }\r\n...\r\n(and repeat)\r\n...\r\nata1: hard resetting link\r\nata1: SATA link down (SStatus 0 SControl 300)\r\n...\r\n(repeat)\r\n...\r\nata1.00: disabled\r\nata1.00: device reported invalid CHS sector 0\r\nsd 0:0:0:0: [sda] tag#21 FAILED Result: hostbye=DID_OK drivebyte=DRIVER_SENSE\r\nsd 0:0:0:0: [sda] tag#21 Sense Key : Illegal Request [current] [descriptor]\r\nsd 0:0:0:0: [sda] tag#21 AddI Sense: Unaligned write command\r\nsd 0:0:0:0: [sda] tag#21 CDB: Write(10) 2a 00 0f 64 4c 78 00 00 10 00\r\nblk_update_request: I/O error, dev sda, sector 258231416\r\n...\r\n(and repeat with increasing tag # and new blk_update sector errors)\r\n...\r\nEXT4-fs warning (device dm-2): ext4_end_bio:332: I/O error -5 writing to inode 13389888 (offset 0 size 28672)\r\nBuffer I/O Error on device dm-2, logical block 58770479\r\n...\r\n(repeat buffer i/o errors)\r\n...\r\nblk_update_request (as above)\r\nsd 0:0:0:0: rejecting I/O to offline device\r\n...\r\nata1: BH  complete\r\n...\r\nAbortin journal on device dm-2-8\r\nEXT4-fs (dm-2): Delayed block allocation failed for inode 14600222 at inode ...\r\nEXT4-fs (dm-2): This should not happen!! Data will be lost\r\n...\r\n(and on)\r\n```\r\n\r\n## My Setup\r\n\r\n- Lenovo X250\r\n- `luks` encrypted drive. (setup following [these instructions](http://bluishcoder.co.nz/2014/05/14/installing-nixos-with-encrypted-root-on-thinkpad-w540.html))\r\n\r\n```\r\n☭ uname -a\r\nLinux nixos 4.3.0 #1-NixOS SMP Thu Jan 1 00:00:01 UTC 1970 x86_64 GNU/Linux\r\n```\r\n```\r\n☭ nixos-version \r\n15.09.706.45128de (Dingo)\r\n```\r\n```\r\n☭ lspci\r\n00:00.0 Host bridge: Intel Corporation Broadwell-U Host Bridge -OPI (rev 09)\r\n00:02.0 VGA compatible controller: Intel Corporation Broadwell-U Integrated Graphics (rev 09)\r\n00:03.0 Audio device: Intel Corporation Broadwell-U Audio Controller (rev 09)\r\n00:14.0 USB controller: Intel Corporation Wildcat Point-LP USB xHCI Controller (rev 03)\r\n00:16.0 Communication controller: Intel Corporation Wildcat Point-LP MEI Controller #1 (rev 03)\r\n00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (3) I218-LM (rev 03)\r\n00:1b.0 Audio device: Intel Corporation Wildcat Point-LP High Definition Audio Controller (rev 03)\r\n00:1c.0 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #6 (rev e3)\r\n00:1c.1 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #3 (rev e3)\r\n00:1d.0 USB controller: Intel Corporation Wildcat Point-LP USB EHCI Controller (rev 03)\r\n00:1f.0 ISA bridge: Intel Corporation Wildcat Point-LP LPC Controller (rev 03)\r\n00:1f.2 SATA controller: Intel Corporation Wildcat Point-LP SATA Controller [AHCI Mode] (rev 03)\r\n00:1f.3 SMBus: Intel Corporation Wildcat Point-LP SMBus Controller (rev 03)\r\n00:1f.6 Signal processing controller: Intel Corporation Wildcat Point-LP Thermal Management Controller (rev 03)\r\n02:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5227 PCI Express Card Reader (rev 01)\r\n03:00.0 Network controller: Intel Corporation Wireless 7265 (rev 59)\r\n```\r\n\r\n```\r\n☭ sudo smartctl -a /dev/sda\r\nsmartctl 6.3 2014-07-26 r3976 [x86_64-linux-4.3.0] (local build)\r\nCopyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org\r\n\r\n=== START OF INFORMATION SECTION ===\r\nDevice Model:     SAMSUNG MZ7LN256HCHP-000L7\r\nSerial Number:    S20HNXBG772065\r\nLU WWN Device Id: 5 002538 d00000000\r\nFirmware Version: EMT03L6Q\r\nUser Capacity:    256,060,514,304 bytes [256 GB]\r\nSector Size:      512 bytes logical/physical\r\nRotation Rate:    Solid State Device\r\nDevice is:        Not in smartctl database [for details use: -P showall]\r\nATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4c\r\nSATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)\r\nLocal Time is:    Thu Nov 26 11:02:36 2015 CET\r\nSMART support is: Available - device has SMART capability.\r\nSMART support is: Enabled\r\n\r\n=== START OF READ SMART DATA SECTION ===\r\nSMART overall-health self-assessment test result: PASSED\r\n\r\nGeneral SMART Values:\r\nOffline data collection status:  (0x00)\tOffline data collection activity\r\n\t\t\t\t\twas never started.\r\n\t\t\t\t\tAuto Offline Data Collection: Disabled.\r\nSelf-test execution status:      (   0)\tThe previous self-test routine completed\r\n\t\t\t\t\twithout error or no self-test has ever \r\n\t\t\t\t\tbeen run.\r\nTotal time to complete Offline \r\ndata collection: \t\t(    0) seconds.\r\nOffline data collection\r\ncapabilities: \t\t\t (0x53) SMART execute Offline immediate.\r\n\t\t\t\t\tAuto Offline data collection on/off support.\r\n\t\t\t\t\tSuspend Offline collection upon new\r\n\t\t\t\t\tcommand.\r\n\t\t\t\t\tNo Offline surface scan supported.\r\n\t\t\t\t\tSelf-test supported.\r\n\t\t\t\t\tNo Conveyance Self-test supported.\r\n\t\t\t\t\tSelective Self-test supported.\r\nSMART capabilities:            (0x0003)\tSaves SMART data before entering\r\n\t\t\t\t\tpower-saving mode.\r\n\t\t\t\t\tSupports SMART auto save timer.\r\nError logging capability:        (0x01)\tError logging supported.\r\n\t\t\t\t\tGeneral Purpose Logging supported.\r\nShort self-test routine \r\nrecommended polling time: \t (   2) minutes.\r\nExtended self-test routine\r\nrecommended polling time: \t ( 133) minutes.\r\nSCT capabilities: \t       (0x003d)\tSCT Status supported.\r\n\t\t\t\t\tSCT Error Recovery Control supported.\r\n\t\t\t\t\tSCT Feature Control supported.\r\n\t\t\t\t\tSCT Data Table supported.\r\n\r\nSMART Attributes Data Structure revision number: 1\r\nVendor Specific SMART Attributes with Thresholds:\r\nID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE\r\n  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0\r\n  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       76\r\n 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       151\r\n170 Unknown_Attribute       0x0032   100   100   010    Old_age   Always       -       0\r\n171 Unknown_Attribute       0x0032   100   100   010    Old_age   Always       -       0\r\n172 Unknown_Attribute       0x0032   100   100   010    Old_age   Always       -       0\r\n173 Unknown_Attribute       0x0033   099   099   005    Pre-fail  Always       -       1\r\n174 Unknown_Attribute       0x0032   099   099   000    Old_age   Always       -       38\r\n178 Used_Rsvd_Blk_Cnt_Chip  0x0013   100   100   010    Pre-fail  Always       -       0\r\n180 Unused_Rsvd_Blk_Cnt_Tot 0x0013   100   100   010    Pre-fail  Always       -       773\r\n184 End-to-End_Error        0x0033   100   100   097    Pre-fail  Always       -       0\r\n187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0\r\n194 Temperature_Celsius     0x0032   077   056   000    Old_age   Always       -       23\r\n199 UDMA_CRC_Error_Count    0x003e   100   100   000    Old_age   Always       -       0\r\n233 Media_Wearout_Indicator 0x0013   099   099   000    Pre-fail  Always       -       16768826\r\n241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       194\r\n242 Total_LBAs_Read         0x0032   099   099   000    Old_age   Always       -       823\r\n249 Unknown_Attribute       0x0032   099   099   000    Old_age   Always       -       256\r\n\r\nSMART Error Log Version: 1\r\nNo Errors Logged\r\n\r\nSMART Self-test log structure revision number 1\r\nNo self-tests have been logged.  [To run self-tests, use: smartctl -t]\r\n\r\nWarning! SMART Selective Self-Test Log Structure error: invalid SMART checksum.\r\nSMART Selective self-test log data structure revision number 1\r\n SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS\r\n    1        0        0  Not_testing\r\n    2        0        0  Not_testing\r\n    3        0        0  Not_testing\r\n    4        0        0  Not_testing\r\n    5        0        0  Not_testing\r\n  255        0    65535  Read_scanning was never started\r\nSelective self-test flags (0x0):\r\n  After scanning selected spans, do NOT read-scan remainder of disk.\r\nIf Selective self-test is pending on power-up, resume after 0 minute delay.\r\n```"},"comment":{"url":"https://api.github.com/repos/NixOS/nixpkgs/issues/comments/161093881","html_url":"https://github.com/NixOS/nixpkgs/issues/11276#issuecomment-161093881","issue_url":"https://api.github.com/repos/NixOS/nixpkgs/issues/11276","id":161093881,"user":{"login":"mbakke","id":1269099,"avatar_url":"https://avatars.githubusercontent.com/u/1269099?v=3","gravatar_id":"","url":"https://api.github.com/users/mbakke","html_url":"https://github.com/mbakke","followers_url":"https://api.github.com/users/mbakke/followers","following_url":"https://api.github.com/users/mbakke/following{/other_user}","gists_url":"https://api.github.com/users/mbakke/gists{/gist_id}","starred_url":"https://api.github.com/users/mbakke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mbakke/subscriptions","organizations_url":"https://api.github.com/users/mbakke/orgs","repos_url":"https://api.github.com/users/mbakke/repos","events_url":"https://api.github.com/users/mbakke/events{/privacy}","received_events_url":"https://api.github.com/users/mbakke/received_events","type":"User","site_admin":false},"created_at":"2015-12-01T20:58:46Z","updated_at":"2015-12-01T20:58:46Z","body":"You can put `libata.force` in either kernel command line or as a module parameter, it should be picked up either way. The libata module is loaded automatically (but it's arguably more nix-y to specify it)."}}