borg.phas.ubc.ca
142.103.236.30 (PUBLIC) - 192.168.1.30 (server)

Prodedure To In Place Remove & Replace Failed Drive

  1. Remove failed drive and insert new drive. Note S/N of new drive.
  2. Find device name for new drive. New drive serial number is WCC4M2645321, 
    # cd /dev/disk
# ls -l by-id | grep <drive S/N, eg. WCC4M2645321>:
lrwxrwxrwx 1 root root  9 Jun 30 11:22 ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M2645321 -> ../../sdu
lrwxrwxrwx 1 root root  9 Jun 30 11:22 scsi-SATA_WDC_WD20EFRX-68_WD-WCC4M2645321 -> ../../sdu
  3. #vim /etc/zfs/vdev_id.conf
    Change entry for relevant slot to point to the new device, eg.: 
    alias Bslot-2       /dev/disk/by-id/ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M2645321
  4. Run udevadm trigger 
    # udevadm trigger
  5. Run zpool replace commmand as follows: 
    # zpool replace -f tank Fslot-8

ata-Hitachi_HUA723020ALA641_YFHUNH5A
16-07-27 Fslot-12 - 10R errors
We are replacing Fslot-12 drive,
ST2000DM001-9YN164 2.0T S/N: W1E0EFSF with
HGST 2TB HUA723020ALA641, S/N: YFHUNH5A 
[root@borg disk]# zpool status
	NAME          STATE     READ WRITE CKSUM
	tank          ONLINE       0     0     0
	  raidz2-0    ONLINE       0     0     0
	  raidz2-1    ONLINE       0     0     0
	    Fslot-12  ONLINE      10     0     0

16-07-25 Fslot-15 - 7R,56w errors
We are replacing Fslot-15 drive,
ST2000DM001-9YN164 2.0T S/N: W1E0EFSF with
HGST 2TB HUA723020ALA641, S/N: YGHJS9EA 
[root@borg disk]# zpool status
	NAME          STATE     READ WRITE CKSUM
	tank          DEGRADED     0     0     0
	  raidz2-0    ONLINE       0     0     0
	  raidz2-1    DEGRADED     0     0     0

16-07-22 Fslot-12 - 2R errors - clear only 
[root@borg ~]# zpool status
  pool: tank
        NAME          STATE     READ WRITE CKSUM
        tank          ONLINE       0     0     0
          raidz2-1    ONLINE       0     0     0
            Fslot-12  ONLINE       2     0     0

16-06-13 Bslot-9 12R, 234W Errors - In Place Remove & Replace Original Drive
We are replacing Bslot-9 drive,
ST2000DM001-9YN1 2.0T S/N: W2405PN2 with
HGST 2TB HUA723020ALA641, S/N: YFHU0XEA 
scan: resilvered 1.15T in 26h36m with 0 errors on Tue Jun 14 15:03:35 2016

16-01-23 Bslot-9 15R Errors - zpool clear tank
Bslot-9 drive, ST2000DM001-9YN1 2.0T S/N: W2405PN2 showed some read errors. Decided to just clear the errors and see what happens with drive. 
zpool clear tank

15-10-30 Fslot-7 6R, 84W Errors - In Place Remove & Replace Original Drive
We are replacing Fslot-7 drive,
ST2000DM001-9YN1 2.0T S/N: S2407DVCY with
Hitachi 2TB HUA723020ALA641, S/N: YFHGPYVA 
 NAME             STATE     READ WRITE CKSUM
	tank             DEGRADED     0     0     0
	  raidz2-0       DEGRADED     0     0     0
	    replacing-1  DEGRADED     0     0     0
	      old        UNAVAIL      6    84     0  corrupted data
	      Fslot-7    ONLINE       0     0     0  (resilvering)

  scan: resilver in progress since Fri Oct 30 09:35:20 2015
    9.70G scanned out of 22.7T at 34.0M/s, 194h27m to go
    410M resilvered, 0.04% done

15-10-15 Bslot-3 17R,74W, 1CKSUM Errors - In Place Remove & Replace Original Drive
We are replacing Bslot-3 drive,
ST2000DM001-9YN1 2.0T S/N: W2F04YPY with
Hitachi 2TB HUA723020ALA641, S/N: YGGNNZPB 
   NAME             STATE     READ WRITE CKSUM
        tank             DEGRADED     0     0     0
          raidz2-2       DEGRADED     0     0     0
	    replacing-3  UNAVAIL      0     0     0
	      old        UNAVAIL     17    74     1  corrupted data
	      Bslot-3    ONLINE       0     0     0  (resilvering)

  scan: resilver in progress since Thu Oct 15 08:54:27 2015
    11.3G scanned out of 23.0T at 36.7M/s, 182h14m to go
    474M resilvered, 0.05% done

    Thu Oct 15 14:58:41 PDT 2015
    10.4T scanned out of 23.0T at 501M/s, 7h17m to go
    443G resilvered, 45.35% done

  scan: resilvered 966G in 20h27m with 0 errors on Fri Oct 16 05:21:46 2015

15-10-13 Bslot-4 1R Errors - In Place Remove & Replace Original Drive
We are replacing Bslot-4 drive,
ST2000DM001-9YN1 2.0T S/N: W24049ER with
Hitachi 2TB HUA723020ALA641, S/N: YGHRUXXA 
   NAME             STATE     READ WRITE CKSUM
	tank             DEGRADED     0     0     0
	  raidz2-2       DEGRADED     0     0     0
	    replacing-4  UNAVAIL      0     0     0
	      old        UNAVAIL      3    11     0  corrupted data
	      Bslot-4    ONLINE       0     0     0  (resilvering)
    3.91G scanned out of 22.8T at 20.7M/s, 321h2m to go
    163M resilvered, 0.02% done

  scan: resilver in progress since Tue Oct 13 09:58:36 2015
    14.2T scanned out of 22.8T at 182M/s, 13h43m to go
    603G resilvered, 62.44% done

  scan: resilvered 968G in 27h48m with 0 errors on Wed Oct 14 13:46:49 2015
  scan was slowed because Bslot-3 failed while resilvering was happening

15-09-23 Bslot-10 1R,14W Errors - In Place Remove & Replace Original Drive
We are replacing Bslot-10 drive,
ST2000DM001-9YN164 2.0T S/N: S2407EHJ with
WD Red 2TB, WD20EFRX-68EUZN0, S/N: WCC4M0ZYTXV4 
    1.13T scanned out of 22.3T at 245M/s, 25h11m to go
    47.8G resilvered, 5.08% done 
    2.79T scanned out of 22.3T at 251M/s, 22h38m to go
    118G resilvered, 12.51% done
    4.24T scanned out of 22.3T at 245M/s, 21h30m to go
    179G resilvered, 19.01% done
    17.6T scanned out of 22.3T at 226M/s, 6h1m to go
    750G resilvered, 79.03% done

15-08-04 Fslot-14 1R,5W Errors - In Place Remove & Replace Original Drive
We are replacing Fslot-14 drive,
ST2000DM001-9YN164 2.0T S/N: S2407EEZ with
WD Red 2TB, WD20EFRX-68EUZN0, S/N: WCC4M2PV99AX (reusing after fixed with SeaTools) 
=== zpool errors =====
[root@borg ~]# zpool status tank
  pool: tank
 state: ONLINE
        NAME          STATE     READ WRITE CKSUM
        tank          ONLINE       0     0     0
	  raidz2-1    DEGRADED     0     0     0
	    Fslot-14  UNAVAIL      1     5     0  corrupted data
After drive replacement: 
[root@borg disk]# zpool status tank
  pool: tank
 state: ONLINE
  scan: resilvered 739G in 15h23m with 0 errors on Wed Aug  5 03:33:02 2015

15-07-03 Bslot-10 4R,42W Errors - In Place Remove & Replace Original Drive
We are replacing Bslot-10 drive,
ST2000DM001-9YN164 2.0T S/N: W2F04YTV with
ST2000DM001-9YN164 2.0T S/N: S2407EHJ 
=== zpool errors =====
[root@borg ~]# zpool status tank
  pool: tank
 state: ONLINE
        NAME          STATE     READ WRITE CKSUM
        tank          ONLINE       0     0     0
	  raidz2-3    DEGRADED     0     0     0
	    Bslot-10  UNAVAIL      4    42     0  corrupted data
After drive replacement: 
[root@borg disk]# zpool status tank
  pool: tank
 state: ONLINE
  scan: resilvered 752G in 15h5m with 0 errors on Sat Jul  4 06:54:24 2015

15-06-10 Fslot-6 1 Read Errors - In Place Remove & Replace Original Drive We are replacing Fslot-6 drive,
ST2000DM001-9YN1 2.0T S/N: S24053NM with
WD Red 2TB, WD20EFRX-68EUZN0, S/N: WCC4M4YR3P5V 
=== zpool errors =====
[root@borg ~]# zpool status tank
  pool: tank
 state: ONLINE
        NAME          STATE     READ WRITE CKSUM
        tank          ONLINE       0     0     0
	  raidz2-0    ONLINE       0     0     0
	    Fslot-6   ONLINE       1     0     0
After drive replacement: 
[root@borg disk]# zpool status tank
  pool: tank
 state: ONLINE
  scan: resilvered 741G in 17h58m with 0 errors on Thu Jun 11 05:25:45 2015

15-06-03 Fslot-13 1 CKSUM Errors - In Place Remove & Replace Original Drive We are replacing Fslot-13 drive,
WD Red 2TB, WD20EFRX-68EUZN0, S/N: WCC4M4YR3HSU with
WD Red 2TB, WD20EFRX-68EUZN0, S/N: WCC4M1XHY05V 
=== zpool errors =====
[root@borg ~]# zpool status tank
  pool: tank
 state: ONLINE
        NAME          STATE     READ WRITE CKSUM
        tank          ONLINE       0     0     0
	  raidz2-1    ONLINE       0     0     0
	    Fslot-13  ONLINE       0     0     1
After drive replacement: 
[root@borg disk]# zpool status tank
  pool: tank
 state: ONLINE
  scan: resilvered 743G in 12h47m with 0 errors on Thu Jun  4 01:13:06 2015

15-06-02 Fslot-16 21 Read, 34 WRITE, 5 CKSUM Errors - In Place Remove & Replace Original Drive We are replacing Fslot-16 drive,
Seagate 2TB, ST2000DM001-9YN1, S/N: W1E0F20C with
WD Red 2TB, WD20EFRX-68EUZN0, S/N: WCC4M1XHYVS6 
=== zpool errors =====
[root@borg ~]# zpool status tank
  pool: tank
 state: ONLINE
        NAME          STATE     READ WRITE CKSUM
        tank          ONLINE       0     0     0
	  raidz2-1    ONLINE       0     0     0
	    Fslot-16  ONLINE      18     0     5
After drive replacement: 
[root@borg disk]# zpool status tank
  pool: tank
 state: ONLINE
  scan: resilvered 746G in 9h40m with 0 errors on Tue Jun  2 18:02:16 2015

15-06-01 Fslot-9 10 Read Errors - In Place Remove & Replace Original Drive We are replacing Fslot-9 drive,
Seagate 2TB, ST2000DM001-9YN164, S/N: S2407EGH (re-used drive) with
WD Red 2TB, WD20EFRX-68EUZN0, S/N: WCC4M4YR3HSU 
=== zpool errors =====
[root@borg ~]# zpool status tank
  pool: tank
 state: ONLINE
        NAME          STATE     READ WRITE CKSUM
	tank          ONLINE       0     0     0
	  raidz2-0    ONLINE       0     0     0
	    Fslot-9   ONLINE      10     0     0
After drive replacement: 
[root@borg disk]# zpool status tank
  pool: tank
 state: ONLINE
  scan: resilvered 742G in 12h6m with 0 errors on Mon Jun  1 23:32:57 2015

15-05-29 Fslot-13 7 Read, 4 write Errors - In Place Remove & Replace Original Drive We are replacing Fslot-13 drive,
Seagate 2TB, ST2000DM001-9YN1, S/N: S2407FB9 with
WD Red 2TB, WD20EFRX-68EUZN0, S/N: WCC4M4YR3HSU 
=== zpool errors =====
[root@borg ~]# zpool status tank
  pool: tank
 state: ONLINE
	NAME          STATE     READ WRITE CKSUM
	tank          DEGRADED     0     0     0
	  raidz2-1    DEGRADED     0     0     0
	    Fslot-12  ONLINE       0     0     0
	    Fslot-13  UNAVAIL      7     4     0  corrupted data
 After drive replacement: 
[root@borg disk]# zpool status tank
  pool: tank
 state: ONLINE
  scan: resilvered 748G in 14h55m with 0 errors on Sat May 30 06:27:26 2015

15-04-22 Fslot-6 11-write Errors - In Place Remove & Replace Original Drive We are replacing Fslot-6 drive,
Seagate 2TB, ST2000DM001-9YN1, S/N: S2407HK8 with
Seagate 2TB, ST2000DM001-9YN1, S/N: S24053NM (pulled because of read errors, but tested OK now) 
=== zpool errors =====
[root@borg ~]# zpool status tank
  pool: tank
 state: ONLINE
        NAME          STATE     READ WRITE CKSUM
        tank          DEGRADED     0     0     0
          raidz2-0    DEGRADED     0     0     0
            Fslot-6   UNAVAIL      0    11     0  corrupted data
 After drive replacement: 
[root@borg disk]# zpool status tank
  pool: tank
 state: ONLINE
  scan: resilvered 779G in 12h8m with 0 errors on Wed Apr 22 20:21:34 2015

15-03-19 Fslot-9 3-read/31-write Errors - In Place Remove & Replace Original Drive We are replacing Fslot-9 drive,
Seagate 2TB, ST2000DM001-9YN164, S/N: S2407F9R with
Seagate 2TB, ST2000DM001-9YN164, S/N: S2407EGH (pulled because of read errors, 15-02-27, but tested OK now) 
=== zpool errors =====
[root@borg ~]# zpool status tank
  pool: tank
 state: ONLINE
	NAME          STATE     READ WRITE CKSUM
	tank          DEGRADED     0     0     0
	  raidz2-0    DEGRADED     0     0     0
	    Fslot-9   UNAVAIL      8    31     0  corrupted data
 After drive replacement: 
[root@borg disk]# zpool status tank
  pool: tank
 state: ONLINE
  scan: resilvered 726G in 16h20m with 0 errors on Fri Mar 20 01:36:55 2015

15-02-27 Fslot-5 read Error - In Place Remove & Replace Original Drive We are replacing Bslot-11 drive,
Seagate 2TB, ST2000DM001-9YN1, S/N: S2407EGH with
Seagate 2TB, ST2000DM001-9YN1, S/N: S2407AFT

See procedure below. 
=== zpool errors =====
[root@borg ~]# zpool status tank
  pool: tank
 state: ONLINE
        NAME          STATE     READ WRITE CKSUM
        tank          ONLINE       0     0     0
	  raidz2-0    ONLINE       0     0     0
	    Fslot-11  ONLINE       2     0     0
 After drive replacement: 
[root@borg disk]# zpool status tank
  pool: tank
 state: ONLINE
  scan: resilvered 736G in 17h18m with 0 errors on Sat Feb 28 03:33:14 2015

15-01-19 Bslot-5 read Error - In Place Remove & Replace Original Drive We are replacing Bslot-5 drive,
Seagate 2TB, ST2000DM001-9YN1, S/N: S2407FRX with a new
WD Red drive, WD20EFRX-68EUZN0 2TB, S/N: WCC4M2LCJKTA 
=== zpool errors =====
[root@borg ~]# zpool status tank
  pool: tank
 state: ONLINE
        NAME          STATE     READ WRITE CKSUM
	tank          ONLINE       0     0     0
	  raidz2-2    ONLINE       0     0     0
	    Bslot-5   ONLINE       4     0     0
 After drive replacement: 
[root@borg disk]# zpool status tank
  pool: tank
 state: ONLINE
  scan: resilvered 667G in 14h48m with 0 errors on Tue Jan 20 02:41:11 2015

15-01-16 Bslot-0 corrupted data Error - In Place Remove & Replace Original Drive We are replacing Bslot-0 drive,
Seagate 2TB, ST2000DM001-9YN1, S/N: S2407FRX with a new
WD Red drive, WD20EFRX 2TB, S/N: WCC4M0DP9UKE 
=== zpool errors =====
[root@borg ~]# zpool status tank
  pool: tank
 state: DEGRADED
	NAME          STATE     READ WRITE CKSUM
	tank          DEGRADED     0     0     0
	  raidz2-2    DEGRADED     0     0     0
	    Bslot-0   UNAVAIL      1     1     0  corrupted data
 After drive replacement: 
[root@borg ~]# zpool status tank

15-01-05 Read Error - Bslot-8 - In Place Remove & Replace Original Drive We are replacing Bslot-8 drive,
Seagate 2TB, ST2000DM001-9YN1, S/N: S2407JEC with a new
WD Red drive, WD20EFRX 2TB, S/N: 68EUZN0 
=== zpool errors =====
  pool: tank
 state: ONLINE
  scan: scrub repaired 96K in 26h45m with 0 errors on Sat Jan  3 17:41:20 2015
	NAME          STATE     READ WRITE CKSUM
	tank          ONLINE       0     0     0
	  raidz2-3    ONLINE       0     0     0
	    Bslot-8   ONLINE      24     0     0
 After drive replacement: 
[root@borg ~]# zpool status tank
  pool: tank
 state: ONLINE
  scan: resilvered 656G in 17h38m with 0 errors on Tue Jan  6 10:02:19 2015

14-09-04 Read Error - Fslot-9 Fslot-9 single read error: 
[root@borg ~]# zpool status -x
  pool: tank
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: resilvered 632G in 12h52m with 0 errors on Wed Aug 27 21:52:54 2014
config:

	NAME          STATE     READ WRITE CKSUM
	tank          ONLINE       0     0     0
	  raidz2-0    ONLINE       0     0     0
	    Fslot-6   ONLINE       0     0     0
	    Fslot-7   ONLINE       0     0     0
	    Fslot-8   ONLINE       0     0     0
	    Fslot-9   ONLINE       1     0     0
 Since it is a single error, I will just clear it for now. rdp 
[root@borg ~]# zpool clear tank Fslot-9
[root@borg ~]# zpool status -x
all pools are healthy 
[root@borg ~]# grep Fslot-9 /etc/zfs/vdev_id.conf
alias Fslot-9       /dev/disk/by-id/ata-ST2000DM001-9YN164_S2407F9R

14-08-27 Failed Disk: In Place - Remove & Replace Original Drive We are replacing Fslot-17 drive, Seagate 2TB, ST2000DM001, S/N: S2407EHJ with Western Digital, WD20EFRX, 2TB, S/N: WCC4M3186107 
[root@borg ~]# zpool status
  pool: tank
 state: DEGRADED
	NAME          STATE     READ WRITE CKSUM
	tank          DEGRADED     0     0     0
	  raidz2-1    DEGRADED     0     0     0
	    Fslot-17  UNAVAIL     24    94     0  corrupted data

14-08-06 Failed Disk: In Place - Remove & Replace Original Drive 
[root@borg ~]# zfs list
NAME                    USED  AVAIL  REFER  MOUNTPOINT
tank                   9.89T  18.6T   336K  /tank
tank/BackupPC          8.03T  18.6T  8.03T  /tank/BackupPC
tank/OwnCloud          1.59G  18.6T  1.59G  /tank/OwnCloud
tank/SQLbackups        29.7G  18.6T  29.7G  /tank/SQLbackups
tank/ServerBackups     1.82T  18.6T  1.82T  /tank/ServerBackups
tmppool                7.31T  1.24T   243K  /tmppool
tmppool/BackupPC       1.87T  1.24T  1.87T  /tmppool/BackupPC
tmppool/ServerBackups  5.44T  1.24T  5.44T  /tmppool/ServerBackups
[root@borg ~]# zpool status
  pool: tank
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
	invalid.  Sufficient replicas exist for the pool to continue
	functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: resilvered 584G in 11h17m with 0 errors on Tue Jul  8 23:10:49 2014
config:

	NAME          STATE     READ WRITE CKSUM
	tank          DEGRADED     0     0     0
	  raidz2-0    ONLINE       0     0     0
	  raidz2-1    ONLINE       0     0     0
	  raidz2-2    ONLINE       0     0     0
	  raidz2-3    DEGRADED     0     0     0
	    Bslot-6   UNAVAIL      5    11     0  corrupted data
We are replacing Bslot-6 drive, Seagate 2TB, ST2000DM001-9YN1, S/N: 9YN164_S2407F0T with Western Digital, WD20EFRX, 2TB, S/N: WCC4M0935377

14-07-08 Failed Disk: In Place - Remove & Replace Original Drive
We are replacing Fslot-10 drive, Seagate 2TB, ST2000DM001-9YN1, S/N: 9YN164_S2407EGL
with Western Digital, WD20EFRX, 2TB, S/N: WCC4M1051126

14-07-02 Failed Disk: In Place - Remove & Replace Original Drive
We are replacing Fslot-17 drive, Seagate 2TB, ST2000DM001-9YN1, S/N: W240EGB6
with drive from Fslot-5, Seagate 2TB, ST2000DM001-9YN1, S/N: S2407EHJ

Fslot-5 is now empty.

14-06-28 Failed Disk: In Place - Remove & Replace Original Drive We are replacing Fslot-8 drive, Seagate 2TB, ST2000DM001-9YN1, S/N: S2407CBF with Western Digital, WD20EFRX, 2TB, S/N: WCC4M2645321
  1. Remove original drive and insert new drive. Note S/N of new drive.
  2. Find device name for new drive. New drive serial number is WCC4M2645321, 
       [root@borg zfs]# cd /dev/disk
   [root@borg disk]# ls -l by-id | grep WCC4M2645321
   lrwxrwxrwx 1 root root  9 Jun 30 11:22 ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M2645321 -> ../../sdu
   lrwxrwxrwx 1 root root  9 Jun 30 11:22 scsi-SATA_WDC_WD20EFRX-68_WD-WCC4M2645321 -> ../../sdu
  3. Edit /etc/zfs/vdev_id.conf
    Change entry for Fslot-8 to point to new device 
    alias Bslot-2       /dev/disk/by-id/ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M2645321
  4. Run udevadm trigger 
    [root@borg disk]# udevadm trigger
  5. Run zpool replace commmand as follows: 
    [root@borg disk]# zpool replace -f tank Fslot-8
[root@borg disk]# zpool status tank
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Mon Jun 30 11:27:44 2014
    227M scanned out of 13.9T at 22.7M/s, 178h14m to go
    8.32M resilvered, 0.00% done
config:

	NAME             STATE     READ WRITE CKSUM
	tank             DEGRADED     0     0     0
	  raidz2-0       DEGRADED     0     0     0
	    Fslot-6      ONLINE       0     0     0
	    Fslot-7      ONLINE       0     0     0
	    replacing-2  UNAVAIL      0     0     0
	      old        UNAVAIL      1     1     0  corrupted data
	      Fslot-8    ONLINE       0     0     0  (resilvering)
	    Fslot-9      ONLINE       0     0     0

14-06-05
Disk read+write errors - Bslot-2
[root@borg ~]# zpool status -x
  pool: tank
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: scrub repaired 0 in 12h16m with 0 errors on Thu Mar 27 02:39:16 2014
config:

	NAME          STATE     READ WRITE CKSUM
	tank          DEGRADED     0     0     0
	  raidz2-0    ONLINE       0     0     0
	  raidz2-1    ONLINE       0     0     0
	  raidz2-2    DEGRADED     0     0     0
	    Bslot-0   ONLINE       4     0     0
	    Bslot-1   ONLINE       0     0     0
	    Bslot-2   OFFLINE      8   263     0

14-06-09 Failed Disk: In Place - Remove & Replace Original Drive

We are replacing Bslot-2 drive, Seagate 2TB, ST2000DM001-9YN1, S/N: S2407DC9, RevLev: CC4C with Western Digital, WD20EFRX, 2TB, S/N: WCC4M2643642
  1. Remove original drive and insert new drive. Note S/N of new drive.
  2. Find device name for new drive. New drive serial number is WCC4M2643642, 
       [root@zfszap zfs]# cd /dev/disk
   [root@zfszap disk]# ls -l by-id | grep WCC4M2643642
   lrwxrwxrwx 1 root root  9 Jun  4 10:44 ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M2643642 -> ../../sdc
   lrwxrwxrwx 1 root root  9 Jun  4 10:44 scsi-SATA_WDC_WD20EFRX-68_WD-WCC4M2643642 -> ../../sdc
  3. Edit /etc/zfs/vdev_id.conf
    Change entry for Bslot-2 to point to new device 
    alias Bslot-2       /dev/disk/by-id/ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M2643642
  4. Run udevadm trigger 
    [root@zfszap disk]# udevadm trigger
  5. Run zpool replace commmand as follows: 
    [root@borg disk]# zpool replace tank Bslot-2
        invalid vdev specification
        use '-f' to override the following errors:
        /dev/disk/by-vdev/Bslot-2 does not contain an EFI label but it may contain partition
        information in the MBR.
        [root@borg disk]# zpool replace -f tank Bslot-2
[root@borg disk]# zpool status tank
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Jun  5 15:11:10 2014
    2.24G scanned out of 13.9T at 21.4M/s, 189h7m to go
    97.0M resilvered, 0.02% done
config:

	NAME             STATE     READ WRITE CKSUM
	tank             DEGRADED     0     0     0
	  raidz2-0       ONLINE       0     0     0
	    Fslot-6      ONLINE       0     0     0
	    Fslot-7      ONLINE       0     0     0
	    Fslot-8      ONLINE       0     0     0
	    Fslot-9      ONLINE       0     0     0
	    Fslot-10     ONLINE       0     0     0
	    Fslot-11     ONLINE       0     0     0
	  raidz2-1       ONLINE       0     0     0
	    Fslot-12     ONLINE       0     0     0
	    Fslot-13     ONLINE       0     0     0
	    Fslot-14     ONLINE       0     0     0
	    Fslot-15     ONLINE       0     0     0
	    Fslot-16     ONLINE       0     0     0
	    Fslot-17     ONLINE       0     0     0
	  raidz2-2       DEGRADED     0     0     0
	    Bslot-0      ONLINE       4     0     0
	    Bslot-1      ONLINE       0     0     0
	    replacing-2  OFFLINE      0     0     0
	      old        OFFLINE      8   263     0
	      Bslot-2    ONLINE       0     0     0  (resilvering)
	    Bslot-3      ONLINE       0     0     0
	    Bslot-4      ONLINE       0     0     0
	    Bslot-5      ONLINE       0     0     0
	  raidz2-3       ONLINE       0     0     0
	    Bslot-6      ONLINE       0     0     0
	    Bslot-7      ONLINE       0     0     0
	    Bslot-8      ONLINE       0     0     0
	    Bslot-9      ONLINE       0     0     0
	    Bslot-10     ONLINE       0     0     0
	    Bslot-11     ONLINE       0     0     0
	logs
	  mirror-4       ONLINE       0     0     0
	    Fslot-2      ONLINE       0     0     0
	    Fslot-3      ONLINE       0     0     0
	cache
	  Fslot-4        ONLINE       0     0     0

errors: No known data errors
After re-silvering has finished: 
[root@borg disk]# zpool status tank
  pool: tank
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: resilvered 586G in 9h58m with 0 errors on Fri Jun  6 01:09:59 2014
config:

	NAME          STATE     READ WRITE CKSUM
	tank          ONLINE       0     0     0
	  raidz2-0    ONLINE       0     0     0
	    Fslot-6   ONLINE       0     0     0
	    Fslot-7   ONLINE       0     0     0
	    Fslot-8   ONLINE       0     0     0
	    Fslot-9   ONLINE       0     0     0
	    Fslot-10  ONLINE       0     0     0
	    Fslot-11  ONLINE       0     0     0
	  raidz2-1    ONLINE       0     0     0
	    Fslot-12  ONLINE       0     0     0
	    Fslot-13  ONLINE       0     0     0
	    Fslot-14  ONLINE       0     0     0
	    Fslot-15  ONLINE       0     0     0
	    Fslot-16  ONLINE       0     0     0
	    Fslot-17  ONLINE       0     0     0
	  raidz2-2    ONLINE       0     0     0
	    Bslot-0   ONLINE       4     0     0
	    Bslot-1   ONLINE       0     0     0
	    Bslot-2   ONLINE       0     0     0
	    Bslot-3   ONLINE       0     0     0
	    Bslot-4   ONLINE       0     0     0
	    Bslot-5   ONLINE       0     0     0
	  raidz2-3    ONLINE       0     0     0
	    Bslot-6   ONLINE       0     0     0
	    Bslot-7   ONLINE       0     0     0
	    Bslot-8   ONLINE       0     0     0
	    Bslot-9   ONLINE       0     0     0
	    Bslot-10  ONLINE       0     0     0
	    Bslot-11  ONLINE       0     0     0
	logs
	  mirror-4    ONLINE       0     0     0
	    Fslot-2   ONLINE       0     0     0
	    Fslot-3   ONLINE       0     0     0
	cache
	  Fslot-4     ONLINE       0     0     0
14-05-27
Disk read errors - Bslot-0
[root@borg ~]# zpool status -x
  pool: tank
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: scrub repaired 0 in 12h16m with 0 errors on Thu Mar 27 02:39:16 2014
config:

	NAME          STATE     READ WRITE CKSUM
	tank          ONLINE       0     0     0
	  raidz2-0    ONLINE       0     0     0
	  raidz2-1    ONLINE       0     0     0
	  raidz2-2    ONLINE       0     0     0
	    Bslot-0   ONLINE       4     0     0
	    Bslot-1   ONLINE       0     0     0
	  raidz2-3    ONLINE       0     0     0
	logs
	  mirror-4    ONLINE       0     0     0
	cache
	  Fslot-4     ONLINE       0     0     0

errors: No known data errors
Since this just occurred, I will monitor it a few days and see if more errors appear. If not, I will use the "zpool clear tank Bslot-0" command to clear them. Otherwise I will have to replace the disk.
See ZFS-8000-9P_ErrorMsg.html
13-06-18
Disk error - Bslot-4 unavailable
zpool offline tank Bslot-4
<replaced disk> (actually just removed and re-inserted the same disk)
zpool online tank Bslot-4
re-silver started immediately. After it was finished (very short time), I ran a scrub: 
zpool scrub tank
13-05-09
Configuring Disks
Create /etc/zfs/vdev_id.conf: 
#     by-vdev
#     name     fully qualified or base name of device link
#     
#     FRONT, left column, bottom to top
alias a0       /dev/disk/by-id/ata-WDC_WD20EFRX-68AX9N0_WD-WMC300674176
alias a1       /dev/disk/by-id/ata-WDC_WD20EFRX-68AX9N0_WD-WMC301574832
alias a2       /dev/disk/by-id/ata-KINGSTON_SVP200S360G_50026B72240C56EC
alias a3       /dev/disk/by-id/ata-KINGSTON_SVP200S360G_50026B72240C58F0
alias a4       /dev/disk/by-id/ata-KINGSTON_SVP200S360G_50026B72240C555D
alias a5       /dev/disk/by-id/ata-ST2000DM001-9YN164_S2407EHJ
#
#     FRONT, 2nd column from left, bottom to top
alias a6       /dev/disk/by-id/ata-ST2000DM001-9YN164_S2407HK8
alias a7       /dev/disk/by-id/ata-ST2000DM001-9YN164_S2407DVC
alias a8       /dev/disk/by-id/ata-ST2000DM001-9YN164_S2407CBF
alias a9       /dev/disk/by-id/ata-ST2000DM001-9YN164_S2407F9R
alias a10      /dev/disk/by-id/ata-ST2000DM001-9YN164_S2407EGL
alias a11      /dev/disk/by-id/ata-ST2000DM001-9YN164_S2407EGH
#
#     FRONT, 3rd column from left, bottom to top
alias a12       /dev/disk/by-id/ata-ST2000DM001-9YN164_S2407FR3
alias a13       /dev/disk/by-id/ata-ST2000DM001-9YN164_S2407FB9
alias a14       /dev/disk/by-id/ata-ST2000DM001-9YN164_S2407EEZ
alias a15       /dev/disk/by-id/ata-ST2000DM001-9YN164_W1E0EFSF
alias a16       /dev/disk/by-id/ata-ST2000DM001-9YN164_W1E0F20C
alias a17       /dev/disk/by-id/ata-ST2000DM001-9YN164_W240EGB6
#
#     FRONT, 4th column from left, bottom to top
alias a18       /dev/disk/by-id/ata-WDC_WD20EFRX-68AX9N0_WD-WCC1T0364107
alias a19       /dev/disk/by-id/ata-WDC_WD20EFRX-68AX9N0_WD-WCC1T0369634
alias a20       /dev/disk/by-id/ata-WDC_WD20EFRX-68AX9N0_WD-WCC1T0374231
alias a21       /dev/disk/by-id/ata-WDC_WD20EFRX-68AX9N0_WD-WCC1T0364751
alias a22       /dev/disk/by-id/ata-WDC_WD20EFRX-68AX9N0_WD-WCC1T0388735
alias a23       /dev/disk/by-id/ata-WDC_WD20EFRX-68AX9N0_WD-WCC1T0373794
#
#     BACK, left column, bottom to top
alias b0       /dev/disk/by-id/ata-ST2000DM001-9YN164_S2407FRX
alias b1       /dev/disk/by-id/ata-ST2000DM001-9YN164_S2F05MTT
alias b2       /dev/disk/by-id/ata-ST2000DM001-9YN164_S2407DC9
#
#     BACK, 2nd column from left, bottom to top
alias b3       /dev/disk/by-id/ata-ST2000DM001-9YN164_W2F04YPY
alias b4       /dev/disk/by-id/ata-ST2000DM001-9YN164_W24049ER
alias b5       /dev/disk/by-id/ata-ST2000DM001-9YN164_W24053NM
#
#     BACK, 3rd column from left, bottom to top
alias b6       /dev/disk/by-id/ata-ST2000DM001-9YN164_S2407F0T
alias b7       /dev/disk/by-id/ata-ST2000DM001-9YN164_S2407CFC
alias b8       /dev/disk/by-id/ata-ST2000DM001-9YN164_S2407JEC
#
#     BACK, 4th column from left, bottom to top
alias b9       /dev/disk/by-id/ata-ST2000DM001-9YN164_W2405PN2
alias b10       /dev/disk/by-id/ata-ST2000DM001-9YN164_W2F04YTV
alias b11       /dev/disk/by-id/ata-ST2000DM001-9YN164_W1E0502S
Run udevadm trigger

13-05-22 zpool history (re-created tank)
2013-05-22.12:23:46 zpool create -f -o ashift=12 tank raidz2 Fslot-6 Fslot-7 Fslot-8 Fslot-9 Fslot-10 Fslot-11 raidz2 Fslot-12 Fslot-13 Fslot-14 Fslot-15 Fslot-16 Fslot-17 raidz2 Bslot-0 Bslot-1 Bslot-2 Bslot-3 Bslot-4 Bslot-5 raidz2 Bslot-6 Bslot-7 Bslot-8 Bslot-9 Bslot-10 Bslot-11 log mirror Fslot-2 Fslot-3 cache Fslot-4

2013-05-22.12:40:13 zfs create tank/BackupPC