mirgserver.phas.ubc.ca 142.103.239.79 |
| Hardware: | Supermicro 4U 4way 64 Core Server |
| Serial Number: | U34970 |
| Processor: | AMD Opteron(TM) Processor 6274 |
| Installed Memory (RAM): | 128 GB |
| System Type: | 64-bit Scientific Linux 6.4 |
| Hard Drive: | 1TB |
| Computer name: | mirgserver |
| Public IP Address & MAC Address: | 142.103.239.79 (00:30:48:FF:DC:9E) |
| Subnet Mask: | 255.255.255.0 |
| Default Gateway: | 142.103.239.254 |
| DNS Servers: | 137.82.1.2, 142.103.1.42 |
| Second IP Address & MAC Address: |
NIS MATLAB license server MATLAB R2013b GATE
Switch /home.mirg back to RAID disks
[root@mirgserver ~]$ cat /etc/fstab # # /etc/fstab # Created by anaconda on Fri Apr 5 09:31:38 2013 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # UUID=cd85bf62-2fe4-49e0-8c9d-762e74f6b1ad / ext4 defaults 1 1 UUID=5b027788-9e1d-4acf-8577-9e0708f5eba0 /boot ext4 defaults 1 2 UUID=d9107fea-c328-4100-bbc1-ac0654df9742 swap swap defaults 0 0 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 /dev/md127 /home.mirg ext4 defaults 1 2 #/dev/mapper/vg_zorok-lv_home /home.mirg ext4 defaults 1 2
MATLAB couldn't start properly
Check /etc/lmboot_TMW, which was still linked to R2013a, so recreated the symbolic link as follows:
[root@mirgserver ~]$ ln -s $MATLAB/etc/lmboot /etc/lmboot_TMW [root@mirgserver ~]$ ln -s $MATLAB/etc/lmdown /etc/lmdown_TMW [root@mirgserver ~]$ ls -l /etc/lm* lrwxrwxrwx. 1 root root 29 Jan 21 08:37 /etc/lmboot_TMW -> /opt/matlab/R2013b/etc/lmboot lrwxrwxrwx. 1 root root 29 Jan 21 08:37 /etc/lmdown_TMW -> /opt/matlab/R2013b/etc/lmdown
Add new user to nis server
[root@mirgserver ~]$ useradd -d /home.mirg/shy -u 538 -U shy [root@mirgserver ~]$ passwd shy [root@mirgserver /]$ cd /var/yp [root@mirgserver yp]$ make gmake[1]: Entering directory `/var/yp/MIRG' gmake[1]: `ypservers' is up to date. gmake[1]: Leaving directory `/var/yp/MIRG' gmake[1]: Entering directory `/var/yp/MIRG' Updating passwd.byname... Updating passwd.byuid... Updating group.byname... Updating group.bygid... Updating netid.byname... gmake[1]: Leaving directory `/var/yp/MIRG' [root@mirgserver yp]$ service ypserv restart Stopping YP server services: [ OK ] Starting YP server services: [ OK ] [root@mirgserver yp]$ ypmatch shy passwd shy:$6$k4RKZWlF$e5RUcGSoh0eMmELA4OfU/IpZeeO4v5oVbnmBU6.1RZ9XqlODhGQqi3jEqAx2/V/1AWA1LIAQWgFO6fm4UJsl.1:538:538::/home.mirg/shy:/bin/bash
Remove user from nis server
[root@mirgserver ~]$ userdel shy [root@mirgserver ~]$ grep shy /etc/passwd [root@mirgserver ~]$ ypmatch shy passwd shy:$6$k4RKZWlF$e5RUcGSoh0eMmELA4OfU/IpZeeO4v5oVbnmBU6.1RZ9XqlODhGQqi3jEqAx2/V/1AWA1LIAQWgFO6fm4UJsl.1:538:538::/home.mirg/shy:/bin/bash [root@mirgserver ~]$ cd /var/yp [root@mirgserver yp]$ make gmake[1]: Entering directory `/var/yp/MIRG' gmake[1]: `ypservers' is up to date. gmake[1]: Leaving directory `/var/yp/MIRG' gmake[1]: Entering directory `/var/yp/MIRG' Updating passwd.byname... Updating passwd.byuid... Updating group.byname... Updating group.bygid... Updating netid.byname... gmake[1]: Leaving directory `/var/yp/MIRG' [root@mirgserver yp]$ ypmatch shy passwd shy:$6$k4RKZWlF$e5RUcGSoh0eMmELA4OfU/IpZeeO4v5oVbnmBU6.1RZ9XqlODhGQqi3jEqAx2/V/1AWA1LIAQWgFO6fm4UJsl.1:538:538::/home.mirg/shy:/bin/bash [root@mirgserver yp]$ service ypserv restart Stopping YP server services: [ OK ] Starting YP server services: [ OK ] [root@mirgserver yp]$ ypmatch shy passwd Can't match key shy in map passwd.byname. Reason: No such key in map
Enable nfs
# service nfs start # vi /etc/exports # # /etc/exports file on mirgserver # /home.mirg carlos-mirg(rw,sync,fsid=0,no_subtree_check,root_squash) /home.mirg dimension(rw,sync,fsid=0,no_subtree_check,root_squash) /home.mirg dosimetry-pc(rw,sync,fsid=0,no_subtree_check,root_squash) /home.mirg march2011(rw,sync,fsid=0,no_subtree_check,root_squash) /home.mirg optiplex(rw,sync,fsid=0,no_subtree_check,root_squash) /home.mirg simba(rw,sync,fsid=0,no_subtree_check,root_squash) /home.mirg tesla(rw,sync,fsid=0,no_subtree_check,root_squash) /home.mirg vostro(rw,sync,no_subtree_check,root_squash) # exportfs -a # vi /etc/sysconfig/iptables # The following rules allow nfs clients to access the nfs folder -A INPUT -s 137.82.90.0/23 -m state --state NEW -m tcp -p tcp --dport 2049 -j ACCEPT -A INPUT -s 137.82.90.0/23 -m state --state NEW -m tcp -p tcp --dport 111 -j ACCEPT -A INPUT -s 137.82.90.0/23 -m state --state NEW -m udp -p udp --dport 111 -j ACCEPT -A INPUT -s 137.82.90.0/23 -m state --state NEW -m tcp -p tcp --dport 32803 -j ACCEPT -A INPUT -s 137.82.90.0/23 -m state --state NEW -m tcp -p tcp --dport 892 -j ACCEPT # service iptables restart
Install and configure ypserv
# rpm -ivh ypserv-2.19-26.el6.x86_64.rpm # vi /etc/sysconfig/network NETWORKING=yes HOSTNAME=mirgserver.vchri.med.ubc.ca NISDOMAIN=MIRG YPSERV_ARGS="-p 834" YPXFRD_ARGS="-p 835" # service ypserv start # /usr/lib64/yp/ypinit -m # vi /etc/sysconfig/iptables # The following rules allow nis clients to access the server -A INPUT -s 137.82.90.0/23 -m state --state NEW -m tcp -p tcp --dport 834 -j ACCEPT -A INPUT -s 137.82.90.0/23 -m state --state NEW -m udp -p udp --dport 834 -j ACCEPT -A INPUT -s 137.82.90.0/23 -m state --state NEW -m tcp -p tcp --dport 835 -j ACCEPT # service iptables restart
Configure matlabe license server
# vi /opt/matlab/R2013a/etc/license.dat SERVER mirgserver 003048FFDC9E 27000 DAEMON MLM "/opt/matlab/R2013a/etc/MLM" port=1711 # vi /etc/sysconfig/iptables # The following rules allow matlab clients to access the license server -A INPUT -s 137.82.90.0/23 -m state --state NEW -m tcp -p tcp --dport 27000 -j ACCEPT -A INPUT -s 137.82.90.0/23 -m state --state NEW -m tcp -p tcp --dport 1711 -j ACCEPT # service iptables restart # cd /opt/matlab/R2013a/etc # ./lmdown # su phasadmin # ./lmstart
Install NVIDIA driver
# rpm --import http://elrepo.org/RPM-GPG-KEY-elrepo.org # rpm -Uvh http://elrepo.org/elrepo-release-6-4.el6.elrepo.noarch.rpm # yum install kmod-nvidia
For more assistance
contact sysadmin@phas.ubc.ca
( Sysadmin
)
| webmaster@phas.ubc.ca | [Dept. Home Page] | last updated: |
[root@mirgserver sbin]$ ./sysinfo_mirg
===================================================================
System informationmirgserver.phas.ubc.ca
===================================================================
-------------------------------------------------------------------
cat /etc/fstab
-------------------------------------------------------------------
#
# /etc/fstab
# Created by anaconda on Fri Apr 5 09:31:38 2013
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=cd85bf62-2fe4-49e0-8c9d-762e74f6b1ad / ext4 defaults 1 1
UUID=5b027788-9e1d-4acf-8577-9e0708f5eba0 /boot ext4 defaults 1 2
UUID=d9107fea-c328-4100-bbc1-ac0654df9742 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
#/dev/md127 /home.mirg ext4 defaults 1 2
LABEL=HOME /home.mirg ext4 defaults 1 2
#/dev/mapper/vg_zorok-lv_home /home.mirg ext4 defaults 1 2
-------------------------------------------------------------------
cat /proc/partitions
-------------------------------------------------------------------
major minor #blocks name
8 0 976762584 sda
8 1 4194304 sda1
8 2 204800 sda2
8 3 972361728 sda3
8 16 976762584 sdb
8 17 4194304 sdb1
8 18 204800 sdb2
8 19 972361728 sdb3
8 64 2930266584 sde
8 48 2930266584 sdd
8 49 2930265543 sdd1
8 32 2930266584 sdc
8 33 2930265543 sdc1
9 1 4192192 md1
9 2 972230464 md2
9 127 2930134272 md127
9 0 204736 md0
-------------------------------------------------------------------
ls -l /dev/disk/by-id
-------------------------------------------------------------------
total 0
lrwxrwxrwx 1 root root 9 Aug 8 11:21 ata-WDC_WD1003FBYX-01Y7B1_WD-WCAW35005894 -> ../../sda
lrwxrwxrwx 1 root root 10 Aug 8 11:21 ata-WDC_WD1003FBYX-01Y7B1_WD-WCAW35005894-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Aug 8 11:21 ata-WDC_WD1003FBYX-01Y7B1_WD-WCAW35005894-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Aug 8 11:21 ata-WDC_WD1003FBYX-01Y7B1_WD-WCAW35005894-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Aug 8 11:21 ata-WDC_WD1003FBYX-01Y7B1_WD-WCAW35009564 -> ../../sdb
lrwxrwxrwx 1 root root 10 Aug 8 11:21 ata-WDC_WD1003FBYX-01Y7B1_WD-WCAW35009564-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Aug 8 11:21 ata-WDC_WD1003FBYX-01Y7B1_WD-WCAW35009564-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Aug 8 11:21 ata-WDC_WD1003FBYX-01Y7B1_WD-WCAW35009564-part3 -> ../../sdb3
lrwxrwxrwx 1 root root 9 Aug 8 11:21 ata-WDC_WD30EFRX-68AX9N0_WD-WMC1T2930945 -> ../../sdc
lrwxrwxrwx 1 root root 10 Aug 8 11:21 ata-WDC_WD30EFRX-68AX9N0_WD-WMC1T2930945-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 9 Aug 8 11:21 ata-WDC_WD30EFRX-68AX9N0_WD-WMC1T2931925 -> ../../sdd
lrwxrwxrwx 1 root root 10 Aug 8 11:21 ata-WDC_WD30EFRX-68AX9N0_WD-WMC1T2931925-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 9 Aug 8 11:21 ata-WDC_WD30EFRX-68AX9N0_WD-WMC1T2932744 -> ../../sde
lrwxrwxrwx 1 root root 9 Aug 8 11:21 md-name-mirgserver:0 -> ../../md0
lrwxrwxrwx 1 root root 9 Aug 8 11:21 md-name-mirgserver:1 -> ../../md1
lrwxrwxrwx 1 root root 9 Aug 8 11:21 md-name-mirgserver:2 -> ../../md2
lrwxrwxrwx 1 root root 11 Aug 8 11:23 md-name-mirgserver.phas.ubc.ca:3 -> ../../md127
lrwxrwxrwx 1 root root 11 Aug 8 11:23 md-uuid-08db0e06:d3f11acd:28630277:8e055f71 -> ../../md127
lrwxrwxrwx 1 root root 9 Aug 8 11:21 md-uuid-339170f7:e8e3fe16:8f2bb838:9182209d -> ../../md0
lrwxrwxrwx 1 root root 9 Aug 8 11:21 md-uuid-65cff1a1:8b9b8133:03d8b05b:0d3a4abf -> ../../md2
lrwxrwxrwx 1 root root 9 Aug 8 11:21 md-uuid-962bd962:a16f16aa:fd68330b:afda3510 -> ../../md1
lrwxrwxrwx 1 root root 9 Aug 8 11:21 scsi-SATA_WDC_WD1003FBYX-_WD-WCAW35005894 -> ../../sda
lrwxrwxrwx 1 root root 10 Aug 8 11:21 scsi-SATA_WDC_WD1003FBYX-_WD-WCAW35005894-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Aug 8 11:21 scsi-SATA_WDC_WD1003FBYX-_WD-WCAW35005894-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Aug 8 11:21 scsi-SATA_WDC_WD1003FBYX-_WD-WCAW35005894-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Aug 8 11:21 scsi-SATA_WDC_WD1003FBYX-_WD-WCAW35009564 -> ../../sdb
lrwxrwxrwx 1 root root 10 Aug 8 11:21 scsi-SATA_WDC_WD1003FBYX-_WD-WCAW35009564-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Aug 8 11:21 scsi-SATA_WDC_WD1003FBYX-_WD-WCAW35009564-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Aug 8 11:21 scsi-SATA_WDC_WD1003FBYX-_WD-WCAW35009564-part3 -> ../../sdb3
lrwxrwxrwx 1 root root 9 Aug 8 11:21 scsi-SATA_WDC_WD30EFRX-68_WD-WMC1T2930945 -> ../../sdc
lrwxrwxrwx 1 root root 10 Aug 8 11:21 scsi-SATA_WDC_WD30EFRX-68_WD-WMC1T2930945-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 9 Aug 8 11:21 scsi-SATA_WDC_WD30EFRX-68_WD-WMC1T2931925 -> ../../sdd
lrwxrwxrwx 1 root root 10 Aug 8 11:21 scsi-SATA_WDC_WD30EFRX-68_WD-WMC1T2931925-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 9 Aug 8 11:21 scsi-SATA_WDC_WD30EFRX-68_WD-WMC1T2932744 -> ../../sde
lrwxrwxrwx 1 root root 9 Aug 8 11:21 wwn-0x50014ee207f578d9 -> ../../sdb
lrwxrwxrwx 1 root root 10 Aug 8 11:21 wwn-0x50014ee207f578d9-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Aug 8 11:21 wwn-0x50014ee207f578d9-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Aug 8 11:21 wwn-0x50014ee207f578d9-part3 -> ../../sdb3
lrwxrwxrwx 1 root root 9 Aug 8 11:21 wwn-0x50014ee2b2a053eb -> ../../sda
lrwxrwxrwx 1 root root 10 Aug 8 11:21 wwn-0x50014ee2b2a053eb-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Aug 8 11:21 wwn-0x50014ee2b2a053eb-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Aug 8 11:21 wwn-0x50014ee2b2a053eb-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Aug 8 11:21 wwn-0x50014ee6033bf0bb -> ../../sde
lrwxrwxrwx 1 root root 9 Aug 8 11:21 wwn-0x50014ee658915322 -> ../../sdd
lrwxrwxrwx 1 root root 10 Aug 8 11:21 wwn-0x50014ee658915322-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 9 Aug 8 11:21 wwn-0x50014ee658915914 -> ../../sdc
lrwxrwxrwx 1 root root 10 Aug 8 11:21 wwn-0x50014ee658915914-part1 -> ../../sdc1
-------------------------------------------------------------------
ls -l /dev/disk/by-label
-------------------------------------------------------------------
total 0
lrwxrwxrwx 1 root root 11 Aug 8 11:23 HOME -> ../../md127
-------------------------------------------------------------------
ls -l /dev/disk/by-path
-------------------------------------------------------------------
total 0
lrwxrwxrwx 1 root root 9 Aug 8 11:21 pci-0000:00:11.0-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 Aug 8 11:21 pci-0000:00:11.0-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Aug 8 11:21 pci-0000:00:11.0-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Aug 8 11:21 pci-0000:00:11.0-scsi-0:0:0:0-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Aug 8 11:21 pci-0000:00:11.0-scsi-1:0:0:0 -> ../../sdb
lrwxrwxrwx 1 root root 10 Aug 8 11:21 pci-0000:00:11.0-scsi-1:0:0:0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Aug 8 11:21 pci-0000:00:11.0-scsi-1:0:0:0-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Aug 8 11:21 pci-0000:00:11.0-scsi-1:0:0:0-part3 -> ../../sdb3
lrwxrwxrwx 1 root root 9 Aug 8 11:21 pci-0000:00:11.0-scsi-2:0:0:0 -> ../../sdc
lrwxrwxrwx 1 root root 10 Aug 8 11:21 pci-0000:00:11.0-scsi-2:0:0:0-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 9 Aug 8 11:21 pci-0000:00:11.0-scsi-3:0:0:0 -> ../../sdd
lrwxrwxrwx 1 root root 10 Aug 8 11:21 pci-0000:00:11.0-scsi-3:0:0:0-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 9 Aug 8 11:21 pci-0000:00:14.1-scsi-0:0:0:0 -> ../../sde
-------------------------------------------------------------------
ls -l /dev/disk/by-uuid
-------------------------------------------------------------------
total 0
lrwxrwxrwx 1 root root 9 Aug 8 11:21 5b027788-9e1d-4acf-8577-9e0708f5eba0 -> ../../md0
lrwxrwxrwx 1 root root 9 Aug 8 11:21 cd85bf62-2fe4-49e0-8c9d-762e74f6b1ad -> ../../md2
lrwxrwxrwx 1 root root 9 Aug 8 11:21 d9107fea-c328-4100-bbc1-ac0654df9742 -> ../../md1
lrwxrwxrwx 1 root root 11 Aug 8 11:23 f571b260-daf2-4f51-bee6-12b41d26869e -> ../../md127
-------------------------------------------------------------------
cat /proc/mdstat
-------------------------------------------------------------------
Personalities : [raid1]
md0 : active raid1 sda2[0] sdb2[1]
204736 blocks super 1.0 [2/2] [UU]
md127 : active raid1 sdc1[0] sdd1[1]
2930134272 blocks super 1.2 [2/2] [UU]
md2 : active raid1 sda3[0] sdb3[1]
972230464 blocks super 1.1 [2/2] [UU]
bitmap: 2/8 pages [8KB], 65536KB chunk
md1 : active raid1 sda1[0] sdb1[1]
4192192 blocks super 1.1 [2/2] [UU]
unused devices:
-------------------------------------------------------------------
cat /etc/mdadm.conf
-------------------------------------------------------------------
# mdadm.conf written out by anaconda
MAILADDR root
MAILFROM mdadm@mirgserver.vchri.med.ubc.ca
AUTO +imsm +1.x -all
#ARRAY /dev/md0 level=raid1 metadata=1.0 name=mirgserver:0 num-devices=2 UUID=339170f7:e8e3fe16:8f2bb838:9182209d
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=339170f7:e8e3fe16:8f2bb838:9182209d
#ARRAY /dev/md0 metadata=1.0 name=mirgserver:0 UUID=339170f7:e8e3fe16:8f2bb838:9182209d
#ARRAY /dev/md1 level=raid1 metadata=1.1 name=mirgserver:1 num-devices=2 UUID=962bd962:a16f16aa:fd68330b:afda3510
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=962bd962:a16f16aa:fd68330b:afda3510
#ARRAY /dev/md1 metadata=1.1 name=mirgserver:1 UUID=962bd962:a16f16aa:fd68330b:afda3510
ARRAY /dev/md2 level=raid1 metadata=1.1 name=mirgserver:2 num-devices=2 UUID=65cff1a1:8b9b8133:03d8b05b:0d3a4abf
#ARRAY /dev/md2 level=raid1 num-devices=2 UUID=65cff1a1:8b9b8133:03d8b05b:0d3a4abf
#ARRAY /dev/md2 metadata=1.1 name=mirgserver:2 UUID=65cff1a1:8b9b8133:03d8b05b:0d3a4abf
ARRAY /dev/md3 metadata=1.2 spares=1 name=mirgserver:3 UUID=1caef7da:0d9a0c70:b11d5b45:80403b08
-------------------------------------------------------------------
mdadm --examine /dev/sda
-------------------------------------------------------------------
/dev/sda:
MBR Magic : aa55
Partition[0] : 8388608 sectors at 2048 (type fd)
Partition[1] : 409600 sectors at 8390656 (type fd)
Partition[2] : 1944723456 sectors at 8800256 (type fd)
-------------------------------------------------------------------
mdadm --examine /dev/sdb
-------------------------------------------------------------------
/dev/sdb:
MBR Magic : aa55
Partition[0] : 8388608 sectors at 2048 (type fd)
Partition[1] : 409600 sectors at 8390656 (type fd)
Partition[2] : 1944723456 sectors at 8800256 (type fd)
-------------------------------------------------------------------
mdadm --examine /dev/sdc
-------------------------------------------------------------------
/dev/sdc:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
-------------------------------------------------------------------
mdadm --examine /dev/sdd
-------------------------------------------------------------------
/dev/sdd:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
===================================================================
[root@mirgserver rc.d]$ cat /etc/mdadm.conf # mdadm.conf written out by anaconda MAILADDR root MAILFROM mdadm@mirgserver.vchri.med.ubc.ca AUTO +imsm +1.x -all #ARRAY /dev/md0 level=raid1 metadata=1.0 name=mirgserver:0 num-devices=2 UUID=339170f7:e8e3fe16:8f2bb838:9182209d ARRAY /dev/md0 level=raid1 num-devices=2 UUID=339170f7:e8e3fe16:8f2bb838:9182209d #ARRAY /dev/md0 metadata=1.0 name=mirgserver:0 UUID=339170f7:e8e3fe16:8f2bb838:9182209d #ARRAY /dev/md1 level=raid1 metadata=1.1 name=mirgserver:1 num-devices=2 UUID=962bd962:a16f16aa:fd68330b:afda3510 ARRAY /dev/md1 level=raid1 num-devices=2 UUID=962bd962:a16f16aa:fd68330b:afda3510 #ARRAY /dev/md1 metadata=1.1 name=mirgserver:1 UUID=962bd962:a16f16aa:fd68330b:afda3510 ARRAY /dev/md2 level=raid1 metadata=1.1 name=mirgserver:2 num-devices=2 UUID=65cff1a1:8b9b8133:03d8b05b:0d3a4abf #ARRAY /dev/md2 level=raid1 num-devices=2 UUID=65cff1a1:8b9b8133:03d8b05b:0d3a4abf #ARRAY /dev/md2 metadata=1.1 name=mirgserver:2 UUID=65cff1a1:8b9b8133:03d8b05b:0d3a4abf ARRAY /dev/md3 metadata=1.2 spares=1 name=mirgserver:3 UUID=1caef7da:0d9a0c70:b11d5b45:80403b08
[root@mirgserver rc.d]$ cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sdd1[1] sdc1[0]
2930134272 blocks super 1.2 [2/2] [UU]
md0 : active raid1 sdb2[1] sda2[0]
204736 blocks super 1.0 [2/2] [UU]
md2 : active raid1 sda3[0] sdb3[1]
972230464 blocks super 1.1 [2/2] [UU]
bitmap: 0/8 pages [0KB], 65536KB chunk
md1 : active raid1 sda1[0] sdb1[1]
4192192 blocks super 1.1 [2/2] [UU]
unused devices: <none>
[root@mirgserver ~]$ mdadm --examine /dev/sda /dev/sda: MBR Magic : aa55 Partition[0] : 8388608 sectors at 2048 (type fd) Partition[1] : 409600 sectors at 8390656 (type fd) Partition[2] : 1944723456 sectors at 8800256 (type fd) [root@mirgserver ~]$ mdadm --examine /dev/sdb /dev/sdb: MBR Magic : aa55 Partition[0] : 8388608 sectors at 2048 (type fd) Partition[1] : 409600 sectors at 8390656 (type fd) Partition[2] : 1944723456 sectors at 8800256 (type fd) [root@mirgserver ~]$ mdadm --examine /dev/sdc /dev/sdc: MBR Magic : aa55 Partition[0] : 4294967295 sectors at 1 (type ee) [root@mirgserver ~]$ mdadm --examine /dev/sdd /dev/sdd: MBR Magic : aa55 Partition[0] : 4294967295 sectors at 1 (type ee)
[root@mirgserver ~]$ cat /proc/partitions major minor #blocks name 8 0 976762584 sda 8 1 4194304 sda1 8 2 204800 sda2 8 3 972361728 sda3 8 32 2930266584 sdc 8 33 2930265543 sdc1 8 16 976762584 sdb 8 17 4194304 sdb1 8 18 204800 sdb2 8 19 972361728 sdb3 8 64 2930266584 sde 8 65 2930265088 sde1 8 48 2930266584 sdd 8 49 2930265543 sdd1 9 1 4192192 md1 9 2 972230464 md2 9 0 204736 md0 9 3 2930134272 md3
[root@mirgserver ~]$ dumpe2fs /dev/md0 | grep -i superblock dumpe2fs 1.41.12 (17-May-2010) Primary superblock at 1, Group descriptors at 2-2 Backup superblock at 8193, Group descriptors at 8194-8194 Backup superblock at 24577, Group descriptors at 24578-24578 Backup superblock at 40961, Group descriptors at 40962-40962 Backup superblock at 57345, Group descriptors at 57346-57346 Backup superblock at 73729, Group descriptors at 73730-73730 [root@mirgserver ~]$ dumpe2fs /dev/md2 | grep -i superblock dumpe2fs 1.41.12 (17-May-2010) Primary superblock at 0, Group descriptors at 1-58 Backup superblock at 32768, Group descriptors at 32769-32826 Backup superblock at 98304, Group descriptors at 98305-98362 Backup superblock at 163840, Group descriptors at 163841-163898 Backup superblock at 229376, Group descriptors at 229377-229434 Backup superblock at 294912, Group descriptors at 294913-294970 Backup superblock at 819200, Group descriptors at 819201-819258 Backup superblock at 884736, Group descriptors at 884737-884794 Backup superblock at 1605632, Group descriptors at 1605633-1605690 Backup superblock at 2654208, Group descriptors at 2654209-2654266 Backup superblock at 4096000, Group descriptors at 4096001-4096058 Backup superblock at 7962624, Group descriptors at 7962625-7962682 Backup superblock at 11239424, Group descriptors at 11239425-11239482 Backup superblock at 20480000, Group descriptors at 20480001-20480058 Backup superblock at 23887872, Group descriptors at 23887873-23887930 Backup superblock at 71663616, Group descriptors at 71663617-71663674 Backup superblock at 78675968, Group descriptors at 78675969-78676026 Backup superblock at 102400000, Group descriptors at 102400001-102400058 Backup superblock at 214990848, Group descriptors at 214990849-214990906 [root@mirgserver ~]$ dumpe2fs /dev/md3 | grep -i superblock dumpe2fs 1.41.12 (17-May-2010) Primary superblock at 0, Group descriptors at 1-175 Backup superblock at 32768, Group descriptors at 32769-32943 Backup superblock at 98304, Group descriptors at 98305-98479 Backup superblock at 163840, Group descriptors at 163841-164015 Backup superblock at 229376, Group descriptors at 229377-229551 Backup superblock at 294912, Group descriptors at 294913-295087 Backup superblock at 819200, Group descriptors at 819201-819375 Backup superblock at 884736, Group descriptors at 884737-884911 Backup superblock at 1605632, Group descriptors at 1605633-1605807 Backup superblock at 2654208, Group descriptors at 2654209-2654383 Backup superblock at 4096000, Group descriptors at 4096001-4096175 Backup superblock at 7962624, Group descriptors at 7962625-7962799 Backup superblock at 11239424, Group descriptors at 11239425-11239599 Backup superblock at 20480000, Group descriptors at 20480001-20480175 Backup superblock at 23887872, Group descriptors at 23887873-23888047 Backup superblock at 71663616, Group descriptors at 71663617-71663791 Backup superblock at 78675968, Group descriptors at 78675969-78676143 Backup superblock at 102400000, Group descriptors at 102400001-102400175 Backup superblock at 214990848, Group descriptors at 214990849-214991023 Backup superblock at 512000000, Group descriptors at 512000001-512000175 Backup superblock at 550731776, Group descriptors at 550731777-550731951 Backup superblock at 644972544, Group descriptors at 644972545-644972719
Now you can use the backup superblock to
check the file system using superblock
# 98304 (warning do not run the following on mounted live file system):
# e2fsck -f -b 98304 /dev/md3
OR
mount the file system using superblock
# 98304:
# mkdir -p /mnt/data5
# mount -o sb=98304 /dev/sda5 /mnt/data5
[root@mirgserver sysconfig]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md2 913G 95G 772G 11% /
tmpfs 64G 224K 64G 1% /dev/shm
/dev/md0 194M 73M 111M 40% /boot
/dev/sde1 2.7T 1.9T 706G 73% /home.mirg
[root@mirgserver sysconfig]$ cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Fri Apr 5 09:31:38 2013
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=cd85bf62-2fe4-49e0-8c9d-762e74f6b1ad / ext4 defaults 1 1
UUID=5b027788-9e1d-4acf-8577-9e0708f5eba0 /boot ext4 defaults 1 2
UUID=d9107fea-c328-4100-bbc1-ac0654df9742 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
#/dev/md3 /home.mirg ext4 defaults 1 2
/dev/sde1 /home.mirg ext4 defaults 1 2
#/dev/mapper/vg_zorok-lv_home /home.mirg ext4 defaults 1 2
[root@mirgserver sysconfig]$ parted /dev/sdc
GNU Parted 2.1
Using /dev/sdc
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: ATA WDC WD30EFRX-68A (scsi)
Disk /dev/sdc: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Number Start End Size File system Name Flags
(parted) unit TB
(parted) mkpart primary 0.00TB 3.00TB
(parted) print
Model: ATA WDC WD30EFRX-68A (scsi)
Disk /dev/sdc: 3.00TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Number Start End Size File system Name Flags
1 0.00TB 3.00TB 3.00TB primary
(parted) quit
================================================================
Basic procedure to create a new GPT partition table and partition using gdisk:
gdisk /dev/sdd
option o (creates a new GPT partition table)
option n (creates a new partition)
2x enter (accepts the default partition parameters, since I just want one 1 partition on the drive)
fd00 (sets partition type as Linux Raid)
w (writes changes to the drive and exits)
Detailed steps with output:
[root@mirgserver sysconfig]$ gdisk /dev/sdd
GPT fdisk (gdisk) version 0.8.8
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Command (? for help): o
This option deletes all partitions and creates a new protective MBR.
Proceed? (Y/N): y
Command (? for help): n
Partition number (1-128, default 1):
First sector (34-5860533134, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-5860533134, default = 5860533134) or {+-}size{KMGTP}:
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): fd00
Changed type of partition to 'Linux RAID'
Command (? for help): w
Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!
Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdd.
The operation has completed successfully.
[root@mirgserver sysconfig]$ parted /dev/sdd
GNU Parted 2.1
Using /dev/sdd
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: ATA WDC WD30EFRX-68A (scsi)
Disk /dev/sdd: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Number Start End Size File system Name Flags
1 1049kB 3001GB 3001GB Linux RAID raid
(parted) quit
[root@mirgserver sysconfig]$ gdisk /dev/sdd
GPT fdisk (gdisk) version 0.8.8
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Command (? for help): p
Disk /dev/sdd: 5860533168 sectors, 2.7 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): D909C92F-3F91-435E-9D62-40F6E7A47C3F
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5860533134
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)
Number Start (sector) End (sector) Size Code Name
1 2048 5860533134 2.7 TiB FD00 Linux RAID
Command (? for help): quit
==============================================================================
[root@mirgserver sysconfig]$ mdadm --create /dev/md3 --level=1 --raid-devices=2 /dev/sdc1 /dev/sdd1
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md3 started.
[root@mirgserver sysconfig]$ mkfs.ext4 /dev/md3
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=1 blocks, Stripe width=0 blocks
183140352 inodes, 732533568 blocks
36626678 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
22356 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 29 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
Fdisk - output
[root@mirgserver rc.d]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md2 913G 95G 772G 11% /
tmpfs 64G 228K 64G 1% /dev/shm
/dev/md0 194M 73M 111M 40% /boot
/dev/sde1 2.7T 1.9T 701G 74% /home.mirg
/dev/md3 2.7T 558G 2.1T 22% /raid1
Note: at this point data was still being rsync'd from /home.mirg to /raid1
[root@mirgserver rc.d]$ fdisk -l
Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0003317d
Device Boot Start End Blocks Id System
/dev/sda1 1 523 4194304 fd Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sda2 * 523 548 204800 fd Linux raid autodetect
Partition 2 does not end on cylinder boundary.
/dev/sda3 548 121602 972361728 fd Linux raid autodetect
WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sdc: 3000.6 GB, 3000592982016 bytes
256 heads, 63 sectors/track, 363376 cylinders
Units = cylinders of 16128 * 512 = 8257536 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdc1 1 266306 2147483647+ ee GPT
Partition 1 does not start on physical sector boundary.
Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00058edd
Device Boot Start End Blocks Id System
/dev/sdb1 1 523 4194304 fd Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sdb2 * 523 548 204800 fd Linux raid autodetect
Partition 2 does not end on cylinder boundary.
/dev/sdb3 548 121602 972361728 fd Linux raid autodetect
WARNING: GPT (GUID Partition Table) detected on '/dev/sde'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sde: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sde1 1 267350 2147483647+ ee GPT
Partition 1 does not start on physical sector boundary.
WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sdd: 3000.6 GB, 3000592982016 bytes
256 heads, 63 sectors/track, 363376 cylinders
Units = cylinders of 16128 * 512 = 8257536 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdd1 1 266306 2147483647+ ee GPT
Partition 1 does not start on physical sector boundary.
Disk /dev/md1: 4292 MB, 4292804608 bytes
2 heads, 4 sectors/track, 1048048 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/md2: 995.6 GB, 995563995136 bytes
2 heads, 4 sectors/track, 243057616 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/md0: 209 MB, 209649664 bytes
2 heads, 4 sectors/track, 51184 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
Disk /dev/md3: 3000.5 GB, 3000457494528 bytes
2 heads, 4 sectors/track, 732533568 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Hi All, I'm late to the game but I still want to play I eventually got to exactly the same losetup command line as Ovidu, but I took a slightly different road. I used testdisk again but this time I choose > proceed > EFI GPT ** > Analyse > Quick search The difference is the EFI GPT partition table type rather than the "Intel" type we tried on Friday. Almost immediately it finds an msdos entry at 262145. > MS Data 262145 4294966912 4294704768 It then jumps to the end of that partition and tries to scan the un-partitioned space on the disk. You can then cancel the search and use "P" to list the files as we did before. In retrospect since we didn't know how the partition table was corrupted we should have tried all of the partition table types offered by testdisk. From that point on, the solution is then the same as Ovidu's... $ losetup -r -o $((262145 * 512)) /dev/loop0 /dev/sdX $ mount /dev/loop0 /mnt/disk3 Or just tell mount to offset by the same amount. It looks like there is 1.9T of 2T used. $ mount -o ro,offset=$((262145 * 512)) /dev/sdX /mnt/disk3 I also agree with Ovidu that we should try to find out they "why" as well as the "what". I guess we should go through the binary data at the beginning of the disk with a fine tooth comb! In addition I'd really like to know what happens when that RAID card is plugged into another UNIMPORTANT machine. Also, once all the dust has settled we should try to write something up about it somewhere. Happy Holiday's and THANK YOU to Ovidu! -Ian. On 12/21/2013, 7:54 AM, Ron wrote: > Nice work Ovidiu! I have added access to aston from opio. Let me know > if there is anything else i can do. Please call me at 604-838-6437 or > 778-878-5412 if needed. > Regards, > Ron > > On 13-12-21 12:29 AM, Ovidiu Toader wrote: >> I managed to recover an ext4 filesystem from disk 4 using pretty much >> the same ideas that we discussed on Fri. Instead of looping over offsets >> with losetup and dumpe2fs I scanned only for the 53EF signature that is >> specific to the ext superblock. As expected with a string this short >> there were lots of false positives but luckily I could use the location >> of the backup superblocks from a good ext4 filesystem (the output of >> dumpe2fs) to learn the expected intervals between real backup >> superblocks and then use those intervals to filter out only the real >> superblocks from disk 4. In the end it paid off because I could find a >> valid superblock at offset 262145*512 followed by many backup >> superblocks at the expected relative offsets. I used only a read-only >> view of the disk so fsck was out of question but based on the superblock >> the fs is clean. My guess is that the large offset of the start of ext4 >> is caused by an mdadm superblock which contains a bitmap. After you get >> the data off this disk I want to hunt for the proper mdadm superblock as >> well but only for pedagogical purposes. >> >> If I get direct access from aston to one of my test file-servers >> (opio.phas.ubc.ca) then I could do a temporary backup of the data. >> >> >> # work only with a read-only view >> root@aston:~/work/images# losetup -r -o $((262145*512)) /dev/loop0 /dev/sdc >> >> # we have a superblock! >> root@aston:~/work/images# dumpe2fs -h /dev/loop0 >> dumpe2fs 1.42.5 (29-Jul-2012) >> Filesystem volume name: <none> >> Last mounted on: /home.mirg >> Filesystem UUID: 55074993-d23f-4c7f-b2ff-1d3b38617d8a >> Filesystem magic number: 0xEF53 >> Filesystem revision #: 1 (dynamic) >> Filesystem features: has_journal ext_attr resize_inode dir_index >> filetype extent flex_bg sparse_super large_file huge_file uninit_bg >> dir_nlink extra_isize >> Filesystem flags: signed_directory_hash >> Default mount options: (none) >> Filesystem state: clean >> Errors behavior: Continue >> Filesystem OS type: Linux >> Inode count: 134209536 >> Block count: 536838096 >> Reserved block count: 26841904 >> Free blocks: 30923799 >> Free inodes: 131939760 >> First block: 0 >> Block size: 4096 >> Fragment size: 4096 >> Reserved GDT blocks: 896 >> Blocks per group: 32768 >> Fragments per group: 32768 >> Inodes per group: 8192 >> Inode blocks per group: 512 >> RAID stride: 1 >> Flex block group size: 16 >> Filesystem created: Fri Apr 12 10:39:47 2013 >> Last mount time: Wed Aug 28 08:52:53 2013 >> Last write time: Wed Dec 18 08:27:38 2013 >> Mount count: 12 >> Maximum mount count: 25 >> Last checked: Fri Apr 12 10:39:47 2013 >> Check interval: 15552000 (6 months) >> Next check after: Wed Oct 9 10:39:47 2013 >> Lifetime writes: 13 TB >> Reserved blocks uid: 0 (user root) >> Reserved blocks gid: 0 (group root) >> First inode: 11 >> Inode size: 256 >> Required extra isize: 28 >> Desired extra isize: 28 >> Journal inode: 8 >> Default directory hash: half_md4 >> Directory Hash Seed: 763ce645-b092-4e22-a80c-dab7e2e481df >> Journal backup: inode blocks >> Journal features: journal_incompat_revoke >> Journal size: 128M >> Journal length: 32768 >> Journal sequence: 0x035261d9 >> Journal start: 0 >> >> root@aston:~/work/images# mount /dev/loop0 /recovery/ >> mount: block device /dev/loop0 is write-protected, mounting read-only >> root@aston:~/work/images# ls -l /recovery/ >> total 34705076 >> drwxr-x---. 2 root phasadmin 4096 Oct 8 08:27 00_archive >> -rw-r--r--. 1 root root 604 Mar 5 2013 00_diskuse_130305 >> -rw-r--r--. 1 root root 523 Oct 1 15:41 00_diskuse_131001 >> drwxr-x---. 25 501 phasadmin 4096 Apr 8 2008 aceller >> drwx------. 95 mce 529 12288 Dec 13 14:23 curibe >> drwxrwxr-x. 9 root map 4096 Oct 13 2006 devlib >> drwx------. 78 528 528 4096 Dec 17 11:26 empire >> drwxrwxrwx. 6 mce 537 4096 Oct 7 14:24 GATE61 >> drwxr-xr-x. 35 lees 537 4096 Oct 30 17:53 greynes >> drwx------. 55 zsergey 534 4096 Dec 17 12:16 hma >> drwxrwxr-x. 8 mce 523 4096 Oct 1 11:48 JADA >> drwx------. 19 539 539 4096 Dec 9 16:19 jesset >> drwx------. 39 538 538 4096 Dec 17 12:05 lisa >> drwxrwsr-x. 8 root phasadmin 4096 Oct 1 15:59 Lookup_Tables >> drwx------ 2 root root 16384 Apr 10 2013 lost+found >> drwxrwxrwx. 8 mce 529 4096 Dec 11 14:03 MIRG_Shared >> drwxr-xr-x. 64 tanaka 533 4096 Dec 17 17:27 plesquinas >> drwxrwx--x. 30 521 521 4096 Oct 4 11:38 rap >> drwxr-xr-x. 15 mce 529 4096 Sep 24 09:28 RECMIRG >> drwxr-x--x. 4 root root 4096 Mar 5 2013 root >> drwxrwxr-x. 52 530 sudo 12288 Dec 17 16:23 snakano >> drwxr-x---. 107 507 phasadmin 12288 Apr 22 2013 sshcherbinin >> -rw-r--r--. 1 root root 35537862402 Oct 9 15:21 sshcherbinin.tgz >> >> # fs full! >> root@aston:~# df -h /recovery/ >> Filesystem Size Used Avail Use% Mounted on >> >> /dev/loop0 2.0T 1.9T 16G 100% /recovery >> ------------------------------------------------------------------------ On 12/21/13 15:45, Ian Allison wrote: > I also agree with Ovidu that we should try to find out they "why" as > well as the "what". I guess we should go through the binary data at the > beginning of the disk with a fine tooth comb! In addition I'd really > like to know what happens when that RAID card is plugged into another > UNIMPORTANT machine. I don't have a scratch computer available at the moment but I would also like to see what is special about this card. This episode has reinforced my reluctance to use a GUI tool for storage related operations. > Also, once all the dust has settled we should try > to write something up about it somewhere. I think that for stage1 (finding the ext file-system) your solution is the one that has to be documented. Mine is too convoluted in comparison. I was in fact wondering why testdisk failed to find this healthy file-system. stage2 is only for play at the moment but i would like to see if one could also recover the mdadm superblock somehow. I used a similar scanning approach (this time 0xa92b4efc little-endian) but could not find a superblock in front of the recovered file-system. If there is a superblock then my guess is that it is located at the end of the partition. The strange part is that this array was supposed to be version 1.2 which should place the superblock at 4K from the start of the device. Unfortunately it looks like my disk is going to be busy transferring the data out until Monday so i won't get a chance to scan the end of the partition till then. > > Happy Holiday's Happy Holidays! o. ------------------------------------------------------------------------ Hi Ovidiu, On 12/21/2013, 4:19 PM, Ovidiu Toader wrote: > I don't have a scratch computer available at the moment but I would also > like to see what is special about this card. This episode has reinforced > my reluctance to use a GUI tool for storage related operations. Agreed! I think there are two possibilities. Either the raid card decided it would "help" by starting to zero the disks, or gparted noticed that the disk had 4k sectors but was formatted using fdisk and saying "here, let me ruin that for you" >> Also, once all the dust has settled we should try >> to write something up about it somewhere. stage2 is only for play at the moment but i would like to > see if one could also recover the mdadm superblock somehow. I used a > similar scanning approach (this time 0xa92b4efc little-endian) but could > not find a superblock in front of the recovered file-system. I had the same thought, but I also couldn't find the raid magic-number at the beginning of the disk. One thing I tried was to make 3 images of the first 1G of the disk. I ran the image files through fdisk to put a partition on them then I ran them through mdadm to create a 2 mirror + 1 spare setup. Running any of the resulting files through hexdump you can find the v1.2 superblock at 69632 into the file. This is 64k + 4k into the "disk". As Ovidiu mentioned the 4k is the offset that is described in the v1.2 docs, I'm not so sure about the 64k bit (alignment maybe?) but the conclusion is that there definitely should be a 0xa92b4efc in the region we were checking. The RedHat installer would have to have over-riden the defaults to use a different superblock version - I doubt they would have done this. > Unfortunately it looks like my disk is going to be busy > transferring the data out until Monday so i won't get a chance to scan > the end of the partition till then. I can do this tonight, my disk isn't doing anything. My money is on not finding anything - i think the partition table and raid superblock got zapped with zeros by either gparted or the raid card firmware. Cheers, Ian. ------------------------------------------------------------------------ On 12/21/13 17:20, Ian Allison wrote: > I can do this tonight, my disk isn't doing anything. My money is on not > finding anything i find it odd that the mdadm would disappear just like that although I am pretty sure that my disk (and looks like yours as well) does not have a valid 1.2 mdadm superblock. The other arrays on the same machine have a 1.1 superblock and that's why I am thinking that it is not impossible to find a pre 1.2 superblock at the end of the partition. If that were indeed the case then it is possible that a failed conversion of the mdadm to 1.2 caused this mess. good luck! o. ------------------------------------------------------------------------ Hi Ovidiu, I've got it running as hexdump -a 2198888837120 /dev/loop0 My offset is the last block reported by dumpe2fs and loop0 is just a read only loopback of /dev/sda. So far it sees nothing - as in all blocks read are identically zero. I'll let it finish the device (100 MB/s for the zeros!) but it looks like the factory formatting from western digital. The last nonzero bytes are about 184k before my offset. I'll go back and reassess that area when this run finishes. > a 1.1 superblock and that's why I am thinking that it is not impossible > to find a pre 1.2 superblock at the end of the partition. If that were > indeed the case then it is possible that a failed conversion of the > mdadm to 1.2 caused this mess. I had another look, it looks like only 1.0 and 0.9 use the end of the device. 1.1 and 1.2 are at the beginning but 1.2 is offset by 4k. Cheers, Ian ------------------------------------------------------------------------ Hi Guys, Just a quick update, I let hexdump finish and here are the results from the last 1TB of the disk hexdump -s 2198888837120 /dev/loop0 1fff7fcf000 0000 0000 0000 0000 0000 0000 0000 0000 * 2ba99475c00 4645 2049 4150 5452 0000 0001 005c 0000 2ba99475c10 45e6 30d4 0000 0000 a3af 5d50 0001 0000 2ba99475c20 0001 0000 0000 0000 0022 0000 0000 0000 2ba99475c30 a38e 5d50 0001 0000 b3bf 5fa9 24cf 4f83 2ba99475c40 14be 0cf5 433d 0f76 a38f 5d50 0001 0000 2ba99475c50 0080 0000 0080 0000 d286 ab54 0000 0000 2ba99475c60 0000 0000 0000 0000 0000 0000 0000 0000 * 2ba99475e00 I haven't removed anything from the output, there was no raid superblock magic number. Everything else is zero'd out. In ASCII the above is |EFI PART....\...| |.E.0......P]....| |........".......| |..P]......._.$.O| |....=Cv...P]....| |..........T.....| |................| * 2ba99475e00 It's a GPT partition table entry, though I think the numbers in the LBA fields might not make sense. Cheers, Ian. On 12/21/2013, 5:31 PM, Ovidiu Toader wrote: > On 12/21/13 17:20, Ian Allison wrote: >> I can do this tonight, my disk isn't doing anything. My money is on not >> finding anything > i find it odd that the mdadm would disappear just like that although I > am pretty sure that my disk (and looks like yours as well) does not have > a valid 1.2 mdadm superblock. The other arrays on the same machine have > a 1.1 superblock and that's why I am thinking that it is not impossible > to find a pre 1.2 superblock at the end of the partition. If that were > indeed the case then it is possible that a failed conversion of the > mdadm to 1.2 caused this mess. > > good luck! > o. > ------------------------------------------------------------------------ Hi All, I took a look at the MBR and the GPT partition entries we found, I ran them through python to see what the contents looked like, here is what my drive says... MBR === The first 512b of the disk are MBR (either traditional or GPT "protected"). For a traditional MBR you would have some bootstrap code and other bits and pieces in the first 446 bytes. Next come the 4 partition table entries at 16 bytes each. Finally there are 2 bytes of "Boot signature" - "55 aa". When I look at my disk I have the first 446 bytes zeroed out, then we have the first partition table entry 00 00 01 00 ee fe ff ff 01 00 00 00 ff ff ff ff Status : 0 CHS Address of first absolute sector : 0,0,1 Partition type : ee CHS Address of last absolute sector : 254,1023,63 LBA of first absolute sector : 16777216 Number of sectors in partition : 4294967295 second and third partition entries are empty, and the last two bytes contain the boot signature as expected "55 aa". We could look it up but I think this is exactly the "protective MBR" that GPT specifies - basically it is a placeholder that is supposed to stop fdisk from trying to mess with the partitions. GPT 1 === The first (GPT) partition entry starts at 512 bytes into the disk, here are the values I decoded First Partition Table Entry Signature : 'EFI PART' Revision : 65536 Header Size : 92 CRC32 of header : 130046590 Reserved : 0 Current LBA : 1 Backup LBA : 5860533167 First Useable LBA : 34 Last Usable LBA : 5860533134 Disk GUID : '\xbf\xb3\xa9_\xcf$\x83O\xbe\x14\xf5\x0c=Cv\x0f' Starting LBA : 2 Number of Partition entries in array : 128 Single partition entry size : 128 CRC32 of partition Array : 2874462854 GPT 2 ===== The second (GPT) partition table entry starts at 3000592981504 bytes (2baa1475e00) into the disk, again, here are the decoded values Second Partition Table Entry Signature : 'EFI PART' Revision : 65536 Header Size : 92 CRC32 of header : 819217894 Reserved : 0 Current LBA : 5860533167 Backup LBA : 1 First Useable LBA : 34 Last Usable LBA : 5860533134 Disk GUID : '\xbf\xb3\xa9_\xcf$\x83O\xbe\x14\xf5\x0c=Cv\x0f' Starting LBA : 5860533135 Number of Partition entries in array : 128 Single partition entry size : 128 CRC32 of partition Array : 2874462854 The first one looks sort of sensible (though I don't really understand the LBA values) the second one is a mystery to me. After the first partition table entry there is just over 128Mb of zeros then the ext4 filesystem starts at 134219264 bytes (0x8000600). In summary it still looks like gparted screwed the disks! Cheers, Ian. http://en.wikipedia.org/wiki/Master_boot_record http://en.wikipedia.org/wiki/GUID_Partition_Table ------------------------------------------------------------------------