已索引
XSKY 硬盘故障解决案例
在 XSKY 管理台看到 dc01-ywsc-xsky-02(193.165.128.133)节点上有三块硬盘告警：
HDD osd.24(sdg) 状态是“重启中”，SSD osd.2(sda) 和 osd.3(sdb) 状态是“已隔离”。
通过 SSH 登录该节点。
查看 ceph 状态：
# ceph -s
cluster:
id: 1918e441-fa2a-4fb0-853a-d8fa1117649e
health: HEALTH_WARN
5519455/1007668863 objects misplaced (0.548%)
Degraded data redundancy: 74055693/1007668863 objects degraded (7.349%), 573 pgs[18.3fd,18.3ec,18.3d8,18.3c6..] degraded, 556 pgs[18.3fd,18.3ec,18.3d8,18.3c2..] undersized
3 osds down <--- 这儿显示3个盘离线了
1 host (2 osds) down
1 pool(s) nearfull
services:
mon: 3 daemons, quorum dc01-ywsc-xsky-01,dc01-ywsc-xsky-02,dc01-ywsc-xsky-03
osd: 108 osds: 105 up, 108 in; 73 remapped pgs
data:
pools: 17 pools, 2496 pgs
objects: 261M objects, 509T
usage: 757T used, 233T / 990T avail
pgs: 74055636/1007668863 objects degraded (7.349%)
5519306/1007668863 objects misplaced (0.548%)
1854 active+clean
553 active+degraded+undersized
57 active+remapped+backfill_wait
15 active+degraded+recovery_wait
5 active+remapped+rebalance+backfill_wait
5 active+remapped+backfilling
2 active+degraded+undersized+remapped+backfill_wait
1 active+degraded+remapped+backfill_wait
1 active+degraded+recovery_wait+remapped
1 active+clean+remapped
1 active+clean+scrubbing
1 active+degraded+undersized+remapped+backfilling
io:
client: [rd] 187 MB/s, 67862 op/s, [wr] 338 B/s, 38 op/s
recovery: 181 MB/s, 21 objects/s查看磁盘的故障代码：
# cat /var/log/ceph/ceph-osd.24.err
5000003 <--- 表示是亚健康被隔离的查看 dmesg 日志：
# dmesg -T | grep -i blk
[Tue Jul 14 16:28:04 2020] Initializing cgroup subsys blkio
[Tue Jul 14 16:28:09 2020] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
[Fri Apr 24 02:17:48 2026] blk_update_request: critical medium error, dev sdg, sector 180295136 <--- 4月24号 sdg 提示 media error
[Sun Jul 26 21:17:06 2026] blk_update_request: critical medium error, dev sde, sector 145924072 <--- 7月26号 sde 提示 media error
--- 这块盘当时是 osd.21，已经更换，更换后 osd.21 盘符变成 sdi
[Tue Aug 11 13:47:37 2026] blk_update_request: I/O error, dev sdg, sector 11164280496 <--- 8月11号 sdg 提示 I/O error，报扇区故障
[Tue Aug 11 13:47:37 2026] blk_update_request: I/O error, dev sdg, sector 7236140288
[Tue Aug 11 13:47:37 2026] blk_update_request: I/O error, dev sdg, sector 7236139008
[Tue Aug 11 13:47:37 2026] blk_update_request: I/O error, dev sdg, sector 7236141056
[Tue Aug 11 13:47:37 2026] blk_update_request: I/O error, dev sdg, sector 11164281600
[Tue Aug 11 13:47:37 2026] blk_update_request: I/O error, dev sdg, sector 48573952
[Tue Aug 11 13:47:37 2026] blk_update_request: I/O error, dev sdg, sector 48236544
[Tue Aug 11 13:47:37 2026] blk_update_request: I/O error, dev sdg, sector 10718341160
[Tue Aug 11 13:47:37 2026] blk_update_request: I/O error, dev sdg, sector 11164281088
[Tue Aug 11 13:47:37 2026] blk_update_request: I/O error, dev sdg, sector 3930886656
[Tue Aug 11 13:49:43 2026] blk_update_request: 31 callbacks suppressed
[Tue Aug 11 13:49:43 2026] blk_update_request: I/O error, dev sdg, sector 23437770496
[Tue Aug 11 13:49:43 2026] blk_update_request: I/O error, dev sdg, sector 20981136
[Tue Aug 11 13:49:43 2026] blk_update_request: I/O error, dev sdg, sector 48236416
[Tue Aug 11 13:49:43 2026] XFS (sdg1): metadata I/O error: block 0x1401d90 ("xlog_iodone") error 5 numblks 64
[Tue Aug 11 13:49:43 2026] blk_update_request: I/O error, dev sdg, sector 0# dmesg -T | grep -i reset
[Tue Jul 14 16:28:09 2020] megaraid_sas 0000:3b:00.0: Online Controller Reset(OCR) : Enabled
[Tue Aug 11 13:48:05 2026] megaraid_sas 0000:3b:00.0: Found FW in FAULT state, will reset adapter scsi0. <--- 8月11号 sdg 提示 I/O error
[Tue Aug 11 13:48:05 2026] megaraid_sas 0000:3b:00.0: resetting fusion adapter scsi0.
[Tue Aug 11 13:49:43 2026] megaraid_sas 0000:3b:00.0: Online Controller Reset(OCR) : Enabled
[Tue Aug 11 13:49:43 2026] megaraid_sas 0000:3b:00.0: Reset successful for scsi0.
[Tue Aug 11 13:49:43 2026] megaraid_sas 0000:3b:00.0: 174807 (839744347s/0x0020/CRIT) - Controller encountered a fatal error and was reset检查带外，有两条告警：
Disk The disk Disk5 state is abnormal (SN:8HJ2B11H). 2026-08-11 06:38:26 0x02000027
Disk The disk Disk5 is missing. 2026-08-11 06:40:46 0x0200002D 在 XSKY 中看到 SN:8HJ2B11H 的磁盘为 sdg。
注意：medium error 要检查下 BMC 有没有硬盘报错。单纯的 blk 故障不会在 BMC 显示。
检查三块磁盘的性能图，三块盘的“存储盘IO利用率”都有一个陡增，从10%以下迅速上升到100%。
综合上面的分析，最终问题为：
一块盘(sdg)损坏后，RAID 卡 RESET 导致 IO 利用率陡增，XSKY 认为硬盘亚健康，为了防止影响业务读写，就把硬盘隔离了。
但是注意：正常坏盘不会导致 RAID 卡被 RESET，这个需要硬件工程师进行分析。
