这里是以华为服务器 RH2288 V3 为例,过时的主机。查看主机是正常重启还是意外重启:last -x|head|tac
正常reboot重启显示如下:
root pts/0 10.5.3.207 Tue Nov 22 11:14 - down (00:04)
shutdown system down 3.10.0-1062.el7. Tue Nov 22 11:18 - 11:18 (00:00)
reboot system boot 3.10.0-1062.el7. Tue Nov 22 11:18 - 11:39 (00:20)
runlevel (to lvl 5) 3.10.0-1062.el7. Tue Nov 22 11:18 - 11:23 (00:04)
正常按电源开关重启显示如下:
root pts/0 10.5.3.207 Tue Nov 22 11:29 - crash (00:02)
reboot system boot 3.10.0-1062.el7. Tue Nov 22 11:31 - 11:39 (00:08)
runlevel (to lvl 5) 3.10.0-1062.el7. Tue Nov 22 11:31 - 11:39 (00:08)
意外重启多数显示如下(非人为或断电,系统自己重启):
reboot system boot 3.10.0-1160.49.1 Tue Nov 22 02:27 - 13:52 (11:25)
runlevel (to lvl 5) 3.10.0-1160.49.1 Tue Nov 22 02:27 - 13:52 (11:24)
那么到底是什么原因引起的重启? 多数是内存条,也可能是电源或CPU等。排查原因:1. 查看message 在重启时间点上下有什么特别的信息 cat /var/log/message在重启时间点往前翻,发现有多条报错如下:
Nov 21 17:43:08 host142 kernel: EDAC MC0: 3 CE memory scrubbing error on CPU_SrcID#1_Ha#0_Chan#0_DIMM#0 or CPU_SrcID#1_Ha#0_Chan#0_DIMM#1 (channel:0 page:0x2b980fe offset:0x0 grain:32 syndrome:0x0 - OVERFLOW area:DRAM err_code:0008:00c0 socket:1 ha:0 channel_mask:1 rank:255)
因为没查出别的问题,所以只能认为是内存有问题了,参照下面的对应表,把有问题的内存换掉或不用。2. 再进一步查看一下硬件信息dmesg |grep -Ei 'error|Fail'如果得到以下信息可能是内存或cpu有问题:
[root@mysql /]# dmesg |grep -Ei 'error|Fail'
[ 1.750394] ERST: Error Record Serialization Table (ERST) support is initialized.
[ 2.893211] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter.
[ 4163.052786] mce: [Hardware Error]: Machine check events logged
[ 4163.052813] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 4163.352856] EDAC MC0: 2 CE memory scrubbing error on CPU_SrcID#1_Ha#0_Chan#0_DIMM#0 or CPU_SrcID#1_Ha#0_Chan#0_DIMM#1 (channel:0 page:0x2b980fa offset:0x0 grain:32 syndrome:0x0 - OVERFLOW area:DRAM err_code:0008:00c0 socket:1 ha:0 channel_mask:1 rank:255)
3. 使用工具检查一下内存是否有报错:yum install -y libsysfs edac-utils#查看内存是否有报错:edac-util -vdmidecode -t memory
华为 RH2288 V3对应内存位置:
参考: https://blog.51cto.com/u_15060511/4572303
补充:华为服务器用上面的对应不上槽位,通过华为自带的硬件管理查看日志fdm_log,查到是 BANK14有问题再通过系统命令查看 bank14对应的槽位,更换内存:lshw -class memory