Oracle RAC集群存储双活方案部署


刚给客户配置完双机双柜存储双活,双活就是双节点RAC+底层双存储.记录一下

存储双活方案介绍：
在这里插入图片描述

详细磁盘规划如下:

存储1 存储2 第三方仲裁
OCR_1盘 OCR_2盘 采用NFS
(DATA1 DATA2 DATA3 DATA4)(故障组1) (DATA5 DATA6 DATA7 DATA8)(故障组2)

存储采用多路径.查看当前多路径链路状态及盘情况

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
[root@his01 ~]# multipath -ll
MS2500G2-5 (3600b3421abe826ed18eeda0d3d0000da) dm-6 MacroSAN,LU
size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 0:0:1:4 sdl 8:176 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 1:0:1:4 sdv 65:80 active ready running
MS3000G2-5 (3600b3421137e245d4b53d171fdf100d2) dm-8 MacroSAN,LU
size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 0:0:0:4 sdg 8:96  active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 1:0:0:4 sdq 65:0  active ready running
MS2500G2-4 (3600b3429b29dc86d944dd270fd0000da) dm-12 MacroSAN,LU
size=500G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 1:0:1:3 sdu 65:64 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 0:0:1:3 sdk 8:160 active ready running
MS3000G2-4 (3600b342259b42f2df415d0a2ed5d00d2) dm-7 MacroSAN,LU
size=500G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 0:0:0:3 sdf 8:80  active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 1:0:0:3 sdp 8:240 active ready running
MS2500G2-3 (3600b3428a361741d4419d14d7d0000da) dm-4 MacroSAN,LU
size=500G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 0:0:1:2 sdj 8:144 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 1:0:1:2 sdt 65:48 active ready running
MS3000G2-3 (3600b3423382ec30d51b1d6375de500d2) dm-10 MacroSAN,LU
size=500G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 1:0:0:2 sdo 8:224 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 0:0:0:2 sde 8:64  active ready running
MS2500G2-2 (3600b34293ffd77ad2885dd57cd0000da) dm-11 MacroSAN,LU
size=500G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 1:0:1:1 sds 65:32 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 0:0:1:1 sdi 8:128 active ready running
MS3000G2-2 (3600b34249eed7dbda56bd28ebd6400d2) dm-5 MacroSAN,LU
size=500G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 0:0:0:1 sdd 8:48  active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 1:0:0:1 sdn 8:208 active ready running
MS2500G2-1 (3600b342bffd0345d78b1d492ad0000da) dm-3 MacroSAN,LU
size=500G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 0:0:1:0 sdh 8:112 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 1:0:1:0 sdr 65:16 active ready running
MS3000G2-1 (3600b342207dc3a0dd34ade09cd6c00d2) dm-9 MacroSAN,LU
size=500G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 1:0:0:0 sdm 8:192 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 0:0:0:0 sdc 8:32  active ready running

磁盘权限绑定:

1
2
3
4
5
6
7
8
9
10
11
vi /etc/udev/rules.d/12-dm-permissions.rules
ENV{DM_NAME}=="MS2500G2-1", OWNER:="grid", GROUP:="oinstall", MODE:="660"
ENV{DM_NAME}=="MS2500G2-2", OWNER:="grid", GROUP:="oinstall", MODE:="660"
ENV{DM_NAME}=="MS2500G2-3", OWNER:="grid", GROUP:="oinstall", MODE:="660"
ENV{DM_NAME}=="MS2500G2-4", OWNER:="grid", GROUP:="oinstall", MODE:="660"
ENV{DM_NAME}=="MS2500G2-5", OWNER:="grid", GROUP:="oinstall", MODE:="660"
ENV{DM_NAME}=="MS3000G2-1", OWNER:="grid", GROUP:="oinstall", MODE:="660"
ENV{DM_NAME}=="MS3000G2-2", OWNER:="grid", GROUP:="oinstall", MODE:="660"
ENV{DM_NAME}=="MS3000G2-3", OWNER:="grid", GROUP:="oinstall", MODE:="660"
ENV{DM_NAME}=="MS3000G2-4", OWNER:="grid", GROUP:="oinstall", MODE:="660"
ENV{DM_NAME}=="MS3000G2-5", OWNER:="grid", GROUP:="oinstall", MODE:="660"

NFS服务器配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
服务端配置:
[root@shuanghuozhongcai /]#yum install -y nfs-utils

[root@shuanghuozhongcai /]# cat /etc/exports
/ocrvote3 192.168.0.0/24(rw,sync,no_wdelay,insecure_locks,no_root_squash)

节点1客户端挂载:
[root@his01 ~]#showmount -e 192.168.0.75
Export list for 192.168.0.75:
/ocrvote3 192.168.0.0/24
[root@his01 ~]#mount -o rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0 192.168.0.75:/ocrvote3 /ocrvote

vi /etc/fstab
192.168.0.75:/ocrvote3 /ocrvote nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0 0 0


节点2客户端挂载:
[root@his02 ~]#showmount -e 192.168.0.75
Export list for 192.168.0.75:
/ocrvote3 192.168.0.0/24
[root@his02 ~]#mount -o rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0 192.168.0.75:/ocrvote3 /ocrvote

vi /etc/fstab
192.168.0.75:/ocrvote3 /ocrvote nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0 0 0

NFS做OCR/Voting file参数配置如下：Mount Options for Oracle files when used with NFS on NAS devices (文档 ID 359515.1)

在NFS共享文件夹中dd一块磁盘做第三块表决盘：

1
2
3
4
[root@his01 ~]# dd if=/dev/zero  of=/ocrvote/ocrvote3 bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 47.6771 s, 22.5 MB/s

创建OCR磁盘组:

MS2500G2-5 存储1
MS3000G2-5 存储2
ocrvote3 NFS
在这里插入图片描述

创建DATA磁盘组：

存储1 MS2500G2的四块盘做故障组F1
存储2 MS3000G2的四块盘做故障组F2
在这里插入图片描述
在这里插入图片描述

关于ASM建立故障组与非建立故障组的冗余模式分析

磁盘与故障组信息查询：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
[grid@his01 ~]$ asmcmd lsdg
State    Type    Rebal  Sector  Block  AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N   512   4096  1048576   4096000  4095766           512000         1791883              0             N  DATA/
MOUNTED  NORMAL  N   512   4096  1048576      3072     2146             1024             561              0             Y  OCR/

select name,path from v$asm_disk_stat order by name;
NAME                           PATH
------------------------------ ------------------------------
DATA_0000                      /dev/mapper/MS2500G2-1
DATA_0001                      /dev/mapper/MS2500G2-2
DATA_0002                      /dev/mapper/MS2500G2-3
DATA_0003                      /dev/mapper/MS2500G2-4
DATA_0004                      /dev/mapper/MS3000G2-1
DATA_0005                      /dev/mapper/MS3000G2-2
DATA_0006                      /dev/mapper/MS3000G2-3
DATA_0007                      /dev/mapper/MS3000G2-4
OCR_0000                       /ocrvote/ocrvote3
OCR_0001                       /dev/mapper/MS2500G2-5
OCR_0002                       /dev/mapper/MS3000G2-5

故障组查询:
select name,path,failgroup from v$asm_disk order by name;
NAME            PATH                           FAILGROUP
--------------- ------------------------------ ------------------------------
DATA_0000       /dev/mapper/MS2500G2-1         F1
DATA_0001       /dev/mapper/MS2500G2-2         F1
DATA_0002       /dev/mapper/MS2500G2-3         F1
DATA_0003       /dev/mapper/MS2500G2-4         F1
DATA_0004       /dev/mapper/MS3000G2-1         F2
DATA_0005       /dev/mapper/MS3000G2-2         F2
DATA_0006       /dev/mapper/MS3000G2-3         F2
DATA_0007       /dev/mapper/MS3000G2-4         F2
OCR_0000        /ocrvote/ocrvote3              OCR_0000
OCR_0001        /dev/mapper/MS2500G2-5         OCR_0001
OCR_0002        /dev/mapper/MS3000G2-5         OCR_0002

存储双活方案部署完成.

关于ASM建立故障组与非建立故障组的冗余模式分析