By now, we all know Oracle is fully supported on VMware. Anyone telling you it’s not supported is either lying to you, or doesn’t know what he is talking about (I keep wondering what’s worse).
VMware support includes Oracle RAC (if it’s version 11.2.0.2.0 or above). However, Oracle may request to reproduce problems on physically deployed systems in case they suspect the problem is related to the hypervisor. The support note says:
Oracle will only provide support for issues that either are known to occur on the native OS, or can be demonstrated not to be as a result of running on VMware.
In case that happens, I recommend to contact VMWare support first because they might be familiar with the issue or can escalate the problem quickly. VMware support will take full ownership of the problem. Still, I have met numerous customers who are afraid of having to reproduce issues quickly and reliably on physical in case the escalation policy does not help. We need to get out of the virtual world, into reality, without making any other changes. How do we do that?
Unfortunately, no one can be told what the Matrix is. You have to see it for yourself.
(Opens a pillbox, empties the contents into his palms, and outstretches his hands)
This is your last chance. After this, there is no turning back.
(opens his right hand, to reveal a translucent blue pill)
You take the blue pill, the story ends, you wake up in your bed and believe whatever you want to believe.
(opens his left hand, revealing a similarly translucent red pill)
You take the red pill, you stay in Wonderland, and I show you how deep the rabbit hole goes.
– Morpheus, The Matrix (1999)
Let’s stay in database Wonderland but with a real, not virtual, view of the world.
One of the issues is that a database can be quite large and if the database uses VMDK files on VMFS, a physical host can not mount those volumes to be able to use that data – VMFS is part of the VMware kernel and not available as a file system within Linux (or any other OS for that matter). An alternative that is often suggested is using RDM (Raw Device Mappings) which allows VMware virtual machines to use SAN volumes directly without VMware’s VMFS file system in between. Although RDMs have advantages (most notably it makes EMC storage replication a lot easier) this causes some administration overhead and is not always preferred by administrators, especially for many small- and medium sized databases. Some of the white papers recommending RDMs for this purpose, also suggest you build an Oracle RAC cluster with (by default) virtual nodes but with the option of adding physical nodes when needed – allowing you to migrate easily between full virtual and full physical deployments. Of course this requires you to deal with the complexity and overhead of RAC to begin with (not even mentioning the high additional license cost). Many of the customers I’ve spoken to have a large number of databases where only a few run Oracle RAC and they prefer to use VMFS for at least a number of their databases.
VMFS volumes would thus normally require the data to be migrated on physical volumes first before a physical host could mount and use the volumes. That takes precious time and effort, and involves risk of running into issues – exactly the kind of thing you don’t need when dealing with other serious production problems!
But maybe… just maybe you don’t have to move that data at all…
Let’s see if we can outsmart Oracle’s support policies and use a virtual machine’s data on VMDK volumes on a physical host. This requires that the virtual and physical machine have similar architectures and OS installed, so that the physical machine could run the database if it had access to the data. But… that was the requirement anyway even if you would physically migrate the data to the physical machine.
The method we are going to use is a storage networking protocol called iSCSI. Basically iSCSI allows you to present a block device (disk) over an IP network to another host. So what we are going to do is set up the “production” virtual machine as an iSCSI target (server) and the “physical” machine as an iSCSI initiator (client). Sounds complicated? Let’s run through the setup and see.
For the sake of this test I have two virtual machines running under VMware ESXi 5.1. “vmwdb1” is the live database server that is supposed to have a production problem of which the issue is suspected to be the hypervisor. The other, “phydb2” is actually a clean virtual machine without databases – but with Oracle ASM and the grid component installed and configured. We will pretend “phydb2” is a physical host without the capability of opening VMware VMFS/VMDK files. Of course you need network access between the machines (if performance is an issue you might as well have a fast LAN backbone).
Other configuration details:
Both VMs: 2 vCPU each, 4GB mem each
OS: CentOS 6.4 (compatible with Red Hat 6.4 and Oracle Enterprise Linux 6.4)
Boot volume: 20GB
Server 1
Hostname: vmwdb1
Database: 11.2.0.3.0.
Database name: proddb
SGA: 1 GB
Disk setup:
# asm list disk: /dev/sda [0:0:0:0] 20.00 GB in use (has partitions) disk: /dev/sdb [0:0:1:0] 4.00 GB asm volume /dev/oracleasm/admn disk: /dev/sdc [0:0:3:0] 4.00 GB asm volume /dev/oracleasm/tmp2 disk: /dev/sdd [1:0:0:0] 8.00 GB asm volume /dev/oracleasm/dat1 disk: /dev/sde [1:0:1:0] 8.00 GB asm volume /dev/oracleasm/dat2 disk: /dev/sdf [2:0:0:0] 3.00 GB asm volume /dev/oracleasm/rdo2
Application: Swingbench
Filesystem for Oracle stuff: mounted as /oracle
ASM is managed by a script I created that mimics ASMLib commands.
In a moment of sheer inspiration I called my script “asm” 😉
This because ASMlib is no longer supported by Oracle under Red Hat / CentOS 6.4 (and not using ASMlib has some advantages). All VMDK volumes reside on a few SATA disks in my test server (this test is to show functionality, not make it to guinness book of records).
Here is a screenshot of Swingbench (actually minibench) running against the virtual machine’s database:
Swingbench uses the standard Order Entry benchmark with User defined scale = 2. This will create about 6,5 GB datasize. I had partitioning and bigfile tablespace enabled and created all SB data on diskgroup “+DATA”.
Server 2
Hostname: phydb2
Database: 11.2.0.3.0 (only binaries installed, no database)
ASM diskgroups:
None (but 1 LUN visible as /dev/oracleasm/foo)
Now we want to access the database on “vmwdb1” from “phydb2”, start it up and open it for processing. This requires a few steps:
– An ORACLE_HOME should be available with the correct version of the software
– ASM diskgroups should be visible on the target host
– Some startup settings need to be provided (i.e. init.ora, oratab and the like).
Let’s start with ORACLE_HOME. Remember we are troubleshooting operational problems so ideally we want the same ORACLE_HOME as we use in production. This guarantees we don’t silently introduce new problems due to different patch levels, or otherwise magically eliminate issues by the same means.
Our “prod” server has an /oracle file system, ORACLE_HOME sits under /oracle/base/db and ORACLE_HOME of ASM (GRID) sits under /oracle/grid.
Let’s NFS mount ORACLE_HOME on the target so we use the exact same binaries. This requires that the prod host exports the ORACLE_HOME as NFS. How to do this:
set up an entry in /etc/exports for the ORACLE_BASE
# cat /etc/exports /oracle/base/ 192.168.0.0/255.255.128.0(rw,sync,no_subtree_check,no_root_squash)
Make sure nfs services are enabled and disable NFS v4 (avoid problems with user rights etc)
# chkconfig nfs on # vi /etc/sysconfig/nfs # Turn off v4 protocol support RPCNFSDARGS="-N 4"
start NFS services
# service nfs start Starting NFS services: [ OK ] Starting NFS mountd: [ OK ] Stopping RPC idmapd: [ OK ] Starting RPC idmapd: [ OK ] Starting NFS daemon: [ OK ]
On the target server
We already have an ORACLE_HOME for ASM/grid and another one for database services. However, we want to replace the one for database with the NFS shared version of our prod server. Assuming we don’t have any databases running on the target server:
# mv /oracle/base/ /oracle/base_temp # mkdir /oracle/base # mount vmwdb1:/oracle/base /oracle/base # df /oracle/base Filesystem 1K-blocks Used Available Use% Mounted on vmwdb1:/oracle/base 10079232 7419904 2556928 75% /oracle/base
Now let’s deal with the ASM volumes.
Setup iSCSI target (prod host)
First we need to install the scsi target utilities:
# yum install -y scsi-target-utils
Now we set up an iSCSI environment with CHAP authentication. First uncomment the “include” line in targets.conf:
# vi /etc/tgt/targets.conf # This one includes other config files: include /etc/tgt/temp/*.conf
We have to set an iSCSI name which is normally “iqn.<year-month>.domain:targetname” – let’s use “iqn.2013-09.local:target01” to keep things simple. In large iSCSI deployments naming needs to be a bit more sophisticated but for now this will do just fine.
We can make a separate file with our iSCSI volumes in /etc/tgt/temp/asm.conf without messing around with the targets.conf (that file might be overwritten during OS updates). Let’s see which volumes we have under /dev/oracleasm and add them all as iSCSI target.
# ls -d /dev/oracleasm/* /dev/oracleasm/admn /dev/oracleasm/dat1 /dev/oracleasm/dat2 /dev/oracleasm/rdo2 /dev/oracleasm/tmp2 # mkdir /etc/tgt/temp # vi /etc/tgt/temp/asm.conf <target iqn.2013-09.local:target01> backing-store /dev/oracleasm/admn backing-store /dev/oracleasm/dat1 backing-store /dev/oracleasm/dat2 backing-store /dev/oracleasm/rdo2 backing-store /dev/oracleasm/tmp2 initiator-address 192.168.2.13 incominguser oracle oracle </target>
Note that we use oracle/oracle here as iSCSI userid/password. In real production environments you might want to use a more secure combi. The only host allowed to get access to the iSCSI volumes is the one using the specified “initiator address” (in our case this is the “physical” host).
Make sure “tgtd” gets started each time we reboot and start it now:
# chkconfig tgtd on # service tgtd start Starting SCSI target daemon: [ OK ] # tgtadm --mode target --op show Target 1: iqn.2013-09.local:target01 System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Prevent removal: No Readonly: No Backing store type: null Backing store path: None Backing store flags: Account information: oracle ACL information: 192.168.2.13
You can see that there are no iSCSI devices presented. This is because ASM and the database are currently still running and keep the devices locked. Note that on my system I have the linux firewall disabled – if you have it enabled, make sure you enable network access to the iscsi port 3260.
Let’s stop the database and ASM and refresh the iSCSI configuration:
$ srvctl stop database -d proddb $ srvctl stop asm -f # service tgtd reload # tgtadm --mode target --op show Target 1: iqn.2013-09.local:target01 System information: Driver: iscsi State: ready I_T nexus information: ... LUN: 1 Type: disk SCSI ID: IET 00010001 SCSI SN: beaf11 Size: 4295 MB, Block size: 512 Online: Yes Removable media: No Prevent removal: No Readonly: No Backing store type: rdwr Backing store path: /dev/oracleasm/admn Backing store flags: ... LUN: 5 Type: disk SCSI ID: IET 00010005 SCSI SN: beaf15 Size: 4295 MB, Block size: 512 Online: Yes Removable media: No Prevent removal: No Readonly: No Backing store type: rdwr Backing store path: /dev/oracleasm/tmp2 Backing store flags: Account information: oracle ACL information: 192.168.2.13
So now all ASM volumes are present.
Now let’s setup the target (physical). Set the initiator name and set CHAP with userid/password in the iscsi configuration:
# vi /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.2013-09.local:target01 InitiatorAlias=initiator01 # vi /etc/iscsi/iscsid.conf # To enable CHAP authentication set node.session.auth.authmethod # to CHAP. The default is None. node.session.auth.authmethod = CHAP # To set a CHAP username and password for initiator # authentication by the target(s), uncomment the following lines: node.session.auth.username = oracle node.session.auth.password = oracle
Now let’s discover the target and login:
# iscsiadm --mode discovery -t sendtargets --portal vmwdb1 Starting iscsid: [ OK ] 192.168.2.11:3260,1 iqn.2013-09.local:target01 # iscsiadm --mode node --targetname iqn.2013-09.local:target01 --portal vmwdb1 --login Logging in to [iface: default, target: iqn.2013-09.local:target01, portal: 192.168.2.11,3260] (multiple) Login to [iface: default, target: iqn.2013-09.local:target01, portal: 192.168.2.11,3260] successful. # dmesg| tail -100 ... scsi4 : iSCSI Initiator over TCP/IP scsi 4:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5 scsi 4:0:0:0: Attached scsi generic sg3 type 12 scsi 4:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5 sd 4:0:0:1: Attached scsi generic sg4 type 0 scsi 4:0:0:2: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5 sd 4:0:0:2: Attached scsi generic sg5 type 0 sd 4:0:0:1: [sdc] 8388608 512-byte logical blocks: (4.29 GB/4.00 GiB) scsi 4:0:0:3: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5 sd 4:0:0:3: Attached scsi generic sg6 type 0 sd 4:0:0:2: [sdd] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) scsi 4:0:0:4: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5 sd 4:0:0:4: Attached scsi generic sg7 type 0 sd 4:0:0:1: [sdc] Write Protect is off ... sde: sdf: sdg: unknown partition table unknown partition table unknown partition table unknown partition table unknown partition table sd 4:0:0:3: [sde] Attached SCSI disk sd 4:0:0:4: [sdf] Attached SCSI disk sd 4:0:0:1: [sdc] Attached SCSI disk sd 4:0:0:5: [sdg] Attached SCSI disk sd 4:0:0:2: [sdd] Attached SCSI disk # ls -al /dev/sd* brw-rw---- 1 root disk 8, 0 Sep 17 09:46 /dev/sda brw-rw---- 1 root disk 8, 1 Sep 17 09:46 /dev/sda1 brw-rw---- 1 root disk 8, 2 Sep 17 09:46 /dev/sda2 brw-rw---- 1 root disk 8, 32 Sep 17 10:18 /dev/sdc brw-rw---- 1 root disk 8, 48 Sep 17 10:18 /dev/sdd brw-rw---- 1 root disk 8, 64 Sep 17 10:18 /dev/sde brw-rw---- 1 root disk 8, 80 Sep 17 10:18 /dev/sdf brw-rw---- 1 root disk 8, 96 Sep 17 10:18 /dev/sdg
You see 5 new disks appeared on the system. The remarks about unknown partition table is correct, we don’t partition ASM volumes.
Now I use my “asm” script to make these ASM devices appear under /dev/oracleasm. But first a special note on how this works. I use the Linux “udev” facility to get consistent device naming and my “asm” script uses the “SCSI identifier” to recognize disks even if they get different names between reboots. So a disk now listed as /dev/sdd might become /dev/sdf after reboot and so on. But the SCSI id does not change. You can see the scsi id using the “scsi_id” tool, but that requires the /etc/scsi_id.config to be set correctly. Without these settings “scsi_id” will report empty scsi IDs.
# vi /etc/scsi_id.config # allow scsi_id /dev/sdX reporting # Also set disk.EnableUUID = "true" in VMware VMX file for this to work # whitelisted and replace-whitespace is required for iscsi initiator compatibility options=-g --whitelisted --replace-whitespace # scsi_id /dev/sda 36000c29794d2bbd16cf370a62bd72236 # scsi_id /dev/sdc 1IET_00010001 # scsi_id /dev/sdg 1IET_00010005 # asm createdisk iscsi1 /dev/sdc running createdisk iscsi1 /dev/sdc iscsi1 1IET_00010001 Please review /etc/asmtab and reconfigure udev with "asmupdate" when done Then check /dev/oracleasm or /dev/iorate for the new volume. It should be disappeared from /dev [root@phydb2 ~]# asm createdisk iscsi2 /dev/sdd running createdisk iscsi2 /dev/sdd iscsi2 1IET_00010002 ... # asm createdisk iscsi3 /dev/sde running createdisk iscsi3 /dev/sde iscsi3 1IET_00010003 ... # asm createdisk iscsi4 /dev/sdf running createdisk iscsi4 /dev/sdf iscsi4 1IET_00010004 ... # asm createdisk iscsi5 /dev/sdg running createdisk iscsi5 /dev/sdg iscsi5 1IET_00010005 ... # asm scandisks # ls -l /dev/oracleasm/* brw-rw---- 1 grid asmdba 8, 16 Sep 17 10:25 /dev/oracleasm/foo brw-rw---- 1 grid asmdba 8, 32 Sep 17 10:25 /dev/oracleasm/iscsi1 brw-rw---- 1 grid asmdba 8, 48 Sep 17 10:25 /dev/oracleasm/iscsi2 brw-rw---- 1 grid asmdba 8, 64 Sep 17 10:25 /dev/oracleasm/iscsi3 brw-rw---- 1 grid asmdba 8, 80 Sep 17 10:25 /dev/oracleasm/iscsi4 brw-rw---- 1 grid asmdba 8, 96 Sep 17 10:25 /dev/oracleasm/iscsi5 [root@phydb2 ~]# ls -l /dev/sd* brw-rw---- 1 root disk 8, 0 Sep 17 10:25 /dev/sda brw-rw---- 1 root disk 8, 1 Sep 17 10:25 /dev/sda1 brw-rw---- 1 root disk 8, 2 Sep 17 10:25 /dev/sda2 # cat /etc/asmtab # This file is the source for asm disks (without ASMLib) in Oracle # Enter on each line: # volname scsi_id # # example: # redo 36000c29f825cd85b5fcc70a1aadebf0c # admn 36000c298afa5c31b47fe76cbd1750937 foo 36000c29706783fc9c5e6accdcc860e63 iscsi1 1IET_00010001 iscsi2 1IET_00010002 iscsi3 1IET_00010003 iscsi4 1IET_00010004 iscsi5 1IET_00010005 # asm list disk: /dev/sda [2:0:0:0] 20.00 GB in use (has partitions) disk: /dev/sdb [2:0:1:0] 4.00 GB asm volume /dev/oracleasm/foo disk: /dev/sdc [4:0:0:1] 4.00 GB asm volume /dev/oracleasm/iscsi1 disk: /dev/sdd [4:0:0:2] 8.00 GB asm volume /dev/oracleasm/iscsi2 disk: /dev/sde [4:0:0:3] 8.00 GB asm volume /dev/oracleasm/iscsi3 disk: /dev/sdf [4:0:0:4] 3.00 GB asm volume /dev/oracleasm/iscsi4 disk: /dev/sdg [4:0:0:5] 4.00 GB asm volume /dev/oracleasm/iscsi5
Note: asmtab is a special file I use myself to keep track of scsi identifiers versus device names under linux. It’s managed by the “asm” script. “asm list” shows the original (/dev/sdX) name, the SCSI ID (handy to figure out which VMDK file it is mapped to), the size and the new linux device name.
“asm” generates the correct rules for udev, as shown here:
# cat /etc/udev/rules.d/99-asm.rules
KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", OWNER="grid", GROUP="asmdba", MODE="0660", ENV{ID_SERIAL}=="36000c29706783fc9c5e6accdcc860e63", NAME="oracleasm/foo"
KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", OWNER="grid", GROUP="asmdba", MODE="0660", ENV{ID_SERIAL}=="1IET_00010001", NAME="oracleasm/iscsi1"
KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", OWNER="grid", GROUP="asmdba", MODE="0660", ENV{ID_SERIAL}=="1IET_00010002", NAME="oracleasm/iscsi2"
KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", OWNER="grid", GROUP="asmdba", MODE="0660", ENV{ID_SERIAL}=="1IET_00010003", NAME="oracleasm/iscsi3"
KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", OWNER="grid", GROUP="asmdba", MODE="0660", ENV{ID_SERIAL}=="1IET_00010004", NAME="oracleasm/iscsi4"
KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", OWNER="grid", GROUP="asmdba", MODE="0660", ENV{ID_SERIAL}=="1IET_00010005", NAME="oracleasm/iscsi5"
So now we have the original ORACLE_HOME and all ASM devices. Let’s make an entry in oratab to make DB startup a bit easier:
# echo proddb:/oracle/base/db:N >> /etc/oratab
Now we need to let ASM mount the diskgroups and start the database
SYS:+ASM > @asm-listdgs NUM NAME BLOCK AU_MB STATE TYPE TOTAL_MB FREE_MB USABLE_FILE_MB OFFLINE_DISKS ---------- ---------- ---------- ---------- ----------- ------ ---------- ---------- -------------- ------------- 0 REDO 4096 0 DISMOUNTED 0 0 0 0 0 DATA 4096 0 DISMOUNTED 0 0 0 0 0 ADMN 4096 0 DISMOUNTED 0 0 0 0 0 TEMP 4096 0 DISMOUNTED 0 0 0 0 SYS:+ASM > alter diskgroup redo mount; Diskgroup altered. SYS:+ASM > alter diskgroup data mount; Diskgroup altered. SYS:+ASM > alter diskgroup admn mount; Diskgroup altered. SYS:+ASM > alter diskgroup temp mount; Diskgroup altered. # su - oracle $ sqlplus / as sysdba SQL*Plus: Release 11.2.0.3.0 Production on Tue Sep 17 10:41:42 2013 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to an idle instance. SYS:proddb > startup ORACLE instance started. Total System Global Area 1068937216 bytes Fixed Size 2235208 bytes Variable Size 629146808 bytes Database Buffers 432013312 bytes Redo Buffers 5541888 bytes Database mounted. Database opened.
Mission accomplished 🙂
(Neo reaches for the red pill)
Remember: all I’m offering is the truth. Nothing more.
– Morpheus, The Matrix (1999)
Let’s start Swingbench on the “physical” node to see if we get it working:
Note that we almost get the same transactions per minute as compared to the original benchmark.
Let’s cleanup and get back to the original state:
SYS:proddb > shutdown immediate; ... $ srvctl stop asm -f ... # iscsiadm -m node -u Logging out of session [sid: 2, target: iqn.2013-09.local:target01, portal: 192.168.2.11,3260] Logout of [sid: 2, target: iqn.2013-09.local:target01, portal: 192.168.2.11,3260] successful. # iscsiadm -m session -o show iscsiadm: No active sessions.
On the prod (virtual) host we can now restart ASM and the database:
$ srvctl start asm $ srvctl start database -d proddb
Some additional thoughts:
- ASM will prevent starting on one server if the other one is active. Don’t worry about data corruption using this method.
- If you want to stay on physical you may use ASM migration to move the data (online) to local (physical attached) disk
- If your data is not using ASM but file systems (i.e. ext3) you can still use iSCSI remount but you need to mount the filesystems under the correct mountpoints yourself
- If you need to run the physical database directly as the new production environment you also need to change the IP and hostname settings. Before you do this make sure this doesn’t interfere with the IP connectivity between iSCSI target and initiator (i.e. use a different IP subnet for iSCSI where IP addresses don’t change)
- In a virtualized production environment you can prepare lots of these steps upfront (i.e. setting up iSCSI with a dummy volume, create scripts that quickly configure the ASM disks etc)
- If you want to keep production running as virtual, VMware has virtual disk cloning features that allow you to provide snapshots/clones next to the primary data. You may now provide iSCSI access to these clones instead of the real data. This has some advantages as I blogged about here: http://dirty-cache.com/2013/01/15/why-clone-db-firefighting/
The Oracle: You know why Morpheus brought you to see me?
Neo: I think so.
I think it’s because The Oracle wants to know if Neo lives in the virtual or physical world 😉
The Oracle: So, what do you think? Do you think you’re The One?
Neo: I don’t know.
(The latin phrase Temet Nosce appears on an inscription over the Oracle’s door)
The Oracle: You know what that means? It’s Latin. Means “Know thyself”. I’m going to let you in on a little secret. Being The One is just like being in love. No one can tell you you’re in love, you just know it. Through and through. Balls to bones.
– Neo and The Oracle, The Matrix (1999)
As long as The Oracle doesn’t know we’re in the virtual or physical world, things don’t matter. As long as we know ourselves, we’re good. But in case The Oracle finds out, you know what to do…
Comments are closed.