Veritas Cluster is responsible to provide high availability for a Application with a minimum downtime. High availability clusters (HAC) improve application availability by failing them over or switching them over in a group of systems.
A cluster can be build with 2 nodes atleast and a maximum of 32nodes. While using VCS we need to create Service groups and in which we need to create Resources.
For a resource we will define its attributes respectively. We need to define on which node,this particular service group should be online. Sometimes according to requirement need to make same service group on both the nodes online at a time.
For this sharing, we use CFS (Cluster FileSystem). Latest Veritas Cluster version : 6.1 VXSFCFSHA.
Three types of clusters:
1) Failover
2) Parallel
3) Hybrid
Failover is one in which when any of the node gets down then we switch particular service group onto the other node. In this case, we say when primary node is down we switch application to failover node.
Parallel is when all the service groups are online on all nodes. Application will be online all the time with zero downtime.
Hybrid is a combination of both Failover and Parallel. It means some service groups will be shared and online on both nodes and some will be switched across nodes whenever a failover occurs.
Three types of Resources:
1) ON-Only
2) ON-OFF
3) Persistent
On-Only
We can only start these resources through VCS, but does not stop them.
For example, VCS requires NFS daemons to be running to export a file system.
VCS starts the daemons if required, but does not stop them if the associated service group is taken offline.
On-Off
We can start and stop On-Off resources as required. For example, VCS imports a disk group when required and deports it when it is no longer needed.
Persistent
These resources cannot be brought online or taken offline.
For example, a network interface card cannot be started or stopped, but it is required to configure an IP address. Failure of a Persistent resource triggers a service group failover.
Attribute and Resource Type:
A resource type is one which states the purpose of a resource by its naming.
For example, Mount Volume Diskgroup Oracle SAPNW04 NFSRestart IP NIC are resources.
Attribute is the value which helps a resource to act according to its type.
For example, resource type Mount will have Attributes like mount point,fstyp,fsck options,block device path.
Low Latency Transmit Protocol (LLT) :
The main purpose of LLT is to transmit heartbeats. It checks the heartbeats between
the nodes in a cluster at time intervals (0.5 sec on high link and 1 sec on low link).
/etc/llthosts file is responsible to specify the hostnames of both nodes.
Start/Stop LLT
# lltconfig -c -> start LLT
# lltconfig -U -> stop LLT (GAB needs to stopped first)
Global Atomic Broadcast (GAB) :
Stands for Group membership services and atomic broadcast.
Group membership services : It tracks the heartbeats sent over LLT. If any nodes fails to send the heartbeat over LLT the GAB module send the information to I/O fencing module to take further action to avoid any split brain condition.
Atomic Broadcast : atomic broadcast ensures that every node in the cluster has same information about every resource and service group in the cluster.
# cat /etc/gabtab
/sbin/gabconfig -c -n 2 ==== command to start the GAB. " -n 2 " -minimum no of nodes required to communicate before starting VCS.
Start/Stop GAB
# gabconfig -c -> start GAB
# gabconfig -U -> stop GAB
High Availability Daemon (HAD) :
HAD, high availability daemon is the main daemon which manages the agents and service group.
hashadow daemon is responsible for this. HAD maintains the resource configuration and state information.
Start/Stop HAD
# hastart -> start HAD
# hastop -> stop HAD comes with many options, "-all" stops HAD in all nodes. "-local" for a single node.
Jeopardy and Split Brain Condition :
When a node in the cluster has only the last LLT link intact, the node forms a Jeopardy membership with that node and regular membership with nodes which has more than one LLT. Hence we achieve a regular membership among all nodes.
Coming to splitbrain condition,
Split brain occurs when all the LLT links fails simultaneously. A particular node fail to identify whether it is a system failure or an interconnect failure.
Each node thinks that it is the only node which is active at the moment and tries to start the service groups on the other node which he think is down.
Same thing happens to the other node and this may lead to a simultaneous access to the storage and can cause data corruption.
I/O Fencing :
In Splitbrain condition to avoid data corruption, we use I/O fencing concept. I/O fencing driver uses SCSI-3 PGR (persistent group reservations) to avoid the data corruption.
In case of a possible split brain scenario, each node tries to access storage. I/O fencing helps to avoid this, and provides the given disks (Quorum disks) to both nodes for writing its data.
A cluster can be build with 2 nodes atleast and a maximum of 32nodes. While using VCS we need to create Service groups and in which we need to create Resources.
For a resource we will define its attributes respectively. We need to define on which node,this particular service group should be online. Sometimes according to requirement need to make same service group on both the nodes online at a time.
For this sharing, we use CFS (Cluster FileSystem). Latest Veritas Cluster version : 6.1 VXSFCFSHA.
Three types of clusters:
1) Failover
2) Parallel
3) Hybrid
Failover is one in which when any of the node gets down then we switch particular service group onto the other node. In this case, we say when primary node is down we switch application to failover node.
Parallel is when all the service groups are online on all nodes. Application will be online all the time with zero downtime.
Hybrid is a combination of both Failover and Parallel. It means some service groups will be shared and online on both nodes and some will be switched across nodes whenever a failover occurs.
Three types of Resources:
1) ON-Only
2) ON-OFF
3) Persistent
On-Only
We can only start these resources through VCS, but does not stop them.
For example, VCS requires NFS daemons to be running to export a file system.
VCS starts the daemons if required, but does not stop them if the associated service group is taken offline.
On-Off
We can start and stop On-Off resources as required. For example, VCS imports a disk group when required and deports it when it is no longer needed.
Persistent
These resources cannot be brought online or taken offline.
For example, a network interface card cannot be started or stopped, but it is required to configure an IP address. Failure of a Persistent resource triggers a service group failover.
Attribute and Resource Type:
A resource type is one which states the purpose of a resource by its naming.
For example, Mount Volume Diskgroup Oracle SAPNW04 NFSRestart IP NIC are resources.
Attribute is the value which helps a resource to act according to its type.
For example, resource type Mount will have Attributes like mount point,fstyp,fsck options,block device path.
Low Latency Transmit Protocol (LLT) :
The main purpose of LLT is to transmit heartbeats. It checks the heartbeats between
the nodes in a cluster at time intervals (0.5 sec on high link and 1 sec on low link).
/etc/llthosts file is responsible to specify the hostnames of both nodes.
Start/Stop LLT
# lltconfig -c -> start LLT
# lltconfig -U -> stop LLT (GAB needs to stopped first)
Global Atomic Broadcast (GAB) :
Stands for Group membership services and atomic broadcast.
Group membership services : It tracks the heartbeats sent over LLT. If any nodes fails to send the heartbeat over LLT the GAB module send the information to I/O fencing module to take further action to avoid any split brain condition.
Atomic Broadcast : atomic broadcast ensures that every node in the cluster has same information about every resource and service group in the cluster.
# cat /etc/gabtab
/sbin/gabconfig -c -n 2 ==== command to start the GAB. " -n 2 " -minimum no of nodes required to communicate before starting VCS.
Start/Stop GAB
# gabconfig -c -> start GAB
# gabconfig -U -> stop GAB
High Availability Daemon (HAD) :
HAD, high availability daemon is the main daemon which manages the agents and service group.
hashadow daemon is responsible for this. HAD maintains the resource configuration and state information.
Start/Stop HAD
# hastart -> start HAD
# hastop -> stop HAD comes with many options, "-all" stops HAD in all nodes. "-local" for a single node.
Jeopardy and Split Brain Condition :
When a node in the cluster has only the last LLT link intact, the node forms a Jeopardy membership with that node and regular membership with nodes which has more than one LLT. Hence we achieve a regular membership among all nodes.
Coming to splitbrain condition,
Split brain occurs when all the LLT links fails simultaneously. A particular node fail to identify whether it is a system failure or an interconnect failure.
Each node thinks that it is the only node which is active at the moment and tries to start the service groups on the other node which he think is down.
Same thing happens to the other node and this may lead to a simultaneous access to the storage and can cause data corruption.
I/O Fencing :
In Splitbrain condition to avoid data corruption, we use I/O fencing concept. I/O fencing driver uses SCSI-3 PGR (persistent group reservations) to avoid the data corruption.
In case of a possible split brain scenario, each node tries to access storage. I/O fencing helps to avoid this, and provides the given disks (Quorum disks) to both nodes for writing its data.
###############################################################################
