VXSFCFSHA 6.1 Installation !!!

In this post, let us discuss the Veritas Cluster Installation procedure including I/O fencing.Now we are going to see installation of VXSFCFSHA (Veritas Storage Foundation Cluster Filesystem with High Availability) .

Version : VXSFHA 6.1
Cluster : 2 node

Unzip the cluster software and start the installation :

root@solaris:/tmp/dvd1-sol_sparc/sol11_sparc#
root@solaris:/tmp/dvd1-sol_sparc/sol11_sparc# ./installer

                     Symantec Storage Foundation and High Availability Solutions 6.1 Install Program

Copyright (c) 2013 Symantec Corporation. All rights reserved.  Symantec, the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.

The Licensed Software and Documentation are deemed to be "commercial computer software" and "commercial computer software documentation" as defined in FAR Sections 12.212 and DFARS Section 227.7202.

Logs are being written to /var/tmp/installer-201405031548hnG while installer is in progress.

                     Symantec Storage Foundation and High Availability Solutions 6.1 Install Program

Symantec Product                       Version Installed on solaris    Licensed
=======================================================================
Symantec Licensing Utilities (VRTSvlic) are not installed due to which products and licenses are not discovered. Use the menu below to continue.

Task Menu:

    P) Perform a Pre-Installation Check       I) Install a Product
    C) Configure an Installed Product         G) Upgrade a Product
    O) Perform a Post-Installation Check      U) Uninstall a Product
    L) License a Product                      S) Start a Product
    D) View Product Descriptions              X) Stop a Product
    R) View Product Requirements              ?) Help

Enter a Task: [P,I,C,G,O,U,L,S,D,X,R,?] i

                     Symantec Storage Foundation and High Availability Solutions 6.1 Install Program

     1)  Symantec Dynamic Multi-Pathing (DMP)
     2)  Symantec Cluster Server (VCS)
     3)  Symantec Storage Foundation (SF)
     4)  Symantec Storage Foundation and High Availability (SFHA)
     5)  Symantec Storage Foundation Cluster File System HA (SFCFSHA)
     6)  Symantec Storage Foundation for Oracle RAC (SF Oracle RAC)
     b)  Back to previous menu

Select a product to install: [1-6,b,q] 5

This Symantec product may contain open source and other third party materials that are subject to a separate license. See the applicable Third-Party Notice at http://www.symantec.com/about/profile/policies/eulas

Do you agree with the terms of the End User License Agreement as specified in the
storage_foundation_cluster_file_system_ha/EULA/en/EULA_SFHA_Ux_6.1.pdf file present on media? [y,n,q,?] y 

                          Symantec Storage Foundation Cluster File System HA 6.1 Install Program

     1)  Install minimal required packages - 661 MB required
     2)  Install recommended packages - 800 MB required
     3)  Install all packages - 829 MB required
     4)  Display packages to be installed for each option

Select the packages to be installed on all systems? [1-4,q,?] (2) 3

Enter the Solaris 11 Sparc system names separated by spaces: [q,?] solaris solaris2

                          Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                     solaris solaris2

Logs are being written to /var/tmp/installer-201405031548hnG while installer is in progress

    Verifying systems: 100%

    Estimated time remaining: (mm:ss) 0:00                                                                      8 of 8

    Checking system communication ............................................................................... Done
    Checking release compatibility .............................................................................. Done
    Checking installed product .................................................................................. Done
    Checking prerequisite patches and packages .................................................................. Done
    Checking platform version ................................................................................... Done
    Checking file system free space ............................................................................. Done
    Checking product licensing .................................................................................. Done
    Performing product prechecks ................................................................................ Done

System verification checks completed successfully

                          Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                     solaris solaris2

The following Symantec Storage Foundation Cluster File System HA packages will be installed on all systems:

Package           Version              Package Description
VRTSllt           6.1.0.0              Low Latency Transport
VRTSgab           6.1.0.0              Group Membership and Atomic Broadcast
VRTSvxfen         6.1.0.0              I/O Fencing
VRTSamf           6.1.0.0              Asynchronous Monitoring Framework
VRTSvcs           6.1.0.0              Cluster Server
VRTScps           6.1.0.0              Cluster Server - Coordination Point Server
VRTSvcsag         6.1.0.0              Cluster Server Bundled Agents
VRTSvcsea         6.1.0.0              Cluster Server Enterprise Agents
VRTSglm           6.1.0.0              Group Lock Manager
VRTScavf          6.1.0.0              Cluster Server Agents for Cluster File System
VRTSgms           6.1.0.0              Group Messaging Services
VRTSvbs           6.1.0.0              Virtual Business Service
VRTSvcswiz        6.1.0.0              Cluster Server Wizards

The following Symantec Storage Foundation Cluster File System HA packages will be installed on solaris:

Package           Version              Package Description
VRTSperl          5.16.1.6             Perl Redistribution
VRTSvlic          3.2.61.10            Licensing
VRTSsfcpi61       6.1.0.0              Storage Foundation Installer
VRTSspt           6.1.0.0              Software Support Tools
VRTSvxvm          6.1.0.0              Volume Manager Binaries
VRTSaslapm        6.1.0.0              Volume Manager - ASL/APM
VRTSsfmh          6.0.0.0              Storage Foundation Managed Host
VRTSvxfs          6.1.0.0              File System
VRTSfsadv         6.1.0.0              File System Advanced Solutions
VRTSfssdk         6.1.0.0              File System Software Developer Kit
VRTSdbed          6.1.0.0              Storage Foundation Databases

Press [Enter] to continue:
VRTSodm           6.1.0.0              Oracle Disk Manager

Press [Enter] to continue:

                          Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                     solaris solaris2

Logs are being written to /var/tmp/installer-201405031548hnG while installer is in progress

    Installing SFCFSHA: 100%

    Estimated time remaining: (mm:ss) 0:00                                                                      3 of 3

    Performing SFCFSHA preinstall tasks ......................................................................... Done
    Installing SFCFSHA packages ................................................................................. Done
    Performing SFCFSHA postinstall tasks ........................................................................ Done

Symantec Storage Foundation Cluster File System HA Install completed successfully

                          Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                     solaris solaris2

To comply with the terms of Symantec's End User License Agreement, you have 60 days to either:

* Enter a valid license key matching the functionality in use on the systems
* Enable keyless licensing and manage the systems with a Management Server. For more details visit http://go.symantec.com/sfhakeyless. The product is fully functional during these 60 days.

     1)  Enter a valid license key
     2)  Enable keyless licensing and complete system licensing later

How would you like to license the systems? [1-2,q] (2) 1

Checking system licensing

SFCFSHA is not licensed on solaris

SFCFSHA is not licensed on solaris2

SFCFSHA is unlicensed on all systems

Enter a SFCFSHA license key: [b,q,?] AJDE-WVQC-NP2Z-FDTB-GTJZ-8APH-R680-404C-P

Storage Foundation for Cluster File System successfully registered on solaris
File System successfully registered on solaris2

Storage Foundation for Cluster File System successfully registered on solaris2
Do you wish to enter additional licenses? [y,n,q,b] (n) n

Would you like to configure SFCFSHA on solaris solaris2? [y,n,q] (n) y

I/O Fencing

It needs to be determined at this time if you plan to configure I/O Fencing in enabled or disabled mode, as well as help in determining the number of network interconnects (NICS) required on your systems. If you configure I/O Fencing in enabled mode, only a single NIC is required, though at least two are recommended.

A split brain can occur if servers within the cluster become unable to communicate for any number of reasons. If I/O Fencing is not enabled, you run the risk of data corruption should a split brain occur. Therefore, to avoid data corruption due to split brain in CFS environments, I/O Fencing has to be enabled.

If you do not enable I/O Fencing, you do so at your own risk

See the Administrator's Guide for more information on I/O Fencing

Do you want to configure I/O Fencing in enabled mode? [y,n,q,?] (y) y

                          Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                     solaris solaris2

To configure VCS, answer the set of questions on the next screen.

When [b] is presented after a question, 'b' may be entered to go back to the first question of the configuration set.

When [?] is presented after a question, '?' may be entered for help or additional information about the question.

Following each set of questions, the information you have entered will be presented for confirmation.  To repeat the set of questions and correct any previous errors, enter 'n' at the confirmation prompt.

No configuration changes are made to the systems until all configuration questions are completed and confirmed.

Press [Enter] to continue:

                          Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                     solaris solaris2

To configure VCS for SFCFSHA the following information is required:

  A unique cluster name
  Two or more NICs per system used for heartbeat links
  A unique cluster ID number between 0-65535

  One or more heartbeat links are configured as private links
  You can configure one heartbeat link as a low-priority link

All systems are being configured to create one cluster.

Enter the unique cluster name: [q,?] MYSOL

                          Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                     solaris solaris2

     1)  Configure the heartbeat links using LLT over Ethernet
     2)  Configure the heartbeat links using LLT over UDP
     3)  Automatically detect configuration for LLT over Ethernet
     b)  Back to previous menu

How would you like to configure heartbeat links? [1-3,b,q,?] (3) 1

Discovering NICs on solaris ............................................. Discovered net0 net1 net2 net3 net4

Enter the NIC for the first private heartbeat link on solaris: [b,q,?] (net0) net3

Would you like to configure a second private heartbeat link? [y,n,q,b,?] (n) y

Enter the NIC for the second private heartbeat link on solaris: [b,q,?] (net0) net4

Would you like to configure a third private heartbeat link? [y,n,q,b,?] (n)

Do you want to configure an additional low-priority heartbeat link? [y,n,q,b,?] (n) y

Enter the NIC for the low-priority heartbeat link on solaris: [b,q,?] (net2) net0

Are you using the same NICs for private heartbeat links on all systems? [y,n,q,b,?] (y)
    Checking media speed for net3 on solaris .............................. Not Applicable (Virtual Device)
    Checking media speed for net4 on solaris .............................. Not Applicable (Virtual Device)
    Checking media speed for net3 on solaris2 ............................ Not Applicable (Virtual Device)
    Checking media speed for net4 on solaris2 ............................ Not Applicable (Virtual Device)

Enter a unique cluster ID number between 0-65535: [b,q,?] (50864) 64325

The cluster cannot be configured if the cluster ID 64325 is in use by another cluster. Installer can perform a check to determine if the cluster ID is duplicate. The check will take less than a minute to complete.

Would you like to check if the cluster ID is in use by another cluster? [y,n,q] (y) y

    Checking cluster ID ......................................................................................... Done

Duplicated cluster ID detection passed. The cluster ID 64325 can be used for the cluster.

Press [Enter] to continue:

                          Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                     solaris solaris2

Cluster information verification:

        Cluster Name:      MYSOL
        Cluster ID Number: 64325

        Private Heartbeat NICs for solaris:
                link1=net0
        Low-Priority Heartbeat NIC for solaris:
                link-lowpri1=net1

        Private Heartbeat NICs for solaris2:
                link1=net4
        Low-Priority Heartbeat NIC for solaris2:
                link-lowpri1=net2

Is this information correct? [y,n,q,?] (y) y

                          Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                     solaris solaris2

The following data is required to configure the Virtual IP of the Cluster:

        A public NIC used by each system in the cluster
        A Virtual IP address and netmask

Do you want to configure the Virtual IP? [y,n,q,?] (n) n

                          Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                     solaris solaris2

Symantec Cluster Server can be configured in secure mode

Running VCS in Secure Mode guarantees that all inter-system communication is encrypted, and users are verified with security credentials.

When running VCS in Secure Mode, NIS and system usernames and passwords are used to verify identity. VCS usernames and passwords are no longer utilized when a cluster is running in Secure Mode.

Would you like to configure the VCS cluster in secure mode? [y,n,q,?] (n) n

                          Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                     solaris solaris2

The following information is required to add VCS users:

        A user name
        A password for the user
        User privileges (Administrator, Operator, or Guest)

Do you wish to accept the default cluster credentials of 'admin/password'? [y,n,q] (y) y

Do you want to add another user to the cluster? [y,n,q] (n) n

                       Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                     solaris solaris2

VCS User verification:

        User: admin         Privilege: Administrators
        Passwords are not displayed

Is this information correct? [y,n,q] (y) y

                          Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                     solaris solaris2

The following information is required to configure SMTP notification:

        The domain-based hostname of the SMTP server
        The email address of each SMTP recipient
        A minimum severity level of messages to send to each recipient

Do you want to configure SMTP notification? [y,n,q,?] (n) n

                          Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                     solaris solaris2

The following information is required to configure SNMP notification:

        System names of SNMP consoles to receive VCS trap messages 
        SNMP trap daemon port numbers for each console
        A minimum severity level of messages to send to each console

Do you want to configure SNMP notification? [y,n,q,?] (n) n

All SFCFSHA processes that are currently running must be stopped

Do you want to stop SFCFSHA processes now? [y,n,q,?] (y) y

                          Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                     solaris solaris2

Logs are being written to /var/tmp/installer-201405031548hnG while installer is in progress

    Stopping SFCFSHA: 100%

    Estimated time remaining: (mm:ss) 0:00                                                                    10 of 10

    Performing SFCFSHA prestop tasks ............................................................................ Done
    Stopping vxgms .............................................................................................. Done
    Stopping vxglm .............................................................................................. Done
    Stopping vxcpserv ........................................................................................... Done
    Stopping had ................................................................................................ Done
    Stopping CmdServer .......................................................................................... Done
    Stopping amf ................................................................................................ Done
    Stopping vxfen .............................................................................................. Done
    Stopping gab ................................................................................................ Done
    Stopping llt ................................................................................................ Done

Symantec Storage Foundation Cluster File System HA Shutdown completed successfully

                          Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                     solaris solaris2

Logs are being written to /var/tmp/installer-201405031548hnG while installer is in progress

    Starting SFCFSHA: 100%

    Estimated time remaining: (mm:ss) 0:00                                                                    23 of 23

    Starting vxio ............................................................................................... Done
    Starting vxspec ............................................................................................. Done
    Starting vxconfigd .......................................................................................... Done
    Starting vxesd .............................................................................................. Done
    Starting vxrelocd ........................................................................................... Done
    Starting vxcached ........................................................................................... Done
    Starting vxconfigbackupd .................................................................................... Done
    Starting vxattachd .......................................................................................... Done
    Starting vxportal ........................................................................................... Done
    Starting fdd ................................................................................................ Done
    Starting llt ................................................................................................ Done
    Starting gab ................................................................................................ Done
    Starting vxfen .............................................................................................. Done
    Starting amf ................................................................................................ Done
    Starting vxglm .............................................................................................. Done
    Starting had ................................................................................................ Done
    Starting CmdServer .......................................................................................... Done
    Starting vxdbd .............................................................................................. Done
    Starting vxgms .............................................................................................. Done
    Starting odm ................................................................................................ Done
    Performing SFCFSHA poststart tasks .......................................................................... Done

Symantec Storage Foundation Cluster File System HA Startup completed successfully

                                Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                                            solaris solaris2

Fencing configuration
     1)  Configure Coordination Point client based fencing
     2)  Configure disk based fencing

Select the fencing mechanism to be configured in this Application Cluster: [1-2,q,?] 2

This I/O fencing configuration option requires a restart of VCS. Installer will stop VCS at a later stage in this run. Note that the service groups will be online only on the systems that are in the 'AutoStartList' after restarting VCS. Do you want to continue? [y,n,q,b,?] y

Do you have SCSI3 PR enabled disks? [y,n,q,b,?] (y)

Since you have selected to configure disk based fencing, you need to provide the existing disk group to be used as coordinator or create a new disk group for it.

Select one of the options below for fencing disk group:
     1)  Create a new disk group
     2)  Using an existing disk group
     b)  Back to previous menu

Enter the choice for a disk group: [1-2,b,q] 1

List of available disks to create a new disk group
A new disk group cannot be created as the number of available free VxVM CDS disks is 0 which is less than three. If there are disks available which are not under VxVM control, use the command vxdisksetup or use the installer to initialize them as VxVM disks.

Do you want to initialize more disks as VxVM disks? [y,n,q,b] (y)

List of disks which can be initialized as VxVM disks:
     1)  c3d0s2
     2)  c3d1s2
     3)  emc0_1b47
     4)  emc0_1b48
     5)  emc0_1b49
     b)  Back to previous menu

Enter the disk options, separated by spaces: [1-5,b,q] 3 4 5
    Intializing disk emc0_1b47 on solaris ................................................................. Done
    Intializing disk emc0_1b48 on solaris ................................................................. Done
    Intializing disk emc0_1b49 on solaris ................................................................. Done

     1)  emc0_1b47
     2)  emc0_1b48
     3)  emc0_1b49
     b)  Back to previous menu

Select odd number of disks and at least three disks to form a disk group. Enter the disk options, separated by spaces: [1-3,b,q] 1 2 3

Enter the new disk group name: [b] TEST_QUORUM
Created disk group TEST_QUORUM

Before you continue with configuration, Symantec recommends that you run the vxfentsthdw utility (I/O fencing test hardware utility), in a separate console, to test whether the shared storage supports I/O fencing.  You can access the utility at '/opt/VRTSvcs/vxfen/bin/vxfentsthdw'.

As per the 'vxfentsthdw' run you performed, do you want to continue with this disk group? [y,n,q] (y) May  5 16:05:05 QA10 sendmail[26588]: [ID 702911 mail.alert] daemon MTA: problem creating SMTP socket
May  5 16:05:55 QA10 last message repeated 10 times
May  5 16:05:55 QA10 sendmail[26588]: [ID 801593 mail.alert] NOQUEUE: SYSERR(root): opendaemonsocket: daemon MTA: server SMTP socket wedged: exiting


Using disk group TEST_QUORUM

Enter disk policy for the disk(s) (raw/dmp): [b,q,?] raw

                       Symantec Storage Foundation Cluster File System HA 6.1 Install Program
                                                                            solaris solaris2

I/O fencing configuration verification

        Disk Group: TEST_QUORUM
        Fencing disk policy: raw

Is this information correct? [y,n,q] (y)

Installer will stop VCS before applying fencing configuration. To make sure VCS shuts down successfully, unfreeze any frozen service group and unmount the mounted filesystems in the cluster.

Are you ready to stop VCS and apply fencing configuration on all nodes at this time? [y,n,q] (y)

    Stopping VCS on solaris2 ............................................................................... Done
    Stopping Fencing on solaris2 .......................................................................... Done
    Stopping VCS on solaris ................................................................................. Done
    Stopping Fencing on solaris ............................................................................ Done
    Starting Fencing on solaris ............................................................................. Done
    Starting Fencing on solaris2 ........................................................................... Done
    Updating main.cf with fencing ........................................................................ Done
    Starting VCS on solaris .................................................................................. Done
    Starting VCS on solaris2 ................................................................................ Done

The Coordination Point Agent monitors the registrations on the coordination points.
Do you want to configure Coordination Point Agent on the client cluster? [y,n,q] (y)
Enter a non-existing name for the service group for Coordination Point Agent: [b] (vxfen)

Additionally the Coordination Point Agent can also monitor changes to the Coordinator Disk Group constitution such as a disk being accidently deleted from the Coordinator Disk Group. The frequency of this detailed monitoring can be tuned with the LevelTwoMonitorFreq attribute.

For example, if you set this attribute to 5, the agent will monitor the Coordinator Disk Group constitution every five monitor cycles. If LevelTwoMonitorFreq attribute is not set, the agent will not monitor any changes to the Coordinator Disk Group.

Do you want to set LevelTwoMonitorFreq? [y,n,q] (y)
Enter the value of the LevelTwoMonitorFreq attribute(0 to 65535): [b,q,?] (5) 50

 Adding Coordination Point Agent via solaris ........................................................... Done

 I/O Fencing configuration ......................................................................................... Done

I/O Fencing configuration completed successfully

The updates to VRTSaslapm package are released via the Symantec SORT web page: https://sort.symantec.com/asl. To make sure you have the latest version of VRTSaslapm
(for up to date ASLs and APMs), download and install the latest package from the SORT web page.

Checking online updates for Symantec Storage Foundation Cluster File System HA 6.1

    A connection attempt to https://sort.symantec.com to check for product updates failed.
    Visit https://sort.symantec.com to check for available product updates and information.

installer log files, summary file, and response file are saved at:

        /opt/VRTS/install/logs/installer-201405051543ein

Would you like to view the summary file? [y,n,q] (n)
root@solaris:/tmp/dvd1-sol_sparc/sol11_sparc#
root@solaris:/tmp/dvd1-sol_sparc/sol11_sparc#
root@solaris:/tmp/dvd1-sol_sparc/sol11_sparc#
root@solaris:/tmp/dvd1-sol_sparc/sol11_sparc#
root@solaris:/tmp/dvd1-sol_sparc/sol11_sparc# pkg info VRTSvcs
          Name: VRTSvcs
       Summary: Veritas Cluster Server by Symantec
   Description: The package contains Veritas Cluster Server by Symantec
      Category: Applications/System Utilities
         State: Installed
     Publisher: Symantec
       Version: 6.1.0.0
 Build Release: 5.11
        Branch: None
Packaging Date: October 21, 2013 08:38:13 PM
          Size: 236.57 MB
          FMRI: pkg://Symantec/VRTSvcs@6.1.0.0,5.11:20131021T203813Z
root@solaris:/tmp/dvd1-sol_sparc/sol11_sparc#

By this we completed the installation of Cluster Software and now it is time to rock n roll with Cluster configuration in Veritas Cluster Console (VCC), GUI.

###############################################################################

Veritas Cluster Concept !!!

Veritas Cluster is responsible to provide high availability for a Application with a minimum downtime. High availability clusters (HAC) improve application availability by failing them over or switching them over in a group of systems.

A cluster can be build with 2 nodes atleast and a maximum of 32nodes. While using VCS we need to create Service groups and in which we need to create Resources.

For a resource we will define its attributes respectively. We need to define on which node,this particular service group should be online. Sometimes according to requirement need to make same service group on both the nodes online at a time.

For this sharing, we use CFS (Cluster FileSystem). Latest Veritas Cluster version : 6.1 VXSFCFSHA.

Three types of clusters:

1) Failover
2) Parallel
3) Hybrid

Failover is one in which when any of the node gets down then we switch particular service group onto the other node. In this case, we say when primary node is down we switch application to failover node.

Parallel is when all the service groups are online on all nodes. Application will be online all the time with zero downtime.

Hybrid is a combination of both Failover and Parallel. It means some service groups will be shared and online on both nodes and some will be switched across nodes whenever a failover occurs.

Three types of Resources:

1) ON-Only
2) ON-OFF
3) Persistent

On-Only

We can only start these resources through VCS, but does not stop them.
For example, VCS requires NFS daemons to be running to export a file system. 
VCS starts the daemons if required, but does not stop them if the associated service group is taken offline.

On-Off

We can start and stop On-Off resources as required. For example, VCS imports a disk group when required and deports it when it is no longer needed.

Persistent

These resources cannot be brought online or taken offline. 
For example, a network interface card cannot be started or stopped, but it is required to configure an IP address. Failure of a Persistent resource triggers a service group failover.

Attribute and Resource Type:

A resource type is one which states the purpose of a resource by its naming.
For example, Mount Volume Diskgroup Oracle SAPNW04 NFSRestart IP NIC are resources.

Attribute is the value which helps a resource to act according to its type.
For example, resource type Mount will have Attributes like mount point,fstyp,fsck options,block device path.

Low Latency Transmit Protocol (LLT) :

The main purpose of LLT is to transmit heartbeats. It checks the heartbeats between
the nodes in a cluster at time intervals (0.5 sec on high link and 1 sec on low link).
/etc/llthosts file is responsible to specify the hostnames of both nodes.

Start/Stop LLT

# lltconfig -c       -> start LLT
# lltconfig -U       -> stop LLT (GAB needs to stopped first)

Global Atomic Broadcast (GAB) :

Stands for Group membership services and atomic broadcast.

Group membership services : It tracks the heartbeats sent over LLT. If any nodes fails to send the heartbeat over LLT the GAB module send the information to I/O fencing module to take further action to avoid any split brain condition. 

Atomic Broadcast : atomic broadcast ensures that every node in the cluster has same information about every resource and service group in the cluster.

# cat /etc/gabtab
/sbin/gabconfig -c -n 2       ==== command to start the GAB.  " -n 2 "  -minimum no of nodes required to communicate before starting VCS.

Start/Stop GAB

# gabconfig -c        -> start GAB
# gabconfig -U       -> stop GAB

High Availability Daemon (HAD) :

HAD, high availability daemon is the main daemon which manages the agents and service group.
hashadow daemon is responsible for this. HAD maintains the resource configuration and state information.

Start/Stop HAD

# hastart            -> start HAD
# hastop             -> stop HAD   comes with many options, "-all" stops HAD in all nodes. "-local" for a single node.

Jeopardy and Split Brain Condition :

When a node in the cluster has only the last LLT link intact, the node forms a Jeopardy membership with that node and regular membership with nodes which has more than one LLT. Hence we achieve a regular membership among all nodes.

Coming to splitbrain condition,

Split brain occurs when all the LLT links fails simultaneously. A particular node fail to identify whether it is a system failure or an interconnect failure. 

Each node thinks that it is the only node which is active at the moment and tries to start the service groups on the other node which he think is down.

Same thing happens to the other node and this may lead to a simultaneous access to the storage and can cause data corruption.

I/O Fencing :

In Splitbrain condition to avoid data corruption, we use I/O fencing concept. I/O fencing driver uses SCSI-3 PGR (persistent group reservations) to avoid the data corruption.

In case of a possible split brain scenario, each node tries to access storage. I/O fencing helps to avoid this, and provides the given disks (Quorum disks) to both nodes for writing its data. 

###############################################################################

Veritas Netbackup Commands !!!

It is better to gain some knowledge in Veritas Netbackup at command level.
Though majority of the work in VNB carried out through GUI, it will be helpful if we have some command knowledge.

Few basic and commonly used commands of Netbackup are :

1. available_media      ------ > To view availability of media
2. robtest                       ------ > To instruct robot manually
3. bpmedialist -p <poolname>   ------ > To view medias assigned to a pool
4. bpexpdate -m <media> -d 0    ------ > To scratch a media
5. bpmedia -unfreeze/freeze <media>  ------ > To unfreeze/freeze a media
6. bpdbjobs -report      ------ > To list all netbackups jobs
7. vmpool -listall           ------ > To list all pools
8. vmquery -m <media>    ----- > To list tape volume details
9. vmchange -exp 12/31/06 23:59:58 -m <media ID>   ----- > Change a tapes expiry date 
10.vmchange -p <pool number> -m <media ID>   ----- > Change a tape's media pool 

All the above listed commands runs from Master Server only.Command to start particular backup from a particular media server should be run in Media Server.

bpbackup -p "policyname" -s "UserBackup" -L "progress log location" -S "master server"

Periodically we need to test our robotic arm, for this we use robtest command and through this command we can manually move slots to drives and slots from drives.

root@MSTSRVR # robtest
1Configured robots with local control supporting test utilities:
  TLD(0)     robotic path = /dev/sg/c0tw500104f0009b5ea3l0

Robot Selection
---------------
  1)  TLD 0
  2)  none/quit
Enter choice: 1                      ------- To access tape library

Robot selected: TLD(0)   robotic path = /dev/sg/c0tw500104f0009b5ea3l0

Invoking robotic test utility:
/usr/openv/volmgr/bin/tldtest -rn 0 -r /dev/sg/c0tw500104f0009b5ea3l0

Opening /dev/sg/c0tw500104f0009b5ea3l0
MODE_SENSE complete
Enter tld commands (? returns help information)
s d
drive 1 (addr 500) access = 0 Contains Cartridge = yes
Source address = 1093 (slot 94)
Barcode = RS1012
drive 2 (addr 501) access = 0 Contains Cartridge = yes
Source address = 1032 (slot 33)
Barcode = L21768
drive 3 (addr 502) access = 0 Contains Cartridge = yes
Source address = 1200 (slot 201)
Barcode = FJ0301
drive 4 (addr 503) access = 0 Contains Cartridge = yes
Source address = 1034 (slot 35)
Barcode = L21752
drive 5 (addr 504) access = 1 Contains Cartridge = no
drive 6 (addr 505) access = 1 Contains Cartridge = no
drive 7 (addr 506) access = 1 Contains Cartridge = no
drive 8 (addr 507) access = 1 Contains Cartridge = no
drive 9 (addr 508) access = 1 Contains Cartridge = no
drive 10 (addr 509) access = 1 Contains Cartridge = no
READ_ELEMENT_STATUS complete
q

Robot Selection
---------------
  1)  TLD 0
  2)  none/quit
Enter choice:
root@MSTSRVR #

To check the availability of media, displays all pools media.

root@MSTSRVR # available_media | more
media   media   robot   robot   robot   side/   ret    size     status/
 ID     type    type      #     slot    face    level  KBytes    multiplexed
----------------------------------------------------------------------------
MYSAPSRV_M02_FRI pool

CI0688  HCART3   TLD      0      111      -       0   63628256     ACTIVE

MYSAPSRV_M02_MON pool

CI0684  HCART3   TLD      0       53      -       0   63636416     ACTIVE

MYSAPSRV_M02_SAT pool

CI0689  HCART3   TLD      0      110      -       0   63602944     ACTIVE

MYSAPSRV_M02_SUN pool

CI0690  HCART3   TLD      0      109      -       0   63646112     ACTIVE

MYSAPSRV_M02_THU pool

CI0687  HCART3   TLD      0      112      -       0   63611488     ACTIVE

MYSAPSRV_M02_TUE pool

CI0685  HCART3   TLD      0       52      -       0   63601472     ACTIVE

MYSAPSRV_M02_WED pool

--More--          OUTPUT TRUNCATED

root@MSTSRVR #

To view medias assigned to a particular pool.

root@MSTSRVR # bpmedialist -p Oraclesrvr_M01_DAILY
Server Host = MED1SRVR

 id     rl  images   allocated        last updated      density  kbytes restores
           vimages   expiration       last read         <------- STATUS ------->
           On Hold
--------------------------------------------------------------------------------
077100   0      2   11/26/2014 08:10  11/26/2014 08:10  hcart3  1237892608     0
                2   12/03/2014 08:10        N/A         FULL
           0

FJ0302   0      2   11/23/2014 14:04  11/23/2014 14:04  hcart3  1234659328     0
                2   11/30/2014 14:04        N/A         FULL
           0

FJ0303   0      3   11/24/2014 18:39  11/25/2014 11:26  hcart3  1208266624     0
                3   12/02/2014 11:26        N/A         FULL
           0

L21752   0      0   11/29/2014 08:38  11/29/2014 08:38  hcart3           0     0
                0   12/06/2014 08:38        N/A       
           0

L21753   0      2   11/24/2014 18:39  11/24/2014 18:39  hcart3  1349768704     0
                2   12/01/2014 18:39        N/A         FULL
           0

$$$$$$$$     &&&     OUTPUT TRUNCATED     &&&   $$$$$$$$$$$$


RS1012   0      0   11/29/2014 08:38  11/29/2014 08:38  hcart3           0     0
                0   12/06/2014 08:38        N/A       
           0
root@MSTSRVR #

Another command which displays media details in other form, here we can get info for a particular schedule. Below is my server backup friday schedule's media info :

root@MSTSRVR # vmquery -pn MYSAPSRV_M02_FRI
================================================================================
media ID:              CI0688
media type:            1/2" cartridge tape 3 (24)
barcode:               CI0688
media description:     Added by Media Manager
volume pool:           MYSAPSRV_M02_FRI (91)
robot type:            TLD - Tape Library DLT (8)
robot number:          0
robot slot:            111
robot control host:    MSTSRVR
volume group:          000_00000_TLD
vault name:            ---
vault sent date:       ---
vault return date:     ---
vault slot:            ---
vault session id:      ---
vault container id:    -
created:               Wed Oct 08 19:58:15 2008
assigned:              Fri Nov 28 02:56:24 2014
last mounted:          Fri Nov 28 02:57:21 2014
first mount:           Sat Mar 28 00:37:54 2009
expiration date:       ---
number of mounts:      298
max mounts allowed:    ---
status:                0x0
================================================================================
root@MSTSRVR #

Media info by giving its id :

root@MSTSRVR # bpmedialist -ev CI0688
Server Host = MED2SRVR

 id     rl  images   allocated        last updated      density  kbytes restores
           vimages   expiration       last read         <------- STATUS ------->
           On Hold
--------------------------------------------------------------------------------
CI0688   0      1   11/28/2014 02:56  11/28/2014 02:56  hcart3    63628256     0
                1   12/05/2014 02:56        N/A       
           0

root@MSTSRVR #

To view reports, like which backups are in progress and which are completed, even provides which were failed and in queue.

root@MSTSRVR # bpdbjobs -report | more
JobID         Type  State Status              Policy               Schedule     Client Dest Media Svr
Active PID FATPipe
74692       Backup Queued              MYSRV1_Daily    MYSRV1_MED01_DAILY MED1SRVR               
                 
74691       Backup Active              MYSRV1_Daily    MYSRV1_MED01_DAILY MED1SRVR     MED1SRVR
     11998      No
74690       Backup Active              MYSRV1_Daily    MYSRV1_MED01_DAILY MED1SRVR     MED1SRVR
     11974      No
74689       Backup Active              MYSRV1_Daily    MYSRV1_MED01_DAILY MED1SRVR     MED1SRVR
     11973      No
74688       Backup Active              MYSRV1_Daily    MYSRV1_MED01_DAILY MED1SRVR     MED1SRVR
     11957      No
74687       Backup Active              MYSRV1_Daily                     - MED1SRVR     MED1SRVR
                No
74686 Image Delete   Done      1                                                                     
      6939       
74685       Backup   Done      0  MYSAPSRV_BCV_MED02 MYSAPSRV_BCV_MED02_SAT MED2SRVR     MED2SRVR
     25690      No
74684       Backup   Done      0  MYSAPSRV_BCV_MED02                      - MED2SRVR     MED2SRVR
                No
74683 Image Delete   Done      1                                                                     
      8993       
74682 Image Delete   Done      1                                                                     
     13972       
74681       Backup   Done      0       MYSRV1_Daily    MYSRV1_MED01_DAILY MED1SRVR     MED1SRVR
     23514      No
74680       Backup   Done      0       MYSRV1_Daily    MYSRV1_MED01_DAILY MED1SRVR     MED1SRVR
     15476      No
74679       Backup   Done      0       MYSRV1_Daily    MYSRV1_MED01_DAILY MED1SRVR     MED1SRVR
--More--

root@MSTSRVR #
root@MSTSRVR # bpdbjobs | grep -i active           ------- we can use grep so that we will get only Active backups report.
JobID         Type  State Status              Policy               Schedule     Client Dest Media Svr Active PID FATPipe
74691       Backup Active              MYSRV1_Daily    MYSRV1_MED01_DAILY MED1SRVR     MED1SRVR      11998      No
74690       Backup Active              MYSRV1_Daily    MYSRV1_MED01_DAILY MED1SRVR     MED1SRVR      11974      No
74689       Backup Active              MYSRV1_Daily    MYSRV1_MED01_DAILY MED1SRVR     MED1SRVR      11973      No
74688       Backup Active              MYSRV1_Daily    MYSRV1_MED01_DAILY MED1SRVR     MED1SRVR      11957      No
74687       Backup Active              MYSRV1_Daily                     - MED1SRVR     MED1SRVR                 No
root@MSTSRVR #
root@MSTSRVR #
root@MSTSRVR # bpdbjobs | grep -i done | more
74686 Image Delete   Done      1                                                                     
      6939       
74685       Backup   Done      0  MYSAPSRV_BCV_MED02 MYSAPSRV_BCV_MED02_SAT MED2SRVR     MED2SRVR
     25690      No
74684       Backup   Done      0  MYSAPSRV_BCV_MED02                      - MED2SRVR     MED2SRVR
                No
74683 Image Delete   Done      1                                                                     
      8993       
74682 Image Delete   Done      1                                                                     
     13972       
74681       Backup   Done      0        MYSRV1_Daily    MYSRV1_MED01_DAILY MED1SRVR     MED1SRVR
     23514      No
74680       Backup   Done      0        MYSRV1_Daily    MYSRV1_MED01_DAILY MED1SRVR     MED1SRVR
     15476      No
74674       Backup   Done      0   MYORA_MED02_DAILY    MYORA_MED02_DAILY  MED2SRVR     MED2SRVR
     11837      No
74673       Backup   Done      0   MYORA_MED02_DAILY    MYORA_MED02_DAILY  MED2SRVR     MED2SRVR
     10218      No

root@MSTSRVR #

To start backup of a particular policy, this can be done in media server only.
Below is the scenario like, my servers backup full backup is scheduled in Media-1 server, so I will run the command from Media-1 server.

SYNTAX : bpbackup -p "policyname" -s "UserBackup" -L "progress log location" -S "master server"

root@MED1SRVR # bpbackup -p Oraclesrvr_Daily -s Oraclesrvr_MED1_Daily -S MSTSRVR
root@MED1SRVR #

Here my policy name is " Oraclesrvr_Daily "
My Schedule name is " Oraclesrvr_MED1_Daily " .... Let's check is it started or not

root@MSTSRVR # bpdbjobs | grep -i active      
JobID         Type  State Status              Policy                 Schedule     Client Dest Media Svr Active PID FATPipe
74691       Backup Active           Oraclesrvr_Daily    Oraclesrvr_MED1_Daily   MED1SRVR       MED1SRVR      11998      No
root@MSTSRVR #
root@MSTSRVR #

If we want to move all db like name of pools,schedules,volumes to new media server from existing media server we can achieve this with one command :

SYNTAX : bpmedia -movedb -allvolumes -newserver <media server> -oldserver <media server>

Similarly if we want to move allocated medias ,

SYNTAX : bpmedia -movedb -m <media_id> -newserver <media server>

These are few netbackup commands which I wanted to know , so that we can use in our daily work. Sometimes our Netbackup console will not open properly so I felt like to learn these.

#################################################################################

Veritas Snapshots !!!

A Veritas Snapshot is used to create the snap of a particular volume.A snap represents the data exists in a volume at a given point of time.Thus using snapshot we can even rollback the current situation of a DG and we can also create a copy of the filesystem at that particular point of time.

# bash
bash-3.2#
bash-3.2#
bash-3.2# df -kh
Filesystem             size   used  avail capacity  Mounted on
rpool/ROOT/s10s_u11wos_24a
                        15G   5.3G   5.8G    48%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                   7.3G   464K   7.3G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
sharefs                  0K     0K     0K     0%    /etc/dfs/sharetab
swap                   7.3G     0K   7.3G     0%    /dev/vx/dmp
swap                   7.3G     0K   7.3G     0%    /dev/vx/rdmp
/platform/SUNW,SPARC-Enterprise-T5120/lib/libc_psr/libc_psr_hwcap2.so.1
                        11G   5.3G   5.8G    48%    /platform/sun4v/lib/libc_psr.so.1
/platform/SUNW,SPARC-Enterprise-T5120/lib/sparcv9/libc_psr/libc_psr_hwcap2.so.1
                        11G   5.3G   5.8G    48%    /platform/sun4v/lib/sparcv9/libc_psr.so.1
fd                       0K     0K     0K     0%    /dev/fd
swap                   7.3G    32K   7.3G     1%    /tmp
swap                   7.3G    40K   7.3G     1%    /var/run
rpool/export            15G    32K   5.8G     1%    /export
rpool/export/home       15G    31K   5.8G     1%    /export/home
rpool                   15G   106K   5.8G     1%    /rpool
/dev/odm                 0K     0K     0K     0%    /dev/odm
/dev/vx/dsk/datadg/vol1
                        20G   2.4G    16G    13%    /mysap
bash-3.2#
bash-3.2# cd /mysap
bash-3.2#
bash-3.2# ls -lrth
total 4194320
drwxr-xr-x   7 root     root          96 Sep  4  2013 ASCS03
drwxr-xr-x   2 root     root          96 Jul 19 05:52 lost+found
-rw-r--r--   1 root     root        2.0G Jul 21 10:31 pacct
bash-3.2#
bash-3.2# vxprint -ht
Disk group: datadg

dg datadg       default      default  10000    1405729182.10.test1

dm disk1        emc_clariion0_192 auto 65535   142524320 -
dm disk2        emc_clariion0_194 auto 65535   142524320 -

v  vol1         -            ENABLED  ACTIVE   41943040 SELECT    -        fsgen
pl vol1-01      vol1         ENABLED  ACTIVE   41943040 CONCAT    -        RW
sd disk2-01     vol1-01      disk2    0        41943040 0         emc_clariion0_194 ENA
bash-3.2#

For the snapshot , first of all we need a SNAP of volume then only we can start a SNAPSHOT.
Now let us start the snap.....

bash-3.2# vxassist -g datadg snapstart vol1
bash-3.2#
bash-3.2#
bash-3.2# vxprint -ht
Disk group: datadg

dg datadg       default      default  10000    1405729182.10.test1

dm disk1        emc_clariion0_192 auto 65535   142524320 -
dm disk2        emc_clariion0_194 auto 65535   142524320 -

v  vol1         -            ENABLED  ACTIVE   41943040 SELECT    -        fsgen
pl vol1-01      vol1         ENABLED  ACTIVE   41943040 CONCAT    -        RW
sd disk2-01     vol1-01      disk2    0        41943040 0         emc_clariion0_194 ENA
pl vol1-02      vol1         ENABLED  SNAPDONE 41943040 CONCAT    -        WO
sd disk1-01     vol1-02      disk1    0        41943040 0         emc_clariion0_192 ENA
bash-3.2#

In above output we can observe that a SNAP is DONE, now we are ready to start a snapshot from the SNAP which we took already.

bash-3.2#
bash-3.2#
bash-3.2# vxassist -g datadg snapshot vol1 snap-vol1
bash-3.2#
bash-3.2#
bash-3.2# vxprint -ht
Disk group: datadg

dg datadg       default      default  10000    1405729182.10.test1

dm disk1        emc_clariion0_192 auto 65535   142524320 -
dm disk2        emc_clariion0_194 auto 65535   142524320 -

v  snap-vol1    -            ENABLED  ACTIVE   41943040 ROUND     -        fsgen
pl vol1-02      snap-vol1    ENABLED  ACTIVE   41943040 CONCAT    -        RW
sd disk1-01     vol1-02      disk1    0        41943040 0         emc_clariion0_192 ENA

v  vol1         -            ENABLED  ACTIVE   41943040 SELECT    -        fsgen
pl vol1-01      vol1         ENABLED  ACTIVE   41943040 CONCAT    -        RW
sd disk2-01     vol1-01      disk2    0        41943040 0         emc_clariion0_194 ENA
bash-3.2#

By this we completed the snapshot of the volume, now this particular snapshot acts as an individual volume.We can even mount this volume as a Filesystem.

bash-3.2#
bash-3.2# mount -F vxfs /dev/vx/dsk/datadg/snap-vol1 /mnt
bash-3.2#
bash-3.2#
bash-3.2# df -kh
Filesystem             size   used  avail capacity  Mounted on
rpool/ROOT/s10s_u11wos_24a
                        15G   5.3G   5.8G    48%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                   7.3G   464K   7.3G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
sharefs                  0K     0K     0K     0%    /etc/dfs/sharetab
swap                   7.3G     0K   7.3G     0%    /dev/vx/dmp
swap                   7.3G     0K   7.3G     0%    /dev/vx/rdmp
/platform/SUNW,SPARC-Enterprise-T5120/lib/libc_psr/libc_psr_hwcap2.so.1
                        11G   5.3G   5.8G    48%    /platform/sun4v/lib/libc_psr.so.1
/platform/SUNW,SPARC-Enterprise-T5120/lib/sparcv9/libc_psr/libc_psr_hwcap2.so.1
                        11G   5.3G   5.8G    48%    /platform/sun4v/lib/sparcv9/libc_psr.so.1
fd                       0K     0K     0K     0%    /dev/fd
swap                   7.3G    32K   7.3G     1%    /tmp
swap                   7.3G    40K   7.3G     1%    /var/run
rpool/export            15G    32K   5.8G     1%    /export
rpool/export/home       15G    31K   5.8G     1%    /export/home
rpool                   15G   106K   5.8G     1%    /rpool
/dev/odm                 0K     0K     0K     0%    /dev/odm
/dev/vx/dsk/datadg/vol1
                        20G   2.4G    16G    13%    /mysap
/dev/vx/dsk/datadg/snap-vol1
                        20G   2.4G    16G    13%    /mnt
bash-3.2#

Some more useful commands related to vxsnaps:

To take snapshot of all volumes of a DG....

bash-3.2# vxassist -g datadg -o allvols snapshot

To clear a snap.... (Remember it is clearing a snap, but not the snapshot)

bash-3.2# vxassist -g datadg snapclear snap-vol1