GAB ports - Veritas Cluster Server

This is a discussion on GAB ports - Veritas Cluster Server ; Does anybody have a list of all possible GAB ports and their function? I know port a is for GAB, h is for HAD, b is for fencing. my gabconfig -a shows: GAB Port Memberships ================================================== ============= Port a gen ...

+ Reply to Thread
Results 1 to 3 of 3

Thread: GAB ports

  1. GAB ports


    Does anybody have a list of all possible GAB ports and their function?
    I know port a is for GAB, h is for HAD, b is for fencing. my gabconfig -a
    shows:

    GAB Port Memberships
    ================================================== =============
    Port a gen 514e0b membership 012345
    Port b gen 514e0d membership 012345
    Port f gen 514e20 membership 012345
    Port h gen 514e23 membership 012345
    Port q gen 514e1e membership 012345
    Port v gen 514e2b membership 012345
    Port w gen 514e2d membership 012345

    thanks,
    leigh


  2. Re: GAB ports

    mmm... seem to have just done it, but here goes again


    GAB is used for communications in a cluster environment. Now to
    communicate, you will have to register with GAB. Each node ni the
    cluster needs to do the registration and the seed number (yeah, will get
    to it) must be the same for all the nodes which belong to the same
    cluster. Although each node has a name, it comes back down to a number
    (via the /etc/llthosts file). So, each GAB port will have a port number
    (like a, or b ....), a usage (like gen) a seed (like 514e0b) and then
    will have a list of all the ports with membership (like 0123 5 --> this
    will indicate that 5 machines have the same port membership, and we're
    likely missing node 4 - could be down).

    So, let's answer your question. What are all the ports, and what are
    they used for.

    Port What is it used for
    ---------------------------
    a This is node-to-node communication. As soon as GAB starts on a
    node, it will look for other nodes in the cluster and
    establish port "a" communication

    b This is used for IO fencing. If you use RAC or VCS 4.x, you can
    use IO fencing to protect data disks. In RAC, as soon as the
    gab port membership changes, we will have a race for the
    coordinator disks, and some nodes will panic when they lose
    the race

    d In RAC, the different Oracle instances need to talk to each
    other. GAB provides port "d" for this. So, port "d" membership
    will statr when Oracle RAC starts

    f This is the main communications port for cluster file system.
    More than 1 machine can mount the same filesystem, but they
    need to communicate to not update the metadata (like inodes,
    super-block, free inode list, free data block list, etc.....)
    at the same time. If they do it at the same time, you will get
    corruption. There is always a primary for any filesystem that
    controls the access to the metadata. This control (locking) is
    done via port "f"

    h Well, now we've got GAB. The different nodes in the cluster
    needs to know what is happening on other nodes (and on itself)
    It needs to know which service groups, resources are online or
    offline or faulted. The program that knows all this info, is
    the "main" vcs program called "had". So on each machine, had
    needs to talk to GAB. This is done via port "h"

    o This is a port used specifically in RAC, and specifically or
    ODM. Let's start by saying what ODM is, and then why it is
    needed. Oracle (like most other database managers) will try to
    cache IO before writing it out to disk (raw volumes or data
    files on a filesystem). The biggest problem comes in when
    Oracle tries to write to a filesystem. Each filesystem has
    it's own cache. As you can think, the general purpose
    filesystem cache is not the same as the very specific Oracle
    cache. The startegy used is very different between Oracle and
    the filesystem. A while ago, Veritas had a close look at how
    the Oracle cache works and how it sends IO to the filesystem.
    Veritas then wrote an extension for their filesystem (called
    Quick IO - QIO). With QIO, they got performance very close to
    the performance Oracle got on raw volumes. The rest of the
    filesystem comunity (read SUN UFS, IBM JFS, .....) thought
    that Oracle gave the information to Veritas and complained
    about it. Oracle then sat down and actually wrote a
    specification. This specification allows everyone to write
    their own library, and the Oracle will call this library to do
    IO. Oracle called this specification ODM (Oracle Disk
    Manager). The best is, that only Veritas ever wrote their own
    libraries for ODM. So, getting back to port "o". Port "o" is
    used for ODM to ODM communication in a RAC cluster. (wow, QIO,
    ODM and port "o" in one go !)

    q This is another port used in Cluster Filesystem. VxFS is a
    journaled filesystem. This means that it keeps a log which
    it will write to, before making changes to the metadata on the
    filesystem. (like Oracle keeps redo logs). Normally this log
    is kept on the same filesystem. This means that for each
    access, the log has to be updated, then the metadata and then
    the data itself. Thus 3 different times VxFS has to access the
    same disk. Normally the metadata is kept close to the file,
    but the log is always kept in a static place (normally close
    to the beginning of the filesystem). This could means that
    there will be a lot of seeking (for the begining of the
    filesystem, then again to the metadata and data). As we all
    know, disk access time is about 100 times slower than memory,
    so we have a slowdown here. Veritas made a plan and developed
    quicklog. This allows you to have the filesystem log on a
    different disk. This helps in speeding things up, because most
    disk operations can happen in parallel. OK, so now you know
    what quicklog is. You can have quicklog on cluster filesystems
    as well. Port "q" is used to coordinate access to quicklog
    (wow, that was a loooong one)

    u Not a port you would normally see, but just to be complete,
    let's mention it here. When a Cluster Volume Manager is
    started, it will need to do a couple of things. The access to
    changing the configuration of volumes, plexes, subdisks and
    diskgroups, needs to be coordinated. This means that a
    "master" will always need to be selected in the cluster (can
    be checked with the "vxdctl -c mode" command). Normally the
    master is the first one to open port "u". Port "u" is an
    exclusive port for registering with the cluster volume manager
    "master". If no master has been established yet, the first
    node to open port "u" will assume the role of master. The
    master controls all access to changes of the cluster volume
    manager configuration. Each node that tries to join the
    cluster (CVM), will need to open (exclusively) port "u",
    search for the master, and make sure that the node and the
    master sees all the same disks for the shared diskgroups.

    v OK, now that we've estabblished that there is a master, we
    need to mention that fact that each instance of volume manager
    running (thus on each node) keeps the configuration in memory
    (regardless if it is part of a cluster or not). This "memory"
    is is managed by the configuration daemon (vxconfigd). We will
    get to the vxconfigd in a minute, but first port "v". So,
    port "v" is actually used to register membership for the
    cluster volume manager. (once the node got port "u"
    membership, the "permanent" membership is done via port "v".
    Only members of the same cluster (cluster volume manager
    cluster that is) are allowed to import and access the (shared)
    disks

    w The last port in cluster volume manager. This is the port used
    for the vxconfigd on each node to communicate with the
    vxconfigd on all the other nodes. The biggest issue is that a
    configuration change needs to be the same across the whole
    cluster (does not help that 1 node thinks we still have a
    mirrored volume and the others don't know a thing about the
    mirror)





    WOW, that was a long one !!

    Hope that explains things a bit.



    leigh reamy wrote:
    > Does anybody have a list of all possible GAB ports and their function?
    > I know port a is for GAB, h is for HAD, b is for fencing. my gabconfig -a
    > shows:
    >
    > GAB Port Memberships
    > ================================================== =============
    > Port a gen 514e0b membership 012345
    > Port b gen 514e0d membership 012345
    > Port f gen 514e20 membership 012345
    > Port h gen 514e23 membership 012345
    > Port q gen 514e1e membership 012345
    > Port v gen 514e2b membership 012345
    > Port w gen 514e2d membership 012345
    >
    > thanks,
    > leigh
    >


  3. Re: GAB ports

    sorry - swapped port "d" and "o"

    Me wrote:
    > mmm... seem to have just done it, but here goes again
    >
    >
    > GAB is used for communications in a cluster environment. Now to
    > communicate, you will have to register with GAB. Each node ni the
    > cluster needs to do the registration and the seed number (yeah, will get
    > to it) must be the same for all the nodes which belong to the same
    > cluster. Although each node has a name, it comes back down to a number
    > (via the /etc/llthosts file). So, each GAB port will have a port number
    > (like a, or b ....), a usage (like gen) a seed (like 514e0b) and then
    > will have a list of all the ports with membership (like 0123 5 --> this
    > will indicate that 5 machines have the same port membership, and we're
    > likely missing node 4 - could be down).
    >
    > So, let's answer your question. What are all the ports, and what are
    > they used for.
    >
    > Port What is it used for
    > ---------------------------
    > a This is node-to-node communication. As soon as GAB starts on a
    > node, it will look for other nodes in the cluster and
    > establish port "a" communication
    >
    > b This is used for IO fencing. If you use RAC or VCS 4.x, you can
    > use IO fencing to protect data disks. In RAC, as soon as the
    > gab port membership changes, we will have a race for the
    > coordinator disks, and some nodes will panic when they lose
    > the race
    >
    > d In RAC, the different Oracle instances need to talk to each
    > other. GAB provides port "d" for this. So, port "d" membership
    > will statr when Oracle RAC starts
    >
    > f This is the main communications port for cluster file system.
    > More than 1 machine can mount the same filesystem, but they
    > need to communicate to not update the metadata (like inodes,
    > super-block, free inode list, free data block list, etc.....)
    > at the same time. If they do it at the same time, you will get
    > corruption. There is always a primary for any filesystem that
    > controls the access to the metadata. This control (locking) is
    > done via port "f"
    >
    > h Well, now we've got GAB. The different nodes in the cluster
    > needs to know what is happening on other nodes (and on itself)
    > It needs to know which service groups, resources are online or
    > offline or faulted. The program that knows all this info, is
    > the "main" vcs program called "had". So on each machine, had
    > needs to talk to GAB. This is done via port "h"
    >
    > o This is a port used specifically in RAC, and specifically or
    > ODM. Let's start by saying what ODM is, and then why it is
    > needed. Oracle (like most other database managers) will try to
    > cache IO before writing it out to disk (raw volumes or data
    > files on a filesystem). The biggest problem comes in when
    > Oracle tries to write to a filesystem. Each filesystem has
    > it's own cache. As you can think, the general purpose
    > filesystem cache is not the same as the very specific Oracle
    > cache. The startegy used is very different between Oracle and
    > the filesystem. A while ago, Veritas had a close look at how
    > the Oracle cache works and how it sends IO to the filesystem.
    > Veritas then wrote an extension for their filesystem (called
    > Quick IO - QIO). With QIO, they got performance very close to
    > the performance Oracle got on raw volumes. The rest of the
    > filesystem comunity (read SUN UFS, IBM JFS, .....) thought
    > that Oracle gave the information to Veritas and complained
    > about it. Oracle then sat down and actually wrote a
    > specification. This specification allows everyone to write
    > their own library, and the Oracle will call this library to do
    > IO. Oracle called this specification ODM (Oracle Disk
    > Manager). The best is, that only Veritas ever wrote their own
    > libraries for ODM. So, getting back to port "o". Port "o" is
    > used for ODM to ODM communication in a RAC cluster. (wow, QIO,
    > ODM and port "o" in one go !)
    >
    > q This is another port used in Cluster Filesystem. VxFS is a
    > journaled filesystem. This means that it keeps a log which
    > it will write to, before making changes to the metadata on the
    > filesystem. (like Oracle keeps redo logs). Normally this log
    > is kept on the same filesystem. This means that for each
    > access, the log has to be updated, then the metadata and then
    > the data itself. Thus 3 different times VxFS has to access the
    > same disk. Normally the metadata is kept close to the file,
    > but the log is always kept in a static place (normally close
    > to the beginning of the filesystem). This could means that
    > there will be a lot of seeking (for the begining of the
    > filesystem, then again to the metadata and data). As we all
    > know, disk access time is about 100 times slower than memory,
    > so we have a slowdown here. Veritas made a plan and developed
    > quicklog. This allows you to have the filesystem log on a
    > different disk. This helps in speeding things up, because most
    > disk operations can happen in parallel. OK, so now you know
    > what quicklog is. You can have quicklog on cluster filesystems
    > as well. Port "q" is used to coordinate access to quicklog
    > (wow, that was a loooong one)
    >
    > u Not a port you would normally see, but just to be complete,
    > let's mention it here. When a Cluster Volume Manager is
    > started, it will need to do a couple of things. The access to
    > changing the configuration of volumes, plexes, subdisks and
    > diskgroups, needs to be coordinated. This means that a
    > "master" will always need to be selected in the cluster (can
    > be checked with the "vxdctl -c mode" command). Normally the
    > master is the first one to open port "u". Port "u" is an
    > exclusive port for registering with the cluster volume manager
    > "master". If no master has been established yet, the first
    > node to open port "u" will assume the role of master. The
    > master controls all access to changes of the cluster volume
    > manager configuration. Each node that tries to join the
    > cluster (CVM), will need to open (exclusively) port "u",
    > search for the master, and make sure that the node and the
    > master sees all the same disks for the shared diskgroups.
    >
    > v OK, now that we've estabblished that there is a master, we
    > need to mention that fact that each instance of volume manager
    > running (thus on each node) keeps the configuration in memory
    > (regardless if it is part of a cluster or not). This "memory"
    > is is managed by the configuration daemon (vxconfigd). We will
    > get to the vxconfigd in a minute, but first port "v". So,
    > port "v" is actually used to register membership for the
    > cluster volume manager. (once the node got port "u"
    > membership, the "permanent" membership is done via port "v".
    > Only members of the same cluster (cluster volume manager
    > cluster that is) are allowed to import and access the (shared)
    > disks
    >
    > w The last port in cluster volume manager. This is the port used
    > for the vxconfigd on each node to communicate with the
    > vxconfigd on all the other nodes. The biggest issue is that a
    > configuration change needs to be the same across the whole
    > cluster (does not help that 1 node thinks we still have a
    > mirrored volume and the others don't know a thing about the
    > mirror)
    >
    >
    >
    >
    >
    > WOW, that was a long one !!
    >
    > Hope that explains things a bit.
    >
    >
    >
    > leigh reamy wrote:
    >
    >> Does anybody have a list of all possible GAB ports and their function?
    >> I know port a is for GAB, h is for HAD, b is for fencing. my
    >> gabconfig -a
    >> shows:
    >>
    >> GAB Port Memberships
    >> ================================================== =============
    >> Port a gen 514e0b membership 012345 Port b
    >> gen 514e0d membership 012345 Port f gen
    >> 514e20 membership 012345 Port h gen 514e23
    >> membership 012345 Port q gen 514e1e
    >> membership 012345 Port v gen 514e2b
    >> membership 012345 Port w gen 514e2d
    >> membership 012345
    >> thanks,
    >> leigh
    >>


+ Reply to Thread