GAB ports

Does anybody have a list of all possible GAB ports and their function?
I know port a is for GAB, h is for HAD, b is for fencing. my gabconfig -a
shows:

GAB Port Memberships
===============================================================
Port a gen 514e0b membership 012345
Port b gen 514e0d membership 012345
Port f gen 514e20 membership 012345
Port h gen 514e23 membership 012345
Port q gen 514e1e membership 012345
Port v gen 514e2b membership 012345
Port w gen 514e2d membership 012345

thanks,
leigh

mmm... seem to have just done it, but here goes again


GAB is used for communications in a cluster environment. Now to
communicate, you will have to register with GAB. Each node ni the
cluster needs to do the registration and the seed number (yeah, will get
to it) must be the same for all the nodes which belong to the same
cluster. Although each node has a name, it comes back down to a number
(via the /etc/llthosts file). So, each GAB port will have a port number
(like a, or b ....), a usage (like gen) a seed (like 514e0b) and then
will have a list of all the ports with membership (like 0123 5 --> this
will indicate that 5 machines have the same port membership, and we're
likely missing node 4 - could be down).

So, let's answer your question. What are all the ports, and what are
they used for.

Port What is it used for
---------------------------
a This is node-to-node communication. As soon as GAB starts on a
node, it will look for other nodes in the cluster and
establish port "a" communication

b This is used for IO fencing. If you use RAC or VCS 4.x, you can
use IO fencing to protect data disks. In RAC, as soon as the
gab port membership changes, we will have a race for the
coordinator disks, and some nodes will panic when they lose
the race

d In RAC, the different Oracle instances need to talk to each
other. GAB provides port "d" for this. So, port "d" membership
will statr when Oracle RAC starts

f This is the main communications port for cluster file system.
More than 1 machine can mount the same filesystem, but they
need to communicate to not update the metadata (like inodes,
super-block, free inode list, free data block list, etc.....)
at the same time. If they do it at the same time, you will get
corruption. There is always a primary for any filesystem that
controls the access to the metadata. This control (locking) is
done via port "f"

h Well, now we've got GAB. The different nodes in the cluster
needs to know what is happening on other nodes (and on itself)
It needs to know which service groups, resources are online or
offline or faulted. The program that knows all this info, is
the "main" vcs program called "had". So on each machine, had
needs to talk to GAB. This is done via port "h"

o This is a port used specifically in RAC, and specifically or
ODM. Let's start by saying what ODM is, and then why it is
needed. Oracle (like most other database managers) will try to
cache IO before writing it out to disk (raw volumes or data
files on a filesystem). The biggest problem comes in when
Oracle tries to write to a filesystem. Each filesystem has
it's own cache. As you can think, the general purpose
filesystem cache is not the same as the very specific Oracle
cache. The startegy used is very different between Oracle and
the filesystem. A while ago, Veritas had a close look at how
the Oracle cache works and how it sends IO to the filesystem.
Veritas then wrote an extension for their filesystem (called
Quick IO - QIO). With QIO, they got performance very close to
the performance Oracle got on raw volumes. The rest of the
filesystem comunity (read SUN UFS, IBM JFS, .....) thought
that Oracle gave the information to Veritas and complained
about it. Oracle then sat down and actually wrote a
specification. This specification allows everyone to write
their own library, and the Oracle will call this library to do
IO. Oracle called this specification ODM (Oracle Disk
Manager). The best is, that only Veritas ever wrote their own
libraries for ODM. So, getting back to port "o". Port "o" is
used for ODM to ODM communication in a RAC cluster. (wow, QIO,
ODM and port "o" in one go !)

q This is another port used in Cluster Filesystem. VxFS is a
journaled filesystem. This means that it keeps a log which
it will write to, before making changes to the metadata on the
filesystem. (like Oracle keeps redo logs). Normally this log
is kept on the same filesystem. This means that for each
access, the log has to be updated, then the metadata and then
the data itself. Thus 3 different times VxFS has to access the
same disk. Normally the metadata is kept close to the file,
but the log is always kept in a static place (normally close
to the beginning of the filesystem). This could means that
there will be a lot of seeking (for the begining of the
filesystem, then again to the metadata and data). As we all
know, disk access time is about 100 times slower than memory,
so we have a slowdown here. Veritas made a plan and developed
quicklog. This allows you to have the filesystem log on a
different disk. This helps in speeding things up, because most
disk operations can happen in parallel. OK, so now you know
what quicklog is. You can have quicklog on cluster filesystems
as well. Port "q" is used to coordinate access to quicklog
(wow, that was a loooong one)

u Not a port you would normally see, but just to be complete,
let's mention it here. When a Cluster Volume Manager is
started, it will need to do a couple of things. The access to
changing the configuration of volumes, plexes, subdisks and
diskgroups, needs to be coordinated. This means that a
"master" will always need to be selected in the cluster (can
be checked with the "vxdctl -c mode" command). Normally the
master is the first one to open port "u". Port "u" is an
exclusive port for registering with the cluster volume manager
"master". If no master has been established yet, the first
node to open port "u" will assume the role of master. The
master controls all access to changes of the cluster volume
manager configuration. Each node that tries to join the
cluster (CVM), will need to open (exclusively) port "u",
search for the master, and make sure that the node and the
master sees all the same disks for the shared diskgroups.

v OK, now that we've estabblished that there is a master, we
need to mention that fact that each instance of volume manager
running (thus on each node) keeps the configuration in memory
(regardless if it is part of a cluster or not). This "memory"
is is managed by the configuration daemon (vxconfigd). We will
get to the vxconfigd in a minute, but first port "v". So,
port "v" is actually used to register membership for the
cluster volume manager. (once the node got port "u"
membership, the "permanent" membership is done via port "v".
Only members of the same cluster (cluster volume manager
cluster that is) are allowed to import and access the (shared)
disks

w The last port in cluster volume manager. This is the port used
for the vxconfigd on each node to communicate with the
vxconfigd on all the other nodes. The biggest issue is that a
configuration change needs to be the same across the whole
cluster (does not help that 1 node thinks we still have a
mirrored volume and the others don't know a thing about the
mirror)





WOW, that was a long one !!

Hope that explains things a bit.



leigh reamy wrote:[color=blue]
> Does anybody have a list of all possible GAB ports and their function?
> I know port a is for GAB, h is for HAD, b is for fencing. my gabconfig -a
> shows:
>
> GAB Port Memberships
> ===============================================================
> Port a gen 514e0b membership 012345
> Port b gen 514e0d membership 012345
> Port f gen 514e20 membership 012345
> Port h gen 514e23 membership 012345
> Port q gen 514e1e membership 012345
> Port v gen 514e2b membership 012345
> Port w gen 514e2d membership 012345
>
> thanks,
> leigh
>[/color]

sorry - swapped port "d" and "o"

Me wrote:[color=blue]
> mmm... seem to have just done it, but here goes again
>
>
> GAB is used for communications in a cluster environment. Now to
> communicate, you will have to register with GAB. Each node ni the
> cluster needs to do the registration and the seed number (yeah, will get
> to it) must be the same for all the nodes which belong to the same
> cluster. Although each node has a name, it comes back down to a number
> (via the /etc/llthosts file). So, each GAB port will have a port number
> (like a, or b ....), a usage (like gen) a seed (like 514e0b) and then
> will have a list of all the ports with membership (like 0123 5 --> this
> will indicate that 5 machines have the same port membership, and we're
> likely missing node 4 - could be down).
>
> So, let's answer your question. What are all the ports, and what are
> they used for.
>
> Port What is it used for
> ---------------------------
> a This is node-to-node communication. As soon as GAB starts on a
> node, it will look for other nodes in the cluster and
> establish port "a" communication
>
> b This is used for IO fencing. If you use RAC or VCS 4.x, you can
> use IO fencing to protect data disks. In RAC, as soon as the
> gab port membership changes, we will have a race for the
> coordinator disks, and some nodes will panic when they lose
> the race
>
> d In RAC, the different Oracle instances need to talk to each
> other. GAB provides port "d" for this. So, port "d" membership
> will statr when Oracle RAC starts
>
> f This is the main communications port for cluster file system.
> More than 1 machine can mount the same filesystem, but they
> need to communicate to not update the metadata (like inodes,
> super-block, free inode list, free data block list, etc.....)
> at the same time. If they do it at the same time, you will get
> corruption. There is always a primary for any filesystem that
> controls the access to the metadata. This control (locking) is
> done via port "f"
>
> h Well, now we've got GAB. The different nodes in the cluster
> needs to know what is happening on other nodes (and on itself)
> It needs to know which service groups, resources are online or
> offline or faulted. The program that knows all this info, is
> the "main" vcs program called "had". So on each machine, had
> needs to talk to GAB. This is done via port "h"
>
> o This is a port used specifically in RAC, and specifically or
> ODM. Let's start by saying what ODM is, and then why it is
> needed. Oracle (like most other database managers) will try to
> cache IO before writing it out to disk (raw volumes or data
> files on a filesystem). The biggest problem comes in when
> Oracle tries to write to a filesystem. Each filesystem has
> it's own cache. As you can think, the general purpose
> filesystem cache is not the same as the very specific Oracle
> cache. The startegy used is very different between Oracle and
> the filesystem. A while ago, Veritas had a close look at how
> the Oracle cache works and how it sends IO to the filesystem.
> Veritas then wrote an extension for their filesystem (called
> Quick IO - QIO). With QIO, they got performance very close to
> the performance Oracle got on raw volumes. The rest of the
> filesystem comunity (read SUN UFS, IBM JFS, .....) thought
> that Oracle gave the information to Veritas and complained
> about it. Oracle then sat down and actually wrote a
> specification. This specification allows everyone to write
> their own library, and the Oracle will call this library to do
> IO. Oracle called this specification ODM (Oracle Disk
> Manager). The best is, that only Veritas ever wrote their own
> libraries for ODM. So, getting back to port "o". Port "o" is
> used for ODM to ODM communication in a RAC cluster. (wow, QIO,
> ODM and port "o" in one go !)
>
> q This is another port used in Cluster Filesystem. VxFS is a
> journaled filesystem. This means that it keeps a log which
> it will write to, before making changes to the metadata on the
> filesystem. (like Oracle keeps redo logs). Normally this log
> is kept on the same filesystem. This means that for each
> access, the log has to be updated, then the metadata and then
> the data itself. Thus 3 different times VxFS has to access the
> same disk. Normally the metadata is kept close to the file,
> but the log is always kept in a static place (normally close
> to the beginning of the filesystem). This could means that
> there will be a lot of seeking (for the begining of the
> filesystem, then again to the metadata and data). As we all
> know, disk access time is about 100 times slower than memory,
> so we have a slowdown here. Veritas made a plan and developed
> quicklog. This allows you to have the filesystem log on a
> different disk. This helps in speeding things up, because most
> disk operations can happen in parallel. OK, so now you know
> what quicklog is. You can have quicklog on cluster filesystems
> as well. Port "q" is used to coordinate access to quicklog
> (wow, that was a loooong one)
>
> u Not a port you would normally see, but just to be complete,
> let's mention it here. When a Cluster Volume Manager is
> started, it will need to do a couple of things. The access to
> changing the configuration of volumes, plexes, subdisks and
> diskgroups, needs to be coordinated. This means that a
> "master" will always need to be selected in the cluster (can
> be checked with the "vxdctl -c mode" command). Normally the
> master is the first one to open port "u". Port "u" is an
> exclusive port for registering with the cluster volume manager
> "master". If no master has been established yet, the first
> node to open port "u" will assume the role of master. The
> master controls all access to changes of the cluster volume
> manager configuration. Each node that tries to join the
> cluster (CVM), will need to open (exclusively) port "u",
> search for the master, and make sure that the node and the
> master sees all the same disks for the shared diskgroups.
>
> v OK, now that we've estabblished that there is a master, we
> need to mention that fact that each instance of volume manager
> running (thus on each node) keeps the configuration in memory
> (regardless if it is part of a cluster or not). This "memory"
> is is managed by the configuration daemon (vxconfigd). We will
> get to the vxconfigd in a minute, but first port "v". So,
> port "v" is actually used to register membership for the
> cluster volume manager. (once the node got port "u"
> membership, the "permanent" membership is done via port "v".
> Only members of the same cluster (cluster volume manager
> cluster that is) are allowed to import and access the (shared)
> disks
>
> w The last port in cluster volume manager. This is the port used
> for the vxconfigd on each node to communicate with the
> vxconfigd on all the other nodes. The biggest issue is that a
> configuration change needs to be the same across the whole
> cluster (does not help that 1 node thinks we still have a
> mirrored volume and the others don't know a thing about the
> mirror)
>
>
>
>
>
> WOW, that was a long one !!
>
> Hope that explains things a bit.
>
>
>
> leigh reamy wrote:
>[color=green]
>> Does anybody have a list of all possible GAB ports and their function?
>> I know port a is for GAB, h is for HAD, b is for fencing. my
>> gabconfig -a
>> shows:
>>
>> GAB Port Memberships
>> ===============================================================
>> Port a gen 514e0b membership 012345 Port b
>> gen 514e0d membership 012345 Port f gen
>> 514e20 membership 012345 Port h gen 514e23
>> membership 012345 Port q gen 514e1e
>> membership 012345 Port v gen 514e2b
>> membership 012345 Port w gen 514e2d
>> membership 012345
>> thanks,
>> leigh
>>[/color][/color]