This is a discussion on Mirrorvg gone bad - Aix ; Hello to one and all, After the routine replacement of an SSA disk in a mirrored data VG on a machine running AIX 4.3.3, the last step of the procedure called for remirroring the disks using the "mirrorvg -m" command ...
Hello to one and all,
After the routine replacement of an SSA disk in a mirrored data VG on
a machine running AIX 4.3.3, the last step of the procedure called for
remirroring the disks using the "mirrorvg -m" command accompanied by a
specific list of hdisks. This step always takes quite a few hours,
and for extra security we always run it directly from the system
console (tty), to avoid any possible network issues.
This time, however, something went wrong, and it looks as though the
console was disconnected while the mirrorvg process was running.
Nobody had access to the server room, so we're certain that this was a
technical glitch and not the result of human error or intervention.
As a result, the mirrorvg process was orphaned and the volume group is
locked. Using the ps command to identify processes running under
'root', we find the following:
root 107538 1 0 11:13:43 - 0:00 ksh /usr/sbin/mirrorvg -m
hdisk10 hdisk18 hdisk20 hdisk23 hdisk25 hdisk29 hdisk31 hdisk32
hdisk6 hdisk7 hdisk8
root 220366 107538 0 12:00:08 - 0:00 ksh /usr/sbin/rmlvcopy
root 93398 220366 0 12:00:10 - 0:00 putlvodm -k
0042051394917f5c –X 0
If we follow the PIDs, we see that the mirrorvg process spawned an
rmlvcopy process, which in turn locked the volume group using the
putlvodm command, all prior to the disconnection of the tty.
Now we need to fix this without damaging the one good copy of the data
and without bringing down the machine. What are the next steps and
what is the damage going to look like? How do we get back to a
'normal' mirrored configuration?
Thanks to all in advance.