[9fans] How to get the diagnostics of fs(3) - Plan9

This is a discussion on [9fans] How to get the diagnostics of fs(3) - Plan9 ; Dear List, I have experienced a disk crash in a mirrored fs(3). It turned out that the mirroring has not been successful since December 2007 which is quite a loss for me now. To prevent a case like this it ...

+ Reply to Thread
Results 1 to 11 of 11

Thread: [9fans] How to get the diagnostics of fs(3)

  1. [9fans] How to get the diagnostics of fs(3)

    Dear List,

    I have experienced a disk crash in a mirrored fs(3). It turned out
    that the mirroring has not been successful since December 2007 which
    is quite a loss for me now.

    To prevent a case like this it would have helped If I had seen the
    error messages by fs(3) earlier/at all. By browsing through the
    code with the intention to add somehting there I found several
    prints that do issue the right warnings. What I don't see at the
    moment: Where do they go to? How is #k started and how can I redirect
    the stdout of #k to a file that I can monitor?

    Thanks for your insights,

    Christian

    --
    You may use my gpg key for replies:
    pub 1024D/47F79788 2005/02/02 Christian Kellermann (C-Keen)

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.8 (OpenBSD)

    iEYEARECAAYFAkj1qZwACgkQXYob3Uf3l4jTgACgqM/stFqstLrwbakWNJoOv8Qt
    Ug0AmgN2ZduCKsmK1+MLEkzHT7vVThiF
    =X48t
    -----END PGP SIGNATURE-----


  2. [9fans] How to get the diagnostics of fs(3)

    We use aux/clog to send the contents of /dev/kprint to /sys/log/$sysname
    We bind '#k' by hand after booting our server, but how you do it it depends
    on the particular config for your machine.

    > From: Christian.Kellermann@nefkom.net
    > To: 9fans@cse.psu.edu
    > Reply-To: 9fans@9fans.net
    > Date: Wed Oct 15 10:31:41 CET 2008
    > Subject: [9fans] How to get the diagnostics of fs(3)
    >
    > Dear List,
    >
    > I have experienced a disk crash in a mirrored fs(3). It turned out
    > that the mirroring has not been successful since December 2007 which
    > is quite a loss for me now.
    >
    > To prevent a case like this it would have helped If I had seen the
    > error messages by fs(3) earlier/at all. By browsing through the
    > code with the intention to add somehting there I found several
    > prints that do issue the right warnings. What I don't see at the
    > moment: Where do they go to? How is #k started and how can I redirect
    > the stdout of #k to a file that I can monitor?
    >
    > Thanks for your insights,
    >
    > Christian
    >
    > --
    > You may use my gpg key for replies:
    > pub 1024D/47F79788 2005/02/02 Christian Kellermann (C-Keen)
    >
    >
    > — 2
    >



  3. Re: [9fans] How to get the diagnostics of fs(3)

    * Fco. J. Ballesteros [081015 10:42]:
    > We use aux/clog to send the contents of /dev/kprint to /sys/log/$sysname
    > We bind '#k' by hand after booting our server, but how you do it it depends
    > on the particular config for your machine.
    >


    As it has been I placed the fs= line in plan9.ini as I had my
    fossil/venti starting off the fs(3) server. This might not be a
    good idea now. Still if you want to keep it this way, can I still
    get to the output?


    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.8 (OpenBSD)

    iEYEARECAAYFAkj1rhUACgkQXYob3Uf3l4gGFQCfYcbc5hmfny x5p+w5fVAuu42k
    DB4Ani85aH85Xs6eSuQo5Tc3Cl/D8Osz
    =wWBZ
    -----END PGP SIGNATURE-----


  4. Re: [9fans] How to get the diagnostics of fs(3)

    All the output sent to the console is available via /dev/kprint.
    If you copy that file somewhere, eg, using aux/clog, all your messages
    should be in that file.

    > From: Christian.Kellermann@nefkom.net
    > To: 9fans@9fans.net
    > Reply-To: 9fans@9fans.net
    > Date: Wed Oct 15 10:48:09 CET 2008
    > Subject: Re: [9fans] How to get the diagnostics of fs(3)
    >
    > * Fco. J. Ballesteros [081015 10:42]:
    > > We use aux/clog to send the contents of /dev/kprint to /sys/log/$sysname
    > > We bind '#k' by hand after booting our server, but how you do it it depends
    > > on the particular config for your machine.
    > >

    >
    > As it has been I placed the fs= line in plan9.ini as I had my
    > fossil/venti starting off the fs(3) server. This might not be a
    > good idea now. Still if you want to keep it this way, can I still
    > get to the output?
    >
    >
    >
    > — 2
    >



  5. Re: [9fans] How to get the diagnostics of fs(3)

    we have three different console servers at coraid.
    so i've changed how consoles and console logging
    works. maybe this will be useful to other people.

    here are the changes that make this work
    1. instead of consoledb use /lib/ndb/consoledb.$sysname.
    and have a general test for this new file in cpurc.
    if(test -f /lib/ndb/consoledb.$sysname){
    aux/consolefs -c /lib/ndb/consoledb.$sysname
    startclog
    }
    2. startclog (attached) uses a new consoledb tuple
    clog=$sysname, e.g. from /lib/ndb/consoledb.baron
    console=ila dev=/dev/eia5
    gid=sys
    clog=baron

    (the -r flag is part of a very sneaky trick involving booting
    from a skeleton kfs and binding large bits of the namespace
    back via aan so that the console logger can survive loosing
    its connection to the fs without having console logging
    end up someplace other than the man fs. the way to start
    it is startclog; startclog -r. this setup survived a switch gone
    wild a few weeks ago.)

    3. add the directory /sys/log/clog & create the console
    files you wish.

    - erik

  6. Re: [9fans] How to get the diagnostics of fs(3)

    simliar story here, lost disk, lost data. In this case because I made
    a mistake.

    I've been wondering -- how much of your file systems are you using out there?

    Seems to me that one could build a fanless wonder with a small via
    embedded system and USB sticks -- 3 of them -- as the disks, running
    in the fs redundant mode. But that means you're limited in size to
    64GB. Performance of the corsair voyagers is actually quite good, so
    on a 100bt network you probably would not feel too much pain.

    It just depends on whether 64G is enough. But the fanless wonder
    removes all motors, which seems a good idea.

    ron


  7. Re: [9fans] How to get the diagnostics of fs(3)

    back when i had mirroring via devfs (some three years ago now) i used
    'cmp' to verify that the disks were being correctly written to and
    that no errors have occurred. i ran cmp from the nigtly log at least
    couple of times a week.


  8. Re: [9fans] How to get the diagnostics of fs(3)

    > back when i had mirroring via devfs (some three years ago now) i used
    > 'cmp' to verify that the disks were being correctly written to and
    > that no errors have occurred. i ran cmp from the nigtly log at least
    > couple of times a week.


    that's a good idea.

    when using the mirror device, no write error is issued unless all mirrored
    copies are unwritable. this would make me nervous since without a
    checksum, it's hard to know which copy is good.

    (/n/sources/plan9/sys/src/9/port/devfs.c:697)

    - erik


  9. Re: [9fans] How to get the diagnostics of fs(3)

    * erik quanstrom [081015 18:23]:
    > > back when i had mirroring via devfs (some three years ago now) i used
    > > 'cmp' to verify that the disks were being correctly written to and
    > > that no errors have occurred. i ran cmp from the nigtly log at least
    > > couple of times a week.

    >
    > that's a good idea.
    >
    > when using the mirror device, no write error is issued unless all mirrored
    > copies are unwritable. this would make me nervous since without a
    > checksum, it's hard to know which copy is good.
    >
    > (/n/sources/plan9/sys/src/9/port/devfs.c:697)


    Also it means that while the writes may be ok, you will find out
    that some sectors are corrupt at the wrong time. For this a cmp
    would not be sufficient would it? An interesting thing in my case
    is that I got cmp errors from the start, even with freshly nulled
    partitions and identical partitions and disks. I still cannot figure
    out why.

    Cheers,

    Christian


  10. Re: [9fans] How to get the diagnostics of fs(3)

    On Wed Oct 15 14:38:47 EDT 2008, Christian.Kellermann@nefkom.net wrote:
    > [...] An interesting thing in my case
    > is that I got cmp errors from the start, even with freshly nulled
    > partitions and identical partitions and disks. I still cannot figure
    > out why.


    were these cmp errors in any particular place on the partition?

    - erik


  11. Re: [9fans] How to get the diagnostics of fs(3)

    * erik quanstrom [081016 03:00]:
    > On Wed Oct 15 14:38:47 EDT 2008, Christian.Kellermann@nefkom.net wrote:
    > > [...] An interesting thing in my case
    > > is that I got cmp errors from the start, even with freshly nulled
    > > partitions and identical partitions and disks. I still cannot figure
    > > out why.

    >
    > were these cmp errors in any particular place on the partition?


    unfortunately this data got lost in the noise...


    --
    You may use my gpg key for replies:
    pub 1024D/47F79788 2005/02/02 Christian Kellermann (C-Keen)

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (OpenBSD)

    iEYEARECAAYFAkj2714ACgkQXYob3Uf3l4hvuwCeIAE02EkT0U hp62Je5/RlTmSu
    e0MAoLO5It3ri5YYS8F73HoEgtIAq3tJ
    =aRPl
    -----END PGP SIGNATURE-----


+ Reply to Thread