FreeBSD 7-STABLE deadlock! - FreeBSD

This is a discussion on FreeBSD 7-STABLE deadlock! - FreeBSD ; Hello, freebsd-stable. Sometimes my FreeBSD 7 machine becomes semi-dead: all network sttack is working (pingable, etc), all LIVE processes are live, but it is impossible to create new (and, it seems, EXIT finished) process. So, I can work in shell ...

+ Reply to Thread
Results 1 to 8 of 8

Thread: FreeBSD 7-STABLE deadlock!

  1. FreeBSD 7-STABLE deadlock!

    Hello, freebsd-stable.

    Sometimes my FreeBSD 7 machine becomes semi-dead: all network sttack
    is working (pingable, etc), all LIVE processes are live, but it is
    impossible to create new (and, it seems, EXIT finished) process.
    So, I can work in shell via SSH, if I have connection, but if I try
    to run any external program session is lost for me -- it try to run it
    forever, and ^C is not working. Same with "real" console. Same with
    scripts -- if script working, it locks on next external command
    forever and can not be killed with ^C.
    It is dual-core (E4500)-based comuter, amd64 arch. It have these
    debug options in kernel:

    options DDB
    options KDB
    options KDB_UNATTENDED
    options INVARIANTS
    options INVARIANT_SUPPORT
    options WITNESS
    options DEBUG_MEMGUARD
    options WITNESS_KDB
    options MUTEX_DEBUG

    I have kernel debugger and here is some information (it is re-typed
    from screen, so is not byte-to-byte exact output and I skipped long
    64-bit addresses):

    (1) show allpcpu
    Current CPU: 1

    cpuid = 0
    curthread = pid 12: "idle: cpu0"
    curpcb =
    fpcurthread = none
    idlethread = pid 12: "idle: cpu0"
    spinlocks held:

    cpuid = 1
    curthread = pid 17: "swi6: Giant taskq"
    curpcb =
    fpcurthread = none
    idlethread = pid 11: "idle: cpu1"
    spinlocks held:

    (2) show locks
    exclusive sleep mutex Giant r = 0 (ADDRESS) locked @
    /usr/src/sys/kern/kern_intr.c:1035

    (3) show alllocks
    process 723 (sshd) thread
    (TID)
    exclusive sx so_rcv_sx r = 1 (ADDRESS) locked @
    /usr/src/sys/kern/uipc_sockbuf.c:148
    process 574 (nmdb) thread
    (TID)
    exclusive sx user map r = 0 (ADDRESS) locked @
    /usr/src/sys/vm/vm_map.c:3111
    process 17 (swi6: Ginat taskq) thread
    (TID)
    exclusive sleep mutex Giant r = 0 (ADDRESS) locked @
    /usr/src/sys/kern/kern_intr.c:1035

    (4) alltrace shows:

    (a) I have MANY (about 20) sendmail processes (on
    computer without mail traffic at all!), and all of them have
    "VOP_LOCK1_AVP()" in trace, which leads to sched_switch() after some
    steps, including _sleep().

    (b) cron is sleeping with sched_switch() after vfork() (Oh!
    Impossibility to create processes!)

    (c) All other userland processes are traced up to sched_switch() from
    different syscalls, via _sleep()...

    I don't reboot this computer for now, and waiting for requests which
    can help diagnose this situation

    --
    // Black Lion AKA Lev Serebryakov

    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

  2. Re: FreeBSD 7-STABLE deadlock!

    Lev Serebryakov wrote:

    >> Sometimes my FreeBSD 7 machine becomes semi-dead: all network sttack
    >> is working (pingable, etc), all LIVE processes are live, but it is
    >> impossible to create new (and, it seems, EXIT finished) process.
    > It is SCHED_ULE
    It seems to be ATA/SATA or UFS2 problem: now I have computer in state,
    when 4 iozone processes are hanged in "Disk wait" state, and I can not
    "cd" to filesystem, which is tested by "iozone".
    But I can create processes, work on system, etc., if I don't touch
    this filesystem.

    --
    // Lev Serebryakov
    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

  3. Re: FreeBSD 7-STABLE deadlock!

    Lev Serebryakov wrote:

    > It seems to be ATA/SATA or UFS2 problem: now I have computer in state,
    > when 4 iozone processes are hanged in "Disk wait" state, and I can not
    > "cd" to filesystem, which is tested by "iozone".
    > But I can create processes, work on system, etc., if I don't touch this
    > filesystem.
    I can reproduce it, creating gmirror on 5 disks (yes, not very useful
    configuration, but I've started from non-base-system RAID5 and need to
    exclude it), FS with 64Kb blocks, and 4 threads of iozone with mixed
    workload (-i 8 -+p 70).

    All 5 disks are ICH9DO-based, SATA-II WD5000AAKS HDDs.

    --
    // Lev Serebryakov
    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

  4. Re: FreeBSD 7-STABLE deadlock!

    On Monday 16 June 2008 07:21:15 am Lev Serebryakov wrote:
    > Lev Serebryakov wrote:
    > > It seems to be ATA/SATA or UFS2 problem: now I have computer in state,
    > > when 4 iozone processes are hanged in "Disk wait" state, and I can not
    > > "cd" to filesystem, which is tested by "iozone".
    > > But I can create processes, work on system, etc., if I don't touch this
    > > filesystem.
    >
    > I can reproduce it, creating gmirror on 5 disks (yes, not very useful
    > configuration, but I've started from non-base-system RAID5 and need to
    > exclude it), FS with 64Kb blocks, and 4 threads of iozone with mixed
    > workload (-i 8 -+p 70).
    >
    > All 5 disks are ICH9DO-based, SATA-II WD5000AAKS HDDs.

    Try getting the 'ps' output from ddb. Also, get a crash dump if you can.

    --
    John Baldwin
    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

  5. Re[2]: FreeBSD 7-STABLE deadlock!

    Hello, John.
    You wrote 23 2008 ., 18:47:33:

    > On Monday 16 June 2008 07:21:15 am Lev Serebryakov wrote:
    >> Lev Serebryakov wrote:
    >> > It seems to be ATA/SATA or UFS2 problem: now I have computer in state,
    >> > when 4 iozone processes are hanged in "Disk wait" state, and I can not
    >> > "cd" to filesystem, which is tested by "iozone".
    >> > But I can create processes, work on system, etc., if I don't touch this
    >> > filesystem.
    >>
    >> I can reproduce it, creating gmirror on 5 disks (yes, not very useful
    >> configuration, but I've started from non-base-system RAID5 and need to
    >> exclude it), FS with 64Kb blocks, and 4 threads of iozone with mixed
    >> workload (-i 8 -+p 70).
    >>
    >> All 5 disks are ICH9DO-based, SATA-II WD5000AAKS HDDs.
    > Try getting the 'ps' output from ddb. Also, get a crash dump if you can.
    It was tracked douwn to known deadlock in buffer allocator when
    buffer map is fragmented (thnx to kib@!). Workaround is known: don't
    use FSes with 16Kb and 64Kb blocks on same system in one time. 16/32
    mixture works well


    --
    // Black Lion AKA Lev Serebryakov

    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

  6. Re: FreeBSD 7-STABLE deadlock!

    Lev Serebryakov wrote:
    > Hello, John.
    > You wrote 23 2008 ., 18:47:33:
    >
    >> On Monday 16 June 2008 07:21:15 am Lev Serebryakov wrote:
    >>> Lev Serebryakov wrote:
    >>>> It seems to be ATA/SATA or UFS2 problem: now I have computer in state,
    >>>> when 4 iozone processes are hanged in "Disk wait" state, and I can not
    >>>> "cd" to filesystem, which is tested by "iozone".
    >>>> But I can create processes, work on system, etc., if I don't touch this
    >>>> filesystem.
    >>> I can reproduce it, creating gmirror on 5 disks (yes, not very useful
    >>> configuration, but I've started from non-base-system RAID5 and need to
    >>> exclude it), FS with 64Kb blocks, and 4 threads of iozone with mixed
    >>> workload (-i 8 -+p 70).
    >>>
    >>> All 5 disks are ICH9DO-based, SATA-II WD5000AAKS HDDs.
    >> Try getting the 'ps' output from ddb. Also, get a crash dump if you can.
    > It was tracked douwn to known deadlock in buffer allocator when
    > buffer map is fragmented (thnx to kib@!). Workaround is known: don't
    > use FSes with 16Kb and 64Kb blocks on same system in one time. 16/32
    > mixture works well
    >
    >

    Is there a PR filed with this bug? Having the specific information
    recorded will be very useful.

    Kris

    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

  7. Re[2]: FreeBSD 7-STABLE deadlock!

    Hello, Kris.
    You wrote 23 2008 ., 19:56:14:

    > Is there a PR filed with this bug? Having the specific information
    > recorded will be very useful.
    Kostik (kib@) says, that I don't need to fill PR for this issue...

    --
    // Black Lion AKA Lev Serebryakov

    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

  8. Re: FreeBSD 7-STABLE deadlock!

    Lev Serebryakov wrote:
    > Hello, Kris.
    > You wrote 23 июня 2008 г., 19:56:14:
    >
    >> Is there a PR filed with this bug? Having the specific information
    >> recorded will be very useful.
    > Kostik (kib@) says, that I don't need to fill PR for this issue...
    >

    OK, that is good enough for me

    Kris
    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

+ Reply to Thread
« Re: CALL FOR TEST: Updates to pixmap, cairo, poppler,and firefox 3.0! | Failure building apache22 and mysql51 »