Strange system hangs - Kernel

This is a discussion on Strange system hangs - Kernel ; Hello, I am experiencing weird system hangs. Once about 2-5 weeks system freezes and stops accepting remote connections, so it is no longer possible to connect to most important services: smtp (postfix), www (squid) or even ssh. Such connection is ...

+ Reply to Thread
Results 1 to 13 of 13

Thread: Strange system hangs

  1. Strange system hangs

    Hello,

    I am experiencing weird system hangs. Once about 2-5 weeks system freezes
    and stops accepting remote connections, so it is no longer possible to
    connect to most important services: smtp (postfix), www (squid) or even
    ssh. Such connection is accepted but then it hangs.

    What is strange, that previously established ssh session is usable. It is
    possible to work on such system until you do something stupid like "less
    /var/log/all.log". Using strace I found that process blocks on:

    --- strace: being ---
    execve("/usr/bin/tail", ["tail", "-f", "/var/log/all.log"], [/* 33 vars */]) = 0
    brk(0) = 0x8052000
    mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x6ff00000
    access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
    open("/etc/ld.so.cache", O_RDONLY) = 3
    fstat64(3, {st_mode=S_IFREG|0644, st_size=20944, ...}) = 0
    mmap2(NULL, 20944, PROT_READ, MAP_PRIVATE, 3, 0) = 0x6fefa000
    close(3) = 0
    open("/lib/libc.so.6", O_RDONLY) = 3
    read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0RY\ 1\0004\0\0\0"..., 512) = 512
    fstat64(3, {st_mode=S_IFREG|0755, st_size=1175920, ...}) = 0
    mmap2(NULL, 1185212, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x6fdd8000
    mmap2(0x6fef4000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x11b) = 0x6fef4000
    mmap2(0x6fef7000, 9660, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x6fef7000
    close(3) = 0
    mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x6fdd7000
    set_thread_area({entry_number:-1 -> 6, base_addr:0x6fdd76b0, limit:1048575,seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
    mprotect(0x6fef4000, 4096, PROT_READ) = 0
    mprotect(0x6ff1c000, 4096, PROT_READ) = 0
    munmap(0x6fefa000, 20944) = 0
    brk(0) = 0x8052000
    brk(0x8073000) = 0x8073000
    open("/var/log/all.log", O_RDONLY|O_LARGEFILE) = 3
    fstat64(3, {st_mode=S_IFREG|0640, st_size=3171841, ...})
    llseek(3, 0,
    --- strace: end ---

    This file is not very big:

    # ls -l /var/log/all.log
    -rw-r----- 1 root root 3171841 Sep 27 04:36 /var/log/all.log

    Also running "dmesg > file" hangs, creating a file with only 4096 bytes.

    --- Show Blocked State: begin ---
    SysRq : Show Blocked State

    free sibling
    task PC stack pid father child younger older
    syslogd D F5C83C60 0 2162 1 (NOTLB)
    f5c83c74 00000082 00000002 f5c83c60 f5c83c5c 00000000 00000000 78538d20
    00000009 00000001 f7f6a070 f7cb8030 82c47e5f 0001cfed 00000a43 f7f6a17c
    7a016980 f705dc80 78404217 7812c708 00000000 00000213 f5c83c84 1e7a64bb
    Call Trace:
    [<78404217>] _spin_unlock_irqrestore+0xf/0x23
    [<7812c708>] __mod_timer+0x92/0x9c
    [<78402b34>] schedule_timeout+0x70/0x8d
    [<7812c521>] process_timeout+0x0/0x5
    [<78402548>] io_schedule_timeout+0x1e/0x28
    [<7814d41e>] congestion_wait+0x50/0x64
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<781493e7>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<78145bd0>] generic_file_buffered_write+0x4ee/0x605
    [<783c55a1>] unix_dgram_recvmsg+0x1b4/0x1c8
    [<78128c8e>] current_fs_time+0x41/0x46
    [<78146167>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7814621b>] generic_file_aio_write+0x55/0xb3
    [<78194b28>] ext3_file_write+0x24/0x8f
    [<7815f34f>] do_sync_readv_writev+0xc1/0xfe
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<784041ae>] _spin_unlock+0xd/0x21
    [<781a8c38>] log_wait_commit+0xc3/0xe3
    [<7814448b>] find_get_pages_tag+0x76/0x80
    [<7815f204>] rw_copy_check_uvector+0x50/0xaa
    [<7815f9d4>] do_readv_writev+0x99/0x164
    [<78194b04>] ext3_file_write+0x0/0x8f
    [<7815fadc>] vfs_writev+0x3d/0x48
    [<7815feb5>] sys_writev+0x41/0x67
    [<78103d6a>] sysenter_past_esp+0x5f/0x85
    =======================
    freshclam D 00000282 0 2866 1 (NOTLB)
    f36e3cc4 00000082 00000009 00000282 7a0173c0 00000002 00000000 0000007b
    00000009 00000001 f7cb8030 f7c72030 82c4884d 0001cfed 000009ee f7cb813c
    7a016980 f66c0b80 78404217 7812c708 00000000 00000213 f36e3cd4 1e7a64bb
    Call Trace:
    [<78404217>] _spin_unlock_irqrestore+0xf/0x23
    [<7812c708>] __mod_timer+0x92/0x9c
    [<78402b34>] schedule_timeout+0x70/0x8d
    [<7812c521>] process_timeout+0x0/0x5
    [<78402548>] io_schedule_timeout+0x1e/0x28
    [<7814d41e>] congestion_wait+0x50/0x64
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<781493e7>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<78145bd0>] generic_file_buffered_write+0x4ee/0x605
    [<7819cdb4>] __ext3_journal_stop+0x19/0x34
    [<7840408f>] _spin_lock+0xd/0x5a
    [<78176f3d>] __mark_inode_dirty+0xdd/0x16f
    [<78128c8e>] current_fs_time+0x41/0x46
    [<78146167>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7814621b>] generic_file_aio_write+0x55/0xb3
    [<78103159>] setup_sigcontext+0x105/0x189
    [<78194b28>] ext3_file_write+0x24/0x8f
    [<7815f453>] do_sync_write+0xc7/0x10a
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<781085d2>] convert_fxsr_from_user+0x15/0xd5
    [<7815f38c>] do_sync_write+0x0/0x10a
    [<7815fbb6>] vfs_write+0x8a/0x10c
    [<78160123>] sys_write+0x41/0x67
    [<78103d6a>] sysenter_past_esp+0x5f/0x85
    =======================
    crond D 00000140 0 3029 1 (NOTLB)
    f1577cc4 00000082 00000008 00000140 7a00f3c0 00000002 00000000 00026b9a
    00000009 00000001 7a1d8030 e89f0070 82c469af 0001cfed 00000ad9 7a1d813c
    7a016980 f73afb40 78404217 7812c708 00000000 00000213 f1577cd4 1e7a64bb
    Call Trace:
    [<78404217>] _spin_unlock_irqrestore+0xf/0x23
    [<7812c708>] __mod_timer+0x92/0x9c
    [<78402b34>] schedule_timeout+0x70/0x8d
    [<7812c521>] process_timeout+0x0/0x5
    [<78402548>] io_schedule_timeout+0x1e/0x28
    [<7814d41e>] congestion_wait+0x50/0x64
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<781493e7>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<78145bd0>] generic_file_buffered_write+0x4ee/0x605
    [<7819cdb4>] __ext3_journal_stop+0x19/0x34
    [<7840408f>] _spin_lock+0xd/0x5a
    [<78176f3d>] __mark_inode_dirty+0xdd/0x16f
    [<78128c8e>] current_fs_time+0x41/0x46
    [<78146167>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7814621b>] generic_file_aio_write+0x55/0xb3
    [<78194b28>] ext3_file_write+0x24/0x8f
    [<7815f453>] do_sync_write+0xc7/0x10a
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<78162281>] sys_stat64+0x1e/0x23
    [<7815f38c>] do_sync_write+0x0/0x10a
    [<7815fbb6>] vfs_write+0x8a/0x10c
    [<78160123>] sys_write+0x41/0x67
    [<78103d6a>] sysenter_past_esp+0x5f/0x85
    [<78400000>] svcauth_gss_accept+0x8a3/0xb2b
    =======================
    ntpd D 00000140 0 3237 1 (NOTLB)
    ed027cc4 00200082 00000006 00000140 7a00f3c0 00000002 00000000 888925f9
    00000009 00000001 7a1c8070 ea1d2a50 82c49c08 0001cfed 00000a11 7a1c817c
    7a016980 f66c0580 78404217 7812c708 00000000 00200213 ed027cd4 1e7a64bb
    Call Trace:
    [<78404217>] _spin_unlock_irqrestore+0xf/0x23
    [<7812c708>] __mod_timer+0x92/0x9c
    [<78402b34>] schedule_timeout+0x70/0x8d
    [<7810477b>] common_interrupt+0x23/0x28
    [<7812c521>] process_timeout+0x0/0x5
    [<78402548>] io_schedule_timeout+0x1e/0x28
    [<7814d41e>] congestion_wait+0x50/0x64
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<781493e7>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<78145bd0>] generic_file_buffered_write+0x4ee/0x605
    [<78128c8e>] current_fs_time+0x41/0x46
    [<78146167>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7814621b>] generic_file_aio_write+0x55/0xb3
    [<781484b8>] get_page_from_freelist+0x278/0x325
    [<78194b28>] ext3_file_write+0x24/0x8f
    [<7815f453>] do_sync_write+0xc7/0x10a
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<7815f38c>] do_sync_write+0x0/0x10a
    [<7815fbb6>] vfs_write+0x8a/0x10c
    [<78160123>] sys_write+0x41/0x67
    [<78103d6a>] sysenter_past_esp+0x5f/0x85
    =======================
    snmpd D EB4EDCB0 0 3506 1 (NOTLB)
    eb4edcc4 00200086 00000002 eb4edcb0 eb4edcac 00000000 ebdbe100 000f4240
    00000007 00000001 eb94e030 ed65a030 82c4e0b2 0001cfed 00000a63 eb94e13c
    7a016980 ebdbe100 78404217 7812c708 00000000 00200213 eb4edcd4 1e7a64bb
    Call Trace:
    [<78404217>] _spin_unlock_irqrestore+0xf/0x23
    [<7812c708>] __mod_timer+0x92/0x9c
    [<78402b34>] schedule_timeout+0x70/0x8d
    [<7812c521>] process_timeout+0x0/0x5
    [<78402548>] io_schedule_timeout+0x1e/0x28
    [<7814d41e>] congestion_wait+0x50/0x64
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<781493e7>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<78145bd0>] generic_file_buffered_write+0x4ee/0x605
    [<78128c8e>] current_fs_time+0x41/0x46
    [<78146167>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7814621b>] generic_file_aio_write+0x55/0xb3
    [<7819cdb4>] __ext3_journal_stop+0x19/0x34
    [<78194b28>] ext3_file_write+0x24/0x8f
    [<7815f453>] do_sync_write+0xc7/0x10a
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<7815f38c>] do_sync_write+0x0/0x10a
    [<7815fbb6>] vfs_write+0x8a/0x10c
    [<78160123>] sys_write+0x41/0x67
    [<78103d6a>] sysenter_past_esp+0x5f/0x85
    [<78400000>] svcauth_gss_accept+0x8a3/0xb2b
    =======================
    squid D 00000140 0 3565 3563 (NOTLB)
    eb2f7cc4 00000086 00000004 00000140 7a00f3c0 00000002 00000000 f7fac000
    00000009 00000001 f7c72030 7a1c8070 82c491f7 0001cfed 000009aa f7c7213c
    7a016980 ebdbeb00 78404217 7812c708 00000000 00000213 eb2f7cd4 1e7a64bb
    Call Trace:
    [<78404217>] _spin_unlock_irqrestore+0xf/0x23
    [<7812c708>] __mod_timer+0x92/0x9c
    [<78402b34>] schedule_timeout+0x70/0x8d
    [<7812c521>] process_timeout+0x0/0x5
    [<78402548>] io_schedule_timeout+0x1e/0x28
    [<7814d41e>] congestion_wait+0x50/0x64
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<781493e7>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<78145bd0>] generic_file_buffered_write+0x4ee/0x605
    [<7819cdb4>] __ext3_journal_stop+0x19/0x34
    [<7840408f>] _spin_lock+0xd/0x5a
    [<78176f3d>] __mark_inode_dirty+0xdd/0x16f
    [<78128c8e>] current_fs_time+0x41/0x46
    [<78146167>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7814621b>] generic_file_aio_write+0x55/0xb3
    [<782bc211>] e1000_alloc_rx_buffers+0x1bb/0x280
    [<78194b28>] ext3_file_write+0x24/0x8f
    [<7815f453>] do_sync_write+0xc7/0x10a
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<78102d86>] __switch_to+0x108/0x127
    [<784023bb>] __sched_text_start+0x78b/0x7b2
    [<78374bb5>] net_rx_action+0x13a/0x17b
    [<7815f38c>] do_sync_write+0x0/0x10a
    [<7815fbb6>] vfs_write+0x8a/0x10c
    [<78160123>] sys_write+0x41/0x67
    [<78103d6a>] sysenter_past_esp+0x5f/0x85
    [<78400000>] svcauth_gss_accept+0x8a3/0xb2b
    =======================
    zabbix_agentd D E9C41E34 0 3687 1 (NOTLB)
    e9c41e48 00200082 00000002 e9c41e34 e9c41e30 00000000 00000310 ea20cb84
    00000008 00000001 eb94e540 f7f92030 ad13acaf 0001ce8d 00012a82 eb94e64c
    7a016980 eab33b40 1e634329 00000003 00000000 00000000 e9c41e60 9d739158
    Call Trace:
    [<7840324c>] __mutex_lock_slowpath+0x10f/0x20e
    [<78146206>] generic_file_aio_write+0x40/0xb3
    [<781484b8>] get_page_from_freelist+0x278/0x325
    [<78194b28>] ext3_file_write+0x24/0x8f
    [<7815f453>] do_sync_write+0xc7/0x10a
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<7815f38c>] do_sync_write+0x0/0x10a
    [<7815fbb6>] vfs_write+0x8a/0x10c
    [<78160123>] sys_write+0x41/0x67
    [<78103d6a>] sysenter_past_esp+0x5f/0x85
    =======================
    zabbix_agentd D EB6C5CB0 0 3754 3687 (NOTLB)
    eb6c5cc4 00200082 00000002 eb6c5cb0 eb6c5cac 00000000 f7c81574 00000001
    00000008 00000001 ed65a030 f7f92030 82c4ea91 0001cfed 000009df ed65a13c
    7a016980 f5cc5c40 1e7a6458 00000003 00000000 00000000 eb6c5cd4 1e7a64bb
    Call Trace:
    [<78402b34>] schedule_timeout+0x70/0x8d
    [<7812c521>] process_timeout+0x0/0x5
    [<78402548>] io_schedule_timeout+0x1e/0x28
    [<7814d41e>] congestion_wait+0x50/0x64
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<781493e7>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<78145bd0>] generic_file_buffered_write+0x4ee/0x605
    [<7819cdb4>] __ext3_journal_stop+0x19/0x34
    [<7840408f>] _spin_lock+0xd/0x5a
    [<78176f3d>] __mark_inode_dirty+0xdd/0x16f
    [<78128c8e>] current_fs_time+0x41/0x46
    [<78146167>] __generic_file_aio_write_nolock+0x480/0x4df
    [<781710e9>] mntput_no_expire+0x11/0x63
    [<7814621b>] generic_file_aio_write+0x55/0xb3
    [<781484b8>] get_page_from_freelist+0x278/0x325
    [<78194b28>] ext3_file_write+0x24/0x8f
    [<7815f453>] do_sync_write+0xc7/0x10a
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<7815f38c>] do_sync_write+0x0/0x10a
    [<7815fbb6>] vfs_write+0x8a/0x10c
    [<78160123>] sys_write+0x41/0x67
    [<78103d6a>] sysenter_past_esp+0x5f/0x85
    =======================
    amavisd D 00000270 0 19280 1 (NOTLB)
    c2563e2c 00000082 00000005 00000270 7a0173c0 00000002 00000000 00000000
    00000008 00000001 f7e86a90 944bc070 82c4b069 0001cfed 000009e0 f7e86b9c
    7a016980 c13f6680 1e7a6458 00000003 00000000 00000270 c2563e3c 1e7a64bb
    Call Trace:
    [<78402b34>] schedule_timeout+0x70/0x8d
    [<7812c521>] process_timeout+0x0/0x5
    [<78402548>] io_schedule_timeout+0x1e/0x28
    [<7814d41e>] congestion_wait+0x50/0x64
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<781493e7>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<7814e32c>] do_wp_page+0x3bb/0x3fd
    [<7814f6da>] __handle_mm_fault+0x7b4/0x82c
    [<7812f815>] __atomic_notifier_call_chain+0x3f/0x5a
    [<78119744>] do_page_fault+0x20d/0x52d
    [<78138135>] do_gettimeofday+0x31/0xd4
    [<78119537>] do_page_fault+0x0/0x52d
    [<784043ea>] error_code+0x72/0x78
    [<78400000>] svcauth_gss_accept+0x8a3/0xb2b
    =======================
    amavisd D 00000001 0 19570 1 (NOTLB)
    97f79ccc 00000006 f7c81574 00000001 0000000a 7822768f 97f79c80 97f79c80
    00000009 00000001 e89f0070 f7f6a070 82c4741c 0001cfed 00000a6d e89f017c
    7a016980 78622bc0 78404217 7812c708 00000000 00000213 97f79cdc 1e7a64bb
    Call Trace:
    [<7822768f>] blk_done_softirq+0x44/0x4f
    [<78404217>] _spin_unlock_irqrestore+0xf/0x23
    [<7812c708>] __mod_timer+0x92/0x9c
    [<78402b34>] schedule_timeout+0x70/0x8d
    [<7810477b>] common_interrupt+0x23/0x28
    [<7812c521>] process_timeout+0x0/0x5
    [<78402548>] io_schedule_timeout+0x1e/0x28
    [<7814d41e>] congestion_wait+0x50/0x64
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<781493e7>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<78145bd0>] generic_file_buffered_write+0x4ee/0x605
    [<7819cdb4>] __ext3_journal_stop+0x19/0x34
    [<7840408f>] _spin_lock+0xd/0x5a
    [<78128c8e>] current_fs_time+0x41/0x46
    [<78146167>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7814621b>] generic_file_aio_write+0x55/0xb3
    [<78176f3d>] __mark_inode_dirty+0xdd/0x16f
    [<78194b28>] ext3_file_write+0x24/0x8f
    [<7815f453>] do_sync_write+0xc7/0x10a
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<7815f38c>] do_sync_write+0x0/0x10a
    [<7815fbb6>] vfs_write+0x8a/0x10c
    [<781601f0>] sys_pwrite64+0x48/0x5f
    [<78103d6a>] sysenter_past_esp+0x5f/0x85
    [<78400000>] svcauth_gss_accept+0x8a3/0xb2b
    =======================
    sadc D 00000140 0 21692 1 (NOTLB)
    a830bcc4 00000082 00000002 00000140 7a00f3c0 00000002 00000000 00000002
    00000007 00000001 ebff0a90 eb94e030 82c4d64f 0001cfed 00000a11 ebff0b9c
    7a016980 f5cc5040 78404217 7812c708 00000000 00000213 a830bcd4 1e7a64bb
    Call Trace:
    [<78404217>] _spin_unlock_irqrestore+0xf/0x23
    [<7812c708>] __mod_timer+0x92/0x9c
    [<78402b34>] schedule_timeout+0x70/0x8d
    [<7812c521>] process_timeout+0x0/0x5
    [<78402548>] io_schedule_timeout+0x1e/0x28
    [<7814d41e>] congestion_wait+0x50/0x64
    [<78134abc>] autoremove_wake_function+0x0/0x35
    [<781493e7>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    --- Show Blocked State: end ---

    --- cat /proc/interrupts: begin ---
    CPU0 CPU1
    0: 541 0 IO-APIC-edge timer
    1: 56 0 IO-APIC-edge i8042
    6: 3 0 IO-APIC-edge floppy
    8: 43 0 IO-APIC-edge rtc
    9: 0 0 IO-APIC-fasteoi acpi
    12: 1418 0 IO-APIC-edge i8042
    14: 26 0 IO-APIC-edge libata
    15: 0 0 IO-APIC-edge libata
    17: 0 0 IO-APIC-fasteoi uhci_hcd:usb3
    18: 0 0 IO-APIC-fasteoi uhci_hcd:usb4
    20: 106711998 0 IO-APIC-fasteoi eth0
    21: 42948 0 IO-APIC-fasteoi eth1
    22: 10816027 0 IO-APIC-fasteoi libata, ehci_hcd:usb1, uhci_hcd:usb2
    NMI: 0 0
    LOC: 86943769 86934941
    ERR: 0
    --- cat /proc/interrupts: end ---

    --- dmesg: begin ---
    Linux version 2.6.22.6 (root@cougar) (gcc version 4.1.2 (Gentoo 4.1.2)) #1 SMP PREEMPT Fri Sep 21 11:41:24 CEST 2007
    BIOS-provided physical RAM map:
    BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
    BIOS-e820: 0000000000100000 - 000000007ffc0000 (usable)
    BIOS-e820: 000000007ffc0000 - 000000007ffcfc00 (ACPI data)
    BIOS-e820: 000000007ffcfc00 - 000000007ffff000 (reserved)
    BIOS-e820: 00000000f0000000 - 00000000f4000000 (reserved)
    BIOS-e820: 00000000fec00000 - 00000000fed00400 (reserved)
    BIOS-e820: 00000000fed13000 - 00000000feda0000 (reserved)
    BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
    BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
    2047MB LOWMEM available.
    found SMP MP-table at 000fe710
    Entering add_active_range(0, 0, 524224) 0 entries of 256 used
    Zone PFN ranges:
    DMA 0 -> 4096
    Normal 4096 -> 524224
    early_node_map[1] active PFN ranges
    0: 0 -> 524224
    On node 0 totalpages: 524224
    DMA zone: 32 pages used for memmap
    DMA zone: 0 pages reserved
    DMA zone: 4064 pages, LIFO batch:0
    Normal zone: 4063 pages used for memmap
    Normal zone: 516065 pages, LIFO batch:31
    DMI 2.3 present.
    ACPI: RSDP 000FD160, 0014 (r0 DELL )
    ACPI: RSDT 000FD174, 0038 (r1 DELL PE830 1 MSFT 100000A)
    ACPI: FACP 000FD1B8, 0074 (r1 DELL PE830 1 MSFT 100000A)
    ACPI: DSDT 7FFC0000, 1C19 (r1 DELL PE830 1 MSFT 100000E)
    ACPI: FACS 7FFCFC00, 0040
    ACPI: APIC 000FD22C, 0074 (r1 DELL PE830 1 MSFT 100000A)
    ACPI: SPCR 000FD2A0, 0050 (r1 DELL PE830 1 MSFT 100000A)
    ACPI: HPET 000FD2F0, 0038 (r1 DELL PE830 1 MSFT 100000A)
    ACPI: MCFG 000FD328, 003C (r1 DELL PE830 1 MSFT 100000A)
    ACPI: PM-Timer IO Port: 0x808
    ACPI: Local APIC address 0xfee00000
    ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
    Processor #0 15:4 APIC version 20
    ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
    Processor #1 15:4 APIC version 20
    ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
    ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
    ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
    IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
    ACPI: IOAPIC (id[0x03] address[0xfec10000] gsi_base[32])
    IOAPIC[1]: apic_id 3, version 32, address 0xfec10000, GSI 32-55
    ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
    ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
    ACPI: IRQ0 used by override.
    ACPI: IRQ2 used by override.
    ACPI: IRQ9 used by override.
    Enabling APIC mode: Flat. Using 2 I/O APICs
    ACPI: HPET id: 0xffffffff base: 0xfed00000
    Using ACPI (MADT) for SMP configuration information
    Allocating PCI resources starting at 80000000 (gap: 7ffff000:70001000)
    Built 1 zonelists. Total pages: 520129
    Kernel command line: auto BOOT_IMAGE=Linux-2.6.22.6 ro root=900 rootflags=data=journal nf_conntrack.hashsize=32768
    mapped APIC to ffffd000 (fee00000)
    mapped IOAPIC to ffffc000 (fec00000)
    mapped IOAPIC to ffffb000 (fec10000)
    Enabling fast FPU save and restore... done.
    Enabling unmasked SIMD FPU exception support... done.
    Initializing CPU#0
    PID hash table entries: 4096 (order: 12, 16384 bytes)
    Detected 3200.405 MHz processor.
    Console: colour VGA+ 80x30
    Dentry cache hash table entries: 262144 (order: 8, 1048576 bytes)
    Inode-cache hash table entries: 131072 (order: 7, 524288 bytes)
    Memory: 2073468k/2096896k available (3092k kernel code, 23040k reserved, 1269k data, 220k init, 0k highmem)
    virtual kernel memory layout:
    fixmap : 0xfffb7000 - 0xfffff000 ( 288 kB)
    vmalloc : 0xf8800000 - 0xfffb5000 ( 119 MB)
    lowmem : 0x78000000 - 0xf7fc0000 (2047 MB)
    .init : 0x7854b000 - 0x78582000 ( 220 kB)
    .data : 0x784051cb - 0x7854280c (1269 kB)
    .text : 0x78100000 - 0x784051cb (3092 kB)
    Checking if this processor honours the WP bit even in supervisor mode... Ok.
    hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
    hpet0: 3 64-bit timers, 14318180 Hz
    Calibrating delay using timer specific routine.. 6404.65 BogoMIPS (lpj=3202326)
    Mount-cache hash table entries: 512
    CPU: After generic identify, caps: bfebfbff 20100000 00000000 00000000 0000649d 00000000 00000001
    monitor/mwait feature present.
    using mwait in idle threads.
    CPU: Trace cache: 12K uops, L1 D cache: 16K
    CPU: L2 cache: 1024K
    CPU: Physical Processor ID: 0
    CPU: Processor Core ID: 0
    CPU: After all inits, caps: bfebfbff 20100000 00000000 0000b180 0000649d 00000000 00000001
    Intel machine check architecture supported.
    Intel machine check reporting enabled on CPU#0.
    CPU0: Intel P4/Xeon Extended MCE MSRs (24) available
    CPU0: Thermal monitoring enabled
    Compat vDSO mapped to ffffe000.
    Checking 'hlt' instruction... OK.
    SMP alternatives: switching to UP code
    ACPI: Core revision 20070126
    CPU0: Intel(R) Pentium(R) D CPU 3.20GHz stepping 07
    SMP alternatives: switching to SMP code
    Booting processor 1/1 eip 3000
    Initializing CPU#1
    Calibrating delay using timer specific routine.. 6400.33 BogoMIPS (lpj=3200168)
    CPU: After generic identify, caps: bfebfbff 20100000 00000000 00000000 0000649d 00000000 00000001
    monitor/mwait feature present.
    CPU: Trace cache: 12K uops, L1 D cache: 16K
    CPU: L2 cache: 1024K
    CPU: Physical Processor ID: 0
    CPU: Processor Core ID: 1
    CPU: After all inits, caps: bfebfbff 20100000 00000000 0000b180 0000649d 00000000 00000001
    Intel machine check architecture supported.
    Intel machine check reporting enabled on CPU#1.
    CPU1: Intel P4/Xeon Extended MCE MSRs (24) available
    CPU1: Thermal monitoring enabled
    CPU1: Intel(R) Pentium(R) D CPU 3.20GHz stepping 07
    Total of 2 processors activated (12804.98 BogoMIPS).
    ENABLING IO-APIC IRQs
    ..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
    checking TSC synchronization [CPU#0 -> CPU#1]: passed.
    Brought up 2 CPUs
    migration_cost=291
    NET: Registered protocol family 16
    ACPI: bus type pci registered
    PCI: Using MMCONFIG
    Setting up standard PCI resources
    ACPI: Interpreter enabled
    ACPI: (supports S0 S4 S5)
    ACPI: Using IOAPIC for interrupt routing
    ACPI: PCI Root Bridge [PCI0] (0000:00)
    PCI: Probing PCI hardware (bus 00)
    PCI quirk: region 0800-087f claimed by ICH6 ACPI/GPIO/TCO
    PCI quirk: region 0880-08bf claimed by ICH6 GPIO
    PCI: PXH quirk detected, disabling MSI for SHPC device
    PCI: Transparent bridge - 0000:00:1e.0
    ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
    ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PES1._PRT]
    ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEP0._PRT]
    ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEP0.PXHA._PRT]
    ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEP1._PRT]
    ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEP2._PRT]
    ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCIS._PRT]
    ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 *10 11 12)
    ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 11 12) *0, disabled.
    ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11 12)
    ACPI: PCI Interrupt Link [LNKD] (IRQs *3 4 5 6 7 9 10 11 12)
    ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 *11 12)
    ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 *10 11 12)
    ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 *5 6 7 9 10 11 12)
    ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 11 12) *0, disabled.
    Linux Plug and Play Support v0.97 (c) Adam Belay
    pnp: PnP ACPI init
    ACPI: bus type pnp registered
    pnp: PnP ACPI: found 13 devices
    ACPI: ACPI bus type pnp unregistered
    SCSI subsystem initialized
    libata version 2.21 loaded.
    usbcore: registered new interface driver usbfs
    usbcore: registered new interface driver hub
    usbcore: registered new device driver usb
    PCI: Using ACPI for IRQ routing
    PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
    pnp: 00:09: ioport range 0x800-0x87f has been reserved
    pnp: 00:09: ioport range 0x880-0x8bf has been reserved
    pnp: 00:09: ioport range 0x8c0-0x8df has been reserved
    Time: tsc clocksource has been installed.
    pnp: 00:09: ioport range 0x8e0-0x8e3 has been reserved
    pnp: 00:09: ioport range 0xc00-0xc0f has been reserved
    pnp: 00:09: ioport range 0xc10-0xc1f has been reserved
    pnp: 00:09: ioport range 0xca0-0xca7 has been reserved
    pnp: 00:09: ioport range 0xca9-0xcab has been reserved
    pnp: 00:0b: iomem range 0xf0000000-0xf3ffffff could not be reserved
    PCI: Bridge: 0000:00:01.0
    IO window: disabled.
    MEM window: disabled.
    PREFETCH window: disabled.
    PCI: Bridge: 0000:02:00.0
    IO window: e000-efff
    MEM window: fe900000-feafffff
    PREFETCH window: disabled.
    PCI: Bridge: 0000:00:1c.0
    IO window: e000-efff
    MEM window: fe800000-feafffff
    PREFETCH window: disabled.
    PCI: Bridge: 0000:00:1c.4
    IO window: d000-dfff
    MEM window: fe600000-fe7fffff
    PREFETCH window: disabled.
    PCI: Bridge: 0000:00:1c.5
    IO window: disabled.
    MEM window: disabled.
    PREFETCH window: disabled.
    PCI: Bridge: 0000:00:1e.0
    IO window: c000-cfff
    MEM window: fe400000-fe5fffff
    PREFETCH window: fd000000-fdffffff
    ACPI: PCI Interrupt 0000:00:01.0[A] -> GSI 16 (level, low) -> IRQ 16
    PCI: Setting latency timer of device 0000:00:01.0 to 64
    ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 21 (level, low) -> IRQ 17
    PCI: Setting latency timer of device 0000:00:1c.0 to 64
    PCI: Setting latency timer of device 0000:02:00.0 to 64
    ACPI: PCI Interrupt 0000:00:1c.4[B] -> GSI 22 (level, low) -> IRQ 18
    PCI: Setting latency timer of device 0000:00:1c.4 to 64
    ACPI: PCI Interrupt 0000:00:1c.5[C] -> GSI 23 (level, low) -> IRQ 19
    PCI: Setting latency timer of device 0000:00:1c.5 to 64
    PCI: Setting latency timer of device 0000:00:1e.0 to 64
    NET: Registered protocol family 2
    IP route cache hash table entries: 65536 (order: 6, 262144 bytes)
    TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
    TCP bind hash table entries: 65536 (order: 7, 786432 bytes)
    TCP: Hash tables configured (established 262144 bind 65536)
    TCP reno registered
    Machine check exception polling timer started.
    IA-32 Microcode Update Driver: v1.14a
    Total HugeTLB memory allocated, 0
    io scheduler noop registered
    io scheduler anticipatory registered
    io scheduler deadline registered (default)
    io scheduler cfq registered
    Boot video device is 0000:06:05.0
    PCI: Setting latency timer of device 0000:00:01.0 to 64
    assign_interrupt_mode Found MSI capability
    Allocate Port Service[0000:00:01.0cie00]
    Allocate Port Service[0000:00:01.0cie03]
    PCI: Setting latency timer of device 0000:00:1c.0 to 64
    assign_interrupt_mode Found MSI capability
    Allocate Port Service[0000:00:1c.0cie00]
    Allocate Port Service[0000:00:1c.0cie03]
    PCI: Setting latency timer of device 0000:00:1c.4 to 64
    assign_interrupt_mode Found MSI capability
    Allocate Port Service[0000:00:1c.4cie00]
    Allocate Port Service[0000:00:1c.4cie03]
    PCI: Setting latency timer of device 0000:00:1c.5 to 64
    assign_interrupt_mode Found MSI capability
    Allocate Port Service[0000:00:1c.5cie00]
    Allocate Port Service[0000:00:1c.5cie02]
    Allocate Port Service[0000:00:1c.5cie03]
    sisfb: Video ROM found
    sisfb: Video RAM at 0xfd000000, mapped to 0xf8880000, size 16384k
    sisfb: MMIO at 0xfe4c0000, mapped to 0xf9900000, size 256k
    sisfb: Memory heap starting at 16160K, size 32K
    sisfb: CRT1 DDC supported
    sisfb: CRT1 DDC level: 2
    Switched to high resolution mode on CPU 1
    Switched to high resolution mode on CPU 0
    sisfb: Monitor range H 24-61KHz, V 56-75Hz, Max. dotclock 80MHz
    sisfb: Default mode is 800x600x8 (60Hz)
    sisfb: Initial vbflags 0x0
    Console: switching to colour frame buffer device 100x37
    sisfb: 2D acceleration is enabled, y-panning enabled (auto-max)
    fb0: XGI Z7 frame buffer device version 1.8.9
    sisfb: Copyright (C) 2001-2005 Thomas Winischhofer
    input: Power Button (FF) as /class/input/input0
    ACPI: Power Button (FF) [PWRF]
    Real Time Clock Driver v1.12ac
    hpet_resources: 0xfed00000 is busy
    intel_rng: FWH not detected
    Hangcheck: starting hangcheck timer 0.9.0 (tick is 180 seconds, margin is 60 seconds).
    Hangcheck: Using get_cycles().
    Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
    serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
    00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
    Floppy drive(s): fd0 is 1.44M
    FDC 0 is a National Semiconductor PC87306
    loop: module loaded
    Intel(R) PRO/1000 Network Driver - version 7.3.20-k2-NAPI
    Copyright (c) 1999-2006 Intel Corporation.
    ACPI: PCI Interrupt 0000:03:02.0[A] -> GSI 35 (level, low) -> IRQ 20
    e1000: 0000:03:02.0: e1000_probe: (PCI-X:133MHz:64-bit) 00:04:23:c7:a4:de
    e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
    ACPI: PCI Interrupt 0000:03:02.1[B] -> GSI 34 (level, low) -> IRQ 21
    e1000: 0000:03:02.1: e1000_probe: (PCI-X:133MHz:64-bit) 00:04:23:c7:a4:df
    e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
    Ethernet Channel Bonding Driver: v3.1.3 (June 13, 2007)
    bonding: Warning: either miimon or arp_interval and arp_ip_target module parameters must be specified, otherwise bonding will not detect link failures! see bonding.txt for details.
    tg3.c:v3.77 (May 31, 2007)
    ACPI: PCI Interrupt 0000:04:00.0[A] -> GSI 16 (level, low) -> IRQ 16
    PCI: Setting latency timer of device 0000:04:00.0 to 64
    eth2: Tigon3 [partno(BCM95721) rev 4101 PHY(5750)] (PCI Express) 10/100/1000Base-T Ethernet 00:13:72:3c:fd:fe
    eth2: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] WireSpeed[1] TSOcap[1]
    eth2: dma_rwctrl[76180000] dma_mask[64-bit]
    netconsole: not configured, aborting
    st: Version 20070203, fixed bufsize 32768, s/g segs 256
    ata_piix 0000:00:1f.1: version 2.11
    ACPI: Unable to derive IRQ for device 0000:00:1f.1
    ACPI: PCI Interrupt 0000:00:1f.1[A]: no GSI
    PCI: Setting latency timer of device 0000:00:1f.1 to 64
    scsi0 : ata_piix
    scsi1 : ata_piix
    ata1: PATA max UDMA/133 cmd 0x000101f0 ctl 0x000103f6 bmdma 0x0001fc00 irq 14
    ata2: PATA max UDMA/133 cmd 0x00010170 ctl 0x00010376 bmdma 0x0001fc08 irq 15
    ata1.00: ATAPI: TSSTcorpDVD-ROM TS-H352C, DE02, max UDMA/33
    ata1.01: ATAPI: Seagate STT3401A, 310B, max PIO3
    ata1.00: configured for UDMA/33
    ata1.01: configured for PIO3
    ata2: port disabled. ignoring.
    scsi 0:0:0:0: CD-ROM TSSTcorp DVD-ROM TS-H352C DE02 PQ: 0 ANSI: 5
    sr0: scsi3-mmc drive: 4x/48x cd/rw xa/form2 cdda tray
    Uniform CD-ROM driver Revision: 3.20
    sr 0:0:0:0: Attached scsi CD-ROM sr0
    sr 0:0:0:0: Attached scsi generic sg0 type 5
    scsi 0:0:1:0: Sequential-Access Seagate STT3401A 310B PQ: 0 ANSI: 2
    st 0:0:1:0: Attached scsi tape st0
    st 0:0:1:0: st0: try direct i/o: yes (alignment 512 B)
    st 0:0:1:0: Attached scsi generic sg1 type 1
    ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ]
    ACPI: PCI Interrupt 0000:00:1f.2[C] -> GSI 20 (level, low) -> IRQ 22
    PCI: Setting latency timer of device 0000:00:1f.2 to 64
    scsi2 : ata_piix
    scsi3 : ata_piix
    ata3: SATA max UDMA/133 cmd 0x0001bc98 ctl 0x0001bc92 bmdma 0x0001bc60 irq 22
    ata4: SATA max UDMA/133 cmd 0x0001bc80 ctl 0x0001bc7a bmdma 0x0001bc68 irq 22
    ata3.00: ATA-7: WDC WD2500JS-75NCB1, 10.02E01, max UDMA/133
    ata3.00: 488281250 sectors, multi 8: LBA48 NCQ (depth 0/32)
    ata3.01: ATA-7: Maxtor 7L250S0, BACE1G10, max UDMA/133
    ata3.01: 488281250 sectors, multi 8: LBA48 NCQ (depth 0/32)
    ata3.00: configured for UDMA/133
    ata3.01: configured for UDMA/133
    ata4.00: ATA-7: Maxtor 7L250S0, BACE1G10, max UDMA/133
    ata4.00: 488281250 sectors, multi 8: LBA48 NCQ (depth 0/32)
    ata4.00: configured for UDMA/133
    scsi 2:0:0:0: Direct-Access ATA WDC WD2500JS-75N 10.0 PQ: 0 ANSI: 5
    sd 2:0:0:0: [sda] 488281250 512-byte hardware sectors (250000 MB)
    sd 2:0:0:0: [sda] Write Protect is off
    sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
    sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
    sd 2:0:0:0: [sda] 488281250 512-byte hardware sectors (250000 MB)
    sd 2:0:0:0: [sda] Write Protect is off
    sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
    sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
    sda: sda1 sda2 sda3 sda4
    sd 2:0:0:0: [sda] Attached SCSI disk
    sd 2:0:0:0: Attached scsi generic sg2 type 0
    scsi 2:0:1:0: Direct-Access ATA Maxtor 7L250S0 BACE PQ: 0 ANSI: 5
    sd 2:0:1:0: [sdb] 488281250 512-byte hardware sectors (250000 MB)
    sd 2:0:1:0: [sdb] Write Protect is off
    sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
    sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
    sd 2:0:1:0: [sdb] 488281250 512-byte hardware sectors (250000 MB)
    sd 2:0:1:0: [sdb] Write Protect is off
    sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
    sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
    sdb: sdb1 sdb2 sdb3 sdb4
    sd 2:0:1:0: [sdb] Attached SCSI disk
    sd 2:0:1:0: Attached scsi generic sg3 type 0
    scsi 3:0:0:0: Direct-Access ATA Maxtor 7L250S0 BACE PQ: 0 ANSI: 5
    sd 3:0:0:0: [sdc] 488281250 512-byte hardware sectors (250000 MB)
    sd 3:0:0:0: [sdc] Write Protect is off
    sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
    sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
    sd 3:0:0:0: [sdc] 488281250 512-byte hardware sectors (250000 MB)
    sd 3:0:0:0: [sdc] Write Protect is off
    sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
    sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
    sdc: sdc1 sdc2 sdc3 sdc4
    sd 3:0:0:0: [sdc] Attached SCSI disk
    sd 3:0:0:0: Attached scsi generic sg4 type 0
    ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 20 (level, low) -> IRQ 22
    PCI: Setting latency timer of device 0000:00:1d.7 to 64
    ehci_hcd 0000:00:1d.7: EHCI Host Controller
    ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
    ehci_hcd 0000:00:1d.7: debug port 1
    PCI: cache line size of 128 is not supported by device 0000:00:1d.7
    ehci_hcd 0000:00:1d.7: irq 22, io mem 0xfeb00400
    ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
    usb usb1: configuration #1 chosen from 1 choice
    hub 1-0:1.0: USB hub found
    hub 1-0:1.0: 6 ports detected
    ohci_hcd: 2006 August 04 USB 1.1 'Open' Host Controller (OHCI) Driver
    USB Universal Host Controller Interface driver v3.0
    ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 20 (level, low) -> IRQ 22
    PCI: Setting latency timer of device 0000:00:1d.0 to 64
    uhci_hcd 0000:00:1d.0: UHCI Host Controller
    uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
    uhci_hcd 0000:00:1d.0: irq 22, io base 0x0000bce0
    usb usb2: configuration #1 chosen from 1 choice
    hub 2-0:1.0: USB hub found
    hub 2-0:1.0: 2 ports detected
    ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 21 (level, low) -> IRQ 17
    PCI: Setting latency timer of device 0000:00:1d.1 to 64
    uhci_hcd 0000:00:1d.1: UHCI Host Controller
    uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
    uhci_hcd 0000:00:1d.1: irq 17, io base 0x0000bcc0
    usb usb3: configuration #1 chosen from 1 choice
    hub 3-0:1.0: USB hub found
    hub 3-0:1.0: 2 ports detected
    ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 22 (level, low) -> IRQ 18
    PCI: Setting latency timer of device 0000:00:1d.2 to 64
    uhci_hcd 0000:00:1d.2: UHCI Host Controller
    uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4
    uhci_hcd 0000:00:1d.2: irq 18, io base 0x0000bca0
    usb usb4: configuration #1 chosen from 1 choice
    hub 4-0:1.0: USB hub found
    hub 4-0:1.0: 2 ports detected
    Initializing USB Mass Storage driver...
    usbcore: registered new interface driver usb-storage
    USB Mass Storage support registered.
    PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
    serio: i8042 KBD port at 0x60,0x64 irq 1
    serio: i8042 AUX port at 0x60,0x64 irq 12
    mice: PS/2 mouse device common for all mice
    input: AT Translated Set 2 keyboard as /class/input/input1
    md: raid0 personality registered for level 0
    md: raid1 personality registered for level 1
    md: raid10 personality registered for level 10
    raid6: int32x1 730 MB/s
    raid6: int32x2 742 MB/s
    raid6: int32x4 667 MB/s
    raid6: int32x8 507 MB/s
    raid6: mmxx1 1660 MB/s
    raid6: mmxx2 1781 MB/s
    raid6: sse1x1 1019 MB/s
    raid6: sse1x2 1113 MB/s
    raid6: sse2x1 2039 MB/s
    raid6: sse2x2 1851 MB/s
    raid6: using algorithm sse2x1 (2039 MB/s)
    md: raid6 personality registered for level 6
    md: raid5 personality registered for level 5
    md: raid4 personality registered for level 4
    raid5: automatically using best checksumming function: pIII_sse
    pIII_sse : 4008.000 MB/sec
    raid5: using function: pIII_sse (4008.000 MB/sec)
    md: faulty personality registered for level -5
    device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised: dm-devel@redhat.com
    device-mapper: multipath: version 1.0.5 loaded
    device-mapper: multipath round-robin: version 1.0.0 loaded
    EDAC MC: Ver: 2.0.1 Sep 21 2007
    input: ImExPS/2 Generic Explorer Mouse as /class/input/input2
    dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.2)
    usbcore: registered new interface driver usbhid
    drivers/hid/usbhid/hid-core.c: v2.6:USB HID core driver
    Netfilter messages via NETLINK v0.30.
    nf_conntrack version 0.5.0 (32768 buckets, 262144 max)
    ctnetlink v0.93: registering with nfnetlink.
    ip_tables: (C) 2000-2006 Netfilter Core Team
    IP_TPROXY: Transparent proxy support initialized, version 4.0.0
    IP_TPROXY: Copyright (c) 2002-2007 BalaBit IT Ltd.
    arp_tables: (C) 2002 David S. Miller
    TCP cubic registered
    NET: Registered protocol family 1
    NET: Registered protocol family 10
    ip6_tables: (C) 2000-2006 Netfilter Core Team
    IPv6 over IPv4 tunneling driver
    NET: Registered protocol family 17
    Using IPI No-Shortcut mode
    BIOS EDD facility v0.16 2004-Jun-25, 3 devices found
    md: Autodetecting RAID arrays.
    md: autorun ...
    md: considering sdc4 ...
    md: adding sdc4 ...
    md: sdc3 has different UUID to sdc4
    md: sdc2 has different UUID to sdc4
    md: adding sdb4 ...
    md: sdb3 has different UUID to sdc4
    md: sdb2 has different UUID to sdc4
    md: adding sda4 ...
    md: sda3 has different UUID to sdc4
    md: sda2 has different UUID to sdc4
    md: created md4
    md: bind
    md: bind
    md: bind
    md: running:
    raid5: device sdc4 operational as raid disk 2
    raid5: device sdb4 operational as raid disk 1
    raid5: device sda4 operational as raid disk 0
    raid5: allocated 3164kB for md4
    raid5: raid level 5 set md4 active with 3 out of 3 devices, algorithm 2
    RAID5 conf printout:
    --- rd:3 wd:3
    disk 0, o:1, dev:sda4
    disk 1, o:1, dev:sdb4
    disk 2, o:1, dev:sdc4
    md: considering sdc3 ...
    md: adding sdc3 ...
    md: sdc2 has different UUID to sdc3
    md: adding sdb3 ...
    md: sdb2 has different UUID to sdc3
    md: adding sda3 ...
    md: sda2 has different UUID to sdc3
    md: created md15
    md: bind
    md: bind
    md: bind
    md: running:
    raid10: raid set md15 active with 3 out of 3 devices
    md15: bitmap initialized from disk: read 12/12 pages, set 0 bits, status: 0
    created bitmap (179 pages) for device md15
    md: considering sdc2 ...
    md: adding sdc2 ...
    md: adding sdb2 ...
    md: adding sda2 ...
    md: created md0
    md: bind
    md: bind
    md: bind
    md: running:
    raid1: raid set md0 active with 3 out of 3 mirrors
    md0: bitmap initialized from disk: read 15/15 pages, set 0 bits, status: 0
    created bitmap (239 pages) for device md0
    md: ... autorun DONE.
    EXT3-fs: INFO: recovery required on readonly filesystem.
    EXT3-fs: write access will be enabled during recovery.
    kjournald starting. Commit interval 5 seconds
    EXT3-fs: md0: orphan cleanup on readonly fs
    ext3_orphan_cleanup: deleting unreferenced inode 130332
    EXT3-fs: md0: 1 orphan inode deleted
    EXT3-fs: recovery complete.
    EXT3-fs: mounted filesystem with journal data mode.
    VFS: Mounted root (ext3 filesystem) readonly.
    Freeing unused kernel memory: 220k freed
    EXT3 FS on md0, internal journal
    kjournald starting. Commit interval 5 seconds
    EXT3 FS on dm-0, internal journal
    EXT3-fs: mounted filesystem with journal data mode.
    kjournald starting. Commit interval 5 seconds
    EXT3 FS on dm-1, internal journal
    EXT3-fs: mounted filesystem with journal data mode.
    kjournald starting. Commit interval 5 seconds
    EXT3 FS on dm-2, internal journal
    EXT3-fs: mounted filesystem with writeback data mode.
    kjournald starting. Commit interval 5 seconds
    EXT3 FS on dm-4, internal journal
    EXT3-fs: mounted filesystem with writeback data mode.
    kjournald starting. Commit interval 5 seconds
    EXT3 FS on dm-3, internal journal
    EXT3-fs: mounted filesystem with ordered data mode.
    Adding 2927608k swap on /dev/md15. Priority:8192 extents:1 across:2927608k
    bonding: bond0: setting mode to active-backup (1).
    bonding: bond0: Setting MII monitoring interval to 100.
    ADDRCONF(NETDEV_UP): bond0: link is not ready
    bonding: bond0: Adding slave eth0.
    e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
    bonding: bond0: making interface eth0 the new active one.
    bonding: bond0: first active interface up!
    bonding: bond0: enslaving eth0 as an active interface with an up link.
    bonding: bond0: Adding slave eth1.
    ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
    ADDRCONF(NETDEV_UP): eth1: link is not ready
    bonding: bond0: enslaving eth1 as a backup interface with a down link.
    bonding: bond0: Setting eth0 as primary slave.
    eth0: no IPv6 routers present
    bond0: no IPv6 routers present
    device eth0 entered promiscuous mode
    device bond0 entered promiscuous mode
    device eth0 left promiscuous mode
    device bond0 left promiscuous mode
    device eth0 entered promiscuous mode
    device bond0 entered promiscuous mode
    device eth0 left promiscuous mode
    device bond0 left promiscuous mode
    --- dmesg: end ---

    Both dmesg and interrupts are from working system.

    This is a quite busy "Intel(R) Pentium(R) D CPU 3.20GHz" running mainly
    squid and postfix. There are three disks (ata_piix) and LVM on top of a
    software (md) raid5, with root fs on raid1 and swap on raid10.

    I have tested 2.6.20, 2.6.21 and 2.6.22 kernels so far and this problem
    exists with each release.

    Any ideas?

    Best regards,


    Krzysztof Olędzki

  2. Re: Strange system hangs

    On Fri, 2007-09-28 at 10:42 +0200, Krzysztof Oledzki wrote:
    > Hello,
    >
    > I am experiencing weird system hangs. Once about 2-5 weeks system freezes
    > and stops accepting remote connections, so it is no longer possible to
    > connect to most important services: smtp (postfix), www (squid) or even
    > ssh. Such connection is accepted but then it hangs.
    >
    > What is strange, that previously established ssh session is usable. It is
    > possible to work on such system until you do something stupid like "less
    > /var/log/all.log".


    So it takes weeks to reproduce this?


    > free sibling
    > task PC stack pid father child younger older
    > syslogd D F5C83C60 0 2162 1 (NOTLB)
    > f5c83c74 00000082 00000002 f5c83c60 f5c83c5c 00000000 00000000 78538d20
    > 00000009 00000001 f7f6a070 f7cb8030 82c47e5f 0001cfed 00000a43 f7f6a17c
    > 7a016980 f705dc80 78404217 7812c708 00000000 00000213 f5c83c84 1e7a64bb
    > Call Trace:
    > [<78404217>] _spin_unlock_irqrestore+0xf/0x23
    > [<7812c708>] __mod_timer+0x92/0x9c
    > [<78402b34>] schedule_timeout+0x70/0x8d
    > [<7812c521>] process_timeout+0x0/0x5
    > [<78402548>] io_schedule_timeout+0x1e/0x28
    > [<7814d41e>] congestion_wait+0x50/0x64
    > [<78134abc>] autoremove_wake_function+0x0/0x35
    > [<781493e7>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    > [<78145bd0>] generic_file_buffered_write+0x4ee/0x605
    > [<783c55a1>] unix_dgram_recvmsg+0x1b4/0x1c8
    > [<78128c8e>] current_fs_time+0x41/0x46
    > [<78146167>] __generic_file_aio_write_nolock+0x480/0x4df
    > [<7814621b>] generic_file_aio_write+0x55/0xb3
    > [<78194b28>] ext3_file_write+0x24/0x8f
    > [<7815f34f>] do_sync_readv_writev+0xc1/0xfe
    > [<78134abc>] autoremove_wake_function+0x0/0x35
    > [<784041ae>] _spin_unlock+0xd/0x21
    > [<781a8c38>] log_wait_commit+0xc3/0xe3
    > [<7814448b>] find_get_pages_tag+0x76/0x80
    > [<7815f204>] rw_copy_check_uvector+0x50/0xaa
    > [<7815f9d4>] do_readv_writev+0x99/0x164
    > [<78194b04>] ext3_file_write+0x0/0x8f
    > [<7815fadc>] vfs_writev+0x3d/0x48
    > [<7815feb5>] sys_writev+0x41/0x67
    > [<78103d6a>] sysenter_past_esp+0x5f/0x85
    > =======================


    This trace puzzles me, what is: unix_dgram_recvmsg doing there.
    Also, it has two invocations of: ext3_file_write
    do you have a stacked filesystem of sorts, ext3 on loopback on ext3?



    > freshclam D 00000282 0 2866 1 (NOTLB)
    > f36e3cc4 00000082 00000009 00000282 7a0173c0 00000002 00000000 0000007b
    > 00000009 00000001 f7cb8030 f7c72030 82c4884d 0001cfed 000009ee f7cb813c
    > 7a016980 f66c0b80 78404217 7812c708 00000000 00000213 f36e3cd4 1e7a64bb
    > Call Trace:
    > [<78404217>] _spin_unlock_irqrestore+0xf/0x23
    > [<7812c708>] __mod_timer+0x92/0x9c
    > [<78402b34>] schedule_timeout+0x70/0x8d
    > [<7812c521>] process_timeout+0x0/0x5
    > [<78402548>] io_schedule_timeout+0x1e/0x28
    > [<7814d41e>] congestion_wait+0x50/0x64
    > [<78134abc>] autoremove_wake_function+0x0/0x35
    > [<781493e7>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    > [<78145bd0>] generic_file_buffered_write+0x4ee/0x605
    > [<7819cdb4>] __ext3_journal_stop+0x19/0x34
    > [<7840408f>] _spin_lock+0xd/0x5a
    > [<78176f3d>] __mark_inode_dirty+0xdd/0x16f
    > [<78128c8e>] current_fs_time+0x41/0x46
    > [<78146167>] __generic_file_aio_write_nolock+0x480/0x4df
    > [<7814621b>] generic_file_aio_write+0x55/0xb3
    > [<78103159>] setup_sigcontext+0x105/0x189
    > [<78194b28>] ext3_file_write+0x24/0x8f
    > [<7815f453>] do_sync_write+0xc7/0x10a
    > [<78134abc>] autoremove_wake_function+0x0/0x35
    > [<781085d2>] convert_fxsr_from_user+0x15/0xd5
    > [<7815f38c>] do_sync_write+0x0/0x10a
    > [<7815fbb6>] vfs_write+0x8a/0x10c
    > [<78160123>] sys_write+0x41/0x67
    > [<78103d6a>] sysenter_past_esp+0x5f/0x85
    > =======================


    single write, no networking, also stuck in balance_dirty_pages().

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  3. Re: Strange system hangs



    On Fri, 28 Sep 2007, Peter Zijlstra wrote:

    > On Fri, 2007-09-28 at 10:42 +0200, Krzysztof Oledzki wrote:
    >> Hello,
    >>
    >> I am experiencing weird system hangs. Once about 2-5 weeks system freezes
    >> and stops accepting remote connections, so it is no longer possible to
    >> connect to most important services: smtp (postfix), www (squid) or even
    >> ssh. Such connection is accepted but then it hangs.
    >>
    >> What is strange, that previously established ssh session is usable. It is
    >> possible to work on such system until you do something stupid like "less
    >> /var/log/all.log".

    >
    > So it takes weeks to reproduce this?


    Unfortunately, yes.

    >> free sibling
    >> task PC stack pid father child younger older
    >> syslogd D F5C83C60 0 2162 1 (NOTLB)
    >> f5c83c74 00000082 00000002 f5c83c60 f5c83c5c 00000000 00000000 78538d20
    >> 00000009 00000001 f7f6a070 f7cb8030 82c47e5f 0001cfed 00000a43 f7f6a17c
    >> 7a016980 f705dc80 78404217 7812c708 00000000 00000213 f5c83c84 1e7a64bb
    >> Call Trace:
    >> [<78404217>] _spin_unlock_irqrestore+0xf/0x23
    >> [<7812c708>] __mod_timer+0x92/0x9c
    >> [<78402b34>] schedule_timeout+0x70/0x8d
    >> [<7812c521>] process_timeout+0x0/0x5
    >> [<78402548>] io_schedule_timeout+0x1e/0x28
    >> [<7814d41e>] congestion_wait+0x50/0x64
    >> [<78134abc>] autoremove_wake_function+0x0/0x35
    >> [<781493e7>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    >> [<78145bd0>] generic_file_buffered_write+0x4ee/0x605
    >> [<783c55a1>] unix_dgram_recvmsg+0x1b4/0x1c8
    >> [<78128c8e>] current_fs_time+0x41/0x46
    >> [<78146167>] __generic_file_aio_write_nolock+0x480/0x4df
    >> [<7814621b>] generic_file_aio_write+0x55/0xb3
    >> [<78194b28>] ext3_file_write+0x24/0x8f
    >> [<7815f34f>] do_sync_readv_writev+0xc1/0xfe
    >> [<78134abc>] autoremove_wake_function+0x0/0x35
    >> [<784041ae>] _spin_unlock+0xd/0x21
    >> [<781a8c38>] log_wait_commit+0xc3/0xe3
    >> [<7814448b>] find_get_pages_tag+0x76/0x80
    >> [<7815f204>] rw_copy_check_uvector+0x50/0xaa
    >> [<7815f9d4>] do_readv_writev+0x99/0x164
    >> [<78194b04>] ext3_file_write+0x0/0x8f
    >> [<7815fadc>] vfs_writev+0x3d/0x48
    >> [<7815feb5>] sys_writev+0x41/0x67
    >> [<78103d6a>] sysenter_past_esp+0x5f/0x85
    >> =======================

    >
    > This trace puzzles me, what is: unix_dgram_recvmsg doing there.
    > Also, it has two invocations of: ext3_file_write
    > do you have a stacked filesystem of sorts, ext3 on loopback on ext3?


    No, no loopback:

    # mount
    /dev/md0 on / type ext3 (rw)
    proc on /proc type proc (rw)
    sysfs on /sys type sysfs (rw,nosuid,nodev,noexec)
    devpts on /dev/pts type devpts (rw,nosuid,noexec)
    /dev/mapper/VolGrp0-usr on /usr type ext3 (rw,nodev,data=journal)
    /dev/mapper/VolGrp0-var on /var type ext3 (rw,nodev,data=journal)
    /dev/mapper/VolGrp0-squid_spool on /var/cache/squid/cd0 type ext3 (rw,nosuid,nodev,noatime,data=writeback)
    /dev/mapper/VolGrp0-squid_spool2 on /var/cache/squid/cd1 type ext3 (rw,nosuid,nodev,noatime,data=writeback)
    /dev/mapper/VolGrp0-news_spool on /var/spool/news type ext3 (rw,nosuid,nodev,noatime)
    shm on /dev/shm type tmpfs (rw,noexec,nosuid,nodev)
    usbfs on /proc/bus/usb type usbfs (rw,noexec,nosuid,devmode=0664,devgid=85)
    owl:/usr/gentoo-nfs on /usr/gentoo-nfs type nfs (ro,nosuid,nodev,noatime,bg,intr,tcp,addr=192.168. 129.26)

    Nothing more.

    >> freshclam D 00000282 0 2866 1 (NOTLB)
    >> f36e3cc4 00000082 00000009 00000282 7a0173c0 00000002 00000000 0000007b
    >> 00000009 00000001 f7cb8030 f7c72030 82c4884d 0001cfed 000009ee f7cb813c
    >> 7a016980 f66c0b80 78404217 7812c708 00000000 00000213 f36e3cd4 1e7a64bb
    >> Call Trace:
    >> [<78404217>] _spin_unlock_irqrestore+0xf/0x23
    >> [<7812c708>] __mod_timer+0x92/0x9c
    >> [<78402b34>] schedule_timeout+0x70/0x8d
    >> [<7812c521>] process_timeout+0x0/0x5
    >> [<78402548>] io_schedule_timeout+0x1e/0x28
    >> [<7814d41e>] congestion_wait+0x50/0x64
    >> [<78134abc>] autoremove_wake_function+0x0/0x35
    >> [<781493e7>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    >> [<78145bd0>] generic_file_buffered_write+0x4ee/0x605
    >> [<7819cdb4>] __ext3_journal_stop+0x19/0x34
    >> [<7840408f>] _spin_lock+0xd/0x5a
    >> [<78176f3d>] __mark_inode_dirty+0xdd/0x16f
    >> [<78128c8e>] current_fs_time+0x41/0x46
    >> [<78146167>] __generic_file_aio_write_nolock+0x480/0x4df
    >> [<7814621b>] generic_file_aio_write+0x55/0xb3
    >> [<78103159>] setup_sigcontext+0x105/0x189
    >> [<78194b28>] ext3_file_write+0x24/0x8f
    >> [<7815f453>] do_sync_write+0xc7/0x10a
    >> [<78134abc>] autoremove_wake_function+0x0/0x35
    >> [<781085d2>] convert_fxsr_from_user+0x15/0xd5
    >> [<7815f38c>] do_sync_write+0x0/0x10a
    >> [<7815fbb6>] vfs_write+0x8a/0x10c
    >> [<78160123>] sys_write+0x41/0x67
    >> [<78103d6a>] sysenter_past_esp+0x5f/0x85
    >> =======================

    >
    > single write, no networking, also stuck in balance_dirty_pages().


    Exactly. Strange, isn't it?

    Thanks.

    Best regards,

    Krzysztof Olędzki

  4. Re: Strange system hangs

    On Friday 28 September 2007 18:42, Krzysztof Oledzki wrote:
    > Hello,
    >
    > I am experiencing weird system hangs. Once about 2-5 weeks system freezes
    > and stops accepting remote connections, so it is no longer possible to
    > connect to most important services: smtp (postfix), www (squid) or even
    > ssh. Such connection is accepted but then it hangs.
    >
    > What is strange, that previously established ssh session is usable. It is
    > possible to work on such system until you do something stupid like "less
    > /var/log/all.log". Using strace I found that process blocks on:


    Is this a regression? If so, what's the most recent kernel that didn't show
    the problem?

    The symptoms could be consistent with some place doing a
    balance_dirty_pages while holding a lock that is required for IO, but I can't
    see a smoking gun (you've got contention on i_mutex, but that should be
    OK).

    Can you see if there is any memory under writeback that isn't being
    completed (sysrq+M), also a list the locks held after the hang might be
    helpful (compile in lockdep and sysrq+D)

    Is anything currently running? (sysrq+P and even a full sysrq+T task list
    could be useful).

    Are any IO errors occurring at all?

    Thanks,
    Nick
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  5. Re: Strange system hangs



    On Sat, 29 Sep 2007, Nick Piggin wrote:

    > On Friday 28 September 2007 18:42, Krzysztof Oledzki wrote:
    >> Hello,
    >>
    >> I am experiencing weird system hangs. Once about 2-5 weeks system freezes
    >> and stops accepting remote connections, so it is no longer possible to
    >> connect to most important services: smtp (postfix), www (squid) or even
    >> ssh. Such connection is accepted but then it hangs.
    >>
    >> What is strange, that previously established ssh session is usable. It is
    >> possible to work on such system until you do something stupid like "less
    >> /var/log/all.log". Using strace I found that process blocks on:

    >
    > Is this a regression? If so, what's the most recent kernel that didn't show
    > the problem?


    I don't know. First kernel I ran was 2.6.20.x. This is quite fresh system.

    > The symptoms could be consistent with some place doing a
    > balance_dirty_pages while holding a lock that is required for IO, but I can't
    > see a smoking gun (you've got contention on i_mutex, but that should be
    > OK).
    >
    > Can you see if there is any memory under writeback that isn't being
    > completed (sysrq+M), also a list the locks held after the hang might be
    > helpful (compile in lockdep and sysrq+D)


    OK. I'll try to do it next time if there will be a chance. It may take
    some time, BTW.

    > Is anything currently running? (sysrq+P and even a full sysrq+T task list
    > could be useful).


    I'll have to check - maybe I have this captured. If not I'll check it next
    time.

    > Are any IO errors occurring at all?


    Didn't notice - so no.

    Thank you.

    Best regards,


    Krzysztof Olędzki

  6. Re: Strange system hangs



    On Sat, 29 Sep 2007, Nick Piggin wrote:

    > On Friday 28 September 2007 18:42, Krzysztof Oledzki wrote:
    >> Hello,
    >>
    >> I am experiencing weird system hangs. Once about 2-5 weeks system freezes
    >> and stops accepting remote connections, so it is no longer possible to
    >> connect to most important services: smtp (postfix), www (squid) or even
    >> ssh. Such connection is accepted but then it hangs.
    >>
    >> What is strange, that previously established ssh session is usable. It is
    >> possible to work on such system until you do something stupid like "less
    >> /var/log/all.log". Using strace I found that process blocks on:

    >
    > Is this a regression? If so, what's the most recent kernel that didn't show
    > the problem?
    >
    > The symptoms could be consistent with some place doing a
    > balance_dirty_pages while holding a lock that is required for IO, but I can't
    > see a smoking gun (you've got contention on i_mutex, but that should be
    > OK).
    >
    > Can you see if there is any memory under writeback that isn't being
    > completed (sysrq+M),


    Mem-info:
    DMA per-cpu:
    CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
    CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
    Normal per-cpu:
    CPU 0: Hot: hi: 186, btch: 31 usd: 167 Cold: hi: 62, btch: 15 usd: 47
    CPU 1: Hot: hi: 186, btch: 31 usd: 148 Cold: hi: 62, btch: 15 usd: 54
    Active:340169 inactive:117325 dirty:48579 writeback:0 unstable:0
    free:25266 slab:28357 mapped:3109 pagetables:816 bounce:0
    DMA free:8784kB min:512kB low:640kB high:768kB active:0kB inactive:72kB present:16256kB pages_scanned:36 all_unreclaimable? no
    lowmem_reserve[]: 0 2015
    Normal free:92280kB min:65016kB low:81268kB high:97524kB active:1360676kB inactive:469228kB present:2064260kB pages_scanned:0 all_unreclaimable? no
    lowmem_reserve[]: 0 0
    DMA: 12*4kB 6*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 2*4096kB = 8784kB
    Normal: 2724*4kB 1863*8kB 95*16kB 0*32kB 1*64kB 3*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 15*4096kB = 92280kB
    Swap cache: add 6396, delete 6331, find 3048/3691, race 0+0
    Free swap = 2927260kB
    Total swap = 2927608kB
    Free swap: 2927260kB
    524224 pages of RAM
    0 pages of HIGHMEM
    6345 reserved pages
    302073 pages shared
    65 pages swap cached
    48579 pages dirty
    0 pages writeback
    3109 pages mapped
    28357 pages slab
    816 pages pagetables
    --------------------------


    > also a list the locks held after the hang might be
    > helpful (compile in lockdep and sysrq+D)


    SysRq : Show Locks Held
    Showing all locks held in the system:
    1 lock held by syslogd/2163:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
    1 lock held by freshclam/2864:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
    1 lock held by named/2925:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
    1 lock held by crond/3027:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
    1 lock held by squid/3559:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
    1 lock held by zabbix_agentd/3701:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
    1 lock held by agetty/3753:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by agetty/3754:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by agetty/3755:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by agetty/3756:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by agetty/3758:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by agetty/3759:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by agetty/3760:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by bash/3851:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by bash/3866:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by bash/3888:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by bash/6622:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by agetty/27417:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by bash/9224:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by amavisd/15049:
    #0: (&mm->mmap_sem){----}, at: [<781196ad>] do_page_fault+0x152/0x535
    1 lock held by amavisd/15169:
    #0: (&mm->mmap_sem){----}, at: [<781196ad>] do_page_fault+0x152/0x535
    1 lock held by sadc/16498:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
    1 lock held by mkpir/16842:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
    1 lock held by fetch_mail_stat/16843:
    #0: (&inode->i_mutex){--..}, at: [<78164ae6>] generic_file_llseek+0x29/0xa0

    =============================================


    > Is anything currently running? (sysrq+P


    SysRq : Show Regs

    Pid: 0, comm: swapper
    EIP: 0060:[<781021a6>] CPU: 0
    EIP is at mwait_idle_with_hints+0x3b/0x3f
    EFLAGS: 00000246 Not tainted (2.6.22.9 #1)
    EAX: 00000000 EBX: 00000000 ECX: 00000000 EDX: 00000000
    ESI: 00000000 EDI: 7855a008 EBP: 00000002 DS: 007b ES: 007b FS: 00d8
    CR0: 8005003b CR2: 6ff2e000 CR3: 7db52000 CR4: 000006d0
    [<78102257>] mwait_idle+0x0/0xf
    [<78102391>] cpu_idle+0x99/0xc6
    [<78561907>] start_kernel+0x2dc/0x2e4
    [<7856117b>] unknown_bootoption+0x0/0x202
    =======================

    > and even a full sysrq+T task list
    > could be useful).


    fault_wake_function+0x0/0xc
    [<78127816>] sys_wait4+0x31/0x34
    [<78127840>] sys_waitpid+0x27/0x2b
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    agetty S 9A8AFEC8 0 27417 1 (NOTLB)
    9a8afedc 00000086 00000002 9a8afec8 9a8afec4 00000000 00000000 00000000
    00000009 00000001 81ab0070 f7fbe030 d7259500 00007d9a 00000000 81ab017c
    7a022980 f6e6b2c0 0836b8d5 00000003 00000000 00000000 7fffffff f7184000
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<7829cf28>] read_chan+0x2fb/0x54b
    [<78297ed7>] tty_ldisc_try+0x2e/0x32
    [<7812027d>] default_wake_function+0x0/0xc
    [<7829cc2d>] read_chan+0x0/0x54b
    [<7829a215>] tty_read+0x67/0xaf
    [<7829a1ae>] tty_read+0x0/0xaf
    [<781649a0>] vfs_read+0x88/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    bash S 00000000 0 9224 27386 (NOTLB)
    bae9bedc 00000086 00000000 00000000 f531248c f7cb8130 00000000 00000000
    0000000a 00000000 f7cb8130 f7c740b0 fe89f040 0001ab8a 00000000 f7cb823c
    7a01a980 f7cb7540 f7cb8668 f7cb8130 00000000 00000000 7fffffff f5312000
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<7829cf28>] read_chan+0x2fb/0x54b
    [<78297ed7>] tty_ldisc_try+0x2e/0x32
    [<7812027d>] default_wake_function+0x0/0xc
    [<7829cc2d>] read_chan+0x0/0x54b
    [<7829a215>] tty_read+0x67/0xaf
    [<7829a1ae>] tty_read+0x0/0xaf
    [<781649a0>] vfs_read+0x88/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    scache S AEB09D0C 0 29135 1 (NOTLB)
    aeb09d20 00200092 00000002 aeb09d0c aeb09d08 00000000 00000000 f6f2bf54
    0000000a 00000001 a2cd4030 f7fbe030 c853d980 0001f475 00000000 a2cd413c
    7a022980 eb12ad00 20c7bfd6 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    pdflush S A30BDF9C 0 8843 2 (L-TLB)
    a30bdfb0 00000092 00000002 a30bdf9c a30bdf98 00000000 00000292 daaee560
    0000000a 00000000 daaee030 785143e0 abd01e80 0001f615 00000000 daaee13c
    7a01a980 ea38bcc0 20e3014d 00000003 00000000 00000000 a30bdfc8 7814e33a
    Call Trace:
    [<7814e33a>] pdflush+0x0/0x1a6
    [<7814e3e8>] pdflush+0xae/0x1a6
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    pdflush S F7E2AB20 0 13129 2 (L-TLB)
    b41bbfb0 00000092 7851c174 f7e2ab20 00000001 786b80f0 b41bbf60 f7e2b050
    0000000a 00000001 f7e2ab20 eb1bd510 8245efc0 0001f3f1 00000000 f7e2ac2c
    7a022980 eb12aa80 7814e33a 00000046 7851c160 7814e33a b41bbfc8 7814e33a
    Call Trace:
    [<7814e33a>] pdflush+0x0/0x1a6
    [<7814e33a>] pdflush+0x0/0x1a6
    [<7814e33a>] pdflush+0x0/0x1a6
    [<7814e3e8>] pdflush+0xae/0x1a6
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    smtpd S D0A83B4C 0 14300 1 (NOTLB)
    d0a83b60 00200096 00000002 d0a83b4c d0a83b48 00000000 7840d6e3 785cde80
    00000007 00000000 d5a75590 785143e0 13b0b200 0001f60f 00000000 d5a7569c
    7a01a980 eba4fd40 20e292aa 00000003 00000000 00000000 d0a83b70 20e72688
    Call Trace:
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<783a2023>] tcp_poll+0x1c/0x129
    [<7812cada>] process_timeout+0x0/0x5
    [<7816ec1d>] do_select+0x399/0x3e7
    [<7816f1be>] __pollwait+0x0/0xac
    [<7812027d>] default_wake_function+0x0/0xc
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<78120686>] run_rebalance_domains+0x74/0x3ae
    [<7813e986>] trace_hardirqs_on+0x10c/0x14c
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<781299b4>] local_bh_enable_ip+0xcd/0xed
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783a3bd7>] tcp_recvmsg+0x63f/0x74b
    [<7840d1f4>] _spin_lock_bh+0x38/0x43
    [<783a3bd7>] tcp_recvmsg+0x63f/0x74b
    [<7816ef17>] core_sys_select+0x2ac/0x2c9
    [<78370fdd>] sock_aio_read+0xc8/0xd4
    [<7813e986>] trace_hardirqs_on+0x10c/0x14c
    [<7816423d>] do_sync_read+0xc7/0x10a
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7816f30a>] sys_select+0xa0/0x188
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    amavisd S 00000000 0 14884 2790 (NOTLB)
    a5367e44 00000086 00000000 00000000 7813e7a9 00000000 00000000 f3333940
    0000000a 00000001 c670eae0 ac2a7510 f3d5ec00 0001f3f9 000f4240 c670ebec
    7a022980 f457a300 c670f010 c670eae0 781299b4 a5367ea0 7fffffff 7fffffff
    Call Trace:
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<781299b4>] local_bh_enable_ip+0xcd/0xed
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<783a0b90>] inet_csk_accept+0x9e/0x214
    [<78371564>] sock_attach_fd+0x53/0xb5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783bb524>] inet_accept+0x1f/0xa4
    [<78371564>] sock_attach_fd+0x53/0xb5
    [<7837262e>] sys_accept+0xb2/0x187
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<78152de2>] do_wp_page+0x3b9/0x406
    [<7840d1b1>] _spin_lock+0x33/0x3e
    [<781541bf>] __handle_mm_fault+0x7c3/0x83b
    [<781197d3>] do_page_fault+0x278/0x535
    [<783727bc>] sys_socketcall+0xb9/0x242
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103e68>] restore_nocheck+0x12/0x15
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    amavisd S 931D3B4C 0 14931 2790 (NOTLB)
    931d3b60 00000096 00000002 931d3b4c 931d3b48 00000000 00000000 00000000
    0000000a 00000000 ce58e070 785143e0 f3d5ec00 0001f3f9 00000000 ce58e17c
    7a01a980 e3809ac0 20bfa251 00000003 00000000 00000000 7fffffff 931d3f9c
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<783a2023>] tcp_poll+0x1c/0x129
    [<7816ec1d>] do_select+0x399/0x3e7
    [<7816f1be>] __pollwait+0x0/0xac
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<783734b9>] sk_reset_timer+0xc/0x16
    [<783aa645>] tcp_rcv_established+0x5c5/0x628
    [<783b1465>] tcp_v4_rcv+0x504/0x787
    [<783b133f>] tcp_v4_rcv+0x3de/0x787
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<783b1465>] tcp_v4_rcv+0x504/0x787
    [<7839949e>] ip_local_deliver+0x1aa/0x1c8
    [<78398c10>] ip_local_deliver_finish+0x0/0x14c
    [<783992bb>] ip_rcv+0x498/0x4d1
    [<7839896c>] ip_rcv_finish+0x0/0x2a4
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7837c73c>] process_backlog+0xef/0xfa
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7837c9d8>] net_rx_action+0x162/0x18f
    [<7813e986>] trace_hardirqs_on+0x10c/0x14c
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<781298c7>] local_bh_enable+0x110/0x130
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7837cbf7>] dev_queue_xmit+0x1f2/0x213
    [<7839dedc>] ip_output+0x270/0x2ac
    [<7839c7d5>] ip_finish_output+0x0/0x212
    [<7839d5e6>] ip_queue_xmit+0x316/0x35b
    [<7839b270>] dst_output+0x0/0x7
    [<7816ef17>] core_sys_select+0x2ac/0x2c9
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78152de2>] do_wp_page+0x3b9/0x406
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<78152de2>] do_wp_page+0x3b9/0x406
    [<7840d1b1>] _spin_lock+0x33/0x3e
    [<781541bf>] __handle_mm_fault+0x7c3/0x83b
    [<781197d3>] do_page_fault+0x278/0x535
    [<7816f340>] sys_select+0xd6/0x188
    [<78103e68>] restore_nocheck+0x12/0x15
    [<7811955b>] do_page_fault+0x0/0x535
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    amavisd D 00000000 0 15049 2790 (NOTLB)
    f5de5e2c 00000082 00000000 00000000 7813e7a9 e9568030 7840d6e3 785cde80
    00000009 00000000 e9568030 f7ca0070 6e5cce80 0001f616 00000000 e956813c
    7a01a980 826e4580 785cde80 7812ccc1 00000000 00000292 f5de5e3c 20e30e6e
    Call Trace:
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812ccc1>] __mod_timer+0x92/0x9c
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812cada>] process_timeout+0x0/0x5
    [<7840b0ce>] io_schedule_timeout+0x1e/0x28
    [<78151eca>] congestion_wait+0x50/0x64
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<78152ded>] do_wp_page+0x3c4/0x406
    [<7840d1b1>] _spin_lock+0x33/0x3e
    [<781541bf>] __handle_mm_fault+0x7c3/0x83b
    [<78138305>] down_read_trylock+0x47/0x4e
    [<7811976d>] do_page_fault+0x212/0x535
    [<7811955b>] do_page_fault+0x0/0x535
    [<7840d96a>] error_code+0x72/0x78
    =======================
    amavisd D 00000000 0 15169 2790 (NOTLB)
    c217de2c 00000082 00000000 00000000 7813e7a9 bf1fcb20 7840d6e3 785cde80
    00000009 00000000 bf1fcb20 daaeea60 6e5cce80 0001f616 00000000 bf1fcc2c
    7a01a980 e9a420c0 785cde80 7812ccc1 00000000 00000292 c217de3c 20e30e6e
    Call Trace:
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812ccc1>] __mod_timer+0x92/0x9c
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812cada>] process_timeout+0x0/0x5
    [<7840b0ce>] io_schedule_timeout+0x1e/0x28
    [<78151eca>] congestion_wait+0x50/0x64
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<78152ded>] do_wp_page+0x3c4/0x406
    [<7840d1b1>] _spin_lock+0x33/0x3e
    [<781541bf>] __handle_mm_fault+0x7c3/0x83b
    [<78138305>] down_read_trylock+0x47/0x4e
    [<7811976d>] do_page_fault+0x212/0x535
    [<7811955b>] do_page_fault+0x0/0x535
    [<7840d96a>] error_code+0x72/0x78
    =======================
    squidauth.pl S 00000122 0 15322 3559 (NOTLB)
    e7b47dd8 00000082 00000000 00000122 00000000 00000000 00000000 ec925cac
    0000000a 00000001 c05340f0 eb1bd510 31a413c0 0001f3d9 00000000 c05341fc
    7a022980 b0c607c0 c0534620 c05340f0 00000000 c05340f0 7fffffff 00000000
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78370fdd>] sock_aio_read+0xc8/0xd4
    [<7814aada>] generic_file_aio_write+0x61/0xb6
    [<7816423d>] do_sync_read+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781648dd>] vfs_write+0xd1/0x10c
    [<781649b4>] vfs_read+0x9c/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    squidauth.pl S F660FDC4 0 15323 3559 (NOTLB)
    f660fdd8 00000082 00000002 f660fdc4 f660fdc0 00000000 00000000 cbcc89ac
    00000007 00000000 f7ca14d0 785143e0 c9f296c0 0001efd2 00000000 f7ca15dc
    7a01a980 b0c602c0 2079f567 00000003 00000000 00000000 7fffffff 00000000
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
    [<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78370fdd>] sock_aio_read+0xc8/0xd4
    [<7814aada>] generic_file_aio_write+0x61/0xb6
    [<7816423d>] do_sync_read+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78165177>] __fput+0x10a/0x134
    [<781649b4>] vfs_read+0x9c/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    squidauth.pl S 00000122 0 15324 3559 (NOTLB)
    ea80fdd8 00000082 00000000 00000122 00000000 00000000 00000000 a2fe36ac
    00000006 00000001 f7f46b60 f7ca14d0 b8d75100 0001efd2 00000000 f7f46c6c
    7a022980 ea38b540 f7f47090 f7f46b60 00000000 f7f46b60 7fffffff 00000000
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
    [<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78370fdd>] sock_aio_read+0xc8/0xd4
    [<7814aada>] generic_file_aio_write+0x61/0xb6
    [<7816423d>] do_sync_read+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78165177>] __fput+0x10a/0x134
    [<781649b4>] vfs_read+0x9c/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    unlinkd S CBEF3E44 0 15325 3559 (NOTLB)
    cbef3e58 00000086 00000002 cbef3e44 cbef3e40 00000000 e13d8c3c e769ea60
    00000007 00000000 e769ea60 785143e0 7fd85180 0001f3f9 000f4240 e769eb6c
    7a01a980 e3809d40 20bf9ab8 00000003 00000000 00000000 db5f7600 e13d8b78
    Call Trace:
    [<7816943d>] pipe_wait+0x53/0x73
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78169b7d>] pipe_read+0x2af/0x321
    [<78174b7c>] file_update_time+0x22/0x6a
    [<7816423d>] do_sync_read+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781648dd>] vfs_write+0xd1/0x10c
    [<78164176>] do_sync_read+0x0/0x10a
    [<781649a0>] vfs_read+0x88/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S A1053D0C 0 15570 1 (NOTLB)
    a1053d20 00200092 00000002 a1053d0c a1053d08 00000000 00000000 f6f2bf54
    0000000a 00000001 c670e0b0 f7fbe030 3a020200 0001f403 000f4240 c670e1bc
    7a022980 b0c60540 20c03deb 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783b2c0d>] ip4_datagram_connect+0x2ad/0x308
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S A199DD0C 0 15949 1 (NOTLB)
    a199dd20 00200092 00000002 a199dd0c a199dd08 00000000 00000000 f6f2bf54
    0000000a 00000000 81ab14d0 785143e0 f85451c0 0001f405 00000000 81ab15dc
    7a01a980 db566cc0 20c06bf2 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783b2c0d>] ip4_datagram_connect+0x2ad/0x308
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 86493D0C 0 16128 1 (NOTLB)
    86493d20 00200092 00000002 86493d0c 86493d08 00000000 00000000 f6f2bf54
    00000008 00000000 e769f490 785143e0 636dde40 0001f401 00000000 e769f59c
    7a01a980 f6b9f300 20c01f12 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783b2c0d>] ip4_datagram_connect+0x2ad/0x308
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    amavisd S 00000000 0 16169 2790 (NOTLB)
    ae5bfe44 00000086 00000000 00000000 7813e7a9 00000000 00000000 f3333940
    0000000a 00000000 e769e030 f7ca0070 b5ba4380 0001f3f9 00000000 e769e13c
    7a01a980 f6c8f340 e769e560 e769e030 781299b4 fffffe00 7fffffff 7fffffff
    Call Trace:
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<781299b4>] local_bh_enable_ip+0xcd/0xed
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<783a0b90>] inet_csk_accept+0x9e/0x214
    [<78371564>] sock_attach_fd+0x53/0xb5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783bb524>] inet_accept+0x1f/0xa4
    [<78371564>] sock_attach_fd+0x53/0xb5
    [<7837262e>] sys_accept+0xb2/0x187
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<78152de2>] do_wp_page+0x3b9/0x406
    [<7840d1b1>] _spin_lock+0x33/0x3e
    [<781541bf>] __handle_mm_fault+0x7c3/0x83b
    [<781197d3>] do_page_fault+0x278/0x535
    [<783727bc>] sys_socketcall+0xb9/0x242
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103e68>] restore_nocheck+0x12/0x15
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S A984DD0C 0 16395 1 (NOTLB)
    a984dd20 00200092 00000002 a984dd0c a984dd08 00000000 00000000 f6f2bf54
    0000000a 00000000 c0535550 785143e0 a06b9a00 0001f409 00000000 c053565c
    7a01a980 eba4f5c0 20c0a94a 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783b2c0d>] ip4_datagram_connect+0x2ad/0x308
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    lmtp S 9FEDBD0C 0 16454 1 (NOTLB)
    9fedbd20 00200092 00000002 9fedbd0c 9fedbd08 00000000 00000000 f6f2bf54
    00000003 00000001 7a1dc030 f7fbe030 ce9997c0 0001f43f 00000000 7a1dc13c
    7a022980 f457a080 20c43648 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    sadc D 00000000 0 16498 1 (NOTLB)
    9bb11cc4 00000082 00000000 00000000 7813e7a9 e9d8d4d0 7840d6e3 785cde80
    00000007 00000000 e9d8d4d0 f7f47590 6e5cce80 0001f616 00000000 e9d8d5dc
    7a01a980 f71d5300 785cde80 7812ccc1 00000000 00000296 9bb11cd4 20e30e6e
    Call Trace:
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812ccc1>] __mod_timer+0x92/0x9c
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812cada>] process_timeout+0x0/0x5
    [<7840b0ce>] io_schedule_timeout+0x1e/0x28
    [<78151eca>] congestion_wait+0x50/0x64
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<7814a483>] generic_file_buffered_write+0x4ee/0x605
    [<78128f60>] current_fs_time+0x41/0x46
    [<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7814aad1>] generic_file_aio_write+0x58/0xb6
    [<7814cb18>] free_hot_cold_page+0x13b/0x167
    [<78199bb4>] ext3_file_write+0x24/0x92
    [<78164133>] do_sync_write+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781648dd>] vfs_write+0xd1/0x10c
    [<7816406c>] do_sync_write+0x0/0x10a
    [<78164896>] vfs_write+0x8a/0x10c
    [<78164e0f>] sys_write+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    lmtp S DAAEF490 0 16638 1 (NOTLB)
    a92dfd20 00200092 00000002 daaef490 daaefa10 00000000 00000000 f6f2bf54
    00000007 00000001 daaef490 86709550 a41a1380 0001f43f 00000000 daaef59c
    7a022980 eb12a800 daaef9c0 daaef490 00000000 daaef490 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S A4427D0C 0 16714 1 (NOTLB)
    a4427d20 00200092 00000002 a4427d0c a4427d08 00000000 00000000 f6f2bf54
    0000000a 00000000 9a946b20 785143e0 49bed3c0 0001f404 00000000 9a946c2c
    7a01a980 b0c60a40 20c04fba 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783b2c0d>] ip4_datagram_connect+0x2ad/0x308
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S DABA3D0C 0 16716 1 (NOTLB)
    daba3d20 00200092 00000002 daba3d0c daba3d08 00000000 00000000 f6f2bf54
    0000000a 00000000 d5a74130 785143e0 0aeb9080 0001f403 00000000 d5a7423c
    7a01a980 f6c8fac0 20c03ad5 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783b2c0d>] ip4_datagram_connect+0x2ad/0x308
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 7813E99C 0 16720 1 (NOTLB)
    e2ebbd20 00200092 00000000 7813e99c 00200046 00000000 00000000 f6f2bf54
    00000007 00000001 ec8e34d0 f7cb0b20 05d792c0 0001f3fd 00000000 ec8e35dc
    7a022980 e9c8cd00 ec8e3a00 ec8e34d0 00000000 ec8e34d0 7fffffff 7fffffff
    Call Trace:
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 841A3D0C 0 16754 1 (NOTLB)
    841a3d20 00200092 00000002 841a3d0c 841a3d08 00000000 00000000 f6f2bf54
    0000000a 00000000 f7d09490 785143e0 5c6753c0 0001f405 00000000 f7d0959c
    7a01a980 9169aa80 20c061ba 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783b2c0d>] ip4_datagram_connect+0x2ad/0x308
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 99069D0C 0 16755 1 (NOTLB)
    99069d20 00200092 00000002 99069d0c 99069d08 00000000 00000000 f6f2bf54
    0000000a 00000000 9a9460f0 785143e0 5a34bf80 0001f409 00000000 9a9461fc
    7a01a980 826e4300 20c0a4b2 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783b2c0d>] ip4_datagram_connect+0x2ad/0x308
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S C6AF7D0C 0 16756 1 (NOTLB)
    c6af7d20 00200092 00000002 c6af7d0c c6af7d08 00000000 00000000 f6f2bf54
    0000000a 00000000 a2cd5490 785143e0 3d1cf640 0001f406 00000000 a2cd559c
    7a01a980 826e4800 20c07072 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783b2c0d>] ip4_datagram_connect+0x2ad/0x308
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S BF57BD0C 0 16757 1 (NOTLB)
    bf57bd20 00200092 00000002 bf57bd0c bf57bd08 00000000 00000000 f6f2bf54
    0000000a 00000000 ecbbc0b0 785143e0 d3205700 0001f400 00000000 ecbbc1bc
    7a01a980 826e4080 20c015a0 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35 [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783b2c0d>] ip4_datagram_connect+0x2ad/0x308
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 9B7E9D0C 0 16758 1 (NOTLB)
    9b7e9d20 00200092 00000002 9b7e9d0c 9b7e9d08 00000000 00000000 f6f2bf54
    0000000a 00000000 f7fb4130 785143e0 8e156240 0001f414 00000000 f7fb423c
    7a01a980 e9c8ca80 20c160a3 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 9766BD0C 0 16759 1 (NOTLB)
    9766bd20 00200092 00000002 9766bd0c 9766bd08 00000000 00000000 f6f2bf54
    0000000a 00000000 f7fb4b60 785143e0 a07adc40 0001f409 00000000 f7fb4c6c
    7a01a980 eba4fac0 20c0a94a 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783b2c0d>] ip4_datagram_connect+0x2ad/0x308
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S B31D9D0C 0 16760 1 (NOTLB)
    b31d9d20 00200092 00000002 b31d9d0c b31d9d08 00000000 00000000 f6f2bf54
    0000000a 00000000 bf1fc0f0 785143e0 5aaed180 0001f409 00000000 bf1fc1fc
    7a01a980 eb12a300 20c0a4ba 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783b2c0d>] ip4_datagram_connect+0x2ad/0x308
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 836E9D0C 0 16761 1 (NOTLB)
    836e9d20 00200092 00000002 836e9d0c 836e9d08 00000000 00000000 f6f2bf54
    00000003 00000000 f7e220b0 785143e0 efd80140 0001f437 00000000 f7e221bc
    7a01a980 f5aa9840 20c3b241 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    cleanup S E9507090 0 16793 1 (NOTLB)
    c546bb60 00200096 e9506b60 e9507090 7a1d2014 e9506b60 7840d6e3 7a1d2000
    0000000a 00000001 e9506b60 ec8e34d0 05d792c0 0001f3fd 00000000 e9506c6c
    7a022980 b0c60040 7a1d2000 7812ccc1 00000000 00200282 c546bb70 20f6c453
    Call Trace:
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812ccc1>] __mod_timer+0x92/0x9c
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<7812cada>] process_timeout+0x0/0x5
    [<7816ec1d>] do_select+0x399/0x3e7
    [<7816f1be>] __pollwait+0x0/0xac
    [<7812027d>] default_wake_function+0x0/0xc
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7817f013>] __find_get_block+0x7c/0x158
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7817f013>] __find_get_block+0x7c/0x158
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7817ede9>] __find_get_block_slow+0x114/0x125
    [<7817ed2f>] __find_get_block_slow+0x5a/0x125
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7817f0cc>] __find_get_block+0x135/0x158
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7817f0e5>] __find_get_block+0x14e/0x158
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<783cdb0e>] unix_stream_sendmsg+0x22f/0x2f7
    [<7840d33a>] _read_lock+0x33/0x3e
    [<7816ef17>] core_sys_select+0x2ac/0x2c9
    [<78370f09>] sock_aio_write+0xcb/0xd7
    [<78164133>] do_sync_write+0xc7/0x10a
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7816f30a>] sys_select+0xa0/0x188
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    mkpir D 00000000 0 16842 3027 (NOTLB)
    98303cc4 00000082 00000000 00000000 7813e7a9 daaeea60 7840d6e3 785cde80
    00000007 00000000 daaeea60 e9d8d4d0 6e5cce80 0001f616 00000000 daaeeb6c
    7a01a980 f457ad00 785cde80 7812ccc1 00000000 00000296 98303cd4 20e30e6e
    Call Trace:
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812ccc1>] __mod_timer+0x92/0x9c
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812cada>] process_timeout+0x0/0x5
    [<7840b0ce>] io_schedule_timeout+0x1e/0x28
    [<78151eca>] congestion_wait+0x50/0x64
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<7814a483>] generic_file_buffered_write+0x4ee/0x605
    [<78128f60>] current_fs_time+0x41/0x46
    [<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7814aad1>] generic_file_aio_write+0x58/0xb6
    [<78199bb4>] ext3_file_write+0x24/0x92
    [<78164133>] do_sync_write+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781649e7>] vfs_read+0xcf/0x10a
    [<7816406c>] do_sync_write+0x0/0x10a
    [<78164896>] vfs_write+0x8a/0x10c
    [<78164e0f>] sys_write+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    fetch_mail_st D C7A0FF14 0 16843 3500 (NOTLB)
    c7a0ff28 00200086 00000002 c7a0ff14 c7a0ff10 00000000 00000000 9185284c
    00000008 00000001 f1c9d4d0 f7fbe030 f9ec5080 0001f3fd 004c4b40 f1c9d5dc
    7a022980 f457aa80 20bfe5cc 00000003 00000000 00000000 c7a0ff40 91852814
    Call Trace:
    [<7840bf07>] __mutex_lock_slowpath+0x14d/0x267
    [<78164ae6>] generic_file_llseek+0x29/0xa0
    [<78164ae6>] generic_file_llseek+0x29/0xa0
    [<78164abd>] generic_file_llseek+0x0/0xa0
    [<78163e17>] vfs_llseek+0x36/0x3a
    [<78164d25>] sys_llseek+0x43/0x85
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 7FC8BD0C 0 16844 1 (NOTLB)
    7fc8bd20 00200012 00000002 7fc8bd0c 7fc8bd08 00000000 00000000 f6f2bf54
    0000000a 00000000 f7cc0030 785143e0 e674adc0 0001f409 00000000 f7cc013c
    7a01a980 b0c60cc0 20c0ade1 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 00000080 0 16845 1 (NOTLB)
    d95d9d20 00200092 00000001 00000080 7a0233e0 00000002 00000000 f6f2bf54
    00000009 00000001 f7cd9550 f7cb0b20 17271c00 0001f40a 000f4240 f7cd965c
    7a022980 f6b9f800 20c0b10f 00000003 00000000 00000080 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S F601BD0C 0 16846 1 (NOTLB)
    f601bd20 00200092 00000002 f601bd0c f601bd08 00000000 00000000 f6f2bf54
    00000009 00000000 f7cc94d0 785143e0 17fcbb80 0001f40a 000f4240 f7cc95dc
    7a01a980 f6e7e300 20c0b11e 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 9E3A7D0C 0 16847 1 (NOTLB)
    9e3a7d20 00200092 00000002 9e3a7d0c 9e3a7d08 00000000 00000000 f6f2bf54
    00000009 00000001 f7cd8b20 f7fbe030 18c318c0 0001f40a 00000000 f7cd8c2c
    7a022980 e38090c0 20c0b12b 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 95655D0C 0 16848 1 (NOTLB)
    95655d20 00200092 00000002 95655d0c 95655d08 00000000 00000000 f6f2bf54
    00000009 00000001 ac2a60b0 f7fbe030 19a7fa80 0001f40a 00000000 ac2a61bc
    7a022980 f6c8f0c0 20c0b13a 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S C5229D0C 0 16849 1 (NOTLB)
    c5229d20 00200092 00000002 c5229d0c c5229d08 00000000 00000000 f6f2bf54
    00000009 00000000 e9fc3510 785143e0 1a7d9a00 0001f40a 00000000 e9fc361c
    7a01a980 f6e7e800 20c0b149 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 840DFD0C 0 16850 1 (NOTLB)
    840dfd20 00200092 00000002 840dfd0c 840dfd08 00000000 00000000 f6f2bf54
    00000008 00000001 e9d8caa0 f7fbe030 1b533980 0001f40a 00000000 e9d8cbac
    7a022980 f5aa9d40 20c0b157 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S C3D33D0C 0 16851 1 (NOTLB)
    c3d33d20 00200092 00000002 c3d33d0c c3d33d08 00000000 00000000 f6f2bf54
    00000008 00000001 ac2a6ae0 f7fbe030 1c381b40 0001f40a 00000000 ac2a6bec
    7a022980 f6c8fd40 20c0b165 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S D3603D0C 0 16852 1 (NOTLB)
    d3603d20 00200092 00000002 d3603d0c d3603d08 00000000 00000000 f6f2bf54
    00000008 00000000 a2cd4a60 785143e0 1d0dbac0 0001f40a 000f4240 a2cd4b6c
    7a01a980 dc7ae5c0 20c0b174 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S BEFAFD0C 0 16853 1 (NOTLB)
    befafd20 00200092 00000002 befafd0c befafd08 00000000 00000000 f6f2bf54
    00000008 00000001 d129f510 f7fbe030 1de35a40 0001f40a 000f4240 d129f61c
    7a022980 dc7aed40 20c0b181 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S B1745D0C 0 16854 1 (NOTLB)
    b1745d20 00200092 00000002 b1745d0c b1745d08 00000000 00000000 f6f2bf54
    00000008 00000001 f1c9c070 f7fbe030 1ef602c0 0001f40a 00000000 f1c9c17c
    7a022980 f6c8f5c0 20c0b193 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S C1985D0C 0 16855 1 (NOTLB)
    c1985d20 00200092 00000002 c1985d0c c1985d08 00000000 00000000 f6f2bf54
    00000008 00000000 d129eae0 785143e0 1f9ddb80 0001f40a 00000000 d129ebec
    7a01a980 f650aa40 20c0b19f 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 84C57D0C 0 16856 1 (NOTLB)
    84c57d20 00200092 00000002 84c57d0c 84c57d08 00000000 00000000 f6f2bf54
    00000008 00000000 f1c9caa0 785143e0 20737b00 0001f40a 00000000 f1c9cbac
    7a01a980 dc7ae0c0 20c0b1af 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S B3A63D0C 0 16857 1 (NOTLB)
    b3a63d20 00200092 00000002 b3a63d0c b3a63d08 00000000 00000000 f6f2bf54
    00000007 00000000 ca81b490 785143e0 21491a80 0001f40a 00000000 ca81b59c
    7a01a980 dc7ae340 20c0b1bb 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 9E141D0C 0 16858 1 (NOTLB)
    9e141d20 00200092 00000002 9e141d0c 9e141d08 00000000 00000000 f6f2bf54
    00000007 00000000 f7cd0ae0 785143e0 221eba00 0001f40a 00000000 f7cd0bec
    7a01a980 e9a42ac0 20c0b1cb 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S A887BD0C 0 16859 1 (NOTLB)
    a887bd20 00200092 00000002 a887bd0c a887bd08 00000000 00000000 f6f2bf54
    00000007 00000001 ce58eaa0 f7fbe030 22f45980 0001f40a 00000000 ce58ebac
    7a022980 e9a425c0 20c0b1d7 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S D1A11D0C 0 16860 1 (NOTLB)
    d1a11d20 00200092 00000002 d1a11d0c d1a11d08 00000000 00000000 f6f2bf54
    00000008 00000000 f7cb1550 785143e0 23c9f900 0001f40a 00000000 f7cb165c
    7a01a980 f457a800 20c0b1e6 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 00000080 0 16861 1 (NOTLB)
    9083fd20 00200092 00000001 00000080 7a0233e0 00000002 00000000 f6f2bf54
    00000008 00000001 d129e0b0 f7cb0b20 249f9880 0001f40a 00000000 d129e1bc
    7a022980 dc7ae840 20c0b1f2 00000003 00000000 00000080 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 00000080 0 16862 1 (NOTLB)
    c5043d20 00200092 00000001 00000080 7a01b3e0 00000002 00000000 f6f2bf54
    00000008 00000000 ec8e2070 f7cb0b20 25847a40 0001f40a 000f4240 ec8e217c
    7a01a980 f457a580 20c0b201 00000003 00000000 00000080 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S C8C75D0C 0 16863 1 (NOTLB)
    c8c75d20 00200092 00000002 c8c75d0c c8c75d08 00000000 00000000 f6f2bf54
    00000007 00000001 dc459550 f7fbe030 265a19c0 0001f40a 00000000 dc45965c
    7a022980 cce05d40 20c0b210 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S B0931D0C 0 16864 1 (NOTLB)
    b0931d20 00200092 00000002 b0931d0c b0931d08 00000000 00000000 f6f2bf54
    00000008 00000000 a13b3590 785143e0 273efb80 0001f40a 00000000 a13b369c
    7a01a980 f67c9040 20c0b221 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 89769D0C 0 16865 1 (NOTLB)
    89769d20 00200092 00000002 89769d0c 89769d08 00000000 00000000 f6f2bf54
    0000000a 00000000 a13b2130 785143e0 c7b523c0 0001f40d 00000000 a13b223c
    7a01a980 f6e6bcc0 20c0eefb 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 991F7D0C 0 16866 1 (NOTLB)
    991f7d20 00200092 00000002 991f7d0c 991f7d08 00000000 00000000 f6f2bf54
    0000000a 00000000 a13b2b60 785143e0 c48cec40 0001f40e 00000000 a13b2c6c
    7a01a980 cce050c0 20c0ff8b 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 8F65BD0C 0 16867 1 (NOTLB)
    8f65bd20 00200092 00000002 8f65bd0c 8f65bd08 00000000 00000000 f6f2bf54
    00000009 00000000 dc458b20 785143e0 b817d600 0001f40e 000f4240 dc458c2c
    7a01a980 cce05340 20c0febb 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78154205>] __handle_mm_fault+0x809/0x83b
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S B6DF7D0C 0 16868 1 (NOTLB)
    b6df7d20 00200092 00000002 b6df7d0c b6df7d08 00000000 00000000 f6f2bf54
    0000000a 00000000 dc4580f0 785143e0 6614bd80 0001f410 00000000 dc4581fc
    7a01a980 cce05ac0 20c11ae8 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S A15E7D0C 0 16869 1 (NOTLB)
    a15e7d20 00200092 00000002 a15e7d0c a15e7d08 00000000 00000000 f6f2bf54
    0000000a 00000000 9a4e3490 785143e0 29398440 0001f410 00000000 9a4e359c
    7a01a980 cce05840 20c116ec 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S D4F43D0C 0 16870 1 (NOTLB)
    d4f43d20 00200092 00000002 d4f43d0c d4f43d08 00000000 00000000 f6f2bf54
    00000009 00000000 9a4e2a60 785143e0 2891ab80 0001f410 00000000 9a4e2b6c
    7a01a980 f6e6ba40 20c116e0 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 846F5D0C 0 16871 1 (NOTLB)
    846f5d20 00200092 00000002 846f5d0c 846f5d08 00000000 00000000 f6f2bf54
    00000006 00000000 df6e20f0 785143e0 15eaaac0 0001f412 00000000 df6e21fc
    7a01a980 9169a080 20c13735 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 88395D0C 0 16872 1 (NOTLB)
    88395d20 00200092 00000002 88395d0c 88395d08 00000000 00000000 f6f2bf54
    00000006 00000000 c670f510 785143e0 f2466900 0001f412 000f4240 c670f61c
    7a01a980 826e4d00 20c145a7 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S CD089D0C 0 16873 1 (NOTLB)
    cd089d20 00200092 00000002 cd089d0c cd089d08 00000000 00000000 f6f2bf54
    00000007 00000000 9a947550 785143e0 d9d6ce40 0001f413 00000000 9a94765c
    7a01a980 e3809340 20c154d5 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 98913D0C 0 16874 1 (NOTLB)
    98913d20 00200092 00000002 98913d0c 98913d08 00000000 00000000 f6f2bf54
    00000006 00000000 9a4e2030 785143e0 9c44f9c0 0001f414 00000000 9a4e213c
    7a01a980 e38095c0 20c16193 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S C3685D0C 0 16875 1 (NOTLB)
    c3685d20 00200092 00000002 c3685d0c c3685d08 00000000 00000000 f6f2bf54
    0000000a 00000000 f5de34d0 785143e0 a6b44300 0001f416 00000000 f5de35dc
    7a01a980 a7484cc0 20c183ce 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S B14A5D0C 0 16876 1 (NOTLB)
    b14a5d20 00200092 00000002 b14a5d0c b14a5d08 00000000 00000000 f6f2bf54
    00000009 00000000 f5de2aa0 785143e0 08466680 0001f416 00000000 f5de2bac
    7a01a980 db566540 20c1796d 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S A20E9D0C 0 16877 1 (NOTLB)
    a20e9d20 00200092 00000002 a20e9d0c a20e9d08 00000000 00000000 f6f2bf54
    00000009 00000000 f5de2070 785143e0 5bc7f6c0 0001f416 00000000 f5de217c
    7a01a980 db566a40 20c17ee6 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 8C27FD0C 0 16878 1 (NOTLB)
    8c27fd20 00200092 00000002 8c27fd0c 8c27fd08 00000000 00000000 f6f2bf54
    00000002 00000000 c978b510 785143e0 3832f740 0001f417 00000000 c978b61c
    7a01a980 db566040 20c18d56 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S A2C15D0C 0 16879 1 (NOTLB)
    a2c15d20 00200092 00000002 a2c15d0c a2c15d08 00000000 00000000 f6f2bf54
    00000006 00000000 c978aae0 785143e0 1e275fc0 0001f418 00000000 c978abec
    7a01a980 db5662c0 20c19c6b 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S C70F5D0C 0 16880 1 (NOTLB)
    c70f5d20 00200092 00000002 c70f5d0c c70f5d08 00000000 00000000 f6f2bf54
    00000008 00000000 c978a0b0 785143e0 2e5dc3c0 0001f418 00000000 c978a1bc
    7a01a980 e3809840 20c19d7b 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 99BE1D0C 0 16881 1 (NOTLB)
    99be1d20 00200092 00000002 99be1d0c 99be1d08 00000000 00000000 f6f2bf54
    00000008 00000000 dbd9d550 785143e0 3cbb2200 0001f418 00000000 dbd9d65c
    7a01a980 f7cb7040 20c19e6a 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S DBD9D058 0 16882 1 (NOTLB)
    90b79d20 00200092 00000000 dbd9d058 7813f8ca 00000000 00000000 f6f2bf54
    00000009 00000000 dbd9cb20 c4d33590 75d50d80 0001f418 00000000 dbd9cc2c
    7a01a980 f67c9a40 dbd9d050 dbd9cb20 00000000 dbd9cb20 7fffffff 7fffffff
    Call Trace:
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S E7CFBD0C 0 16883 1 (NOTLB)
    e7cfbd20 00200092 00000002 e7cfbd0c e7cfbd08 00000000 00000000 f6f2bf54
    0000000a 00000000 c4d33590 785143e0 897131c0 0001f418 00000000 c4d3369c
    7a01a980 cce055c0 20c1a370 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S B2CE9D0C 0 16884 1 (NOTLB)
    b2ce9d20 00200092 00000002 b2ce9d0c b2ce9d08 00000000 00000000 f6f2bf54
    00000002 00000000 c4d32130 785143e0 c8ec4dc0 0001f419 00000000 c4d3223c
    7a01a980 e9fb9d00 20c1b860 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 7FE3FD0C 0 16885 1 (NOTLB)
    7fe3fd20 00200012 00000002 7fe3fd0c 7fe3fd08 00000000 00000000 f6f2bf54
    0000000a 00000000 c4d32b60 785143e0 2a04df00 0001f41a 00000000 c4d32c6c
    7a01a980 f7cb77c0 20c1bebd 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S CA9E7D0C 0 16886 1 (NOTLB)
    ca9e7d20 00200092 00000002 ca9e7d0c ca9e7d08 00000000 00000000 f6f2bf54
    0000000a 00000000 dbd9c0f0 785143e0 82e43980 0001f41b 00000000 dbd9c1fc
    7a01a980 f71d5580 20c1d559 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 94943D0C 0 16887 1 (NOTLB)
    94943d20 00200092 00000002 94943d0c 94943d08 00000000 00000000 f6f2bf54
    0000000a 00000000 84393490 785143e0 97673e80 0001f41f 00000000 8439359c
    7a01a980 da41fd40 20c219cb 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 8EFF7D0C 0 16888 1 (NOTLB)
    8eff7d20 00200092 00000002 8eff7d0c 8eff7d08 00000000 00000000 f6f2bf54
    00000009 00000000 84392a60 785143e0 7c0bf580 0001f420 00000000 84392b6c
    7a01a980 9169ad00 20c228c7 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 8B12FD0C 0 16889 1 (NOTLB)
    8b12fd20 00200092 00000002 8b12fd0c 8b12fd08 00000000 00000000 f6f2bf54
    0000000a 00000000 84392030 785143e0 0b7719c0 0001f426 00000000 8439213c
    7a01a980 9169a300 20c28611 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S C61A9D0C 0 16890 1 (NOTLB)
    c61a9d20 00200092 00000002 c61a9d0c c61a9d08 00000000 00000000 f6f2bf54
    00000006 00000000 9963f4d0 785143e0 a4f47a80 0001f42a 00000000 9963f5dc
    7a01a980 da41f0c0 20c2d33d 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S C0EDFD0C 0 16891 1 (NOTLB)
    c0edfd20 00200092 00000002 c0edfd0c c0edfd08 00000000 00000000 f6f2bf54
    0000000a 00000000 9963eaa0 785143e0 d4efcdc0 0001f42a 00000000 9963ebac
    7a01a980 da41fac0 20c2d662 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 9AEC1D0C 0 16892 1 (NOTLB)
    9aec1d20 00200092 00000002 9aec1d0c 9aec1d08 00000000 00000000 f6f2bf54
    0000000a 00000000 a25c7510 785143e0 3c1cc480 0001f42b 00000000 a25c761c
    7a01a980 a1257cc0 20c2dd23 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S DC5A3D0C 0 16893 1 (NOTLB)
    dc5a3d20 00200092 00000002 dc5a3d0c dc5a3d08 00000000 00000000 f6f2bf54
    00000009 00000001 a25c6ae0 f7fbe030 1cc2a500 0001f42b 000f4240 a25c6bec
    7a022980 a1257a40 20c2db17 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S D8409D0C 0 16894 1 (NOTLB)
    d8409d20 00200092 00000002 d8409d0c d8409d08 00000000 00000000 f6f2bf54
    0000000a 00000000 9963e070 785143e0 894ce640 0001f42b 00000000 9963e17c
    7a01a980 da41f840 20c2e233 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S E2FEBD0C 0 16895 1 (NOTLB)
    e2febd20 00200092 00000002 e2febd0c e2febd08 00000000 00000000 f6f2bf54
    00000009 00000000 87dc9550 785143e0 b77e7600 0001f42b 000f4240 87dc965c
    7a01a980 da41f5c0 20c2e53b 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S D6923D0C 0 16896 1 (NOTLB)
    d6923d20 00200092 00000002 d6923d0c d6923d08 00000000 00000000 f6f2bf54
    0000000a 00000000 87dc8b20 785143e0 8792e4c0 0001f42c 00000000 87dc8c2c
    7a01a980 da41f340 20c2f2df 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S C6CC1D0C 0 16897 1 (NOTLB)
    c6cc1d20 00200092 00000002 c6cc1d0c c6cc1d08 00000000 00000000 f6f2bf54
    0000000a 00000000 87dc80f0 785143e0 3f38b840 0001f430 00000000 87dc81fc
    7a01a980 a1257540 20c3313a 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S B98C7D0C 0 16898 1 (NOTLB)
    b98c7d20 00200092 00000002 b98c7d0c b98c7d08 00000000 00000000 f6f2bf54
    00000006 00000000 a25c60b0 785143e0 3f75c140 0001f430 00000000 a25c61bc
    7a01a980 a12572c0 20c3313e 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S D2203D0C 0 16899 1 (NOTLB)
    d2203d20 00200092 00000002 d2203d0c d2203d08 00000000 00000000 f6f2bf54
    00000007 00000000 9a747590 785143e0 07413740 0001f431 00000000 9a74769c
    7a01a980 a12577c0 20c33e56 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 80687D0C 0 16900 1 (NOTLB)
    80687d20 00200092 00000002 80687d0c 80687d08 00000000 00000000 f6f2bf54
    00000006 00000000 9a746b60 785143e0 414f46c0 0001f431 000f4240 9a746c6c
    7a01a980 e9fb9a80 20c34226 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S C473DD0C 0 16901 1 (NOTLB)
    c473dd20 00200092 00000002 c473dd0c c473dd08 00000000 00000000 f6f2bf54
    0000000a 00000000 9a746130 785143e0 c935d080 0001f432 00000000 9a74623c
    7a01a980 e9fb9800 20c35bd4 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S DA57DD0C 0 16902 1 (NOTLB)
    da57dd20 00200092 00000002 da57dd0c da57dd08 00000000 00000000 f6f2bf54
    0000000a 00000000 cd433490 785143e0 62942d00 0001f436 00000000 cd43359c
    7a01a980 c01f2d00 20c39835 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 83283D0C 0 16903 1 (NOTLB)
    83283d20 00200092 00000002 83283d0c 83283d08 00000000 00000000 f6f2bf54
    00000009 00000001 cd432030 f7fbe030 48a799c0 0001f438 00000000 cd43213c
    7a022980 e9fb9300 20c3b810 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S E3FE1D0C 0 16904 1 (NOTLB)
    e3fe1d20 00200092 00000002 e3fe1d0c e3fe1d08 00000000 00000000 f6f2bf54
    00000009 00000000 ce74d4d0 785143e0 49126980 0001f438 00000000 ce74d5dc
    7a01a980 9cfd0d40 20c3b817 00000001 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 8F511D0C 0 16905 1 (NOTLB)
    8f511d20 00200092 00000002 8f511d0c 8f511d08 00000000 00000000 f6f2bf54
    00000009 00000001 ce74caa0 f7fbe030 49f74b40 0001f438 00000000 ce74cbac
    7a022980 a74847c0 20c3b827 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S F26E9D0C 0 16906 1 (NOTLB)
    f26e9d20 00200092 00000002 f26e9d0c f26e9d08 00000000 00000000 f6f2bf54
    00000009 00000000 cd432a60 785143e0 4adc2d00 0001f438 00000000 cd432b6c
    7a01a980 9cfd0ac0 20c3b838 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 8A7EBD0C 0 16907 1 (NOTLB)
    8a7ebd20 00200092 00000002 8a7ebd0c 8a7ebd08 00000000 00000000 f6f2bf54
    00000007 00000001 ce74c070 f7fbe030 a1d4bec0 0001f43c 00000000 ce74c17c
    7a022980 e9fb9580 20c40104 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 912A9D0C 0 16908 1 (NOTLB)
    912a9d20 00200092 00000002 912a9d0c 912a9d08 00000000 00000000 f6f2bf54
    0000000a 00000000 89695510 785143e0 82018cc0 0001f43e 00000000 8969561c
    7a01a980 9cfd0840 20c4207d 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S A1B39D0C 0 16909 1 (NOTLB)
    a1b39d20 00200092 00000002 a1b39d0c a1b39d08 00000000 00000000 f6f2bf54
    0000000a 00000000 89694ae0 785143e0 a8272cc0 0001f43e 00000000 89694bec
    7a01a980 9cfd00c0 20c422fd 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 9A793D0C 0 16910 1 (NOTLB)
    9a793d20 00200092 00000002 9a793d0c 9a793d08 00000000 00000000 f6f2bf54
    0000000a 00000000 896940b0 785143e0 15c41680 0001f43f 00000000 896941bc
    7a01a980 a063ccc0 20c42a2c 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S E2A39D0C 0 16912 1 (NOTLB)
    e2a39d20 00200092 00000002 e2a39d0c e2a39d08 00000000 00000000 f6f2bf54
    00000002 00000000 867080f0 785143e0 acfca240 0001f441 00000000 867081fc
    7a01a980 9cfd0340 20c455a3 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S E278FD0C 0 16913 1 (NOTLB)
    e278fd20 00200092 00000002 e278fd0c e278fd08 00000000 00000000 f6f2bf54
    00000004 00000000 86708b20 785143e0 badfee80 0001f441 00000000 86708c2c
    7a01a980 a063c540 20c4568c 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 8B41DD0C 0 16914 1 (NOTLB)
    8b41dd20 00200092 00000002 8b41dd0c 8b41dd08 00000000 00000000 f6f2bf54
    0000000a 00000000 bf519590 785143e0 e2edd680 0001f441 000f4240 bf51969c
    7a01a980 a063c7c0 20c4592c 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S BCB57D0C 0 16915 1 (NOTLB)
    bcb57d20 00200092 00000002 bcb57d0c bcb57d08 00000000 00000000 f6f2bf54
    00000002 00000000 bf518130 785143e0 cda772c0 0001f443 00000000 bf51823c
    7a01a980 e33f5d00 20c47955 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S CD79FD0C 0 16916 1 (NOTLB)
    cd79fd20 00200092 00000002 cd79fd0c cd79fd08 00000000 00000000 f6f2bf54
    00000006 00000000 bf518b60 785143e0 3c388080 0001f444 00000000 bf518c6c
    7a01a980 a7484a40 20c48096 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S ED8E7D0C 0 16917 1 (NOTLB)
    ed8e7d20 00200092 00000002 ed8e7d0c ed8e7d08 00000000 00000000 f6f2bf54
    00000007 00000000 b4993490 785143e0 46790380 0001f444 00000000 b499359c
    7a01a980 a7484540 20c48142 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S A9E2FD0C 0 16918 1 (NOTLB)
    a9e2fd20 00200092 00000002 a9e2fd0c a9e2fd08 00000000 00000000 f6f2bf54
    0000000a 00000000 b4992030 785143e0 d8724980 0001f445 000f4240 b499213c
    7a01a980 a74842c0 20c49b96 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S C3213D0C 0 16919 1 (NOTLB)
    c3213d20 00200092 00000002 c3213d0c c3213d08 00000000 00000000 f6f2bf54
    0000000a 00000000 b4992a60 785143e0 c9414ac0 0001f449 00000000 b4992b6c
    7a01a980 e6fbdd40 20c4ddb7 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S F29BDD0C 0 16920 1 (NOTLB)
    f29bdd20 00200092 00000002 f29bdd0c f29bdd08 00000000 00000000 f6f2bf54
    0000000a 00000000 88d9f4d0 785143e0 d1e650c0 0001f44a 00000000 88d9f5dc
    7a01a980 a063c2c0 20c4ef0c 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 857EBD0C 0 16921 1 (NOTLB)
    857ebd20 00200092 00000002 857ebd0c 857ebd08 00000000 00000000 f6f2bf54
    00000008 00000000 afcaf510 785143e0 e8405c80 0001f44a 00000000 afcaf61c
    7a01a980 e6fbdac0 20c4f085 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S E1483D0C 0 16922 1 (NOTLB)
    e1483d20 00200092 00000002 e1483d0c e1483d08 00000000 00000000 f6f2bf54
    00000009 00000000 afcaeae0 785143e0 3c978c40 0001f44b 00000000 afcaebec
    7a01a980 a7484040 20c4f60d 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S AAABBD0C 0 16923 1 (NOTLB)
    aaabbd20 00200092 00000002 aaabbd0c aaabbd08 00000000 00000000 f6f2bf54
    00000008 00000000 afcae0b0 785143e0 5127d480 0001f44b 00000000 afcae1bc
    7a01a980 a063ca40 20c4f764 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 8C039D0C 0 16924 1 (NOTLB)
    8c039d20 00200092 00000002 8c039d0c 8c039d08 00000000 00000000 f6f2bf54
    0000000a 00000000 8a4ab550 785143e0 8479a5c0 0001f44b 00000000 8a4ab65c
    7a01a980 a063c040 20c4fac1 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S CB7EBD0C 0 16925 1 (NOTLB)
    cb7ebd20 00200092 00000002 cb7ebd0c cb7ebd08 00000000 00000000 f6f2bf54
    0000000a 00000000 8a4aab20 785143e0 3df6ffc0 0001f44c 00000000 8a4aac2c
    7a01a980 88ea2cc0 20c506e8 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S D105FD0C 0 16926 1 (NOTLB)
    d105fd20 00200092 00000002 d105fd0c d105fd08 00000000 00000000 f6f2bf54
    00000009 00000001 8a4aa0f0 f7fbe030 15da5540 0001f44d 00000000 8a4aa1fc
    7a022980 e33f5a80 20c5150f 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S D59B9D0C 0 16927 1 (NOTLB)
    d59b9d20 00200092 00000002 d59b9d0c d59b9d08 00000000 00000000 f6f2bf54
    00000009 00000000 88d9eaa0 785143e0 a145a380 0001f44f 00000000 88d9ebac
    7a01a980 88ea2040 20c53fc0 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smtpd S 93F03D0C 0 16928 1 (NOTLB)
    93f03d20 00200092 00000002 93f03d0c 93f03d08 00000000 00000000 f6f2bf54
    00000009 00000000 88d9e070 785143e0 a22a8540 0001f44f 00000000 88d9e17c
    7a01a980 e33f5300 20c53fcf 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================



    Best regards,

    Krzysztof Olędzki

  7. Re: Strange system hangs



    On Sat, 29 Sep 2007, Nick Piggin wrote:

    > On Friday 28 September 2007 18:42, Krzysztof Oledzki wrote:
    >> Hello,
    >>
    >> I am experiencing weird system hangs. Once about 2-5 weeks system freezes
    >> and stops accepting remote connections, so it is no longer possible to
    >> connect to most important services: smtp (postfix), www (squid) or even
    >> ssh. Such connection is accepted but then it hangs.
    >>
    >> What is strange, that previously established ssh session is usable. It is
    >> possible to work on such system until you do something stupid like "less
    >> /var/log/all.log". Using strace I found that process blocks on:

    >
    > Is this a regression? If so, what's the most recent kernel that didn't show
    > the problem?
    >
    > The symptoms could be consistent with some place doing a
    > balance_dirty_pages while holding a lock that is required for IO, but I can't
    > see a smoking gun (you've got contention on i_mutex, but that should be
    > OK).


    OK. It did it again , so I send additional debug information. Today I
    discovered that "echo i > /proc/sysrq-trigger" makes my system being
    usable.

    > Can you see if there is any memory under writeback that isn't being
    > completed (sysrq+M),


    SysRq : Show Memory
    Mem-info:
    DMA per-cpu:
    CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
    CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
    Normal per-cpu:
    CPU 0: Hot: hi: 186, btch: 31 usd: 11 Cold: hi: 62, btch: 15 usd: 8
    CPU 1: Hot: hi: 186, btch: 31 usd: 170 Cold: hi: 62, btch: 15 usd: 23
    Active:281859 inactive:140551 dirty:46161 writeback:0 unstable:0
    free:39081 slab:50571 mapped:3261 pagetables:356 bounce:0
    DMA free:8704kB min:512kB low:640kB high:768kB active:48kB inactive:0kB present:16256kB pages_scanned:12 all_unreclaimable? no
    lowmem_reserve[]: 0 2015
    Normal free:147620kB min:65016kB low:81268kB high:97524kB active:1127388kB inactive:562204kB present:2064260kB pages_scanned:0 all_unreclaimable? no
    lowmem_reserve[]: 0 0
    DMA: 4*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 2*4096kB = 8704kB
    Normal: 15781*4kB 522*8kB 264*16kB 56*32kB 9*64kB 8*128kB 6*256kB 1*512kB 1*1024kB 2*2048kB 16*4096kB = 147620kB
    Swap cache: add 5022, delete 4928, find 2281/2804, race 0+0
    Free swap = 2927208kB
    Total swap = 2927608kB
    Free swap: 2927208kB
    524224 pages of RAM
    0 pages of HIGHMEM
    6345 reserved pages
    249411 pages shared
    94 pages swap cached
    46161 pages dirty
    0 pages writeback
    3261 pages mapped
    50571 pages slab
    356 pages pagetables

    > also a list the locks held after the hang might be
    > helpful (compile in lockdep and sysrq+D)


    SysRq : Show Locks Held

    Showing all locks held in the system:
    1 lock held by syslogd/2197:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
    1 lock held by freshclam/2898:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
    1 lock held by crond/3061:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
    1 lock held by snmpd/3546:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
    1 lock held by squid/3606:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
    1 lock held by zabbix_agentd/3728:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
    1 lock held by agetty/3791:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by agetty/3793:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by agetty/3794:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by agetty/3795:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by agetty/3796:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by agetty/3797:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by agetty/3798:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    1 lock held by agetty/3799:
    #0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
    2 locks held by bash/3887:
    #0: (sysrq_key_table_lock){....}, at: [<782a6ea9>] __handle_sysrq+0x17/0x109
    #1: (tasklist_lock){..-?}, at: [<7813d883>] debug_show_all_locks+0x3b/0x12d
    1 lock held by amavisd/7227:
    #0: (&mm->mmap_sem){----}, at: [<781196ad>] do_page_fault+0x152/0x535
    1 lock held by sadc/8202:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
    1 lock held by emerge/8204:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
    1 lock held by mkpir/8297:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
    1 lock held by screen/8365:
    #0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6

    =============================================


    > Is anything currently running? (sysrq+P


    Empty.

    > and even a full sysrq+T task list
    > could be useful).


    SysRq : Show State

    free sibling
    task PC stack pid father child younger older
    init S 7A1C1B4C 0 1 0 (NOTLB)
    7a1c1b60 00000016 00000002 7a1c1b4c 7a1c1b48 00000000 7840d6e3 7a1d2000
    0000000a 00000001 f7fbf490 f7fbe030 9c45a103 0001ff61 00005b8a f7fbf59c
    7a022980 f71bdd00 219904e6 00000003 00000000 00000000 7a1c1b70 2199186c
    Call Trace:
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7812cada>] process_timeout+0x0/0x5
    [<7816ec1d>] do_select+0x399/0x3e7
    [<7816f1be>] __pollwait+0x0/0xac
    [<7812027d>] default_wake_function+0x0/0xc
    [<781298c7>] local_bh_enable+0x110/0x130
    [<7813e986>] trace_hardirqs_on+0x10c/0x14c
    [<7837cb10>] dev_queue_xmit+0x10b/0x213
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78172a90>] __d_lookup+0xbb/0x10b
    [<78172a3f>] __d_lookup+0x6a/0x10b
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<78172ac3>] __d_lookup+0xee/0x10b
    [<7816a323>] do_lookup+0x4f/0x143
    [<78171ebc>] dput+0x16/0xe4
    [<7816be16>] __link_path_walk+0xa1a/0xb5e
    [<7816ef17>] core_sys_select+0x2ac/0x2c9
    [<7816c003>] link_path_walk+0xa9/0xb3
    [<78160ec2>] kmem_cache_free+0x9a/0xa1
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<7816f30a>] sys_select+0xa0/0x188
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    kthreadd S F7FBEA60 0 2 0 (L-TLB)
    7a1c3fd0 00000016 7851aa94 f7fbea60 00000001 786b7ec8 f7fbef90 f7fbef90
    0000000a 00000000 f7fbea60 edcd0030 e200f0d9 0001df92 00000d92 f7fbeb6c
    7a01a980 f7cafa40 00001890 00000246 7851aa80 00001890 7851aa60 00001890
    Call Trace:
    [<78134f4d>] kthreadd+0x61/0xfe
    [<78134eec>] kthreadd+0x0/0xfe
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    migration/0 S 7A022980 0 3 2 (L-TLB)
    7a1c9fa4 00000002 7811e4eb 7a022980 f7c7d550 7a1c9fa4 7811f199 7a1c7a00
    00000001 00000000 7a1c74d0 f7e84b60 78531ca2 0001ff5b 00001431 7a1c75dc
    7a01a980 e913a340 7a0229e8 00000046 7a01a980 7a01a980 7a01b2f4 7a01a980
    Call Trace:
    [<7811e4eb>] resched_task+0x3b/0x59
    [<7811f199>] move_tasks+0x1f5/0x2bd
    [<781221cf>] migration_thread+0x14a/0x1cd
    [<78122085>] migration_thread+0x0/0x1cd
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    ksoftirqd/0 S 7A1CBFB4 0 4 2 (L-TLB)
    7a1cbfc8 00000002 00000002 7a1cbfb4 7a1cbfb0 00000000 edcd1490 edcd1490
    00000009 00000000 7a1c6aa0 785143e0 f1a2fd1e 0001e9af 0000065e 7a1c6bac
    7a01a980 c053d340 202bebb7 00000003 00000000 00000000 7a018100 00000000
    Call Trace:
    [<78129a29>] ksoftirqd+0x55/0x134
    [<781299d4>] ksoftirqd+0x0/0x134
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    watchdog/0 S 7A01A980 0 5 2 (L-TLB)
    7a1cdfcc 00000006 7840d6e3 7a01a980 7a1cdfcc 7813e99c 00000002 00000046
    00000001 00000000 7a1c6070 f7fbf490 35f147b4 0000000a 00002c3f 7a1c617c
    7a01a980 78514180 7a1c6530 ffffffff 78596a0c 01a85000 00000000 78146910
    Call Trace:
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78146910>] watchdog+0x0/0x58
    [<78146958>] watchdog+0x48/0x58
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    migration/1 S 7A1D1F90 0 6 2 (L-TLB)
    7a1d1fa4 00000002 00000002 7a1d1f90 7a1d1f8c 00000000 7a1cfa40 7a1cfa40
    00000001 00000001 7a1cf510 f7fbe030 04d4c284 0001ff60 00001b5e 7a1cf61c
    7a022980 e8d34580 2198ea1b 00000003 00000000 00000000 7a0232f4 7a022980
    Call Trace:
    [<781221cf>] migration_thread+0x14a/0x1cd
    [<78122085>] migration_thread+0x0/0x1cd
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    ksoftirqd/1 S 7A1D5FB4 0 7 2 (L-TLB)
    7a1d5fc8 00000002 00000002 7a1d5fb4 7a1d5fb0 00000000 00000046 00000000
    0000000a 00000001 7a1ceae0 f7fbe030 07b31f2a 0001ff3a 0004c80b 7a1cebec
    7a022980 e913a340 21966ac7 00000003 00000000 00000000 7a020100 00000001
    Call Trace:
    [<78129a29>] ksoftirqd+0x55/0x134
    [<781299d4>] ksoftirqd+0x0/0x134
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    watchdog/1 S 7A022980 0 8 2 (L-TLB)
    7a1d7fcc 00000006 7840d6e3 7a022980 7a1d7fcc 7813e99c 00000002 00000046
    00000001 00000001 7a1ce0b0 7a1ceae0 372914ac 0000000a 00003a96 7a1ce1bc
    7a022980 78514180 7a1ce570 ffffffff 78596a0c 01a8d000 00000001 78146910
    Call Trace:
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78146910>] watchdog+0x0/0x58
    [<78146958>] watchdog+0x48/0x58
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    events/0 S F7FBDFA0 0 9 2 (L-TLB)
    f7fbdfb4 00000086 00000002 f7fbdfa0 f7fbdf9c 00000000 f7ea1a80 00000000
    0000000a 00000000 f7ea1550 785143e0 959db3cd 0001ff61 0000158b f7ea165c
    7a01a980 c053d840 21990475 00000003 00000000 00000000 f7f940c0 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    events/1 S F7FB9FA0 0 10 2 (L-TLB)
    f7fb9fb4 00000086 00000002 f7fb9fa0 f7fb9f9c 00000000 f7ea1050 00000000
    0000000a 00000001 f7ea0b20 f7fbe030 992e5f8c 0001ff61 000013bc f7ea0c2c
    7a022980 e913a340 219904b2 00000003 00000000 00000000 f7c4df40 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    khelper S 00000000 0 11 2 (L-TLB)
    f7fbbfb4 00000086 00000046 00000000 00000000 f7c4defc f7ea0620 00000000
    00000009 00000001 f7ea00f0 f7cc8130 6fcbad27 0001df91 000011c4 f7ea01fc
    7a022980 f6b15840 00000046 f7c4dee8 00000286 f7c4dee8 f7c4dec0 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kblockd/0 S F7E3BFA0 0 64 2 (L-TLB)
    f7e3bfb4 00000086 00000002 f7e3bfa0 f7e3bf9c 00000000 f7e85090 00000000
    0000000a 00000000 f7e84b60 785143e0 6ff1ad71 0001ff61 00001779 f7e84c6c
    7a01a980 e913a340 219901fd 00000003 00000000 00000000 f7e951c0 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kblockd/1 S F7E3DFA0 0 65 2 (L-TLB)
    f7e3dfb4 00000086 00000002 f7e3dfa0 f7e3df9c 00000000 00000000 00000000
    0000000a 00000001 f7e85590 f7fbe030 593eee3f 0001ff61 00001932 f7e8569c
    7a022980 d3e4acc0 2199007e 00000002 00000000 00000000 f7e95140 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kacpid S 00000000 0 66 2 (L-TLB)
    f7e3ffb4 00000086 00000046 00000000 00000000 f7c76cfc f7c7c620 00000000
    00000001 00000000 f7c7c0f0 f7fbf490 3bbce431 0000000a 0000072f f7c7c1fc
    7a01a980 78514180 00000046 f7c76ce8 00000286 f7c76ce8 f7c76cc0 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kacpi_notify S 00000000 0 67 2 (L-TLB)
    f7e41fb4 00000086 00000046 00000000 00000000 f7c76bfc f7c7d050 00000000
    00000001 00000000 f7c7cb20 f7fbf490 3bbd4590 0000000a 00000720 f7c7cc2c
    7a01a980 78514180 00000046 f7c76be8 00000286 f7c76be8 f7c76bc0 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    ata/0 S 00000000 0 147 2 (L-TLB)
    f7e21fb4 00000086 00000046 00000000 00000000 f7c4bd7c f7e23a40 00000000
    0000000a 00000000 f7e23510 f7cd0030 a23c198a 0000000b 0002d0fb f7e2361c
    7a01a980 78514180 00000046 f7c4bd68 00000286 f7c4bd68 f7c4bd40 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    ata/1 S 00000000 0 148 2 (L-TLB)
    f7e19fb4 00000086 00000046 00000000 00000000 f7c4bcfc f7e2a620 00000000
    0000000a 00000001 f7e2a0f0 f7fbf490 7e41e2a8 0000000b 00005ba5 f7e2a1fc
    7a022980 78514180 00000046 f7c4bce8 00000286 f7c4bce8 f7c4bcc0 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    ata_aux S F7E43FA0 0 149 2 (L-TLB)
    f7e43fb4 00000086 00000002 f7e43fa0 f7e43f9c 00000000 f7e1a5a0 00000000
    00000001 00000000 f7e1a070 785143e0 3dbcad9d 0000000a 00000e19 f7e1a17c
    7a01a980 78514180 fffb6d29 00000003 00000000 00000000 f7c4bc40 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    ksuspend_usbd S F7E29FA0 0 150 2 (L-TLB)
    f7e29fb4 00000086 00000002 f7e29fa0 f7e29f9c 00000000 f7e84660 00000000
    00000001 00000000 f7e84130 785143e0 3dbd4cb7 0000000a 00001042 f7e8423c
    7a01a980 78514180 fffb6d29 00000003 00000000 00000000 f7c4bb40 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    khubd S 00000000 0 153 2 (L-TLB)
    f7e05f54 00000086 00000046 00000000 00000000 78538a34 f7e225e0 00000000
    0000000a 00000000 f7e220b0 f7fbf490 d1ac4454 0000000b 00005717 f7e221bc
    7a01a980 78514180 00000046 78538a20 00000286 78538a20 00000000 f7d36200
    Call Trace:
    [<78316b2e>] hub_thread+0x98f/0x9f9
    [<7840d732>] _spin_unlock_irq+0x2b/0x41
    [<7840ae59>] __sched_text_start+0x701/0x810
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7831619f>] hub_thread+0x0/0x9f9
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kseriod S 00000000 0 155 2 (L-TLB)
    7a1fdf9c 00000002 00000046 00000000 00000000 7853bcd4 f7e0a660 00000000
    0000000a 00000000 f7e0a130 f7fbf490 0dfac2e3 0000000c 00003730 f7e0a23c
    7a01a980 78514180 00000046 7853bcc0 00000216 7853bcc0 f7d50580 7832efe5
    Call Trace:
    [<7832efe5>] serio_thread+0x0/0x2bb
    [<7832f255>] serio_thread+0x270/0x2bb
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7832efe5>] serio_thread+0x0/0x2bb
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kswapd0 S F7E59F44 0 187 2 (L-TLB)
    f7e59f58 00000086 00000002 f7e59f44 f7e59f40 00000000 7a1fb010 00000000
    00000009 00000001 7a1faae0 f7fbe030 1a81a860 0001e6a0 00e7fbcc 7a1fabec
    7a022980 abc7dd40 1ff863fd 00000003 00000000 00000000 00000001 ffffffff
    Call Trace:
    [<78150a2e>] kswapd+0xb0/0x40f
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7815097e>] kswapd+0x0/0x40f
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    aio/0 S 00000000 0 188 2 (L-TLB)
    f7e5dfb4 00000086 00000046 00000000 00000000 f7ed547c 7a1fba40 00000000
    00000001 00000000 7a1fb510 f7fbea60 42056d94 0000000a 00000dba 7a1fb61c
    7a01a980 78514180 00000046 f7ed5468 00000286 f7ed5468 f7ed5440 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    aio/1 S 00000000 0 189 2 (L-TLB)
    f7e5ffb4 00000086 00000046 00000000 00000000 f7ed53fc 7a1f25a0 00000000
    00000001 00000001 7a1f2070 f7fbf490 4205dc87 0000000a 000009cd 7a1f217c
    7a022980 78514180 00000046 f7ed53e8 00000286 f7ed53e8 f7ed53c0 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    cryptd S F7E7FFA8 0 190 2 (L-TLB)
    f7e7ffbc 00000082 00000002 f7e7ffa8 f7e7ffa4 00000000 787ce0d8 f7fb5590
    00000001 00000000 f7fb5590 785143e0 42116cfc 0000000a 000076c7 f7fb569c
    7a01a980 78514180 fffb6d71 00000003 00000000 00000000 00000000 00000000
    Call Trace:
    [<7820a221>] cryptd_thread+0x95/0xa5
    [<7820a18c>] cryptd_thread+0x0/0xa5
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    scsi_eh_0 S 7A1EA030 0 320 2 (L-TLB)
    f7ca1f80 00000092 f7e0f058 7a1ea030 7840d6e3 f7e0f000 f7cef000 7813e99c
    0000000a 00000000 7a1ea030 f7fbf490 7ebe3537 0000000b 00005ddb 7a1ea13c
    7a01a980 78514180 00000292 f7cef000 f7e0f000 00000000 00000282 f7e0f000
    Call Trace:
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<782e9a32>] scsi_error_handler+0x3b/0x28d
    [<782e99f7>] scsi_error_handler+0x0/0x28d
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    scsi_eh_1 S F7CA5F6C 0 322 2 (L-TLB)
    f7ca5f80 00000092 00000002 f7ca5f6c f7ca5f68 00000000 00000000 7813e99c
    00000003 00000000 f7cf94d0 785143e0 7e3733ad 0000000b 0000c8a9 f7cf95dc
    7a01a980 78514180 fffb8236 00000003 00000000 00000000 00000282 f7cbd800
    Call Trace:
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<782e9a32>] scsi_error_handler+0x3b/0x28d
    [<782e99f7>] scsi_error_handler+0x0/0x28d
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    scsi_eh_2 S F7E0B590 0 339 2 (L-TLB)
    f7ca7f80 00000092 7a1d8058 f7e0b590 7840d6e3 7a1d8000 00000000 7813e99c
    00000006 00000000 f7e0b590 f7fbf490 9835b106 0000000b 004284a4 f7e0b69c
    7a01a980 78514180 00000292 00000282 7a1d8000 00000000 00000282 7a1d8000
    Call Trace:
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<782e9a32>] scsi_error_handler+0x3b/0x28d
    [<782e99f7>] scsi_error_handler+0x0/0x28d
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    scsi_eh_3 S F7CD0030 0 341 2 (L-TLB)
    f7ca3f80 00000092 f7cbd058 f7cd0030 7840d6e3 f7cbd000 00000000 7813e99c
    00000008 00000000 f7cd0030 f7fbf490 a28957bc 0000000b 003e72b9 f7cd013c
    7a01a980 78514180 00000292 00000282 f7cbd000 00000000 00000282 f7cbd000
    Call Trace:
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<782e9a32>] scsi_error_handler+0x3b/0x28d
    [<782e99f7>] scsi_error_handler+0x0/0x28d
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kpsmoused S 00000000 0 407 2 (L-TLB)
    f7d75fb4 00000086 00000046 00000000 00000000 f7d1767c f7d005e0 00000000
    00000009 00000000 f7d000b0 f7fbf490 d3b0a6af 0000000b 0000087a f7d001bc
    7a01a980 78514180 00000046 f7d17668 00000286 f7d17668 f7d17640 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kcryptd/0 S 00000000 0 412 2 (L-TLB)
    f7d77fb4 00000086 00000046 00000000 00000000 f7d170fc f7e33090 00000000
    00000009 00000000 f7e32b60 f7fbf490 e6cc0d2f 0000000b 000007f4 f7e32c6c
    7a01a980 78514180 00000046 f7d170e8 00000286 f7d170e8 f7d170c0 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kcryptd/1 S F7D71FA0 0 413 2 (L-TLB)
    f7d71fb4 00000086 00000002 f7d71fa0 f7d71f9c 00000000 f7d01010 00000000
    00000008 00000001 f7d00ae0 f7fbe030 e6cc9219 0000000b 000010ad f7d00bec
    7a022980 78514180 fffb8914 00000003 00000000 00000000 f7d5cf40 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kmpathd/0 S 00000000 0 414 2 (L-TLB)
    f7d73fb4 00000086 00000046 00000000 00000000 f7d5cd7c f7cf85a0 00000000
    00000009 00000000 f7cf8070 f7fbf490 e6cd348c 0000000b 00000786 f7cf817c
    7a01a980 78514180 00000046 f7d5cd68 00000286 f7d5cd68 f7d5cd40 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kmpathd/1 S F7D6DFA0 0 415 2 (L-TLB)
    f7d6dfb4 00000086 00000002 f7d6dfa0 f7d6df9c 00000000 f7cf8fd0 00000000
    00000008 00000001 f7cf8aa0 f7fbe030 e6cda902 0000000b 00000f2f f7cf8bac
    7a022980 78514180 fffb8914 00000003 00000000 00000000 f7d5ccc0 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    ksnapd S 00000000 0 416 2 (L-TLB)
    f7d31fb4 00000086 00000046 00000000 00000000 f7d5c9fc f7d01a40 00000000
    00000008 00000000 f7d01510 f7fbf490 e72c4b66 0000000b 0000070f f7d0161c
    7a01a980 78514180 00000046 f7d5c9e8 00000286 f7d5c9e8 f7d5c9c0 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kedac S F7D7BF80 0 419 2 (L-TLB)
    f7d7bf94 00000086 00000002 f7d7bf80 f7d7bf7c 00000000 7840d6e3 7a1d2000
    0000000a 00000001 f7e0ab60 f7fbe030 acabac43 0001ff61 00000d5e f7e0ac6c
    7a022980 e913a340 219905f9 00000003 00000000 00000000 f7d7bfa4 219909e0
    Call Trace:
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<78363dd9>] edac_kernel_thread+0x0/0xf3
    [<7812cada>] process_timeout+0x0/0x5
    [<78363dd9>] edac_kernel_thread+0x0/0xf3
    [<78363ea0>] edac_kernel_thread+0xc7/0xf3
    [<78363dd9>] edac_kernel_thread+0x0/0xf3
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    md4_raid5 S F7E2AB20 0 434 2 (L-TLB)
    7a255f84 00000006 7a21a920 f7e2ab20 7840d727 00000000 00000000 7a224d5c
    0000000a 00000000 f7e2ab20 f6928030 63aadcb9 0001ff61 00002f44 f7e2ac2c
    7a01a980 e913a340 00000000 00000000 f7e2ab20 f7e2b050 7fffffff 7fffffff
    Call Trace:
    [<7840d727>] _spin_unlock_irq+0x20/0x41
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<78354f55>] md_thread+0xab/0xdc
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78354eaa>] md_thread+0x0/0xdc
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    md15_raid10 S 7A1F3A00 0 437 2 (L-TLB)
    7a253f84 00000006 7a1f34d0 7a1f3a00 785cde94 7a1f34d0 7840d6e3 785cde80
    0000000a 00000000 7a1f34d0 f6928a60 39bb729c 0001ff61 000011b8 7a1f35dc
    7a01a980 f58bf040 785cde80 7812ccc1 00000000 00000216 7a253f94 219911f1
    Call Trace:
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812ccc1>] __mod_timer+0x92/0x9c
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7812cada>] process_timeout+0x0/0x5
    [<78354f55>] md_thread+0xab/0xdc
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78354eaa>] md_thread+0x0/0xdc
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    md0_raid1 S F7C7DA80 0 439 2 (L-TLB)
    f719ff84 00000086 f7c7d550 f7c7da80 785cde94 f7c7d550 7840d6e3 785cde80
    0000000a 00000000 f7c7d550 f6928030 706bbe1a 0001ff61 00002cc3 f7c7d65c
    7a01a980 e913a340 785cde80 7812ccc1 00000000 00000296 f719ff94 2199158a
    Call Trace:
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812ccc1>] __mod_timer+0x92/0x9c
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7812cada>] process_timeout+0x0/0x5
    [<78354f55>] md_thread+0xab/0xdc
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78354eaa>] md_thread+0x0/0xdc
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kjournald S 00000000 0 441 2 (L-TLB)
    f717dfa8 00000082 00000046 00000000 00000000 f7199d3c f7c74608 f7c745e0
    0000000a 00000000 f7c740b0 f6928030 575613ff 0001ff61 000092f3 f7c741bc
    7a01a980 f6e31800 00000046 00000246 f7199c14 f7199c00 f7199c14 f7199c00
    Call Trace:
    [<781aeb09>] kjournald+0x16d/0x1ec
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781ae99c>] kjournald+0x0/0x1ec
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kjournald S F6DD5F94 0 584 2 (L-TLB)
    f6dd5fa8 00000082 00000002 f6dd5f94 f6dd5f90 00000000 f7fb4688 f7fb4660
    0000000a 00000001 f7fb4130 f7fbe030 545ba352 0001ff61 0000700b f7fb423c
    7a022980 e632d2c0 2199002b 00000003 00000000 00000000 f68c8814 f68c8800
    Call Trace:
    [<781aeb09>] kjournald+0x16d/0x1ec
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781ae99c>] kjournald+0x0/0x1ec
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kjournald S F588DF94 0 585 2 (L-TLB)
    f588dfa8 00000082 00000002 f588df94 f588df90 00000000 f7cd19e8 f7cd19c0
    0000000a 00000000 f7cd1490 785143e0 f0b5365d 0001ff24 00001614 f7cd159c
    7a01a980 f7cafa40 219507da 00000003 00000000 00000000 f58c1814 f58c1800
    Call Trace:
    [<781aeb09>] kjournald+0x16d/0x1ec
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781ae99c>] kjournald+0x0/0x1ec
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kjournald S 00000000 0 586 2 (L-TLB)
    f6fdbfa8 00000082 00000046 00000000 00000000 f58c113c 7a1dcfb8 7a1dcf90
    00000009 00000000 7a1dca60 f7cb80b0 53c6eaba 0000000d 00001d05 7a1dcb6c
    7a01a980 f68e9ac0 00000046 00000246 f58c1014 f58c1000 f58c1014 f58c1000
    Call Trace:
    [<781aeb09>] kjournald+0x16d/0x1ec
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781ae99c>] kjournald+0x0/0x1ec
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kjournald S 00000000 0 587 2 (L-TLB)
    f587bfa8 00000082 00000046 00000000 00000000 f64c993c f7c6da28 f7c6da00
    0000000a 00000001 f7c6d4d0 f7ea0b20 f44b1328 0001e632 0000149c f7c6d5dc
    7a022980 e632d2c0 00000046 00000246 f64c9814 f64c9800 f64c9814 f64c9800
    Call Trace:
    [<781aeb09>] kjournald+0x16d/0x1ec
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781ae99c>] kjournald+0x0/0x1ec
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    kjournald S 00000000 0 588 2 (L-TLB)
    f58c9fa8 00000082 00000046 00000000 00000000 f64c913c 7a1dc588 7a1dc560
    00000008 00000000 7a1dc030 f7cb80b0 5d289ec0 0000000d 00001bb0 7a1dc13c
    7a01a980 f68e9ac0 00000046 00000246 f64c9014 f64c9000 f64c9014 f64c9000
    Call Trace:
    [<781aeb09>] kjournald+0x16d/0x1ec
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781ae99c>] kjournald+0x0/0x1ec
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    syslogd D 00000000 0 2197 1 (NOTLB)
    f6267c74 00000082 00000000 00000000 7813e7a9 f7cc8130 7840d6e3 785cde80
    00000007 00000000 f7cc8130 f7c74ae0 abd727e3 0001ff61 00002fdc f7cc823c
    7a01a980 f6b15840 785cde80 7812ccc1 00000000 00000292 f6267c84 2199064e
    Call Trace:
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812ccc1>] __mod_timer+0x92/0x9c
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812cada>] process_timeout+0x0/0x5
    [<7840b0ce>] io_schedule_timeout+0x1e/0x28
    [<78151eca>] congestion_wait+0x50/0x64
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<7814a483>] generic_file_buffered_write+0x4ee/0x605
    [<7814a8cf>] __generic_file_aio_write_nolock+0x335/0x4df
    [<78128f60>] current_fs_time+0x41/0x46
    [<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7814aad1>] generic_file_aio_write+0x58/0xb6
    [<78199bb4>] ext3_file_write+0x24/0x92
    [<7816402f>] do_sync_readv_writev+0xc1/0xfe
    [<78139494>] getnstimeofday+0x30/0xbc
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78163ee4>] rw_copy_check_uvector+0x50/0xaa
    [<781646b4>] do_readv_writev+0x99/0x164
    [<78199b90>] ext3_file_write+0x0/0x92
    [<781647bc>] vfs_writev+0x3d/0x48
    [<78164b9e>] sys_writev+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    klogd S F5F37DB8 0 2211 1 (NOTLB)
    f5f37dcc 00000086 00000002 f5f37db8 f5f37db4 00000000 00000000 f6f77f54
    00000007 00000000 f7c6caa0 785143e0 579dd14b 0001ff41 000c175e f7c6cbac
    7a01a980 f62ada40 2196e5cc 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78370f09>] sock_aio_write+0xcb/0xd7
    [<7813e986>] trace_hardirqs_on+0x10c/0x14c
    [<78164133>] do_sync_write+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781649e7>] vfs_read+0xcf/0x10a
    [<781648aa>] vfs_write+0x9e/0x10c
    [<78164e0f>] sys_write+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    acpid S F5C35BE8 0 2265 1 (NOTLB)
    f5c35bfc 00000082 00000002 f5c35be8 f5c35be4 00000000 78528e74 f7cc1550
    00000001 00000000 f7cc1550 785143e0 87711dce 0000000f 0000efdb f7cc165c
    7a01a980 f6b150c0 fffbc622 00000003 00000000 00000000 7fffffff 00000000
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<783cbaf6>] unix_poll+0x1a/0x9a
    [<7816e77e>] do_sys_poll+0x258/0x327
    [<7816f1be>] __pollwait+0x0/0xac
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840d732>] _spin_unlock_irq+0x2b/0x41
    [<7840af41>] __sched_text_start+0x7e9/0x810
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<783cc862>] unix_dgram_sendmsg+0x3b3/0x431
    [<7840d33a>] _read_lock+0x33/0x3e
    [<7840d699>] _read_unlock+0x25/0x3b
    [<783cc862>] unix_dgram_sendmsg+0x3b3/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78152de2>] do_wp_page+0x3b9/0x406
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<78152de2>] do_wp_page+0x3b9/0x406
    [<7840d1b1>] _spin_lock+0x33/0x3e
    [<781541bf>] __handle_mm_fault+0x7c3/0x83b
    [<781197d3>] do_page_fault+0x278/0x535
    [<781383ba>] up_read+0x14/0x26
    [<781197d3>] do_page_fault+0x278/0x535
    [<78103e68>] restore_nocheck+0x12/0x15
    [<7811955b>] do_page_fault+0x0/0x535
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7816e881>] sys_poll+0x34/0x37
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    amavisd S F31C9B4C 0 2823 1 (NOTLB)
    f31c9b60 00000096 00000002 f31c9b4c f31c9b48 00000000 7840d6e3 7a1d2000
    00000007 00000001 f7e32130 f7fbe030 65ed4f6c 0001ff60 000151f2 f7e3223c
    7a022980 f64f1800 2198f07d 00000003 00000000 00000000 f31c9b70 2199178d
    Call Trace:
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7812cada>] process_timeout+0x0/0x5
    [<7816ec1d>] do_select+0x399/0x3e7
    [<781299b4>] local_bh_enable_ip+0xcd/0xed
    [<7813e986>] trace_hardirqs_on+0x10c/0x14c
    [<7838b171>] __nf_ct_refresh_acct+0xcf/0x10a
    [<7816f1be>] __pollwait+0x0/0xac
    [<7838b171>] __nf_ct_refresh_acct+0xcf/0x10a
    [<7838eaf9>] tcp_packet+0xa0f/0xa3e
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<78160ec2>] kmem_cache_free+0x9a/0xa1
    [<7813e986>] trace_hardirqs_on+0x10c/0x14c
    [<78160ec2>] kmem_cache_free+0x9a/0xa1
    [<7813e986>] trace_hardirqs_on+0x10c/0x14c
    [<783bb722>] inet_sock_destruct+0x179/0x1bb
    [<783b147b>] tcp_v4_rcv+0x51a/0x787
    [<783994b1>] ip_local_deliver+0x1bd/0x1c8
    [<78398c10>] ip_local_deliver_finish+0x0/0x14c
    [<783992bb>] ip_rcv+0x498/0x4d1
    [<7839896c>] ip_rcv_finish+0x0/0x2a4
    [<78139494>] getnstimeofday+0x30/0xbc
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7816ef17>] core_sys_select+0x2ac/0x2c9
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<78160ec2>] kmem_cache_free+0x9a/0xa1
    [<7813e986>] trace_hardirqs_on+0x10c/0x14c
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<782be370>] e1000_clean_tx_irq+0xbc/0x2c5
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7812e2a3>] group_send_sig_info+0x12/0x56
    [<7812e378>] kill_pid_info+0x61/0x7b
    [<7812f183>] sys_kill+0x108/0x125
    [<7816f30a>] sys_select+0xa0/0x188
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    clamd S 00000000 0 2889 1 (NOTLB)
    f32d1e58 00000086 00000002 00000000 00010822 00000000 00000000 ee9a69ac
    00000001 00000001 f7cb80b0 db4c8030 6f522ff7 0001e655 00010e05 f7cb81bc
    7a022980 f68e9ac0 00000000 00000000 f7cb80b0 f7cb85e0 7fffffff f5e2d380
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<783787ab>] skb_recv_datagram+0x13f/0x1c4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783ccdf1>] unix_accept+0x51/0xde
    [<7837262e>] sys_accept+0xb2/0x187
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<781671ca>] sys_fstat64+0x1e/0x23
    [<783727bc>] sys_socketcall+0xb9/0x242
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    freshclam D 00000000 0 2898 1 (NOTLB)
    f2f43cc4 00000082 00000000 00000000 7813e7a9 f6928030 7840d6e3 785cde80
    00000008 00000000 f6928030 f7cc8130 abd6f807 0001ff61 00003332 f692813c
    7a01a980 f58bfa40 785cde80 7812ccc1 00000000 00000296 f2f43cd4 2199064e
    Call Trace:
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812ccc1>] __mod_timer+0x92/0x9c
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812cada>] process_timeout+0x0/0x5
    [<7840b0ce>] io_schedule_timeout+0x1e/0x28
    [<78151eca>] congestion_wait+0x50/0x64
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<7814a483>] generic_file_buffered_write+0x4ee/0x605
    [<78128f60>] current_fs_time+0x41/0x46
    [<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7814aad1>] generic_file_aio_write+0x58/0xb6
    [<78199bb4>] ext3_file_write+0x24/0x92
    [<78164133>] do_sync_write+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7840d727>] _spin_unlock_irq+0x20/0x41
    [<7840d732>] _spin_unlock_irq+0x2b/0x41
    [<7816406c>] do_sync_write+0x0/0x10a
    [<78164896>] vfs_write+0x8a/0x10c
    [<78164e0f>] sys_write+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    named S F0989D0C 0 2959 1 (NOTLB)
    f0989d20 00200092 00000002 f0989d0c f0989d08 00000000 00000000 f6f77f54
    0000000a 00000000 f7e13490 785143e0 0e816649 0001e659 00013c17 f7e1359c
    7a01a980 f58bf2c0 1ff3b86f 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<78171ebc>] dput+0x16/0xe4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<78169ed9>] path_release+0xa/0x20
    [<78166c60>] cp_new_stat64+0xfc/0x10e
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    crond D 00000000 0 3061 1 (NOTLB)
    f1367cc4 00000082 00000000 00000000 7813e7a9 f7e02b20 7840d6e3 7a1d2000
    00000008 00000001 f7e02b20 edcd0a60 abd708dc 0001ff61 0000367c f7e02c2c
    7a022980 f58bfcc0 7a1d2000 7812ccc1 00000000 00000296 f1367cd4 2199064e
    Call Trace:
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812ccc1>] __mod_timer+0x92/0x9c
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812cada>] process_timeout+0x0/0x5
    [<7840b0ce>] io_schedule_timeout+0x1e/0x28
    [<78151eca>] congestion_wait+0x50/0x64
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<7814a483>] generic_file_buffered_write+0x4ee/0x605
    [<78128f60>] current_fs_time+0x41/0x46
    [<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7814aad1>] generic_file_aio_write+0x58/0xb6
    [<78199bb4>] ext3_file_write+0x24/0x92
    [<78164133>] do_sync_write+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78160ec2>] kmem_cache_free+0x9a/0xa1
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7816406c>] do_sync_write+0x0/0x10a
    [<78164896>] vfs_write+0x8a/0x10c
    [<78164e0f>] sys_write+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    mdadm S F1ADDB4C 0 3110 1 (NOTLB)
    f1addb60 00000096 00000002 f1addb4c f1addb48 00000000 7840d6e3 785cde80
    00000007 00000000 f6928a60 785143e0 39bb937f 0001ff61 000020e3 f6928b6c
    7a01a980 f58bf040 2198fe6a 00000003 00000000 00000000 f1addb70 2199e8c9
    Call Trace:
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7812cada>] process_timeout+0x0/0x5
    [<7816ec1d>] do_select+0x399/0x3e7
    [<7816f1be>] __pollwait+0x0/0xac
    [<7812027d>] default_wake_function+0x0/0xc
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7840d727>] _spin_unlock_irq+0x20/0x41
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840d732>] _spin_unlock_irq+0x2b/0x41
    [<7840af41>] __sched_text_start+0x7e9/0x810
    [<7840ae59>] __sched_text_start+0x701/0x810
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7811e3ba>] __wake_up+0x32/0x43
    [<7840b09e>] preempt_schedule+0x46/0x58
    [<7834dc68>] md_wakeup_thread+0x26/0x28
    [<78354e25>] md_ioctl+0x1261/0x12e6
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7840d727>] _spin_unlock_irq+0x20/0x41
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7816ef17>] core_sys_select+0x2ac/0x2c9
    [<7840bd93>] __mutex_unlock_slowpath+0x105/0x127
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78352e61>] md_open+0x51/0x58
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<781335c2>] call_rcu+0x64/0x69
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7816f30a>] sys_select+0xa0/0x188
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    portmap S F1571BE8 0 3161 1 (NOTLB)
    f1571bfc 00000082 00000002 f1571be8 f1571be4 00000000 00000000 00000000
    00000007 00000001 7a1dd490 f7fbe030 e696d8aa 00000013 000281e5 7a1dd59c
    7a022980 f6e31080 fffc0fae 00000003 00000000 00000000 7fffffff 00000000
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<783a2023>] tcp_poll+0x1c/0x129
    [<783b435a>] udp_poll+0x10/0xdb
    [<7816e77e>] do_sys_poll+0x258/0x327
    [<7816f1be>] __pollwait+0x0/0xac
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<7840d1f4>] _spin_lock_bh+0x38/0x43
    [<783b57a5>] udp_sendmsg+0x4a7/0x5b9
    [<783ba7cb>] inet_sendmsg+0x3b/0x45
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78128f60>] current_fs_time+0x41/0x46
    [<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
    [<78377ed6>] verify_iovec+0x3e/0x70
    [<783719e4>] sys_sendmsg+0x194/0x1f9
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7814cb18>] free_hot_cold_page+0x13b/0x167
    [<7814cb24>] free_hot_cold_page+0x147/0x167
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7814cb58>] __pagevec_free+0x14/0x1a
    [<7814ecc5>] release_pages+0x126/0x12e
    [<78159060>] anon_vma_unlink+0x13/0x53
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7816e881>] sys_poll+0x34/0x37
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    rpciod/0 S 00000000 0 3216 2 (L-TLB)
    ec88dfb4 00000086 00000046 00000000 00000000 f6343c7c 7a1eb9c0 00000000
    0000000a 00000000 7a1eb490 89333550 78dbc731 0001e655 0000401b 7a1eb59c
    7a01a980 c053d840 00000046 f6343c68 00000286 f6343c68 f6343c40 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    rpciod/1 S 00000000 0 3217 2 (L-TLB)
    ec8e7fb4 00000086 00000046 00000000 00000000 f5a6867c f69299c0 00000000
    0000000a 00000001 f6929490 a79ec0b0 2e2b55c5 00000834 0000ce3a f692959c
    7a022980 a71d7340 00000046 f5a68668 00000286 f5a68668 f5a68640 78132a2b
    Call Trace:
    [<78132a2b>] worker_thread+0x0/0xc5
    [<78132ab3>] worker_thread+0x88/0xc5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    lockd S 00000001 0 3218 2 (L-TLB)
    ec865f18 00000082 7813534c 00000001 784d6071 f7e02628 00000000 00000000
    00000001 00000000 f7e020f0 f7ca8a60 e69b3708 00000013 0000a1e7 f7e021fc
    7a01a980 f68e90c0 f7e02620 f7e020f0 781299b4 f0fecd7c 7fffffff f0fec000
    Call Trace:
    [<7813534c>] add_wait_queue+0x12/0x30
    [<781299b4>] local_bh_enable_ip+0xcd/0xed
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e986>] trace_hardirqs_on+0x10c/0x14c
    [<783ffbab>] svc_recv+0x249/0x3ca
    [<7812027d>] default_wake_function+0x0/0xc
    [<781e6643>] lockd+0xe8/0x1fd
    [<78121e77>] schedule_tail+0x0/0xb6
    [<78103e68>] restore_nocheck+0x12/0x15
    [<781e655b>] lockd+0x0/0x1fd
    [<781e655b>] lockd+0x0/0x1fd
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    ntpd S EC959B4C 0 3269 1 (NOTLB)
    ec959b60 00200096 00000002 ec959b4c ec959b48 00000000 f7cb05a0 00200046
    0000000a 00000000 f7cb0070 785143e0 a27c1eef 0001ff61 0000aea1 f7cb017c
    7a01a980 f7cafa40 2199054d 00000003 00000000 00000000 7fffffff ec959f9c
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<783b435a>] udp_poll+0x10/0xdb
    [<7816ec1d>] do_select+0x399/0x3e7
    [<7816f1be>] __pollwait+0x0/0xac
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<783760ad>] skb_dequeue+0x39/0x3f
    [<783786ed>] skb_recv_datagram+0x81/0x1c4
    [<783b5b8f>] udp_recvmsg+0x57/0x1cd
    [<78372cc9>] sock_common_recvmsg+0x3e/0x54
    [<78371763>] sock_recvmsg+0xcf/0xe8
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7816ef17>] core_sys_select+0x2ac/0x2c9
    [<7837221d>] sys_recvmsg+0x11d/0x1cf
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<7840d732>] _spin_unlock_irq+0x2b/0x41
    [<78103986>] do_notify_resume+0x511/0x5f3
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7816f340>] sys_select+0xd6/0x188
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    sqlgrey S 00000000 0 3323 1 (NOTLB)
    ec79fb60 00200096 786b7ed0 00000000 7837cbf7 00200046 00000000 00000000
    00000004 00000001 f7cb0aa0 f7fbf490 b29583a6 0001f67f 00068968 f7cb0bac
    7a022980 f68e90c0 7813e99c 00000000 00200046 ec975398 7fffffff ec79ff9c
    Call Trace:
    [<7837cbf7>] dev_queue_xmit+0x1f2/0x213
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<783a2023>] tcp_poll+0x1c/0x129
    [<7816ec1d>] do_select+0x399/0x3e7
    [<7816f1be>] __pollwait+0x0/0xac
    [<7812027d>] default_wake_function+0x0/0xc
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<781299b4>] local_bh_enable_ip+0xcd/0xed
    [<7813e986>] trace_hardirqs_on+0x10c/0x14c
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<783b1465>] tcp_v4_rcv+0x504/0x787
    [<783b133f>] tcp_v4_rcv+0x3de/0x787
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<783b1465>] tcp_v4_rcv+0x504/0x787
    [<7839949e>] ip_local_deliver+0x1aa/0x1c8
    [<78398c10>] ip_local_deliver_finish+0x0/0x14c
    [<783992bb>] ip_rcv+0x498/0x4d1
    [<7839896c>] ip_rcv_finish+0x0/0x2a4
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7837c73c>] process_backlog+0xef/0xfa
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7837c9d8>] net_rx_action+0x162/0x18f
    [<7813e986>] trace_hardirqs_on+0x10c/0x14c
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<781298c7>] local_bh_enable+0x110/0x130
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7837cbf7>] dev_queue_xmit+0x1f2/0x213
    [<7816ef17>] core_sys_select+0x2ac/0x2c9
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78160ec2>] kmem_cache_free+0x9a/0xa1
    [<78160ec2>] kmem_cache_free+0x9a/0xa1
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7816f340>] sys_select+0xd6/0x188
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    smartd S EA68DD0C 0 3487 1 (NOTLB)
    ea68dd20 00000092 00000002 ea68dd0c ea68dd08 00000000 00000000 f6f77f54
    00000009 00000000 ec907490 785143e0 ad75944f 0001e66c 00034dc2 ec90759c
    7a01a980 f6e11040 1ff502a1 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783cc30c>] unix_wait_for_peer+0x75/0x9a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
    [<78371838>] sock_sendmsg+0xbc/0xd4
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813facb>] lock_release_non_nested+0xec/0x14d
    [<783cdc63>] unix_dgram_connect+0x8d/0x157
    [<783cdd00>] unix_dgram_connect+0x12a/0x157
    [<783cdc63>] unix_dgram_connect+0x8d/0x157
    [<78371b64>] sys_sendto+0x11b/0x13b
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<783cdd00>] unix_dgram_connect+0x12a/0x157
    [<7837135e>] sys_connect+0x72/0x9c
    [<78371bbb>] sys_send+0x37/0x3b
    [<78372830>] sys_socketcall+0x12d/0x242
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    snmpd D 00000000 0 3546 1 (NOTLB)
    ea333cc4 00200082 00000000 00000000 7813e7a9 f7c74ae0 7840d6e3 785cde80
    00000007 00000000 f7c74ae0 89333550 abd756a0 0001ff61 00002ebd f7c74bec
    7a01a980 f6e31800 785cde80 7812ccc1 00000000 00200296 ea333cd4 2199064e
    Call Trace:
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812ccc1>] __mod_timer+0x92/0x9c
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812cada>] process_timeout+0x0/0x5
    [<7840b0ce>] io_schedule_timeout+0x1e/0x28
    [<78151eca>] congestion_wait+0x50/0x64
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<7814a483>] generic_file_buffered_write+0x4ee/0x605
    [<78128f60>] current_fs_time+0x41/0x46
    [<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7814aad1>] generic_file_aio_write+0x58/0xb6
    [<78199bb4>] ext3_file_write+0x24/0x92
    [<78164133>] do_sync_write+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78160ec2>] kmem_cache_free+0x9a/0xa1
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7816406c>] do_sync_write+0x0/0x10a
    [<78164896>] vfs_write+0x8a/0x10c
    [<78164e0f>] sys_write+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    squid S E9A41F20 0 3604 1 (NOTLB)
    e9a41f34 00200082 00000002 e9a41f20 e9a41f1c 00000000 e9e6da40 e9e6da40
    00000005 00000000 e9e6d510 785143e0 74bc2237 00000016 00014e53 e9e6d61c
    7a01a980 f6e31d00 fffc3aae 00000003 00000000 00000000 ffffffff 00000001
    Call Trace:
    [<781276ea>] do_wait+0x8eb/0x9e6
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7812027d>] default_wake_function+0x0/0xc
    [<78127816>] sys_wait4+0x31/0x34
    [<78127840>] sys_waitpid+0x27/0x2b
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    squid D 00000000 0 3606 3604 (NOTLB)
    e9a65cc4 00000082 00000000 00000000 7813e7a9 f7cb9510 7840d6e3 7a1d2000
    00000007 00000001 f7cb9510 edcd1490 abd7ba4d 0001ff61 000021db f7cb961c
    7a022980 f6ce25c0 7a1d2000 7812ccc1 00000000 00000296 e9a65cd4 2199064e
    Call Trace:
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812ccc1>] __mod_timer+0x92/0x9c
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812cada>] process_timeout+0x0/0x5
    [<7840b0ce>] io_schedule_timeout+0x1e/0x28
    [<78151eca>] congestion_wait+0x50/0x64
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<7814a483>] generic_file_buffered_write+0x4ee/0x605
    [<78128f60>] current_fs_time+0x41/0x46
    [<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7814aad1>] generic_file_aio_write+0x58/0xb6
    [<78199bb4>] ext3_file_write+0x24/0x92
    [<78164133>] do_sync_write+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781648dd>] vfs_write+0xd1/0x10c
    [<7816406c>] do_sync_write+0x0/0x10a
    [<78164896>] vfs_write+0x8a/0x10c
    [<78164e0f>] sys_write+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    sshd S E9239B4C 0 3663 1 (NOTLB)
    e9239b60 00200096 00000002 e9239b4c e9239b48 00000000 00000000 00000000
    00000006 00000000 e9b7e0f0 785143e0 61b30795 0001ff2d 000245bd e9b7e1fc
    7a01a980 f62ad540 219595ea 00000003 00000000 00000000 7fffffff e9239f9c
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<783a2023>] tcp_poll+0x1c/0x129
    [<7816e4d2>] free_poll_entry+0xe/0x16
    [<7816ec1d>] do_select+0x399/0x3e7
    [<7816f1be>] __pollwait+0x0/0xac
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<783b1465>] tcp_v4_rcv+0x504/0x787
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<783b1465>] tcp_v4_rcv+0x504/0x787
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<78160ec2>] kmem_cache_free+0x9a/0xa1
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<78160ec2>] kmem_cache_free+0x9a/0xa1
    [<7813e986>] trace_hardirqs_on+0x10c/0x14c
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<782be370>] e1000_clean_tx_irq+0xbc/0x2c5
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7816ef17>] core_sys_select+0x2ac/0x2c9
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78160ec2>] kmem_cache_free+0x9a/0xa1
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7840d727>] _spin_unlock_irq+0x20/0x41
    [<7816f340>] sys_select+0xd6/0x188
    [<78103e68>] restore_nocheck+0x12/0x15
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    zabbix_agentd S 7A1F2AA0 0 3722 1 (NOTLB)
    e9369f34 00200082 7854ec14 7a1f2aa0 00000001 786b7e90 7a1f2fd0 7a1f2fd0
    00000001 00000001 7a1f2aa0 f7fb4b60 0ec74ab6 00000017 0000b6ef 7a1f2bac
    7a022980 e9314540 00000000 00200246 7854ec00 00000001 00000e8c 00000001
    Call Trace:
    [<781276ea>] do_wait+0x8eb/0x9e6
    [<781197d3>] do_page_fault+0x278/0x535
    [<7812027d>] default_wake_function+0x0/0xc
    [<78127816>] sys_wait4+0x31/0x34
    [<78127840>] sys_waitpid+0x27/0x2b
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    zabbix_agentd S E93D1F2C 0 3724 3722 (NOTLB)
    e93d1f40 00200092 00000002 e93d1f2c e93d1f28 00000000 7813e99c 7a02391c
    00000008 00000001 ec8de0f0 f7fbe030 aa5c0061 0001ff61 0004acea ec8de1fc
    7a022980 e9314cc0 219905d2 00000003 00000000 00000000 e93d1f58 00000001
    Call Trace:
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840c5af>] do_nanosleep+0x48/0x73
    [<78137d84>] hrtimer_nanosleep+0x39/0xdc
    [<78137879>] hrtimer_wakeup+0x0/0x18
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78137e70>] sys_nanosleep+0x49/0x59
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    zabbix_agentd S 00000070 0 3725 3722 (NOTLB)
    e8ccbe44 00200086 00000001 00000070 7a01b3e0 00000002 00000000 ead5e140
    00000001 00000000 e90b8b60 e9e6cae0 0f7701b8 00000017 000051d5 e90b8c6c
    7a01a980 e913a0c0 fffc44d7 00000003 00000000 00000070 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<783a0b90>] inet_csk_accept+0x9e/0x214
    [<78371564>] sock_attach_fd+0x53/0xb5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783bb524>] inet_accept+0x1f/0xa4
    [<78371564>] sock_attach_fd+0x53/0xb5
    [<7837262e>] sys_accept+0xb2/0x187
    [<78153c96>] __handle_mm_fault+0x29a/0x83b
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<78153d89>] __handle_mm_fault+0x38d/0x83b
    [<781197d3>] do_page_fault+0x278/0x535
    [<783727bc>] sys_socketcall+0xb9/0x242
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103e68>] restore_nocheck+0x12/0x15
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    zabbix_agentd S E8CCDE30 0 3726 3722 (NOTLB)
    e8ccde44 00200086 00000002 e8ccde30 e8ccde2c 00000000 00000000 ead5e140
    00000001 00000000 e9e6cae0 785143e0 0fa415f1 00000017 0001181a e9e6cbec
    7a01a980 e93147c0 fffc44d7 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<783a0b90>] inet_csk_accept+0x9e/0x214
    [<78371564>] sock_attach_fd+0x53/0xb5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783bb524>] inet_accept+0x1f/0xa4
    [<78371564>] sock_attach_fd+0x53/0xb5
    [<7837262e>] sys_accept+0xb2/0x187
    [<78153c96>] __handle_mm_fault+0x29a/0x83b
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<78153d89>] __handle_mm_fault+0x38d/0x83b
    [<781197d3>] do_page_fault+0x278/0x535
    [<783727bc>] sys_socketcall+0xb9/0x242
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103e68>] restore_nocheck+0x12/0x15
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    zabbix_agentd S 00000000 0 3727 3722 (NOTLB)
    e8cede44 00200086 00000000 00000000 7813e7a9 00000000 00000000 ead5e140
    00000001 00000001 f7cb8ae0 e9e6c0b0 0fbbed36 00000017 0003651f f7cb8bec
    7a022980 f62ad040 f7cb9010 f7cb8ae0 781299b4 e8cedea0 7fffffff 7fffffff
    Call Trace:
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<781299b4>] local_bh_enable_ip+0xcd/0xed
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<783a0b90>] inet_csk_accept+0x9e/0x214
    [<78371564>] sock_attach_fd+0x53/0xb5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783bb524>] inet_accept+0x1f/0xa4
    [<78371564>] sock_attach_fd+0x53/0xb5
    [<7837262e>] sys_accept+0xb2/0x187
    [<78153c96>] __handle_mm_fault+0x29a/0x83b
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<78153d89>] __handle_mm_fault+0x38d/0x83b
    [<781197d3>] do_page_fault+0x278/0x535
    [<783727bc>] sys_socketcall+0xb9/0x242
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103e68>] restore_nocheck+0x12/0x15
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    zabbix_agentd D E8CEFCB0 0 3728 3722 (NOTLB)
    e8cefcc4 00200082 00000002 e8cefcb0 e8cefcac 00000000 7840d6e3 7a1d2000
    00000008 00000001 e9e6c0b0 f7fbe030 abd7fd2c 0001ff61 00002173 e9e6c1bc
    7a022980 e913a340 219905eb 00000003 00000000 00000000 e8cefcd4 2199064e
    Call Trace:
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812cada>] process_timeout+0x0/0x5
    [<7840b0ce>] io_schedule_timeout+0x1e/0x28
    [<78151eca>] congestion_wait+0x50/0x64
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<7814a483>] generic_file_buffered_write+0x4ee/0x605
    [<7814a8cf>] __generic_file_aio_write_nolock+0x335/0x4df
    [<78128f60>] current_fs_time+0x41/0x46
    [<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7814aad1>] generic_file_aio_write+0x58/0xb6
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7814ce3d>] get_page_from_freelist+0x293/0x35e
    [<78199bb4>] ext3_file_write+0x24/0x92
    [<78164133>] do_sync_write+0xc7/0x10a
    [<78153e4d>] __handle_mm_fault+0x451/0x83b
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781197d3>] do_page_fault+0x278/0x535
    [<7816406c>] do_sync_write+0x0/0x10a
    [<78164896>] vfs_write+0x8a/0x10c
    [<7811955b>] do_page_fault+0x0/0x535
    [<78164e0f>] sys_write+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    agetty S E8849EC8 0 3791 1 (NOTLB)
    e8849edc 00000086 00000002 e8849ec8 e8849ec4 00000000 00000000 00000000
    00000007 00000001 f7fb4b60 f7fbe030 6d0455f6 00000017 0018bcb8 f7fb4c6c
    7a022980 f6b15d40 fffc4afe 00000003 00000000 00000000 7fffffff f539f000
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<7829cf28>] read_chan+0x2fb/0x54b
    [<78297ed7>] tty_ldisc_try+0x2e/0x32
    [<7812027d>] default_wake_function+0x0/0xc
    [<7829cc2d>] read_chan+0x0/0x54b
    [<7829a215>] tty_read+0x67/0xaf
    [<7829a1ae>] tty_read+0x0/0xaf
    [<781649a0>] vfs_read+0x88/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    agetty S 00000000 0 3793 1 (NOTLB)
    e8d19edc 00000086 00000000 00000000 f53a5c8c ea120a60 00000000 00000000
    00000007 00000001 ea120a60 f7e12030 70e57c7e 00000017 0010cf04 ea120b6c
    7a022980 e9314040 ea120f98 ea120a60 00000000 00000000 7fffffff f53a5800
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<7829cf28>] read_chan+0x2fb/0x54b
    [<78297ed7>] tty_ldisc_try+0x2e/0x32
    [<7812027d>] default_wake_function+0x0/0xc
    [<7829cc2d>] read_chan+0x0/0x54b
    [<7829a215>] tty_read+0x67/0xaf
    [<7829a1ae>] tty_read+0x0/0xaf
    [<781649a0>] vfs_read+0x88/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    agetty S 00000000 0 3794 1 (NOTLB)
    e8d61edc 00000086 00000000 00000000 f56dcc8c f7e12030 00000000 00000000
    00000007 00000001 f7e12030 ec8deb20 70f48c60 00000017 000f0fe2 f7e1213c
    7a022980 f62adcc0 f7e12568 f7e12030 00000000 00000000 7fffffff f56dc800
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<7829cf28>] read_chan+0x2fb/0x54b
    [<78297ed7>] tty_ldisc_try+0x2e/0x32
    [<7812027d>] default_wake_function+0x0/0xc
    [<7829cc2d>] read_chan+0x0/0x54b
    [<7829a215>] tty_read+0x67/0xaf
    [<7829a1ae>] tty_read+0x0/0xaf
    [<781649a0>] vfs_read+0x88/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    agetty S 00000000 0 3795 1 (NOTLB)
    e8d23edc 00000086 00000000 00000000 f52fcc8c ec8deb20 00000000 00000000
    00000007 00000001 ec8deb20 ec8ff590 71036a29 00000017 000eddc9 ec8dec2c
    7a022980 e913a840 ec8df058 ec8deb20 00000000 00000000 7fffffff f52fc800
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<7829cf28>] read_chan+0x2fb/0x54b
    [<78297ed7>] tty_ldisc_try+0x2e/0x32
    [<7812027d>] default_wake_function+0x0/0xc
    [<7829cc2d>] read_chan+0x0/0x54b
    [<7829a215>] tty_read+0x67/0xaf
    [<7829a1ae>] tty_read+0x0/0xaf
    [<781649a0>] vfs_read+0x88/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    agetty S 00000000 0 3796 1 (NOTLB)
    e8903edc 00000086 00000000 00000000 f583748c ec8ff590 00000000 00000000
    00000007 00000001 ec8ff590 e8eea130 71118e5d 00000017 000e2434 ec8ff69c
    7a022980 f6f5b080 ec8ffac8 ec8ff590 00000000 00000000 7fffffff f5837000
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<7829cf28>] read_chan+0x2fb/0x54b
    [<78297ed7>] tty_ldisc_try+0x2e/0x32
    [<7812027d>] default_wake_function+0x0/0xc
    [<7829cc2d>] read_chan+0x0/0x54b
    [<7829a215>] tty_read+0x67/0xaf
    [<7829a1ae>] tty_read+0x0/0xaf
    [<781649a0>] vfs_read+0x88/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    agetty S E8921EC8 0 3797 1 (NOTLB)
    e8921edc 00000086 00000002 e8921ec8 e8921ec4 00000000 00000000 00000000
    00000007 00000001 e8eea130 f7fbe030 713eaf24 00000017 00023594 e8eea23c
    7a022980 e913a5c0 fffc4b41 00000003 00000000 00000000 7fffffff f7242000
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<7829cf28>] read_chan+0x2fb/0x54b
    [<78297ed7>] tty_ldisc_try+0x2e/0x32
    [<7812027d>] default_wake_function+0x0/0xc
    [<7829cc2d>] read_chan+0x0/0x54b
    [<7829a215>] tty_read+0x67/0xaf
    [<7829a1ae>] tty_read+0x0/0xaf
    [<781649a0>] vfs_read+0x88/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    agetty S 00000000 0 3798 1 (NOTLB)
    e8923edc 00000086 00000000 00000000 f5d5048c ec906030 00000000 00000000
    00000007 00000001 ec906030 ea024070 712da585 00000017 000ebb02 ec90613c
    7a022980 e913aac0 ec906568 ec906030 00000000 00000000 7fffffff f5d50000
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<7829cf28>] read_chan+0x2fb/0x54b
    [<78297ed7>] tty_ldisc_try+0x2e/0x32
    [<7812027d>] default_wake_function+0x0/0xc
    [<7829cc2d>] read_chan+0x0/0x54b
    [<7829a215>] tty_read+0x67/0xaf
    [<7829a1ae>] tty_read+0x0/0xaf
    [<781649a0>] vfs_read+0x88/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    agetty S 00000000 0 3799 1 (NOTLB)
    e8925edc 00000086 00000000 00000000 f6f2348c ea024070 00000000 00000000
    00000007 00000001 ea024070 e8eea130 713c7990 00000017 000ed40b ea02417c
    7a022980 f6e31580 ea0245a8 ea024070 00000000 00000000 7fffffff f6f23000
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<7829cf28>] read_chan+0x2fb/0x54b
    [<78297ed7>] tty_ldisc_try+0x2e/0x32
    [<7812027d>] default_wake_function+0x0/0xc
    [<7829cc2d>] read_chan+0x0/0x54b
    [<7829a215>] tty_read+0x67/0xaf
    [<7829a1ae>] tty_read+0x0/0xaf
    [<781649a0>] vfs_read+0x88/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    sshd S E3429DC4 0 3835 3663 (NOTLB)
    e3429dd8 00000082 00000002 e3429dc4 e3429dc0 00000000 00000000 e12799ac
    00000007 00000000 7a1eaa60 785143e0 1e0f5b36 00000025 00001291 7a1eab6c
    7a01a980 e8d34d00 fffd3160 00000003 00000000 00000000 7fffffff 00000000
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78370fdd>] sock_aio_read+0xc8/0xd4
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7816423d>] do_sync_read+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78160ec2>] kmem_cache_free+0x9a/0xa1
    [<781649b4>] vfs_read+0x9c/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    sshd S E6C59B4C 0 3838 3835 (NOTLB)
    e6c59b60 00000096 00000002 e6c59b4c e6c59b48 00000000 e90b8688 e90b8668
    0000000a 00000000 e90b8130 785143e0 28faff08 0001ff23 0001a008 e90b823c
    7a01a980 f71bd580 2194e9e6 00000003 00000000 00000000 7fffffff e6c59f9c
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<7829cb0e>] normal_poll+0x0/0x11f
    [<78298d67>] tty_poll+0x56/0x63
    [<7816ec1d>] do_select+0x399/0x3e7
    [<7816f1be>] __pollwait+0x0/0xac
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7837cbf7>] dev_queue_xmit+0x1f2/0x213
    [<7839dedc>] ip_output+0x270/0x2ac
    [<7839c7d5>] ip_finish_output+0x0/0x212
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7839b270>] dst_output+0x0/0x7
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<781299b4>] local_bh_enable_ip+0xcd/0xed
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783a3340>] tcp_sendmsg+0x91c/0xa0d
    [<7840d1f4>] _spin_lock_bh+0x38/0x43
    [<783a3340>] tcp_sendmsg+0x91c/0xa0d
    [<7816ef17>] core_sys_select+0x2ac/0x2c9
    [<78370f09>] sock_aio_write+0xcb/0xd7
    [<78164133>] do_sync_write+0xc7/0x10a
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7816f340>] sys_select+0xd6/0x188
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78164e0f>] sys_write+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    bash S EA120030 0 3839 3838 (NOTLB)
    e3c81f34 00000082 7854ec14 ea120030 00000001 786b7e90 ea120560 ea120560
    00000009 00000000 ea120030 e7428030 3ac85a37 00000028 00019fc2 ea12013c
    7a01a980 f71bd800 00000000 00000246 7854ec00 00000001 ffffffff 00000001
    Call Trace:
    [<781276ea>] do_wait+0x8eb/0x9e6
    [<7812027d>] default_wake_function+0x0/0xc
    [<78127816>] sys_wait4+0x31/0x34
    [<78127840>] sys_waitpid+0x27/0x2b
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    bash S E7428030 0 3866 3839 (NOTLB)
    e5f65f34 00000082 7854ec14 e7428030 00000001 786b7e90 e7428560 e7428560
    00000008 00000000 e7428030 f7c6c070 8cd13ab9 0001ff22 0001ae9d e742813c
    7a01a980 f71bda80 00000000 00000246 7854ec00 00000001 ffffffff 00000001
    Call Trace:
    [<781276ea>] do_wait+0x8eb/0x9e6
    [<7812027d>] default_wake_function+0x0/0xc
    [<78127816>] sys_wait4+0x31/0x34
    [<78127840>] sys_waitpid+0x27/0x2b
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    sshd S 00000127 0 3878 3663 (NOTLB)
    e3d2bdd8 00000082 00000000 00000127 00000000 00000000 00000000 e4c4c0ac
    00000007 00000001 e21fa0b0 e21fb510 5abe5861 0000002b 00001417 e21fa1bc
    7a022980 f71bd080 e21fa5e0 e21fa0b0 00000000 e21fa0b0 7fffffff 00000000
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78370fdd>] sock_aio_read+0xc8/0xd4
    [<7816423d>] do_sync_read+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78160ec2>] kmem_cache_free+0x9a/0xa1
    [<781649b4>] vfs_read+0x9c/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    sshd S 00000000 0 3880 3878 (NOTLB)
    e406fb60 00000096 00000046 00000000 00000002 00000001 e21fba68 e21fba48
    0000000a 00000000 e21fb510 e4632130 adc501bc 0001ff61 00010239 e21fb61c
    7a01a980 e93142c0 e91624c0 e1acb80c 7813e99c e91624c0 7fffffff e406ff9c
    Call Trace:
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<7829cb0e>] normal_poll+0x0/0x11f
    [<78298d67>] tty_poll+0x56/0x63
    [<7816ec1d>] do_select+0x399/0x3e7
    [<7816f1be>] __pollwait+0x0/0xac
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7837cbf7>] dev_queue_xmit+0x1f2/0x213
    [<7839dedc>] ip_output+0x270/0x2ac
    [<7839c7d5>] ip_finish_output+0x0/0x212
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7839b270>] dst_output+0x0/0x7
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
    [<781299b4>] local_bh_enable_ip+0xcd/0xed
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783a3340>] tcp_sendmsg+0x91c/0xa0d
    [<7840d1f4>] _spin_lock_bh+0x38/0x43
    [<783a3340>] tcp_sendmsg+0x91c/0xa0d
    [<78120273>] try_to_wake_up+0x393/0x39d
    [<7816ef17>] core_sys_select+0x2ac/0x2c9
    [<78370f09>] sock_aio_write+0xcb/0xd7
    [<78164133>] do_sync_write+0xc7/0x10a
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7816f340>] sys_select+0xd6/0x188
    [<78103dc6>] sysenter_past_esp+0x8f/0x99
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78164e0f>] sys_write+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    bash S E401D550 0 3881 3880 (NOTLB)
    e401ff34 00000082 7854ec14 e401d550 00000001 786b7e90 e401da80 e401da80
    00000009 00000000 e401d550 e8eeab60 53749886 0000002c 0001b062 e401d65c
    7a01a980 e4743d40 00000000 00000246 7854ec00 00000001 ffffffff 00000001
    Call Trace:
    [<781276ea>] do_wait+0x8eb/0x9e6
    [<7812027d>] default_wake_function+0x0/0xc
    [<78127816>] sys_wait4+0x31/0x34
    [<78127840>] sys_waitpid+0x27/0x2b
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    bash R running 0 3887 3881 (NOTLB)
    unlinkd S 00000218 0 1139 3606 (NOTLB)
    c1d17e58 00000086 00000002 00000218 7a0233e0 00000002 00000000 89332b20
    00000007 00000001 89332b20 f7cd1490 861dbffc 0001e62a 00045f0d 89332c2c
    7a022980 f6e11a40 1ff0a94d 00000003 00000000 00000218 f5a60c00 f7cb5b78
    Call Trace:
    [<7816943d>] pipe_wait+0x53/0x73
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78169b7d>] pipe_read+0x2af/0x321
    [<78174b7c>] file_update_time+0x22/0x6a
    [<7816423d>] do_sync_read+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781648dd>] vfs_write+0xd1/0x10c
    [<78164176>] do_sync_read+0x0/0x10a
    [<781649a0>] vfs_read+0x88/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    pdflush S ACC9FF9C 0 944 2 (L-TLB)
    acc9ffb0 00000092 00000002 acc9ff9c acc9ff98 00000000 9c505590 9c505590
    00000007 00000001 7a1fa0b0 f7fbe030 6ce9b6b3 0001e655 0000065b 7a1fa1bc
    7a022980 f68e9340 1ff37b50 00000003 00000000 00000000 acc9ffc8 7814e33a
    Call Trace:
    [<7814e33a>] pdflush+0x0/0x1a6
    [<7814e3e8>] pdflush+0xae/0x1a6
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    amavisd S 00000000 0 6000 2823 (NOTLB)
    929dfe44 00000086 00000000 00000000 7813e7a9 00000000 00000000 f532f940
    0000000a 00000001 e8eeb590 f7e03550 a54ef8a7 0001e62a 000028bb e8eeb69c
    7a022980 d3e4a040 e8eebac0 e8eeb590 781299b4 fffffe00 7fffffff 7fffffff
    Call Trace:
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<781299b4>] local_bh_enable_ip+0xcd/0xed
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<783a0b90>] inet_csk_accept+0x9e/0x214
    [<78371564>] sock_attach_fd+0x53/0xb5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783bb524>] inet_accept+0x1f/0xa4
    [<78371564>] sock_attach_fd+0x53/0xb5
    [<7837262e>] sys_accept+0xb2/0x187
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<781332be>] __rcu_process_callbacks+0xe3/0x172
    [<781332a7>] __rcu_process_callbacks+0xcc/0x172
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<783727bc>] sys_socketcall+0xb9/0x242
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103e68>] restore_nocheck+0x12/0x15
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    squidauth.pl S 95039DC4 0 6285 3606 (NOTLB)
    95039dd8 00000082 00000002 95039dc4 95039dc0 00000000 00000000 c31076ac
    00000005 00000001 7f498aa0 f7fbe030 84d480c7 0001e4d0 00001b01 7f498bac
    7a022980 f6e117c0 1fd9ea3c 00000003 00000000 00000000 7fffffff 00000000
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78370fdd>] sock_aio_read+0xc8/0xd4
    [<7814aada>] generic_file_aio_write+0x61/0xb6
    [<7816423d>] do_sync_read+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781648dd>] vfs_write+0xd1/0x10c
    [<781649b4>] vfs_read+0x9c/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    squidauth.pl S 00000127 0 6286 3606 (NOTLB)
    bc6c7dd8 00000082 00000000 00000127 00000000 00000000 00000000 d18f63ac
    00000007 00000001 e7429490 f7e2b550 15274906 0001df92 0004a752 e742959c
    7a022980 f68e9d40 e74299c0 e7429490 00000000 e7429490 7fffffff 00000000
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
    [<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78370fdd>] sock_aio_read+0xc8/0xd4
    [<7814aada>] generic_file_aio_write+0x61/0xb6
    [<7816423d>] do_sync_read+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78165177>] __fput+0x10a/0x134
    [<781649b4>] vfs_read+0x9c/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    squidauth.pl S 00000127 0 6287 3606 (NOTLB)
    e31d5dd8 00000082 00000000 00000127 00000000 00000000 00000000 9cb76cac
    00000007 00000001 f7e2b550 7f498aa0 152ad54d 0001df92 00038c47 f7e2b65c
    7a022980 c053dac0 f7e2ba80 f7e2b550 00000000 f7e2b550 7fffffff 00000000
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
    [<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78370fdd>] sock_aio_read+0xc8/0xd4
    [<7814aada>] generic_file_aio_write+0x61/0xb6
    [<7816423d>] do_sync_read+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78165177>] __fput+0x10a/0x134
    [<781649b4>] vfs_read+0x9c/0x10a
    [<78164da8>] sys_read+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    pdflush S F6A75F9C 0 6288 2 (L-TLB)
    f6a75fb0 00000092 00000002 f6a75f9c f6a75f98 00000000 00000292 edcd0560
    0000000a 00000001 edcd0030 f7fbe030 bc332535 0001ff60 00002604 edcd013c
    7a022980 f6e31800 2198f62b 00000003 00000000 00000000 f6a75fc8 7814e33a
    Call Trace:
    [<7814e33a>] pdflush+0x0/0x1a6
    [<7814e3e8>] pdflush+0xae/0x1a6
    [<781350d5>] kthread+0x38/0x5f
    [<7813509d>] kthread+0x0/0x5f
    [<78104a8f>] kernel_thread_helper+0x7/0x10
    =======================
    amavisd S 85DD5E30 0 6350 2823 (NOTLB)
    85dd5e44 00000086 00000002 85dd5e30 85dd5e2c 00000000 00000000 f532f940
    00000007 00000001 a163eaa0 f7fbe030 bcc842a9 0001e62a 000024c3 a163ebac
    7a022980 f62ad2c0 1ff0acdf 00000003 00000000 00000000 7fffffff 7fffffff
    Call Trace:
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<783a0b90>] inet_csk_accept+0x9e/0x214
    [<78371564>] sock_attach_fd+0x53/0xb5
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<783bb524>] inet_accept+0x1f/0xa4
    [<78371564>] sock_attach_fd+0x53/0xb5
    [<7837262e>] sys_accept+0xb2/0x187
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<78152de2>] do_wp_page+0x3b9/0x406
    [<7840d1b1>] _spin_lock+0x33/0x3e
    [<781541bf>] __handle_mm_fault+0x7c3/0x83b
    [<781197d3>] do_page_fault+0x278/0x535
    [<783727bc>] sys_socketcall+0xb9/0x242
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103e68>] restore_nocheck+0x12/0x15
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    amavisd S F7E03550 0 6922 2823 (NOTLB)
    d7bdfb60 00000096 786b7ed0 f7e03550 00000001 00000046 00000000 00000000
    0000000a 00000001 f7e03550 edcd0a60 6a543cc5 0001e655 0008e72a f7e0365c
    7a022980 f6e11cc0 7813e99c 00000000 00000046 f5829698 7fffffff d7bdff9c
    Call Trace:
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<783a2023>] tcp_poll+0x1c/0x129
    [<7816ec1d>] do_select+0x399/0x3e7
    [<7816f1be>] __pollwait+0x0/0xac
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<783b1465>] tcp_v4_rcv+0x504/0x787
    [<783b133f>] tcp_v4_rcv+0x3de/0x787
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<783b1465>] tcp_v4_rcv+0x504/0x787
    [<7839949e>] ip_local_deliver+0x1aa/0x1c8
    [<78398c10>] ip_local_deliver_finish+0x0/0x14c
    [<783992bb>] ip_rcv+0x498/0x4d1
    [<7839896c>] ip_rcv_finish+0x0/0x2a4
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7837c73c>] process_backlog+0xef/0xfa
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7837c9d8>] net_rx_action+0x162/0x18f
    [<7813e986>] trace_hardirqs_on+0x10c/0x14c
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<781298c7>] local_bh_enable+0x110/0x130
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7837cbf7>] dev_queue_xmit+0x1f2/0x213
    [<7838108c>] neigh_resolve_output+0x1f2/0x224
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7816ef17>] core_sys_select+0x2ac/0x2c9
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78152de2>] do_wp_page+0x3b9/0x406
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<78152de2>] do_wp_page+0x3b9/0x406
    [<7840d1b1>] _spin_lock+0x33/0x3e
    [<781541bf>] __handle_mm_fault+0x7c3/0x83b
    [<7812938d>] _local_bh_enable+0xb1/0xbe
    [<781197d3>] do_page_fault+0x278/0x535
    [<7816f340>] sys_select+0xd6/0x188
    [<78103e68>] restore_nocheck+0x12/0x15
    [<7811955b>] do_page_fault+0x0/0x535
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    amavisd D 00000000 0 7227 2823 (NOTLB)
    d1953e2c 00000082 00000000 00000000 7813e7a9 edcd0a60 7840d6e3 7a1d2000
    00000007 00000001 edcd0a60 f7cb14d0 abd73b81 0001ff61 000032a5 edcd0b6c
    7a022980 e632d2c0 7a1d2000 7812ccc1 00000000 00000292 d1953e3c 2199064e
    Call Trace:
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812ccc1>] __mod_timer+0x92/0x9c
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812cada>] process_timeout+0x0/0x5
    [<7840b0ce>] io_schedule_timeout+0x1e/0x28
    [<78151eca>] congestion_wait+0x50/0x64
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<78152ded>] do_wp_page+0x3c4/0x406
    [<7840d1b1>] _spin_lock+0x33/0x3e
    [<781541bf>] __handle_mm_fault+0x7c3/0x83b
    [<78138305>] down_read_trylock+0x47/0x4e
    [<7811976d>] do_page_fault+0x212/0x535
    [<7811955b>] do_page_fault+0x0/0x535
    [<7840d96a>] error_code+0x72/0x78
    =======================
    amavisd S 00000002 0 7836 2823 (NOTLB)
    98947b60 00000096 00000000 00000002 7840ae59 00000000 00000000 00000000
    0000000a 00000000 ec8feb60 89333550 bca917ba 0001e62a 00001d50 ec8fec6c
    7a01a980 e8d34a80 1ff0ab63 00000003 00000000 00000000 7fffffff 98947f9c
    Call Trace:
    [<7840ae59>] __sched_text_start+0x701/0x810
    [<7840b667>] schedule_timeout+0x13/0x8d
    [<783a2023>] tcp_poll+0x1c/0x129
    [<7816ec1d>] do_select+0x399/0x3e7
    [<7816f1be>] __pollwait+0x0/0xac
    [<7812027d>] default_wake_function+0x0/0xc
    [<7812027d>] default_wake_function+0x0/0xc
    [<7813e986>] trace_hardirqs_on+0x10c/0x14c
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<781298c7>] local_bh_enable+0x110/0x130
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7837cbf7>] dev_queue_xmit+0x1f2/0x213
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<781299b4>] local_bh_enable_ip+0xcd/0xed
    [<7816ef17>] core_sys_select+0x2ac/0x2c9
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<78152de2>] do_wp_page+0x3b9/0x406
    [<7840d623>] _spin_unlock+0x25/0x3b
    [<78152de2>] do_wp_page+0x3b9/0x406
    [<7840d1b1>] _spin_lock+0x33/0x3e
    [<781541bf>] __handle_mm_fault+0x7c3/0x83b
    [<781197d3>] do_page_fault+0x278/0x535
    [<7816f340>] sys_select+0xd6/0x188
    [<78103e68>] restore_nocheck+0x12/0x15
    [<7811955b>] do_page_fault+0x0/0x535
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    sadc D 00000000 0 8202 1 (NOTLB)
    ccff9cc4 00000082 00000000 00000000 7813e7a9 edcd1490 7840d6e3 7a1d2000
    00000007 00000001 edcd1490 e9e6c0b0 abd7dbb9 0001ff61 0000216c edcd159c
    7a022980 c053d340 7a1d2000 7812ccc1 00000000 00000296 ccff9cd4 2199064e
    Call Trace:
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812ccc1>] __mod_timer+0x92/0x9c
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812cada>] process_timeout+0x0/0x5
    [<7840b0ce>] io_schedule_timeout+0x1e/0x28
    [<78151eca>] congestion_wait+0x50/0x64
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<7814a483>] generic_file_buffered_write+0x4ee/0x605
    [<78128f60>] current_fs_time+0x41/0x46
    [<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7814aad1>] generic_file_aio_write+0x58/0xb6
    [<7814cb18>] free_hot_cold_page+0x13b/0x167
    [<78199bb4>] ext3_file_write+0x24/0x92
    [<78164133>] do_sync_write+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781648dd>] vfs_write+0xd1/0x10c
    [<7816406c>] do_sync_write+0x0/0x10a
    [<78164896>] vfs_write+0x8a/0x10c
    [<78164e0f>] sys_write+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    emerge D B7BF9CB0 0 8204 3061 (NOTLB)
    b7bf9cc4 00200082 00000002 b7bf9cb0 b7bf9cac 00000000 7840d6e3 785cde80
    00000007 00000000 89333550 785143e0 abd783fe 0001ff61 00002d5e 8933365c
    7a01a980 c053d840 219905ed 00000003 00000000 00000000 b7bf9cd4 2199064e
    Call Trace:
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812cada>] process_timeout+0x0/0x5
    [<7840b0ce>] io_schedule_timeout+0x1e/0x28
    [<78151eca>] congestion_wait+0x50/0x64
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<7814a483>] generic_file_buffered_write+0x4ee/0x605
    [<78128f60>] current_fs_time+0x41/0x46
    [<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7814aad1>] generic_file_aio_write+0x58/0xb6
    [<7813f8ca>] __lock_acquire+0xaf0/0xb84
    [<7814ce3d>] get_page_from_freelist+0x293/0x35e
    [<78199bb4>] ext3_file_write+0x24/0x92
    [<78164133>] do_sync_write+0xc7/0x10a
    [<78153e4d>] __handle_mm_fault+0x451/0x83b
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<781197d3>] do_page_fault+0x278/0x535
    [<7816406c>] do_sync_write+0x0/0x10a
    [<78164896>] vfs_write+0x8a/0x10c
    [<7811955b>] do_page_fault+0x0/0x535
    [<78164e0f>] sys_write+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    mkpir D 00000000 0 8297 3061 (NOTLB)
    9c507cc4 00000082 00000000 00000000 7813e7a9 9c505590 7840d6e3 7a1d2000
    00000007 00000001 9c505590 f7cb9510 abd79872 0001ff61 00002bf0 9c50569c
    7a022980 f68e9340 7a1d2000 7812ccc1 00000000 00000296 9c507cd4 2199064e
    Call Trace:
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812ccc1>] __mod_timer+0x92/0x9c
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812cada>] process_timeout+0x0/0x5
    [<7840b0ce>] io_schedule_timeout+0x1e/0x28
    [<78151eca>] congestion_wait+0x50/0x64
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<7814a483>] generic_file_buffered_write+0x4ee/0x605
    [<781a8658>] journal_stop+0x1ce/0x1da
    [<78128f60>] current_fs_time+0x41/0x46
    [<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7814aad1>] generic_file_aio_write+0x58/0xb6
    [<78148be7>] file_read_actor+0x0/0xde
    [<78199bb4>] ext3_file_write+0x24/0x92
    [<78164133>] do_sync_write+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<78165177>] __fput+0x10a/0x134
    [<7816406c>] do_sync_write+0x0/0x10a
    [<78164896>] vfs_write+0x8a/0x10c
    [<78164e0f>] sys_write+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    screen S 8DBAFF98 0 8364 3866 (NOTLB)
    8dbaffac 00000086 00000002 8dbaff98 8dbaff94 00000000 00000000 7812850a
    00000007 00000000 f7c6c070 785143e0 455dd02b 0001ff61 00005ab9 f7c6c17c
    7a01a980 e913ad40 2198ff2e 00000003 00000000 00000000 ffffffff 00000000
    Call Trace:
    [<7812850a>] do_setitimer+0x16e/0x30b
    [<7812f315>] sys_pause+0x11/0x17
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================
    screen D 00000000 0 8365 8364 (NOTLB)
    eca0fcc4 00000082 00000000 00000000 7813e7a9 f7cb14d0 7840d6e3 7a1d2000
    00000007 00000001 f7cb14d0 9c505590 abd76c82 0001ff61 00003101 f7cb15dc
    7a022980 d3e4acc0 7a1d2000 7812ccc1 00000000 00000296 eca0fcd4 2199064e
    Call Trace:
    [<7813e7a9>] mark_held_locks+0x39/0x53
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812ccc1>] __mod_timer+0x92/0x9c
    [<7840b6c4>] schedule_timeout+0x70/0x8d
    [<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
    [<7812cada>] process_timeout+0x0/0x5
    [<7840b0ce>] io_schedule_timeout+0x1e/0x28
    [<78151eca>] congestion_wait+0x50/0x64
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
    [<7814a483>] generic_file_buffered_write+0x4ee/0x605
    [<7814a8cf>] __generic_file_aio_write_nolock+0x335/0x4df
    [<78128f60>] current_fs_time+0x41/0x46
    [<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
    [<7813e99c>] trace_hardirqs_on+0x122/0x14c
    [<7814aad1>] generic_file_aio_write+0x58/0xb6
    [<78199bb4>] ext3_file_write+0x24/0x92
    [<78164133>] do_sync_write+0xc7/0x10a
    [<7813518d>] autoremove_wake_function+0x0/0x35
    [<7840c019>] __mutex_lock_slowpath+0x25f/0x267
    [<7816406c>] do_sync_write+0x0/0x10a
    [<78164896>] vfs_write+0x8a/0x10c
    [<78164e0f>] sys_write+0x41/0x67
    [<78103d96>] sysenter_past_esp+0x5f/0x99
    =======================

    > Are any IO errors occurring at all?

    No.

    Best regards,

    Krzysztof Olędzki

  8. Re: Strange system hangs



    On Sat, 29 Sep 2007, Nick Piggin wrote:

    > On Friday 28 September 2007 18:42, Krzysztof Oledzki wrote:
    >> Hello,
    >>
    >> I am experiencing weird system hangs. Once about 2-5 weeks system freezes
    >> and stops accepting remote connections, so it is no longer possible to
    >> connect to most important services: smtp (postfix), www (squid) or even
    >> ssh. Such connection is accepted but then it hangs.
    >>
    >> What is strange, that previously established ssh session is usable. It is
    >> possible to work on such system until you do something stupid like "less
    >> /var/log/all.log". Using strace I found that process blocks on:

    >
    > Is this a regression? If so, what's the most recent kernel that didn't show
    > the problem?
    >
    > The symptoms could be consistent with some place doing a
    > balance_dirty_pages while holding a lock that is required for IO, but I can't
    > see a smoking gun (you've got contention on i_mutex, but that should be
    > OK).
    >
    > Can you see if there is any memory under writeback that isn't being
    > completed (sysrq+M), also a list the locks held after the hang might be
    > helpful (compile in lockdep and sysrq+D)
    >
    > Is anything currently running? (sysrq+P and even a full sysrq+T task list
    > could be useful).
    >
    > Are any IO errors occurring at all?


    It seems that 2.6.23.x still fails but somehow different. I updated my
    bugreport at: http://bugzilla.kernel.org/show_bug.cgi?id=9182. There are
    new attachments with traces and an oops that happened while I was taking
    the debugging data.

    Thank you.

    Best regards,


    Krzysztof Olędzki

  9. Re: [Bug 9182] Critical memory leak (dirty pages)



    On Mon, 3 Dec 2007, Thomas Osterried wrote:

    > On the machine which has troubles, the bug occured within about 10 days
    > During these days, the amount of dirty pages increased, up to 400MB.
    > I have testet kernel 2.6.19, 2.6.20, 2.6.22.1 and 2.6.22.10 (with our config),
    > and even linux-2.6.20 from ubuntu-sever. They have all shown that behaviour.




    > 10 days ago, i installed kernel 2.6.18.5 on this machine (with backported
    > 3ware controller code). I'm quite sure that this kernel will now fixes our
    > severe stability problems on this production machine (currently:
    > Dirty: 472 kB, nr_dirty 118).
    > If so, it's the "lastest" kernel i found usable, after half of a year of pain.


    Strange, my tests show that both 2.6.18(.8) and 2.6.19(.7) are OK and the
    first wrong kernel is 2.6.20.

    BTW: Could someone please look at this problem? I feel little ignored and
    in my situation this is a critical regression.

    Best regards,

    Krzysztof Olędzki

  10. Re: [Bug 9182] Critical memory leak (dirty pages)


    On Thu, 2007-12-13 at 16:17 +0100, Krzysztof Oledzki wrote:
    >


    > BTW: Could someone please look at this problem? I feel little ignored and
    > in my situation this is a critical regression.


    I was hoping to get around to it today, but I guess tomorrow will have
    to do :-/

    So, its ext3, dirty some pages, sync, and dirty doesn't fall to 0,
    right?

    Does it happen with other filesystems as well?

    What are you ext3 mount options?



    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  11. Re: [Bug 9182] Critical memory leak (dirty pages)



    On Thu, 13 Dec 2007, Peter Zijlstra wrote:

    >
    > On Thu, 2007-12-13 at 16:17 +0100, Krzysztof Oledzki wrote:
    >>

    >
    >> BTW: Could someone please look at this problem? I feel little ignored and
    >> in my situation this is a critical regression.

    >
    > I was hoping to get around to it today, but I guess tomorrow will have
    > to do :-/


    Thanks.

    > So, its ext3, dirty some pages, sync, and dirty doesn't fall to 0,
    > right?


    Not only doesn't fall but continuously grows.

    > Does it happen with other filesystems as well?


    Don't know. I generally only use ext3 and I'm afraid I'm not able to
    switch this system to other filesystem.

    > What are you ext3 mount options?

    /dev/root / ext3 rw,data=journal 0 0
    /dev/VolGrp0/usr /usr ext3 rw,nodev,data=journal 0 0
    /dev/VolGrp0/var /var ext3 rw,nodev,data=journal 0 0
    /dev/VolGrp0/squid_spool /var/cache/squid/cd0 ext3 rw,nosuid,nodev,noatime,data=writeback 0 0
    /dev/VolGrp0/squid_spool2 /var/cache/squid/cd1 ext3 rw,nosuid,nodev,noatime,data=writeback 0 0
    /dev/VolGrp0/news_spool /var/spool/news ext3 rw,nosuid,nodev,noatime,data=ordered 0 0

    Best regards,

    Krzysztof Olędzki

  12. Re: [Bug 9182] Critical memory leak (dirty pages)



    On Thu, 13 Dec 2007, Krzysztof Oledzki wrote:

    >
    >
    > On Thu, 13 Dec 2007, Peter Zijlstra wrote:
    >
    >>
    >> On Thu, 2007-12-13 at 16:17 +0100, Krzysztof Oledzki wrote:
    >>>

    >>
    >>> BTW: Could someone please look at this problem? I feel little ignored and
    >>> in my situation this is a critical regression.

    >>
    >> I was hoping to get around to it today, but I guess tomorrow will have
    >> to do :-/

    >
    > Thanks.
    >
    >> So, its ext3, dirty some pages, sync, and dirty doesn't fall to 0,
    >> right?

    >
    > Not only doesn't fall but continuously grows.
    >
    >> Does it happen with other filesystems as well?

    >
    > Don't know. I generally only use ext3 and I'm afraid I'm not able to switch
    > this system to other filesystem.
    >
    >> What are you ext3 mount options?

    > /dev/root / ext3 rw,data=journal 0 0
    > /dev/VolGrp0/usr /usr ext3 rw,nodev,data=journal 0 0
    > /dev/VolGrp0/var /var ext3 rw,nodev,data=journal 0 0
    > /dev/VolGrp0/squid_spool /var/cache/squid/cd0 ext3
    > rw,nosuid,nodev,noatime,data=writeback 0 0
    > /dev/VolGrp0/squid_spool2 /var/cache/squid/cd1 ext3
    > rw,nosuid,nodev,noatime,data=writeback 0 0
    > /dev/VolGrp0/news_spool /var/spool/news ext3
    > rw,nosuid,nodev,noatime,data=ordered 0 0


    BTW: this regression also exists in 2.6.24-rc5. I'll try to find when it
    was introduced but it is hard to do it on a highly critical production
    system, especially since it takes ~2h after a reboot, to be sure.

    However, 2h is quite good time, on other systems I have to wait ~2 months
    to get 20MB of leaked memory:

    # uptime
    13:29:34 up 58 days, 13:04, 9 users, load average: 0.38, 0.27, 0.31

    # sync;sync;sleep 1;sync;grep Dirt /proc/meminfo
    Dirty: 23820 kB

    Best regards,

    Krzysztof Olędzki

  13. Re: [Bug 9182] Critical memory leak (dirty pages)


    http://bugzilla.kernel.org/show_bug.cgi?id=9182


    On Sat, 15 Dec 2007, Krzysztof Oledzki wrote:

    >
    >
    > On Thu, 13 Dec 2007, Krzysztof Oledzki wrote:
    >
    >>
    >>
    >> On Thu, 13 Dec 2007, Peter Zijlstra wrote:
    >>
    >>>
    >>> On Thu, 2007-12-13 at 16:17 +0100, Krzysztof Oledzki wrote:
    >>>>
    >>>
    >>>> BTW: Could someone please look at this problem? I feel little ignored and
    >>>> in my situation this is a critical regression.
    >>>
    >>> I was hoping to get around to it today, but I guess tomorrow will have
    >>> to do :-/

    >>
    >> Thanks.
    >>
    >>> So, its ext3, dirty some pages, sync, and dirty doesn't fall to 0,
    >>> right?

    >>
    >> Not only doesn't fall but continuously grows.
    >>
    >>> Does it happen with other filesystems as well?

    >>
    >> Don't know. I generally only use ext3 and I'm afraid I'm not able to switch
    >> this system to other filesystem.
    >>
    >>> What are you ext3 mount options?

    >> /dev/root / ext3 rw,data=journal 0 0
    >> /dev/VolGrp0/usr /usr ext3 rw,nodev,data=journal 0 0
    >> /dev/VolGrp0/var /var ext3 rw,nodev,data=journal 0 0
    >> /dev/VolGrp0/squid_spool /var/cache/squid/cd0 ext3
    >> rw,nosuid,nodev,noatime,data=writeback 0 0
    >> /dev/VolGrp0/squid_spool2 /var/cache/squid/cd1 ext3
    >> rw,nosuid,nodev,noatime,data=writeback 0 0
    >> /dev/VolGrp0/news_spool /var/spool/news ext3
    >> rw,nosuid,nodev,noatime,data=ordered 0 0

    >
    > BTW: this regression also exists in 2.6.24-rc5. I'll try to find when it was
    > introduced but it is hard to do it on a highly critical production system,
    > especially since it takes ~2h after a reboot, to be sure.
    >
    > However, 2h is quite good time, on other systems I have to wait ~2 monthsto
    > get 20MB of leaked memory:
    >
    > # uptime
    > 13:29:34 up 58 days, 13:04, 9 users, load average: 0.38, 0.27, 0.31
    >
    > # sync;sync;sleep 1;sync;grep Dirt /proc/meminfo
    > Dirty: 23820 kB


    More news, I hope this time my problem get more attention from developers
    since now I have much more information.

    So far I found that:
    - 2.6.20-rc4 - bad: http://bugzilla.kernel.org/attachment.cgi?id=14057
    - 2.6.20-rc2 - bad: http://bugzilla.kernel.org/attachment.cgi?id=14058
    - 2.6.20-rc1 - OK (probably, I need to wait little more to be 100% sure).

    2.6.20-rc1 with 33m uptime:
    ~$ grep Dirt /proc/meminfo ;sync ; sleep 1 ; sync ; grep Dirt /proc/meminfo
    Dirty: 10504 kB
    Dirty: 0 kB

    2.6.20-rc2 was released Dec 23/24 2006 (BAD)
    2.6.20-rc1 was released Dec 13/14 2006 (GOOD?)

    It seems that this bug was introduced exactly one year ago. Surprisingly,
    dirty memory in 2.6.20-rc2/2.6.20-rc4 leaks _much_ more faster than in
    2.6.20-final and later kernels as it took only about 6h to reach 172MB.
    So, this bug might be cured afterward, but only a little.

    There are three commits that may be somehow related:

    http://git.kernel.org/?p=linux/kerne...9dd8fe40ae7bd9
    http://git.kernel.org/?p=linux/kerne...153dca9a43b11d
    http://git.kernel.org/?p=linux/kerne...38434f9c25aed2

    I'm going to check 2.6.20-rc1-git... releases but it would be *very* nice
    if someone could finally give ma a hand and point some hints helping
    debugging this problem.

    Please note that none of my systems with kernels >= 2.6.20-rc1 is able to
    reach 0 kb of dirty memory, even after many synces, even when idle.

    Best regards,

    Krzysztof Olędzki

+ Reply to Thread