[PATCH] x86_64: fix delayed signals - Kernel

This is a discussion on [PATCH] x86_64: fix delayed signals - Kernel ; On Fri, 11 Jul 2008, Linus Torvalds wrote: > > Btw, did any of the impacted people test -rc9? Edwin's report is about > -rc2 and -rc8, and one of the things we fixed since -rc8 is that incorrect > ...

+ Reply to Thread
Page 2 of 3 FirstFirst 1 2 3 LastLast
Results 21 to 40 of 45

Thread: [PATCH] x86_64: fix delayed signals

  1. Re: [PATCH] x86_64: fix delayed signals



    On Fri, 11 Jul 2008, Linus Torvalds wrote:
    >
    > Btw, did any of the impacted people test -rc9? Edwin's report is about
    > -rc2 and -rc8, and one of the things we fixed since -rc8 is that incorrect
    > and unintentional nr_zones zeroing that effectively disabled kswapd - and
    > made everybody do synchronous memory freeing when they wanted to allocate
    > more memory.. That can play havoc with any interactive stuff.


    No, after testing more, I think Edwin is right.

    There _does_ seem to be something wrong in signal handling when the signal
    happens during 'D' state (although I guess it could also be an artifact of
    other scheduling activity). I just put my machine under heavy read load by
    having a background process that does

    while : ; do echo 3 > /proc/sys/vm/drop_caches ; sleep 5; done

    to flush the caches all the time, and then doing endless loops of "git
    grep" and other things, and I can definitely re-create the bad latency.

    The best example is doing just

    [torvalds@woody ~]$ ls -l /usr/bin
    ^Ctotal 367720
    -rwxr-xr-x 1 root root 43648 2008-07-04 09:35 [
    [torvalds@woody ~]$

    and there are several seconds between the '^C' and the printout of the two
    lines (after which the 'ls' finally dies).

    Perhaps more interestingly: when this happens (it doesn't happen all the
    time), it will _always_ die at the first two lines of printout for me, on
    two different machines: ie I *always* get that exact pattern of those two
    lines printed out (except when it reacts immediately, or when I press ^C
    so late that it has printed out much more).

    And it's not some single system call latency that is long. I checked with
    strace, and while the IO load is high and causes the whole 'ls' to take a
    long time (up to 10 seconds - /usr/bin is *big*), each individual system
    call never takes very long. The longest system call latency I saw was 0.24
    seconds, which was for some very unlucky lstat() calls. Most of them were
    in the millisecond range, even though they obviously had to do IO.

    More tellingly, those two writes are simply not even done with a single
    system call. The 'strace' for an 'ls -l' shows

    ...
    write(1, "total 367720\n", 13) = 13 <0.000085>
    open("/etc/localtime", O_RDONLY) = 3 <0.000016>
    fstat(3, {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0 <0.000004>
    fstat(3, {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0 <0.000004>
    mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6d996dd000 <0.000006>
    read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\4\0\0\0\ 4\0\0\0\0"..., 4096) = 2819 <0.041416>
    lseek(3, -1802, SEEK_CUR) = 1017 <0.000005>
    read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0\ 5\0\0\0\0"..., 4096) = 1802 <0.000006>
    close(3) = 0 <0.000006>
    munmap(0x7f6d996dd000, 4096) = 0 <0.000014>
    write(1, "-rwxr-xr-x 1 root root 436"..., 54) = 54 <0.000008>
    ...

    so there is actually *many* system calls in there between the header line
    that does the 'total' and the first line of actual real output, but
    despite that, it reacts to the ^C consistently after the second line.

    Very odd.

    And the problem definitely happens on x86-32 too.

    It also happens on the text console, so this is not some bad interaction
    with X and pty's and xterm or gnome-terminal or something like that. At
    first I thought "no way can that actually happen, I think the terminal is
    broken and only sends the signal after it has seen output", but no, it
    happens for me in text-mode too.

    Oleg added to the Cc as the master of signal handling code.

    Linus
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  2. Re: [PATCH] x86_64: fix delayed signals

    On Fri, 11 Jul 2008 11:31:26 -0700 (PDT)
    Linus Torvalds wrote:

    >
    >
    > On Fri, 11 Jul 2008, Linus Torvalds wrote:
    > >
    > > Btw, did any of the impacted people test -rc9? Edwin's report is
    > > about -rc2 and -rc8, and one of the things we fixed since -rc8 is
    > > that incorrect and unintentional nr_zones zeroing that effectively
    > > disabled kswapd - and made everybody do synchronous memory freeing
    > > when they wanted to allocate more memory.. That can play havoc with
    > > any interactive stuff.

    >
    > Hmm. Edwin's latencytop output includes this (ignoring the _very_ top
    > entries that are all either CD-ROM media change tests or are
    > interruptible pipe/select things) at the top:
    >
    > 21 10264428 915514 get_request_wait __make_request
    > generic_make_request submit_bio xfs_submit_ioend_bio xfs_submit_ioend
    > xfs_page_state_convert xfs_vm_writepage __writepage
    > write_cache_pages generic_writepages xfs_vm_writepages



    argh. well I guess this might be useful for this case, but normally
    latencytop gives you much more humanly readable data... maybe Edwin
    forgot to do "make install" :-(
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  3. Re: [PATCH] x86_64: fix delayed signals



    On Fri, 11 Jul 2008, Linus Torvalds wrote:
    >
    > No, after testing more, I think Edwin is right.


    Naah. It is true that there is a huge delay in doing

    ls -l /usr/bin

    and then pressing ^C, but after having hit my head against this for a
    while, I realized that it has nothing to do with the kernel.

    Doing an "strace ls" showed that ls doesn't play any games with signals
    etc, which fooled me into looking for a kernel reason.

    HOWEVER, it looks like at least fedora does a

    alias ls='ls --color=auto'

    and it turns out that if you do that --color=auto, then ls will indeed
    catch all normal signals and set a "please stop now" flag, instead of
    dying immediately. The reason is probably to avoiding leaving the terminal
    with some odd color if interrupted in an inconvenient place.

    So I was chasing this latency thing totally unnecessarily. It's in user
    space (or at least _my_ particular issue was).

    Other user cases may obviously be elsewhere.

    Linus
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  4. Re: [PATCH] x86_64: fix delayed signals

    On 2008-07-12 02:22, Linus Torvalds wrote:
    > On Fri, 11 Jul 2008, Linus Torvalds wrote:
    >
    >> No, after testing more, I think Edwin is right.
    >>

    >
    > Naah. It is true that there is a huge delay in doing
    >
    > ls -l /usr/bin
    >
    > and then pressing ^C, but after having hit my head against this for a
    > while, I realized that it has nothing to do with the kernel.
    >
    > Doing an "strace ls" showed that ls doesn't play any games with signals
    > etc, which fooled me into looking for a kernel reason.
    >
    > HOWEVER, it looks like at least fedora does a
    >
    > alias ls='ls --color=auto'
    >
    > and it turns out that if you do that --color=auto, then ls will indeed
    > catch all normal signals and set a "please stop now" flag, instead of
    > dying immediately. The reason is probably to avoiding leaving the terminal
    > with some odd color if interrupted in an inconvenient place.
    >
    > So I was chasing this latency thing totally unnecessarily. It's in user
    > space (or at least _my_ particular issue was).
    >
    > Other user cases may obviously be elsewhere.


    Thanks for looking into the problem.
    I did some more testing, and here are my results so far (with my
    testcase of running dd and find):

    On my 32-bit box (slow disks, SMP, XFS filesystem) 2.6.26-rc9 behaves
    the same as 2.6.26-rc8, I can reliably reproduce a 2-3 second latency
    [1] between pressing ^C the first time, and the shell returning (on the
    text console too).
    Using ftrace available from tip/master, I see up to 3 seconds of delay
    between kill_pgrp and detach_pid (and during that time I can press ^C
    again, leading to 2-3 kill_pgrp calls)

    On my 64-bit box (2 disks in raid-0, UP, reiserfs filesystem) 2.6.25 and
    2.6.26-rc9 behave the same, and most of the time (10-20 times in a row)
    find responds to ^C instantly.
    However in _some_ cases find doesn't respond to ^C for a very long time
    (~30 seconds), and when this happens I can't do anything else but switch
    consoles,
    starting another process (latencytop -d) hangs, and so does any other
    external command.
    I have tried to get a latencytop output from the 64-bit box, but I
    couldn't reproduce the long latency when latencytop was running.
    I have reproduced the latency after launching/killing find 20+ times,
    but after I started latencytop I couldn't reproduce.
    But I can no longer reproduce it now, even if I stop latencytop (and
    latencytop sysctl _is_ 0).

    I haven't yet tried ftrace on this box, and neither did I try Roland's
    patch yet. I will try that now, and hopefuly come back with some numbers
    shortly.


    [1]
    I switch to text console, then I run this script in the background:

    cd /debug/tracing
    echo kill_pgrp detach_pid\
    >set_ftrace_filter

    echo ftrace >current_tracer
    echo 1 >tracing_enabled
    sleep 60
    echo 0 >tracing_enabled
    cat trace >/tmp/trace.out
    cat latency_trace >/tmp/latencytrace.out

    Then I run this script in the background, and wait until in shows
    'Waiting for dd to finish':
    dd if=/dev/zero of=xt bs=100M count=4&
    dd if=/dev/zero of=yt bs=100M count=4&
    rm xt
    rm yt
    echo "Waiting for dd to finish..."
    wait %1 %2
    echo "Done"

    Then I start 'find / >/dev/null 2>/dev/null, and type ^C after about a
    second'.
    trace and latencytrace outputs below (for the 32-bit box):

    # tracer: ftrace
    #
    # TASK-PID CPU# TIMESTAMP FUNCTION
    # | | | | |
    events/0-9 [00] 450.146629: kill_pgrp <-n_tty_receive_buf
    bash-5821 [00] 450.386653: detach_pid <-release_task
    events/0-9 [00] 451.631327: kill_pgrp <-n_tty_receive_buf
    events/0-9 [00] 452.394820: kill_pgrp <-n_tty_receive_buf
    events/0-9 [00] 453.127932: kill_pgrp <-n_tty_receive_buf
    bash-5821 [00] 453.143686: detach_pid <-release_task
    bash-5821 [00] 453.143686: detach_pid <-release_task
    bash-5821 [00] 453.143686: detach_pid <-release_task
    ^^^^^^^^^^^^ I had to press ^C 3 times here, and there is a ~2 sec delay

    events/0-9 [00] 454.164452: kill_pgrp <-n_tty_receive_buf
    events/0-9 [00] 455.027774: kill_pgrp <-n_tty_receive_buf
    events/0-9 [00] 455.762679: kill_pgrp <-n_tty_receive_buf
    bash-5821 [00] 455.770332: detach_pid <-release_task
    bash-5821 [00] 455.770332: detach_pid <-release_task
    bash-5821 [00] 455.770332: detach_pid <-release_task
    ^^^^^^^ again, a 1 second delay
    events/0-9 [00] 456.718620: kill_pgrp <-n_tty_receive_buf
    bash-5821 [00] 456.862658: detach_pid <-release_task
    bash-5821 [00] 456.862658: detach_pid <-release_task
    bash-5821 [00] 456.862658: detach_pid <-release_task
    events/0-9 [00] 457.880204: kill_pgrp <-n_tty_receive_buf
    bash-5821 [00] 457.908467: detach_pid <-release_task
    bash-5821 [00] 457.908467: detach_pid <-release_task
    bash-5821 [00] 457.908467: detach_pid <-release_task
    events/0-9 [00] 459.238903: kill_pgrp <-n_tty_receive_buf
    bash-5821 [00] 460.081542: detach_pid <-release_task
    bash-5821 [00] 460.081542: detach_pid <-release_task
    bash-5821 [00] 460.081542: detach_pid <-release_task
    events/0-9 [00] 460.083701: kill_pgrp <-n_tty_receive_buf
    events/0-9 [00] 462.246407: kill_pgrp <-n_tty_receive_buf
    events/0-9 [00] 462.975581: kill_pgrp <-n_tty_receive_buf
    bash-5821 [00] 463.583139: detach_pid <-release_task
    ^^^^^^^^^^^^^^^^^^^ 3.5 second delay here
    bash-5821 [00] 463.583139: detach_pid <-release_task
    bash-5821 [00] 463.583139: detach_pid <-release_task
    events/0-9 [00] 463.699181: kill_pgrp <-n_tty_receive_buf
    events/0-9 [00] 464.735022: kill_pgrp <-n_tty_receive_buf
    bash-5821 [00] 465.094654: detach_pid <-release_task
    bash-5821 [00] 465.094654: detach_pid <-release_task
    bash-5821 [00] 465.094654: detach_pid <-release_task
    events/0-9 [00] 466.537663: kill_pgrp <-n_tty_receive_buf
    bash-5821 [00] 466.945909: detach_pid <-release_task

    # tracer: ftrace
    #
    ftrace latency trace v1.1.5 on 2.6.26-rc9-tip-01669-g639a570
    --------------------------------------------------------------------
    latency: 0 us, #210/210, CPU#0 | (M:unknown VP:0, KP:0, SP:0 HP:0 #P:2)
    -----------------
    | task: -0 (uid:0 nice:0 policy:0 rt_prio:0)
    -----------------

    # _------=> CPU#
    # / _-----=> irqs-off
    # | / _----=> need-resched
    # || / _---=> hardirq/softirq
    # ||| / _--=> preempt-depth
    # |||| /
    # ||||| delay
    # cmd pid ||||| time | caller
    # \ / ||||| \ | /
    kstopmac-6225 0d..3 10us : detach_pid (release_task)
    kstopmac-6225 0d..3 10us : detach_pid (release_task)
    kstopmac-6225 0d..3 10us : detach_pid (release_task)
    init-1 0d..3 10us : detach_pid (release_task)
    init-1 0d..3 10us : detach_pid (release_task)
    init-1 0d..3 10us!: detach_pid (release_task)
    sh-6228 1d..3 3944272us+: detach_pid (release_task)
    sh-6228 1d..3 3944273us : detach_pid (release_task)
    sh-6228 1d..3 3944274us!: detach_pid (release_task)
    sh-6228 1d..3 3945744us+: detach_pid (release_task)
    sh-6228 1d..3 3945745us : detach_pid (release_task)
    sh-6228 1d..3 3945746us!: detach_pid (release_task)
    hal-syst-6233 0d..3 4156674us : detach_pid (release_task)
    hal-syst-6233 0d..3 4156674us : detach_pid (release_task)
    hal-syst-6233 0d..3 4156674us!: detach_pid (release_task)
    hald-run-5416 0d..3 4160008us : detach_pid (release_task)
    hald-run-5416 0d..3 4160008us : detach_pid (release_task)
    hald-run-5416 0d..3 4160008us!: detach_pid (release_task)
    events/0-9 0.... 9523335us!: kill_pgrp (n_tty_receive_buf)
    bash-5821 0d..3 9763359us : detach_pid (release_task)
    bash-5821 0d..3 9763359us : detach_pid (release_task)
    bash-5821 0d..3 9763359us!: detach_pid (release_task)
    hal-syst-6236 0d..3 10167777us : detach_pid (release_task)
    hal-syst-6236 0d..3 10167779us : detach_pid (release_task)
    hal-syst-6236 0d..3 10167779us!: detach_pid (release_task)
    hald-run-5416 0d..3 10172764us : detach_pid (release_task)
    hald-run-5416 0d..3 10172765us : detach_pid (release_task)
    hald-run-5416 0d..3 10172765us!: detach_pid (release_task)
    events/0-9 0.... 11008033us!: kill_pgrp (n_tty_receive_buf)
    events/0-9 0.... 11771525us!: kill_pgrp (n_tty_receive_buf)
    events/0-9 0.... 12504637us!: kill_pgrp (n_tty_receive_buf)
    bash-5821 0d..3 12520392us : detach_pid (release_task)
    bash-5821 0d..3 12520392us : detach_pid (release_task)
    bash-5821 0d..3 12520392us!: detach_pid (release_task)
    events/0-9 0.... 13541157us!: kill_pgrp (n_tty_receive_buf)
    events/0-9 0.... 14404479us!: kill_pgrp (n_tty_receive_buf)
    events/0-9 0.... 15139385us!: kill_pgrp (n_tty_receive_buf)
    bash-5821 0d..3 15147037us : detach_pid (release_task)
    bash-5821 0d..3 15147037us : detach_pid (release_task)
    bash-5821 0d..3 15147037us!: detach_pid (release_task)
    events/0-9 0.... 16095325us!: kill_pgrp (n_tty_receive_buf)
    hal-syst-6242 0d..3 16180193us : detach_pid (release_task)
    hal-syst-6242 0d..3 16180193us : detach_pid (release_task)
    hal-syst-6242 0d..3 16180193us!: detach_pid (release_task)
    hald-run-5416 0d..3 16183420us : detach_pid (release_task)
    hald-run-5416 0d..3 16183420us : detach_pid (release_task)
    hald-run-5416 0d..3 16183420us!: detach_pid (release_task)
    bash-5821 0d..3 16239363us : detach_pid (release_task)
    bash-5821 0d..3 16239363us : detach_pid (release_task)
    bash-5821 0d..3 16239363us!: detach_pid (release_task)
    events/0-9 0.... 17256910us!: kill_pgrp (n_tty_receive_buf)
    bash-5821 0d..3 17285172us : detach_pid (release_task)
    bash-5821 0d..3 17285172us : detach_pid (release_task)
    bash-5821 0d..3 17285172us!: detach_pid (release_task)
    events/0-9 0.... 18615608us!: kill_pgrp (n_tty_receive_buf)
    bash-5821 0d..3 19458248us : detach_pid (release_task)
    bash-5821 0d..3 19458248us : detach_pid (release_task)
    bash-5821 0d..3 19458248us!: detach_pid (release_task)
    events/0-9 0.... 19460406us!: kill_pgrp (n_tty_receive_buf)
    events/0-9 0.... 21623112us!: kill_pgrp (n_tty_receive_buf)
    hal-syst-6247 0d..3 22194451us : detach_pid (release_task)
    hal-syst-6247 0d..3 22194453us : detach_pid (release_task)
    hal-syst-6247 0d..3 22194453us!: detach_pid (release_task)
    hald-run-5416 0d..3 22199518us : detach_pid (release_task)
    hald-run-5416 0d..3 22199520us : detach_pid (release_task)
    hald-run-5416 0d..3 22199520us!: detach_pid (release_task)
    events/0-9 0.... 22352287us!: kill_pgrp (n_tty_receive_buf)
    bash-5821 0d..3 22959844us : detach_pid (release_task)
    bash-5821 0d..3 22959844us : detach_pid (release_task)
    bash-5821 0d..3 22959844us!: detach_pid (release_task)
    events/0-9 0.... 23075886us!: kill_pgrp (n_tty_receive_buf)
    events/0-9 0.... 24111727us!: kill_pgrp (n_tty_receive_buf)
    bash-5821 0d..3 24471359us : detach_pid (release_task)
    bash-5821 0d..3 24471359us : detach_pid (release_task)
    bash-5821 0d..3 24471359us!: detach_pid (release_task)
    events/0-9 0.... 25914368us!: kill_pgrp (n_tty_receive_buf)
    bash-5821 0d..3 26322615us : detach_pid (release_task)
    bash-5821 0d..3 26322615us : detach_pid (release_task)
    bash-5821 0d..3 26322615us!: detach_pid (release_task)
    hal-syst-6252 0d..3 28207618us : detach_pid (release_task)
    hal-syst-6252 0d..3 28207619us : detach_pid (release_task)
    hal-syst-6252 0d..3 28207620us!: detach_pid (release_task)
    hald-run-5416 0d..3 28212746us : detach_pid (release_task)
    hald-run-5416 0d..3 28212747us : detach_pid (release_task)
    hald-run-5416 0d..3 28212748us!: detach_pid (release_task)
    events/0-9 0.... 28473354us!: kill_pgrp (n_tty_receive_buf)
    events/0-9 0.... 29208378us!: kill_pgrp (n_tty_receive_buf)
    pdflush-236 1d..3 29463682us+: detach_pid (release_task)
    pdflush-236 1d..3 29463684us : detach_pid (release_task)
    pdflush-236 1d..3 29463685us!: detach_pid (release_task)
    bash-5821 0d..3 29633203us : detach_pid (release_task)
    bash-5821 0d..3 29633203us : detach_pid (release_task)
    bash-5821 0d..3 29633203us!: detach_pid (release_task)
    events/0-9 0.... 31177193us!: kill_pgrp (n_tty_receive_buf)
    events/0-9 0.... 32081149us!: kill_pgrp (n_tty_receive_buf)
    bash-5821 0d..3 32096726us : detach_pid (release_task)
    bash-5821 0d..3 32096726us : detach_pid (release_task)
    bash-5821 0d..3 32096726us!: detach_pid (release_task)
    hal-syst-6255 0d..3 34221092us : detach_pid (release_task)
    hal-syst-6255 0d..3 34221094us : detach_pid (release_task)
    hal-syst-6255 0d..3 34221095us!: detach_pid (release_task)
    hald-run-5416 0d..3 34226008us+: detach_pid (release_task)
    hald-run-5416 0d..3 34226009us : detach_pid (release_task)
    hald-run-5416 0d..3 34226010us!: detach_pid (release_task)
    events/0-9 0.... 34943377us!: kill_pgrp (n_tty_receive_buf)
    bash-5821 0d..3 34953441us : detach_pid (release_task)
    bash-5821 0d..3 34953441us : detach_pid (release_task)
    bash-5821 0d..3 34953441us!: detach_pid (release_task)
    sh-6228 1d..3 38434166us : detach_pid (release_task)
    sh-6228 1d..3 38434166us : detach_pid (release_task)
    sh-6228 1d..3 38434166us!: detach_pid (release_task)
    sh-6228 1d..3 38740305us+: detach_pid (release_task)
    sh-6228 1d..3 38740307us : detach_pid (release_task)
    sh-6228 1d..3 38740308us!: detach_pid (release_task)
    bash-5821 0d..3 38744188us : detach_pid (release_task)
    bash-5821 0d..3 38744188us : detach_pid (release_task)
    bash-5821 0d..3 38744188us!: detach_pid (release_task)
    bash-5821 0d..3 40161326us : detach_pid (release_task)
    bash-5821 0d..3 40161327us : detach_pid (release_task)
    bash-5821 0d..3 40161328us!: detach_pid (release_task)
    hal-syst-6259 0d..3 40233791us : detach_pid (release_task)
    hal-syst-6259 0d..3 40233792us : detach_pid (release_task)
    hal-syst-6259 0d..3 40233793us!: detach_pid (release_task)
    hald-run-5416 0d..3 40238677us : detach_pid (release_task)
    hald-run-5416 0d..3 40238678us+: detach_pid (release_task)
    hald-run-5416 0d..3 40238680us!: detach_pid (release_task)
    bash-5821 1d..3 43913077us : detach_pid (release_task)
    bash-5821 1d..3 43913077us : detach_pid (release_task)
    bash-5821 1d..3 43913077us!: detach_pid (release_task)
    bash-5821 0d..3 44968815us : detach_pid (release_task)
    bash-5821 0d..3 44968815us : detach_pid (release_task)
    bash-5821 0d..3 44968815us!: detach_pid (release_task)
    hal-syst-6263 0d..3 46247842us : detach_pid (release_task)
    hal-syst-6263 0d..3 46247844us : detach_pid (release_task)
    hal-syst-6263 0d..3 46247845us!: detach_pid (release_task)
    hald-run-5416 0d..3 46252860us : detach_pid (release_task)
    hald-run-5416 0d..3 46252861us : detach_pid (release_task)
    hald-run-5416 0d..3 46252862us!: detach_pid (release_task)
    nullmail-5260 1d..3 46591091us : detach_pid (release_task)
    nullmail-5260 1d..3 46591092us : detach_pid (release_task)
    nullmail-5260 1d..3 46591093us!: detach_pid (release_task)
    nullmail-5260 1d..3 46594439us : detach_pid (release_task)
    nullmail-5260 1d..3 46594439us : detach_pid (release_task)
    nullmail-5260 1d..3 46594439us!: detach_pid (release_task)
    nullmail-5260 1d..3 46596485us : detach_pid (release_task)
    nullmail-5260 1d..3 46596486us : detach_pid (release_task)
    nullmail-5260 1d..3 46596487us!: detach_pid (release_task)
    nullmail-5260 1d..3 46599932us : detach_pid (release_task)
    nullmail-5260 1d..3 46599933us : detach_pid (release_task)
    nullmail-5260 1d..3 46599934us!: detach_pid (release_task)
    nullmail-5260 0d..3 46627792us : detach_pid (release_task)
    nullmail-5260 0d..3 46627792us : detach_pid (release_task)
    nullmail-5260 0d..3 46627792us!: detach_pid (release_task)
    nullmail-5260 0d..3 46636851us : detach_pid (release_task)
    nullmail-5260 0d..3 46636851us : detach_pid (release_task)
    nullmail-5260 0d..3 46636851us!: detach_pid (release_task)
    nullmail-5260 0d..3 46647738us : detach_pid (release_task)
    nullmail-5260 0d..3 46647740us : detach_pid (release_task)
    nullmail-5260 0d..3 46647740us!: detach_pid (release_task)
    nullmail-5260 0d..3 46656857us : detach_pid (release_task)
    nullmail-5260 0d..3 46656857us : detach_pid (release_task)
    nullmail-5260 0d..3 46656857us!: detach_pid (release_task)
    nullmail-5260 0d..3 46666791us : detach_pid (release_task)
    nullmail-5260 0d..3 46666791us : detach_pid (release_task)
    nullmail-5260 0d..3 46666791us!: detach_pid (release_task)
    nullmail-5260 0d..3 46680090us : detach_pid (release_task)
    nullmail-5260 0d..3 46680090us : detach_pid (release_task)
    nullmail-5260 0d..3 46680090us!: detach_pid (release_task)
    nullmail-5260 0d..3 46683330us : detach_pid (release_task)
    nullmail-5260 0d..3 46683330us : detach_pid (release_task)
    nullmail-5260 0d..3 46683330us!: detach_pid (release_task)
    nullmail-5260 1d..3 46684182us : detach_pid (release_task)
    nullmail-5260 1d..3 46684182us : detach_pid (release_task)
    nullmail-5260 1d..3 46684182us!: detach_pid (release_task)
    nullmail-5260 1d..3 46687500us : detach_pid (release_task)
    nullmail-5260 1d..3 46687500us : detach_pid (release_task)
    nullmail-5260 1d..3 46687500us!: detach_pid (release_task)
    nullmail-5260 1d..3 46692670us : detach_pid (release_task)
    nullmail-5260 1d..3 46692670us : detach_pid (release_task)
    nullmail-5260 1d..3 46692670us!: detach_pid (release_task)
    bash-5821 1d..3 47327646us : detach_pid (release_task)
    bash-5821 1d..3 47327647us : detach_pid (release_task)
    bash-5821 1d..3 47327648us!: detach_pid (release_task)
    bash-6280 1d..3 47331181us : detach_pid (release_task)
    bash-6280 1d..3 47331181us : detach_pid (release_task)
    bash-6280 1d..3 47331181us!: detach_pid (release_task)
    bash-5821 1d..3 47331699us : detach_pid (release_task)
    bash-5821 1d..3 47331700us : detach_pid (release_task)
    bash-5821 1d..3 47331701us!: detach_pid (release_task)
    bash-6282 1d..3 47334166us : detach_pid (release_task)
    bash-6282 1d..3 47334166us : detach_pid (release_task)
    bash-6282 1d..3 47334166us : detach_pid (release_task)
    bash-5821 1d..3 47334166us : detach_pid (release_task)
    bash-5821 1d..3 47334166us : detach_pid (release_task)
    bash-5821 1d..3 47334166us!: detach_pid (release_task)
    hal-syst-6284 0d..3 52261078us : detach_pid (release_task)
    hal-syst-6284 0d..3 52261079us : detach_pid (release_task)
    hal-syst-6284 0d..3 52261080us!: detach_pid (release_task)
    hald-run-5416 0d..3 52266033us+: detach_pid (release_task)
    hald-run-5416 0d..3 52266034us : detach_pid (release_task)
    hald-run-5416 0d..3 52266035us!: detach_pid (release_task)
    hal-syst-6286 0d..3 58273784us : detach_pid (release_task)
    hal-syst-6286 0d..3 58273786us : detach_pid (release_task)
    hal-syst-6286 0d..3 58273786us!: detach_pid (release_task)
    hald-run-5416 0d..3 58278681us : detach_pid (release_task)
    hald-run-5416 0d..3 58278682us : detach_pid (release_task)
    hald-run-5416 0d..3 58278683us!: detach_pid (release_task)
    sh-6224 0d..3 60000916us+: detach_pid (release_task)
    sh-6224 0d..3 60000918us : detach_pid (release_task)
    sh-6224 0d..3 60000919us : detach_pid (release_task)


    vim:ft=help

    Best regards,
    --Edwin
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  5. Re: [PATCH] x86_64: fix delayed signals

    On 2008-07-12 01:53, Arjan van de Ven wrote:
    > On Fri, 11 Jul 2008 11:31:26 -0700 (PDT)
    > Linus Torvalds wrote:
    >
    >
    >> On Fri, 11 Jul 2008, Linus Torvalds wrote:
    >>
    >>> Btw, did any of the impacted people test -rc9? Edwin's report is
    >>> about -rc2 and -rc8, and one of the things we fixed since -rc8 is
    >>> that incorrect and unintentional nr_zones zeroing that effectively
    >>> disabled kswapd - and made everybody do synchronous memory freeing
    >>> when they wanted to allocate more memory.. That can play havoc with
    >>> any interactive stuff.
    >>>

    >> Hmm. Edwin's latencytop output includes this (ignoring the _very_ top
    >> entries that are all either CD-ROM media change tests or are
    >> interruptible pipe/select things) at the top:
    >>
    >> 21 10264428 915514 get_request_wait __make_request
    >> generic_make_request submit_bio xfs_submit_ioend_bio xfs_submit_ioend
    >> xfs_page_state_convert xfs_vm_writepage __writepage
    >> write_cache_pages generic_writepages xfs_vm_writepages
    >>

    >
    >
    > argh. well I guess this might be useful for this case, but normally
    > latencytop gives you much more humanly readable data... maybe Edwin
    > forgot to do "make install" :-(


    I forgot to use 'latencytop -d', I was running latencytop and when I've
    seen something interesting I did a 'cat /proc/latency_stats' instead of
    'latencytop -d'.
    I'll try to use latencytop -d next time.

    Best regards,
    --Edwin
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  6. Re: [PATCH] x86_64: fix delayed signals

    Linus Torvalds writes:
    >
    > That said, seeing the full history of this same thing on the x86-32 side
    > actually makes me _much_ happier about the patch. Because now I can tell
    > when we did it, and what problems it seems to have caused (answer:
    > "apparently none that are possibly relevant on x86-64").


    FWIW I agree with Roland that the 64bit behaviour has a race window
    that can lead to delayed signals. I had actually fixed this some
    months ago myself in my tree because this was noticed by David (who
    was comparing to sparc I thin) during code review, but never pushed it
    out because it didn't get enough testing (it only ran on my
    workstation for some time)

    I think his patch is actually doing way too much work though, especially
    it is not needed to repeat all checks for the full work mask. I'm
    appending the patch I was using, which only rechecks signals.

    Obviously it's still not something that should be used without a lot
    of testing.

    But in theory it's ok.

    > So now I'm considering just putting it in before the 2.6.26 release after
    > all


    That would seem rather hazardous.

    -Andi

    ---

    Fix lost signal race on x86-64

    Originally noticed by Dave Miller during code review

    Signed-off-by: Andi Kleen

    ---
    arch/x86/kernel/entry_64.S | 6 +++---
    1 file changed, 3 insertions(+), 3 deletions(-)

    Index: linux/arch/x86/kernel/entry_64.S
    ================================================== =================
    --- linux.orig/arch/x86/kernel/entry_64.S
    +++ linux/arch/x86/kernel/entry_64.S
    @@ -305,7 +305,7 @@ sysret_signal:
    leaq -ARGOFFSET(%rsp),%rdi # &pt_regs -> arg1
    xorl %esi,%esi # oldset -> arg2
    call ptregscall_common
    -1: movl $_TIF_NEED_RESCHED,%edi
    +1: movl $_TIF_NEED_RESCHED|_TIF_SIGPENDING,%edi
    /* Use IRET because user could have changed frame. This
    works because ptregscall_common has called FIXUP_TOP_OF_STACK. */
    DISABLE_INTERRUPTS(CLBR_NONE)
    @@ -393,7 +393,7 @@ int_signal:
    movq %rsp,%rdi # &ptregs -> arg1
    xorl %esi,%esi # oldset -> arg2
    call do_notify_resume
    -1: movl $_TIF_NEED_RESCHED,%edi
    +1: movl $_TIF_NEED_RESCHED|_TIF_SIGPENDING,%edi
    int_restore_rest:
    RESTORE_REST
    DISABLE_INTERRUPTS(CLBR_NONE)
    @@ -647,7 +647,7 @@ retint_signal:
    RESTORE_REST
    DISABLE_INTERRUPTS(CLBR_NONE)
    TRACE_IRQS_OFF
    - movl $_TIF_NEED_RESCHED,%edi
    + movl $_TIF_NEED_RESCHED|_TIF_SIGPENDING,%edi
    GET_THREAD_INFO(%rcx)
    jmp retint_check

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  7. Re: [PATCH] x86_64: fix delayed signals

    Linus Torvalds writes:

    Linus, the fact that is not explained by your theory is why
    Ctrl-Z+kill works but Ctrl-C doesn't. On the signal side these
    both should be implemented in the same way -- if there was
    really a missing signal_pending() check somewhere then Ctrl-Z
    (and kill) should be delayed too.

    Now doing faster signal processing is always a good thing in general,
    but I'm a little doubtful your patch will help with that
    particular problem.

    -Andi
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  8. Re: [PATCH] x86_64: fix delayed signals

    On 2008-07-12 13:32, Török Edwin wrote:
    > I haven't yet tried ftrace on this box, and neither did I try Roland's
    > patch yet. I will try that now, and hopefuly come back with some numbers
    > shortly.
    >


    tip/master of today already includes Roland's patch, so I have tested it
    without even knowing on the 64-bit box.
    [After gathering all the traces, I wanted to repeat everything with the
    patch applied, but there was no need since its already there].
    I can say that I didn't notice any difference in the delays (I still get
    24 - 30 second delays).
    I also tried Linus's readdir patch, the latency is around 27 seconds
    (was 30 before). I am not sure if it is an improvement, or measurement
    noise.

    Also I wasn't able to reproduce the 'Ctrl+Z works faster than Ctrl+C'
    symptom, perhaps it is just a coincidence: by the time I press Ctrl+Z to
    interrupt, enough time has passed (30+ secs), that the same effect would
    have been obtained by another Ctrl+C, or by just waiting.

    On 32-bit, I think the 2-3 seconds latency can be attributed to I/O
    delays, and I think its acceptable given the slow disk (5k4 rpm sata).
    On 64-bit, which has faster disks, the latency is usually much smaller
    (responds to Ctrl+C instantly), but ocasionally the maximum latency is
    30 seconds.
    That is too much, I am sure the disks would be able to seek and give the
    data required much sooner (a 7k2 rpm, and a 10k rpm disk in raid0).

    I think there is something more going on here than just possibly
    delaying a signal, I cannot even start the dump_all.sh script when the
    30 second latency occurs,
    and its just a short script (that *was* launched before, should be in
    cache), and it mostly reads /proc stuff.

    What should I do next? Is there anything else I could trace to help you
    figure out where is the delay?

    I am using 2.6.26-rc9-tip for tracing, latest commit is:
    commit d47d504b9a39555b9dce1dc2d1974795a87534ba
    Author: Ingo Molnar
    Date: Sat Jul 12 09:37:25 2008 +0200

    .config is appended to this mail[6].

    To reproduce the latency I do the following on a text console (scripts
    below [1][2]):
    # ./tracing.sh &
    # ./test.sh &

    When I see the latency [3]:
    # . ./dump_all.sh

    As another user on a different tty:
    $ find / >/dev/null 2>/dev/null
    ^C

    Repeat until I see that find won't respond for more than a few seconds
    (10 sec+):

    $ find / >/dev/null 2>/dev/null
    ^C^C ^C^C ^C^C ^C^C ^C^C ^C^C ^C^C ^C^C ^C^C ^C^C ^C^C

    Switch to the root tty, execute the dump_all command, unfortunately the
    command only launches *after* find is killed (so after the huge latency
    finished), but
    I think it still gathers some useful info [CFS debugging is turned on,
    and so is kernel.latencytop].

    Here are some snippets from the same latency trace, 2.6.26-rc9-tip +
    Linus's readdir patch (see the full output at the end [4], [5], [6]):
    latencytop -d
    [max 27565.9msec] Walking directory tree
    - 344.92 msec (18.7%)
    [max 23876.6msec] stat() operation
    - 548.89 msec (6.2%)
    [max 23778.5msec] Creating block layer request
    - 1119.56 msec (11.5%)
    [max 23669.0msec] Reading directory content
    - 985.11 msec (4.0%)
    [max 23575.4msec] Page fault
    - 211.78 msec (4.5%)
    [max 20232.7msec] Pagecache sync readahead
    - 4058.77 msec (3.1%)
    [max 3020.4msec] fsync() on a file
    - 572.14 msec (1.1%)

    /debug/tracing/trace
    events/0-5 [00] 163.271126: kill_pgrp <-n_tty_receive_buf
    events/0-5 [00] 164.148165: kill_pgrp <-n_tty_receive_buf
    events/0-5 [00] 164.402391: kill_pgrp <-n_tty_receive_buf
    [... more of these ...]

    events/0-5 [00] 175.516874: kill_pgrp <-n_tty_receive_buf
    gzip-4600 [00] 175.710875: detach_pid <-release_task
    gzip-4600 [00] 175.710876: detach_pid <-release_task
    gzip-4600 [00] 175.710877: detach_pid <-release_task
    events/0-5 [00] 176.068271: kill_pgrp <-n_tty_receive_buf
    gzip-4600 [00] 176.104521: detach_pid <-release_task
    gzip-4600 [00] 176.104523: detach_pid <-release_task
    gzip-4600 [00] 176.104523: detach_pid <-release_task
    events/0-5 [00] 176.633643: kill_pgrp <-n_tty_receive_buf
    events/0-5 [00] 177.229960: kill_pgrp <-n_tty_receive_buf
    events/0-5 [00] 177.837424: kill_pgrp <-n_tty_receive_buf
    events/0-5 [00] 178.430948: kill_pgrp <-n_tty_receive_buf
    events/0-5 [00] 178.982075: kill_pgrp <-n_tty_receive_buf
    events/0-5 [00] 179.415018: kill_pgrp <-n_tty_receive_buf
    tracing.sh-4779 [00] 180.920587: detach_pid <-release_task
    tracing.sh-4779 [00] 180.920588: detach_pid <-release_task
    tracing.sh-4779 [00] 180.920588: detach_pid <-release_task
    bash-4745 [00] 188.162442: detach_pid <-release_tas

    This is a 25 second latency.

    I am not sure what the time units in the scheduler debug info are, but
    assuming they are miliseconds, they show a 24s se.block_max:
    bash (4677, #threads: 1)
    ---------------------------------------------------------
    se.exec_start : 739851.985209
    se.vruntime : 242558.101836
    se.sum_exec_runtime : 844.881816
    se.avg_overlap : 0.021806
    se.wait_start : 0.000000
    se.sleep_start : 739851.985209
    se.block_start : 0.000000
    se.sleep_max : 93374.991561
    se.block_max : 24867.990819

    [1]: tracing.sh
    #!/bin/sh
    cd /debug/tracing
    sysctl kernel.latencytop=1
    echo kill_pgrp detach_pid\
    >set_ftrace_filter

    echo ftrace >current_tracer
    echo 1 >tracing_enabled
    sleep 60
    echo 0 >tracing_enabled
    cat trace >/tmp/trace.out
    cat latency_trace >/tmp/latencytrace.out
    echo "DONE"

    [2]: test.sh
    dd if=/dev/zero of=x1 bs=100M count=20&
    dd if=/dev/zero of=x2 bs=100M count=20&
    dd if=/dev/zero of=x3 bs=100M count=20&
    dd if=/dev/zero of=x4 bs=100M count=20&
    dd if=/dev/zero of=x5 bs=100M count=20&
    wait %1 %2 %3 %4 $5
    rm x1 x2 x3 x4 x5

    [3] dump_all.sh
    #!/bin/sh
    #try to launch everything asap
    ../cfs-debug-info.sh&
    cat /proc/latency_stats >latency.raw&
    latencytop -d >latency.dump&

    [4] latency.dump
    [max 27565.9msec] Walking directory tree
    - 344.92 msec (18.7%)
    [max 23876.6msec] stat() operation
    - 548.89 msec (6.2%)
    [max 23778.5msec] Creating block layer request
    - 1119.56 msec (11.5%)
    [max 23669.0msec] Reading directory content
    - 985.11 msec (4.0%)
    [max 23575.4msec] Page fault
    - 211.78 msec (4.5%)
    [max 20232.7msec] Pagecache sync readahead
    - 4058.77 msec (3.1%)
    [max 3020.4msec] fsync() on a file
    - 572.14 msec (1.1%)
    [max 1326.8msec] Syncing inodes
    - 315.55 msec (44.3%)
    [max 1181.5msec] Writing back inodes
    - 29.98 msec (1.6%)
    [max 1151.1msec] Deleting an inode
    - 36.61 msec (0.3%)
    [max 973.9msec] Writing a page to disk
    - 43.24 msec (0.3%)
    [max 701.8msec] Writing buffer to disk (synchronous)
    - 41.91 msec (0.7%)
    [max 123.0msec] Executing a program
    - 29.36 msec (0.2%)
    [max 97.8msec] synchronous write
    - 8.32 msec (3.3%)
    [max 41.5msec] queue_log_writer do_journal_end do_journal_begin_r
    journal_begin reiserfs_persistent_transaction reiserfs_get_block
    __block_prepare_write block_write_begin reiserfs_write_begin
    generic_file_buffered_write __generic_file_aio_write_nolock
    generic_file_aio_write - 9.12 msec (0.0%)
    [max 18.5msec] congestion_wait __alloc_pages_internal __alloc_pages
    __do_page_cache_readahead force_page_cache_readahead sys_readahead
    system_call_after_swapgs - 18.41 msec (0.0%)
    [max 17.1msec] congestion_wait __alloc_pages_internal __alloc_pages
    find_or_create_page __getblk do_journal_end journal_end
    reiserfs_end_persistent_transaction reiserfs_write_end
    generic_file_buffered_write __generic_file_aio_write_nolock
    generic_file_aio_write - 11.38 msec (0.0%)
    [max 16.7msec] Fork() system call
    - 8.98 msec (0.0%)
    [max 13.0msec] Writing to file
    - 10.03 msec (0.0%)
    [max 7.4msec] Executing raw SCSI command
    - 0.51 msec (0.0%)
    [max 4.7msec] Waiting for a process to die
    - 0.07 msec (0.0%)
    [max 4.5msec] Reading from a pipe
    - 0.64 msec (0.0%)
    [max 4.4msec] Waiting for event (poll)
    - 4.31 msec (0.0%)
    [max 3.9msec] opening cdrom device
    - 0.61 msec (0.0%)
    [max 3.2msec] Waiting for event (select)
    - 0.80 msec (0.0%)
    [max 1.4msec] congestion_wait __alloc_pages_internal __alloc_pages
    find_or_create_page __getblk get_empty_nodes fix_nodes
    reiserfs_paste_into_item reiserfs_get_block __block_prepare_write
    block_write_begin reiserfs_write_begin - 1.36 msec (0.0%)
    [max 0.5msec] SCSI cdrom ioctl
    - 0.25 msec (0.0%)
    [max 0.0msec] Waiting for event (epoll)
    - 0.01 msec (0.0%)
    [max 0.0msec] Waiting for TTY input
    - 0.01 msec (0.0%)
    [max 0.0msec] Writing data to TTY
    - 0.00 msec (0.0%)

    [5] Everything else is available as a .tar.gz from here (it is 3.1M
    unpacked, and didn't want to flood your inboxes):
    http://edwintorok.googlepages.com/traces.tar.gz

    [6]
    #
    # Automatically generated make config: don't edit
    # Linux kernel version: 2.6.26-rc9
    # Sat Jul 12 13:39:35 2008
    #
    CONFIG_64BIT=y
    # CONFIG_X86_32 is not set
    CONFIG_X86_64=y
    CONFIG_X86=y
    CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
    # CONFIG_GENERIC_LOCKBREAK is not set
    CONFIG_GENERIC_TIME=y
    CONFIG_GENERIC_CMOS_UPDATE=y
    CONFIG_CLOCKSOURCE_WATCHDOG=y
    CONFIG_GENERIC_CLOCKEVENTS=y
    CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
    CONFIG_LOCKDEP_SUPPORT=y
    CONFIG_STACKTRACE_SUPPORT=y
    CONFIG_HAVE_LATENCYTOP_SUPPORT=y
    CONFIG_FAST_CMPXCHG_LOCAL=y
    CONFIG_MMU=y
    CONFIG_ZONE_DMA=y
    CONFIG_GENERIC_ISA_DMA=y
    CONFIG_GENERIC_IOMAP=y
    CONFIG_GENERIC_BUG=y
    CONFIG_GENERIC_HWEIGHT=y
    # CONFIG_GENERIC_GPIO is not set
    CONFIG_ARCH_MAY_HAVE_PC_FDC=y
    CONFIG_RWSEM_GENERIC_SPINLOCK=y
    # CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
    # CONFIG_ARCH_HAS_ILOG2_U32 is not set
    # CONFIG_ARCH_HAS_ILOG2_U64 is not set
    CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
    CONFIG_GENERIC_CALIBRATE_DELAY=y
    CONFIG_GENERIC_TIME_VSYSCALL=y
    CONFIG_ARCH_HAS_CPU_RELAX=y
    CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
    # CONFIG_HAVE_SETUP_PER_CPU_AREA is not set
    # CONFIG_HAVE_CPUMASK_OF_CPU_MAP is not set
    CONFIG_ARCH_HIBERNATION_POSSIBLE=y
    CONFIG_ARCH_SUSPEND_POSSIBLE=y
    CONFIG_ZONE_DMA32=y
    CONFIG_ARCH_POPULATES_NODE_MAP=y
    CONFIG_AUDIT_ARCH=y
    CONFIG_ARCH_SUPPORTS_AOUT=y
    CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
    CONFIG_GENERIC_HARDIRQS=y
    CONFIG_GENERIC_IRQ_PROBE=y
    CONFIG_X86_BIOS_REBOOT=y
    CONFIG_X86_TRAMPOLINE=y
    # CONFIG_KTIME_SCALAR is not set
    CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"

    #
    # General setup
    #
    CONFIG_EXPERIMENTAL=y
    CONFIG_BROKEN_ON_SMP=y
    CONFIG_INIT_ENV_ARG_LIMIT=32
    CONFIG_LOCALVERSION=""
    # CONFIG_LOCALVERSION_AUTO is not set
    CONFIG_SWAP=y
    CONFIG_SYSVIPC=y
    CONFIG_SYSVIPC_SYSCTL=y
    CONFIG_POSIX_MQUEUE=y
    CONFIG_BSD_PROCESS_ACCT=y
    CONFIG_BSD_PROCESS_ACCT_V3=y
    # CONFIG_TASKSTATS is not set
    CONFIG_AUDIT=y
    # CONFIG_AUDITSYSCALL is not set
    CONFIG_IKCONFIG=y
    CONFIG_IKCONFIG_PROC=y
    CONFIG_LOG_BUF_SHIFT=16
    # CONFIG_CGROUPS is not set
    CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
    CONFIG_GROUP_SCHED=y
    CONFIG_FAIR_GROUP_SCHED=y
    # CONFIG_RT_GROUP_SCHED is not set
    CONFIG_USER_SCHED=y
    # CONFIG_CGROUP_SCHED is not set
    # CONFIG_SYSFS_DEPRECATED_V2 is not set
    CONFIG_RELAY=y
    CONFIG_NAMESPACES=y
    # CONFIG_UTS_NS is not set
    # CONFIG_IPC_NS is not set
    # CONFIG_USER_NS is not set
    # CONFIG_PID_NS is not set
    CONFIG_BLK_DEV_INITRD=y
    CONFIG_INITRAMFS_SOURCE=""
    # CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
    CONFIG_SYSCTL=y
    # CONFIG_EMBEDDED is not set
    CONFIG_UID16=y
    CONFIG_SYSCTL_SYSCALL=y
    CONFIG_SYSCTL_SYSCALL_CHECK=y
    CONFIG_KALLSYMS=y
    CONFIG_KALLSYMS_ALL=y
    # CONFIG_KALLSYMS_EXTRA_PASS is not set
    CONFIG_HOTPLUG=y
    CONFIG_PRINTK=y
    CONFIG_BUG=y
    CONFIG_ELF_CORE=y
    CONFIG_PCSPKR_PLATFORM=y
    # CONFIG_COMPAT_BRK is not set
    CONFIG_BASE_FULL=y
    CONFIG_FUTEX=y
    CONFIG_ANON_INODES=y
    CONFIG_EPOLL=y
    CONFIG_SIGNALFD=y
    CONFIG_TIMERFD=y
    CONFIG_EVENTFD=y
    CONFIG_SHMEM=y
    CONFIG_VM_EVENT_COUNTERS=y
    CONFIG_SLUB_DEBUG=y
    # CONFIG_SLAB is not set
    CONFIG_SLUB=y
    # CONFIG_SLOB is not set
    CONFIG_PROFILING=y
    CONFIG_MARKERS=y
    CONFIG_OPROFILE=m
    CONFIG_HAVE_OPROFILE=y
    CONFIG_KPROBES=y
    CONFIG_KRETPROBES=y
    CONFIG_HAVE_KPROBES=y
    CONFIG_HAVE_KRETPROBES=y
    # CONFIG_HAVE_DMA_ATTRS is not set
    # CONFIG_USE_GENERIC_SMP_HELPERS is not set
    CONFIG_PROC_PAGE_MONITOR=y
    # CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
    CONFIG_SLABINFO=y
    CONFIG_RT_MUTEXES=y
    # CONFIG_TINY_SHMEM is not set
    CONFIG_BASE_SMALL=0
    CONFIG_MODULES=y
    # CONFIG_MODULE_FORCE_LOAD is not set
    CONFIG_MODULE_UNLOAD=y
    CONFIG_MODULE_FORCE_UNLOAD=y
    CONFIG_MODVERSIONS=y
    # CONFIG_MODULE_SRCVERSION_ALL is not set
    CONFIG_KMOD=y
    CONFIG_BLOCK=y
    CONFIG_BLK_DEV_IO_TRACE=y
    # CONFIG_BLK_DEV_BSG is not set
    CONFIG_BLOCK_COMPAT=y

    #
    # IO Schedulers
    #
    CONFIG_IOSCHED_NOOP=y
    CONFIG_IOSCHED_AS=y
    CONFIG_IOSCHED_DEADLINE=y
    CONFIG_IOSCHED_CFQ=y
    # CONFIG_DEFAULT_AS is not set
    # CONFIG_DEFAULT_DEADLINE is not set
    CONFIG_DEFAULT_CFQ=y
    # CONFIG_DEFAULT_NOOP is not set
    CONFIG_DEFAULT_IOSCHED="cfq"
    CONFIG_CLASSIC_RCU=y

    #
    # Processor type and features
    #
    CONFIG_TICK_ONESHOT=y
    CONFIG_NO_HZ=y
    CONFIG_HIGH_RES_TIMERS=y
    CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
    # CONFIG_SMP is not set
    CONFIG_X86_FIND_SMP_CONFIG=y
    CONFIG_X86_MPPARSE=y
    CONFIG_X86_PC=y
    # CONFIG_X86_ELAN is not set
    # CONFIG_X86_VOYAGER is not set
    # CONFIG_X86_GENERICARCH is not set
    # CONFIG_X86_RDC321X is not set
    # CONFIG_X86_VSMP is not set
    # CONFIG_PARAVIRT_GUEST is not set
    # CONFIG_MEMTEST is not set
    # CONFIG_M386 is not set
    # CONFIG_M486 is not set
    # CONFIG_M586 is not set
    # CONFIG_M586TSC is not set
    # CONFIG_M586MMX is not set
    # CONFIG_M686 is not set
    # CONFIG_MPENTIUMII is not set
    # CONFIG_MPENTIUMIII is not set
    # CONFIG_MPENTIUMM is not set
    # CONFIG_MPENTIUM4 is not set
    # CONFIG_MK6 is not set
    # CONFIG_MK7 is not set
    CONFIG_MK8=y
    # CONFIG_MCRUSOE is not set
    # CONFIG_MEFFICEON is not set
    # CONFIG_MWINCHIPC6 is not set
    # CONFIG_MWINCHIP2 is not set
    # CONFIG_MWINCHIP3D is not set
    # CONFIG_MGEODEGX1 is not set
    # CONFIG_MGEODE_LX is not set
    # CONFIG_MCYRIXIII is not set
    # CONFIG_MVIAC3_2 is not set
    # CONFIG_MVIAC7 is not set
    # CONFIG_MPSC is not set
    # CONFIG_MCORE2 is not set
    # CONFIG_GENERIC_CPU is not set
    CONFIG_X86_CPU=y
    CONFIG_X86_L1_CACHE_BYTES=64
    CONFIG_X86_INTERNODE_CACHE_BYTES=64
    CONFIG_X86_CMPXCHG=y
    CONFIG_X86_L1_CACHE_SHIFT=6
    CONFIG_X86_WP_WORKS_OK=y
    CONFIG_X86_GOOD_APIC=y
    CONFIG_X86_INTEL_USERCOPY=y
    CONFIG_X86_USE_PPRO_CHECKSUM=y
    CONFIG_X86_TSC=y
    CONFIG_X86_CMPXCHG64=y
    CONFIG_X86_CMOV=y
    CONFIG_X86_MINIMUM_CPU_FAMILY=64
    CONFIG_X86_DEBUGCTLMSR=y
    CONFIG_X86_DS=y
    CONFIG_X86_PTRACE_BTS=y
    CONFIG_HPET_TIMER=y
    CONFIG_HPET_EMULATE_RTC=y
    CONFIG_DMI=y
    CONFIG_GART_IOMMU=y
    # CONFIG_CALGARY_IOMMU is not set
    # CONFIG_AMD_IOMMU is not set
    CONFIG_SWIOTLB=y
    CONFIG_IOMMU_HELPER=y
    # CONFIG_PREEMPT_NONE is not set
    CONFIG_PREEMPT_VOLUNTARY=y
    # CONFIG_PREEMPT is not set
    CONFIG_X86_LOCAL_APIC=y
    CONFIG_X86_IO_APIC=y
    CONFIG_X86_MCE=y
    CONFIG_X86_MCE_INTEL=y
    CONFIG_X86_MCE_AMD=y
    # CONFIG_I8K is not set
    CONFIG_MICROCODE=m
    CONFIG_MICROCODE_OLD_INTERFACE=y
    CONFIG_X86_MSR=m
    CONFIG_X86_CPUID=m
    CONFIG_ARCH_SPARSEMEM_DEFAULT=y
    CONFIG_ARCH_SPARSEMEM_ENABLE=y
    CONFIG_ARCH_SELECT_MEMORY_MODEL=y
    CONFIG_ILLEGAL_POINTER_VALUE=0xffffc10000000000
    CONFIG_SELECT_MEMORY_MODEL=y
    # CONFIG_FLATMEM_MANUAL is not set
    # CONFIG_DISCONTIGMEM_MANUAL is not set
    CONFIG_SPARSEMEM_MANUAL=y
    CONFIG_SPARSEMEM=y
    CONFIG_HAVE_MEMORY_PRESENT=y
    # CONFIG_SPARSEMEM_STATIC is not set
    CONFIG_SPARSEMEM_EXTREME=y
    CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
    CONFIG_SPARSEMEM_VMEMMAP=y

    #
    # Memory hotplug is currently incompatible with Software Suspend
    #
    CONFIG_PAGEFLAGS_EXTENDED=y
    CONFIG_SPLIT_PTLOCK_CPUS=4
    CONFIG_RESOURCES_64BIT=y
    CONFIG_ZONE_DMA_FLAG=1
    CONFIG_BOUNCE=y
    CONFIG_VIRT_TO_BUS=y
    CONFIG_MTRR=y
    CONFIG_MTRR_SANITIZER=y
    CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
    CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
    # CONFIG_X86_PAT is not set
    # CONFIG_EFI is not set
    CONFIG_SECCOMP=y
    # CONFIG_CC_STACKPROTECTOR is not set
    # CONFIG_HZ_100 is not set
    # CONFIG_HZ_250 is not set
    CONFIG_HZ_300=y
    # CONFIG_HZ_1000 is not set
    CONFIG_HZ=300
    CONFIG_SCHED_HRTICK=y
    CONFIG_KEXEC=y
    # CONFIG_CRASH_DUMP is not set
    CONFIG_PHYSICAL_START=0x200000
    # CONFIG_RELOCATABLE is not set
    CONFIG_PHYSICAL_ALIGN=0x200000
    # CONFIG_COMPAT_VDSO is not set
    CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

    #
    # Power management options
    #
    CONFIG_ARCH_HIBERNATION_HEADER=y
    CONFIG_PM=y
    # CONFIG_PM_DEBUG is not set
    CONFIG_PM_SLEEP=y
    CONFIG_SUSPEND=y
    CONFIG_SUSPEND_FREEZER=y
    CONFIG_HIBERNATION=y
    CONFIG_PM_STD_PARTITION=""
    CONFIG_ACPI=y
    CONFIG_ACPI_SLEEP=y
    CONFIG_ACPI_PROCFS=y
    # CONFIG_ACPI_PROCFS_POWER is not set
    CONFIG_ACPI_SYSFS_POWER=y
    CONFIG_ACPI_PROC_EVENT=y
    CONFIG_ACPI_AC=m
    CONFIG_ACPI_BATTERY=m
    CONFIG_ACPI_BUTTON=m
    CONFIG_ACPI_VIDEO=m
    CONFIG_ACPI_FAN=m
    CONFIG_ACPI_DOCK=m
    # CONFIG_ACPI_BAY is not set
    CONFIG_ACPI_PROCESSOR=m
    CONFIG_ACPI_THERMAL=m
    CONFIG_ACPI_WMI=y
    CONFIG_ACPI_ASUS=m
    CONFIG_ACPI_TOSHIBA=m
    # CONFIG_ACPI_CUSTOM_DSDT is not set
    CONFIG_ACPI_BLACKLIST_YEAR=0
    # CONFIG_ACPI_DEBUG is not set
    CONFIG_ACPI_EC=y
    CONFIG_ACPI_POWER=y
    CONFIG_ACPI_SYSTEM=y
    CONFIG_X86_PM_TIMER=y
    CONFIG_ACPI_CONTAINER=m
    CONFIG_ACPI_SBS=m

    #
    # CPU Frequency scaling
    #
    CONFIG_CPU_FREQ=y
    CONFIG_CPU_FREQ_TABLE=m
    # CONFIG_CPU_FREQ_DEBUG is not set
    CONFIG_CPU_FREQ_STAT=m
    # CONFIG_CPU_FREQ_STAT_DETAILS is not set
    CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
    # CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
    # CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
    # CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
    # CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
    CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
    CONFIG_CPU_FREQ_GOV_POWERSAVE=m
    CONFIG_CPU_FREQ_GOV_USERSPACE=y
    # CONFIG_CPU_FREQ_GOV_ONDEMAND is not set
    CONFIG_CPU_FREQ_GOV_CONSERVATIVE=m

    #
    # CPUFreq processor drivers
    #
    CONFIG_X86_ACPI_CPUFREQ=m
    CONFIG_X86_POWERNOW_K8=m
    CONFIG_X86_POWERNOW_K8_ACPI=y
    # CONFIG_X86_SPEEDSTEP_CENTRINO is not set
    # CONFIG_X86_P4_CLOCKMOD is not set

    #
    # shared options
    #
    # CONFIG_X86_ACPI_CPUFREQ_PROC_INTF is not set
    # CONFIG_X86_SPEEDSTEP_LIB is not set
    CONFIG_CPU_IDLE=y
    CONFIG_CPU_IDLE_GOV_LADDER=y
    CONFIG_CPU_IDLE_GOV_MENU=y

    #
    # Bus options (PCI etc.)
    #
    CONFIG_PCI=y
    CONFIG_PCI_DIRECT=y
    CONFIG_PCI_MMCONFIG=y
    CONFIG_PCI_DOMAINS=y
    # CONFIG_DMAR is not set
    CONFIG_PCIEPORTBUS=y
    CONFIG_HOTPLUG_PCI_PCIE=m
    CONFIG_PCIEAER=y
    # CONFIG_PCIEASPM is not set
    CONFIG_ARCH_SUPPORTS_MSI=y
    CONFIG_PCI_MSI=y
    CONFIG_PCI_LEGACY=y
    # CONFIG_PCI_DEBUG is not set
    CONFIG_HT_IRQ=y
    CONFIG_ISA_DMA_API=y
    CONFIG_K8_NB=y
    # CONFIG_PCCARD is not set
    CONFIG_HOTPLUG_PCI=m
    CONFIG_HOTPLUG_PCI_FAKE=m
    CONFIG_HOTPLUG_PCI_ACPI=m
    CONFIG_HOTPLUG_PCI_ACPI_IBM=m
    CONFIG_HOTPLUG_PCI_CPCI=y
    CONFIG_HOTPLUG_PCI_CPCI_ZT5550=m
    CONFIG_HOTPLUG_PCI_CPCI_GENERIC=m
    CONFIG_HOTPLUG_PCI_SHPC=m

    #
    # Executable file formats / Emulations
    #
    CONFIG_BINFMT_ELF=y
    CONFIG_COMPAT_BINFMT_ELF=y
    CONFIG_BINFMT_MISC=m
    CONFIG_IA32_EMULATION=y
    CONFIG_IA32_AOUT=y
    CONFIG_COMPAT=y
    CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
    CONFIG_SYSVIPC_COMPAT=y

    #
    # Networking
    #
    CONFIG_NET=y

    #
    # Networking options
    #
    CONFIG_PACKET=y
    CONFIG_PACKET_MMAP=y
    CONFIG_UNIX=y
    CONFIG_XFRM=y
    CONFIG_XFRM_USER=m
    # CONFIG_XFRM_SUB_POLICY is not set
    # CONFIG_XFRM_MIGRATE is not set
    # CONFIG_XFRM_STATISTICS is not set
    CONFIG_NET_KEY=m
    # CONFIG_NET_KEY_MIGRATE is not set
    CONFIG_INET=y
    CONFIG_IP_MULTICAST=y
    CONFIG_IP_ADVANCED_ROUTER=y
    CONFIG_ASK_IP_FIB_HASH=y
    # CONFIG_IP_FIB_TRIE is not set
    CONFIG_IP_FIB_HASH=y
    CONFIG_IP_MULTIPLE_TABLES=y
    CONFIG_IP_ROUTE_MULTIPATH=y
    CONFIG_IP_ROUTE_VERBOSE=y
    # CONFIG_IP_PNP is not set
    CONFIG_NET_IPIP=m
    CONFIG_NET_IPGRE=m
    CONFIG_NET_IPGRE_BROADCAST=y
    CONFIG_IP_MROUTE=y
    CONFIG_IP_PIMSM_V1=y
    CONFIG_IP_PIMSM_V2=y
    # CONFIG_ARPD is not set
    CONFIG_SYN_COOKIES=y
    CONFIG_INET_AH=m
    CONFIG_INET_ESP=m
    CONFIG_INET_IPCOMP=m
    CONFIG_INET_XFRM_TUNNEL=m
    CONFIG_INET_TUNNEL=m
    CONFIG_INET_XFRM_MODE_TRANSPORT=m
    CONFIG_INET_XFRM_MODE_TUNNEL=m
    CONFIG_INET_XFRM_MODE_BEET=m
    CONFIG_INET_LRO=m
    CONFIG_INET_DIAG=m
    CONFIG_INET_TCP_DIAG=m
    CONFIG_TCP_CONG_ADVANCED=y
    CONFIG_TCP_CONG_BIC=y
    CONFIG_TCP_CONG_CUBIC=m
    CONFIG_TCP_CONG_WESTWOOD=m
    CONFIG_TCP_CONG_HTCP=m
    CONFIG_TCP_CONG_HSTCP=m
    CONFIG_TCP_CONG_HYBLA=m
    CONFIG_TCP_CONG_VEGAS=m
    CONFIG_TCP_CONG_SCALABLE=m
    CONFIG_TCP_CONG_LP=m
    CONFIG_TCP_CONG_VENO=m
    # CONFIG_TCP_CONG_YEAH is not set
    # CONFIG_TCP_CONG_ILLINOIS is not set
    CONFIG_DEFAULT_BIC=y
    # CONFIG_DEFAULT_CUBIC is not set
    # CONFIG_DEFAULT_HTCP is not set
    # CONFIG_DEFAULT_VEGAS is not set
    # CONFIG_DEFAULT_WESTWOOD is not set
    # CONFIG_DEFAULT_RENO is not set
    CONFIG_DEFAULT_TCP_CONG="bic"
    # CONFIG_TCP_MD5SIG is not set
    CONFIG_IP_VS=m
    # CONFIG_IP_VS_DEBUG is not set
    CONFIG_IP_VS_TAB_BITS=12

    #
    # IPVS transport protocol load balancing support
    #
    CONFIG_IP_VS_PROTO_TCP=y
    CONFIG_IP_VS_PROTO_UDP=y
    CONFIG_IP_VS_PROTO_ESP=y
    CONFIG_IP_VS_PROTO_AH=y

    #
    # IPVS scheduler
    #
    CONFIG_IP_VS_RR=m
    CONFIG_IP_VS_WRR=m
    CONFIG_IP_VS_LC=m
    CONFIG_IP_VS_WLC=m
    CONFIG_IP_VS_LBLC=m
    CONFIG_IP_VS_LBLCR=m
    CONFIG_IP_VS_DH=m
    CONFIG_IP_VS_SH=m
    CONFIG_IP_VS_SED=m
    CONFIG_IP_VS_NQ=m

    #
    # IPVS application helper
    #
    CONFIG_IP_VS_FTP=m
    CONFIG_IPV6=m
    CONFIG_IPV6_PRIVACY=y
    # CONFIG_IPV6_ROUTER_PREF is not set
    # CONFIG_IPV6_OPTIMISTIC_DAD is not set
    CONFIG_INET6_AH=m
    CONFIG_INET6_ESP=m
    CONFIG_INET6_IPCOMP=m
    # CONFIG_IPV6_MIP6 is not set
    CONFIG_INET6_XFRM_TUNNEL=m
    CONFIG_INET6_TUNNEL=m
    CONFIG_INET6_XFRM_MODE_TRANSPORT=m
    CONFIG_INET6_XFRM_MODE_TUNNEL=m
    CONFIG_INET6_XFRM_MODE_BEET=m
    CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION=m
    CONFIG_IPV6_SIT=m
    CONFIG_IPV6_NDISC_NODETYPE=y
    CONFIG_IPV6_TUNNEL=m
    # CONFIG_IPV6_MULTIPLE_TABLES is not set
    # CONFIG_IPV6_MROUTE is not set
    # CONFIG_NETLABEL is not set
    CONFIG_NETWORK_SECMARK=y
    CONFIG_NETFILTER=y
    # CONFIG_NETFILTER_DEBUG is not set
    # CONFIG_NETFILTER_ADVANCED is not set

    #
    # Core Netfilter Configuration
    #
    CONFIG_NETFILTER_NETLINK=m
    CONFIG_NETFILTER_NETLINK_LOG=m
    CONFIG_NF_CONNTRACK=m
    CONFIG_NF_CONNTRACK_SECMARK=y
    CONFIG_NF_CONNTRACK_FTP=m
    CONFIG_NF_CONNTRACK_IRC=m
    CONFIG_NF_CONNTRACK_SIP=m
    CONFIG_NF_CT_NETLINK=m
    CONFIG_NETFILTER_XTABLES=m
    CONFIG_NETFILTER_XT_TARGET_MARK=m
    CONFIG_NETFILTER_XT_TARGET_NFLOG=m
    CONFIG_NETFILTER_XT_TARGET_SECMARK=m
    CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
    CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
    CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
    CONFIG_NETFILTER_XT_MATCH_MARK=m
    CONFIG_NETFILTER_XT_MATCH_POLICY=m
    CONFIG_NETFILTER_XT_MATCH_STATE=m

    #
    # IP: Netfilter Configuration
    #
    CONFIG_NF_CONNTRACK_IPV4=m
    CONFIG_NF_CONNTRACK_PROC_COMPAT=y
    CONFIG_IP_NF_IPTABLES=m
    CONFIG_IP_NF_FILTER=m
    CONFIG_IP_NF_TARGET_REJECT=m
    CONFIG_IP_NF_TARGET_LOG=m
    CONFIG_IP_NF_TARGET_ULOG=m
    CONFIG_NF_NAT=m
    CONFIG_NF_NAT_NEEDED=y
    CONFIG_IP_NF_TARGET_MASQUERADE=m
    CONFIG_NF_NAT_FTP=m
    CONFIG_NF_NAT_IRC=m
    # CONFIG_NF_NAT_TFTP is not set
    # CONFIG_NF_NAT_AMANDA is not set
    # CONFIG_NF_NAT_PPTP is not set
    # CONFIG_NF_NAT_H323 is not set
    CONFIG_NF_NAT_SIP=m
    CONFIG_IP_NF_MANGLE=m

    #
    # IPv6: Netfilter Configuration
    #
    CONFIG_NF_CONNTRACK_IPV6=m
    CONFIG_IP6_NF_IPTABLES=m
    CONFIG_IP6_NF_MATCH_IPV6HEADER=m
    CONFIG_IP6_NF_FILTER=m
    CONFIG_IP6_NF_TARGET_LOG=m
    CONFIG_IP6_NF_TARGET_REJECT=m
    CONFIG_IP6_NF_MANGLE=m
    CONFIG_IP_DCCP=m
    CONFIG_INET_DCCP_DIAG=m
    CONFIG_IP_DCCP_ACKVEC=y

    #
    # DCCP CCIDs Configuration (EXPERIMENTAL)
    #
    CONFIG_IP_DCCP_CCID2=m
    # CONFIG_IP_DCCP_CCID2_DEBUG is not set
    CONFIG_IP_DCCP_CCID3=m
    # CONFIG_IP_DCCP_CCID3_DEBUG is not set
    CONFIG_IP_DCCP_CCID3_RTO=100
    CONFIG_IP_DCCP_TFRC_LIB=m

    #
    # DCCP Kernel Hacking
    #
    # CONFIG_IP_DCCP_DEBUG is not set
    # CONFIG_NET_DCCPPROBE is not set
    CONFIG_IP_SCTP=m
    # CONFIG_SCTP_DBG_MSG is not set
    # CONFIG_SCTP_DBG_OBJCNT is not set
    # CONFIG_SCTP_HMAC_NONE is not set
    # CONFIG_SCTP_HMAC_SHA1 is not set
    CONFIG_SCTP_HMAC_MD5=y
    CONFIG_TIPC=m
    CONFIG_TIPC_ADVANCED=y
    CONFIG_TIPC_ZONES=3
    CONFIG_TIPC_CLUSTERS=1
    CONFIG_TIPC_NODES=255
    CONFIG_TIPC_SLAVE_NODES=0
    CONFIG_TIPC_PORTS=8191
    CONFIG_TIPC_LOG=0
    # CONFIG_TIPC_DEBUG is not set
    CONFIG_ATM=y
    CONFIG_ATM_CLIP=y
    # CONFIG_ATM_CLIP_NO_ICMP is not set
    CONFIG_ATM_LANE=m
    CONFIG_ATM_MPOA=m
    CONFIG_ATM_BR2684=m
    # CONFIG_ATM_BR2684_IPFILTER is not set
    CONFIG_BRIDGE=m
    CONFIG_VLAN_8021Q=m
    CONFIG_DECNET=m
    # CONFIG_DECNET_ROUTER is not set
    CONFIG_LLC=m
    CONFIG_LLC2=m
    CONFIG_IPX=m
    # CONFIG_IPX_INTERN is not set
    CONFIG_ATALK=m
    CONFIG_DEV_APPLETALK=m
    CONFIG_IPDDP=m
    CONFIG_IPDDP_ENCAP=y
    CONFIG_IPDDP_DECAP=y
    CONFIG_X25=m
    CONFIG_LAPB=m
    CONFIG_ECONET=m
    CONFIG_ECONET_AUNUDP=y
    CONFIG_ECONET_NATIVE=y
    CONFIG_WAN_ROUTER=m
    CONFIG_NET_SCHED=y

    #
    # Queueing/Scheduling
    #
    CONFIG_NET_SCH_CBQ=m
    CONFIG_NET_SCH_HTB=m
    CONFIG_NET_SCH_HFSC=m
    CONFIG_NET_SCH_ATM=m
    CONFIG_NET_SCH_PRIO=m
    CONFIG_NET_SCH_RED=m
    CONFIG_NET_SCH_SFQ=m
    CONFIG_NET_SCH_TEQL=m
    CONFIG_NET_SCH_TBF=m
    CONFIG_NET_SCH_GRED=m
    CONFIG_NET_SCH_DSMARK=m
    CONFIG_NET_SCH_NETEM=m
    CONFIG_NET_SCH_INGRESS=m

    #
    # Classification
    #
    CONFIG_NET_CLS=y
    CONFIG_NET_CLS_BASIC=m
    CONFIG_NET_CLS_TCINDEX=m
    CONFIG_NET_CLS_ROUTE4=m
    CONFIG_NET_CLS_ROUTE=y
    CONFIG_NET_CLS_FW=m
    CONFIG_NET_CLS_U32=m
    CONFIG_CLS_U32_PERF=y
    CONFIG_CLS_U32_MARK=y
    CONFIG_NET_CLS_RSVP=m
    CONFIG_NET_CLS_RSVP6=m
    # CONFIG_NET_CLS_FLOW is not set
    CONFIG_NET_EMATCH=y
    CONFIG_NET_EMATCH_STACK=32
    CONFIG_NET_EMATCH_CMP=m
    CONFIG_NET_EMATCH_NBYTE=m
    CONFIG_NET_EMATCH_U32=m
    CONFIG_NET_EMATCH_META=m
    CONFIG_NET_EMATCH_TEXT=m
    CONFIG_NET_CLS_ACT=y
    CONFIG_NET_ACT_POLICE=m
    CONFIG_NET_ACT_GACT=m
    CONFIG_GACT_PROB=y
    CONFIG_NET_ACT_MIRRED=m
    CONFIG_NET_ACT_IPT=m
    CONFIG_NET_ACT_NAT=m
    CONFIG_NET_ACT_PEDIT=m
    CONFIG_NET_ACT_SIMP=m
    CONFIG_NET_CLS_IND=y
    CONFIG_NET_SCH_FIFO=y

    #
    # Network testing
    #
    CONFIG_NET_PKTGEN=m
    # CONFIG_NET_TCPPROBE is not set
    # CONFIG_HAMRADIO is not set
    # CONFIG_CAN is not set
    # CONFIG_IRDA is not set
    # CONFIG_BT is not set
    CONFIG_AF_RXRPC=m
    # CONFIG_AF_RXRPC_DEBUG is not set
    # CONFIG_RXKAD is not set
    CONFIG_FIB_RULES=y

    #
    # Wireless
    #
    # CONFIG_CFG80211 is not set
    # CONFIG_WIRELESS_EXT is not set
    # CONFIG_MAC80211 is not set
    # CONFIG_IEEE80211 is not set
    # CONFIG_RFKILL is not set
    # CONFIG_NET_9P is not set

    #
    # Device Drivers
    #

    #
    # Generic Driver Options
    #
    CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
    CONFIG_STANDALONE=y
    CONFIG_PREVENT_FIRMWARE_BUILD=y
    CONFIG_FW_LOADER=m
    # CONFIG_DEBUG_DRIVER is not set
    # CONFIG_DEBUG_DEVRES is not set
    # CONFIG_SYS_HYPERVISOR is not set
    CONFIG_CONNECTOR=m
    CONFIG_MTD=m
    # CONFIG_MTD_DEBUG is not set
    # CONFIG_MTD_CONCAT is not set
    CONFIG_MTD_PARTITIONS=y
    # CONFIG_MTD_REDBOOT_PARTS is not set
    # CONFIG_MTD_AR7_PARTS is not set

    #
    # User Modules And Translation Layers
    #
    # CONFIG_MTD_CHAR is not set
    # CONFIG_MTD_BLKDEVS is not set
    # CONFIG_MTD_BLOCK is not set
    # CONFIG_MTD_BLOCK_RO is not set
    # CONFIG_FTL is not set
    # CONFIG_NFTL is not set
    # CONFIG_INFTL is not set
    # CONFIG_RFD_FTL is not set
    # CONFIG_SSFDC is not set
    # CONFIG_MTD_OOPS is not set

    #
    # RAM/ROM/Flash chip drivers
    #
    # CONFIG_MTD_CFI is not set
    CONFIG_MTD_JEDECPROBE=m
    CONFIG_MTD_GEN_PROBE=m
    # CONFIG_MTD_CFI_ADV_OPTIONS is not set
    CONFIG_MTD_MAP_BANK_WIDTH_1=y
    CONFIG_MTD_MAP_BANK_WIDTH_2=y
    CONFIG_MTD_MAP_BANK_WIDTH_4=y
    # CONFIG_MTD_MAP_BANK_WIDTH_8 is not set
    # CONFIG_MTD_MAP_BANK_WIDTH_16 is not set
    # CONFIG_MTD_MAP_BANK_WIDTH_32 is not set
    CONFIG_MTD_CFI_I1=y
    CONFIG_MTD_CFI_I2=y
    # CONFIG_MTD_CFI_I4 is not set
    # CONFIG_MTD_CFI_I8 is not set
    # CONFIG_MTD_CFI_INTELEXT is not set
    CONFIG_MTD_CFI_AMDSTD=m
    # CONFIG_MTD_CFI_STAA is not set
    CONFIG_MTD_CFI_UTIL=m
    CONFIG_MTD_RAM=m
    # CONFIG_MTD_ROM is not set
    # CONFIG_MTD_ABSENT is not set

    #
    # Mapping drivers for chip access
    #
    # CONFIG_MTD_COMPLEX_MAPPINGS is not set
    # CONFIG_MTD_PHYSMAP is not set
    # CONFIG_MTD_TS5500 is not set
    # CONFIG_MTD_AMD76XROM is not set
    # CONFIG_MTD_ICHXROM is not set
    # CONFIG_MTD_ESB2ROM is not set
    # CONFIG_MTD_CK804XROM is not set
    # CONFIG_MTD_SCB2_FLASH is not set
    # CONFIG_MTD_NETtel is not set
    # CONFIG_MTD_L440GX is not set
    # CONFIG_MTD_INTEL_VR_NOR is not set
    # CONFIG_MTD_PLATRAM is not set

    #
    # Self-contained MTD device drivers
    #
    # CONFIG_MTD_PMC551 is not set
    # CONFIG_MTD_SLRAM is not set
    # CONFIG_MTD_PHRAM is not set
    # CONFIG_MTD_MTDRAM is not set
    # CONFIG_MTD_BLOCK2MTD is not set

    #
    # Disk-On-Chip Device Drivers
    #
    # CONFIG_MTD_DOC2000 is not set
    # CONFIG_MTD_DOC2001 is not set
    # CONFIG_MTD_DOC2001PLUS is not set
    # CONFIG_MTD_NAND is not set
    # CONFIG_MTD_ONENAND is not set

    #
    # UBI - Unsorted block images
    #
    # CONFIG_MTD_UBI is not set
    CONFIG_PARPORT=m
    CONFIG_PARPORT_PC=m
    CONFIG_PARPORT_SERIAL=m
    CONFIG_PARPORT_PC_FIFO=y
    CONFIG_PARPORT_PC_SUPERIO=y
    # CONFIG_PARPORT_GSC is not set
    CONFIG_PARPORT_AX88796=m
    CONFIG_PARPORT_1284=y
    CONFIG_PARPORT_NOT_PC=y
    CONFIG_PNP=y
    # CONFIG_PNP_DEBUG is not set

    #
    # Protocols
    #
    CONFIG_PNPACPI=y
    CONFIG_BLK_DEV=y
    CONFIG_BLK_DEV_FD=m
    # CONFIG_PARIDE is not set
    # CONFIG_BLK_CPQ_DA is not set
    # CONFIG_BLK_CPQ_CISS_DA is not set
    # CONFIG_BLK_DEV_DAC960 is not set
    # CONFIG_BLK_DEV_UMEM is not set
    # CONFIG_BLK_DEV_COW_COMMON is not set
    CONFIG_BLK_DEV_LOOP=m
    CONFIG_BLK_DEV_CRYPTOLOOP=m
    CONFIG_BLK_DEV_NBD=m
    # CONFIG_BLK_DEV_SX8 is not set
    # CONFIG_BLK_DEV_UB is not set
    CONFIG_BLK_DEV_RAM=y
    CONFIG_BLK_DEV_RAM_COUNT=16
    CONFIG_BLK_DEV_RAM_SIZE=65536
    # CONFIG_BLK_DEV_XIP is not set
    CONFIG_CDROM_PKTCDVD=m
    CONFIG_CDROM_PKTCDVD_BUFFERS=8
    # CONFIG_CDROM_PKTCDVD_WCACHE is not set
    # CONFIG_ATA_OVER_ETH is not set
    CONFIG_MISC_DEVICES=y
    # CONFIG_IBM_ASM is not set
    # CONFIG_PHANTOM is not set
    # CONFIG_EEPROM_93CX6 is not set
    # CONFIG_SGI_IOC4 is not set
    CONFIG_TIFM_CORE=m
    # CONFIG_TIFM_7XX1 is not set
    # CONFIG_FUJITSU_LAPTOP is not set
    # CONFIG_MSI_LAPTOP is not set
    # CONFIG_SONY_LAPTOP is not set
    # CONFIG_THINKPAD_ACPI is not set
    # CONFIG_INTEL_MENLOW is not set
    # CONFIG_EEEPC_LAPTOP is not set
    # CONFIG_ENCLOSURE_SERVICES is not set
    CONFIG_HAVE_IDE=y
    CONFIG_IDE=m
    CONFIG_BLK_DEV_IDE=m

    #
    # Please see Documentation/ide/ide.txt for help/info on IDE drives
    #
    # CONFIG_BLK_DEV_IDE_SATA is not set
    # CONFIG_BLK_DEV_IDEDISK is not set
    # CONFIG_IDEDISK_MULTI_MODE is not set
    CONFIG_BLK_DEV_IDECD=m
    CONFIG_BLK_DEV_IDECD_VERBOSE_ERRORS=y
    # CONFIG_BLK_DEV_IDETAPE is not set
    CONFIG_BLK_DEV_IDEFLOPPY=m
    # CONFIG_BLK_DEV_IDESCSI is not set
    CONFIG_BLK_DEV_IDEACPI=y
    # CONFIG_IDE_TASK_IOCTL is not set
    CONFIG_IDE_PROC_FS=y

    #
    # IDE chipset support/bugfixes
    #
    # CONFIG_IDE_GENERIC is not set
    # CONFIG_BLK_DEV_PLATFORM is not set
    # CONFIG_BLK_DEV_CMD640 is not set
    # CONFIG_BLK_DEV_IDEPNP is not set
    CONFIG_BLK_DEV_IDEDMA_SFF=y

    #
    # PCI IDE chipsets support
    #
    CONFIG_BLK_DEV_IDEPCI=y
    # CONFIG_BLK_DEV_OFFBOARD is not set
    CONFIG_BLK_DEV_GENERIC=m
    # CONFIG_BLK_DEV_OPTI621 is not set
    # CONFIG_BLK_DEV_RZ1000 is not set
    CONFIG_BLK_DEV_IDEDMA_PCI=y
    # CONFIG_BLK_DEV_AEC62XX is not set
    # CONFIG_BLK_DEV_ALI15X3 is not set
    CONFIG_BLK_DEV_AMD74XX=m
    # CONFIG_BLK_DEV_ATIIXP is not set
    # CONFIG_BLK_DEV_CMD64X is not set
    # CONFIG_BLK_DEV_TRIFLEX is not set
    # CONFIG_BLK_DEV_CY82C693 is not set
    # CONFIG_BLK_DEV_CS5520 is not set
    # CONFIG_BLK_DEV_CS5530 is not set
    # CONFIG_BLK_DEV_HPT34X is not set
    # CONFIG_BLK_DEV_HPT366 is not set
    # CONFIG_BLK_DEV_JMICRON is not set
    # CONFIG_BLK_DEV_SC1200 is not set
    # CONFIG_BLK_DEV_PIIX is not set
    # CONFIG_BLK_DEV_IT8213 is not set
    # CONFIG_BLK_DEV_IT821X is not set
    # CONFIG_BLK_DEV_NS87415 is not set
    # CONFIG_BLK_DEV_PDC202XX_OLD is not set
    # CONFIG_BLK_DEV_PDC202XX_NEW is not set
    # CONFIG_BLK_DEV_SVWKS is not set
    # CONFIG_BLK_DEV_SIIMAGE is not set
    # CONFIG_BLK_DEV_SIS5513 is not set
    # CONFIG_BLK_DEV_SLC90E66 is not set
    # CONFIG_BLK_DEV_TRM290 is not set
    # CONFIG_BLK_DEV_VIA82CXXX is not set
    # CONFIG_BLK_DEV_TC86C001 is not set
    CONFIG_BLK_DEV_IDEDMA=y
    # CONFIG_BLK_DEV_HD_ONLY is not set
    # CONFIG_BLK_DEV_HD is not set

    #
    # SCSI device support
    #
    CONFIG_RAID_ATTRS=m
    CONFIG_SCSI=y
    CONFIG_SCSI_DMA=y
    CONFIG_SCSI_TGT=m
    CONFIG_SCSI_NETLINK=y
    CONFIG_SCSI_PROC_FS=y

    #
    # SCSI support type (disk, tape, CD-ROM)
    #
    CONFIG_BLK_DEV_SD=m
    # CONFIG_CHR_DEV_ST is not set
    # CONFIG_CHR_DEV_OSST is not set
    CONFIG_BLK_DEV_SR=m
    CONFIG_BLK_DEV_SR_VENDOR=y
    CONFIG_CHR_DEV_SG=m
    CONFIG_CHR_DEV_SCH=m

    #
    # Some SCSI devices (e.g. CD jukebox) support multiple LUNs
    #
    CONFIG_SCSI_MULTI_LUN=y
    CONFIG_SCSI_CONSTANTS=y
    CONFIG_SCSI_LOGGING=y
    CONFIG_SCSI_SCAN_ASYNC=y
    CONFIG_SCSI_WAIT_SCAN=m

    #
    # SCSI Transports
    #
    CONFIG_SCSI_SPI_ATTRS=m
    CONFIG_SCSI_FC_ATTRS=m
    # CONFIG_SCSI_FC_TGT_ATTRS is not set
    CONFIG_SCSI_ISCSI_ATTRS=m
    CONFIG_SCSI_SAS_ATTRS=m
    CONFIG_SCSI_SAS_LIBSAS=m
    # CONFIG_SCSI_SAS_ATA is not set
    # CONFIG_SCSI_SAS_HOST_SMP is not set
    # CONFIG_SCSI_SAS_LIBSAS_DEBUG is not set
    CONFIG_SCSI_SRP_ATTRS=m
    CONFIG_SCSI_SRP_TGT_ATTRS=y
    CONFIG_SCSI_LOWLEVEL=y
    # CONFIG_ISCSI_TCP is not set
    # CONFIG_BLK_DEV_3W_XXXX_RAID is not set
    # CONFIG_SCSI_3W_9XXX is not set
    # CONFIG_SCSI_ACARD is not set
    # CONFIG_SCSI_AACRAID is not set
    # CONFIG_SCSI_AIC7XXX is not set
    # CONFIG_SCSI_AIC7XXX_OLD is not set
    # CONFIG_SCSI_AIC79XX is not set
    # CONFIG_SCSI_AIC94XX is not set
    # CONFIG_SCSI_DPT_I2O is not set
    # CONFIG_SCSI_ADVANSYS is not set
    # CONFIG_SCSI_ARCMSR is not set
    # CONFIG_MEGARAID_NEWGEN is not set
    # CONFIG_MEGARAID_LEGACY is not set
    # CONFIG_MEGARAID_SAS is not set
    # CONFIG_SCSI_HPTIOP is not set
    # CONFIG_SCSI_BUSLOGIC is not set
    # CONFIG_SCSI_DMX3191D is not set
    # CONFIG_SCSI_EATA is not set
    # CONFIG_SCSI_FUTURE_DOMAIN is not set
    # CONFIG_SCSI_GDTH is not set
    # CONFIG_SCSI_IPS is not set
    # CONFIG_SCSI_INITIO is not set
    # CONFIG_SCSI_INIA100 is not set
    # CONFIG_SCSI_PPA is not set
    # CONFIG_SCSI_IMM is not set
    # CONFIG_SCSI_MVSAS is not set
    # CONFIG_SCSI_STEX is not set
    # CONFIG_SCSI_SYM53C8XX_2 is not set
    # CONFIG_SCSI_IPR is not set
    # CONFIG_SCSI_QLOGIC_1280 is not set
    # CONFIG_SCSI_QLA_FC is not set
    # CONFIG_SCSI_QLA_ISCSI is not set
    # CONFIG_SCSI_LPFC is not set
    # CONFIG_SCSI_DC395x is not set
    # CONFIG_SCSI_DC390T is not set
    # CONFIG_SCSI_DEBUG is not set
    # CONFIG_SCSI_SRP is not set
    CONFIG_ATA=m
    # CONFIG_ATA_NONSTANDARD is not set
    CONFIG_ATA_ACPI=y
    CONFIG_SATA_PMP=y
    CONFIG_SATA_AHCI=m
    CONFIG_SATA_SIL24=m
    CONFIG_ATA_SFF=y
    # CONFIG_SATA_SVW is not set
    # CONFIG_ATA_PIIX is not set
    # CONFIG_SATA_MV is not set
    CONFIG_SATA_NV=m
    # CONFIG_PDC_ADMA is not set
    # CONFIG_SATA_QSTOR is not set
    # CONFIG_SATA_PROMISE is not set
    # CONFIG_SATA_SX4 is not set
    CONFIG_SATA_SIL=m
    # CONFIG_SATA_SIS is not set
    # CONFIG_SATA_ULI is not set
    # CONFIG_SATA_VIA is not set
    # CONFIG_SATA_VITESSE is not set
    # CONFIG_SATA_INIC162X is not set
    # CONFIG_PATA_ACPI is not set
    # CONFIG_PATA_ALI is not set
    CONFIG_PATA_AMD=m
    # CONFIG_PATA_ARTOP is not set
    # CONFIG_PATA_ATIIXP is not set
    # CONFIG_PATA_CMD640_PCI is not set
    # CONFIG_PATA_CMD64X is not set
    # CONFIG_PATA_CS5520 is not set
    # CONFIG_PATA_CS5530 is not set
    # CONFIG_PATA_CYPRESS is not set
    # CONFIG_PATA_EFAR is not set
    # CONFIG_ATA_GENERIC is not set
    # CONFIG_PATA_HPT366 is not set
    # CONFIG_PATA_HPT37X is not set
    # CONFIG_PATA_HPT3X2N is not set
    # CONFIG_PATA_HPT3X3 is not set
    # CONFIG_PATA_IT821X is not set
    # CONFIG_PATA_IT8213 is not set
    # CONFIG_PATA_JMICRON is not set
    # CONFIG_PATA_TRIFLEX is not set
    # CONFIG_PATA_MARVELL is not set
    # CONFIG_PATA_MPIIX is not set
    # CONFIG_PATA_OLDPIIX is not set
    # CONFIG_PATA_NETCELL is not set
    # CONFIG_PATA_NINJA32 is not set
    # CONFIG_PATA_NS87410 is not set
    # CONFIG_PATA_NS87415 is not set
    # CONFIG_PATA_OPTI is not set
    # CONFIG_PATA_OPTIDMA is not set
    # CONFIG_PATA_PDC_OLD is not set
    # CONFIG_PATA_RADISYS is not set
    # CONFIG_PATA_RZ1000 is not set
    # CONFIG_PATA_SC1200 is not set
    # CONFIG_PATA_SERVERWORKS is not set
    # CONFIG_PATA_PDC2027X is not set
    # CONFIG_PATA_SIL680 is not set
    # CONFIG_PATA_SIS is not set
    # CONFIG_PATA_VIA is not set
    # CONFIG_PATA_WINBOND is not set
    # CONFIG_PATA_SCH is not set
    CONFIG_MD=y
    CONFIG_BLK_DEV_MD=y
    CONFIG_MD_LINEAR=m
    CONFIG_MD_RAID0=y
    CONFIG_MD_RAID1=y
    CONFIG_MD_RAID10=m
    CONFIG_MD_RAID456=m
    CONFIG_MD_RAID5_RESHAPE=y
    CONFIG_MD_MULTIPATH=m
    CONFIG_MD_FAULTY=m
    CONFIG_BLK_DEV_DM=m
    # CONFIG_DM_DEBUG is not set
    CONFIG_DM_CRYPT=m
    CONFIG_DM_SNAPSHOT=m
    CONFIG_DM_MIRROR=m
    CONFIG_DM_ZERO=m
    CONFIG_DM_MULTIPATH=m
    CONFIG_DM_MULTIPATH_EMC=m
    # CONFIG_DM_MULTIPATH_RDAC is not set
    # CONFIG_DM_MULTIPATH_HP is not set
    # CONFIG_DM_DELAY is not set
    # CONFIG_DM_UEVENT is not set
    # CONFIG_FUSION is not set

    #
    # IEEE 1394 (FireWire) support
    #

    #
    # Enable only one of the two stacks, unless you know what you are doing
    #
    # CONFIG_FIREWIRE is not set
    # CONFIG_IEEE1394 is not set
    CONFIG_I2O=m
    CONFIG_I2O_LCT_NOTIFY_ON_CHANGES=y
    CONFIG_I2O_EXT_ADAPTEC=y
    CONFIG_I2O_EXT_ADAPTEC_DMA64=y
    CONFIG_I2O_CONFIG=m
    CONFIG_I2O_CONFIG_OLD_IOCTL=y
    CONFIG_I2O_BUS=m
    CONFIG_I2O_BLOCK=m
    CONFIG_I2O_SCSI=m
    CONFIG_I2O_PROC=m
    # CONFIG_MACINTOSH_DRIVERS is not set
    CONFIG_NETDEVICES=y
    # CONFIG_NETDEVICES_MULTIQUEUE is not set
    CONFIG_IFB=m
    CONFIG_DUMMY=m
    CONFIG_BONDING=m
    CONFIG_MACVLAN=m
    CONFIG_EQUALIZER=m
    CONFIG_TUN=m
    CONFIG_VETH=m
    # CONFIG_NET_SB1000 is not set
    # CONFIG_ARCNET is not set
    CONFIG_PHYLIB=m

    #
    # MII PHY device drivers
    #
    CONFIG_MARVELL_PHY=m
    CONFIG_DAVICOM_PHY=m
    CONFIG_QSEMI_PHY=m
    CONFIG_LXT_PHY=m
    CONFIG_CICADA_PHY=m
    CONFIG_VITESSE_PHY=m
    CONFIG_SMSC_PHY=m
    CONFIG_BROADCOM_PHY=m
    CONFIG_ICPLUS_PHY=m
    CONFIG_REALTEK_PHY=m
    # CONFIG_MDIO_BITBANG is not set
    CONFIG_NET_ETHERNET=y
    CONFIG_MII=m
    # CONFIG_HAPPYMEAL is not set
    # CONFIG_SUNGEM is not set
    # CONFIG_CASSINI is not set
    # CONFIG_NET_VENDOR_3COM is not set
    # CONFIG_NET_TULIP is not set
    # CONFIG_HP100 is not set
    # CONFIG_IBM_NEW_EMAC_ZMII is not set
    # CONFIG_IBM_NEW_EMAC_RGMII is not set
    # CONFIG_IBM_NEW_EMAC_TAH is not set
    # CONFIG_IBM_NEW_EMAC_EMAC4 is not set
    CONFIG_NET_PCI=y
    # CONFIG_PCNET32 is not set
    # CONFIG_AMD8111_ETH is not set
    # CONFIG_ADAPTEC_STARFIRE is not set
    # CONFIG_B44 is not set
    CONFIG_FORCEDETH=m
    # CONFIG_FORCEDETH_NAPI is not set
    # CONFIG_EEPRO100 is not set
    # CONFIG_E100 is not set
    # CONFIG_FEALNX is not set
    # CONFIG_NATSEMI is not set
    # CONFIG_NE2K_PCI is not set
    # CONFIG_8139CP is not set
    # CONFIG_8139TOO is not set
    # CONFIG_R6040 is not set
    # CONFIG_SIS900 is not set
    # CONFIG_EPIC100 is not set
    # CONFIG_SUNDANCE is not set
    # CONFIG_VIA_RHINE is not set
    # CONFIG_SC92031 is not set
    # CONFIG_NET_POCKET is not set
    # CONFIG_NETDEV_1000 is not set
    # CONFIG_NETDEV_10000 is not set
    # CONFIG_TR is not set

    #
    # Wireless LAN
    #
    # CONFIG_WLAN_PRE80211 is not set
    # CONFIG_WLAN_80211 is not set
    # CONFIG_IWLWIFI_LEDS is not set

    #
    # USB Network Adapters
    #
    CONFIG_USB_CATC=m
    CONFIG_USB_KAWETH=m
    CONFIG_USB_PEGASUS=m
    CONFIG_USB_RTL8150=m
    CONFIG_USB_USBNET=m
    CONFIG_USB_NET_AX8817X=m
    CONFIG_USB_NET_CDCETHER=m
    CONFIG_USB_NET_DM9601=m
    CONFIG_USB_NET_GL620A=m
    CONFIG_USB_NET_NET1080=m
    CONFIG_USB_NET_PLUSB=m
    CONFIG_USB_NET_MCS7830=m
    CONFIG_USB_NET_RNDIS_HOST=m
    CONFIG_USB_NET_CDC_SUBSET=m
    CONFIG_USB_ALI_M5632=y
    CONFIG_USB_AN2720=y
    CONFIG_USB_BELKIN=y
    CONFIG_USB_ARMLINUX=y
    CONFIG_USB_EPSON2888=y
    # CONFIG_USB_KC2190 is not set
    CONFIG_USB_NET_ZAURUS=m
    # CONFIG_WAN is not set
    # CONFIG_ATM_DRIVERS is not set
    # CONFIG_FDDI is not set
    # CONFIG_HIPPI is not set
    # CONFIG_PLIP is not set
    CONFIG_PPP=m
    CONFIG_PPP_MULTILINK=y
    CONFIG_PPP_FILTER=y
    CONFIG_PPP_ASYNC=m
    CONFIG_PPP_SYNC_TTY=m
    CONFIG_PPP_DEFLATE=m
    CONFIG_PPP_BSDCOMP=m
    CONFIG_PPP_MPPE=m
    CONFIG_PPPOE=m
    CONFIG_PPPOATM=m
    # CONFIG_PPPOL2TP is not set
    # CONFIG_SLIP is not set
    CONFIG_SLHC=m
    # CONFIG_NET_FC is not set
    CONFIG_NETCONSOLE=m
    # CONFIG_NETCONSOLE_DYNAMIC is not set
    CONFIG_NETPOLL=y
    # CONFIG_NETPOLL_TRAP is not set
    CONFIG_NET_POLL_CONTROLLER=y
    # CONFIG_ISDN is not set
    # CONFIG_PHONE is not set

    #
    # Input device support
    #
    CONFIG_INPUT=y
    CONFIG_INPUT_FF_MEMLESS=m
    CONFIG_INPUT_POLLDEV=m

    #
    # Userland interfaces
    #
    CONFIG_INPUT_MOUSEDEV=y
    CONFIG_INPUT_MOUSEDEV_PSAUX=y
    CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
    CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
    CONFIG_INPUT_JOYDEV=m
    CONFIG_INPUT_EVDEV=m
    # CONFIG_INPUT_EVBUG is not set

    #
    # Input Device Drivers
    #
    CONFIG_INPUT_KEYBOARD=y
    CONFIG_KEYBOARD_ATKBD=y
    CONFIG_KEYBOARD_SUNKBD=m
    CONFIG_KEYBOARD_LKKBD=m
    CONFIG_KEYBOARD_XTKBD=m
    CONFIG_KEYBOARD_NEWTON=m
    CONFIG_KEYBOARD_STOWAWAY=m
    CONFIG_INPUT_MOUSE=y
    CONFIG_MOUSE_PS2=m
    CONFIG_MOUSE_PS2_ALPS=y
    CONFIG_MOUSE_PS2_LOGIPS2PP=y
    CONFIG_MOUSE_PS2_SYNAPTICS=y
    CONFIG_MOUSE_PS2_LIFEBOOK=y
    CONFIG_MOUSE_PS2_TRACKPOINT=y
    # CONFIG_MOUSE_PS2_TOUCHKIT is not set
    CONFIG_MOUSE_SERIAL=m
    # CONFIG_MOUSE_APPLETOUCH is not set
    CONFIG_MOUSE_VSXXXAA=m
    CONFIG_INPUT_JOYSTICK=y
    CONFIG_JOYSTICK_ANALOG=m
    CONFIG_JOYSTICK_A3D=m
    CONFIG_JOYSTICK_ADI=m
    CONFIG_JOYSTICK_COBRA=m
    CONFIG_JOYSTICK_GF2K=m
    CONFIG_JOYSTICK_GRIP=m
    CONFIG_JOYSTICK_GRIP_MP=m
    CONFIG_JOYSTICK_GUILLEMOT=m
    CONFIG_JOYSTICK_INTERACT=m
    CONFIG_JOYSTICK_SIDEWINDER=m
    CONFIG_JOYSTICK_TMDC=m
    CONFIG_JOYSTICK_IFORCE=m
    CONFIG_JOYSTICK_IFORCE_USB=y
    CONFIG_JOYSTICK_IFORCE_232=y
    CONFIG_JOYSTICK_WARRIOR=m
    CONFIG_JOYSTICK_MAGELLAN=m
    CONFIG_JOYSTICK_SPACEORB=m
    CONFIG_JOYSTICK_SPACEBALL=m
    CONFIG_JOYSTICK_STINGER=m
    CONFIG_JOYSTICK_TWIDJOY=m
    # CONFIG_JOYSTICK_ZHENHUA is not set
    CONFIG_JOYSTICK_DB9=m
    CONFIG_JOYSTICK_GAMECON=m
    CONFIG_JOYSTICK_TURBOGRAFX=m
    CONFIG_JOYSTICK_JOYDUMP=m
    # CONFIG_JOYSTICK_XPAD is not set
    # CONFIG_INPUT_TABLET is not set
    # CONFIG_INPUT_TOUCHSCREEN is not set
    CONFIG_INPUT_MISC=y
    CONFIG_INPUT_PCSPKR=m
    # CONFIG_INPUT_ATLAS_BTNS is not set
    # CONFIG_INPUT_ATI_REMOTE is not set
    # CONFIG_INPUT_ATI_REMOTE2 is not set
    # CONFIG_INPUT_KEYSPAN_REMOTE is not set
    # CONFIG_INPUT_POWERMATE is not set
    # CONFIG_INPUT_YEALINK is not set
    CONFIG_INPUT_UINPUT=m

    #
    # Hardware I/O ports
    #
    CONFIG_SERIO=y
    CONFIG_SERIO_I8042=y
    CONFIG_SERIO_SERPORT=m
    CONFIG_SERIO_CT82C710=m
    CONFIG_SERIO_PARKBD=m
    CONFIG_SERIO_PCIPS2=m
    CONFIG_SERIO_LIBPS2=y
    CONFIG_SERIO_RAW=m
    CONFIG_GAMEPORT=m
    CONFIG_GAMEPORT_NS558=m
    CONFIG_GAMEPORT_L4=m
    CONFIG_GAMEPORT_EMU10K1=m
    CONFIG_GAMEPORT_FM801=m

    #
    # Character devices
    #
    CONFIG_VT=y
    CONFIG_VT_CONSOLE=y
    CONFIG_HW_CONSOLE=y
    # CONFIG_VT_HW_CONSOLE_BINDING is not set
    CONFIG_DEVKMEM=y
    # CONFIG_SERIAL_NONSTANDARD is not set
    # CONFIG_NOZOMI is not set

    #
    # Serial drivers
    #
    CONFIG_SERIAL_8250=y
    CONFIG_SERIAL_8250_CONSOLE=y
    CONFIG_FIX_EARLYCON_MEM=y
    CONFIG_SERIAL_8250_PCI=y
    CONFIG_SERIAL_8250_PNP=y
    CONFIG_SERIAL_8250_NR_UARTS=16
    CONFIG_SERIAL_8250_RUNTIME_UARTS=4
    CONFIG_SERIAL_8250_EXTENDED=y
    CONFIG_SERIAL_8250_MANY_PORTS=y
    CONFIG_SERIAL_8250_SHARE_IRQ=y
    # CONFIG_SERIAL_8250_DETECT_IRQ is not set
    CONFIG_SERIAL_8250_RSA=y

    #
    # Non-8250 serial port support
    #
    CONFIG_SERIAL_CORE=y
    CONFIG_SERIAL_CORE_CONSOLE=y
    CONFIG_SERIAL_JSM=m
    CONFIG_UNIX98_PTYS=y
    # CONFIG_LEGACY_PTYS is not set
    CONFIG_PRINTER=m
    # CONFIG_LP_CONSOLE is not set
    CONFIG_PPDEV=m
    CONFIG_IPMI_HANDLER=m
    # CONFIG_IPMI_PANIC_EVENT is not set
    CONFIG_IPMI_DEVICE_INTERFACE=m
    CONFIG_IPMI_SI=m
    CONFIG_IPMI_WATCHDOG=m
    CONFIG_IPMI_POWEROFF=m
    CONFIG_HW_RANDOM=y
    CONFIG_HW_RANDOM_INTEL=m
    CONFIG_HW_RANDOM_AMD=m
    CONFIG_NVRAM=m
    # CONFIG_R3964 is not set
    # CONFIG_APPLICOM is not set
    # CONFIG_MWAVE is not set
    # CONFIG_PC8736x_GPIO is not set
    # CONFIG_RAW_DRIVER is not set
    CONFIG_HPET=y
    CONFIG_HPET_MMAP=y
    CONFIG_HANGCHECK_TIMER=m
    # CONFIG_TCG_TPM is not set
    # CONFIG_TELCLOCK is not set
    CONFIG_DEVPORT=y
    CONFIG_I2C=m
    CONFIG_I2C_BOARDINFO=y
    CONFIG_I2C_CHARDEV=m
    CONFIG_I2C_ALGOBIT=m

    #
    # I2C Hardware Bus support
    #
    CONFIG_I2C_ALI1535=m
    CONFIG_I2C_ALI1563=m
    CONFIG_I2C_ALI15X3=m
    CONFIG_I2C_AMD756=m
    CONFIG_I2C_AMD756_S4882=m
    CONFIG_I2C_AMD8111=m
    CONFIG_I2C_I801=m
    CONFIG_I2C_I810=m
    CONFIG_I2C_PIIX4=m
    CONFIG_I2C_NFORCE2=m
    CONFIG_I2C_OCORES=m
    CONFIG_I2C_PARPORT=m
    CONFIG_I2C_PARPORT_LIGHT=m
    CONFIG_I2C_PROSAVAGE=m
    CONFIG_I2C_SAVAGE4=m
    # CONFIG_I2C_SIMTEC is not set
    CONFIG_I2C_SIS5595=m
    CONFIG_I2C_SIS630=m
    CONFIG_I2C_SIS96X=m
    # CONFIG_I2C_TAOS_EVM is not set
    CONFIG_I2C_STUB=m
    # CONFIG_I2C_TINY_USB is not set
    CONFIG_I2C_VIA=m
    CONFIG_I2C_VIAPRO=m
    CONFIG_I2C_VOODOO3=m
    # CONFIG_I2C_PCA_PLATFORM is not set

    #
    # Miscellaneous I2C Chip support
    #
    CONFIG_DS1682=m
    CONFIG_SENSORS_EEPROM=m
    CONFIG_SENSORS_PCF8574=m
    CONFIG_PCF8575=m
    CONFIG_SENSORS_PCF8591=m
    CONFIG_SENSORS_MAX6875=m
    # CONFIG_SENSORS_TSL2550 is not set
    # CONFIG_I2C_DEBUG_CORE is not set
    # CONFIG_I2C_DEBUG_ALGO is not set
    # CONFIG_I2C_DEBUG_BUS is not set
    # CONFIG_I2C_DEBUG_CHIP is not set
    # CONFIG_SPI is not set
    # CONFIG_W1 is not set
    CONFIG_POWER_SUPPLY=y
    # CONFIG_POWER_SUPPLY_DEBUG is not set
    # CONFIG_PDA_POWER is not set
    # CONFIG_BATTERY_DS2760 is not set
    CONFIG_HWMON=y
    CONFIG_HWMON_VID=m
    CONFIG_SENSORS_ABITUGURU=m
    # CONFIG_SENSORS_ABITUGURU3 is not set
    # CONFIG_SENSORS_AD7418 is not set
    CONFIG_SENSORS_ADM1021=m
    CONFIG_SENSORS_ADM1025=m
    CONFIG_SENSORS_ADM1026=m
    # CONFIG_SENSORS_ADM1029 is not set
    CONFIG_SENSORS_ADM1031=m
    CONFIG_SENSORS_ADM9240=m
    CONFIG_SENSORS_ADT7470=m
    CONFIG_SENSORS_ADT7473=m
    CONFIG_SENSORS_K8TEMP=m
    CONFIG_SENSORS_ASB100=m
    CONFIG_SENSORS_ATXP1=m
    CONFIG_SENSORS_DS1621=m
    CONFIG_SENSORS_I5K_AMB=m
    CONFIG_SENSORS_F71805F=m
    CONFIG_SENSORS_F71882FG=m
    CONFIG_SENSORS_F75375S=m
    CONFIG_SENSORS_FSCHER=m
    CONFIG_SENSORS_FSCPOS=m
    CONFIG_SENSORS_FSCHMD=m
    CONFIG_SENSORS_GL518SM=m
    CONFIG_SENSORS_GL520SM=m
    # CONFIG_SENSORS_CORETEMP is not set
    # CONFIG_SENSORS_IBMAEM is not set
    CONFIG_SENSORS_IBMPEX=m
    CONFIG_SENSORS_IT87=m
    CONFIG_SENSORS_LM63=m
    CONFIG_SENSORS_LM75=m
    CONFIG_SENSORS_LM77=m
    CONFIG_SENSORS_LM78=m
    CONFIG_SENSORS_LM80=m
    CONFIG_SENSORS_LM83=m
    CONFIG_SENSORS_LM85=m
    CONFIG_SENSORS_LM87=m
    CONFIG_SENSORS_LM90=m
    CONFIG_SENSORS_LM92=m
    CONFIG_SENSORS_LM93=m
    CONFIG_SENSORS_MAX1619=m
    # CONFIG_SENSORS_MAX6650 is not set
    CONFIG_SENSORS_PC87360=m
    CONFIG_SENSORS_PC87427=m
    CONFIG_SENSORS_SIS5595=m
    CONFIG_SENSORS_DME1737=m
    CONFIG_SENSORS_SMSC47M1=m
    CONFIG_SENSORS_SMSC47M192=m
    CONFIG_SENSORS_SMSC47B397=m
    CONFIG_SENSORS_ADS7828=m
    CONFIG_SENSORS_THMC50=m
    CONFIG_SENSORS_VIA686A=m
    CONFIG_SENSORS_VT1211=m
    CONFIG_SENSORS_VT8231=m
    CONFIG_SENSORS_W83781D=m
    CONFIG_SENSORS_W83791D=m
    CONFIG_SENSORS_W83792D=m
    CONFIG_SENSORS_W83793=m
    CONFIG_SENSORS_W83L785TS=m
    CONFIG_SENSORS_W83L786NG=m
    CONFIG_SENSORS_W83627HF=m
    CONFIG_SENSORS_W83627EHF=m
    CONFIG_SENSORS_HDAPS=m
    # CONFIG_SENSORS_APPLESMC is not set
    # CONFIG_HWMON_DEBUG_CHIP is not set
    CONFIG_THERMAL=y
    # CONFIG_THERMAL_HWMON is not set
    CONFIG_WATCHDOG=y
    # CONFIG_WATCHDOG_NOWAYOUT is not set

    #
    # Watchdog Device Drivers
    #
    CONFIG_SOFT_WATCHDOG=m
    CONFIG_ACQUIRE_WDT=m
    CONFIG_ADVANTECH_WDT=m
    CONFIG_ALIM1535_WDT=m
    CONFIG_ALIM7101_WDT=m
    CONFIG_SC520_WDT=m
    CONFIG_EUROTECH_WDT=m
    CONFIG_IB700_WDT=m
    CONFIG_IBMASR=m
    CONFIG_WAFER_WDT=m
    CONFIG_I6300ESB_WDT=m
    CONFIG_ITCO_WDT=m
    # CONFIG_ITCO_VENDOR_SUPPORT is not set
    CONFIG_IT8712F_WDT=m
    # CONFIG_HP_WATCHDOG is not set
    CONFIG_SC1200_WDT=m
    CONFIG_PC87413_WDT=m
    CONFIG_60XX_WDT=m
    CONFIG_SBC8360_WDT=m
    CONFIG_CPU5_WDT=m
    CONFIG_SMSC37B787_WDT=m
    CONFIG_W83627HF_WDT=m
    CONFIG_W83697HF_WDT=m
    CONFIG_W83877F_WDT=m
    CONFIG_W83977F_WDT=m
    CONFIG_MACHZ_WDT=m
    CONFIG_SBC_EPX_C3_WATCHDOG=m

    #
    # PCI-based Watchdog Cards
    #
    CONFIG_PCIPCWATCHDOG=m
    CONFIG_WDTPCI=m
    CONFIG_WDT_501_PCI=y

    #
    # USB-based Watchdog Cards
    #
    CONFIG_USBPCWATCHDOG=m

    #
    # Sonics Silicon Backplane
    #
    CONFIG_SSB_POSSIBLE=y
    # CONFIG_SSB is not set

    #
    # Multifunction device drivers
    #
    # CONFIG_MFD_SM501 is not set
    # CONFIG_HTC_PASIC3 is not set

    #
    # Multimedia devices
    #

    #
    # Multimedia core support
    #
    CONFIG_VIDEO_DEV=m
    CONFIG_VIDEO_V4L2_COMMON=m
    CONFIG_VIDEO_ALLOW_V4L1=y
    CONFIG_VIDEO_V4L1_COMPAT=y
    # CONFIG_DVB_CORE is not set
    CONFIG_VIDEO_MEDIA=m

    #
    # Multimedia drivers
    #
    # CONFIG_MEDIA_ATTACH is not set
    CONFIG_MEDIA_TUNER=m
    # CONFIG_MEDIA_TUNER_CUSTOMIZE is not set
    CONFIG_MEDIA_TUNER_SIMPLE=m
    CONFIG_MEDIA_TUNER_TDA8290=m
    CONFIG_MEDIA_TUNER_TDA9887=m
    CONFIG_MEDIA_TUNER_TEA5761=m
    CONFIG_MEDIA_TUNER_TEA5767=m
    CONFIG_MEDIA_TUNER_MT20XX=m
    CONFIG_MEDIA_TUNER_XC2028=m
    CONFIG_MEDIA_TUNER_XC5000=m
    CONFIG_VIDEO_V4L2=m
    CONFIG_VIDEO_V4L1=m
    # CONFIG_VIDEO_CAPTURE_DRIVERS is not set
    # CONFIG_RADIO_ADAPTERS is not set
    # CONFIG_DAB is not set

    #
    # Graphics support
    #
    CONFIG_AGP=y
    CONFIG_AGP_AMD64=y
    # CONFIG_AGP_INTEL is not set
    # CONFIG_AGP_SIS is not set
    # CONFIG_AGP_VIA is not set
    # CONFIG_DRM is not set
    CONFIG_VGASTATE=m
    CONFIG_VIDEO_OUTPUT_CONTROL=m
    CONFIG_FB=y
    CONFIG_FIRMWARE_EDID=y
    # CONFIG_FB_DDC is not set
    CONFIG_FB_CFB_FILLRECT=y
    CONFIG_FB_CFB_COPYAREA=y
    CONFIG_FB_CFB_IMAGEBLIT=y
    # CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
    # CONFIG_FB_SYS_FILLRECT is not set
    # CONFIG_FB_SYS_COPYAREA is not set
    # CONFIG_FB_SYS_IMAGEBLIT is not set
    # CONFIG_FB_FOREIGN_ENDIAN is not set
    # CONFIG_FB_SYS_FOPS is not set
    # CONFIG_FB_SVGALIB is not set
    # CONFIG_FB_MACMODES is not set
    # CONFIG_FB_BACKLIGHT is not set
    CONFIG_FB_MODE_HELPERS=y
    CONFIG_FB_TILEBLITTING=y

    #
    # Frame buffer hardware drivers
    #
    # CONFIG_FB_CIRRUS is not set
    # CONFIG_FB_PM2 is not set
    # CONFIG_FB_CYBER2000 is not set
    # CONFIG_FB_ARC is not set
    # CONFIG_FB_ASILIANT is not set
    # CONFIG_FB_IMSTT is not set
    CONFIG_FB_VGA16=m
    # CONFIG_FB_UVESA is not set
    CONFIG_FB_VESA=y
    # CONFIG_FB_EFI is not set
    # CONFIG_FB_N411 is not set
    # CONFIG_FB_HGA is not set
    # CONFIG_FB_S1D13XXX is not set
    # CONFIG_FB_NVIDIA is not set
    # CONFIG_FB_RIVA is not set
    # CONFIG_FB_LE80578 is not set
    # CONFIG_FB_INTEL is not set
    # CONFIG_FB_MATROX is not set
    # CONFIG_FB_RADEON is not set
    # CONFIG_FB_ATY128 is not set
    # CONFIG_FB_ATY is not set
    # CONFIG_FB_S3 is not set
    # CONFIG_FB_SAVAGE is not set
    # CONFIG_FB_SIS is not set
    # CONFIG_FB_NEOMAGIC is not set
    # CONFIG_FB_KYRO is not set
    # CONFIG_FB_3DFX is not set
    # CONFIG_FB_VOODOO1 is not set
    # CONFIG_FB_VT8623 is not set
    # CONFIG_FB_TRIDENT is not set
    # CONFIG_FB_ARK is not set
    # CONFIG_FB_PM3 is not set
    # CONFIG_FB_GEODE is not set
    # CONFIG_FB_VIRTUAL is not set
    CONFIG_BACKLIGHT_LCD_SUPPORT=y
    # CONFIG_LCD_CLASS_DEVICE is not set
    CONFIG_BACKLIGHT_CLASS_DEVICE=y
    # CONFIG_BACKLIGHT_CORGI is not set
    # CONFIG_BACKLIGHT_PROGEAR is not set

    #
    # Display device support
    #
    # CONFIG_DISPLAY_SUPPORT is not set

    #
    # Console display driver support
    #
    CONFIG_VGA_CONSOLE=y
    # CONFIG_VGACON_SOFT_SCROLLBACK is not set
    CONFIG_VIDEO_SELECT=y
    CONFIG_DUMMY_CONSOLE=y
    CONFIG_FRAMEBUFFER_CONSOLE=y
    # CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set
    CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
    # CONFIG_FONTS is not set
    CONFIG_FONT_8x8=y
    CONFIG_FONT_8x16=y
    # CONFIG_LOGO is not set

    #
    # Sound
    #
    CONFIG_SOUND=m

    #
    # Advanced Linux Sound Architecture
    #
    CONFIG_SND=m
    CONFIG_SND_TIMER=m
    CONFIG_SND_PCM=m
    CONFIG_SND_HWDEP=m
    CONFIG_SND_RAWMIDI=m
    CONFIG_SND_SEQUENCER=m
    CONFIG_SND_SEQ_DUMMY=m
    CONFIG_SND_OSSEMUL=y
    CONFIG_SND_MIXER_OSS=m
    CONFIG_SND_PCM_OSS=m
    CONFIG_SND_PCM_OSS_PLUGINS=y
    CONFIG_SND_SEQUENCER_OSS=y
    # CONFIG_SND_DYNAMIC_MINORS is not set
    CONFIG_SND_SUPPORT_OLD_API=y
    CONFIG_SND_VERBOSE_PROCFS=y
    # CONFIG_SND_VERBOSE_PRINTK is not set
    # CONFIG_SND_DEBUG is not set
    CONFIG_SND_VMASTER=y

    #
    # Generic devices
    #
    # CONFIG_SND_PCSP is not set
    CONFIG_SND_MPU401_UART=m
    CONFIG_SND_OPL3_LIB=m
    CONFIG_SND_VX_LIB=m
    CONFIG_SND_AC97_CODEC=m
    CONFIG_SND_DUMMY=m
    CONFIG_SND_VIRMIDI=m
    # CONFIG_SND_MTPAV is not set
    # CONFIG_SND_MTS64 is not set
    CONFIG_SND_SERIAL_U16550=m
    CONFIG_SND_MPU401=m
    # CONFIG_SND_PORTMAN2X4 is not set
    CONFIG_SND_SB_COMMON=m

    #
    # PCI devices
    #
    CONFIG_SND_AD1889=m
    CONFIG_SND_ALS300=m
    CONFIG_SND_ALS4000=m
    CONFIG_SND_ALI5451=m
    CONFIG_SND_ATIIXP=m
    CONFIG_SND_ATIIXP_MODEM=m
    CONFIG_SND_AU8810=m
    CONFIG_SND_AU8820=m
    CONFIG_SND_AU8830=m
    # CONFIG_SND_AW2 is not set
    CONFIG_SND_AZT3328=m
    CONFIG_SND_BT87X=m
    # CONFIG_SND_BT87X_OVERCLOCK is not set
    CONFIG_SND_CA0106=m
    CONFIG_SND_CMIPCI=m
    # CONFIG_SND_OXYGEN is not set
    CONFIG_SND_CS4281=m
    CONFIG_SND_CS46XX=m
    CONFIG_SND_CS46XX_NEW_DSP=y
    # CONFIG_SND_CS5530 is not set
    CONFIG_SND_DARLA20=m
    CONFIG_SND_GINA20=m
    CONFIG_SND_LAYLA20=m
    CONFIG_SND_DARLA24=m
    CONFIG_SND_GINA24=m
    CONFIG_SND_LAYLA24=m
    CONFIG_SND_MONA=m
    CONFIG_SND_MIA=m
    CONFIG_SND_ECHO3G=m
    CONFIG_SND_INDIGO=m
    CONFIG_SND_INDIGOIO=m
    CONFIG_SND_INDIGODJ=m
    CONFIG_SND_EMU10K1=m
    CONFIG_SND_EMU10K1X=m
    CONFIG_SND_ENS1370=m
    CONFIG_SND_ENS1371=m
    CONFIG_SND_ES1938=m
    CONFIG_SND_ES1968=m
    CONFIG_SND_FM801=m
    CONFIG_SND_FM801_TEA575X_BOOL=y
    CONFIG_SND_FM801_TEA575X=m
    CONFIG_SND_HDA_INTEL=m
    CONFIG_SND_HDA_HWDEP=y
    CONFIG_SND_HDA_CODEC_REALTEK=y
    CONFIG_SND_HDA_CODEC_ANALOG=y
    CONFIG_SND_HDA_CODEC_SIGMATEL=y
    CONFIG_SND_HDA_CODEC_VIA=y
    CONFIG_SND_HDA_CODEC_ATIHDMI=y
    CONFIG_SND_HDA_CODEC_CONEXANT=y
    CONFIG_SND_HDA_CODEC_CMEDIA=y
    CONFIG_SND_HDA_CODEC_SI3054=y
    CONFIG_SND_HDA_GENERIC=y
    CONFIG_SND_HDA_POWER_SAVE=y
    CONFIG_SND_HDA_POWER_SAVE_DEFAULT=200
    CONFIG_SND_HDSP=m
    CONFIG_SND_HDSPM=m
    # CONFIG_SND_HIFIER is not set
    CONFIG_SND_ICE1712=m
    CONFIG_SND_ICE1724=m
    CONFIG_SND_INTEL8X0=m
    CONFIG_SND_INTEL8X0M=m
    CONFIG_SND_KORG1212=m
    CONFIG_SND_KORG1212_FIRMWARE_IN_KERNEL=y
    CONFIG_SND_MAESTRO3=m
    CONFIG_SND_MAESTRO3_FIRMWARE_IN_KERNEL=y
    CONFIG_SND_MIXART=m
    CONFIG_SND_NM256=m
    CONFIG_SND_PCXHR=m
    CONFIG_SND_RIPTIDE=m
    CONFIG_SND_RME32=m
    CONFIG_SND_RME96=m
    CONFIG_SND_RME9652=m
    CONFIG_SND_SONICVIBES=m
    CONFIG_SND_TRIDENT=m
    CONFIG_SND_VIA82XX=m
    CONFIG_SND_VIA82XX_MODEM=m
    # CONFIG_SND_VIRTUOSO is not set
    CONFIG_SND_VX222=m
    CONFIG_SND_YMFPCI=m
    CONFIG_SND_YMFPCI_FIRMWARE_IN_KERNEL=y
    CONFIG_SND_AC97_POWER_SAVE=y
    CONFIG_SND_AC97_POWER_SAVE_DEFAULT=200

    #
    # USB devices
    #
    CONFIG_SND_USB_AUDIO=m
    CONFIG_SND_USB_USX2Y=m
    # CONFIG_SND_USB_CAIAQ is not set

    #
    # System on Chip audio support
    #
    # CONFIG_SND_SOC is not set

    #
    # ALSA SoC audio for Freescale SOCs
    #

    #
    # SoC Audio for the Texas Instruments OMAP
    #

    #
    # Open Sound System
    #
    # CONFIG_SOUND_PRIME is not set
    CONFIG_AC97_BUS=m
    CONFIG_HID_SUPPORT=y
    CONFIG_HID=m
    # CONFIG_HID_DEBUG is not set
    CONFIG_HIDRAW=y

    #
    # USB Input Devices
    #
    CONFIG_USB_HID=m
    CONFIG_USB_HIDINPUT_POWERBOOK=y
    # CONFIG_HID_FF is not set
    CONFIG_USB_HIDDEV=y

    #
    # USB HID Boot Protocol drivers
    #
    CONFIG_USB_KBD=m
    CONFIG_USB_MOUSE=m
    CONFIG_USB_SUPPORT=y
    CONFIG_USB_ARCH_HAS_HCD=y
    CONFIG_USB_ARCH_HAS_OHCI=y
    CONFIG_USB_ARCH_HAS_EHCI=y
    CONFIG_USB=y
    # CONFIG_USB_DEBUG is not set
    CONFIG_USB_ANNOUNCE_NEW_DEVICES=y

    #
    # Miscellaneous USB options
    #
    CONFIG_USB_DEVICEFS=y
    CONFIG_USB_DEVICE_CLASS=y
    # CONFIG_USB_DYNAMIC_MINORS is not set
    # CONFIG_USB_SUSPEND is not set
    # CONFIG_USB_OTG is not set

    #
    # USB Host Controller Drivers
    #
    # CONFIG_USB_C67X00_HCD is not set
    CONFIG_USB_EHCI_HCD=m
    CONFIG_USB_EHCI_ROOT_HUB_TT=y
    # CONFIG_USB_EHCI_TT_NEWSCHED is not set
    CONFIG_USB_ISP116X_HCD=m
    # CONFIG_USB_ISP1760_HCD is not set
    CONFIG_USB_OHCI_HCD=m
    # CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
    # CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
    CONFIG_USB_OHCI_LITTLE_ENDIAN=y
    CONFIG_USB_UHCI_HCD=m
    CONFIG_USB_U132_HCD=m
    CONFIG_USB_SL811_HCD=m
    # CONFIG_USB_R8A66597_HCD is not set

    #
    # USB Device Class drivers
    #
    CONFIG_USB_ACM=m
    CONFIG_USB_PRINTER=m
    # CONFIG_USB_WDM is not set

    #
    # NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support'
    #

    #
    # may also be needed; see USB_STORAGE Help for more information
    #
    CONFIG_USB_STORAGE=m
    # CONFIG_USB_STORAGE_DEBUG is not set
    CONFIG_USB_STORAGE_DATAFAB=y
    CONFIG_USB_STORAGE_FREECOM=y
    CONFIG_USB_STORAGE_ISD200=y
    CONFIG_USB_STORAGE_DPCM=y
    CONFIG_USB_STORAGE_USBAT=y
    CONFIG_USB_STORAGE_SDDR09=y
    CONFIG_USB_STORAGE_SDDR55=y
    CONFIG_USB_STORAGE_JUMPSHOT=y
    CONFIG_USB_STORAGE_ALAUDA=y
    # CONFIG_USB_STORAGE_ONETOUCH is not set
    CONFIG_USB_STORAGE_KARMA=y
    # CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
    # CONFIG_USB_LIBUSUAL is not set

    #
    # USB Imaging devices
    #
    CONFIG_USB_MDC800=m
    CONFIG_USB_MICROTEK=m
    CONFIG_USB_MON=y

    #
    # USB port drivers
    #
    CONFIG_USB_USS720=m
    CONFIG_USB_SERIAL=m
    CONFIG_USB_EZUSB=y
    CONFIG_USB_SERIAL_GENERIC=y
    CONFIG_USB_SERIAL_AIRCABLE=m
    CONFIG_USB_SERIAL_AIRPRIME=m
    CONFIG_USB_SERIAL_ARK3116=m
    CONFIG_USB_SERIAL_BELKIN=m
    # CONFIG_USB_SERIAL_CH341 is not set
    CONFIG_USB_SERIAL_WHITEHEAT=m
    CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
    CONFIG_USB_SERIAL_CP2101=m
    CONFIG_USB_SERIAL_CYPRESS_M8=m
    CONFIG_USB_SERIAL_EMPEG=m
    CONFIG_USB_SERIAL_FTDI_SIO=m
    CONFIG_USB_SERIAL_FUNSOFT=m
    CONFIG_USB_SERIAL_VISOR=m
    CONFIG_USB_SERIAL_IPAQ=m
    CONFIG_USB_SERIAL_IR=m
    CONFIG_USB_SERIAL_EDGEPORT=m
    CONFIG_USB_SERIAL_EDGEPORT_TI=m
    CONFIG_USB_SERIAL_GARMIN=m
    CONFIG_USB_SERIAL_IPW=m
    # CONFIG_USB_SERIAL_IUU is not set
    # CONFIG_USB_SERIAL_KEYSPAN_PDA is not set
    # CONFIG_USB_SERIAL_KEYSPAN is not set
    CONFIG_USB_SERIAL_KLSI=m
    CONFIG_USB_SERIAL_KOBIL_SCT=m
    CONFIG_USB_SERIAL_MCT_U232=m
    CONFIG_USB_SERIAL_MOS7720=m
    CONFIG_USB_SERIAL_MOS7840=m
    # CONFIG_USB_SERIAL_MOTOROLA is not set
    CONFIG_USB_SERIAL_NAVMAN=m
    CONFIG_USB_SERIAL_PL2303=m
    # CONFIG_USB_SERIAL_OTI6858 is not set
    # CONFIG_USB_SERIAL_SPCP8X5 is not set
    CONFIG_USB_SERIAL_HP4X=m
    CONFIG_USB_SERIAL_SAFE=m
    # CONFIG_USB_SERIAL_SAFE_PADDED is not set
    CONFIG_USB_SERIAL_SIERRAWIRELESS=m
    CONFIG_USB_SERIAL_TI=m
    CONFIG_USB_SERIAL_CYBERJACK=m
    CONFIG_USB_SERIAL_XIRCOM=m
    CONFIG_USB_SERIAL_OPTION=m
    CONFIG_USB_SERIAL_OMNINET=m
    CONFIG_USB_SERIAL_DEBUG=m

    #
    # USB Miscellaneous drivers
    #
    # CONFIG_USB_EMI62 is not set
    # CONFIG_USB_EMI26 is not set
    CONFIG_USB_ADUTUX=m
    CONFIG_USB_AUERSWALD=m
    CONFIG_USB_RIO500=m
    CONFIG_USB_LEGOTOWER=m
    CONFIG_USB_LCD=m
    # CONFIG_USB_BERRY_CHARGE is not set
    CONFIG_USB_LED=m
    CONFIG_USB_CYPRESS_CY7C63=m
    CONFIG_USB_CYTHERM=m
    CONFIG_USB_PHIDGET=m
    CONFIG_USB_PHIDGETKIT=m
    CONFIG_USB_PHIDGETMOTORCONTROL=m
    CONFIG_USB_PHIDGETSERVO=m
    CONFIG_USB_IDMOUSE=m
    CONFIG_USB_FTDI_ELAN=m
    CONFIG_USB_APPLEDISPLAY=m
    CONFIG_USB_SISUSBVGA=m
    CONFIG_USB_SISUSBVGA_CON=y
    CONFIG_USB_LD=m
    CONFIG_USB_TRANCEVIBRATOR=m
    CONFIG_USB_IOWARRIOR=m
    CONFIG_USB_TEST=m
    # CONFIG_USB_ISIGHTFW is not set
    CONFIG_USB_ATM=m
    CONFIG_USB_SPEEDTOUCH=m
    CONFIG_USB_CXACRU=m
    CONFIG_USB_UEAGLEATM=m
    CONFIG_USB_XUSBATM=m
    CONFIG_USB_GADGET=m
    # CONFIG_USB_GADGET_DEBUG is not set
    # CONFIG_USB_GADGET_DEBUG_FILES is not set
    # CONFIG_USB_GADGET_DEBUG_FS is not set
    CONFIG_USB_GADGET_SELECTED=y
    # CONFIG_USB_GADGET_AMD5536UDC is not set
    # CONFIG_USB_GADGET_ATMEL_USBA is not set
    # CONFIG_USB_GADGET_FSL_USB2 is not set
    CONFIG_USB_GADGET_NET2280=y
    CONFIG_USB_NET2280=m
    # CONFIG_USB_GADGET_PXA2XX is not set
    # CONFIG_USB_GADGET_M66592 is not set
    # CONFIG_USB_GADGET_PXA27X is not set
    # CONFIG_USB_GADGET_GOKU is not set
    # CONFIG_USB_GADGET_LH7A40X is not set
    # CONFIG_USB_GADGET_OMAP is not set
    # CONFIG_USB_GADGET_S3C2410 is not set
    # CONFIG_USB_GADGET_AT91 is not set
    # CONFIG_USB_GADGET_DUMMY_HCD is not set
    CONFIG_USB_GADGET_DUALSPEED=y
    CONFIG_USB_ZERO=m
    CONFIG_USB_ETH=m
    CONFIG_USB_ETH_RNDIS=y
    CONFIG_USB_GADGETFS=m
    CONFIG_USB_FILE_STORAGE=m
    # CONFIG_USB_FILE_STORAGE_TEST is not set
    CONFIG_USB_G_SERIAL=m
    CONFIG_USB_MIDI_GADGET=m
    # CONFIG_USB_G_PRINTER is not set
    # CONFIG_MMC is not set
    # CONFIG_MEMSTICK is not set
    # CONFIG_NEW_LEDS is not set
    # CONFIG_ACCESSIBILITY is not set
    # CONFIG_INFINIBAND is not set
    # CONFIG_EDAC is not set
    CONFIG_RTC_LIB=m
    CONFIG_RTC_CLASS=m

    #
    # RTC interfaces
    #
    CONFIG_RTC_INTF_SYSFS=y
    CONFIG_RTC_INTF_PROC=y
    CONFIG_RTC_INTF_DEV=y
    # CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
    # CONFIG_RTC_DRV_TEST is not set

    #
    # I2C RTC drivers
    #
    CONFIG_RTC_DRV_DS1307=m
    CONFIG_RTC_DRV_DS1374=m
    CONFIG_RTC_DRV_DS1672=m
    # CONFIG_RTC_DRV_MAX6900 is not set
    CONFIG_RTC_DRV_RS5C372=m
    CONFIG_RTC_DRV_ISL1208=m
    CONFIG_RTC_DRV_X1205=m
    CONFIG_RTC_DRV_PCF8563=m
    # CONFIG_RTC_DRV_PCF8583 is not set
    # CONFIG_RTC_DRV_M41T80 is not set
    # CONFIG_RTC_DRV_S35390A is not set
    # CONFIG_RTC_DRV_FM3130 is not set

    #
    # SPI RTC drivers
    #

    #
    # Platform RTC drivers
    #
    CONFIG_RTC_DRV_CMOS=m
    CONFIG_RTC_DRV_DS1511=m
    CONFIG_RTC_DRV_DS1553=m
    CONFIG_RTC_DRV_DS1742=m
    CONFIG_RTC_DRV_STK17TA8=m
    CONFIG_RTC_DRV_M48T86=m
    CONFIG_RTC_DRV_M48T59=m
    CONFIG_RTC_DRV_V3020=m

    #
    # on-CPU RTC drivers
    #
    CONFIG_DMADEVICES=y

    #
    # DMA Devices
    #
    CONFIG_INTEL_IOATDMA=m
    CONFIG_DMA_ENGINE=y

    #
    # DMA Clients
    #
    CONFIG_NET_DMA=y
    CONFIG_DCA=m
    # CONFIG_AUXDISPLAY is not set
    CONFIG_UIO=m
    # CONFIG_UIO_CIF is not set
    # CONFIG_UIO_SMX is not set

    #
    # Firmware Drivers
    #
    CONFIG_EDD=m
    # CONFIG_EDD_OFF is not set
    CONFIG_FIRMWARE_MEMMAP=y
    CONFIG_DELL_RBU=m
    CONFIG_DCDBAS=m
    CONFIG_DMIID=y
    # CONFIG_ISCSI_IBFT_FIND is not set

    #
    # File systems
    #
    CONFIG_EXT2_FS=y
    CONFIG_EXT2_FS_XATTR=y
    CONFIG_EXT2_FS_POSIX_ACL=y
    CONFIG_EXT2_FS_SECURITY=y
    # CONFIG_EXT2_FS_XIP is not set
    CONFIG_EXT3_FS=m
    CONFIG_EXT3_FS_XATTR=y
    CONFIG_EXT3_FS_POSIX_ACL=y
    CONFIG_EXT3_FS_SECURITY=y
    CONFIG_EXT4DEV_FS=m
    CONFIG_EXT4DEV_FS_XATTR=y
    CONFIG_EXT4DEV_FS_POSIX_ACL=y
    CONFIG_EXT4DEV_FS_SECURITY=y
    CONFIG_JBD=m
    # CONFIG_JBD_DEBUG is not set
    CONFIG_JBD2=m
    # CONFIG_JBD2_DEBUG is not set
    CONFIG_FS_MBCACHE=y
    CONFIG_REISERFS_FS=m
    # CONFIG_REISERFS_CHECK is not set
    # CONFIG_REISERFS_PROC_INFO is not set
    CONFIG_REISERFS_FS_XATTR=y
    CONFIG_REISERFS_FS_POSIX_ACL=y
    CONFIG_REISERFS_FS_SECURITY=y
    CONFIG_JFS_FS=m
    CONFIG_JFS_POSIX_ACL=y
    CONFIG_JFS_SECURITY=y
    # CONFIG_JFS_DEBUG is not set
    # CONFIG_JFS_STATISTICS is not set
    CONFIG_FS_POSIX_ACL=y
    CONFIG_XFS_FS=m
    CONFIG_XFS_QUOTA=y
    CONFIG_XFS_POSIX_ACL=y
    CONFIG_XFS_RT=y
    # CONFIG_XFS_DEBUG is not set
    CONFIG_GFS2_FS=m
    CONFIG_GFS2_FS_LOCKING_NOLOCK=m
    CONFIG_GFS2_FS_LOCKING_DLM=m
    # CONFIG_OCFS2_FS is not set
    CONFIG_DNOTIFY=y
    CONFIG_INOTIFY=y
    CONFIG_INOTIFY_USER=y
    CONFIG_QUOTA=y
    CONFIG_QUOTA_NETLINK_INTERFACE=y
    # CONFIG_PRINT_QUOTA_WARNING is not set
    CONFIG_QFMT_V1=m
    CONFIG_QFMT_V2=m
    CONFIG_QUOTACTL=y
    CONFIG_AUTOFS_FS=m
    CONFIG_AUTOFS4_FS=m
    CONFIG_FUSE_FS=m
    CONFIG_GENERIC_ACL=y

    #
    # CD-ROM/DVD Filesystems
    #
    CONFIG_ISO9660_FS=m
    CONFIG_JOLIET=y
    CONFIG_ZISOFS=y
    CONFIG_UDF_FS=m
    CONFIG_UDF_NLS=y

    #
    # DOS/FAT/NT Filesystems
    #
    CONFIG_FAT_FS=m
    CONFIG_MSDOS_FS=m
    CONFIG_VFAT_FS=m
    CONFIG_FAT_DEFAULT_CODEPAGE=437
    CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
    CONFIG_NTFS_FS=m
    # CONFIG_NTFS_DEBUG is not set
    CONFIG_NTFS_RW=y

    #
    # Pseudo filesystems
    #
    CONFIG_PROC_FS=y
    CONFIG_PROC_KCORE=y
    CONFIG_PROC_SYSCTL=y
    CONFIG_SYSFS=y
    CONFIG_TMPFS=y
    CONFIG_TMPFS_POSIX_ACL=y
    # CONFIG_HUGETLBFS is not set
    # CONFIG_HUGETLB_PAGE is not set
    CONFIG_CONFIGFS_FS=m

    #
    # Miscellaneous filesystems
    #
    CONFIG_ADFS_FS=m
    # CONFIG_ADFS_FS_RW is not set
    CONFIG_AFFS_FS=m
    CONFIG_ECRYPT_FS=m
    CONFIG_HFS_FS=m
    CONFIG_HFSPLUS_FS=m
    CONFIG_BEFS_FS=m
    # CONFIG_BEFS_DEBUG is not set
    CONFIG_BFS_FS=m
    CONFIG_EFS_FS=m
    # CONFIG_JFFS2_FS is not set
    CONFIG_CRAMFS=y
    CONFIG_VXFS_FS=m
    CONFIG_MINIX_FS=m
    CONFIG_HPFS_FS=m
    CONFIG_QNX4FS_FS=m
    CONFIG_ROMFS_FS=m
    CONFIG_SYSV_FS=m
    CONFIG_UFS_FS=m
    # CONFIG_UFS_FS_WRITE is not set
    # CONFIG_UFS_DEBUG is not set
    CONFIG_NETWORK_FILESYSTEMS=y
    # CONFIG_NFS_FS is not set
    CONFIG_NFSD=m
    CONFIG_NFSD_V3=y
    # CONFIG_NFSD_V3_ACL is not set
    # CONFIG_NFSD_V4 is not set
    CONFIG_LOCKD=m
    CONFIG_LOCKD_V4=y
    CONFIG_EXPORTFS=m
    CONFIG_NFS_COMMON=y
    CONFIG_SUNRPC=m
    # CONFIG_SUNRPC_BIND34 is not set
    # CONFIG_RPCSEC_GSS_KRB5 is not set
    # CONFIG_RPCSEC_GSS_SPKM3 is not set
    # CONFIG_SMB_FS is not set
    # CONFIG_CIFS is not set
    # CONFIG_NCP_FS is not set
    # CONFIG_CODA_FS is not set
    # CONFIG_AFS_FS is not set

    #
    # Partition Types
    #
    CONFIG_PARTITION_ADVANCED=y
    CONFIG_ACORN_PARTITION=y
    # CONFIG_ACORN_PARTITION_CUMANA is not set
    # CONFIG_ACORN_PARTITION_EESOX is not set
    CONFIG_ACORN_PARTITION_ICS=y
    # CONFIG_ACORN_PARTITION_ADFS is not set
    # CONFIG_ACORN_PARTITION_POWERTEC is not set
    CONFIG_ACORN_PARTITION_RISCIX=y
    CONFIG_OSF_PARTITION=y
    CONFIG_AMIGA_PARTITION=y
    CONFIG_ATARI_PARTITION=y
    CONFIG_MAC_PARTITION=y
    CONFIG_MSDOS_PARTITION=y
    CONFIG_BSD_DISKLABEL=y
    CONFIG_MINIX_SUBPARTITION=y
    CONFIG_SOLARIS_X86_PARTITION=y
    CONFIG_UNIXWARE_DISKLABEL=y
    CONFIG_LDM_PARTITION=y
    # CONFIG_LDM_DEBUG is not set
    CONFIG_SGI_PARTITION=y
    CONFIG_ULTRIX_PARTITION=y
    CONFIG_SUN_PARTITION=y
    CONFIG_KARMA_PARTITION=y
    CONFIG_EFI_PARTITION=y
    # CONFIG_SYSV68_PARTITION is not set
    CONFIG_NLS=y
    CONFIG_NLS_DEFAULT="iso8859-1"
    CONFIG_NLS_CODEPAGE_437=m
    CONFIG_NLS_CODEPAGE_737=m
    CONFIG_NLS_CODEPAGE_775=m
    CONFIG_NLS_CODEPAGE_850=m
    CONFIG_NLS_CODEPAGE_852=m
    CONFIG_NLS_CODEPAGE_855=m
    CONFIG_NLS_CODEPAGE_857=m
    CONFIG_NLS_CODEPAGE_860=m
    CONFIG_NLS_CODEPAGE_861=m
    CONFIG_NLS_CODEPAGE_862=m
    CONFIG_NLS_CODEPAGE_863=m
    CONFIG_NLS_CODEPAGE_864=m
    CONFIG_NLS_CODEPAGE_865=m
    CONFIG_NLS_CODEPAGE_866=m
    CONFIG_NLS_CODEPAGE_869=m
    CONFIG_NLS_CODEPAGE_936=m
    CONFIG_NLS_CODEPAGE_950=m
    CONFIG_NLS_CODEPAGE_932=m
    CONFIG_NLS_CODEPAGE_949=m
    CONFIG_NLS_CODEPAGE_874=m
    CONFIG_NLS_ISO8859_8=m
    CONFIG_NLS_CODEPAGE_1250=m
    CONFIG_NLS_CODEPAGE_1251=m
    CONFIG_NLS_ASCII=m
    CONFIG_NLS_ISO8859_1=m
    CONFIG_NLS_ISO8859_2=m
    CONFIG_NLS_ISO8859_3=m
    CONFIG_NLS_ISO8859_4=m
    CONFIG_NLS_ISO8859_5=m
    CONFIG_NLS_ISO8859_6=m
    CONFIG_NLS_ISO8859_7=m
    CONFIG_NLS_ISO8859_9=m
    CONFIG_NLS_ISO8859_13=m
    CONFIG_NLS_ISO8859_14=m
    CONFIG_NLS_ISO8859_15=m
    CONFIG_NLS_KOI8_R=m
    CONFIG_NLS_KOI8_U=m
    CONFIG_NLS_UTF8=m
    CONFIG_DLM=m
    # CONFIG_DLM_DEBUG is not set

    #
    # Kernel hacking
    #
    CONFIG_TRACE_IRQFLAGS_SUPPORT=y
    CONFIG_PRINTK_TIME=y
    # CONFIG_ENABLE_WARN_DEPRECATED is not set
    # CONFIG_ENABLE_MUST_CHECK is not set
    CONFIG_FRAME_WARN=2048
    CONFIG_MAGIC_SYSRQ=y
    CONFIG_UNUSED_SYMBOLS=y
    CONFIG_DEBUG_FS=y
    # CONFIG_HEADERS_CHECK is not set
    CONFIG_DEBUG_KERNEL=y
    # CONFIG_DEBUG_SHIRQ is not set
    CONFIG_DETECT_SOFTLOCKUP=y
    # CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set
    CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
    CONFIG_SCHED_DEBUG=y
    CONFIG_SCHEDSTATS=y
    # CONFIG_TIMER_STATS is not set
    # CONFIG_DEBUG_OBJECTS is not set
    # CONFIG_SLUB_DEBUG_ON is not set
    # CONFIG_SLUB_STATS is not set
    # CONFIG_DEBUG_RT_MUTEXES is not set
    # CONFIG_RT_MUTEX_TESTER is not set
    # CONFIG_DEBUG_SPINLOCK is not set
    # CONFIG_DEBUG_MUTEXES is not set
    # CONFIG_DEBUG_LOCK_ALLOC is not set
    # CONFIG_PROVE_LOCKING is not set
    # CONFIG_LOCK_STAT is not set
    # CONFIG_DEBUG_SPINLOCK_SLEEP is not set
    # CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
    CONFIG_STACKTRACE=y
    # CONFIG_DEBUG_KOBJECT is not set
    CONFIG_DEBUG_BUGVERBOSE=y
    # CONFIG_DEBUG_INFO is not set
    # CONFIG_DEBUG_VM is not set
    # CONFIG_DEBUG_VIRTUAL is not set
    # CONFIG_DEBUG_WRITECOUNT is not set
    # CONFIG_DEBUG_LIST is not set
    # CONFIG_DEBUG_SG is not set
    CONFIG_FRAME_POINTER=y
    # CONFIG_BOOT_PRINTK_DELAY is not set
    # CONFIG_RCU_TORTURE_TEST is not set
    # CONFIG_KPROBES_SANITY_TEST is not set
    # CONFIG_BACKTRACE_SELF_TEST is not set
    # CONFIG_LKDTM is not set
    # CONFIG_FAULT_INJECTION is not set
    CONFIG_LATENCYTOP=y
    CONFIG_HAVE_FTRACE=y
    CONFIG_HAVE_DYNAMIC_FTRACE=y
    CONFIG_TRACER_MAX_TRACE=y
    CONFIG_TRACING=y
    CONFIG_FTRACE=y
    # CONFIG_IRQSOFF_TRACER is not set
    # CONFIG_SYSPROF_TRACER is not set
    CONFIG_SCHED_TRACER=y
    CONFIG_CONTEXT_SWITCH_TRACER=y
    CONFIG_DYNAMIC_FTRACE=y
    # CONFIG_FTRACE_STARTUP_TEST is not set
    # CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
    # CONFIG_SAMPLES is not set
    CONFIG_HAVE_ARCH_KGDB=y
    # CONFIG_KGDB is not set
    # CONFIG_NONPROMISC_DEVMEM is not set
    CONFIG_X86_VERBOSE_BOOTUP=y
    CONFIG_EARLY_PRINTK=y
    # CONFIG_DEBUG_STACKOVERFLOW is not set
    # CONFIG_DEBUG_STACK_USAGE is not set
    # CONFIG_DEBUG_PAGEALLOC is not set
    # CONFIG_X86_PTDUMP is not set
    CONFIG_DEBUG_RODATA=y
    # CONFIG_DIRECT_GBPAGES is not set
    # CONFIG_DEBUG_RODATA_TEST is not set
    # CONFIG_DEBUG_NX_TEST is not set
    # CONFIG_IOMMU_DEBUG is not set
    # CONFIG_MMIOTRACE is not set
    CONFIG_IO_DELAY_TYPE_0X80=0
    CONFIG_IO_DELAY_TYPE_0XED=1
    CONFIG_IO_DELAY_TYPE_UDELAY=2
    CONFIG_IO_DELAY_TYPE_NONE=3
    CONFIG_IO_DELAY_0X80=y
    # CONFIG_IO_DELAY_0XED is not set
    # CONFIG_IO_DELAY_UDELAY is not set
    # CONFIG_IO_DELAY_NONE is not set
    CONFIG_DEFAULT_IO_DELAY_TYPE=0
    # CONFIG_KMEMCHECK is not set
    # CONFIG_DEBUG_BOOT_PARAMS is not set
    # CONFIG_CPA_DEBUG is not set

    #
    # Security options
    #
    CONFIG_KEYS=y
    # CONFIG_KEYS_DEBUG_PROC_KEYS is not set
    CONFIG_SECURITY=y
    CONFIG_SECURITY_NETWORK=y
    CONFIG_SECURITY_NETWORK_XFRM=y
    CONFIG_SECURITY_CAPABILITIES=y
    # CONFIG_SECURITY_FILE_CAPABILITIES is not set
    # CONFIG_SECURITY_ROOTPLUG is not set
    CONFIG_SECURITY_DEFAULT_MMAP_MIN_ADDR=65536
    CONFIG_SECURITY_SELINUX=y
    CONFIG_SECURITY_SELINUX_BOOTPARAM=y
    CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=0
    CONFIG_SECURITY_SELINUX_DISABLE=y
    CONFIG_SECURITY_SELINUX_DEVELOP=y
    CONFIG_SECURITY_SELINUX_AVC_STATS=y
    CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
    # CONFIG_SECURITY_SELINUX_ENABLE_SECMARK_DEFAULT is not set
    # CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set
    CONFIG_XOR_BLOCKS=m
    CONFIG_ASYNC_CORE=m
    CONFIG_ASYNC_MEMCPY=m
    CONFIG_ASYNC_XOR=m
    CONFIG_CRYPTO=y

    #
    # Crypto core or helper
    #
    CONFIG_CRYPTO_ALGAPI=y
    CONFIG_CRYPTO_AEAD=m
    CONFIG_CRYPTO_BLKCIPHER=m
    CONFIG_CRYPTO_HASH=y
    CONFIG_CRYPTO_MANAGER=y
    CONFIG_CRYPTO_GF128MUL=m
    CONFIG_CRYPTO_NULL=m
    # CONFIG_CRYPTO_CRYPTD is not set
    CONFIG_CRYPTO_AUTHENC=m
    CONFIG_CRYPTO_TEST=m

    #
    # Authenticated Encryption with Associated Data
    #
    CONFIG_CRYPTO_CCM=m
    CONFIG_CRYPTO_GCM=m
    CONFIG_CRYPTO_SEQIV=m

    #
    # Block modes
    #
    CONFIG_CRYPTO_CBC=m
    CONFIG_CRYPTO_CTR=m
    # CONFIG_CRYPTO_CTS is not set
    CONFIG_CRYPTO_ECB=m
    CONFIG_CRYPTO_LRW=m
    CONFIG_CRYPTO_PCBC=m
    # CONFIG_CRYPTO_XTS is not set

    #
    # Hash modes
    #
    CONFIG_CRYPTO_HMAC=y
    CONFIG_CRYPTO_XCBC=m

    #
    # Digest
    #
    CONFIG_CRYPTO_CRC32C=m
    CONFIG_CRYPTO_MD4=m
    CONFIG_CRYPTO_MD5=y
    CONFIG_CRYPTO_MICHAEL_MIC=m
    CONFIG_CRYPTO_SHA1=m
    CONFIG_CRYPTO_SHA256=m
    CONFIG_CRYPTO_SHA512=m
    CONFIG_CRYPTO_TGR192=m
    CONFIG_CRYPTO_WP512=m

    #
    # Ciphers
    #
    CONFIG_CRYPTO_AES=m
    CONFIG_CRYPTO_AES_X86_64=m
    CONFIG_CRYPTO_ANUBIS=m
    CONFIG_CRYPTO_ARC4=m
    CONFIG_CRYPTO_BLOWFISH=m
    # CONFIG_CRYPTO_CAMELLIA is not set
    CONFIG_CRYPTO_CAST5=m
    CONFIG_CRYPTO_CAST6=m
    CONFIG_CRYPTO_DES=m
    # CONFIG_CRYPTO_FCRYPT is not set
    CONFIG_CRYPTO_KHAZAD=m
    CONFIG_CRYPTO_SALSA20=m
    CONFIG_CRYPTO_SALSA20_X86_64=m
    # CONFIG_CRYPTO_SEED is not set
    CONFIG_CRYPTO_SERPENT=m
    CONFIG_CRYPTO_TEA=m
    CONFIG_CRYPTO_TWOFISH=m
    CONFIG_CRYPTO_TWOFISH_COMMON=m
    CONFIG_CRYPTO_TWOFISH_X86_64=m

    #
    # Compression
    #
    CONFIG_CRYPTO_DEFLATE=m
    CONFIG_CRYPTO_LZO=m
    # CONFIG_CRYPTO_HW is not set
    CONFIG_HAVE_KVM=y
    # CONFIG_VIRTUALIZATION is not set

    #
    # Library routines
    #
    CONFIG_BITREVERSE=y
    CONFIG_GENERIC_FIND_FIRST_BIT=y
    CONFIG_GENERIC_FIND_NEXT_BIT=y
    CONFIG_CRC_CCITT=m
    CONFIG_CRC16=m
    CONFIG_CRC_ITU_T=m
    CONFIG_CRC32=y
    # CONFIG_CRC7 is not set
    CONFIG_LIBCRC32C=m
    CONFIG_ZLIB_INFLATE=y
    CONFIG_ZLIB_DEFLATE=m
    CONFIG_LZO_COMPRESS=m
    CONFIG_LZO_DECOMPRESS=m
    CONFIG_TEXTSEARCH=y
    CONFIG_TEXTSEARCH_KMP=m
    CONFIG_TEXTSEARCH_BM=m
    CONFIG_TEXTSEARCH_FSM=m
    CONFIG_PLIST=y
    CONFIG_HAS_IOMEM=y
    CONFIG_HAS_IOPORT=y
    CONFIG_HAS_DMA=y

    Best regards,
    --Edwin

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  9. Re: [PATCH] x86_64: fix delayed signals

    On Sat, 12 Jul 2008 16:42:13 +0300
    Török Edwin wrote:

    > On 2008-07-12 13:32, Török Edwin wrote:
    > > I haven't yet tried ftrace on this box, and neither did I try
    > > Roland's patch yet. I will try that now, and hopefuly come back
    > > with some numbers shortly.
    > >

    >
    > tip/master of today already includes Roland's patch, so I have tested
    > it without even knowing on the 64-bit box.
    > [After gathering all the traces, I wanted to repeat everything with
    > the patch applied, but there was no need since its already there].
    > I can say that I didn't notice any difference in the delays (I still
    > get 24 - 30 second delays).
    > I also tried Linus's readdir patch, the latency is around 27 seconds
    > (was 30 before). I am not sure if it is an improvement, or measurement
    > noise.
    >
    > Also I wasn't able to reproduce the 'Ctrl+Z works faster than Ctrl+C'
    > symptom, perhaps it is just a coincidence: by the time I press Ctrl+Z
    > to interrupt, enough time has passed (30+ secs), that the same effect
    > would have been obtained by another Ctrl+C, or by just waiting.
    >
    > On 32-bit, I think the 2-3 seconds latency can be attributed to I/O
    > delays, and I think its acceptable given the slow disk (5k4 rpm sata).


    I see really bad delays on 32 bit as well, but they go away for me if I
    do
    echo 4096 > /sys/block/sda/queue/nr_requests



    --
    If you want to reach me at my work email, use arjan@linux.intel.com
    For development, discussion and tips for power savings,
    visit http://www.lesswatts.org
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  10. Re: [PATCH] x86_64: fix delayed signals



    On Sat, 12 Jul 2008, Török Edwin wrote:
    >
    > On my 32-bit box (slow disks, SMP, XFS filesystem) 2.6.26-rc9 behaves
    > the same as 2.6.26-rc8, I can reliably reproduce a 2-3 second latency
    > [1] between pressing ^C the first time, and the shell returning (on the
    > text console too).
    > Using ftrace available from tip/master, I see up to 3 seconds of delay
    > between kill_pgrp and detach_pid (and during that time I can press ^C
    > again, leading to 2-3 kill_pgrp calls)


    The thing is, it's important to see what happens in between.

    In particular, 2-3 second latencies can be entirely _normal_ (although
    obviously very annoying) with most log-based filesystems when they decide
    they have to flush the log. A lot of filesystems are not designed for
    latency - every single filesystem test I have ever seen has always been
    either a throughput test, or a "average random-seek latency" kind of test.

    The exact behavior will depend on the filesystem, for example. It will
    also easily depend on things like whether you update 'atime' or not. Many
    ostensibly read-only loads end up writing some data, especially inode
    atimes, and that's when they can get caught up in having to wait for a log
    to flush (to make up for that atime thing).

    You can try to limit the amount of dirty data in flight by tweaking
    /proc/sys/vm/dirty*ratio, but from a latency standpoint the thing that
    actually matters more is often not the amount of dirty data, but the size
    of the requests queues - because you often care about read latency, but if
    you have big requests and especially if you have a load that does lots of
    big _contiguous_ writes (like your 'dd's would do), then what can easily
    happen is that the read ends up being behind a really big write in the
    request queues.

    And 2-3 second latencies by no means means that each individual IO is 2-3
    seconds long. No - it just means that you ended up having to do multiple
    reads synchronously, and since the reads depended on each other (think a
    pathname lookup - reading each directory entry -> inode -> data), you can
    easily have a single system call causing 5-10 reads (bad cases are _much_
    more, but 5-10 are perfectly normal for even well-behaved things), and now
    if each of those reads end up being behind a fairly big write...

    > On my 64-bit box (2 disks in raid-0, UP, reiserfs filesystem) 2.6.25 and
    > 2.6.26-rc9 behave the same, and most of the time (10-20 times in a row)
    > find responds to ^C instantly.
    >
    > However in _some_ cases find doesn't respond to ^C for a very long time
    > (~30 seconds), and when this happens I can't do anything else but switch
    > consoles, starting another process (latencytop -d) hangs, and so does
    > any other external command.


    Ok, that is definitel not related to signals at all. You're simply stuck
    waiting for IO - or perhaps some fundamental filesystem semaphore which is
    held while some IO needs to be flushed. That's why _unrelated_ processes
    hang: they're all waiting for a global resource.

    And it may be worse on your other box for any number of reasons: raid
    means, for example, that you have two different levels of queueing, and
    thus effectively your queues are longer. And while raid-0 is better for
    throughput, it's not necessarily at all better for latency. The filesystem
    also makes a difference, as does the amount of dirty data under write-back
    (do you also have more memory in your x86-64 box, for example? That makes
    the kernel do bigger writeback buffers by default)

    > I haven't yet tried ftrace on this box, and neither did I try Roland's
    > patch yet. I will try that now, and hopefuly come back with some numbers
    > shortly.


    Trust me, roland's patch will make no difference what-so-ever. It's purely
    a per-thread thing, and your behaviour is clearly not per-thread.

    Signals are _always_ delayed until non-blocking system calls are done, and
    that means until the end of IO.

    This is also why your trace on just 'kill_pgrp' and 'detach_pid' is not
    interesting. It's _normal_ to have a delay between them. It can happen
    because the process blocks (or catches) signals, but it will also happen
    if some system call waits for disk.

    (The waiting for disk may be indirect - it might be due to needing more
    memory and needing to write out dirty studd. So it's not necessarily doing
    IO per se, although it's quite likely that that is part of it).

    You could try 'noatime' and see if that helps behaviour a bit.

    Linus
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  11. Re: [PATCH] x86_64: fix delayed signals



    On Sat, 12 Jul 2008, Andi Kleen wrote:
    >
    > Linus, the fact that is not explained by your theory is why
    > Ctrl-Z+kill works but Ctrl-C doesn't.


    Umm. Read the reports more carefully. Many of the complaints are about ^Z
    too.

    More-over, ^Z and ^C are very different from an app standpoint. A _lot_ of
    applications block or catch ^C (SIGINT), while doing the same for ^Z
    (SIGTSTP) and SIGTERM is much less common (but possible: a tty sends
    SIGTSTP, which is blockable, not SIGSTOP, which is not).

    The reason? It's quite common to catch ^C in order to do cleanup. The
    common handler case is that you catch SIGINT, do cleanup, and then kill
    yourself with SIGINT in order to make it _look_ like you reacted directly
    to the ^C.


    So there are a _lot_ of loads where ^C will not kill things immediately,
    but ^Z + SIGKILL (or SIGTERM, the default for "sig") will.

    Try doing a "yum update", for example.

    Or use 'git', for that matter. git does exactly that kind of thing:
    catches SIGINT and removes temp-files etc. But if you kill it manually
    using some "stronger" signal (and using ^Z to get the shell back before
    you do that is obviously one way), that won't happen.

    And trust me, git is not at all unusual in this respect.

    Linus
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  12. Re: [PATCH] x86_64: fix delayed signals



    On Sat, 12 Jul 2008, Arjan van de Ven wrote:
    >
    > I see really bad delays on 32 bit as well, but they go away for me if I
    > do
    > echo 4096 > /sys/block/sda/queue/nr_requests


    Hmm. I think the default is 128, and in many cases latencies should
    actually go up with bigger requests queues - especially if it means that
    you can have a lot more writes in front of the read. You see the opposite
    behaviour.

    That could easily happen if the scheduler is crazy and lets writes use up
    all of the request queue, or if the limited queue means that it cannot
    effectively merge requests. But request merging should happen trivially
    for the contiguous 'dd' case almost regardless of queue size, so I wonder
    if something else is going on.

    Ahh.. I see something _very_ suspicious.

    Look at block/blk-core.c: get_request(). It starts throttling and batching
    requests when it gets

    if (rl->count[rw]+1 >= queue_congestion_on_threshold(q)) {

    and notice how this is independent of whether it's a read or a write (but
    it does count them separately). But on the wakeup path, it uses different
    limits for reads than for writes.

    That batching looks pretty bogus for reads to begin with, and then
    behaving similarly on throttling but differently on wakup sounds bogus.

    The blk_alloc_request() also ends up allocating all requests from one
    mempool, so if that mempool runs out (due to writes having used them all
    up), then those writes will block reads too, even though reads should have
    much higher priority.

    I dunno. But there _has_ been a lot of churn in the different block queues
    over the last few months. I wouldn't be surprised at all if something got
    broken in the process. And as with filesystems, almost all performance
    tests are for throughput, not "bad latency" in the presense of other
    heavy IO.

    Linus
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  13. Re: [PATCH] x86_64: fix delayed signals

    On Sat, 12 Jul 2008 11:00:06 -0700 (PDT)
    Linus Torvalds wrote:

    >
    >
    > On Sat, 12 Jul 2008, Arjan van de Ven wrote:
    > >
    > > I see really bad delays on 32 bit as well, but they go away for me
    > > if I do
    > > echo 4096 > /sys/block/sda/queue/nr_requests

    >
    > Hmm. I think the default is 128, and in many cases latencies should
    > actually go up with bigger requests queues - especially if it means
    > that you can have a lot more writes in front of the read. You see the
    > opposite behaviour.


    well... so far my assumption on this has been this:
    CFQ is good at fairness, and manages to control latency between
    processes in some fair way (eg if one guy is slamming the disk and
    someone else needs to page in a 4Kb page but is otherwise not doing
    IO... CFQ will let the pagefault skip ahead to a large degree) .
    However... the 128 limit happens BEFORE CFQ gets involved and suddenly
    the 4kb pagefault has to wait for everything that's pending


    > Look at block/blk-core.c: get_request(). It starts throttling and
    > batching requests when it gets
    >
    > if (rl->count[rw]+1 >= queue_congestion_on_threshold(q)) {
    >
    > and notice how this is independent of whether it's a read or a write
    > (but it does count them separately). But on the wakeup path, it uses
    > different limits for reads than for writes.
    >
    > That batching looks pretty bogus for reads to begin with, and then
    > behaving similarly on throttling but differently on wakup sounds
    > bogus.
    >
    > The blk_alloc_request() also ends up allocating all requests from one
    > mempool, so if that mempool runs out (due to writes having used them
    > all up), then those writes will block reads too, even though reads
    > should have much higher priority.
    >
    > I dunno. But there _has_ been a lot of churn in the different block
    > queues over the last few months. I wouldn't be surprised at all if
    > something got broken in the process. And as with filesystems, almost
    > all performance tests are for throughput, not "bad latency" in the
    > presense of other heavy IO.


    what I'm seeing is not super new; even 2.6.25 has the behavior already
    (and with latencytop it's very visible behavior; before changing the
    tunable I see 2+ second delays in user apps, after the tunable change
    they're down significantly)

    --
    If you want to reach me at my work email, use arjan@linux.intel.com
    For development, discussion and tips for power savings,
    visit http://www.lesswatts.org
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  14. Re: [PATCH] x86_64: fix delayed signals



    On Sat, 12 Jul 2008, Arjan van de Ven wrote:
    >
    > what I'm seeing is not super new; even 2.6.25 has the behavior already
    > (and with latencytop it's very visible behavior; before changing the
    > tunable I see 2+ second delays in user apps, after the tunable change
    > they're down significantly)


    Yeah, I think people said it's more visible, and pointed back to 2.6.23 or
    something being better. So no, I don't think anybody really claimed this
    as a regression from 25.

    Linus
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  15. Re: [PATCH] x86_64: fix delayed signals

    On 2008-07-12 20:29, Linus Torvalds wrote:
    > On Sat, 12 Jul 2008, Török Edwin wrote:
    >
    >> On my 32-bit box (slow disks, SMP, XFS filesystem) 2.6.26-rc9 behaves
    >> the same as 2.6.26-rc8, I can reliably reproduce a 2-3 second latency
    >> [1] between pressing ^C the first time, and the shell returning (on the
    >> text console too).
    >> Using ftrace available from tip/master, I see up to 3 seconds of delay
    >> between kill_pgrp and detach_pid (and during that time I can press ^C
    >> again, leading to 2-3 kill_pgrp calls)
    >>

    >
    > The thing is, it's important to see what happens in between.
    >
    > In particular, 2-3 second latencies can be entirely _normal_ (although
    > obviously very annoying) with most log-based filesystems when they decide
    > they have to flush the log.


    A bit off-topic, but something I noticed during the tests:
    In my original test I have rm-ed the files right after launching dd in
    the background, yet it still continued to write to the disk.
    I can understand that if the file is opened O_RDWR, you might seek back
    and read what you wrote, so Linux needs to actually do the write,
    but why does it insist on writing to the disk, on a file opened with
    O_WRONLY, after the file itself got unlinked?

    > A lot of filesystems are not designed for
    > latency - every single filesystem test I have ever seen has always been
    > either a throughput test, or a "average random-seek latency" kind of test.
    >
    > The exact behavior will depend on the filesystem, for example. It will
    > also easily depend on things like whether you update 'atime' or not. Many
    > ostensibly read-only loads end up writing some data, especially inode
    > atimes, and that's when they can get caught up in having to wait for a log
    > to flush (to make up for that atime thing).
    >


    I have my filesystems mounted as noatime already.
    But yes, I am using different filesystems, the x86-64 box has reiserfs,
    and the x86-32 box has xfs.

    > You can try to limit the amount of dirty data in flight by tweaking
    > /proc/sys/vm/dirty*ratio


    I have these in my /etc/rc.local:
    echo 5 > /proc/sys/vm/dirty_background_ratio
    echo 10 >/proc/sys/vm/dirty_ratio


    > , but from a latency standpoint the thing that
    > actually matters more is often not the amount of dirty data, but the size
    > of the requests queues - because you often care about read latency, but if
    > you have big requests and especially if you have a load that does lots of
    > big _contiguous_ writes (like your 'dd's would do), then what can easily
    > happen is that the read ends up being behind a really big write in the
    > request queues.
    >


    I will try tweaking these (and /sys/block/sda/queue/nr_requests that
    Arjan suggested).

    > And 2-3 second latencies by no means means that each individual IO is 2-3
    > seconds long. No - it just means that you ended up having to do multiple
    > reads synchronously, and since the reads depended on each other (think a
    > pathname lookup - reading each directory entry -> inode -> data), you can
    > easily have a single system call causing 5-10 reads (bad cases are _much_
    > more, but 5-10 are perfectly normal for even well-behaved things), and now
    > if each of those reads end up being behind a fairly big write...
    >


    I'll try blktrace tomorrow, it should tell me when I/Os are queued /
    completed.

    >
    >> On my 64-bit box (2 disks in raid-0, UP, reiserfs filesystem) 2.6.25 and
    >> 2.6.26-rc9 behave the same, and most of the time (10-20 times in a row)
    >> find responds to ^C instantly.
    >>
    >> However in _some_ cases find doesn't respond to ^C for a very long time
    >> (~30 seconds), and when this happens I can't do anything else but switch
    >> consoles, starting another process (latencytop -d) hangs, and so does
    >> any other external command.
    >>

    >
    > Ok, that is definitel not related to signals at all. You're simply stuck
    > waiting for IO - or perhaps some fundamental filesystem semaphore which is
    > held while some IO needs to be flushed.


    AFAICT reiserfs still uses the BKL, could that explain why one I/O
    delays another?

    > That's why _unrelated_ processes
    > hang: they're all waiting for a global resource.
    >
    > And it may be worse on your other box for any number of reasons: raid
    > means, for example, that you have two different levels of queueing, and
    > thus effectively your queues are longer. And while raid-0 is better for
    > throughput, it's not necessarily at all better for latency. The filesystem
    > also makes a difference, as does the amount of dirty data under write-back
    > (do you also have more memory in your x86-64 box, for example? That makes
    > the kernel do bigger writeback buffers by default)
    >
    >


    Yes, I have more memory on x86-64 (2G), and x86-32 has 1G.
    >> I haven't yet tried ftrace on this box, and neither did I try Roland's
    >> patch yet. I will try that now, and hopefuly come back with some numbers
    >> shortly.
    >>

    >
    > Trust me, roland's patch will make no difference what-so-ever. It's purely
    > a per-thread thing, and your behaviour is clearly not per-thread.
    >


    Indeed.
    [Roland's patch is included in tip/master so I actually tried it even if
    I didn't know I did]

    > Signals are _always_ delayed until non-blocking system calls are done, and
    > that means until the end of IO.
    >
    > This is also why your trace on just 'kill_pgrp' and 'detach_pid' is not
    > interesting. It's _normal_ to have a delay between them. It can happen
    > because the process blocks (or catches) signals, but it will also happen
    > if some system call waits for disk.
    >


    Is there a way to trace what happens between those 2 functions?
    Maybe if I don't use the trace filter (and thus trace all functions),
    and modify the kernel sources to start tracing on kill_pgrp, and stop
    tracing on detach_pid.
    Would that provide useful info?


    Best regards,
    --Edwin

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  16. Re: [PATCH] x86_64: fix delayed signals



    On Sat, 12 Jul 2008, Török Edwin wrote:
    >
    > A bit off-topic, but something I noticed during the tests:
    > In my original test I have rm-ed the files right after launching dd in
    > the background, yet it still continued to write to the disk.
    > I can understand that if the file is opened O_RDWR, you might seek back
    > and read what you wrote, so Linux needs to actually do the write,
    > but why does it insist on writing to the disk, on a file opened with
    > O_WRONLY, after the file itself got unlinked?


    Linux itself doesn't insist on writing to disk. In fact, at least with
    traditional UNIX filesystems (eg minix, ext2) the deleted writes would be
    undone.

    But some filesystems can't just invalidate dirty buffers (some won't do it
    for meta-data, others won't do it for _any_ data). So again, this
    behaviour depends on the filesystem. And sadly, the more "advanced"
    filesystem, the worse it usually behaves here.


    > I have my filesystems mounted as noatime already.
    > But yes, I am using different filesystems, the x86-64 box has reiserfs,
    > and the x86-32 box has xfs.
    >
    > > You can try to limit the amount of dirty data in flight by tweaking
    > > /proc/sys/vm/dirty*ratio

    >
    > I have these in my /etc/rc.local:
    > echo 5 > /proc/sys/vm/dirty_background_ratio
    > echo 10 >/proc/sys/vm/dirty_ratio


    That matches the modern defaults. You can try playing with them if you
    want to. And yes, it's worth testing nr_requests too.

    > > Ok, that is definitel not related to signals at all. You're simply stuck
    > > waiting for IO - or perhaps some fundamental filesystem semaphore which is
    > > held while some IO needs to be flushed.

    >
    > AFAICT reiserfs still uses the BKL, could that explain why one I/O
    > delays another?


    The BKL should be ok in this respect - it gets automatically dropped when
    doing synchronous waiting (this is somethign that will possibly go away as
    we try to convince people to get rid of the BKL, but it certainly hasn't
    happened yet).

    So it actually gets worse with other locks - semaphores or mutexes - that
    stay held over IO. And reiserfs has a journal lock (and a "commit" lock),
    but I don't know how they are held and whether this could be part of the
    issue.

    > > This is also why your trace on just 'kill_pgrp' and 'detach_pid' is not
    > > interesting. It's _normal_ to have a delay between them. It can happen
    > > because the process blocks (or catches) signals, but it will also happen
    > > if some system call waits for disk.

    >
    > Is there a way to trace what happens between those 2 functions?


    You could try to trace not just those functions, but scheduling events
    too. Or yes, do something special-caed.

    Trying to figure out latencies in the block trace is likely also going to
    be interesting (although you won't see any signal issues there - but any
    long read latencies will automatically tend to imply latency issues not
    just for signals, but for pretty much any operations).

    Linus
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  17. Re: [PATCH] x86_64: fix delayed signals

    On Saturday 12 July 2008 22:26, Török Edwin wrote:
    > A bit off-topic, but something I noticed during the tests:
    > In my original test I have rm-ed the files right after launching dd in
    > the background, yet it still continued to write to the disk.
    > I can understand that if the file is opened O_RDWR, you might seek back
    > and read what you wrote, so Linux needs to actually do the write,
    > but why does it insist on writing to the disk, on a file opened with
    > O_WRONLY, after the file itself got unlinked?


    Because process can do

    fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | O_RDWR)

    at any time.
    --
    vda
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  18. Re: [PATCH] x86_64: fix delayed signals

    Linus Torvalds wrote:
    >
    > On Sat, 12 Jul 2008, Andi Kleen wrote:
    >> Linus, the fact that is not explained by your theory is why
    >> Ctrl-Z+kill works but Ctrl-C doesn't.

    >
    > Umm. Read the reports more carefully. Many of the complaints are about ^Z
    > too.


    At least the original report was about Ctrl-C only versus Ctrl-Z.

    I see the problem regularly myself that Ctrl-C doesn't work,
    but Ctrl-Z+kill does (although I unfortunately cannot
    reproduce it on demand). But it was with programs who shouldn't catch Ctrl-C,
    like find. I confirmed with ltrace that at least my version of find doesn't
    catch any signals.

    Take the original report for example:

    http://marc.info/?l=linux-kernel&m=121464952615807&w=2

    It also had trouble interrupting find:

    $ find / >/dev/null
    find: `/boot/lost+found': Permission denied
    ^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C ^C^C^C^C^C^C

    but Ctrl-Z+kill worked immediately.

    > The reason? It's quite common to catch ^C in order to do cleanup.


    Yes I understand that, but I don't think it's the case here.

    I'm afraid it's more a "has a simple explanation that is wrong"
    case here.

    Or we might be talking about multiple different bugs.

    -Andi
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  19. Re: [PATCH] x86_64: fix delayed signals

    On 07/12, Denys Vlasenko wrote:
    >
    > On Saturday 12 July 2008 22:26, T?r?k Edwin wrote:
    > > A bit off-topic, but something I noticed during the tests:
    > > In my original test I have rm-ed the files right after launching dd in
    > > the background, yet it still continued to write to the disk.
    > > I can understand that if the file is opened O_RDWR, you might seek back
    > > and read what you wrote, so Linux needs to actually do the write,
    > > but why does it insist on writing to the disk, on a file opened with
    > > O_WRONLY, after the file itself got unlinked?

    >
    > Because process can do
    >
    > fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | O_RDWR)


    Is it?

    SETFL_MASK doesn't have O_RDWR, and in any case setfl() changes ->f_flags,
    not ->f_mode.

    Oleg.

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  20. Re: [PATCH] x86_64: fix delayed signals

    On Sunday 13 July 2008 12:46, Oleg Nesterov wrote:
    > On 07/12, Denys Vlasenko wrote:
    > >
    > > On Saturday 12 July 2008 22:26, T?r?k Edwin wrote:
    > > > A bit off-topic, but something I noticed during the tests:
    > > > In my original test I have rm-ed the files right after launching dd in
    > > > the background, yet it still continued to write to the disk.
    > > > I can understand that if the file is opened O_RDWR, you might seek back
    > > > and read what you wrote, so Linux needs to actually do the write,
    > > > but why does it insist on writing to the disk, on a file opened with
    > > > O_WRONLY, after the file itself got unlinked?

    > >
    > > Because process can do
    > >
    > > fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | O_RDWR)

    >
    > Is it?
    >
    > SETFL_MASK doesn't have O_RDWR, and in any case setfl() changes ->f_flags,
    > not ->f_mode.


    Just tested it and you are right.

    I distinctly remember seeing such code somewhere. Interesting.
    Now I wonder whether it was a bug, or those were not file descriptors,
    but sockets?...
    --
    vda
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

+ Reply to Thread
Page 2 of 3 FirstFirst 1 2 3 LastLast