vfat BKL/lock_super regression in v2.6.26-rc3-g8f59342 - Kernel

This is a discussion on vfat BKL/lock_super regression in v2.6.26-rc3-g8f59342 - Kernel ; I've seen some strange things happen USB keys connected to my laptop since I upgraded the kernel the last time. The system: - Thinkpad X41 / Pentium M - Debian/Lenny - libc6 2.7-10 - gcc 4.2.4 The scenario: - I ...

+ Reply to Thread
Results 1 to 12 of 12

Thread: vfat BKL/lock_super regression in v2.6.26-rc3-g8f59342

  1. vfat BKL/lock_super regression in v2.6.26-rc3-g8f59342

    I've seen some strange things happen USB keys connected to my laptop
    since I upgraded the kernel the last time.

    The system:
    - Thinkpad X41 / Pentium M
    - Debian/Lenny
    - libc6 2.7-10
    - gcc 4.2.4

    The scenario:
    - I insert the USB key or CF card to a USB reader
    - udev mounts it for me (it's a vfat fs)
    - I move a file from the key to my main ext3 disk
    - the file is created on disk and all contents are copied
    - the mv command never exits and cannot be killed
    - after that any syscall attempting to access the USB key will lockup

    Running an strace shows that the command hangs on the unlink() of the
    file being moved. Here are the last two few lines...

    close(4) = 0
    close(3) = 0
    lstat64("/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
    fstatat64(AT_FDCWD, "/media/usb0/dcim/100ncd70/20080817-175746-361.jpg", {st_mode=S_IFREG|0660, st_size=830429, ...}, AT_SYMLINK_NOFOLLOW) = 0
    unlinkat(AT_FDCWD, "/media/usb0/dcim/100ncd70/20080817-175746-361.jpg", 0

    I was able to bisect it to the commit 8f5934278d1d86590244c2791b28f77d67466007
    which claims to "Replace BKL with superblock lock in fat/msdos/vfat".

    When I run with lock debugging I get...

    =============================================
    [ INFO: possible recursive locking detected ]
    2.6.27-rc3-bisect-00448-ga7f5aaf #16
    ---------------------------------------------
    mv/4020 is trying to acquire lock:
    (&type->s_lock_key#9){--..}, at: [] lock_super+0x1e/0x20

    but task is already holding lock:
    (&type->s_lock_key#9){--..}, at: [] lock_super+0x1e/0x20

    other info that might help us debug this:
    3 locks held by mv/4020:
    #0: (&sb->s_type->i_mutex_key#9/1){--..}, at: [] do_unlinkat+0x66/0x140
    #1: (&sb->s_type->i_mutex_key#9){--..}, at: [] vfs_unlink+0x84/0x110
    #2: (&type->s_lock_key#9){--..}, at: [] lock_super+0x1e/0x20

    stack backtrace:
    Pid: 4020, comm: mv Not tainted 2.6.27-rc3-bisect-00448-ga7f5aaf #16
    [] validate_chain+0x984/0xea0
    [] ? native_sched_clock+0x0/0xf0
    [] __lock_acquire+0x2ec/0x9b0
    [] lock_acquire+0x6f/0x90
    [] ? lock_super+0x1e/0x20
    [] mutex_lock_nested+0xad/0x300
    [] ? lock_super+0x1e/0x20
    [] ? lock_super+0x1e/0x20
    [] lock_super+0x1e/0x20
    [] fat_write_inode+0x60/0x2b0 [fat]
    [] ? _spin_unlock_irqrestore+0x48/0x80
    [] ? fat_sync_inode+0x3/0x20 [fat]
    [] fat_sync_inode+0x12/0x20 [fat]
    [] fat_remove_entries+0xbe/0x120 [fat]
    [] vfat_unlink+0x5f/0x90 [vfat]
    [] ? vfat_unlink+0x0/0x90 [vfat]
    [] vfs_unlink+0x98/0x110
    [] do_unlinkat+0x130/0x140
    [] ? audit_syscall_entry+0x105/0x150
    [] sys_unlinkat+0x3b/0x40
    [] sysenter_do_call+0x12/0x3f
    =======================

    It looks like the call trace is:

    - do_unlinkat
    - vfs_unlink
    - vfat_unlink
    * lock_super
    - fat_remove_entries
    - fat_sync_inode
    - fat_write_inode
    * lock_super

    So this code really really liked BKL because it was recursive. I am
    testing a naive patch to address this problem and will follow up on it
    in a bit.

    Below is some more debug info that I've collected if it helps.

    Here is the bad commit:

    8f5934278d1d86590244c2791b28f77d67466007 is first bad commit
    commit 8f5934278d1d86590244c2791b28f77d67466007
    Author: Linus Torvalds
    Date: Mon May 19 19:53:01 2008 -0700

    Replace BKL with superblock lock in fat/msdos/vfat

    This replaces the use of the BKL in the FAT family of filesystems with the
    existing superblock lock instead.

    The code already appears to do mostly proper locking with its own private
    spinlocks (and mutexes), but while the BKL could possibly have been
    dropped entirely, converting it to use the superblock lock (which is just
    a regular mutex) is the conservative thing to do.

    As a per-filesystem mutex, it not only won't have any of the possible
    latency issues related to the BKL, but the lock is obviously private to
    the particular filesystem instance and will thus not cause problems for
    entirely unrelated users like the BKL can.

    Signed-off-by: Linus Torvalds
    Cc: OGAWA Hirofumi
    Signed-off-by: Jonathan Corbet

    :040000 040000 8b8d1a0d574e953de4be1462ddcd228781371528 aef9012aa95032bc6f8e45365c8052c2ab5096f3 M fs

    Here is the bisect log:

    git-bisect start
    # bad: [a7f5aaf36ded825477c4d7167cc6eb1bcdc63191] Merge branch 'x86-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
    git-bisect bad a7f5aaf36ded825477c4d7167cc6eb1bcdc63191
    # good: [bce7f793daec3e65ec5c5705d2457b81fe7b5725] Linux 2.6.26
    git-bisect good bce7f793daec3e65ec5c5705d2457b81fe7b5725
    # bad: [6e86841d05f371b5b9b86ce76c02aaee83352298] Linux 2.6.27-rc1
    git-bisect bad 6e86841d05f371b5b9b86ce76c02aaee83352298
    # bad: [d20b27478d6ccf7c4c8de4f09db2bdbaec82a6c0] V4L/DVB (8415): gspca: Infinite loop in i2c_w() of etoms.
    git-bisect bad d20b27478d6ccf7c4c8de4f09db2bdbaec82a6c0
    # bad: [666484f0250db2e016948d63b3ef33e202e3b8d0] Merge branch 'core/softirq' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
    git-bisect bad 666484f0250db2e016948d63b3ef33e202e3b8d0
    # good: [d59fdcf2ac501de99c3dfb452af5e254d4342886] Merge commit 'v2.6.26' into x86/core
    git-bisect good d59fdcf2ac501de99c3dfb452af5e254d4342886
    # good: [a3da5bf84a97d48cfaf66c6842470fc403da5121] Merge branch 'x86/for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
    git-bisect good a3da5bf84a97d48cfaf66c6842470fc403da5121
    # good: [6712e299b7dc78aa4971b85e803435ee6d49a9dd] Merge branch 'tracing/ftrace' into auto-ftrace-next
    git-bisect good 6712e299b7dc78aa4971b85e803435ee6d49a9dd
    # bad: [b7e3e1fbf69d1b6abfd337460ea17d7230e3a6e5] mpt: fasync BKL pushdown
    git-bisect bad b7e3e1fbf69d1b6abfd337460ea17d7230e3a6e5
    # bad: [2861ead38b8a376888c3f63b9c8e45d4cee02117] apm-emulation: BKL pushdown
    git-bisect bad 2861ead38b8a376888c3f63b9c8e45d4cee02117
    # good: [20613f24bcd1cbfb08e64f0bb00c44481313b448] usbcore: cdev lock_kernel() pushdown
    git-bisect good 20613f24bcd1cbfb08e64f0bb00c44481313b448
    # good: [dea3f665d6fa263a9870a54e9f8cfd146016f140] dpt_i20: BKL pushdown
    git-bisect good dea3f665d6fa263a9870a54e9f8cfd146016f140
    # good: [6606470dd1d628878383c96d10b52a77986ddac7] videodev: BKL pushdown
    git-bisect good 6606470dd1d628878383c96d10b52a77986ddac7
    # good: [9514dff918b947ae43b66517dc90d0e05548bd6a] Remove the lock_kernel() call from chrdev_open()
    git-bisect good 9514dff918b947ae43b66517dc90d0e05548bd6a
    # bad: [8f5934278d1d86590244c2791b28f77d67466007] Replace BKL with superblock lock in fat/msdos/vfat
    git-bisect bad 8f5934278d1d86590244c2791b28f77d67466007
    # good: [5ca6a93d802a9d110127556e5d3ed032fd273e03] bkl-removal viotape fixup
    git-bisect good 5ca6a93d802a9d110127556e5d3ed032fd273e03
    # bad: [8f5934278d1d86590244c2791b28f77d67466007] Replace BKL with superblock lock in fat/msdos/vfat
    git-bisect bad 8f5934278d1d86590244c2791b28f77d67466007

    And lastly here is my config (w/o the comments)...

    CONFIG_X86_32=y
    CONFIG_X86=y
    CONFIG_ARCH_DEFCONFIG="arch/x86/configs/i386_defconfig"
    CONFIG_GENERIC_TIME=y
    CONFIG_GENERIC_CMOS_UPDATE=y
    CONFIG_CLOCKSOURCE_WATCHDOG=y
    CONFIG_GENERIC_CLOCKEVENTS=y
    CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
    CONFIG_LOCKDEP_SUPPORT=y
    CONFIG_STACKTRACE_SUPPORT=y
    CONFIG_HAVE_LATENCYTOP_SUPPORT=y
    CONFIG_FAST_CMPXCHG_LOCAL=y
    CONFIG_MMU=y
    CONFIG_ZONE_DMA=y
    CONFIG_GENERIC_ISA_DMA=y
    CONFIG_GENERIC_IOMAP=y
    CONFIG_GENERIC_BUG=y
    CONFIG_GENERIC_HWEIGHT=y
    CONFIG_GENERIC_GPIO=y
    CONFIG_ARCH_MAY_HAVE_PC_FDC=y
    CONFIG_RWSEM_XCHGADD_ALGORITHM=y
    CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
    CONFIG_GENERIC_CALIBRATE_DELAY=y
    CONFIG_ARCH_HAS_CPU_RELAX=y
    CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
    CONFIG_ARCH_HIBERNATION_POSSIBLE=y
    CONFIG_ARCH_SUSPEND_POSSIBLE=y
    CONFIG_ARCH_POPULATES_NODE_MAP=y
    CONFIG_ARCH_SUPPORTS_AOUT=y
    CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
    CONFIG_GENERIC_HARDIRQS=y
    CONFIG_GENERIC_IRQ_PROBE=y
    CONFIG_X86_BIOS_REBOOT=y
    CONFIG_KTIME_SCALAR=y
    CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
    CONFIG_EXPERIMENTAL=y
    CONFIG_BROKEN_ON_SMP=y
    CONFIG_LOCK_KERNEL=y
    CONFIG_INIT_ENV_ARG_LIMIT=32
    CONFIG_LOCALVERSION="-bisect"
    CONFIG_LOCALVERSION_AUTO=y
    CONFIG_SWAP=y
    CONFIG_SYSVIPC=y
    CONFIG_SYSVIPC_SYSCTL=y
    CONFIG_POSIX_MQUEUE=y
    CONFIG_BSD_PROCESS_ACCT=y
    CONFIG_BSD_PROCESS_ACCT_V3=y
    CONFIG_TASKSTATS=y
    CONFIG_TASK_DELAY_ACCT=y
    CONFIG_TASK_XACCT=y
    CONFIG_TASK_IO_ACCOUNTING=y
    CONFIG_AUDIT=y
    CONFIG_AUDITSYSCALL=y
    CONFIG_AUDIT_TREE=y
    CONFIG_IKCONFIG=y
    CONFIG_IKCONFIG_PROC=y
    CONFIG_LOG_BUF_SHIFT=14
    CONFIG_CGROUPS=y
    CONFIG_CGROUP_NS=y
    CONFIG_CGROUP_DEVICE=y
    CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
    CONFIG_GROUP_SCHED=y
    CONFIG_FAIR_GROUP_SCHED=y
    CONFIG_RT_GROUP_SCHED=y
    CONFIG_USER_SCHED=y
    CONFIG_CGROUP_CPUACCT=y
    CONFIG_RESOURCE_COUNTERS=y
    CONFIG_MM_OWNER=y
    CONFIG_CGROUP_MEM_RES_CTLR=y
    CONFIG_RELAY=y
    CONFIG_NAMESPACES=y
    CONFIG_UTS_NS=y
    CONFIG_IPC_NS=y
    CONFIG_USER_NS=y
    CONFIG_PID_NS=y
    CONFIG_BLK_DEV_INITRD=y
    CONFIG_INITRAMFS_SOURCE=""
    CONFIG_SYSCTL=y
    CONFIG_EMBEDDED=y
    CONFIG_UID16=y
    CONFIG_KALLSYMS=y
    CONFIG_KALLSYMS_ALL=y
    CONFIG_HOTPLUG=y
    CONFIG_PRINTK=y
    CONFIG_BUG=y
    CONFIG_ELF_CORE=y
    CONFIG_BASE_FULL=y
    CONFIG_FUTEX=y
    CONFIG_ANON_INODES=y
    CONFIG_EPOLL=y
    CONFIG_SIGNALFD=y
    CONFIG_TIMERFD=y
    CONFIG_EVENTFD=y
    CONFIG_SHMEM=y
    CONFIG_VM_EVENT_COUNTERS=y
    CONFIG_SLUB_DEBUG=y
    CONFIG_SLUB=y
    CONFIG_PROFILING=y
    CONFIG_MARKERS=y
    CONFIG_OPROFILE=m
    CONFIG_HAVE_OPROFILE=y
    CONFIG_KPROBES=y
    CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
    CONFIG_KRETPROBES=y
    CONFIG_HAVE_IOREMAP_PROT=y
    CONFIG_HAVE_KPROBES=y
    CONFIG_HAVE_KRETPROBES=y
    CONFIG_PROC_PAGE_MONITOR=y
    CONFIG_HAVE_GENERIC_DMA_COHERENT=y
    CONFIG_SLABINFO=y
    CONFIG_RT_MUTEXES=y
    CONFIG_BASE_SMALL=0
    CONFIG_MODULES=y
    CONFIG_MODULE_UNLOAD=y
    CONFIG_MODULE_FORCE_UNLOAD=y
    CONFIG_MODVERSIONS=y
    CONFIG_KMOD=y
    CONFIG_BLOCK=y
    CONFIG_LBD=y
    CONFIG_BLK_DEV_BSG=y
    CONFIG_BLK_DEV_INTEGRITY=y
    CONFIG_IOSCHED_NOOP=y
    CONFIG_IOSCHED_AS=y
    CONFIG_IOSCHED_DEADLINE=y
    CONFIG_IOSCHED_CFQ=y
    CONFIG_DEFAULT_AS=y
    CONFIG_DEFAULT_IOSCHED="anticipatory"
    CONFIG_CLASSIC_RCU=y
    CONFIG_TICK_ONESHOT=y
    CONFIG_NO_HZ=y
    CONFIG_HIGH_RES_TIMERS=y
    CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
    CONFIG_X86_FIND_SMP_CONFIG=y
    CONFIG_X86_MPPARSE=y
    CONFIG_X86_PC=y
    CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y
    CONFIG_PARAVIRT_GUEST=y
    CONFIG_XEN_SAVE_RESTORE=y
    CONFIG_LGUEST_GUEST=y
    CONFIG_PARAVIRT=y
    CONFIG_PARAVIRT_DEBUG=y
    CONFIG_MEMTEST=y
    CONFIG_MPENTIUMM=y
    CONFIG_X86_CPU=y
    CONFIG_X86_CMPXCHG=y
    CONFIG_X86_L1_CACHE_SHIFT=6
    CONFIG_X86_XADD=y
    CONFIG_X86_WP_WORKS_OK=y
    CONFIG_X86_INVLPG=y
    CONFIG_X86_BSWAP=y
    CONFIG_X86_POPAD_OK=y
    CONFIG_X86_INTEL_USERCOPY=y
    CONFIG_X86_USE_PPRO_CHECKSUM=y
    CONFIG_X86_P6_NOP=y
    CONFIG_X86_TSC=y
    CONFIG_X86_CMOV=y
    CONFIG_X86_MINIMUM_CPU_FAMILY=6
    CONFIG_X86_DEBUGCTLMSR=y
    CONFIG_HPET_TIMER=y
    CONFIG_HPET_EMULATE_RTC=y
    CONFIG_DMI=y
    CONFIG_PREEMPT=y
    CONFIG_X86_UP_APIC=y
    CONFIG_X86_UP_IOAPIC=y
    CONFIG_X86_LOCAL_APIC=y
    CONFIG_X86_IO_APIC=y
    CONFIG_X86_MCE=y
    CONFIG_X86_MCE_NONFATAL=m
    CONFIG_X86_MCE_P4THERMAL=y
    CONFIG_VM86=y
    CONFIG_MICROCODE=m
    CONFIG_MICROCODE_OLD_INTERFACE=y
    CONFIG_X86_MSR=m
    CONFIG_X86_CPUID=m
    CONFIG_HIGHMEM4G=y
    CONFIG_VMSPLIT_3G=y
    CONFIG_PAGE_OFFSET=0xC0000000
    CONFIG_HIGHMEM=y
    CONFIG_ARCH_FLATMEM_ENABLE=y
    CONFIG_ARCH_SPARSEMEM_ENABLE=y
    CONFIG_ARCH_SELECT_MEMORY_MODEL=y
    CONFIG_SELECT_MEMORY_MODEL=y
    CONFIG_FLATMEM_MANUAL=y
    CONFIG_FLATMEM=y
    CONFIG_FLAT_NODE_MEM_MAP=y
    CONFIG_SPARSEMEM_STATIC=y
    CONFIG_PAGEFLAGS_EXTENDED=y
    CONFIG_SPLIT_PTLOCK_CPUS=4
    CONFIG_ZONE_DMA_FLAG=1
    CONFIG_BOUNCE=y
    CONFIG_VIRT_TO_BUS=y
    CONFIG_MTRR=y
    CONFIG_SECCOMP=y
    CONFIG_HZ_250=y
    CONFIG_HZ=250
    CONFIG_SCHED_HRTICK=y
    CONFIG_KEXEC=y
    CONFIG_KEXEC_JUMP=y
    CONFIG_PHYSICAL_START=0x100000
    CONFIG_PHYSICAL_ALIGN=0x100000
    CONFIG_COMPAT_VDSO=y
    CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
    CONFIG_PM=y
    CONFIG_PM_SLEEP=y
    CONFIG_SUSPEND=y
    CONFIG_SUSPEND_FREEZER=y
    CONFIG_HIBERNATION=y
    CONFIG_PM_STD_PARTITION="/dev/sda1"
    CONFIG_ACPI=y
    CONFIG_ACPI_SLEEP=y
    CONFIG_ACPI_PROCFS=y
    CONFIG_ACPI_PROCFS_POWER=y
    CONFIG_ACPI_SYSFS_POWER=y
    CONFIG_ACPI_PROC_EVENT=y
    CONFIG_ACPI_AC=y
    CONFIG_ACPI_BATTERY=y
    CONFIG_ACPI_BUTTON=y
    CONFIG_ACPI_VIDEO=m
    CONFIG_ACPI_FAN=y
    CONFIG_ACPI_DOCK=y
    CONFIG_ACPI_BAY=m
    CONFIG_ACPI_PROCESSOR=y
    CONFIG_ACPI_THERMAL=y
    CONFIG_ACPI_WMI=m
    CONFIG_ACPI_ASUS=m
    CONFIG_ACPI_TOSHIBA=m
    CONFIG_ACPI_BLACKLIST_YEAR=2001
    CONFIG_ACPI_EC=y
    CONFIG_ACPI_PCI_SLOT=m
    CONFIG_ACPI_POWER=y
    CONFIG_ACPI_SYSTEM=y
    CONFIG_X86_PM_TIMER=y
    CONFIG_ACPI_CONTAINER=m
    CONFIG_ACPI_SBS=m
    CONFIG_CPU_FREQ=y
    CONFIG_CPU_FREQ_TABLE=y
    CONFIG_CPU_FREQ_STAT=y
    CONFIG_CPU_FREQ_STAT_DETAILS=y
    CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
    CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
    CONFIG_CPU_FREQ_GOV_POWERSAVE=y
    CONFIG_CPU_FREQ_GOV_USERSPACE=y
    CONFIG_CPU_FREQ_GOV_ONDEMAND=y
    CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
    CONFIG_X86_ACPI_CPUFREQ=y
    CONFIG_X86_SPEEDSTEP_CENTRINO=y
    CONFIG_X86_SPEEDSTEP_CENTRINO_TABLE=y
    CONFIG_X86_SPEEDSTEP_ICH=y
    CONFIG_X86_SPEEDSTEP_SMI=y
    CONFIG_X86_P4_CLOCKMOD=m
    CONFIG_X86_SPEEDSTEP_LIB=y
    CONFIG_X86_SPEEDSTEP_RELAXED_CAP_CHECK=y
    CONFIG_CPU_IDLE=y
    CONFIG_CPU_IDLE_GOV_LADDER=y
    CONFIG_CPU_IDLE_GOV_MENU=y
    CONFIG_PCI=y
    CONFIG_PCI_GOANY=y
    CONFIG_PCI_BIOS=y
    CONFIG_PCI_DIRECT=y
    CONFIG_PCI_MMCONFIG=y
    CONFIG_PCI_DOMAINS=y
    CONFIG_PCIEPORTBUS=y
    CONFIG_PCIEAER=y
    CONFIG_ARCH_SUPPORTS_MSI=y
    CONFIG_PCI_MSI=y
    CONFIG_PCI_LEGACY=y
    CONFIG_HT_IRQ=y
    CONFIG_ISA_DMA_API=y
    CONFIG_ISA=y
    CONFIG_PCCARD=m
    CONFIG_PCMCIA=m
    CONFIG_PCMCIA_LOAD_CIS=y
    CONFIG_PCMCIA_IOCTL=y
    CONFIG_CARDBUS=y
    CONFIG_YENTA=m
    CONFIG_YENTA_O2=y
    CONFIG_YENTA_RICOH=y
    CONFIG_YENTA_TI=y
    CONFIG_YENTA_ENE_TUNE=y
    CONFIG_YENTA_TOSHIBA=y
    CONFIG_PD6729=m
    CONFIG_I82092=m
    CONFIG_I82365=m
    CONFIG_TCIC=m
    CONFIG_PCMCIA_PROBE=y
    CONFIG_PCCARD_NONSTATIC=m
    CONFIG_BINFMT_ELF=y
    CONFIG_BINFMT_AOUT=m
    CONFIG_BINFMT_MISC=m
    CONFIG_NET=y
    CONFIG_PACKET=m
    CONFIG_PACKET_MMAP=y
    CONFIG_UNIX=y
    CONFIG_XFRM=y
    CONFIG_XFRM_USER=m
    CONFIG_XFRM_MIGRATE=y
    CONFIG_XFRM_STATISTICS=y
    CONFIG_XFRM_IPCOMP=y
    CONFIG_NET_KEY=m
    CONFIG_INET=y
    CONFIG_IP_MULTICAST=y
    CONFIG_IP_ADVANCED_ROUTER=y
    CONFIG_ASK_IP_FIB_HASH=y
    CONFIG_IP_FIB_HASH=y
    CONFIG_IP_MULTIPLE_TABLES=y
    CONFIG_IP_ROUTE_VERBOSE=y
    CONFIG_NET_IPIP=m
    CONFIG_NET_IPGRE=m
    CONFIG_NET_IPGRE_BROADCAST=y
    CONFIG_IP_MROUTE=y
    CONFIG_IP_PIMSM_V1=y
    CONFIG_IP_PIMSM_V2=y
    CONFIG_SYN_COOKIES=y
    CONFIG_INET_AH=y
    CONFIG_INET_ESP=y
    CONFIG_INET_IPCOMP=y
    CONFIG_INET_XFRM_TUNNEL=y
    CONFIG_INET_TUNNEL=y
    CONFIG_INET_XFRM_MODE_TRANSPORT=y
    CONFIG_INET_XFRM_MODE_TUNNEL=y
    CONFIG_INET_XFRM_MODE_BEET=y
    CONFIG_INET_LRO=y
    CONFIG_INET_DIAG=m
    CONFIG_INET_TCP_DIAG=m
    CONFIG_TCP_CONG_CUBIC=y
    CONFIG_DEFAULT_TCP_CONG="cubic"
    CONFIG_IPV6=m
    CONFIG_IPV6_PRIVACY=y
    CONFIG_INET6_AH=m
    CONFIG_INET6_ESP=m
    CONFIG_INET6_IPCOMP=m
    CONFIG_IPV6_MIP6=m
    CONFIG_INET6_XFRM_TUNNEL=m
    CONFIG_INET6_TUNNEL=m
    CONFIG_INET6_XFRM_MODE_TRANSPORT=m
    CONFIG_INET6_XFRM_MODE_TUNNEL=m
    CONFIG_INET6_XFRM_MODE_BEET=m
    CONFIG_IPV6_SIT=m
    CONFIG_IPV6_NDISC_NODETYPE=y
    CONFIG_IPV6_TUNNEL=m
    CONFIG_NETFILTER=y
    CONFIG_NETFILTER_ADVANCED=y
    CONFIG_BRIDGE_NETFILTER=y
    CONFIG_NETFILTER_NETLINK=m
    CONFIG_NETFILTER_NETLINK_QUEUE=m
    CONFIG_NETFILTER_NETLINK_LOG=m
    CONFIG_NF_CONNTRACK=m
    CONFIG_NF_CT_ACCT=y
    CONFIG_NF_CONNTRACK_MARK=y
    CONFIG_NF_CONNTRACK_EVENTS=y
    CONFIG_NF_CT_PROTO_DCCP=m
    CONFIG_NF_CT_PROTO_GRE=m
    CONFIG_NF_CT_PROTO_SCTP=m
    CONFIG_NF_CT_PROTO_UDPLITE=m
    CONFIG_NF_CONNTRACK_AMANDA=m
    CONFIG_NF_CONNTRACK_FTP=m
    CONFIG_NF_CONNTRACK_H323=m
    CONFIG_NF_CONNTRACK_IRC=m
    CONFIG_NF_CONNTRACK_NETBIOS_NS=m
    CONFIG_NF_CONNTRACK_PPTP=m
    CONFIG_NF_CONNTRACK_SANE=m
    CONFIG_NF_CONNTRACK_SIP=m
    CONFIG_NF_CONNTRACK_TFTP=m
    CONFIG_NF_CT_NETLINK=m
    CONFIG_NETFILTER_XTABLES=y
    CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
    CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
    CONFIG_NETFILTER_XT_TARGET_DSCP=m
    CONFIG_NETFILTER_XT_TARGET_MARK=m
    CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
    CONFIG_NETFILTER_XT_TARGET_NFLOG=m
    CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
    CONFIG_NETFILTER_XT_TARGET_RATEEST=m
    CONFIG_NETFILTER_XT_TARGET_TRACE=m
    CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
    CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m
    CONFIG_NETFILTER_XT_MATCH_COMMENT=m
    CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
    CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
    CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
    CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
    CONFIG_NETFILTER_XT_MATCH_DCCP=m
    CONFIG_NETFILTER_XT_MATCH_DSCP=m
    CONFIG_NETFILTER_XT_MATCH_ESP=m
    CONFIG_NETFILTER_XT_MATCH_HELPER=m
    CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
    CONFIG_NETFILTER_XT_MATCH_LENGTH=m
    CONFIG_NETFILTER_XT_MATCH_LIMIT=m
    CONFIG_NETFILTER_XT_MATCH_MAC=m
    CONFIG_NETFILTER_XT_MATCH_MARK=m
    CONFIG_NETFILTER_XT_MATCH_OWNER=m
    CONFIG_NETFILTER_XT_MATCH_POLICY=m
    CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
    CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
    CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
    CONFIG_NETFILTER_XT_MATCH_QUOTA=m
    CONFIG_NETFILTER_XT_MATCH_RATEEST=m
    CONFIG_NETFILTER_XT_MATCH_REALM=m
    CONFIG_NETFILTER_XT_MATCH_SCTP=m
    CONFIG_NETFILTER_XT_MATCH_STATE=m
    CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
    CONFIG_NETFILTER_XT_MATCH_STRING=m
    CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
    CONFIG_NETFILTER_XT_MATCH_TIME=m
    CONFIG_NETFILTER_XT_MATCH_U32=m
    CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
    CONFIG_NF_CONNTRACK_IPV4=m
    CONFIG_NF_CONNTRACK_PROC_COMPAT=y
    CONFIG_IP_NF_QUEUE=m
    CONFIG_IP_NF_IPTABLES=y
    CONFIG_IP_NF_MATCH_RECENT=m
    CONFIG_IP_NF_MATCH_ECN=m
    CONFIG_IP_NF_MATCH_AH=m
    CONFIG_IP_NF_MATCH_TTL=m
    CONFIG_IP_NF_MATCH_ADDRTYPE=m
    CONFIG_IP_NF_FILTER=m
    CONFIG_IP_NF_TARGET_REJECT=m
    CONFIG_IP_NF_TARGET_LOG=m
    CONFIG_IP_NF_TARGET_ULOG=m
    CONFIG_NF_NAT=m
    CONFIG_NF_NAT_NEEDED=y
    CONFIG_IP_NF_TARGET_MASQUERADE=m
    CONFIG_IP_NF_TARGET_REDIRECT=m
    CONFIG_IP_NF_TARGET_NETMAP=m
    CONFIG_NF_NAT_SNMP_BASIC=m
    CONFIG_NF_NAT_PROTO_DCCP=m
    CONFIG_NF_NAT_PROTO_GRE=m
    CONFIG_NF_NAT_PROTO_UDPLITE=m
    CONFIG_NF_NAT_PROTO_SCTP=m
    CONFIG_NF_NAT_FTP=m
    CONFIG_NF_NAT_IRC=m
    CONFIG_NF_NAT_TFTP=m
    CONFIG_NF_NAT_AMANDA=m
    CONFIG_NF_NAT_PPTP=m
    CONFIG_NF_NAT_H323=m
    CONFIG_NF_NAT_SIP=m
    CONFIG_IP_NF_MANGLE=m
    CONFIG_IP_NF_TARGET_ECN=m
    CONFIG_IP_NF_TARGET_TTL=m
    CONFIG_IP_NF_TARGET_CLUSTERIP=m
    CONFIG_IP_NF_RAW=m
    CONFIG_IP_NF_ARPTABLES=m
    CONFIG_IP_NF_ARPFILTER=m
    CONFIG_IP_NF_ARP_MANGLE=m
    CONFIG_NF_CONNTRACK_IPV6=m
    CONFIG_IP6_NF_QUEUE=m
    CONFIG_IP6_NF_IPTABLES=m
    CONFIG_IP6_NF_MATCH_RT=m
    CONFIG_IP6_NF_MATCH_OPTS=m
    CONFIG_IP6_NF_MATCH_FRAG=m
    CONFIG_IP6_NF_MATCH_HL=m
    CONFIG_IP6_NF_MATCH_IPV6HEADER=m
    CONFIG_IP6_NF_MATCH_AH=m
    CONFIG_IP6_NF_MATCH_MH=m
    CONFIG_IP6_NF_MATCH_EUI64=m
    CONFIG_IP6_NF_FILTER=m
    CONFIG_IP6_NF_TARGET_LOG=m
    CONFIG_IP6_NF_TARGET_REJECT=m
    CONFIG_IP6_NF_MANGLE=m
    CONFIG_IP6_NF_TARGET_HL=m
    CONFIG_IP6_NF_RAW=m
    CONFIG_BRIDGE_NF_EBTABLES=m
    CONFIG_BRIDGE_EBT_BROUTE=m
    CONFIG_BRIDGE_EBT_T_FILTER=m
    CONFIG_BRIDGE_EBT_T_NAT=m
    CONFIG_BRIDGE_EBT_802_3=m
    CONFIG_BRIDGE_EBT_AMONG=m
    CONFIG_BRIDGE_EBT_ARP=m
    CONFIG_BRIDGE_EBT_IP=m
    CONFIG_BRIDGE_EBT_IP6=m
    CONFIG_BRIDGE_EBT_LIMIT=m
    CONFIG_BRIDGE_EBT_MARK=m
    CONFIG_BRIDGE_EBT_PKTTYPE=m
    CONFIG_BRIDGE_EBT_STP=m
    CONFIG_BRIDGE_EBT_VLAN=m
    CONFIG_BRIDGE_EBT_ARPREPLY=m
    CONFIG_BRIDGE_EBT_DNAT=m
    CONFIG_BRIDGE_EBT_MARK_T=m
    CONFIG_BRIDGE_EBT_REDIRECT=m
    CONFIG_BRIDGE_EBT_SNAT=m
    CONFIG_BRIDGE_EBT_LOG=m
    CONFIG_BRIDGE_EBT_ULOG=m
    CONFIG_BRIDGE_EBT_NFLOG=m
    CONFIG_IP_SCTP=m
    CONFIG_SCTP_HMAC_MD5=y
    CONFIG_STP=m
    CONFIG_GARP=m
    CONFIG_BRIDGE=m
    CONFIG_VLAN_8021Q=m
    CONFIG_VLAN_8021Q_GVRP=y
    CONFIG_LLC=m
    CONFIG_NET_SCHED=y
    CONFIG_NET_SCH_CBQ=m
    CONFIG_NET_SCH_HTB=m
    CONFIG_NET_SCH_HFSC=m
    CONFIG_NET_SCH_PRIO=m
    CONFIG_NET_SCH_RED=m
    CONFIG_NET_SCH_SFQ=m
    CONFIG_NET_SCH_TEQL=m
    CONFIG_NET_SCH_TBF=m
    CONFIG_NET_SCH_GRED=m
    CONFIG_NET_SCH_DSMARK=m
    CONFIG_NET_SCH_NETEM=m
    CONFIG_NET_SCH_INGRESS=m
    CONFIG_NET_CLS=y
    CONFIG_NET_CLS_BASIC=m
    CONFIG_NET_CLS_TCINDEX=m
    CONFIG_NET_CLS_ROUTE4=m
    CONFIG_NET_CLS_ROUTE=y
    CONFIG_NET_CLS_FW=m
    CONFIG_NET_CLS_U32=m
    CONFIG_CLS_U32_PERF=y
    CONFIG_CLS_U32_MARK=y
    CONFIG_NET_CLS_RSVP=m
    CONFIG_NET_CLS_RSVP6=m
    CONFIG_NET_CLS_FLOW=m
    CONFIG_NET_EMATCH=y
    CONFIG_NET_EMATCH_STACK=32
    CONFIG_NET_EMATCH_CMP=m
    CONFIG_NET_EMATCH_NBYTE=m
    CONFIG_NET_EMATCH_U32=m
    CONFIG_NET_EMATCH_META=m
    CONFIG_NET_EMATCH_TEXT=m
    CONFIG_NET_CLS_ACT=y
    CONFIG_NET_ACT_POLICE=y
    CONFIG_NET_ACT_GACT=m
    CONFIG_GACT_PROB=y
    CONFIG_NET_ACT_MIRRED=m
    CONFIG_NET_ACT_IPT=m
    CONFIG_NET_ACT_NAT=m
    CONFIG_NET_ACT_PEDIT=m
    CONFIG_NET_SCH_FIFO=y
    CONFIG_NET_PKTGEN=m
    CONFIG_NET_TCPPROBE=m
    CONFIG_IRDA=m
    CONFIG_IRLAN=m
    CONFIG_IRNET=m
    CONFIG_IRCOMM=m
    CONFIG_IRDA_ULTRA=y
    CONFIG_IRDA_CACHE_LAST_LSAP=y
    CONFIG_IRDA_FAST_RR=y
    CONFIG_IRDA_DEBUG=y
    CONFIG_IRTTY_SIR=m
    CONFIG_DONGLE=y
    CONFIG_ESI_DONGLE=m
    CONFIG_ACTISYS_DONGLE=m
    CONFIG_TEKRAM_DONGLE=m
    CONFIG_TOIM3232_DONGLE=m
    CONFIG_LITELINK_DONGLE=m
    CONFIG_MA600_DONGLE=m
    CONFIG_GIRBIL_DONGLE=m
    CONFIG_MCP2120_DONGLE=m
    CONFIG_OLD_BELKIN_DONGLE=m
    CONFIG_ACT200L_DONGLE=m
    CONFIG_KINGSUN_DONGLE=m
    CONFIG_KSDAZZLE_DONGLE=m
    CONFIG_KS959_DONGLE=m
    CONFIG_USB_IRDA=m
    CONFIG_SIGMATEL_FIR=m
    CONFIG_NSC_FIR=m
    CONFIG_WINBOND_FIR=m
    CONFIG_TOSHIBA_FIR=m
    CONFIG_SMC_IRCC_FIR=m
    CONFIG_ALI_FIR=m
    CONFIG_VLSI_FIR=m
    CONFIG_VIA_FIR=m
    CONFIG_MCS_FIR=m
    CONFIG_AF_RXRPC=m
    CONFIG_RXKAD=m
    CONFIG_FIB_RULES=y
    CONFIG_CFG80211=m
    CONFIG_NL80211=y
    CONFIG_WIRELESS_EXT=y
    CONFIG_WIRELESS_EXT_SYSFS=y
    CONFIG_MAC80211=m
    CONFIG_MAC80211_RC_PID=y
    CONFIG_MAC80211_RC_DEFAULT_PID=y
    CONFIG_MAC80211_RC_DEFAULT="pid"
    CONFIG_MAC80211_MESH=y
    CONFIG_MAC80211_LEDS=y
    CONFIG_MAC80211_DEBUGFS=y
    CONFIG_IEEE80211=m
    CONFIG_IEEE80211_CRYPT_WEP=m
    CONFIG_IEEE80211_CRYPT_CCMP=m
    CONFIG_IEEE80211_CRYPT_TKIP=m
    CONFIG_RFKILL=m
    CONFIG_RFKILL_INPUT=m
    CONFIG_RFKILL_LEDS=y
    CONFIG_NET_9P=m
    CONFIG_NET_9P_VIRTIO=m
    CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
    CONFIG_STANDALONE=y
    CONFIG_PREVENT_FIRMWARE_BUILD=y
    CONFIG_FW_LOADER=m
    CONFIG_FIRMWARE_IN_KERNEL=y
    CONFIG_EXTRA_FIRMWARE=""
    CONFIG_CONNECTOR=m
    CONFIG_PARPORT=m
    CONFIG_PARPORT_PC=m
    CONFIG_PARPORT_SERIAL=m
    CONFIG_PARPORT_PC_FIFO=y
    CONFIG_PARPORT_PC_PCMCIA=m
    CONFIG_PARPORT_1284=y
    CONFIG_PARPORT_NOT_PC=y
    CONFIG_PNP=y
    CONFIG_ISAPNP=y
    CONFIG_PNPBIOS=y
    CONFIG_PNPBIOS_PROC_FS=y
    CONFIG_PNPACPI=y
    CONFIG_BLK_DEV=y
    CONFIG_BLK_DEV_FD=m
    CONFIG_BLK_DEV_LOOP=y
    CONFIG_BLK_DEV_CRYPTOLOOP=m
    CONFIG_BLK_DEV_NBD=m
    CONFIG_BLK_DEV_RAM=y
    CONFIG_BLK_DEV_RAM_COUNT=16
    CONFIG_BLK_DEV_RAM_SIZE=8192
    CONFIG_CDROM_PKTCDVD=m
    CONFIG_CDROM_PKTCDVD_BUFFERS=8
    CONFIG_ATA_OVER_ETH=m
    CONFIG_VIRTIO_BLK=m
    CONFIG_MISC_DEVICES=y
    CONFIG_IBM_ASM=m
    CONFIG_PHANTOM=m
    CONFIG_EEPROM_93CX6=m
    CONFIG_TIFM_CORE=m
    CONFIG_TIFM_7XX1=m
    CONFIG_THINKPAD_ACPI=m
    CONFIG_THINKPAD_ACPI_BAY=y
    CONFIG_THINKPAD_ACPI_VIDEO=y
    CONFIG_ENCLOSURE_SERVICES=m
    CONFIG_HAVE_IDE=y
    CONFIG_IDE=m
    CONFIG_BLK_DEV_IDE=m
    CONFIG_IDE_TIMINGS=y
    CONFIG_IDE_ATAPI=y
    CONFIG_BLK_DEV_IDEDISK=m
    CONFIG_BLK_DEV_IDECS=m
    CONFIG_BLK_DEV_IDECD=m
    CONFIG_BLK_DEV_IDECD_VERBOSE_ERRORS=y
    CONFIG_BLK_DEV_IDETAPE=m
    CONFIG_BLK_DEV_IDEFLOPPY=m
    CONFIG_BLK_DEV_IDESCSI=m
    CONFIG_IDE_PROC_FS=y
    CONFIG_IDE_GENERIC=m
    CONFIG_BLK_DEV_CMD640=m
    CONFIG_BLK_DEV_IDEDMA_SFF=y
    CONFIG_BLK_DEV_IDEPCI=y
    CONFIG_BLK_DEV_GENERIC=m
    CONFIG_BLK_DEV_IDEDMA_PCI=y
    CONFIG_BLK_DEV_AEC62XX=m
    CONFIG_BLK_DEV_PIIX=m
    CONFIG_BLK_DEV_IDEDMA=y
    CONFIG_SCSI=y
    CONFIG_SCSI_DMA=y
    CONFIG_SCSI_NETLINK=y
    CONFIG_SCSI_PROC_FS=y
    CONFIG_BLK_DEV_SD=y
    CONFIG_CHR_DEV_ST=m
    CONFIG_CHR_DEV_OSST=m
    CONFIG_BLK_DEV_SR=m
    CONFIG_CHR_DEV_SG=m
    CONFIG_CHR_DEV_SCH=m
    CONFIG_SCSI_ENCLOSURE=m
    CONFIG_SCSI_MULTI_LUN=y
    CONFIG_SCSI_CONSTANTS=y
    CONFIG_SCSI_LOGGING=y
    CONFIG_SCSI_WAIT_SCAN=m
    CONFIG_SCSI_SPI_ATTRS=m
    CONFIG_SCSI_FC_ATTRS=y
    CONFIG_SCSI_ISCSI_ATTRS=m
    CONFIG_SCSI_SAS_ATTRS=m
    CONFIG_SCSI_LOWLEVEL=y
    CONFIG_SCSI_ACARD=m
    CONFIG_ATA=y
    CONFIG_ATA_ACPI=y
    CONFIG_ATA_SFF=y
    CONFIG_ATA_PIIX=y
    CONFIG_PATA_ACPI=m
    CONFIG_PATA_MPIIX=y
    CONFIG_PATA_SCH=y
    CONFIG_IEEE1394=m
    CONFIG_IEEE1394_OHCI1394=m
    CONFIG_IEEE1394_PCILYNX=m
    CONFIG_IEEE1394_SBP2=m
    CONFIG_IEEE1394_ETH1394_ROM_ENTRY=y
    CONFIG_IEEE1394_ETH1394=m
    CONFIG_IEEE1394_RAWIO=m
    CONFIG_IEEE1394_VIDEO1394=m
    CONFIG_IEEE1394_DV1394=m
    CONFIG_I2O=m
    CONFIG_I2O_LCT_NOTIFY_ON_CHANGES=y
    CONFIG_I2O_EXT_ADAPTEC=y
    CONFIG_I2O_CONFIG=m
    CONFIG_I2O_CONFIG_OLD_IOCTL=y
    CONFIG_I2O_BUS=m
    CONFIG_I2O_BLOCK=m
    CONFIG_I2O_SCSI=m
    CONFIG_I2O_PROC=m
    CONFIG_NETDEVICES=y
    CONFIG_IFB=m
    CONFIG_DUMMY=m
    CONFIG_BONDING=m
    CONFIG_MACVLAN=m
    CONFIG_EQUALIZER=m
    CONFIG_TUN=m
    CONFIG_VETH=m
    CONFIG_NET_SB1000=m
    CONFIG_PHYLIB=y
    CONFIG_MARVELL_PHY=y
    CONFIG_DAVICOM_PHY=y
    CONFIG_QSEMI_PHY=y
    CONFIG_LXT_PHY=y
    CONFIG_CICADA_PHY=y
    CONFIG_VITESSE_PHY=y
    CONFIG_SMSC_PHY=y
    CONFIG_BROADCOM_PHY=y
    CONFIG_ICPLUS_PHY=y
    CONFIG_REALTEK_PHY=y
    CONFIG_FIXED_PHY=y
    CONFIG_MII=m
    CONFIG_NETDEV_1000=y
    CONFIG_E1000=m
    CONFIG_E1000E=m
    CONFIG_IP1000=m
    CONFIG_IGB=m
    CONFIG_TIGON3=y
    CONFIG_BNX2=m
    CONFIG_WLAN_PRE80211=y
    CONFIG_STRIP=m
    CONFIG_ARLAN=m
    CONFIG_WAVELAN=m
    CONFIG_PCMCIA_WAVELAN=m
    CONFIG_PCMCIA_NETWAVE=m
    CONFIG_WLAN_80211=y
    CONFIG_PCMCIA_RAYCS=m
    CONFIG_IPW2100=m
    CONFIG_IPW2100_MONITOR=y
    CONFIG_IPW2200=m
    CONFIG_IPW2200_MONITOR=y
    CONFIG_IPW2200_RADIOTAP=y
    CONFIG_IPW2200_PROMISCUOUS=y
    CONFIG_IPW2200_QOS=y
    CONFIG_LIBERTAS=m
    CONFIG_LIBERTAS_USB=m
    CONFIG_LIBERTAS_CS=m
    CONFIG_LIBERTAS_SDIO=m
    CONFIG_AIRO=m
    CONFIG_HERMES=m
    CONFIG_PLX_HERMES=m
    CONFIG_TMD_HERMES=m
    CONFIG_NORTEL_HERMES=m
    CONFIG_PCI_HERMES=m
    CONFIG_PCMCIA_HERMES=m
    CONFIG_PCMCIA_SPECTRUM=m
    CONFIG_ATMEL=m
    CONFIG_PCI_ATMEL=m
    CONFIG_PCMCIA_ATMEL=m
    CONFIG_AIRO_CS=m
    CONFIG_PCMCIA_WL3501=m
    CONFIG_PRISM54=m
    CONFIG_USB_ZD1201=m
    CONFIG_USB_NET_RNDIS_WLAN=m
    CONFIG_RTL8180=m
    CONFIG_RTL8187=m
    CONFIG_ADM8211=m
    CONFIG_P54_COMMON=m
    CONFIG_P54_USB=m
    CONFIG_P54_PCI=m
    CONFIG_ATH5K=m
    CONFIG_HOSTAP=m
    CONFIG_HOSTAP_FIRMWARE=y
    CONFIG_HOSTAP_PLX=m
    CONFIG_HOSTAP_PCI=m
    CONFIG_HOSTAP_CS=m
    CONFIG_B43=m
    CONFIG_B43_PCI_AUTOSELECT=y
    CONFIG_B43_PCICORE_AUTOSELECT=y
    CONFIG_B43_PCMCIA=y
    CONFIG_B43_PIO=y
    CONFIG_B43_LEDS=y
    CONFIG_B43_RFKILL=y
    CONFIG_B43LEGACY=m
    CONFIG_B43LEGACY_PCI_AUTOSELECT=y
    CONFIG_B43LEGACY_PCICORE_AUTOSELECT=y
    CONFIG_B43LEGACY_LEDS=y
    CONFIG_B43LEGACY_RFKILL=y
    CONFIG_B43LEGACY_DEBUG=y
    CONFIG_B43LEGACY_DMA=y
    CONFIG_B43LEGACY_PIO=y
    CONFIG_B43LEGACY_DMA_AND_PIO_MODE=y
    CONFIG_ZD1211RW=m
    CONFIG_USB_CATC=m
    CONFIG_USB_KAWETH=m
    CONFIG_USB_PEGASUS=m
    CONFIG_USB_RTL8150=m
    CONFIG_USB_USBNET=m
    CONFIG_USB_NET_AX8817X=m
    CONFIG_USB_NET_CDCETHER=m
    CONFIG_USB_NET_DM9601=m
    CONFIG_USB_NET_GL620A=m
    CONFIG_USB_NET_NET1080=m
    CONFIG_USB_NET_PLUSB=m
    CONFIG_USB_NET_MCS7830=m
    CONFIG_USB_NET_RNDIS_HOST=m
    CONFIG_USB_NET_CDC_SUBSET=m
    CONFIG_USB_ALI_M5632=y
    CONFIG_USB_AN2720=y
    CONFIG_USB_BELKIN=y
    CONFIG_USB_ARMLINUX=y
    CONFIG_USB_EPSON2888=y
    CONFIG_USB_KC2190=y
    CONFIG_USB_NET_ZAURUS=m
    CONFIG_NET_PCMCIA=y
    CONFIG_PCMCIA_3C589=m
    CONFIG_PCMCIA_3C574=m
    CONFIG_PCMCIA_FMVJ18X=m
    CONFIG_PCMCIA_PCNET=m
    CONFIG_PCMCIA_NMCLAN=m
    CONFIG_PCMCIA_SMC91C92=m
    CONFIG_PCMCIA_XIRC2PS=m
    CONFIG_PCMCIA_AXNET=m
    CONFIG_PPP=m
    CONFIG_PPP_MULTILINK=y
    CONFIG_PPP_FILTER=y
    CONFIG_PPP_ASYNC=m
    CONFIG_PPP_SYNC_TTY=m
    CONFIG_PPP_DEFLATE=m
    CONFIG_PPP_BSDCOMP=m
    CONFIG_PPP_MPPE=m
    CONFIG_PPPOE=m
    CONFIG_PPPOL2TP=m
    CONFIG_SLHC=m
    CONFIG_NETCONSOLE=m
    CONFIG_NETPOLL=y
    CONFIG_NET_POLL_CONTROLLER=y
    CONFIG_VIRTIO_NET=m
    CONFIG_INPUT=y
    CONFIG_INPUT_POLLDEV=m
    CONFIG_INPUT_MOUSEDEV=m
    CONFIG_INPUT_MOUSEDEV_PSAUX=y
    CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
    CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
    CONFIG_INPUT_JOYDEV=m
    CONFIG_INPUT_EVDEV=y
    CONFIG_INPUT_EVBUG=m
    CONFIG_INPUT_KEYBOARD=y
    CONFIG_KEYBOARD_ATKBD=y
    CONFIG_KEYBOARD_SUNKBD=m
    CONFIG_KEYBOARD_LKKBD=m
    CONFIG_KEYBOARD_XTKBD=m
    CONFIG_KEYBOARD_NEWTON=m
    CONFIG_KEYBOARD_GPIO=m
    CONFIG_INPUT_MOUSE=y
    CONFIG_MOUSE_PS2=m
    CONFIG_MOUSE_PS2_ALPS=y
    CONFIG_MOUSE_PS2_LOGIPS2PP=y
    CONFIG_MOUSE_PS2_SYNAPTICS=y
    CONFIG_MOUSE_PS2_LIFEBOOK=y
    CONFIG_MOUSE_PS2_TRACKPOINT=y
    CONFIG_MOUSE_SERIAL=m
    CONFIG_MOUSE_INPORT=m
    CONFIG_MOUSE_LOGIBM=m
    CONFIG_MOUSE_PC110PAD=m
    CONFIG_MOUSE_VSXXXAA=m
    CONFIG_MOUSE_GPIO=m
    CONFIG_INPUT_JOYSTICK=y
    CONFIG_JOYSTICK_ANALOG=m
    CONFIG_JOYSTICK_A3D=m
    CONFIG_JOYSTICK_ADI=m
    CONFIG_JOYSTICK_COBRA=m
    CONFIG_JOYSTICK_GF2K=m
    CONFIG_JOYSTICK_GRIP=m
    CONFIG_JOYSTICK_GRIP_MP=m
    CONFIG_JOYSTICK_GUILLEMOT=m
    CONFIG_JOYSTICK_INTERACT=m
    CONFIG_JOYSTICK_SIDEWINDER=m
    CONFIG_JOYSTICK_TMDC=m
    CONFIG_JOYSTICK_IFORCE=m
    CONFIG_JOYSTICK_IFORCE_USB=y
    CONFIG_JOYSTICK_IFORCE_232=y
    CONFIG_JOYSTICK_WARRIOR=m
    CONFIG_JOYSTICK_MAGELLAN=m
    CONFIG_JOYSTICK_SPACEORB=m
    CONFIG_JOYSTICK_SPACEBALL=m
    CONFIG_JOYSTICK_STINGER=m
    CONFIG_JOYSTICK_TWIDJOY=m
    CONFIG_JOYSTICK_DB9=m
    CONFIG_JOYSTICK_GAMECON=m
    CONFIG_JOYSTICK_TURBOGRAFX=m
    CONFIG_JOYSTICK_JOYDUMP=m
    CONFIG_INPUT_MISC=y
    CONFIG_INPUT_ATI_REMOTE=m
    CONFIG_INPUT_ATI_REMOTE2=m
    CONFIG_INPUT_KEYSPAN_REMOTE=m
    CONFIG_INPUT_POWERMATE=m
    CONFIG_INPUT_YEALINK=m
    CONFIG_INPUT_UINPUT=m
    CONFIG_SERIO=y
    CONFIG_SERIO_I8042=y
    CONFIG_SERIO_SERPORT=m
    CONFIG_SERIO_CT82C710=m
    CONFIG_SERIO_PARKBD=m
    CONFIG_SERIO_PCIPS2=m
    CONFIG_SERIO_LIBPS2=y
    CONFIG_SERIO_RAW=m
    CONFIG_GAMEPORT=m
    CONFIG_GAMEPORT_NS558=m
    CONFIG_GAMEPORT_L4=m
    CONFIG_GAMEPORT_EMU10K1=m
    CONFIG_GAMEPORT_FM801=m
    CONFIG_VT=y
    CONFIG_CONSOLE_TRANSLATIONS=y
    CONFIG_VT_CONSOLE=y
    CONFIG_HW_CONSOLE=y
    CONFIG_VT_HW_CONSOLE_BINDING=y
    CONFIG_DEVKMEM=y
    CONFIG_SERIAL_8250=y
    CONFIG_SERIAL_8250_CONSOLE=y
    CONFIG_FIX_EARLYCON_MEM=y
    CONFIG_SERIAL_8250_PCI=m
    CONFIG_SERIAL_8250_PNP=m
    CONFIG_SERIAL_8250_CS=m
    CONFIG_SERIAL_8250_NR_UARTS=4
    CONFIG_SERIAL_8250_RUNTIME_UARTS=4
    CONFIG_SERIAL_8250_EXTENDED=y
    CONFIG_SERIAL_8250_MANY_PORTS=y
    CONFIG_SERIAL_8250_SHARE_IRQ=y
    CONFIG_SERIAL_8250_RSA=y
    CONFIG_SERIAL_CORE=y
    CONFIG_SERIAL_CORE_CONSOLE=y
    CONFIG_CONSOLE_POLL=y
    CONFIG_UNIX98_PTYS=y
    CONFIG_LEGACY_PTYS=y
    CONFIG_LEGACY_PTY_COUNT=256
    CONFIG_PRINTER=m
    CONFIG_PPDEV=m
    CONFIG_HVC_DRIVER=y
    CONFIG_VIRTIO_CONSOLE=y
    CONFIG_IPMI_HANDLER=m
    CONFIG_IPMI_DEVICE_INTERFACE=m
    CONFIG_IPMI_SI=m
    CONFIG_IPMI_WATCHDOG=m
    CONFIG_IPMI_POWEROFF=m
    CONFIG_HW_RANDOM=y
    CONFIG_HW_RANDOM_INTEL=m
    CONFIG_HW_RANDOM_AMD=y
    CONFIG_HW_RANDOM_GEODE=y
    CONFIG_HW_RANDOM_VIA=y
    CONFIG_HW_RANDOM_VIRTIO=m
    CONFIG_NVRAM=m
    CONFIG_DTLK=m
    CONFIG_SYNCLINK_CS=m
    CONFIG_MWAVE=m
    CONFIG_RAW_DRIVER=m
    CONFIG_MAX_RAW_DEVS=256
    CONFIG_HPET=y
    CONFIG_HPET_MMAP=y
    CONFIG_HANGCHECK_TIMER=m
    CONFIG_TCG_TPM=m
    CONFIG_TCG_TIS=m
    CONFIG_TCG_NSC=m
    CONFIG_TCG_ATMEL=m
    CONFIG_DEVPORT=y
    CONFIG_I2C=y
    CONFIG_I2C_BOARDINFO=y
    CONFIG_I2C_CHARDEV=m
    CONFIG_I2C_HELPER_AUTO=y
    CONFIG_I2C_ALGOBIT=m
    CONFIG_I2C_ALGOPCF=m
    CONFIG_I2C_ALGOPCA=m
    CONFIG_I2C_ALI1535=m
    CONFIG_I2C_ALI1563=m
    CONFIG_I2C_ALI15X3=m
    CONFIG_I2C_AMD756=m
    CONFIG_I2C_AMD756_S4882=m
    CONFIG_I2C_AMD8111=m
    CONFIG_I2C_I801=y
    CONFIG_I2C_ISCH=m
    CONFIG_I2C_PIIX4=m
    CONFIG_I2C_NFORCE2=m
    CONFIG_I2C_SIS5595=m
    CONFIG_I2C_SIS630=m
    CONFIG_I2C_SIS96X=m
    CONFIG_I2C_VIA=m
    CONFIG_I2C_VIAPRO=m
    CONFIG_I2C_GPIO=m
    CONFIG_I2C_OCORES=m
    CONFIG_I2C_SIMTEC=m
    CONFIG_I2C_PARPORT=m
    CONFIG_I2C_PARPORT_LIGHT=m
    CONFIG_I2C_TAOS_EVM=m
    CONFIG_I2C_TINY_USB=m
    CONFIG_I2C_VOODOO3=m
    CONFIG_I2C_ELEKTOR=m
    CONFIG_I2C_PCA_ISA=m
    CONFIG_I2C_STUB=m
    CONFIG_SCx200_ACB=m
    CONFIG_AT24=m
    CONFIG_SENSORS_EEPROM=m
    CONFIG_SENSORS_PCF8591=m
    CONFIG_TPS65010=m
    CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
    CONFIG_GPIOLIB=y
    CONFIG_GPIO_SYSFS=y
    CONFIG_GPIO_MAX732X=m
    CONFIG_GPIO_PCA953X=m
    CONFIG_GPIO_PCF857X=m
    CONFIG_GPIO_BT8XX=m
    CONFIG_W1=m
    CONFIG_W1_CON=y
    CONFIG_W1_MASTER_GPIO=m
    CONFIG_W1_SLAVE_THERM=m
    CONFIG_W1_SLAVE_SMEM=m
    CONFIG_W1_SLAVE_DS2433=m
    CONFIG_POWER_SUPPLY=y
    CONFIG_HWMON=y
    CONFIG_HWMON_VID=m
    CONFIG_SENSORS_AD7418=m
    CONFIG_SENSORS_ADM1021=m
    CONFIG_SENSORS_ADM1025=m
    CONFIG_SENSORS_ADM1026=m
    CONFIG_SENSORS_ADM1029=m
    CONFIG_SENSORS_ADM1031=m
    CONFIG_SENSORS_ADM9240=m
    CONFIG_SENSORS_ADT7470=m
    CONFIG_SENSORS_ADT7473=m
    CONFIG_SENSORS_ASB100=m
    CONFIG_SENSORS_ATXP1=m
    CONFIG_SENSORS_DS1621=m
    CONFIG_SENSORS_I5K_AMB=m
    CONFIG_SENSORS_F71882FG=m
    CONFIG_SENSORS_F75375S=m
    CONFIG_SENSORS_FSCHER=m
    CONFIG_SENSORS_FSCPOS=m
    CONFIG_SENSORS_FSCHMD=m
    CONFIG_SENSORS_GL518SM=m
    CONFIG_SENSORS_GL520SM=m
    CONFIG_SENSORS_CORETEMP=m
    CONFIG_SENSORS_IBMPEX=m
    CONFIG_SENSORS_IT87=m
    CONFIG_SENSORS_LM63=m
    CONFIG_SENSORS_LM75=m
    CONFIG_SENSORS_LM77=m
    CONFIG_SENSORS_LM78=m
    CONFIG_SENSORS_LM80=m
    CONFIG_SENSORS_LM83=m
    CONFIG_SENSORS_LM85=m
    CONFIG_SENSORS_LM87=m
    CONFIG_SENSORS_LM90=m
    CONFIG_SENSORS_LM92=m
    CONFIG_SENSORS_LM93=m
    CONFIG_SENSORS_MAX1619=m
    CONFIG_SENSORS_MAX6650=m
    CONFIG_SENSORS_PC87360=m
    CONFIG_SENSORS_PC87427=m
    CONFIG_SENSORS_SIS5595=m
    CONFIG_SENSORS_DME1737=m
    CONFIG_SENSORS_SMSC47M1=m
    CONFIG_SENSORS_SMSC47B397=m
    CONFIG_SENSORS_ADS7828=m
    CONFIG_SENSORS_THMC50=m
    CONFIG_SENSORS_VIA686A=m
    CONFIG_SENSORS_VT1211=m
    CONFIG_SENSORS_W83781D=m
    CONFIG_SENSORS_W83792D=m
    CONFIG_SENSORS_W83L785TS=m
    CONFIG_SENSORS_W83L786NG=m
    CONFIG_SENSORS_W83627HF=m
    CONFIG_SENSORS_W83627EHF=m
    CONFIG_SENSORS_HDAPS=m
    CONFIG_THERMAL=y
    CONFIG_THERMAL_HWMON=y
    CONFIG_WATCHDOG=y
    CONFIG_SOFT_WATCHDOG=m
    CONFIG_ACQUIRE_WDT=m
    CONFIG_ADVANTECH_WDT=m
    CONFIG_ALIM1535_WDT=m
    CONFIG_ALIM7101_WDT=m
    CONFIG_SC520_WDT=m
    CONFIG_EUROTECH_WDT=m
    CONFIG_IB700_WDT=m
    CONFIG_IBMASR=m
    CONFIG_WAFER_WDT=m
    CONFIG_I6300ESB_WDT=m
    CONFIG_ITCO_WDT=m
    CONFIG_ITCO_VENDOR_SUPPORT=y
    CONFIG_IT8712F_WDT=m
    CONFIG_HP_WATCHDOG=m
    CONFIG_SC1200_WDT=m
    CONFIG_60XX_WDT=m
    CONFIG_SBC7240_WDT=m
    CONFIG_CPU5_WDT=m
    CONFIG_W83627HF_WDT=m
    CONFIG_W83877F_WDT=m
    CONFIG_MACHZ_WDT=m
    CONFIG_PCWATCHDOG=m
    CONFIG_MIXCOMWD=m
    CONFIG_WDT=m
    CONFIG_WDT_501=y
    CONFIG_PCIPCWATCHDOG=m
    CONFIG_WDTPCI=m
    CONFIG_WDT_501_PCI=y
    CONFIG_USBPCWATCHDOG=m
    CONFIG_SSB_POSSIBLE=y
    CONFIG_SSB=m
    CONFIG_SSB_SPROM=y
    CONFIG_SSB_BLOCKIO=y
    CONFIG_SSB_PCIHOST_POSSIBLE=y
    CONFIG_SSB_PCIHOST=y
    CONFIG_SSB_B43_PCI_BRIDGE=y
    CONFIG_SSB_PCMCIAHOST_POSSIBLE=y
    CONFIG_SSB_PCMCIAHOST=y
    CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
    CONFIG_SSB_DRIVER_PCICORE=y
    CONFIG_MFD_SM501=m
    CONFIG_AGP=y
    CONFIG_AGP_INTEL=y
    CONFIG_DRM=y
    CONFIG_DRM_I810=y
    CONFIG_DRM_I915=y
    CONFIG_VGASTATE=m
    CONFIG_VIDEO_OUTPUT_CONTROL=m
    CONFIG_FB=m
    CONFIG_FIRMWARE_EDID=y
    CONFIG_FB_DDC=m
    CONFIG_FB_CFB_FILLRECT=m
    CONFIG_FB_CFB_COPYAREA=m
    CONFIG_FB_CFB_IMAGEBLIT=m
    CONFIG_FB_SVGALIB=m
    CONFIG_FB_MODE_HELPERS=y
    CONFIG_FB_TILEBLITTING=y
    CONFIG_FB_VGA16=m
    CONFIG_FB_UVESA=m
    CONFIG_FB_I810=m
    CONFIG_FB_I810_GTF=y
    CONFIG_FB_I810_I2C=y
    CONFIG_FB_LE80578=m
    CONFIG_FB_CARILLO_RANCH=m
    CONFIG_FB_INTEL=m
    CONFIG_FB_INTEL_I2C=y
    CONFIG_FB_VT8623=m
    CONFIG_FB_ARK=m
    CONFIG_FB_PM3=m
    CONFIG_BACKLIGHT_LCD_SUPPORT=y
    CONFIG_LCD_CLASS_DEVICE=m
    CONFIG_LCD_PLATFORM=m
    CONFIG_BACKLIGHT_CLASS_DEVICE=y
    CONFIG_BACKLIGHT_PROGEAR=m
    CONFIG_BACKLIGHT_CARILLO_RANCH=m
    CONFIG_VGA_CONSOLE=y
    CONFIG_VGACON_SOFT_SCROLLBACK=y
    CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
    CONFIG_VIDEO_SELECT=y
    CONFIG_MDA_CONSOLE=m
    CONFIG_DUMMY_CONSOLE=y
    CONFIG_FRAMEBUFFER_CONSOLE=m
    CONFIG_FONT_8x8=y
    CONFIG_FONT_8x16=y
    CONFIG_LOGO=y
    CONFIG_LOGO_LINUX_MONO=y
    CONFIG_LOGO_LINUX_VGA16=y
    CONFIG_LOGO_LINUX_CLUT224=y
    CONFIG_SOUND=y
    CONFIG_SND=y
    CONFIG_SND_TIMER=y
    CONFIG_SND_PCM=y
    CONFIG_SND_HWDEP=m
    CONFIG_SND_RAWMIDI=m
    CONFIG_SND_SEQUENCER=m
    CONFIG_SND_SEQ_DUMMY=m
    CONFIG_SND_OSSEMUL=y
    CONFIG_SND_MIXER_OSS=y
    CONFIG_SND_PCM_OSS=y
    CONFIG_SND_PCM_OSS_PLUGINS=y
    CONFIG_SND_SEQUENCER_OSS=y
    CONFIG_SND_SUPPORT_OLD_API=y
    CONFIG_SND_VERBOSE_PROCFS=y
    CONFIG_SND_VMASTER=y
    CONFIG_SND_MPU401_UART=m
    CONFIG_SND_OPL3_LIB=m
    CONFIG_SND_VX_LIB=m
    CONFIG_SND_AC97_CODEC=y
    CONFIG_SND_DRIVERS=y
    CONFIG_SND_DUMMY=m
    CONFIG_SND_VIRMIDI=m
    CONFIG_SND_MTPAV=m
    CONFIG_SND_SERIAL_U16550=m
    CONFIG_SND_MPU401=m
    CONFIG_SND_AC97_POWER_SAVE=y
    CONFIG_SND_AC97_POWER_SAVE_DEFAULT=0
    CONFIG_SND_SB_COMMON=m
    CONFIG_SND_SB16_DSP=m
    CONFIG_SND_PCI=y
    CONFIG_SND_ALS4000=m
    CONFIG_SND_ALI5451=m
    CONFIG_SND_ATIIXP=m
    CONFIG_SND_ATIIXP_MODEM=m
    CONFIG_SND_AU8810=m
    CONFIG_SND_AU8820=m
    CONFIG_SND_AU8830=m
    CONFIG_SND_AZT3328=m
    CONFIG_SND_BT87X=m
    CONFIG_SND_CA0106=m
    CONFIG_SND_CMIPCI=m
    CONFIG_SND_OXYGEN_LIB=m
    CONFIG_SND_OXYGEN=m
    CONFIG_SND_CS4281=m
    CONFIG_SND_CS46XX=m
    CONFIG_SND_CS46XX_NEW_DSP=y
    CONFIG_SND_CS5530=m
    CONFIG_SND_EMU10K1=m
    CONFIG_SND_EMU10K1X=m
    CONFIG_SND_ENS1370=m
    CONFIG_SND_ENS1371=m
    CONFIG_SND_ES1938=m
    CONFIG_SND_ES1968=m
    CONFIG_SND_FM801=m
    CONFIG_SND_HDA_INTEL=m
    CONFIG_SND_HDA_CODEC_REALTEK=y
    CONFIG_SND_HDA_CODEC_ANALOG=y
    CONFIG_SND_HDA_CODEC_SIGMATEL=y
    CONFIG_SND_HDA_CODEC_VIA=y
    CONFIG_SND_HDA_CODEC_ATIHDMI=y
    CONFIG_SND_HDA_CODEC_CONEXANT=y
    CONFIG_SND_HDA_CODEC_CMEDIA=y
    CONFIG_SND_HDA_CODEC_SI3054=y
    CONFIG_SND_HDA_GENERIC=y
    CONFIG_SND_HDSP=m
    CONFIG_SND_HIFIER=m
    CONFIG_SND_ICE1712=m
    CONFIG_SND_ICE1724=m
    CONFIG_SND_INTEL8X0=y
    CONFIG_SND_INTEL8X0M=y
    CONFIG_SND_KORG1212=m
    CONFIG_SND_MAESTRO3=m
    CONFIG_SND_MIXART=m
    CONFIG_SND_NM256=m
    CONFIG_SND_RME32=m
    CONFIG_SND_RME96=m
    CONFIG_SND_RME9652=m
    CONFIG_SND_SIS7019=m
    CONFIG_SND_SONICVIBES=m
    CONFIG_SND_TRIDENT=m
    CONFIG_SND_VIA82XX=m
    CONFIG_SND_VIA82XX_MODEM=m
    CONFIG_SND_VIRTUOSO=m
    CONFIG_SND_VX222=m
    CONFIG_SND_YMFPCI=m
    CONFIG_SND_USB=y
    CONFIG_SND_USB_AUDIO=m
    CONFIG_SND_USB_USX2Y=m
    CONFIG_SND_USB_CAIAQ=m
    CONFIG_SND_SOC=m
    CONFIG_AC97_BUS=y
    CONFIG_HID_SUPPORT=y
    CONFIG_HID=y
    CONFIG_HIDRAW=y
    CONFIG_USB_HID=m
    CONFIG_USB_HIDDEV=y
    CONFIG_USB_KBD=m
    CONFIG_USB_MOUSE=m
    CONFIG_USB_SUPPORT=y
    CONFIG_USB_ARCH_HAS_HCD=y
    CONFIG_USB_ARCH_HAS_OHCI=y
    CONFIG_USB_ARCH_HAS_EHCI=y
    CONFIG_USB=m
    CONFIG_USB_ANNOUNCE_NEW_DEVICES=y
    CONFIG_USB_DEVICEFS=y
    CONFIG_USB_DEVICE_CLASS=y
    CONFIG_USB_SUSPEND=y
    CONFIG_USB_MON=y
    CONFIG_USB_EHCI_HCD=m
    CONFIG_USB_EHCI_ROOT_HUB_TT=y
    CONFIG_USB_ISP116X_HCD=m
    CONFIG_USB_ISP1760_HCD=m
    CONFIG_USB_OHCI_HCD=m
    CONFIG_USB_OHCI_HCD_SSB=y
    CONFIG_USB_OHCI_LITTLE_ENDIAN=y
    CONFIG_USB_UHCI_HCD=m
    CONFIG_USB_SL811_HCD=m
    CONFIG_USB_SL811_CS=m
    CONFIG_USB_R8A66597_HCD=m
    CONFIG_USB_ACM=m
    CONFIG_USB_PRINTER=m
    CONFIG_USB_WDM=m
    CONFIG_USB_STORAGE=m
    CONFIG_USB_STORAGE_DATAFAB=y
    CONFIG_USB_STORAGE_FREECOM=y
    CONFIG_USB_STORAGE_ISD200=y
    CONFIG_USB_STORAGE_DPCM=y
    CONFIG_USB_STORAGE_USBAT=y
    CONFIG_USB_STORAGE_SDDR09=y
    CONFIG_USB_STORAGE_SDDR55=y
    CONFIG_USB_STORAGE_JUMPSHOT=y
    CONFIG_USB_STORAGE_ALAUDA=y
    CONFIG_USB_STORAGE_ONETOUCH=y
    CONFIG_USB_STORAGE_KARMA=y
    CONFIG_USB_STORAGE_SIERRA=y
    CONFIG_USB_MDC800=m
    CONFIG_USB_MICROTEK=m
    CONFIG_USB_USS720=m
    CONFIG_USB_SERIAL=m
    CONFIG_USB_EZUSB=y
    CONFIG_USB_SERIAL_GENERIC=y
    CONFIG_USB_SERIAL_AIRCABLE=m
    CONFIG_USB_SERIAL_ARK3116=m
    CONFIG_USB_SERIAL_BELKIN=m
    CONFIG_USB_SERIAL_CH341=m
    CONFIG_USB_SERIAL_WHITEHEAT=m
    CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
    CONFIG_USB_SERIAL_CP2101=m
    CONFIG_USB_SERIAL_CYPRESS_M8=m
    CONFIG_USB_SERIAL_EMPEG=m
    CONFIG_USB_SERIAL_FTDI_SIO=m
    CONFIG_USB_SERIAL_FUNSOFT=m
    CONFIG_USB_SERIAL_VISOR=m
    CONFIG_USB_SERIAL_IPAQ=m
    CONFIG_USB_SERIAL_IR=m
    CONFIG_USB_SERIAL_EDGEPORT=m
    CONFIG_USB_SERIAL_EDGEPORT_TI=m
    CONFIG_USB_SERIAL_GARMIN=m
    CONFIG_USB_SERIAL_IPW=m
    CONFIG_USB_SERIAL_IUU=m
    CONFIG_USB_SERIAL_KEYSPAN_PDA=m
    CONFIG_USB_SERIAL_KEYSPAN=m
    CONFIG_USB_SERIAL_KEYSPAN_MPR=y
    CONFIG_USB_SERIAL_KEYSPAN_USA28=y
    CONFIG_USB_SERIAL_KEYSPAN_USA28X=y
    CONFIG_USB_SERIAL_KEYSPAN_USA28XA=y
    CONFIG_USB_SERIAL_KEYSPAN_USA28XB=y
    CONFIG_USB_SERIAL_KEYSPAN_USA19=y
    CONFIG_USB_SERIAL_KEYSPAN_USA18X=y
    CONFIG_USB_SERIAL_KEYSPAN_USA19W=y
    CONFIG_USB_SERIAL_KEYSPAN_USA19QW=y
    CONFIG_USB_SERIAL_KEYSPAN_USA19QI=y
    CONFIG_USB_SERIAL_KEYSPAN_USA49W=y
    CONFIG_USB_SERIAL_KEYSPAN_USA49WLC=y
    CONFIG_USB_SERIAL_KLSI=m
    CONFIG_USB_SERIAL_KOBIL_SCT=m
    CONFIG_USB_SERIAL_MCT_U232=m
    CONFIG_USB_SERIAL_MOS7720=m
    CONFIG_USB_SERIAL_MOS7840=m
    CONFIG_USB_SERIAL_MOTOROLA=m
    CONFIG_USB_SERIAL_NAVMAN=m
    CONFIG_USB_SERIAL_PL2303=m
    CONFIG_USB_SERIAL_OTI6858=m
    CONFIG_USB_SERIAL_SPCP8X5=m
    CONFIG_USB_SERIAL_HP4X=m
    CONFIG_USB_SERIAL_SAFE=m
    CONFIG_USB_SERIAL_SAFE_PADDED=y
    CONFIG_USB_SERIAL_SIERRAWIRELESS=m
    CONFIG_USB_SERIAL_TI=m
    CONFIG_USB_SERIAL_CYBERJACK=m
    CONFIG_USB_SERIAL_XIRCOM=m
    CONFIG_USB_SERIAL_OPTION=m
    CONFIG_USB_SERIAL_OMNINET=m
    CONFIG_USB_RIO500=m
    CONFIG_USB_LEGOTOWER=m
    CONFIG_USB_LCD=m
    CONFIG_USB_BERRY_CHARGE=m
    CONFIG_USB_LED=m
    CONFIG_USB_CYTHERM=m
    CONFIG_USB_IDMOUSE=m
    CONFIG_USB_SISUSBVGA=m
    CONFIG_USB_IOWARRIOR=m
    CONFIG_USB_TEST=m
    CONFIG_USB_GADGET=m
    CONFIG_USB_GADGET_DEBUG_FS=y
    CONFIG_USB_GADGET_SELECTED=y
    CONFIG_USB_GADGET_NET2280=y
    CONFIG_USB_NET2280=m
    CONFIG_USB_GADGET_DUALSPEED=y
    CONFIG_USB_ZERO=m
    CONFIG_USB_ETH=m
    CONFIG_USB_ETH_RNDIS=y
    CONFIG_USB_GADGETFS=m
    CONFIG_USB_FILE_STORAGE=m
    CONFIG_USB_G_SERIAL=m
    CONFIG_USB_G_PRINTER=m
    CONFIG_USB_CDC_COMPOSITE=m
    CONFIG_MMC=m
    CONFIG_MMC_BLOCK=m
    CONFIG_MMC_BLOCK_BOUNCE=y
    CONFIG_MMC_SDHCI=m
    CONFIG_MMC_SDHCI_PCI=m
    CONFIG_MMC_WBSD=m
    CONFIG_MMC_TIFM_SD=m
    CONFIG_MMC_SDRICOH_CS=m
    CONFIG_MEMSTICK=m
    CONFIG_MEMSTICK_TIFM_MS=m
    CONFIG_MEMSTICK_JMICRON_38X=m
    CONFIG_NEW_LEDS=y
    CONFIG_LEDS_CLASS=m
    CONFIG_LEDS_PCA9532=m
    CONFIG_LEDS_GPIO=m
    CONFIG_LEDS_PCA955X=m
    CONFIG_LEDS_TRIGGERS=y
    CONFIG_LEDS_TRIGGER_TIMER=m
    CONFIG_LEDS_TRIGGER_IDE_DISK=y
    CONFIG_LEDS_TRIGGER_HEARTBEAT=m
    CONFIG_LEDS_TRIGGER_DEFAULT_ON=m
    CONFIG_RTC_LIB=y
    CONFIG_RTC_CLASS=y
    CONFIG_RTC_HCTOSYS=y
    CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
    CONFIG_RTC_INTF_SYSFS=y
    CONFIG_RTC_INTF_PROC=y
    CONFIG_RTC_INTF_DEV=y
    CONFIG_RTC_INTF_DEV_UIE_EMUL=y
    CONFIG_RTC_DRV_TEST=m
    CONFIG_RTC_DRV_DS1307=m
    CONFIG_RTC_DRV_DS1374=m
    CONFIG_RTC_DRV_DS1672=m
    CONFIG_RTC_DRV_MAX6900=m
    CONFIG_RTC_DRV_RS5C372=m
    CONFIG_RTC_DRV_ISL1208=m
    CONFIG_RTC_DRV_X1205=m
    CONFIG_RTC_DRV_PCF8563=m
    CONFIG_RTC_DRV_PCF8583=m
    CONFIG_RTC_DRV_M41T80=m
    CONFIG_RTC_DRV_CMOS=m
    CONFIG_RTC_DRV_DS1511=m
    CONFIG_RTC_DRV_DS1553=m
    CONFIG_RTC_DRV_DS1742=m
    CONFIG_RTC_DRV_STK17TA8=m
    CONFIG_RTC_DRV_M48T86=m
    CONFIG_RTC_DRV_M48T59=m
    CONFIG_RTC_DRV_V3020=m
    CONFIG_DMADEVICES=y
    CONFIG_INTEL_IOATDMA=m
    CONFIG_DMA_ENGINE=y
    CONFIG_NET_DMA=y
    CONFIG_DCA=m
    CONFIG_UIO=m
    CONFIG_UIO_CIF=m
    CONFIG_UIO_PDRV=m
    CONFIG_UIO_SMX=m
    CONFIG_EDD=m
    CONFIG_FIRMWARE_MEMMAP=y
    CONFIG_DMIID=y
    CONFIG_EXT2_FS=m
    CONFIG_EXT3_FS=y
    CONFIG_EXT3_FS_XATTR=y
    CONFIG_EXT3_FS_POSIX_ACL=y
    CONFIG_EXT3_FS_SECURITY=y
    CONFIG_JBD=y
    CONFIG_FS_MBCACHE=y
    CONFIG_FS_POSIX_ACL=y
    CONFIG_XFS_FS=y
    CONFIG_XFS_QUOTA=y
    CONFIG_XFS_POSIX_ACL=y
    CONFIG_XFS_RT=y
    CONFIG_DNOTIFY=y
    CONFIG_INOTIFY=y
    CONFIG_INOTIFY_USER=y
    CONFIG_QUOTA=y
    CONFIG_QUOTA_NETLINK_INTERFACE=y
    CONFIG_PRINT_QUOTA_WARNING=y
    CONFIG_QFMT_V1=m
    CONFIG_QFMT_V2=m
    CONFIG_QUOTACTL=y
    CONFIG_AUTOFS_FS=m
    CONFIG_AUTOFS4_FS=m
    CONFIG_FUSE_FS=m
    CONFIG_ISO9660_FS=m
    CONFIG_JOLIET=y
    CONFIG_ZISOFS=y
    CONFIG_UDF_FS=m
    CONFIG_UDF_NLS=y
    CONFIG_FAT_FS=m
    CONFIG_MSDOS_FS=m
    CONFIG_VFAT_FS=m
    CONFIG_FAT_DEFAULT_CODEPAGE=437
    CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
    CONFIG_NTFS_FS=m
    CONFIG_NTFS_RW=y
    CONFIG_PROC_FS=y
    CONFIG_PROC_KCORE=y
    CONFIG_PROC_SYSCTL=y
    CONFIG_SYSFS=y
    CONFIG_TMPFS=y
    CONFIG_CONFIGFS_FS=m
    CONFIG_ECRYPT_FS=m
    CONFIG_CRAMFS=y
    CONFIG_ROMFS_FS=m
    CONFIG_NETWORK_FILESYSTEMS=y
    CONFIG_NFS_FS=m
    CONFIG_NFS_V3=y
    CONFIG_NFS_V3_ACL=y
    CONFIG_NFS_V4=y
    CONFIG_NFSD=m
    CONFIG_NFSD_V2_ACL=y
    CONFIG_NFSD_V3=y
    CONFIG_NFSD_V3_ACL=y
    CONFIG_NFSD_V4=y
    CONFIG_LOCKD=m
    CONFIG_LOCKD_V4=y
    CONFIG_EXPORTFS=m
    CONFIG_NFS_ACL_SUPPORT=m
    CONFIG_NFS_COMMON=y
    CONFIG_SUNRPC=m
    CONFIG_SUNRPC_GSS=m
    CONFIG_RPCSEC_GSS_KRB5=m
    CONFIG_RPCSEC_GSS_SPKM3=m
    CONFIG_SMB_FS=m
    CONFIG_CIFS=m
    CONFIG_CIFS_STATS=y
    CONFIG_CIFS_STATS2=y
    CONFIG_CIFS_XATTR=y
    CONFIG_CIFS_POSIX=y
    CONFIG_CODA_FS=m
    CONFIG_9P_FS=m
    CONFIG_MSDOS_PARTITION=y
    CONFIG_NLS=y
    CONFIG_NLS_DEFAULT="cp437"
    CONFIG_NLS_CODEPAGE_437=m
    CONFIG_NLS_CODEPAGE_737=m
    CONFIG_NLS_CODEPAGE_775=m
    CONFIG_NLS_CODEPAGE_850=m
    CONFIG_NLS_CODEPAGE_852=m
    CONFIG_NLS_CODEPAGE_855=m
    CONFIG_NLS_CODEPAGE_857=m
    CONFIG_NLS_CODEPAGE_860=m
    CONFIG_NLS_CODEPAGE_861=m
    CONFIG_NLS_CODEPAGE_862=m
    CONFIG_NLS_CODEPAGE_863=m
    CONFIG_NLS_CODEPAGE_864=m
    CONFIG_NLS_CODEPAGE_865=m
    CONFIG_NLS_CODEPAGE_866=m
    CONFIG_NLS_CODEPAGE_869=m
    CONFIG_NLS_CODEPAGE_936=m
    CONFIG_NLS_CODEPAGE_950=m
    CONFIG_NLS_CODEPAGE_932=m
    CONFIG_NLS_CODEPAGE_949=m
    CONFIG_NLS_CODEPAGE_874=m
    CONFIG_NLS_ISO8859_8=m
    CONFIG_NLS_CODEPAGE_1250=m
    CONFIG_NLS_CODEPAGE_1251=m
    CONFIG_NLS_ASCII=m
    CONFIG_NLS_ISO8859_1=m
    CONFIG_NLS_ISO8859_2=m
    CONFIG_NLS_ISO8859_3=m
    CONFIG_NLS_ISO8859_4=m
    CONFIG_NLS_ISO8859_5=m
    CONFIG_NLS_ISO8859_6=m
    CONFIG_NLS_ISO8859_7=m
    CONFIG_NLS_ISO8859_9=m
    CONFIG_NLS_ISO8859_13=m
    CONFIG_NLS_ISO8859_14=m
    CONFIG_NLS_ISO8859_15=m
    CONFIG_NLS_KOI8_R=m
    CONFIG_NLS_KOI8_U=m
    CONFIG_NLS_UTF8=m
    CONFIG_DLM=m
    CONFIG_TRACE_IRQFLAGS_SUPPORT=y
    CONFIG_ENABLE_WARN_DEPRECATED=y
    CONFIG_ENABLE_MUST_CHECK=y
    CONFIG_FRAME_WARN=1024
    CONFIG_MAGIC_SYSRQ=y
    CONFIG_UNUSED_SYMBOLS=y
    CONFIG_DEBUG_FS=y
    CONFIG_DEBUG_KERNEL=y
    CONFIG_DETECT_SOFTLOCKUP=y
    CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
    CONFIG_SCHED_DEBUG=y
    CONFIG_SCHEDSTATS=y
    CONFIG_TIMER_STATS=y
    CONFIG_SLUB_STATS=y
    CONFIG_DEBUG_PREEMPT=y
    CONFIG_DEBUG_MUTEXES=y
    CONFIG_STACKTRACE=y
    CONFIG_DEBUG_BUGVERBOSE=y
    CONFIG_DEBUG_INFO=y
    CONFIG_FRAME_POINTER=y
    CONFIG_LATENCYTOP=y
    CONFIG_HAVE_FTRACE=y
    CONFIG_HAVE_DYNAMIC_FTRACE=y
    CONFIG_TRACER_MAX_TRACE=y
    CONFIG_TRACING=y
    CONFIG_FTRACE=y
    CONFIG_SYSPROF_TRACER=y
    CONFIG_SCHED_TRACER=y
    CONFIG_CONTEXT_SWITCH_TRACER=y
    CONFIG_DYNAMIC_FTRACE=y
    CONFIG_HAVE_ARCH_KGDB=y
    CONFIG_KGDB=y
    CONFIG_KGDB_SERIAL_CONSOLE=m
    CONFIG_X86_VERBOSE_BOOTUP=y
    CONFIG_X86_PTDUMP=y
    CONFIG_DOUBLEFAULT=y
    CONFIG_IO_DELAY_TYPE_0X80=0
    CONFIG_IO_DELAY_TYPE_0XED=1
    CONFIG_IO_DELAY_TYPE_UDELAY=2
    CONFIG_IO_DELAY_TYPE_NONE=3
    CONFIG_IO_DELAY_0X80=y
    CONFIG_DEFAULT_IO_DELAY_TYPE=0
    CONFIG_KEYS=y
    CONFIG_CRYPTO=y
    CONFIG_CRYPTO_ALGAPI=y
    CONFIG_CRYPTO_AEAD=y
    CONFIG_CRYPTO_BLKCIPHER=y
    CONFIG_CRYPTO_HASH=y
    CONFIG_CRYPTO_MANAGER=y
    CONFIG_CRYPTO_GF128MUL=m
    CONFIG_CRYPTO_NULL=m
    CONFIG_CRYPTO_CRYPTD=m
    CONFIG_CRYPTO_AUTHENC=y
    CONFIG_CRYPTO_TEST=m
    CONFIG_CRYPTO_CBC=y
    CONFIG_CRYPTO_CTS=m
    CONFIG_CRYPTO_ECB=m
    CONFIG_CRYPTO_LRW=m
    CONFIG_CRYPTO_PCBC=m
    CONFIG_CRYPTO_XTS=m
    CONFIG_CRYPTO_HMAC=y
    CONFIG_CRYPTO_XCBC=m
    CONFIG_CRYPTO_CRC32C=m
    CONFIG_CRYPTO_MD4=m
    CONFIG_CRYPTO_MD5=y
    CONFIG_CRYPTO_MICHAEL_MIC=m
    CONFIG_CRYPTO_RMD128=m
    CONFIG_CRYPTO_RMD160=m
    CONFIG_CRYPTO_RMD256=m
    CONFIG_CRYPTO_RMD320=m
    CONFIG_CRYPTO_SHA1=y
    CONFIG_CRYPTO_SHA256=m
    CONFIG_CRYPTO_SHA512=m
    CONFIG_CRYPTO_TGR192=m
    CONFIG_CRYPTO_WP512=m
    CONFIG_CRYPTO_AES=m
    CONFIG_CRYPTO_AES_586=m
    CONFIG_CRYPTO_ANUBIS=m
    CONFIG_CRYPTO_ARC4=m
    CONFIG_CRYPTO_BLOWFISH=m
    CONFIG_CRYPTO_CAMELLIA=m
    CONFIG_CRYPTO_CAST5=m
    CONFIG_CRYPTO_CAST6=m
    CONFIG_CRYPTO_DES=y
    CONFIG_CRYPTO_FCRYPT=m
    CONFIG_CRYPTO_KHAZAD=m
    CONFIG_CRYPTO_SEED=m
    CONFIG_CRYPTO_SERPENT=m
    CONFIG_CRYPTO_TEA=m
    CONFIG_CRYPTO_TWOFISH=m
    CONFIG_CRYPTO_TWOFISH_COMMON=m
    CONFIG_CRYPTO_TWOFISH_586=m
    CONFIG_CRYPTO_DEFLATE=y
    CONFIG_CRYPTO_HW=y
    CONFIG_CRYPTO_DEV_PADLOCK=y
    CONFIG_CRYPTO_DEV_PADLOCK_AES=m
    CONFIG_CRYPTO_DEV_PADLOCK_SHA=m
    CONFIG_CRYPTO_DEV_GEODE=m
    CONFIG_HAVE_KVM=y
    CONFIG_VIRTUALIZATION=y
    CONFIG_LGUEST=m
    CONFIG_VIRTIO=y
    CONFIG_VIRTIO_RING=y
    CONFIG_VIRTIO_PCI=m
    CONFIG_VIRTIO_BALLOON=m
    CONFIG_BITREVERSE=y
    CONFIG_GENERIC_FIND_FIRST_BIT=y
    CONFIG_GENERIC_FIND_NEXT_BIT=y
    CONFIG_CRC_CCITT=y
    CONFIG_CRC16=m
    CONFIG_CRC_T10DIF=y
    CONFIG_CRC_ITU_T=m
    CONFIG_CRC32=y
    CONFIG_CRC7=m
    CONFIG_LIBCRC32C=m
    CONFIG_AUDIT_GENERIC=y
    CONFIG_ZLIB_INFLATE=y
    CONFIG_ZLIB_DEFLATE=y
    CONFIG_TEXTSEARCH=y
    CONFIG_TEXTSEARCH_KMP=m
    CONFIG_TEXTSEARCH_BM=m
    CONFIG_TEXTSEARCH_FSM=m
    CONFIG_PLIST=y
    CONFIG_HAS_IOMEM=y
    CONFIG_HAS_IOPORT=y
    CONFIG_HAS_DMA=y
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  2. [PATCH] make lock_super recursive to simulate BKL

    This fixes a regression introduced when BKL was removed from the
    vfat driver in commit 8f5934278d1d86590244c2791b28f77d67466007.

    With vfat, the unlink syscall would result in the following call chain
    that caused deadlock.

    - do_unlinkat
    - vfs_unlink
    - vfat_unlink
    * lock_super
    - fat_remove_entries
    - fat_sync_inode
    - fat_write_inode
    * lock_super

    This is not the ideal fix, but it should reverse some regressions
    caused by BKL removal in code that depended on BKL being recursive.

    Signed-off-by: Bart Trojanowski
    ---
    fs/super.c | 20 ++++++++++++++++++--
    include/linux/fs.h | 2 ++
    2 files changed, 20 insertions(+), 2 deletions(-)

    diff --git a/fs/super.c b/fs/super.c
    index e931ae9..8f813db 100644
    --- a/fs/super.c
    +++ b/fs/super.c
    @@ -73,6 +73,8 @@ static struct super_block *alloc_super(struct file_system_type *type)
    INIT_LIST_HEAD(&s->s_dentry_lru);
    init_rwsem(&s->s_umount);
    mutex_init(&s->s_lock);
    + s->s_lock_cookie = NULL;
    + s->s_lock_refcnt = 0;
    lockdep_set_class(&s->s_umount, &type->s_umount_key);
    /*
    * The locking rules for s_lock are up to the
    @@ -227,14 +229,28 @@ static int grab_super(struct super_block *s) __releases(sb_lock)
    */
    void lock_super(struct super_block * sb)
    {
    + if (sb->s_lock_heldby == current) {
    + sb->s_lock_refcnt ++;
    + return;
    + }
    +
    get_fs_excl();
    mutex_lock(&sb->s_lock);
    +
    + sb->s_lock_heldby = current;
    + sb->s_lock_refcnt = 1;
    }

    void unlock_super(struct super_block * sb)
    {
    - put_fs_excl();
    - mutex_unlock(&sb->s_lock);
    + BUG_ON(sb->s_lock_heldby != current);
    +
    + if (! -- sb->s_lock_refcnt) {
    + sb->s_lock_heldby = NULL;
    +
    + put_fs_excl();
    + mutex_unlock(&sb->s_lock);
    + }
    }

    EXPORT_SYMBOL(lock_super);
    diff --git a/include/linux/fs.h b/include/linux/fs.h
    index 580b513..d88178e 100644
    --- a/include/linux/fs.h
    +++ b/include/linux/fs.h
    @@ -1076,6 +1076,8 @@ struct super_block {
    struct dentry *s_root;
    struct rw_semaphore s_umount;
    struct mutex s_lock;
    + void *s_lock_heldby;
    + u32 s_lock_refcnt;
    int s_count;
    int s_syncing;
    int s_need_sync_fs;
    --
    1.5.6.2.221.g3c6e79

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  3. Re: vfat BKL/lock_super regression in v2.6.26-rc3-g8f59342



    On Tue, 19 Aug 2008, Bart Trojanowski wrote:
    >
    > I was able to bisect it to the commit 8f5934278d1d86590244c2791b28f77d67466007
    > which claims to "Replace BKL with superblock lock in fat/msdos/vfat".
    >
    > When I run with lock debugging I get...
    >
    > =============================================
    > [ INFO: possible recursive locking detected ]
    > 2.6.27-rc3-bisect-00448-ga7f5aaf #16
    > ---------------------------------------------
    > mv/4020 is trying to acquire lock:
    > (&type->s_lock_key#9){--..}, at: [] lock_super+0x1e/0x20
    >
    > but task is already holding lock:
    > (&type->s_lock_key#9){--..}, at: [] lock_super+0x1e/0x20


    Thanks for the excellent debug.

    > It looks like the call trace is:
    >
    > - do_unlinkat
    > - vfs_unlink
    > - vfat_unlink
    > * lock_super
    > - fat_remove_entries
    > - fat_sync_inode
    > - fat_write_inode
    > * lock_super


    Very much so.

    And I think you hit this issue because you probably mounted the USB stick
    as a "sync" (or dirsync) mount - which is what some distros do by default,
    even if it is known to cause problems for some flash cards that don't do a
    good job at wear levelling.

    But it's good that you did that, because all _my_ testing (which was
    admittedly very deficient) had been done with a default mount without that
    thing.

    > So this code really really liked BKL because it was recursive.


    Yeah, we had some other cases like that. It's the main source of BKL
    problems by far (if it wasn't for the recursion, BKL removal would
    generally be trivial).

    The other example of this was 9c20616c385ebeaa30257ef5d35e8f346db4ee32,
    where fat_setattr->fat_truncate caused a deadlock.

    > I am testing a naive patch to address this problem and will follow up on
    > it in a bit.


    Thanks.

    Btw, quite often, the right solution may be to remove one of the locks
    entirely. FAT should actually have been largely BKL free, and my
    conversion of BKL to super-lock was "overly eager" exactly because it's
    easier to find deadlocks (and debug things carefully and handle them as
    they pop up) than it is to find races (which are almost impossible to
    debug and pinpoint).

    In particular, I think fat_write_inode() really is safe. It already uses

    spin_lock(&sbi->inode_hash_lock);
    ..
    spin_unlock(&sbi->inode_hash_lock);

    to protect its internal data structures, and all that [un]lock_super()
    protects is really just local variables and code that is already SMP-safe
    (ie "sb_bread()" certainly doesn't need locking.

    So I'm pretty sure the right fix is to just remove [un]lock_super()
    entirely from fat_write_inode().

    Linus
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  4. Re: vfat BKL/lock_super regression in v2.6.26-rc3-g8f59342

    * Linus Torvalds [080819 18:19]:
    > And I think you hit this issue because you probably mounted the USB stick
    > as a "sync" (or dirsync) mount - which is what some distros do by default,
    > even if it is known to cause problems for some flash cards that don't do a
    > good job at wear levelling.


    You're right. I turned on sync a while back so I could just pull the
    stick out w/o the need to do a manual sync... probably not a good idea
    on my part with respect to the wear of the media.

    > But it's good that you did that, because all _my_ testing (which was
    > admittedly very deficient) had been done with a default mount without that
    > thing.


    My ignorance paid off!

    > Btw, quite often, the right solution may be to remove one of the locks
    > entirely. FAT should actually have been largely BKL free, and my
    > conversion of BKL to super-lock was "overly eager" exactly because it's
    > easier to find deadlocks (and debug things carefully and handle them as
    > they pop up) than it is to find races (which are almost impossible to
    > debug and pinpoint).


    I totally agree. After spending a couple of hours (while doing other
    things) bisecting the tree, I found the commit that introduced the
    regression. In contrast, I was able to find the double-lock in 5
    minutes.

    > In particular, I think fat_write_inode() really is safe.


    I wasn't sure there.

    > So I'm pretty sure the right fix is to just remove [un]lock_super()
    > entirely from fat_write_inode().


    Ok, I will test your patch and let you know in a few minutes.

    -Bart

    --
    WebSig: http://www.jukie.net/~bart/sig/
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  5. Re: vfat BKL/lock_super regression in v2.6.26-rc3-g8f59342



    On Tue, 19 Aug 2008, Bart Trojanowski wrote:
    >
    > Ok, I will test your patch and let you know in a few minutes.


    Btw, while you're at it, can you try doing some random writes to that
    thing, since the whole sync mount is quite possibly going to find a few
    other cases like this.

    IOW, _maybe_ you hit the only case that is ever going to be an issue for
    sync mounts, and maybe you didn't. In my eternal quest to never actually
    test anything myself, I'm hoping you can try some writing (and
    over-writing) of files, since writes to the filesystem is where the whole
    "sync" thing is going to show up (both metadata ie file creation and
    removal, and "real" data ie normal write/truncate calls).

    Linus
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  6. Re: vfat BKL/lock_super regression in v2.6.26-rc3-g8f59342

    * Linus Torvalds [080819 18:19]:
    > And I think you hit this issue because you probably mounted the USB stick
    > as a "sync" (or dirsync) mount - which is what some distros do by default,
    > even if it is known to cause problems for some flash cards that don't do a
    > good job at wear levelling.


    Hmm... I was considering the costs and benefits of sync mounts. I don't
    care about the performance hit, as I don't move that much data around.
    Primarily, I like the fact that I can pull the media out w/o any special
    action (which sometimes requires that I unlock my screensaver first).
    But, on the other hand, it's nice to have the media last longer.

    So, maybe it would be a good idea to have a 'delaysync=60' to force a
    sync after 60 seconds of inactivity. Unless, there is something else
    that would do that for me already.

    Thoughts?

    -Bart

    --
    WebSig: http://www.jukie.net/~bart/sig/
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  7. Re: vfat BKL/lock_super regression in v2.6.26-rc3-g8f59342

    * Linus Torvalds [080819 20:12]:
    > Btw, while you're at it, can you try doing some random writes to that
    > thing, since the whole sync mount is quite possibly going to find a few
    > other cases like this.


    My bisect test script moved files back and forth between the device and
    my hard disk. If the patch works, I'll modify the script to also do
    some truncate and overwrite tests, and maybe do a few things in
    parallel.

    > IOW, _maybe_ you hit the only case that is ever going to be an issue for
    > sync mounts, and maybe you didn't. In my eternal quest to never actually
    > test anything myself, I'm hoping you can try some writing (and
    > over-writing) of files, since writes to the filesystem is where the whole
    > "sync" thing is going to show up (both metadata ie file creation and
    > removal, and "real" data ie normal write/truncate calls).


    I am glad to help you in your eternal quest.

    -Bart

    --
    WebSig: http://www.jukie.net/~bart/sig/
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  8. Re: vfat BKL/lock_super regression in v2.6.26-rc3-g8f59342



    On Tue, 19 Aug 2008, Bart Trojanowski wrote:
    >
    > So, maybe it would be a good idea to have a 'delaysync=60' to force a
    > sync after 60 seconds of inactivity. Unless, there is something else
    > that would do that for me already.


    Oh, it could be even shorter.

    The problem with using 'sync' is that it easily ends up overwriting things
    like the sector that contains a particular inode thousands of times for
    even trivial operations. Or things like the file allocation table etc.

    For example, something as trivial as copying a single big file, if the
    copy program just copies it a few kB at a time, then a file that is a few
    megabytes in size will actually end up rewriting the inode block (just
    because the size grows) thousands of time.

    With any kind of half-way decent wear leveling, this isn't a problem at
    all, and most flash drives have that. But if they don't, then that means
    that the file allocation table sectors and the inode sectors get rewritten
    over and over and over again thousands of times.

    Just making it do the sync once per _second_ or something like that would
    already make the "thousands of times" go away. The sectors would probably
    be rewritten a few times per big file, and just once per couple of tens of
    files for small files being written.

    So we don't even need anything like 60 seconds, we literally would just
    need some trivial delays.

    But no, we don't have that kind of "half-sync" behavior. Right now, it's
    pretty much all or nothing. Either we're fully synchronous (and that
    really is bad for crappy flash), or we end up depending on bdflush writing
    things back in the background.

    Of course, pdflush already syncs within 60s (in fact, 30s by default,
    iirc), but then things like "laptop_mode" will actually make that
    potentially much less frequent (I think the default value for that is 5
    minutes).

    I do think this is something we could do better, no question about it.

    But I don't know exactly what the timeout should be, though (although I
    suspect that it should involve _ignoring_ non-data writes like the atime
    updates, and trigger a timeout on data writes so that when you actually
    write a file, you'll know that the sync will happen within five seconds of
    you having finished the write or whatever).

    And no, no such mount option currently exists. And the pdflush things are
    all global, not per-device, iirc.

    Linus
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  9. Re: vfat BKL/lock_super regression in v2.6.26-rc3-g8f59342



    On Tue, 19 Aug 2008, Linus Torvalds wrote:
    >
    > But I don't know exactly what the timeout should be, though (although I
    > suspect that it should involve _ignoring_ non-data writes like the atime
    > updates, and trigger a timeout on data writes so that when you actually
    > write a file, you'll know that the sync will happen within five seconds of
    > you having finished the write or whatever).


    ... and by "finished the write" I mean "closed the file", not "end of
    write() system call". Ie it's one of those things where it really doesn't
    mostly make much sense to try to give any kinds of flush guarantees until
    the user has basically shown that he's all done with writing.

    If you have something like removable media, and you actually remove it
    while you have a "cp -R" in progress, it damn well won't matter whether we
    were synchronous or not. But if you remove it after the "cp" has actually
    finished, it's a lot more understandable if somebody expects it to be on
    disk.

    So one thing we could perhaps consider is to make FAT in particular
    consider "sync" mounts to be about open/close consistency, not about
    per-write-system-call consistency. So the "close()" wouldn't return until
    the file is on disk, but we wouldn't force a synchronous rewrite the inode
    or the file allocation table thousands of times just because the file was
    big.

    FAT really is kind of different. I suspect we could just change what
    "sync" means for it. But it would probably be good to have a VFS-level
    notion of open-close consistency. It is, after all, what NFS is already
    supposed to give you, so there is precedence for that being a useful IO
    serialization model.

    Al, what do you think?

    Linus
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  10. Re: [PATCH] make lock_super recursive to simulate BKL

    * Linus Torvalds [080819 18:23]:
    >
    >
    > On Tue, 19 Aug 2008, Bart Trojanowski wrote:
    > >
    > > This fixes a regression introduced when BKL was removed from the
    > > vfat driver in commit 8f5934278d1d86590244c2791b28f77d67466007.

    >
    > I agree that it's going to almost certainly fix the regression, but could
    > you test the following patch instead as an alternative? I'd rather remove
    > the broken recursive lockign than introduce it as an acceptable concept.


    Looks good. I ran 10 parallel processes creating 1M files truncating
    them, writing to them again and then deleting them. This patch fixes
    the issue I ran into.

    Signed-off-by: Bart Trojanowski

    > ---
    > fs/fat/inode.c | 10 +++-------
    > 1 files changed, 3 insertions(+), 7 deletions(-)
    >
    > diff --git a/fs/fat/inode.c b/fs/fat/inode.c
    > index 6d266d7..80ff338 100644
    > --- a/fs/fat/inode.c
    > +++ b/fs/fat/inode.c
    > @@ -562,26 +562,23 @@ static int fat_write_inode(struct inode *inode, int wait)
    > struct buffer_head *bh;
    > struct msdos_dir_entry *raw_entry;
    > loff_t i_pos;
    > - int err = 0;
    > + int err;
    >
    > retry:
    > i_pos = MSDOS_I(inode)->i_pos;
    > if (inode->i_ino == MSDOS_ROOT_INO || !i_pos)
    > return 0;
    >
    > - lock_super(sb);
    > bh = sb_bread(sb, i_pos >> sbi->dir_per_block_bits);
    > if (!bh) {
    > printk(KERN_ERR "FAT: unable to read inode block "
    > "for updating (i_pos %lld)\n", i_pos);
    > - err = -EIO;
    > - goto out;
    > + return -EIO;
    > }
    > spin_lock(&sbi->inode_hash_lock);
    > if (i_pos != MSDOS_I(inode)->i_pos) {
    > spin_unlock(&sbi->inode_hash_lock);
    > brelse(bh);
    > - unlock_super(sb);
    > goto retry;
    > }
    >
    > @@ -607,11 +604,10 @@ retry:
    > }
    > spin_unlock(&sbi->inode_hash_lock);
    > mark_buffer_dirty(bh);
    > + err = 0;
    > if (wait)
    > err = sync_dirty_buffer(bh);
    > brelse(bh);
    > -out:
    > - unlock_super(sb);
    > return err;
    > }
    >
    > --
    > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    > the body of a message to majordomo@vger.kernel.org
    > More majordomo info at http://vger.kernel.org/majordomo-info.html
    > Please read the FAQ at http://www.tux.org/lkml/


    --
    WebSig: http://www.jukie.net/~bart/sig/
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  11. Re: vfat BKL/lock_super regression in v2.6.26-rc3-g8f59342

    * Linus Torvalds [080819 20:56]:
    > So one thing we could perhaps consider is to make FAT in particular
    > consider "sync" mounts to be about open/close consistency, not about
    > per-write-system-call consistency. So the "close()" wouldn't return until
    > the file is on disk, but we wouldn't force a synchronous rewrite the inode
    > or the file allocation table thousands of times just because the file was
    > big.


    I was reading the vfat code, and it turns out that vfat has a "flush"
    mount option. Which is documented in the code (not in the manpage) as:

    struct fat_mount_options {
    ...
    unsigned
    ...
    flush:1, /* write things quickly */

    Since that was very informative I looked at the usage. It's used
    in fat_file_release() to do almost what you describe. But it seems to
    be a best effort thing. If my data doesn't hit the disk (or flash) in
    HZ/10, then all bets are off.

    -Bart

    --
    WebSig: http://www.jukie.net/~bart/sig/
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  12. Re: vfat BKL/lock_super regression in v2.6.26-rc3-g8f59342

    El Tue, 19 Aug 2008 22:27:12 -0400, Bart Trojanowski escribió:

    > I was reading the vfat code, and it turns out that vfat has a "flush"
    > mount option. Which is documented in the code (not in the manpage) as:


    Yeah, probably the one place where it was documented was
    http://kernelnewbies.org/Linux_2_6_19 (commit ae78bf9c4f5fde3c67e2829505f195d7347ce3e4).

    Apparently at least one person must have read it, because there's a lot
    of dmesgs in the interweb with this string in it:

    "FAT: Unrecognized mount option "flush" or missing value"

    due to HAL trying to mount vfat filesystems with that option in kernels
    that don't support it. Which broke USB stick support for them, BTW, and
    generated a lot of posts in forums asking for help.

    IOW: not only this option exist, it's also being used by default in all
    the vfat mounts on the linux systems that use HAL, ie. almost all of them.

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

+ Reply to Thread