[PATCH 00/10] NR_CPUS: third reduction of NR_CPUS memory usage x86-version v2 - Kernel

This is a discussion on [PATCH 00/10] NR_CPUS: third reduction of NR_CPUS memory usage x86-version v2 - Kernel ; Wii, isn't this fun...! This is a resubmission of yesterday's patches based on the x86.git/latest tree. Yes, it _is_ a maze of twisty litle passages. ;-) Patches that are actually different are marked "v2". (I did not recalculate the memory ...

+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 20 of 28

Thread: [PATCH 00/10] NR_CPUS: third reduction of NR_CPUS memory usage x86-version v2

  1. [PATCH 00/10] NR_CPUS: third reduction of NR_CPUS memory usage x86-version v2


    Wii, isn't this fun...! This is a resubmission of yesterday's patches
    based on the x86.git/latest tree. Yes, it _is_ a maze of twisty litle
    passages. ;-)

    Patches that are actually different are marked "v2". (I did not recalculate
    the memory usages changes but I did reverify the defconfig and akpm2
    configs with both 255 and 4096 NR_CPUS.)
    ....................

    Here's the third round of removing static allocations of arrays using
    NR_CPUS to size the length. The change is to use PER_CPU variables in
    place of the static tables, or allocate the array based on nr_cpu_ids.

    In addition, there's a cleanup of x86 non-smp code, the movement of
    setting nr_cpu_ids to setup_per_cpu_areas() so it's available as soon
    as possible, and a new function cpumask_scnprintf_len() to return the
    number of characters needed to display "len" cpumask bits.

    Affected files:

    arch/ia64/kernel/acpi.c
    arch/ia64/kernel/setup.c
    arch/powerpc/kernel/setup_64.c
    arch/sparc64/mm/init.c
    arch/x86/kernel/cpu/intel_cacheinfo.c
    arch/x86/kernel/genapic_64.c
    arch/x86/kernel/mpparse_64.c
    arch/x86/kernel/setup64.c
    arch/x86/kernel/smpboot_32.c
    arch/x86/mm/numa_64.c
    arch/x86/oprofile/nmi_int.c
    drivers/acpi/processor_core.c
    drivers/acpi/processor_idle.c
    drivers/acpi/processor_perflib.c
    drivers/acpi/processor_throttling.c
    drivers/base/cpu.c
    drivers/cpufreq/cpufreq.c
    drivers/cpufreq/cpufreq_stats.c
    drivers/cpufreq/freq_table.c
    include/acpi/processor.h
    include/asm-x86/smp_32.h
    include/asm-x86/smp_64.h
    include/asm-x86/topology.h
    include/linux/bitmap.h
    include/linux/cpumask.h
    init/main.c
    kernel/sched.c
    lib/bitmap.c
    net/core/dev.c

    Based on:
    git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
    git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git


    Cc: Alexey Kuznetsov
    Cc: Andi Kleen
    Cc: Anton Blanchard
    Cc: Christoph Lameter
    Cc: Dave Jones
    Cc: David S. Miller
    Cc: H. Peter Anvin
    Cc: Ingo Molnar
    Cc: Ingo Molnar
    Cc: James Morris
    Cc: Len Brown
    Cc: Patrick McHardy
    Cc: Paul Jackson
    Cc: Paul Mackerras
    Cc: Philippe Elie
    Cc: Thomas Gleixner
    Cc: Tony Luck
    Cc: William L. Irwin

    Signed-off-by: Mike Travis
    ---

    I moved the x86_64 cleanup and move-set-nr_cpu_ids from the zero-based
    percpu variables patchset to this one, as I was encountering a panic
    from system_call_after_swapgs() after an unknown device interrupt during
    module loading. That problem will be dealt with in another patch.


    Here's the various effects of the patches on memory usages using the
    akpm2 config file with NR_CPUS=4096 and MAXNODES=512:

    ====== Data (-l 500)
    1 - initial
    2 - cleanup
    4 - nr_cpus-in-cpufreq-cpu_alloc
    5 - nr_cpus-in-acpi-driver-cpu_alloc
    7 - nr_cpus-in-intel_cacheinfo
    8 - nr_cpus-in-cpu_c
    11 - nr_cpus-in-kernel_sched

    .1. .2. .4. .5. .7. .8. .11.
    32768 . -32768 . . . . show_table(.bss)
    32768 . . . . . -32768 sched_group_nodes_bycpu(.bss)
    32768 . . -32768 . . . processors(.bss)
    32768 . . -32768 . . . processor_device_array(.bss)
    32768 . . . . . -32768 init_sched_entity_p(.bss)
    32768 . . . . . -32768 init_cfs_rq_p(.bss)
    32768 . . .-32768 . . index_kobject(.bss)
    32768 . . .-32768 . . cpuid4_info(.bss)
    32768 . -32768 . . . . cpufreq_cpu_governor(.bss)
    32768 . -32768 . . . . cpufreq_cpu_data(.bss)
    32768 . . . . -32768 . cpu_sys_devices(.bss)
    32768 . . .-32768 . . cache_kobject(.bss)

    ====== Text/Data ()
    1 - initial
    4 - nr_cpus-in-cpufreq-cpu_alloc
    5 - nr_cpus-in-acpi-driver-cpu_alloc
    7 - nr_cpus-in-intel_cacheinfo
    8 - nr_cpus-in-cpu_c
    11 - nr_cpus-in-kernel_sched

    .1. .4. .5. .7. .8. .11. ..final..
    3373056 . +2048 . . . 3375104 <1% TextSize
    1656832 . +2048 . . . 1658880 <1% DataSize
    1855488 -98304 -65536 -98304 -32768 -98304 1462272 -21% BssSize
    10395648 . +4096 . . . 10399744 <1% OtherSize
    17281024 -98304 -57344 -98304 -32768 -98304 16896000 -2% Totals

    ====== Stack (-l 500)
    .... files 11 vars 928 all 0 lim 500 unch 0

    1 - initial
    7 - nr_cpus-in-intel_cacheinfo
    11 - nr_cpus-in-kernel_sched

    .1. .7. .11. ..final..
    4648 . -4080 568 -87% cpu_attach_domain
    4104 -4104 . . -100% show_shared_cpu_map
    8752 -4104 -4080 568 -93% Totals

    --
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  2. [PATCH 05/10] cpumask: Add cpumask_scnprintf_len function

    Add a new function cpumask_scnprintf_len() to return the number of
    characters needed to display "len" cpumask bits. The current method
    of allocating NR_CPUS bytes is incorrect as what's really needed is
    9 characters per 32-bit word of cpumask bits (8 hex digits plus the
    seperator [','] or the terminating NULL.) This function provides the
    caller the means to allocate the correct string length.

    Based on:
    git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
    git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git

    Cc: Paul Jackson

    Signed-off-by: Mike Travis
    ---
    include/linux/bitmap.h | 1 +
    include/linux/cpumask.h | 7 +++++++
    lib/bitmap.c | 16 ++++++++++++++++
    3 files changed, 24 insertions(+)

    --- linux.trees.git.orig/include/linux/bitmap.h
    +++ linux.trees.git/include/linux/bitmap.h
    @@ -108,6 +108,7 @@ extern int __bitmap_weight(const unsigne

    extern int bitmap_scnprintf(char *buf, unsigned int len,
    const unsigned long *src, int nbits);
    +extern int bitmap_scnprintf_len(unsigned int len);
    extern int __bitmap_parse(const char *buf, unsigned int buflen, int is_user,
    unsigned long *dst, int nbits);
    extern int bitmap_parse_user(const char __user *ubuf, unsigned int ulen,
    --- linux.trees.git.orig/include/linux/cpumask.h
    +++ linux.trees.git/include/linux/cpumask.h
    @@ -273,6 +273,13 @@ static inline int __cpumask_scnprintf(ch
    return bitmap_scnprintf(buf, len, srcp->bits, nbits);
    }

    +#define cpumask_scnprintf_len(len) \
    + __cpumask_scnprintf_len((len))
    +static inline int __cpumask_scnprintf_len(int len)
    +{
    + return bitmap_scnprintf_len(len);
    +}
    +
    #define cpumask_parse_user(ubuf, ulen, dst) \
    __cpumask_parse_user((ubuf), (ulen), &(dst), NR_CPUS)
    static inline int __cpumask_parse_user(const char __user *buf, int len,
    --- linux.trees.git.orig/lib/bitmap.c
    +++ linux.trees.git/lib/bitmap.c
    @@ -316,6 +316,22 @@ int bitmap_scnprintf(char *buf, unsigned
    EXPORT_SYMBOL(bitmap_scnprintf);

    /**
    + * bitmap_scnprintf_len - return buffer length needed to convert
    + * bitmap to an ASCII hex string.
    + * @len: number of bits to be converted
    + */
    +int bitmap_scnprintf_len(unsigned int len)
    +{
    + /* we need 9 chars per word for 32 bit words (8 hexdigits + sep/null) */
    + int bitslen = ALIGN(len, CHUNKSZ);
    + int wordlen = CHUNKSZ / 4;
    + int buflen = (bitslen / wordlen) * (wordlen + 1) * sizeof(char);
    +
    + return buflen;
    +}
    +EXPORT_SYMBOL(bitmap_scnprintf_len);
    +
    +/**
    * __bitmap_parse - convert an ASCII hex string into a bitmap.
    * @buf: pointer to buffer containing string.
    * @buflen: buffer size in bytes. If string is smaller than this

    --
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  3. [PATCH 08/10] net: remove NR_CPUS arrays in net/core/dev.c v2

    Remove the fixed size channels[NR_CPUS] array in
    net/core/dev.c and dynamically allocate array based on
    nr_cpu_ids.

    Based on:
    git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
    git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git

    Cc: David S. Miller
    Cc: Alexey Kuznetsov
    Cc: James Morris
    Cc: Patrick McHardy
    Signed-off-by: Mike Travis
    ---
    v2: fixed logic error in netdev_dma_register().
    ---
    net/core/dev.c | 15 +++++++++++----
    1 file changed, 11 insertions(+), 4 deletions(-)

    --- linux.trees.git.orig/net/core/dev.c
    +++ linux.trees.git/net/core/dev.c
    @@ -162,7 +162,7 @@ struct net_dma {
    struct dma_client client;
    spinlock_t lock;
    cpumask_t channel_mask;
    - struct dma_chan *channels[NR_CPUS];
    + struct dma_chan **channels;
    };

    static enum dma_state_client
    @@ -2444,7 +2444,7 @@ static struct netif_rx_stats *softnet_ge
    {
    struct netif_rx_stats *rc = NULL;

    - while (*pos < NR_CPUS)
    + while (*pos < nr_cpu_ids)
    if (cpu_online(*pos)) {
    rc = &per_cpu(netdev_rx_stat, *pos);
    break;
    @@ -4316,7 +4316,7 @@ netdev_dma_event(struct dma_client *clie
    spin_lock(&net_dma->lock);
    switch (state) {
    case DMA_RESOURCE_AVAILABLE:
    - for (i = 0; i < NR_CPUS; i++)
    + for (i = 0; i < nr_cpu_ids; i++)
    if (net_dma->channels[i] == chan) {
    found = 1;
    break;
    @@ -4331,7 +4331,7 @@ netdev_dma_event(struct dma_client *clie
    }
    break;
    case DMA_RESOURCE_REMOVED:
    - for (i = 0; i < NR_CPUS; i++)
    + for (i = 0; i < nr_cpu_ids; i++)
    if (net_dma->channels[i] == chan) {
    found = 1;
    pos = i;
    @@ -4358,6 +4358,13 @@ netdev_dma_event(struct dma_client *clie
    */
    static int __init netdev_dma_register(void)
    {
    + net_dma.channels = kzalloc(nr_cpu_ids * sizeof(struct net_dma),
    + GFP_KERNEL);
    + if (unlikely(!net_dma.channels)) {
    + printk(KERN_NOTICE
    + "netdev_dma: no memory for net_dma.channels\n");
    + return -ENOMEM;
    + }
    spin_lock_init(&net_dma.lock);
    dma_cap_set(DMA_MEMCPY, net_dma.client.cap_mask);
    dma_async_client_register(&net_dma.client);

    --
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  4. [PATCH 02/10] init: move setup of nr_cpu_ids to as early as possible v2

    Move the setting of nr_cpu_ids from sched_init() to setup_per_cpu_areas(),
    so that it's available as early as possible.

    Based on:
    git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
    git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git

    # ia64
    Cc: Tony Luck

    # powerpc
    Cc: Paul Mackerras
    Cc: Anton Blanchard

    # sparc
    Cc: David S. Miller
    Cc: William L. Irwin

    # x86
    Cc: Thomas Gleixner
    Cc: Ingo Molnar
    Cc: H. Peter Anvin

    Signed-off-by: Mike Travis
    ---

    Moved from the zero-based percpu variables patchset and redone to be
    integrated with setup_per_cpu_areas instead of being called before
    that function. This had to be done because some arch's call
    prefill_possible_map() from setup_per_cpu_areas() which may increase
    the number of possible cpus.

    v2: rebased on linux-2.6.git + linux-2.6-x86.git
    ---

    arch/ia64/kernel/acpi.c | 4 ++++
    arch/ia64/kernel/setup.c | 7 +++++++
    arch/powerpc/kernel/setup_64.c | 5 ++++-
    arch/sparc64/mm/init.c | 10 +++++++++-
    arch/x86/kernel/setup.c | 10 +++++++---
    init/main.c | 15 ++++++++++++---
    kernel/sched.c | 7 -------
    7 files changed, 43 insertions(+), 15 deletions(-)

    --- linux.trees.git.orig/arch/ia64/kernel/acpi.c
    +++ linux.trees.git/arch/ia64/kernel/acpi.c
    @@ -831,6 +831,10 @@ __init void prefill_possible_map(void)

    for (i = 0; i < possible; i++)
    cpu_set(i, cpu_possible_map);
    +
    +#ifdef CONFIG_SMP
    + nr_cpu_ids = possible;
    +#endif
    }

    int acpi_map_lsapic(acpi_handle handle, int *pcpu)
    --- linux.trees.git.orig/arch/ia64/kernel/setup.c
    +++ linux.trees.git/arch/ia64/kernel/setup.c
    @@ -765,6 +765,13 @@ setup_per_cpu_areas (void)
    /* start_kernel() requires this... */
    #ifdef CONFIG_ACPI_HOTPLUG_CPU
    prefill_possible_map();
    +#elif defined(CONFIG_SMP)
    + int cpu, highest_cpu = 0;
    +
    + for_each_possible_cpu(cpu)
    + highest_cpu = cpu;
    +
    + nr_cpu_ids = highest_cpu + 1;
    #endif
    }

    --- linux.trees.git.orig/arch/powerpc/kernel/setup_64.c
    +++ linux.trees.git/arch/powerpc/kernel/setup_64.c
    @@ -576,7 +576,7 @@ void cpu_die(void)
    #ifdef CONFIG_SMP
    void __init setup_per_cpu_areas(void)
    {
    - int i;
    + int i, highest_cpu = 0;
    unsigned long size;
    char *ptr;

    @@ -594,7 +594,10 @@ void __init setup_per_cpu_areas(void)

    paca[i].data_offset = ptr - __per_cpu_start;
    memcpy(ptr, __per_cpu_start, __per_cpu_end - __per_cpu_start);
    + if (i > highest_cpu)
    + highest_cpu = i;
    }
    + nr_cpu_ids = highest_cpu + 1;

    /* Now that per_cpu is setup, initialize cpu_sibling_map */
    smp_setup_cpu_sibling_map();
    --- linux.trees.git.orig/arch/sparc64/mm/init.c
    +++ linux.trees.git/arch/sparc64/mm/init.c
    @@ -1292,10 +1292,18 @@ pgd_t swapper_pg_dir[2048];
    static void sun4u_pgprot_init(void);
    static void sun4v_pgprot_init(void);

    -/* Dummy function */
    +#ifdef CONFIG_SMP
    +/* set nr_cpu_ids */
    void __init setup_per_cpu_areas(void)
    {
    + int cpu, highest_cpu = 0;
    +
    + for_each_possible_cpu(cpu)
    + highest_cpu = cpu;
    +
    + nr_cpu_ids = highest_cpu + 1;
    }
    +#endif

    void __init paging_init(void)
    {
    --- linux.trees.git.orig/arch/x86/kernel/setup.c
    +++ linux.trees.git/arch/x86/kernel/setup.c
    @@ -54,7 +54,7 @@ EXPORT_SYMBOL(__per_cpu_offset);
    */
    void __init setup_per_cpu_areas(void)
    {
    - int i;
    + int i, highest_cpu = 0;
    unsigned long size;

    #ifdef CONFIG_HOTPLUG_CPU
    @@ -80,15 +80,19 @@ void __init setup_per_cpu_areas(void)
    else
    ptr = alloc_bootmem_pages_node(NODE_DATA(node), size);
    #endif
    - if (!ptr)
    - panic("Cannot allocate cpu data for CPU %d\n", i);
    +
    #ifdef CONFIG_X86_64
    cpu_pda(i)->data_offset = ptr - __per_cpu_start;
    #else
    __per_cpu_offset[i] = ptr - __per_cpu_start;
    #endif
    memcpy(ptr, __per_cpu_start, __per_cpu_end - __per_cpu_start);
    +
    + if (i > highest_cpu)
    + highest_cpu = i;
    }
    + nr_cpu_ids = highest_cpu + 1;
    + printk(KERN_DEBUG "NR_CPUS:%d (nr_cpu_ids:%d)\n", NR_CPUS, nr_cpu_ids);

    /* Setup percpu data maps */
    setup_per_cpu_maps();
    --- linux.trees.git.orig/init/main.c
    +++ linux.trees.git/init/main.c
    @@ -364,16 +364,20 @@ static inline void smp_prepare_cpus(unsi

    #else

    +int nr_cpu_ids __read_mostly = NR_CPUS;
    +EXPORT_SYMBOL(nr_cpu_ids);
    +
    #ifndef CONFIG_HAVE_SETUP_PER_CPU_AREA
    unsigned long __per_cpu_offset[NR_CPUS] __read_mostly;
    -
    EXPORT_SYMBOL(__per_cpu_offset);

    +/* nr_cpu_ids is set as a side effect */
    static void __init setup_per_cpu_areas(void)
    {
    - unsigned long size, i;
    - char *ptr;
    + unsigned long size;
    + int i, highest_cpu = 0;
    unsigned long nr_possible_cpus = num_possible_cpus();
    + char *ptr;

    /* Copy section for each CPU (we discard the original) */
    size = ALIGN(PERCPU_ENOUGH_ROOM, PAGE_SIZE);
    @@ -383,7 +387,12 @@ static void __init setup_per_cpu_areas(v
    __per_cpu_offset[i] = ptr - __per_cpu_start;
    memcpy(ptr, __per_cpu_start, __per_cpu_end - __per_cpu_start);
    ptr += size;
    + if (i > highest_cpu)
    + highest_cpu = i;
    }
    +
    + nr_cpu_ids = highest_cpu + 1;
    + printk(KERN_DEBUG "NR_CPUS:%d (nr_cpu_ids:%d)\n", NR_CPUS, nr_cpu_ids);
    }
    #endif /* CONFIG_HAVE_SETUP_PER_CPU_AREA */

    --- linux.trees.git.orig/kernel/sched.c
    +++ linux.trees.git/kernel/sched.c
    @@ -5923,10 +5923,6 @@ void __init migration_init(void)

    #ifdef CONFIG_SMP

    -/* Number of possible processor ids */
    -int nr_cpu_ids __read_mostly = NR_CPUS;
    -EXPORT_SYMBOL(nr_cpu_ids);
    -
    #ifdef CONFIG_SCHED_DEBUG

    static int sched_domain_debug_one(struct sched_domain *sd, int cpu, int level)
    @@ -7152,7 +7148,6 @@ static void init_tg_rt_entry(struct rq *

    void __init sched_init(void)
    {
    - int highest_cpu = 0;
    int i, j;

    #ifdef CONFIG_SMP
    @@ -7207,7 +7202,6 @@ void __init sched_init(void)
    #endif
    init_rq_hrtick(rq);
    atomic_set(&rq->nr_iowait, 0);
    - highest_cpu = i;
    }

    set_load_weight(&init_task);
    @@ -7217,7 +7211,6 @@ void __init sched_init(void)
    #endif

    #ifdef CONFIG_SMP
    - nr_cpu_ids = highest_cpu + 1;
    open_softirq(SCHED_SOFTIRQ, run_rebalance_domains, NULL);
    #endif


    --
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  5. [PATCH 09/10] x86: oprofile: remove NR_CPUS arrays in arch/x86/oprofile/nmi_int.c

    Change the following arrays sized by NR_CPUS to be PERCPU variables:

    static struct op_msrs cpu_msrs[NR_CPUS];
    static unsigned long saved_lvtpc[NR_CPUS];

    Also some minor complaints from checkpatch.pl fixed.

    Based on:
    git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
    git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git

    Cc: Philippe Elie

    Signed-off-by: Mike Travis
    ---

    All changes were transparent except for:

    static void nmi_shutdown(void)
    {
    + struct op_msrs *msrs = &__get_cpu_var(cpu_msrs);
    nmi_enabled = 0;
    on_each_cpu(nmi_cpu_shutdown, NULL, 0, 1);
    unregister_die_notifier(&profile_exceptions_nb);
    - model->shutdown(cpu_msrs);
    + model->shutdown(msrs);
    free_msrs();
    }

    The existing code passed a reference to cpu 0's instance of struct op_msrs
    to model->shutdown, whilst the other functions are passed a reference to
    instance of a struct op_msrs. This seemed to be a bug to me
    even though as long as cpu 0 and are of the same type it would
    have the same effect...?
    ---
    arch/x86/oprofile/nmi_int.c | 49 ++++++++++++++++++++++++--------------------
    1 file changed, 27 insertions(+), 22 deletions(-)

    --- linux.trees.git.orig/arch/x86/oprofile/nmi_int.c
    +++ linux.trees.git/arch/x86/oprofile/nmi_int.c
    @@ -23,8 +23,8 @@
    #include "op_x86_model.h"

    static struct op_x86_model_spec const *model;
    -static struct op_msrs cpu_msrs[NR_CPUS];
    -static unsigned long saved_lvtpc[NR_CPUS];
    +static DEFINE_PER_CPU(struct op_msrs, cpu_msrs);
    +static DEFINE_PER_CPU(unsigned long, saved_lvtpc);

    static int nmi_start(void);
    static void nmi_stop(void);
    @@ -89,7 +89,7 @@ static int profile_exceptions_notify(str

    switch (val) {
    case DIE_NMI:
    - if (model->check_ctrs(args->regs, &cpu_msrs[cpu]))
    + if (model->check_ctrs(args->regs, &per_cpu(cpu_msrs, cpu)))
    ret = NOTIFY_STOP;
    break;
    default:
    @@ -126,7 +126,7 @@ static void nmi_cpu_save_registers(struc
    static void nmi_save_registers(void *dummy)
    {
    int cpu = smp_processor_id();
    - struct op_msrs *msrs = &cpu_msrs[cpu];
    + struct op_msrs *msrs = &per_cpu(cpu_msrs, cpu);
    nmi_cpu_save_registers(msrs);
    }

    @@ -134,10 +134,10 @@ static void free_msrs(void)
    {
    int i;
    for_each_possible_cpu(i) {
    - kfree(cpu_msrs[i].counters);
    - cpu_msrs[i].counters = NULL;
    - kfree(cpu_msrs[i].controls);
    - cpu_msrs[i].controls = NULL;
    + kfree(per_cpu(cpu_msrs, i).counters);
    + per_cpu(cpu_msrs, i).counters = NULL;
    + kfree(per_cpu(cpu_msrs, i).controls);
    + per_cpu(cpu_msrs, i).controls = NULL;
    }
    }

    @@ -149,13 +149,15 @@ static int allocate_msrs(void)

    int i;
    for_each_possible_cpu(i) {
    - cpu_msrs[i].counters = kmalloc(counters_size, GFP_KERNEL);
    - if (!cpu_msrs[i].counters) {
    + per_cpu(cpu_msrs, i).counters = kmalloc(counters_size,
    + GFP_KERNEL);
    + if (!per_cpu(cpu_msrs, i).counters) {
    success = 0;
    break;
    }
    - cpu_msrs[i].controls = kmalloc(controls_size, GFP_KERNEL);
    - if (!cpu_msrs[i].controls) {
    + per_cpu(cpu_msrs, i).controls = kmalloc(controls_size,
    + GFP_KERNEL);
    + if (!per_cpu(cpu_msrs, i).controls) {
    success = 0;
    break;
    }
    @@ -170,11 +172,11 @@ static int allocate_msrs(void)
    static void nmi_cpu_setup(void *dummy)
    {
    int cpu = smp_processor_id();
    - struct op_msrs *msrs = &cpu_msrs[cpu];
    + struct op_msrs *msrs = &per_cpu(cpu_msrs, cpu);
    spin_lock(&oprofilefs_lock);
    model->setup_ctrs(msrs);
    spin_unlock(&oprofilefs_lock);
    - saved_lvtpc[cpu] = apic_read(APIC_LVTPC);
    + per_cpu(saved_lvtpc, cpu) = apic_read(APIC_LVTPC);
    apic_write(APIC_LVTPC, APIC_DM_NMI);
    }

    @@ -203,13 +205,15 @@ static int nmi_setup(void)
    */

    /* Assume saved/restored counters are the same on all CPUs */
    - model->fill_in_addresses(&cpu_msrs[0]);
    + model->fill_in_addresses(&per_cpu(cpu_msrs, 0));
    for_each_possible_cpu(cpu) {
    if (cpu != 0) {
    - memcpy(cpu_msrs[cpu].counters, cpu_msrs[0].counters,
    + memcpy(per_cpu(cpu_msrs, cpu).counters,
    + per_cpu(cpu_msrs, 0).counters,
    sizeof(struct op_msr) * model->num_counters);

    - memcpy(cpu_msrs[cpu].controls, cpu_msrs[0].controls,
    + memcpy(per_cpu(cpu_msrs, cpu).controls,
    + per_cpu(cpu_msrs, 0).controls,
    sizeof(struct op_msr) * model->num_controls);
    }

    @@ -249,7 +253,7 @@ static void nmi_cpu_shutdown(void *dummy
    {
    unsigned int v;
    int cpu = smp_processor_id();
    - struct op_msrs *msrs = &cpu_msrs[cpu];
    + struct op_msrs *msrs = &__get_cpu_var(cpu_msrs);

    /* restoring APIC_LVTPC can trigger an apic error because the delivery
    * mode and vector nr combination can be illegal. That's by design: on
    @@ -258,23 +262,24 @@ static void nmi_cpu_shutdown(void *dummy
    */
    v = apic_read(APIC_LVTERR);
    apic_write(APIC_LVTERR, v | APIC_LVT_MASKED);
    - apic_write(APIC_LVTPC, saved_lvtpc[cpu]);
    + apic_write(APIC_LVTPC, per_cpu(saved_lvtpc, cpu));
    apic_write(APIC_LVTERR, v);
    nmi_restore_registers(msrs);
    }

    static void nmi_shutdown(void)
    {
    + struct op_msrs *msrs = &__get_cpu_var(cpu_msrs);
    nmi_enabled = 0;
    on_each_cpu(nmi_cpu_shutdown, NULL, 0, 1);
    unregister_die_notifier(&profile_exceptions_nb);
    - model->shutdown(cpu_msrs);
    + model->shutdown(msrs);
    free_msrs();
    }

    static void nmi_cpu_start(void *dummy)
    {
    - struct op_msrs const *msrs = &cpu_msrs[smp_processor_id()];
    + struct op_msrs const *msrs = &__get_cpu_var(cpu_msrs);
    model->start(msrs);
    }

    @@ -286,7 +291,7 @@ static int nmi_start(void)

    static void nmi_cpu_stop(void *dummy)
    {
    - struct op_msrs const *msrs = &cpu_msrs[smp_processor_id()];
    + struct op_msrs const *msrs = &__get_cpu_var(cpu_msrs);
    model->stop(msrs);
    }


    --
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  6. [PATCH 06/10] x86: reduce memory and stack usage in intel_cacheinfo

    * Change the following static arrays sized by NR_CPUS to
    per_cpu data variables:

    _cpuid4_info *cpuid4_info[NR_CPUS];
    _index_kobject *index_kobject[NR_CPUS];
    kobject * cache_kobject[NR_CPUS];

    * Remove the local NR_CPUS array with a kmalloc'd region in
    show_shared_cpu_map().

    Also some minor complaints from checkpatch.pl fixed.

    Based on:
    git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
    git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git

    Cc: Thomas Gleixner
    Cc: Ingo Molnar
    Cc: H. Peter Anvin
    Cc: Andi Kleen

    Signed-off-by: Mike Travis
    ---
    arch/x86/kernel/cpu/intel_cacheinfo.c | 70 +++++++++++++++++++---------------
    1 file changed, 40 insertions(+), 30 deletions(-)

    --- linux.trees.git.orig/arch/x86/kernel/cpu/intel_cacheinfo.c
    +++ linux.trees.git/arch/x86/kernel/cpu/intel_cacheinfo.c
    @@ -129,7 +129,7 @@ struct _cpuid4_info {
    union _cpuid4_leaf_ebx ebx;
    union _cpuid4_leaf_ecx ecx;
    unsigned long size;
    - cpumask_t shared_cpu_map;
    + cpumask_t shared_cpu_map; /* future?: only cpus/node is needed */
    };

    unsigned short num_cache_leaves;
    @@ -451,8 +451,8 @@ unsigned int __cpuinit init_intel_cachei
    }

    /* pointer to _cpuid4_info array (for each cache leaf) */
    -static struct _cpuid4_info *cpuid4_info[NR_CPUS];
    -#define CPUID4_INFO_IDX(x,y) (&((cpuid4_info[x])[y]))
    +static DEFINE_PER_CPU(struct _cpuid4_info *, cpuid4_info);
    +#define CPUID4_INFO_IDX(x, y) (&((per_cpu(cpuid4_info, x))[y]))

    #ifdef CONFIG_SMP
    static void __cpuinit cache_shared_cpu_map_setup(unsigned int cpu, int index)
    @@ -474,7 +474,7 @@ static void __cpuinit cache_shared_cpu_m
    if (cpu_data(i).apicid >> index_msb ==
    c->apicid >> index_msb) {
    cpu_set(i, this_leaf->shared_cpu_map);
    - if (i != cpu && cpuid4_info[i]) {
    + if (i != cpu && per_cpu(cpuid4_info, i)) {
    sibling_leaf = CPUID4_INFO_IDX(i, index);
    cpu_set(cpu, sibling_leaf->shared_cpu_map);
    }
    @@ -505,8 +505,8 @@ static void __cpuinit free_cache_attribu
    for (i = 0; i < num_cache_leaves; i++)
    cache_remove_shared_cpu_map(cpu, i);

    - kfree(cpuid4_info[cpu]);
    - cpuid4_info[cpu] = NULL;
    + kfree(per_cpu(cpuid4_info, cpu));
    + per_cpu(cpuid4_info, cpu) = NULL;
    }

    static int __cpuinit detect_cache_attributes(unsigned int cpu)
    @@ -519,9 +519,9 @@ static int __cpuinit detect_cache_attrib
    if (num_cache_leaves == 0)
    return -ENOENT;

    - cpuid4_info[cpu] = kzalloc(
    + per_cpu(cpuid4_info, cpu) = kzalloc(
    sizeof(struct _cpuid4_info) * num_cache_leaves, GFP_KERNEL);
    - if (cpuid4_info[cpu] == NULL)
    + if (per_cpu(cpuid4_info, cpu) == NULL)
    return -ENOMEM;

    oldmask = current->cpus_allowed;
    @@ -546,8 +546,8 @@ static int __cpuinit detect_cache_attrib

    out:
    if (retval) {
    - kfree(cpuid4_info[cpu]);
    - cpuid4_info[cpu] = NULL;
    + kfree(per_cpu(cpuid4_info, cpu));
    + per_cpu(cpuid4_info, cpu) = NULL;
    }

    return retval;
    @@ -561,7 +561,7 @@ out:
    extern struct sysdev_class cpu_sysdev_class; /* from drivers/base/cpu.c */

    /* pointer to kobject for cpuX/cache */
    -static struct kobject * cache_kobject[NR_CPUS];
    +static DEFINE_PER_CPU(struct kobject *, cache_kobject);

    struct _index_kobject {
    struct kobject kobj;
    @@ -570,8 +570,8 @@ struct _index_kobject {
    };

    /* pointer to array of kobjects for cpuX/cache/indexY */
    -static struct _index_kobject *index_kobject[NR_CPUS];
    -#define INDEX_KOBJECT_PTR(x,y) (&((index_kobject[x])[y]))
    +static DEFINE_PER_CPU(struct _index_kobject *, index_kobject);
    +#define INDEX_KOBJECT_PTR(x, y) (&((per_cpu(index_kobject, x))[y]))

    #define show_one_plus(file_name, object, val) \
    static ssize_t show_##file_name \
    @@ -593,9 +593,16 @@ static ssize_t show_size(struct _cpuid4_

    static ssize_t show_shared_cpu_map(struct _cpuid4_info *this_leaf, char *buf)
    {
    - char mask_str[NR_CPUS];
    - cpumask_scnprintf(mask_str, NR_CPUS, this_leaf->shared_cpu_map);
    - return sprintf(buf, "%s\n", mask_str);
    + int n = 0;
    + int len = cpumask_scnprintf_len(nr_cpu_ids);
    + char *mask_str = kmalloc(len, GFP_KERNEL);
    +
    + if (mask_str) {
    + cpumask_scnprintf(mask_str, len, this_leaf->shared_cpu_map);
    + n = sprintf(buf, "%s\n", mask_str);
    + kfree(mask_str);
    + }
    + return n;
    }

    static ssize_t show_type(struct _cpuid4_info *this_leaf, char *buf) {
    @@ -684,10 +691,10 @@ static struct kobj_type ktype_percpu_ent

    static void __cpuinit cpuid4_cache_sysfs_exit(unsigned int cpu)
    {
    - kfree(cache_kobject[cpu]);
    - kfree(index_kobject[cpu]);
    - cache_kobject[cpu] = NULL;
    - index_kobject[cpu] = NULL;
    + kfree(per_cpu(cache_kobject, cpu));
    + kfree(per_cpu(index_kobject, cpu));
    + per_cpu(cache_kobject, cpu) = NULL;
    + per_cpu(index_kobject, cpu) = NULL;
    free_cache_attributes(cpu);
    }

    @@ -703,13 +710,14 @@ static int __cpuinit cpuid4_cache_sysfs_
    return err;

    /* Allocate all required memory */
    - cache_kobject[cpu] = kzalloc(sizeof(struct kobject), GFP_KERNEL);
    - if (unlikely(cache_kobject[cpu] == NULL))
    + per_cpu(cache_kobject, cpu) =
    + kzalloc(sizeof(struct kobject), GFP_KERNEL);
    + if (unlikely(per_cpu(cache_kobject, cpu) == NULL))
    goto err_out;

    - index_kobject[cpu] = kzalloc(
    + per_cpu(index_kobject, cpu) = kzalloc(
    sizeof(struct _index_kobject ) * num_cache_leaves, GFP_KERNEL);
    - if (unlikely(index_kobject[cpu] == NULL))
    + if (unlikely(per_cpu(index_kobject, cpu) == NULL))
    goto err_out;

    return 0;
    @@ -733,7 +741,8 @@ static int __cpuinit cache_add_dev(struc
    if (unlikely(retval < 0))
    return retval;

    - retval = kobject_init_and_add(cache_kobject[cpu], &ktype_percpu_entry,
    + retval = kobject_init_and_add(per_cpu(cache_kobject, cpu),
    + &ktype_percpu_entry,
    &sys_dev->kobj, "%s", "cache");
    if (retval < 0) {
    cpuid4_cache_sysfs_exit(cpu);
    @@ -745,13 +754,14 @@ static int __cpuinit cache_add_dev(struc
    this_object->cpu = cpu;
    this_object->index = i;
    retval = kobject_init_and_add(&(this_object->kobj),
    - &ktype_cache, cache_kobject[cpu],
    + &ktype_cache,
    + per_cpu(cache_kobject, cpu),
    "index%1lu", i);
    if (unlikely(retval)) {
    for (j = 0; j < i; j++) {
    kobject_put(&(INDEX_KOBJECT_PTR(cpu,j)->kobj));
    }
    - kobject_put(cache_kobject[cpu]);
    + kobject_put(per_cpu(cache_kobject, cpu));
    cpuid4_cache_sysfs_exit(cpu);
    break;
    }
    @@ -760,7 +770,7 @@ static int __cpuinit cache_add_dev(struc
    if (!retval)
    cpu_set(cpu, cache_dev_map);

    - kobject_uevent(cache_kobject[cpu], KOBJ_ADD);
    + kobject_uevent(per_cpu(cache_kobject, cpu), KOBJ_ADD);
    return retval;
    }

    @@ -769,7 +779,7 @@ static void __cpuinit cache_remove_dev(s
    unsigned int cpu = sys_dev->id;
    unsigned long i;

    - if (cpuid4_info[cpu] == NULL)
    + if (per_cpu(cpuid4_info, cpu) == NULL)
    return;
    if (!cpu_isset(cpu, cache_dev_map))
    return;
    @@ -777,7 +787,7 @@ static void __cpuinit cache_remove_dev(s

    for (i = 0; i < num_cache_leaves; i++)
    kobject_put(&(INDEX_KOBJECT_PTR(cpu,i)->kobj));
    - kobject_put(cache_kobject[cpu]);
    + kobject_put(per_cpu(cache_kobject, cpu));
    cpuid4_cache_sysfs_exit(cpu);
    }


    --
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  7. [PATCH 10/10] sched: Remove fixed NR_CPUS sized arrays in kernel_sched.c

    Change fixed size arrays to per_cpu variables or dynamically allocated
    arrays in sched_init() and sched_init_smp().

    (1) static struct sched_entity *init_sched_entity_p[NR_CPUS];
    (1) static struct cfs_rq *init_cfs_rq_p[NR_CPUS];
    (1) static struct sched_rt_entity *init_sched_rt_entity_p[NR_CPUS];
    (1) static struct rt_rq *init_rt_rq_p[NR_CPUS];
    static struct sched_group **sched_group_nodes_bycpu[NR_CPUS];
    char str[NR_CPUS];
    int ints[NR_CPUS], i;

    (1 - these arrays are allocated via alloc_bootmem_low())

    Also in sched_create_group() we allocate new arrays based on nr_cpu_ids.

    Based on:
    git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
    git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git

    Cc: Ingo Molnar
    Signed-off-by: Mike Travis
    ---
    kernel/sched.c | 92 +++++++++++++++++++++++++++++++++++++++------------------
    1 file changed, 63 insertions(+), 29 deletions(-)

    --- linux.trees.git.orig/kernel/sched.c
    +++ linux.trees.git/kernel/sched.c
    @@ -66,6 +66,7 @@
    #include
    #include
    #include
    +#include

    #include
    #include
    @@ -193,17 +194,11 @@ struct task_group {
    static DEFINE_PER_CPU(struct sched_entity, init_sched_entity);
    /* Default task group's cfs_rq on each cpu */
    static DEFINE_PER_CPU(struct cfs_rq, init_cfs_rq) ____cacheline_aligned_in_smp;
    -
    -static struct sched_entity *init_sched_entity_p[NR_CPUS];
    -static struct cfs_rq *init_cfs_rq_p[NR_CPUS];
    #endif

    #ifdef CONFIG_RT_GROUP_SCHED
    static DEFINE_PER_CPU(struct sched_rt_entity, init_sched_rt_entity);
    static DEFINE_PER_CPU(struct rt_rq, init_rt_rq) ____cacheline_aligned_in_smp;
    -
    -static struct sched_rt_entity *init_sched_rt_entity_p[NR_CPUS];
    -static struct rt_rq *init_rt_rq_p[NR_CPUS];
    #endif

    /* task_group_lock serializes add/remove of task groups and also changes to
    @@ -227,17 +222,7 @@ static int init_task_group_load = INIT_T
    /* Default task group.
    * Every task in system belong to this group at bootup.
    */
    -struct task_group init_task_group = {
    -#ifdef CONFIG_FAIR_GROUP_SCHED
    - .se = init_sched_entity_p,
    - .cfs_rq = init_cfs_rq_p,
    -#endif
    -
    -#ifdef CONFIG_RT_GROUP_SCHED
    - .rt_se = init_sched_rt_entity_p,
    - .rt_rq = init_rt_rq_p,
    -#endif
    -};
    +struct task_group init_task_group;

    /* return group to which a task belongs */
    static inline struct task_group *task_group(struct task_struct *p)
    @@ -3518,7 +3503,7 @@ static inline void trigger_load_balance(
    */
    int ilb = first_cpu(nohz.cpu_mask);

    - if (ilb != NR_CPUS)
    + if (ilb < nr_cpu_ids)
    resched_cpu(ilb);
    }
    }
    @@ -5470,11 +5455,11 @@ static void move_task_off_dead_cpu(int d
    dest_cpu = any_online_cpu(mask);

    /* On any allowed CPU? */
    - if (dest_cpu == NR_CPUS)
    + if (dest_cpu >= nr_cpu_ids)
    dest_cpu = any_online_cpu(p->cpus_allowed);

    /* No more Mr. Nice Guy. */
    - if (dest_cpu == NR_CPUS) {
    + if (dest_cpu >= nr_cpu_ids) {
    cpumask_t cpus_allowed = cpuset_cpus_allowed_locked(p);
    /*
    * Try to stay on the same cpuset, where the
    @@ -5929,9 +5914,16 @@ static int sched_domain_debug_one(struct
    {
    struct sched_group *group = sd->groups;
    cpumask_t groupmask;
    - char str[NR_CPUS];
    + int len = cpumask_scnprintf_len(nr_cpu_ids);
    + char *str = kmalloc(len, GFP_KERNEL);
    + int ret = 0;
    +
    + if (!str) {
    + printk(KERN_DEBUG "Cannot load-balance (no memory)\n");
    + return -1;
    + }

    - cpumask_scnprintf(str, NR_CPUS, sd->span);
    + cpumask_scnprintf(str, len, sd->span);
    cpus_clear(groupmask);

    printk(KERN_DEBUG "%*s domain %d: ", level, "", level);
    @@ -5941,6 +5933,7 @@ static int sched_domain_debug_one(struct
    if (sd->parent)
    printk(KERN_ERR "ERROR: !SD_LOAD_BALANCE domain"
    " has parent");
    + kfree(str);
    return -1;
    }

    @@ -5984,7 +5977,7 @@ static int sched_domain_debug_one(struct

    cpus_or(groupmask, groupmask, group->cpumask);

    - cpumask_scnprintf(str, NR_CPUS, group->cpumask);
    + cpumask_scnprintf(str, len, group->cpumask);
    printk(KERN_CONT " %s", str);

    group = group->next;
    @@ -5997,6 +5990,8 @@ static int sched_domain_debug_one(struct
    if (sd->parent && !cpus_subset(groupmask, sd->parent->span))
    printk(KERN_ERR "ERROR: parent span is not a superset "
    "of domain->span\n");
    +
    + kfree(str);
    return 0;
    }

    @@ -6198,7 +6193,7 @@ __setup("isolcpus=", isolated_cpu_setup)
    /*
    * init_sched_build_groups takes the cpumask we wish to span, and a pointer
    * to a function which identifies what group(along with sched group) a CPU
    - * belongs to. The return value of group_fn must be a >= 0 and < NR_CPUS
    + * belongs to. The return value of group_fn must be a >= 0 and < nr_cpu_ids
    * (due to the fact that we keep track of groups covered with a cpumask_t).
    *
    * init_sched_build_groups will build a circular linked list of the groups
    @@ -6396,7 +6391,7 @@ cpu_to_phys_group(int cpu, const cpumask
    * gets dynamically allocated.
    */
    static DEFINE_PER_CPU(struct sched_domain, node_domains);
    -static struct sched_group **sched_group_nodes_bycpu[NR_CPUS];
    +static struct sched_group ***sched_group_nodes_bycpu;

    static DEFINE_PER_CPU(struct sched_domain, allnodes_domains);
    static DEFINE_PER_CPU(struct sched_group, sched_group_allnodes);
    @@ -7039,6 +7034,11 @@ void __init sched_init_smp(void)
    {
    cpumask_t non_isolated_cpus;

    +#if defined(CONFIG_NUMA)
    + sched_group_nodes_bycpu = kzalloc(nr_cpu_ids * sizeof(void **),
    + GFP_KERNEL);
    + BUG_ON(sched_group_nodes_bycpu == NULL);
    +#endif
    get_online_cpus();
    arch_init_sched_domains(&cpu_online_map);
    cpus_andnot(non_isolated_cpus, cpu_possible_map, cpu_isolated_map);
    @@ -7056,6 +7056,11 @@ void __init sched_init_smp(void)
    #else
    void __init sched_init_smp(void)
    {
    +#if defined(CONFIG_NUMA)
    + sched_group_nodes_bycpu = kzalloc(nr_cpu_ids * sizeof(void **),
    + GFP_KERNEL);
    + BUG_ON(sched_group_nodes_bycpu == NULL);
    +#endif
    sched_init_granularity();
    }
    #endif /* CONFIG_SMP */
    @@ -7149,6 +7154,35 @@ static void init_tg_rt_entry(struct rq *
    void __init sched_init(void)
    {
    int i, j;
    + unsigned long alloc_size = 0, ptr;
    +
    +#ifdef CONFIG_FAIR_GROUP_SCHED
    + alloc_size += 2 * nr_cpu_ids * sizeof(void **);
    +#endif
    +#ifdef CONFIG_RT_GROUP_SCHED
    + alloc_size += 2 * nr_cpu_ids * sizeof(void **);
    +#endif
    + /*
    + * As sched_init() is called before page_alloc is setup,
    + * we use alloc_bootmem().
    + */
    + if (alloc_size) {
    + ptr = (unsigned long)alloc_bootmem_low(alloc_size);
    +
    +#ifdef CONFIG_FAIR_GROUP_SCHED
    + init_task_group.se = (struct sched_entity **)ptr;
    + ptr += nr_cpu_ids * sizeof(void **);
    +
    + init_task_group.cfs_rq = (struct cfs_rq **)ptr;
    + ptr += nr_cpu_ids * sizeof(void **);
    +#endif
    +#ifdef CONFIG_RT_GROUP_SCHED
    + init_task_group.rt_se = (struct sched_rt_entity **)ptr;
    + ptr += nr_cpu_ids * sizeof(void **);
    +
    + init_task_group.rt_rq = (struct rt_rq **)ptr;
    +#endif
    + }

    #ifdef CONFIG_SMP
    init_defrootdomain();
    @@ -7394,10 +7428,10 @@ static int alloc_fair_sched_group(struct
    struct rq *rq;
    int i;

    - tg->cfs_rq = kzalloc(sizeof(cfs_rq) * NR_CPUS, GFP_KERNEL);
    + tg->cfs_rq = kzalloc(sizeof(cfs_rq) * nr_cpu_ids, GFP_KERNEL);
    if (!tg->cfs_rq)
    goto err;
    - tg->se = kzalloc(sizeof(se) * NR_CPUS, GFP_KERNEL);
    + tg->se = kzalloc(sizeof(se) * nr_cpu_ids, GFP_KERNEL);
    if (!tg->se)
    goto err;

    @@ -7477,10 +7511,10 @@ static int alloc_rt_sched_group(struct t
    struct rq *rq;
    int i;

    - tg->rt_rq = kzalloc(sizeof(rt_rq) * NR_CPUS, GFP_KERNEL);
    + tg->rt_rq = kzalloc(sizeof(rt_rq) * nr_cpu_ids, GFP_KERNEL);
    if (!tg->rt_rq)
    goto err;
    - tg->rt_se = kzalloc(sizeof(rt_se) * NR_CPUS, GFP_KERNEL);
    + tg->rt_se = kzalloc(sizeof(rt_se) * nr_cpu_ids, GFP_KERNEL);
    if (!tg->rt_se)
    goto err;


    --
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  8. Re: [PATCH 02/10] init: move setup of nr_cpu_ids to as early as possible v2


    * Mike Travis wrote:

    > Move the setting of nr_cpu_ids from sched_init() to
    > setup_per_cpu_areas(), so that it's available as early as possible.


    hm, why not a separate call before setup_per_cpu_areas(), so that we can
    avoid spreading this from generic kernel into a bunch of architectures
    that happen to have their own version of setup_per_cpu_areas():

    > 7 files changed, 43 insertions(+), 15 deletions(-)


    Ingo
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  9. Re: [PATCH 00/10] NR_CPUS: third reduction of NR_CPUS memory usage x86-version v2


    * Mike Travis wrote:

    > Wii, isn't this fun...! This is a resubmission of yesterday's patches
    > based on the x86.git/latest tree. Yes, it _is_ a maze of twisty litle
    > passages. ;-)


    just to make patch dependencies clear: most of the patches here can be
    applied to their base trees as-is, without depending on any other patch,
    correct?

    the only undeclared dependency i found was the cpumask_scnprintf_len()
    patch - please prominently list dependencies in the changelog like this:

    [ this patch depends on "cpumask: Add cpumask_scnprintf_len function" ]

    Ingo
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  10. Re: [PATCH 01/10] x86_64: Cleanup non-smp usage of cpu maps v2


    * Mike Travis wrote:

    > Cleanup references to the early cpu maps for the non-SMP configuration
    > and remove some functions called for SMP configurations only.


    thanks, applied.

    one observation:

    > +#ifdef CONFIG_SMP
    > extern int x86_cpu_to_node_map_init[];
    > extern void *x86_cpu_to_node_map_early_ptr;
    > +#else
    > +#define x86_cpu_to_node_map_early_ptr NULL
    > +#endif


    Right now all these early_ptrs are in essence open-coded "early
    per-cpu", right? But shouldnt we solve that in a much cleaner way: by
    explicitly adding an early-per-cpu types and accessors, and avoid all
    that #ifdeffery?

    Ingo
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  11. Re: [PATCH 09/10] x86: oprofile: remove NR_CPUS arrays in arch/x86/oprofile/nmi_int.c


    * Mike Travis wrote:

    > Change the following arrays sized by NR_CPUS to be PERCPU variables:
    >
    > static struct op_msrs cpu_msrs[NR_CPUS];
    > static unsigned long saved_lvtpc[NR_CPUS];
    >
    > Also some minor complaints from checkpatch.pl fixed.


    thanks, applied.

    > All changes were transparent except for:
    >
    > static void nmi_shutdown(void)
    > {
    > + struct op_msrs *msrs = &__get_cpu_var(cpu_msrs);
    > nmi_enabled = 0;
    > on_each_cpu(nmi_cpu_shutdown, NULL, 0, 1);
    > unregister_die_notifier(&profile_exceptions_nb);
    > - model->shutdown(cpu_msrs);
    > + model->shutdown(msrs);
    > free_msrs();
    > }
    >
    > The existing code passed a reference to cpu 0's instance of struct
    > op_msrs to model->shutdown, whilst the other functions are passed a
    > reference to instance of a struct op_msrs. This seemed
    > to be a bug to me even though as long as cpu 0 and are of
    > the same type it would have the same effect...?


    i dont think this has any real effect in practice (the model pointers
    are not expected to change across cpus on the same system) - but in any
    case i've promoted your observation to the main portion of the changelog
    so that we'll have notice of this.

    (someone might want to play with simulating a weaker CPU on a secondary
    core, but we've got tons of other assumptions on CPU type symmetry.)

    Ingo
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  12. Re: [PATCH 06/10] x86: reduce memory and stack usage in intel_cacheinfo


    * Mike Travis wrote:

    > * Change the following static arrays sized by NR_CPUS to
    > per_cpu data variables:
    >
    > _cpuid4_info *cpuid4_info[NR_CPUS];
    > _index_kobject *index_kobject[NR_CPUS];
    > kobject * cache_kobject[NR_CPUS];
    >
    > * Remove the local NR_CPUS array with a kmalloc'd region in
    > show_shared_cpu_map().


    thanks Travis, i've applied this to x86.git.

    one observation:

    > static ssize_t show_shared_cpu_map(struct _cpuid4_info *this_leaf, char *buf)
    > {
    > - char mask_str[NR_CPUS];
    > - cpumask_scnprintf(mask_str, NR_CPUS, this_leaf->shared_cpu_map);
    > - return sprintf(buf, "%s\n", mask_str);
    > + int n = 0;
    > + int len = cpumask_scnprintf_len(nr_cpu_ids);
    > + char *mask_str = kmalloc(len, GFP_KERNEL);
    > +
    > + if (mask_str) {
    > + cpumask_scnprintf(mask_str, len, this_leaf->shared_cpu_map);
    > + n = sprintf(buf, "%s\n", mask_str);
    > + kfree(mask_str);
    > + }
    > + return n;


    the other changes look good, but this one looks a bit ugly and complex.
    We basically want to sprintf shared_cpu_map into 'buf', but we do that
    by first allocating a temporary buffer, print a string into it, then
    print that string into another buffer ...

    this very much smells like an API bug in cpumask_scnprintf() - why dont
    you create a cpumask_scnprintf_ptr() API that takes a pointer to a
    cpumask? Then this change would become a trivial and much more readable:

    - char mask_str[NR_CPUS];
    - cpumask_scnprintf(mask_str, NR_CPUS, this_leaf->shared_cpu_map);
    - return sprintf(buf, "%s\n", mask_str);
    + return cpumask_scnprintf_ptr(buf, NR_CPUS, &this_leaf->shared_cpu_map);

    Ingo
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  13. Re: [PATCH 02/10] init: move setup of nr_cpu_ids to as early as possible v2

    Ingo Molnar wrote:
    > * Mike Travis wrote:
    >
    >> Move the setting of nr_cpu_ids from sched_init() to
    >> setup_per_cpu_areas(), so that it's available as early as possible.

    >
    > hm, why not a separate call before setup_per_cpu_areas(), so that we can
    > avoid spreading this from generic kernel into a bunch of architectures
    > that happen to have their own version of setup_per_cpu_areas():
    >
    >> 7 files changed, 43 insertions(+), 15 deletions(-)

    >
    > Ingo


    I had this before but I then discovered that an arch would increase
    (and possible decrease) it's number of possible cpus in setup_per_cpu_areas().
    So I figured that setting nr_cpu_ids (and the cpumask_of_cpu map) should
    be a side effect of setup_per_cpu_areas().

    Thanks,
    Mike
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  14. Re: [PATCH 00/10] NR_CPUS: third reduction of NR_CPUS memory usage x86-version v2

    Ingo Molnar wrote:
    > * Mike Travis wrote:
    >
    >> Wii, isn't this fun...! This is a resubmission of yesterday's patches
    >> based on the x86.git/latest tree. Yes, it _is_ a maze of twisty litle
    >> passages. ;-)

    >
    > just to make patch dependencies clear: most of the patches here can be
    > applied to their base trees as-is, without depending on any other patch,
    > correct?
    >
    > the only undeclared dependency i found was the cpumask_scnprintf_len()
    > patch - please prominently list dependencies in the changelog like this:
    >
    > [ this patch depends on "cpumask: Add cpumask_scnprintf_len function" ]
    >
    > Ingo



    Ahh, ok. I was under the assumption that an entire patchset would be
    applied en-mass and only divided up by bi-sect debugging...?

    The second patchset (cpumask) is highly incremental and I did it like
    this to show memory gains (or losses). I tossed a few patches that
    didn't have any overall goodness (and have a few more to help with
    the memory footprint or performance in the queue.)

    Thanks,
    Mike
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  15. Re: [PATCH 06/10] x86: reduce memory and stack usage in intel_cacheinfo

    Ingo Molnar wrote:
    > * Mike Travis wrote:
    >
    >> * Change the following static arrays sized by NR_CPUS to
    >> per_cpu data variables:
    >>
    >> _cpuid4_info *cpuid4_info[NR_CPUS];
    >> _index_kobject *index_kobject[NR_CPUS];
    >> kobject * cache_kobject[NR_CPUS];
    >>
    >> * Remove the local NR_CPUS array with a kmalloc'd region in
    >> show_shared_cpu_map().

    >
    > thanks Travis, i've applied this to x86.git.
    >
    > one observation:
    >
    >> static ssize_t show_shared_cpu_map(struct _cpuid4_info *this_leaf, char *buf)
    >> {
    >> - char mask_str[NR_CPUS];
    >> - cpumask_scnprintf(mask_str, NR_CPUS, this_leaf->shared_cpu_map);
    >> - return sprintf(buf, "%s\n", mask_str);
    >> + int n = 0;
    >> + int len = cpumask_scnprintf_len(nr_cpu_ids);
    >> + char *mask_str = kmalloc(len, GFP_KERNEL);
    >> +
    >> + if (mask_str) {
    >> + cpumask_scnprintf(mask_str, len, this_leaf->shared_cpu_map);
    >> + n = sprintf(buf, "%s\n", mask_str);
    >> + kfree(mask_str);
    >> + }
    >> + return n;

    >
    > the other changes look good, but this one looks a bit ugly and complex.
    > We basically want to sprintf shared_cpu_map into 'buf', but we do that
    > by first allocating a temporary buffer, print a string into it, then
    > print that string into another buffer ...
    >
    > this very much smells like an API bug in cpumask_scnprintf() - why dont
    > you create a cpumask_scnprintf_ptr() API that takes a pointer to a
    > cpumask? Then this change would become a trivial and much more readable:
    >
    > - char mask_str[NR_CPUS];
    > - cpumask_scnprintf(mask_str, NR_CPUS, this_leaf->shared_cpu_map);
    > - return sprintf(buf, "%s\n", mask_str);
    > + return cpumask_scnprintf_ptr(buf, NR_CPUS, &this_leaf->shared_cpu_map);
    >
    > Ingo


    The main goal was to avoid allocating 4096 bytes when only 32 would do
    (characters needed to represent nr_cpu_ids cpus instead of NR_CPUS cpus.)
    But I'll look at cleaning it up a bit more. It wouldn't have to be
    a function if CHUNKSZ in cpumask_scnprintf() were visible (or a non-changeable
    constant.)

    Thanks,
    Mike
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  16. Re: [PATCH 06/10] x86: reduce memory and stack usage in intel_cacheinfo

    Mike Travis wrote:
    > Ingo Molnar wrote:
    >
    >> * Mike Travis wrote:
    >>
    >>
    >>> * Change the following static arrays sized by NR_CPUS to
    >>> per_cpu data variables:
    >>>
    >>> _cpuid4_info *cpuid4_info[NR_CPUS];
    >>> _index_kobject *index_kobject[NR_CPUS];
    >>> kobject * cache_kobject[NR_CPUS];
    >>>
    >>> * Remove the local NR_CPUS array with a kmalloc'd region in
    >>> show_shared_cpu_map().
    >>>

    >> thanks Travis, i've applied this to x86.git.
    >>
    >> one observation:
    >>
    >>
    >>> static ssize_t show_shared_cpu_map(struct _cpuid4_info *this_leaf, char *buf)
    >>> {
    >>> - char mask_str[NR_CPUS];
    >>> - cpumask_scnprintf(mask_str, NR_CPUS, this_leaf->shared_cpu_map);
    >>> - return sprintf(buf, "%s\n", mask_str);
    >>> + int n = 0;
    >>> + int len = cpumask_scnprintf_len(nr_cpu_ids);
    >>> + char *mask_str = kmalloc(len, GFP_KERNEL);
    >>> +
    >>> + if (mask_str) {
    >>> + cpumask_scnprintf(mask_str, len, this_leaf->shared_cpu_map);
    >>> + n = sprintf(buf, "%s\n", mask_str);
    >>> + kfree(mask_str);
    >>> + }
    >>> + return n;
    >>>

    >> the other changes look good, but this one looks a bit ugly and complex.
    >> We basically want to sprintf shared_cpu_map into 'buf', but we do that
    >> by first allocating a temporary buffer, print a string into it, then
    >> print that string into another buffer ...
    >>
    >> this very much smells like an API bug in cpumask_scnprintf() - why dont
    >> you create a cpumask_scnprintf_ptr() API that takes a pointer to a
    >> cpumask? Then this change would become a trivial and much more readable:
    >>
    >> - char mask_str[NR_CPUS];
    >> - cpumask_scnprintf(mask_str, NR_CPUS, this_leaf->shared_cpu_map);
    >> - return sprintf(buf, "%s\n", mask_str);
    >> + return cpumask_scnprintf_ptr(buf, NR_CPUS, &this_leaf->shared_cpu_map);
    >>
    >> Ingo
    >>

    >
    > The main goal was to avoid allocating 4096 bytes when only 32 would do
    > (characters needed to represent nr_cpu_ids cpus instead of NR_CPUS cpus.)
    > But I'll look at cleaning it up a bit more. It wouldn't have to be
    > a function if CHUNKSZ in cpumask_scnprintf() were visible (or a non-changeable
    > constant.)
    >


    It's a pity you can't take advantage of kasprintf to handle all this.

    Hm, I would say that bitmap_scnprintf is a candidate for implementation
    as a printk format specifier so you could get away from needing a
    special function to print bitmaps...

    Eh? What's the difference between snprintf and scnprintf?

    J
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  17. Re: [PATCH 01/10] x86_64: Cleanup non-smp usage of cpu maps v2

    Ingo Molnar wrote:
    > * Mike Travis wrote:
    >
    >> Cleanup references to the early cpu maps for the non-SMP configuration
    >> and remove some functions called for SMP configurations only.

    >
    > thanks, applied.
    >
    > one observation:
    >
    >> +#ifdef CONFIG_SMP
    >> extern int x86_cpu_to_node_map_init[];
    >> extern void *x86_cpu_to_node_map_early_ptr;
    >> +#else
    >> +#define x86_cpu_to_node_map_early_ptr NULL
    >> +#endif

    >
    > Right now all these early_ptrs are in essence open-coded "early
    > per-cpu", right? But shouldnt we solve that in a much cleaner way: by
    > explicitly adding an early-per-cpu types and accessors, and avoid all
    > that #ifdeffery?
    >
    > Ingo


    I was thinking of something similar but had to put it on the back
    burner until we got to the point of being able to boot a kernel
    with NR_CPUS set to 4096. It should pop back up on the priority
    queue very soon... ;-)

    Thanks!
    Mike
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  18. Re: [PATCH 06/10] x86: reduce memory and stack usage in intel_cacheinfo

    Jeremy Fitzhardinge wrote:
    > Mike Travis wrote:
    >> Ingo Molnar wrote:
    >>
    >>> * Mike Travis wrote:
    >>>
    >>>
    >>>> * Change the following static arrays sized by NR_CPUS to
    >>>> per_cpu data variables:
    >>>>
    >>>> _cpuid4_info *cpuid4_info[NR_CPUS];
    >>>> _index_kobject *index_kobject[NR_CPUS];
    >>>> kobject * cache_kobject[NR_CPUS];
    >>>>
    >>>> * Remove the local NR_CPUS array with a kmalloc'd region in
    >>>> show_shared_cpu_map().
    >>>>
    >>> thanks Travis, i've applied this to x86.git.
    >>>
    >>> one observation:
    >>>
    >>>
    >>>> static ssize_t show_shared_cpu_map(struct _cpuid4_info *this_leaf,
    >>>> char *buf)
    >>>> {
    >>>> - char mask_str[NR_CPUS];
    >>>> - cpumask_scnprintf(mask_str, NR_CPUS, this_leaf->shared_cpu_map);
    >>>> - return sprintf(buf, "%s\n", mask_str);
    >>>> + int n = 0;
    >>>> + int len = cpumask_scnprintf_len(nr_cpu_ids);
    >>>> + char *mask_str = kmalloc(len, GFP_KERNEL);
    >>>> +
    >>>> + if (mask_str) {
    >>>> + cpumask_scnprintf(mask_str, len, this_leaf->shared_cpu_map);
    >>>> + n = sprintf(buf, "%s\n", mask_str);
    >>>> + kfree(mask_str);
    >>>> + }
    >>>> + return n;
    >>>>
    >>> the other changes look good, but this one looks a bit ugly and
    >>> complex. We basically want to sprintf shared_cpu_map into 'buf', but
    >>> we do that by first allocating a temporary buffer, print a string
    >>> into it, then print that string into another buffer ...
    >>>
    >>> this very much smells like an API bug in cpumask_scnprintf() - why
    >>> dont you create a cpumask_scnprintf_ptr() API that takes a pointer to
    >>> a cpumask? Then this change would become a trivial and much more
    >>> readable:
    >>>
    >>> - char mask_str[NR_CPUS];
    >>> - cpumask_scnprintf(mask_str, NR_CPUS, this_leaf->shared_cpu_map);
    >>> - return sprintf(buf, "%s\n", mask_str);
    >>> + return cpumask_scnprintf_ptr(buf, NR_CPUS,
    >>> &this_leaf->shared_cpu_map);
    >>>
    >>> Ingo
    >>>

    >>
    >> The main goal was to avoid allocating 4096 bytes when only 32 would do
    >> (characters needed to represent nr_cpu_ids cpus instead of NR_CPUS cpus.)
    >> But I'll look at cleaning it up a bit more. It wouldn't have to be
    >> a function if CHUNKSZ in cpumask_scnprintf() were visible (or a
    >> non-changeable
    >> constant.)
    >>

    >
    > It's a pity you can't take advantage of kasprintf to handle all this.
    >
    > Hm, I would say that bitmap_scnprintf is a candidate for implementation
    > as a printk format specifier so you could get away from needing a
    > special function to print bitmaps...


    Hmm, I hadn't thought of that. There is commonly a format spec called
    %b for diags, etc. to print bit strings. Maybe something like:

    "... %*b ...", nr_cpu_ids, ptr_to_bitmap

    where the length arg is rounded up to 32 or 64 bits...?

    >
    > Eh? What's the difference between snprintf and scnprintf?


    Good question... I'll have to ask the cpumask person. ;-)
    >
    > J


    Thanks!
    Mike
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  19. Re: [PATCH 06/10] x86: reduce memory and stack usage in intel_cacheinfo

    Mike Travis wrote:
    > Hmm, I hadn't thought of that. There is commonly a format spec called
    > %b for diags, etc. to print bit strings. Maybe something like:
    >
    > "... %*b ...", nr_cpu_ids, ptr_to_bitmap
    >
    > where the length arg is rounded up to 32 or 64 bits...?
    >


    I think that would need to be %.*b, but I always need to try it both
    ways anyway...

    But yes, that seems like the right way to go.

    >> Eh? What's the difference between snprintf and scnprintf?
    >>

    >
    > Good question... I'll have to ask the cpumask person. ;-)
    >


    It's in generic lib/vsprintf.c. The two functions are pretty much
    identical... Oh, I see; snprintf returns the total output size,
    regardless of whether it fits into the provided buffer, but scnprintf
    returns the actual output size, clipped by the buffer length.

    J
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  20. Re: [PATCH 02/10] init: move setup of nr_cpu_ids to as early as possible v2


    * Mike Travis wrote:

    > Ingo Molnar wrote:
    > > * Mike Travis wrote:
    > >
    > >> Move the setting of nr_cpu_ids from sched_init() to
    > >> setup_per_cpu_areas(), so that it's available as early as possible.

    > >
    > > hm, why not a separate call before setup_per_cpu_areas(), so that we can
    > > avoid spreading this from generic kernel into a bunch of architectures
    > > that happen to have their own version of setup_per_cpu_areas():
    > >
    > >> 7 files changed, 43 insertions(+), 15 deletions(-)

    > >
    > > Ingo

    >
    > I had this before but I then discovered that an arch would increase
    > (and possible decrease) it's number of possible cpus in
    > setup_per_cpu_areas(). So I figured that setting nr_cpu_ids (and the
    > cpumask_of_cpu map) should be a side effect of setup_per_cpu_areas().


    well, then why not do it shortly after setup_per_cpu_areas()? That still
    moves it earlier than sched_init() but doesnt export all this code and
    complexity toevery setup_per_cpu_areas() implementation. (which clearly
    didnt need this complexity before)

    Ingo
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

+ Reply to Thread
Page 1 of 2 1 2 LastLast