[RFC PATCH 0/1]: timers: Timer Migration - Kernel

This is a discussion on [RFC PATCH 0/1]: timers: Timer Migration - Kernel ; Hi, An idle cpu on which device drivers have initialized timers, has to be frequently woken up to service the timers. So, consolidation of timers onto a fewer number of cpus is important. Migration of timers from idle cpus onto ...

+ Reply to Thread
Results 1 to 7 of 7

Thread: [RFC PATCH 0/1]: timers: Timer Migration

  1. [RFC PATCH 0/1]: timers: Timer Migration

    Hi,

    An idle cpu on which device drivers have initialized timers, has to be frequently woken up to service the timers. So, consolidation of timers onto a fewer number of cpus is important. Migration of
    timers from idle cpus onto lesser idle cpus is necessary. Currently, timers are migrated during the cpu offline operation. However cpu-hotplug for the sake of idle system power management is too
    heavy. So, this patch implements a lightweight timer migration framework.

    Also, in a multi-core, multi-package system, it is always desirable to have all the timers firing on the cpus present in the same package. This would enable us to place the idle package in deep sleep
    state. So, migration of timers is required for this.

    A per_cpu sysfs hook is created at /sys/devices/system/cpu/cpuX/timer_migration. By echo-ing 1 to cpuX/timer_migration, all regular and hrtimers from cpuX are migrated to cpu0. The fact that all
    timers are migrated from cpuX to cpu0 is demonstrated by making a note of /proc/interrupts.

    $:echo 1 > /sys/devices/system/cpu/cpu3/timer_migration
    $:cat /proc/interrupts > file1; sleep 30 ; cat /proc/interrupts > file2
    $:diff file1 file2 | grep LOC
    < LOC: 5276 2100 2676 1647 2415 1803 2275 2977 Local timer interrupts
    > LOC: 6991 2195 2774 1647 2562 1900 2497 3075 Local timer interrupts


    As we can observe, first we are enabling timer migration in cpu3. As a result, the number of local timer interrupts on cpu3 over the next 30 seconds are zero. cpu3 has been totally rid of all regular
    and hrtimers.

    Please refer to the following paper for details:

    http://ols.fedoraproject.org/OLS/Rep...n1-reprint.pdf


    thanks,
    Arun
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  2. Re: [RFC PATCH 0/1]: timers: Timer Migration

    On Tue, 2008-09-16 at 14:42 +0530, Arun R Bharadwaj wrote:
    > Hi,


    80 char lines please.

    > An idle cpu on which device drivers have initialized timers, has to be
    > frequently woken up to service the timers. So, consolidation of timers
    > onto a fewer number of cpus is important. Migration of
    > timers from idle cpus onto lesser idle cpus is necessary. Currently,
    > timers are migrated during the cpu offline operation. However
    > cpu-hotplug for the sake of idle system power management is too
    > heavy. So, this patch implements a lightweight timer migration
    > framework.
    >
    > Also, in a multi-core, multi-package system, it is always desirable to
    > have all the timers firing on the cpus present in the same package.
    > This would enable us to place the idle package in deep sleep
    > state. So, migration of timers is required for this.
    >
    > A per_cpu sysfs hook is created
    > at /sys/devices/system/cpu/cpuX/timer_migration. By echo-ing 1 to
    > cpuX/timer_migration, all regular and hrtimers from cpuX are migrated
    > to cpu0. The fact that all
    > timers are migrated from cpuX to cpu0 is demonstrated by making a note
    > of /proc/interrupts.


    This seems like an exceedingly dumb idea - cpu 0 might be a long long
    way from x and by pushing all timers to cpu0 you might actually overload
    cpu 0.

    Initially you talked about packages an moving timers to busy cpus -
    which is something else altogether.

    > $:echo 1 > /sys/devices/system/cpu/cpu3/timer_migration
    > $:cat /proc/interrupts > file1; sleep 30 ; cat /proc/interrupts >
    > file2
    > $:diff file1 file2 | grep LOC
    > < LOC: 5276 2100 2676 1647 2415
    > 1803 2275 2977 Local timer interrupts
    > > LOC: 6991 2195 2774 1647 2562

    > 1900 2497 3075 Local timer interrupts
    >
    > As we can observe, first we are enabling timer migration in cpu3. As a
    > result, the number of local timer interrupts on cpu3 over the next 30
    > seconds are zero. cpu3 has been totally rid of all regular
    > and hrtimers.
    >
    > Please refer to the following paper for details:
    >
    > http://ols.fedoraproject.org/OLS/Rep...n1-reprint.pdf


    If there's anything worth reading in there it should have been mentioned
    in this email and preferably in some comment in the code and Kconfig
    help as well.

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  3. Re: [RFC PATCH 0/1]: timers: Timer Migration

    On Tue, Sep 16, 2008 at 02:42:31PM +0530, Arun R Bharadwaj wrote:
    > Hi,
    >
    > An idle cpu on which device drivers have initialized timers, has to be frequently woken up to service the timers. So, consolidation of timers onto a fewer number of cpus is important. Migration of
    > timers from idle cpus onto lesser idle cpus is necessary. Currently, timers are migrated during the cpu offline operation. However cpu-hotplug for the sake of idle system power management is too
    > heavy. So, this patch implements a lightweight timer migration framework.



    Who would trigger such migrations?

    I'm not sure it's a good idea to make it fully manually triggered. It would be nicer
    if it was bound to the scheduler's power saving heuristic. So if the scheduler consolidates
    threads on a single package for power saving the timers should follow too.

    >
    > Also, in a multi-core, multi-package system, it is always desirable to have all the timers firing on the cpus present in the same package. This would enable us to place the idle package in deep sleep
    > state. So, migration of timers is required for this.
    >
    > A per_cpu sysfs hook is created at /sys/devices/system/cpu/cpuX/timer_migration. By echo-ing 1 to cpuX/timer_migration, all regular and hrtimers from cpuX are migrated to cpu0. The fact that all
    > timers are migrated from cpuX to cpu0 is demonstrated by making a note of /proc/interrupts.



    I suspect moving all to cpu 0 is too inflexible as Peter already mentioned.

    (haven't looked at the actual code yet sorry)

    -Andi
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  4. Re: [RFC PATCH 0/1]: timers: Timer Migration

    On Tue, 16 Sep 2008 14:42:31 +0530
    Arun R Bharadwaj wrote:

    > Hi,
    >
    > An idle cpu on which device drivers have initialized timers, has to
    > be frequently woken up to service the timers. So, consolidation of
    > timers onto a fewer number of cpus is important. Migration of timers
    > from idle cpus onto lesser idle cpus is necessary. Currently, timers
    > are migrated during the cpu offline operation. However cpu-hotplug
    > for the sake of idle system power management is too heavy. So, this
    > patch implements a lightweight timer migration framework.


    while I absolutely like reducing power consumption... I'm not sure this
    helps or is the right approach.

    First of all, it's of course absolutely better to fix timers and apps
    that cause them (and with PowerTOP we fixed basically all the bad
    stuff).

    Second, in terms of power; a wake up is a wake up, it doesn't really
    matter where it happens.

    Now we could do some consolidation (which realistically needs the range
    timer feature that's aimed for 2.6.28), but I would much rather do that
    in a different way: rather than actively moving stuff, I would instead
    suggest sharing the timer queues between logical cpus that share the
    same cache. (Now its an admin choice if he wants this on a "shared L1",
    "shared L2" or "shared L3" basis).

    Or if you want to forcefully migrate timers, don't move existing ones,
    just on queue/requeue put the on not-the-local-cpu. Sure they'll fire
    once on the "wrong" cpu, but that's a very short term problem!

    --
    Arjan van de Ven Intel Open Source Technology Centre
    For development, discussion and tips for power savings,
    visit http://www.lesswatts.org
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  5. Re: [RFC PATCH 0/1]: timers: Timer Migration

    Peter Zijlstra wrote:
    > On Tue, 2008-09-16 at 14:42 +0530, Arun R Bharadwaj wrote:
    >
    >> Hi,
    >>

    >
    > 80 char lines please.
    >
    >
    >> An idle cpu on which device drivers have initialized timers, has to be
    >> frequently woken up to service the timers. So, consolidation of timers
    >> onto a fewer number of cpus is important. Migration of
    >> timers from idle cpus onto lesser idle cpus is necessary. Currently,
    >> timers are migrated during the cpu offline operation. However
    >> cpu-hotplug for the sake of idle system power management is too
    >> heavy. So, this patch implements a lightweight timer migration
    >> framework.
    >>
    >> Also, in a multi-core, multi-package system, it is always desirable to
    >> have all the timers firing on the cpus present in the same package.
    >> This would enable us to place the idle package in deep sleep
    >> state. So, migration of timers is required for this.
    >>
    >> A per_cpu sysfs hook is created
    >> at /sys/devices/system/cpu/cpuX/timer_migration. By echo-ing 1 to
    >> cpuX/timer_migration, all regular and hrtimers from cpuX are migrated
    >> to cpu0. The fact that all
    >> timers are migrated from cpuX to cpu0 is demonstrated by making a note
    >> of /proc/interrupts.
    >>

    >
    > This seems like an exceedingly dumb idea - cpu 0 might be a long long
    > way from x and by pushing all timers to cpu0 you might actually overload
    > cpu 0.
    >
    > Initially you talked about packages an moving timers to busy cpus -
    > which is something else altogether.
    >
    >


    Here, I've chosen cpu0 just to demonstrate migration of timers. I'm
    working on the algorithm to choose the best cpu or the best set of cpus
    to migrate the timers to. This is just the first iteration on which i
    intend to build upon. What could be a possible way is that, a package
    can be chosen as the destination for the migrated timers and any one or
    many cpus in that package can be the target(s). So, this will prevent
    the possible overloading of any particular cpu.

    >> $:echo 1 > /sys/devices/system/cpu/cpu3/timer_migration
    >> $:cat /proc/interrupts > file1; sleep 30 ; cat /proc/interrupts >
    >> file2
    >> $:diff file1 file2 | grep LOC
    >> < LOC: 5276 2100 2676 1647 2415
    >> 1803 2275 2977 Local timer interrupts
    >>
    >>> LOC: 6991 2195 2774 1647 2562
    >>>

    >> 1900 2497 3075 Local timer interrupts
    >>
    >> As we can observe, first we are enabling timer migration in cpu3. As a
    >> result, the number of local timer interrupts on cpu3 over the next 30
    >> seconds are zero. cpu3 has been totally rid of all regular
    >> and hrtimers.
    >>
    >> Please refer to the following paper for details:
    >>
    >> http://ols.fedoraproject.org/OLS/Rep...n1-reprint.pdf
    >>

    >
    > If there's anything worth reading in there it should have been mentioned
    > in this email and preferably in some comment in the code and Kconfig
    > help as well.
    >
    >

    Thanks for the hint. I'll do that next time.

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  6. Re: [RFC PATCH 0/1]: timers: Timer Migration

    Arjan van de Ven wrote:
    > On Tue, 16 Sep 2008 14:42:31 +0530
    > Arun R Bharadwaj wrote:
    >
    >
    >> Hi,
    >>
    >> An idle cpu on which device drivers have initialized timers, has to
    >> be frequently woken up to service the timers. So, consolidation of
    >> timers onto a fewer number of cpus is important. Migration of timers
    >> from idle cpus onto lesser idle cpus is necessary. Currently, timers
    >> are migrated during the cpu offline operation. However cpu-hotplug
    >> for the sake of idle system power management is too heavy. So, this
    >> patch implements a lightweight timer migration framework.
    >>

    >
    > while I absolutely like reducing power consumption... I'm not sure this
    > helps or is the right approach.
    >
    > First of all, it's of course absolutely better to fix timers and apps
    > that cause them (and with PowerTOP we fixed basically all the bad
    > stuff).
    >
    > Second, in terms of power; a wake up is a wake up, it doesn't really
    > matter where it happens.
    >



    This is a very valid point. The algorithm which decides which cpu to
    migrate the timers to should be intelligent enough in the sense that it
    migrates *only* when there is some benefit out of it. That is, the cpu
    to which the timers are being migrated, is sufficiently busy and it is
    not unnecessarily brought out of its idle state. Else, as you pointed
    out, the penalty of wake up is same for either of the cpu.


    > Now we could do some consolidation (which realistically needs the range
    > timer feature that's aimed for 2.6.28), but I would much rather do that
    > in a different way: rather than actively moving stuff, I would instead
    > suggest sharing the timer queues between logical cpus that share the
    > same cache. (Now its an admin choice if he wants this on a "shared L1",
    > "shared L2" or "shared L3" basis).
    >



    While migrating range timers, I would migrate only if the time range of
    the source and destination cpus match. Else, there is no point in
    migrating. I would rather look at timer migration as an optimization
    strategy to increase the cpu idle time and not as a pro-active method to
    rid the cpu of timers, no matter what. I would like to migrate timers
    only if the wake up penalty of the system as a whole decreases.


    > Or if you want to forcefully migrate timers, don't move existing ones,
    > just on queue/requeue put the on not-the-local-cpu. Sure they'll fire
    > once on the "wrong" cpu, but that's a very short term problem!
    >
    >


    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  7. Re: [RFC PATCH 0/1]: timers: Timer Migration

    Peter Zijlstra chello.nl> writes:

    > This seems like an exceedingly dumb idea - cpu 0 might be a long long
    > way from x and by pushing all timers to cpu0 you might actually overload
    > cpu 0.


    You have to give credit to Solaris for doing that first . They implemented
    usleep() using nanosleep() which uses the kernel callout queue. That queue is
    single threaded and only processed on CPU0 in the clock interrupt handler,
    causing CPU0 to spin 100% if enough calls to usleep() are made across the system.

    http://bugs.opensolaris.org/view_bug.do?bug_id=6565503

    -Andrew

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

+ Reply to Thread