* Yinghai Lu wrote:

> calling pcie_portdrv_init+0x0/0x77
> BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
> IP: [] hrtick_start_fair+0x8e/0x178
> PGD 0
> Oops: 0000 [1] SMP
> CPU 4
> Modules linked in:
> Pid: 103, comm: events/4 Not tainted 2.6.26-rc9-tip-01834-g99e624f-dirty #354
> RIP: 0010:[] []
> hrtick_start_fair+0x8e/0x178
> RSP: 0018:ffff88082486dbd0 EFLAGS: 00010046
> RAX: 0000000000000000 RBX: ffff880824870000 RCX: 0000000000000006
> RDX: ffff88084423e700 RSI: ffff880824870000 RDI: ffff88084423e700
> RBP: ffff88082486dc00 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff88084423e700
> R13: ffff880844239700 R14: ffff880824870000 R15: 0000000000000c18
> FS: 0000000000000000(0000) GS:ffff881024c3a600(0000) knlGS:0000000000000000
> CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> CR2: 0000000000000000 CR3: 0000000000201000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process events/4 (pid: 103, threadinfo ffff88082486c000, task ffff880824864230)
> Stack: ffff880844248118 0000000060ce4c84 ffff880844239700 ffff880824955878
> ffff88084423e700 0000000000000000 ffff88082486dc40 ffffffff80258339
> ffff88082486dc30 ffffffff805196dd 0000000060ce4c84 ffff880824955840
> Call Trace:
> [] dequeue_task_fair+0x64/0x83
> [] ? __first_cpu+0x26/0x49
> [] dequeue_task+0xd3/0xf5
> [] deactivate_task+0x36/0x55
> [] pull_task+0x31/0x7d
> [] load_balance_fair+0x192/0x27c
> [] schedule+0x3e0/0x96d
> [] ? vmstat_update+0x0/0x5e
> [] ? schedule_delayed_work+0x31/0x48
> [] worker_thread+0xbb/0x114
> [] ? autoremove_wake_function+0x0/0x63
> [] ? worker_thread+0x0/0x114
> [] kthread+0x61/0xa4
> [] ? schedule_tail+0x3b/0x8a
> [] child_rip+0xa/0x11
> [] ? kthread+0x0/0xa4

i've triggered a similar crash yesterday - see it below.

Ingo

[ 9.230000] initcall ahci_init+0x0/0x47 returned 0 after 0 msecs
[ 9.240000] calling piix_init+0x0/0x55
[ 9.240000] ata_piix 0000:00:1f.1: version 2.12
[ 9.240000] IOAPIC[0]: Set routing entry (2-18 -> 0x59 -> IRQ 18 Mode:1 Active:1)
[ 9.250000] ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 18
[ 9.260000] PCI: Setting latency timer of device 0000:00:1f.1 to 64
[ 9.270000] BUG: unable to handle kernel paging request at ffff88080101c1f8
[ 9.270000] IP: [] dequeue_task+0x73/0xf0
[ 9.270000] PGD 202063 PUD 0
[ 9.270000] Thread overran stack, or stack corrupted
[ 9.270000] Oops: 0000 [1] SMP
[ 9.270000] CPU 0
[ 9.270000] Modules linked in:
[ 9.270000] Pid: 1, comm: swapper Tainted: G M 2.6.26-rc9-01344-gfa93e3e-dirty #13956
[ 9.270000] RIP: 0010:[] [] dequeue_task+0x73/0xf0
[ 9.270000] RSP: 0018:ffff88003f9b7420 EFLAGS: 00010046
[ 9.270000] RAX: ffff88000101c200 RBX: ffff88003f9b4000 RCX: ffffffff80db5d40
[ 9.270000] RDX: 00000000ffffffff RSI: ffff88003f9b6000 RDI: ffff880001026d40
[ 9.270000] RBP: ffff88003f9b7440 R08: 0000000000000000 R09: 0000000000000001
[ 9.270000] R10: 00000000ffff8e6e R11: 0000000000000000 R12: 0000000000000000
[ 9.270000] R13: ffff880001026d40 R14: 0000000000000002 R15: ffff88003f9b76c0
[ 9.270000] FS: 0000000000000000(0000) GS:ffffffff80c56640(0000) knlGS:0000000000000000
[ 9.270000] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[ 9.270000] CR2: ffff88080101c1f8 CR3: 0000000000201000 CR4: 00000000000006e0
[ 9.270000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 9.270000] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 9.270000] Process swapper (pid: 1, threadinfo ffff88003f9b6000, task ffff88003f9b4000)
[ 9.270000] Stack: 0000000076a041aa 0000000000000046 0000000076a041aa ffff880001026d40
[ 9.270000] ffff88003f9b7470 ffffffff80230674 0000000000000000 ffffffff8088c020
[ 9.270000] 0000000076a041aa 7fffffffffffffff ffff88003f9b7570 ffffffff80889898
[ 9.270000] Call Trace:
[ 9.270000] [] deactivate_task+0x31/0x50
[ 9.270000] [] ? _spin_lock+0xd/0x4c
[ 9.270000] [] schedule+0x20a/0x6af
[ 9.270000] [] ? __enqueue_entity+0x9d/0xb3
[ 9.270000] [] ? enqueue_entity+0x1c5/0x1e6
[ 9.270000] [] schedule_timeout+0x36/0xf7
[ 9.270000] [] wait_for_common+0xee/0x17a
[ 9.270000] [] ? default_wake_function+0x0/0x36
[ 9.270000] [] wait_for_completion+0x2b/0x41
[ 9.270000] [] kthread_create+0xc7/0x14d
[ 9.270000] [] ? scsi_error_handler+0x0/0x552
[ 9.270000] [] ? snprintf+0x70/0x89
[ 9.270000] [] ? kref_init+0xd/0x3b
[ 9.270000] [] ? kobject_init+0x5b/0xb1
[ 9.270000] [] ? klist_init+0xd/0x4c
[ 9.270000] [] ? device_initialize+0x57/0xf5
[ 9.270000] [] ? __mutex_init+0xd/0x4b
[ 9.270000] [] scsi_host_alloc+0x2e6/0x343
[ 9.270000] [] ata_scsi_add_hosts+0x4e/0x130
[ 9.270000] [] ata_host_register+0xb0/0x296
[ 9.270000] [] ? devres_open_group+0xb1/0xd5
[ 9.270000] [] ata_pci_sff_activate_host+0x1af/0x1f0
[ 9.270000] [] ? ata_sff_interrupt+0x0/0x26a
[ 9.270000] [] piix_init_one+0x69e/0x6cb
[ 9.270000] [] pci_call_probe+0x92/0xdf
[ 9.270000] [] pci_device_probe+0x69/0xa9
[ 9.270000] [] driver_probe_device+0xb3/0x178
[ 9.270000] [] ? down+0x4b/0x68
[ 9.270000] [] __driver_attach+0x63/0xa4
[ 9.270000] [] ? __driver_attach+0x0/0xa4
[ 9.270000] [] bus_for_each_dev+0x63/0xb1
[ 9.270000] [] driver_attach+0x34/0x4a
[ 9.270000] [] bus_add_driver+0xcc/0x22c
[ 9.270000] [] ? kset_find_obj+0x4c/0x97
[ 9.270000] [] driver_register+0xbc/0x153
[ 9.270000] [] __pci_register_driver+0x67/0xb6
[ 9.270000] [] ? piix_init+0x0/0x55
[ 9.270000] [] piix_init+0x31/0x55
[ 9.270000] [] kernel_init+0x1c6/0x333
[ 9.270000] [] ? finish_task_switch+0x41/0xe0
[ 9.270000] [] child_rip+0xa/0x11
[ 9.270000] [] ? restore_args+0x0/0x30
[ 9.270000] [] ? kernel_init+0x0/0x333
[ 9.270000] [] ? child_rip+0x0/0x11
[ 9.270000]
[ 9.270000]
[ 9.270000] Code: 29 c8 48 29 d0 48 c1 f8 03 48 01 d0 48 89 86 a0 00 00 00 48 8b 73 08 48 8b 05 9f 66 a2 00 48 c7 c1 40 5d db 80 45 31 c0 8b 56 1c <48> 8b 04 d0 48 8b 93 10 04 00 00 48 8b 40 08 48 8b 84 01 10 08
[ 9.270000] RIP [] dequeue_task+0x73/0xf0
[ 9.270000] RSP
[ 9.270000] CR2: ffff88080101c1f8
[ 9.270000] Kernel panic - not syncing: Fatal exception
Press any key to enter the menu
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Sun, Jul 13, 2008 at 11:26 AM, Ingo Molnar wrote:
> [ 9.270000] BUG: unable to handle kernel paging request at ffff88080101c1f8
> [ 9.270000] IP: [] dequeue_task+0x73/0xf0

[...]

> [ 9.270000] RIP: 0010:[] [] dequeue_task+0x73/0xf0
> [ 9.270000] RSP: 0018:ffff88003f9b7420 EFLAGS: 00010046
> [ 9.270000] RAX: ffff88000101c200 RBX: ffff88003f9b4000 RCX: ffffffff80db5d40
> [ 9.270000] RDX: 00000000ffffffff RSI: ffff88003f9b6000 RDI: ffff880001026d40

I don't think this is per-cpu related.

It seems that task_thread_info(p)->cpu returns -1. This is the number
that is loaded into %rdx:

mov 0x1c(%rsi),%edx
mov (%rax,%rdx,8),%rax <-- faulting instruction

You also had this:

[ 9.270000] Thread overran stack, or stack corrupted

so it seems likely that your thread_info was corrupted as well.


Vegard

--
"The animistic metaphor of the bug that maliciously sneaked in while
the programmer was not looking is intellectually dishonest as it
disguises that the error is the programmer's own creation."
-- E. W. Dijkstra, EWD1036
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Hi,

On Sun, Jul 13, 2008 at 10:16 AM, Yinghai Lu wrote:
> calling pcie_portdrv_init+0x0/0x77
> BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
> IP: [] hrtick_start_fair+0x8e/0x178
> PGD 0
> Oops: 0000 [1] SMP
> CPU 4
> Modules linked in:
> Pid: 103, comm: events/4 Not tainted 2.6.26-rc9-tip-01834-g99e624f-dirty #354
> RIP: 0010:[] []
> hrtick_start_fair+0x8e/0x178

I find that this corresponds to timer->base being NULL:

static inline int hrtimer_is_hres_active(struct hrtimer *timer)
{
return timer->base->cpu_base->hres_active;
}

Apparently timer->base can legitimately be NULL in some cases:

/*
* [...]
* When the timer's base is locked, and the timer removed from list, it is
* possible to set timer->base = NULL and drop the lock: the timer remains
* locked.
*/

I don't know any more than this yet, I hope I didn't waste your time.


Vegard

--
"The animistic metaphor of the bug that maliciously sneaked in while
the programmer was not looking is intellectually dishonest as it
disguises that the error is the programmer's own creation."
-- E. W. Dijkstra, EWD1036
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Sun, Jul 13, 2008 at 4:55 AM, Vegard Nossum wrote:
> Hi,
>
> On Sun, Jul 13, 2008 at 10:16 AM, Yinghai Lu wrote:
>> calling pcie_portdrv_init+0x0/0x77
>> BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
>> IP: [] hrtick_start_fair+0x8e/0x178
>> PGD 0
>> Oops: 0000 [1] SMP
>> CPU 4
>> Modules linked in:
>> Pid: 103, comm: events/4 Not tainted 2.6.26-rc9-tip-01834-g99e624f-dirty #354
>> RIP: 0010:[] []
>> hrtick_start_fair+0x8e/0x178
>
> I find that this corresponds to timer->base being NULL:
>
> static inline int hrtimer_is_hres_active(struct hrtimer *timer)
> {
> return timer->base->cpu_base->hres_active;
> }
>
> Apparently timer->base can legitimately be NULL in some cases:

thanks

could be caused by max_low_pfn_mapped patch...because hpet still use
fixmap address that is cleared by others.

will check that.

YH
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

* Yinghai Lu wrote:

> thanks
>
> could be caused by max_low_pfn_mapped patch...because hpet still use
> fixmap address that is cleared by others.
>
> will check that.

got another weird crash:

[ 12.431572] device: 'sda1': device_add
[ 12.435498] PM: Adding info for No Bus:sda1
[ 12.439795] device: 'sda2': device_add
[ 12.443606] PM: Adding info for No Bus:sda2
[ 12.447945] ------------[ cut here ]------------
[ 12.452585] WARNING: at kernel/mutex.c:134 mutex_lock_nested+0x327/0x350()
[ 12.459481] Modules linked in:
[ 12.462597] Pid: 1, comm: swapper Not tainted 2.6.26-rc9-tip #14453
[ 12.468884]
[ 12.468885] Call Trace:
[ 12.472882] [] warn_on_slowpath+0x73/0xc0
[ 12.478565] [] ? trace_hardirqs_on+0x20/0x40
[ 12.484509] [] ? debug_locks_off+0x8/0x70
[ 12.490190] [] ? ftrace_record_ip+0x193/0x2b0
[ 12.496220] [] ? trace_hardirqs_on+0x20/0x40
[ 12.502159] [] ? mcount_call+0x5/0x31
[ 12.507494] [] ? scsi_disk_put+0x3d/0x80
[ 12.513089] [] mutex_lock_nested+0x327/0x350
[ 12.519030] [] ? scsi_disk_put+0x3d/0x80
[ 12.524624] [] scsi_disk_put+0x3d/0x80
[ 12.530044] [] sd_release+0x5b/0xa0
[ 12.535207] [] __blkdev_put+0x1d8/0x200
[ 12.540713] [] ? mcount_call+0x5/0x31
[ 12.546047] [] blkdev_put+0x23/0x40
[ 12.551209] [] register_disk+0x1a5/0x1c0
[ 12.556806] [] add_disk+0x5a/0xc0
[ 12.561794] [] sd_probe+0x2e4/0x440
[ 12.566955] [] driver_probe_device+0x104/0x260
[ 12.573068] [] ? __device_attach+0x0/0x40
[ 12.578750] [] __device_attach+0x21/0x40
[ 12.584347] [] bus_for_each_drv+0x7b/0xc0
[ 12.590027] [] device_attach+0xa9/0xc0
[ 12.595450] [] bus_attach_device+0x75/0xa0
[ 12.601216] [] device_add+0x599/0x670
[ 12.606550] [] ? mcount_call+0x5/0x31
[ 12.611884] [] scsi_sysfs_add_sdev+0x75/0x2a0
[ 12.617914] [] scsi_probe_and_add_lun+0xbbb/0xc70
[ 12.624289] [] __scsi_add_device+0x13a/0x150
[ 12.630229] [] ata_scsi_scan_host+0xeb/0x330
[ 12.636171] [] ata_host_register+0x286/0x310
[ 12.642111] [] ? devres_open_group+0xc0/0xf0
[ 12.648053] [] ? ata_sff_interrupt+0x0/0x2c0
[ 12.653995] [] ata_pci_sff_activate_host+0x12f/0x240
[ 12.660628] [] piix_init_one+0x334/0x7f0
[ 12.666225] [] pci_call_probe+0xa3/0x100
[ 12.671817] [] pci_device_probe+0x99/0xd0
[ 12.677499] [] driver_probe_device+0x104/0x260
[ 12.683615] [] __driver_attach+0xab/0xc0
[ 12.689208] [] ? __driver_attach+0x0/0xc0
[ 12.694891] [] bus_for_each_dev+0x7b/0xc0
[ 12.700570] [] driver_attach+0x34/0x50
[ 12.705991] [] bus_add_driver+0x116/0x2a0
[ 12.711670] [] driver_register+0x87/0x1a0
[ 12.717353] [] ? __spin_lock_init+0x47/0x90
[ 12.723206] [] ? piix_init+0x0/0x60
[ 12.728368] [] __pci_register_driver+0x7e/0xe0
[ 12.734483] [] piix_init+0x31/0x60
[ 12.739558] [] kernel_init+0x26f/0x39c
[ 12.744981] [] ? _spin_unlock_irq+0x3f/0x70
[ 12.750834] [] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 12.757297] [] ? trace_hardirqs_on_caller+0x169/0x1d0
[ 12.764019] [] child_rip+0xa/0x11
[ 12.769006] [] ? restore_args+0x0/0x30
[ 12.774429] [] ? kernel_init+0x0/0x39c
[ 12.779850] [] ? child_rip+0x0/0x11
[ 12.785012]
[ 12.786538] ---[ end trace 0361c3a3643346eb ]---
[ 12.791214] BUG: scheduling while atomic: swapper/1/0x92899fb1
[ 12.797072] INFO: lockdep is turned off.
[ 12.801020] Modules linked in:

task structure got corrupted here too - the preempt count is 0x92899fb1.

Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

* Yinghai Lu wrote:

> On Sun, Jul 13, 2008 at 11:55 AM, Ingo Molnar wrote:
> >
> > * Yinghai Lu wrote:
> >
> >> thanks
> >>
> >> could be caused by max_low_pfn_mapped patch...because hpet still use
> >> fixmap address that is cleared by others.
>
> still got
>
> calling __stack_chk_test+0x0/0x89
> Testing -fstack-protector-all feature
> -fstack-protector-all test failed
> ------------[ cut here ]------------
> WARNING: at kernel/panic.c:393 __stack_chk_test+0x61/0x89()
> Modules linked in:
> Pid: 1, comm: swapper Not tainted 2.6.26-rc9-tip-01873-ga9827e7-dirty #359
>
> Call Trace:
> [] warn_on_slowpath+0x6c/0xa7
> [] ? __stack_chk_test+0x0/0x89
> [] ? __stack_chk_test+0x0/0x89
> [] ? mcount_call+0x5/0x31
> [] ? __stack_chk_test+0x0/0x89
> [] __stack_chk_test+0x61/0x89
> [] kernel_init+0x1de/0x346
> [] ? mcount_call+0x5/0x31
> [] ? finish_task_switch+0x14/0xe3
> [] child_rip+0xa/0x11
> [] ? kernel_init+0x0/0x346
> [] ? child_rip+0x0/0x11
>
> ---[ end trace 0f276ea63a4e83de ]---

please try latest tip/master - the self-test had to be removed. (gcc
will treat the stackprotector-failure function as ((noreturn)) attribute
and will optimize out the return path, making a reliable self-test
impossible in the way we tried to)

Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Sun, Jul 13, 2008 at 11:55 AM, Ingo Molnar wrote:
>
> * Yinghai Lu wrote:
>
>> thanks
>>
>> could be caused by max_low_pfn_mapped patch...because hpet still use
>> fixmap address that is cleared by others.

still got

calling __stack_chk_test+0x0/0x89
Testing -fstack-protector-all feature
-fstack-protector-all test failed
------------[ cut here ]------------
WARNING: at kernel/panic.c:393 __stack_chk_test+0x61/0x89()
Modules linked in:
Pid: 1, comm: swapper Not tainted 2.6.26-rc9-tip-01873-ga9827e7-dirty #359

Call Trace:
[] warn_on_slowpath+0x6c/0xa7
[] ? __stack_chk_test+0x0/0x89
[] ? __stack_chk_test+0x0/0x89
[] ? mcount_call+0x5/0x31
[] ? __stack_chk_test+0x0/0x89
[] __stack_chk_test+0x61/0x89
[] kernel_init+0x1de/0x346
[] ? mcount_call+0x5/0x31
[] ? finish_task_switch+0x14/0xe3
[] child_rip+0xa/0x11
[] ? kernel_init+0x0/0x346
[] ? child_rip+0x0/0x11

---[ end trace 0f276ea63a4e83de ]---
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Sun, Jul 13, 2008 at 2:04 PM, Ingo Molnar wrote:
>
> * Yinghai Lu wrote:
>
>> On Sun, Jul 13, 2008 at 11:55 AM, Ingo Molnar wrote:
>> >
>> > * Yinghai Lu wrote:
>> >
>> >> thanks
>> >>
>> >> could be caused by max_low_pfn_mapped patch...because hpet still use
>> >> fixmap address that is cleared by others.
>>
>> still got
>>
>> calling __stack_chk_test+0x0/0x89
>> Testing -fstack-protector-all feature
>> -fstack-protector-all test failed
>> ------------[ cut here ]------------
>> WARNING: at kernel/panic.c:393 __stack_chk_test+0x61/0x89()
>> Modules linked in:
>> Pid: 1, comm: swapper Not tainted 2.6.26-rc9-tip-01873-ga9827e7-dirty #359
>>
>> Call Trace:
>> [] warn_on_slowpath+0x6c/0xa7
>> [] ? __stack_chk_test+0x0/0x89
>> [] ? __stack_chk_test+0x0/0x89
>> [] ? mcount_call+0x5/0x31
>> [] ? __stack_chk_test+0x0/0x89
>> [] __stack_chk_test+0x61/0x89
>> [] kernel_init+0x1de/0x346
>> [] ? mcount_call+0x5/0x31
>> [] ? finish_task_switch+0x14/0xe3
>> [] child_rip+0xa/0x11
>> [] ? kernel_init+0x0/0x346
>> [] ? child_rip+0x0/0x11
>>
>> ---[ end trace 0f276ea63a4e83de ]---
>
> please try latest tip/master - the self-test had to be removed. (gcc
> will treat the stackprotector-failure function as ((noreturn)) attribute
> and will optimize out the return path, making a reliable self-test
> impossible in the way we tried to)
>
ok, you fixed it...
that warning is gone.

YH
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/