VServer schmiert ab / BUG: soft lockup - CPU#1 stuck for 26s!

Hallo,

ich bin vor kurzem von einem dedizierten Rootserver auf einen virtuellen gewechselt, und habe seitdem immer wieder Aussetzer bzw. Abstürze.

Die Virtualisierungstechnik ist KVM.

Im Syslog findet sich folgendes:

Code:

---

Dec  9 12:34:32 server1 kernel: [726172.400038] hrtimer: interrupt took 19235578 ns
Dec  9 12:34:32 server1 kernel: [726148.000014] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 2, t=16983 jiffies)
Dec  9 12:34:32 server1 kernel: [726148.000014] sending NMI to all CPUs:
Dec  9 12:34:32 server1 kernel: [726148.000014] NMI backtrace for cpu 2
Dec  9 12:34:32 server1 kernel: [726148.000014] CPU 2
Dec  9 12:34:32 server1 kernel: [726148.000014] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:34:32 server1 kernel: [726148.000014]
Dec  9 12:34:32 server1 kernel: [726148.000014] Pid: 14, comm: kworker/2:0 Not tainted 3.2.0-4-amd64 #1 Debian 3.2.68-1+deb7u6 Red Hat KVM
Dec  9 12:34:32 server1 kernel: [726148.000014] RIP: 0010:[<ffffffff8102b3e5>]  [<ffffffff8102b3e5>] native_write_msr_safe+0x6/0x9
Dec  9 12:34:32 server1 kernel: [726148.000014] RSP: 0018:ffff88033fd03dc0  EFLAGS: 00000046
Dec  9 12:34:32 server1 kernel: [726148.000014] RAX: 0000000000000400 RBX: 0000000000000400 RCX: 0000000000000830
Dec  9 12:34:32 server1 kernel: [726148.000014] RDX: 0000000000000002 RSI: 0000000000000400 RDI: 0000000000000830
Dec  9 12:34:32 server1 kernel: [726148.000014] RBP: 000000000000cc42 R08: 0000000000000000 R09: ffffffff8168f1e0
Dec  9 12:34:32 server1 kernel: [726148.000014] R10: 0000000000000000 R11: 0000000034a882d1 R12: 0000000000000002
Dec  9 12:34:32 server1 kernel: [726148.000014] R13: 0000000000000400 R14: 0000000000000002 R15: 0000000000080000
Dec  9 12:34:32 server1 kernel: [726148.000014] FS:  0000000000000000(0000) GS:ffff88033fd00000(0000) knlGS:0000000000000000
Dec  9 12:34:32 server1 kernel: [726148.000014] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec  9 12:34:32 server1 kernel: [726148.000014] CR2: 00007f5b7bb88000 CR3: 0000000331946000 CR4: 00000000001406e0
Dec  9 12:34:32 server1 kernel: [726148.000014] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  9 12:34:32 server1 kernel: [726148.000014] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec  9 12:34:32 server1 kernel: [726148.000014] Process kworker/2:0 (pid: 14, threadinfo ffff8803328e4000, task ffff8803328c4740)
Dec  9 12:34:32 server1 kernel: [726148.000014] Stack:
Dec  9 12:34:32 server1 kernel: [726148.000014]  ffffffff81027884 0000000000000096 ffffffff8102795c 0000000000000002
Dec  9 12:34:32 server1 kernel: [726148.000014]  ffffffff8168f1e0 ffff88033fd03e38 0000000000002710 ffffffff81622280
Dec  9 12:34:32 server1 kernel: [726148.000014]  ffffffff81622380 0000000000000001 ffffffff81624380 0000000000000000
Dec  9 12:34:32 server1 kernel: [726148.000014] Call Trace:
Dec  9 12:34:32 server1 kernel: [726148.000014]  <IRQ>
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81027884>] ? native_x2apic_icr_write+0x12/0x16
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8102795c>] ? __x2apic_send_IPI_mask+0x7b/0xb7
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8102514e>] ? arch_trigger_all_cpu_backtrace+0x4d/0x7b
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81096648>] ? __rcu_pending+0x21a/0x358
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81011a6d>] ? arch_local_irq_save+0x5/0x13
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8106c560>] ? tick_nohz_handler+0xd0/0xd0
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81096ab3>] ? rcu_check_callbacks+0x90/0xcc
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81052dde>] ? update_process_times+0x31/0x63
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8106c5ca>] ? tick_sched_timer+0x6a/0x90
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8106275a>] ? __run_hrtimer+0xac/0x135
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81062e44>] ? hrtimer_interrupt+0xd7/0x1b1
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff813584ec>] ? call_softirq+0x1c/0x30
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81024757>] ? smp_apic_timer_interrupt+0x6f/0x82
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81356d5e>] ? apic_timer_interrupt+0x6e/0x80
Dec  9 12:34:32 server1 kernel: [726148.000014]  <EOI>
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff810713ce>] ? arch_local_irq_restore+0x2/0x8
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81351267>] ? _raw_spin_unlock_irqrestore+0xe/0xf
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffffa00c0192>] ? e1000_update_stats+0x926/0x931 [e1000]
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8103665c>] ? should_resched+0x5/0x23
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffffa00c04dc>] ? e1000_watchdog+0x33f/0x4ac [e1000]
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:34:32 server1 kernel: [726148.000014] Code: 89 d0 89 0f 48 c1 e0 20 48 09 f0 c3 89 f9 0f 32 45 31 c0 89 c7 48 89 d0 44 89 06 48 c1 e0 20 89 f9 48 09 c8 c3 89 f0 89 f9 0f 30 <31> c0 c3 89 f9 0f 33 89 c7 48 89 d0 48 c1 e0 20 89 f9 48 09 c8
Dec  9 12:34:32 server1 kernel: [726148.000014] Call Trace:
Dec  9 12:34:32 server1 kernel: [726148.000014]  <IRQ>  [<ffffffff81027884>] ? native_x2apic_icr_write+0x12/0x16
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8102795c>] ? __x2apic_send_IPI_mask+0x7b/0xb7
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8102514e>] ? arch_trigger_all_cpu_backtrace+0x4d/0x7b
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81096648>] ? __rcu_pending+0x21a/0x358
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81011a6d>] ? arch_local_irq_save+0x5/0x13
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8106c560>] ? tick_nohz_handler+0xd0/0xd0
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81096ab3>] ? rcu_check_callbacks+0x90/0xcc
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81052dde>] ? update_process_times+0x31/0x63
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8106c5ca>] ? tick_sched_timer+0x6a/0x90
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8106275a>] ? __run_hrtimer+0xac/0x135
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81062e44>] ? hrtimer_interrupt+0xd7/0x1b1
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff813584ec>] ? call_softirq+0x1c/0x30
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81024757>] ? smp_apic_timer_interrupt+0x6f/0x82
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81356d5e>] ? apic_timer_interrupt+0x6e/0x80
Dec  9 12:34:32 server1 kernel: [726148.000014]  <EOI>  [<ffffffff810713ce>] ? arch_local_irq_restore+0x2/0x8
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81351267>] ? _raw_spin_unlock_irqrestore+0xe/0xf
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffffa00c0192>] ? e1000_update_stats+0x926/0x931 [e1000]
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8103665c>] ? should_resched+0x5/0x23
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffffa00c04dc>] ? e1000_watchdog+0x33f/0x4ac [e1000]
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:34:32 server1 kernel: [726212.901000] NMI backtrace for cpu 0
Dec  9 12:34:32 server1 kernel: [726212.901000] CPU 0
Dec  9 12:34:32 server1 kernel: [726212.901000] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:34:32 server1 kernel: [726212.901000]
Dec  9 12:34:32 server1 kernel: [726212.901000] Pid: 0, comm: swapper/0 Not tainted 3.2.0-4-amd64 #1 Debian 3.2.68-1+deb7u6 Red Hat KVM
Dec  9 12:34:32 server1 kernel: [726212.901000] RIP: 0010:[<ffffffffa00be800>]  [<ffffffffa00be800>] e1000_alloc_rx_buffers+0x27b/0x28a [e1000]
Dec  9 12:34:32 server1 kernel: [726212.901000] RSP: 0018:ffff88033fc03ca0  EFLAGS: 00010286
Dec  9 12:34:32 server1 kernel: [726212.901000] RAX: ffffc900025c2818 RBX: ffff88032fa50740 RCX: 00000000bcc7a000
Dec  9 12:34:32 server1 kernel: [726212.901000] RDX: 00000000000005f2 RSI: 00000000bcc7a000 RDI: ffff88032fa50740
Dec  9 12:34:32 server1 kernel: [726212.901000] RBP: ffffc900025b0ca0 R08: 0000000000000000 R09: 00000000000005f2
Dec  9 12:34:32 server1 kernel: [726212.901000] R10: ffff8801da22b9c0 R11: ffff880332a24090 R12: ffff88015fb692c0
Dec  9 12:34:32 server1 kernel: [726212.901000] R13: 00000000000000ed R14: ffff88032f8a7780 R15: 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.901000] FS:  0000000000000000(0000) GS:ffff88033fc00000(0000) knlGS:0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.901000] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec  9 12:34:32 server1 kernel: [726212.901000] CR2: 00007f21535ba000 CR3: 000000020fd05000 CR4: 00000000001406f0
Dec  9 12:34:32 server1 kernel: [726212.901000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.901000] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec  9 12:34:32 server1 kernel: [726212.901000] Process swapper/0 (pid: 0, threadinfo ffffffff81600000, task ffffffff8160d020)
Dec  9 12:34:32 server1 kernel: [726212.901000] Stack:
Dec  9 12:34:32 server1 kernel: [726212.901000]  ffff88033fc03cd0 00000000bcc7a000 00000000000005f2 ffffffff00000000
Dec  9 12:34:32 server1 kernel: [726212.901000]  000005f229977bc0 ffff880332a24090 ffff88032fa50000 00000000000005f2
Dec  9 12:34:32 server1 kernel: [726212.901000]  ffff88003743fee0 ffff88032fa50740 ffff880329977bc0 ffff88032f8a7780
Dec  9 12:34:32 server1 kernel: [726212.901000] Call Trace:
Dec  9 12:34:32 server1 kernel: [726212.901000]  <IRQ>
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffffa00beb25>] ? e1000_clean_rx_irq+0x316/0x39e [e1000]
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffffa00be433>] ? e1000_clean+0x2a9/0x3fb [e1000]
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8102b2d2>] ? kvm_clock_read+0x17/0x1a
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8103b0c7>] ? check_preempt_curr+0x36/0x5f
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff810713ed>] ? arch_local_irq_save+0x11/0x17
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff81291d23>] ? net_rx_action+0xa1/0x1aa
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff81245fe3>] ? add_interrupt_randomness+0x38/0x155
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8104c48e>] ? __do_softirq+0xb9/0x177
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff813584ec>] ? call_softirq+0x1c/0x30
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8100fa91>] ? do_softirq+0x3c/0x7b
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8104c6f6>] ? irq_exit+0x3c/0x99
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8100f66a>] ? do_IRQ+0x82/0x98
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff813514ee>] ? common_interrupt+0x6e/0x6e
Dec  9 12:34:32 server1 kernel: [726212.901000]  <EOI>
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8102b36c>] ? native_safe_halt+0x2/0x3
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff81014938>] ? default_idle+0x47/0x7f
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8100d24c>] ? cpu_idle+0xaf/0xf2
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff816aab3b>] ? start_kernel+0x3bd/0x3c8
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff816aa140>] ? early_idt_handlers+0x140/0x140
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff816aa3c4>] ? x86_64_start_kernel+0x104/0x111
Dec  9 12:34:32 server1 kernel: [726212.901000] Code: fe ff ff 45 39 6e 18 74 22 45 85 ed 45 89 6e 18 75 04 45 8b 6e 14 41 ff cd 0f ae f8 41 0f b7 46 36 48 03 83 58 03 00 00 44 89 28 <48> 83 c4 48 5b 5d 41 5c 41 5d 41 5e 41 5f c3 41 57 48 8d 87 58
Dec  9 12:34:32 server1 kernel: [726212.901000] Call Trace:
Dec  9 12:34:32 server1 kernel: [726212.901000]  <IRQ>  [<ffffffffa00beb25>] ? e1000_clean_rx_irq+0x316/0x39e [e1000]
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffffa00be433>] ? e1000_clean+0x2a9/0x3fb [e1000]
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8102b2d2>] ? kvm_clock_read+0x17/0x1a
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8103b0c7>] ? check_preempt_curr+0x36/0x5f
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff810713ed>] ? arch_local_irq_save+0x11/0x17
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff81291d23>] ? net_rx_action+0xa1/0x1aa
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff81245fe3>] ? add_interrupt_randomness+0x38/0x155
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8104c48e>] ? __do_softirq+0xb9/0x177
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff813584ec>] ? call_softirq+0x1c/0x30
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8100fa91>] ? do_softirq+0x3c/0x7b
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8104c6f6>] ? irq_exit+0x3c/0x99
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8100f66a>] ? do_IRQ+0x82/0x98
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff813514ee>] ? common_interrupt+0x6e/0x6e
Dec  9 12:34:32 server1 kernel: [726212.901000]  <EOI>  [<ffffffff8102b36c>] ? native_safe_halt+0x2/0x3
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff81014938>] ? default_idle+0x47/0x7f
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8100d24c>] ? cpu_idle+0xaf/0xf2
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff816aab3b>] ? start_kernel+0x3bd/0x3c8
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff816aa140>] ? early_idt_handlers+0x140/0x140
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff816aa3c4>] ? x86_64_start_kernel+0x104/0x111
Dec  9 12:34:32 server1 kernel: [726212.900571] NMI backtrace for cpu 1
Dec  9 12:34:32 server1 kernel: [726212.900571] CPU 1
Dec  9 12:34:32 server1 kernel: [726212.900571] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:34:32 server1 kernel: [726212.900571]
Dec  9 12:34:32 server1 kernel: [726212.900571] Pid: 2471, comm: nginx Not tainted 3.2.0-4-amd64 #1 Debian 3.2.68-1+deb7u6 Red Hat KVM
Dec  9 12:34:32 server1 kernel: [726212.900571] RIP: 0010:[<ffffffffa00be109>]  [<ffffffffa00be109>] e1000_xmit_frame+0xa9a/0xb1b [e1000]
Dec  9 12:34:32 server1 kernel: [726212.900571] RSP: 0018:ffff88032f50b988  EFLAGS: 00010286
Dec  9 12:34:32 server1 kernel: [726212.900571] RAX: ffffc900025c3818 RBX: ffff88003768fc70 RCX: ffff88003743e430
Dec  9 12:34:32 server1 kernel: [726212.900571] RDX: 0000000000000044 RSI: 0000000022100000 RDI: 000000002210001f
Dec  9 12:34:32 server1 kernel: [726212.900571] RBP: ffff88032f8a77c0 R08: 0000000000000000 R09: 0000000025e2adc7
Dec  9 12:34:32 server1 kernel: [726212.900571] R10: 0000000000000000 R11: ffff88032fa50000 R12: ffff88032fa50000
Dec  9 12:34:32 server1 kernel: [726212.900571] R13: ffffc900025aac90 R14: 0000000000000000 R15: 0000000000000043
Dec  9 12:34:32 server1 kernel: [726212.900571] FS:  00007f4adf9f7700(0000) GS:ffff88033fc80000(0000) knlGS:0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.900571] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  9 12:34:32 server1 kernel: [726212.900571] CR2: 00007f2153410000 CR3: 00000002cf5b7000 CR4: 00000000001406e0
Dec  9 12:34:32 server1 kernel: [726212.900571] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.900571] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec  9 12:34:32 server1 kernel: [726212.900571] Process nginx (pid: 2471, threadinfo ffff88032f50a000, task ffff88032f139100)
Dec  9 12:34:32 server1 kernel: [726212.900571] Stack:
Dec  9 12:34:32 server1 kernel: [726212.900571]  ffff880059adeab8 ffff88032fa50000 0000000000000dc7 0000100000000009
Dec  9 12:34:32 server1 kernel: [726212.900571]  ffff88030000001f ffff88032fa50740 0000000000000001 0000000000000001
Dec  9 12:34:32 server1 kernel: [726212.900571]  0000004100000002 ffff880332a24000 ffff880332a24090 0000000000000001
Dec  9 12:34:32 server1 kernel: [726212.900571] Call Trace:
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff81292b85>] ? dev_hard_start_xmit+0x3d2/0x518
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812a920f>] ? sch_direct_xmit+0x61/0x135
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff81292f8d>] ? dev_queue_xmit+0x2c2/0x46b
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff8102bb5c>] ? pvclock_clocksource_read+0x42/0xb2
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812bbca6>] ? ip_finish_output2+0x1bc/0x1f9
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812bd06d>] ? ip_queue_xmit+0x28e/0x2b9
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812ce3a1>] ? tcp_transmit_skb+0x6e6/0x719
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812cee36>] ? tcp_write_xmit+0x6a0/0x7a9
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810eba4e>] ? virt_to_head_page+0x6/0x29
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810ebcd7>] ? virt_to_cache+0x7/0x23
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810ebd09>] ? ksize+0x16/0x1c
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff8134fe9c>] ? _cond_resched+0x7/0x1c
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812cef80>] ? __tcp_push_pending_frames+0x18/0x44
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812c476b>] ? tcp_sendmsg+0x64d/0x72d
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff81280812>] ? sock_aio_write+0x121/0x135
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810fae8c>] ? do_sync_write+0xb4/0xec
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff8102bb5c>] ? pvclock_clocksource_read+0x42/0xb2
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff81165799>] ? security_file_permission+0x16/0x2d
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810fb584>] ? vfs_write+0xa9/0xe9
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810fb75a>] ? sys_write+0x45/0x6b
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff813562b2>] ? system_call_fastpath+0x16/0x1b
Dec  9 12:34:32 server1 kernel: [726212.900571] Code: 41 0f 44 d0 41 ff ca 45 85 d2 75 ca 41 8b 84 24 8c 09 00 00 09 41 08 0f ae f8 0f b7 45 2a 89 55 18 49 03 84 24 98 0a 00 00 89 10 <8b> 45 1c 8b 4d 18 31 d2 39 c8 77 03 8b 55 14 ff c8 45 31 ed 29
Dec  9 12:34:32 server1 kernel: [726212.900571] Call Trace:
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff81292b85>] ? dev_hard_start_xmit+0x3d2/0x518
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812a920f>] ? sch_direct_xmit+0x61/0x135
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff81292f8d>] ? dev_queue_xmit+0x2c2/0x46b
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff8102bb5c>] ? pvclock_clocksource_read+0x42/0xb2
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812bbca6>] ? ip_finish_output2+0x1bc/0x1f9
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812bd06d>] ? ip_queue_xmit+0x28e/0x2b9
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812ce3a1>] ? tcp_transmit_skb+0x6e6/0x719
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812cee36>] ? tcp_write_xmit+0x6a0/0x7a9
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810eba4e>] ? virt_to_head_page+0x6/0x29
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810ebcd7>] ? virt_to_cache+0x7/0x23
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810ebd09>] ? ksize+0x16/0x1c
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff8134fe9c>] ? _cond_resched+0x7/0x1c
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812cef80>] ? __tcp_push_pending_frames+0x18/0x44
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812c476b>] ? tcp_sendmsg+0x64d/0x72d
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff81280812>] ? sock_aio_write+0x121/0x135
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810fae8c>] ? do_sync_write+0xb4/0xec
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff8102bb5c>] ? pvclock_clocksource_read+0x42/0xb2
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff81165799>] ? security_file_permission+0x16/0x2d
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810fb584>] ? vfs_write+0xa9/0xe9
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810fb75a>] ? sys_write+0x45/0x6b
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff813562b2>] ? system_call_fastpath+0x16/0x1b
Dec  9 12:34:32 server1 kernel: [726212.912006] NMI backtrace for cpu 3
Dec  9 12:34:32 server1 kernel: [726212.912006] CPU 3
Dec  9 12:34:32 server1 kernel: [726212.912006] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:34:32 server1 kernel: [726212.912006]
Dec  9 12:34:32 server1 kernel: [726212.912006] Pid: 0, comm: swapper/3 Not tainted 3.2.0-4-amd64 #1 Debian 3.2.68-1+deb7u6 Red Hat KVM
Dec  9 12:34:32 server1 kernel: [726212.912006] RIP: 0010:[<ffffffff8102b36c>]  [<ffffffff8102b36c>] native_safe_halt+0x2/0x3
Dec  9 12:34:32 server1 kernel: [726212.912006] RSP: 0018:ffff88033291fef0  EFLAGS: 00000246
Dec  9 12:34:32 server1 kernel: [726212.912006] RAX: 0000000000000000 RBX: ffff88033291e000 RCX: 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.912006] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff88033291e000
Dec  9 12:34:32 server1 kernel: [726212.912006] RBP: ffffffff8168f1e0 R08: 0000000000000000 R09: ffffffff8168f1e0
Dec  9 12:34:32 server1 kernel: [726212.912006] R10: 0000000000000601 R11: ffff88033291e000 R12: 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.912006] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.912006] FS:  0000000000000000(0000) GS:ffff88033fd80000(0000) knlGS:0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.912006] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec  9 12:34:32 server1 kernel: [726212.912006] CR2: 00007f2145e73008 CR3: 00000002dd877000 CR4: 00000000001406e0
Dec  9 12:34:32 server1 kernel: [726212.912006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.912006] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec  9 12:34:32 server1 kernel: [726212.912006] Process swapper/3 (pid: 0, threadinfo ffff88033291e000, task ffff8803328fe100)
Dec  9 12:34:32 server1 kernel: [726212.912006] Stack:
Dec  9 12:34:32 server1 kernel: [726212.912006]  ffffffff81014938 ffffffff8168f1e0 ffffffff8100d24c 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.912006]  bd9e6a7c2cb84c47 00000000b1c7b947 0000000000000003 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.912006]  ffffffff8134328a 0000000000000000 b93446556384bb8b 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.912006] Call Trace:
Dec  9 12:34:32 server1 kernel: [726212.912006]  [<ffffffff81014938>] ? default_idle+0x47/0x7f
Dec  9 12:34:32 server1 kernel: [726212.912006]  [<ffffffff8100d24c>] ? cpu_idle+0xaf/0xf2
Dec  9 12:34:32 server1 kernel: [726212.912006]  [<ffffffff8134328a>] ? start_secondary+0x1da/0x1e0
Dec  9 12:34:32 server1 kernel: [726212.912006] Code: ee 48 c7 c7 1f 89 4c 81 31 c0 e8 7e f5 31 00 48 83 c4 18 89 d8 5b 5d 41 5c 41 5d c3 90 90 90 9c 58 c3 57 9d c3 fa c3 fb c3 fb f4 <c3> f4 c3 0f 1f 80 00 00 00 00 c3 0f 1f 80 00 00 00 00 c3 0f 06
Dec  9 12:34:32 server1 kernel: [726212.912006] Call Trace:
Dec  9 12:34:32 server1 kernel: [726212.912006]  [<ffffffff81014938>] ? default_idle+0x47/0x7f
Dec  9 12:34:32 server1 kernel: [726212.912006]  [<ffffffff8100d24c>] ? cpu_idle+0xaf/0xf2
Dec  9 12:34:32 server1 kernel: [726212.912006]  [<ffffffff8134328a>] ? start_secondary+0x1da/0x1e0
Dec  9 12:35:12 server1 kernel: [726222.800034] ------------[ cut here ]------------
Dec  9 12:35:12 server1 kernel: [726222.800049] WARNING: at /build/linux-Tvajqd/linux-3.2.68/net/sched/sch_generic.c:256 dev_watchdog+0xf2/0x151()
Dec  9 12:35:12 server1 kernel: [726222.800051] Hardware name: KVM
Dec  9 12:35:12 server1 kernel: [726222.800053] NETDEV WATCHDOG: eth0 (e1000): transmit queue 0 timed out
Dec  9 12:35:12 server1 kernel: [726222.800054] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:35:12 server1 kernel: [726222.800101] Pid: 0, comm: swapper/2 Not tainted 3.2.0-4-amd64 #1 Debian 3.2.68-1+deb7u6
Dec  9 12:35:12 server1 kernel: [726222.800102] Call Trace:
Dec  9 12:35:12 server1 kernel: [726222.800104]  <IRQ>  [<ffffffff81046de1>] ? warn_slowpath_common+0x78/0x8c
Dec  9 12:35:12 server1 kernel: [726222.800113]  [<ffffffff81046e8d>] ? warn_slowpath_fmt+0x45/0x4a
Dec  9 12:35:12 server1 kernel: [726222.800121]  [<ffffffff812a8d51>] ? netif_tx_lock+0x40/0x75
Dec  9 12:35:12 server1 kernel: [726222.800124]  [<ffffffff812a8ec1>] ? dev_watchdog+0xf2/0x151
Dec  9 12:35:12 server1 kernel: [726222.800127]  [<ffffffff81052618>] ? run_timer_softirq+0x19a/0x261
Dec  9 12:35:12 server1 kernel: [726222.800129]  [<ffffffff812a8dcf>] ? netif_tx_unlock+0x49/0x49
Dec  9 12:35:12 server1 kernel: [726222.800133]  [<ffffffff81066439>] ? timekeeping_get_ns+0xd/0x2a
Dec  9 12:35:12 server1 kernel: [726222.800136]  [<ffffffff8104c48e>] ? __do_softirq+0xb9/0x177
Dec  9 12:35:12 server1 kernel: [726222.800141]  [<ffffffff813584ec>] ? call_softirq+0x1c/0x30
Dec  9 12:35:12 server1 kernel: [726222.800147]  [<ffffffff8100fa91>] ? do_softirq+0x3c/0x7b
Dec  9 12:35:12 server1 kernel: [726222.800149]  [<ffffffff8104c6f6>] ? irq_exit+0x3c/0x99
Dec  9 12:35:12 server1 kernel: [726222.800154]  [<ffffffff8102475c>] ? smp_apic_timer_interrupt+0x74/0x82
Dec  9 12:35:12 server1 kernel: [726222.800169]  [<ffffffff81356d5e>] ? apic_timer_interrupt+0x6e/0x80
Dec  9 12:35:12 server1 kernel: [726222.800171]  <EOI>  [<ffffffff81096b8d>] ? rcu_needs_cpu+0x50/0x1bb
Dec  9 12:35:12 server1 kernel: [726222.800178]  [<ffffffff8102b36c>] ? native_safe_halt+0x2/0x3
Dec  9 12:35:12 server1 kernel: [726222.800182]  [<ffffffff81014938>] ? default_idle+0x47/0x7f
Dec  9 12:35:12 server1 kernel: [726222.800185]  [<ffffffff8100d24c>] ? cpu_idle+0xaf/0xf2
Dec  9 12:35:12 server1 kernel: [726222.800190]  [<ffffffff8134328a>] ? start_secondary+0x1da/0x1e0
Dec  9 12:35:12 server1 kernel: [726222.800192] ---[ end trace 877820c1f31d5380 ]---
Dec  9 12:35:40 server1 kernel: [726280.565671] BUG: soft lockup - CPU#1 stuck for 26s! [mysqld:4141]
Dec  9 12:35:40 server1 kernel: [726280.566088] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:35:40 server1 kernel: [726280.566142] CPU 1
Dec  9 12:35:40 server1 kernel: [726280.566143] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:35:40 server1 kernel: [726280.566173]
Dec  9 12:35:40 server1 kernel: [726280.566177] Pid: 4141, comm: mysqld Tainted: G        W    3.2.0-4-amd64 #1 Debian 3.2.68-1+deb7u6 Red Hat KVM
Dec  9 12:35:40 server1 kernel: [726280.566181] RIP: 0010:[<ffffffff813515f6>]  [<ffffffff813515f6>] retint_careful+0xe/0x32
Dec  9 12:35:40 server1 kernel: [726280.566193] RSP: 0000:ffff880329751f88  EFLAGS: 00000203
Dec  9 12:35:40 server1 kernel: [726280.566195] RAX: 00007fc0c5e74000 RBX: ffffffff81066439 RCX: ffff880329750000
Dec  9 12:35:40 server1 kernel: [726280.566197] RDX: 0000000000000008 RSI: ffff880329751f78 RDI: 000000000000fe2e
Dec  9 12:35:40 server1 kernel: [726280.566199] RBP: 00007fc0c5e74000 R08: ffffffff814052a0 R09: 0000000000000000
Dec  9 12:35:40 server1 kernel: [726280.566201] R10: 0000000000000000 R11: 00007fc0d2205e38 R12: ffff88032f1a4080
Dec  9 12:35:40 server1 kernel: [726280.566203] R13: ffff88015fbeb400 R14: ffff88032f6cb1c0 R15: 00000000000127c0
Dec  9 12:35:40 server1 kernel: [726280.566206] FS:  00007fc0bea41700(0000) GS:ffff88033fc80000(0000) knlGS:0000000000000000
Dec  9 12:35:40 server1 kernel: [726280.566208] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  9 12:35:40 server1 kernel: [726280.566210] CR2: 00007f2153410000 CR3: 000000031c0f8000 CR4: 00000000001406e0
Dec  9 12:35:40 server1 kernel: [726280.566215] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  9 12:35:40 server1 kernel: [726280.566217] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec  9 12:35:40 server1 kernel: [726280.566220] Process mysqld (pid: 4141, threadinfo ffff880329750000, task ffff88032f6cb1c0)
Dec  9 12:35:40 server1 kernel: [726280.566222] Stack:
Dec  9 12:35:40 server1 kernel: [726280.566503]  00007fc0c5e74000 0000000000000000 0000000000000000 0000000000000000
Dec  9 12:35:40 server1 kernel: [726280.566508]  a4a46aee384fff9f 00007fc0c5e75000 ddc05329c75d11a3 ddc05329c75d11a3
Dec  9 12:35:40 server1 kernel: [726280.566512]  00007fc0c5e77ff8 ffffffffffffff02 00007fc0d1aae9a0 0000000000000033
Dec  9 12:35:40 server1 kernel: [726280.566516] Call Trace:
Dec  9 12:35:40 server1 kernel: [726280.569621] Code: 00 00 ff ff 5f 65 48 0b 04 25 08 a0 00 00 0f 01 f8 0f 1f 00 48 89 c4 58 e9 9f ff ff ff 0f ba e2 03 73 2c fb 0f 1f 80 00 00 00 00 <57> e8 48 ea ff ff 5f 65 48 8b 0c 25 c8 b6 00 00 48 81 e9 d8 1f
Dec  9 12:35:40 server1 kernel: [726280.569621] Call Trace:
Dec  9 12:37:56 server1 kernel: [726416.567402] BUG: soft lockup - CPU#2 stuck for 33s! [cron:2299]
Dec  9 12:37:56 server1 kernel: [726381.850950] ------------[ cut here ]------------
Dec  9 12:37:56 server1 kernel: [726381.850950] WARNING: at /build/linux-Tvajqd/linux-3.2.68/kernel/watchdog.c:241 watchdog_overflow_callback+0x93/0x9e()
Dec  9 12:37:56 server1 kernel: [726381.850950] Hardware name: KVM
Dec  9 12:37:56 server1 kernel: [726381.850950] Watchdog detected hard LOCKUP on cpu 1
Dec  9 12:37:56 server1 kernel: [726381.850950] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:37:56 server1 kernel: [726381.850950] Pid: 160, comm: kworker/1:1 Tainted: G        W    3.2.0-4-amd64 #1 Debian 3.2.68-1+deb7u6
Dec  9 12:37:56 server1 kernel: [726381.850950] Call Trace:
Dec  9 12:37:56 server1 kernel: [726381.850950]  <NMI>  [<ffffffff81046de1>] ? warn_slowpath_common+0x78/0x8c
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff81046e8d>] ? warn_slowpath_fmt+0x45/0x4a
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff81013e64>] ? sched_clock+0x5/0x8
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff81090c78>] ? watchdog_overflow_callback+0x93/0x9e
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff81090be5>] ? touch_nmi_watchdog+0x60/0x60
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff810b0dd1>] ? __perf_event_overflow+0xf9/0x17b
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff810641b2>] ? local_clock+0x24/0x2b
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff810af21f>] ? perf_event_update_userpage+0x9/0x9d
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff8101a3e3>] ? x86_perf_event_set_period+0x101/0x10b
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff8101d7c6>] ? intel_pmu_handle_irq+0x222/0x283
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff8100fd13>] ? dump_trace+0x243/0x252
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff81352222>] ? nmi_handle.isra.0+0x41/0x64
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff813522c2>] ? do_nmi+0x7d/0x258
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff81351ae0>] ? nmi+0x20/0x30
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff81071422>] ? do_raw_spin_lock+0x15/0x1b
Dec  9 12:37:56 server1 kernel: [726381.850950]  <<EOE>>  [<ffffffff810ec17a>] ? drain_array+0x4b/0xc9
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff810ec3b1>] ? cache_reap+0xc8/0x1a2
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:37:56 server1 kernel: [726381.850950] ---[ end trace 877820c1f31d5381 ]---
Dec  9 12:37:56 server1 kernel: [726416.571367] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:37:56 server1 kernel: [726416.571367] CPU 2
Dec  9 12:37:56 server1 kernel: [726416.571367] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:37:56 server1 kernel: [726416.571367]
Dec  9 12:37:56 server1 kernel: [726416.571367] Pid: 2299, comm: cron Tainted: G        W    3.2.0-4-amd64 #1 Debian 3.2.68-1+deb7u6 Red Hat KVM
Dec  9 12:37:56 server1 kernel: [726416.571367] RIP: 0033:[<00007fa6fc429478>]  [<00007fa6fc429478>] 0x7fa6fc429477
Dec  9 12:37:56 server1 kernel: [726416.571367] RSP: 002b:00007fff4e3a6868  EFLAGS: 00010217
Dec  9 12:37:56 server1 kernel: [726416.571367] RAX: 0000000000000000 RBX: 000000000000fe2e RCX: 00007fa6fc429470
Dec  9 12:37:56 server1 kernel: [726416.571367] RDX: 0000000000000000 RSI: 00007fff4e3a6a10 RDI: 00007fff4e3a6a10
Dec  9 12:37:56 server1 kernel: [726416.571367] RBP: 00007fff4e3a6910 R08: 00007fff4e3a6870 R09: 0000000000000000
Dec  9 12:37:56 server1 kernel: [726416.571367] R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000
Dec  9 12:37:56 server1 kernel: [726416.571367] R13: 0000000000000000 R14: 00007fff4e3a6b70 R15: 00000000ffffffff
Dec  9 12:37:56 server1 kernel: [726416.571367] FS:  00007fa6fcd4b7a0(0000) GS:ffff88033fd00000(0000) knlGS:0000000000000000
Dec  9 12:37:56 server1 kernel: [726416.571367] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  9 12:37:56 server1 kernel: [726416.571367] CR2: 00007f5b7bb88000 CR3: 0000000330cb5000 CR4: 00000000001406e0
Dec  9 12:37:56 server1 kernel: [726416.571367] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  9 12:37:56 server1 kernel: [726416.571367] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec  9 12:37:56 server1 kernel: [726416.571367] Process cron (pid: 2299, threadinfo ffff880330ccc000, task ffff88032fa410c0)
Dec  9 12:37:56 server1 kernel: [726416.571367]
Dec  9 12:37:56 server1 kernel: [726416.571367] Call Trace:
Dec  9 12:38:04 server1 kernel: [726424.620114] INFO: task kworker/2:0:14 blocked for more than 120 seconds.
Dec  9 12:38:04 server1 kernel: [726424.625127] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  9 12:38:04 server1 kernel: [726424.625864] kworker/2:0    D ffff88033fd127c0    0    14      2 0x00000000
Dec  9 12:38:04 server1 kernel: [726424.625874]  ffff8803328c4740 0000000000000046 0000000000000000 ffff8803328f0780
Dec  9 12:38:04 server1 kernel: [726424.625878]  00000000000127c0 ffff8803328e5fd8 ffff8803328e5fd8 ffff8803328c4740
Dec  9 12:38:04 server1 kernel: [726424.625880]  0000000000000002 0000000181041d01 0000000000000000 7fffffffffffffff
Dec  9 12:38:04 server1 kernel: [726424.625883] Call Trace:
Dec  9 12:38:04 server1 kernel: [726424.625893]  [<ffffffff8135030b>] ? schedule_timeout+0x2c/0xdb
Dec  9 12:38:04 server1 kernel: [726424.625899]  [<ffffffff81038139>] ? set_next_entity+0x32/0x55
Dec  9 12:38:04 server1 kernel: [726424.625905]  [<ffffffff8100d02f>] ? load_TLS+0x7/0xa
Dec  9 12:38:04 server1 kernel: [726424.625908]  [<ffffffff8100d66c>] ? __switch_to+0x101/0x265
Dec  9 12:38:04 server1 kernel: [726424.625910]  [<ffffffff8134ff51>] ? wait_for_common+0xa0/0x119
Dec  9 12:38:04 server1 kernel: [726424.625913]  [<ffffffff8103f751>] ? try_to_wake_up+0x197/0x197
Dec  9 12:38:04 server1 kernel: [726424.625917]  [<ffffffff8105ae16>] ? wait_on_work+0xe6/0x11c
Dec  9 12:38:04 server1 kernel: [726424.625919]  [<ffffffff8105a450>] ? worker_set_flags+0x8f/0x8f
Dec  9 12:38:04 server1 kernel: [726424.625922]  [<ffffffff8105ba76>] ? __cancel_work_timer+0xb2/0xf4
Dec  9 12:38:04 server1 kernel: [726424.626089]  [<ffffffffa00bc889>] ? e1000_down_and_stop+0x2f/0x48 [e1000]
Dec  9 12:38:04 server1 kernel: [726424.626094]  [<ffffffffa00bf1da>] ? e1000_down+0x108/0x15e [e1000]
Dec  9 12:38:04 server1 kernel: [726424.626098]  [<ffffffffa00c1373>] ? e1000_reset_task+0x53/0x6c [e1000]
Dec  9 12:38:04 server1 kernel: [726424.626100]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:38:04 server1 kernel: [726424.626103]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:38:04 server1 kernel: [726424.626105]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:38:04 server1 kernel: [726424.626108]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:38:04 server1 kernel: [726424.626112]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:38:04 server1 kernel: [726424.626115]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:38:04 server1 kernel: [726424.626117]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:38:04 server1 kernel: [726424.626158] INFO: task kworker/2:2:171 blocked for more than 120 seconds.
Dec  9 12:38:04 server1 kernel: [726424.626416] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  9 12:38:04 server1 kernel: [726424.627023] kworker/2:2    D ffff88033fd127c0    0  171      2 0x00000000
Dec  9 12:38:04 server1 kernel: [726424.627031]  ffff88032f6cb8c0 0000000000000046 00000000000127c0 ffff880330e13100
Dec  9 12:38:04 server1 kernel: [726424.627034]  00000000000127c0 ffff88032fa53fd8 ffff88032fa53fd8 ffff88032f6cb8c0
Dec  9 12:38:04 server1 kernel: [726424.627037]  ffffffff81351267 ffffffff81042c7b ffff88032f6cb8c0 ffff88032fa51000
Dec  9 12:38:04 server1 kernel: [726424.627039] Call Trace:
Dec  9 12:38:04 server1 kernel: [726424.627044]  [<ffffffff81351267>] ? _raw_spin_unlock_irqrestore+0xe/0xf
Dec  9 12:38:04 server1 kernel: [726424.627048]  [<ffffffff81042c7b>] ? set_cpus_allowed_ptr+0xdd/0xec
Dec  9 12:38:04 server1 kernel: [726424.627050]  [<ffffffff8135079c>] ? __mutex_lock_common.isra.5+0xff/0x164
Dec  9 12:38:04 server1 kernel: [726424.627052]  [<ffffffff8135068a>] ? mutex_lock+0x1a/0x2d
Dec  9 12:38:04 server1 kernel: [726424.627055]  [<ffffffff81037240>] ? test_tsk_need_resched+0xa/0x13
Dec  9 12:38:04 server1 kernel: [726424.627058]  [<ffffffff8103b0e3>] ? check_preempt_curr+0x52/0x5f
Dec  9 12:38:04 server1 kernel: [726424.627066]  [<ffffffffa00c01e5>] ? e1000_watchdog+0x48/0x4ac [e1000]
Dec  9 12:38:04 server1 kernel: [726424.627069]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:38:04 server1 kernel: [726424.627073]  [<ffffffff81059ff6>] ? need_to_create_worker+0x9/0x1c
Dec  9 12:38:04 server1 kernel: [726424.627075]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:38:04 server1 kernel: [726424.627077]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:38:04 server1 kernel: [726424.627080]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:38:04 server1 kernel: [726424.627082]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:38:04 server1 kernel: [726424.627085]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:38:04 server1 kernel: [726424.627089]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:39:01 server1 /USR/SBIN/CRON[23643]: (root) CMD (  [ -x /usr/lib/php5/sessionclean ] && /usr/lib/php5/sessionclean)
Dec  9 12:40:01 server1 /USR/SBIN/CRON[23666]: (root) CMD (/usr/local/maldetect/maldet --mkpubpaths >> /dev/null 2>&1)
Dec  9 12:40:04 server1 kernel: [726544.624119] INFO: task kworker/2:0:14 blocked for more than 120 seconds.
Dec  9 12:40:04 server1 kernel: [726544.627939] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  9 12:40:04 server1 kernel: [726544.628526] kworker/2:0    D ffff88033fd127c0    0    14      2 0x00000000
Dec  9 12:40:04 server1 kernel: [726544.628538]  ffff8803328c4740 0000000000000046 0000000000000000 ffff8803328f0780
Dec  9 12:40:04 server1 kernel: [726544.628542]  00000000000127c0 ffff8803328e5fd8 ffff8803328e5fd8 ffff8803328c4740
Dec  9 12:40:04 server1 kernel: [726544.628545]  0000000000000002 0000000181041d01 0000000000000000 7fffffffffffffff
Dec  9 12:40:04 server1 kernel: [726544.628548] Call Trace:
Dec  9 12:40:04 server1 kernel: [726544.628578]  [<ffffffff8135030b>] ? schedule_timeout+0x2c/0xdb
Dec  9 12:40:04 server1 kernel: [726544.628591]  [<ffffffff81038139>] ? set_next_entity+0x32/0x55
Dec  9 12:40:04 server1 kernel: [726544.628598]  [<ffffffff8100d02f>] ? load_TLS+0x7/0xa
Dec  9 12:40:04 server1 kernel: [726544.628601]  [<ffffffff8100d66c>] ? __switch_to+0x101/0x265
Dec  9 12:40:04 server1 kernel: [726544.628603]  [<ffffffff8134ff51>] ? wait_for_common+0xa0/0x119
Dec  9 12:40:04 server1 kernel: [726544.628609]  [<ffffffff8103f751>] ? try_to_wake_up+0x197/0x197
Dec  9 12:40:04 server1 kernel: [726544.628618]  [<ffffffff8105ae16>] ? wait_on_work+0xe6/0x11c
Dec  9 12:40:04 server1 kernel: [726544.628620]  [<ffffffff8105a450>] ? worker_set_flags+0x8f/0x8f
Dec  9 12:40:04 server1 kernel: [726544.628623]  [<ffffffff8105ba76>] ? __cancel_work_timer+0xb2/0xf4
Dec  9 12:40:04 server1 kernel: [726544.628645]  [<ffffffffa00bc889>] ? e1000_down_and_stop+0x2f/0x48 [e1000]
Dec  9 12:40:04 server1 kernel: [726544.628649]  [<ffffffffa00bf1da>] ? e1000_down+0x108/0x15e [e1000]
Dec  9 12:40:04 server1 kernel: [726544.628653]  [<ffffffffa00c1373>] ? e1000_reset_task+0x53/0x6c [e1000]
Dec  9 12:40:04 server1 kernel: [726544.628655]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:40:04 server1 kernel: [726544.628658]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:40:04 server1 kernel: [726544.628660]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:40:04 server1 kernel: [726544.628664]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:40:04 server1 kernel: [726544.628671]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:40:04 server1 kernel: [726544.628674]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:40:04 server1 kernel: [726544.628676]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:40:04 server1 kernel: [726544.628687] INFO: task kworker/2:2:171 blocked for more than 120 seconds.
Dec  9 12:40:04 server1 kernel: [726544.628971] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  9 12:40:04 server1 kernel: [726544.629430] kworker/2:2    D ffff88033fd127c0    0  171      2 0x00000000
Dec  9 12:40:04 server1 kernel: [726544.629434]  ffff88032f6cb8c0 0000000000000046 00000000000127c0 ffff880330e13100
Dec  9 12:40:04 server1 kernel: [726544.629436]  00000000000127c0 ffff88032fa53fd8 ffff88032fa53fd8 ffff88032f6cb8c0
Dec  9 12:40:04 server1 kernel: [726544.629439]  ffffffff81351267 ffffffff81042c7b ffff88032f6cb8c0 ffff88032fa51000
Dec  9 12:40:04 server1 kernel: [726544.629441] Call Trace:
Dec  9 12:40:04 server1 kernel: [726544.629445]  [<ffffffff81351267>] ? _raw_spin_unlock_irqrestore+0xe/0xf
Dec  9 12:40:04 server1 kernel: [726544.629449]  [<ffffffff81042c7b>] ? set_cpus_allowed_ptr+0xdd/0xec
Dec  9 12:40:04 server1 kernel: [726544.629451]  [<ffffffff8135079c>] ? __mutex_lock_common.isra.5+0xff/0x164
Dec  9 12:40:04 server1 kernel: [726544.629454]  [<ffffffff8135068a>] ? mutex_lock+0x1a/0x2d
Dec  9 12:40:04 server1 kernel: [726544.629456]  [<ffffffff81037240>] ? test_tsk_need_resched+0xa/0x13
Dec  9 12:40:04 server1 kernel: [726544.629459]  [<ffffffff8103b0e3>] ? check_preempt_curr+0x52/0x5f
Dec  9 12:40:04 server1 kernel: [726544.629464]  [<ffffffffa00c01e5>] ? e1000_watchdog+0x48/0x4ac [e1000]
Dec  9 12:40:04 server1 kernel: [726544.629467]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:40:04 server1 kernel: [726544.629470]  [<ffffffff81059ff6>] ? need_to_create_worker+0x9/0x1c
Dec  9 12:40:04 server1 kernel: [726544.629472]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:40:04 server1 kernel: [726544.629474]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:40:04 server1 kernel: [726544.629476]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:40:04 server1 kernel: [726544.629479]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:40:04 server1 kernel: [726544.629482]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:40:04 server1 kernel: [726544.629484]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:42:04 server1 kernel: [726664.628090] INFO: task kworker/2:0:14 blocked for more than 120 seconds.
Dec  9 12:42:04 server1 kernel: [726664.631165] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  9 12:42:04 server1 kernel: [726664.631612] kworker/2:0    D ffff88033fd127c0    0    14      2 0x00000000
Dec  9 12:42:04 server1 kernel: [726664.631628]  ffff8803328c4740 0000000000000046 0000000000000000 ffff8803328f0780
Dec  9 12:42:04 server1 kernel: [726664.631632]  00000000000127c0 ffff8803328e5fd8 ffff8803328e5fd8 ffff8803328c4740
Dec  9 12:42:04 server1 kernel: [726664.631634]  0000000000000002 0000000181041d01 0000000000000000 7fffffffffffffff
Dec  9 12:42:04 server1 kernel: [726664.631639] Call Trace:
Dec  9 12:42:04 server1 kernel: [726664.631668]  [<ffffffff8135030b>] ? schedule_timeout+0x2c/0xdb
Dec  9 12:42:04 server1 kernel: [726664.631693]  [<ffffffff81038139>] ? set_next_entity+0x32/0x55
Dec  9 12:42:04 server1 kernel: [726664.631701]  [<ffffffff8100d02f>] ? load_TLS+0x7/0xa
Dec  9 12:42:04 server1 kernel: [726664.631704]  [<ffffffff8100d66c>] ? __switch_to+0x101/0x265
Dec  9 12:42:04 server1 kernel: [726664.631706]  [<ffffffff8134ff51>] ? wait_for_common+0xa0/0x119
Dec  9 12:42:04 server1 kernel: [726664.631712]  [<ffffffff8103f751>] ? try_to_wake_up+0x197/0x197
Dec  9 12:42:04 server1 kernel: [726664.631720]  [<ffffffff8105ae16>] ? wait_on_work+0xe6/0x11c
Dec  9 12:42:04 server1 kernel: [726664.631722]  [<ffffffff8105a450>] ? worker_set_flags+0x8f/0x8f
Dec  9 12:42:04 server1 kernel: [726664.631725]  [<ffffffff8105ba76>] ? __cancel_work_timer+0xb2/0xf4
Dec  9 12:42:04 server1 kernel: [726664.631856]  [<ffffffffa00bc889>] ? e1000_down_and_stop+0x2f/0x48 [e1000]
Dec  9 12:42:04 server1 kernel: [726664.631860]  [<ffffffffa00bf1da>] ? e1000_down+0x108/0x15e [e1000]
Dec  9 12:42:04 server1 kernel: [726664.631864]  [<ffffffffa00c1373>] ? e1000_reset_task+0x53/0x6c [e1000]
Dec  9 12:42:04 server1 kernel: [726664.631866]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:42:04 server1 kernel: [726664.631869]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:42:04 server1 kernel: [726664.631871]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:42:04 server1 kernel: [726664.631874]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:42:04 server1 kernel: [726664.631881]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:42:04 server1 kernel: [726664.631884]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:42:04 server1 kernel: [726664.631886]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:42:04 server1 kernel: [726664.631921] INFO: task kworker/2:2:171 blocked for more than 120 seconds.
Dec  9 12:42:04 server1 kernel: [726664.632248] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  9 12:42:04 server1 kernel: [726664.632651] kworker/2:2    D ffff88033fd127c0    0  171      2 0x00000000
Dec  9 12:42:04 server1 kernel: [726664.632659]  ffff88032f6cb8c0 0000000000000046 00000000000127c0 ffff880330e13100
Dec  9 12:42:04 server1 kernel: [726664.632662]  00000000000127c0 ffff88032fa53fd8 ffff88032fa53fd8 ffff88032f6cb8c0
Dec  9 12:42:04 server1 kernel: [726664.632664]  ffffffff81351267 ffffffff81042c7b ffff88032f6cb8c0 ffff88032fa51000
Dec  9 12:42:04 server1 kernel: [726664.632667] Call Trace:
Dec  9 12:42:04 server1 kernel: [726664.632671]  [<ffffffff81351267>] ? _raw_spin_unlock_irqrestore+0xe/0xf
Dec  9 12:42:04 server1 kernel: [726664.632675]  [<ffffffff81042c7b>] ? set_cpus_allowed_ptr+0xdd/0xec
Dec  9 12:42:04 server1 kernel: [726664.632677]  [<ffffffff8135079c>] ? __mutex_lock_common.isra.5+0xff/0x164
Dec  9 12:42:04 server1 kernel: [726664.632679]  [<ffffffff8135068a>] ? mutex_lock+0x1a/0x2d
Dec  9 12:42:04 server1 kernel: [726664.632682]  [<ffffffff81037240>] ? test_tsk_need_resched+0xa/0x13
Dec  9 12:42:04 server1 kernel: [726664.632685]  [<ffffffff8103b0e3>] ? check_preempt_curr+0x52/0x5f
Dec  9 12:42:04 server1 kernel: [726664.632690]  [<ffffffffa00c01e5>] ? e1000_watchdog+0x48/0x4ac [e1000]
Dec  9 12:42:04 server1 kernel: [726664.632693]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:42:04 server1 kernel: [726664.632695]  [<ffffffff81059ff6>] ? need_to_create_worker+0x9/0x1c
Dec  9 12:42:04 server1 kernel: [726664.632697]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:42:04 server1 kernel: [726664.632699]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:42:04 server1 kernel: [726664.632702]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:42:04 server1 kernel: [726664.632705]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:42:04 server1 kernel: [726664.632707]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:42:04 server1 kernel: [726664.632710]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:43:28 server1 shutdown[23719]: shutting down for system reboot
Dec  9 12:43:28 server1 init: Switching to runlevel: 6
Dec  9 12:43:29 server1 vnstatd[2046]: SIGTERM received, exiting.
Dec  9 12:43:30 server1 mysqld: 151209 12:43:30 [Note] /usr/sbin/mysqld: Normal shutdown
Dec  9 12:43:30 server1 mysqld:
Dec  9 12:43:30 server1 mysqld: 151209 12:43:30 [Note] Event Scheduler: Purging the queue. 0 events
Dec  9 12:43:30 server1 mysqld: 151209 12:43:30  InnoDB: Starting shutdown...
Dec  9 12:43:33 server1 mysqld: 151209 12:43:33  InnoDB: Shutdown completed; log sequence number 3625647798
Dec  9 12:43:33 server1 mysqld: 151209 12:43:33 [Note] /usr/sbin/mysqld: Shutdown complete
Dec  9 12:43:33 server1 mysqld:
Dec  9 12:43:33 server1 mysqld:
Dec  9 12:43:33 server1 mysqld: Memory status:
Dec  9 12:43:33 server1 mysqld: Non-mmapped space allocated from system: 10186752
Dec  9 12:43:33 server1 mysqld: Number of free chunks:#011#011#011 67
Dec  9 12:43:33 server1 mysqld: Number of fastbin blocks:#011#011 0
Dec  9 12:43:33 server1 mysqld: Number of mmapped regions:#011#011 17
Dec  9 12:43:33 server1 mysqld: Space in mmapped regions:#011#011 49676288
Dec  9 12:43:33 server1 mysqld: Maximum total allocated space:#011#011 0
Dec  9 12:43:33 server1 mysqld: Space available in freed fastbin blocks: 0
Dec  9 12:43:33 server1 mysqld: Total allocated space:#011#011#011 8657696
Dec  9 12:43:33 server1 mysqld: Total free space:#011#011#011 1529056
Dec  9 12:43:33 server1 mysqld: Top-most, releasable space:#011#011 548544
Dec  9 12:43:33 server1 mysqld: Estimated memory (with thread stack):    59863040
Dec  9 12:43:33 server1 mysqld:
Dec  9 12:43:33 server1 mysqld:
Dec  9 12:43:33 server1 mysqld:
Dec  9 12:43:33 server1 mysqld: Events status:
Dec  9 12:43:33 server1 mysqld: LLA = Last Locked At  LUA = Last Unlocked At
Dec  9 12:43:33 server1 mysqld: WOC = Waiting On Condition  DL = Data Locked
Dec  9 12:43:33 server1 mysqld:
Dec  9 12:43:33 server1 mysqld: Event scheduler status:
Dec  9 12:43:33 server1 mysqld: State      : INITIALIZED
Dec  9 12:43:33 server1 mysqld: Thread id  : 0
Dec  9 12:43:33 server1 mysqld: LLA        : n/a:0
Dec  9 12:43:33 server1 mysqld: LUA        : n/a:0
Dec  9 12:43:33 server1 mysqld: WOC        : NO
Dec  9 12:43:33 server1 mysqld: Workers    : 0
Dec  9 12:43:33 server1 mysqld: Executed  : 0
Dec  9 12:43:33 server1 mysqld: Data locked: NO
Dec  9 12:43:33 server1 mysqld:
Dec  9 12:43:33 server1 mysqld: Event queue status:
Dec  9 12:43:33 server1 mysqld: Element count  : 0
Dec  9 12:43:33 server1 mysqld: Data locked    : NO
Dec  9 12:43:33 server1 mysqld: Attempting lock : NO
Dec  9 12:43:33 server1 mysqld: LLA            : drop_schema_events:382
Dec  9 12:43:33 server1 mysqld: LUA            : drop_schema_events:384
Dec  9 12:43:33 server1 mysqld: WOC            : NO
Dec  9 12:43:33 server1 mysqld: Next activation : never
Dec  9 12:43:33 server1 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Dec  9 12:43:34 server1 acpid: exiting

---

Für mich sieht es so aus , als wenn mein MySQL-Server dieses Problem hervorruft:

Code:

---

BUG: soft lockup - CPU#1 stuck for 26s! [mysqld:4141]

---

Hatte jemand mal ein ähnliches Problem in Zusammenhang mit MySQL und einem KVM-Server?