Hi,
We are facing some issue on oracle linux which is running on OVM. We have 2 different server which has the same issue. the production server is using raw disk, and not production is using virtual disk. below is the message we are seeing in the message log and we eventually not able to run any lvm command. Everything single command will hang. Based on the message, it seems related to storage, however, the storage team confirms there is no problem with the storage. Checked with oracle and oracle could not find the issue.
Does anyone see the issue before and how you solve the problem?
Sep 21 23:04:45 kc1prdb01 kernel: INFO: task jbd2/dm-21-8:1635 blocked for more than 120 seconds.
Sep 21 23:04:45 kc1prdb01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message
Sep 21 23:04:45 kc1prdb01 kernel: jbd2/dm-21-8 D ffff882e23252180 0 1635 2 0x00000080
Sep 21 23:04:45 kc1prdb01 kernel: ffff882da6f87be0 0000000000000246 ffff882da6f87b60 ffff882d00000000
Sep 21 23:04:45 kc1prdb01 kernel: 0000000000012180 ffff882da6f87fd8 ffff882da6f86010 0000000000012180
Sep 21 23:04:45 kc1prdb01 kernel: ffff882da6f87fd8 0000000000012180 ffff882dacbe20c0 ffff882da6ae65c0
Sep 21 23:04:45 kc1prdb01 kernel: Call Trace:
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffff8119b710>] ? __wait_on_buffer+0x30/0x30
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffff8150ad1f>] schedule+0x3f/0x60
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffff8150adcc>] io_schedule+0x8c/0xd0
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffff8119b71e>] sleep_on_buffer+0xe/0x20
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffff8150b4bf>] __wait_on_bit+0x5f/0x90
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffff8119b710>] ? __wait_on_buffer+0x30/0x30
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffff8150b568>] out_of_line_wait_on_bit+0x78/0x90
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffff810914d0>] ? autoremove_wake_function+0x50/0x50
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffff8119b70e>] __wait_on_buffer+0x2e/0x30
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffffa0044b8b>] jbd2_journal_commit_transaction+0xb5b/0x1690 [jbd2]
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffff8150cd7e>] ? _raw_spin_unlock_irqrestore+0x1e/0x30
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffff8107e876>] ? try_to_del_timer_sync+0x86/0xe0
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffffa004aa86>] kjournald2+0xd6/0x3b0 [jbd2]
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffff81091480>] ? wake_up_bit+0x40/0x40
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffffa004a9b0>] ? commit_timeout+0x10/0x10 [jbd2]
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffff81090db6>] kthread+0x96/0xa0
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffff81516304>] kernel_thread_helper+0x4/0x10
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffff815153e3>] ? int_ret_from_sys_call+0x7/0x1b
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffff8150d261>] ? retint_restore_args+0x5/0x6
Sep 21 23:04:45 kc1prdb01 kernel: [<ffffffff81516300>] ? gs_change+0x13/0x13
Thanks,
Denny