After upgrading Ubuntu LTS 14.04 to 16.04 system stopped booting and proposed entering maintenance shell to solve a problem. There are two problem indicators in logs:
[ 721.242651] INFO: task bfad_worker:952 blocked for more than 120 seconds. [ 721.272889] Not tainted 4.4.0-31-generic #50-Ubuntu [ 721.302231] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 721.332218] bfad_worker D ffff880c45767ea8 0 952 2 0x00000000 [ 721.332220] ffff880c45767ea8 ffff8818443939c0 ffff881848f5cb00 ffff880c43d59900 [ 721.332222] ffff880c45768000 ffff881842780000 ffffffffc00df450 0000000000000000 [ 721.332223] 0000000000000000 ffff880c45767ec0 ffffffff81829a25 ffff880c48aa5b40 [ 721.332225] Call Trace: [ 721.332231] [<ffffffffc00df450>] ? bfad_read_firmware+0xd0/0xd0 [bfa] [ 721.332233] [<ffffffff81829a25>] schedule+0x35/0x80 [ 721.332235] [<ffffffff810a07e3>] kthread+0xb3/0xf0 [ 721.332236] [<ffffffff810a0730>] ? kthread_create_on_node+0x1e0/0x1e0 [ 721.332238] [<ffffffff8182decf>] ret_from_fork+0x3f/0x70 [ 721.332239] [<ffffffff810a0730>] ? kthread_create_on_node+0x1e0/0x1e0
and (the more important):
[ 6.070145] bfa 0000:15:00.0: Running firmware version is incompatible with the driver version [ 6.295256] bfa 0000:15:00.0: bfa init failed
According to this https://bugzilla.kernel.org/show_bug.cgi?id=104191 bug report, bfa driver loads incorrect (old) firmware. Simple diagnostic command confirms it:
# modinfo bfa filename: /lib/modules/4.4.0-31-generic/kernel/drivers/scsi/bfa/bfa.ko version: 3.2.23.0 author: Brocade Communications Systems, Inc. description: Brocade Fibre Channel HBA Driver fcpim license: GPL firmware: ct2fw-3.2.3.0.bin firmware: ctfw-3.2.3.0.bin firmware: cbfw-3.2.3.0.bin [...]
So it’s version 3.2.3.0 instead of 3.2.5.1. The fast workaround for it could be:
modprobe -r bfa cd /lib/firmware mv ct2fw-3.2.3.0.bin ct2fw-3.2.3.0.bin-ORIG mv ctfw-3.2.3.0.bin ctfw-3.2.3.0.bin-ORIG mv cbfw-3.2.3.0.bin cbfw-3.2.3.0.bin-ORIG ln -s ct2fw-3.2.5.1.bin ct2fw-3.2.3.0.bin ln -s ctfw-3.2.5.1.bin ctfw-3.2.3.0.bin ln -s cbfw-3.2.5.1.bin cbfw-3.2.3.0.bin modprobe bfa update-initramfs -u |
This fixed the problem indeed. I confirm that this also works on ubuntu 16.10.
Be aware that you have to redo this fix each time the linux-firmware package gets updated. You can avoid this by setting the package to hold (sudo apt-mark hold linux-firmware)
This is the kernel bug for this issue. Not much has happened.
https://bugzilla.kernel.org/show_bug.cgi?id=104191
Thank you for hint with linux-firmware package upgrade hold.