Hi,
I'm trying to troubleshoot a problem with a test machine under VirtualBox. The guest OS is a minimal install of OL 7.6, configured by the OS installer to mount the root file system (sdb1) via iSCSI. The remote storage system is a Synology NAS. The /boot and swap partitions (sda1/sda2) reside on a standard local disk image.
So far it works except during reboot. I have to pause the boot menu for 30 seconds, otherwise the system fails to establish the iSCSI session. The problem appears to be that the iSCSI session is not disconnecting during shutdown.
I see the following during system start up:
Apr 18 03:54:26 localhost iscsid: iscsid: st_start 10.0.1.111:3260 0
Apr 18 03:54:26 localhost iscsid: iscsid: re-opening session 1 (reopen_cnt 0)
Apr 18 03:54:28 localhost iscsid: iscsid: Poll was woken by an alarm
Apr 18 03:54:28 localhost iscsid: iscsid: re-opening session 1 (reopen_cnt 0)
Apr 18 03:54:28 localhost iscsid: iscsid: connecting to 10.0.1.111:3260
Apr 18 03:54:28 localhost iscsid: iscsid: connected local port 34277 to 10.0.1.111:3260
Apr 18 03:54:28 localhost iscsid: iscsid: login response status 0000
Apr 18 03:54:28 localhost iscsid: iscsid: deleting a scheduled/waiting thread!
Apr 18 03:54:28 localhost iscsid: iscsid: connection1:0 is operational after recovery (1 attempts)
Journalctl reports the following during a reboot:
csiadm[10081]: iscsiadm: No matching sessions found
stemd[1]: Stopped Logout off all iSCSI sessions on shutdown.
csid[3723]: iscsid: pid 3723 caught signal 15
csid[3723]: iscsid: event_loop interrupted
csid[3723]: iscsid: pid 3723 caught signal 15
csid[3723]: iscsid: iscsid shutting down.
stemd[1]: Stopping Open-iSCSI...
stemd[1]: Stopped Open-iSCSI.
And journalctl -x shows:
Apr 18 04:24:27 localhost.localdomain iscsiadm[10081]: iscsiadm: No matching ses
Apr 18 04:24:27 localhost.localdomain systemd[1]: Stopped Logout off all iSCSI s
-- Subject: Unit iscsi-shutdown.service has finished shutting down
However, there is a session:
[root@localhost iscsi]# iscsiadm -m session
tcp: [1] 10.0.1.111:3260,0 iqn.2000-01.com.synology:ds1812.70.2 (non-flash)
I corrected the issue modifying the iscsi-shutdown.service by replacing logoutall=all with logout:
sed -i 's/logoutall=all/logout/g' /usr/lib/systemd/system/iscsi-shutdown.service
But now it shows:
csid[3718]: iscsid: Session is actively in use for mounted storage, and iscsid.s
csiadm[7609]: iscsiadm: Could not logout of [sid: 1, target: iqn.2000-01.com.syn
csiadm[7609]: iscsiadm: initiator reported error (28 - device or resource in use
csiadm[7609]: iscsiadm: Could not logout of all requested sessions
csiadm[7609]: Logging out of session [sid: 1, target: iqn.2000-01.com.synology:d
From what I understand, the iscsi session cannot logout while root is still mounted.
So I changed iscsi-shutdown.service attempting to run the iscsi-shutdown after dismounting the file systems:
Old:
After=systemd-remount-fs.service network.target iscsid.service iscsiuio.service
Before=remote-fs-pre.target
Wants=remote-fs-pre.target
New:
After=local-fs.target systemd-tmpfiles-setup.service
Before=sysinit.target shutdown.target
Now it runs much later during shutdown, showing no error, but it may not be doing anything at all. I can see on the Synology console that the session is still connected and timed out or aborted after about 30 seconds.
Is iSCSI some exotic set up, or why the trouble? I'm out of ideas for now.
Thanks!