Skip to Main Content
Forums

Infrastructure Software

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

Interested in getting your voice heard by members of the Developer Marketing team at Oracle? Check out this post for AppDev or this post for AI focus group information.

LACP PDU Timeout

DHeliosAug 19 2016 — edited Aug 22 2016

Hi, I have problem with Solaris 11.3 LAG technology and Juniper QFX5100 switches.

Sometimes solaris does not respond to LACP PDU packets in time and link aggregation become unavailable.

Solaris configuration:

The physical processor has 14 cores and 28 virtual processors (0-13,28-41)

x86 (GenuineIntel 306F2 family 6 model 63 step 2 clock 2600 MHz)

  Intel(r) Xeon(r) CPU E5-2697 v3 @ 2.60GHz

The physical processor has 14 cores and 28 virtual processors (14-27,42-55)

x86 (GenuineIntel 306F2 family 6 model 63 step 2 clock 2600 MHz)

  Intel(r) Xeon(r) CPU E5-2697 v3 @ 2.60GHz

root@srv-da-zfs-02:~# dladm show-aggr -x aggr3

LINK PORT SPEED DUPLEX STATE ADDRESS PORTSTATE

aggr3 -- 10000Mb full up 90:e2:ba:86:54:c0 --

       net6           10000Mb full   up        90:e2:ba:86:54:c0  attached

       net8           10000Mb full   up        90:e2:ba:86:5b:28  attached

root@srv-da-zfs-02:/etc/driver/drv# dladm show-aggr -L aggr3

LINK PORT AGGREGATABLE SYNC COLL DIST DEFAULTED EXPIRED

aggr3 net8 yes yes yes yes no no

-- net6 yes yes yes yes no no

root@srv-da-zfs-02:/etc/driver/drv# dladm show-aggr -P aggr3

LINK MODE POLICY ADDRPOLICY LACPACTIVITY LACPTIMER

aggr3 trunk L3,L4 auto passive short

root@srv-da-zfs-02:/etc/driver/drv# dladm show-vlan vlan616

LINK VID SVID PVLAN-TYPE FLAGS OVER

vlan616 616 -- -- ----- aggr3

root@srv-da-zfs-02:~# ipmpstat -a

ADDRESS STATE GROUP INBOUND OUTBOUND

:: down sc_ipmp3 -- --

zclu01-616-1 up sc_ipmp3 vlan616 vlan616

root@srv-da-zfs-02:~# kstat -p | grep alloc_fail | ggrep -v '0$'

root@srv-da-zfs-02:~#

Juniper Configuration

Switches uses MC-LAG technology.

jpqfx5100-1-sdn> show forwarding-options enhanced-hash-key

Slot 0

Current RTAG7 Settings

----------------------

Hash-Mode : layer2-payload

inet RTAG7 settings-

inet packet fields

   Protocol                        : Yes

   Destination L4 Port             : Yes

   Source L4 Port                  : Yes

   Destination IPv4 Addr           : Yes

   Source IPv4 Addr                : Yes

   Vlan id                         : No

jpqfx5100-1-sdn> show configuration interfaces ae2

description da-zfs02;

mtu 9216;

aggregated-ether-options {

minimum-links 1;

link-speed 10g;

lacp {

    active;

    periodic fast;

    system-id 00:01:02:03:04:05;

    admin-key 4;

}

mc-ae {

    mc-ae-id 2;

    chassis-id 0;

    mode active-active;

    status-control active;

    init-delay-time 2;

}

}

unit 0 {

family ethernet-switching {

    interface-mode trunk;

    vlan {

        members SDN\_DC\_STORE\_DVLP;

    }

}

}

jpqfx5100-1-sdn> show configuration interfaces xe-0/0/20

description ae2;

ether-options {

802.3ad ae2;

}

ERRORS

Juniper side

Aug 18 04:12:18.555 2016 jpqfx5100-1-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:18.554 2016 jpqfx5100-1-sdn mcsnoopd[1352]: krt_decode_iflogical: xe-0/0/20.0 has got color 0

Aug 18 04:12:18.680 2016 jpqfx5100-1-sdn lacpd[1321]: LACPD_TIMEOUT: xe-0/0/20: lacp current while timer expired current Receive State: CURRENT

Aug 18 04:12:18.684 2016 jpqfx5100-1-sdn /kernel: KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: cifd xe-0/0/20 - ATTACHED state - acting as standby link

Aug 18 04:12:18.684 2016 jpqfx5100-1-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:18.682 2016 jpqfx5100-1-sdn lacpd[1321]: LACP_INTF_DOWN: ae2: Interface marked down due to lacp timeout on member xe-0/0/20

Aug 18 04:12:18.683 2016 jpqfx5100-1-sdn rpd[1327]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:18.684 2016 jpqfx5100-1-sdn mcsnoopd[1352]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:18.684 2016 jpqfx5100-1-sdn mcsnoopd[1352]: krt_decode_iflogical: xe-0/0/20.0 has got color 0

Aug 18 04:12:19.274 2016 jpqfx5100-1-sdn /kernel: KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: cifd xe-0/0/20 - CD state - ready to carry traffic

Aug 18 04:12:19.274 2016 jpqfx5100-1-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:19.269 2016 jpqfx5100-1-sdn rpd[1327]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:19.275 2016 jpqfx5100-1-sdn mcsnoopd[1352]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:19.275 2016 jpqfx5100-1-sdn mcsnoopd[1352]: krt_decode_iflogical: xe-0/0/20.0 has got color 0

Aug 18 04:12:19.308 2016 jpqfx5100-1-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:19.310 2016 jpqfx5100-1-sdn mcsnoopd[1352]: krt_decode_iflogical: xe-0/0/20.0 has got color 2

Aug 18 04:12:26.278 2016 jpqfx5100-1-sdn lacpd[1321]: LACPD_TIMEOUT: xe-0/0/20: lacp current while timer expired current Receive State: CURRENT

Aug 18 04:12:26.283 2016 jpqfx5100-1-sdn /kernel: KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: cifd xe-0/0/20 - ATTACHED state - acting as standby link

Aug 18 04:12:26.283 2016 jpqfx5100-1-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:26.280 2016 jpqfx5100-1-sdn lacpd[1321]: LACP_INTF_DOWN: ae2: Interface marked down due to lacp timeout on member xe-0/0/20

Aug 18 04:12:26.281 2016 jpqfx5100-1-sdn rpd[1327]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:26.282 2016 jpqfx5100-1-sdn mcsnoopd[1352]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:26.282 2016 jpqfx5100-1-sdn mcsnoopd[1352]: krt_decode_iflogical: xe-0/0/20.0 has got color 2

Aug 18 04:12:26.336 2016 jpqfx5100-1-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:26.334 2016 jpqfx5100-1-sdn mcsnoopd[1352]: krt_decode_iflogical: xe-0/0/20.0 has got color 0

Aug 18 04:12:29.334 2016 jpqfx5100-1-sdn rpd[1327]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:29.338 2016 jpqfx5100-1-sdn /kernel: KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: cifd xe-0/0/20 - DETACHED state - will not carry traffic

Aug 18 04:12:29.338 2016 jpqfx5100-1-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:29.335 2016 jpqfx5100-1-sdn mcsnoopd[1352]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:29.335 2016 jpqfx5100-1-sdn mcsnoopd[1352]: krt_decode_iflogical: xe-0/0/20.0 has got color 0

Aug 18 04:12:30.553 2016 jpqfx5100-1-sdn rpd[1327]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:30.553 2016 jpqfx5100-1-sdn mcsnoopd[1352]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:30.571 2016 jpqfx5100-1-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:30.569 2016 jpqfx5100-1-sdn mcsnoopd[1352]: krt_decode_iflogical: xe-0/0/20.0 has got color 2

Aug 18 04:12:31.542 2016 jpqfx5100-1-sdn rpd[1327]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:31.543 2016 jpqfx5100-1-sdn mcsnoopd[1352]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:32.559 2016 jpqfx5100-1-sdn /kernel: KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: cifd xe-0/0/20 - ATTACHED state - acting as standby link

Aug 18 04:12:32.559 2016 jpqfx5100-1-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:32.559 2016 jpqfx5100-1-sdn /kernel: KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: cifd xe-0/0/20 - CD state - ready to carry traffic

Aug 18 04:12:32.559 2016 jpqfx5100-1-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:32.554 2016 jpqfx5100-1-sdn rpd[1327]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:32.555 2016 jpqfx5100-1-sdn mcsnoopd[1352]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:32.555 2016 jpqfx5100-1-sdn mcsnoopd[1352]: krt_decode_iflogical: xe-0/0/20.0 has got color 2

Aug 18 04:12:32.558 2016 jpqfx5100-1-sdn rpd[1327]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:32.559 2016 jpqfx5100-1-sdn mcsnoopd[1352]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:32.559 2016 jpqfx5100-1-sdn mcsnoopd[1352]: krt_decode_iflogical: xe-0/0/20.0 has got color 2

Aug 18 04:12:32.565 2016 jpqfx5100-1-sdn rpd[1327]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:32.566 2016 jpqfx5100-1-sdn mcsnoopd[1352]: Decode ifd xe-0/0/20 index 741: ifdm_flags 0xc000

Aug 18 04:12:18.512 2016 jpqfx5100-2-sdn lacpd[2664]: LACPD_TIMEOUT: xe-0/0/20: lacp current while timer expired current Receive State: CURRENT

Aug 18 04:12:18.516 2016 jpqfx5100-2-sdn /kernel: KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: cifd xe-0/0/20 - ATTACHED state - acting as standby link

Aug 18 04:12:18.516 2016 jpqfx5100-2-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:18.514 2016 jpqfx5100-2-sdn lacpd[2664]: LACP_INTF_DOWN: ae2: Interface marked down due to lacp timeout on member xe-0/0/20

Aug 18 04:12:18.515 2016 jpqfx5100-2-sdn mcsnoopd[1353]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Aug 18 04:12:18.515 2016 jpqfx5100-2-sdn mcsnoopd[1353]: krt_decode_iflogical: xe-0/0/20.0 has got color 2

Aug 18 04:12:18.515 2016 jpqfx5100-2-sdn rpd[1327]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Aug 18 04:12:18.727 2016 jpqfx5100-2-sdn mcsnoopd[1353]: krt_decode_iflogical: xe-0/0/20.0 has got color 0

Aug 18 04:12:18.729 2016 jpqfx5100-2-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:19.271 2016 jpqfx5100-2-sdn /kernel: KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: cifd xe-0/0/20 - CD state - ready to carry traffic

Aug 18 04:12:19.271 2016 jpqfx5100-2-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:19.270 2016 jpqfx5100-2-sdn rpd[1327]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Aug 18 04:12:19.277 2016 jpqfx5100-2-sdn mcsnoopd[1353]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Aug 18 04:12:19.277 2016 jpqfx5100-2-sdn mcsnoopd[1353]: krt_decode_iflogical: xe-0/0/20.0 has got color 0

Aug 18 04:12:19.306 2016 jpqfx5100-2-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:19.305 2016 jpqfx5100-2-sdn mcsnoopd[1353]: krt_decode_iflogical: xe-0/0/20.0 has got color 2

Aug 18 04:12:26.287 2016 jpqfx5100-2-sdn lacpd[2664]: LACPD_TIMEOUT: xe-0/0/20: lacp current while timer expired current Receive State: CURRENT

Aug 18 04:12:26.291 2016 jpqfx5100-2-sdn /kernel: KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: cifd xe-0/0/20 - ATTACHED state - acting as standby link

Aug 18 04:12:26.291 2016 jpqfx5100-2-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:26.289 2016 jpqfx5100-2-sdn lacpd[2664]: LACP_INTF_DOWN: ae2: Interface marked down due to lacp timeout on member xe-0/0/20

Aug 18 04:12:26.320 2016 jpqfx5100-2-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:26.290 2016 jpqfx5100-2-sdn mcsnoopd[1353]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Aug 18 04:12:26.290 2016 jpqfx5100-2-sdn mcsnoopd[1353]: krt_decode_iflogical: xe-0/0/20.0 has got color 2

Aug 18 04:12:26.290 2016 jpqfx5100-2-sdn rpd[1327]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Aug 18 04:12:26.318 2016 jpqfx5100-2-sdn mcsnoopd[1353]: krt_decode_iflogical: xe-0/0/20.0 has got color 0

Aug 18 04:12:29.291 2016 jpqfx5100-2-sdn mcsnoopd[1353]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Aug 18 04:12:29.294 2016 jpqfx5100-2-sdn /kernel: KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: cifd xe-0/0/20 - DETACHED state - will not carry traffic

Aug 18 04:12:29.294 2016 jpqfx5100-2-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:29.291 2016 jpqfx5100-2-sdn mcsnoopd[1353]: krt_decode_iflogical: xe-0/0/20.0 has got color 0

Aug 18 04:12:29.291 2016 jpqfx5100-2-sdn rpd[1327]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Aug 18 04:12:30.554 2016 jpqfx5100-2-sdn mcsnoopd[1353]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Aug 18 04:12:30.554 2016 jpqfx5100-2-sdn rpd[1327]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Aug 18 04:12:30.583 2016 jpqfx5100-2-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:30.582 2016 jpqfx5100-2-sdn mcsnoopd[1353]: krt_decode_iflogical: xe-0/0/20.0 has got color 2

Aug 18 04:12:31.542 2016 jpqfx5100-2-sdn mcsnoopd[1353]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Aug 18 04:12:31.542 2016 jpqfx5100-2-sdn rpd[1327]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Aug 18 04:12:32.560 2016 jpqfx5100-2-sdn /kernel: KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: cifd xe-0/0/20 - ATTACHED state - acting as standby link

Aug 18 04:12:32.560 2016 jpqfx5100-2-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:32.560 2016 jpqfx5100-2-sdn /kernel: KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: cifd xe-0/0/20 - CD state - ready to carry traffic

Aug 18 04:12:32.560 2016 jpqfx5100-2-sdn /kernel: if_pfe_mcae_color_pfe_update: xe-0/0/20.0: need to send mcae color to pfe

Aug 18 04:12:32.555 2016 jpqfx5100-2-sdn mcsnoopd[1353]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Aug 18 04:12:32.555 2016 jpqfx5100-2-sdn mcsnoopd[1353]: krt_decode_iflogical: xe-0/0/20.0 has got color 2

Aug 18 04:12:32.555 2016 jpqfx5100-2-sdn rpd[1327]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Aug 18 04:12:32.558 2016 jpqfx5100-2-sdn mcsnoopd[1353]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Aug 18 04:12:32.558 2016 jpqfx5100-2-sdn mcsnoopd[1353]: krt_decode_iflogical: xe-0/0/20.0 has got color 2

Aug 18 04:12:32.559 2016 jpqfx5100-2-sdn rpd[1327]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Aug 18 04:12:32.568 2016 jpqfx5100-2-sdn mcsnoopd[1353]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Aug 18 04:12:32.568 2016 jpqfx5100-2-sdn rpd[1327]: Decode ifd xe-0/0/20 index 740: ifdm_flags 0xc000

Solaris side

root@srv-da-zfs-02:/etc/driver/drv# egrep "mpath|mac:" /var/adm/messages

Aug 18 04:12:30 srv-da-zfs-02.net.billing.ru mac: [ID 486395 kern.info] NOTICE: aggr2 link down

Aug 18 04:12:30 srv-da-zfs-02.net.billing.ru mac: [ID 486395 kern.info] NOTICE: vlan612 link down

Aug 18 04:12:30 srv-da-zfs-02.net.billing.ru in.mpathd[115]: [ID 215189 daemon.error] The link has gone down on vlan612

Aug 18 04:12:30 srv-da-zfs-02.net.billing.ru in.mpathd[115]: [ID 773107 daemon.error] All IP interfaces in group sc_ipmp0 are now unusable

Aug 18 04:12:30 srv-da-zfs-02.net.billing.ru mac: [ID 486395 kern.info] NOTICE: aggr3 link down

Aug 18 04:12:30 srv-da-zfs-02.net.billing.ru mac: [ID 486395 kern.info] NOTICE: vlan616 link down

Aug 18 04:12:30 srv-da-zfs-02.net.billing.ru in.mpathd[115]: [ID 215189 daemon.error] The link has gone down on vlan616

Aug 18 04:12:30 srv-da-zfs-02.net.billing.ru in.mpathd[115]: [ID 773107 daemon.error] All IP interfaces in group sc_ipmp3 are now unusable

Aug 18 04:12:32 srv-da-zfs-02.net.billing.ru mac: [ID 435574 kern.info] NOTICE: aggr3 link up, 10000 Mbps, full duplex

Aug 18 04:12:32 srv-da-zfs-02.net.billing.ru mac: [ID 435574 kern.info] NOTICE: vlan616 link up, 10000 Mbps, unknown duplex

Aug 18 04:12:32 srv-da-zfs-02.net.billing.ru in.mpathd[115]: [ID 820239 daemon.error] The link has come up on vlan616

Aug 18 04:12:32 srv-da-zfs-02.net.billing.ru in.mpathd[115]: [ID 561795 daemon.error] At least 1 IP interface (vlan616) in group sc_ipmp3 is now usable

Aug 18 04:12:32 srv-da-zfs-02.net.billing.ru mac: [ID 435574 kern.info] NOTICE: aggr2 link up, 10000 Mbps, full duplex

Aug 18 04:12:32 srv-da-zfs-02.net.billing.ru mac: [ID 435574 kern.info] NOTICE: vlan612 link up, 10000 Mbps, unknown duplex

Aug 18 04:12:32 srv-da-zfs-02.net.billing.ru in.mpathd[115]: [ID 820239 daemon.error] The link has come up on vlan612

Aug 18 04:12:32 srv-da-zfs-02.net.billing.ru in.mpathd[115]: [ID 561795 daemon.error] At least 1 IP interface (vlan612) in group sc_ipmp0 is now usable

LOAD AVG 1min when LACP TIMEOUT appeared. It only happend when system in hight load.

LACP_TIMEOUT01.PNG

Network Traffic

LACP_TIMEOUT02.PNG

Comments

Locked Post
New comments cannot be posted to this locked post.

Post Details

Locked on Sep 19 2016
Added on Aug 19 2016
#oracle-solaris, #solaris-11
1 comment
3,898 views