Infrastructure Software

The majority of my systems were built with u5. We've recently added a few x86 VMs and a few zones running u10 and I'm having ssh issues with them. When I try to ssh to them, I'm unable to login using our keys and get presented with the password prompt. I've been looking at previous posts similar to this situation, but none seem fitting. I've tried creating a new dsa key as a test and below are the results.

$ ssh -i id_dsa2 -vvv newuser@newserver
Sun_SSH_1.1.2, SSH protocols 1.5/2.0, OpenSSL 0x0090704f
debug1: Reading configuration data /opt/build/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Rhosts Authentication disabled, originating port will not be trusted.
debug1: ssh_connect: needpriv 0
debug1: Connecting to newserver [10.10.100.99] port 22.
debug1: Connection established.
debug3: Not a RSA1 key file id_dsa2.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: no key found
debug3: key_read: no space
debug3: key_read: no space
debug3: key_read: no space
debug3: key_read: no space
debug3: key_read: no space
debug3: key_read: no space
debug3: key_read: no space
debug3: key_read: no space
debug3: key_read: no space
debug3: key_read: no space
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: no key found
debug1: identity file id_dsa2 type -1
debug1: Remote protocol version 2.0, remote software version Sun_SSH_1.1.3
debug1: match: Sun_SSH_1.1.3 pat Sun_SSH_1.1.*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-Sun_SSH_1.1.2
debug1: use_engine is 'yes'
debug1: pkcs11 engine initialized, now setting it as default for RSA, DSA, and symmetric ciphers
debug1: pkcs11 engine initialization complete
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: en-US
debug2: kex_parse_kexinit: en-US
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug1: Failed to acquire GSS-API credentials for any mechanisms (No credentials were supplied, or the credentials were unavailable or inaccessible
Unknown code 0
)
debug1: SSH2_MSG_KEXINIT sent
debug3: kex_reset_dispatch -- should we dispatch_set(KEXINIT) here? 0 && !0
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: en-US
debug2: kex_parse_kexinit: en-US
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: gss-group1-sha1-toWM5Slw5Ew8Mqkay+al2g==,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: en-CA,en-US,es-MX,es,fr,fr-CA,i-default
debug2: kex_parse_kexinit: en-CA,en-US,es-MX,es,fr,fr-CA,i-default
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_init: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_init: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: Peer sent proposed langtags, ctos: en-CA,en-US,es-MX,es,fr,fr-CA,i-default
debug1: Peer sent proposed langtags, stoc: en-CA,en-US,es-MX,es,fr,fr-CA,i-default
debug1: We proposed langtags, ctos: en-US
debug1: We proposed langtags, stoc: en-US
debug1: Negotiated lang: en-US
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: Remote: Negotiated main locale: en_US.UTF-8
debug1: Remote: Negotiated messages locale: en_US.UTF-8
debug1: dh_gen_key: priv key bits set: 134/256
debug1: bits set: 1600/3191
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: check_host_in_hostfile: filename /opt/build/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 68
debug1: Host 'newserver' is known and matches the RSA host key.
debug1: Found key in /opt/build/.ssh/known_hosts:68
debug1: bits set: 1576/3191
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug3: kex_reset_dispatch -- should we dispatch_set(KEXINIT) here? 0 && !0
debug1: newkeys: mode 1
debug1: set_newkeys: setting new keys for 'out' mode
debug3: aes-128-ctr NID found
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: newkeys: mode 0
debug1: set_newkeys: setting new keys for 'in' mode
debug3: aes-128-ctr NID found
debug1: SSH2_MSG_NEWKEYS received
debug1: done: ssh_kex2.
debug1: send SSH2_MSG_SERVICE_REQUEST
debug2: service_accept: ssh-userauth
debug1: got SSH2_MSG_SERVICE_ACCEPT
debug1: Authentications that can continue: gssapi-keyex,gssapi-with-mic,publickey,password,keyboard-interactive
debug3: start over, passed a different list gssapi-keyex,gssapi-with-mic,publickey,password,keyboard-interactive
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Failed to acquire GSS-API credentials for any mechanisms (No credentials were supplied, or the credentials were unavailable or inaccessible
Unknown code 0
)
debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: id_dsa2
debug1: read PEM private key done: type DSA
debug3: sign_and_send_pubkey
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: gssapi-keyex,gssapi-with-mic,publickey,password,keyboard-interactive
debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password:

####################################
On the server
# /usr/lib/ssh/sshd -ddd
debug1: sshd version Sun_SSH_1.1.3
debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key.
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug3: Not a RSA1 key file /etc/ssh/ssh_host_dsa_key.
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug1: Server will not fork when running in debugging mode.
Connection from 10.10.100.70 port 42731
debug1: Client protocol version 2.0; client software version Sun_SSH_1.1.2
debug1: match: Sun_SSH_1.1.2 pat Sun_SSH_1.1.*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-Sun_SSH_1.1.3
debug2: Waiting for monitor
monitor debug1: list_hostkey_types: ssh-rsa,ssh-dss
monitor debug2: Monitor pid 3595, unprivileged child pid 3596
debug2: Monitor signalled readiness
debug1: use_engine is 'yes'
debug1: pkcs11 engine initialized, now setting it as default for RSA, DSA, and symmetric ciphers
debug1: pkcs11 engine initialization complete
debug1: list_hostkey_types: ssh-rsa,ssh-dss
monitor debug1: reading the context from the child
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: en-CA,en-US,es-MX,es,fr,fr-CA,i-default
debug2: kex_parse_kexinit: en-CA,en-US,es-MX,es,fr,fr-CA,i-default
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: GSS-API Mechanism encoded as toXXXXlw5Ew8Mqkay+al2g==
debug1: SSH2_MSG_KEXINIT sent
debug3: kex_reset_dispatch -- should we dispatch_set(KEXINIT) here? 0 && !0
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: gss-group1-sha1-toWMXXXXEw8Mqkay+al2g==,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: en-CA,en-US,es-MX,es,fr,fr-CA,i-default
debug2: kex_parse_kexinit: en-CA,en-US,es-MX,es,fr,fr-CA,i-default
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: en-US
debug2: kex_parse_kexinit: en-US
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_init: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug2: mac_init: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: Peer sent proposed langtags, ctos: en-US
debug1: Peer sent proposed langtags, stoc: en-US
debug1: We proposed langtags, ctos: en-CA,en-US,es-MX,es,fr,fr-CA,i-default
debug1: We proposed langtags, stoc: en-CA,en-US,es-MX,es,fr,fr-CA,i-default
debug1: Negotiated main locale: en_US.UTF-8
debug1: Negotiated messages locale: en_US.UTF-8
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received
debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
debug1: dh_gen_key: priv key bits set: 125/256
debug1: bits set: 1582/3191
debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
debug1: bits set: 1602/3191
debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
debug2: kex_derive_keys
debug3: kex_reset_dispatch -- should we dispatch_set(KEXINIT) here? 0 && !0
debug1: newkeys: mode 1
debug1: set_newkeys: setting new keys for 'out' mode
debug3: aes-128-ctr NID found
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: newkeys: mode 0
debug1: set_newkeys: setting new keys for 'in' mode
debug3: aes-128-ctr NID found
debug1: SSH2_MSG_NEWKEYS received
debug1: KEX done
debug1: userauth-request for user newuser service ssh-connection method none
debug1: attempt 0 initial attempt 0 failures 0 initial failures 0
debug2: input_userauth_request: setting up authctxt for newuser
debug2: input_userauth_request: try method none
Failed none for newuser from 10.10.100.70 port 42731 ssh2
debug1: userauth-request for user newuser service ssh-connection method publickey
debug1: attempt 1 initial attempt 0 failures 1 initial failures 0
debug2: input_userauth_request: try method publickey
debug1: temporarily_use_uid: 516/516 (e=0/0)
debug1: trying public key file /opt/newuser/.ssh/authorized_keys
debug3: secure_filename: checking '/opt/newuser/.ssh'
debug3: secure_filename: checking '/opt/newuser'
debug3: secure_filename: terminating check at '/opt/newuser'
debug1: matching key found: file /opt/newuser/.ssh/authorized_keys, line 2
Found matching DSA key: 3a:73:9a:93:dd:6e:ae:fb:08:05:e1:16:32:c5:b0:db
debug1: restore_uid: 0/0
debug1: ssh_dss_verify: signature correct
debug2: Starting PAM service sshd-pubkey for method publickey
debug3: Trying to reverse map address 10.10.100.70.
debug2: userauth_pubkey: authenticated 0 pkalg ssh-dss
Failed publickey for newuser from 10.10.100.70 port 42731 ssh2
debug1: userauth-request for user newuser service ssh-connection method keyboard-interactive
debug1: attempt 2 initial attempt 0 failures 2 initial failures 0
debug2: input_userauth_request: try method keyboard-interactive
debug1: keyboard-interactive devs
debug2: Starting PAM service sshd-kbdint for method keyboard-interactive
debug2: Calling pam_authenticate()
debug2: PAM echo off prompt: Password:
debug2: Nesting dispatch_run loop
Connection closed by 10.10.100.70
debug1: Calling cleanup 0x8065f39(0x80c7358)
debug1: Calling cleanup 0x806057f(0x80c8418)
debug1: Calling cleanup 0x80810c0(0x0)
monitor debug1: child closed the communication pipe before user auth was finished
monitor debug1: Calling cleanup 0x80810c0(0x0)
monitor debug1: Calling cleanup 0x80810c0(0x0)