ssh certificate authentication fails

[WellRedPandit]

sshj fails where openssh succeeds without a password using two files private-key and private-key-cert.pub. What am I coding wrongly with sshj?

(if sshj does not support certificate authentication, am willing to contribute, if pointed in the right direction...)

Test

object Sshj {
  def ping() = {
    val ssh = new SSHClient
    ssh.loadKnownHosts()
    ssh.connect("555.16.13.210", 26)
    try {
      ssh.authPublickey("root","~/.ssh/private-key")
      val session = ssh.startSession()
      try {
        val cmd = session.exec("ping -c 1 google.com")
        println(IOUtils.readFully(cmd.getInputStream()).toString())
        cmd.join(5, TimeUnit.SECONDS)
        println("\n** exit status: " + cmd.getExitStatus())
      } finally {
        session.close()
      }
    } finally {
      ssh.disconnect()
    }
  }
}

Scala REPL

scala> Sshj.ping
[INFO ] [15:05:15.179] [run-main-4] [n.schmizz.sshj.common.SecurityUtils] BouncyCastle already registered as a JCE provider
[WARN ] [15:05:15.195] [run-main-4] [net.schmizz.sshj.DefaultConfig] Disabling high-strength ciphers: cipher strengths apparently limited by JCE policy
[INFO ] [15:05:15.232] [run-main-4] [n.s.sshj.transport.TransportImpl] Client identity string: SSH-2.0-SSHJ_0_9_0
[INFO ] [15:05:15.236] [run-main-4] [n.s.sshj.transport.TransportImpl] Server identity string: SSH-2.0-OpenSSH_5.9
[DEBUG] [15:05:15.237] [run-main-4] [net.schmizz.concurrent.Promise] Setting <<kex done>> to `null`
[DEBUG] [15:05:15.237] [run-main-4] [n.s.sshj.transport.KeyExchanger] Sending SSH_MSG_KEXINIT
[DEBUG] [15:05:15.248] [run-main-4] [net.schmizz.concurrent.Promise] Setting <<kexinit sent>> to `SOME`
[DEBUG] [15:05:15.249] [run-main-4] [net.schmizz.concurrent.Promise] Awaiting <<kex done>>
[DEBUG] [15:05:15.249] [reader] [n.s.sshj.transport.KeyExchanger] Received SSH_MSG_KEXINIT
[DEBUG] [15:05:15.251] [reader] [n.s.sshj.transport.KeyExchanger] Negotiated algorithms: [ kex=diffie-hellman-group14-sha1; sig=ssh-rsa; c2sCipher=aes128-ctr; s2cCipher=aes128-ctr; c2sMAC=hmac-sha1; s2cMAC=hmac-sha1; c2sComp=none; s2cComp=none ]
[DEBUG] [15:05:15.290] [reader] [net.schmizz.sshj.transport.kex.DHG14] Sending SSH_MSG_KEXDH_INIT
[DEBUG] [15:05:15.301] [reader] [n.s.sshj.transport.KeyExchanger] Received kex followup data
[DEBUG] [15:05:15.301] [reader] [net.schmizz.sshj.transport.kex.DHG14] Received SSH_MSG_KEXDH_REPLY
[DEBUG] [15:05:15.326] [reader] [n.s.sshj.transport.KeyExchanger] Trying to verify host key with net.schmizz.sshj.transport.verification.OpenSSHKnownHosts@576882e3
[DEBUG] [15:05:15.327] [reader] [n.s.sshj.transport.KeyExchanger] Sending SSH_MSG_NEWKEYS
[DEBUG] [15:05:15.327] [reader] [n.s.sshj.transport.KeyExchanger] Received SSH_MSG_NEWKEYS
[DEBUG] [15:05:15.328] [reader] [net.schmizz.concurrent.Promise] Setting <<kexinit sent>> to `null`
[DEBUG] [15:05:15.328] [reader] [net.schmizz.concurrent.Promise] Setting <<kex done>> to `SOME`
[DEBUG] [15:05:15.328] [run-main-4] [net.schmizz.sshj.SSHClient] Key exchange took 0.091 seconds
[DEBUG] [15:05:15.328] [run-main-4] [net.schmizz.sshj.SSHClient] Attempting to load key from: ~/.ssh/private-key
[DEBUG] [15:05:15.342] [run-main-4] [net.schmizz.concurrent.Promise] Setting <<service accept>> to `null`
[DEBUG] [15:05:15.343] [run-main-4] [n.s.sshj.transport.TransportImpl] Sending SSH_MSG_SERVICE_REQUEST for ssh-userauth
[DEBUG] [15:05:15.343] [run-main-4] [net.schmizz.concurrent.Promise] Awaiting <<service accept>>
[DEBUG] [15:05:15.373] [reader] [net.schmizz.concurrent.Promise] Setting <<service accept>> to `SOME`
[DEBUG] [15:05:15.374] [run-main-4] [n.s.sshj.transport.TransportImpl] Setting active service to ssh-userauth
[DEBUG] [15:05:15.375] [run-main-4] [net.schmizz.concurrent.Promise] Setting <<authenticated>> to `null`
[DEBUG] [15:05:15.375] [run-main-4] [n.schmizz.sshj.userauth.UserAuthImpl] Trying `publickey` auth...
[DEBUG] [15:05:15.376] [run-main-4] [n.s.s.userauth.method.AuthPublickey] Attempting authentication using PKCS8KeyFile{resource=[PrivateKeyFileResource] ~/.ssh/private-key}
[DEBUG] [15:05:15.437] [run-main-4] [net.schmizz.concurrent.Promise] Awaiting <<authenticated>>
[DEBUG] [15:05:15.441] [reader] [net.schmizz.concurrent.Promise] Setting <<authenticated>> to `false`
[DEBUG] [15:05:15.442] [run-main-4] [n.schmizz.sshj.userauth.UserAuthImpl] `publickey` auth failed
[INFO ] [15:05:15.443] [run-main-4] [n.s.sshj.transport.TransportImpl] Disconnected - BY_APPLICATION
[DEBUG] [15:05:15.443] [run-main-4] [n.schmizz.sshj.userauth.UserAuthImpl] Notified of net.schmizz.sshj.transport.TransportException: [BY_APPLICATION] Disconnected
[DEBUG] [15:05:15.443] [run-main-4] [n.s.sshj.transport.TransportImpl] Sending SSH_MSG_DISCONNECT: reason=[BY_APPLICATION], msg=[]
[DEBUG] [15:05:15.443] [run-main-4] [net.schmizz.concurrent.Promise] Setting <<transport close>> to `SOME`
[DEBUG] [15:05:15.444] [reader] [net.schmizz.sshj.transport.Reader] Stopping
net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods
  at net.schmizz.sshj.SSHClient.auth(SSHClient.java:217)
  at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:316)
  at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:365)
  at Sshj$.ping(basics.scala:59)
  ... 43 elided

OpenSSH

> ssh -v -p 26 -i ~/.ssh/private-key root@555.16.13.210
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 555.16.13.210 [555.16.13.210] port 26.
debug1: Connection established.
debug1: identity file ~/.ssh/private-key type -1
debug1: ssh_rsa_verify: signature correct
debug1: identity file ~/.ssh/private-key-cert type 5
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9
debug1: match: OpenSSH_5.9 pat OpenSSH_5* compat 0x0c000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: RSA 1c:e0:42:f3:af:6b:15:59:1c:19:8d:44:c8:f2:99:fb
debug1: Host '[555.16.13.210]:26' is known and matches the RSA host key.
debug1: Found key in ~/.ssh/known_hosts:1
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering RSA-CERT public key: ~/.ssh/private-key
debug1: Server accepts key: pkalg ssh-rsa-cert-v01@openssh.com blen 1150
debug1: ssh_rsa_verify: signature correct
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to 555.16.13.210 ([555.16.13.210]:26).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Last login: Thu Oct 16 14:47:53 2014 from 192.168.225.2
Unauthorized access to this system is not permitted.
User activity may be logged for auditing or other purposes.
UPS_FAKE=2
[14:48:15] #

[hierynomus]

Is it a PKCS8 format key file? Look at https://github.com/hierynomus/sshj/blob/master/src/main/java/net/schmizz/sshj/userauth/keyprovider/PKCS8KeyFile.java and associated test, and try load your data in the test, that would be a good first step.

[WellRedPandit]

Thank you for looking into this! Otherwise - way over my head...

I don't think it's a PKCS8 format key file issue, but I did as you suggested: the type test passes, the keys test fails (I pushed my tests into this repo).

I believe this to be a certificate authentication issue, because for openssh to succeed (i.e. to log in or run a remote command without a password) it has to have two files in ~/.ssh:
private-v2-key
private-v2-key-cert.pub
and I can't find a way to tell sshj to use both...

Please do let me know if there is anything else I can do to help in troubleshooting this...

[hierynomus]

Could you also provide a sample certificate, or provide the commandline to build one with?

[WellRedPandit]

Will try. I am just a user (given a key and a certificate; will try to hunt down those in the know).

[dkocher]

For reference we are tracking issue #8371 that sounds similar.

[hierynomus]

@dkocher Sounds pretty similar indeed... From what I gather in that issue, you also have no key/certificate yet to reproduce this?

[dkocher]

@hierynomus No, unfortunately not.

[WellRedPandit]

@hierynomus To clarify: I do of course have a key and a certificate, but, unfortunately, can't share...

[hierynomus]

@WellRedPandit I got that 😉. But I would like to get my hands on a sample certificate that shows this behaviour so that we can fix it. Or the ssh-keygen commandline to obtain one.

[WellRedPandit]

Got it. Will do my best. Our sysadmins are not very talkative :-(

[hierynomus]

@WellRedPandit Could you give me the header of both files? I mean the part between the dashes if you view the contents. From that I might already be able to deduce a lot on what's going wrong.

[WellRedPandit]

@hierynomus Did ask. Lightning fast response: "Under no circumstances!!!" - sorry...
Will share what I can as soon as I know it myself.

[hierynomus]

The header of both files contains no private data, it merely indicates what kind of key it is, i.e.:

-----BEGIN RSA PRIVATE KEY-----

[WellRedPandit]

This is what I see:

...-v2-key starts with:
-----BEGIN RSA PRIVATE KEY-----

...-v2-key-cert.pub starts with:
ssh-rsa-cert-v01@openssh.com

[hierynomus]

Ok, thx! Now I can finally find some stuff online using that string. You're using this method: http://blog.habets.pp.se/2011/07/OpenSSH-certificates

I need to look into that. It's pretty OpenSSH specific I think.