FCC: +outgoing
From: Michael Richardson <mcr@sandelman.ca>
To: <openssl-users@openssl.org>
cc:
Subject: configuring callbacks (or not) and SNI vs not... no shared cipher from server end
X-Attribution: mcr
X-Mailer: MH-E 8.6; nmh 1.7+dev; GNU Emacs 24.5.1
X-Face:  $\n1pF)h^`}$H>Hk{L"x@)JS7<%Az}5RyS@k9X%29-lHB$Ti.V>2bi.~ehC0;<'$9xN5Ub#
 z!G,p`nR&p7Fz@^UXIn156S8.~^@MJ*mMsD7=QFeq%AL4m<nPbLgmtKK-5dC@#:k
--------

EXEC SUMMARY:
     https://www.openssl.org/docs/manmaster/man3/SSL_set_tlsext_host_name.html
and: https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_client_hello_cb.html

are pretty vague.  I think that SSL_set_tlsext_host_name() is probably
intended to be used on the client to set the SNI, but I'm not sure.
The legacy cb function returns int, but it's values are not documented.
The newer cb function is better documented, but I'm a bit at a loss as to
what SSL_client_hello_get0_ext() extracts.  Is the CB expected to parse the
extensions itself?

I guess the point is that CB can set the server certificate to something
appropriate, or I think, it could just decide to ignore the SNI value
completely and force the certificate regardless.

What is the SNI functionality otherwise on the server?  Is there any support
for picking a certificate based upon the SNI name?

I am asking because I seem to have run into a situation where it does not
seem to do the right thing, but I'm not sure that the error that I'm getting
is really about selecting the right certificate, or if there is something
else going on.  Well, I'm pretty sure that the no shared cipher is wrong.

Things I have tried (described below):
1) making sure that I'm running 1.1.1, which has ECDSA support, and not
   getting 1.0 shared object by mistake (this has happened before)
2) making sure that the SubjectName contains the target SNI.
   (with working certificate, it does not matter to server if I use wrong name)
3) observed private key was in different (SEC1 vs PKCS8) format, tried
   switching that.

Many details at: http://www.sandelman.ca/tmp/certprob201905/



The long story
--------------

In testing of a rails-based HTTPS server I have typically just configured a
keypair without a lot of thought to the DN used for the server.  As I have
some (experimental) patches to openssl and ruby-openssl, I often struggle
with having the wrong shared object pulled in and then some things do not
work.  In particular, I would get a message about no shared cipher on the
server emitted when something linked in openssl 1.0.x rather than 1.1.x,
and I had configured an ECDSA keypair.  I'm not sure if it was precisely:

   140639813764864:error:1417A0C1:SSL routines:tls_post_process_client_hello:no
                            shared cipher:ssl/statem/statem_srvr.c:2131:

(1) which I'm now getting, but it was close to that. I seem to recall that the
file name mentioned was one that was in 1.0.x, but not in 1.1.x, which was
the clue that I had made a linking error.  I got this again yesterday while
testing, and wondered if I could excise 1.0.x completely from my laptop, and
finally found that it was only an old version of libpq5 that linked against
it, and an upgrade (via postgresql.org debian package) eliminated my ruby
process from ever linking 1.0.x.

Yet the error persisted.  I test with the hostname target of
"fountain-test.example.com", which I put into my /etc/hosts as ::2
(an alias on lo), and bind against.

The private keys are test keys, and I could share them if that was useful.

NOTE that both working and failing certificates are generated by ruby code.

(2) I went back to a known working situation where a locally generated
certificate with that name as the CN was present:
(file: http://www.sandelman.ca/tmp/certprob201905/working-cert.txt and .pem)

       Issuer: DC = ca, DC = sandelman, CN = fountain-test.example.com\0A
       Unstrung Fountain Root CA
       Validity
            Not Before: May  7 22:56:23 2019 GMT
            Not After : Jun  7 08:56:23 2019 GMT
       Subject: DC = ca, DC = sandelman, CN = fountain-test.example.com

And this one works regardless of what name I use to access it.  That is,
given:
    ::2             fountain-test.sandelman.ca fountain-test.example.com n3CE618.router.securehomegateway.ca

all three of:
%curl -k https://fountain-test.sandelman.ca:8443/version.json
{"version":"0.7","revision":"devel"}%
%curl -k https://n3CE618.router.securehomegateway.ca:8443/version.json
{"version":"0.7","revision":"devel"}%
%curl -k https://fountain-test.example.com:8443/version.json
{"version":"0.7","revision":"devel"}%

work.  Using what I think is a similar certificate:

/corp/projects/shg/shg_mud_supervisor/spec/files/product/Smarkaklink-n3ce618/jrc_prime256v1.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 840664151 (0x321b8457)
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: C = Canada, ST = Ontario, OU = Sandelman, CN = highway-test.example.com CA
        Validity
            Not Before: May  8 17:18:37 2019 GMT
            Not After : Dec 31 00:00:00 2999 GMT
        Subject: CN = n3CE618.router.securehomegateway.ca
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:56:83:ba:e9:0d:9f:08:f4:b3:67:26:9d:52:b2:
                    75:5a:47:a2:44:ad:12:14:e4:7e:23:71:84:9b:6c:
                    2b:f5:69:91:8e:b8:ed:ec:9b:ec:9c:02:2c:b5:03:
                    7e:58:30:95:50:35:5f:f8:83:32:89:d6:7b:88:63:
                    cf:bb:d3:96:c9
                ASN1 OID: prime256v1
                NIST CURVE: P-256

it fails:

dooku-[projects/shg/shg_reach](2.6.2) mcr 10035 %curl -k https://n3CE618.router.securehomegateway.ca:8443/version.json
curl: (35) Unknown SSL protocol error in connection to n3CE618.router.securehomegateway.ca:8443


...
dooku-[projects/shg/shg_mud_supervisor](2.6.2) mcr 10171 %bin/startshg
Using rack adapter
Thin web server (v1.7.2 codename Bachmanity)
Maximum connections set to 1024
Listening on ::2:8443, CTRL+C to stop
139838954243840:error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher:ssl/statem/statem_srvr.c:2131:

(3) I observed that one private key (working one) was in SEC1 format
("BEGIN EC PRIVATE KEY"), while the not working one was in PKCS8 format
("BEGIN PRIVATE KEY").

I tried converting:
   openssl ec -in key.pem -out keysec1.pem

%openssl ec -in key.pem -out keysec1.pem
read EC key
writing EC key
%cat keysec1.pem
-----BEGIN EC PRIVATE KEY-----
...

%cat key.pem
-----BEGIN PRIVATE KEY-----

but that did not help.