'CKR_MECHANISM_INVALID' Error using a Nitrokey Pro

Got a problem with Viscosity or need help? Ask here!


Posts: 2
Joined: Tue Mar 26, 2024 7:58 pm

Post by sebastian.hofmann » Tue Mar 26, 2024 8:34 pm

I’m trying to connect to an openvpn server running on an opnsense using an Nitrokey Pro and viscosity 1.11. I’m getting the following error:
Mrz 25 2:12:05 PM: Status auf Authenticating geändert
Mrz 25 2:12:09 PM: ERROR: An unknown error occured while trying to logon to this token. Please contact your device provider for further assistance.
Method C_SignInit returned CKR_MECHANISM_INVALID Retrying…
Mrz 25 2:12:12 PM: Status auf Trenne Verbindung (PKCS#11 Cancel) geändert
Mrz 25 2:12:12 PM: ERROR: PKCS#11 signing failed.
Mrz 25 2:12:12 PM: OpenSSL: error:0A080006:SSL routines::EVP lib:
Mrz 25 2:12:12 PM: TLS_ERROR: BIO read tls_read_plaintext error
Mrz 25 2:12:12 PM: TLS Error: TLS object → incoming plaintext read error
Mrz 25 2:12:12 PM: TLS Error: TLS handshake failed
Versions server:
opnsense 24.1.4
openssl 3.0.13
openvpn 2.6.9

Versions client:
windows 11 home
viscosity 1.11
opensc 0.25.0

Asking google, the problem seems to be, that current versions of openvpn and openssl use TLS 1.3 and this requires RSA-PSS, which might not be supported by the NitroKey.

On Linux I got the same error, but was able to connect with openvpn client, when downgrading the connection to TLS1.2 in the openssl.cnf:
SignatureAlgorithms = RSA+SHA512:ECDSA+SHA512:RSA+SHA384:ECDSA+SHA384:RSA+SHA256:ECDSA+SHA256"
MaxProtocol = TLSv1.2
MinProtocol = TLSv1.2
Now my question is, if there is a way to apply an openssl.cnf to the viscosity client? I already tried to apply the additional parameters
tls-version-max 1.2
tls-version-min 1.2
without any luck.

By the way, I also don’t see a way to force the server to use TLS1.2, since opnsense apparently has no config option for this.

Thanks in advance, Sebastian


User avatar
Posts: 2341
Joined: Thu Sep 04, 2008 9:27 pm

Post by James » Wed Mar 27, 2024 3:01 pm
Hi Sebastian,

Setting tls-version-max and tls-version-min should be sufficient to force the TLS version used by OpenVPN, however it won't necessarily cause OpenSSL to use ciphers/hashes in that TLS suite version it considers insecure.

TLS 1.2 also optionally supports RSA PSS if the client TLS session announces support for it (which will be happening with Viscosity 1.11 as it's part of OpenSSL 3's behaviour) and the server supports it (which yours should as it is also using OpenSSL 3). TLS 1.3 requires support for PSS.

Your Linux OpenSSL config file is essentially doing two things: forcing the use of TLS 1.2 (so making RSA PSS optional), and then disabling the use of RSA PSS by not including it in the list of allowed signature algorithms (e.g. "RSA-PSS+SHA256"/"rsa_pss_pss_sha256").

There are a number of things you can try:

1. Adjust the OpenSSL configuration file on the server to add the commands in your Linux OpenSSL configuration file above (i.e. force TLS 1.2 and specify a signature algorithms list without the PSS items). This has the added benefit that connecting clients shouldn't need to make any changes. A quick internet search seems to indicate this is possible on OpnSense.

2. Use TLS 1.1 instead. You may need to enable this on the server using tls-version-min. PSS isn't supported by TLS 1.1, so it'll definitely prevent PSS signing. Obviously there are security considerations to using older TLS versions.

3. Try lowering the OpenSSL security level to allow algorithms OpenSSL 3 considers insecure. Please note the security implications of doing so. The command for this is "tls-cipher "DEFAULT:@SECLEVEL=0"

It's unusual for a PKCS#11 token to not support PSS signing considering the age of TLS 1.2 and TLS 1.3. I would recommend seeing if the manufacturer of your token has a newer version available (either firmware or hardware) that has PSS support, or consider some of the alternatives available (Yubikeys seem quite popular among Viscosity's userbase).

Web: https://www.sparklabs.com
Support: https://www.sparklabs.com/support
Twitter: https://twitter.com/sparklabs


Posts: 2
Joined: Tue Mar 26, 2024 7:58 pm

Post by sebastian.hofmann » Thu Mar 28, 2024 11:52 pm
Hi James,

Thank you very much for your detailed explanations and your suggestions.

I ordered a Yubikey for a test and it works immediately without any problems. So in my opinion it is the best solution to switch to this token, since it works without any hacks on the opnsense and uses the state-of-the-art security features.

3 posts Page 1 of 1