Installing the SSL certificates in ESX environment


I cannot say it is difficult to install trusted CA certificates in ESX environment. However, I’ve read many VMware KB articles to install them properly. Also, I’ve got one corrupted certificate, so the Troubleshooting section in this article is fully tested :o)

1. Validating the Certificate

In order to check out the certificates before or after the certificate installation, use a browser and navigate to the URL of vCenter Server or ESX(i) host server. Use the following URL

https://FQDN

because certificate name must match the server fully qualified domain name

1.1. If the Certificate Issuer is Unknown

If the certificate issuer is unknown, you will see the warning

Click on Continue to this website (not recommended)

On the next screen

Click on the Certificate Error button, then click on View certificates

Take a look at the certificate

It cannot be verified because it was issued by VMware Installer, which is not the trusted certification authority.
Click OK

1.2. If certificate issuer is well-known

If certificate issuer is well-known, you will see the VMware welcome screen without any warning

Click on the Security Report button, then click on View certificates

This certificate is issued by the well-known CA and has been successfully verified

2. Prerequisites

2.1. Required Software

Download and install on VMware vCenter Server the following software:
•    Microsoft Visual C++ 2008 Redistributable Package (http://www.microsoft.com/download/en/details.aspx?id=5582 or http://www.microsoft.com/download/en/details.aspx?id=2092 )
•    Open-SSL (http://slproweb.com/products/Win32OpenSSL.html )
•    VMware vSphere Command Line Interface
•    (http://communities.vmware.com/community/vmtn/server/vsphere/automationtools/vsphere_cli )
•    VMware vClient

Note: All applications can be installed on any Windows computer
Note: Microsoft Visual C++ 2008 Redistributable Package is prerequisite for OpenSSL installation

2.2. Software Configuration

1.    Find the openssl.cfg file in OpenSSL\bin folder (i.e. C:\OpenSSL-Win32\bin). Open it with WordPad and make the following changes in the [ req_distinguished_name ] section:

countryName            = Country Name (2 letter code)
countryName_default        = Your_Country_Code
countryName_min            = 2
countryName_max            = 2

stateOrProvinceName        = State of Province Name (full name)
stateOrProvinceName_default    = Your_State_Province_Name

localityName            = Locality Name (i.e. city name)

0.organizationName        = Organization Name
0.organizationName_default    = Your_Company

# we can do this but it is not needed normally 🙂
#1.organizationName        = Second Organization Name (eg, company)
#1.organizationName_default    = World Wide Web Pty Ltd

organizationalUnitName        = Organizational Unit Name
organizationalUnitName_default    = Your_OU

commonName            = Common Name (e.g. server FQDN or YOUR name)
commonName_max            = 64

emailAddress            = Email Address
emailAddress_max        = 64

2.    Save the openssl.cfg file

2.3. Generating the Certificate Signing Request

The Certificate Signing Request (CSR) can be generated on any computer. The process does not depend on the operating system the CSR is being generated for.

1.    Login to the computer with OpenSSL installed
2.    Open a command prompt and go to the OpenSSL\bin folder
3.    Execute the following command. Replace FQDN in this example with host fully qualified domain name:

openssl req –newkey rsa:2048 -nodes -out FQDN.csr -keyout FQDN.key -config C:\OpenSSL-Win32\bin\openssl.cfg

Loading ‘screen’ into random state – done
Generating a 1024 bit RSA private key
………..++++++
.++++++
writing new private key to ‘FQDN.key’
—–
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter ‘.’, the field will be left blank.
—–
Country Name (2 letter code) [Your_Country_Code]:
State of Province Full Name [Your_State_Province_Name]:
Locality Name (i.e. city name) []:Your_City
Organization Name [ Your_Company]:
Organizational Unit Name [Your_OU]:
Common Name (e.g. server FQDN or YOUR name) []:FQDN
Email Address []:your_email@company.com

Please enter the following ‘extra’ attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

4.    When OpenSSL finishes, find two files (FQDN.csr and FQDN.key) in the OpenSSL/bin folder and send them out to the Certification Authority to generate the certificate or use your local CA server.

Note: Keep FQDN.key file, it is needed to use with the generated certificate

3. Installing the SSL certificate on VMware vCenter

When the certificate has been received from the Certification Authority, unzip the file into a new folder. Usually there are four files in the folder, all with host FQDN names and different extensions

1.    Copy FQDN.cer into FQDN.crt
2.    Add FQDN.key (received as described in chapter 2.3. Generating the Certificate Signing Request, step 4) to the folder with other certificate files.
Now the folder with certificate files should contain the following files:

3.1. Converting the Certificate

Run OpenSSL to convert the certificate into PFX format

OpenSSL pkcs12 -export -in FQDN.crt -inkey FQDN.key -name rui -passout pass:testpassword -out FQDN.pfx

Note: The certificate name must be rui and the password must be testpassword. These values are hardcoded in vCenter. In fact, they are not, but it is not easy to find them in the configuration and not safe to change them

3.2. Installing the Certificate

1.    Rename all certificate files into rui.*. Keep their file extensions. The vCenter has the certificate file names hard coded and can use only rui.* name
2.    Log into vCenter server
3.    Go to the vCenter certificate folder

  •   for Windows 2003

C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\SSL

  • for Windows 2008

C:\ProgramData\VMware\VMware VirtualCenter\SSL
or
C:\Users\All Users\VMware\VMware VirtualCenter\SSL

4.    Create the OLDSSL folder and copy all the existing certificate files to it
5.    Stop the VMware VirtualCenter Management Webservices and VMware VirtualCenter Server services
6.    Copy new certificate files into SSL folder, replacing the existing files in the directory. Actually, only rui.crt, rui.key and rui.pfx are needed
7.    Start the VMware VirtualCenter Server and VMware VirtualCenter Management Webservices services

3.3. Fixing the Database Connection

Note: These steps are needed only for vCenter 4.0

If you try to start vCenter services after replacing the certificate files, they will fail with the service specific error.In vCenter Server 4.x logs, you will see errors similar to this:

[2012-04-20 10:11:02.751 07108 error ‘App’] [VpxKey::Decrypt] crypto failure: error:0406506C:rsa routines:RSA_EAY_PRIVATE_DECRYPT:data greater than mod len
[2012-04-20 10:11:02.751 07108 error ‘App’] [VpxdCert] Failed to decrypt password: applying key to encrypted data failed (likely the wrong key)
[2012-04-20 10:11:02.751 07108 error ‘App’] ODBC error: () –
[2012-04-20 10:11:02.751 07108 error ‘App’] Error getting configuration info from the database
[2012-04-20 10:11:02.751 07108 error ‘App’] [Vpxd::ServerApp::Init] Init failed: VpxdVdb::Init(Vdb::GetInstance(), false, false)
[2012-04-20 10:11:02.751 07108 error ‘App’] Failed to intialize VMware VirtualCenter. Shutting down…
[2012-04-20 10:11:02.751 07108 info ‘App’] Forcing shutdown of VMware VirtualCenter now

This issue occurs because the database password was encrypted using the certificate you replaced. To resolve this issue, re-enter the database password.

1.    Stop the VMware VirtualCenter Management Webservices and VMware VirtualCenter Server services
2.    Open command prompt window as an administrator
3.    Go to the C:\Program Files\VMware\Infrastructure\VirtualCenter Server folder
4.    Run this command to reset the database password:

vpxd.exe -p

Note: This command rehashes the passwords for the database users from the ODBC connection.
Note: Pay attention to the command output

[2012-04-20 12:52:18.129 03820 info ‘App’] Current working directory: C:\Program
Files\VMware\Infrastructure\VirtualCenter Server
[2012-04-20 12:52:18.129 03820 info ‘Libs’] HOSTINFO: Seeing Intel CPU, numCores
PerCPU 1 numThreadsPerCore 2.
[2012-04-20 12:52:18.129 03820 info ‘Libs’] HOSTINFO: numPhysCPUs is 0, bumping
to 1.
[2012-04-20 12:52:18.129 03820 info ‘Libs’] HOSTINFO: numCores is 0, bumping to
1.
[2012-04-20 12:52:18.129 03820 info ‘Libs’] HOSTINFO: This machine has 1 physica
l CPUS, 1 total cores, and 1 logical CPUs.
[2012-04-20 12:52:18.129 03820 info ‘App’] Log path: C:\ProgramData\VMware\VMwar
e VirtualCenter\Logs
[2012-04-20 12:52:18.144 03820 info ‘App’] Initializing SSL
[2012-04-20 12:52:18.144 03820 info ‘Libs’] Using system libcrypto, version 9080
CF
[2012-04-20 12:52:19.519 03820 info ‘App’] Vmacore::InitSSL: doVersionCheck = tr
ue, handshakeTimeoutUs = 120000000

Note: When prompted, enter the new password.

Enter new DB password:
again:
[2012-04-20 12:52:36.848 03820 info ‘App’] Reset DB password succeeded.

5.    Start the VMware VirtualCenter Server and VMware VirtualCenter Management Webservices services

3.4. Reconnecting the ESX(i) hosts to vCenter

1.    Verify if the certificate is valid. See chapter 1. Validating the Certificates earlier in this document
2.    Start vClient, connect it to the vCenter Server and switch to Hosts and Clusters view
3.    All hosts are labeled with a red cross that means that they lost connection to vCenter. It happened, because the SSL trust between vCenter server and a host has changed and a new trust relationship must  be established
4.    Right click on host and select Connect

5.    On the warning message

Click Close

6.    On the Specify Connection Settings screen

Enter host root account Username and Password and click Next

7.    On Security Alert screen

Click Yes

8.    Follow the Add Host Wizard, just checking the information and clicking Next. Click Finish at the end
9.    Repeat the steps for each ESX(i) host

4. Installing the SSL certificate on ESXi host

When the certificate has been received from the Certification Authority, unzip the file into a new folder. Usually there are four files in the folder, all with host FQDN names and different extensions

1.    Copy FQDN.cer into FQDN.crt
2.    Add FQDN.key (received as described in chapter 2.3. Generating the Certificate Signing Request, step 4) to the folder with other certificate files.
3.    Run vClient and connect to the vCenter, which manages the ESX(i) host
4.    vMotion all the virtual machines to other hosts or shut them down if they cannot be vMotioned
5.    Ensure that Lockdown Mode is disabled for the ESX(i) host

  • ESX 4.0

The option is in ESX(i) console

  • ESX 4.1

The option is on Configuration tab, Security Profile in vCenter

6.    Ensure that SSH is allowed. The option is on Configuration tab, Security Profile in vCenter.

  • ESX 4.0

Firewall Properties, SSH Server

  • ESX 4.1

Remote Tech Support (SSH)

7.    Put the ESXi host in maintenance mode
8.    Run VMware vSphere CLI
9.    Go to the bin folder

C:\Program Files\VMware\VMware vSphere CLI>cd bin
C:\Program Files\VMware\VMware vSphere CLI\bin>

10.    Run vifs.pl to copy the new FQDN.crt and FQDN.key files to the ESX(i) host. Enter the root account password when prompted

vifs.pl –server <FQDN or IP Address> –username root –put FQDN.key /host/ssl_key
Enter password:
Uploaded file FQDN.key to ssl_key successfully.

vifs.pl –server < FQDN or IP Address > –username root –put FQDN.crt /host/ssl_cert
Enter password:
Uploaded file FQDN.crt to ssl_cert successfully.

11.    Go to the ESX(i) host console

  • ESX 4.0

Scroll down to Restart Management Agents
Press Enter

  • ESX 4.1

Scroll down to Troubleshooting Options
Press Enter
Scroll down to Restart Management Agents
Press Enter.

12.    Press F11 to restart the management agents (vpxa etc).
13.    After the management agents are restarted press Escape a few times until you log out
14.    In vClient exit Maintenance Mode

15. Reconnect the host in vCenter

  • ESX 4.0

Reconnect host to the vCenter (see 3.4. Reconnecting the ESX(i) hosts to vCenter earlier in this document)

  • ESX 4.1

Reconnect host to the vCenter, if necessary (see 3.4. Reconnecting the ESX(i) hosts to vCenter earlier in this article). Usually vCenter 4.1 is able to re-establish the communication to the host

16.    Return the virtual machines vMotioned in step 4

5. Troubleshooting

5.1. Checking the Certificate Expiration Date on ESX(i) Host

To check the expiration date of SSL certificates on ESX(i) host

1.    Log in to an ESX(i) host console as the root user (locally or via SSH)
2.    Run the following command:

openssl x509 -noout -in /etc/vmware/ssl/rui.crt -enddate

3.    You see an output similar to:

# openssl x509 -noout -in /etc/vmware/ssl/rui.crt -enddate
notAfter=Apr 24 20:08:24 2025 GMT

5.2. Restoring the Default VMware Certificate on ESX(i) Host

If certificate is corrupted or expires, an ESX(i) host does not allow any communications to it but SSH or local login. In this case you may need to restore the default VMware SSL certificate

1.    Connect to the ESX(i) host using iLO/DRAC
2.    Log in to the ESXi host using Tech Support Mode. You can also try to use SSH login
3.    Run the following commands to regenerate the SSL certificate:

cd sbin
generate-certificates.sh

4.    Restart the management agents (See steps 12 to 14 in 4. Installing the SSL certificate on ESXi host)
5.    Check out that the VMware Installer certificate is presented (See 1. Validating the Certificate)

6. References

1.    Replacing vCenter Server 4.0 Certificates
2.    KB 1023688: Generating Domain Root CA signed certificates for vCenter Server
3.    KB 1003070: vCenter Server fails to start after replacing the default SSL certificates with custom SSL certificates
4.    KB 2015600: Determining the expiry period of default SSL certificates of vCenter Server and ESX/ESXi
5.    KB 1013632: vCenter Service Status does not work when SSL certificates are replaced
6.    KB 2006210: After upgrading to vSphere 5, you see the HA error: vSphere HA Cannot be configured on this host because its SSL thumbprint has not been verified
7.    KB 1030661: Replacing vCenter Server 4.1 SSL certificates using the vpxd -p command fails with the error: failed to do early initialization
8.    KB 1018624: Discovering the ESXi 4.0 host using EMC Control Center 6.1 fails with the error: SSL certificate verification failed
9.    KB 2013423: Locating and troubleshooting information about using SSL certificates in VMware Products
10. KB 1018624: Discovering the ESXi 4.0 host using EMC Control Center 6.1 fails with the error: SSL certificate verification failed
11. KB 1003677: Tech Support Mode for Emergency Support
12. KB 1003490: Restarting the Management agents on an ESX or ESXi Server

Advertisements

7 Responses to Installing the SSL certificates in ESX environment

  1. Could you not have just used “certreq -new” instead of the openssl install?

    • I’ve never used this one. I think any tool that can create a certificate request is good to be used

      • Yes, I agree… Yet I needed a solution that allowed a certificate request to be generated with out the need of installing a third party software. I am in no way taking away from your Guide, its perfect. I just happen to be in an environment that 3rd party software is always a hassle. The solution I used was simply using Windows command “certreq -new” and with the proper .inf file from the CA this worked flawlessly.

        I suppose I am just trying to say you could accomplish the same thing without installing

        -Microsoft Visual C++ 2008 Redistributable Package
        -Open-SSL

        —————–retype quotes (sample .inf) ——————-

        [Version]
        Signature=”$Windows NT$”

        [NewRequest]
        Subject = “CN=SERVER.CONTOSO.COM” ; For a wildcard use “CN=*.CONTOSO.COM” for example
        ; For an empty subject use the following line instead or remove the Subject line entierely
        ; Subject =
        Exportable = FALSE ; Private key is not exportable
        KeyLength = 2048 ; Common key sizes: 512, 1024, 2048, 4096, 8192, 16384
        KeySpec = 1 ; AT_KEYEXCHANGE
        KeyUsage = 0xA0 ; Digital Signature, Key Encipherment
        MachineKeySet = True ; The key belongs to the local computer account
        ProviderName = “Microsoft RSA SChannel Cryptographic Provider”
        ProviderType = 12
        SMIME = FALSE
        RequestType = CMC

        ——————————-end—————————————-

        Just thought it might help someone else.

  2. PaperCraft says:

    PaperCraft…

    […]Installing the SSL certificates in ESX environment « Notes of Windows Admin[…]…

  3. website visitors…

    […]Installing the SSL certificates in ESX environment « Notes of Windows Admin[…]…

  4. wechsel says:

    wechsel…

    […]Installing the SSL certificates in ESX environment « Notes of Windows Admin[…]…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: