UPDATE 9/29/12: Thanks to reader feedback, I've included a command line method to automate the submission of the CSR and saving of the certificate file from a Microsoft CA. Plus there's a new batch file at the very end of this post, in small text, that automates most of the certificate generation process for you.
Also, using the "pre-staging" method of placing trusted SSL certificates in specific directories prior to software installs seems like a reliable way to work around some issues/bugs, and requires less work as well. So go ahead and try the trusted SSL certificates, but only do the pre-staging option. Inventory Service still breaks for me when replacing SSL certificates POST install, but is fine with pre-staging.
UPDATE 9/24/12: After consulting with VMware tech support, and confirmed by user "Terafirma" on the VMware forums, I've updated the RSA private key generation process. vCenter relies on specific header information to recognize the key file, and if not present, will not use the private key. This of course leads to various fatal problems. Variables may include the version of OpenSSL you are using, so I've included screenshots of the 'bad' and 'good' headers so you can manually verify.
There is also an open question regarding the "KeyUsage" requirements for the trusted certificates. If you look at the VMware self-signed certificates many of them have "Data Encipherment" in addition to the standard "Digital Signature" and "Key Encipherment" options. If you are using a Microsoft CA with the standard "Web Server" certificate template it is NOT configured to issue certificates with "Data Encipherment" KeyUsage. I created a custom certificate template which can issue such certificates, just in case vCenter really needed it. I'm posing this question to VMware, so if I get a definitive answer I'll post an update. You can check out my blog post here for how to create such a custom certificate template.
Another interesting fact is that in the VMware certificate replacement documention for vSphere 5.1 they direct users to configure both serverAuth and clientAuth for the extendedkeyusage field. However, none of the VMware self-signed certificates have the clientAuth extended key usage, and the default Microsoft Web Server template will not issue certs with a clientAuth key field. So in the OpenSSL configuration files included below, I've removed the clientAuth field as I don't think it is needed and your CA probably won't issue a certificate with that usage anyway, without a custom template.
UPDATE 9/22/12: Due to numerous problems with the RTM release of vCenter 5.1 components regarding SSL certificates I've revised the SSL certificate replacement process. I want to acknowledge a lot of great work by the VMware forums user "Terafirma" for spending hours of looking through vCenter code and SQL databases to help debug all of the problems. The SSL certificate replacement process is quite painful, error prone and lengthy. If you want a fast install for your lab that has a high chance of succeeding, just skip installing all of the trusted SSL certificate steps and just use my blog posts for installing/configuring the vCenter Server 5.1 components.
It appears that using a single trusted SSL certificate for all vCenter services (SSO, vCenter, vCenter web client, inventory service) flat out will NOT work, even if they are installed on the same server. So don't think, "Hey I'm installing all the vCenter components on one server, so forget making lots of certificates I'll just use one." The "organizationalUnitName" needs to be unique for each service certificate, while the rest of the properties including subjectname can be the same. Viewing the VMware self-signed certificates confirms that the OU property is unique for each of the services that gets installed.
----
In Part 1 of my series on installing VMware vCenter 5.1, we got the SSO Service installed using a remote Microsoft SQL 2008 R2 database using an AD based service account. Next up you might think would be a blog post on installing the Inventory Service, but you would be wrong!
In production environments you should replace all of the default self-signed SSL certificates with trusted certificates. This is a tedious process, and each component has different ways of replacing the certificate. The VMware documentation for replacing SSL certificates in vSphere 5.1 is a bit lacking, to put it kindly.
Certificates are tricky, and depending on how you "mint" them (OpenSSL root CA, Microsoft CA, commercial CA, etc.) the steps may differ significantly. This blog series assumes a Microsoft Windows Server 2008 R2 Enterprise root CA that has the web certificate services installed.
The scope of this post will be only to create the SSL certificates for the vCenter 5.1 services. Follow-on posts cover the ins and outs of replacing the service specific certificates, as the process differs for each service.
To generate the required vCenter trusted certificates follow these steps:
1. Download and install the Windows OpenSSL binary. I would suggest the Win32 OpenSSL v1.0.1c (or later) package. This process requires OpenSSL 0.98r or later. Remember to install the appropriate Visual C++ Redistributable package prior to installing OpenSSL.
2. Install OpenSSL with the default path, and then create a directory called Certs in C:\OpenSSL-Win32. This directory is optional, but I find it helps me keep the subdirectories we need in one consistent place throughout all of the steps.
3. VMware requires that the vCenter related certificates contain a SAN (subject alternative name) field in them (which BTW is not required on ESXi hosts). To my knowledge this is a new requirement in vCenter 5.1. OpenSSL is not very user friendly, so we will need to create configuration file with all of the required parameters from which the certificate signing request (CSR) is made. The usual command prompt certificate generation does NOT prompt you for SAN fields, thus the requirement to create a custom configuration file.
Given the new information regarding the requirement for unique OU information in each certificate below are a series of OpenSSL configuration files for all of the certificates needed. Plus, with the need to create several certificates using configuration files make the process less error prone. You must change the subjectAltName and commonName fields. For the subjectAltName I used both the short name and FQDN of my vCenter server (e.g. D001VCTR02 and D001VCTR02.contoso.net). For the commonName I used the FQDN of the vCenter server (e.g. D001VCTR02.contoso.net).
The "organizationalUnitName" MUST be unique for each of the certificates, as I mentioned before. So feel free to change that property name, but make sure they are all unique. Since certificate filenames will all be the same, I created a folder structure as shown below, in my OpenSSL directory to put all of the certificate related files.
This five OpenSSL configuration files are listed below:
Inventory.cfg
[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:ServerShortName, DNS:Server.DomainName
[ req_distinguished_name ]
countryName = US
stateOrProvinceName = YourState
localityName = YourCity
0.organizationName = YourCompanyName
organizationalUnitName = InventoryService
commonName = Server.DomainName
SSO.cfg
[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:ServerShortName, DNS:Server.DomainName
[ req_distinguished_name ]
countryName = US
stateOrProvinceName = YourState
localityName = YourCity
0.organizationName = YourCompanyName
organizationalUnitName = SingleSignOn
commonName = Server.DomainName
vCenter.cfg
[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:ServerShortName, DNS:Server.DomainName
[ req_distinguished_name ]
countryName = US
stateOrProvinceName = YourState
localityName = YourCity
0.organizationName = YourCompanyName
organizationalUnitName = vCenterServer
commonName = Server.DomainName
WebClient.cfg
[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:ServerShortName, DNS:Server.DomainName
[ req_distinguished_name ]
countryName = US
stateOrProvinceName = YourState
localityName = YourCity
0.organizationName = YourCompanyName
organizationalUnitName = vCenter_Web_Client
commonName = Server.DomainName
VUM.cfg
[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:ServerShortName, DNS:Server.DomainName
[ req_distinguished_name ]
countryName = US
stateOrProvinceName = YourState
localityName = YourCity
0.organizationName = YourCompanyName
organizationalUnitName = vCenterUpdateManager
commonName = Server.DomainName
4. After you tweak and save your OpenSSL configuration files you need to generate the actual CSR files so you can submit that to your CA. First, we need to generate the RSA 2048 bit private keys.
WARNING: vSphere 5.1 is extremely picky about the format of the RSA private key file (rui.key) and you will most certainly have a smoking vCenter VM if you don't have the right format. If the header of the RSA key file only reads "-----BEGIN PRIVATE KEY-----" then Houston, you have a major problem.
The format of your RSA private key file should have a header of "-----BEGIN RSA PRIVATE KEY-----". This is the one and ONLY format that vSphere will accept.
Use the following OpenSSL command to create a file called rui.key. Run this in each directory, as each certificate should have a unique private key.
c:\OpenSSL-Win32\bin\openssl.exe genrsa 2048 > rui.key
Note: I've added a script at the bottom of this post that automates the entire certificate process, after you create the OpenSSL configuration files. Scroll down to the end if you wish to try that out instead of individual commands that are in the next several steps.
5. Using the RSA private key and the service-specific configuration file we need to generate CSRs (certificate signing request) for each service. Run the command below in each service certificate folder, changing the name of the configuration file for each invocation.
c:\OpenSSL-Win32\bin\openssl.exe req -out rui.csr -key rui.key -new -config inventory.cfg
6. After running both commands you should now see rui.csr and rui.key files in each service folder.
Create_CSR.bat
----
CD /d c:\OpenSSL-Win32\certs\vcenter
c:\OpenSSL-Win32\bin\openssl.exe genrsa 2048 > rui.key
c:\OpenSSL-Win32\bin\openssl.exe req -out rui.csr -key rui.key -new -config vcenter.cfg
CD /d c:\OpenSSL-Win32\certs\Inventory
c:\OpenSSL-Win32\bin\openssl.exe genrsa 2048 > rui.key
c:\OpenSSL-Win32\bin\openssl.exe req -out rui.csr -key rui.key -new -config inventory.cfg
CD /d c:\OpenSSL-Win32\certs\SSO
c:\OpenSSL-Win32\bin\openssl.exe genrsa 2048 > rui.key
c:\OpenSSL-Win32\bin\openssl.exe req -out rui.csr -key rui.key -new -config SSO.cfg
CD /d c:\OpenSSL-Win32\certs\VUM
c:\OpenSSL-Win32\bin\openssl.exe genrsa 2048 > rui.key
c:\OpenSSL-Win32\bin\openssl.exe req -out rui.csr -key rui.key -new -config VUM.cfg
CD /d c:\OpenSSL-Win32\certs\webclient
c:\OpenSSL-Win32\bin\openssl.exe genrsa 2048 > rui.key
c:\OpenSSL-Win32\bin\openssl.exe req -out rui.csr -key rui.key -new -config webclient.cfg
---
7. There are a couple of ways you can mint the SSL certificates from a Microsoft CA. One is using the traditional web interface. The other, which a reader pointed out in the comments to their article, is a command line method to automate the process.
To use the command line method you need to know the hostname of your CA, the "name" of your CA (see first screenshot below), and the certificate "template name" which may NOT the same as the certificate "template display name". The "template name" is usually the template display name without any spaces (see screenshot below).
In one of the certificate directories run the following command, of course changing the properties as needed for your CA. This will produce the rui.crt file, a newly minted SSL certificate.
certreq -submit -config "D001DC01\Contoso-D001DC01-CA" -attrib "CertificateTemplate:VMwareSSL" rui.csr rui.crt
If you want to create the certificate the old fashion method then open the first rui.csr file with NotePad and copy the contents to the clipboard.
8. To create the certificate submit the CSR to the Microsoft CA then download the certificate. Navigate to the homepage of the Microsoft CA and you should see a screen just like the one below. Select "Request a certificate."
A. Select "advanced certificate request."
B. Select the second option shown below:
C. Paste the CSR you generated from OpenSSL into the request window. Change the certificate template to "Web Server" or whatever you have defined as your SSL certificate template. If you only see "User" and "Basic EFS" options, then the account you are using lacks the proper permissions on the CA to request certificates. Use a more privileged account to perform the request.
D. Submit the certificate request (Base 64 encoded) then Save As and use a file of rui.crt and place it into the appropriate folder (e.g. C:\OpenSSL-Win32\Certs\Inventory) directory.
E. Further in this process we will also need the public root certificate file in Base-64 encoded format. One way to accomplish this is to select "Download Certificate chain" (in addition to Download Certificate) for one of your SSL certificates. Download this file as chain.p7b.
F. Double click the chain.P7B file the CertMgr snap-in should appear. Open the Certificates node then find the root CA certificate, right click, and export it. Save the Base-64 encoded file as Root64.cer. Your directory structure should look like the following:
8. To validate the certificate has all of the right fields, double click on the rui.crt file and it should open up. Click on the Details tab and verify that the "OU" sub-field of the Subject name matches the service for which you created the certificate for. I would also double check the Subject Alternative Name to ensure both the short hostname and the server's FQDN is listed.
9. Repeat the certificate minting process for the remaining services. Each folder should contain rui.csr, rui.key, rui.crt the OpenSSL configuration files.
10. To create the required PKCS#12 PFX files use the following command for each service:
C:\openssl-Win32\bin\openssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx
To make life a little easier below is a little batch file that automates the PFX creation process. Of course change any paths that you need to.
Create_PFX.bat
---
CD /d c:\OpenSSL-Win32\certs\SSO
C:\openssl-Win32\bin\openssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx
CD /d c:\OpenSSL-Win32\certs\Inventory
C:\openssl-Win32\bin\openssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx
CD /d c:\OpenSSL-Win32\certs\vCenter
C:\openssl-Win32\bin\openssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx
CD /d c:\OpenSSL-Win32\certs\VUM
C:\openssl-Win32\bin\openssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx
CD /d c:\OpenSSL-Win32\certs\WebClient
C:\openssl-Win32\bin\openssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx
----
11. To validate the PFX files were created correctly you can run the following command from one of the service certificate folders:
C:\openssl-win32\bin\openssl pkcs12 -in rui.pfx -info
When you run that command you will be prompted a few times to enter the password, which is 'testpassword'. Validate all of the fields look appropriate.
12. Copy your Certs\Root64.cer file to C:\ProgramData\VMware\SSL (you will need to create the SSL directory) and rename the file to ca_certificates.crt.
13. Next up is creating a trust store with your CA certificates, both the root and any intermediary CAs. vCenter requires certificate files with a specific name in a specific location. The file names of the associated certificate are in the format of hash.0 where hash is the result of the OpenSSL hash command (see below) and each file has an extension of 0 (zero).
14. Compute the hash for your base-64 encoded root certificate file (root64.cer), then copy the root64.cer certificate file to the C:\ProgramData\VMware\SSL directory, and change the file name to your hash value and change the file extension to 0 (e.g. 8c3f9174.0).
c:\OpenSSL-Win32\bin\openssl.exe x509 -subject_hash_old -noout -in c:\OpenSSL-Win32\certs\root64.cer
The output should be a short string of numbers and letter, as shown below.
15. Repeat the hashing process for any intermediary CAs, such that the full certificate chain has a set of files named with their hash value. You do not need to hash any other certificates that you have created. My example is shown below.
Now that we have generated all of the required key pairs and certificates we need to replace the Single Sign On and Lookup Service certificates. Those tedious steps are covered in Part 3.
UPDATE 9/29/12: Here is the batch file that will create the private RSA keys, create CSRs based on the OpenSSL configuration file, get a minted certificate from a MS CA, then create the PFX file with the VMware password. Fully automated! The Certreq command is very picky about the format of the CA name and certificate template. See the steps above for screenshots and more details on what to put there. It is not as obvious as it may first appear.
---
Set OpenSSL_BIN=c:\OpenSSL-Win32\bin\openssl.exe
Set Cert_Path=c:\OpenSSL-Win32\certs
Set Cert_Template=VMwareSSL
Set CA_Name=D001DC01\Contoso-D001DC01-CA
CD /d %Cert_Path%\vcenter
%OpenSSL_BIN% genrsa 2048 > rui.key
%OpenSSL_BIN% req -out rui.csr -key rui.key -new -config vcenter.cfg
certreq -submit -config "%CA_NAME%" -attrib "CertificateTemplate:%Cert_Template%" rui.csr rui.crt
%OpenSSL_BIN% pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx
CD /d %Cert_Path%\Inventory
%OpenSSL_BIN% genrsa 2048 > rui.key
%OpenSSL_BIN% req -out rui.csr -key rui.key -new -config inventory.cfg
certreq -submit -config "%CA_NAME%" -attrib "CertificateTemplate:%Cert_Template%" rui.csr rui.crt
%OpenSSL_BIN% pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx
CD /d %Cert_Path%\SSO
%OpenSSL_BIN% genrsa 2048 > rui.key
%OpenSSL_BIN% req -out rui.csr -key rui.key -new -config SSO.cfg
certreq -submit -config "%CA_NAME%" -attrib "CertificateTemplate:%Cert_Template%" rui.csr rui.crt
%OpenSSL_BIN% pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx
CD /d %Cert_Path%\VUM
%OpenSSL_BIN% genrsa 2048 > rui.key
%OpenSSL_BIN% req -out rui.csr -key rui.key -new -config VUM.cfg
certreq -submit -config "%CA_NAME%" -attrib "CertificateTemplate:%Cert_Template%" rui.csr rui.crt
%OpenSSL_BIN% pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx
CD /d %Cert_Path%\webclient
%OpenSSL_BIN% genrsa 2048 > rui.key
%OpenSSL_BIN% req -out rui.csr -key rui.key -new -config webclient.cfg
certreq -submit -config "%CA_NAME%" -attrib "CertificateTemplate:%Cert_Template%" rui.csr rui.crt
%OpenSSL_BIN% pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx