In my previous post about security related updates that are coming in Juno, I mentioned that Keystone itself is a poor identity management solution. I feel that this is a topic that deserves a more thorough discussion.
If you ask people familiar with OpenStack what Keystone’s purpose is, I’m willing to bet many of the answers include the term authentication. In my mind, Keystone’s main purpose is authorization within an OpenStack deployment. It achieves this by issuing authorization tokens to authenticated users. It’s true that Keystone can perform authentication itself, but that doesn’t mean that it should be used for authentication.
Let’s consider what Keystone natively offers from an authentication perspective. At it’s simplest, Keystone uses a SQL backend for identity. This involves storing a user entry that contains a hash of the user’s password. When a user authenticates to Keystone to obtain a token, they provide their clear-text password, which Keystone hashes and compares with it’s stored copy. This is a common approach for simple password based authentication. The big downsides here are that the authentication is a single factor, and the user actually has to send their clear text password over the network. The transport should of course be protected with SSL/TLS to prevent their password from being intercepted, but it’s still possible for an administrator on the Keystone system to get a user’s clear-text password by modifying Keystone or dumping memory contents. If a user uses this same password for other authentication systems outside of OpenStack, the user has opened up their exposure to attack from a malicious Keystone administrator. Most password-based authentication services also have additional capabilities that attempt to improve security. This includes capabilities such as:
password syntax and dictionary checking to prevent against weak passwords
password expiration to force a regular password change interval
password history to prevent against password reuse
account lockout after a fixed number of failed authentication attempts
Keystone itself has none of the capabilities mentioned above, and I feel that it’s not worth adding them given that these capabilities are widely available in services that are designed to be centralized authentication services. The typical answer for someone looking to have things like password policies with Keystone is to use LDAP as an identity backend instead of SQL.
Most people I talk to using OpenStack have their users stored in an existing centralized authentication service. This is typically an LDAP server, or Active Directory (which offers LDAP capabilities). Keystone has an LDAP driver for the identity backend to allow it to use LDAP for authentication and storage of users and groups. Let’s consider what benefits we gain over using the SQL backend from an authentication perspective.
Keystone’s LDAP code currently only supports the LDAP simple bind operation. This works exactly as Keystone’s SQL authentication does, which is described above. A user provides Keystone with their clear-text password, and Keystone then sends this clear-text password to the LDAP server, which then hashes the password and compares it to the stored hash to authenticate the user. This really doesn’t give us any benefit, as the user still gives up their clear-text password which is sent over the network. LDAP has support for stronger authentication mechanisms via SASL, but Keystone doesn’t have any support for SASL bind operations. Most LDAP servers have password policy and account lockout capabilities as described above, so this does provide some additional security over what Keystone itself offers. From an identity management standpoint, using an LDAP server also gets Keystone out of the business of user provisioning and maintenance and allows for a centralized identity source that can be shared with other applications.
Keystone has the ability to allow for stronger forms of authentication than simple password-based authentication via it’s “external” authentication method. When the “external” authentication method is used, Keystone expects that the web server that Keystone is running behind performs user authentication and supplies Keystone with the authenticated username via the “REMOTE_USER” environment variable. This allows one to run Keystone within Apache HTTPD, utilizing the myriad of available Apache authentication modules to allow for strong forms of authentication such as Kerberos or X.509 client certificates. This gets us away from the downsides of password-based authentication, as the user is no longer giving up their secret. A user entry is still stored within Keystone (or in an LDAP server that Keystone utilizes), but a password credential does not need to be stored. If we extract authentication out in this way, Keystone is really just responsible for authorization, which is the way it should be.
What about situations where you do not want to store all of your users within a single database that is used by Keystone? If we think about it, all that Keystone really needs to know is the identity of the authenticated user and any information that it needs to map OpenStack specific roles and projects to that user. Determining which roles to apply to a user is often based on group membership, though other attributes could be taken into account. Instead of requiring Keystone to handle the many variations of LDAP schema and LDAP connection management itself, it would be ideal to have the additional user information supplied in a simple form by the web server along with “REMOTE_USER”. This is exactly how Keystone’s federation extension works to provide SAML support. From Keystone’s perspective, it is simply provided with all of the pertinent information about an authenticated user along with the request for a token. Keystone is then able to map the user information it is provided with to the project and roles that apply to the user. This is much simpler for Keystone than having to go out and look that information up itself as it does when LDAP is used. This same model should be usable for any authentication method or identity source, not just SAML federation. A very nice benefit of this approach is that it makes Keystone itself agnostic of the authentication method.
There is a very useful Apache HTTPD module that was written by a colleague of mine called mod_lookup_identity that helps to eliminate the need for identity lookup functionality in Keystone itself. This module utilizes SSSD from the underlying platform to provide user and group information from various identity providers. If you’re not familiar with SSSD, it’s a robust set of client-side daemons designed to use identity providers such as FreeIPA, Active Directory, or LDAP for authentication and identity information. It provides some nice features that are a big step up from Keystone’s LDAP code, such as caching, connection pooling, and the ability to simultaneously provide identities from multiple sources. Using SSSD offloads quite a bit of complexity from Keystone, which leaves Keystone to focus on it’s main purpose of providing authorization tokens. This should result in a more scalable and performant Keystone. SSSD is available on most common Linux platforms (Fedora, RHEL, Debian, Ubuntu, OpenSUSE, Gentoo, etc.), so using mod_lookup_identity is a nice general purpose approach for Keystone to use.
A picture is worth a thousand words, so let’s see what this whole approach looks like.
What’s nice about this approach is that Keystone doesn’t know or care what authentication method was used or where the user information comes from. This means that you can simply swap out the Apache modules to support different methods of authentication. You want to use X.509 client-certificates for authentication? Configure mod_ssl instead of mod_auth_kerb. You want to use SAML? Configure mod_shib or mod_mellon. As far as Keystone is concerned, you would simply need to update the mappings to apply role assignments based on the way the environment variables are provided by the Apache modules.
A very common use-case for Keystone is to leverage users from Active Directory. With the current LDAP identity driver, this requires putting service users in Active Directory and potentially needing to create additional OpenStack specific groups. This is often a non-starter for many environments, as this additional information is not allowed to be added to Active Directory. The Juno changes I’ve described in my previous post will solve the service user portion of this, but not the groups portion. In addition, authentication is still restricted to LDAP simple bind. A much better solution is to use FreeIPA’s cross-realm Kerberos trust functionality to establish a trust relationship with Active Directory. This approach allows users from trusted Active Directory domains to access Linux/Unix resources that are managed in a FreeIPA realm. An Active Directory user can use their Kerberos TGT that was obtained when logging into their Windows system to be used to fetch service tickets for Kerberos enabled services that are set up in FreeIPA, such as Keystone and Horizon. OpenStack specific groups can also be defined locally in FreeIPA. These groups can contain external users and groups from Active Directory in addition to local users that are defined in FreeIPA.
SSSD has been designed to take advantage of the cross-realm Kerberos trust functionality in FreeIPA. SSSD is able to extract the group information contained in the PAC structure in the Kerberos tickets, performing the group lookups against FreeIPA. This allows FreeIPA groups that have external members (such as Active Directory groups) to be resolved, which results in a new PAC being generated that represents the combined group memberships from multiple identity sources. In effect, this functionality can be used to bridge multiple identity ‘silos’ without resorting to messy solutions that duplicate user data.
The area of cross-realm Kerberos trusts is complex, and likely a bit confusing if you are not already familiar with these areas. What it boils down to is that you can allow Active Directory users to use Kerberos single-sign on for authentication to Keystone, with the ability to create additional groups for role assignment purposes outside of Active Directory. In addition, you could create other accounts in FreeIPA which would also be able to authenticate via Kerberos. You could even establish multiple cross-realm trusts to allow users from multiple Active Directory forests to utilize the same Keystone server. Here is a revised version of the previous diagram that shows cross-realm Kerberos trusts in use with Keystone:
As you can see, offloading the authentication and identity lookup from Keystone can allow for some much more complex scenarios that are very useful for allowing existing identity providers to be used. It also provides some very nice security benefits by providing stronger forms of authentication than Keystone is currently capable of, some of which provide for a much nicer user experience than passwords due to single-sign on.