One platform — standardized, built and operated by many.
One platform — standardized, built and operated by many.
The mission of the Sovereign Cloud Stack (SCS) is to define standards that enable to build and operate fully federated, state-of-the-art open source solutions that enables cloud service providers to deploy Infrastructure-as-a-Service and Container-as-a-Service in a sovereign manner, i.e. independent of other vendors. As such SCS is comprised of a set of standards as well as a reference implementation of these standards.
The word “Stack” already implies that there are several technical components layered upon each other to build the solution and they need to interact with each other with open standard protocols to construct a solution that is both lean and efficient towards delivering the advertised services, yet robust with respect to security and interoperability in the context of heterogenous existing IT infrastructures. The latter is important to stand a chance to be considered as a real new alternative to pre-existing alternative solutions, without the need to create an entirely new compound infrastructure from scratch, which would reduce attractiveness significantly for any operative cloud service provider (CSP).
One of the ingredients of a secure solution is identity and access management (IAM). Trivially you need identities for that and you don’t want to grow a new pool of identities for each new software solution. The preferred solution is to delegate the identity management to a central service, which offers open standard protocols for authentication and authorization, like OAuth 2.0 (and OpenID Connect on top), SAML or even plain LDAP. And there is a high probability that an operative cloud service provider already has a solution like that, just as well as the customer organizations that use its service may already are using an IAM solution of their choice, either on-premise or maybe hosted by yet some other CSP. This is the context where a solution like Sovereign Cloud Stack needs to be able to integrate into as seamlessly as possible.
That is the scenario, which the SCS SIG IAM (Sovereign Cloud Stack Special Interest Group for Identity and Access Management) had to consider when thinking about which software components, layers and protocols to make use of to get a implement a sane architecture for IAM aspects woven into the SCS fabric.
One of the central components of Sovereign Cloud Stack on the other hand is OpenStack. OpenStack has a component named Keystone that can issue OpenStack specific tokens (so called fernet tokens) in exchange for successful authentication. These tokens can then be used in the OpenStack cosmos of services to access APIs to manage cloud resources. Keystone can delegate the task of authentication to specialized IAM services, like LDAP in general or originally proprietary solutions like Active Directory. Keystone can also federate with other, external Keystone instances (“K2K”) and it can federate out towards Single-Sign-On (SSO) Identity Provider (IdP) solutions that speak one of the established SSO protocols like SAML, OIDC (OpenID Connect based on Oauth 2.0) or Shibboleth.
Quite early in the journey, the SCS “Operations and IAM” Team nourished the idea to use the latter option, i.e. to make Keystone federate out to an independent IdP solution that speaks these SSO protocols natively and supports corresponding flows for login, token renewal and logout. IdP solutions usually are also capable to do “Identity Brokering” – a term that is not very precisely defined and its interpretation is often specific to the software solution – which generally means that IdPs can be configured to again federate out to any number of external IdPs, forming a daisy chain. There are caveats and common misunderstandings connected to this term (and maybe I’m suffering from misunderstanding something too), so it may we worth to point out that the purpose of this daisy chained federation is that a user is guided towards the IAM solution that is authoritative for checking their identity. In this process the user is only expected to trust that final IAM authority to properly handle his or her password. The user is not expected to enter the password to some intermediary IdP which could be compromised and run around with the collected user credentials. That’s a common misunderstanding, frequently observable e.g. in a number of online payment service solutions which expect you to hand over the PIN for your bank account to them. With SCS we don’t want to offer inherently compromisable solutions like this to users. Anyway, there are several IdP solutions and a number of them is open source. The SCS project initially chose Keycloak as this IdP component for our first rounds of integration, but since we rely on open standard protocols, we may switch to a different solution here, if there are reasons to reconsider the initial choice. Solutions like these offer additional possibly desirable optional features, like 2nd factor authentication for specific usage contexts (specific clients or user groups) and some offer hooking up to an LDAP compliant backend directory, if desired. One of the core tasks of these solutions is to allow mapping attributes/claims from incoming tokens to other attributes/claims. This mapping can happen during federation or when issuing tokens towards clients. Btw, Keycloak uses the generic term “Client” for both, SAML Service Providers (SP) and OIDC Relying Parties (RP).
In summary from the perspective of an OpenStack API, it consumes a Keystone token to make access decisions and Keystone in turn consumes OIDC access tokens (JWT, Json Web Tokens) to decide if it should issue an OpenStack specific token. While Keystone supports hosting users locally (or in an LDAP backend), it can also federate out the request for and OIDC access token to an external IdP solution that supports protocols like OIDC or SAML. That secondary IdP is then external to Sovereign Cloud Stack and controlled by the respective customer. And it may be in turn backed by a customer specific IAM solution, like a directory service. The only thing that Sovereign Cloud Stack needs to offer, is a means to hook up OpenStack to a component that speaks one of the well established SSO protocols and that can in turn federate out towards whatever the customer sues to manage his user accounts. In this process SCS doesn’t see the user passwords, neither in plain text nor in hashed form. It only consumes derived credentials in the form of SSO tokens.
Sounds nice, and OpenStack also supports this kind of federation to “log in” and use Web-Services like Horizon. But, to make SCS usable for CSP business, it also needs to be able to use this kind of mechanism, or login flow when a customer wants to access the OpenStack APIs via non-web interfaces, like the openstack CLI client.
So the SCS SIG IAM started to dive into the nitty gritty details of authentication and authorization flows as well as attribute mappings. As a first step we started out with the triangle between OpenStack CLI, Keystone and Keycloak, without any “identity brokering” towards an external IdP. OpenStack (or rather the keystoneauth library of python modules) offers a range of authentication plugins and one of them, the v3oidcpassword was instrumental to make this triangle work. The goal was to be able to run openstack --os-auth-type v3oidcpassword token issue (and using some additional options to point it towards the SCS Keycloak IdP service), passing the name of a Keycloak local test user account “alice” and its plain text password to Keycloak, receive an access token in exchange and hand that over to Keystone to obtain an OpenStack specific token. To be clear again, this was the first step, and the second step would be to make something similar work with federation (and without handing over the clear text password to the SCS local Keycloak during the process). To cut a longer series of technical debugging sessions short, we managed to do this, by configuring the Keystone internal web proxy to allow using mod_oauth2 at and an alternative endpoint instead of the mod_auth_openidc that is used to serve Web-SSO requests. After succeeding with this minimal proof on concept and including the required configuration changes into the SCS repositories, we also contacted the Debian mainlines of mod_auth_openidc to propose also consider picking up mod_oauth2 for maintenance. Both modules are originally developed by the same author, Hans Zandbelt. These modules are now used in the SCS configuration for the Keystone container in the apache reverse proxy WSGI configuration.
During this journey the SCS SIG IAM repeatedly touched topics in the meetings of whether the limitations of apache WSGI handling would justify the effort to take the initiative to replace it by nginx. Discussions like these are easily possible with openstack components and they take time and carful consideration of pros and cons and the question of maintenance and if the upstream OpenStack project would also be interested and benefit from such a change. – I digress, but this also reflects how hard it is to maintain focus and use the invested developer resources wisely and for the benefit of all stakeholders of the project.
As the next, and currently latest step the SCS SIG IAM considered how openstack CLI could be used in the context of federated IdPs. At first we got stuck with the fact, that the v3oidcpassword auth plugin makes use of the “Resource Owner Password Flow” (summarized as “Direct Access Grant” in Keycloak) and Keycloak natively doesn’t support federation with this flow. We found a request in the Keycloak issue tracker (yeah, you guessed it, right, via StackOverflow) requesting implementation of this, but upstream developers were only moderately interested in investing into the implementation of this option. And, in hindsight, it makes sense to be skeptical regarding the sanity of this approach. As already mentioned above, that’s not really the idea of federation, to hand over your plain test password via “Resource Owner Password Flow” to the local Keycloak to let it in turn pass that on to the remote IdP. As a user, I want to be directed towards my “home” IdP and then return only the obtained token back to the SovereignCloudStack. So, implementing that option in Keycloak looked like a bad investment, which would also have the possible drawback that we would invest into this specific IdP solution rather than finding a solution that would leave it open to switch out this component in the future in case valid reasons come up for that.
When using OIDC for Web-SSO, technically the Authorization Code Flow is used. It’s called like that, because in exchange for successful authentication you (or rather your browser) receives an authorization code that is then handed over to a validated “redirect URI”, which can then exchange that for an access token. The details are off topic here, but the point is that there is no commonly established way to transfer the authorization code from the browser to a command line tool (running possibly in an entirely different context than the user browser).
With these puzzles in my head I traveled to Cologne to attend the SCS Hackathon hosted by plusserver. Not only did we get to have a guided tour through the elaborate technical details of professionally running a CSP infrastructure at high standards of technical security, but also we made proper use of the welcome “post”-COVID opportunity to have casual, spontaneous face to face exchange between the project participants. In one of these exchanges Joshua Mühlfort from Gonicus asked me for feedback for the architecture diagram that showed his plans for the Status Page application he was designing and implementing. Specifically he pointed out, that he considered to make use of the OAuth 2.0 “Device Authorization Flow”. This exchange alone made the effort of traveling to Cologne a time well spent (in the end it was a lot of time, thanks, Deutsche Bahn, it gave me ample of time to write down notes and start investigating). In the following sessions of the SCS SIG IAM meetings we intensified the frequency to weekly meetings and quickly were able to hack together a first implementation of the OAuth 2.0 “Device Authorization Flow” in keystoneauth. First tests without federation were successful (primarily in finding stupid bugs in my few lines of python code) and so we went on for the final PoC test which took place shortly before end of the year (actually it was the 31. of December), where we configured a kind of short cut federation between the standard “OSISM” realm in the testbed Keycloak and some fake “remote” realm, both served my the same Keycloak instance. This time we created the test user “bob” not in the primary OSISM realm, but in the “remote” realm. And instead of v3oidcpassword we now used openstack --os-auth-type v3oidcdeviceauthz token issue, where v3oidcdeviceauthz is the preliminary name of the new keystoneauth plugin implementation of the OAuth 2.0 “Device Authorization Flow”. Once you hit return, the openstack client context the appropriate endpoint
served by Keycloak and receives a device auth code in turn and a URL which the user shall visit. The plugin prints the URL including the code on the terminal and the user can right-click open it and is shown a login form and can choose to be redirected to the “remote” IdP instead. Once he does this and successfully authenticated there, the browser redirection dance takes place and the SCS local Keycloak obtains the token. All the while the openstack CLI auth plugin polls Keycloak at a Keycloak-specified frequency for the token, which is finally handed out to the openstack client. Finally the openstack client just does the final mile and contacts Keystone to exchange the access token for an OpenStack native token.
As next step we will propose our extension to keystoneauth for inclusion in the upstream project. That would be a win-win scenario, as other users may also find this new auth plugin useful and on the other hand we don’t have to maintain the patches locally in the SCS codebase.
And then, we will need get back to the topic of mapping attributes/claims and documenting how to configure Keycloak properly for this use case and make recommendations how to coordinate this from the external IAM to SCS to achieve the desired results. The general idea is to allow customers to self service configure the federation according to their needs. This in turn plays into the greater topic of what the desired role model would be and how to match that with the current OpenStack policy model. But that will be the topic of a future post.
Enjoy, Arvid Requate (Member SCS “Operations and IAM” Team)