Upgrading from Havana to Juno - Under the Covers
Upgrade from one OpenStack release to the next is a daunting task. Experienced OpenStack operators usually only do so reluctantly. After all, it took days (for some - weeks) to get OpenStack to stand up correctly in the first place and now they want to upgrade it? At the last OpenStack Summit in Vancouver, it wasn't uncommon to hear about companies with large clouds still running Havana. Moving forward to Icehouse, Juno or Kilo was to be an epic undertaking with lots of downtime for users and lots of frustration for operators.
In Solaris engineering, not only are we dealing with upgrading from Havana but we actually skipped Icehouse entirely. This means we had to move people from Havana directly to Juno which isn't officially supported upstream. Upstream only supports moving from X to X+1 so we were mostly on our own for this. Luckily, the Juno code base for each component carried the database starting point from Havana and carried SQLAlchemy-Migrate scripts through Icehouse to Juno. This ends up being a huge headache saver because 'component-manage db sync' will simply do the right thing and convert the schema automatically.
We created new SMF services for each component to handle any of the upgrade duties. Each service does comparable tasks: prepare the database for migration and update configuration files for Juno. The component-upgrade service is now a dependency for each other component-* service. This way, Juno versions of OpenStack software won't run until after the upgrade service completes.
The Databases
For the most part, migration of the databases from Havana to Juno is straight-forward. Since the components deliver the appropriate SQLAlchemy-Migrate scripts, we can simply enable the proper component-db SMF service and let the 'db sync' calls handle the database. We did hit a few snags along the way, however. Migration of SQLite-backed databases became increasingly error-prone as we worked on Juno. Upstream, there's a strong push to do away with SQLite support entirely. We decided that we would not support migration of SQLite databases explicitly. That is, if an operator chose to run one or more of the components with SQLite, we would try to upgrade the database automatically for them but there were no guarantees. It's well documented both in Oracle's documentation and the upstream documentation that SQLite isn't robust enough to handle what OpenStack needs for throughput to the database.
The second major snag we hit was the forced change to 'charset = utf8' in Glance 2014.2.2 for MySQL. This required our upgrade SMF services to introspect into each component's configuration files, extract their SQLAlchemy connection string, and, if MySQL, convert all the databases to use utf8. With these checks done and any MySQL databases converted, our databases could migrate cleanly and be ready for Juno
The Configuration Files
Each component's configuration files had to examined to look for deprecations or changes from Havana to Juno. We started off simply examining the default configuration files for Juno 2014.2.2 and looking for 'deprecated'. A simple Python dictionary was created to contain the renames and deprecations for Juno. We then examine each configuration file and, if necessary, move configuration names and values to the proper place. As an example, the Havana components typically set DEFAULT.sql_connection = <SQLAlchemy Connection String>. In Juno, those were all changed to database.connection = <SQLAlchemy Connection String> so we had to make sure the upgraded configuration file for Juno brought the new variable along including the rename.
The Safety Net
"Change configuration values automatically?!"
"Update database character sets?!!"
"Are you CRAZY?! You can't DO that!"
Oh, but remember that you're running Solaris where we have the best enterprise OS tools. Upgrading to Juno will create a new boot environment for operators. For anyone unfamiliar with boot environments, please examine the awesome magic here. What this means is an upgrade to Juno is completely safe. The Havana deployment and all of the instances, databases and configurations will be saved in the current BE while Juno will be installed into a brand new BE for you. The new BE activates on the next boot where the upgrade process will happen automatically. If the upgrade goes sideways, the old BE is calmly sitting there ready to called back into action.
Hopefully this takes the hand-wringing out of upgrading OpenStack for operators running Solaris. OpenStack is complicated enough as it is without also incurring additional headaches around upgrading from one release to the next.