Cloud Foundry foundation
Earlier this week there was an announcement about the Cloud Foundry foundation, which Adrian Coyle from Pivotal referred to as "the worst kept secret."
"We have been working on how we can open up Cloud Foundry as a foundation and put it in a place to be owned and developed as a shared industry asset. The hope is to have the foundation officially launched later this year."
Work is going into drafting bylaws for which there is a first draft. There is a lot of work to incorporate the foundation. Gathering feedback from the group is of paramount importance.
Chris Ferris from IBM said, "There have been some great posts and coverage [on the foundation news]. Everyone is very excited. It's an important next step for Cloud Foundry. It is an important project for Enterprise PaaS."
James Watters from Pivotal gave his thoughts, saying, "We wanted to show our intention to move to a foundation model. This opens up [PaaS] to companies who do not want to make a $100mill R&D investment. It was a big day for PaaS in general, because it was a crystallized message of all these companies moving in that direction. IBM did a really good job of getting customers out [for the announcement]. We need to do more of that. It was a great day for PaaS in general to be validated by all these people."
Chris indicated that, "We will be going through the process over the next few weeks for the outline of the structure of the organization and the governance model. We are looking at a model that we are calling 'inspired by Eclipse'. It is not exactly the same model, but we will have projects or programs that roughly map to what with we have today - with the services, runtimes, and BOSH."
"These projects will be run by a PMC committee", said Chris. "The influence of individual PMC members will be based on the resources they are putting into it. Technical guidelines will be set for the PMC members to follow. A technical committee will be elected by the development community. The board will have influence mostly over operations. For instance, the financial aspects, managing the budget, paying the staff, hiring the staff and branding."
Adrian said, "We envision the board having non-technical oversight. There will be a technical committee, whose primary responsibility will be to say how things are to be done. This includes coding standards, architectural guidelines and what should a contribution look like."
"Day-to-day a lot of power will be in the project teams. There will be an oversight structure for groups of related projects. This will be for life-cycles events, new projects coming through incubation, projects graduating from incubation and release reviews," said Adrian.
"In individual projects, where the work is being done, we are looking at two fundamental models that projects might use. One being an Agile or DOJO development flow and the other being more of a traditional distributed open-source team."
"How does the CAB play into all of this?", asked Chris. "As we transition into the foundation the CAB continues to give the users and operators of Cloud Foundry a voice."
James Bayer, Director of Product Management at Pivotal changed the discussion and introduced the "The dev side of things..."
CLI
Scott Truit, Product Manager at Pivotal, talked about what has happened with command-line interface since last month.
"Since the last call we went GA (general availability) with the 6.0.1 release. Most people are currently pulling from the edge release, but please use 6.0.1 unless you are actively doing development. We would prefer everyone be on dot release."
There is also a 6.1 roadmap that can be shared.
Iron Foundry Incubation
Brian Button from CenturyLink/Tier3 announced that Iron Foundry, the open-source project, would like to be entered into the Cloud Foundry incubation project.
Intel have been running Iron Foundry for some length of time and gave some feedback prior to the meeting. Iron Foundry have working code, but there are still some feature gaps. These gaps include Loggregator integration and the directory server that gives you the read-only file system. There is a desire for users to have the same experience whether on a Linux or a Windows stack, so filling these gaps would help with that. There is some "hardening" that is needed.
James Bayer said that the project is mature from his point-of-view and he is really happy about it.
James mentioned that he would like to make a small tweak to incubation project generally. If somebody has a signed CLA and already has working code, then they should be able to move that working code into the incubator prior to the next CAB meeting. There was no objection to this.
Cathy Spence from Intel asked "What new features are being worked on in Iron Foundry?"
"We're working on bringing it up to feature parity with Cloud Foundry," Brian replied. "There are a few things that Windows just can't do that are holding this back, but other than that feature parity is our initial goal."
Renat Khasanshyn from Altoros asked Brian about the work that has been done in the last 6-8 months since CFv2 came out, and if there is a SQL Server Broker in Iron Foundry. He suggested that Altoros might contribute to it.
Brian only joined the project at the beginning of the year, but he does know that a lot of work was put in during the second half of last year. This consisted of building the pieces that enabled Cloud Foundry to run on Windows, including the DEAs, the directory service, and Warden. This was done mid-way through last fall and was based on Cloud Foundry v154.
The current work that is being done is targeted at compatibility with v159. They are also working on refactoring components of the Cloud Foundry project, to ease future integration and make it easier for them to keep up with developments of Cloud Foundry core code.
Work is also being done to get things stable.
In the case of service brokers, Brian said that they are going to start working on implementing the service broker for SQL Server in the coming week. He expects that to be available very soon.
Iron Foundry's roadmap includes supporting more services throughout the rest of the year.
Renat stated they would like to be involved and would discuss further on the mailing list.
During the meeting IronFoundry.org posted Iron Foundry Now in Cloud Foundry Incubation Program on their blog.
Eclipse Tools
Ryan Morgan's Eclipse Tools were discussed. They have been around for a long time, but are not currently part of the Cloud Foundry project. The Cloud Foundry plugin for Eclipse has been developed under the SpringSource GitHub organization. It is preferable to bring this under the Cloud Foundry GitHub organization. It also makes sense to change the license to Apache 2, to be inline with the rest of the Cloud Foundry project.
Questions were raised as to whether this should go straight into the incubator or straight into the Cloud Foundry project? This project has unofficially been part of the Cloud Foundry project for a while and is only in need of a license change.
Making this change would enable people to contribute. IBM is excited to engage and contribute to this.
Intel's Cathy Spence said, "I'm in agreement with this approach. It's well established and I think we have little risk there."
Jeff Hobbs from ActiveState agreed. "A lot of people already consider this part of core."
Dr. Nic said, "It sounds great."
There were no objections.
Pull Requests On CF-Release
James Bayer discussed pull requests on the CF-release repository, around which there has recently been some confusion.
If you are contributing and sending pull requests to CF-release, then they will be first merged into a "develop" branch. This will go through a CI pipeline. If it comes out green from this pipeline, then (and only then) will it merged into master.
"We can make sure all code passes the tests before it makes it onto the master branch. This makes it a lot cleaner as it goes forward," said James. "Anyone checking out the master branch can be sure that the tests pass."
It does make it easier for the developers merging submitted changes into this repository if the changes are submitted directly to the "develop" branch. James clarified that this is desired, but not necessary, and that all changes will make it in, albeit via a more manual merge on their side.
There was a question as to whether contributors will still get attribution if a manual merge is required for non-develop pull requests. James did not think this would make any difference to attribution.
Open-Sourcing Releases
Matthew Kocher, Director of Engineering at Pivotal, said, "We are looking at separating the creation of final BOSH releases for CF release out of Pivotal's internal operations and making it part of the open-source code-base."
This will base cutting of the final releases on when all the features have been accepted and the test have passed. It is expected that there will be a lot more releases more often. For instance, BOSH releases are currently in the 1000s. These should be thought more of build numbers.
There is a strong desire to get new releases into the hands of committers as soon as possible, including their submitted pull requests.
Release Meta Data
Previously the releases have represented that Pivotal have made a deployment to their public instance on Amazon Web Services. Matthew described this as "mixing the streams." Instead, they hope to separate the two.
Dr. Nic suggested including some secondary meta data for "who is running what and what success they had."
Doug from IBM asked, "What version of all the dependencies should be guaranteed to work? Is there any plan to include that type of information in the meta data?"
Meta data around Cloud Foundry builds is being incrementally improved. Currently this information can only be found on the mailing-list which James admits is not ideal, but it is better than nothing.
One potential solution is "CF-Hub". This would be a place where anyone can annotate releases with additional meta data that would then be searchable. This would be a centralized repository with multiple vendors sharing common data. Obviously a format would need to be defined for expressing the meta data. James describes this as a "centralized source of truth" that any vendor can submit information to.
Doug suggested adding machine readable meta data during the build, such as an XML file. This would give all the information specific to that build.
Dr. Nic pointed out, "That would only give info on what happened in the CI," but he did agree that it is still useful.
Colin from Cloud Credo agreed with Doug's idea around machine-readable meta data.
The general conclusion was that it was a good start to have a wiki that anyone can edit with release announcements linking to the wiki.
Dr. Nic is looking forward to a fully automated build pipelines and hopes that Cloud Foundry continues on its current path of speeding it up. He suggested, "It would be nice if we can reduce the cost of owning Cloud Foundry. It would be good to know which version we should upgrade to."
Spiff Incubator Status
Guillaume Berche, a Software Architect at Orange, asked about the status of Spiff. This follows his attempts to get Cloud Foundry up and running using BOSH and upgrading via Spiff.
James Bayer said he still considers Spiff to be incubator quality. He said it was originally created to address the difficulty of creating a BOSH manifest. It was adopted by the Runtime Team, but work was never put into its design and he is not satisfied with its workflow.
James said that at the BOSH Summit there was talk about having something that was more aware of the BOSH release it was being included in. Meta data would include what things were required and what things are optional. This would also incorporate more validation.
"There were no shortage of opinions on this at the BOSH Summit," said James.
File System Service
Jeff Hobbs spoke about ActiveState's desire to open-source their file-system service, which has been a part of Stackato and has been battle tested in production. There were also questions, from Jeff, as to whether the incubation process will change as the Cloud Foundry foundation evolves.
Jeff discussed the sshfs quota system that is enabled if you put it onto a system that support quotas.
"The file-system service fits into the standard services subsystem," he explained. "Some applications, such as Wordpress or Drupal, need to share a portion of the filesystem and this service provides that. It is not a system for throwing everything in and should be limited to the files that need to be shared across the instances. It works well, but is definitely not designed for throwing in terabytes of data."
One limitation, expressed by Jeff, was that currently the file-system service is designed to work with Docker containers. Some work will be required to generalize this for use with Warden.
James Bayer suggested that the incubator is a great location for this.
NATS
James Bayer mentioned that there are some issues with using gnatsd in a clustered way, so it is currently being limited to single node usage. They are working closely with gnatsd's creators, Apcera, to resolve the issue.
Health Manager
James said that they are recommending that people use HM9000 (Health Manager 9000) for production use in multi-node mode. He suggested they may rename it to just "Health Manager."
On HA9000, Dr Nic said, "I like it! The code is really good and docs are easy to read."
Naming Confusion
There was discussion on whether components that were rewritten should have new names, or whether they should replace the previous versions.
There is confusion either way. A new project being named the same thing is hard to differentiate, but someone trying to understand which components make up Cloud Foundry finds it very hard when these names keep changing. Examples given were "Health Manager" to "HA9000", "DEA" to "Diego", and "Warden" to "Garden".
Buildpack Management
The latest cf command-line client now has enhanced buildpack management. This allows adding buildpacks with a priority order.
Prior to the latest release, the cf client allowed for buildpacks to be specified from a local directory. Now you can specify a git URL. This will work in the same way as a local file that you are uploading. Cloud Foundry will download the buildpack on the server-side. URLs can be found on GitHub on the right-hand side, where it says "Download a zip file".
James Bayer stated that IBM has helped them a lot with contributions and getting this feature rolled out.
CF Audit Events
When you type "cf events" the details you get are being slowly expanded. Currently user ids are seen in the JSON. James said they would like to get the full usernames, that come from the OAuth token, in there too.
Service Brokers
There are a lot of service brokers being written with v2, according to James Bayer. They are coming from open-source contributions and partners. James said that there are also many people talking to them about writing brokers.
He said that there is also work going into being able to pass arbitrary service-specific information to the broker.
BOSH Errands
BOSH Errands is a new capability that James said they are working on. This is a way to run "temporary jobs."
BOSH jobs are assumed to be running forever. For instance, if you just want to run a job that validates that the release is working as expected, or you are deploying a service broker and just want to register it with the Cloud Foundry instance, you can use BOSH Errands.
CPI Extraction
CPI extraction work is underway. Follow along on the BOSH tracker for more information on this.
Single Login Server
James Bayer informed us that the Identity Team have created a simpler set of repositories for the UAA and the login server. There is now only a single login server, so you do not have to track the Pivotal login server, Cloud Foundry login server and SAML login server.
This single login server can be skinned in different ways. There is a Pivotal skin and an open-source skin available.
LDAP and Active Directory
There is now the ability to configure LDAP and Active Directory in the UAA. James thanked Intel for helping move this along. This uses Spring LDAP under the covers.
There is also an IBM customer that is interested in incorporating OpenStack's Keystone for authentication.
Docker vs Warden
Jeff Hobbs mentioned a discussion that came up recently about Docker and Warden. ActiveState's Stackato uses Docker in preference to Warden. Jeff suggested that "The effort that is going into Warden could be going into Docker."
He also suggested that if the containerization usage was more configurable, we could open it up and allow people to vote with their feet.
Renat from Altoros agreed to the idea of considering Docker and suggested that it "Will help adoption, too."
Glyn Normington from Pivotal, pointed to a recent post he had written on Warden vs Docker and suggested that, "For production deployments, especially of supported commercial products, diagnostics are critical."
There is Google Document for the differences between Docker and Warden.
Dmitry also suggested hypervisor integration and that for full stack it may be preferable to use a lower level hypervisor rather than higher level virtualization.
Decker
The discussion quickly turned to Decker, which is something Colin at CloudCredo has been working on.
Decker is a new project that allows running a Docker stack in parallel with Warden, not replacing the Warden stack. This might allow for uploading Docker containers from the command-line and running them in parallel. The idea is that it can use much of the architecture of Cloud Foundry for scaling, orchestrating and utilizing Cloud Foundry services.
James Watters said, "I met with Docker's CEO to help get them involved with this project."
The general consensus was that it "seems reasonable to have alternative container technologies side-by-side."
TOSCA In BOSH
There was a question on whether there is a requirement for integrating TOSCA or CAMP into BOSH.
James Bayer said, "I've looked at CAMP before. I didn't really see it as a problem that needed solving at the time. I've talked to many users, but the need for OASIS never came up."
Conclusion
That's all from this month's Cloud Foundry Advisory Board Notes. Do not be surprised if the title for this regular update includes the word "foundation" in the near future.
We are already seeing some buzz on Twitter around "Decker". Is it stepping stone to Docker in Cloud Foundry?
The next meeting will be 26th March at 8am PST.