Diego Progress
Alex Suraci from Pivotal gave an update on Diego. They are currently working in AWS and are able to successfully stage applications within their development environment. They hope to move to a larger scale staging environment and then move forward to usage in production. They are looking to 2 orders of magnitude higher scaling, but are hoping to take it even further.
Diego Simulator is a project that is used to test Diego's algorithm. Alex said that this is the first area that they are looking at for bottlenecks. The simulators do not actually push any load.
You can find the source of Diego Simulator here and here. These repositories will likely be removed at some point, since the simulator was just an experiment, accordingly to Matthew Kocher.
Michael Behrendt from IBM asked what the current limit was if they were looking for higher scaling. Alex said that 100 Executors (similar to DEAs) is the current limit where things work well. At about 400 Executors, etcd starts to fall over due to the number of long-polling HTTP connections to etcd. These long-polling connections allow listeners to receive instant updates to etcd changes. They plan to switch to some other tool to handle this part. They have already seen that by doing so this problem with etcd goes away.
Within the next couple of months they hope to have production environments using Diego to stage applications, such as in Pivotal's environment on AWS where they host thousands of applications. Actually running the applications would be a later milestone, but proving the staging would pave the way for this.
There have been weekly updates from the Diego team to the vcap-dev@cloudfoundry.org mailing list.
Epics and Inceptions
The trackers for the various components of Cloud Foundry now have "epics" which show a more zoomed-out view of an area of development without having to read through all the tasks and see how they fit together.
James Bayer from Pivotal said Scott Truitt's CLI team had an "inception" to plan a few months' worth of work. Notice of this was given on the mailing list with a plan so that feedback could be given. The inception involved their team meeting for 4-6 hours without phones or laptops. The result of the inception was posted on the vcap-dev@cloudfoundry.org list.
James was pleased with the way Scott's team did this openly with a week's notice for feedback and hopes this style of communication is adopted more widely across other Cloud Foundry teams.
Chris Ferris from IBM also praised Scott's handling of the inception, but requested that a little more notice be given in future. Chris wants to be involved in other inceptions and suggested possibly using a wiki for feedback, so that we do not have to dig through the mailing list to find these things. James agreed and said this would be similar to the design docs.
James also said that the next inception would be around May or June and we could start planning now. Scott noted that the Cloud Foundry Summit is actually taking place in June, so it might be possible do something there too.
Runtime Pull Request Burn-down
IBM has had people on-site with Pivotal to help in a concerted effort to get the pull request backlog burned all the way down or as much as possible. They started the week with 26 pull requests and the last count was 13.
James said that getting down to zero is not possible, since more pull requests have been coming in, but they hope to be mostly caught up by the end of the week.
Front-loading Contributions
James stated they there was a desire to "front-load" the conversations around contributions.
Significant changes to the code are sometimes initiated as pull requests. If they can be started in the form of a GitHub Issue, then it gives people a chance to discuss how those changes might be best shaped prior to the code being written. This would involve Mark Kropf agreeing that it is a product fit from a Runtime Product Manager's perspective and then having the Pivotal Engineering Team providing advice and guidance on the best implementation.
Some pull requests have been open for a while because of this lack of front-loading, said James. This is aggravating for the contributors and it also causes churn when these pull requests go on longer than necessary. This new process should reduce a lot of the back-and-forth.
Chris Ferris from IBM said this sounded good, but there should also be a little bit of flexibility in the system, for instance, if somebody is submitting a security patch. Chris gave the example of credentials in the logs via stack-traces that they had wanted to address immediately.
James said that a few lines of code should not be an issue and that this was more for the larger pieces of code where somebody needs to decide. For example, where the best place for it to live is. He would like to try to front-load as much as possible and then address the situations where it does not fit. James stressed that the Runtime Engineers have a lot of experience with the code-base. They know where it is going, where it has been and the patterns they are using to shape certain parts of it.
Matthew Kocher from Pivotal joined the conversation to say that this is not a change in policy in how contributions have to be made, but it is merely an offer from Pivotal's Runtime Team to help people get their changes in faster.
Disclosed Exploit
James Bayer said that IBM had discovered a vulnerability in one of the BOSH stem-cells that was exposed in Ubuntu last summer. There was a well known CVE in Ubuntu for this vulnerability. IBM disclosed this to Pivotal privately. This became a "privileged escalation".
The vulnerability meant that some untrusted code in a Warden container could get access to higher privileges on the Linux host that it was running on.
James said that they applied the kernel patch and then gave 48 hours advanced notice to public clouds who are running on the Cloud Foundry code base. They also gave a strong recommendation for applying the patch. After 48 hours, it was disclosed on the public mailing list.
The takeaway from this, said James, is that more formal processing is needed from the development team to review already existing security vulnerabilities. James said that this is something they will be doing.
The second thing that James found from this was that there was no formal list of public PaaS providers that were using Cloud Foundry. The list they compiled was based on personal knowledge. Therefore, they intend to add more structure around this, so that public PaaS providers can register themselves to receive early notifications on vulnerabilities.
James indicated that there will be a high bar for acceptance onto this list and they will need to verify that a public PaaS is actually being run. Obviously, we would never want those that aim to profit from exploits to have early access to this information.
For everyone else, this will be similar to the way other open-source projects handle this in the fact that everyone gets the information at the same time with the appropriate remediation steps. James acknowledged that this process is not as mature as he would like it to be.
Health Manager
James discussed the HM9000 project and said that they have been running it in production with no issue. They recommend that people start moving over to that. It may be renamed to just "Health Manager".
There was discussion about etcd and whether there was an issue with running a single instance of this with HM9000, but this seemed not to be the case. Although, it is recommended to run more than one instance of etcd in production.
"There will probably be a few bumps in the road, but we're running it in production and we're happy with it. We think we'll be able to work through all the issues and get everyone on this new code-base," said James.
BOSH
The BOSH Team have done a rewrite of the documentation and added a lot of new documentation for BOSH. This can be found at docs.cloudfoundry.org/bosh
James noted that documentation is all in Markdown format, so it is easy to edit. Elisabeth Hendrickson from Pivotal is involved in creating the documentation and said they appreciate either GitHub Issues or pull requests for changes.
CLI
The Ruby "cf" command-line will be deprecated, said James. This is also known as version 5 of the CLI. This will happen on April 4th and it will be moved to the attic.
So far, there have been no objections to this on the mailing list, but any strong opinions are still invited.
The cfoundry Ruby SDK for the REST API, which the Ruby cf client used will no longer be needed. It will either be passed off to a community maintainer or also moved to the attic. James said that anyone with a strong interest in maintaining this should announce themselves on the mailing list.
Version 6 of the CLI is written in Go.
Performance Acceptance Tests
Andy Bravery from IBM introduced "pat", which is a performance acceptance tool for which they have submitted a proposal for inclusion into the Cloud Foundry Incubator.
Andy said that they wanted something that did not rely on any other external tools and could be used as part of a build process, such as regression testing.
pat is written in Go.
Behind the scenes it will run experiments either against the cf CLI or the Cloud Foundry REST API. Over time, Andy hopes that they will support configuring a wide range of experiments that users can run. At the moment it is fairly limited set of experiments. This includes targeting, login and pushing applications. There is also a "dummy mode" which enables people to download, install and run it without actually requiring a Cloud Foundry cluster. You can run it against BOSH-lite or an actual Cloud Foundry install.
pat supports exporting results to CSV. In the future, Andy hopes that you will be able to do a diff against previous results to see any regressions that have been introduced. Multiple application support is also on the road-map. Currently, it will only push the one test application. Similarly, he wants to see the ability to log in as multiple different users.
It was asked what insights have been found with this tool, but Andy said it has not been run in any serious way to find any significant insights, yet.
James Bayer said that he has not had a chance to look much at the project yet, but he is glad it is in incubation. He thinks it will be useful to get baseline performance metrics of Cloud Foundry running on different environments, such as AWS.
Chris Ferris said it would be interesting to have something to catch performance regression before a release.
Elisabeth Hendrickson said she has not had a chance to look at this particular test-suite, but thinks it is a great concept. She said we have a fair bit of work to do before we are in agreement as to what we are testing.
Andy is looking for feedback and collaboration on this project. There is a Google Doc which gives more information.
Summit
The Cloud Foundry Summit takes place in San Francisco from June 9th to June 11th. Chris reminded everyone that there is a call for speakers and said that the conference will be roughly along the lines of what the 2013 PlatformCF conference was in the fall.
There will be 30 minute keynotes from some of the higher level sponsors. Chris said that there will be some other 30 minute talks, but it will be predominately 10-15 minute lightning talks. This is so they can get in as many talks as possible.
The talks will be followed by the unconference portion of the event.
Chris said he is keen to see this Summit as a way to augment the inception process (see "Epics and Inceptions" above).
James Bayer said that this year they will have multiple tracks. Last year they needed to ensure that all talks were appropriate for all audiences, since there was only a single track. This change will allow the conference to have some more low-level technical talks, while also have some very high-level talks regarding things like the adoption of Cloud Foundry.
Service Packs
Renat Khasanshyn from Altoros brought up the idea of "Service Packs", which would be similar to buildpacks. This would be a way to productize services.
Renat said that buildpacks have been very successful in standardizing the upper tiers of the stacks. He thinks the role of buildpacks is huge.
When it comes to service brokers, Renat thinks we can do better. This is not something that he sees as being limited to the Cloud Foundry community, but as a building block that could span the entire PaaS community. Renat says that this kind of standardization is something that customers want.
Renat's main goal is to bring this up for discussion and get ideas from other Cloud Foundry community members. He listed the following questions in the chat.
Questions:
1. Do you think other PaaS-es could benefit from a unified standard around PaaS Services?
2. What are the most common and biggest limitations of Service Brokers that you face today, feature wise?
3. What are the bottlenecks for adoption of Service Brokers in other PaaS-es?
Matthew Kocher from Pivotal gave his thoughts on the subject. Matthew said that they have been putting a lot of work into the v2 service broker API and trying to drive that standard forward. He said that they have seen a high rate of adoption from other people building service brokers, both open-source and commercially. The feedback has been really positive.
"The way to make it a standard is to drive adoption forward," said Matthew.
He conceded that there are parts of the v2 API that are still being fleshed out and they will make it more powerful as the versions increase. The majority of the use-cases are satisfied, but there are some use-cases that are not there yet. One example of services that are not there yet, is dealing with data services that take a long time to provision and are not multi-tenant.
Matthew pointed out that there is nothing Cloud Foundry specific in the v2 API and we could easily have other tools using it outside of Cloud Foundry, once there is enough critical mass adoption.
It was pointed out that service brokers as applications is going to become more popular. One major API difference between CFv1 and CFv2 was that the service broker no longer needs to talk to the Cloud Controller's API. It was this previous bidirectional communication that was preventing creating service brokers as applications. Now service brokers are only required to implement a set of inbound APIs.
Renat asked if other areas of the API need to be improved to fit more use-cases. The response was that the API has obviously been designed with Cloud Foundry in mind. It is based on the idea of binding a service to an application and giving that application unique credentials. That seems to be a model that has worked well up to this point.
In conclusion, a RESTful API seems to be the best way to go as a minimal point of integration and if it can work in a Cloud Foundry context, then there is no reason that it cannot also work well in other contexts.
Juju Charms
Renat gave an update on the work being carried out around Juju Charms, which is a joint effort currently being undertaken by Canonical, Pivotal and Altoros. The purpose is to bring Cloud Foundry to the Canonical and OpenStack ecosystem in the form of Juju Charm deployments. It is a bundle of Charms that together deploy all the components of Cloud Foundry. This uses the Juju deployment tool-chain as an alternative to BOSH. It is also an Ubuntu-native way of doing things, said Renat.
Teams from Canonical, Pivotal and Altoros are currently working in the Pivotal offices in San Francisco and Renat announced that there should be a release of these Juju Charms within the next couple weeks.
Renat intends to write a blog post with more details and a discussion of what this might mean for the Cloud Foundry community. He sees the biggest challenge as being that most deployments are currently using BOSH as a layer of deployment management. He sees the overlap of features as important. He also wants everyone to understand what it is, what it is not, where you use one or the other or where you use both BOSH and Juju Charms together. He intends to work with Canonical on helping to frame and communicate where this fits.
Dr Nic asked Renat if they are basing the Juju Charms off of the cf-release or whether they are building Cloud Foundry from source. Renat confirmed that they are building from source.
Manuel Garcia from Altoros, who is currently working in the Pivotal offices, said they are building the Ubuntu packages to match the cf-release. They are currently working on release v153 of Cloud Foundry, but they intend to also build other versions. To build them, they are matching the git commit ids for each component. He said that the first version will be a developer evaluation local deployment.
JBoss Buildpack
Colin Humphreys from CloudCredo said that there was some interest from the community in the JBoss buildpack. Therefore, they are moving it into a Cloud Foundry community repository. This is based on the Cloud Foundry Java buildpack.
Decker
Colin also gave an update on Decker, which was announced at the previous CAB meeting.
The repository for Decker is currently here, but will be transferred to the CloudCredo organization and then on to incubation as it becomes more usable.
Colin said that Decker is working, but there are "a few sharp edges". The dependencies are hard to resolve with the custom stem-cells being used. They are currently tidying it up and putting some documentation around it so that people can deploy it.
Colin hopes to talk about Decker at the CF Summit.
Conclusion
At the end there was some mention of Mesos for use in deployment. There are some git repos from cf-platform-eng and CloudCredo. I think it is likely we will hear more about this at the next meeting, which is scheduled for Wednesday 30th April at 8am PST.
Follow @ActiveState
Related Posts
Cloud Foundry Advisory Board Meeting - 2014 February
Earlier this week there was an announcement about the Cloud Foundry foundation, which Adrian Coyle from Pivotal referred to as "the worst kept secret."
"We have been working on how we can open up Cloud Foundry as a foundation and put it in a place to be owned and developed as a shared industry asset. The hope is to have the foundation officially launched later this year."
Work is going into drafting bylaws for which there is a first draft. There is a lot of work to incorporate the foundation. Gathering feedback from the group is of paramount importance.
Chris Ferris from IBM said, "There have been some great posts and coverage [on the foundation news]. Everyone is very excited. It's an important next step for Cloud Foundry. It is an important project for Enterprise PaaS."
Read more...
Cloud Foundry Advisory Board Meeting - 2014 January
Earlier today the Cloud Foundry Advisory Board met to discuss the Cloud Foundry project. Attendees, in no particular order, were present from IBM, Altoros, Pivotal, ActiveState, CenturyLink/Tier3, Intel, Ubuntu, Cloud Credo, Stark & Wayne, NTT and Piston. What follows is overview of the meeting, information that was shared and decisions that were made.
Read more...