Photo by Ryan Lerch
This talk was part of the Fedora Users’ and Developers’ Conference in Lawrence, Kansas this past weekend. A full video of the talk is available.
The Problem
Spot gave this talk; it was essentially a proposal to change the model under which we release Fedora – not how we actually produce the bits and push them out, but more of the logistics around the release schedule and what names we give various versions of Fedora.
The problem Spot was looking to solve with this problem is related to Fedora 18′s slippage. Fedora 18 had the longest slip in Fedora history – we were roughly two and a half months off of our schedule. One of the reasons for this is that we had a very large feature slated for Fedora 18 – the new Anaconda installer. Fitting something that large into a 6 month release cycle is challenging. If the installer ships broken, every part of the release is affected, so multiple slips were granted this release cycle to make sure we released with a solid and working installer. Slipping, though, causes a lot of problems, but so does missing out on opportunities to make large scale changes and feature developments for fear they won’t fit into the 6-month model. So what is the solution? How can we foster innovation in Fedora without continual and unpredictable slippage?
Longer Release Cycles Won’t Necessarily Solve the Problem
The obvious solution you might point out is to simply lengthen the release cycle. Spot recently hosted a Reddit thread asking for suggestions for Fedora in 2013 and one participant suggested exactly that: “In a similar vein, I would love to see the release cycle extended to 9 months like OpenSUSE does. Fedora always misses their release deadlines, so why not pad the release cycle by a couple months?”
Simply pushing the release cycle’s length outwards is not necessarily the right solution to enable larger, more cohesive feature development in Fedora. “I think if we push the release cycle out to 9 months,” Spot explained, “I think we’ll find ourselves in a similar situation, but slipping to 12 months. It’s not that the time window we’re shooting for is wrong, it’s more that we’re trying to shove too big things into that window of time.” You could keep expanding the time window to make it large enough to accomodate the largest feature you might want to put into a release, but then your release cycle would be unnecessarily long most of the time, users would have to wait far too long between releases to get upgrades, and you’d really have a whole new set of problems.
Rolling Release Models
Spot said pretty much up-front that he didn’t want to discuss rolling release models in this session, and that he had an alternate proposal he wanted to spend the majority of the session on. If you have an interest in exploring a rolling release model for Fedora, you can feel free to explore it or hold a hackfest or whatnot, but it was pretty much out for this talk.
(You can read some discussion about rolling release models and Fedora in the aforementioned Reddit thread.)
The Proposal
What if we changed the schedule to better handle large features that span multiple releases? Let’s take Anaconda’s new UI as an example feature for this proposed new model. The amount of work that Anaconda needed for the new UI really couldn’t possible have landed in just six months.
“If you didn’t know, and you looked from a distance at our feature page, you’d think that we were trying to land it in six months,” Spot said. “We need to better communicate to our community and our users what it means for these big features to land in Fedora, and how we indicate to them where they are in the cycle.”
The Red Hat Linux model
Spot then explained a little bit about Red Hat Linux’s old model. When Red Hat Linux 6.0 came out, you would expect that if you installed it later on, at the 6.2 release, it would be more polished, more feature-complete, and more stable than the initial .0 release. You’d expect the beginnings of big changes in a .0 release.
“Users knew there was a 6.1 coming after 6.0. People who wanted the newest and the greatest immediately could go to that 6.0, but users who weren’t quite willing to jump at that point could wait [for 6.1 or 6.2] and know that the corners would be polished a little bit nicer, etc.,” Spot explained.
Applying larger release cycles to Fedora
So what if we think about Fedora’s releases in terms of larger cycles, similar to the old way the Red Hat Linux cycles worked? We can stick with releases every 6 months – it works. Supporting each release for 13 months afterwards also works. We can continue with those same timelines. We have the community to continue maintaining that, and we’re able to put out enough updates for releases.
So let’s talk about this in specifics. What we would call Fedora 20, we’d call Fedora 20.0. What we’d normally call Fedora 21, we’d call Fedora 20.1. We’d go all the way to Fedora 20.3. Each release cycle includes four releases – each release gets 6 months, so it’s a 2-year release cycle total for the whole series or ‘cycle’ of releases. At the very beginning of the Fedora 20.0 planning process, we pitch big picture ideas and features (think systemd, think new Anaconda), and talk about how those ideas/features would fit across those 4 20.x releases. Which chunks of the feature would land in 20.0? Which parts would land in 20.1? (That’s not to say that any given feature would take all four point releases in a single 20.x release cycle to be feature complete. Some might just take one release, some might take two or three.)
Users coming into the 20.x cycle though, could look at the feature tracking for the 20.x series of releases and decide at which point they’d like to jump in. The earlier in the full release cycle a user jumps into installing Fedora, the more they can expect things to not quite be feature complete and not quite stable – the later on in a full release cycle a user jumps in, the more fully baked and polished they can expect it to be.
Longer-term planning enablement
The problem we have with our release model today, as Tom explained it, is that “we don’t think about Fedora++. We’re always only thinking about what Fedora is next; we have all of our focus on that one future Fedora release and not beyond that. Going to a cycle model forces us to have to be thinking bigger, and broader, from that perspective.”
Usability testing enablement
(Inkscape SVGZ source)
This model also would enable us to do better things like usability testing. For example, a feature could land in a .0 release, then go through usability testing, and the issues discovered during that process could be addressed and the feature improved for the .1 and .2 releases.
Without more users, we can’t grow more contributors. One path to growing more contributors for Fedora is to increase the user base from which we gain contributors. (See more discussion on this here.) A path worth pursuing in gaining more users is bettering the usability of Fedora itself, so a release cycle that better accommodates usability testing and improvements will, in this roundabout way, potentially help us gain more contributors.
“It’s hard to do usability improvements when every six months, everything is different,” Spot said. This proposal tempers the breakneck pace of how every six months, everything changes completely, hopefully without killing the excitement that comes from using an OS like Fedora, where new technology and features come out quite often first or at least towards the head of the pack.
Upgradeability
Upgradeability is a huge thing for users. We have to pair this plan with in-place upgradeability across the cycle release. If you start at Fedora 20.0, you should be able to in-place upgrade to Fedora 20.1. When 20.1 pushes, we tell you right on your desktop that an upgrade is available. Users will be given the choice to in-place upgrade to 20.1 and move to the next track, or stay on the 20.0 track and continue receiving updates for it until it’s no longer supported.
Planning Fedora releases in this new model
Let’s say we’re about to start planning for the Fedora 20 release cycle, which will be comprised of four releases:
Fedora 20.0 (6 month cycle, 13 months supported)
Fedora 20.1 (6 month cycle, 13 months supported)
Fedora 20.2 (6 month cycle, 13 months supported)
Fedora 20.3 (6 month cycle, 13 months supported)
When plan for Fedora 20, we won’t just be thinking about what we want Fedora 20.0 to look like at the end of its 6-month cycle. We’ll start talking and thinking about what we want Fedora 20.3 to look like, the last in the 2-year long Fedora 20 cycle. We can think bigger about where we want Fedora to look and feel like, and plan for the long-term future of Fedora as well as the immediate 6-month cycle.
We can commit towards big changes, set our sights high on the big problems that need to be fixed and start making progress on them, release-by-release. In this new model, we can unite around large features, large goals that we aim to accomplish over the entire cycle, not just in one 6-month period.
“There are always a large number of big problems that a lot of people are afraid to tackle because they know that there’s no way to fit them into one release. I think a lot of people are afraid to admit they don’t know how it would fit into multiple releases, either,” Spot explained.
The majority of our features are things that can be fit into 6-month releases. That’s completely fine – we’ll continue doing those. The big change here is that we’ll be enabled to think bigger-picture for larger changes, and we’ll have a model that makes it possible.
Questions
1. It seems that some users will only want to use x.3 or x.4 releases. They will want x.4 supported until the next x.4 comes out. What do we do to continue supporting them?
(Asked by Dennis Gilmore)
Spot pointed out that we’ll know if that expectation is true or not when we get there, “When we look at where we are at 20.2 / 20.3… if there are users who like the approach and want to treat the last release as an LTS [Long-Term Support] release, we can consider that.”
Spot was against promising some kind of LTS model as a hook, however. An LTS-style release model requires more significant resource investment than what this proposal would require – this proposal would require basically the same resources that today’s Fedora release model requires.
“It is worth considering a slightly longer cycle for that last release, though. Maybe 18 months,” Spot concluded. [Our current support term per release is 13 months.]
2. How will a user upgrade between minor releases without rebooting?
(I’m not sure who asked this.)
The proposal isn’t to allow upgrading between point releases without a reboot. The idea is more that you’d get a notification on your desktop that a new point release is available, and it might guide you into using fedup to perform the upgrade then and there.
One thing Spot pointed out here, is that something that would take longer than the period of time remaining in the cycle to be completed would not be accepted for that cycle, and would be pushed to the next cycle. So if you’re at the last release in the cycle, 20.3, and propose a feature like systemd that would take 2-3 releases to get done, we’d suggest waiting until 21.0 to release that feature.
3. Won’t we not have as much testing for updates in the older releases in the expanded release cycle, as users upgrade to the new point releases or stick only to the ones at the end of the cycle?
(Asked by Jon Dulaney)
Spot answered, “If you read what people write about Fedora – [they ask] ‘Is this the stable release, or should I wait a few more?’ People are already doing that kind of guesswork as to which is the ‘good’ release to stick on.” If we go to a cycle model, small pieces move from point release to point release and big pieces don’t move quite so much. So you’ll have a better idea of what the release will look like in the end, so there will be less random guesswork about which release is the ‘good’ release.
4. Would a KDE 3 to 4 upgrade then happen only as part of a x.0 release?
(Asked by Dennis Gilmore)
“We have to figure out where that line is,” answered Spot. “For each desktop SIG, we’ll have to figure out where that line is.”
Dennis pointed out, he didn’t mean just a point release for a desktop environment, but a major release, like KDE 3 to 4, or GNOME 2 to GNOME 3.
Spot didn’t want to nail it down to that level. He said we should look at what each desktop upstream’s plans are for the duration of the cycle and work that into our vision for the final release in that cycle. We’ll be going to the upstreams and asking them what their planning model looks like 2 years out – some of them will be able to answer that, some won’t be planned that far out. We have to figure out how we involve that into our plan.
If a desktop upstream decides to change something fundamentally in the middle of a cycle, and we see it as being disruptive, we can say we won’t adopt it in the current release cycle and that we will wait until the next cycle. That being said, we could potentially have a branch open for working on that next cycle depending on when in the cycle this disruptive change is proposed.
5. So there was no way to release Anaconda in the current release cycle model? There are features that have to land complete.
(Asked by Joe Brockmeier)
Spot pointed out that Anaconda didn’t land feature-complete in Fedora 18, and that there are features (advanced network storage device configuration is one of them) which will land in future iterations of the feature.
Adam Williamson pointed out that there are just some things that take longer than six months to get to a releasable point.
The Anaconda team tried to work outside of the release cycle for a long time, Peter Jones pointed out. It simply doesn’t work, you can’t do it, because you’re overwhelmed with additional changes you have to do to merge back into the tree. Six months is not long enough to do that kind of merging back. Peter also said that he failed to see how the proposal addressed that problem.
Spot conceded that Peter had a valid point, but that Anaconda as an example is an outlier. Peter agreed, but not as much of an outlier as Spot thought.
“Let’s look at systemd instead of Anaconda,” Spot answered. “It was developed across multiple releases, every new release had another new major set of functionality. We’re still not feature complete with it.”
Peter pointed out that, “not all of those new features were necessary to put the distro out.”
“That’s true,” replied Spot. “If we look at which features can we not release the distribution without that take longer than six month to get to a complete enough state to release, I think Anaconda might be the only one.”
6. What if I came up with a major anaconda- or systemd-sized thing, and you’re at 20.1 already. You’re telling me that I have to hold it for 18 months? Doesn’t that kill the whole benefit of the 6 month release cycle?
(Asked by Joe Brockmeier)
If you can do it in less than the amount of time available, you can still do it, Spot explained. The window narrows the further you go. If it’s a 12-month (or a 18-month) feature, there is time for you to do it at 20.1. This assumes that you could break the feature into 6-month chunks, and that the community has weighed the balance, to see if it’s okay to add to the long-term vision since it was proposed after the initial release cycle vision planning before the x.0 release.
However, as Spot explained, “eventually the point comes where the window because too narrow for anything longer than 6 months, but the wait of time you have is also less.”
7. In the Red Hat Linux days, most of the time a .3 happened, it wasn’t planned beforehand.
(Pointed out by Peter Jones)
Peter further suggested that, “If we move to a model like this, you don’t want to nail down, ‘there are x many each time.’ Instead, plan out what we intend to do and then figure out how many releases it would take to get there.”
Spot then drew a diagram on the board, which I’ve taken a number of liberties with:
(Inkscape SVG source)
“If we have an annual conference that is big and focused at each of these 6 month points,” Spot explained, pointing at the 20.0 planning, and 20.1 release, and the 20.3 release points on the diagram, “we end up with a conference schedule where the first annual conference is planning for the full release, a check-up checkpoint in the middle, and a post-mortem at the end and a start to the next cycle.” Spot explained this annual conference 3-checkpoint cycle is why he chose a 2-year long cycle.
8. What if we did something where even releases were stable releases, and odd releases were unstable?
(I’m not sure who asked this.)
Spot basically answered that he didn’t know that the value of that outweighs the complications it introduces, although it’s a neat idea.
9. We should look at what other linux distros with an LTS model have done well and where they’ve made mistakes.
(Suggested by Jon Masters)
(I couldn’t hear everything Jon was saying, I think he asked something about Anaconda too but the microphone just didn’t pick it up well enough to understand.)
Spot pointed out that we’ve had some major features, like GNOME 3, that shipped an initial version that, over additional iterations, matured and gained much more polish release-after-release.
I think Jon was concerned about working past the 20.0 release to continue feature development, and it was pointed out that there would be a rawhide-like branch for 20.1 open during the 20.0 cycle if need be.
(I apologize greatly that I couldn’t follow this part of the discussion very well. If someone was there live and can clarify, please do so in the comments and I’m happy to fix this section.)
10. Linux moves a lot faster than it did in the Red Hat Linux days. We have a new kernel coming out every 2 months roughly. How do we know we can go back to this release model in a much faster world?
(Asked by Bill Nottingham)
Spot explained he looked at all of the features submitted for recent Fedora releases, and analyzed how they would have fit into his proposed release model. The vast majority of the features that we have are the ones that can be done in the 6 month window. He gave some examples of these types of features – updating PHP, rebuilding all of the Ruby packages, and updating glibc. None of the folks working on these features are asking for more time to finish them, so for most features under this new model it should be business as usual.
Where the new model will change things, Spot explained, is on big ticket features that need more time. It’ll enable us to focus (finally) on a consistent user experience. It’s a hard problem, but it’s still a solvable problem, and by nature we already what we need to do today in a weird sort of way. We haven’t solved it yet simply because there’s a lack of people tasked to solve this problem and working towards a solution.
11. Why would anybody install 20.0 or 20.1 in this new model? Are users going to abandon those releases?
(Asked by David Cantrell)
Spot explained that historically, if you look at Red Hat Linux, or even other distros with an LTS model, you don’t see a huge drop of users for the earlier or non-LTS releases. If we did see user abandonment like that, though, we could change the model again. Spot doesn’t think it would kill Fedora to try it though.
12. What’s the plan for point releases and their life cycle?
(Asked by Matt Miller)
Point releases under this proposal will still get 13 months of updates. I think someone in the crowd suggested making that time period shorter. Spot responded, “If we say we didn’t need those extra months, we could drop them next time.”
Peter Jones suggested that we did automatic migration between point releases to lessen the amount of time older point releases would have to be supported.
“I’m not opposed to an automatic migration,” Spot replied. “We want the user to know that is what they are opting into by default, though. It’s a marketing thing. There are going to be users who don’t want that, so the user notification could ask them to go to the next point release… I think it’s less about who wants to stick with 20.0, and more about when they are ready to go to 20.1.”
The proposed model doesn’t change the number of concurrently supported releases. “If these numbers were 20, 21, 22, 23, [instead of 20.0, 20.1, 20.2, 20.3] it’d be the same as today’s model.”
There did seem to be some confusion in the room at this point, with many folks assuming the last point release would necessarily be an LTS release. Spot pointed out that LTS is not a part of this proposal necessarily.
“What if as soon as a point release is declared stable, you make the previous point releases security-fix only?” asked Stephen Gallagher.
“It’s an option, definitely,” responded Spot.
13. People stick to old releases because they have a $5000 copy of MatLab, and it works on Fedora 14 but it doesn’t work on F16 because of library changes.
(I don’t know who pointed this out, I didn’t recognize their voice.)
Someone in the crowd asked, “Why aren’t you running RHEL?”
“That’s great,” replied the questioner, “but not everybody does that. Take something like Nvidia’s Cuda tool. They provide a version that runs on FEdora. Take Steam, they’re talking about Steam on Fedora. It seems to me that’s a lot of the reason why people stick to an old version for longer than what you’d expect.”
“I agree,” said Spot, “I think that’s valid. I don’t know how to solve that problem without being RHEL. I guess it depends how we define major change.”
“You’re developing against libraries that change over time,” pointed out the questioner. “Cuda will not install on Fedora 17 because it requires a specific library version.”
Spot responded, “It depends where we draw the line on major changes. I think where you want to draw the line isn’t where I want to draw the line. RHEL solves that problem; it does an excellent job of doing that. The audience for Fedora is less interested in a $5000 copy of MatLab and more interested in GNOME++ and more interested in the newest, latest, greatest free software. We’re not going to take Gimp away and replace it with something new in the middle of the cycle. If Gimp’s menu’s move slightly, that’s more acceptable than replacing Gimp with something else.”
14. Who is it that you are trying to satisfy with this proposal?
(Asked by Joe Brockmeier)
“That’s a good question,” replied Spot. “I’m trying to get the people who run OS X and not Linux to give us a shot.”
15. Doesn’t this change the nature of Fedora?
(Asked by someone in the crowd, possibly Jesse Keating.)
“By the 20.3 release, it’s very likely that other distros at that point have a newer version of our features than us,” the questioner continued.
“That happens all the time depending on other distros’ timing,” Spot responded. “That’s why this isn’t a build a 5-year Linux on Fedora proposal.”
Spot concluded the session by saying that he is happy to hear “ideas, alternate proposals, and rotten fruit” in response to his proposal.