“Lions and tigers and bears! Oh my!” – Wizard of Oz.
Every so often there is a twitter thread asking how the new Magento Connect is going to work in combination with Composer, extension reviews, and patches. There often feels like there is quite a bit of concern in how this is all going to work. It can all sound a bit scary. The good news is everyone I have talked to so far when they understand our plans leave happy. So I am personally pretty confident we are on the right path here.
I have posted on parts of this before a few times, but here is a brain dump of where things are at right now.
Installation and upgrades have enough gotchas that I would not be surprised if there are a few things still to sort out as we go. Feedback is always welcome!
Disclaimer: This post is my own and does not necessarily reflect that of my employer. More relevantly, I knocked this up quickly without checking with the team doing the real work! This is not officially binding communication, just sharing how it is likely to be to help people plan for the future.
The New Connect
Magento Connect is getting a major work-over. The current Magento Connect is a directory of extensions (you cannot purchase them online), unlike the new Magento Connect which will allow you to purchase extensions directly from the site.
There are lots of objectives for the new Connect, but the major focus is around trust. Trust of reviews, trust of code quality, trust of security, trust of reviews (by knowing who has purchased what, it is much easier to trace reviews to purchasers), and more. There is lots of goodness possible here. The first release will be just that, a first release, with more planned to continually improve the experience for both Merchants and developers.
(One outcome that I would personally enjoy seeing is making it easier for developers to get rewarded financially for good work they do. With Connect doing most of the financial paperwork, hopefully it will be easier for new developers to enter the ecosystem. Of course, the counter is to make sure they do so with good quality extensions, which is where the extension verification program kicks in.)
The Connect Experience
The Merchant experience will be pretty straight forward for Magento 2 purchases. There will be a Magento based web store for browsing and purchasing extensions. You can purchase them directly from Connect, which will give you access to the code for the extensions. (Connect will allow you to purchase Magento 1 extensions as well, but the installation approach in Magento 1 has not changed so I focus more on Magento 2 in this post.)
In the future it may be possible to search Connect for additional extensions from inside your Admin panel, but this is unlikely to be available by GA. To me it is not clear it is required anyway. A link taking the user off to the official web site seems adequate (and potentially more secure when it comes to payments), the only question being around whether we can securely avoid requiring the user to authenticate again when they hit the Magento Connect site.
Composer Based
With the new Connect, each Merchant will have an account. All Magento 2 modules will be available via Composer. All of the CE modules will be available to the public, but all for fee modules will also be made available to those who have purchased them. Connect will track who has purchased what. The current plan is for the only official way to download code is via Composer. This avoids problems with multiple release and upgrade paths. A goal is to simplify and unify the experience.
(One area of concern here for technical folks, is downloading a new site from scratch is slower than the old approach of grabbing a tar file. For now, the plan is to have a single release approach (install via Composer), but if this proves too slow in practice, we may need to explore having prebuilt images to kick start an install. For now however, my personal preference is there will only be one installation approach, which is Composer. The slower initial install is offset by the benefits of easier upgrades and patch installations.)
Installing Extensions
The web installation wizard in Magento 2 already allows the user to include or exclude optional modules (e.g. you can remove the FedEx shipping module if you are not going to offer FedEx as a shipping option). This will be extended to support purchased extensions as well. That is, purchasing an extension makes it available – it does not automatically add it into your code base. You have to explicitly add it to your site.
Which modules get downloaded is ultimately controlled by the composer.json file. Developers may choose to edit this file directly instead of using web interfaces. Any extension they have purchased (on behalf of the Merchant) can also be added using command line tools. Composer is used to download all extensions and patches, whether via the command line tools or the web installer.
User Types
One class of Magento users I will roughly categorize as the DIY Merchant. They want to get Magento installed by themselves, without a partner. We want to improve support such Merchants by reducing some of the pain from Magento 1. Platform changes have been made to reduce the number of extension conflicts, which should make it easier to combine different extensions. But a key goal of Connect is to highlight “quality” extensions – extensions that are known to work reliably and well, supported by quality organizations committed to help their customers.
On the other hand, the goal of Connect is not to exclude other parties from the community. There are many good quality open source extensions. The challenge then is to help the DIY Merchant know when they are selecting a quality extension versus when they have come across something of dubious quality. I can imagine a bit of tweaking of Connect going on here post GA.
Another key goal for DIY Merchants is to reduce the risk and pain of installing upgrades. By reducing conflicts and adopting Semantic Versioning for modules, it should be safer to apply module patches when available. This is very important as security vulnerabilities will also be released as patches. The goal is to make the experience for a DIY Merchant so they can apply patches with confidence, and so keep up with the latest patch for their current release level.
Another broad category of Merchant is those who use a partner or in-house development team to build a site. The key difference here is they have much deeper technical skills available to draw upon. A common request we have received from this class of users is to make it easier to lock down a site from accidental change by a Merchant in the Admin panel. As a part of this work, the installation process code has been placed under the ‘setup’ directory. As part of the deployment process to production, it is safe to remove the ‘setup’ directory, thus making it impossible for a Merchant to make further changes to their production installation. (Note: I have not actually got around to trying this yet, but it is the plan.)
Account Management
Coming back to Connect, there are some interesting questions around account management. The recent Magento Meetup hosted by WebShopApps in Austin had some interesting discussion in this area, bringing up one or two points I had not thought of before.
Sure, we want a partner to be able to purchase extensions on behalf of a Merchant. But what controls should there be in place? And what happens if the Merchant decides to switch to a new partner? Purchases should be primarily tied to the Merchant making the purchases, but need to be accessible to the partner working on behalf of the Merchant.
Another interesting point that I had not thought of before was one job for a partner is to recommend what extensions to use on a site. The reality is partners build up a list of “known good” extensions that they tend to reuse on multiple projects. This makes life more efficient for the partner. Having a single place where you can purchase all the extensions can save the partner quite a bit of time compared to interacting with each extension developer one by one. But it is also useful for the partner to work out what extensions they have purchased in the past (even though the purchase was on behalf of a Merchant). Interesting.
Another goal of a higher quality Connect is to reduce the time partners need to spend worrying about code reviews of extensions (because Connect will have done them already). Magento 2 also includes many features to reduce pain in this area, encouraging the use of new modules to override existing behavior (e.g. via plugins) rather than directly editing the code of someone else’s module (be it a core module or an extension). This allows partners to focus on work delivering higher quality value to Merchants rather than spending time performing code reviews.
Timeline
So when is all this new Connect goodness available? The new Connect will be available by the Magento 2 GA (official date is Q4 2015, as publicly stated for the last 2 years). We currently have a Merchant Beta program going on, and there have been a few distractions with the sale of eBay Enterprise, so it’s hard to be more precise. The good news is in spite of distractions, Magento 2 and Magento Connect are still both on track for a 2015 Q4 release.
But what about Connect? When will it be available to the developer community for listing extensions? They need to be available before Magento 2 GA. That is, Connect also includes the extension developer experience of submitting extensions for review and release. This will be made available to extension developers incrementally, starting with a small set of top partners, then working down the various tiers (gold, silver, etc). The reason for this phased approach is simple – to get feedback from smaller groups of developers, fix the problems, and then expand to wider audience bases. So just because you have not seen much happening in public, I can assure you work is in progress here. (And it is not my place to provide dates in this post.)
Oh, and while the release is going to line up with Magento 2 GA, there will also be Magento 1 extensions in the new Connect site.
Extension Review Process
Some of you may have heard about the new review process for extensions to be listed in Connect. The goal here is around quality and trust. For Magento 1, paid extensions already have places they can be purchased, which will not go away. But the goal is to have Connect become a trusted source of extensions. For details on the review process, when different tiers of developers can submit extensions for review, for Magento 1 and Magento 2, I recommend you contact Magento through official channels so you can get information directed at you specific situation. But priority is being given to top tier extension developers (it is not first come first served, at least until the verification backlog clears).
One interesting group of tweets I am noticing as I type is making an interesting point. How fast can an extension developer get their changes on to Connect and available for their customers? Minutes? Hours? Days? The goal of Connect is to get extensions verified to ensure the quality of contents of the site. Verification can be performed with automatic tools like code sniffers, security scans, copy paste detection, and so on. For major releases, it seems reasonable to require more detailed (possibly human) review of code. But should this block a major release until completely addressed, or can the extension be on the site with any problems addressed in a patch? And what if the release is a minor code patch to an existing module? How much re-verification is required (adding humans to the review process will always slow down releases). I don’t have great answers here yet – something that needs digging into further.
Encryption
How can I talk about Connect without touching on the sensitive topic of encryption? The suggested policy that makes the most sense to me is the one where Connect does not prohibit code with encryption sections, but Connect does report what percentage of the code base has been encrypted. This allows Merchants at least to make informed purchasing decisions. I do not believe Magento should prohibit developers from protecting their intellectual property (their code) from theft – but the more code that is encrypted, the less Magento can do to verify the quality of the encrypted code, making it a discouraged practice.
Porting Extensions
So when to start porting your extension? This is a common question, but there is no blanket answer as it is primarily driven by business decisions. Should you get your extension into Connect earlier so you are there from day 1 of the new Connect? Should you get your Magento 2 extension into Connect to gain recognition of your dedication to keeping up with the needs of the Magento community? Or do you aim to reduce churn and port later, waiting for the first few Magento 2 patches and for the customer base to grow first? Each has its own merits.
From a technical perspective porting earlier means you have a longer runway to deal with issues that may arise, allowing you to more easily influence the Magento 2 ode base. It also shows your dedication to the platform. A negative however is there is likely to be more churn in keeping up with the final stages of Magento 2 development. In addition, the longer you leave porting, the more you can learn from the experience of others. Personally, while the technical details matter, I think it is more of a business decision of when to work on a Magento 2 extension. The technical aspects are becoming less important as we get closer to GA in Q4.
Community Edition (CE) and Enterprise Edition (EE)
Changing tack, so how are CE and EE going to be released? There will be a base set of modules that make up CE, plus additional modules only available if you purchase EE.
The current plan is there will be a meta-package for each CE and EE patch level. This meta-package will identify the exact version numbers of each module to include in your site. This makes the job of support easier. Identifying CE 2.0.1 will identify the exact version numbers of all modules included in your installation.
So what will the module version numbers look like? This is still being discussed internally, but a possible outcome is that all modules will at GA have a release version number of 100.0.0. The reason to start from 100 for modules is to simplify support discussions. If someone mentions a version number of 2.1, it is clearly CE or EE; if someone mentions a version number of 100.1.0, then it is clearly a module being referred to. I am sure some developers may consider this to be a bit “unclean”, and they may be right. However the benefits of simplifying support (by Magento or extension developers) should not be undervalued. This change has not been committed to yet, and is not yet available in the code base. But if you start seeing module version numbers starting from 100, at least you will know why.
Currently the CE code base is all located together in a single GIT repository. When 2.0 is released, all modules will be released with the same version number. This may be repeated for the 2.1 release (all modules will have the version number incremented), but we are currently exploring releasing each Magento module with its own version numbering. The best way to achieve this appears to be to create one GIT repository per module. This is a fairly major change to development and the test automation framework, so is being reviewed carefully before being committed to. The negative of incrementing the version number of all modules (even if unchanged) for CE 2.1 is that all extensions declaring dependencies on CE 2.0 will need to be re-released, to include the 2.1 version number as also being supported (after being tested). This imposition on extension developers may accelerate the investigation around whether to split the code base into multiple independent repositories. (I can hear the screams of pain now!) But this investigation is unlikely to occur before GA – but that is OK, as it is CE 2.1 when the issue will first be encountered.
EE Hot Fixes
The Magento Support organization provides support to Magento EE customers (and not CE customers). This is a part of the service offering for EE. There are competing pressures regarding patches – get a customer going again as fast as possible, and the overall controlled management and release of patches.
The proposed strategy for Magento 2 is to eliminate hot fixes. All patches will be released in the same way, building upon all previous patches. All patches will also be made available on Connect, so it is as simple as running “composer update” to download all available patches. This patching approach is part of the reason that Magento 2 has invested heavily in allowing modules to extend and override (via plugins etc) existing modules, rather than editing the source code of modules directly. If a site customization is in module B and which adjusts a core module A, patching A (replacing it with the code of the new patch level) will not lose the customizations made in module B.
The challenge here will be the rate at which a patch can be made available to a customer. Streamlining the patch release process for EE customers is a goal for the core Magento development team, just as it is a goal for extension developers in getting patches out to their customers (as described above).
But what of producing patches for specific customers? It is currently planned to not have any official patches for specific customers. Instead, provide a patch for all customers. If functionality change is required for a particular customer (which would probably break the semantic versioning numbering policy) it is proposed to provide that code with a feature switch, which by default is turned off. That is, anyone installing the patch will not see any difference, unless some configuration file setting is changed to enable the new code. This code then represents new functionality, rather than being customer specific. The default is existing customers would not pick up the new functionality.
If a quick fix is required on site and cannot wait for an official patch to be made available, the next best approach is to create a new module that uses plugins, dependency injection, or similar to override the broken behavior of the core module. This will get the customer up and going until the official tested patch is available.
Getting CE into GitHub has also raised some interesting questions around support interactions. As we increase the communication with the general CE community, we are looking to see how we can similarly improve the experience and responsiveness to change for the EE customer base as well. EE obviously has the additional complexity that the code is private (not shared with the full community).
The end result is simple. All patches are released to everyone at the same time, via one channel (Connect). There would be no more hot fixes being released to just one merchant, and no more requiring merchants and partners to work out what patches exist. All patches will be available to everyone at the same time via Connect. The goal is to accelerate the adoption of patches made available by all customers, which is particularly important for security patches.
Conclusions
This post touches on a range of issues around Magento Connect, Composer, and the user experience. I believe Magento Connect represents a major step forwards for the Magento community. A lot of work has gone into both the platform and Connect to lift the user experience, making it easier to purchase and upgrade extensions with confidence. Hopefully this post has helped connect some of the dots for how this is all planned to fit together. (And my apologies for the rambling nature of this post.)
And if I have missed something, just leave a comment below and I will try to merge into the above.
