This is another exploration of development environment setup for Magento 2 development. This will feed into an official recommendation, but sharing some experiments along the way that can be used immediately. Feedback and comments are always welcome, as ultimately it is you the reader that this effort is for.
What is Vagrant?
In a nutshell, Vagrant uses a Vagrantfile to describe how to set up a Virtual Machine (VM). Vagrant is not virtualization software itself, it relies on virtualization software. (I use the free Virtualbox provider, available for Mac and Windows.). Vagrant allows you to preinstall the exact set of tools you need for a project, and share that environment between different team members. Typically Linux is in the VM.
I do not describe how to install Vagrant and Virtualbox in this blog post – visit the Vagrant site (https://www.vagrantup.com/) or do some internet searches for some good tutorials. This post assumes you have Vagrant installed and working from a Windows prompt.
What Goes Where
This blog post focuses on Windows and Vagrant. Why is it not more OS portable? I believe you can use the same approach on a Mac, but this blog post uses “rsync” to synchronize the local (host) file system contents with the Vagrant (guest) file system contents. On a Mac the “nfs” synchronization approach seems better (there is no copying – just one set of shared files), but this does not work on Windows. So while the approach here will work on a Mac, you may be better adjusting the Vagrantfile to use NFS and not rsync. (I have not tested out what changes would be required yet.)
Why did I pick having the master files on my local (host) machine? Because I like running the IDE natively and I feel “safer” having my master source files survive even if I destroy the Vagrant VM. At the same time, I want to be able to run Gulp or Grunt inside the VM and have file watching work correctly. This works with rsync because Vagrant watches the local file system for changes, then immediately copies changed files via rsync into the VM. Inside the VM it is a normal file system operation, so Grunt/Gulp file watching inside the VM spots the changes reliably.
Why am I not using Alex’s Vagrant project described in a previous blog post? Alex’s project is geared around lots of automation in terms of registering a project with PHP Storm. It also leverages PHP Storm capabilities to sync file changes into the Vagrant box. It also detects the OS and changes strategies automatically. And on top of that, it is primarily set up for CE and EE core development with the ability to switch easily between them. The end result is powerful but a bit complicated. The approach I explore in this blog post I have tried to keep minimal and able to work with editors other than PHP Storm. For example, you can use Sublime or Vim to edit files, and they will be copied into the VM on save.
Project Directory Structure
I have described my recommended directory structure for projects in previous posts. I do not recommend using git clone from the CE repository, but rather Composer to do downloads. Yes, this does involve getting some keys set up from the Magento Marketplace, but as soon as you want extensions from the Magento Marketplace you are going to have to do this anyway.
I have set up my ~/.composer/auth.json file once, and never touch it again. When I run Composer from the command line that is one of the directories Composer looks in. It also keeps the access tokens out of the Git repository.
The project I will be using in this post can be found on GitHub (https://github.com/alankent/magento2-alexa). Just be aware that the project may move along (or fall dormant), so it could get out of sync with this post.
This project was originally set up using composer create-project, and then tweaked (e.g. the .gitignore file was changed). (We may merge some of the tweaks back into the template project if they work well at some future time. Feedback on good and bad aspects of these files are welcome.)
The key parts of the project are the root composer.json file that depends on CE, the .gitignore file that causes directories like vendor to be excluded from the repository, and app which holds local project files. For the Alexa project, I created app/code/AlanKent/Alexa to hold my locally developed code.
This approach can be used both for projects (merchant sites) and extension development. The final extension that may be shared on Marketplace would be built from ZIP files created from the code under the app directory (not a ZIP of the whole project). Composer normally recommends having one module per Git repo, but I do not do that here as treating it as a project containing all of the modules for the extension I personally find makes development easier. (Smart people can find their own solutions for their own circumstances.)
Vagrantfile Walkthrough
The following is a walkthrough of the Vagrantfile from the project. If you take a copy of this file and try it on your project, please let me know of any problems you have with it. One idea is to include this (or a similar) Vagrantfile in the default project directory created by composer create-project.
I am not going to describe every line of the file, just highlight particular points of interest. See the Vagrant documentation if you want to learn more about Vagrantfiles.
The first point of interest is the last line above. While I have not used the Vagrantfile and supporting shell scripts from Alex’s project, I have used the same base box (VM image).
Next comes the network setup.
You may want to change the last number of the IP address above for different projects. It will be the IP address you use to access the web server. For example, the above would be accessed using http://192.168.33.33/. If you want to Vagrant VMs running concurrently, you will need to create them with different IP addresses.
Next comes the rules for synchronizing the local file system with files inside the VM. The “scripts” directory holds some scripts that are needed inside the VM. These are currently not created by composer create-project, if you want them you will need to copy them into your project. They only need to be copied into the VM once at startup and does not need to be synchronized later.
Next, this Vagrantfile copies your host ~/.composer directory into the VM’s ~vagrant/.composer directory (you run as user “vagrant” in the VM). I never want files to be copied from the VM back to my home directory (to avoid pollution), so the “auto rsync” flag (see below) is turned off. Copying this directory across shares the auth.json file authentication tokens and your local composer cache as a starting point for the VM.
Next comes the main synchronization directory, including enabling of the “auto rsync” mode. There are certain files and directories excluded from synchronization because they are either not needed or not wanted.
A quick diversion on how the “auto rsync” mode works. The Vagrant command line includes a command you can run on your host to watch the local file system for changes. When a file system change is detected, rsync is run to synchronize the local file system with the files inside the virtual machine. Vagrant worries about getting the SSH keys etc set up correctly for this to work. However, the process slows down as the number of files being synchronized increases. So I exclude the vendor directory here as it is large and changes rarely. The .git directory can also be big, and there is no need to have the git files in the VM (it can cause confusion running git commands both inside the VM and outside, so I always run git commands from the host).
So, back to the Vagrantfile, you can see it is synchronizing the whole directory tree into /vagrant (I am still considering whether this should instead be under the ~vagrant home directory), but excluding git files, the var directory (we don’t want Windows scratch files copied into the VM), the vendor directory, and the app/etc/env.php file (which contains local configuration settings such as the MySQL host and port which does not make sense sharing between environments). (Oh, the .idea directory is used by PHP Storm to store project settings. You may add other directories if you use a different IDE.)
Next, you can set up some settings specific to providers. I gave my application 2GB of memory for the VM to use.
Finally Alex’s base box did not include NodeJS (this may change in the future) so I install that in case Grunt or Gulp is wanted, I also install the “frontool” project for Gulp support, and finally run a Magento setup script to create the database, set up app/etc/env.php and app/etc/config.php, and so on.
Basic Tool Installation
Before proceeding, you need to make sure you have the following installed:
You must install Vagrant (https://www.vagrantup.com/). I use Virtualbox with Vagrant to provide the virtualization technology (https://www.virtualbox.org/wiki/Downloads).
Next you need to install some command line tools such as rsync, ssh, and git. I personally use Cygwin (https://cygwin.com/install.html), but there are other similar environments around.
While I run the web server and MySQL inside the Vagrant VM, I still run some commands on the host, such as Composer (https://getcomposer.org/download/). There is a Windows based install for Composer that includes everything, otherwise you need to install both a version of PHP (say in Cygwin) and Composer itself.
I briefly mention PHP Storm configuration later in this post as it appears to be the most popular IDE being used in the Magento space, but you can just use a normal text editor as well.
You need to be able to start up a command prompt and run commands such as git, ssh, and composer. Once you can do that you are good to proceed with the following steps.
Setting up a New Project
The following is a walk through how you could create your own project from scratch. It is a little long, but you don’t do it very often. If the approach gains traction, we may reduce some of the steps by merging into composer create-project. Feedback here is welcome.
First, create a new empty project using composer create-project. (You can set up the directory structure by copying from another project instead if you prefer.) Here I specify a new directory called “magento2-alexa” to create the project in.
For Enterprise Edition customers, just use magento/project-enterprise-edition instead. (For Magento Cloud customers, I recommend you can a new project git repository using the Magento Cloud web interface which sets all the files up ready for immediate cloud deployment.)
If this is this is the first time you have used Composer in this way, you may be asked for credentials for the Magento repository. Composer will prompt you for a username and password – these correspond to the “public key” and “private key” mentioned in the Magento Marketplace for extensions. To get keys, go to http://marketplace.magento.com/ and log in (register if you have not done so before). When you log on for the first time you will start on the “My Account” page. Select the “My Access Keys” link. If you don’t have any keys, click the “Generate New” button to create new keys. If you want different keys per project, use the project name as the label for the keys. Copy the public key into the Composer prompt for your username and the private key string into the Composer prompt for you password. Composer will store these files in your ~/.composer/auth.json file for later use.
The composer create-project command will take a while to run as it not only creates the template directory structure, but also downloads all the modules for the first time as well. I like having these files all available on my host file system so I can view them easily, even though we are only going to run the code inside the VM. (It allows better file navigation inside PHP Storm for example.)
Next, create a .gitignore file. Copy from my Alexa project if you like https://raw.githubusercontent.com/alankent/magento2-alexa/master/.gitignore. There may be some local changes you want for your project – for example, currently I removed the Gruntfile.js from the Alexa project. If you use Grunt you would want to keep that file.
Currently the composer create-project command creates a few files that you may not want in your project – these may be removed in the future, but just in case I suggest ignoring/removing files like CHANGELOG.md and CONTRIBUTING.md. You can browse the Alexa project and compare your list of files to your project to help make decisions on what to keep and what to remove.
Next, put the project under source code control.
You may want to push to a GitHub or similar repository at this stage (git push). The exact instructions depends on where you host you repository, so I leave that as an exercise for the reader.
Great, so we have the project under source code control, without too many excess files in the git repository. Next add the Vagrantfile. Again, you can copy from the Alexa project and then make any local edits (such as changing the IP address). https://raw.githubusercontent.com/alankent/magento2-alexa/master/Vagrantfile
The default Vagrantfile also requires the files in the scripts directory. https://github.com/alankent/magento2-alexa/tree/master/scripts
Add the extra files to your git project once you are happy with them (git add, git commit, etc).
(At this stage your may have decided to skip most of the above and just fork the Alexa project as a starting point! Your call! If this whole project goes well, these extra files may find their way into the project that composer create-project set up, avoiding these extra steps.)
Now the fun part – start up your VM which will include the web server, MySQL, and so on. The first time this is run it will be quite slow downloading the base image, setting up the VM, adding NodeJS, and so forth. It will also create the MySQL database and run composer install. If you use the Vagrantfile as supplied, it will copy your ~/.composer directory inside the Vagrant VM picking up your auth.json file and all the files in your Composer cache.
If anything goes wrong, you can blow away the VM using vagrant destroy and start again. This is safe as the master files for your project are on the host file system, not inside the VM.
Before you start up rsync, there is one important file you need to get out of the VM and add to your project, which is the config.php file. This is created inside the VM during the database creation process. Since we are not running the Magento install script on the host, we need to grab it from the VM. (If you don’t do this before starting rsync, then rsync will delete the file in order to make the VM file system match the host file system.)
Add and commit this file to your git repository as well.
Next, start up synchronization from your host into the VM using the vagrant rsync-auto command. This will run rsync whenever the local file system changes. You need to have rsync in your PATH for this to work. I have only tried Cygwin, but I assume other versions of rsync will work as well.
I leave this window running so I can keep an eye on it, minimizing it when I am sure things are working fine. So you may need to start up another shell to continue. Make sure you restart this command if you reboot your machine. (I wasted a very frustrating period of time wondering why my changes where not working, only to discover I had not started this command up after a reboot.)
Check that you can get into the VM using ssh. If you started up a new window, make sure you cd into the local project directory first. It needs access to files in this directory to work out which VM to connect to.
If you browse around, you will discover you start in the vagrant account home directory (/home/vagrant). The Magento files are synchronized under the /vagrant directory, so you will probably immediately cd over to there. You should find the complete project there. You can check if things seem okay by running the Magento CLI. (It has been added to your PATH for you.)
You can exit out of this shell once done.
Next, fire up your web browser and point it to the IP address in your Vagrantfile. After a short time the home page of your store should come up. The administration interface is available from /admin. The user name and password is admin and admin123. You can change these in scripts/install-magento if you don’t like these defaults.
Easy! (Well, okay, a bit longwinded but not that bad and only has to be done when setting up a new project.)
Checking out an Existing Project
What about other team members? Once you have your project created, what would another team member do to get going on the same project (where both developers save code into the project’s git repository)? Good news is it is a lot simpler than the above. (If you want to try this, you can always git clone my Alexa project to experiment on.)
git clone the project onto your own machine.
vagrant up to create the VM
vagrant rsync-auto to copy files into the VM
You can now edit files on your host and they will be automatically copied into the VM. I use git and my IDE from my host, never inside the VM. Any edits inside the VM could be overwritten at any time, so generally I do not make many edits there.
To do an upgrade, I edit the composer.json file to update the version number, run composer update on my host, then run composer update inside the VM as well (because I do not use rsync to synchronize the vendor directory for speed).
PHP Storm
The above steps allows any IDE to be used – when files are saved to disk, they are copied into the VM automatically. Since PHP Storm is so commonly used, I thought it worth mentioning a few useful features. PHP Storm has built in support for Vagrant.
There are different versions of PHP Storm, so you may need to make some adjustments to the following based on your specific version of the IDE you are running.
First, we need to create a PHP Storm project. Under the File menu, select “New Project from Existing Files…”. I selected “Source files are available locally, no web server is yet configured”. This is because the two “remote web server” options listed (with Vagrant, the web server is “remote”) both result in file downloading, which is not wanted. So create the project from local files and add the remote web server by hand later.
Navigate to the root directory of your project, click the “Project Root” button, and then click “Finish”.
I am not going to go through all of the PHP Storm configuration here, but I will step through getting unit tests running remotely to give you a taste. This is based on https://confluence.jetbrains.com/display/PhpStorm/Working+with+Remote+PHP+Interpreters+in+PhpStorm.
Navigate to “File”, “Settings”. In the popup box under “Languages & Frameworks” find “PHP”. For “Interpreter” we are going to register the Vagrant VM as where PHP is available. Click the “…” button next to the currently selected interpreter name, and in the popup click “+” in the top left corner to add a new interpreter. Select “Remote” from the drop down list. This will bring up another dialog box. Select the “Vagrant” radio button at the top, for “Vagrant Instance Folder” navigate to your project home directory again (where the Vagrantfile is), then wait for a moment. You should see the “Vagrant Host URL” field just below be automatically populated after a few seconds. Click “OK” and wait for PHP Storm to do its magic. (It runs several commands against the Vagrant VM.) I suggest clicking the “Visible to only this project” checkbox before finally accepting the new interpreter as you should only use the interpreter on the current project.
Once you have done all of the above, you can just use this interpreter in PHP Storm. It is smart enough when you run code to realize it has to access PHP via Vagrant.
If you check out my Alex project to play with the above, try running the unit test FrontControllerTest. You should see it run without error.
Gotcha’s
I have been caught by a few issues using this approach that I though worth mentioning.
I have to remember that hitting “Save” in the editor does not save the file into the VM immediately. I often have to pause for a second to give rsync a chance to notice the file change and copy it. If I run say a unit test too quickly, the changed file might not have been transferred yet. For me, around 1 second is the normal pause I now do after saving a file – so it’s not too bad.
I tore my hair out at one stage until I realized I forgot to start up the vagrant rsync-auto command after a reboot. Without this command your local file system changes won’t be copied into the VM. To me this was interesting from the perspective that the VM was becoming somewhat transparent in my thinking. That is a good sign.
You do need to ssh into the VM at times. I have not found it a big problem. It seems simpler to understand than trying to do too much magic on the host OS with scripts that ssh over to the VM.
On Windows, often when I kill the vagrant rsync-auto process using ^C, it does not fully exit. Use Windows Task Manager to see if you can see a “ruby.exe” process still running – and if so kill it. (Vagrant is implemented in Ruby.)
Conclusions
This post described one way to use Vagrant to manage a Magento 2 development project. I am using it to develop my Alexa integration module, to go through the real development experience. There are a few little gotchas, but in generally it is working pretty well for me.
But what about Docker? Isn’t Docker better? Docker have some cool features coming out soon to help with high performance file syncing without using rsync. However for Windows users I don’t think this will support Grunt/Gulp file watching initially. You may notice that my Alexa project also has a Docker Composer configuration file in it. A later experiment is to see if I can use both technologies on the same project in Git, meaning individual developers can pick which ever technology they prefer – Docker or Vagrant (or even just native installation of tools on the host). The PHP Storm Vagrant integration support was nice however – that is not available with Docker yet.
Another topic that is of course missing from this post is how to take your project from development to production. That may be a topic for a future blog post.
Feel free to leave a comment if you hit any problems using the above approach on your project. And if it works well, leave a comment as well! That will help with deciding whether to merge some of the additional files in the default Composer created project.
