alvinashcraft
shared this story
from 
Jeff Wilcox.
In this post I'm going to walk through using Visual Studio Team Services Package Management as a private npm
registry for scoped npm packages using a custom deployment script for Azure App
Service.
Many Node.js developers are probably familiar with npm private feeds but may not have
used them yet in the Microsoft Cloud + Node.js stack. I wanted to share the experience that
I have had: my team is running a ton of small-to-medium Node.js services within Microsoft
and we're eager to share our experiences for the good of the community.
I'll also show how my team is able to use Azure KeyVault at deployment time to
enable credential rotation without having to manage secrets at the individual app level.
While this is a long post in explanation and detail, someone using App Service for their Node.js
deployments today should be able to connect to a private npm feed in about 15-30 minutes if they have
not yet adopted a custom deployment script. Of course KeyVault and other aspects could add time.
Let's get started.
Private NPM modules
Node.js developers know that "there's an npm module for nearly everything", but it turns
out that sometimes you need to host your own private npm registry to store private modules
specific to your organization, policies or business rules, or to share custom code
amongst your internal projects.
Common scenarios include corporate-specific Express middleware, private business
logic plugins to open source systems, supporting configuration-as-code,
and getting an npm module ready to ship to the world ahead of release.
Scoped packages
When looking to use private npm registries, you'll need to decide whether you
intend for the private npm registry to be the "primary" registry for all packages,
including popular packages you would otherwise get from the public registry -
express, moment, etc. - or if you want to use scoping.
Scopes are built into npm and are the only native
way to support multiple registries. You can configure npm to visit a specific registry for a
given scope instead of the global registry.
A downside to using scopes is that modules intending to go public will cause some churn in package.json files. A
private npm that uses a scope today but ships to the world tomorrow will likely lose or change its scope, requiring
you to edit your various package and source files.
For this post I'm using a custom @jeffwilcox scope; within the Open Source Programs Office
at Microsoft we use the @ospo scope, since that's the short name we use for our team.
Scopes are not unique to private repos. For example, the @microsoft scope is for
Microsoft open source npm modules that are shipping to the world.
Paid NPMJS
The official NPMJS registry sells private npm registry feeds for $7/user/mo. or so and
this may be sufficient for many users.
However, those using Visual Studio Team Services may want to use the great
package management integration and private feed support available in that product. On our
team we like that the private NPM feeds live alongside our source code in
VSTS, are contained, and the existing permission settings we use for the team just
work with the private feed.
Configuring your scoped npm environment
To enable private npm use you'll need to login to npm (for the official private npm feed support) or
configure your Node environment to use a specific credential or credential helper (i.e. .npmrc file
customizations).
If you didn't want to use scoping but were replacing the entire npm environment, you would leave
off the scope part of the commands.
For VSTS Package Management users on Windows, there is also a credential helper, but for the
deployment-focused purposes of this post, I am skipping that detail.
Deployment environment
On a deployment or CI server, you will often need to commit a .npmrc file directly into
your environment; while that is out of scope for this post, do note that such files can
use environment variable replacement:
For this guide, I'm going to use a custom script to customize the Azure App Service
environment at deployment time to handle other scenarios including KeyVault. If you are
interested in simple CI/CD environments you can read about the scenarios at npmjs.com.
The custom script for this post is called configure-azure-appservice-private-npm-feed. Very exciting name.
VSTS Package Management
Visual Studio Team Services (VSTS), located at visualstudio.com, is a set of
clcoud-scale hosted Microsoft services for engineers to share code, track work, ship software, etc.
VSTS is fast, easy, and offers private source code hosting of Git repositories free for up to 5 users.
Package Management is a VSTS add-on, available as an extension that can be licensed,
and today it provides both NuGet and npm private feed/registry hosting capabilities. Your VSTS
administrator needs to install the extension and make it available to you.
Adding the Package Management extension
VSTS administrators have an "extensions" menu under the project-wide settings.
From the extensions page, you can then browse the VSTS extensions marketplace.
Package Management is a key extension, you should be able to quickly locate it on one of the main pages
of the VSTS extensions marketplace.
On the Package Management extension info page you can learn plenty more, but we are just interested in the "buy" button.
You can also start a 30-day trial easily.
The process will confirm your Azure subscription as part of the licensing process. It took me about 20 seconds to install.
After installation, the Build & Release menus within the project will light up with the new Packages extension. Let's go there now.
Creating a new private feed
From a project's Packages extension, you'll find a mail delivery person character if you do not yet have any feeds.
Select "New feed".
You can then configure the feed:
The name for the feed
Whether it will be open to your entire team, including who can publish to the feed
Whether to include upstream sources (only select this if you are intending to completely replace the npm registy)
Now we're ready to go! We just need credentials to authorize our local NPM environment to be able to publish packages to the private feed hosted by VSTS.
Generating an access token
From the private feed's page within the Packages extension, select "Connect to feed". This
screen will share the basic .npmrc configuration settings that you will need to get to the
feed:
The registry URL
The always-auth flag that should be true for private feeds
While there's a command for Windows users to install the credential helper, vsts-npm-auth,
for this blog post I'm just going to use the non-Windows token. Part of the reason - I use a
MacBook Pro most of the day, so I'm going to need that token - but we'll also save it to use
with our deployment environment (though in a real prod scenario you will want to use a service
account to probably generate such a token).
Select "Generate npm credentials" to get the non-Windows token.
Make sure to safely and securely store the token from the generated .npmrc file. I
stored my token inside Azure KeyVault, since that is eventually where I will be grabbing
this token from in a more advanced, automated scenario.
Configuring your development environment
If you are configuring for scoped private npm packages, here are the commands you will
want to enter.
First, follow the instructions from the token generation to update your .npmrc file with
the token for your private registry.
Second, configure the user-global environment using npm (though you could also just edit the same rc):
While you can confirm your settings with npm config list, note that private feed tokens will not be shown
in this view:
Publishing a private npm
It's time to ship some private code!
I've created a super simple npm module: a function that will return a string value when called to "prove" that it
came from the module. After creating this index.js file, I run npm init to create a new package.json file
in the folder, give the component a name and version, etc.
Include the scope in the name; in my case this means my package name is @jeffwilcox/private-npm-package. Here's my
package.json file:
Here is my index.js source file:
Time to publish! Since my token to publish and consume the private feed is in my .npmrc environment,
I can simply publish and it's good to go.
Now I can go back to the Package Management extension within VSTS and refresh the feed to see my npm package.
VSTS provides some options including unpublishing and deprecating the package that will help in the
lifetime of the npm module.
Using the module with your app
I have a basic Express web app created using express-generator.
This generates a new folder called "website" that I can initialize as a Git repository to
deploy.
Now I'm ready to use my private NPM module. Let's install it, saving the module, too.
And then I should get confirmation that this package was installed OK:
And to integrate with my private npm, this is what my ./routes/index.js file ended up as; it will
set the title of the page, used by the view engine, to the value returned from the function exported
by my private npm:
I can now run npm start and navigate to localhost:3000 to confirm that the app is working just fine.
Deploying to Azure App Service
Selecting the deployment type
Azure App Service really is my favorite way of hosting Node.js apps on Azure. It's
so easy to configure the web site as a Git repository itself, or connect it to GitHub, or
use VSTS and its build system to deploy to the site.
Here I am using the Git repository hosted by the App Service instance, meaning
when I push to the repo, I am literally pushing and deploying right then onto the app service's
storage. The deployment experience will be echoed back to my Git client, making it easy to watch how
that goes.
I do not want to commit my node_modules/ folder, however, because packages change and take up
space that is not my code - I want to use npm to resolve those packages at runtime. So my .gitignore has
the modules folder in it.
After committing I add my Git remote as the origin of this repo and push:
Within Git's stdout, since I am using an App Service-hosted deployment, I will
get updates about the deployment. If I had connected the service instead up to
GitHub or similar, I'd have to go to the "Deployment Options" part of the Azure
portal to see how it's doing... but this is a _#fail# visual:
The deployment failed because the @jeffwilcox/private-npm-package scoped
npm module isn't in the global npmjs registry.
Because it's a private module.
And App Service does not have any idea about my scoped private registry nor what credential/token
it would even use to connect to it.
Deployment scripts
Automatic Kudu deployment script
By default the Kudu engine used for deployments actually dynamically generates a
deployment script based on how it identifies your application. If it looks like your
code is a Node.js app, it will generate a pretty good default.
You can retrieve the script that was generated for your environment by going to the
"Advanced Tools" part of the web site in the portal to open up the Kudu site
administrator tools where there's an option to download the script to take a peek.
The script generator itself is open source, too.
Custom deployment scripts
To enable private NPM feed use, I'm going to need to go ahead and https://github.com/projectkudu/kudu/wiki/Customizing-deployments with
a custom deploy.cmd script. I'll fork this from the best-known current generated
version for Node.js apps, adding value where I need to.
I first need to create a file called .deployment and commit it to my repo. The content of
the file points the deployment engine at my custom deployment script:
Here is my "current favorite" deployment script, reproduced here for anyone to use,
with the caveat that this is not in source control. That's another future blog post
I hope.
This is for Node.js deployment and it does a few things:
It installs npm prod and development packages (since I want to use grunt and some other tools)
It uses my npm deployment script for private repos, described at the end of this post, configure-azure-appservice-private-npm-feed to configure private npm feed support
If you have a Gruntfile it will grunt
If you have a bower.json file it will install any bower dependencies
It installs npm packages before copying over the actively deployed site
Yes it is a batch file, because 2017. You can also write a deploy.sh bash script.
The important step for this blog post is just that it installs the npm script I wrote and executes it before installing npm modules. This way, scoped private feeds are configured either through the environment and/or configuration before the npm modules are officially installed.
Configuring private feed settings
Now all that is left to do is to actually tell our custom deployment script about
the private npm feed token that we want to use.
By default the script used is configured to look for environment variables named:
NPM_PRIVATE_FEED_TOKEN: The token credential from VSTS Package Management that we generated earlier
NPM_PRIVATE_FEED_SCOPE: The scoped name to use, so jeffwilcox in my example. You can omit the @ symbol.
NPM_PRIVATE_FEED: The path to the private feed hosted by VSTS for my npm scope
The custom script run during deployment will use these values, if present, to configure the .npmrc specific
to this single app service.
Here's a look at what this looks like, redacted partially:
After setting these values I'm ready to deploy again. Fingers crossed.
Publishing with custom deployment
So I just need to commit these deployment files and push to the remote to kick off
another build attempt.
And here it goes...
Partly through the deployment process you should see that it installed the custom npm
script to help things get configured, and then it executed, since it recognized the
presence of the environment variables.
I get some feedback here about the private scoped feed being configured.
Also some good news: I see that the private npm package is successfully installed now
that the .npmrc on the App Service side is properly configured inside the sandboxed
environment.
Finally I get that great "deployment successful" message and know it's time to go visit
the site.
Success!
Navigating to the App Service web URL I will now see the title that is generated
by my private npm function.
At this point we're done covering the basics and you could successfully use this
just fine to deploy App Services using your private npm package(s) for your Node.js apps.
Azure KeyVault support
While storing the private feed token to get to my package inside the Application Settings
for the App Service works to an extent, it has some issues.
The token has been manually placed inside the configuration. A typo or extra space
could cause problems.
If I have 50 web site deployments that use the private npm feed, I have 50 places to configure the token
If the credential leaks, is exposed, or as a general practice I want to reguarly rotate my secrets, I will have to update the secret token in at least 50 places. This will take time.
If I accidentally grant someone broad access to my web site instance, they may easily get to the token
The KeyVault support in this post, I should mention, is what works for my team, but is not official guidance. Use at your own risk and level of understanding of your needs and vault scenarios.
While KeyVault can be used for many scenarios, I'm most interested here in these:
Preventing developer mistakes, especially from manual operation or settings data entry
Potentially being able to build a production environment that very few, or no one, have access to, including the token
Authorizing an individual app and its deployment to have access to a token, instead of having to configure more
Being able to rotate a secret, knowing that the next deployment will pick up the latest secret version and use that to connect to my feed, enabling great cred rotation options
Enabling configuration-as-code scenarios... I can actually store KeyVault secret paths inside private source control repositories without worrying the same way I would about committing an actual secret value
Authoring the vault in the portal
There are many ways to authorize an app to have access to KeyVault, from client ID and secret pairs
to certificates and other methods. In the most simple case, supported in this script, my web site
has an Azure Active Directory application registration that provides an app client ID and a client secret
key that is good for 1-2 years.
I'm able to authorize the application identity to have access to the KeyVault, so it can read the
secrets, and then I can use a custom protocol scheme supported by this script to point the various
Application Settings to the vault.
So I'll end up with something similar to these values for the 3 environment variables to configure
the private npm feed, where the actual secret value is my token and then I use tags on the KeyVault entry
for the other values, keeping the feed URL and the secret token value pair together:
NPM_PRIVATE_FEED: keyvault://feed@my_vault_name.vault.azure.net/secrets/my-vsts-private-feed
NPM_PRIVATE_FEED_SCOPE: jeffwilcox
NPM_PRIVATE_FEED_TOKEN: keyvault://my_vault_name.vault.azure.net/secrets/my-vsts-private-feed
Here's the secret entry inside the vault in the Azure portal. You can see some of the tags I've used to help
organize secrets.
Helping manage secrets
Another great value that KeyVault provide is a good way to gather and review
metadata about the secrets that I'm using in my set of apps.
By tagging secrets with my own representation for the type of secret, I'm able
to generate a rather useful dashboard that identifies secret age and other
properties, grouping similar types of secrets together.
Here in my custom portal I have a view that shows the metadata for a specific
secret version, including the tags I've defined, and also the custom keyvault://
scheme we've designed for this scenario, making it easy to copy and paste the
vault secret URL as needed.
And here is a color-coded look at the current broader dashboard for a simple
application deployment covering about 10 different types of secrets that are
stored in this particular vault.
The private NPM feed key is stored under the "NPMJS Tokens" category to help
aid in understanding.
I'll cover some of this dashboard work in a future post but wanted to share an early
look at some of this.
Configuration as code
Finally, the custom deployment script supports "yet another" opinionated "painless" configuration-as-code story.
By defining a CONFIGURATION_ENVIRONMENT variable pointing to a known environment file, similar to a NODE_ENV setting,
the configuration system is able to read variable values from the right local file.
Since the configuration resolution code supports KeyVault, we're able to safely commit configuration information internally,
just pointing at KeyVault secrets, and then we can treat config changes as code. New secrets and environment changes can
be code reviewed and move through Git like any other set of changes.
For any given secret we have a choice: we can point to a secret name, resolving the latest version at
deployment or runtime, or we can point at a specific version. There are scenarios for both, but for the
private npm feed example, we're using the versionless secret URI. This enables token credential rotation easier.
Here's a quick look inside Visual Studio Code at the configuration environment settings for the secrets
used by this app:
Thanks to config-as-code, we no longer need to configure the private NPM information on each app service, as
long as the app's identity is authorized to the vault. This helps reduce errors.
Open source
The configure-azure-appservice-private-npm-feed
script is licensed MIT and is a small tool/utility. At this time it is also published as an npm, configure-azure-appservice-private-npm-feed.
It's worth noting that at this time only a single private scoped feed is supported by the script. The script has two top-level dependencies that flow to others:
- async
- painless-config-resolver
Your contributions and feedback are always welcome.
Closing
Hopefully you found this post interesting. It's really fun how we're able to piece together
so many great technologies: Node.js, npm; Azure App Service, Active Directory, KeyVault; VSTS
and VSTS Package Management.
