Node.js native addon binary install tool
$ npm install node-pre-gyp
node-pre-gyp
stands between npm and node-gyp and offers a cross-platform method of binary deployment.
node-pre-gyp
that can install your package's C++ module from a binary.require('node-pre-gyp').find
For a hello world example of a module packaged with node-pre-gyp
see https://github.com/springmeyer/node-addon-example and the wiki for real world examples.
See the Frequently Ask Questions.
node-pre-gyp
is designed to be installed as a local dependency of your Node.js C++ addon and accessed like:
./node_modules/.bin/node-pre-gyp --help
But you can also install it globally:
npm install node-pre-gyp -g
View all possible commands:
node-pre-gyp --help
You can also chain commands:
node-pre-gyp clean build unpublish publish info
Options include:
-C/--directory
: run the command in this directory--build-from-source
: build from source instead of using pre-built binary--update-binary
: reinstall by replacing previously installed local binary with remote binary--runtime=node-webkit
: customize the runtime: node
, electron
and node-webkit
are the valid options--fallback-to-build
: fallback to building from source if pre-built binary is not available--target=0.4.0
: Pass the target node or node-webkit version to compile against--target_arch=ia32
: Pass the target arch and override the host arch
. Valid values are 'ia32','x64', or arm
.--target_platform=win32
: Pass the target platform and override the host platform
. Valid values are linux
, darwin
, win32
, sunos
, freebsd
, openbsd
, and aix
.Both --build-from-source
and --fallback-to-build
can be passed alone or they can provide values. You can pass --fallback-to-build=false
to override the option as declared in package.json. In addition to being able to pass --build-from-source
you can also pass --build-from-source=myapp
where myapp
is the name of your module.
For example: npm install --build-from-source=myapp
. This is useful if:
myapp
is referenced in the package.json of a larger app and therefore myapp
is being installed as a dependency with npm install
.node-pre-gyp
myapp
and the other modules.This is a guide to configuring your module to use node-pre-gyp.
package.json
node-pre-gyp
to dependencies
aws-sdk
as a devDependency
install
scriptbinary
objectThis looks like:
"dependencies" : {
"node-pre-gyp": "0.6.x"
},
"devDependencies": {
"aws-sdk": "2.x"
}
"scripts": {
"install": "node-pre-gyp install --fallback-to-build"
},
"binary": {
"module_name": "your_module",
"module_path": "./lib/binding/",
"host": "https://your_module.s3-us-west-1.amazonaws.com"
}
For a full example see node-addon-examples's package.json.
Let's break this down:
node-pre-gyp
aws-sdk
so that you can run node-pre-gyp publish
locally or a CI system. We recommend using devDependencies
only since aws-sdk
is large and not needed for node-pre-gyp install
since it only uses http to fetch binariesscripts
section should override the install
target with "install": "node-pre-gyp install --fallback-to-build"
. This allows node-pre-gyp to be used instead of the default npm behavior of always source compiling with node-gyp
directly.binary
section describing key properties you provide to allow node-pre-gyp to package optimally. They are detailed below.Note: in the past we recommended putting node-pre-gyp
in the bundledDependencies
, but we no longer recommend this. In the past there were npm bugs (with node versions 0.10.x) that could lead to node-pre-gyp not being available at the right time during install (unless we bundled). This should no longer be the case. Also, for a time we recommended using "preinstall": "npm install node-pre-gyp"
as an alternative method to avoid needing to bundle. But this did not behave predictably across all npm versions - see https://github.com/mapbox/node-pre-gyp/issues/260 for the details. So we do not recommend using preinstall
to install node-pre-gyp
. More history on this at https://github.com/strongloop/fsevents/issues/157#issuecomment-265545908.
binary
object has three required propertiesThe name of your native node module. This value must:
-
).node
extension.The location your native module is placed after a build. This should be an empty directory without other Javascript files. This entire directory will be packaged in the binary tarball. When installing from a remote package this directory will be overwritten with the contents of the tarball.
Note: This property supports variables based on Versioning.
A url to the remote location where you've published tarball binaries (must be https
not http
).
It is highly recommended that you use Amazon S3. The reasons are:
publish
and info
only work with an S3 host.Why then not require S3? Because while some applications using node-pre-gyp need to distribute binaries as large as 20-30 MB, others might have very small binaries and might wish to store them in a GitHub repo. This is not recommended, but if an author really wants to host in a non-S3 location then it should be possible.
It should also be mentioned that there is an optional and entirely separate npm module called node-pre-gyp-github which is intended to complement node-pre-gyp and be installed along with it. It provides the ability to store and publish your binaries within your repositories GitHub Releases if you would rather not use S3 directly. Installation and usage instructions can be found here, but the basic premise is that instead of using the node-pre-gyp publish
command you would use node-pre-gyp-github publish
.
binary
object has two optional propertiesIt is recommended that you customize this property. This is an extra path to use for publishing and finding remote tarballs. The default value for remote_path
is ""
meaning that if you do not provide it then all packages will be published at the base of the host
. It is recommended to provide a value like ./{name}/v{version}
to help organize remote packages in the case that you choose to publish multiple node addons to the same host
.
Note: This property supports variables based on Versioning.
It is not recommended to override this property unless you are also overriding the remote_path
. This is the versioned name of the remote tarball containing the binary .node
module and any supporting files you've placed inside the module_path
directory. Unless you specify package_name
in your package.json
then it defaults to {module_name}-v{version}-{node_abi}-{platform}-{arch}.tar.gz
which allows your binary to work across node versions, platforms, and architectures. If you are using remote_path
that is also versioned by ./{module_name}/v{version}
then you could remove these variables from the package_name
and just use: {node_abi}-{platform}-{arch}.tar.gz
. Then your remote tarball will be looked up at, for example, https://example.com/your-module/v0.1.0/node-v11-linux-x64.tar.gz
.
Avoiding the version of your module in the package_name
and instead only embedding in a directory name can be useful when you want to make a quick tag of your module that does not change any C++ code. In this case you can just copy binaries to the new version behind the scenes like:
aws s3 sync --acl public-read s3://mapbox-node-binary/sqlite3/v3.0.3/ s3://mapbox-node-binary/sqlite3/v3.0.4/
Note: This property supports variables based on Versioning.
node-pre-gyp
calls out to node-gyp
to compile the module and passes variables along like module_name and module_path.
A new target must be added to binding.gyp
that moves the compiled .node
module from ./build/Release/module_name.node
into the directory specified by module_path
.
Add a target like this at the end of your targets
list:
{
"target_name": "action_after_build",
"type": "none",
"dependencies": [ "<(module_name)" ],
"copies": [
{
"files": [ "<(PRODUCT_DIR)/<(module_name).node" ],
"destination": "<(module_path)"
}
]
}
For a full example see node-addon-example's binding.gyp.
.node
Inside the main js file that requires your addon module you are likely currently doing:
var binding = require('../build/Release/binding.node');
or:
var bindings = require('./bindings')
Change those lines to:
var binary = require('node-pre-gyp');
var path = require('path');
var binding_path = binary.find(path.resolve(path.join(__dirname,'./package.json')));
var binding = require(binding_path);
For a full example see node-addon-example's index.js
Now build your module from source:
npm install --build-from-source
The --build-from-source
tells node-pre-gyp
to not look for a remote package and instead dispatch to node-gyp to build.
Now node-pre-gyp
should now also be installed as a local dependency so the command line tool it offers can be found at ./node_modules/.bin/node-pre-gyp
.
Now npm test
should work just as it did before.
Then package your app:
./node_modules/.bin/node-pre-gyp package
Once packaged, now you can publish:
./node_modules/.bin/node-pre-gyp publish
Currently the publish
command pushes your binary to S3. This requires:
aws-sdk
with npm install aws-sdk
host
points to an S3 http or https endpoint.You can also host your binaries elsewhere. To do this requires:
package
command to an https
endpointhost
value points to your custom https
endpoint.Now you need to publish builds for all the platforms and node versions you wish to support. This is best automated.
Now publish your module to the npm registry. Users will now be able to install your module from a binary.
What will happen is this:
npm install <your package>
will pull from the npm registryinstall
script which will call out to node-pre-gyp
node-pre-gyp
will fetch the binary .node
module and unpack in the right placeIf a a binary was not available for a given platform and --fallback-to-build
was used then node-gyp rebuild
will be called to try to source compile the module.
N-API is an ABI-stable alternative to previous technologies such as nan which are tied to a specific Node runtime engine. N-API is Node runtime engine agnostic and guarantees modules created today will continue to run, without changes, into the future.
Using node-pre-gyp
with N-API projects requires a handful of additional configuration values and imposes some additional requirements.
The most significant difference is that an N-API module can be coded to target multiple N-API versions. Therefore, an N-API module must declare in its package.json
file which N-API versions the module is designed to run against. In addition, since multiple builds may be required for a single module, path and file names must be specified in way that avoids naming conflicts.
napi_versions
array propertyAn N-API modules must declare in its package.json
file, the N-API versions the module is intended to support. This is accomplished by including an napi-versions
array property in the binary
object. For example:
"binary": {
"module_name": "your_module",
"module_path": "your_module_path",
"host": "https://your_bucket.s3-us-west-1.amazonaws.com",
"napi_versions": [1,3]
}
If the napi_versions
array property is not present, node-pre-gyp
operates as it always has. Including the napi_versions
array property instructs node-pre-gyp
that this is a N-API module build.
When the napi_versions
array property is present, node-pre-gyp
fires off multiple operations, one for each of the N-API versions in the array. In the example above, two operations are initiated, one for N-API version 1 and second for N-API version 3. How this version number is communicated is described next.
napi_build_version
valueFor each of the N-API module operations node-pre-gyp
initiates, it ensures that the napi_build_version
is set appropriately.
This value is of importance in two areas:
node-pre-gyp
itself which must assign appropriate path and file names to avoid collisions.NAPI_VERSION
for the C/C++ codeThe napi_build_version
value is communicated to the C/C++ code by adding this code to the binding.gyp
file:
"defines": [
"NAPI_VERSION=<(napi_build_version)",
]
This ensures that NAPI_VERSION
, an integer value, is declared appropriately to the C/C++ code for each build.
Note that earlier versions of this document recommended defining the symbol
NAPI_BUILD_VERSION
.NAPI_VERSION
is prefered because it used by the N-API C/C++ headers to configure the specific N-API veriosn being requested.
package.json
Since node-pre-gyp
fires off multiple operations for each request, it is essential that path and file names be created in such a way as to avoid collisions. This is accomplished by imposing additional path and file naming requirements.
Specifically, when performing N-API builds, the {napi_build_version}
text configuration value must be present in the module_path
property. In addition, the {napi_build_version}
text configuration value must be present in either the remote_path
or package_name
property. (No problem if it's in both.)
Here's an example:
"binary": {
"module_name": "your_module",
"module_path": "./lib/binding/napi-v{napi_build_version}",
"remote_path": "./{module_name}/v{version}/{configuration}/",
"package_name": "{platform}-{arch}-napi-v{napi_build_version}.tar.gz",
"host": "https://your_bucket.s3-us-west-1.amazonaws.com",
"napi_versions": [1,3]
}
You may have a legacy native add-on that you wish to continue supporting for those versions of Node that do not support N-API, as you add N-API support for later Node versions. This can be accomplished by specifying the node_napi_label
configuration value in the package.json binary.package_name
property.
Placing the configuration value node_napi_label
in the package.json binary.package_name
property instructs node-pre-gyp
to build all viable N-API binaries supported by the current Node instance. If the current Node instance does not support N-API, node-pre-gyp
will request a traditional, non-N-API build.
The configuration value node_napi_label
is set by node-pre-gyp
to the type of build created, napi
or node
, and the version number. For N-API builds, the string contains the N-API version nad has values like napi-v3
. For traditional, non-N-API builds, the string contains the ABI version with values like node-v46
.
Here's how the binary
configuration above might be changed to support both N-API and NAN builds:
"binary": {
"module_name": "your_module",
"module_path": "./lib/binding/{node_napi_label}",
"remote_path": "./{module_name}/v{version}/{configuration}/",
"package_name": "{platform}-{arch}-{node_napi_label}.tar.gz",
"host": "https://your_bucket.s3-us-west-1.amazonaws.com",
"napi_versions": [1,3]
}
The C/C++ symbol NAPI_VERSION
can be used to distinguish N-API and non-N-API builds. The value of NAPI_VERSION
is set to the integer N-API version for N-API builds and is set to 0
for non-N-API builds.
For example:
#if NAPI_VERSION
// N-API code goes here
#else
// NAN code goes here
#endif
The following two configuration values, which were implemented in previous versions of node-pre-gyp
, continue to exist, but have been replaced by the node_napi_label
configuration value described above.
napi_version
If N-API is supported by the currently executing Node instance, this value is the N-API version number supported by Node. If N-API is not supported, this value is an empty string.
node_abi_napi
If the value returned for napi_version
is non empty, this value is 'napi'
. If the value returned for napi_version
is empty, this value is the value returned for node_abi
.
These values are present for use in the binding.gyp
file and may be used as {napi_version}
and {node_abi_napi}
for text substituion in the binary
properties of the package.json
file.
You can host wherever you choose but S3 is cheap, node-pre-gyp publish
expects it, and S3 can be integrated well with Travis.ci to automate builds for OS X and Ubuntu, and with Appveyor to automate builds for Windows. Here is an approach to do this:
First, get setup locally and test the workflow:
And have your key and secret key ready for writing to the bucket.
It is recommended to create a IAM user with a policy that only gives permissions to the specific bucket you plan to publish to. This can be done in the IAM console by: 1) adding a new user, 2) choosing Attach User Policy
, 3) Using the Policy Generator
, 4) selecting Amazon S3
for the service, 5) adding the actions: DeleteObject
, GetObject
, GetObjectAcl
, ListBucket
, PutObject
, PutObjectAcl
, 6) adding an ARN of arn:aws:s3:::bucket/*
(replacing bucket
with your bucket name), and finally 7) clicking Add Statement
and saving the policy. It should generate a policy like:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1394587197000",
"Effect": "Allow",
"Action": [
"s3:DeleteObject",
"s3:GetObject",
"s3:GetObjectAcl",
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::node-pre-gyp-tests/*"
]
}
]
}
Either install it globally:
npm install node-pre-gyp -g
Or put the local version on your PATH
export PATH=`pwd`/node_modules/.bin/:$PATH
There are several ways to do this.
You can use any of the methods described at http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/node-configuring.html.
Or you can create a ~/.node_pre_gyprc
Or pass options in any way supported by RC
A ~/.node_pre_gyprc
looks like:
{
"accessKeyId": "xxx",
"secretAccessKey": "xxx"
}
Another way is to use your environment:
export node_pre_gyp_accessKeyId=xxx
export node_pre_gyp_secretAccessKey=xxx
You may also need to specify the region
if it is not explicit in the host
value you use. The bucket
can also be specified but it is optional because node-pre-gyp
will detect it from the host
value.
Install the aws-sdk
:
npm install aws-sdk
Then publish:
node-pre-gyp package publish
Note: if you hit an error like Hostname/IP doesn't match certificate's altnames
it may mean that you need to provide the region
option in your config.
Appveyor can build binaries and publish the results per commit and supports:
For an example of doing this see node-sqlite3's appveyor.yml.
Below is a guide to getting set up:
Go to https://ci.appveyor.com/signup/free and sign in with your GitHub account.
Go to https://ci.appveyor.com/projects/new and select the GitHub repo for your module
Once you have committed an appveyor.yml
(appveyor.yml reference) to your GitHub repo and pushed it AppVeyor should automatically start building your project.
Encrypt your S3 AWS keys by going to https://ci.appveyor.com/tools/encrypt and hitting the encrypt
button.
Then paste the result into your appveyor.yml
environment:
node_pre_gyp_accessKeyId:
secure: Dn9HKdLNYvDgPdQOzRq/DqZ/MPhjknRHB1o+/lVU8MA=
node_pre_gyp_secretAccessKey:
secure: W1rwNoSnOku1r+28gnoufO8UA8iWADmL1LiiwH9IOkIVhDTNGdGPJqAlLjNqwLnL
NOTE: keys are per account but not per repo (this is difference than Travis where keys are per repo but not related to the account used to encrypt them).
Just put node-pre-gyp package publish
in your appveyor.yml
after npm install
.
You might wish to publish binaries only on a specific commit. To do this you could borrow from the Travis CI idea of commit keywords and add special handling for commit messages with [publish binary]
:
SET CM=%APPVEYOR_REPO_COMMIT_MESSAGE%
if not "%CM%" == "%CM:[publish binary]=%" node-pre-gyp --msvs_version=2013 publish
If your commit message contains special characters (e.g. &
) this method might fail. An alternative is to use PowerShell, which gives you additional possibilities, like ignoring case by using ToLower()
:
ps: if($env:APPVEYOR_REPO_COMMIT_MESSAGE.ToLower().Contains('[publish binary]')) { node-pre-gyp --msvs_version=2013 publish }
Remember this publishing is not the same as npm publish
. We're just talking about the binary module here and not your entire npm package.
Travis can push to S3 after a successful build and supports both:
For an example of doing this see node-add-example's .travis.yml.
Note: if you need 32 bit binaries, this can be done from a 64 bit Travis machine. See the node-sqlite3 scripts for an example of doing this.
Below is a guide to getting set up:
gem install travis
Make sure you run this command from within the directory of your module.
Use travis-encrypt
like:
travis encrypt node_pre_gyp_accessKeyId=${node_pre_gyp_accessKeyId}
travis encrypt node_pre_gyp_secretAccessKey=${node_pre_gyp_secretAccessKey}
Then put those values in your .travis.yml
like:
env:
global:
- secure: F+sEL/v56CzHqmCSSES4pEyC9NeQlkoR0Gs/ZuZxX1ytrj8SKtp3MKqBj7zhIclSdXBz4Ev966Da5ctmcTd410p0b240MV6BVOkLUtkjZJyErMBOkeb8n8yVfSoeMx8RiIhBmIvEn+rlQq+bSFis61/JkE9rxsjkGRZi14hHr4M=
- secure: o2nkUQIiABD139XS6L8pxq3XO5gch27hvm/gOdV+dzNKc/s2KomVPWcOyXNxtJGhtecAkABzaW8KHDDi5QL1kNEFx6BxFVMLO8rjFPsMVaBG9Ks6JiDQkkmrGNcnVdxI/6EKTLHTH5WLsz8+J7caDBzvKbEfTux5EamEhxIWgrI=
More details on Travis encryption at http://about.travis-ci.org/docs/user/encryption-keys/.
Just put node-pre-gyp package publish
in your .travis.yml
after npm install
.
If you want binaries for OS X in addition to linux you can enable multi-os for Travis
Use a configuration like:
language: cpp
os:
- linux
- osx
env:
matrix:
- NODE_VERSION="4"
- NODE_VERSION="6"
before_install:
- rm -rf ~/.nvm/ && git clone --depth 1 https://github.com/creationix/nvm.git ~/.nvm
- source ~/.nvm/nvm.sh
- nvm install $NODE_VERSION
- nvm use $NODE_VERSION
See Travis OS X Gotchas for why we replace language: node_js
and node_js:
sections with language: cpp
and a custom matrix.
Also create platform specific sections for any deps that need install. For example if you need libpng:
- if [ $(uname -s) == 'Linux' ]; then apt-get install libpng-dev; fi;
- if [ $(uname -s) == 'Darwin' ]; then brew install libpng; fi;
For detailed multi-OS examples see node-mapnik and node-sqlite3.
First, unlike the Travis Linux machines, the OS X machines do not put node-pre-gyp
on PATH by default. To do so you will need to:
export PATH=$(pwd)/node_modules/.bin:${PATH}
Second, the OS X machines do not support using a matrix for installing different Node.js versions. So you need to bootstrap the installation of Node.js in a cross platform way.
By doing:
env:
matrix:
- NODE_VERSION="4"
- NODE_VERSION="6"
before_install:
- rm -rf ~/.nvm/ && git clone --depth 1 https://github.com/creationix/nvm.git ~/.nvm
- source ~/.nvm/nvm.sh
- nvm install $NODE_VERSION
- nvm use $NODE_VERSION
You can easily recreate the previous behavior of this matrix:
node_js:
- "4"
- "6"
You might wish to publish binaries only on a specific commit. To do this you could borrow from the Travis CI idea of commit keywords and add special handling for commit messages with [publish binary]
:
COMMIT_MESSAGE=$(git log --format=%B --no-merges -n 1 | tr -d '\n')
if [[ ${COMMIT_MESSAGE} =~ "[publish binary]" ]]; then node-pre-gyp publish; fi;
Then you can trigger new binaries to be built like:
git commit -a -m "[publish binary]"
Or, if you don't have any changes to make simply run:
git commit --allow-empty -m "[publish binary]"
WARNING: if you are working in a pull request and publishing binaries from there then you will want to avoid double publishing when Travis CI builds both the push
and pr
. You only want to run the publish on the push
commit. See https://github.com/Project-OSRM/node-osrm/blob/8eb837abe2e2e30e595093d16e5354bc5c573575/scripts/is_pr_merge.sh which is called from https://github.com/Project-OSRM/node-osrm/blob/8eb837abe2e2e30e595093d16e5354bc5c573575/scripts/publish.sh for an example of how to do this.
Remember this publishing is not the same as npm publish
. We're just talking about the binary module here and not your entire npm package. To automate the publishing of your entire package to npm on Travis see http://about.travis-ci.org/docs/user/deployment/npm/
The binary
properties of module_path
, remote_path
, and package_name
support variable substitution. The strings are evaluated by node-pre-gyp
depending on your system and any custom build flags you passed.
node_abi
: The node C++ ABI
number. This value is available in Javascript as process.versions.modules
as of >= v0.10.4 >= v0.11.7
and in C++ as the NODE_MODULE_VERSION
define much earlier. For versions of Node before this was available we fallback to the V8 major and minor version.platform
matches node's process.platform
like linux
, darwin
, and win32
unless the user passed the --target_platform
option to override.arch
matches node's process.arch
like x64
or ia32
unless the user passes the --target_arch
option to override.libc
matches require('detect-libc').family
like glibc
or musl
unless the user passes the --target_libc
option to override.configuration
- Either 'Release' or 'Debug' depending on if --debug
is passed during the build.module_name
- the binary.module_name
attribute from package.json
.version
- the semver version
value for your module from package.json
(NOTE: ignores the semver.build
property).major
, minor
, patch
, and prelease
match the individual semver values for your module's version
build
- the sevmer build
value. For example it would be this.that
if your package.json version
was v1.0.0+this.that
prerelease
- the semver prerelease
value. For example it would be alpha.beta
if your package.json version
was v1.0.0-alpha.beta
The options are visible in the code at https://github.com/mapbox/node-pre-gyp/blob/612b7bca2604508d881e1187614870ba19a7f0c5/lib/util/versioning.js#L114-L127
S3 is broken in China for the well known reason.
Using the npm
config argument: --{module_name}_binary_host_mirror
can download binary files through a mirror.
e.g.: Install v8-profiler from npm
.
$ npm install v8-profiler --profiler_binary_host_mirror=https://npm.taobao.org/mirrors/node-inspector/
© 2010 - cnpmjs.org x YWFE | Home | YWFE