Native module

Some contents are borrowed from https://xcoder.in/2017/07/01/nodejs-addon-history/ (opens in a new tab)

The Nature of Native Modules

Let's start with the most essential C++ module development for Node.js. For example, we have a legitimate native module pinyin.linux-x64-gnu.node under Linux, which is actually a binary file that couldn't be seen properly in a text editor, until we came across the binary viewer.

The sharp-eyed reader will see that its Magic Number¹ is 0x7F454C46 and the ASCII code it presses is ELF, so the answer is obvious: it is a DLL file for Linux.

In fact, not just on Linux. When a C++ module of Node.js is compiled under OSX, you get a DLL with the suffix *.node which is essentially *.dylib, and under Windows, you get a DLL with the suffix *.node which is essentially *.dll.

Such a module, when required in Node.js, is required via process.dlopen(). Let's take a look at the DLOpen² function in Node.js v10.23.0 (opens in a new tab):

// DLOpen is process.dlopen(module, filename, flags).
void DLOpen(const FunctionCallbackInfo<Value>& args) {
  Environment* env = Environment::GetCurrent(args);
  auto context = env->context();
 
  Local<Object> module;
  Local<Object> exports;
  Local<Value> exports_v;
  // initialize `module`, `module.exports` values
  if (!args[0]->ToObject(context).ToLocal(&module) ||
      // this line is equal to `exports = module.exports`
      !module->Get(context, env->exports_string()).ToLocal(&exports_v) ||
      !exports_v->ToObject(context).ToLocal(&exports)) {
    return;  // Exception pending.
  }
 
  node::Utf8Value filename(env->isolate(), args[1]);  // Cast
  DLib dlib(*filename, flags);
  bool is_opened = dlib.Open();
 
  node_module* const mp = static_cast<node_module*>(
      uv_key_get(&thread_local_modpending));
  uv_key_set(&thread_local_modpending, nullptr);
 
  ...
 
  // transfer the handle in dynamic lib to the `mp`
  mp->nm_dso_handle = dlib.handle_;
  mp->nm_link = modlist_addon;
  modlist_addon = mp;
 
  if (mp->nm_context_register_func != nullptr) {
    mp->nm_context_register_func(exports, module, context, mp->nm_priv);
  } else if (mp->nm_register_func != nullptr) {
    mp->nm_register_func(exports, module, mp->nm_priv);
  } else {
    dlib.Close();
    env->ThrowError("Module has no declared entry point.");
    return;
  }
 
}

Logically, the loading process actually looks like this.

• Load the link library via uv_dlopen.
• Hook the loaded library into the native module chain table.
• Initialize the module with mp->nm_register_func(), and get the module and module.exports that are there.

The flow down is similar to this flowchart:

How to build native module

node-waf

Before Node.js 0.8, developers used the node-waf to build their library. Of course the node-waf is not the node-waf in npm registry, the original node-waf has been fallen to disrepair for years.

This thing was configured with a file named wscript. From Node.js 0.8, it had node-gyp builtin, so people didn't need wscript anymore.

But because this temporary shortage, many libraries using C++ to build Node.js addon contains both binding.gyp and wscript in that time.

You can see files back to that age in this library node-mysql-libmysqlclient (opens in a new tab). For node-gyp support it had binding.gyp and still preserved the wscript file.

node-gyp

This stuff It has been with Node.js since Node.js v0.8, before that its default compilation helper package was node-waf(see below), which should be familiar to old Noder.

GYP

node-gyp is based on GYP³. It recognizes the binding.gyp⁴ file in a package or project, and then generates compilable projects for each system based on that configuration file, such as Visual Studio project files (*.sln, etc.) for Windows and Makefiles for Unix. node-gyp can also invoke system compilation tools (such as GCC) to compile the project to a final DLL *.node file.

As you can see from the above description, compiling C++ native modules on Windows relies on Visual Studio, which is why you will need to have Visual Studio pre-installed to install some Node.js packages.
In fact, for users who don't need Visual Studio, it's not necessary, since node-gyp only relies on its compiler, not the IDE. Those who want to streamline the installation can visit https://download.microsoft.com/download/5/f/7/5f7acaeb-8363-451f-9425-68a90f98b238/visualcppbuildtools_full.exe (opens in a new tab) directly. cpp-build-tools Download the Visual CPP Build Tools installation, or install it from the npm install --global --production windows-build-tools command line to get the compilation tools you deserve.

Now that we have that out of the way, let's take a look at the basic structure of binding.gyp:

{
  "targets": [{
    "target_name": "addon1",
    "sources": [ "1/addon.cc", "1/myobject.cc" ]
  }, {
    "target_name": "addon2",
    "sources": [ "2/addon.cc", "2/myobject.cc" ]
  }, {
    "target_name": "addon3",
    "sources": [ "3/addon.cc", "3/myobject.cc" ]
  }, {
    "target_name": "addon4",
    "sources": [ "4/addon.cc", "4/myobject.cc" ]
  }]
}

This configuration tells the following story:

• Four C++ native modules are defined.
• The source code for each module is *.addon.cc and *.myobject.cc, respectively.
• The names of the four modules are addon1 to addon4.
• The hidden story: These modules exist in build/Release/addon*.node after compiling.

For more information on the GYP configuration file, you can go to the official documentation, which has a link to GYP in a footnote.

Something more in node-gyp

In addition to being GYP-based itself, node-gyp does a few extra things. First of all, when we compile a C++ native extension, it goes to the specified directory (usually ~/.node-gyp) and searches for our current version of Node.js headers and statically linked libraries, and if they don't exist, it feverishly goes to the Node.js website to download them.

This is a directory structure for the specific version of Node.js headers and libraries downloaded from node-gyp on macOS:

/Users/napi-rs/.node-gyp
└── 14.15.1
    └── include
        └── node
            ├── common.gypi
            ├── config.gypi
            ├── cppgc
            ├── js_native_api.h
            ├── js_native_api_types.h
            ├── libplatform
            ├── node.h
            ├── node_api.h
            ├── node_api_types.h
            ├── node_buffer.h
            ├── node_object_wrap.h
            ├── node_version.h
            ├── openssl
            ├── uv
            ├── uv.h
            ├── v8-fast-api-calls.h
            ├── v8-internal.h
            ├── v8-platform.h
            ├── v8-profiler.h
            ├── v8-util.h
            ├── v8-value-serializer-version.h
            ├── v8-version-string.h
            ├── v8-version.h
            ├── v8-wasm-trap-handler-posix.h
            ├── v8-wasm-trap-handler-win.h
            ├── v8.h
            └── v8config.h

This header directory will be merged into our binding.gyp in the form of an "include_dirs" field when node-gyp is compiled; in short, all the headers can be #include directly.

node-gyp is a command line program that can be run directly from $ node-gyp after installation. It has some subcommands for you to use.

• $ node-gyp configure: generates project files, such as Makefiles, from binding.gyp in the current directory.
• $ node-gyp build: build and compile the current project, which must be preceded by configure.
• $ node-gyp clean: cleans the resulting build files and output directories, in other words, cleans the directories.
• $ node-gyp rebuild: This is equivalent to the execution of clean, configure and build in sequence.
• $ node-gyp install: Manually download the header files and library files of the current version of Node.js to the appropriate directory.

Conclusion

In this chapter, we introduced what the Node.js native addon is and how to compile it. In the next chapter, we will review the history of API changes related to the native addon in Node.js and formally introduce our protagonist: the N-API.

References