Hemanth's Scribes

node

HTTP Imports on Node

Author Photo

Hemanth HM

Thumbnail

Node v18 shall have ESM import support over HTTPS remotely and HTTP locally and is currently under a flag.

This is experimental and things can change drastically, some of the features might be removed as well, so take it with a pinch of salt.

Example

import helloWorld from "https://h3manth.com/hello-world.mjs";
console.log(helloWorld()); // Log Hello World
import helloWorld from "http://localhost:1337/hello-world.mjs";
console.log(helloWorld()); // Log Hello World

As you might have noticed, this is still under an experimental flag: --experimental-network-imports

For now, the --experimental-network-imports flag is required to enable loading resources over http: or https:. In the future, a different mechanism will be used to enforce this. Opt-in is required to prevent transitive dependencies inadvertently using potentially mutable state that could affect reliability of Node.js applications.

The contents of https://h3manth.com/hello-world.mjs:

export default function helloWorld () {
  console.log('Hello, World!')
}

MIME Type Important

The most important thing to notice here is the MIME type of the URL.

$ curl -s -o /dev/null -w '%{content_type}' 'https://h3manth.com/hello-world.mjs'
application/javascript

The hosted mjs should respond with application/javascript as its content-type or else it will fail to load.

Authorization, Cookie, and Proxy-Authorization headers are not sent to the server and CORS policies/headers are neither sent nor enforced.

Common Errors

ERR_NETWORK_IMPORT_BAD_RESPONSE

Occurs when the URL gives a bad response, like 404:

import helloWorld from "https://h3manth.com/foo";
Error [ERR_NETWORK_IMPORT_BAD_RESPONSE]: import 'https://h3manth.com/foo' received a bad response: HTTP response returned status code of 404

ERR_NETWORK_IMPORT_DISALLOWED

Will occur when the request cannot redirect to non-network location, if HTTP is used for non-local resources:

import helloWorld from "https://h3manth.com/foo.mjs";
// Say this redirects to ftp://h3manth.com/bar.mjs
Error [ERR_NETWORK_IMPORT_DISALLOWED]: import of 'ftp://h3manth.com/bar.mjs' by 'https://h3manth.com/foo.mjs' is not support: cannot redirect to non-network location

ERR_UNKNOWN_MODULE_FORMAT

Occurs when the URL is not an ESM:

import helloWorld from "https://h3manth.com/hello.js";
RangeError [ERR_UNKNOWN_MODULE_FORMAT]: Unknown module format: null for URL https://h3manth.com/hello.js

ERR_UNSUPPORTED_ESM_URL_SCHEME

If your URL scheme is not supported by the default ESM loader:

import helloWorld from "ftp://foo.js";
console.log(helloWorld())
Error [ERR_UNSUPPORTED_ESM_URL_SCHEME]: Only URLs with a scheme in: file, data, https, http are supported by the default ESM loader. Received protocol 'ftp:'

How the Fetch Works

fetchModule does the basic sanity checks and also checks if it is already present in cache (cacheForGET) if not it invokes fetchWithRedirects:

function fetchModule(parsed, { parentURL }) {
  const { href } = parsed;
  const existing = cacheForGET.get(href);
  
  if (existing) {
    return existing;
  }
  
  if (parsed.protocol === 'http:') {
    return PromisePrototypeThen(isLocalAddress(parsed.hostname), (is) => {
      // Makes few checks for ERR_NETWORK_IMPORT_DISALLOWED
      return fetchWithRedirects(parsed);
    });
  }
  
  return fetchWithRedirects(parsed);
}
function fetchWithRedirects(parsed) {
  const existing = cacheForGET.get(parsed.href);
  
  if (existing) {
    return existing;
  }
  
  const handler = parsed.protocol === 'http:' ? HTTPGet : HTTPSGet;
  
  const result = new Promise((fulfill, reject) => {
    const req = handler(parsed, { headers: { Accept: '*/*' } })
      .on('error', reject)
      .on('response', (res) => {
        // Checks for multiple error cases that were mentioned above
        // Caches the content if required
        // Returns a promise which shall resolve to the module's content
      })
  })
}
let HTTPSAgent;

function HTTPSGet(url, opts) {
  const https = require('https'); // For HTTPGet, we use the `http` builtin
  HTTPSAgent ??= new https.Agent({ keepAlive: true });
  return https.get(url, { agent: HTTPSAgent, ...opts });
}

Summary

Summarizing the status of this experimental feature:

  • Disabled by default - requires --experimental-https-modules flag
  • Redirects - act like web
  • Selective HTTP support - only works on loopback
  • Debugging HTTPS on a local machine is painful - use loopback + http:
  • Cannot access node: builtins - banned anything except http: and https: deps.

Source of truth is the commit on the main branch of node src.

#node#javascript#esm#http
Author Photo

About Hemanth HM

Hemanth HM is a Sr. Machine Learning Manager at PayPal, Google Developer Expert, TC39 delegate, FOSS advocate, and community leader with a passion for programming, AI, and open-source contributions.