Repository navigation
Worker eval as ES Module #30682
Description
Activity
Adding
execArgv: ['--input-type=module']to the options doesn't help (and wouldn't be the best UX).Can I use Es6 module in Worker by using
new Worker("./foo.js", { type: "module" })Chrome 80 will enable this feature by default https://bugs.chromium.org/p/chromium/issues/detail?id=680046Reacted by Marvin Hagemeister, Jimmy Wärting and Zekiah@nodejs/modules-active-members
@mko-io if your file would be interpreted as a module when run with
node foo.js(in a"type": "module"package, or if the extension is.mjs), yes.Reacted by LinzI hope we'll support
type: modulefor workers. I assume in that case we could treat it as an implicit "also treat the source text as.mjs" (as opposed to supporting different kinds of modules ineval: true):import { Worker } from 'worker_threads'; new Worker( ` import { join } from 'path'; console.log(join('hello', 'world')); `, { eval: true, type: 'module' }, );
(Not working code, just as a possible example of what it would look like in node.)
Reacted by Anna Henningsen, Marvin Hagemeister, Dan Bornstein and Jimmy WärtingIf we do add a
type: 'module'option to the Worker constructor, should that also apply to non-eval code, i.e. be treated like a package.jsontype:option?I assume it should only
importinstead ofrequire(in terms of loading semantics). Generally speaking we've been hesitant to allow the importer to change the static semantics of the imported resource.import-semantics may also imply that the "filename" is a URL (e.g. support fordata:URLs).Since we don't support bare specifiers for workers and we don't support conditional mappings of effectively "absolute URLs", I assume that
type: moduleis mostly opting into an async worker bootstrap but not much else outside ofeval.Do workers already support the package.json
"type": "module"ok? If so then perhaps we don't needtype: 'module'as an option at all actually. The major argument for that would be web compatibility but we already don't have web compatibility for this API as a goal right?Perhaps eval can be supported via either
eval: 'module'or using a data URI with a MIME type corresponding to a module like we already support in the module loader as well?I think we should try to match the web worker API, which uses
type: classicandtype: module. lacking URL.createObjectURL, combining type:module with eval:true makes the most sense to me. The collision with the "type" name in package.json is unfortunate but not a huge deal imo.Reacted by Anna Henningsen, Timothy Gu, Marvin Hagemeister and Jimmy WärtingThe question is - why though?
why work towards compat with other runtimes? it's annoying and confusing when there's a difference. we also do try to work in compat where possible, like supporting on{event} properties.
Reacted by Anna Henningsen and PauanAs mentioned
Workeris already not web compatible though as it uses paths not URLs and also has API differences. So since a compatibility layer is needed regardless any attempt towards web compat is just "going through the motions", while also invalidating the work we've done that already ensures module types are well-defined.I guess one question is, could
Workersomeday be Web compatible? Like we may be lackingURL.createObjectURLright now, but it seems like something we could conceivably add someday, isn’t it? Since we have data URIs already. Likewise we’ve been batting around the idea ofimportfromhttpURLs, which isn’t possible now but my understanding is that that was mostly punted for difficulty (working out the mechanics of what allowing that would mean) rather than some philosophical opposition.Anyway my point is, if theoretically
Workermight be web-compatible someday once we work out all the other incompatible pieces around it, then we might as well stay compatible whenever possible now, to avoid breaking changes later when the only parts that are incompatible are the APIs we deliberately designed differently. And I agree with @devsnek, it’s a better UX to have the same (or as similar as possible) API forWorkerin both environments, if nothing else to help users remember it.Reacted by Michaël Zasso and snek12 remaining items
I ran into this issue today and there is a "workaround" to be able to execute ESM in worker as explained in #36985 (comment)
import { Worker } from "node:worker_threads" const worker = new Worker(new URL('data:text/javascript,import fs from "fs"'));
However the code inside the data url throws when it contains relative urls.
import { Worker } from 'node:worker_threads' const worker = new Worker(new URL('data:text/javascript,import "./file.mjs"'));
Ideally I could do something as below:
new Worker(` import value from "./file.js"; console.log(value); `, { eval: true, type: 'module', url: import.meta.url }, );
I understand it's not possible today but is there something I can do with the API offered by Node.js?
I don't know if that fits your use-case, but you should be able to do the following:
const toDataUrl = js => new URL(`data:text/javascript,${encodeURIComponent(js)}`); const toAbsoluteUrl = relativeUrl => JSON.stringify(new URL(relativeUrl, import.meta.url)); new Worker(toDataUrl(` import value from ${toAbsoluteUrl('./file.js')}; console.log(value); `) );
Reacted by icetbrConverting to absolute urls would work indeed!
In my use case the string passed to the worker is dynamic. But I can use babel to transform and resolve all relative import specifiers found in the code 🤔 .Good enough for me.
Edit: Actually no 😬 . There is several other cases where node throw
ERR_INVALID_URLerrorsnew Worker(toDataUrl(` import value from "leftpad"; console.log(value); `) );
☝️ In the code above
"leftpad"would be a package declared in my package.json. But as code is executed in the context of a data url, Node.js won't find my package.json.I guess I could resolve all specifiers with import.meta.resolve but it's experimental so I would be forced to put
--experimental-import-meta-resolvein all my node commands using this technic 😬It also doesn’t work properly, even with the flag.
If ESM is unavoidable, it may be sensible to write the eval'd code to file first before calling the worker.
// pseudo-code import {writeFile} from 'fs/promises'; await writeFile( './module-eval.js', `import something from 'somewhere'; console.log(something.doSomething()); `, 'utf8' ); const worker = new Worker('./module-eval.js', {type: 'module'});
Note: This assumes top-level await support.
We are now at 2023, and still no
{type:"module"}support, and it waste three engineers one hour to struggle and finally I find this issue. 😒Reacted by RobMinReacted by intrnl and Pablo FernándezAnd of course those three engineers are going to invest their time to contribute a fix, or they are simply going to waste everyone’s time complaining it’s 2023?
Reacted by Moshe Atlow, Ben Noordhuis, Fabio Spampinato, Geoffrey Booth, James Mortensen, Inaiat Henrique, DeepJoyPo, Jordan Harband, Pablo Fernández and Patrick KerschbaumA lot reasons to complain about it being 2023, to be fair.
Reacted by Antoine du Hamel, Pauan and Troy Kellynew (require('worker_threads').Worker)( ` import { join } from 'path'; console.log(join('hello', 'world')); `, { eval: true, type: 'module' }, );
import { Worker } from 'worker_threads'; new Worker( ` import { join } from 'path'; console.log(join('hello', 'world')); `, { eval: true, type: 'module' }, );
AFAICT both these work, so I've closed this issue. LMK if I'm missing something.
I'll have a look. Please don't close my issues unless there's an obvious reason (like a PR that fixes it)
It also works without
type: 'module'. This is unexpected.It also works without
type: 'module'. This is unexpected.Not really,
type: 'module'is completely ignored – it's not an option the constructor supports. The snippet works unless you run it with the--no-experimental-detect-moduleCLI flag.I know it's ignored. What I find unexpected is that module detection happens here (in the eval path)
I don't think it's unexpected (at least I don't find it surprinsing), string inputs being affected was mentioned in the original PR that added the flag (#50096 (comment)). However, what I find surprising is that the snippet is also broken if you pass
--input-type=modulein the CLI flags:$ node <<'EOF' import { Worker } from 'worker_threads'; new Worker( ` import { join } from 'path'; console.log(join('hello', 'world')); `, { eval: true }, ); EOF hello/world $ node --input-type=module <<'EOF' import { Worker } from 'worker_threads'; new Worker( ` import { join } from 'path'; console.log(join('hello', 'world')); `, { eval: true }, ); EOF node:internal/event_target:1090 process.nextTick(() => { throw err; }); ^ [worker eval]:2 import { join } from 'path'; ^^^^^^ SyntaxError: Cannot use import statement outside a module at makeContextifyScript (node:internal/vm:185:14) at node:internal/process/execution:107:22 at [worker eval]-wrapper:6:24 at runScript (node:internal/process/execution:101:62) at evalScript (node:internal/process/execution:136:3) at MessagePort.<anonymous> (node:internal/main/worker_thread:167:9) at [nodejs.internal.kHybridDispatch] (node:internal/event_target:816:20) at MessagePort.<anonymous> (node:internal/per_context/messageport:23:28) Node.js v23.0.0-pre
Unless I missed it, there doesn't seem to be a way of creating a Worker from an ESM source string (with
eval: true).Example that should be possible somehow:
Related: #21502