GDExporter cannot find GDevelop Core

hiran · May 10, 2022, 6:07am

Is GDExporter still a supported component of the GDevelop experience?

Quite some time ago I added gd-exporter into my ci-cd pipeline. Recently the pipeline fails with this output, and I guess the fix is to change permissions on the published GDevelop installation.

+ /usr/local/lib/node_modules/gdexporter/bin/cli -p /project/game.json -o /output
📓 No config file found!
⌛ Loading plugins...
⌛ Loading GDCore...
🕗 Getting latest release tag...
❌ The GDevelop version was not found, downloading it!
🕗 Starting download of GDevelop Runtime 'v5.0.132'...
🕗 Starting download of GDevelop Core...
/usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/downloadGD.js:12
          throw new Error(
          ^

Error: ❌ Cannot download https://s3.amazonaws.com/gdevelop-gdevelop.js/master/commit/bb3abdb1fa85a2d9e89e5053ae47deef1008299a/libGD.js! Error 403: Forbidden
    at RedirectableRequest.<anonymous> (/usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/downloadGD.js:12:17)
    at RedirectableRequest.emit (node:events:527:28)
    at RedirectableRequest._processResponse (/usr/local/lib/node_modules/gdexporter/node_modules/follow-redirects/index.js:346:10)
    at ClientRequest.RedirectableRequest._onNativeResponse (/usr/local/lib/node_modules/gdexporter/node_modules/follow-redirects/index.js:57:10)
    at Object.onceWrapper (node:events:642:26)
    at ClientRequest.emit (node:events:527:28)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (node:_http_client:631:27)
    at HTTPParser.parserOnHeadersComplete (node:_http_common:128:17)
    at TLSSocket.socketOnData (node:_http_client:494:22)
    at TLSSocket.emit (node:events:527:28)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:234:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)

Node.js v17.9.0
+ exit 1

Gruk · May 10, 2022, 8:26pm

Well, I’m not sure if it has ever really been a component of GDevelop, but let’s ask the author.
@arthuro555, are you maintaining GDExporter? Is it possible that something got… outdated? See OP.

arthuro555 · May 10, 2022, 9:31pm

GDexporter is and always was a community project, not a part of the GDevelop Company’s official ecosystem.

This error is quite odd and I’ll investigate it later. That URL is part of the GDevelop infrastructure where I fetch prebuilt binaries from. As far as I am aware, this should still be existing as it is as far as I recall also in use in the GDevelop repository

Yes, I still maintain and use GDexporter, and will update it if it has to be.

hiran · May 13, 2022, 7:03am

I was not aware there is a company behind GDevelop.
But yes, if that URL is part of GDevelop infrastructure it may need a fix anyway.

The GDExporter is doing a marvellous job - I am really happy about it.

arthuro555 · May 13, 2022, 7:08am

That’s understandable, it has been founded not too long ago for the maintainer and a few other people to get to work full-time on GDevelop.

I’m glad it can be useful to you

arthuro555 · May 13, 2022, 4:30pm

So, I just took a look and think I pinpointed the issue. The commit linked to this release is the following one:

As you can see, it contains a [skip ci] tag, which prevented the CI responsible for building and uploading this library from running for this commit.
I’ve fixed this on version 3.2.7, by making it use the nearest parent commit without the skip-ci tag.

hiran · May 13, 2022, 8:30pm

Good match. My last successful run was on APR22, the first failure on APR30.
For the root cause I am not sure I can follow. But I can let you know once it works again.

Just now it still failed. BTW, version number 3.2.7 refers to GDExporter, doesn’t it? Does GDExporter also output that version? So far I am sure to run the latest available version but cannot tell exactly which one it is…

+ /usr/local/lib/node_modules/gdexporter/bin/cli -p /project/game.json -o /output
📓 No config file found!
⌛ Loading plugins...
⌛ Loading GDCore...
🕗 Getting latest release tag...
❌ The GDevelop version was not found, downloading it!
🕗 Starting download of GDevelop Runtime 'v5.0.132'...
🕗 Starting download of GDevelop Core...
/usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/downloadGD.js:12
          throw new Error(
          ^

Error: ❌ Cannot download https://s3.amazonaws.com/gdevelop-gdevelop.js/master/commit/bb3abdb1fa85a2d9e89e5053ae47deef1008299a/libGD.js! Error 403: Forbidden
    at RedirectableRequest.<anonymous> (/usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/downloadGD.js:12:17)
    at RedirectableRequest.emit (node:events:527:28)
    at RedirectableRequest._processResponse (/usr/local/lib/node_modules/gdexporter/node_modules/follow-redirects/index.js:346:10)
    at ClientRequest.RedirectableRequest._onNativeResponse (/usr/local/lib/node_modules/gdexporter/node_modules/follow-redirects/index.js:57:10)
    at Object.onceWrapper (node:events:642:26)
    at ClientRequest.emit (node:events:527:28)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (node:_http_client:631:27)
    at HTTPParser.parserOnHeadersComplete (node:_http_common:128:17)
    at TLSSocket.socketOnData (node:_http_client:494:22)
    at TLSSocket.emit (node:events:527:28)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:234:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)

Node.js v17.9.0
+ exit 1

hiran · June 24, 2022, 5:53am

So something indeed changed. Trying to compile again I have a new error. I am not sure whether it is connected or a different issue, but I’ll share it here:

TypeError: Failed to parse URL from /usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/Versions/v5.0.137/libGD.wasm
    at new Request (node:internal/deps/undici/undici:4813:19)
    at Agent.fetch2 (node:internal/deps/undici/undici:5505:29)
    ... 6 lines matching cause stack trace ...
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Emitted 'error' event on WrappedGD instance at:
    at Object.onAbort (/usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/WrappedGD.js:51:28)
    at process.abort (/usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/Versions/v5.0.137/libGD.js:9:15232)
    at process.emit (node:events:527:28)
    at emit (node:internal/process/promises:149:20)
    at processPromiseRejections (node:internal/process/promises:283:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:96:32) {
  [cause]: TypeError [ERR_INVALID_URL]: Invalid URL
      at new NodeError (node:internal/errors:377:5)
      at URL.onParseError (node:internal/url:563:9)
      at new URL (node:internal/url:643:5)
      at new Request (node:internal/deps/undici/undici:4811:25)
      at Agent.fetch2 (node:internal/deps/undici/undici:5505:29)
      at Object.fetch (node:internal/deps/undici/undici:6332:20)
      at fetch (node:internal/bootstrap/pre_execution:196:25)
      at instantiateAsync (/usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/Versions/v5.0.137/libGD.js:9:17055)
      at createWasm (/usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/Versions/v5.0.137/libGD.js:9:17655)
      at /usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/Versions/v5.0.137/libGD.js:9:95296
      at getGD (/usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/downloadGD.js:208:57)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
    input: '/usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/Versions/v5.0.137/libGD.wasm',
    code: 'ERR_INVALID_URL'
  }
}

arthuro555 · June 24, 2022, 6:07am

try deleting node_modules/gdcore-tools and reinstalling gdexporter

hiran · June 24, 2022, 8:42pm

Thank you for the hint. Actually I never delete stuff - I build from scratch. My pipeline triggers automatically once a month. It builds a docker container for gdexporter on it’s own using this dockerfile:

FROM node

RUN mkdir /project
RUN mkdir /output
RUN npm i -g gdexporter

COPY entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
#CMD [ "-p", "/project", "-o", "/output" ]

VOLUME ["/usr/local/lib/node_modules/gdexporter/node_modules"]

With this container freshly created I then compiled my projects and got this output:

TypeError: Failed to parse URL from /usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/Versions/v5.0.138/libGD.wasm
    at new Request (node:internal/deps/undici/undici:4832:19)
    at Agent.fetch2 (node:internal/deps/undici/undici:5524:29)
    ... 6 lines matching cause stack trace ...
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Emitted 'error' event on WrappedGD instance at:
    at Object.onAbort (/usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/WrappedGD.js:51:28)
    at process.abort (/usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/Versions/v5.0.138/libGD.js:9:15232)
    at process.emit (node:events:537:28)
    at emit (node:internal/process/promises:149:20)
    at processPromiseRejections (node:internal/process/promises:283:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:96:32) {
  [cause]: TypeError [ERR_INVALID_URL]: Invalid URL
      at new NodeError (node:internal/errors:388:5)
      at URL.onParseError (node:internal/url:564:9)
      at new URL (node:internal/url:644:5)
      at new Request (node:internal/deps/undici/undici:4830:25)
      at Agent.fetch2 (node:internal/deps/undici/undici:5524:29)
      at Object.fetch (node:internal/deps/undici/undici:6351:20)
      at fetch (node:internal/bootstrap/pre_execution:197:25)
      at instantiateAsync (/usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/Versions/v5.0.138/libGD.js:9:17055)
      at createWasm (/usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/Versions/v5.0.138/libGD.js:9:17655)
      at /usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/Versions/v5.0.138/libGD.js:9:95296
      at getGD (/usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/downloadGD.js:208:57)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
    input: '/usr/local/lib/node_modules/gdexporter/node_modules/gdcore-tools/src/Versions/v5.0.138/libGD.wasm',
    code: 'ERR_INVALID_URL'
  }
}

Node.js v18.4.0

This time I added one more line of output, displaying the node version number.
It seems to be that apart from the gdexporter version number nothing changed.

hiran · June 24, 2022, 8:49pm

Oh, I see there is a reference to a docker volume that might contain the files as a cache. But the pipeline that compiles the gdevelop projects (there are a handful) creates the volume, runs gdexporter for each project and then removes the volume again. Thus even if that volume would contain wrong files - it gets recreated from scatch every time I compile.

Hence I believe it is not a caching issue.

arthuro555 · June 24, 2022, 9:12pm

I think I use an older LTS version of node, either v14 or v16. Maybe there was a breaking change in modules resolution causing this to happen?

hiran · June 25, 2022, 10:09am

Indeed, that fixed it. I just changed the first line in Dockerfile to
FROM node:lts

Now the whole setup is based on node 16.15.1 and the error is gone.