How to use RetryAgent with Agent that has multiple Pool connections

[vitalygashkov]

I ran into a problem when trying to use RetryAgent with regular Agent when specifying multiple Pool connections.

My goal is to use concurrent download when downloading a large number of small 1-20 MB files. If I understand correctly, there is a connections option in the Agent class for this. But when using it in this way, the SocketError (like in issue #2412) occurs after some time. So I wanted to use RetryAgent, but I also got a different error: ClientClosedError: The client is closed.

Here's my part of the code, just for more understanding:

const { Agent, RetryAgent, stream } = require('undici');

const agent = new Agent({ maxRedirections: 3, connections: 8 });
const agentWithRetry = new RetryAgent(agent);

const urls = [...]
for (const url of urls) {
  stream(url, { dispatcher: agentWithRetry }, ...)
}

await agentWithRetry.close();

Could you please suggest how to solve this problem? Maybe I'm not using these classes correctly, or maybe it's a bug, or maybe there's just a more concise solution for this.

[metcoder95]

Hmm, haven't dig into it yet but have you tried using the interceptor instead?

[vitalygashkov]

Tried to do this today. It seems to have gotten rid of the error, but I encountered another problem: for some reason the final file is corrupted. I am downloading an mp4 file for example, and when downloading incorrectly, the MediaInfo program indicates IsTruncated=true property.

If I use something like fastq library with global dispatcher instead of Agent with connections option, the problem disappears and the file is downloaded correctly.

I haven't really figured out what the problem might be related to yet. I need to somehow debug the download process to understand at what point something unnecessary is written to the file (or maybe, on the contrary, something underwritten). Maybe later I'll try to make an reproduce example, if it's a bug.

[metcoder95]

That would be really appreciated