Bash crashes on execution of command.run script when a large number of files need to be staged [AWS Batch]

[fstrozzi]

Bug report

Expected behavior and actual behavior

When using AWS Batch executor and running a single Nextflow step that needs to stage thousands of files from S3, the execution of the bash script command.run gives a segmentation fault error.

This is most probably due to the fact that all the names and paths of the files that need to be staged on the AWS Batch job container are put in a single bash array for the execution of multiple download function. The process then, once executed, goes out of the bash stack size limit.

Steps to reproduce the problem

mkdir many_files_test && cd many_files_test
for ((n=0;n<100000;n++)); do touch dummy_file_${n}.txt; done
aws s3 sync ./ s3://bucket/tmp-dir/

Then simply run a Nextflow process that retrieves all the files using a

Channel.fromPath("s3://bucket/tmp-dir/dummy_file_*").collect()

Program output

nxf-scratch-dir ip-10-0-2-243:/tmp/nxf.naKegDjemO
bash: line 1:    Done  aws --region eu-west-1 s3 cp --only-show-errors s3://bucket/nextflow-tmp/63/00e73447d2269f625d32e1b1d8a081/.command.run -
Segmentation fault      | bash 2>&1
| tee .command.log

Environment

• Nextflow version: 19.07
• Java version: 11
• Operating system: macOS Mojave
• Executor: AWS Batch

Additional context

Setting ulimit -s unlimited in the .command.run script has proven to be a workaround solution for this problem.

[gsainsbury86]

I accidentally opened a duplicate issue. I have provided some more details here:
#1385

In my testing, you only needed about 5000 input files before the problem arose, though that might depend on the lengths of the paths.

Another workaround was to run the body of the nxf_stage() function as a separate bash script from within the correct work directory.

I'm now trying to see if there is a way to put the ulimit -s unlimited command within the nextflow configuration or process somewhere such that it is executed before the staging step.

[gsainsbury86]

Successful workaround adding beforeScript ulimit -Ss unlimited to process.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tamuanand]

Hi @pditommaso @gsainsbury86 @fstrozzi

I am having a similar issue and I added this below to my awsbatch profile within nextflow.config

 awsbatch {
        process {
           container = 'ecr_container'
           executor = 'awsbatch'
           queue = 'named-queue'
           errorStrategy = 'retry'
           maxRetries = 2
           beforeScript = 'ulimit -s unlimited'
           }

However, I get this error when I try to launch my job:

main: line 35178: ulimit: stack size: cannot modify limit: Invalid argument

I would appreciate your help.

Thanks in advance.

[fstrozzi]

Hi @tamuanand,
It is possible that this might be due to a newer version of Docker being used on the AMI where your Batch jobs run on. Basically the limit is enforced directly by Docker and it is not modifiable via the bash command.
I’ve seen this happening already, the solution is to specify the unlimit parameter directly for Docker via the AWS Batch Job Defition, have a look here https://docs.aws.amazon.com/batch/latest/userguide/job_definition_parameters.html#containerProperties . You can then specify an ulimit which is reasonably high, such as 1 or 2 GB for instance.
That should work and I hope it helps.

[tamuanand]

Thanks @fstrozzi I am using Amazon Linux 2 AMI and that could possibly explain things as you stated in your answer

Paolo moved my GH issue to a discussion and suggested using containerOptions I have asked Paolo a question there about how to mimic ulimit -s unlimited with containerOptions - probably you might know the answer as well. If you happen to know the answer, I suggest you answer it there so that it would be all in one place and benefit all. To try what Paolo says, for now, we have to use the latest edge release which is now 21.12.1-edge.5653

I am going to try that and I will post my findings in that discussion page.

Thanks once again.

[njspix]

Hello all, I just ran into this issue on a local HPC (not AWS), and I was wondering about the feasibility of rewriting the nxf_stage function. Would it be simple to just write the commands to a file (e.g. .command.stage) and then exec the file?