Fix mount points conflict causing empty project DLL in research environment

[efJerryYang]

Problem Description
When running lean research, the /LeanCLI directory inside the Docker container was missing all .cs source files, leaving only an generated .csproj by the tool. This caused dotnet build to emit the warning CS2008: No source files specified, resulting in an invalid project DLL (the content can be checked by ilspycmd) placed in /Lean/Launcher/bin/Debug/ that prevented referencing namespaced C# code of the project from Jupyter notebook.

Root Cause
The Lean CLI attempted to mount the project directory to two paths in the container:

1. /LeanCLI for C# compilation. This is needed for the correct compilation.
2. /Lean/Launcher/bin/Debug/Notebooks for Jupyter Lab working directory. This is also needed for accessing the host project directory content directly.

However, the volumes python dict uses host paths as keys. Subsequent mounts overwrote earlier ones, leaving only the last mount (to /Lean/Launcher/bin/Debug/Notebooks) active.

Additionally, the _ensure_csproj_is_valid method (called inside get_basic_docker_config) would generate temporary .csproj files and mounted them to /LeanCLI, which means we already have files mounted in /LeanCLI directory. Another relevant mount point that cause shadowing is the config.json for the Notebooks/ directory, therefore we'd better have the mounting for Notebooks/ prior to any other mounting in the Notebooks/ directory to ensure the shadowing working as expected.

Expected Behavior

We need to have /LeanCLI to be compatible with backtest/live, and allow it to compile the project DLL correctly. This compilation is still needed for research because in C# jupyter kernel, we can only load script files, those C# source files containing namespace are not allowed to be used directly in notebook cells and therefore can only be referenced by loading the corresponding project DLL. User can then run #r ../<project_name>.dll after the #load "../QuantConnect.csx".

Or even better, add a command to commands then we can echo "Assembly.LoadFrom($project_name);" >> /Lean/Launcher/bin/Debug/QuantConnect.csx to load the DLL automatically. But I am not sure if this would break anything, so for simplicity didn't do that in this PR.

Chosen Solution
Replaced volumes with explicit mounts for the project directory and move the mounting logic prior to all other Notebooks related mounting.

[jhonabreul]

Nice catch, thank you! This looks good, minor comment about the test below

[jhonabreul]

/LeanCLI is mounted as a volume, not as a bind mount, you will find it kwargs["volumes"][{project name}] as a dictionary {'bind': '/LeanCLI', 'mode': 'rw'}

[efJerryYang]

Hi @jhonabreul , thanks for your feedback!

I know /LeanCLI directory is mounted as volume, but /LeanCLI/<project_name>.csproj would be mounted using mounts, which was shown at run_options["mounts"].append(Mount(target=f"/LeanCLI/{csproj_path.relative_to(compile_root).as_posix()}", in above quoted line 315,

# ./lean/components/docker/lean_runner.py:829:        run_options["mounts"].append(Mount(target=f"/LeanCLI/{csproj_path.relative_to(compile_root).as_posix()}",

That's why I am trying to access the *.csproj file from kwargs["mounts"].

What I commented from line 314 to 317 was trying to show, the code for mounting the /LeanCLI/<project_name>.csproj would not be run during mock test which would actually run if we launch the real image (that's how I figured out the solution). I also noticed _ensure_csproj_is_valid is never been called during mock tests even I run for all tests, and then I will add a new comment there for clarification to avoid any confusion.

[jhonabreul]

Thank you!

I updated your unit test so that your commented assertions run.

Also the addition of Assembly.LoadFrom() was removed since I tested it and the notebook still needs the #r instruction to load the assembly. It won't work for Python notebooks either, so it would be better to find a more generic solution in the future.

[Martin-Molinero]

Thank you @efJerryYang! 🚀