Update outputs of conda commands to match whatever platform we decide to use

[SJaffa]

My quick test of the new JupyterHub instance found that conda does work (the command exists, I can list and create environments, install packages) but things like the default channels are different.

[SJaffa]

After creating the environment when I try to activate I get the error CondaError: Run 'conda init' before 'conda activate'. I ran conda init bash and retry activating, it still gives the error. I close nad repoen terminal and try activating, it still gives the error.

However source ~/.bashrc does work to activate the conda base env and allow me to then activate my env.

[SJaffa]

Also a couple of times when trying to install packages (conda install astropy pint) the terminal crashes or hangs.

When it crashed and I open a new terminal my environment no longer exists (conda env list only shows base)

[ianhinder]

conda init adds stuff to your shell startup files, and can't modify your current environment, so I think you would always have to log out and back in again to see the changes, as you did. It's odd that it doesn't find them when you logged back in. Can you check you actually are in bash, not something else like zsh? If it crashes and you open a new terminal, do you still see the environments in ~/.conda/envs? It's possible that your jupyter session is running in a container which crashes, and when you get a new one, you don't have the content of your home directory anymore, as you get a fresh container. Is anything else still there, after such a crash? The home directory would need to be persisted in a volume, rather than just existing in the ephemeral container itself.

[ianhinder]

Turns out I can log into that system, so I poked around. Is it open to everyone?

You do get a container for your session, and it has no .bashrc file or other shell startup files at all.

This link explains the reason that .bashrc might not be run in Terminal on a mac, and I suspect it is the same thing happening here.

The Jupyter Terminal is probably giving you a "login shell", and when bash is a "login shell", it doesn't run .bashrc, it runs .profile or .bash_profile. Usually, these would then source .bashrc. But since there is no .profile or .bash_profile, .bashrc is not being run.

Creating .bash_profile with "source .bashrc" in it does the trick for me, and when I open a new Terminal, it runs .bashrc.

[ianhinder]

Also a couple of times when trying to install packages (conda install astropy pint) the terminal crashes or hangs.

This might be due to the system being overloaded, or perhaps memory limits being placed on the containers, and these being too low for conda. You could run "watch free" in one Terminal while you try to install packages in another, to see if the memory is going up too much.

When it crashed and I open a new terminal my environment no longer exists (conda env list only shows base)

When you have an environment, where does it end up being created? I previously said .conda/envs, but when I create an environment, it goes in /opt:

jovyan@jupyter-ian-hinder---ee0c9d7a:~$ conda env list

# conda environments:
#
base                 * /opt/conda
ian                    /opt/conda/envs/ian

As an aside, to avoid the "conda init" fiasco of modifying your startup files and tying you to always use a specific conda installation for everything on your system, I prefer to activate environments with "source activate ". This is deprecated, as they want you to use the "conda" function and the shell startup thing, but I prefer it, and I doubt it is going away soon.

[ianhinder]

If I stop my jupyter server with File -> Hub control panel -> Stop server, and then restart it, I also lose my conda env in /opt/conda/envs, and I'm left with just the base environment, consistent with what you saw. So probably your server container crashed and had to be restarted, and /opt/conda is not being persisted.

Looking at the output of the "mount" command, we have

jovyan@jupyter-ian-hinder---ee0c9d7a:~$ mount|grep home
127.0.0.1:/ on /home/jovyan type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,noresvport,proto=tcp,port=20651,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1)

showing that /home/jovyan (your home directory) is an NFS mount from the host, and is likely persistent. However, there is nothing for /opt, so this will be obtained from the container image when it is started. You probably want to ask them to either configure conda to use your home directory, or to make /opt/conda persistent somehow. The former is probably more straightforward.

[ianhinder]

Here is a link that explains about how this can be done (on their end).

You could make conda store its envs in your home dir by creating .condarc in your home dir and adding

envs_dirs:
  - ~/.conda/envs

But we need to get this effect for all the course participants, if we want their environments to persist across possible server crashes.

[SJaffa]

Thanks for all the help @ianhinder, I'll get back to the admins and see if they can fix this for us.

[douglowe]

This does look to be a very promising solution.

Rather than create an extra config file we could ask the participants to move the .bashrc file:

mv .bashrc .profile

Getting the participants to add to their environment directory list can be done using conda config:

conda config --add envs_dirs /home/jovyan/.conda/envs

(and then conda config --show envs_dirs).

I think this is a nice example of how to adapt your conda configuration - and shouldn't be too much extra mental load for the learners.

[ianhinder]

For the admins, a better solution than storing the envs path in .condarc is to set the environment variable

CONDA_ENVS_PATH=/home/jovyan/.conda/envs

in the container when it starts. In the absence of their help, we can ask the participants to set the path directly, as you say, which sounds like a good idea.

Regarding .bashrc, are you meaning to get them to mv .bashrc .profile after conda init has written to it? I guess that would work. I can see them getting confused and doing things in the wrong order, though.

It would be better to get the admins to provide a more standard shell environment, so we don't have to make the participants do this sort of thing.

I know Doug sees these things as learning opportunities, but I'm hesitant. They are learning opportunities for the experts; they are just confusion and friction for the beginners who are struggling to follow as it is.

[douglowe]

I can consistently crash the IT-Services VM when trying to install astropy in the base environment. Attached is the system info as I'm trying to do this. Even though only one core and ~1/3 of the RAM is being used, it seems that the VM doesn't like having 1 CPU working heavily.

[ianhinder]

The conda process itself is using between 1 and 2 GB, and the other processes are using some as well. If I look at

jovyan@jupyter-ian-hinder---ee0c9d7a:~$ cat /sys/fs/cgroup/memory.max 
2147483648

then I see a 2 GB limit. (Found this here -- I assume they are using Kubernetes on AWS, though EKS or similar) Apparently the memory limitation stuff in containers is quite complicated, and there are lots of different places to look, but this suggests that the admins have imposed a 2 GB limit on each server container. @nigeldgreen, if you are in contact with the admins of this system, please can you ask them to increase the limit to 4 GB? I don't know if they should do this just for course participants, or globally. The former would need them to maintain a list of participants, whereas at the moment it seems to be open to everyone? At least, it was open to me without requesting anything. Maybe because I am in ITS?

[douglowe]

That's a strange limit to have, given the VM seems to have 4 cores and 16Gb RAM.

[ianhinder]

But how many users?

[ianhinder]

I don't think this is mainly for us, right?