NEET Spring 2019 Server

Keep this in mind!

The NEET Server is a shared resource! Coordinate the times you are using the server with the other NEET teams so that you do not overlap. And remember to shut down your Jupyter Notebook or training script after you are done. Do not leave a training script running - it will reserve all available GPU memory, and prevent other teams from training.

Concurrent usage by multiple teams: If you and another team agree beforehand, you can limit the memory TensorFlow allocates while training, and both teams can train on the server simultaneously. Again remember to shut down jupyter and/or your python training scripts when you are done. Additionally, note that if both teams want to concurrently run Jupyter notebook instances, you cannot share the same port (by default, 8888). Different ports will be auto-assigned. You can manually specify a port with jupyter notebook --no-browser --port ####, for example, --port 8765). You will need to ensure the port for your notebook instance is forwarded via ssh.

Please ask the instructors before running any commands with sudo rights.

Lastly, please do not reboot the server! Again, ask if you have questions.

Installing Dependencies

There's nothing to do! The dependencies you need for the Imitation learning lab and the Reinforcement learning lab are already installed in the base conda environment. This environment is automatically activated when you log in, so there are no extra steps necessary (i.e., you do not need to run conda activate base).

(conda list will show all dependencies installed in the environment if you are curious.)

Connecting to the server over SSH

At the terminal, the following command will SSH into the server, enable x-forwarding, and also forward a port for Jupyter Notebook access:

Note

You will need to replace $USERNAME and $IP_ADDRESS below with the appropriate values (ask the instructors in the Slack chatroom)

ssh -L 8888:localhost:8888 -X  $USERNAME@$IP_ADDRESS

Once you are connected, run ls - you have an empty directory for your team already created in the home directory. (mx0 is the "instructor" directory)

$ ls
donkey_simulator mx0  mx1  mx2  mx3

Adding a shortcut for easy SSH access:

This step is optional

You can simplify the process of connecting to the NEET server further by adding the following information to your local SSH config file:

Warning

Make sure you are editing the config file on your personal laptop (not the server!) before proceeding:

$ vi ~/.ssh/config

Note

You will need to replace $USERNAME and $IP_ADDRESS below with the appropriate values (ask the instructors in the Slack chatroom)

Host neet
    User $USERNAME
    HostName $IP_ADDRESS
    ForwardX11 yes
    IdentityFile ~/.ssh/id_rsa
    LocalForward 8888 127.0.0.1:8888
    ControlPath ~/.ssh/controlmasters_%r@%h:%p
    ControlMaster auto

This sets up port-forwarding (for Jupyter), SSH multiplexing, and X-forwarding once and for all.

Afterwards you can SSH into the server just by typing:

$ ssh neet

at your local command prompt.

Convenient SCP

With the shortcut, you can also SCP directories and files to the server easily, e.g., to copy to your team's directory mxN (where N is 1,2, or 3):

$ scp -r local_directory_w_training_images neet:~/mxN/

No-password logins

If you would like to avoid being asked for a password each time, you can generate a local identity file:

https://help.github.com/en/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent

Then, use the following command to connect to the server.

# only required once
$ ssh-copy-id neet

which will copy your public key to the server.

Opening Jupyter Notebooks over SSH

Note

If you have Jupyter Notebook running locally, shut it down first (or restart it and change the port to something besides the default port 8888, so that the local port used by Jupyter does not conflict with the SSH-forwarded one)

On your local computer:

jupyter notebook --port 8889

for example.

Once you have SSHed into the computer, you can start jupyter in your team's folder (include --no-browser so that Firefox does not try to load over SSH - which will be annoyingly slow)

$ cd mxN/
$ jupyter notebook --no-browser

Jupyter will provide a URL for you to use in your local computer's browser. Copy and paste it into your browser, e.g.,

Or copy and paste one of these URLs:
   http://localhost:8888/?token=r4nd0mh3xstring

Note

It is highly recommended to run a program like tmux or screen after first logging in, so that your work is not lost in case your SSH connection is interrupted. tmux is already installed on the server.

  • Here is a simple guide that introduces tmux.
  • Also, here is a cheatsheet for quick reference.
$ cd mxN/
$ tmux
$ jupyter notebook --no-browser

Running the RL Simulation Environment

You can collect data and train a policy for the RL lab in person on the server (contact the instructors over Slack regarding server access).

There is a folder already created in the home directory called ~/donkey_simulator/ - you can start the simulator by double-clicking the executable build_sdsandbox.x86_64 in the Ubuntu file explorer or by running it in a new terminal.

Once the simulator is running, you can train a policy in another terminal or over SSH.

Remember

Shut down the simulator and your training scripts when you are done!