Commit b37d16b3 authored by Alexander Rosenberg's avatar Alexander Rosenberg

more tweaking and restructuring

- added setup section
- moved SSH port forwarding after the example job script
- Replaced pointer to Jupyter Notebooks with a sentence on interactive jobs.
parent 5b8b21ac
...@@ -93,7 +93,7 @@ You can find more job examples where we run TensorFlow and PyTorch containers at ...@@ -93,7 +93,7 @@ You can find more job examples where we run TensorFlow and PyTorch containers at
## Interactive jobs ## Interactive jobs
For setting up interactive Jupyter Notebook sessions, see our [Jupyter Notebooks Guide]({{site.baseurl}}{% link software/jupyter-notebook.md %}). Interactive jobs are those were the user needs to provide input to the application through an interactive pseudo-terminal. For example, this includes the shell and Ncurses-based TUI utilities.
### Starting an Interactive job ### Starting an Interactive job
......
...@@ -6,29 +6,28 @@ sort: 4 ...@@ -6,29 +6,28 @@ sort: 4
Jupyter Notebook is an interactive web application that provides an environment where you can create and share documents with live code, equations, visualizations, and narrative text. It is great for data analysis, scientific computing, and machine learning tasks - you can run Python code in cells, see results right away, and document your work all in one place. Jupyter Notebook is an interactive web application that provides an environment where you can create and share documents with live code, equations, visualizations, and narrative text. It is great for data analysis, scientific computing, and machine learning tasks - you can run Python code in cells, see results right away, and document your work all in one place.
## Using Jupyter Notebook ## Running Jupyter Notebook
To start Jupyter Notebook on the cluster, you would use either the `sbatch` or `srun` commands. `sbatch` is typically used to run scripts or commands that can execute without user intervention. `srun` is used to launch applications in an interactive pseudo-terminal session. As the Jupyter Notebook server process does not require any interactivity itself, `sbatch` is sufficient. `sbatch` has the added benefit that you can set the options at the top of the job script to make its reuse more convenient. To start Jupyter Notebook on the cluster, you would use either the `sbatch` or `srun` commands. `sbatch` is typically used to run scripts or commands that can execute without user intervention. `srun` is used to launch applications in an interactive pseudo-terminal session. As the Jupyter Notebook server process does not require any interactivity itself, `sbatch` is sufficient. `sbatch` has the added benefit that you can set the options at the top of the job script to make its reuse more convenient.
Once Jupyter Notebook is running, you will need to run an SSH command to be able to access it.
```note ### Setup
### Using Your Storage Effectively
Usually, for most of your work you should store your files at `/fs1/projects/{project-name}/`, which lives on the parallel file-system storage. You can also use your home directory (`/home/{username}/`) for quick experiments and convenient access to scripts, but keep in mind that your home directory has limited storage space and performance. The parallel file-system storable is much faster and has way more space for your notebooks and data. Before jumping in and running Jupyter Notebook, you may need to install required packages and stage your data.
```
### SSH Port Forwarding Remember: The compute nodes do not have access the Internet themselves, so you need to transfer any files you need through the login node first.
As the compute nodes where workloads run on the cluster are not directly reachable from the campus network, you'll need to use SSH port forwarding through the login node to access your Jupyter Notebook instances on the cluster. Also, as the login node itself is not currently reachable off campus, either SSH port forwarding through the Linux lab machines or VPN access is needed to access the login node when off campus. Please see our guide on using conda and [how to transfer files]({{site.baseurl}}{% link storage/file_transfer.md %}).
1. The job script (shown in the next section) will generate an SSH command in your output file ```note
2. Run this command from your local machine to establish the connection through the Linux lab machine #### Using Your Storage Effectively
3. Access Jupyter through your local web browser
## Running Jupyter Notebook Usually, for most of your work you should store your files at `/fs1/projects/{project-name}/`, which lives on the parallel file-system storage. You can also use your home directory (`/home/{username}/`) for quick experiments and convenient access to scripts, but keep in mind that your home directory has limited storage space and performance. The parallel file-system storable is much faster and has way more space for your notebooks and data.
```
### Job Script
Here's a script to launch Jupyter. Save it as `jupyter.sbatch`: You'll typically use a job script to launch Jupyter Notebook and most other applications after performing any initial setup. Below is an example that you can just copy and paste to get started. Save it as `jupyter.sbatch`:
```bash ```bash
#!/bin/bash #!/bin/bash
...@@ -71,7 +70,15 @@ To get started: ...@@ -71,7 +70,15 @@ To get started:
4. Find the Jupyter URL with token in your error file (`jupyter_<jobid>.err`) 4. Find the Jupyter URL with token in your error file (`jupyter_<jobid>.err`)
5. Open that URL in your local computer's browser 5. Open that URL in your local computer's browser
Remember: The compute nodes can't access the internet, so transfer any files you need through the login node first. Once Jupyter Notebook is running, you'll need to run one or more SSH commands to setup SSH port forwarding so you can access it.
### SSH Port Forwarding
As the compute nodes where workloads run on the cluster are not directly reachable from the campus network, you'll need to use SSH port forwarding through the login node to access your Jupyter Notebook instances on the cluster. Also, as the login node itself is not currently reachable off campus, either SSH port forwarding through the Linux lab machines or VPN access is needed to access the login node when off campus.
1. The job script (shown in the next section) will generate an SSH command in your output file
2. Run this command from your local machine to establish the connection through the Linux lab machine
3. Access Jupyter through your local web browser
## Working on the Same Node ## Working on the Same Node
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment