Managing software dependencies and configurations can be challenging in an HPC environment. Users often need different versions of the same software or libraries, leading to conflicts and complex setups. Virtual environments provide a solution by allowing users to isolate their project and its dependencies.This simplifies the setup process, ensures that users have the correct software environment for their applications, and reduces conflicts and errors caused by incompatible software versions. This guide provides different methods for installing virtual environments in multiple languages including
Python, R, Julia, Rust, C, C++, and other languages. This will allow you to create projects in isolated environments that will not require root or sudo access.
Managing software dependencies and configurations can be challenging in an HPC environment. Users often need different versions of the same software or libraries, leading to conflicts and complex setups. Virtual environments provide a solution by allowing users to isolate their project and its dependencies. This simplifies the setup process, ensures that users have the correct software environment for their applications, and reduces conflicts and errors caused by incompatible software versions. This guide provides different methods for installing or simulating virtual environments in multiple languages including Python, R, Julia, Rust, C, C++, and other languages. This will allow you to create projects in isolated environments that will not require root or sudo access.
## General Tips (New Section)
## General Tips
- Always use version control (e.g., Git) alongside virtual environments
- Document your project setup and dependencies
- Regularly update your virtual environment and dependencies
- Be consistent with naming conventions for your projects and environments
## What is a virtual environment?
A virtual environment is an environment or directory that isolates packages, libraries, and dependencies for a specific project. This prevents any dependencies from being permanently installed on the actual system while still allowing it to be managed within the project's directory.
## Python
### How to create and use a virtual environment in Python
This section of the guide explains how to use, create, and manage virtual environments for Python as well as install and manage packages within that environment.
#### Setup environment
This subsection walks you through creating a new Python virtual environment. A virtual environment in Python is an isolated working copy of Python which allows you to work on a specific project without affecting other projects.
1. Ensure Python is installed:
```
...
...
@@ -22,12 +27,12 @@ Python, R, Julia, Rust, C, C++, and other languages. This will allow you to crea
```
If not installed, use your distribution's package manager to install Python 3.
2. Create directory for project:
2. Create a new directory for your project:
```
mkdir Project
```
3. Navigate to directory:
3. Navigate to the new directory:
```
cd Project
```
...
...
@@ -37,7 +42,7 @@ Python, R, Julia, Rust, C, C++, and other languages. This will allow you to crea
python3 -m venv research1
```
5. Ensure the environment is created:
5. Check if the environment was created:
```
ls
```
...
...
@@ -46,7 +51,7 @@ Python, R, Julia, Rust, C, C++, and other languages. This will allow you to crea
research1
```
6. Activate the environment:
6. Activate the environment for use:
```
source research1/bin/activate
```
...
...
@@ -55,14 +60,15 @@ Python, R, Julia, Rust, C, C++, and other languages. This will allow you to crea
(research1) user@super-computer
```
7. Deactivate the environment (when you're done):
```
deactivate
```
#### Deactivate environment
When you're done working in the virtual environment, you can deactivate it:
#### Installing packages
```
deactivate
```
(While virtual environment is activated)
#### Installing packages
Once your virtual environment is set up and activated, you can install packages specific to your project. This keeps your project dependencies separate from other projects and the global Python installation.
1. Upgrade pip to the latest version:
```
...
...
@@ -86,8 +92,7 @@ Python, R, Julia, Rust, C, C++, and other languages. This will allow you to crea
```
#### Saving dependencies
(While virtual environment is activated)
Saving your project's dependencies allows you to easily recreate your environment on another machine or share it with others.
1. Save packages to a file:
```
...
...
@@ -105,6 +110,7 @@ Python, R, Julia, Rust, C, C++, and other languages. This will allow you to crea
```
#### Using dependencies
This section explains how to install dependencies from a requirements file, which is useful when setting up a project on a new machine or collaborating with others.
1. Install dependencies from requirements file:
```
...
...
@@ -121,7 +127,18 @@ Python, R, Julia, Rust, C, C++, and other languages. This will allow you to crea
pip list
```
#### Running Python scripts
To run a Python script within the virtual environment:
1. Ensure the virtual environment is activated
2. Run the script:
```
python your_script.py
```
#### Delete virtual environment
When you're done with a project or want to start fresh, you can delete the virtual environment. This removes all installed packages and the environment itself.
1. Deactivate the environment if it's active:
```
...
...
@@ -130,20 +147,17 @@ Python, R, Julia, Rust, C, C++, and other languages. This will allow you to crea
2. Delete directory with virtual environment:
```
rm -r research1
rm -rf research1
```
**Warning:**
* Do not store source code inside virtual environment directories
* All source code should be stored in main directory (project in this example)
* Always use `.gitignore` to exclude the virtual environment directory from version control
## R
### How to create and use a virtual environment in R
This section explains how to use, create, and manage virtual environments for R as well as install and manage packages within that environment. R uses the 'renv' package to create project-specific libraries, which function similarly to virtual environments in other languages.
#### Setup environment
Setting up an R environment involves creating a new R project and initializing it with renv, which will manage project-specific package libraries.
1. Ensure R is installed:
```
...
...
@@ -157,7 +171,7 @@ Python, R, Julia, Rust, C, C++, and other languages. This will allow you to crea
3. Navigate to directory:
```
cd /Project
cd Project
```
4. Create an R project:
...
...
@@ -178,24 +192,25 @@ Python, R, Julia, Rust, C, C++, and other languages. This will allow you to crea
This creates a project-specific library and .Rprofile
#### Activate an environment
Activating an environment in R loads the project-specific library, ensuring that your code uses the correct versions of packages for your project.
```R
> renv::activate()
```
#### Deactivate an environment
Deactivating an environment returns you to the global R library, which is useful when switching between projects.
```R
> renv::deactivate()
```
#### Installing Packages
(While virtual environment is activated)
Installing packages in an renv environment adds them to your project-specific library, keeping your project dependencies isolated.
1. Install a package:
```R
> install.packages("package")
> install.packages("package_name")
```
2. Record the new package in the lockfile:
...
...
@@ -204,6 +219,7 @@ Python, R, Julia, Rust, C, C++, and other languages. This will allow you to crea
```
#### Using dependencies
These commands allow you to save the state of your environment, restore it on another machine, or update all packages in your project.
1. Save environment state to lock file:
```R
...
...
@@ -220,13 +236,35 @@ Python, R, Julia, Rust, C, C++, and other languages. This will allow you to crea
> renv::update()
```
#### Running R scripts
To run an R script within the project environment:
1. Ensure you're in the project directory
2. Run the script:
```
Rscript your_script.R
```
#### Managing different R versions
While renv doesn't manage R versions directly, you can use tools like `rig` or `RSwitch` to manage multiple R versions on your system.
#### Sharing environments
To share your project environment:
1. Include both renv.lock and .Rprofile in your version control system
2. Others can recreate your exact environment using:
```R
> renv::restore()
```
#### Delete virtual environment
Deleting an R virtual environment involves removing the renv directory and associated files. This completely removes the isolated environment and its packages.
1. Exit R if you're in an R session
2. Delete directory with virtual environment:
```
rm -r renv
rm -rf renv
```
3. Remove .Rprofile and renv.lock files:
...
...
@@ -238,7 +276,10 @@ Python, R, Julia, Rust, C, C++, and other languages. This will allow you to crea
### How to create and use a virtual environment in Julia
This section outlines how to set up, use, and manage project-specific environments in Julia. Julia's built-in package manager provides functionality similar to virtual environments, allowing you to create isolated project environments.
#### Setup environment
Setting up a Julia environment involves creating a new project directory and activating it as a Julia environment. This isolates the project's dependencies from other projects.
1. Check Julia version:
```
...
...
@@ -252,7 +293,7 @@ Python, R, Julia, Rust, C, C++, and other languages. This will allow you to crea
3. Navigate to directory:
```
cd /Project
cd Project
```
4. Start Julia and enter package manager mode:
...
...
@@ -272,15 +313,23 @@ Python, R, Julia, Rust, C, C++, and other languages. This will allow you to crea
(@v1.10) pkg> instantiate
```
#### Activate environment
To activate an existing environment:
```
julia> ]
(@v1.10) pkg> activate /path/to/your/project
```
#### Deactivate environment
Deactivating an environment in Julia returns you to the default (global) environment. This is useful when switching between projects.
```
(@v1.10) pkg> activate
```
#### Install packages
(While virtual environment is activated)
Installing packages in a Julia environment adds them to your project-specific manifest, keeping your project dependencies isolated.
1. Install package:
```
...
...
@@ -295,6 +344,7 @@ Python, R, Julia, Rust, C, C++, and other languages. This will allow you to crea
```
#### Using dependencies
These commands allow you to install dependencies from your project files, ensuring consistency across different machines or for different users of your project.
1. Enter package manager:
```
...
...
@@ -311,15 +361,56 @@ Python, R, Julia, Rust, C, C++, and other languages. This will allow you to crea
(@v1.10) pkg> update
```
#### Running Julia scripts
To run a Julia script from within the project environment:
1. Activate the environment in Julia REPL
2. Run the script:
```julia
julia> include("your_script.jl")
```
Or from the command line:
```
julia --project=. your_script.jl
```
#### Managing multiple environments
Julia allows you to have multiple environments for different purposes:
1. Create a test environment:
```
(@v1.10) pkg> activate --temp
```
2. Switch between environments:
```
(@v1.10) pkg> activate /path/to/environment
```
#### Delete virtual environment
In Julia, deleting a virtual environment simply involves removing the project-specific files. This removes the isolated environment and its package information.
Simply delete the Project.toml and Manifest.toml files from your project directory.
#### Sharing environments
To share your project environment:
1. Include both Project.toml and Manifest.toml in your version control system
2. Others can recreate your exact environment using:
```
(@v1.10) pkg> instantiate
```
## Node JS
### How to create and use a virtual environment in Node JS
This section explains how to set up and manage project-specific environments in Node.js. While Node.js doesn't have traditional virtual environments, it uses npm (Node Package Manager) to manage dependencies on a per-project basis, which serves a similar purpose.
#### Setup environment
Setting up a Node.js project environment involves creating a new directory and initializing it with npm. This creates a package.json file to manage your project's dependencies.
1. Check node version:
```
...
...
@@ -333,7 +424,7 @@ Simply delete the Project.toml and Manifest.toml files from your project directo
3. Navigate to directory:
```
cd /Project
cd Project
```
4. Initialize the project:
...
...
@@ -347,6 +438,7 @@ Simply delete the Project.toml and Manifest.toml files from your project directo
```
#### Install packages
Installing packages in a Node.js project adds them to your package.json file and the node_modules directory, keeping your project dependencies isolated and easily reproducible.
1. Install a package and save it to package.json:
```
...
...
@@ -359,6 +451,7 @@ Simply delete the Project.toml and Manifest.toml files from your project directo
```
#### Using dependencies
These commands allow you to use a specific Node.js version for your project and install or update all dependencies listed in your package.json file.
1. Use .nvmrc for node version:
```
...
...
@@ -376,6 +469,7 @@ Simply delete the Project.toml and Manifest.toml files from your project directo
```
#### Managing different Node.js versions
Node Version Manager (nvm) allows you to install and use different versions of Node.js for different projects, providing a way to isolate not just packages but the JavaScript runtime itself.
1. Install a specific Node.js version:
```
...
...
@@ -387,9 +481,35 @@ Simply delete the Project.toml and Manifest.toml files from your project directo
nvm use 14.17.0
```
#### Delete virtual environment
#### Running your Node.js application
To run your Node.js application:
In Node.js, the "virtual environment" is essentially your `node_modules` directory and `package.json` file. To remove:
```
node your-app.js
```
Or, if you've defined a start script in your package.json:
```
npm start
```
#### Creating and using an .npmrc file
An .npmrc file can be used to configure npm behavior for your project:
1. Create an .npmrc file in your project root:
```
touch .npmrc
```
2. Add configuration to the file, for example:
```
save-exact=true
```
This will save exact versions of packages instead of using npm's default semantic versioning range operator.
#### Delete virtual environment
In Node.js, removing the "virtual environment" involves deleting the node_modules directory and optionally the package.json file. This removes all installed packages and dependency information.
1. Delete the node_modules directory:
```
...
...
@@ -401,7 +521,8 @@ In Node.js, the "virtual environment" is essentially your `node_modules` directo
rm package.json package-lock.json
```
Remember to keep your `package.json` if you want to reinstall your dependencies later.
You can keep the `package.json` if you want to reinstall your dependencies later.
# C and C++ Development Guide Using Home Directory
...
...
@@ -409,7 +530,10 @@ Remember to keep your `package.json` if you want to reinstall your dependencies
### How to use the home directory for development with C
This section provides guidance on setting up a local development environment for C programming without requiring root access. It explains how to configure your home directory to install and use libraries, as well as how to structure and compile C projects.
#### Setup environment
Setting up a C development environment in your home directory involves creating directories for binaries, libraries, and include files, and setting up environment variables to use these directories.
1. Verify installation:
```
...
...
@@ -436,6 +560,7 @@ Remember to keep your `package.json` if you want to reinstall your dependencies
```
#### Install packages
Installing packages (libraries) for C development in your home directory involves downloading source code, compiling it, and installing it in your local directories.
Install a library with example libcurl:
```
...
...
@@ -448,6 +573,7 @@ make install
```
#### Create project
Creating a C project involves setting up a directory structure, creating source files, and setting up a Makefile for easy compilation.
1. Set up project structure:
```
...
...
@@ -503,11 +629,49 @@ make install
make clean
```
#### Using the project
To run your compiled C program:
```
./main
```
#### Updating libraries
To update a library, you typically need to download the new version, compile, and install it similarly to the initial installation process. For example, to update libcurl:
1. Download and extract the new version
2. Navigate to the extracted directory
3. Run the installation commands again:
```
./configure --prefix=$HOME/local
make
make install
```
#### Deleting the environment
In C, there's no virtual environment to delete. However, you can remove your local installations and project files:
1. Remove local installations:
```
rm -rf ~/local
```
2. Remove project directory:
```
rm -rf my_c_project
```
Remember to also remove or comment out the environment variable settings in your `~/.bashrc` or `~/.bash_profile` if you no longer need them.
## C++
### How to use the home directory for development with C++
Similar to the C section, this part of the guide shows how to set up a local C++ development environment using your home directory. It covers installing libraries, creating projects, and compiling C++ code without needing system-wide installations.
#### Setup environment
Setting up a C++ development environment in your home directory is similar to C, involving creating directories for binaries, libraries, and include files, and setting up environment variables.
1. Verify installation:
```
...
...
@@ -534,6 +698,7 @@ make install
```
#### Install packages
Installing packages (libraries) for C++ development in your home directory involves downloading source code, compiling it, and installing it in your local directories.
Install a library with example boost:
```
...
...
@@ -545,6 +710,7 @@ cd boost_1_76_0
```
#### Create project
Creating a C++ project involves setting up a directory structure, creating source files, and setting up a Makefile for easy compilation.
1. Set up project structure:
```
...
...
@@ -601,13 +767,49 @@ cd boost_1_76_0
5. Clean using:
```
make clean
```
#### Using the project
To run your compiled C++ program:
```
./main
```
#### Updating libraries
To update a library, you typically need to download the new version, compile, and install it similarly to the initial installation process. For example, to update Boost:
1. Download and extract the new version
2. Navigate to the extracted directory
3. Run the installation commands again:
```
./bootstrap.sh --prefix=$HOME/local
./b2 install
```
#### Deleting the environment
In C++, there's no virtual environment to delete. However, you can remove your local installations and project files:
1. Remove local installations:
```
rm -rf ~/local
```
2. Remove project directory:
```
rm -rf my_cpp_project
```
Remember to also remove or comment out the environment variable settings in your `~/.bashrc` or `~/.bash_profile` if you no longer need them.
# Rust
### How to simulate a virtual environment with rust
This guide outlines how to create a project-specific environment in Rust, similar to virtual environments in other languages. (Most similar to node)
This guide outlines how to create a project-specific environment in Rust, similar to virtual environments in other languages. Rust uses Cargo, its package manager and build system, to manage dependencies and create isolated project environments.
## Setup Environment
#### Setup Environment
Setting up a Rust project environment involves creating a new directory and initializing it with Cargo. This creates a new Rust project with its own Cargo.toml file for managing dependencies.
1. Ensure Rust and Cargo are installed:
```bash
...
...
@@ -631,15 +833,35 @@ This guide outlines how to create a project-specific environment in Rust, simila
cd my_project
```
## Install Packages
#### Install Packages
Installing packages in a Rust project adds them to your Cargo.toml file and downloads them to a local cache, keeping your project dependencies isolated and easily reproducible.
Install a package:
```bash
cargo add package
cargo add package_name
```
## Using Dependencies
#### Using Dependencies
Building your Rust project with Cargo automatically downloads and compiles all necessary dependencies specified in your Cargo.toml file.
Install packages using Cargo.toml:
```bash
cargo build
```
#### Delete virtual environment
In Rust, there isn't a traditional "virtual environment" to delete. The project isolation is achieved through the project-specific Cargo.toml file and the target directory. To "delete" this environment:
1. Remove the entire project directory:
```bash
cd ..
rm -rf my_project
```
This will remove the project files, Cargo.toml, and all compiled artifacts.
Note: If you want to keep the source code but remove all compiled artifacts and dependencies, you can instead run:
```bash
cargo clean
```
This will remove the target directory, which contains all compiled files and downloaded dependencies.
