Skip to main content

Development Setup

For new developers, here are the steps to get started with the development of SAGE3. This guide adresses development of the Nodejs backend server and the React frontend user interface.

Prerequisites​

  • Node.js V20.xx, the current LTS version
    • We use 'Yarn' v1.xx as our package manager, so have it installed as well
    • npm install --global yarn
  • Docker with docker-compose enabled (most recent version have it already enabled)
    • you can install 'docker desktop' for windows and mac
    • for linux, you can install docker packages from your package manager
  • A code editor of your choice, but VSCode would be recommended if you don't have a preference
  • Terminal knowledge
  • WSL2 (Windows Subsystem for Linux) if you are on Windows

WSL2 and Docker Desktop​

On Windows, it is recommended to use WSL2 to run the development environment. WSL2 is a Linux environment that runs on Windows. It is faster than the previous version of WSL and it is compatible with Docker Desktop.

In a Windows terminal, enable WSL2 by running the following command:

wsl --install   # Answer yes to all the prompts

Reboot the PC after the installation is complete. Upon reboot, answer the prompts to set up a new user and password for the WSL2 environment.

Now you can open WSL terminal by typing wsl in the command prompt or by opening the Ubuntu terminal from the start menu.

Update the Ubuntu packages:

sudo apt-get update
sudo apt-get upgrade

Install Docker Desktop for Windows and enable WSL2 integration. Docker Desktop for Windows can be downloaded from the Docker website: Docker Desktop. You can find the WSL documentation here. Restart the PC after the Docker installation is complete.

Open Docker Desktop and skip the registration questions. Check if there are any updated available. Make sure to close your WSL terminal before you start the update. Also, 'Enable Start Docker Desktop when you sign in'. Select 'WSL 2 base engine' as the default engine (it should be the default option).

To prove that Docker is working, open a WSL terminal and type:

docker ps
docker images
docker run hello-world

Install Visual Studio Code in Windows and install the 'Remote - WSL' extension. This extension allows you to open a WSL terminal in VSCode. You should see the WSL installation on the left sidear inside the 'Remote Explorer' tab.

On the WSL, we need to install a few software packages:

sudo apt-get install -y curl git
curl -fsSL https://deb.nodesource.com/setup_20.x -o nodesource_setup.sh
sudo -E bash nodesource_setup.sh
sudo apt-get install -y nodejs

To verify that Node.js is installed, type:

node -v

To install Yarn, a Nodejs package manager, type:

sudo npm install --global yarn

MacOS​

  • Install Docker Desktop for Mac
    • Docker.com
    • Select the right version for your Mac: Apple Silicon or Intel
  • Install Node.js
  • Install yarn using npm in a Terminal window:
    • npm install --global yarn

Linux - Ubuntu​

sudo apt-get install -y curl git
curl -fsSL https://deb.nodesource.com/setup_20.x -o nodesource_setup.sh
sudo -E bash nodesource_setup.sh
sudo apt-get install -y nodejs
  • Install yarn using npm in a Terminal window:
    • npm install --global yarn
  • Install dependecies for the 'canvas' package in Node.js

Get the code​

The development occurs in the 'dev' branch of the repository. The main branch is for production code only.

On Windowsm, make sure to install the SAGE3 source code inside the WSL environment. Once in a WSL terminal, you can change folder with the 'cd' command that will return you to your home directory.

In a terminal window, clone the repository and checkout the 'dev' branch:

git clone https://github.com/SAGE-3/next
cd next
git checkout dev
git pull origin dev

Backend​

We run the 'backend' services in docker containers: this includes the REDIS server, the Fluentd logging server, and the jupyter server. Make sure you have docker installed and running on your machine.

In a terminal window, navigate to the 'deployment' directory and run the following commands:

docker compose -f docker-compose-backend.yml pull

This will download the docker images. It will take a while to download the images the first time you run this command. You will need around 10GB free to store the docker images.

Webstack​

The webstack is a React application that runs in a custom Nodejs server called. The webstack is a React application that runs in a custom Nodejs server called 'sage3-server'. The server is an Express server that serves the React application and manages the collaboration using the backend services (redis, jupyter, ...).

In another terminal window, navigate to the 'webstack' directory and run the following commands to setup the dependencies for TypeScript and React: It will require around 1.5GB of space to store the node_modules.

yarn install

The first time that you setup a new development environment, you will need to stage some files and generate some keys. Run the following command to stage the files:

yarn stage

To generate the JSON Web Token (JWT) keys, run the following command:

cd webstack/keys
./genJWT_keys.sh  # this will generate the key to sign the JWT tokens
./genJWT_token.sh # this will generate a JWT token for development

The file token.json will be generated in the 'webstack/keys' directory. This token is used to authenticate the user in the development environment. Open the file and copy the token value (long string, around 5 lines) and paste it in the 'next/deployment/.env' file as the value for the TOKEN variable.

Start the development environment​

Start the backend services​

You will need 3 terminal windows open to start the development environment. In the first terminal window, start the backend services:

cd next/deployment
docker compose -f docker-compose-backend.yml up

There is a utility script to start and to stop the backend services: './Backend' in the 'deployment' directory.

Start the web server​

In the second terminal window, start the Node.js server:

cd next/webstack
yarn start

It will start the server on the port 3333. It will take a minute to compile the Typescript code. The server will restart automatically when you change the server code or any of its librariees (SAGE3 apps are built into one library).

Start the React application​

In the third terminal window, start the React application:

cd next/webstack
yarn webapp

This will start the React application on the port 4200. The React development server will proxy automatically all the request to the Node.js server on port 3333. It takes a minute to compile the React and Typescript code. You can access the application at http://localhost:4200. The React server will automatically reload the page when you change the code.

To stop the development environment, you can type 'Ctrl-C' in each terminal window.

Code Organization​

The code is organized in the following directories:

  • 'webstack' contains the React application and Nodejs server into a monorepo
    • the subfolder apps contains the nodejs server code homebase and the React application code webapp
  • Libraries are stored in the 'libs' directory
    • 'libs' contains the SAGE3 libraries that are used by the 'homebase' server and the 'webapp' application
    • Shared types and functions using by both apps are stored in the 'libs/shared' directory
    • The 'libs/applications' directory contains the SAGE3 applications
    • 'libs/backend' contains the backend services that are used by the 'homebase' server
    • 'libs/frontend' contains the frontend services that are used by the 'webapp' application
    • 'libs/sagebase' contains the 'SAGEBase' module that defines distributed collections defined over Redis. The collections have HTTP and WebSocket interfaces to access the data.
    • A few extras modules

Application Development​

To develop a SAGE3 application, you need to use the scalfolding tool that we provide. It creates template files for your applications and adds it to the SAGE3 environment.

To create a new application, run the following command in the 'webstack' directory and answer the questions:

yarn newapp

Your application will reside in the folder webstack/libs/applications/src/lib/apps:

  • The 'index.ts' file contains the schema for the shared state of your application. It is defined using the 'zod' type module.
  • 'styling.css' is file where you can store CSS styles for your application.
  • '[AppName].tsx' is the main file for your application. It exports two React components: the application window and the toolbar (a grouped toolbar can also be defined for group functions).

More information in the 'Application-Development' page.

If you ever need to reset the SAGE3 application list, run the command yarn regen in the 'webstack' directory.