Architecture

Overview

SAGE3 utilizes NodeJs as the server-side platform, React as a single page application frontend, and Redis as the real time in-memory database. Redis is known for its performance and scalability, making it an ideal choice for realtime applications like SAGE3. NodeJs is used to serve the React web application and provide a simple and intuitive API to access the database through an interface called SAGEBase. SAGEBase utilizes REDIS as a non-relational database and stores data as JSON-like documents in collections. These design choices provide a flexible and scalable solution for managing data, with fast and efficient access to data stored in the database.

Within the SAGE3 codebase we have 3 core components.

• Homebase

  • NodeJs server that manages the communication between the server and clients.
    • HTTP/Websocket : Serves the Webapp to clients, provides API endpoints using Express.js, and a websocket channel for client data subscriptions.
    • SAGEBase : Redis database interface layer that manages SAGE3's state and provides a simplified API for storing JSON documents within collections in Redis. Handles authentication through OAuth providers using Passport.
    • Asset Manager : Handles file processing for various types and provides a file upload endpoint.

• Webapp

  • React Web Application
    • HTTP/Websocket : Communication channels between the client and the server. Handles sending CRUD operations to the JSON documents stored on the server. Manages subscriptions to collections or documents on the server using web sockets.
    • Stores : Zustand stores that manages the state of SAGE3 throughout the React application. These are singleton stores that all the various React components interface with.
    • Components : React functional components that structure the SAGE3 GUI. Components only communicate with the Zustand stores or various React hooks to interact with SAGE3's state.

• Foresight

  • Python AI Functionality

1. Backend

1.A Homebase

Homebase is the backend server of SAGE3 and serves as a communication bridge between the clients and the database. It also handles asset uploads and serves the React webapp. Homebase is built using Node.js and Express, which are commonly used together as a backend server for web applications. Node.js provides the runtime environment for executing JavaScript code on the server, while Express is a popular and flexible framework for building and organizing the backend of a web application. Homebase uses NodeJs and Express to create various APIs, handle authentication, manage session information, provide websocket connections, and process file uploads.

The Homebase code is located here:

/webstack/apps/homebase
/webstack/libs/backend

1.A.1 HTTP/Websocket

Homebase uses Express to help build a REST API endpoints to interact with the state within SAGE3. When a client makes a HTTP request to one of the API endpoints, such as a GET, POST, PUT, DELETE, the request is passed to SAGEBase for authentication and to perform the requested action. CRUD operations and Collection/Document subscriptions can be requested over Websockets.

Homebase also enables clients to upload files to the server using standard HTTP methods such as POST. The server will receive the file and then pass the file to the Asset Manager to process, validate, store, and document the newly uploaded file.

The HTTP/Websocket code is located here:

/webstack/apps/homebase/src/api
/webstack/libs/backend/src/libs/generics

1.A.2 SAGEBase

SAGEBase is a database interface for Redis developed for SAGE3. It leverages Redis to create a fast a NoSQL document-oriented database. SAGEbase allows developers to create Collections and store Documents within them. Documents within SAGEBase are JSON objects. Documents and Collections support CRUD style operations and also support subscriptions for Document EVENTS: 'CREATE', 'UPDATE', 'DELETE'.

A Car Collection example:

// Create a SAGEBase configuration
const sbConfig: SAGEBaseConfig = {
    projectName: 'Automobiles', // Name of the project. Can use the same Redis database for multiple SAGEBase projects.
    redisUrl: 'redis://localhost:6379', // URL of the Redis server.
  };

// Initialize SAGEBase
await SAGEBase.init(sbConfig);

// Create a Car type. This defines the Car Document schema.
type Car = {
  make: string;
  model: string;
  year: number;
};

// Create a Car Collection using SAGEBase.
// The first parameter, 'cars', is the name of the collection.
// The second parameter are the fields to index so they are queryable. So this collection only indexes 'model' and 'year'
const carCollection = await SAGEBase.Database.collection<Car>('cars', { model: '', year: 0 });

// Subscribe to Document EVENTS ('CREATE', 'UPDATE', 'DELETE') on the Car Collection.
carCollection.subscribe((message) => {
  // This would be called after the code below runs.
  // 1. First Event message would be CREATE and contain the new Document state.
  // 2. Second Event message would be UPDATE and contain the updated Document state.
  // 3. Third Event message would be DELETE
})

// CREATE a new Car Document in the Collection. This returns a REF to the Document in the database.
const wrxRef = await carCollection.addDoc({ make: 'Subaru', model: 'WRX', year: 2019 }, '');

// Read the Document from the database.
const wrxDoc = await wrxRef.read();

// UPDATE the Document
await wrxRef.update({year: 2020});

// DELETE the Document
await wrxRef.delete();

The SAGEBase library is located here:

/webstack/libs/sagebase

SAGEBase is hosted on NPM for use in other projects. @sage3/sagebase

1.A.3 Asset Manager

Asset Manager is responsible for managing and organizing the uploaded assets such as images, videos, audio files, csv data, pdfs...etc. The goal of the asset manager is to ensure that the assets are properly organized, optimized, and delivered to the client efficiently. It uses SAGEBase to organize all the files uploaded and their hosted URLs.

The Asset Manager code is located here:

/webstack/apps/homebase/src/api/routes/custom/asset.ts
/webstack/apps/homebase/src/processors/*
/webstack/apps/homebase/src/connectors/*

1.B Foresight

1.C Redis

Redis Stack on Docker Hub

SAGEBase uses the Redis Stack Docker image. Redis Stack was created to allow developers to build real-time applications with a backend data platform that can reliably process requests in milliseconds or less. The RedisJson and RedisSearch modules are enabled for our configuration.

SAGEBase leverages the Redis Stack as Collection and Document style database. Documents are organized within Redis using the standard Redis prefix naming conventions. From the SAGEBase example above, the WRX Car would be stored with this Redis key:

//PROJECT_NAME:MODULE:COLLECTION_NAME:DOC_UID
AutoTown:DB:cars:ec3dcbe8-8e0f-4fa8-89e1-b8c9663eae64

Module refers to the SAGEBase module, in this case the Database DB module. Each Document is created with a unique UID automatically using the UUID library.

1.D JupterLab

2. Frontend

SAGE3's frontend is a React web application written in Typescript. It is hosted on the SAGE3 Homebase server and sent to connected clients over HTTP. Clients connect to Homebase using the SAGE3 Electron desktop application or a Chrome web browser.

2.A Electron

Electron allows developers to create desktop applications using HTML, CSS, and JavaScript, making it possible to build cross-platform applications that can run on multiple operating systems. SAGE3's Electron application supports MacOS (Intel or Apple Silicon), Ubuntu, and Windows.

2.B Webapp

SAGE3's Frontend GUI is a React Webapp written in Typescript.

2.B.1 HTTP/WS

Manages sending user CRUD requests to the server using HTTP or Websockets.

HTTP

/webstack/libs/frontend/src/lib/api/http/api-https.ts

Websocket

/webstack/libs/frontend/src/lib/api/ws/api-socket.ts

2.B.2 Stores/Hooks

2.B.2.A Stores

/webstack/libs/frontend/src/lib/stores/*

The stores are the Clients local state of the SAGE3 server state. The various GUI Components only interface with the Stores when creating, updating, deleting, or subscribing to documents through the stores. The stores interface with the HTTP/WS to perform the actual communication.

• App
  • Description
• Asset
  • Description
• Board
  • Description
• Panel
  • Description
• Presence
  • Description
• Room
  • Description
• UI
  • Description
• User
  • Description

2.B.2.B Stores

/webstack/libs/frontend/src/lib/hooks/*

React hooks allows users to reuse stateful logic in their functional components and provide a way to manage state an side effects in your components without writing a class component. Essentially they are functions that allow functional components to "hook" into React state and lifecycle.

• useAuth
  • Description
• useCursorBoardPosition
  • Description
• useData
  • Description
• useHexColor
  • Description
• useHotkeys
  • Description
• useKeyPress
  • Description
• useMeasure
  • Description
• usePeer
  • Description
• useRouteNav
  • Description
• useUser
  • Description
• useWindowResize
  • Description

2.B.3 Components

SAGE3's uses React Functional components for rendering the GUI to the user. These components listen to the Stores to keep their state up to date and to perform any CRUD operations.

2.B.3.A Pages

• Landing (Electron)
  • Description
• Server Login
  • Description
• Account Creation
  • Description
• Home
  • Description
• Board
  • Description

2.B.3.B Panels `/webstack/apps/webapp/src/app/

• Annotations
• Applications
• Assets
• Navigation
• Users

2.B.3.C Reusable UI

• Board, Room, User Cards
• Context Menu
• Color Picker
• Main Button
• Board, Room, User Lists
• Modals

2.B.3.D Apps

• AppWindow
• Apps
• AppToolbar

3. Cloud Services

3.A AI

3.B OAuth

3.C Twilio