Installing the development version of NACSOS

This is a guide on how to install the entire platform locally for development. Please read the entire guide before running any of the suggested commands, since there are different options you can take and define the way to do things.

Prerequisites

• Up-to-date GNU/Linux operating system
  • Ubuntu 18.x is known to cause issues due to outdated packages
  • We recommend Ubuntu 22.04, Arch, or Alpine
• Python 3.12 (with matching pip and virtualenv)
• NodeJS (>= 18.10.0) / NPM (>= 8.19.2) Official installation guide
• PyCharm Pro IDE recommended (more here)
• All necessary repositories cloned into the same directory side-by-side
  • NACSOS Core
  • NACSOS Data Lib
  • NACSOS Pipelines
  • NACSOS Web Interface
  • NACSOS Documentation

Setting up Python environments

Although you could use a single environment for all projects, there are advantages to encapsulate all three python projects. Below are notes on how to set up virtual environments using virtualenv. It is recommended to use your favourite IDE and have it manage your environments.

nacsos_data in editable mode

The requirements.txt of nacsos-pipes and nacsos-core is set up to install a fixed version of nacsos_data directly from git. If you are actively working on the data lib and nacsos-core, it makes sense to install nacsos_data in the editable mode. If you want that, you should remove the respective line in the requirements.txt when initialising the project and instead run

pip install -e ../nacsos-data/

If you installed nacsos_data in the editable mode, make sure to add the line back in and update the requirements once your changes are merged!

NACSOS Core

Follow these steps to set up the backend server

cd nacsos-core/
virtualenv venv
source venv/bin/activate
pip install -r requirements.txt
# additional requirements for development/testing
pip install -r requirements_dev.txt
deactivate

NACSOS Pipelines

Follow these steps to set up the pipeline backend

cd nacsos-pipes/
virtualenv venv
source venv/bin/activate
pip install -r requirements.txt
pip install -r requirements_dev.txt
deactivate

NACSOS Data

Follow these steps to set up the data library

cd nacsos-data/
virtualenv venv
source venv/bin/activate
pip install -r requirements.txt
pip install -r requirements_dev.txt
deactivate

Setting up nodejs environments

Go to the NACSOS web directory and install the requirements:

cd nacsos-web/
npm install

Initialise the database

You can use a natively running postgres database. Alternatively you can use docker to spin up a postgres instance (make sure to change ports in the docker file to not conflict with other services running on you machine).

Postgres with DockerNative Postgres

Running Postgres

For a seemingly easier way to keep different versions of Postgres or different databases separated, you may wish to run Postgres via docker. Be advised, that the provided config may lead to conflicts with ports of a locally running Postgres server or other service. In that case, adapt the ports in the docker-compose.yml file before doing anything else! Also make sure you know about the implications of the default mounting points/volumes.

Make sure you have the latest version of docker installed and the host service running.

ArchUbuntu

Consult ArchWiki for additional support.

sudo pacman -S docker docker-compose
sudo systemctl start docker

Consult Docker Wiki for additional support.

sudo apt install docker docker-compose
sudo systemctl start docker

Once you ensured the docker service is running, you may proceed by running the container

# navigate to nacsos-core repository
cd nacsos-core/
docker-compose up -d

In the future, you probably only need to run the docker service and the image will spin up automatically. You can check whether the image is running with

sudo docker containers ls

The docker image comes with PGAdmin, available at: http://localhost:5050/browser/#
The first time, you have to configure the server (host: nacsos_postgres, port: 5432)

Some more info:

• https://towardsdatascience.com/how-to-run-postgresql-and-pgadmin-using-docker-3a6a8ae918b5
• https://github.com/lifeparticle/PostgreSql-Snippets/blob/main/pgAdmin/docker-compose.yml

Running Postgres

ArchUbuntu

sudo pacman -S postgresql
sudo systemctl start postgresql

sudo apt install postgresql
sudo systemctl start postgresql

TODO: Creating database and user

Import initialisation data

For development purposes, it is helpful to have a small pre-filled database. The most recent initialisation dump is here. In order to import the dump, you need an SQL client, for example:

sudo apt-get install postgresql-client
psql -h 0.0.0.0 -U root -W -d nacsos_core -f ~/Downloads/dump.sql

Run migrations

There may have been changes to the schema since the last version of the dump. To make sure everything is up-to-date, run

alembic upgrade head

Run the services

Adapt configuration (optional)

If you run a custom configuration, copy the default config (and never commit your custom config)

• NACOS Core
  • Default config location: nacsos-core/config/default.env
  • See nacsos-core/server/util/config.py for available options
  • Use export NACSOS_CONFIG=config/default.env to set the config file to be used
• NACSOS Pipelines
  • Default config location: nacsos-pipelines/config/default.env
  • See nacsos-pipelines/server/util/config.py for available options
  • Use export NACSOS_CONFIG=config/default.env to set the config file to be used (NOTE: this should be a different name, needs changing!)
• NACSOS Web (only needs config for build and development)
  • If you are running NACSOS Core or Pipelines at different locations, DO NOT update the config in package.json and commit!
  • See the node documentation on how to inject settings as CLI parameters
• NACSOS Data
  • This is never actually run stand-alone, so no config needed

Build / run frontend

Host by nacsos-coreHost by node

This builds the frontend and puts everything needed in nacsos-web/dist/. The default config for nacsos-core will serve static files from that directory.

cd nacsos-web/
npm run build

Alternatively, you can run the frontend as a standalone and auto-refresh on code changes.

cd nacsos-web/
npm run serve

The frontend is likely at http://localhost:8080/ (but npm should tell you where it is served).

Run NACSOS Core (aka "the backend")

Open a new terminal and run

cd nacsos-core/

# activate environment (one of...)
source venv/bin/activate
conda activate nacsos2env

# set this in case you want to use a different config (optional)
export NACSOS_CONFIG=config/default.env

# for development, using the optional --reload is helpful
hypercorn --config=config/hypercorn.toml --reload main:app

Swagger documentation of the backend API is available here:
http://127.0.0.1:8081/docs

If you built the frontend, it will be served at the root route:
http://127.0.0.1:8081/

Run NACSOS Pipelines

Open a new terminal and run

cd nacsos-pipelines/

# activate environment (one of...)
source venv/bin/activate
conda activate nacsos2env

# set this in case you want to use a different config (optional)
export NACSOS_CONFIG=config/default.env

# for development, using the optional --reload is helpful
hypercorn --config=config/hypercorn.toml --reload nacsos_pipes.app:app

Swagger documentation of the backend API is available here:
http://127.0.0.1:8000/docs