Server deployment

This page is supposed to document how the system can be deployed on an Ubuntu server.

Prerequisites:

• GitLab Runner service is running and at least two executors are registered (one as "docker" (no tag), and one as "script" (with "bare-metal" tag)).
• System user nacsos (default shell /sbin/nologin) with home directory (use sudo su -l nacsos -s /bin/bash to open a shell as that user).
• PostgreSQL 15 installed and running, database initialised
• milvus is installed and running (see below, NO DOCKER!)

Overview

Deployment is handled via GitLab CI/CD. Necessary secrets are configured in group settings or respective project settings. We follow these conventions:

• The branch production is that running on the server, so we can cherry-pick from the master branch if needed
• nacsos-data version has to be updated in nacsos-core
• nacsos-core deployment runs database migrations
• nacsos-data does not need deployment, it's a dependency of nacsos-core
• nacsos-web is deployed independent of the backends
• pipeline workers need restart when task-specific code was changed; this does not happen automatically so we don't accidentally kill long-running workers

Common practice for deployment:

1. Open a new merge request from master to production.
2. Call the MR Version <semantic version>
3. Have the MR approved by someone and merge
4. Build/test jobs finish, trigger deploy-to-production stage manually

PostgreSQL

Parallel to PostgreSQL 12, the VM is running a v15 server on a different port (see config secrets).

Service management via systemd

The service is handled via systemd, common commands below.

# See status of the service
$ systemctl status postgresql@15-main.service

# Start or restart the service
$ sudo systemctl start postgresql@15-main.service
$ sudo systemctl restart postgresql@15-main.service

# Permanently enable the service, so it will automatically restart on server boot)
$ sudo systemctl enable postgresql@15-main.service

Database migration via alembic

When running alembic for migrating, you have to set the following environment variables. Note, that changing the connection URL in the alembic.ini might not work for complex passwords.

# Set connection string to override that stored in `alembic.ini`
NACSOS_DB__USER=nacsos
NACSOS_DB__PASSWORD=??
NACSOS_DB__HOST=localhost
NACSOS_DB__PORT=5432
NACSOS_DB__DATABASE=nacsos_core

Automatic migrations

Note, that migrations are automatically executed during deployment of nacsos-core! Check the respective section below for details.

Be careful with migrations!

Note, that migrations will be handled automatically during deployment of nacsos-core. The required environment variables are set in the GitLab CI/CD Variables settings. It might be, that manual intervention is necessary in case of failure. Never change the database schema outside of alembic migrations!

In order to manually trigger upgrade migrations, install nacsos_data and use

# Get help
$ nacsos_migrate --help
# Upgrade to head
$ nacsos_migrate upgrade --revision head --root-path=<...>/site-packages/nacsos_data --ini-file=<...>/alembic.ini

See below for more information. For more commands (e.g. downgrading), consult the Alembic Documentation.

Database permissions

At the moment, we have two users: nacsos ("admin") and nacsos_user. You should always use the nacsos_user account unless you actually need special permissions.

Following permissions are set

-- GRANTs for nacsos superuser
GRANT select, insert, update, delete, references, trigger, truncate
ON "user", auth_tokens, project, project_permissions, 
    annotation_scheme, assignment_scope, assignment,  annotation, bot_annotation, bot_annotation_metadata, 
    item,generic_item, academic_item, twitter_item, import,  m2m_import_item, tasks, highlighters
TO nacsos;

-- GRANTs for nacsos system user (and used in scripts)
GRANT select, insert, update, delete
ON  "user", auth_tokens, project, project_permissions, 
    annotation_scheme, assignment_scope, assignment,  annotation, bot_annotation, bot_annotation_metadata, 
    item,generic_item, academic_item, twitter_item, import,  m2m_import_item, tasks, highlighters
TO nacsos_user;

-- GRANTs for nacsos read only user
GRANT select
ON "user", auth_tokens, project, project_permissions, 
    annotation_scheme, assignment_scope, assignment,  annotation, bot_annotation, bot_annotation_metadata, 
    item,generic_item, academic_item, twitter_item, import,  m2m_import_item, tasks, highlighters
TO nacsos_read;

Service deployment

The nacsos-core and nacsos-dramatiq services, are managed with systemd. The compiled nacsos-web frontend is served directly with nginx.

You can control the services via

# Reload service configuration files (required after editing systemd configs)
$ sudo systemctl daemon-reload

# Common commands for handling a service
$ sudo systemctl enable nacsos-core.service
$ sudo systemctl start nacsos-core.service
$ sudo systemctl stop nacsos-core.service
$ sudo systemctl status nacsos-core.service
$ sudo journalctl -u nacsos-core.service

NACSOS-core

NACSOS-core exposes the nacsos_data library via an API. It also provides authentication and permission management to the different endpoints and mailing system.

Local configuration

Create config/hypercorn-server.toml and config/logging-server.toml. make sure to set at least

debug = false
workers = 1
accesslog = '../logs/hypercorn-core.access'
errorlog = '../logs/hypercorn-core.error'

Manually running the server from the virtual environment (assuming your working directory is the project root):

/var/www/nacsos2/nacsos-core/venv/bin/python -m hypercorn main:app --config=config/hypercorn-server.toml

systemd configuration

/etc/systemd/system/nacsos-core.service /etc/systemd/system/nacsos-dramatiq.service

[Unit]
Description=NACSOS core server
After=network.target

[Service]
User=nacsos
Type=simple
Environment="NACSOS_CONFIG=/var/www/nacsos2/nacsos-core/config/server.env"
WorkingDirectory=/var/www/nacsos2/nacsos-core
LimitNOFILE=4096
ExecStart=/var/www/nacsos2/nacsos-core/venv/bin/python -m hypercorn main:app --config=config/hypercorn-server.toml
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

[Unit]
Description=dramatiq workers
After=network.target

[Service]
Type=simple
User=nacsos
Group=nacsos
Environment="NACSOS_CONFIG=/var/www/nacsos2/nacsos-core/config/server.env"
WorkingDirectory=/var/www/nacsos2/nacsos-core
#ExecStart=/var/www/nacsos2/nacsos-core/venv/bin/dramatiq dramatiq server.pipelines.tasks -t 2 -p 3 -Q default nacsos-pipes --watch /var/www/nacsos2/nacsos-core/server --pid-file /var/www/nacsos2/dramatiq.pid  
ExecStart=/var/www/nacsos2/nacsos-core/venv/bin/dramatiq dramatiq server.pipelines.tasks -t 2 -p 3 -Q default nacsos-pipes --pid-file /var/www/nacsos2/dramatiq.pid
Restart=always
RestartSec=30s
PIDFile=/var/www/nacsos2/dramatiq.pid
KillMode=process
KillSignal=SIGHUP
TimeoutStopSec=30
FinalKillSignal=SIGKILL

[Install]
WantedBy=multi-user.target

Database migration

The nacsos_data library exposes the nacsos_migrate script (see entry points in setup.cfg). This in turn exposes some basic alembic functions. They rely on the same environment variables to be set as described in the database section above! However, you need to copy the alembic.ini from the nacsos_data to /var/www/nacsos2/nacsos-core/config/alembic.ini. This is not pretty, but only has to be done once during setup and keeps all configuration files in one place. During deployment (see .gitlab-ci in nacsos-core), the scrip will be called to essentially trigger alembic upgrade head.

NACSOS-web

NACSOS web is hosted via apache. During deployment, a GitLab runner executes npm run build, which creates all frontend files in the dist/ folder. This is cached as an "artefact" and copied to a folder apache can see and statically serve.

Create a file in /var/www/nacsos2/html/. This will be available to the plugins/api/index.ts for setting up the clients. Adapt the public-facing URLs accordingly (example below)

.env.production.local

VITE_NACSOS_BASE_URL=http://127.0.0.1:8080/
VITE_NACSOS_CORE_URL=http://127.0.0.1:8081

Permissions for systemd

Use sudo visudo to allow the gitlab-runner user to start/stop the services and update the server repository.

# Allow GitLab Runner to start/stop nacsos-core service
gitlab-runner ALL= NOPASSWD: /bin/systemctl restart nacsos-core.service
gitlab-runner ALL= NOPASSWD: /bin/systemctl stop nacsos-core.service
gitlab-runner ALL= NOPASSWD: /bin/systemctl start nacsos-core.service
gitlab-runner ALL= NOPASSWD: /bin/systemctl status nacsos-core.service

# Allow GitLab Runner to start/stop nacsos-pipelines service
gitlab-runner ALL= NOPASSWD: /bin/systemctl restart nacsos-dramatiq.service
gitlab-runner ALL= NOPASSWD: /bin/systemctl stop nacsos-dramatiq.service
gitlab-runner ALL= NOPASSWD: /bin/systemctl start nacsos-dramatiq.service
gitlab-runner ALL= NOPASSWD: /bin/systemctl status nacsos-dramatiq.service

# Allow GitLab Runner to transfer file ownership
gitlab-runner ALL= NOPASSWD: /bin/chown -R nacsos /var/www/nacsos2/nacsos-core
gitlab-runner ALL= NOPASSWD: /bin/chgrp -R nacsos /var/www/nacsos2/nacsos-core
gitlab-runner ALL= NOPASSWD: /bin/chown -R gitlab-runner /var/www/nacsos2/nacsos-core
gitlab-runner ALL= NOPASSWD: /bin/chgrp -R gitlab-runner /var/www/nacsos2/nacsos-core

Control flow is as follows (example):

rept@srv-mcc-apsis:~$ sudo -u nacsos -s
Agent pid 838
nacsos@srv-mcc-apsis:~$ sudo systemctl stop nacsos-core.service
# or
rept@srv-mcc-apsis:~$ sudo -u nacsos bash
Agent pid 39786
nacsos@srv-mcc-apsis:~$

Database Backups

Daily backups are handled by scheduled pipelines in GitLab. The pipeline is defined in the nacsos_data/.gitlab_ci ([backup-nacsos]) repository. The respective environment variables are not the group or project variables, but they are specified in a separate location in GitLab.

For each database (nacsos-legacy in postgres 12 and nacsos_core in postgres 15), we run a separate scheduled pipeline with the same script but different connection details. The script uses pg_dump to write the current snapshot to disk. We run it in parallelised mode, so it produces multiple files that are written to $BACKUP_LOCATION. This directory is then zipped and copied with a timestamp to $BACKUP_PATH. We then keep the last $KEEP_DAILY archive files in that folder. On Sunday (or the day specified in $REMOTE_DAY), the latest archive is copied to a remote location.

milvus

Running milvus in docker seems unreliable (it did just stop without restarting in the past...) Running it as a proper systemd service (via binary or deb package) would be preferred, but currently that seems to be impossible (but it used to be supported and they say it will be supported again).

For now, deployment works as such:

cd /srv/milvus
git clone https://github.com/milvus-io/milvus.git
cd milvus
git checkout v2.4.9
cd ..
cp milvus/configs/milvus.yml user.yml
vim user.yml # change minio port to 9100 (otherwise it clashes with other things)
milvus/scripts/standalone_embed.sh start
# verify things are running via
sudo docker ps 

Assuming milvus crashed

# check it's running
sudo docker ps

# start again
cd /srv/milvus
sudo milvus/scripts/standalone_embed.sh start

Failed notes trying to build it

Try 1

Build and install milvus as described here: https://github.com/milvus-io/milvus/tree/671112d17b2004cb3c93c76e35eff62c8da8b810/build/deb Currently (Aug 2024), we installed v2.4.9

Try 2

Note, that the docker image seems to be out of date regularly (based on issues people have). It's just as easy to create a quick python environment and pip install "conan==1.65.0". Also make sure you have apt install golang installed. Then edit the config according to the sed commands here and create the directories mkdir -p milvus-deb/milvus/milvus-bin and mkdir -p milvus-deb/milvus/milvus-lib and create the embedEtcd.yaml. Then run make milvus and wait.

Try 3

Alternatively build with docker:

# Get the source
git clone git@github.com:milvus-io/milvus.git
cd milvus
git checkout v2.4.9

# Run the development docker container and connect to it
sudo ./scripts/devcontainer.sh up
sudo docker exec -ti milvus-builder-1 bash

# Build milvus binary
make milvus

# Build package
cd build/deb
bash 

# see https://github.com/milvus-io/milvus/tree/master/build
# see https://github.com/milvus-io/milvus/issues/35329
# see https://github.com/milvus-io/milvus/blob/master/DEVELOPMENT.md
# see https://github.com/milvus-io/milvus/discussions/33654

Additional notes

Serving

• nacsos-web and nacsos-docs are served as static files from /var/www/nacsos2/html and /var/www/nacsos2/docs via nginx.

Logs

• Most logs are written to /var/www/nacsos2/logs/.
• and via journalctl -u <service-name>