Model testing and integration

Guide specification
Guide type:	Wasm code
Requirements:	None
Recommended reading:	Managing models, OSQL Testing

Introduction

In Managing models we learned how models can be packages into model releases and how model releases can be imported into an SA Engine instance.

In this guide we look at how automated testing could be set up for such models. An integration test using SA Engine will be built up step-by-step. The goal is to have a setup like:

1. Every push to the git repository triggers a test job
2. The test job installs SA Engine and loads the model
3. The test.osql written for the model is executed
4. The test job should run on different platforms

To see the complete code, refer to https://github.com/streamanalyze/ci-demo-public.

While this setup is using GitHub Actions, the test scripts used could be called from other CI tools as well.

1. Setup your git repo

Create a git repo and clone it on the computer where you will be developing your models. In the repo create the following folder structure:

SA/
├─ models/
└─ models-releases/

Set the SA_HOME environment variable on the computer to point to the SA folder. User Models in SA Studio will now point to the models folder.

SA Studio automatically creates some files that do not need to be shared between users. It is recommended to add these to .gitignore in SA folder:

bin
docs
federations
logs
temp
tls
API_AUTH
federation.localhost

Test existing model

If you have already created some models that you want to test, you can just push your SA_HOME folder to git.

2. Create the model to test

In SA Studio create a model eg. my-model. Add an example function in master.osql:

create function prime(Integer n) -> boolean
    as select true where notany(select d
       from Integer d
      where d in range(2,floor(sqrt(n)))
        and 0 = mod(n,d));

In test.osql, add a test. (Read more about testing in OSQL Testing.)

validate "my prime"
    check prime(0) => true
    check prime(13) => true
    check prime(4) => null
    check prime(20) => null
    check prime(31) => true;

3. Create the GitHub Actions workflow

In a top level folder called .github\workflows, create a file named github-actions.yml. Choose name, trigger, and runner using instructions from GitHub. The ci-demo runs on one of the GitHub provided runners:

name: GitHub Actions Demo
on: [push]
jobs:
  TestModel:
    runs-on: ubuntu-latest

4. Install SA Engine and the model

To place the model code on the test runner, one of the pre-defined actions can be used. Note that the checkout action will clean the folder, so it should be called before downloading SA Engine.

- name: Check out repository code
        uses: actions/checkout@v3

The test runner should install SA Engine to be able to test the model. To do this, create steps that download the correct version of SA Engine from our download site. When you are logged in as user of SA Studio, you can retrieve a personal download link on studio.streamanalyze.com/download/.

As the runner is a linux machine, select an SA Engine for linux. To install it, unpack it using tar.

- name: Download sa.engine release
    run: |
      curl -L -o "sa_engine_core_linux_x64.tar.gz" "https://studio.streamanalyze.com/download/Y2ktZGVtb0BzdHJlYW1hbmFseXplLmNvbQ==/3dae5506d45c299cb0b86a8e321d5f74/v4.16.2/linux/sa_engine_core_linux_x64.tar.gz"
- name: install SA Engine
    run: |
      tar xzvf sa_engine_core_linux_x64.tar.gz

5. Create the test script

Create a script file for the first test script. In the ci-demo repo, test scripts are placed in a top level folder named ci and the script is called test-linux.sh. Call the script from GitHub Actions, after setting the executable file permission.

- name: Test models
    run: |
      chmod +x ci/test-linux.sh && ci/test-linux.sh

Add the following code to test-linux.sh for testing my-model:

./sa.engine/bin/sa.engine -f SA -o "models:test('my-model')" -o "quit;"

These are the parts of the command:

./sa.engine/bin/sa.engine   : call the sa.engine instance we installed
-f SA                       : set the SA_HOME of this sa.engine run to the SA folder we checked out
-o "models:test('my-model')" : run the command models:test('my-model'), this will load my-model and run all osql in test.osql
-o "quit;"                  : close sa.engine with an exit code. This is important to make the test run fail if any validate fails.

Now, pushing to git should trigger a test of my-model.

To verify that the model is indeed tested, lets add a failing test to test.osql:

validate "my prime failing"
    check prime(20) => true;

Push the updated test file to git. The test should now fail, displaying the failing validate statement from test.osql.

6. Test on an emulated Raspberry Pi

Another example, implemented as a second job in github-actions.yml, is to run the tests on another platform.

In this example, a docker-container called qemu-pi is used. It contains a QEMU-based emulation of a Raspberry Pi 3 Model B with a 32 bit Bullseye Raspberry OS. When the docker container is started, the QEMU emulation is started inside and SA Engine is automatically installed.

The container is stored and retrieved from GitHub container repository:

TestModelonEmulatedPi:
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/streamanalyze/sa.qemu-pi:1.0.1

When starting docker containers like this, GitHub overrides the entrypoint. To execute the default entrypoint script, call it manually.

steps:
  - name: Run entrypoint
    run: |
        screen -dm bash -c  "entrypoint.sh" & 

Since the container already installs SA Engine, it is enough to check out the code and call the test script:

steps:
  - name: Check out repository code
    uses: actions/checkout@v3
  - name: Test models
    run: |
        chmod +x ci/test-pi.sh && ci/test-pi.sh

The test script for the emulated Pi uses the same command as on Ubuntu. To run the command inside the Raspberry OS, we use copy-to-pi and execute-on-pi helper scripts available in the container.

TESTCMD= "./sa.engine/bin/sa.engine -f SA -o 'models:test(\"my-model\")' -o 'quit;' ";

./copy-to-pi.sh SA
./execute-on-pi.sh "$TESTCMD"