SA Engine Python Interfaces

Version 1.0 (Updated: 2023-06-21)

This document offers a look at the external interfaces between SA Engine and the programming language Python. Broadly, there are two key ways to connect SA Engine with Python programs:

Client Interface: In this setup, Python programs call upon SA Engine.
Plugin Interface: Here, foreign OSQL functions are implemented as Python methods. You can also employ a combination of the two, wherein foreign functions in Python call SA Engine back through the client interface.

Prerequisites

To call Python from SA Engine, you need SA Engine (5.0.2+) and a supported Python runtime available to the SA Engine process.

Installation instructions for SA Engine can be found in the SA Engine Quick start guide.

The supported Python versions may vary by platform. While other Python versions might work, our extension has been extensively tested on the following Python versions:

Platform	Python version
Windows	3.8
Mac OS X	3.8
Linux x86	3.8
Raspberry Pi 32 bit	3.7
Raspberry Pi 64 bit	3.9

Should you require support for a different version of Python, or need Python support for an alternative platform, please submit an issue on our community repo.

Install Python (recommended: Miniconda)

We recommend using Miniconda to manage the exact Python version and packages that SA Engine will use. This avoids conflicts with system Python and makes it easy to reproduce environments.

Linux / WSL / macOS (shell)

Download and install Miniconda (adjust the installer name for your OS/arch if needed):

# Example for Linux x86_64 (WSL included):
cd ~
curl -LO https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh

# Initialize conda in your shell and reload it:
~/miniconda3/bin/conda init
exec "$SHELL"

Windows (PowerShell)

If you prefer command-line installation on Windows, you can use winget:

winget install --id Anaconda.Miniconda3 -e

Then open a new terminal and ensure conda works:

conda --version

Ensure that the bin/ directory under SA Engine is included in your environment. Start by setting SA_ENGINE_HOME to your installation directory (modify the path to match your setup):

export SA_ENGINE_HOME="$HOME/sa.engine"

export PYTHONPATH="${SA_ENGINE_HOME}/bin:${PYTHONPATH}"
export PATH="${SA_ENGINE_HOME}/bin:${PATH}"
export LD_LIBRARY_PATH="${SA_ENGINE_HOME}/bin:${LD_LIBRARY_PATH}"

To make these settings persist for new terminals, add the exports to ~/.bashrc and reload it:

nano ~/.bashrc
# (paste the exports above, save, exit)
source ~/.bashrc

On Windows (PowerShell), you can set these for the current session like this:

$env:PYTHONPATH = "$env:SA_ENGINE_HOME\bin;$env:PYTHONPATH"
$env:Path = "$env:SA_ENGINE_HOME\bin;$env:Path"

Managing Python packages

SA Engine runs Python code in the Python environment visible to the SA Engine process (Python version + installed packages). The most reliable approach is:

Create a dedicated environment (conda or venv)
Install your required packages into that environment
Start SA Engine tooling from an activated environment, so the process inherits the right PATH/PYTHONPATH

Option A (recommended): conda environment

Create an environment using one of the tested Python versions (example uses Python 3.8):

conda create -n sa-python python=3.8 pip
conda activate sa-python

Linux / WSL / macOS: conda runtime library path (recommended)

On non-Windows platforms, SA Engine’s Python integration may need help locating shared libraries from your active conda environment (for example, libpython and other .so/.dylib dependencies). After you activate your conda environment, CONDA_PREFIX points at that environment directory.

Run this after conda activate ... in the same terminal where you will start SA Engine tooling:

export LD_LIBRARY_PATH="${SA_ENGINE_HOME}/bin:${CONDA_PREFIX}/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}"

To print the active conda environment path (useful for copy/paste when hardcoding), run:

echo "$CONDA_PREFIX"

If you want this to be persistent, add it to your ~/.bashrc (for example, /home/<YOUR_USERNAME>/.bashrc) as a hardcoded path to a specific env:

# Example: hardcode one specific conda env path (edit to match your env name/location)
export LD_LIBRARY_PATH="${SA_ENGINE_HOME}/bin:$HOME/miniconda3/envs/sa-python/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}"

# Install packages your foreign functions depend on:
python -m pip install --upgrade pip
python -m pip install numpy

Start the SA Engine client (SA Engine CLI, VS Code, or SA Studio) from the same activated environment:

conda activate sa-python
code

If you are activating conda from a non-interactive shell (e.g., scripts), you may need to initialize it first:

source ~/miniconda3/etc/profile.d/conda.sh
conda activate sa-python

Option B: venv + pip (no conda)

If you prefer the standard library venv:

python3 -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install numpy

Quick sanity checks

From the same terminal you’ll use to start your SA Engine tooling, verify Python is the one you expect and that imports work:

which python
python --version
python -c "import numpy; print('OK')"

Note

When executing queries or functions on an edge, it is the Python environment installed on that edge that will run the query.

Mac OS-specific setup

For the RPATH in Python extender, a Python distribution under /usr/local/opt/python@3.8 is necessary. To install Python 3.8 using Homebrew, follow these commands:

Install HomeBrew

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

and follow the instructions.

Install Python 3.8 using HomeBrew

brew install python@3.8

Create a symbolic link to the Python installation

ln -s /usr/local/bin/python3.8@  /usr/local/opt/python@3.8

(Assuming that HomeBrew installs Python 3.8 to /usr/local/bin/.)

Install required dependencies

Dependencies can be installed by running the pip command from within the 3.8 installation. For example:

python3.8 -m pip install h5py

Python client interface

The Python client interface setup is used when Python functions call upon SA Engine functions. SA Engine acts as an embedding to Python.

Here's an example on how to initialize a connection to the local SA Engine in the Python process and execute various operations:

import sa_python as sa_engine

# Initialize a connection to the local SA Engine in the python process:
local = sa_engine.connect("")

# All results from SA Engine are iterators:
# Let's create an OSQL function that will call the python built in function abs:
[x for x in local.query("""create function foo(Number a) -> Number as foreign "py:abs";""")]

# And call it as a query:
[x for x in local.query("foo(-1)")]
# Or as a function:
[x for x in local.call("foo",-2)]

# Run a query that starta a nameserver and edge. Will not yield output.
local.query("start_nameserver('s');wait_for('s');").run()
local.query("start_edge('e1');").run()

# You can now open connections to the peers in your local federation
s = sa_engine.connect("s")
e1 = sa_engine.connect("e1")

# Then run queris on each of these peers.
[x for x in s.query("this_peerid()")]
[x for x in e1.query("this_peerid()")]

# This means that you can access any peer in the world that is 
# connected to the same federation as your python instance!
# Let's kill the federation:
local.query("kill_the_federation()").run()

Python plugin interface

The Python plugin interface setup is used when SA Engine call upon Python, using foreign functions. A foreign function is defined as call to a python function, with a syntax as in this example:

create function myabs(Number a) -> number 
  as foreign 'py:abs';

myabs(-1);

The function after py: can be either a built-in python funtion, or a function defined in a .py file. If its the later, the foreign function should define a relative path to the file from a location in the PYTHONPATH.

When running SA Engine the models folder models:folder() is always added to the PYTHONPATH of the process. This means that if you create a model my_python_model in OSQL and put a file my_python.py with the following python in it:

import numpy as np
# Lets create a python function that we call from OSQL!
def my_generator(l, u):
    for x in range(l,u):
        yield np.array(range(l,x))

You will be able to call in from SA Engine by declaring the following function:

create function my_py_generator(Number l, Number u) -> Stream of Array 
 as foreign "py:my_python_model.my_python.my_generator";

my_py_generator(1,10);

If everything worked you should get the following output:

[sa.engine] >my_py_generator(1,10);

3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 05:59:45) [MSC v.1929 64 bit (AMD64)]
array([])
array('I32',[1])
array('I32',[1,2])
array('I32',[1,2,3])
array('I32',[1,2,3,4])
array('I32',[1,2,3,4,5])
array('I32',[1,2,3,4,5,6])
array('I32',[1,2,3,4,5,6,7])
array('I32',[1,2,3,4,5,6,7,8])
1.141 s
[sa.engine] >

Note that we support passing Numpy arrays by reference, which significantly improves the efficiency of transferring Array objects between Python and SA Engine!

API

Object	Method	Docs
sa_python	`connect(peerspec: string) -> connection`	Opens a connection to a reachable peer. It's crucial to at least call `sa_python.connect("")` to initialize the local client in the Python process
connection	`query(q: string) -> Iterator`	Executes query `q` on the SA Engine instance associated with the `connection`
	`call(fn: string,args...) -> Iterator`	Calls function `fn` with `args` on the SA Engine instance associated with the `connection`
	`run(q: string) -> void`	Executes query on the SA Engine instance associated with the `connection`

SA Engine Python Interfaces

Prerequisites​

Install Python (recommended: Miniconda)​

Linux / WSL / macOS (shell)​

Windows (PowerShell)​

Managing Python packages​

Option A (recommended): conda environment​

Linux / WSL / macOS: conda runtime library path (recommended)​

Option B: venv + pip (no conda)​

Quick sanity checks​

Mac OS-specific setup​

Install HomeBrew​

Install Python 3.8 using HomeBrew​

Create a symbolic link to the Python installation​

Install required dependencies​

Python client interface​

Python plugin interface​

API​