SA Engine Python Interfaces

Version 1.0 (Updated: 2023-06-21)

This document offers a look at the external interfaces between SA Engine and the programming language Python. Broadly, there are two key ways to connect SA Engine with Python programs:

1. Client Interface: In this setup, Python programs call upon SA Engine.
2. Plugin Interface: Here, foreign OSQL functions are implemented as Python methods. You can also employ a combination of the two, wherein foreign functions in Python call SA Engine back through the client interface.

Prerequisite

To call Python from SA Engine, it's essential to install Python and SA Engine.

SA Engine should be version 5.0.2 or later.

The Python versions supported may vary by platform. While other Python versions might work, our extension has been extensively tested on the following Python versions:

Platform	Python version
Windows	3.8
Mac OS X	3.8
Linux x86	3.8
Raspberry Pi 32 bit	3.7
Raspberry Pi 64 bit	3.9

Should you require support for a different version of Python or need Python support for an alternative platform, please submit an issue on our community repo.

Ensure that the bin directory under sa.engine is included in your PYTHONPATH variable. For example, if SA_ENGINE_HOME points to the SA Engine directory:

export PYTHONPATH=${SA_ENGINE_HOME}/bin:${PYTHONPATH}

Also make sure to include ${SA_ENGINE_HOME}/bin in your PATH.

Mangaging python packages

The SA Engine process needs to run in a python environment with the correct version and with all needed python packages installed. If, for instance, an Anaconda environment is used, ensure that the envrionment is active in the SA Engine context. To achieve this, the SA Engine client (SA Engine CLI, VS Code, or SA Studio) can be started from within the environment.

conda activate my-python-env
code

Note

When executing queries or functions on an edge, it is the python environment installed on that edge that will run the query.

Mac OS-specific setup

For the RPATH in Python extender, a Python distribution under /usr/local/opt/python@3.8 is necessary. To install Python 3.8 using Homebrew, follow these commands:

Install HomeBrew

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

and follow the instructions.

Install Python 3.8 using HomeBrew

brew install python@3.8

Create a symbolic link to the Python installation

ln -s /usr/local/bin/python3.8@  /usr/local/opt/python@3.8

(Assuming that HomeBrew installs Python 3.8 to /usr/local/bin/.)

Install required dependencies

Dependencies can be installed by running the pip command from within the 3.8 installation. For example:

python3.8 -m pip install h5py

Python client interface

The Python client interface setup is used when Python functions call upon SA Engine functions. SA Engine acts as an embedding to Python.

Here's an example on how to initialize a connection to the local SA Engine in the Python process and execute various operations:

import sa_python as sa_engine

# Initialize a connection to the local SA Engine in the python process:
local = sa_engine.connect("")

# All results from SA Engine are iterators:
# Let's create an OSQL function that will call the python built in function abs:
[x for x in local.query("""create function foo(Number a) -> Number as foreign "py:abs";""")]

# And call it as a query:
[x for x in local.query("foo(-1)")]
# Or as a function:
[x for x in local.call("foo",-2)]

# Run a query that starta a nameserver and edge. Will not yield output.
local.query("start_nameserver('s');wait_for('s');").run()
local.query("start_edge('e1');").run()

# You can now open connections to the peers in your local federation
s = sa_engine.connect("s")
e1 = sa_engine.connect("e1")

# Then run queris on each of these peers.
[x for x in s.query("this_peerid()")]
[x for x in e1.query("this_peerid()")]

# This means that you can access any peer in the world that is 
# connected to the same federation as your python instance!
# Let's kill the federation:
local.query("kill_the_federation()").run()

Python plugin interface

The Python plugin interface setup is used when SA Engine call upon Python, using foreign functions. A foreign function is defined as call to a python function, with a syntax as in this example:

create function myabs(Number a) -> number 
  as foreign 'py:abs';

myabs(-1);

The function after py: can be either a built-in python funtion, or a function defined in a .py file. If its the later, the foreign function should define a relative path to the file from a location in the PYTHONPATH.

When running SA Engine the models folder models:folder() is always added to the PYTHONPATH of the process. This means that if you create a model my_python_model in OSQL and put a file my_python.py with the following python in it:

import numpy as np
# Lets create a python function that we call from OSQL!
def my_generator(l, u):
    for x in range(l,u):
        yield np.array(range(l,x))

You will be able to call in from SA Engine by declaring the following function:

create function my_py_generator(Number l, Number u) -> Stream of Array 
 as foreign "py:my_python_model.my_python.my_generator";

my_py_generator(1,10);

If everything worked you should get the following output:

[sa.engine] >my_py_generator(1,10);

3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 05:59:45) [MSC v.1929 64 bit (AMD64)]
array([])
array('I32',[1])
array('I32',[1,2])
array('I32',[1,2,3])
array('I32',[1,2,3,4])
array('I32',[1,2,3,4,5])
array('I32',[1,2,3,4,5,6])
array('I32',[1,2,3,4,5,6,7])
array('I32',[1,2,3,4,5,6,7,8])
1.141 s
[sa.engine] >

Note that we support passing Numpy arrays by reference, which significantly improves the efficiency of transferring Array objects between Python and SA Engine!

API

Object	Method	Docs
sa_python	connect(peerspec: string) -> connection	Opens a connection to a reachable peer. It's crucial to at least call sa_python.connect("") to initialize the local client in the Python process
connection	query(q: string) -> Iterator	Executes query q on the SA Engine instance associated with the connection
	call(fn: string,args...) -> Iterator	Calls function fn with args on the SA Engine instance associated with the connection
	run(q: string) -> void	Executes query on the SA Engine instance associated with the connection