Arrays and tensors

A multi-dimensional array is a collection that represents a tensor whose elements are numeric values arranged as a cube in a number of dimensions. For example, a numerical vector is a 1-dimensional (1D) array, while a matrix is a 2D array.

Multi-dimensional arrays are represented as objects of type Array. The difference between objects of type Array and type Vector is that type Vector represent 1D sequences of objects of any kind, while 1D arrays very efficiently represent dense numerical vectors.

All elements in an array have the same data representation or format. The format specifies the internal binary representation of the elements. There is a built-in subtype of Array for each format. The format is specified by strings:

Format	Internal representation	Type signature
f64/F64	64-bit float (Real)	Array of F64
f32/F32	32-bit float	Array of F32
i64/I64	64-bit integer (Integer)	Array of I64
i32/I32	32-bit integer	Array of I32
u32/U32	32-bit unsigned integer	Array of U32
i16/I16	16-bit integer	Array of I16
u16/U16	16-bit unsigned integer	Array of U16
i8/I8	8-bit integer	Array of I8
u8/U8	8-bit unsigned integer	Array of U8

Creating arrays

An array can be created with the array(Vector elements)->Array constructor function, where elements is a vector holding the elements of the new array. The default type for arrays is 64-bit integers (format 'I64') for integer elements, and 64-bit floats (format 'F64') for floating point elements.

Example: The following statement constructs a numerical vector of 64-bits integers:

set :a1 = array([10,20,30])

typesig(:a1)

Example: The following statement constructs a numerical vector of 64-bits real numbers:

set :a2 = array([10.0, 20, 30, 40])

typesig(:a2)

The elements of array :a2 are real numbers since one of the array elements in the array() call is a real number.

The function array(Charstring format, Vector elements)->Array takes an extra first argument being the format of the elements.

For example:

set :a2 = array('I8',[1,2,3])

typesig(:a2)  -- a numerical vector of 8-bit integers

You can create arrays of higher rank than one by nesting the elements vector.

Example: The following expression creates a matrix :m having three columns and two rows.

set :m = array([[1,2,3],
                [4,5,6]])

typesig(:m)

To copy an array you pass the array you want to copy into the array(Array a)->Array or array(Charstring fmt, Array a)->Arrayfunction.

set :a3 = array([1,2,3])

set :a4 = array("U8", :a3)

set :a5 = array(format(:a4), :a4)

Array metadata

An array object contains metadata describing its shape, number of elements, number of dimensions, and datatype.

The array metadata functions:

shape(Array a)->Vector of Integer    - Shape of array (e.g., [3,2] for a 3x2 array)
dim(Array a)->Integer                - Number of elements in array 
rank(Array a)->Integer               - Number of dimensions
format(Array a)->Charstring          - Format of array elements (e.g., 'F64', `I8`)
length(Array a,Integer ax)->Integer  - The length of dimension ax in array a

Example:

set :d = array("I16", [[1,2,3],
                       [4,5,6]])

typesig(:d)

shape(:d)

dim(:d)

rank(:d)

format(:d)

length(:d, 2)

Indexing arrays

Arrays are indexed with square brackets [].

set :arr = array([1,2,3,4,5,6,7,8,9,10])

:arr[3]

set :arr = array([[1,2,3],
                  [4,5,6]])

:arr[1,2]

:arr[2,3]

OSQL also supports linear indexing of multidimensional arrays.

set :arr = array([[1,2,3],
                  [4,5,6]])

:arr[5]

The character * can be used to omit dimensions to form subarrays, called array slicing.

set :arr = array([[ 1,  2,  3,  4,  5],
                  [ 6,  7,  8,  9, 10],
                  [11, 12, 13, 14, 15],
                  [16, 17, 18, 19, 20]])

:arr[1,*]

:arr[*,3]

Arithmetics

The following infix operators and functions over arrays a and b, and numbers lambda are defined:

operation	description	implemented as function
a + b	element-wise addition	plus
a .+ b	-"-	plus
lambda + a	add number to elements	plus
a + lambda	-"-	plus
a - b	element-wise subtractions	minus
a .- b	-"-	minus
a - lambda	subtract number from elements	minus
lambda - a	subtract elements from number	minus
-a	negate elements	uminus
a .* b	element-wise multiplications	elemtimes
lambda * a	multiply elements with number	times
a * lambda	-"-	times
dot(a,b)	dot produt	dot
matmul(a,b)	matrix multiplication	matmul
a ./ b	element-wise division	elemdiv
a / lambda	divide elements with number	div
lambda / a	divide number with elements	div
a .^ lambda	element-wise exponent	elempower

To exemplify the array operators and function we use the following two arrays:

set :a = array("f32", [[1,2],[2,4]])

typesig(:a)

set :b = array("f32", [[-1,-2],[-4,-8]])

Addition

Example: Element-wise addition.

:a + :b

is the same as

:a .+ :b

Example: Add number to elements

5 + :a

is the same as

:a + 5

Subtraction

Example: Element-wise subtraction

:a - :b

is the same as

:a .- :b

Example: Subtract number from elements

:a - 5

Example: Subtract elements from number

5 - :a

Example: Negate elements

- :a

Multiplication

Example: Element-wise multiplication

:a .* :b

Example: Multiply elements with number

:a * 2

is the same as

2 * :a

Example: Dot product

dot(:a, :b)

Example: Matrix multiplication

matmul(:a, :b)

Division

Example: Element-wise division

:a ./ :b

Example: Divide elements with number

:a / 2

Example: Divide number with elements

2 / :a

Exponent

Example: Element-wise exponent

:a .^ 2

Aggregation

Arrays can be used as arguments to aggregate functions. If an aggregate function is called with an array as argument it will aggregate over all array elements, independent of the array's shape.

We exemplify aggregation with the following array:

set :a8 = array('i8',[[1,2,3],
                      [4,5,6]])

typesig(:a8)

Example: The largest element in :a8.

max(:a8)

Example: The smallest element in :a8.

min(:a8)

Example: The sum of the elements in :a8.

sum(:a8)

Example: The sum of the first column elements of :a8.

sum(:a8[*,1])

Example: The mean of the elements in the second row of :a8.

avg(:a8[2,*])

Select array queries

You can specify queries that construct new arrays as array comprehension queries, where you specify both shape and format of the result along with index variables that are bound when specifying the values of array elements.

Example: Constructs a 1D array named out with shape [5] indexed by the variable i

select Array out[i..5] of U8 e
 where e = i

The array name is optional and can be omitted:

select Array[i..5] of U8 e
 where e = i + 3

Example: Construct a 5x5 unity matrix of integers with format ´U8´

select Array[i..5, j..5] of U8 e
 where e = case when i = j then 1
                else 0 end

You can use arrays in regular queries just like with all other data types.

Example:

select a
  from Array a
 where a = array('U8',[1,2,3])

Defining array functions

Derived functions can be defined to create and transform arrays.

Example: This function create a new array

create function my_arr() -> Array
  as array("U8", [1,2,3])

my_arr()

Variables can be used for defining the shape in a select array query.

Example:

create function inc_2d(Integer sz) -> Array
  as select Array[y..sz, x..sz] of U8 val
      where val = (y-1) + (x-1)

inc_2d(2)

inc_2d(5)

The array size does not have to be a scalar. It can be an expression.

Example:

create function inc(Integer sz) -> Array
  as select Array[i..(sz + 3)] of U8 val
      where val = i

inc(5)

Arrays as parameters

Functions can take arrays as arguments.

Examples:

create function my_fun(Array a) -> Integer
  as a[2]*10

my_fun(array([1,2,3]))

create function my_fun2(Array a) -> Integer
  as select a[i]*10
      from Integer i
     where i = shape(a)[1]/2

my_fun2(array([1,2,3,4,5,6]))

Functions are type-aware, so if you use the generic type Array as result type (without the data type format) the system will deduce the exact result type automatically.

Example:

create function my_fun(Array a) -> Array
  as a + 2

my_fun(array("U8",[1,2,3]))

You can also specify the format and declare array shape variables directly in the function signature.

Example: The following takes an Array of U8 array as argument and creates a nes Array of U8 object with the same shape.

create function transpose(Array arr[ys,xs] of U8) -> Array
  as select Array[x..xs, y..ys] of U8 val
      where val = arr[y,x]

set :a2 = array("U8", [[1,2,3,4],
                       [5,6,7,8],
                       [9,10,11,12]])

transpose(:a2)

Overloaded array functions

OSQL supports overloading of array functions based on array formats (but not on shapes).

Example:

create function overload(Array of F64 a, Array of F64 b) -> Real
  as sum(a .* b)

create function overload(Array of F32 a, Array of F32 b) -> Real
  as sum(a ./ b)

overload(array("F64", [1.0,2,3]), array("F64", [4.0,5,6]))

roundto(overload(array("F32", [1.0,2,3]), array("F32", [2.0,4.0,5])), 2)

It is possible to use the same shape variable for multiple arguments. This will put constraints on the arguments requiring matching shapes.

Example: The following Boolean function tests that arrays u and v are numerical vectors of the same size.

create function sameshape(Array u[dim], Array v[dim])
                        -> Boolean
   as true

sameshape(array([1,2,3,4]), array([5,6,7,8]))

sameshape(array([1,2,3,4]), array([5,6,7,8,9]))

sameshape(array([1,2,3,4]), array([[5,6],[7,8]]))

Example: The following Boolean function tests that an array a is a square matrix.

create function is_square(Array a[dim,dim]) -> Boolean
   as true

is_square(array([[1,2],
                 [3,4]]))

is_square(array([[1,2],
                 [3,4],
                 [5,6]]))

Reshaping arrays

Arrays can be reshaped with the function reshape(Array a,Vector shape)->Array. You can pass a desired shape, but the number of elements have to be the same as the original.

Example:

set :arr = array("u8", [[1,  2,  3,  4],
                        [5,  6,  7,  8],
                        [9, 10, 11, 12],
                        [13, 14, 15, 16]])

reshape(:arr, [8,2])

You can reshape a multi-dimenasional array to a 1D numerical vector by the function flatten(Array)->Array:

set :f = flatten(:arr)

:f[10]

Plotting arrays

The default plot for arrays is Text. But when the arrays represent images you can plot the images by using the Bitmap plot. It works for both gray level images (arrays with shape [Y,X]) and color images (arrays with shape [Y,X,3]). Almost all array data types are supported. Negative element values will be displayed as black.

Let's define two functions that generate images, one gray level and one color image:

create function inc_2d(Integer sz) -> Array
  as select Array[y..sz, x..sz] of U8 val
      where val = (y-1) + (x-1)

create function rgb_arr(Integer sz) -> Array
  as select Array[y..sz,x..sz,c..3] of U8 val
      where val = case when c = 1 then mod(x+y,256)
                       when c = 2 then mod(abs(x-y),256)
                       else mod((x + y)/2,256) end

Now we can plot the generated arrays with the //plot: Bitmap directive.

//plot: Bitmap
inc_2d(200)

//plot: Bitmap
rgb_arr(500)

Note that the channel order is RGB (unlike, e.g., OpenCV, which uses BGR).

To illustrate this we can create a function that returns a "red" image:

create function red(Integer sz) -> Array
  as select Array[y..sz,x..sz,c..3] of U8 val
      where val = case when c = 1 then mod(x + y,256)
                       else 0 end

Now we can plot the red image with the //plot: Bitmap directive.

//plot: Bitmap
red(200)

Functions

There is documentation of the system functions over arrays in Array functions.