SA Lisp User’s Guide

Stream Analyze Sweden AB
Version 3.0
2023-12-09

SA Lisp is an interpreter for a subset of CommonLisp built on top of the storage manager of the SA Engine system. The storage manager in scalable and extensible, which allows data structures to grow very large gracefully and dynamically without performance degradation. Its garbage collector is incremental and based on reference counting techniques. This means that the system never needs to stop for storage reorganization and makes the behavior of SA Lisp very predictable. SA Lisp is written in C99 and is tightly connected with C. Thus Lisp data structures can be shared and exchanged with C programs without data copying and there are primitives to call C functions from SA Lisp and vice versa. The storage manager is extensible so that new data structures shared between C and Lisp can be introduced to the system. SA Lisp runs under Windows, Unix, and OS X and can easily be ported to any architecture having a C99 compiler. It is delivered as either an executable or a C library (DLL, shared library), which makes it easy to embed SA Lisp in other systems. This report documents how to use the SA Lisp system.

Introduction

SA Lisp is a small but scalable Lisp interpreter that has been developed with the aim of being embedded in other systems. It is tightly interfaced with C and can share data structures and code with C to avoid unnecessary copying. SA Lisp supports a subset of CommonLisp and conforms to the CommonLisp standard [1][2] when possible. However, it is not a full CommonLisp implementation, but rather such CommonLisp constructs are not implemented that are felt not being critical or are difficult to implement efficiently. These restrictions make SA Lisp small and light-weight, which is important when embedding it in other systems.

SA Lisp is designed to be embedded in other systems, in particular the SA Engine [3] federated data analytics system. However, SA Lisp is a general system and can be used as a stand-alone Lisp engine as well. This document describes the stand-alone SA Lisp system and its differences to CommonLisp. SA Lisp includes a main-memory storage manager used for processing data in real-time. Thus all data structures are dynamic and can grow without performance degradation. The data structures grow gracefully so that there are never any significant delays for data reorganization, garbage collection, or data copying. (Except that the OS might sometimes do this, outside the control of SA Lisp). There are no limitations on how large the data area can grow, except OS address space limitations. The performance is of course dependent on the size of the available main memory and thrashing may occur when the amount of memory used by SA Lisp is larger than the main memory.

A critical component in a Lisp system is its garbage collector. Lisp programs often generate large amounts of temporary data areas that need to be reclaimed by the garbage collector. Furthermore, SA Lisp is designed to be used in a real-time main-memory data storage system and therefore it is essential that the garbage collection is predictable, i.e. it is not acceptable if the system would stop for garbage collection now and then. The garbage collector must therefore be incremental and continuously reclaim freed storage. Another requirement for SA Lisp is that it can share data structures with C, in order to be tightly embedded in other systems without causing delays by copying data between subsystems. Therefore, unlike most other implementations of Lisp, both C and Lisp data structures are allocated in the same memory area and there is no need for expensive copying of large data areas between C and Lisp. This is essential for a predictable interface between C and Lisp, in particular if it is used for managing large database objects.

The CommonLisp standard functions are defined in [1] and [2]. Significant differences between a SA Lisp functions and the corresponding CommonLisp functions are described in this document. It is organized as follows:

Starting SA Lisp explains how to get started to run SA Lisp.

Basic Primitives describes the basic system primitives in SA Lisp.

Basic Functions describes basic built in data types and the primitive functions that operate on them.

Second Order Functions describes how to use and define different kinds of functional expressions. In particular functional programming mechanisms such as higher order functions, control structures, and macros are explained.

Input and Output describes the I/O and inter-process communication systems.

Error Handling describes the exception handling mechanisms.

Lisp Debugging describes the debugging and performance tuning facilities.

Code Search and Analysis describes the code search, analysis and documentation system that allows for searching for Lisp code having certain properties. A rule driven validation system searches through Lisp code to find possible errors such as unbound variables, undefined functions, or other questionable Lisp code. A plug-in to the Emacs editor provides and interface between the editor and SA Engine.

In this document, the notation x... indicates that the expression x can be repeated one or several times, while [x] indicates that x is optional.

Starting SA Lisp

SA Lisp is a subsystem of SA Engine. When you download and install SA Engine you also get an executable sa.engine, which includes an SA Lisp. Start SA Lisp from the console with:

sa.engine -q lisp
Database image: myfile.dmp
Release 5, v4, 64 bits
Lisp >

The system first reads a database image file sa.engine.dmp that must be in the same folder as where the sa.engine executable is located. After the engine has been started, the system enters the SA Lisp Read Eval Print Loop, REPL. In the SA Lisp REPL the system reads S-expressions, evaluates them, and prints the results from the evaluation. The prompt Lisp > indicates that the Lisp REPL is run from the console, for example:

Lisp >(+ 1 2)     ;; Call Lisp function +
====>             ;; Indicates printed output from Lisp in this document
3                 ;; The REPL prints the result of the evaluation

Quit SA Lisp by evaluating:

Lisp >(quit)

Since SA Lisp is a subsystem of SA Engine the Lisp REPL can also be entered from the OSQL REPL of SA Engine by issuing the OSQL command lisp;:

[sa.engine] >lisp; ;; In OSQL REPL
Lisp >             ;; In Lisp REPL

To get into the OSQL REPL from the Lisp REPL evaluate the form :osql:

Lisp >:osql
[sa.engine] >      ;; Back in OSQL REPL

We now go through some simple interactions with the Lisp REPL.

First, variables are set using the macro setf that assigns one or several variables. It is a CommonLisp generalization of the classical Lisp assignment function setq:

Lisp >(setf a '(a b c))
====>
WARNING - Setting undeclared global variable: A ;; A warning message
(A B C)                                         ;; The result of the evaluation

As the example shows, SA Lisp warns the user when an undeclared global variable is set. To avoid this error message all global variables should be declared before being used. For example, to avoid the warning above one can first declare the variable _a_ to be global using defglobal:

Lisp >(defglobal _a_)      ;; Declare _A_ as global
====>
_A_                        ;; The value of DEFGLOBAL

Here a coding convention is used is used in SA Lisp that global variables always have _ (underline) as first character.

Lisp >(setf _a_ '(a b c))  ;; no warning this time
====>
(A B C)

Now, let's call the built-in function reverse:

Lisp >(reverse _a_)
====>
(C B A)

Let’s define a function:

Lisp >(defun foo (x)(cons x a))
====>
WARNING - Undeclared free variable A in FOO  ;; The system warns you about possible bugs
FOO 

As the example shows, SA Lisp warns the user when it encounters forms in function definitions that may contain questionable code. In this case we forgot that we renamed variable a to globally declared _a_. Let’s correct the error:

Lisp >(defun foo (x)(cons x _a_))
====>
WARNING - Redefined Lisp function: FOO
FOO
Lisp >(foo 1)
====>
(1 A B C)

This time the system just warned that foo was redefined.

All Lisp code and data are stored the database image. The image can be saved on disk in a file named myfile.dmp by evaluating:

Lisp >(rollout "myfile.dmp")

We can now exit SA Lisp by calling:

Lisp >(quit)

To later run SA Lisp with the saved image, issue the OS command:

sa.engine -q lisp myfile.dmp
Database image: myfile.dmp
Release 5, v4, 64 bits
Lisp >

The function FOO and the global variable _A_ are saved in the database image so we can evaluate:

Lisp >(foo 2)
====>
(2 A B C)

The SA Lisp system includes a command line debugger which is enabled by default when you enter the Lisp REPL. If debugging is enabled and you make an error the system will enter a break loop which is a special Lisp REPL where the error can be investigated.

For example, let's define a buggy function and call it, causing the break loop to be entered:

Lisp >(defun fie (x) (+ x b))
====>
WARNING - Undeclared free variable B in FIE 
FIE 
Lisp >(fie 1)
====>
ERROR - Unbound variable: B
Broken call to ERROR-HANDLER
Inside FIE brk>

At this point you are in the break loop where the message Broken call to ERROR-HANDLER indicates that the break loop was called from the system's error handler and the prompt Inside FIE brk> indicates that the error happened somewhere inside the function FIE.

Lisp forms are evaluated in the break loop as in the main Lisp REPL, so you can, e.g., use values of local variables in forms:

Broken call to ERROR-HANDLER
Inside FIE brk>(+ x 2)
====>
3

In the break loop you can issue a number of break commands. For example, the command :r aborts the break loop and returns the control to the main REPL. We now leave the break loop by the command :r and then correct the definition of FIE:

Broken call to ERROR-HANDLER
Inside FIE brk>:r              ;; Reset to top level REPL
====> NIL
Lisp >(defglobal _b_ 5)        ;; Define _B_ in main Lisp REPL
====>
_B_
Lisp >(defun fie (x)(+ x _b_)) ;; Correct FIE
====>
WARNING - Redefined Lisp function: FIE
FIE
Lisp >(fie 1)
====>
6                               ;; FIE OK now

The debugging system is documented in Section Lisp Debugging. The break command :help lists the break commands.

You also can search for Lisp functions whose names include a given substring by calling the function apropos, for example:

(apropos 'doc)  
(apropos 'calling)

Basic Primitives

Every object in SA Lisp belongs to exactly one data type. There is a type tag stored with each object that identifies its data type. Each data type has an associated type name as a symbol. The symbolic name of the data type of an SA Lisp object x can be accessed with the Lisp function:

(type-of x)

For example:

(type-of 123)
====>
INTEGER

A symbol (data type name SYMBOL) is a unique text string identifier with which various system objects can be associated. In particular functions and variables are identified by symbols.

SA Lisp provides a set of built-in basic data types. Furthermore, through its C-interface SA Lisp can be extended with new data types implemented in C. The system tightly interoperates with C so that:

• Data structures can be shared between C and SA Lisp.
• SA Lisp can call functions implemented in C.
• SA Lisp functions can be called from C.
• C can utilize SA Lisp’s error handling.
• C can utilize SA Lisp's real-time garbage collector.

Defining functions

Functions define various kinds of computations over Lisp objects. There are five different kinds of functions in SA Lisp:

1. A special form is a built-in Lisp function with varying number of arguments and where the arguments are not evaluated the standard way. The most basic special form is (quote x) returning the S-expression x without evaluating it. For example:

(quote hay)
====>
HAY
        
'hay             ;; Alternative syntax for (quote hay)
====>
HAY

The system function (symbol-function fn) returns the definition of the function fn. For example:

(symbol-function 'quote)
====>
#[EXTFN-2 QUOTE]

The definition of a special form is represented as an object of type EXTFN and printed as #[EXTFN-2 fn] as in the example. The EXTFN object contains a pointer to the C definition.

2. A lambda function is a function defined in Lisp using the special form defun. For example:

(defun rev (x)(cons (cdr x)(car x)))
====>
REV

(rev '(1 2))
====>
((2) . 1)

The representation of a lambda function is a list (LAMBDA arguments . body). For example:

(symbol-function 'rev)
====>
(LAMBDA (X) (CONS (CDR X) (CAR X)))

In debug mode the system will automatically analyze the new function definition to warn about eventual fishinesses.

3. An external function is implemented in C. An external Lisp function is represented by the data type EXTFN and printed as #[EXTFNn fn], where n is the arity of the function and fn is its name. For example:

(symbol-function 'car)
====>
#[EXTFN1 CAR]

4. A variadic external function can take a variable number of actual arguments. It definition is printed as #[EXTFN-1 fn]. For example:

(symbol-function 'list)
====>
#[EXTFN-1 LIST]

5. A macro is a Lisp function that takes a form as its argument and produces a new equivalent form. Macros are defined by the special form defmacro. Many system functions are defined as macros. They are Lisp’s rewrite rules. A fundamental capability of Lisp is its very powerful macro definitions overviewed in Macros.

The Type of a function can be EXTFN (defined in C), SPECIAL (special form), LAMBDA (defined in Lisp), or MACRO (Lisp macro) (Sec. 3.14). The type of functions that are similar or equivalent to standard CommonLisp functions in [1] and [2] are prefixed with a *.

The following system functions define and access functions:

Function	Type	Time Description
(DEFC fn def)	*EXTFN	Associate the function definition def with the symbol fn. Same as symbol-setfunction.
(DEFUN fn args form...)	SPECIAL	Define a new Lisp lambda function.
(DEFMACRO fn args form...)	SPECIAL	Define a new Lisp macro (Sec. 3.14).
(EXTFNP x)	LAMBDA	Return T if x is a function defined in C.
(FLET ((fn def) ...) form...)	*MACRO	Bind local function definitions and evaluate the forms form...
(FBOUNDP fn)	*EXTFN	Return T if fn is a function, macro or special form.
(FMAKEUNDEF fn)	LAMBDA	Remove function, macro or special form definition of fn.
(GETD fn)	*EXTFN	Get function definition of symbol fn. Same as symbol-function.
(LAMBDAP x)	LAMBDA	Return T if x is a lambda expression.
(MOVD fn1 fn2)	*EXTFN	Make fn2 have the same function or macro definition as fn1.
(SYMBOL-FUNCTION s)	*EXTFN	Get the function definition associated with the symbol s. Same as getd.
(SYMBOL-SETFUNCTION s d)	*EXTFN	Set the function definition of symbol s to d. Same as defc.
(TYPE-OF o)	*EXTFN	The name of the data type of object o. If o is a structure, the name of the structure is returned.

Binding variables

Symbols hold variable bindings. Variables bindings can be either global or bound locally. Local variables are bound when defined as formal parameters of functions or when locally introduced when a new code block is defined using let or other variable binding expressions. Both local and global variables can be (re)assigned using the macro setf.

The macro let defines a new code block with new variables. For example:

(let ((x 1)     ;; Local X initialized to 1
      Y         ;; Local Y initialized to nil
      (z 2))    ;; Local Z initialized to 2
  (list x y z)) ;; Body returns list of the local variables
====>
(1 NIL 2)

In SA Lisp global variables should be declared before they are used, by calling defvar, defparameter, defconstant or defglobal. SA Lisp gives warnings when setting undeclared global variables or using them in functions. There are a number of built-in global variables that store various system information and system objects.

For example:

(defglobal _g_)         ;; declare _G_ as global
====>
NIL

(setf _g_ 1)            ;; assign number 1 to _G_
====>
1

_g_;; evaluate _G_
====>
1

It is bad programming practice to rebind global variables in code blocks:

(defun fishy() (let ((_g_ 3))
                 _g_))            
====>
WARNING - Global variable used as local: _G_ in FISHY
FISHY                       ;; Warning if global variables re-bound in functions

(fishy)
====>
3                           ;; The local value of _g_ is returned

_g_
====>
1                           ;; Global value did not change

Unlike most other programming languages, global Lisp variables can also be dynamically scoped so that they are rebound when a code block is entered and automatically restored back to their old values when the code block is exited. In CommonLisp dynamically scoped variables are declared using the special form defvar or defparameter. Dynamically scoped variables provide a controlled way to handle global variables as they are restored in the same way as local variables when a code block is exited. As a convention, dynamically scoped variables have * as first character. For example, assume we have a package to do trigonometric computations using radians, degrees, or new degrees:

(defvar *angle-unit* 1)      ;; Units in radians to measure angles
====>
*ANGLE-UNIT*


(defun mysin(x)(sin (* x *angle-unit*)))
====>
MYSIN

(defun hl (ang x)(/ x (mysin ang)))
====>
HL

(hl 0.785398 10)             ;; Compute length of hypotenuse using radians
====>
14.1421


(setf *angle-unit* (/ 3.1415926 180)) ;; Let’s use degrees instead
====>
0.0174533

(hl 45 10)
====>
14.1421

(setf *angle-unit* (/ 3.1415926 200)) ;; Let’s use new degrees instead
====>
0.015708

(hl 50 10)
====>
14.1421

Now suppose we want to have a special version of HL that computes the hypotenuse for regular degrees:

(defun hyplen (ang x)
  (let ((*angle-unit* (/ 3.1415926 180))) ;; Rebind *angle-unit* inside HL
    (hl ang x)))
====>
HYPLEN

(hyplen 45 10)          ;; Using degrees inside HYPLEN
====>
14.1421

(hl 50 10)
====>
14.1421                 ;; Automatically restored back to new degrees outside HYPLEN

Variables declared constant cannot be changed. For example:

(defconstant _pi_ 3.414)
====>
PI

(setf _PI_ 3.415)
ERROR - Incorrect variable assignment: _PI_

The following system functions handle variable bindings.

Function	Type	Description
(BOUNDP var)	*EXTFN	Return T if the variable var is bound. Unlike CommonLisp, boundp works not only for special and global variables but also for local variables, allowing to test whether a variable is locally bound.
(CONSTANTP x)	*SUBR	Return T if x evaluates to itself.
(DEFCONSTANT var val [doc])	*SPECIAL	Bind symbol var permanently to a constant value val that cannot be changed, optionally having a documentation string doc.
(DEFGLOBAL var [val][doc])	MACRO	Declare var to be a global variable with optional initial value val and optional documentation string doc. Unlike dynamically scoped variables global variables are not temporarily reset locally with let or let*. They are much faster to look up than dynamically scoped variables. See also defvar.
(DEFPARAMETER var val [doc])	*SPECIAL	Declare var to be a special variable set to val with optional documentation string doc.
(DEFVAR var [val][doc])	*SPECIAL	Declare var to be a special variable optionally initialized to val set unless var is previously assigned, and an optional documentation stringdoc. Special variables are dynamically scoped. See also defglobal.
(DESTRUCTURING-BIND args l form…..)	*MACRO	Evaluate form... after binding the variables specified in lambda-list args to the corresponding values in the list resulting from the evaluation of l.
(GLOBAL-VARIABLE-P var)	LAMBDA	Return T if var is declared to be a global variable.
(LET ((var init)...) form...)	*MACRO	Bind local variables var to initial values init in parallel and evaluate the forms form.... Instead of a pair the binding can also be just a variable, which binds the variable to nil.
(LET* ((var init)...) form...)	*MACRO	As let but local variables are initialized in sequence.
(PROG-LET ((var init...)...) form...)	MACRO	As let but (return v) must be called in form... to return the value v from prog-let. Notice that the classical Lisp special forms prog and go are NOT implemented in SA Lisp.
(QUOTE x)	*SPECIAL	Return x unevaluated. Same as 'x.
(RESETVAR var val form...)	MACRO	Temporarily reset global value of var to val while evaluating form.... The value of the last evaluated form is returned. After the evaluation var is reset to its old global value. Usually special variables combined with let/let* are used instead.
(SET var val)	*EXTFN	Bind the variable bound to the value of var to val. This function is normally not used; the normal way to set variable values is to call the macro setf that does not evaluate its first argument.
(SETQ var val)	*SPECIAL	Change the value of the variable var to val. The macro setf is a generalization of setq for updating many different kinds of data structures rather than just setting variable values.
(PSETQ var_1 val_1 ….. var_k val_k)	*MACRO	Set in parallel variables var_1 to val_1, ... , var_k to val_k using setq.
(SPECIAL-VARIABLE-P v)	*EXTFN	Return T if the variable v is declared as special with defvar or defparameter.
(SYMBOL-VALUE s)	*EXTFN	Get the global value of the symbol s. Return the symbol NOBIND if no global value is assigned.

Basic Functions

This section describes the basic data types in SA Lisp along with the functions for operating on them.

Symbols

Lists

Lists (data type LIST) represent linked lists. There are many system functions for manipulating lists. Two classical Lisp functions are car to get the head of a list, and cdr to get the tail. For example:

(defglobal _xx_ '(a b c))
====>
_XX_

(car _xx_)
====>
A

(cdr _xx_)
====>
(B C)

Function	Type	Description
(ADJOIN x l)	*EXTFN	Similar to (cons x l) but does not add x if it is already a member of l (tests with equal).
(APPEND l...)	*MACRO	Make a copy of the concatenated lists l... With one argument, (append l) copies the top level elements of l.
(ASSOC x ali)	*EXTFN	Search association list ali for a pair (x . y). Tests with equal.
(ASSQ x ali)	EXTFN	As assoc but tests with eq.
(ATOM x)	*EXTFN	Return T if x is not a list or if it is nil.
(BUTLAST l)	*EXTFN	A copy of list l minus its last element.
(CAAAR x)	*EXTFN	Same as (car (car (car x))). Can be updated with setf.
(CAADR x)	*EXTFN	Same as (car (car (cdr x))). Can be updated with setf.
(CAAR x)	*EXTFN	Same as (car (car x)). Can be updated with setf.
(CADAR x)	*EXTFN	Same as (car (cdr (car x))). Can be updated with setf.
(CADDR x)	*EXTFN	Same as (car (cdr (cdr x))) or (third x). Can be updated with setf.
(CADR x)	*EXTFN	Same as (car (cdr x)) or (second x). Can be updated with setf.
(CAR x)	*EXTFN	The head of the list x, same as (first x). Can be updated with setf.
(CDAAR x)	*EXTFN	Same as (cdr (car (car x))). Can be updated with setf.
(CDADR x)	*EXTFN	Same as (cdr (car (cdr x))). Can be updated with setf.
(CDAR x)	*EXTFN	Same as (cdr (car x)). Can be updated with setf.
(CDDAR x)	*EXTFN	Same as (cdr (cdr (car x))). Can be updated with setf.
(CDDDDR x)	*LAMBDA	Same as (cdr (cdr (cdr (cdr x)))). Can be updated with setf.
(CDDDR x)	*EXTFN	Same as (cdr (cdr (cdr x))). Can be updated with setf.
(CDDR x)	*EXTFN	Same as (cdr (cdr x)). Can be updated with setf.
(CDR x)	*EXTFN	The tail of the list x, same as (rest x). Can be updated with setf.
(CONS x y)	*EXTFN	Construct new list cell.
(CONSP x)	*EXTFN	Test if x is a list cell.
(COPY-TREE l)	*EXTFN	Make a copy of all levels in list structure l. To copy the top level only, use (append l).
(EIGHTH l)	*LAMBDA	The 8th element in list l. Can be updated with setf.
(FIFTH l)	*LAMBDA	The 5th element in list l. Can be updated with setf.
(FIRST l)	*EXTFN	The first element in list l, same as (car l). Can be updated with setf.
(FIRSTN n l)	LAMBDA	A new list consisting of the first n elements in list l.
(FOURTH l)	*LAMBDA	Get fourth element in list l. Can be updated with setf.
(GETF l i)	*EXTFN	Get value stored under the property indicator l in the property list l. Can be updated with setf.
(IN x l)	EXTFN	Return T if there is some substructure in l that is eq to x.
(INTERSECTION x y)	*EXTFN	A list of the elements occurring in both the lists x and y. Tests with equal.
(INTERSECTIONL l)	LAMBDA	Make the intersection of the lists in list l. Tests with equal.
(LAST l)	*EXTFN	Return the last tail of the list l. E.g., (last ’(1 2 3)) => (3)
(LDIFF l tl)	*LAMBDA	Make copy of l up to, but not including, its tail tl.
(LENGTH x)	*EXTFN	Compute the number of elements in a list, the number of characters in a string, or the size of a vector.
(LIST x...)	*EXTFN	Make a list of the elements x...
(LIST* x...)	*EXTFN	Similar to list except that the last argument is used as the end of the list. For example: (list* 1 2 3) => (1 2 . 3) (list* 1 2 ’(a)) => (1 2 A)
(LISTP x)	*EXTFN	Return T if x is a list cell or nil.
(MEMBER x l)	*EXTFN	Tests with equal if element x is member of list l. Return the tail of l where x is found first. For example: (member 1.2 ’(1 1.2 1.2 3)) => (1.2 1.2 3)
(MEMQ x l)	EXTFN	As member but tests with eq instead of equal.
(MERGE lx ly fn)	*LAMBDA	Merge the two lists lx and ly with fn as comparison function. For example: (merge ’(1 3) ’(2 4) (function <)) => (1 2 3 4)
(MKLIST x)	EXTFN	Return x if it is nil or a list. Otherwise (list x) is returned.
(NATOM x)	*EXTFN	Return T if x is not an atom and not nil. Anything not being a list is an atom.
(NINTH l)	*LAMBDA	The 9th element in list l. Can be updated with setf.
(NOT x)	*EXTFN	Return T if x is nil, same as null.
(NTH n l)	*EXTFN	The nth element in list l with enumeration starting at 0. Can be updated with setf.
(NTHCDR n l)	*EXTFN	Get the nth tail of the list l with enumeration starting at 0.
(NULL x)	*EXTFN	Return T if x is nil, same as not.
(PAIR x y)	EXTFN	Same as pairlis.
(PAIRLIS x y)	*EXTFN	Form an association list by pairing the elements of the lists x and y.
(POP l)	*SPECIAL	Remove front of list l, same as (setf l (cdr l)).
(PUSH x l)	*MACRO	Add x to the front of list l, same as (setf l (cons x l)).
(REMOVE x l)	*EXTFN	Remove elements equal to x from list l.
(REMOVE-DUPLICATES l)	*EXTFN	Remove all duplicate elements in the list l. Tests with equal.
(REST l)	*LAMBDA	Same as cdr. Can be updated with setf (Sec. 3.4).
(REVERSE l)	*EXTFN	A list of the elements of l in reverse order.
(SECOND l)	*EXTFN	The 2nd element in list l. Same as cadr. Can be updated with setf.
(SET-DIFFERENCE x y [equalflag])	*EXTFN	A list of the elements in x that are not member of the list y. Tests with eq unless equalflag is true.
(SEVENTH l)	*LAMBDA	The 7th element of the list l. Can be updated with setf.
(SIXTH l)	*LAMBDA	The 6th element of the list l. Can be updated with setf.
(SORT l fn)	*LAMBDA	Sort the elements in the list l using fn as comparison function.
(SUBLIS ali l)	*EXTFN	Substitute elements in the list structure l as specified by the association list ali.

Destructive functions

The list functions introduced so far are constructing new lists out of other objects. For example, (APPEND x y) makes a new list by copying the list x and then concatenating the copied list with the list y. The old x is deallocated by the garbage collector when it is no longer needed. If x is long append will generate quite a lot of garbage. This is not very serious because SA Lisp has a very efficient real-time garbage collector that immediately discards no longer used objects. However, sometimes one needs to actually modify list structures by physically replacing pointers. One may also wish to do so for efficiency reasons as, after all, the generation of garbage has its cost. Another reason is that one wants to implement data structures that are updated destructively. For this Lisp has some destructive list manipulating functions that replace pointers rather than copying list cells. Notice that such destructive functions may cause bugs that can be difficult to find. Therefore destructive functions should be avoided if possible.

To make destructive operations more transparent, most destructive functions are in CommonLisp (and SA Lisp) replaced with the generic setf macro. setf calls a setter macro (mutator) to update data objects associated with a corresponding getter (accessor) that accesses the object. The setter macro will transparently update the getter value destructively. It is a generic mutator.

For example:

(setf a '(1 2 3 4))
====>
WARNING - Setting undeclared global variable: A
(1 2 3 4)

(setf (third a) 8) ;; Replace (third a) with 8
====>
8

a                  ;; The list bound to A has changed
====>
(1 2 8 4)

The macro (setf $p_1$ $v_1$ $p_2$ $v_2$ ...) updates the value of each getter $p_i$ to become EQ to $v_i$. That is, after executing setf all $p_i=v_i$. A getter is an expression accessing data. It can be a variable in which case setf sets the variables $p_i$ like multiple calls to setq, e.g.: (setf x 1 y 2). It can also be a call to a getter function (accessor) that has a corresponding setter macro defined, e.g. (setf (third a) 8). In this case there is a setter macro associated with the getter function third. There are predefined setter macros defined for the getters aref (arrays), getl (property lists), getprop (symbol property lists), gethash (hash tables), get-btree (B-trees), and car…cdddr, first…tenth, rest, nth (lists). When structures are defined (Sec. 3.15) the system will generate getter functions and setter macros to access and update their fields.

The user can extend the update rules for setf by defining new setter macros. If a getter function (fn ...) is called for accessing an updatable location a corresponding setter macro sm can be associated with fn by calling (setfmethod fn sm).

The following destructive system list functions are supported:

Function	Type	Description
(ATTACH x l)	EXTFN	Similar to (cons x l) but is destructive, i.e. the head of the list l is modified so that all pointers to l will point to the extended list after the attachment. Notice that this does not work if `l is not a list, in which case attach is not destructive and creates a new list cell just like cons.
(DELETE x l)	*EXTFN	Remove destructively the elements in the list l that are eq to x. The result is the updated l. Notice that if x is the only remaining element in l the operation is not destructive and nil is returned.
(DMERGE x y fn)	LAMBDA	Merge lists x and y destructively with fn as comparison function. For example: (dmerge ’(1 3 5) ’(2 4 6) #’<) => (1 2 3 4 5 6). The value is the merged list; the merged lists are destroyed. See also merge.
(LCONC hdr l)	SUBR	Add elements in list l to list header hdr. The concatenated list in maintained in (car hdr) as for function tconc.
(NCONC l...)	*MACRO	Destructive concatenate the lists l... (i.e. a destructive append) and return the concatenated list. This classical Lisp function is known to be error prone. It can also be slow when l is long. If possible, use lconc instead.
(NCONC1 l x)	EXTFN	Add x to the end of l destructively, i.e. same as (nconc l (list x)). This function is known to be error prone. It can also be slow when l is long. If possible, use tconc instead.
(NREVERSEi l)	*EXTFN	Destructively reverse the list l. The value is the reversed list. The original list l will be destroyed. Very fast destructive reverse.
(PUTF l i v)	EXTFN	Set the value of the property indicator i in the property list l to v. If possible, use (setf (getf l i) v) instead.
(RPLACA l x)	*EXTFN	Destructively replace the head of list l with x. If possible, use (setf (car l) x) instead.
(RPLACD l x)	*EXTFN	Destructively replace the tail of list l with x. If possible, use (setf (cdr l) x) instead.
(SETF p1 v1 p2 v2 ...)	*EXTFN	General mutator macro for destructively updating variable bindings or data accesses, as explained above. Called generalized variables in CommonLisp terminology [1][2].
(SETFMETHOD access-fn setf-macro)	LAMBDA	Define setter macro for a getter function.
(TCONC hdr x)	EXTFN	Efficient tail concatenation of elements at the end of a list. First a list header hdr is created by calling (tconc) without arguments. Then a successive new element x is concatenated to the tail of hdr each time tconc is called. The concatenated list is maintained in (car hdr). See also lconc.

Strings

Strings (data type STRING) represent text strings of arbitrary length. Strings are always immutable, i.e. they cannot be destructively modified. Notice that, unlike CommonLisp, C etc., there is no special data type for single characters, which are represented as single character strings.

Strings containing the characters " and \ must precede these with the escape character, \. Some special characters are also prefixed with \: \a (alert, beep), \b (backspace), \f (form feed, new page), \n (new line), \r (carriage return, restart line), and \t (tabulation).

UTF-8 is used for representing characters in strings.

Examples of strings:

(setf a "This is a string")
====>
WARNING - Setting undeclared global variable: A
"This is a string"

(setf b "String with string delimiter \" and the escape character \\")
====>
WARNING - Setting undeclared global variable: B
"String with string delimiter \" and the escape character \\"

(concat a " " b)
====>
"This is a string String with string delimiter \" and the escape character \\"

SA Lisp provides the functions (string-like str pat) and (string-like-i str pat)to match the string str against the regular expression pat. In the regular expression pat the character * matches any sequence of characters (zero or more), ? matches a single character, [set] matches any character in the specified set of characters, and [!set] or [^set] matches any character not in the specified set.

The following built-in functions operate on strings:

Function	Type	Description
(CHAR-INT str)	*EXTFN	The UTF-8 integer encoding the first character in string str.
(CONCAT str...)	EXTFN	Coerce the arguments str... to strings and concatenate them.
(EXPLODE str)	EXTFN	A list of strings representing the characters in str.
(INT-CHAR i)	*EXTFN	The single character string encoded as UTF-8 integer i. nil is returned if there is no such character.
(LENGTH str)	*EXTFN	The number of characters in string str.
(MKSTRING x)	EXTFN	Convert object x to a string.
(STRING-CAPITALIZE str)	*EXTFN	Capitalize string str.
(STRING-DOWNCASE str)	*EXTFN	Upper case string str.
(STRING-UPCASE str)	*EXTFN	Lower case string str.
(STRING< s1 s2)	*EXTFN	Return T if the string s1 alphabetically precedes s2.
(STRING= s1 s2)	*EXTFN	Return T if the strings s1 and s2 are the same. They will also be equal.
(STRING-LEFT-TRIM ch str)	*EXTFN	Remove the initial characters in str that also occur in ch.
(STRING-LIKE str pat)	EXTFN	Return T if pat matches string str.
(STRING-LIKE-I str pat)	EXTFN	Case insensitive string-like.
(STRING-POS str x)	EXTFN	The character position of the first occurrence of the string x in str. The character positions are enumerated from 0 and up.
(STRING-RIGHT-TRIM ch str)	*EXTFN	Remove the trailing characters in str that also occur in ch.
(STRING-TRIM ch str)	*EXTFN	Remove those initial and trailing characters in str also occurring in ch.
(STRINGP x)	*EXTFN	Return T if x is a string.
(SUBSTRING p1 p2 str)	EXTFN	The substring in str starting at character position p1 and ending in p2. The character positions are enumerated from 0 and up.

Numbers

Numbers represent numeric values. Numeric values can be either long integers (data type INTEGER) or double precision floating point numbers (data type REAL). Integers are entered to the Lisp reader as an optional sign followed by a sequence of digits, e.g.

1234 -1234 +123

Examples of legal floating point numbers:

1.1 1.0 1. -1. -2.1 +2.3 1.2E3 1.e4 -1.2e-20

The following system functions operate on numbers.

Function	Type	Description
(+ x...)	*EXTFN	Add the numbers x...
(- x y)	*EXTFN	Subtract y from x.
(1+ x)	*MACRO	Add one to number x.
(1- x)	*MACRO	Subtract one from number x.
(* x...)	*EXTFN	Multiply the numbers x...
(/ x y)	*EXTFN	Divide x with y.
(ACOS x)	*EXTFN	Arc cosine of x.
(ASIN x)	*EXTFN	Arc sine of x.
(ATAN x)	*EXTFN	Arc tangent of x.
(CEILING x)	*EXTFN	The smallest integer larger than or equal to x.
(COS x)	*EXTFN	Cosine of x.
(DECF x)	*MACRO	Decrement the variable x with delta, default 1.
(EXP x)	*EXTFN	Natural exponent $e^x$.
(EXPT x y)	*EXTFN	Exponent $x^y$.
(FLOOR x)	*EXTFN	The largest integer less than or equal to x.
(FRAND low high)	EXTFN	A floating-point random number in interval [low, high).
(INCF x [delta])	*MACRO	Increment the variable x with delta, default 1.
(INTEGERP x)	*EXTFN	Return T if x is an integer.
(LOG x)	*EXTFN	The natural logarithm of $log_e(x)$.
(MAX x...)	*EXTFN	Return the largest of the numbers x...
(MIN x...)	*EXTFN	Return the smallest of the numbers x...
(MINUS x)	EXTFN	Negate the number x. Same as (- x).
(MINUSP x)	*LAMBDA	Return T if x is a number less than 0.
(MOD x y)	*EXTFN	The remainder when dividing x with y. x and y can be both integers or floating-point numbers.
(NUMBERP x)	*EXTFN	Return T if x is a number.
(PLUSP x)	*LAMBDA	Return T if x is larger than 0.
(RANDOM n)	*EXTFN	A random integer in interval [0, n).
(RANDOMINIT n)	EXTFN	Generate a new seed for the random number generator.
(ROUND x)	*EXTFN	Round x to the closest integer.
(SQRT x)	*EXTFN	The square root of x.
(SIN x)	*EXTFN	Sine of x.
(TAN x)	*EXTFN	Tangent of x.
(ZEROP x)	*LAMBDA	Return T if x is equal to 0.

Logical Functions

In CommonLisp nil is regarded as false and any other value as true. The global constant T is usually used for representing true. For example:

(setf x t)   ;; regarded as true
WARNING - Setting undeclared global variable: X
====>
T

(setf y nil) ;; regarded as false
====>
WARNING - Setting undeclared global variable: Y

(setf z 1)   ;; regarded as true
====>
WARNING - Setting undeclared global variable: Z
1

(or x y z)
====>
T            ;; X = T is the first true value

(and x y z)
====>
NIL          ;; Y is nil

(or z x y)
1            ;; Z = 1 is the first true value

(not y)
====>
T            ;; Y is nil

(not z)
====>
NIL          ;; Z is 1 (i.e. true)

The following functions return or operate on logical values.

Function	Type	Description
(< x y)	*EXTFN	Return T if the number x is strictly less than y.
(<= x y)	*EXTFN	Return T if the number x is less than or equal to y.
(= x y)	*EXTFN	Return T if the numbers x and y are equal.
(> x y)	*EXTFN	Return T if the number x is strictly greater than y.
(>= x y)	*EXTFN	Return T if the number x is greater than or equal to y.
(AND x...)	*SPECIAL	Evaluate the forms x... and return nil when the first form evaluated to nil is encountered. If no form evaluates to nil, the value of the last form is returned.
(COMPARE x y)	EXTFN	Compare the order of two objects. Return 0 if they are equal, -1 if less, and 1 if greater. All kinds of objects can be compared.
(EQ x y)	*EXTFN	Return T if x and y are the same objects, i.e., having the same address in memory.
(EQUAL x y)	*EXTFN	Return T if objects x and y are equivalent. Notice that, in difference to CommonLisp, arrays are equal if all their elements are equal, and equality can be defined for user-defined data types in C as well.
(EVENP x)	*LAMBDA	Return T if x is an even number.
(NEQ x y)	*EXTFN	Same as (not (eq x y)).
(ODDP x)	*LAMBDA	Return T if x is an odd number.
(OR x...)	*SPECIAL	Evaluate the forms x... until some form does not evaluate to nil. Return the value of that form.
(NOT x)	*EXTFN	Return T if x is nil; same as null.

Arrays

Arrays (data type ARRAY) in SA Lisp represent one-dimensional sequences. The elements of an array can be of any type. Arrays are printed using the notation #(e1 e2 ...). For example:

(setf a #(1 2 3))
====>
WARNING - Setting undeclared global variable: A
#(1 2 3)

Arrays are allocated with the macro (make-array size). For example:

(make-array 3)
====>
#(NIL NIL NIL)

Notice that SA Lisp supports only one-dimensional arrays (vectors) while CommonLisp allows arrays of any dimensionality.

Adjustable arrays are arrays that can be dynamically increased in size. They are allocated with the call:

(make-array size :adjustable t)

Arrays can be enlarged with the call:

(adjust-array array newsize)

Enlargement of adjustable arrays is incremental and does not copy the original array. Non-adjustable arrays can be enlarged as well, but the enlarged array may or may not be a copy of the original one, depending on its size. In other words, you have to rebind non-adjustable arrays after you enlarge them.

For example:

(setf a (make-array 3))
====>
WARNING - Setting undeclared global variable: A
#(NIL NIL NIL)

(adjust-array a 6)
====>
#(NIL NIL NIL NIL NIL NIL)

a
====>
#(NIL NIL NIL)     ;; a was not adjustable

(setf a (make-array 3 :adjustable t))
====>
WARNING - Setting undeclared global variable: A
#(NIL NIL NIL)

(adjust-array a 6)
====>
#(NIL NIL NIL NIL NIL NIL)

a
====>
#(NIL NIL NIL NIL NIL NIL)

Function	Type	Description
(ADJUST-ARRAY a newsize)	*EXTFN	Increase the size of the array a to newsize. If the array is declared to be adjustable at allocation time it is adjusted in-place, otherwise, an array copy may or may not be returned.
(ADJUSTABLE-ARRAY-p a)	*EXTFN	Return T if a is an adjustable array.
(AREF a i)	*MACRO	Access element i of the array a. Enumeration starts at 0. Unlike CommonLisp, only one-dimensional arrays are supported. Can be updated with setf.
(ARRAY-TOTAL-SIZE a)	*EXTFN	The number of elements in the one-dimensional array a, same as (length a).
(ARRAYP x)	*EXTFN	Return T if x is an array (fixed or adjustable).
(ARRAYTOLIST a)	EXTFN	Convert array a to a list.
(CONCATVECTOR x y)	LAMBDA	Concatenate arrays x and y.
(COPY-ARRAY a)	*EXTFN	Make a copy of array a.
(ELT a i)	EXTFN	Same as (aref a i). Can be updated with setf.
(LISTTOARRAY l)	EXTFN	Convert list l to a non-adjustable array.
(LENGTH v)	*EXTFN	The number of elements in vector v.
(MAKE-ARRAY size [:INITIAL-ELEMENT v] [:ADJUSTABLE aflag])	*MACRO	Allocate a one-dimensional array of pointers with size elements. :INITIAL-ELEMENT specifies optional initial element values v. If :ADJUSTABLE is true, an adjustable array is created; the default is a non-adjustable array.
(PUSH-VECTOR a x)	EXTFN	Add x to the end of array a by adjusting it.
(SETA a i v)	EXTFN	Set element i in array a to v. Return v. If possible, use (setf (aref a i) v) instead.
(VECTOR x...)	*EXTFN	Make an array with elements x...

Memory areas

The data type MEMORY represents references to binary memory areas (1-D byte arrays) stored in main memory outside the database image. This is used for storing buffers and other binary data structures. The memory areas are pinned in memory meaning that their contents is not moved by the system. Memory areas not referenced from other objects will be automatically freed by the garbage collector. Memory area objects are saved as other Lisp objects when rollout is called. read and print can handle memory area objects and they can be shipped between peers.

The following SA Lisp function handle memory areas:

Function	Type	Description
(MALLOC sz)	EXTFN	Allocate a new memory area having sz bytes. The memory area is automatically deallocated by the garbage collector when no longer needed.
(READ-FILE-RAW f)	LAMBDA	Make a memory area object of the contents of file f.
(REALLOC m sz)	EXTFN	Increase the size of memory area object m to sz.
(WRITE-FILE f m)	EXTFN	Write memory area m as the contents of file f.

Hash Tables

Hash tables (data type HASHTAB) are unordered dynamic tables that associate values with SA Lisp objects as keys. Hash tables are allocated with:

(make-hash-table)

Notice that, unlike standard CommonLisp, no initial size is given when hash tables are allocated. Instead the system will automatically and incrementally grow (or shrink) hash tables as they evolve.

Elements of a hash table are accessed with:

(gethash key hashtab)

Elements of hash tables are updated with

(setf (gethash key hashtab) new-value)

Iteration over all elements in a hash table is made with:

(maphash (function (lambda (key val) ...)) hashtab)

Notice that comparisons of hash table keys in CommonLisp is by default using eq and not equal. Thus, e.g., two strings with the same contents do not match as hash table keys unless they are pointers to the same string. Normally eq comparisons are useful only when the keys are symbols. To specify a hash table comparing keys with equal (e.g. for numeric keys or strings) use

(make-hash-table :test (function equal))

Example:

(setf ht1 (make-hash-table))
====>
WARNING - Setting undeclared global variable: HT1
#[O HASHTAB 233568 10 2]

(setf (gethash "hello" ht1) "world")
====>
"world"

(gethash "hello" ht1)
====>
NIL

(setf ht2 (make-hash-table :test (function equal)))
====>
WARNING - Setting undeclared global variable: HT2
#[O HASHTAB 233600 10 2]

(setf (gethash "hello" ht2) "world")
====>
"world"

(gethash "hello" ht2)
====>
"world"

The following system functions operate on hash tables:

Function	Type	Description
(CLRHASH ht)	EXTFN	Clear all entries from hash-table ht and return the emptied table.
(GETHASH k ht)	*EXTFN	Get value with key k in hash table ht. Can be updated with setf.
(HASH-BUCKET-FIRSTVAL ht)	EXTFN	The value for the first key stored in hash table ht. What is the first key is undefined and depends on the internal hash function used.
(HASH-BUCKETS ht)	EXTFN	The number of hash buckets in hash table ht.
(HASH-TABLE-COUNT ht)	*EXTFN	The number of elements stored in hash table ht.
(MAKE-HASH-TABLE [:SIZE s] [:TEST eqfn])	*MACRO	Allocate a new hash table. The CommonLisp parameter :SIZE is ignored as the hash tables in SA Lisp are dynamic and scalable. The keyword parameter :TEST specifies the function to be used for testing equivalence of hash keys. :TEST can be (function eq) (default) or (function equal).
(MAPHASH fn ht v)	*EXTFN	Apply (fn key value v) on each pair of key and value in hash table ht.
(PUTHASH k ht v)	EXTFN	Set the value stored in hash table ht under the key k to v. If possible, use (setf (gethash k ht) v) instead.
(REMHASH k ht)	EXTFN	Remove the value stored in hash table ht under the key k.
(SXHASH x)	*EXTFN	Compute a hash code for object x as a non-negative integer. (equal x y) => (= (sxhash x) (sxhash y)).

Main memory B-trees

Main memory B-trees are ordered dynamic tables that associate values with SA Lisp objects as keys. The interfaces to B-trees are very similar to those of hash tables. The main difference between B-trees and hash tables are that B-trees are ordered by the keys and that there are efficient tree search algorithms for finding all keys in a given interval. B-trees are slower than hash tables for equality searches.

B-trees are allocated with:

(make-btree)

Elements of a B-tree are accessed with:

(get-btree key btree)

setf is used for modifying accessed B-tree elements.

For example:

(setf bt (make-btree))
====>
#[MEXI: MBTREE ]   ;; This indicates that B-trees are implemented as a plug-in

(setf (get-btree "hello" bt) "world")
====>
"world"

(get-btree "hello" bt)
====>
"world"

System functions operating on main memory B-trees:

Function	Type	Description
(GET-BTREE k bt)	EXTFN	Get value associated with key k in B-tree bt. The comparison uses function compare. Can be updated with setf.
(MAKE-BTREE)	EXTFN	Allocate a new B-tree.
(MAP-BTREE bt lower upper fn)	EXTFN	Apply Lisp function (fn key val) on each key-value pair in B-tree bt whose key is larger than or equal to lower and less than or equal to upper. If any of lower or upper is the symbol * it means that the interval is open in that end. If both lower and upper are * the entire B-tree is scanned.
(PUT-BTREE k bt v)	EXTFN	Set the value stored in the B-tree bt under the key k to v. If possible, use (setf (get-btree k bt) v) instead. If v is nil the element is deleted.

Time

Time points are represented in SA Lisp by the data type TIMEVAL. It represents UTC time points as number of microseconds since EPOC (midnight GMT 1970-01-01). A TIMEVAL object has two components, sec and usec, representing seconds since EPOC and micro seconds beyond the second, respectively. A TIMEVAL object is printed as #[T sec usec], e.g. #[T 1569255397 470000] . Time differences are usually represented as seconds represented as floating point numbers.

The following Lisp functions operate on time points:

Function	Type	Description
(CLOCK)	EXTFN	Compute the number of wall clock seconds spent so far during the run as a floating point number.
(DAYLIGHT-SAVINGP)	EXTFN	Return T if daylight saving time is active.
(GETTIMEOFDAY)	EXTFN	The TIMEVAL object representing the present wall time.
(GET-TIME-OFFSET)	EXTFN	The current time adjustment relative to the OS clock in seconds as a floating point number.
(LOCAL-TIME [tval])	EXTFN	The location-dependent local UTC time string representing the TIMEVAL object tval. Current local wall time if tval omitted.
(MKTIMEVAL sec usec)	EXTFN	Create a new TIMEVAL object.
(PARSE-UTC-TIME str)	EXTFN	Convert a UTC time string str into a TIMEVAL object.
(RNOW)	EXTFN	The number of seconds since EPOC as a floating-point number.
(SET-TIMER fn period)	EXTFN	Starts a timer function, calling Lisp function fn regularly. The real number period specifies the interval in seconds between calls.
(SLEEP sec)	EXTFN	Makes the system sleep for sec seconds as a floating point number. Can be interrupted with CTRL-C.
(timevalp tval)	EXTFN	True if tval is a TIMEVAL object.
(SET-TIME-OFFSET sec)	EXTFN	Adjust the current system time sec seconds relative to system clock.
(STOP-TIMER tid)	EXTFN	Stop the timer with identifier tid.
(TIMEVAL-SEC tval)	EXTFN	The number of seconds since EPOC for a TIMEVAL object tval.
(TIMEVAL-USEC tval)	EXTFN	The number of microseconds after the timeval-sec part of a TIMEVAL object tval.
(TIMEVAL-SHIFT tval sec)	EXTFN	Construct a new TIMEVAL object by adding sec seconds to tval.
(TIMEVAL-SPAN tv1 tv2)	EXTFN	The difference in seconds between TIMEVAL tv2 and tv1.
(UTC-OFFSET)	EXTFN	The offset in seconds from UTC in the computer's setting, including time zone and daylight saving time.
(UTC-TIME [tval])	EXTFN	The location-independent UTC time string of TIMEVAL object tval. Return the current UTC wall time if tval omitted.

Structures

SA Lisp includes a subset of the structure definition package of CommonLisp. Structures are implemented in SA Lisp using fixed size arrays. You are recommended to use structures instead of lists when defining data structures because of their more efficient and compact representation.

A new structure s is defined with the macro defstruct, for example:

(defstruct person name address)
====>
PERSON

The macro (defstruct s field...) defines a new structure s with fields specified by field.... It generates a number of macros and functions to create and update instances of the new structure. New instances of a new structure named s are created with the generated macro:

(make-s :field1 value1 :field2 value2 ...)

For example:

(setf p (make-person :name "Tore" :address "Uppsala"))
====>
#(PERSON "Tore" "Uppsala")

The fields of a structure are updated and accessed using accessor functions generated for each field s. The field f of the structure s bound to the object o is accessed with (s-f o). For example:

(person-name p)
====>
"Tore"

A field f for a structure s bound to an object o is updated to a new value val by calling setf for the accessor function, (setf (s-f o) val). For example:

(setf (person-name p) "Kalle")
====>
"Kalle"

(person-name p)
====>
"Kalle"

An object o can be tested to be a structure of type s using the accessor function (s-p o)

For example:

(person-p p)
====>
T

A struct is also regarded as a Lisp type, so the Lisp function (type-of o) returns the name of a structure if o is bound to a structure object. For example:

(type-of p)
====>
PERSON

Control Structures

This section describes system functions, macros, and special forms used for implementing syntactic sugar and control structures in SA Lisp.

Compound expressions

The functions progn, prog1, and prog2 are used for forming a single form out of several forms. This makes sense only if some of the forms have side effects. For example:

(progn (print "A") "B")
====>
"A"
"B"    ;; Value of progn is value of last argument

(prog1 (print "A") "B")
====>
"A"
"A"    ;; Value of prog1 is value of first argument

Compound expressions can also be formed by lambda and let expressions described in Binding Variables.

Function	Type	Description
(PROG1 x...)	*EXTFN	The value of the first form in x...
(PROG2 x...)	*LAMBDA	The value of the second form in x...
(PROGN x...)	*SPECIAL	The value of the last form in x...

Conditional expressions

Conditional expressions are special forms that evaluate expressions conditionally depending on the truth value of some form. The following conditional statements are available in SA Lisp:

*SPECIAL cond

(cond (test form...)...) 

The classical Lisp conditional expression is cond. For example:

(setf x 1
      y 2
      z nil)
====>
WARNING - Setting undeclared global variable: X
WARNING - Setting undeclared global variable: Y
WARNING - Setting undeclared global variable: Z
NIL

(cond (x)
      (t y))
====>
1

(cond (z (print "NO"))
      (y (print "YES") 5)
      (t (print "NO")))
====>
"YES"
5

*SPECIAL if

(if pred then else) 

Evaluate the form pred. If it evaluates to true the then form is evaluated, otherwise evaluate else.

*MACRO case

(case test (cases form...)(cases form...) ...(otherwise default...)

For example:

(case (+ 1 2)  
      (1 'HEY)  
      ((2 3) 'HALLO)  
      (otherwise 'DEFAULT))
====>   
HALLO

The test form is evaluated and successively compared with the symbols in each cases expressions until a match is found. Then the corresponding form... are evaluated, and the last one is returned as the value of case. Atomic cases expressions match if they are eq to the value of test, while lists match if the value of test is member of the list. If no cases list matches, the default... forms are evaluated and the value of the last one returned as the value of case. If no otherwise clause is present the default result is nil.

*SPECIAL selectq

This is Interlisp's variant of case.

(selectq test (cases form...)... default)

(selectq (+ 1 2)  
         (1 'hey)  
         ((2 3) 'hallo)  
         'default)  
====>
hallo  

Same as

(case test (cases form...)... (otherwise default))

*MACRO unless

(unless test form...)

Evaluate form... if test is nil and return value of last form.

*MACRO when

(when test form...)

Evaluate form... if test is true and return value of last form.

Iterative statements

As in other programming languages Lisp provides iterative control structures, normally as macros. However, in most cases map functions provide the same functionality in a cleaner and often more general way.

Function	Type	Description
(DO inits (endtest form...) form...)	*MACRO	General CommonLisp iterative control structure. The loop can be terminated with (return val) in addition to when the endtest form is true.
(DO* inits (endtest form...) form...)	*MACRO	Similar to do but the initializations inits are done in sequence rather than in parallel.
(DOLIST (x l) form...)	*MACRO	Evaluate form... for each element x in list l. Same as (mapc #(lambda (x) form...) l).
(DOTIMES (i n [res]) form...)	*MACRO	Evaluate form... n times varying i from 0 to n-1. The optional form res returns the result of the iteration. In res, the variable i is bound to the number of iterations made.
(LOOP form...)	*MACRO	Evaluate form... repeatedly. The loop can be terminated with the result val returned by calling (return val).
(RETURN [val])	*LAMBDA	Return value val from the block in which return is called. A block can be a prog-let, prog-let*, dolist, dotimes, do, do*, loop, or while expressions.
(RPTQ n form)	SPECIAL	Evaluate form n times. Recommended for performance measurements in combination with the time macro.
(WHILE test form...)	MACRO	Evaluate the forms form... while test is true or until return is called.

Non-local returns

Non-local returns allows to bypass the regular function application order. The classical Lisp forms for this are catch and throw. The special form (catch tag form) evaluates tag to a catcher, which must be a symbol. Then form is evaluated and, if thereby the function (throw tag value) is called somewhere with the same catcher, the value is returned. If throw is not called the value of form is returned from catch. For example:

(defun foo (x)(catch 'foo-catch (fie (+ 1 x))))
====>
FOO

(defun fie (y)(cond ((= y 2)(throw 'foo-catch -1))

                    (t y)))
====>
FIE

(foo 1)
====>
-1

(foo 2)
====>
3

A related subject is how to catch errors. In particular the special form unwind-protect is the general mechanism to handle any kind of non-local return and error trapping in SA Lisp.

Function	Type	Description
(catch tag form)	*SPECIAL	Catch calls to throw inside form matching tag.
(throw tag val)	*EXTFN	Return val as the value of a previous call to catch with the same catcher tag, having called throw directly or indirectly.

Hooks

Hooks are lists of Lisp forms executed at particular states of the system. For example, there is an initialization hook evaluating forms just after the system has been initialized, and a shutdown hook evaluating forms when the system is terminated.

To register a form to be executed just after the database image has been read call:

(register-init-form form [where])

The Lisp expression form is inserted into a list of forms stored in the global variable after-rollin-forms. The forms are evaluated by the system just after a database image has been read from the disk. If where is the symbol FIRST the form is added in front of the list; otherwise it is added to the end. For example:

(register-init-form '(formatl t "Welcome!" t))
====>
OK

To register a form to be evaluated when the system is exited call:

(register-shutdown-form form [where])

In this case, the Lisp expression form is evaluated just before the system is to be exited by calling (quit). The shutdown hook will not be executed if (exit) or (quit-now) is called. The global variable shutdown-forms contains a list of the shutdown hook forms. For example:

(register-shutdown-form '(formatl t "Goodbye!" t))
====>
OK

The hooks are saved in the database image. For example, given that we have registered to above two hooks we can do the following:

(rollout "myimage.dmp")    ;; Save the database image in a file
====>
T

(quit)
====>
Goodbye!                   ;; The shutdown hook is evaluated.

At some later occasion, after SA Lisp is started with the OS command sa.engine -q lisp myimage.dmp, the following messages will be printed in the console window:

Database image: myimage.dmp
Welcome!
Release 5, v4, 32 bits
Lisp >

Second Order Functions

Variables bound to functions or even entire expression can be invoked or evaluated by the system. Functional arguments (higher order functions) provide a very powerful abstraction mechanism that can replace many control structures in conventional programming languages. Map functions are examples of elaborate use of functional arguments.

The simplest case for functional arguments is when a function is passed as arguments to some other function. For example, assume we want to make a function (sum2 x y fn) that calls the function fn with arguments x and y and then adds together the two results (i.e. sum2 = fn(x) + fn(y)):

(defun sum2 (x y fn)
  (+ (funcall fn x)(funcall fn y))) ;; The system function FUNCALL calls FN
====>
SUM2

(sum2 1 2 (function sqrt))          ;; sqrt(1) + sqrt(2)
====>
2.41421

In CommonLisp, the system function funcall must be used to call a function bound to a functional argument. Also notice that the special form function should be used when passing a functional argument, to be explained next.

Closures

In previous example the special form function is used when passing functional arguments.

Notice that quote should not be used when passing functional arguments. The reason is that otherwise the system does not know that the argument is a function. This matters particularly if the functional argument is a lambda expression. Consider a function to compute $(x + y)^n$ using sum2:

(defun sumpow (x y n)
  (sum2 x y (function         ; FUNCTION **must** be used here
             (lambda (z)      ; lambda expression = anonymous function
               (expt z n))))) ; N is free variable in the lambda expression
====>
SUMPOW

(sumpow 1 2 2)
====>
5

Free lambda expressions as this one are useful when passing free variables like n into a functional argument. Now, let’s see what happens if quote was used instead of function:

(defun sumpow (x y n)
  (sum2 x y '(lambda (z)                ; Should use #' instead
               (expt z n)))))
====>
WARNING - Redefined Lisp function: SUMPOW
WARNING - Suspicious use of QUOTE rather than FUNCTION: (QUOTE (LAMBDA (Z) (EXPT Z N))) in SUMPOW
SUMPOW

(sumpow 1 2 2)
====>
ERROR - Unbound variable: N

As you can see, the system first warns that quote is used instead of function. Then the variable n becomes unbound when sumpow is called. The reason is that quote returns its argument unchanged while function makes a closure of its argument if it is a lambda expression. A closure is a data type that holds a function (lambda expression) together with the local variables bound where it is called. In our example, the local variable n is bound by the closure when sum2 is called inside sumpow.

Map functions

Map functions are functions and macros taking other functions as arguments and applying them repeatedly on elements in lists and other data structures. Map functions provide a general and clean way to iterate over data structures in a functional programming style. They are often a good alternative to the more conventional iterative statements. They are also a preferred alternative to recursive functions as they don’t eat stack as recursive functions do.

The classical Lisp map function is mapcar. It applies a function on every element of a list and returns a new list formed by the values of the applications. For example:

(mapcar (function 1+) '(1 2 3))
====>
(2 3 4)

The function mapc is similar, but does not return any value. It is useful when the applied function has side effects. For example:

(mapc (function print) '(1 2 3))
====>
1
2
3
NIL     ;; mapc always returns nil

In CommonLisp the basic map functions may take more than one argument to allow parallel iteration of several lists. For example:

(mapcar (function +)
        '(1 2 3) 
        '(10 20 30))
====>
(11 22 33)

Lambda expressions are often useful when iterating using map functions. For example:

(defun rev2 (a b)
  (let (ra rb)
    (mapc #'(lambda (x y)
              (push x ra)
              (push y rb))
          a b)
    (list ra rb)))
====>
REV2

(rev2 '(1 2 3) '(a b c))
====>
((3 2 1) (C B A))

The following system map functions are available in SA Lisp:

Function	Type	Description
(MEMBER-IF fn l)	*EXTFN	The function fn is applied to each element in list l. If fn returns true for some element in l, then member-if will return the corresponding tail of l.
(MAPC fn l...)	*MACRO	Apply fn to each of the elements of the lists l... in parallel. Returns nil.
(MAPCAN fn l...)	*MACRO	Apply fn to each of the elements of the lists l... in parallel and nconc together the results.
(MAPCAR fn l...)	*MACRO	Apply fn to each of the elements of the lists l... in parallel and build a list of the results.
(MAPL fn l...)	*MACRO	Apply fn to each tail of the lists l.... Returns nil.
(EVERY fn l...)	*MACRO	Return T if fn returns true when applied to every element in the lists l... in parallel.
(NOTANY fn l...)	*MACRO	Return T if fn applied to each element in the lists l... in parallel never returns true.
(REDUCE fn l)	*LAMBDA	Aggregate the values in l by applying the binary function fn pairwise between the elements in l. If l has a single element it is returned without applying fn. nil is returned if l is nil.
(REMOVE-IF-NOT fn l)	*EXTFN	The subset of the list l for which the function fn returns true.
(REMOVE-IF fn l)	*EXTFN	The subset of the list l for which the function fn returns nil.
(SOME fn l...)	*MACRO	True if fn applied to each element in the lists l... in parallel returns true for some application.

Variadic function calls

The macro funcall does not work if we don’t know before run time the number of arguments of the function to call. In particular funcall cannot be used if we want to call a variadic function like + (plus). What we need is a way to construct a dynamic argument list before we call a function. For this the system macro apply is used. For example, assume we define a function (combinel x y fn) that applies fn on the elements of lists x and y and combines the results also using fn:

(defun combinel (x y fn)
  (funcall fn
           (apply fn x)
           (apply fn y)))
====>
COMBINEL

(combinel '(1 2 3) '(4 5 6) (function +))
====>
21

In this case we have to construct the arguments as a list to the inner function applications, and therefore apply has to be used in the definition of combinel. This is also correct, but you get a warning:

(defun combinel (x y fn)
  (apply fn
         (list (apply fn x)
               (apply fn y))))
====>
WARNING - Redefined Lisp function: COMBINEL 
WARNING - Use (funcall fn x ...) instead of (apply fn (list x ...)): (APPLY FN (LIST (APPLY FN X) (APPLY FN Y))) in COMBINEL
COMBINEL

(combinel '(1 2 3) '(4 5 6) (function +))
====>
21

Dynamic evaluation

The most general way to execute dynamic expressions in Lisp is to call the system function eval. It takes as argument any Lisp form (i.e. expression) and evaluates it. For example:

(defglobal _a_ 1)
====>
_A_

(eval (list _a_))  ;; This fails because we are trying to evaluate the form (1)
====>
ERROR - Undefined function: 1

(eval '(list _a_))       ;; This succeeds
====>
(1)

Lisp >(list _a_)         ;; This gives the same result since the Lisp REPL calls eval
====>
(1)

The function eval is actually very seldom used. It is useful when writing Lisp programming utilities, like e.g. the Lisp REPL, the Lisp debugger, or remote evaluation.

Notice that you should avoid using eval unless you really need it, as the code executed by eval is not known until run-time and this is unpredictable, unsafe and prohibits compilation and program analysis. If possible, use funcall or apply instead. In most other cases macros replace the need for eval while at the same time producing compilable and analyzable programs.

The following system functions apply functions on arguments and do dynamic evaluation:

Function	Type	Description
(APPLY fn argl)	*MACRO	Apply the function fn on the list of arguments argl. The macro apply is used to call a function where the argument list is dynamically constructed with varying arity, a variadic function call.
(APPLY fn arg arg...)	*MACRO	The macro apply can have more than two arguments arg arg…... In this case, the dynamic argument list is constructed by prepending the first forms of arg arg... with the last one. That is, the call is the same as (apply fn (list* arg1…..argk)).
(APPLYARRAY fn a)	EXTFN	Apply the Lisp function fn on the arguments in the array a.
(EVAL form)	*EXTFN	Evaluate form. Notice that unlike CommonLisp, the form is evaluated in the lexical environment of the eval call.
(F/L fn args form...)	MACRO	(f/l (x) form...) <=> (function(lambda(x) form...)). This is equivalent to the CommonLisp read macro (also supported): #’(lambda (x) form...).
(FUNCALL fn arg...)	*MACRO	Call function fn with arguments arg... The macro funcall is used when the called function fn is not known until run-time (e.g., bound to a variable).
(FUNCTION fn)	*SPECIAL	Make a closure of the function fn.

Macros

Lisp macros provide a very powerful mechanism to extend Lisp with new control structures and syntactic sugar. Because programs are represented as data in Lisp it is particularly simple to make Lisp programs that transform other Lisp programs. Macros provide the hook to make such code transforming programs available as first class objects. A macro should be seen as a rewrite rule that takes a Lisp expression as argument and produces another equivalent Lisp expression as result. For example, assume we want to define a new control structure for to implement for loops, where e.g. (for i 2 10 (print i)) prints the natural numbers from 2 to 10. The macro for can be defined as:

(defmacro for (var from to do)
  (subpair '(_var _from _to _do) ; _var, _from, to, and _do are substituted
           (list var from to do) ; with these actual values
           '(let ((_var _from))  ; This is the code skeleton
              (while (<= _var _to)
                _do
                (incf _var)))))
====>
FOR

(for i 2 4 (print i))            ;; Macros are expanded once by interpreter
====>
2
3
4
NIL                              ;; Value of FOR

When defining macros as in the example there is usually a code skeleton in which one substitutes elements with actual argument expressions. In the example we use subpair to do the substitution. A more convenient CommonLisp facility to define code skeletons is to use back quote (`), which is a variant of quote where subexpressions can be marked for evaluation. By using backquote, for can been written as:

(defmacro for (var from to do)
  `(let ((,var ,from))
     (while (<= ,var ,to)
       ,do  
       (incf ,var))))
====>
FOR

(for i 2 4 (print i))
====>
2
3
4
NIL

Prefixing a form x with the backquote character, `x, marks the form x to be back quoted. The form x is substituted with a new expression by recognizing the symbols , (comma) and ,@ (comma at sign) in x as special markers. A ,form is replaced with the value of the evaluation of form. In ,@form the form is evaluated and 'spliced’ into the list where it appears. For example:

(setf a `(1 2 3)
      b `(3 4 5))
====>
WARNING - Setting undeclared global variable: A
WARNING - Setting undeclared global variable: B
(3 4 5)

`(a (b ,a ,@b)) 
====>
(a (b (1 2 3) 3 4 5))

Macros can be debugged like any other Lisp code. In particular it might be interesting to find out how a macro transform a given call. For this the system functions macroexpand and macroexpand-all can be used, normally in combination with pretty-printing the macro expanded code with ppv. The function (macroexpand form) expands the top level macro of the form while (macroexpand-all form) expands all macros in the form.

For example:

(macroexpand '(for i 2 4 (print i))) ; macroexpand expands the top level only
====>
((LAMBDA (I) (WHILE (<= I 4) (PRINT I) (INCF I))) 2)

(ppv (macroexpand '(for i 2 4 (print i)))) ; ppv makes readable printing of code
====>
((LAMBDA (I)
  (WHILE
     (<= I 4)
     (PRINT I)
     (INCF I)))
 2)
NIL

(ppv (macroexpand-all '(for i 2 4 (print i)))) ; macroexpand-all expands all levels
====>
((LAMBDA (I)
  (CATCH 'PROG-RETURN
     (INT-WHILE
        (<= I 4)
        (PRINT I)
        (SETQ I
           (+ I 1)))))
 2)
NIL

Macros can be used for defining functions whose arguments are always quoted. One such function is (pp fn1...fnk) that pretty-prints function definitions. For example:

(pp pp ppv)
====>
(DEFMACRO PP (&REST FNS)
  "Pretty prints function definitions"
  (LIST 'PPF
     (LIST 'QUOTE FNS)))
(DEFUN PPV (X &OPTIONAL STREAM)
  "Prettyprint X and return NIL"
  (PPS X STREAM T)
  NIL)
(PP PPV)

(ppv (macroexpand '(pp pp ppv)))
====>
(PPF '(PP PPV))   ;; The function ppf does the pretty-printing

Macros are very efficient in SA Lisp because the first time the interpreter encounters a macro call it will modify the code and replace the original form with the macro-expanded one (just-in-time expansion). Thus a macro is normally evaluated only once.

Notice that macros normally should not have side effects, since they are evaluated only once by the interpreter and thereby replaced with the expanded code! They should be side effect free Lisp code that transforms one piece of code to another equivalent piece of code.

The following functions are useful when defining macros:

Function	Type	Description
(ANDIFY l)	LAMBDA	Make an and form of the forms in l.
(DEFMACRO name args form...)	*SPECIAL	Define a new macro analogous to lambda functions with defun.
(KWOTE x)	EXTFN	Make x a quoted form. For example, (kwote t) => T, (kwote 1) => 1, (kwote 'a) => (QUOTE A), (kwote '(+ 1 2)) => (QUOTE (+ 1 2)).
(KWOTED x)	EXTFN	True if x is a quoted form. For example: (kwoted 1) => T, (kwoted '(quote (1))) => T, (kwoted '(1)) => nil. Same as constantp.
(MACRO-FUNCTION fn)	*EXTFN	The function definition of fn if it is a macro; otherwise nil.
(MACROEXPAND form)	*EXTFN	If form is a macro, return what it rewrites form into; otherwise form is returned unchanged.
(MACROEXPAND-ALL form)	LAMBDA	Macroexpand form and all sub forms in it.
(PROGNIFY forms)	LAMBDA	Make a single form from a list of forms forms.

Input and Output

The I/O system is based on various kinds of byte streams. A byte stream is a data type with certain attributes that allows its instances to be supplied as argument to the basic Lisp I/O functions, such as print and read. Examples of byte streams are: i) file streams (type STREAM) for terminal/file I/O, ii) text streams (type TEXTSTREAM) for reading and writing into text buffers, and iii) socket streams (type SOCKET) for communicating with other SA Engine/SA Lisp systems. The storage manager allows the programmer to define new kinds of byte stream. A byte stream argument nil or T represents standard input or standard output (i.e. the console).

Byte streams normally have functions providing the following operations:

• Open a new byte stream, e.g. (openstream "foo.txt" "rw") creates a new file stream to the file named foo.txt for both reading and writing.
• Print objects to byte stream str. For example, (print x str) prints x to byte stream str open for writing. The object x is marshaled as an S-expression.
• Read bytes from byte stream str. For example, (read str) reads one S-expression from a byte stream open for reading. Notice that print and read are compatible so that a copy of a form written with print will be recreated by read.
• Send the contents of byte stream str to its destination by calling (flush str). This will empty the buffer associated with most kinds of byte streams.
• Close byte stream str by calling (closestream str).

The following functions work on any kind of byte streams, including standard input or output:

Function	Type	Description
(CLOSESTREAM str)	EXTFN	Close byte stream str.
(DRIBBLE [FILE])	*LAMBDA	Log both standard input and output to file. Stop logging and close the file by calling (dribble).
(FORMATL str form...)	LAMBDA	Simple replacement for some functionality of CommonLisp's format function. Prints values of form... into byte stream str. A T in form...marks a new line and "~pp" indicates that the next form is pretty-printed.`
(FRESH-LINE [str])	*EXTFN	Print a line feed into str unless the output position is just after a new line.
(PP fn...)	MACRO	Pretty-print the definitions of the unquoted functions fn... on standard output.
(PPV s [str])	LAMBDA	Pretty-print s and return nil.
(PRIN1 s [str])	*EXTFN	Print object s into byte stream str with escape characters and string delimiters inserted when needed.
(PRINC s [str])	*EXTFN	Print object s into byte stream str without escape characters and string delimiters.
(PRINC-CHARCODE n [str])	EXTFN	Print UTF-8 character number n into byte stream str.
(PRINT s [str])	*EXTFN	Prints object s into byte stream str so it can be read back to produce an object equal to s.
(PRINTL l...)	LAMBDA	Print objects l... as a list on standard output.
(READ [str])	*EXTFN	Read S-expression from byte stream str. If str is a string, it reads an expression from the string.
(READ-BYTES n [str])	EXTFN	Read n bytes from byte stream str as a string.
(READ-CHARCODE [str])	EXTFN	Read one UTF-8 character from stream str and return its UTF-8 integer representation.
(READ-LINE [str eolchar])	*EXTFN	Read UTF-8 characters up to just before the next end-of-line character as a string from str.
(READ-TOKEN str [delims] [brks] [strnum] [nostrings])	EXTFN	Read the next token from byte stream str. delims is a string of delimiters that separate tokens and are skipped. Default delimiters are " \t\n\r". The string brks are break characters that are tokens by themselves. Default break characters are "()[]\"';,". If strnum is nil the system will parse numbers; otherwise they are returned as strings. Analogously the function will parse strings if nostrings is true.
(SPACES n [str])	LAMBDA	Print n spaces into byte stream str.
(TERPRI [str])	*EXTFN	Print a line feed into byte stream str.
(TEXTUAL-STREAMP str)	EXTFN	Return T if the byte stream str is open in textual mode.
(TYPE-READER tpe fn)	EXTFN	Define Lisp function (fn tpe args stream) to be a type reader for objects printed as #[tpe x...].
(UNREAD-CHARCODE c str)	EXTFN	Put character c back into byte stream str.
(Y-OR-N-P prompt)	*LAMBDA	Prompt user for y/n or yes/no on standard input.

File I/O

File streams are used for writing to and reading to/from files. Their type name is STREAM. Standard output and standard input are regarded as file streams represented as nil.

A new file stream is opened with:

(openstream filename mode)

where mode can be (combinations of) "r" for reading, "w" for writing, or "a" for appending. Furthermore, the option "b" indicates that the byte stream accepts writing or reading of binary data. For example:

(setf s (openstream "foo.txt" "w"))
====>
WARNING - Setting undeclared global variable: S
#[O STREAM 114670 8 3]

(print '(hello world 1) s)
====>
(HELLO WORLD 1)

(closestream s)
====>
#[O STREAM 114670 8 3]

(setf s (openstream "foo.txt" "r"))
====>
WARNING - Setting undeclared global variable: S
#[O STREAM 114680 8 3]

(read s)
====>
(HELLO WORLD 1)

(closestream s)
====>
#[O STREAM 114680 8 3]

The following system functions and variables handle file I/O and file streams:

Function	Type	Description
(DELETE-FILE nm)	*EXTFN	Delete the file named nm. Return T if successful.
(FILE-EXISTS-P nm)	*EXTFN	Return T if file named nm exists.
(FILE-LENGTH nm)	*EXTFN	The number of bytes in the file named nm.
(LOAD NM)	*EXTFN	Evaluate the forms in the file named nm.
(OPENSTREAM nm mode)	EXTFN	Open a file stream against a file named nm. mode is some Unix file mode(s) i.e., "r", "w", or "a".
(REDIRECT-BASIC-STDOUT nm)	EXTFN	Redirect all standard output to file named nm. Useful for logging in a file when standard output is disabled.
(WITH-OPEN-FILE (str nm [:direction d]) form...)	*MACRO	Open the file stream str for reading, writing, or appending of a file named nm, then evaluate the forms form..., and finally close the stream. The file is opened for reading if d is :input (default), for writing if d is :output, and for appending if d is :append.

Text streams

Text streams (data type TEXTSTREAM) allow the I/O functions to work against dynamically expanded main memory buffers instead of files. This provides an efficient way to manipulate large strings. Each text stream internally stores its data as a MEMORY object.

The following functions provide capabilities for creating and manipulating text streams, essential for handling text data in memory efficiently in Lisp.

Function	Type	Description
(MAKETEXTSTREAM [sz binary])	EXTFN	Create a new text stream with an optional initial buffer size sz. Text streams are textual by default and can be made binary by setting binary to true.
(MAKETEXTSTREAM mem [binary])	EXTFN	Create a new text stream using the memory area mem as buffer. The stream can be made binary if binary is true.
(TEXTSTREAMBUFFER tstr [trim])	LAMBDA	Retrieve the entire internal memory area buffer of text stream tstr. If trim is true, the text stream buffer is trimmed up to the current cursor position.
(TEXTSTREAMPOS tstr)	EXTFN	Get the current cursor position in text stream tstr.
(TEXTSTREAMPOS tstr pos)	EXTFN	Move the cursor to position pos in text stream tstr.
(TEXTSTREAMSTRING tstr)	EXTFN	Retrieve the text stream buffer of text stream tstr as a string. This function requires the text stream to be non-binary.
(CLOSESTREAM tstr)	EXTFN	Reset the cursor of text stream tstr to position 0, equivalent to (textstreampos tstr 0).
(WITH-TEXTSTREAM tstr str form...)	MACRO	Open a text stream tstr for the string str, evaluate the forms form... with tstr open, and then close tstr. The value is the evaluation of the last S-expression in form.... For example: (with-textstream s “(a)(b)” (read s)(read s)) => (B)

Sockets

SA Lisp systems can communicate via TCP sockets. Essentially socket streams are represented as conventional I/O streams where the usual SA Lisp I/O functions work. The Lisp functions print and read are used for sending forms between SA Lisp systems using sockets.

Point to point communication

With point-to-point communication two SA Lisp servers can communicate via sockets by establishing direct TCP/IP socket connections.

The first thing to do is to identify the TCP host on which an SA Lisp system is running by calling: (gethostname).

Server side:

The first step on the server (receiving) side is to open a socket listening for incoming connections. Two calls are needed on the server side:

A new socket object must be first be created, which is going to accept client port registrations. This is done with:

(open-socket nil portno)

For example:

(setf sock (open-socket nil 1235))
====>
WARNING - Setting undeclared global variable: SOCK
#[socket NIL 1235 NIL 500]

open-socket returns a new socket object that will listen on TCP port number portno. If portno is 0 it means that the OS assigns a free port for incoming messages. The port number of socket sock can be obtained with the function :

(socket-portno sock)
====>
1235

Then the server must then wait for some client to request connections by calling (accept-socket sock [timeout]). For example:

(accept-socket sock 1)
====>
NIL

The function accept-socket waits for an open-socket call from some client to establish a new connection. If timeout is omitted the waiting is forever (it can be interrupted with CTRL-C), otherwise it specifies a time-out in seconds. If an incoming connection request is received, accept-socket returns a new socket stream to use for communication with the client that issued the open-socket request. accept-socket returns nil if no open-socket request was received within the time-out period.

Client side:

On the client side a call to:

(open-socket host portno [timeout])

The function open-socket opens a socket stream to the server that is listening on port number portno on host hostname. The parameter hostname can be a logical host name or an IP address, but must not be nil (which would indicate creating a server socket). The result of open-socket is an object of type SOCKET, which is a byte stream. Thus, once open-socket is called the regular Lisp I/O functions can be used for communication.

Notice that since socket stream are buffered, data is not sent on a socket stream s before calling the function (flush s).

To check whether there is something to read in an array of sockets sa with an optional timeout call

(poll-socket sa [timeout][writesock])

The function poll-sockets returns vector of sockets with pending data if something has arrived on any of the sockets in sa within timeout seconds, and nil otherwise. If writesock is nil the polling is for reading; otherwise it is for writing. Polling can be interrupted with CTRL-C.

When a client has finished using a socket s it can be closed and deallocated with:

(close-socket s)

The garbage collector automatically calls close-socket when a socket object is deallocated, but you are advised to do this explicitly to guarantee that the socket is closed.

Notice that pending data in the socket stream may be lost when close-socket is called.

You can associate an object val as property prop of socket s by calling :

(socket-put s prop val)

The property prop of socket s is accessed by:

(socket-get s prop)

Socket migration

A TCP socket opened by a process on a machine (computer, VM, container, …) may be transferred (migrated) to another process on the same machine. Both processes must be running before migration, as the PID of the receiving process must be known by the sending process. Furthermore, a TCP socket between the sending and receiving process is required for the transfer.

In the following example, a TCP socket s on the sending process is migrated to the receiving process. The PID of the receiving process is receiving-pid. The TCP connection between the sending process and the receiving process is called

To migrate a socket s from a sending process to a receiving process (with PID receiving-pid), the sending process first prepares a socket mig for migration:

(setf mig (make-migration-socket s receiving-pid))

This adds an attribute sockinfo-os to the property list of the socket, where os represent socket properties specific to the current OS, e.g. SOCKINFO-WINDOWS. It will contains OS dependent information necessary to migrate the socket from the sending process to the receiving process. Next, the sending process packages the socket information and sends it to the receiving process over the TCP connection to the receiving process socket-to-receiver by calling

(pf (export-socket-minimal mig) socket-to-receiver)

The receiving process receives this information over the TCP connection to the sending process socket-to-sender, and establishes a new socket, by the call:

(setf new-socket (import-socket-minimal (read socket-to-sender)))

Now the receiving process can use new-socket as if new-socket was opened by the receiving process. The sending process should not use its socket s (or mig) any more after migration. Note that no buffer content of socket s is transferred during socket migration. Any content in the read buffer of s in the sending process will not be present in the read buffer of new-socket in the receiving process.

Socket migration is currently available on Windows, Mac OS X, and Linux. Note that Linux and Mac OS X utilize unix domain sockets for transferring socket information. (Unix can only use a domain socket for migration of TCP sockets between processes.) This unix domain socket is opened when calling make-migration-socket. The life time of this domain socket is one minute. Thus, the receiving process must call import-socket-minimal within one minute after the sending process called make-migration-socket. Furthermore, only one socket can be migrated to a receiving process at any given time.

Remote evaluation

There is a higher level remote evaluation mechanism in SA Engine servers (not in basic SA Lisp), where the system can be set up as a server evaluating incoming Lisp forms from other SA Engine peers. With remote evaluation, Lisp forms are sent from one SA Engine/SA Lisp peer to another for evaluation there after which the result is shipped back to the caller. The remote evaluation requires the receiving peer to listen for incoming forms to be evaluated.

Server side:

On the server side the following makes an SA Engine server (SAS) named SRV behave as a remote evaluation server, accepting incoming forms to evaluate remotely.

First let’s start an SA Engine name server named SRV in some shell. A name server is a SAS that keeps track of what SA Engine peers listen to what ports for remote evaluation and can also be used as a regular SAS. To start a name server named SRV (always upper case), execute the shell command:

sa.engine -N srv

When the name server SRV is up and running it starts listening on incoming remote evaluations by default on port number 35021. You can specify another listening port number p by suffixing the SAS name with :p, e.g. the following command makes name server named S listen on port 1234:

sa.engine -N s:1234

Client side:

Start a stand-alone SA Lisp system in another shell with:

sa.engine -q lisp

Now we can open a socket connection _c_ to the SAS named SRV from the client by calling the function (open-socket-to 'srv). For example:

(defglobal _c_ nil)
(setf _c_ (open-socket-to 'srv))

To ship an S-expression form from the client to the SA Engine server listening on connection c for remote evaluation, simply call:

(socket-eval form c)

The function socket-eval ships a Lisp form to the connected server for evaluation. For example, the following form prints the string “Hey” on the standard output of the name server SRV and ships back the result “Hey Joe”:

(socket-eval '(concat (print "Hey") " Joe") _c_)

Errors occurring on server are shipped back to client, for example try this:

(socket-eval '(/ 3 0) _c_)

The function socket-eval is synchronous and blocks until the result is received. For non-blocking messages use instead:

(socket-send form c)

The difference to socket-eval is that form is evaluated on the server on its own; the client does not wait for the result and is thus non-blocking. Errors are not sent back. The function socket-send is much faster than socket-eval since it does not wait for the result to return. If you want to synchronize after many non-blocking messages sent with socket-send, end with a call to socket-eval. For example, the following form will return the number 10000:

(progn (socket-send '(defglobal _cnt_ 0) _c_)
       (dotimes (i 10000)(socket-send '(incf _cnt_) _c_))
       (socket-eval '_cnt_ _c_))

Error handling

When the system detects an error it will call the Lisp function:

(faulteval errno msg o form frame)

where

ERRNO is an error number ( -1 for dynamic errors)

MSG is an error message

O is the failing Lisp object

FORM is the latest Lisp form evaluated when the error was detected.

FRAME is the variable stack frame where the error occurred.

The SA Lisp behavior of faulteval when debugging is disabled first prints the error message and then calls the function (reset) to signal an error to the system, called an error signal. To reset Lisp means to jump to a pre-specified reset point of the system. By default this reset point is the top level read-eval-print loop. It can also be an unwind protection to be explained next.

Trapping exceptions

The special form unwind-protect is the basic system primitive for trapping all kinds of exceptions.

(unwind-protect form [cleanup])

The form is evaluated until it is terminated, whether naturally or by means of an error exception. The cleanup form is then always evaluated before control is handed back to the system. Note that the cleanup form is not protected by the current unwind-protect, so errors produced during evaluation of cleanup will be thrown out from the unwind-protect call. In such cases, errors in cleanup can be caught in some other unwind-protect above the present one.

The special form unwind-protect traps any local or non-local exit, including error exceptions and throw. For example, a call to throw may cause a catcher to be exited leaving a file open. This is clearly undesirable, so a mechanism is needed to close an open file and do any other essential cleaning up on termination of a construct, no matter how or when the termination is caused, which is the purpose of unwind-protect.

Errors raised during the evaluation of a form can also be caught by using the macro:

(catch-error form repair)

It evaluates form and, if successful, returns the result of the evaluation. In case an error exception happened while evaluating form an error condition is returned from catch-error. Such an error condition is a list that contains a numeric identifier for the exception errno, an error message errmsg, and the object that caused the exception to be raised:

(:ERRCOND (errno errmsg o))

For example:

(catch-error a)
====>
(:ERRCOND (1 "Unbound variable" A))

The optional cleanup form in (catch-error form cleanup) is evaluated if an error occurred during the evaluation of form. In cleanup form the global variable _error-condition_ holds the error condition.

The function (error? ec) tests if ec is an error condition. It can be used for testing if catch-error returned an error condition. For a given error condition ec, the function all (errcond-arg ec)returns the object causing the error, (errcond-number ec) returns the error number, and (errcond-msg ec) returns the error message.

Raising errors

The function (error msg x) prints error message msg for object x and raises an error.

To cause an error exception without any error message, an error bang, call (reset).

As any other error these functions will go through the regular error management mechanisms, so user errors can be caught with unwind-protect or catch-error.

User interrupts

After an interrupt is generated by CTRL-C the system calls the Lisp function

(catchinterrupt)

By default catchinterrupt resets SA Lisp. In debug mode a break loop is entered when CTRL-C is typed.

For disable (i.e. postpone) CTRL-C during evaluation of a form, use:

(douninterrupted form)

Error management functions

Below follows short descriptions of system functions and variables for error management.

Function	Type	Description
(CATCH-ERROR form [cleanup])	MACRO	Trap and repair errors. Execute form and run cleanup if an error occurs.
(CATCHINTERRUPT)	LAMBDA	Called whenever the user hits CTRL-C. Different actions are taken depending on the system state.
(DOUNITERRUPTED form)	MACRO	Delay interrupts happening during the evaluation of form until douniterrupted is exited.
(ERRCOND-ARG ec)	LAMBDA	Get the argument of error condition ec.
(ERRCOND-MSG ec)	LAMBDA	Get the error message of error condition ec.
(ERRCOND-NUMBER ec)	LAMBDA	Get the error number of error condition ec.
(ERROR msg x)	EXTFN	Print message msg followed by ': ' and object x, then generate an error.
(ERRORMESSAGE no)	EXTFN	The error message for error number no.
(ERRORNUMBER msg)	EXTFN	The error number for error message msg. Return -1 if the message is not in the system's error table.
(ERROR-AT msg x fn)	LAMBDA	Raise error in the context where the function fn was called.
(ERROR? x)	LAMBDA	True if x is an error condition.
(FAULTEVAL errno errmsg x form env)	LAMBDA	Called by the system when an error is detected. Enters a break loop in debug mode or prints the error message and calls (reset) otherwise.
(RESET)	EXTFN	Signals an unspecified exception. Return control to the latest reset point, executing all cleanup forms on the way.
(UNWIND-PROTECT form cleanup)	*SPECIAL	Basic system primitive to trap exceptions. Executes form and ensures cleanup is run regardless of whether form completes normally or an exception occurs.

Lisp Debugging

This section documents the debugging and profiling facilities of SA Lisp.

To enable run time debugging of SA Lisp programs the system should be put in debug mode. This is automatically done when entering the Lisp REPL. To enable Lisp debugging also in the OSQL REPL call (debugging t) in Lisp or debugging(true) in OSQL. To disable debugging, call (debugging nil) in Lisp or debugging(false) in OSQL. In debug mode the system checks assertions at run time and analyzes Lisp function definitions for semantic errors, and thus runs slightly slower. Also, in debug mode the system will enter a break loop when an error occurs instead of resetting Lisp, as described next.

The interactive break loop for debugging is difficult or even impossible if you are using the system in a batch environment or an environment where an interactive break loop cannot be entered (e.g. in a web server). For debugging in batch environments set the global variable _batch_ to true: `

(setf _batch_ t)`

When _batch_ is set and the system is in debug mode errors are trapped and cause a backtrace to be printed printed on standard output, after which the error is thrown without entering the break loop.

The break loop

The break loop is a Lisp read-eval-print loop (REPL) where some special debugging commands are available. This happens either when

• The user has explicitly specified break points for debugging specific broken functions.
• Explicit break points are introduced in the code by calling help.
• An error exception happens when debug mode is activated. The debug mode is activated by default in the Lisp REPL.

The system maintains variable binding stacks that hold associations between Lisp variables and their values. If multiple Lisp threads are active, there is one variable binding stack per thread. Every time a Lisp function is called it pushes a frame on a variable binding stack. A frame contains information about the called function, the actual arguments is the call, and in what context it is called. The debugger allows inspecting and changing the frames on the stack. The focused frame defines the current function call in the debugger. The function where the break loop is entered is called the broken function. If an error is trapped the broken function is called ERROR-HANDLER:

For example:

Lisp >(defun foo (x) (fie x))      ; Define FOO in top level Lisp REPL
====>
FOO
Lisp >(defun fie (y) x)                 ; Define buggy FIE
====>
WARNING - Undeclared free variable X in FIE. ;; Warning that we ignore
FIE
Lisp >(foo 1)
====>
ERROR - Unbound variable: X    ; Run time error.
Broken call to ERROR-HANDLER   ; The break loop is entered from the error handler
Inside FIE brk>:bt             ; Make a backtrace
====>
FIE
FOO
**BOTTOM**

Since the error happened inside FIE, let's make the call to FIE our focused frame:

Broken call to ERROR-HANDLER
Inside FIE brk>(:f fie)        ; find call to FIE on stack
At FIE brk>                    ; the fcused frame is now the call to FIE

We can inspect the focused frame with the command :fr:

Broken call to ERROR-HANDLER
At FIE brk>:fr                 ; Inspect the focused frame
6  Y <-> 1
5 --- FIE --- @ 0

Forms are evaluated in the lexical environment of the focused frame. For example, the variables bound in the focused frame can be inspected and changed in the break loop:

At FIE brk> y
====>
1
At FIE brk>(setf y 3)          ;; Change the value of variable Y
====>
3
At FIE brk>:fr
====>
6  Y <-> 3
5 --- FIE --- @ 0

We can correct the bug in the focused frame and evaluate the updated FIE:

At FIE brk>(defun fie (y) (1+ y))
====>
WARNING - Redefined Lisp function: FIE
FIE
At FIE >:eval
====>
FIE re-applied
At FIE brk>!value        ;; Get the value of the evaluation of (FIE 3)
====>
4

The command :a returns the control to the original broken function, which was ERROR-HANDLER.

FIE re-applied
At FIE brk>:a 
====>
Broken call to ERROR-HANDLER
Inside FIE brk>:help  ;; List the breakloop commands:
:help

You are in the break loop. Commands:

:help    print this text
:fr      print variables bound in focused frame  (same as ?=)
:fp      print file position and documentation of function at focused frame
:lvars   names of local variables bound at focused frame
(:arg N) get N:th argument in focused frame
:bt      print a backtrace of Lisp function calls below focused frame
:bto     print a backtrace of SLOG operators below focused frame
:btv     print a detailed backtrace of the frames below the focused frame
:btv*    print detailed backtrace including unevaluated and SUBR arguments
:eval    evaluate focused frame
:eval!   unbreak broken function while evaluating its body
!value   variable bound to value of the evaluation of focused frame
:a       change frame frame to previous one or reset to Lisp top loop
:r       reset to Lisp top loop
:c       continue from broken function
(return X) return value X from broken function
:nx      set focused frame one step up the stack
:pr      set focused frame one step down the stack
(:f FN)  set focused frame to first frame down the stack calling FN
:ub      unbreak the function at focused frame
(:b VAR) enter new breakloop whenever VAR becomes bound
:su      print number of objects allocated while evaluating body of focused frame

otherwise evaluate form in environment of current frame

Broken call to ERROR-HANDLER
Inside FIE brk>:r                ; Back to top REPL
=====>
NIL
Lisp >

The following break commands are available in the break loop:

:help Print summary of available debugging commands.

:fr Print the variables bound at the focused frame and their values.

:lvars Print names of local variables bound at the focused frame, without showing their values as :fr.

:fp Print file position of the function at the focused frame. The file position describes file and line number of the function if it has been defined in a file.

:ub Unbreak the function at the focused frame.

:bt Print a backtrace of functions below the focused frame on the variable binding stack. The depth of the backtrace is controlled by the special variable *backtrace-depth* that tells how many function frames should be printed. Its default is 500.

:btv Print a detailed backtrace of the frames below the focused frame.

:btv* Print a long backtrace including all stack contents.

:eval Evaluate the function call at the focused frame.

!value After an evaluation with the :eval command, the Lisp variable !value is bound to the value of evaluation. Its value is the symbol !UNEVALUATED if :eval has not yet been called.

(return x) Return value x from the broken function of the break loop.

:c Continue evaluation from the broken function of the break loop. The !value is directly returned from the break if :eval has been called in the broken function beforehand, which is indicated with the message (function EVALUATED).

:r Reset the system to the top level Lisp REPL.

:a Change focused frame to the frame of the broken function or reset if there is no previous broken function.

(:f fn) Set focused frame to first frame down the stack calling fn.

:nx Set new focused frame one step up the stack.

:pr Set new focused frame one step down the stack.

:su Evaluate body at focused frame and print list on how much storage was used during the evaluation. The value from the evaluation is discarded.

(:arg n) Function that returns the n:th argument in the focused frame.

(:b var) Enter new break loop when var becomes bound.

Explicit break points

It is possible to explicitly insert a break loop in a Lisp function by calling the macro:

(help [tag])

When help is called, the break loop is entered to allow the user to investigate the environment with the usual break commands. The local variables in the environment where help was called are also available. The optional tag is printed to identify the help call.

Breaking functions

Break points can be put around calls to Lisp functions fn... by the Lisp macro:

(break fn...)

When such a broken function is called the system will enter the break loop.

For example:

Lisp >(defun foo (x)(fie x))
====> 
FOO
Lisp >(defun fie (y)(1+ y))
====> 
FIE
Lisp >(foo 1)
====>
2
Lisp >(break foo fie)                       ; Put break points on FOO and FIE
====>
(FOO FIE)

Notice that the arguments of (BREAK fn...) are not evaluated.

Lisp >(foo t)      ;; This will cause an error break
====>
Broken call to FOO ;; In break loop of FOO
At FOO brk>:fr     ;; Inspect arguments of focused function call
====>
4  X <-> 1
3 --- FOO --- @ 0
Broken call to FOO
At FOO brk>:eval    ;; Evaluate the body of FOO
====>
Broken call to FIE  ;; The break loop of FIE entered by :EVAL
At FIE brk>:fr      ;; Inspect arguments of focused function call
====>
70  Y <-> 1
69 --- FIE --- @ 0
Broken call to FIE
At FIE brk>y        ;; Evaluate variable Y in scope of FIE
====>
1
Broken call to FIE
At FIE brk>:eval    ;; Evaluate body of FIE
====>
ERROR - Not a number: T
Broken call to ERROR-HANDLER
Inside FIE brk>:a   ;; Move back to FIE's entry break
====>
Broken call to FIE
At FIE brk>(setf y 2) ;; Change value of argument Y
====>
Broken call to FIE
At FIE brk>:eval      ;; Evaluate body of corrected FIE
====>
Broken call to FIE evaluated
At FIE brk>!value     ;; Inspect the result of the evaluation
====>
3
Broken call to FIE evaluated
At FIE brk>:c         ;; Continue evaluation 
====>
Leaving broken FIE    ;; Leaving break loop of FIE
Broken call to FOO evaluated ;; Back in break loop of FOO
At FOO brk>:c         ;; Continue evaluation
====>
Leaving broken FOO
3                     ;; Result 3 returned from FOO
Lisp >

If you break an external function (type EXTFN) the argument list is in the variable !args.

Breaks on macros mean testing how they are expanded.

The break points on functions fn... can be removed with:

(unbreak fn...)

For example:

(unbreak foo fie)

To remove all current function break points do:

(unbreak)

Conditional break points

The Lisp debugger also permits conditional break points where the break loop is entered only when certain conditions are fulfilled. A conditional break point on a function fn is specified by pairing fn with a precondition function, precode:

(break ... (fn precond) ...)

When fn is called the function precond is first called with the same parameters as fn. If precond returns nil no break loop is entered, otherwise it is.

For example:

(break (+ floatp))
(break (createtype (lambda (tp)(eq tp ’person))) )

In this case no break loop is entered by the call:

(+ 1 2 3)

By contrast, this call enters a break loop:

(+ 1.1 2 3)

Tracing functions

It is possible to trace Lisp functions fn... with the macro:

(trace fn...)

When such a traced function is called the system will print its arguments on entry and its result on exit. The tracing is indented to clarify nested calls.

Macros can also be traced or brokn to inspect that they expand correctly.

Remove function traces with:

(untrace fn...)

To remove all currently active traces do:

(untrace)

Analogous to conditional break points, conditional tracing is supported by specifying a function name fn in trace with a pair of functions (fn precond), for example:

(trace (+ floatp))
====>
(+)

(+ 1 2)
====>
3

(+ 1.1 2)
====>
--> + ( !ARGS=(1.1 2) )
<-- + = 3.1
3.1

(+ 1 2.1)
====>
3.1

Profiling

There are two ways to profile Lisp programs for identifying performance problems:

• The statistical profiler is the easiest way to find performance bottlenecks. It works by collecting statistics on what Lisp functions were executing at periodically sampled time points. It produces a ranking of the most commonly called Lisp functions. The statistical profiler has the advantage not to disturb the execution significantly, at the expense of not being completely exact.
• The function profiler is useful when one wants to measure how much wall time is spent inside a particular function. By the function profiler the user can dynamically wrap Lisp functions with code to collect statistics on how much time is spent inside particular functions. The function profiler is useful to exactly measure how much time is spent in specific functions. Notice that the wrapping makes the instrumented function run slower so the function profiler can slow down the system significantly if the profiled function does not use much time per call.

The Statistical Profiler

The statistical profiler is turned on by:

(start-profile)

After this the system will start a background timer process that regularly updates statistics on what function was executing at that time. After starting the statistical profiler you simply run the program you wish to profile.

When the statistics is collected, the percentage most called SA Lisp functions is printed with:

(profile)

You may collect more statistics to get better statistics by re-running the program and then call profile again.

Statistical profiling is turned off with:

(stop-profile)

The function stop-profile also clears the table of call statistics.

For example;

(start-profile)
====>
STAT-FUNCTION

(defun fib (x)
  (if (< x 2) 1 (+ (fib (- x 1))(fib (- x 2)))))
====>
FIB

(fib 30)
====>
1346269

(profile)
====>
(120 (FIB . 99.1) (DEFUN . 0.8))

(stop-profile)

The function profile returns a list where the first element is the number of samples and the rest lists the percentage spent in each function. profile takes as argument an optional cut-off percentage. For example:

(profile 1)
====>
(120 (FIB . 99.1))

The sampling frequency is controlled with the global variable _profiler-frequency_. It is by default set to 0.005 meaning that up to 200 samples are made per second. In practice the actual number of samples can be smaller.

The sampling is also influenced by the value of the global variable _exclude-profile_ containing a list of functions excluded from sampling. The sampler registers the first call on the execution stack not in this list. For advanced profiling it is sometimes useful to exclude commonly called uninteresting functions by adding more functions to _exclude-profile_.

The Function Profiler

To collect statistics on how much real time is spent in specific SA Lisp functions fn... and how many times they are called use the function profiler:

(profile-functions fn...)

For example:

(profile-functions subset getfunction)

The calling statistics for the profiled functions are printed (optionally into a file) with:

(print-function-profiles [file])

The calling statistics are cleared with:

(clear-function-profiles)

Function profiling can be removed from specific functions fn... with:

(unprofile-functions fn...)

To remove all function profiles do:

(unprofile-functions)

Analogously to conditional break points, conditional function profiling can be specified by pairs (fn precond) as arguments to profile-functions, e.g.

(profile-functions (createtype (lambda(x)(eq x 'person))) )

The function profiler does not double measure recursive functions calls. When a function call causes an error exception it is not measured.

System functions for debugging

We conclude this chapter with a list of all Lisp system functions useful for debugging:

Function	Type	Description
(ALLOCSTAT flag)	LAMBDA	Toggle to print new Lisp object allocations and deallocations per evaluated form in the Lisp REPL when flag is true.
(ALLOCCNT tp form)	EXTFN	Count how many new objects of Lisp type tp were allocated when evaluating form. Returns nil.
(BACKTRACE depth [frame filtered])	EXTFN	Print a backtrace of function frames up to depth. Optionally start from frame and filter arguments of external function calls if filtered is true.
_BATCH_	GLOBAL	If true, no break loop is entered after errors. Instead a backtrace is printed and the system resets. Useful in batch or server nodes.
(BREAK fn...)	MACRO	Set breakpoints on Lisp functions fn... (not evaluated) for debugging.
(CATCHDEMON loc val)	LAMBDA	Used in conjunction with setdemon.
(CLEAR-FUNCTION-PROFILES)	LAMBDA	Clear statistics for wrapping function profiling.
(DEBUGGING flag)	EXTFN	Toggle debug mode with warnings and assertions based on flag.
(DUMPSTACK)	EXTFN	Print the entire contents of the variable binding stack.
(FRAMENO)	EXTFN	The frame number of the top frame of the stack.
(HELP [tag])	MACRO	Insert explicit breakpoints in Lisp code, identified by tag.
(IMAGE-EXPANSION rate [move])	EXTFN	Control database image expansion rate and memory relocation behavior.
(LOC x)	EXTFN	Return the memory location (handle) of Lisp object x as an integer.
(PRINT-FUNCTION-PROFILES [file])	LAMBDA	Print statistics on time spent in profiled functions.
(PRINTFRAME frameno)	EXTFN	Print the variable stack frame numbered frameno.
(PRINTSTAT)	EXTFN	Print storage usage since the last call to printstat.
(PROFILE)	LAMBDA	Print statistics of time spent in Lisp functions after profiling.
(PROFILE-FUNCTIONS fn...)	MACRO	Collect statistics on time spent in specified Lisp functions fn... (not evaluated).
(REFCNT x)	EXTFN	Return the reference count of object x.
(SETDEMON loc val)	EXTFN	Set up a system trap at memory location loc for integer val, calling (catchdemon loc val) when triggered.
(START-PROFILE)	LAMBDA	Start statistical profiling of this Lisp program.
(STOP-PROFILE)	LAMBDA	Stop profiling the Lisp program.
(STORAGE-USED form)	SPECIAL	Return an association list of net allocated objects of different types during the evaluation of form. Returns nil.
(TIME form)	*MACRO	Print the real time spent evaluating form, returning its value.
(TRACE fn...)	MACRO	Trace specified functions fn... (not evaluated) by printing actual arguments and results in calls.
(TRACEALL flag)	EXTFN	Trace all function calls when flag is true. Warning: this causes a very massive trace.
(TRAPDEALLOC x)	EXTFN	Set up a demon to enter the break loop when object x is deallocated.
(UNBREAK fn...)	MACRO	Remove breakpoints from specified functions fn... (not evaluated).
(UNPROFILE-FUNCTIONS fn...)	MACRO	Remove function profiles from specified functions fn... (not evaluated).
(VAG x)	EXTFN	Return the Lisp object at image location x.
(VIRGINFN fn)	LAMBDA	Retrieve the original definition of function fn, even if it is traced or broken.

Code search and analysis

As Lisp code is also data it is stored in the internal database image. A number of system functions are available for searching and analyzing Lisp code in the image. This can be used for finding functions, printing function documentation, cross-referencing functions, analysing correctness of functions, etc.

Emacs subsystem

A very convenient way to develop SA Lisp code is to run from a shell within Emacs. Emacs should thereby be configured using the file init.el that provides extensions to Emacs for finding Lisp code and for evaluating Lisp by SA Lisp. Place init.el in the initialization folder of Emacs (on Linux the file /home/.emacs).

When Emacs is started give the command:

M-x-shell

This will start a new Windows (or Unix) shell inside Emacs. You can there give the usual Windows (Unix) commands.

First check that the init.el file was loaded correctly by typing F1. If it was loaded correctly there should be a message:

Error: ‘’ is not a file

When Emacs initializes OK, run SA Lisp in the Emacs shell by issuing the command:

sa.engine -q lisp

You can also start the Lisp REPL from the OSQL REPL by entering:

sa.engine
[sa.engine] >lisp;
Lisp >

If you are developing Lisp code in sa.engine, you can enter the OSQL REPL with the command lisp;.

Finding source code

The system contains many Lisp functions and it may be difficult to find their source code. To alleviate this, there are Lisp code search functions for locating the source codes of Lisp functions and macros loaded in the database image having certain properties. Most code search functions print their results as file positions consisting of file names followed by the line number of the source for the searched function. Only source code of lambda functions and macros has file positions.

If Emacs is configured properly, the Emacs key F1 (defined in init.el) can be used for jumping to the source code of the file position at the mouse pointer. For example, the function (fp fn) prints the file position of a function:

(fp 'printl)
====>
PRINTL C:/Users/torer/Documents/sa.engine/lsp/basic.lsp 737
T

If you place the mouse pointer over the file name and press F1 you should be placed in a separate Emacs window at the file position where the function printl is defined. If F1 is undefined you have not installed init.el properly.

If you have edited a function with Emacs it can be redefined in Lisp by cut-and-paste. The key F2 will send the form starting at the pointer position in the file source window to the shell window for evaluation.

If you don’t have the source code you can still look at the definition in the database image using (pp fn):

(pp printl)
====>
(DEFUN PRINTL (&REST L)
  "Print list of arguments on standard output"
  (PRINT L))
(PRINTL)

The macro (pp fn...) pretty-prints the definitions of functions in their internal representation in the database image. The appearance in the source file is normally more informative, e.g. including comment lines and with no macros expanded.

Often you vaguely know the name of a function you are looking for. To search for a function where you only know a part of its name use the CommonLisp function (apropos fn). For example:

(apropos 'ddd)
====>
CDDDDR C:/Users/torer/Documents/sa.engine/lsp/basic.lsp 53
   (defmacro CDDDDR (X)...)

CDDDR EXTFN

CADDDR C:/Users/torer/Documents/sa.engine/lsp/basic.lsp 55
   (defmacro CADDDR (X)...)

Here we see that the function cdddr is an external function with no source code. We can inspect the definition of cadddr with:

(pp cadddr)
====>
(DEFMACRO CADDDR (X)
  (LIST 'CADR
     (LIST 'CDDR X)))
(CADDDR)

The function apropos also prints the documentation of lambda functions and macros. For example:

(apropos 'printl)
====>
PRINTL C:/Users/torer/Documents/sa.engine/lsp/basic.lsp 737
   (defun PRINTL (&REST L)...)
   Print list of arguments on standard output
NIL

The main documentation of a function should be given as a string directly after the formal parameter list, as is done for printl.

To find where a structure is defined you can search for its construction. For example:

(apropos 'make-plan)
====>
MAKE-PLAN C:/Users/torer/Documents/sa.engine/lsp/plan.lsp 3
   (defmacro MAKE-PLAN (&REST ARGS)...)
NIL

Summary of Lisp code documentation and search functions:

Function	Type	Description
(DOC fn)	LAMBDA	The documentation string for function fn.
(FP fn)	LAMBDA	Print the file position of the definition of function fn. Accessible via the break loop command :fp.
(GREP string)	LAMBDA	Print lines matching string in all source files currently loaded in the database image.
(CALLING fn [levels file])	LAMBDA	Print file positions for functions calling to fn. Optional levels indicates depth of indirect callers. Optional file for output redirection.
(CALLS fn [levels file])	LAMBDA	Print file positions for functions called from fn. Optional levels for depth of indirect callees. Optional file for output redirection.
(USING var)	LAMBDA	Print file positions for functions using variable var in their definitions.
(MATCHING pat)	LAMBDA	Print file positions of functions matching the code pattern pat. Regular expressions can be used to match symbols. For example, (matching '(map* '* . *)) matches functions with forms like (mapcar 'print l).

Code verification

SA Lisp has a subsystem for verifying Lisp code. The code verification goes through function definitions to search for code patterns that seem erroneous. It also looks for calls to undefined functions, undefined variables, etc. The code verifier is automatically enabled incrementally when in debug mode. However, full code verification requires that all functions in the image are analyzed, e.g. to verify that all called functions are also defined. To verify fully all functions in the image, call:

(verify-all)

It goes through all code in the database image and prints a report. If everything is OK T is returned and nil is returned if something incorrect is found. For example:

(verify-all)
====>
T              ;; All Lisp functions in image OK

(defun foo (x)(fie x))
====>
FOO

(verify-all)
====>
WARNING - Call to undefined function FIE in FOO. 
NIL

References

[1] Guy L.Steele Jr.: Common LISP, the language, Digital Press, http://www.cs.cmu.edu/Groups/AI/html/cltl/cltl2.html

[2] Common Lisp HyperSpec https://www.lispworks.com/documentation/HyperSpec/Front/.

[3] SA Engine Under the Hood https://www.streamanalyze.com/under-the-hood