Accessing external data
When accessing external data sources there are two important issues to be addressed:
There must be a physical interface to the source.
The data formats provided by the source needs to be converted into a format that SA Engine can process.
There are a number of different kinds of data sources for which physical interfaces are predefined:
External data from other SA Engine peers are communicated using its internal communication protocols.
External data stored in regular files can be accessed using predefined file access primitives.
Data from external systems can be communicated using predefined external communication primitives. For example, streams of sensor readings can be received using serial ports or HTTP-based protocols.
Data can be piped from processes running on the same computer as the SA Engine peer.
Foreign OSQL functions may call embedded programs accessing the data. For example, You can access the microphone on your PC through Java interface functions.
SA Engine peer access​
Other peers can be managed from SA Engine peers through a number of peer management functions.
SA Engine systems running on edges are managed by edge functions.
Connections to other peers are managed by by connection configuration functions
Accessing files​
The file system of your peer can be accessed through file system functions.
You can stream data to and from external log files using logging functions.
The data sored in files need to be converted into SA Engine's internal data formats. There are two Major such file formats supported, JSON files and CSV files.
CSV data files​
A common file and stream data format is Comma Separated Value (CSV)
files. Any
Stream of Vector or Bag of Vector can be saved to a CSV log file
recording using one of the csv:write_file functions:
signature("csv:write_file")
Each row written to a CSV file is a vector. For instance to save 10 rows where each row has a number from 1 to 10 the following query can be used:
csv:write_file("test.csv",(select [i]
from Number i
where i in iota(1,10)))
You can convert a CSV log file to a live vector stream by calling
csv:file_stream, for example:
csv:file_stream("test.csv");
To save a stream to a CSV log file and at the same time get the
output, set the feedback argument to 1:
csv:write_file("test.csv",1,(select [i, sin(i)]
from Number i
where i in iota(1,10)))
csv:file_stream("test.csv");
JSON data files​
The JSON data format is commonly used to represent data in files, repositories and data stream sources.
JSON data is in OSQL represented by
records and
vectors, The types
Record and Vectorare therefore below type Json in the type
hierarchy.
JSON functions handle JSON data access.
Communication interfaces​
The system can communicate with other systems through an extensible number of communication interfaces.
HTTP interface​
You can access Internet data through the HTTP interface. This is the basis for many communication protocols.
Serial streamer​
The serial streamer is an interface to stream data from a serial port:
serial:streamer(Charstring serial_port, Integer baud_rate, Integer start_byte,
Integer stop_byte, Integer data_len) -> Stream of Binary
The function returns a stream of
binary data objects from a
serial port named serial_port. The streamer works by looking for
start_byte in the serial stream, when it finds a start_byte it
will look data_len+1 bytes ahead to check if it is a stop_byte. If
this is the case the data_len bytes between start_byte and
`stop_byte is emitted.
Example binary stream for serial:sreamer("COM8",115200,0x55,0xAA,8):
Start byte Stop byte
| |
0x55 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x08 0AAA
| |
|---------- 8 bytes of data -------|
You can use unpack to get individual numbers from the binary object.