Skip to main content

Release notes for SA Engine

SA Engine 6.0.0​

SA Engine Version 6.0.0 is out And it is truly a new milestone in edge analytics.

SQL+OSQL = True​

We have adapted our query language OSQL to become a seamless extension of SQL with mathematical objects, functions and streams for easy and powerful analytics. OSQL keywords are now SQL compatible and case insensitive. This extension does not break any existing models or queries unless you are using a keyword from SQL in it.

During version 6 we aspire to become fully SQL 92 compatible. To learn how Check out our new onboarding tutorial at https://docs.streamanalyze.com/guides/tutorial/database to learn more and test SA Engines SQL in your browser!

Efficient representation of dense arrays​

Another major feature in version 6 is Array comprehension. Array comprehension is the ability to write highly efficient queries doing computation over and generating multi-dimensional arrays. Arrays has been in experimental-mode for about a year now and we have shown that Array comprehension combined with our query optimization and dynamic compiler generates execution plans that actually outperforms Tensorflow Lite; and is on par with Tensorflow Lite with XNNPACK delegates. It does all this with a super simple implementation of all neural network operators, all done in 31 lines of OSQL. You can verify this yourself by downloading version 6 of SA Engine and reading the readme inside the folder models/nn/test/benchmark inside. The benchmark and performance has been verified on x86-64, AARCH64 and Arm T2.

Thanks to our seamless extension of SQL we now allow any database user to easily deploy and run any type of analytical models on real-time data streams on a high level while even gaining performance compared to conventional methods

The extras​

In addition to these two major features we have many improvements to type checking, optimization rules and increase in the number of compilable predicates.

  • Stream Processing Performance: Several enhancements have been made to stream timers and execution speeds. Enjoy faster, more efficient analytics.
  • Advanced Typechecking and Predicate Compilation: Improvements to the typechecker enable more advanced query rewrites and expand the number of predicates available for compilation. This increases the performance of most queries.
  • Meet our newest merge function: pred_merge, which allows you to select streams for element extraction, enabling fully customized merging schemes.
  • Enhanced OSQL and Python Integration: SA Engine now better handles Python modules and foreign function naming. It is now possible to reload python modules during model deployment.
  • support 2 more ARM variants, 32-bit FPU and soft-fp calling convention with FPU
  • compile bit-field operations s_bits, u_bits, and CANbus frame extraction
  • support for bitwise operators; ~ | & ⊕ << >> ntoh16/32/64

SA Engine 5.6.10​

The 5.6.10 release focuses on expanding support for TFLite, improving compiler logic, and resolving bugs to enhance overall system stability and functionality.

Bugs​

  • Inequality Over Arrays Fail Fixed the bug where inequality operations over arrays failed.
  • Memory Leak in Kafka Plugin Resolved a memory leak issue in the Kafka plugin.
  • Slog-Comp: DIVIDE(x,4)=(1,q) Compilation Issue Resolved the issue where DIVIDE(x,4)=(1,q) was not compiled.
  • Too Many Arguments CGB Bug Addressed the bug where providing too many arguments caused issues in CGB.
  • 4-Byte Alignment from Linux Build Tables Fixed crashes in Arduino platforms caused by 4-byte alignment (integer-cell) from Linux build tables.

Enhancements​

  • Add TFLite Support to MX4 Enhanced MX4 by adding TFLite support, broadening its capabilities.
  • Python Extension: Enable Reloading of Loaded Python Module Enabled reloading of already loaded Python modules in the Python extension.
  • Publish in Store-and-Forward Query Made it possible to use publish@server in store-and-forward queries.
  • Support for publish("flow@server") for Publishing Flows to a Peer Added support for the publish("flow@server") syntax to publish flows to a peer.

SA Engine 5.6.7​

The 5.6.7 release addresses bugs and enhancements to improve performance, reliability, and functionality.

Bugs​

  • Fixed several bugs causing crashes on nanocore

Improvements​

  • Unnecessary Array Conversion
    • Removed unnecessary array conversion processes.
  • Support Multiple Python Versions Added support for multiple versions of Python embeddings.
  • Timestamp Interface in Python Added a timestamp interface in Python.
  • Example Code for Python Call-In Interface Provided example code for the Python call-in interface.
  • Example Code for Foreign Functions in Python Provided example code for different types of foreign functions in Python.

SA Engine 5.6.6​

The 5.6.6 release addresses several minor issues and enhancements to improve performance, reliability, and functionality.

Bug fixes​

  • Edge CQS Performance: Resolved slowness in subscribing to flow inside queries.
  • Nameserver Subscription Failure: Fixed issues with subscribe("query@edge") notation on the nameserver not working
  • Implicit Unnesting in Case Statements: Fixed the bug affecting implicit unnesting in case statements within queries.
  • Function Invocation with Case-Statement: Resolved the bug in invoking calls on functions with case statements.
  • CANBus Starvation on Linux: Resolved the starvation issue on CANBus when data rates are too high.
  • CSV Conversion Bug: Fixed the bug related to converting CSV strings with national characters to CSV vectors.

Enhancements​

  • TF-Lite Interface Update: Updated the TF-Lite interface to use arrays improving performance.
  • Re-enable 2D Operators in nn.osql: Re-enabled 2D operators in the nn.osql module.
  • Generic Slicing: Implemented generic slicing for arrays.

SA Engine 5.5.0​

The 5.5.0 release brings a host of enhancements and new features focused on improving performance, expanding functionality, and fixing key issues. Significant speed improvements in various operations, advanced system models, and robust type checking are some of the highlights of this release. This release marks a significant step forward in the system's capabilities, emphasizing performance, scalability, and advanced analytical features.

New Features​

  • MQTT on RPI64-bit: MQTT protocol support included for Raspberry Pi 64-bit, expanding IoT capabilities and connectivity options.
  • Array Support in DBSCAN and KMEANS: Both DBSCAN and KMEANS now support Arrays as data points, broadening the scope of these algorithms for diverse data types.
  • New System Models:
    • DENSTREAM: Introduced the DENSTREAM model for density-based clustering over evolving data streams with noise. Learn more about DenStream.
    • Locality Sensitive Hashing (LSH): Integrated LSH as a system model, enabling efficient high-dimensional vector search. Learn more about LSH.

Improvements​

  • XTREE Indices for Arrays: Introduced support for XTREE indices over Arrays, offering more efficient data handling and retrieval.
  • Enhanced Subscription Syntax: subscribe can now target any peer using the "flow@peer" syntax, increasing flexibility in data flow management.
  • Performance Boosts:
    • Otsu's binarization process is now approximately 100 times faster, significantly optimizing image processing tasks.
    • Improved the speed of cv:hist and cv:gray2rgb, both by about 100 times, greatly enhancing performance in computer vision operations.
  • JDBC Driver Enhancements: The JDBC driver now supports multithreaded connection pools, improving database connectivity and performance in multi-threaded environments.
  • Type Checking and Inferencing: Stricter type checking and improved type inference mechanisms have been implemented, ensuring more robust and error-free code.

Bug Fixes​

  • XTREE Indexes in WASM: Fixed critical issues where XTREE indices could crash in the WebAssembly (WASM) environment, enhancing stability and reliability.

SA Engine 5.4.0​

New Features and Improvements​

  • Slog Compiler Refinements: Major improvements in register allocation and memory management, alongside fixes in unwind table handling for Windows, enhancing compiler efficiency and robustness.
  • Enhanced Array and Matrix Functionalities: New functions for window aggregation in arrays, along with the introduction of generic matrix multiplication.
  • New array counting window function: Introduced awinagg which builds windows over stream of arrays.
  • New array constructor: array(format, shape, initial-value) create array of format with shape and set every element to initial-value.
  • System Enhancements: Reduced memory object size overhead.

Bug Fixes​

  • WASM HTTP Download Issue: Resolved a critical bug where files weren't saved correctly in memory when downloaded via WASM HTTP, particularly for non-octet stream types.
  • Permissions in Release Files: Addressed the issue with incorrect permissions on .dmp files in the release tarball, enhancing system security and integrity.

SA Engine 5.3.0​

New Features and Improvements​

  • PNG Handling: Added support for reading and writing PNG files across most platforms, enhancing media processing capabilities.
  • FFT Performance: Achieved a significant 6x improvement in FFT (Fast Fourier Transform) performance, ensuring faster signal processing and data analysis.
  • JIT Compiler: Made several internal enhancements to the JIT (Just-In-Time) compiler, optimizing runtime performance.

Bug Fixes​

  • Python API Stability: Addressed a critical issue in the Python API where calling an undefined function could lead to a system crash.
  • UTF-8 Reading: Fixed a bug where EOF (End of File) was not detected accurately when reading specific UTF-8 characters. This ensures accurate and efficient text processing.

SA Engine 5.2.0​

New Features​

  • Code and Network Downloads: Microedge-listeners now support downloading of missing code and SANN-networks.
  • Array Flattening: A handy new flatten function has been introduced which can transform nested vectors into rank 1 arrays.
  • Computer Vision Capabilities: Meet the new system model 'CV' - equipped with utility functions tailored specifically for computer vision applications.
  • Array Support: Arrays have been enhanced with support for special values - Infinity (Inf) and Not-a-Number (NaN).
  • Array Slicing: Slicing ranges in arrays is now possible using the a[arange(high,low),*] notation. A more intuitive syntax for this feature will be introduced in the future.
  • MQTT-extension for RPI64: RPI64 now comes bundled with an MQTT-extension, providing more seamless integration with IoT protocols.
  • JSON Type in OSQL: We’ve introduced the OSQL type Json as a union of Vector and Record types, facilitating easier handling of JSON data.
  • Array Function Type Inference: Type inference for generic array functions has been added, allowing you to define one function for all types of arrays.

Improvements​

  • Python API: The Python API is now thread safe.
  • Error Messaging: Error messages, especially when running documentation, now properly display line breaks for better clarity.
  • Array Comprehension: Compilation of almost all array comprehension queries is now fully supported.
  • Verification: A verification step has been added to Store and Forward uploads to ensure the accuracy of data transfers.
  • Platform Support: SA Studio is now supported on RPI64.
  • SLAP Block Size: The system can now handle much larger SLAP blocks, enabling the compilation of more extensive queries directly into machine code.
  • Query Processing: Added the notion of query blocks for separation between rewrites and cost-based query optimization.
  • Query Rewriting: We’ve relaxed the tr rewrite guards and also introduced rewrites over stateful predicates (#2881).
  • Query Compilation: Simple disjunctive conditions are now fully compiled, providing faster query execution.
  • Tuples in streams between threads: Addressed an issue related to handling tuples over streams between coroutines.

Bug Fixes​

  • Transcriber Issues: Several AARCH64-related bugs in the transcriber have been addressed.
  • Topic Merging: Fixed the subscribe:merge functionality to ensure it works perfectly when merging topics from both local and remote sources.
  • Array Type System: Made necessary improvements to the type system concerning arrays, enhancing both performance and accuracy.

Removed features​

  • Removed support for numarrays which have been replaced by arrays.

SA Engine 5.1.0​

New Features​

  • Implemented mean(Array) and div(Array, Real) functions
  • Many new functions for array of real (f64) added
  • Introduced basic element-wise arithmetic operators over arrays
  • Webhooks feature introduced

Improvements​

  • Switched to arange() from vrange() for enhanced performance
  • Added performance tests to array regressions
  • Optimized construction of vectors and arrays
  • Improved optimizer to move constant expressions out of loops
  • Enabled compiling assignment of Charstring
  • Enabled full compilation of simple disjunctive conditions
  • Enabled compiling MAX/MIN/ABS and NDARRAY literals
  • Improved type inference for generic array functions
  • Casting to Real in generic array functions to avoid type Number
  • Added better reconnection logic for JDBC driver

Bug Fixes​

  • Fixed ND-Arrays not being deallocated correctly
  • Fixed constant OR expressions not being compiled
  • Fixed error thrown when setting ndarrays from python to session variables
  • Fixed deadlocks and/or crashes when using threads when calling python.

Removed Features​

  • Removed new_array(bag b) function

SA Engine 5.0.2​

  • Performance Improvements: Enhanced efficiency in array comprehensions.
  • New Platform Support: Added support for RPI 64-bit (ARMv8 AARCH64) with Python call-in and call-out.
  • Cross-platform Compatibility: Enabled WASM edge on iOS and OSX.
  • Array Comprehensions: More components are now compiled, improving performance.
  • Database Support: Federation Services now compatible with both MSSQL and MYSQL.

SA Engine 5.0.1​

  • Enhanced array management.
  • Optimized resource use in merging stream vectors.
  • sa.engine version now in user-agent HTTP header.
  • Added sa.engine peername to sa-peer HTTP header.
  • Fixed an issue where our CI/CD pipeline deployed the wrong version of the windows release

SA Engine 5.0.0​

Key Additions:​

  • Arrays: We're introducing a new data type for managing densely packed numeric arrays.
  • Array Comprehension: An experimental feature to allow advanced array operations, translating high-level commands into machine code for maximum performance. Note that it's still under development, with continuous improvements planned for future releases.
  • The integration of Arrays and Array Comprehension allows our query optimizer and JIT compiler to produce highly efficient machine code. In the near future, it will also support SIMD instructions.
  • This will enable users to create complex models, such as deep learning, in the high-level OSQL language without compromising on performance or simplicity.
  • Scalable Federations: We've revamped the connection logic in the SA Engine to support scalability using Kubernetes, harnessing the power of web scale technology.

Noteworthy Improvements:​

  • Enhanced Unicode support.
  • Refined CAN bus wrapper featuring improved stopping logic.
  • SA Engine can now run in WebAssembly.
  • Model management API has undergone a name change and refactor.
  • A greater number of vector operations are now compiled.

GUI Changes:​

  • We've given SA Studio a complete makeover. The Bitmap visualization feature has been added for efficient array visualization. Also, SA Studio is now independent from SA Engine, with separate life cycles. Consequently, this will be the last release note combining updates for both SA Engine and SA Studio.