LogicBlox 4.0


LogicBlox 4.0.8

Release Date: March 3rd 2014

Executive Summary

LogicBlox 4.0.8 includes the following scalability, performance and functional enhancements:

  • Data loading has been benchmarked to load data up to TPC-H scale factor 1000, with LogicBlox being 33% faster than representative competitors on equivalent hardware.
  • Sorting is now incremental, where the work done to update a sorted predicate is proportional to the amount of data changed.
  • Transaction throughput continues to be tuned, with 4.0.8 showing at 2x throughput improvement through lb-web over 4.0.7.
  • Tabular Data Exchange is mostly functionally complete with respect to TDX in LogicBlox 3.x. Th release notes highlight the details that will be addressed in the next release.

What's New

Services Framework

LogicBlox 4.0.8 includes support for the following Tabular Data Exchange features:

  • GET (export), POST (append) and PUT (replace) HTTP methods.
  • File formats with extensive checking (including booleans and dates).
  • Error handling: rows that contain errors in imports are sent back to the client with cause columns added to the file.
  • Optional columns, bound to values of functional predicates.
  • Export functions (functions, that you can use to transform values on export), including constructors and inverse functions.
  • Auto refmode conversions.
  • Entity accumulation policies (NONE and ACCUMULATE).
  • Retraction of empty optional values on POST.

Known Limitations and Workarounds

TDX support in LogicBlox 4.0.8 is still in beta release, with some limitations:

  • Auto primitive conversions: format checking is currently very strict. E.g., one can only bind an "integer" column to an "int" value. In the future we will allow certain conversions, such as binding a "float" column to an "int" value.
  • Import functions (functions, that you can use to transform values on import) are currently not supported.
  • Import filters: these are predicates bound to a file to work as a filter for the import, as opposed to being filled by the import. The filter flag is currently ignored, so the filter predicate will be modified by an import.
  • Ordered columns are currently not supported.
  • Abort on error: currently it is not possible to abort a transaction when there is an error in any of the import rows.
  • HTTP error codes: currently any error that prevents import (such as missing headers) causes a status 500 to be sent to the client. In 3.x, these errors would lead to a status 4xx.

Performance Improvements

  • LogicBlox 4.0.8 has been tuned to load data up to TPC-H Scale Factor 1000. Figure 1 demonstrates data load benchmarks of LogicBlox against a number of competitors on the same hardware, up to Scale Factor 100. LogicBlox outperforms the closest competitor by around 33%. We are in the process of benchmarking competitors on equivalent hardware for scale factor 1000.
  • The transaction throughput performance of lb-web has increased by ~2x from LogicBlox 4.0.7, using the TinyBank benchmark.

Figure 1.  TPC-H Data Load

TPC-H Data Load

Other Improvements

  • Improvements in predicate sorting (list <<>>):
    • Added incremental maintenance to predicate sorting. The cost of updating a sorted predicate is now proportional to the size of the changes rather than to the size of the predicate.
    • list <<>> exposes last (optional).

      Example 1. 

      In addition to the old syntax (which is also supported):
      first(y,x), next(y,x,u,v) <- list<<>> b(x,y).
      
      it is now possible to write:
      first(y,x), next(y,x,u,v), last(y,z) <- list<<>> b(x,y).
      
  • Autonumbered refmodes have been removed from generated protobuf messages.

    Example 2. 

    A protobuf message named Msg used to have both an autonumbered refmode and a constructor. As of 4.0.8, only the constructor is available. To get the auto-generated number for an individual message, you should now use the constructor:
    MsgContructor[i]=m -> int(i), Msg(m).
    
  • The limit of strings stored in predicates is increased to ~50% of a page size. In earlier 4.x releases, the strings stored in predicates could at most be 255 characters long.
  • Setting logdir_access in lb-web-server.config to an empty value now makes lb-web-server print access logs to the standard error instead of to a file.

Corrected Issues

The issues listed below have been corrected since the 4.0.7 release.

  • Resolved an issue with max aggregations over negative values.
  • Removed some unnecessary stack traces from the lb-compiler.log.

Installation and Upgrade information

Installing LogicBlox 4.0.8 is as simple as following the steps outlined below:

  1. Download the installation package
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.0.8/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Release Information

Table 1. 

Server requirements
Operating System: 64 bit Linux
Java Runtime Environment 1.7. You can check your Java version by executing java -version.
Python 2.7.

LogicBlox 4.0.7

Release Date: February 3rd 2014

What's New

Services Framework

LogicBlox 4.0.7 includes more complete support for the following TDX export features:

  • Column Formats: 4.0.7 TDX export supports all column formats previously supported in LogicBlox 3.x, as well as 3 additional formats:
    • raw_string: as opposed to regular strings, raw_string column values are not trimmed
    • non-negative floats: specified using 0.0+. Columns of this type accept any number greater than or equal to 0.
    • positive floats: specified using >0.0. Columns of this type accept any number greater than, but not equal to, 0.
    Furthermore, formats are now enforced on export. For example, rows for alphanum columns are only exported if the value contains only letters and numbers.
  • Optional Columns: Optional columns must be bound to the value of functional predicates.
  • Transform Functions: File bindings may attach functions to columns so that values are transformed upon export. Both regular (binds keys) and inverse (binds value) functions are supported.

Known Limitations and Workarounds

TDX support in LogicBlox 4.0.7 is still in beta release, with some limitations:
  • The features mentioned above apply to export only (GET requests).
  • Transform functions on optional columns are not yet supported.
  • Files still must not contain a header line.
If you find issues with the TDX export in 4.0.7, you can fall back to the implementation in 4.0.6 or prior by including the following parameter declaration in your service definition:
delim_services(x) {
  ...
  service_parameter["new"] = "false"
}

Database Administration

The lb export-workspace command now allows users to create an online (hot) copy of the database. The LogicBlox services do not need to be stopped to perform this activity, as the export feature knows when the database is at a consistent state to be copied.

Corrected Issues

The issues listed below have been corrected since the 4.0.6 release.

  • Resolved an issue where a failure to create a response protobuf message incorrectly reported an HTTP 200 OK response.
  • Resolved an issue where deltas for pulse predicates at stage @init did not correctly combine with deltas at stage @final.
  • Resolved an issue that caused an internal error in the lb-webserver log when monitoring pulse predicates using the lb web-server monitor command.
  • Resolved an issue with the lb-web internal confirm-reset-password handler that caused an error when parsing "Z" as a timezone.

    <time>Z

    Z is the zone designator for the zero UTC offset in the ISO 8601 standard.

Installation and Upgrade information

Installing LogicBlox 4.0.7 is as simple as following the steps outlined below:

  1. Download the installation package
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.0.7/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Release Information

Table 2. 

Server requirements
Operating System: 64 bit Linux
Java Runtime Environment 1.7. You can check your Java version by executing java -version.
Python 2.7.

LogicBlox 4.0.6

Release Date: January 7th 2014

Executive Summary

LogicBlox 4.0.6 introduces support for automatic parallel evaluation of a query across multiple cores. Domain parallelism, a mechanism for splitting data into pieces such that a query can be evaluated on each piece in parallel, allows LogicBlox to scale up its performance with the number of cores available in a single machine. Applications should expect dramatic performance improvements of both data load as well as query evaluation.

In addition to domain parallelism, 4.0.6 includes enhancements to the services framework that enforce stricter checking of incoming service messages, as well as support for specifying Java manifest files in the lb config tool.

What's New

Domain Parallelism

We are excited to introduce domain parallelism with LogicBlox 4.0.6. With domain parallelism the data needed for a given query is on-the-fly split up in pieces, such that a query can be evaluated for all pieces of data in parallel, using all available processor cores. Applications built using LogicBlox 4.0.6 can expect significant performance gains on multi-core machines, with data load, queries, and inactive blocks.

Figure 1 and Figure 2 illustrate the performance of LogicBlox 4.0.6 on the data-load of TPC-H, a standard decision support benchmark. The scale factor on the x-axis corresponds to the size of the data-set in gigabytes. The dark blue line in both charts is LogicBlox. The systems we compare ourselves to are PostgresSQL, MySQL, Amazon Redshift, MonetDB, and a commercial in-memory column store (legend left out for anonymization). We compare the duration of the load on two different AWS EC2 instances types: the hs1.8xlarge instance has 8 physical cores, and the cr1.8xlarge instance has 16 cores. All experiments are configured to use all cores. It is clear that with domain parallelism, LogicBlox outcompetes all systems on the data load, and that the gap grows bigger when more processor cores are available.

Figure 2.  TPC-H Data Load - cr1.8xlarge

TPC-H Data Load - cr1.8xlarge

Figure 3.  TPC-H Data Load - hs1.8xlarge

TPC-H Data Load - hs1.8xlarge

Figure 3 illustrates the performance gains of LogicBlox 4.0.6 compared to LogicBlox 3.x for the 22 queries of TPC-H, translated into LogiQL. The combined benefit of domain parallelism, our novel join algorithm, and accurate cost estimation results in a dramatic speed-up in total query evaluation time.

Figure 4. LogicBlox 4.0.6 vs. 3.x for 22 TPC-H queries

LogicBlox 4.0.6 vs. 3.x for 22 TPC-H queries

LogicBlox has also evaluated domain parallelism on several other benchmarks, such as a variety of graph queries, and queries using complex joins. Consistently, we have seen good scaling of the query with the number of available cores. Some of these experiments were earlier presented at the LogicBlox User Days 2013.

Note

Note that domain parallelism is currently enabled only for data loading and for query or inactive block evaluation. Evaluation and maintenance of IDB predicates will be parallelized in a subsequent release.

Services Framework

It is now possible to specify on a protobuf service whether it should parse the JSON strictly or loosely. The default, strict, behavior causes an error, if the JSON contains fields that are not specified in the protobuf specification. To specify that JSON messages should be parsed loosely, use the json_parsing predicate in configuration. The following example illustrates how to specify a service using loose parsing.

Example 3. Example of loose JSON parsing of a service

service_by_prefix["/loose-time"] = x,
    default_protobuf_service(x) {
      protobuf_protocol[] = "time",
      json_parsing[] = "loose",
      protobuf_request_message[] = "Request",
      protobuf_response_message[] = "Response"
    }.

Developer Tools

The jar() function of the LB configuration tool now takes a manifest parameter that specifies how to generate the manifest. It can use a static file, or it can configure the manifest with a main class and getting jars from the classpath.

Corrected Issues

The issues listed below have been corrected since the 4.0.5 release.

  • The default search path for Measure Service handlers changed to $(...)/lib/java/handlers, making it impossible for .jar files to be loaded by the LB web-server when not intended.
  • The lb services status command now prints the status OFF when services are not started.

Installation and Upgrade information

Installing LogicBlox 4.0.6 is as simple as following the steps outlined below:

  1. Download the installation package
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.0.6/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Release Information

Table 3. 

Server requirements
Operating System: 64 bit Linux
Java Runtime Environment 1.7. You can check your Java version by executing java -version.
Python 2.7.

LogicBlox 4.0.5

Release Date: December 2nd 2013

What's New

Services Framework

LogicBlox 4.0.5 introduces support for HTTP methods GET and PUT for protobuf based web services. Additionally, web services can be defined to support only certain subsets of methods. When defining a service for a specific method, you can define both the URL and the HTTP method.

Example 4. 

Below you can find an example of the new syntax:
service_by_prefix_and_method["/pst-time", "GET"] = x,
    default_protobuf_service(x) {
      inactive_block_name[] = "sample:pst_time",
      protobuf_response_protocol[] = "time",
      protobuf_response_message[] = "Response"
    }.

You will notice a new constructor for the service service_by_prefix_and_method which allows you to specify an HTTP method for a particular service, in this example we are using the GET method.

Accessing a service with an HTTP method that is not configured will return an HTTP 405 (Method Not Allowed) error.

Backward compatibility is maintained for alll use cases, except for services involving CORS rules and custom Java handlers. For details please refer to the LogicBlox 4.0 Migration Guide.

Developer Tools
  • The clean-workspaces target in the LB configuration tool can now be used to remove previously installed workspaces.
  • The new command lb web-server home-dir can be used to print the assumed home directory for the LB web-server.

Corrected Issues

The issues listed below have been corrected since the 4.0.4 release.

  • Resolved an issue that caused LogiQL scripts to run, even after an error.
  • Resolved an issue where an error in the lb command in interactive mode terminated lb interactive.
  • Resolved an issue where certain admin commands failed when a log level was configured.
  • Resolved an issue with importing data into pulse predicates.
  • Resolved an issue with the monitoring of predicates using the lb web-server monitor command that resulted in an exception when monitoring a predicate of type datetime
  • It is now possible to add or remove predicates to the list of monitored predicates instead of resetting the whole list

    Example 5. 

    You can list all monitors by running the following command:
    $ lb web-server monitor
    To add monitors on predicates p1, p2 and p3, run:
    $ lb web-server monitor p1 p2 p3
    To remove monitors on p1 and p2, run:
    $ lb web-server monitor --remove p1 p2
    Finally, you can run the command below to remove all monitors:
    $ lb web-server monitor --remove
  • Resolved an issue where the batch configuration option tcp_max_connections_per_address in BatchConfig, that is used as default value for individual statements, was not propagated correctly.

Installation and Upgrade information

Installing LogicBlox 4.0.5 is as simple as following the steps outlined below:

  1. Download the installation package
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.0.5/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Release Information

Table 4. 

Server requirements
Operating System: 64 bit Linux
Java Runtime Environment 1.7, update 11 or higher
Python 2.6.4 or higher

LogicBlox 4.0.4

Release Date: November 1st 2013

What's New

User-friendly Command-line Tools.  LogicBlox 4.0.4 includes a set of clean and intuitive commands for developing and managing the LogicBlox database and services. All things LogicBlox-related can be managed through one top level command, lb. The extensive help text that accompanies the lb command-line tool as well as tab-completion guide developers to the appropriate subcommands and options for their specific use cases. LogicBlox 4.0 Migration Guide provides detailed instructions on mapping old commands onto the new ones.

Decimal Built-ins.  We continue to mature the simplified numeric implementation by providing useful built-ins for the decimal type. LogicBlox 4.0.4 includes the following built-in's:

  • decimal:round
  • decimal:floor
  • decimal:abs
  • decimal:int:eq and int:decimal:eq
  • decimal:range

For a complete overview of all the built-ins supported in LogicBlox, please visit the Reference Manual.

Corrected Issues

The issues listed below have been corrected since the 4.0.3 release.

  • Resolved an issue where cross-origin resource sharing did not work on Chrome.
  • The ServiceBlox proxy service can now be configured in lb-web-server.config with regard to the HttpClient it uses. This was previously only applicable to the Global Protobuf Service. The following configuration options are available:
    • tcp_client_threads : Size of the thread pool for the HttpClient (shared among all clients).
    • tcp_timeout: Max time the whole exchange can take (in ms).
    • tcp_connect_timeout: Max time to wait when establishing a connection to the server (in ms).
    • tcp_idle_timeout: Max time in a connection without processing anything (in ms). Processing is defined to be "parsing or generating".
    • tcp_max_connections_per_address: Max number of connections the HttpClient can hold to an address.
  • Resolved an issue where lb unit returned an exit status of 0 when detecting a failing test. It now reports a non-zero status, so that the shell can catch the failing cases.
  • Improvements in compiler stability and error reporting, for numeric types, polymorphic atoms, and head existentials.

Installation and Upgrade information

Installing LogicBlox 4.0.4 is as simple as following the steps outlined below:

  1. Download the installation package
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.0.4/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Release Information

Table 5. 

Server requirements
Operating System: 64 bit Linux
Java Runtime Environment 1.7, update 11 or higher
Python 2.6.4 or higher

LogicBlox 4.0.3

Release Date: October 1st 2013

What's New

Database

Logging

Various logging options have been added to the lb command line. When executing a command, users may indicate the logs of interest using the following the following format:

	    --loglevel LOGLEVEL@CONTEXT[:LOGLEVEL@CONTEXT...] 
	  

  • LOGLEVEL indicates the level of logging. Valid options are, from the least verbose to the most verbose: error, warning, info, debug
  • CONTEXT indicates the type of activity you would like to see logs for. Valid options are:
    • initial: activities during stage initial will be logged
    • final: activities during stage final will be logged
    • query: activities during stage query -- the stage after final, where queries after fixpoint are evaluated -- will be logged
    • clean: activities during the clean-up phase of a transaction will be logged
    • maintenance: activities related to the maintenance of IDB predicates will be logged
    • fixpoint: activities related to the fixpoint computation of recursive rules will be logged
    • Workspace NAME: activities against a specific workspace, indicated by NAME, will be logged

Services Framework

Configurable Workspace Scans

BloxWeb now supports the configuration of which workspaces to scan during start up, through the scan_workspaces option in bloxweb.config. Developers can specify a regular expression, indicating the workspaces to scan. This change can dramatically speed up the BloxWeb start-up process, especially for applications with a large amount of workspaces that do not host services, and thus do not need to be scanned (e.g. Blade based applications with a large amount of workbooks that do not host services).

Example 6. 

The following configuration in bloxweb.config indicates that BloxWeb should only scan workspaces whose names start with "delim-" or "protobuf", in addition to installing static workspaces:

		  scan_workspaces = delim-.*|protobuf.*
		  

Handler Configuration

To simplify development, our releases now come by default configured with commonly used extensions, such as the measure service.

Development Support

  • To help developers debug memory related issues, we now create a memory dump every time there is an OutOfMemory error. It can be found by default in $LB_DEPLOYMENT/logs/current.
  • Constraints have been added to avoid errors when services are declared in the same prefix and group.

Delimited File Services

The delimited file services now support the HTTP HEAD method, in addition to GET, PUT, and POST. HEAD is often used by reporting and integration tools (e.g. Kettle, MS Excel) to check whether a particular service exists.

Enhancements have also been made to our error reporting. The string* format for delimited files cannot be used in optional columns. However, the constraint violation presented to the developer was not easy to interpret. The example below shows the old vs. the new error.

Example 7. 

Old:

Constraint with values that caused the violation:
!(bloxweb:delim:schema:column_optional(8), 
   bloxweb:delim:schema:format_empty_string[] = 6, 
   bloxweb:delim:schema:column_format[8] = 6).

New:

Constraint with values that caused the violation:
!(bloxweb:config:validation:configuration_error(
"File '/delim-format/emptystring-optional-emptystring', column 'STRING2':
        Optional column cannot use the string* format.")).

Developer Tools

lb-config now supports the execution of individual test cases.

Example 8. 

Assuming that the test suite that needs to be run takes a command line argument for the test case that you would like to run, for example test_bar, you would execute the following:

		make check-foo testcase=test_bar
	      

Corrected Issues

The issues listed below have been corrected since the 4.0.2 release.

Database

  • Issues proper error message when an inactive block is executed at a stage different than one that it is compiled for.
  • Various stability improvements.

Services Framework

  • The error message reported when the same URL is used for multiple queue specifications has been improved.
  • Resolved an issue in BloxWeb that leaked the results of monitoring returned by ConnectBlox to handlers. This caused the measure service to interpret monitoring results as query results.
  • To facilitate development, localhost is now by default allowed to do cross-origin requests.
  • The buffer that BloxWeb uses when serializing an HTTP message to post into a queue is increased to be comparable to the buffers used in HttpClient, as longer URLs were prevented from being serialized.
  • Resolved an issue with the behavior of proxies when there are errors and made sure that exceptions and timeouts are logged correctly.

Installation and Upgrade information

Installing LogicBlox 4.0.3 is as simple as following the steps outlined below:

  1. Download the installation package
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.0.3/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Release Information

Table 6. 

Server requirements
Operating System: 64 bit Linux
Java Runtime Environment 1.7, update 11 or higher
Python 2.6.4 or higher

LogicBlox 4.0.2

Release Date: September 2nd 2013

What's New

LogicBlox 4.0.2 includes a further clean up to the semantics of negated formulas. The compiler guarantees that negated formulas are translated in a way that respects the quantifiers of variables. In the case that a complex, negated formula cannot be evaluated natively by the runtime, or cannot be supported by the compiler with a rewrite, the formula will be rejected at compile time.

Other improvements include:

  • Added conversion function int:decimal:convert to allow conversion between two simplified numeric types introduced since 4.0.
  • Improvements to the optimizer that dramatically lowered the amount of memory consumption for small rules.
  • General stability improvements.

Corrected Issues

The issues listed below have been corrected since the 4.0.1 release.

Database Runtime

  • Fixed handling of transaction:id and datetime:now. transaction:id is a predicate containing the identifier of the current transaction, while datetime:now is a predicate containing the time at which a transaction is being evaluated. Both predicates were errorneously keeping data from previous transactions, rather than reflecting the fresh values of the active transactions. The issue has now been fixed.
  • Fixed minor bugs in the support for inactive and afterfixpoint blocks when loading from separately compiled projects, and when executed via execblock.

Services Framework

  • lb-services now waits for bloxweb to start before continuing. A timeout is introduced, to make sure that lb-services do not start, if bloxweb takes longer than the timeout.

    Tip

    The timeout is configurable in bloxweb.config, using the startup_timeout configuration option. The default timeout is set to 60 seconds.

  • Resolved a security issue where the BloxWeb login service was allowed to be used with the SAML authentication realm. The security vulnerability addressed by this release only affects applications that are using SAML authentication.

Installation and Upgrade information

Installing LogicBlox 4.0.2 is as simple as following the steps outlined below:

  1. Download the installation package
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.0.2/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Release Information

Table 7. 

Server requirements
Operating System: 64 bit Linux
Java Runtime Environment 1.7, update 11 or higher
Python 2.6.4 or higher

LogicBlox 4.0.1

Release Date: August 1st 2013

Executive Summary

LogicBlox 4.0.1 introduced a set of drastically simplified numeric types: the types int, float, and decimal are to replace numeric types with explicit bitwidths (e.g. int[32]). While backward compatibility will be maintained, these new numeric types are preferred. We encourage all applications to start using them exclusively.

LogicBlox 4.0 supports applications built using the BLoxWeb service-oriented architecture. Some modification of logic may be necessary. Please refer to the migration guide for details.

What's New

LogicBlox 4.0.1 introduces a set of drastically simplified numeric types: the types int, float, and decimal are to replace numeric types with explicit bitwidths (e.g. int[32]). While backward compatibility will be maintained, these new numeric types are preferred. We encourage all applications to start using them exclusively. Please refer to the LogicBlox 4.0 Migration Guide for details.

The LogicBlox Connectitivity protocol, LogicBlox-distributed services such as Delimited File Service and Measure Service, the BloxWeb service framework, and the protocol buffer support, have all been modified to be consistent with the simplified numeric types. This implies that all protocol buffer message specifications must be modified to only use supported numeric types: int64, sfixed64, sint64, and double.

Lastly, our effort to clean up the language continues by discontinuing the support for setting predicate or system properties through delta rules that write into system:* predicates. These rules have been deprecated since LogicBlox 3.4. All predicate properties should be set using lang:* pragmas.

Corrected Issues

The issues listed below have been corrected since the 4.0.0 release.

  • Corrected a memory leak that could cause significant heap usage, and possibly swapping, for large workspaces.
  • Corrected the handling of LB_LIBRARY_PATH, LB_STANDARD_LIBRARY, and libpath in the connectivity server.
  • Corrected the exit status of lbunit during incorrect execution. It now returns a non-zero exit code for a failing test.

Installation and Upgrade information

Installing LogicBlox 4.0.1 is as simple as following the steps outlined below:

  1. Download the installation package
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.0.1/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Release Information

Table 8. 

Server requirements
Operating System: 64 bit Linux
Java Runtime Environment 1.7, update 11 or higher
Python 2.6.4 or higher

LogicBlox 4.0

Release Date: July 2nd 2013

Executive Summary

LogicBlox 4.0 unveils the foundation for the LogicBlox Smart Database technology. The Smart Database delivers raw performance, a simplified programming model and reduced database administration overhead, together with full ACID-compliance.

  • Innovations in the core query evaluation algorithm and the concurrency model enable LogicBlox 4.0 to out-perform LogicBlox 3.x across a wide spectrum of query workloads (transactional, analytical, as well as graph). For certain query workloads, LogicBlox 4.0 out-competes industry leaders.
  • Innovations in query optimization and continued improvement in the programming model reduce the need for manual tuning of queries as well as the database. Preliminary benchmarks show that for natural queries -- those that have not been hand-optimized by a human expert -- LogicBlox 4.0 delivers query evaluation times that are competitive with the evaluation times of hand-optimized queries on LogicBlox 3.x.
  • Beyond full ACID-compliance, LogicBlox 4.0 delivers full serializability as the default isolation level, further relieving programmers from having to reconcile anomalies that may arise from weaker isolation levels.

LogicBlox 4.0 supports applications built using the BloxWeb service-oriented architecture. Some modification of logic may be necessary. Please refer to the migration guide for details.

What's New

The more appropriate question for this release is, "What's not new?" The LogicBlox 4.0 runtime was reinvented from the ground up to improve raw query evaluation as well as concurrent read/write transaction performance. We show benchmarks that demonstrate the improvements in query evaluation and the handling of concurrent transactions. LogicBlox 4.0 continues to evolve the language, as well. The number of parameters a programmer needs to tune in order to meet an application's performance requirements has been dramatically reduced. We outline these simplifications. We also use benchmarks to demonstrate the effectiveness of automatic query optimization and tuning over manual efforts in LogicBlox 4.0.

On Performance

On Query Evaluation

One of the core innovations in LogicBlox 4.0 is the join algorithm. We demonstrate its current performance characteristics using three sets of benchmarks: TPC-H, some realistic queries extracted from an existing application that have proven to be problematic for LogicBlox 3.10, and the 4-clique query which demonstrates the applicability of LogicBlox 4.0 on graph algorithms.

Figure 1 illustrates the performance of LogicBlox 4.0 on TPC-H against LogicBlox 3.10, and LogicBlox 4.0-parallel. (Note that parallel evaluation is not included in LogicBlox 4.0 but is expected for a subsequent release). TPC-H is the industry standard benchmark suite for OLAP databases; it includes 22 complex queries over large volumes of data. These queries are representative of the queries seen on typical LogicBlox applications. Figure 1 shows that LogicBlox 4.0 takes only half the amount of time that LogicBlox 3.10 takes to complete the entire TPC-H suite. The median query time for LogicBlox 4.0 is 24.6 seconds, while 115.1 seconds on LogicBlox 3.10.

Figure 5. TPC-H Scale 10

TPC-H Scale 10

Figure 2 demonstrates the performance of LogicBlox 4.0 over LogicBlox 3.10 on two queries extracted from an existing project. These queries were found to cause application performance issues on LogicBlox 3.10. The figure demonstrates that LogicBlox 4.0 improves upon LogicBlox 3.10 performance by an order of a magnitude. It further demonstrates that running the same query the second time (columns 2 and 4) yields further gains due to the indexes that have already been created during the queries' first runs.

Figure 6. Sample problematic queries on 3.10

Sample problematic queries on 3.10

Finally, we demonstrate the performance of LogicBlox 4.0 on a graph query, 4-clique. A clique is a graph where every vertex is connected to every other vertex in the clique. Graph queries, such as 4-clique, are typical in its use of self-joins. Specialized databases have been implemented specifically to support graph queries. As shown in Figure 3, LogicBlox 4.0 (red line) outperforms PostgreSQL, Amazon Redshift, commercial in-memory column-store, as well as graph databases. This benchmark illustrates the unique promise for the LogicBlox database to unify not only the processing of transactional and analytical, but also graph workloads.

Figure 7. 4-Clique

4-Clique
On Concurrency

LogicBlox 4.0 implements a novel, lock-free concurrency model. This model is key in enabling high transaction throughput for mixed read and write, long- and short-running concurrent transactions. LogicBlox 4.0 currently supports fully concurrent read transactions, while serializing the writes. In future releases, we will be implementing concurrent writes, as well.

Figure 4 shows the scalability of LogicBlox 4.0 using a micro-benchmark, where we show the number of trivial read/write transactions per second on an small database, while increasing the number of threads available to the runtime. The blue and yellow lines show that for concurrent access with all readers, or one writer along with readers, LogicBlox 4.0 achieves nearly perfect, linear speedup as more threads are allocated to it. The green line demonstrates the impact of performing ad-hoc queries without pre-compilation; its performance indicates the cost of compiling queries. The ad hoc query scenario is a less likely scenario for users of deployed applications, and is more likely to impact application support personnel, who may use ad hoc queries to inspect the state of an application.

Figure 8. Perfect scaling of read/write concurrent transactions

Perfect scaling of read/write concurrent transactions

We also include three benchmarks showing transaction throughput at various levels af connectivity:

  • Figure 5 demonstrates read-only transaction throughput of LogicBlox 4.0 when transactions are executed directly against the lowest level, C++ API into the workspace. This benchmark provides a good indicator for what the runtime is capable of, independent of the overhead imposed by higher level layers such as the database server (ConnectBlox) layer and the service container (BloxWeb) layer.
  • Figure 6 demonstrates the throughput of a simple read-only transaction, when executed through the database server layer. Observe that we maintain a near perfect scaling to the number of threads. However, compared to Figure 5, one can observe around 25% reduction in transaction throughput in this scenario, which can be attributed to the overhead of the database server, communication over TCP sockets, as well as to additional resources needed to run separate clients applications.
  • Figure 7 demonstrates the throughput of a slightly more involved read-only transaction, which takes an input parameter from the client and producing output data. Again, while maintaining near-perfect scaling with respect to the number of threads, additional overhead of marshalling input and output data decreases performance.

Both Figure 6 and Figure 7 further demonstrate that LogicBlox 4.0 achieves orders of magnitude improvement in transaction throughput over LogicBlox 3.10.

Figure 9. Transaction throughput at lowest database connectivity layer

Transaction throughput at lowest database connectivity layer

Figure 10. Transaction throughput of a simple read-only transaction through ConnectBlox server layer

Transaction throughput of a simple read-only transaction through ConnectBlox server layer

Figure 11. Transaction throughput of a more involved read-only transaction through ConnectBlox server layer

Transaction throughput of a more involved read-only transaction through ConnectBlox server layer

On Simplifying Programming Model and Database Administration

LogicBlox 4.0 includes changes to the programming language that simplifies the number of choices a programmer has to make to produce a correct, performant program.

Simplified numeric types

LogicBlox 4.0 supports one simple decimal type, which is a fixed-point decimal type that replaces floating point decimals decimal[64] and decimal[128]. Fixed-point decimal provides exact representations for additions and substractions, and allows for optimization for certain aggregations. It is recommended over floating point types, i.e. float[32] and float[64].

Furthermore, LogicBlox 4.0 implements aggressive data compression. Thus, it is no longer necessary to declare integer types of various bit-width in an attempt to save storage space. We encourage the use of int[64].

Simplified predicate properties

LogicBlox 4.0 no longer requires the declaration of certain predicate properties regarding data storage and locking. The following predicate properties are no longer necessary, and will be removed in subsquent 4.x releases:

  • entity capacities
  • storage models
  • locking policies
  • scalable types
  • supplemental indexes

On automatic query optimization

We conclude with a benchmark illustrating the effectiveness of automated query optimization provided by LogicBlox 4.0. The first two columns of Figure 8 ("query1-first" and "query1-second") show the running times of a query on 3.10 as well as 4.0, the first time it is run, and the second. The second two columns ("query2-first" and "query2-second") show the running times of logically the same query, but hand-tuned for 3.10.

The chart illustrates that without programmer hand-tuning a query, LogicBlox 4.0 achieves 4x performance gains over LogicBlox 3.10. After going through the effort of hand tuning, however, LogicBlox 3.10 is able to evaluate the query faster than the non-parallel LogicBlox 4.0 (but still slower than the parallel version, expected for subsequent 4.x releases).

The key takeaway from the benchmark is that LogicBlox 4.0 is capable of providing very good performance without incurring the cost of programmers hand-tuning their queries. In addition to allowing programmers to express logic more naturally, system-generated optimizations is capable of adapting itself, and re-optimize, to the characteristics of the data as they change in the database. We encourage programmers to evaluate the performance of their natural queries on LogicBlox 4.0 without applying any manual optimization.

Figure 12. Effect of Tuning

Effect of Tuning

Known Limitations and Discontinued Feature

LogicBlox 4.0 supports applications build on the service-oriented framework only. Blade-based applications are not currently supported.

Additionally, the following features are not supported, but are planned for subsequent releases:

  • Delimited File Services can only be used to import/export simple files. Features such as error reporting or optional columns are not yet supported.
  • Mathematical programming and machine learning extensions.
  • Default value predicates.
  • Ordered entities.
  • Replace and remove block.
  • choice and ambig predicate-to-predicate mappings.
  • String concatenation aggregations.
  • argmin and argmax. To get the argument, an additional join can be used.
  • There is limited support for user-level logging. More detailed logging levels will be supported in subsequent releases.
  • Strings stored in predicates can be at most 255 characters long. Post-4.0 versions will remove this limit.
  • Predicates can have arity at most 64. Post-4.0 versions will remove this limit.
  • A few primitive predicates are missing. These include: boolean:hash, int64:hash, floatXX:hash, datetime:hash, float64:isFinite, and floatXX:round2.
  • We do not detect whether transactions that are marked as read-only are indeed read-only. Possible changes by transactions marked as read-only are most of the times not committed into the database; however, once in a while we do commit read-only marked transactions (so that we can make subsequent use of indices created in the transaction). Until we verify that read-only marked transactions do not change the database, it is the responsibility of the user to do so.
  • For functional EDB predicates, retractions are performed based on key match only, rather than on key-value match. For example, -a["foo"]="bar" removes a["foo"]="quz"; i.e., it behaves like -a["foo"]=_. This will be corrected in a future release.

The following features are removed, and will not be supported in subsequent releases:

  • Floating-point decimal primitive types (decimal[64] and decimal[128]) are replaced by a single fixed-precision decimal type.
  • Primitive type color is not supported.
  • Meta-predicates such as system:Predicate are not supported. Future versions of meta-predicates will likely differ substantially from how meta predicates are handled in 3.X.
  • MoReBlox. While continuing to be supported in the LogicBlox 3.x series, support for generic programming in future releases is undergoing a redesign process and will be different enough that existing MoReBlox programs will require a rewrite.
  • MochaBlox and WrappedControls. While continuing to be supported in the LogicBlox 3.x series, we encourage application programmers to choose the user interface framework of their choice, and communicate with the workspace over the service interface.

Installation and Upgrade information

Installing LogicBlox 4.0 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.0.0/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Release Information

Table 9. 

Server requirements
Operating System: 64 bit Linux
Java Runtime Environment 1.7, update 11 or higher
Python 2.6.4