LogicBlox 4.3.7

Release Date: March 1st 2016

What's New

Database
  • The performance of frame rules has significantly improved for larger incremental data loads. There was a problem with the parallel evaluation of frame rules, which would be used for larger deltas. As part of this tuning we have also lowered the delta size limit for the usage of parallel evaluation. This will mean better core utilization for some applications.

LogiQL Language
  • Bitwise integer operations:

    • LogiQL now supports "unsigned" hexadecimal and binary literals (0xABCD and 0b1010111).

    • Added support for several new bitwise operations on integers.

    Note

    Please refer to the Reference Manual for an overview of all the integer bit manipulation functions.

modeler-js User Interface Components
  • Vega Charting Plugin: The new Vega Charting plugin allows developers to configure a sheet containing a chart. Chart configuration must be specified using the Vega visualization language. Data used to populate the chart can be specified using the sheet configuration language already supported for modeler-js sheets. Charts can be configured to interact with other visualization components, such as grids. When linked, changing the data in grids will cause chart displays to change and reflect the new data, as well.

    Example 31. 

    The figure below illustrates how a Pareto chart can be displayed on the same canvas as a grid. The grid and the chart interact with each other, allowing the user to immediately see changes in the grid being reflected in the chart.

  • Locking: This feature allows an end user to lock certain values, such that they are not affected by the spreading that may result from other edits. A lock applies to any edit performed by the user, regardless of on which view the edit is performed. Locks persists throughout the user’s browser session. Locks will be reset upon closing of the browser tab.

  • Individual resizing of rows and columns: Resizing can be now be done individually for the innermost headers (i.e. those next to the data cells). The outer headers are resized based on the inner headers. For cross producted headers all instances of the resized header will be resized. To resize all columns, users can hold down the <CTRL> key while resizing.

    Developers can also configure the individual column/row sizes via the pivot configuration.

    Example 32. 

    The example below allows to apply custom height to subclass1 and width for the Sales column. The members object is used to define the innermost headers sizes. The rollups object is used to define sizes for aggregated heading:

    {
        "gridOptions": {
            "display": {
                "Product:subclass": {
                    "members": {
                        "subclass1": {
                            "height": 50
                        }
                    },
                    "rollups": {
                        "Product:class": {
                            "class1": {
                                "height": 60
                            }
                        }
                    },
                },
                "-:Measures": {
                    "members": {
                        "Sales": {
                            "width": 99
                        }
                    }
                }
            }
        }
    }

  • Logout notification over multiple tabs: The logout notification now also works over multiple tabs. The new configuration option ignoreBroadcastingTime allows developers to configure after how many seconds of idle time the processing of broadcasting messages should stop.

    The notification around user inactivity and automatic logout can now be configured as follows:

    modelingFeatures: {
        ...
        idleTimeValues: {
                //in seconds
                idleNotificationTime: 1800,  //in seconds
                idleLogoutTime: 60,		 //in seconds
                ignoreBroadcastingTime: 300  //in seconds
            }
        ...
    }

  • Level mapping measures: The measure model configuration utility distributed with modeler-js now defines all level mappings to be measures. Every level, consequently, is also declared to be an intersection.

    Example 33. 

    For instance, if a Product hierarchy contains levels Product:Sku, which maps to Product:Subclass, then a measure, Product:Sku:subclass, is automatically configured, backed by the predicate Product:Sku:subclass.

    For level Product:Sku the corresponding intersection, Product:Sku, is declared and can be used to define measures.

  • Improved Dimension Browser: The new Dimension Browser is a more compact representation of all the levels in the schema.

  • Warmups: The new lbconfig.modeler python module can be used for generating warmups for both measure rules and views.

    Example 34. 

    To use this new module, the following needs to be added to the config.py script:

    from lbconfig import modeler
    
    modeler.configure()

    This will create targets to copy over all assets, JavaScript libraries as well as scripts from the platform build installation. For warmups it will create the following make targets:

    • rule-queries: generates queries for rule warmup

    • view-queries: generates queries for view warmup

    • warmup-rules: submits the generated queries to warmup rules

    Note

    Application developers are advised to using this module instead of application specific scripts, to be able to automatically benefit from future enhancements.

Services Framework
  • lb-web allows services to declare an inactive block to be executed in the same transaction as a request. It is now possible to declare multiple such inactive blocks, which allows handling code to be better modularized and reused.

  • The raw file-mode is now supported for TDX imports. It was previously only implemented for TDX export. The raw file-mode does no escaping or unquoting whatsoever, but does not support the delimiter in fields.

  • The lb-web server now prints a clear message when startup is done and it is ready to serve requests.

    INFO lb-web         	- lb web-server initialized and ready to accept requests (code: LB_WEB_INIT_COMPLETE)

  • New json_printing option is now available for protobuf services to configure whether strict or loose mode should be used.

  • It is now possible to configure the force_authn flag on the SAML request that lb-web sends to IDPs when a user attempts to log into a SAML realm. This forces the IDP to re-request authentication even if the user has an ongoing session with the IDP.

  • lb-web services normally obtain the username for a service request from request parameters that are imported into the database using a protobuf message. In active logic, this populates the pulse predicate lb:web:auth:username. We earlier introduced an optional setting enable_pulse_auth on services, which directly populates this predicate at stage initial. This makes the username also easily accessible to inactive blocks. This feature is now also available for TDX services (earlier it only was supported by protobuf services).

  • New features to control lb web-server logs:

    • A new log_file config property can be used to determine where to store the logs.

    • The new systemd property (also settable via the IN_SYSTEMD environment variable) determines 1) whether logs will contain timestamps (if in systemd timestamps are redundant so they are omitted) and 2) if explicitly set to false, then even when running as a systemd unit, the specified log_file will be used. This can be convenient for experiments that produce very large logs.

    • The log_elapsed flag makes the logs contain an additional column with the number of milliseconds since startup.

    • UTC is now the default timezone for timestamps in lb-web server and client, including server access logs. There's a special host timezone that can be used to select the default timezone of the running host.

Measure Service
  • It is now possible to use a CubiQL expression as the input to an update rather than a LiteralTypedRelation.

    Note

    Please refer to the the section called “Upgrade Information” for more information on the deprecation of LiteralTypedRelation.

  • Optimizations for default-valued boolean min/max aggregations.

Workflow

  • lb workflow is now integrated into the lb command line.

  • The new command lb workflow purge allows the removal of workflow instances from the workspace.

    Example 35. 

    You can use the following command to remove all workflow instances of the daily workflow that are older than 7 days:

    $ lb workflow purge --name daily --older-than 7days

    Example of removing a workflow instance with a specific ID:

    $ lb workflow purge --root 10000000063 

    It is also possible to remove all workflow instances at once:

    $ lb workflow purge --all

  • All duration values in lb workflow tasks can now be specified as strings that contain the value plus a unit. For backwards compatibility, tasks still accept integers, which will be interpreted in the same unit as before (seconds for most tasks; milliseconds for lb.tdx.* tasks).

    Example 36. 

    The expected syntax is an integer (specifying the length) followed by an optional resolution specification. The supported values are as follows:

    • ms: milliseconds

    • s: seconds

    • m: minutes

    • h: hours

    • d: days

    • M: months

    • y: years

    Examples:

    "5"  	 //5 milliseconds for lb.tdx.* tasks, 5 seconds otherwise
    "50ms"   //50 milliseconds
    "4h"    //4 hours

    Note

    Please refer to the the section called “Upgrade Information” for more information on the deprecation of using integers as the duration values.

  • It is now possible to list workflow instances using lb workflow status --list-roots. This will use the status service to lookup the roots of workflow instances and list them. It can be used together with the --name option to select instances of a particular workflow.

  • lb workflow has now command line completion like all other lb commands.

    Tip

    To enable locally, source [path_to_lb_home]/etc/bash_completion.d/logicblox.sh in your bash profile or similar.

  • It is now possible to pass the --failed-restartables option to the lb workflow status command to get the status for the restartables that have a failed descendant.

  • The new workflow lb.wb.SetUsers can be used to add and remove users that have access to workbooks in one transaction. Previously, these activities had to be performed via two separate tasks.

Workbook Framework

  • Cross-branch Commit/Refresh: This feature supports an alternative method of performing commit/refresh between workbooks. Whereas the default method of commit/refresh currently is through exchanging data between master and workbooks by exporting/importing CSVs, the cross-branch commit/refresh executes cross-branch logic: rules that refer to data in one branch of the workbook, and derive it into another branch.

    Cross-branch commit/refresh represent performance wins, but are only applicable for workbooks that are co-located on the same instance as their parent.

Developer Tools

  • The lb export-workspace command now also works on secondary workspaces.

  • Various improvements to the profileDiskSpace command:

    • The --branch option can be used to show disk usage only for a specific branch of the workspace.

    • The --versions option has been improved to show the total number of reachable pages for every branch (earlier it only showed differences between versions in the listing).

    • The command can now be used online using lb batch-script. It does block write transactions.

  • lb print-rules (introduced in the 4.3.6 release) now also shows delta rules when applied to EDB predicates.

  • All command line arguments that accept a log level now have a better help message with some examples.

  • Several lb commands that accept a --csv parameter to create a CSV file now support a --delimiter parameter to configure the file's delimiter. This includes lb exec, lb query, lb execblock, lb workbook list and list-templates.

  • lb info now supports the --json option to print the information in JSON format.

Corrected Issues

The issues listed below have been corrected since the 4.3.6 release.

  • Resolved an issue introduced in 4.3.6 where some disjunctive rules would not evaluate correctly. The latest 4.3.6 patch releases also included this fix.

  • The lb batch-script command did not correctly use security tokens and required the LB_CONNECTBLOX_ENABLE_ADMIN environment variable to be set. This is not the case anymore.

  • Resolved an issue where lb export-workspace would hang if the lb-server was terminated unexpectedly during the export.

  • lb export-workspace now respects the port configuration in lb-server.config.

  • A transaction abort in earlier releases would print an ERROR line in the log, which complicated monitoring for errors. This error line has been removed.

  • Resolved an issue with executing write transactions on ws vs. ws@master, which are both the same branch. The transactions would occasionally fail with an error that a concurrent write is detected.

  • The composition of configuration files in lb-web-server has been fixed. Previously, configuration sections in lb_deployment would only be able to send new options to handlers, but never change an existing one. Also, dynamically loaded jars and config files would completely overwrite anything from lb_deployment. This has been fixed such that lb_deployment files have precedence over jar config files, and dynamically loaded config files have precedence over lb_deployment.

  • modeler-js User Interface Components:
    • Resolved an issue that could cause a pivot grid to be rendered empty for incorrectly selected slices on initial render when the slice was set via a measure.

    • The scroll position is now correctly maintained when the user switches between views.

    • Resolved an issue that could cause an incorrect sorting behavior when editing a previously sorted column.

    • Resolved an issue that caused pending calculations not to be marked as such when switching between views.
  • Measure Service:

    • Resolved issues with computing updates with respect to locked recalc metrics.

    • Improved handling of aborts and no longer blocks lb-web threads to handle requests.

    • Resolved an issue that could cause a crash stemming from invalid relabel and split expressions.

Installation and Upgrade information

Installation Instructions

Installing LogicBlox 4.3.7 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.7/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Upgrade Information

  • Java 8: LogicBlox requires Java 8 starting with the 4.3.7 release. All Java code is compiled for Java 8, so on older Java versions the components will not run (e.g. the compiler server). If accidentally deployed on an older Java version, then an error "Unsupported major.minor version 52.0" should be reported by the JVM.

  • Workflow:

    • The lb-workflow command has been deprecated in favour of lb workflow.

    • All duration values in lb workflow tasks should now be specified as strings containing the value and their unit (e.g., "300ms", "5s", "2h", etc). The use of integers is discouraged and the compiler and driver will raise deprecation warnings.

  • All uses of QueueTransport and SQS support for transports has been removed: Only TCP is now available.

  • Deprecated Measure Service Protocol Changes: To facilitate work on removing deprecated features from projects, we re-announce in this release previously-announced deprecations, accompanied with migration advice. We aim to remove them in the 4.3.9 release.

    To check for absence of these deprecated features in a project, set fatal_warnings=true in the project's lb-measure-service.config.

    • ad hoc fields for textual CubiQL are deprecated (since 4.3.3): Most string fields that have been variously available to hold CubiQL expressions in textual form are now deprecated in favor of replacement fields holding a MeasureExpr.

      For example, expr_str in ExprSubst message is deprecated in favor of the expr field and measure_str in QueryRequest in favor of measure.

      Since MeasureExpr now has the str string field for textual CubiQL, the transition amounts to moving the content of the deprecated field to the replacement field, after wrapping it as a MeasureExpr.

      Example 37. 

      For example,

      { "query_request": {
                "measure_str": ["1+2", "total Sales"]
              }}

      becomes

            { "query_request": {
                "measure": [{"str": "1+2"}, {"str": "total Sales"}]
              }}

    • map_pred is deprecated in Edge and Path (since 4.3.3): The string field map_pred is deprecated in favor of the Predicate field pred, in the Dimension.Edge and Dimension.Hierarchy.Path messages.

      Projects that use the config library do not need adjustments to deal with this deprecation (the config predicates dimension_hasEdge and levelMap have been adjusted to produce the pred field). Clients that create protobuf directly should change, e.g.,

      path {
        map_pred: "MyPred", 
        //… 
      }
      to 
      path {
        pred: {name: "MyPred"}, 
        //… 
      }

    • pred field in ExprSubst is deprecated (since 4.3.3): The field pred holding a Predicate within an ExprSubst entry in a model has been deprecated. It was used to specify a predicate in the workspace to be used as a replacement of a CubiQL expression, instead of any optimization that measure service would compute for it. The preferred alternative in protobuf is the replacement_expr field that holds a MeasureExpr, with an anonymous Metric defined for the predicate. In the configuration library, the alternative would also require defining a named metric in the model based on the predicate.

      Example 38. 

      For example, if a substitution was defined as

        +measure:config:exprSubst_replacesExpr["total Returns by to Location.region", false]=subst,
        +measure:config:exprSubst(subst) {
          +measure:config:exprSubst_usesPredicateName[]="pdx:sku:region:week:returns"
        }.

      then the new definitions would be

        +measure:config:exprSubst_replacesExpr["total Returns by to Location.region", false]=subst,
        +measure:config:exprSubst(subst) {
          +measure:config:exprSubst_usesExpression[]="TotalReturnsToRegion"
        }.

      with the named metric for the predicate defined as

        +measure:config:metric(TotalReturnsToRegion) {
          +measure:config:metric_hasName[]="TotalReturnsToRegion",
          +measure:config:metric_hasSignature[]= +measure:config:baseSignature(sig){
            +measure:config:baseSignature_singleton("sku,region,week", "FLOAT")},
          +measure:config:metric_usesPredicate[]="pdx:sku:region:week:returns"
        }.

    • metric field in UpdateExpr is deprecated (since 4.3.4): metric field in UpdateExpr is deprecated, as it is now a special case of the target field holding an UpdateExpr.Target message.

      Example 39. 

      For example, an UpdateExpr

        {"kind": "SPREAD", 
         "metric": "Sales", 
         //...
        } 

      should now be written

        {"kind": "SPREAD", 
         "target": {"metric": "Sales"},
         //...
        } 

    • signature field in UpdateExpr is deprecated (since 4.3.7): The field signature in UpdateExpr is now deprecated as it can be inferred from the source or input fields of the enclosing UpdateRequest.

    • source field in UpdateRequest is deprecated (since 4.3.7): The field source in UpdateRequest that holds a LiteralTypedRelation with input data for the update is now deprecated. It is replaced with the new field input, holding a MeasureExpr. It is more general, in allowing the input data to be computable. The same effect as source can be achieved by using a ParamExpr as input, coupled with a RelationBinding in the enclosing Request, to hold a LiteralTypedExpression with the data. A LiteralExpr (rather than a ParamExpr) could be used as well, but in this case the data would be installed as logic. Unless these data are used more than once, this would likely be undesirable.

    • Using LiteralTypedRelations as the input for updates is deprecated: To eliminate the use of LiteralTypedRelations as the input of updates, use a ParamExpr as the input expression.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 7, update 11 or higher
Python 2.7 or higher
CPU compatible with the corei7 architecture
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.