LogicBlox 4.4.6

Release Date: August 3rd 2017

Executive Summary

LogicBlox 4.4.6 introduces the following enhancements:

  • The database includes parsing and formatting patterns for fractional seconds, support for generating UUID, improvements to the lb popcount and lb list commands as well as various stability improvements.

  • lb web-server includes a number of enhancements to TDX: improved expressivity in validation criteria for column data, the ability to integrate with the user permissions model and whitelisting predicates. Improvements to lb-web-client include full SSL support as well as support for Websockets.

  • The Measure Service is now more tightly integrated with lb-web's authentication and offers a number of locking related enhancements as well as some performance optimizations.

  • For lb-workflow we are introducing support for JSON structured values as input to tasks, as well as support for automatic retries for failed tasks.

  • The release also comes with some exciting features for users of Modeler-js based applications, such as:
    • Support for freezing rows and/or columns.

    • Support for CubiQL formula-defined measures.

    • Dimension splitting for recalc measures.

    • Conditional formatting based on expressions.

    • Level member creation directly from dropdowns.

    • Ability to remove a level member from its hierarchy parent.

What's New

Database

  • There are 3 new specifiers available for parsing fractional seconds:

    • %s - Matches seconds as two digits (0-59), followed by '.' (dot), and fractional seconds as 1 to 6 digits.

    • %F - Matches '.' (dot) followed by fractional seconds as 1 to 6 digits, or empty (if no leading '.').

    • %f - Matches fractional seconds as 1 to 6 digits.

    The format used by string:datetime:convert is now %Y-%m-%d %H:%M:%S%F %Q (added %F).

    For a complete overview of all the built-ins supported in LogicBlox, please visit the Reference Manual.

  • Redundant entity atoms in rule bodies are now optimized away before rule evaluation.

Language

  • UUID can now be derived in logic using the uuid series: uid<<n = uuid()>>. The value of a uuid number has the type int128, that can be converted to the string representation of UUID using predicate int128:to_uuid_string; the string representation can be converted back to an int128 value using int128:from_uuid_string.

  • It is now possible to write functional predicates with more than one value, p(x,y;z,w), relationally, p(x,y,z,w). Previously this only applied to functional predicates with a single value.

  • Writing type constraints over predicates with deltas or stage annotations do not declare them anymore.

  • The compiler reports a better error message when attempting to query the retracted key of refmode predicates in rule bodies.

Services Framework

  • TDX enhancements:
    • TDX services now support enhanced validations in the FileDefinition's format, including regular expressions, greater/lesser than, and precision.

      Example 33. 

      In the example below we show the file definition for SkuPrice. The pipe delimited file is expected to have two columns, SKU and PRICE. The values for SKU are expected to be strings, where we indicate via a regular expression that we expect the values in this column to only contain alphanumeric characters. Furthermore, PRICE is expected to be a decimal, with the following three validations specified:

      • >=0: indicating the the value must be non-negative.
      • <=20: indicating the the value must not be greater than 2000.
      • precision 2: indicating that this decimal must not have more than 2 digits after the decimal point.

        file_definition_by_name["SkuPrice"] = fd,
        file_definition(fd) {
          file_delimiter[] = "|",
          column_headers[] = "SKU,PRICE",
          column_formats[] = "string([a-zA-Z0-9]*),decimal(>=0; <=2000; precision 2)"
        }.

      Note

      Please refer to the chapter on Data Exchange Services in the Reference Manual for an overview of this feature.

    • Dynamic TDX services now support role based authorization and predicate whitelisting.

  • lb web-client improvements:
    • The lb-web client now accepts configuration for the connection timeout and the maximum number of connections to maintain to a given address.

    • lb web-client now supports full SSL, with certificate validation, using standard JDK keystore.

    • lb web-client now supports WebSockets.

  • lb-web's global configuration can now be dynamically changed via the command-line, including tracing parameters.

    Example 34. 

    Below you can find the output of the lb web-server set command, which shows the list of parameters that can be changed with their current values.

      $ lb web-server set
      ------------------------- -------
                 Key             Value
      ------------------------- -------
       log_http                  false
       log_max_message_length    -1
       debug                     false
       tracing_impl              none
       log_connectblox           false
       log_messages              false
       tcp_abort_on_disconnect   true
       profile                   false
       log_tdx_generated_rules   false
       db_log_level
       tracing_detail
       log_exception_details     false
       keep_tdx_files            false 

    A single parameter can be queried as follows:

      $ lb web-server set debug
      false

    A parameter can be set as follows:

    $ lb web-server set debug true

    A parameter can be cleared using the --unset option:

    lb web-server set debug --unset

  • Added support to return custom HTTP status code in Protobuf services without aborting the transaction.

Measure Service

  • The measure service is now more tightly integrated with lb-web’s authentication. This is mostly in preparation for data access permissions that will be available in a future release, but it is already possible to write locks that depend on the authenticated user sending the UpdateRequests.

    The measure service REPL now supports authentication with the :login, :logout, and :whoami commands. The URI of the authentication service (it defaults to /login) can be specified via the --login-uri command line parameter.

  • The new {{ install = false }} expression annotation gives finer grain control over whether an expression will be cached by installation.

  • The measure service now allows locks to be defined on all targets an UpdateRequest can modify.

  • The lb measure-service convert command now accepts an --expand option to expand any measure expressions defined using the MeasureExpr str field into their structured form.

  • The measure service now performs a one-to-one analysis so that it can mark the predicates it generates as being one-to-one. This should expose more opportunities for optimization in the runtime.

  • It is now possible to configure the size of the thread pools the measure service uses with the handler_thread_pool_size and env_thread_pool_size config file options.

  • If the log level is increased to debug, the measure service generates detailed reports explaining how the response for a given EditabilityRequest was derived.

  • The measure service now caches some of its internal context types, which should improve performance especially in the desugaring and optimization phases.

  • The measure service model now maintains a version number. Changes to model predicates will increment the version. As part of measure service requests the version will be checked and the measure service will automatically refresh its internal model if the version number differs.

    Currently this feature is disabled by default to avoid any unexpected interactions with existing projects. To enable it set autorefresh_model = true in the measure service configuration file.

    The Response message now also includes the version number of the model that the measure service is currently using. However, this number will not change unless auto-refresh is enabled, as we will not install the logic to do so otherwise.

  • In the measure language protocol, VarExpr has been renamed to VariableExpr to avoid overlap with the CubiQL VarExpr message.

  • The measure service will now use read-only transactions to retrieve the set of installed blocks.

  • Locks in the model are now desugared at load type, so it is possible to use various sugared forms when writing locks that would have caused failures before.

  • We now better cache some internal structures to make rewriting and some other tasks more efficient.

  • Some initial optimizations have been made to reduce model refresh latency.

Workflow

  • Lb-workflow now supports JSON structured values as input to tasks.

  • The lb-workflow driver will now poll multiple workspaces in parallel, so that slow transactions from one workspace will not hold other workspaces' tasks to run.

  • Support for automatic retries for failed lb-workflow tasks.

    Example 35. 

    The example below shows the specification of a json task, that is configured to get retried up to 5 times if it fails, with an interval of 1000ms:

    workflow json()[json_output] {
      ($json_output = output) <~  lb.JsonService(
        uri="/my_service",
        method="GET",
        driver_meta={"retry": 5, "retry_interval": "1000ms"},
        transport="http://localhost:8080")
    }

  • The lb.ExecuteLogiQL task now supports a file parameter, to indicate the file in which the logic can be found that needs to be executed. In earlier releases, the logic had to be included in the task definition.

Modeler-js

  • Freeze Pane: It is now possible freeze columns and/or rows to make sure that they stay visible while you are scrolling. The options to freeze rows, columns or both, as well as to unfreeze them, are available as a right-mouse click option in the grid.

  • CubiQL formula-defined measures: Developers of Modeler-js based applications can now also write CubiQL formulas instead of only Measure Language rules to define how a measure's value needs to be calculated.

  • Dimension splitting for recalc measures: Support for dimension splitting, where users can view a measure's values by aggregating a key along two different hierarchies of the same dimension, has been around for a while already, except for measures that aggregate via a custom formula (also called recalc measures, as their value gets recalculated at each aggregate level). This restriction has been removed now, thereby supporting dimension splitting for all measures, independent of their aggregation method. Note that the recalc formula has to be specified via CubiQL and not via the measure language.

  • Support for position-only filters: Filter predicates, for instance mask filters, can now also be position-only predicates and do not need to be declared as booleans anymore. To define a position-only predicate via the Measures.csv file, the DataType column for that predicate needs to be left empty.

    Example 36. 

    Below we show two examples. The first rule is for a position-only filter while the second one illustrates the definition of boolean filter that was required in earlier releases.

    //position-only filter predicate
    RedSkus(sku) <-
      SkuColor[sku] = "red".
    
    //boolean typed filter predicate
    RedSku[sku] = true <-
      SkuColor[sku] = "red". 

  • Conditional formatting based on expressions: Conditional formatting has been extended to include the possibility of formatting a measure based on the value of another measure. The measure that is used in the expression does not need to be visible on the sheet, as long as it is a displayableMeasure.

    Example 37. 

    In the example below, we specify the background color of the SkuComplete measure, if the value is false. Conditional formatting for the measures SkuDescription and Product:Sku:color is specified such that they are displayed with a background color, if the value of SkuComplete is equal to false.

    ...
    ,"gridOptions": {
      "conditionalFormatting": {
        "SkuComplete": [
            {
                "condition": "=",
                "value": false,
                "style": {
                    "background": "#ffffe0"
                }
            }
        ],
        "SkuDescription": [
            {
                "conditionalMeasure": "SkuComplete",
                "condition": "=",
                "value": false,
                "style": {
                    "background": "#ffffe0"
                }
            }
        ],
        "Product:Sku:color": [
            {
                "conditionalMeasure": "SkuComplete",
                "condition": "=",
                "value": false,
                "style": {
                    "background": "#ffffe0"
                }
            }
        ]
      }
    },
    ....

    The figure bellow illustrates how the formatting of of these measures changes, based on changes to SkuComplete. In this example, a SKU is deemed as "complete", once it has both a description and a color assigned.

  • Ability to remove a level member from its hierarchy parent: In an earlier release we have introduced a feature that allows users to assign a level member to a hierarchy parent directly from the grid. In this release, we have added support for allowing users to remove a level member from its hierarchy parent.

  • Level member creation directly from dropdowns: Dropdowns are used to display measure values that are entities. Now users can directly create a new level member when entering a value for the measure, that does not yet exist. If a user chooses to create a new level member, the creation form for creating level members for the level is displayed.

    Note

    To enable this feature in a modeler-based application the editSchema property needs to be set to true in the modelingFeatures section which is a part of the configuration passed to AuthenticatedModelerController.

  • Navigation tree enhancements: A navigation item can now contain a URI to a page inside the Modeler as well as to an external resource.

    Example 38. 

    To define a URI to some page, an object of the following shape can be added to the navigation JSON:

    {
        "id": "navigation-item-with-url",
        "properties": [{
            "key": "type",
            "value": "url"
        }, {
            "key": "url",
            "value": "/my/navigation/url"
        }, {
            "key": "caption",
            "value": "Navigation Item With URL"
        }]
    }

    As a result the link will navigate the user to: "/:workbookId?/app/:appId" + "/my/navigation/url".

    It is also possible to add a link to an external resource:

    {
        "id": "navigation-item-with-external-url",
        "properties": [{
            "key": "type",
            "value": "url"
        }, {
            "key": "url",
            "value": "http://logicblox.com/"
        }, {
            "key": "caption",
            "value": "LogicBlox"
        }]
    }

    In this example, we have added a direct link to the LogicBlox website from the navigation tree.

  • JSON Sheet View: It is now possible for users to see and edit the JSON of their sheet configurations, which can be in particular useful during development. This feature can be enabled if desired via the jsonSheetViewEnabled property in the modelerConfig object. This will add another button on the sheet menu bar that switches the sheet to JSON view.

  • Reorganization of Modeler libraries: The Modeler project has re-organized it's LogiQL libraries in order to better support partitioned deployments.

    Note

    Please follow the upgrade steps listed in the the Upgrade Information section.

  • Partitioned Deployments: In order to support partitioned deployments, the services in the modeler_platform library are designed to run in a separate workspace and therefore do not need to be partitioned. To do so:

    • Create a library that includes the modeler_platform library that is intended to be hosted in a non-partitioned workspace.

    • Configure the authentication realm for these services by writing to modeler_init:authentication:app_realm.

    • Configure the URL prefix realm for these services by writing to modeler_init:prefix:prefix.

  • Custom application inside Modeler: We've added an experimental feature allowing the development of a custom application inside Modeler.

    Note

    This feature is still under development and the API we introduce is likely to be changed.

    You can connect your custom application to Modeler by passing your CustomApp view to the modelerConfig.views.

    {
        ...
        views: {
            customApp: CustomApp
        }
    }

    The CustomApp can be accessed then by the following route: /:workbookId?/app/:appId/custom. The list of props that will be passed to your custom application is the following:

    CustomApp.propTypes = {
        /* The top position of the container. */
        top: PropTypes.number.isRequired,
        /* The bottom position of the container. */
        bottom: PropTypes.number.isRequired,
        /* The left position of the container. */
        left: PropTypes.number.isRequired,
        /* The right position of the container. */
        right: PropTypes.number.isRequired,
    
        /* The height of the container. */
        height: PropTypes.number.isRequired,
        /* The width of the container. */
        width: PropTypes.number.isRequired,
    
        /*
         * The term "history" and "history object" in this documentation refers to the history package,
         * which is one of only 2 major dependencies of React Router (besides React itself),
         * and which provides several different implementations for managing session history in JavaScript in various environments.
         **/
        history: PropTypes.object.isRequired,
        /* Locations represent where the app is now, where you want it to go, or even where it was. */
        location: PropTypes.object.isRequired,
        /* A match object contains information about how a <Route path> matched the URL. See react-router docs. */
        match: PropTypes.object.isRequired,
        /* A params object contains all route params collected from all parent routes. */
        params: PropTypes.object.isRequired,
        /* A routeHelper object allows you to navigate between different routes. See CHANGELOG for the list of all available modeler routes. */
        routeHelper: PropTypes.object.isRequired,
    
        /* A modelerApp object has access to dispatcher, urlDriver, all stores and actions of the modeler application. */
        modelerApp: PropTypes.object.isRequired,
        /* The initial configuration of the modeler application. */
        modelerConfig: PropTypes.object.isRequired,
    };

    While developing the custom application you're likely to need several libraries such as React, ReactRouter, ReactRouterDOM, Immutable. You don't need to include them into your bundle for the CustomApp because we export it globally.

Developer Tools

  • The lb list and lb popcount commands now both support an additional option --kind to limit the kind of predicates to be listed (for instance only EDBs or IDBs). The commands do not list predicates anymore that are generated internally by the runtime.

Corrected Issues

The issues listed below have been corrected since the 4.4.5 release:

  • Resolved a data structure (alpha-tree) related bug that caused Unexpected missing versions and no tree has been added for key errors. A related warning for Discrepancy in changes is now an error. Please contact us for support if you receive this error on 4.4.6 or later.

  • Fixed performance issues in a component called the refgraph that caused excessive memory usage for some workloads.s

  • Fixed a bug in the evaluation of rules involving default values that caused Unknown variable errors.

  • Fixed a bug in retractions on scalar predicates that caused Atom … does not match predicate signature errors.

  • Fixed a correctness issue in the maintenance of rules involving negated formulas.

  • Using lang:oneToOne on functional predicates with more than one value now works as expected instead of reporting an error.

  • Using lb compiler --serialize now generates the correct LB0 envelope message.

  • Fixed an issue in lb-web client that could cause threads to accumulate if the scheduler created to monitor transaction timeouts was not properly shutdown.

  • Fixed an issue that prevented lb web-server from logging HTTP requests for services that disabled pulsing the HTTP control messages.

  • Various checks relating to delta logic and stages were not being properly performed, and are now correctly checked. As such it is possible, though unlikely, that developers may have written constraints that were in the past allowed that will now cause a compiler error.

  • Fix for type dependencies and some predicate dependencies that were not being checked for before removing blocks.

  • Fixed invalid HTTP status code for unauthorized WebSocket access.

  • The measure service may now return an HTTP 400 status code instead of HTTP 500 in various situations.

  • Modeler-js:
    • By-value filters now get cleared when the intersection of the table changes.

    • Resolved an issue where a measure could not be removed from the Visible Measures panel when it had a filter applied and the filter panel was open.

    • Resolved an issue with the creation forms, where the Create button could be enabled under certain circumstances, without having populated all the required fields.

    • Resolved an issue where the order attribute of an entity was not respected when sorting the grid on a column with an entity-based measure.

Installation and Upgrade information

Installation Instructions

Install LogicBlox 4.4.6 by following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.4.6/etc/profile.d/logicblox.sh
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Upgrade Information

  • The library containing the modeler migration tool is included in the distribution under lib/npm/modeler-migrations-<version>.tgz and contains a command line tool for doing various transformations/migrations of modeler configurations.

    To use the npm module, you first need to install it by running npm install path/to/modeler-migrations-<version>.tgz. You will then be able to use the command line tool. The module contains a README.md file that lists all available transformations as well as detailed instructions for use.

    To automatically upgrade a Modeler-js based application from one version to another, run the upgrade script to upgrade between versions:
    ./node_modules/.bin/migrate-modeler --fromVersion <prevVer> --toVersion <toVer> /path/to/my/modelerapp
    where the version numbers refer to LogicBlox releases.

    Note

    If you installed the npm module globally, you won't need the ./node_modules/.bin prefix on the command.

    Example 39. 

    To upgrade a Modeler-js based application from LogicBlox 4.4.3 to 4.4.6, you can run the following command:

    ./node_modules/.bin/migrate-modeler --fromVersion 4.4.3 --toVersion 4.4.6 /path/to/my/modelerapp

  • Library Changes: Due to the Modeler library reorganization, the modeler_platform project needs to be added to every Modeler-js based LogiQL project.

    Example 40. 

    Before:

    my_project,projectname
    
    lb_web,library
    lb_measure_service, library
    pivot, library
    modeler_config, library
    generated_schema, library

    After:

    my_project,projectname
    
    lb_web,library
    lb_measure_service, library
    pivot, library
    modeler_config, library
    modeler_platform, library
    generated_schema, library

  • Deprecations: The predicates in the 1st column have been deprecated. Their replacements are listed in the second column. For now, all the deprecated predicates either derive into the new versions or are no-ops if no longer used. The old predicates will be removed in the next major version change.

    Deprecated Predicate New Predicate
    modeler_config:services:connectblox:enable_internal modeler_init:connectblox:enable_internal
    modeler_config:services:connectblox:enable_public modeler_init:connectblox:enable_public
    modeler_config:services:authentication:modeler_app_realm modeler_init:authentication:app_realm
    modeler_config:services:authentication:use_sso modeler_init:authentication:use_sso
    modeler_config:services:measure_service:allowed_origin No longer used
    modeler_config:services:prefix:master modeler_init:prefix:prefix
    modeler_config:services:prefix:global modeler_init:prefix:workbook_prefix
    modeler_config:services:prefix:service modeler_init:prefix:service
    pivot:config:service_config:enabled No longer used
    pivot:config:service_config:modeler_prefix modeler_init:prefix:workbook_prefix
    pivot:config:service_config:modeler_realm modeler_init:authentication:app_realm
    pivot:views:* modeler_platform:views:*

    You can remove any call to the modeler_config:services:workbook block in your config.py. The block is now active and no longer needs to be called as part of the build process. An empty block has been left for backwards compatibility but will be removed at the next major version change.

  • Address Mapping: For projects using the Address cell type released in LB 4.4.4, the block that needs to be executed during build time has changed.

    Note

    Projects that do not use the Address cell type do not need to perform these changes.

    1. The following change will need to be made in config.py:

      'lb execblock ' + WS + ' pivot:config:addressmodel:address_mapping'

      needs to be changed to:

      'lb execblock ' + WS + ' modeler_config:metamodel:address_mapping'

    2. Workflows/batches that call the address services will need to be updated, too. They changed from:

      /addressmodel/address_def
      /addressmodel/country_config

      To:

      $prefix/addressmodel/address_def
      $prefix/addressmodel/country_config

      Where $prefix is the same modeler_prefix used for every other service in the application.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 8
Python 2.7 or higher
Client requirements
Applications using modeler-js User Interface Components: Google Chrome or Internet Explorer 11+
Requirements for applications using non-modeler-js components may vary per application.