LogicBlox 4.3


LogicBlox 4.3.17

Release Date: December 1st 2016

Executive Summary

LogicBlox 4.3.17 introduces the following enhancements:

  • The measure service includes a number of robustness and feature enhancements to support the expanded usage of CubiQL expressions.

  • The workbook framework now supports a new merge method for merging workbooks with the master workspace, which only transfers data that changed in the source since the last merge was performed.

  • Workbook template instantiations can now be tagged, to support querying of only a subset of all the possible instantiations of a template.

  • Modeler-js based applications should see performance improvements to various features, as well as a lower memory usage due to some bug fixes.

  • Additionally, the shortcut button functionality was extended to support overriding existing pivot configurations.

What's New

Database
  • We enhanced the perf logging level to additionally log the total duration of request (next to the existing transaction duration). For transaction requests, the logging includes information on whether a transaction is read or write. For all request, high-level information on the type of request is logged. The logging of request durations now also logs operations like creating or exporting a workspace, adding a mirror etc.

Measure Service
  • It is now possible to use entities defined by constructors in many places within the measure model and queries. If it is not possible to unambiguously infer the constructor predicate to use, it must be supplied via the conversion_pred field of the TypeDefinition message.

    Note

    Note that there is currently no support for sorting (along with header sort, key sort, etc.) on entities defined by constructors.

  • String length is now a built-in CubiQL operator.

  • Various robustness and developer support improvements:

    • The measure service now can continue to load even if there are syntax errors in CubiQL expressions defining metrics or inverses.

    • Improved the logging of exceptions during measure service initialization.

    • Performance improvements to certain CubiQL optimizations.

    • Improved error checking in the REPL.

    • Improved the checking of valid aggregation types with regard to measure types.

    • The measure:config / schema translation is more robust in the presence of dynamic changes to the config predicates.

Workbook Services
  • Delta Merge: Workbooks can now be committed/refreshed using the DELTA merge method. When using this method, only data changes in the source since the last merge (commit or refresh) are being sent, as opposed to the default behavior, where all the data in a workbook is refreshed/committed.

    The default merge_method is FULL, where all data that is refreshable/committable is updated using cross-branch logic. This behavior is the same as in earlier releases.

    Developers can configure the merge_method in either the handler config, the template configuration file or in the workbook message.

    Please refer to the chapter on Merge Methods in the Reference Manual for a more detailed overview of this feature.

    Example 1. 

    In a workbook using the DELTA merge method, on commit, only data (new/removed positions and updated measures) that changed since the last commit or refresh between the master and the workbook will be sent back to the master. Any changes in the master for data that was also changed in the workbook will be overwritten, but data that was not changed in the workbook is kept intact. Let's take a look at a concrete example:

    • The following shows the initial values for the Sales predicate in both the workbook and the master:

      SKU    | STORE   | WEEK   | Sales
      apples | atlanta | week-1 | 50
      apples | atlanta | week-2 | 0 

    • In the workbook the user makes the following update:

      SKU    | STORE   | WEEK   | Sales
      apples | atlanta | week-1 | 100 

    • In the meanwhile, the data in the master is also updated:

      SKU    | STORE   | WEEK   | Sales
      apples | atlanta | week-1 | 75
      apples | atlanta | week-2 | 200 

    • When the user commits the workbook, only Sales at the intersection that was updated in the workbook will be transferred to the master, resulting in the following:

      SKU    | STORE   | WEEK   | Sales
      apples | atlanta | week-1 | 100
      apples | atlanta | week-2 | 200 

      Notice how the value for week-1 was replaced by the value from the workbook, while the value for week-2 was left intact.

    • As a comparison, the values in the master after the commit would look as follows, if the workbook would have used the default (FULL) merge method:

      SKU    | STORE   | WEEK   | Sales
      apples | atlanta | week-1 | 100
      apples | atlanta | week-2 | 0 

      Notice how the change in the master for week-2 was overwritten by the value from the workbook.

    The behavior is the same for the refresh action. Data that has not been updated within a master since the last commit or refresh does not get refreshed and therefore any changes in the workbook to those metrics will not be overwritten:

    • The following shows the initial values for the Sales predicate in both the workbook and the master:

      SKU    | STORE   | WEEK   | Sales
      apples | atlanta | week-1 | 50
      apples | atlanta | week-2 | 0 

    • In the workbook the user makes the following updates:

      SKU    | STORE   | WEEK   | Sales
      apples | atlanta | week-1 | 100
      apples | atlanta | week-2 | 100 

    • The master gets updated with the following:

      SKU    | STORE   | WEEK   | Sales
      apples | atlanta | week-1 | 200

    • When the workbook is refreshed, only the deltas from the master are being refreshed into the workbook, resulting in the following values within the workbook:

      SKU    | STORE   | WEEK   | Sales
      apples | atlanta | week-1 | 100
      apples | atlanta | week-2 | 200 

      Notice how the value for week-1 was left untouched, while the value for week-2 was updated.

    • In a workbook with the default refresh behavior using the FULL merge method, the values would have been as follows after the refresh:

      SKU    | STORE   | WEEK   | Sales
      apples | atlanta | week-1 | 50
      apples | atlanta | week-2 | 200 

      Notice how the change in the workbook to Sales in week-1 was reverted and replaced by the value from the master.

    Note

    Note that the handler configuration option use_branch_transfer is now deprecated.

  • Tagging: The workbook_util API now allows developers to add logic to set a tag on template instantiations. This allows querying of a subset of all possible instantiations of a template, for instance, to create a workflow that only builds workbooks with a certain tag.

    Please refer to the chapter on Tagged Template Instantiation in the Reference Manual for a more detailed overview of this feature.

Modeler-js
  • Override type for shortcut buttons: The behavior of shortcut buttons is now configurable. While in previous releases a shortcut button's configuration was always merged with the existing pivot configuration, it is now possible to configure the behavior of the button such that it overrides the entire configuration with the one specified in the shortcut button.

    Example 2. 

    Lets assume the following sheet configuration:

    {
      "pivotConfig": {
        "axis": {
          "x": [{ "qualifiedName": "Product:Sku" }],
          "y": [{ "qualifiedName": "-:Measures" }],
          "z": [
            { "qualifiedName": "Sales" },
            { qualifiedName: "Returns" }
          ]
        },
        "filter": {
          "Product:Sku": {
            "sku-2": true,
            "sku-3": true,
            "sku-4": true
          }
        }
      }
    }

    In this configuration, we have defined a filter so that only sku-2, sku-3 and 4 are being displayed. With the following regular shortcut button configuration, we can add two more SKU's to the filter:

    {
      "buttonConfig": {
        "label": "Add sku-1 and sku-3 to the filter",
        "type": "merge",
        "config": {
          "pivotConfig": {
            "filter": {
              "Product:Sku": {
                "sku-1": true,
                "sku-3": false
                }
            }
          }
        }
      }
    }

    Notice the new type property. The default type, which is also the behavior from previous releases, is merge and can be omitted. As result of clicking this button, the existing pivot configuration of the sheet will be merged with the one from the shortcut button, resulting in the following:

    {
      "pivotConfig": {
        "axis": {
          "x": [{ "qualifiedName": "Product:Sku" }],
          "y": [{ "qualifiedName": "-:Measures" }],
          "z": [
            { "qualifiedName": "Sales" },
            { qualifiedName: "Returns" }
          ]
        },
        "filter": {
          "Product:Sku": {
            "sku-2": true,
            "sku-3": false,
            "sku-4": true,
            "sku-1": true
          }
        }
      }
    }

    Instead of merging the configuration, it is now also possible to override the existing pivot configuration by using the new type override. In the example below, we have defined a new shortcut button with the label Reset filter to sku-3, which will override the existing pivot configuration:

    {
      "buttonConfig": {
        "label": "Reset filter to sku-3",
        "type": "override",
        "config": {
          "pivotConfig": {
            "axis": {
              "x": [{ "qualifiedName": "Product:Sku" }],
              "y": [{ "qualifiedName": "-:Measures" }],
              "z": [
                { "qualifiedName": "Sales" },
                { qualifiedName: "Returns" }
              ]
            },
            "filter": {
              "Product:Sku": {
                "sku-3": true
              }
            }
          }
        }
      }
    }

    As result of clicking this shortcut button, only sku-3 will be displayed in the view and the pivot configuration will be updated to the following:

    {
      "pivotConfig": {
        "axis": {
          "x": [{ "qualifiedName": "Product:Sku" }],
          "y": [{ "qualifiedName": "-:Measures" }],
          "z": [
            { "qualifiedName": "Sales" },
            { qualifiedName: "Returns" }
          ]
        },
        "filter": {
          "Product:Sku": {
            "sku-3": true
          }
        }
      }
    }

Corrected Issues

The issues listed below have been corrected since the 4.3.16 release.

  • Fixed an evaluation problem that resulted in an incorrect functional dependency violation error.

  • Fixed an internal error in aggregation rules that effectively do not aggregate anything, e.g. p[]=a <- agg<<a = total(x)>> x = 13.0f.

  • Fixed an internal error "object of type ScalarPredicate cannot be cast to SplittableInterface" that occurred while evaluating some aggregations into scalar predicates when parallel evaluation is used.

  • Fixed an internal error "Unexpected missing versions for 'f$2_0_1'" that occurred in rules involving default values when the query optimizer incorrectly attempted to create an index on the value of the default value predicate.

  • Fixed an issue where delta rules involving default values and predicates on different branches would not be evaluated correctly.

  • Resolved an issue where it was not possible to create a workbook template when there were multiple intersections defined with the same levels, but in a different order.

  • Measure Service:
    • For dimensions of primitive type (e.g. string), the hierarchy now has the same name as the dimension.

    • The timeout on acquiring the measure state has been bumped to five minutes.

  • Modeler-js:
    • Resolved a scrolling related performance problem.

    • The performance of both scrolling as well as filtering using the filter panel has drastically improved for large header values.

    • Resolved an issue where data updates that were entered at a certain speed were reverted in the UI, even though the changes were sent correctly to the database.

    • It is now possible to paste into a cell of type wrapped-text.

    • HTML in the label of levels is now correctly escaped.

    • Resolved an issue that could cause an exception when clicking outside of the color picker window in the conditional formatting screen.

    • Resolved an issue where the axis configuration menu closed with no input under certain circumstances.

    • Resolved an issue where the measure filter window did not query higher levels correctly.

    • Resolved an issue where opening the measure filter panel more than 2 times resulted in an exception.

Installation and Upgrade information

Installation Instructions

Install LogicBlox 4.3.17 by following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.17/etc/profile.d/logicblox.sh
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Upgrade Information

  • The lb-web handler configuration option use_branch_transfer is now deprecated in favor the new merge_method option that can be used at either the handler, the workbook template or the workbook level.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 8, update 101 or higher
Python 2.7 or higher
CPU compatible with the corei7 architecture
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.

LogicBlox 4.3.16

Release Date: November 4th 2016

Executive Summary

LogicBlox 4.3.16 introduces database performance improvements, new service configuration options, as well as expanded visualization options and usability improvements to modeler-js.

  • Database workloads with random insertions or updates should expect dramatic performance improvements.
  • Web services can now be run in asynchronous mode; a higher level API makes specifying workbook template instantiations much more concise and easier to manage.
  • Modeler-js applications will receive a number of features automatically, such as right-click menu for quick access to various functionalities, as well as new display options for images, booleans, and strings.

Take a look at the LogicBlox 4.3.16 - New Features Playlist for video highlights of these new features.

What's New

Database
  • Pages with little data not only impact database performance negatively, but also unnecessarily waste disk space. We improved the handling of such underfull btree pages such that they no longer occur. This improvement significantly improves the performance of database workloads that have random insertions or updates. The duration of transactions with random insertions will significantly improve. Disk usage is also reduced consequently.

Services Framework
  • Asynchronous services: long running services can now be run asynchronously. In the asychronous mode, services can continue running server-side without the need for an open connection between client and server, therefore minimizing the risk of transaction aborts due to connection timeouts.

    Please refer to the Asynchronous Services chapter in the reference manual for a more detailed overview of this feature.

  • Tabular Data Exchange (TDX) now supports int128 types bound to string columns. Furthermore, it supports a new column format called uuid that accepts UUID strings and can be bound both to string as well as int128 types.

Measure Service
  • The measure service can now infer a default hierarchy for a dimension if the dimension is linear (where there is bottom level, and there are unique level mappings to each parent level).

  • Default value inference improvements for CubiQL:

    • A CubiQL expression can now have a default value, yet still be considered for inlining.

    • Default value inference will now propagate default values through many more kinds of CubiQL expressions.

      Note

      If you encounter a situation where an expression does not receive a default value, let us know and we can investigate whether the use case can be supported.

Workbook Services
  • A higher level API is now available for developers to more easily configure the creation of workbooks from templates.

    Please refer to the Generating template instantiation data with workbook util chapter in the Reference Manual for a more detailed overview of this feature.

  • The list of measures included in a workbook can now be configured through the new measure_access attribute in the Workbook and WorkbookTemplate configuration message. The configuration of measures through the measures field in the Workbook message is no longer supported.

    Note

    Please refer to the Upgrade Instructions on how to migrate existing JSON workbook configurations.

Modeler-js
  • Right-click menu: Commonly accessed functionalities such as copy, paste, sort, etc., are now also available through right-click on mouse. The menu options are context-sensitive: depending on the placement of the mouse when right-click is initiated, different options are enabled.

    Example 3. 

    The figure below illustrates all the options that are now available in a central place, only one right-mouse click away. In earlier versions, these options were available in various places throughout the UI, such as the settings cogwheel, or as a right-click option on a level pill.

  • Charting Improvements: We've added the following improvements to the chart mode of the grid:

    • It is now possible to configure charts where multiple measures appear side-by-side. In previous releases we only supported stacked bar charts when there were multiple measures to display.

      Limitations

      Note that stacked bar charts are currently not supported, nor the ability to use different colors for different measures. We expect to add these capabilities in an upcoming release.

    • It is now also possible to use the name of Measures as a "column". For instance, in the chart above, the name of measures each became a column, nested inside a Region. In previous releases, Measures only indicated the measure values.

      By placing the Measures pill on an axis, e.g. the columns, the following is rendered:

      • Measure names are displayed on the axis.

      • Measure values are displayed on the opposing axis.

      As a comparison to the chart above, in the example below we swapped the order of the Measures and the Region pills. As a result, we now get a column for each measure name (Returns and Sales), and then one more column per region nested inside.

  • Various new formatting options:

    • Image formatting option: Images can now be displayed directly within the grid, by using the new image formatting option for string. Note that the value of the cell is expected to hold the URL of the image.

      Example 4. 

      The figure below illustrates in the "Sku Photo (image)" column how an image can be displayed inlined in the grid. The column to its right, called "Sku Photo", shows how the same image is displayed using the mouseover formatting option, which used to be the main displaying option for images in past releases.

    • Dropdown formatting option for booleans: It is now possible to display boolean measures as a dropdown, instead of a checkbox. Dropdown formatting allows users to clearly distinguish true, false, or no value at all. This option only makes sense for boolean measures without default values. To display the measure as a dropdown, the Format property of the measure needs to be configured as booleandropdown in the measure configuration file. The default display behavior for booleans remains a checkbox.

      Example 5. 

      The figure below illustrates both display options for booleans. In the column to the left, the measure is formatted using the default formatting option, while the measure to the right is formatted using the new booleandropdown formatting option.

    • Wrap Text formatting option for strings: Strings can now be formatted such that the text is wrapped to a new line, if the cell is not wide enough to display the complete string. This feature simplifies long text viewing and editing.

      To display the string as wrapped text, the Format property of the measure needs to be set to wrapped-text in the measure configuration file.

      Example 6. 

      In the example below we display two measures of type string. The measure in the first column is formatted as wrapped text, while the measure in the second column is formatted using the default formatting option for strings,

      Note

      Note that the width and height of a cell does not automatically change based on the size of the text. Users need to resize the cell manually.

  • Collaboration support for distributed deployments: In past releases, collaboration was only supported on single node deployments. We have removed this restriction in this release.

    Note

    Applications in distributed setting need to configure two additional parameters:
    1. The broadcast service has to be configured in the application's front end server. The example below illustrates how this can be done.

      clauses(`{
        lb:web:config:service_abbr:service_by_prefix["/modelerjs-proxy/example_project" + "/websocket/broadcast_auth/*"] = x,
        lb:web:config:websocket:broadcast_service(x) {
          lb:web:config:service:auth_realm[] = "modeler-proxy-realm"
        }.
      
        lb:web:config:service_abbr:service_by_prefix["/modelerjs-proxy/example_project" + "/websocket/broadcast_noauth"] = x,
        lb:web:config:websocket:broadcast_service(x).
      }),

    2. The URL for the broadcast service should be set in the new broadcastUrlPrefix property in the application configuration object that is input to the AuthenticatedModelerController object. Below you can find an example on how to set the broadcastUrlPrefix property.

          ...
          {
              ...
              appPrefix: "/modelerjs/example_project/partition-1",
              broadcastUrlPrefix: "/modelerjs-proxy/example_project",
              loginUrl: ...,
              modelingFeatures: {
                  ...
              },
              ...
          },
          ...

Workflow
  • Applications can now expose lb-workflow's control service on the lb:web:public endpoint simply by setting the realm name in lb:workflow:control:service_config:public_realm[].

  • The new lb workflow check command can now be used to compile workflows and check for errors.

  • We now provide a number of workflows that use the new workbook utilities to simplify the instantiation of workbook templates.

Corrected Issues

The issues listed below have been corrected since the 4.3.15 release.

  • Resolved an issue where seq could give incorrect results under certain circumstances.

  • Resolved an issue where entities were not retracted from the workbook upon workbook refresh, after they were moved across partitions.

  • Measure Service:
    • Resolved an issue with the measure service state not being properly guarded if there were in-flight requests while lb-web reloaded the measure service handler.

    • Resolved an issue where providing a spread method for a measure language recalc metric would be silently ignored. An error will now be reported indicating that it is disallowed.

    • Fixed a source of non-determinism that could cause some locks and edits to be ignored in some cases.

    • Resolved an issue that would sometimes prevent CubiQL let expressions and some abstractions from normalizing correctly.

  • modeler-js User Interface Components:
    • When an area is copied to the clipboard, it is now highlighted in the grid and the "Data copied to clipboard" infobox appears as expected.

    • Resolved an issue that caused the grid to not load correctly when no positions were passing the by-mask filter.

    • Resolved an issue that caused an exception under certain circumstances, when the Measures pill was on the slice.

    • When locking a measure at an aggregated level, the lock now correctly propagates down to the base-level of the measure.

    • Resolved an issue where charts did not draw correctly when two hierarchically related levels were on the same axis.

    • The link cell type now shows an empty cell when there is no value.

    • Resolved an issue that caused an exception when running a warm-up query on a sheet that contained a dimension split.

    • Resolved an issue where under certain circumstances the slice filter became unresponsive.

    • Resolved an issue where the filter icon was not updating correctly, when a level is removed from the axis.

    • Resolved various issues with keyboard navigation on the headers.

    • It is now possible to use the Level filter panel when the Measures pill is on the slice.

    • Resolved an issue that caused an exception when a measure was locked at a certain intersection and the user tried to update the measure at an unlocked position.

Installation and Upgrade information

Installation Instructions

Install LogicBlox 4.3.16 by following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.16/etc/profile.d/logicblox.sh
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Upgrade Information

  • To upgrade the application sheet JSON configurations, as well as the workbook JSON configurations, you can simply run the following command:

    $ migrate-modeler 4.3.15 4.3.16

  • Due to an internal upgrade of modeler-js to React 15, all html pages will need to be updated. Since Modeler-js applications today all write some of their own Javascript code to hook into modeler-js, the upgrade could not get automated. Please refer to the React documentation for the upgrade instructions and get in touch with us if you have any questions or problems!

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 8, update 101 or higher
Python 2.7 or higher
CPU compatible with the corei7 architecture
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.

LogicBlox 4.3.15

Release Date: October 4th 2016

Executive Summary

LogicBlox 4.3.15 introduces database performance improvements through new evaluation strategies and a new primitive type (int128) that can be more effectively used in place of string values. Modeler-js based applications should see expanded support for end-user data discovery, through dimension splitting and attributes, as well as other usability improvements. Check LogicBlox 4.3.15 - New Features for video highlights of these new features.

What's New

Database
  • A new evaluation strategy is introduced with this release: prefix join. Prefix-join applies to rules in which the key ordering selected by the optimizer for every body atom form a prefix. Our benchmarks show that rules where prefix-join strategy can be applied see up to a 4x performance improvement. The overall improvement to an entire application workload will vary depending on the rule characteristics of the application.

    The following are a couple of examples of rules where prefix-join evaluation would typically apply:

    • Rule with prefix (sku,store,week):

      		netsales[sku, store, week] = v <-
      	      v = sales[sku,store,week] - returns[sku,store,week].

    • Rule with prefix (sku):

      		Sales_by_brand[brand, store, week] = tv <-
      		agg<< tv = total(v) >>
      		sales[sku,store,week] = v,
      	      brand[sku] = brand.

    These patterns are ubiquitous in LogicBlox applications, either as LogiQL written by developers, or those generated by services such as Measure Service. The prefix join evaluation strategy is applicable to both the full and incremental evaluation of IDB rules. It currently does not apply to rules that involve negation.

  • A new primitive type int128 is now available for signed 128-bit integer values. The existing primitive type int supports signed 64-bit integer values and should be sufficient for virtually all applications. The int128 primitive type is particularly useful for storing UUIDs, which previously would typically be stored as string values. string values have some significant performance overhead associated with them for very large data sets, which can be avoided through the use of int128.

    The UUID usage of int128 is supported by the new built-ins int128:from_uuid_string and int128:to_uuid_string.

    Note

    We currently do not support constants for int128 (use string:int128:convert) and some overloaded infix operators, such as < and >.

  • Optimized handling of min/max aggregations of default valued predicates, using the new min_default and max_default aggregations. Please refer to the Aggregations chapter in the Reference Manual for an extensive overview of all the supported aggregations and their usage.

Modeler-js
  • Dimension Splitting: Dimension-splitting refers to viewing a measure's values by aggregating a key along two different hierarchies of the same dimension. It is a common data discovery activity and we are very pleased to finally bring it to LogicBlox users!

    Example 7. 

    Given Sales measured at the base intersection of Sku, dimension splitting allows a user to view Sales for each SubClass and Vendor combination, produced by aggregating the Sales values along the Vendor hierarchy, as well as the SubClass hierarchy.

    Note the following restrictions

    • Measures are not editable when they are placed on a view that is using dimension splitting.

    • Dimension splitting is currently only supported for non-recalc measures.

  • Attributes: Measures can now be put directly on a row or column axis, to be displayed as attributes. This allows users to configure views where some measures are shown at a different intersection than others.

    Example 8. 

    For instance, users can configure on the same view, the measure Sku to Color to be shown for each Sku, whereas the measures Sales, Returns and Net Sales shown for each Sku and Year. To do this, the user simply needs to drag the measure Sku to Color directly onto the axis where the Measures pill is already displayed or by clicking on the Plus (+) button on that axis and checking the Attribute checkbox when selecting the measure.

  • Improved links and mouseover cells: We have added an editor and link box for link and mouseover cell types. Clicking a link or mouseover cell now shows the link in a popup box along with an edit button. The edit button will bring up the editor, allowing both the URL and the caption for the cell to be changed.

  • Usability improvements for configuring rollups: Users can now quickly configure a rollup level by adding that level to an axis with the Plus (+) button on the axis. As the rollup level is represented as a pill on the axis, users can easily change the way the level is displayed.

    • Display vs. Hide Outline: The outline mode creates a new column (or row) for the respective level member, where as hiding it provides more screen real estate, especially when rollup is configured on the level.
    • Display vs. Hide Rollup: The rollup mode creates an additional row (or column) for rollup values.

    The following image demonstrates the difference between various combinations of outline vs. rollup configuration, using Subclass as an example level:

  • Usability improvements for adding measures: Adding measures to a view has never been this easy. Users can now simply click on the Plus (+) button on the axis that contains the Measures pill to directly add a measure to the grid. The user has the option to choose between displaying it as a Measure or as an Attribute.

  • Changed behavior of Filters: Measure filters now get reset when a new level is added to or removed from one of the axis panels.

  • Vega Plugin: Vega plugin configurations now have the ability to refer to either .id or .label in labeling the axis, this needs to be specified in the configuration. Previously, .id was automatically shown as label. To preserve the old behavior and display the label, .id needs to be updated to .label.

Services Framework
  • The lb-web TXN Service now allows downloading .csv files directly from an URL, without the need to send a Header in the request. This allows the result of individual parts to be listed in HTML so that the browser can download the resulting files.

  • lb-web's broadcast service has been extended to support dynamically defined channels. This feature allows clients to connect to channels determined by the URL. This enables applications, such as the Modeler, to dynamically define channels for relevant concepts: in the case of Modeler, a channel for each workbook.

    Example 9. 

    For example, if the broadcast service is hosted on "/broadcast/*", a client connecting to "/broadcast/my_channel" is connecting to the "my_channel" channel.

Measure Service
  • When the measure service fails to find a requested metric, dimension, level, etc. in the model it will now look for close matches and report them in the error message.

    Example 10. 

    When trying to aggregate the metric Sale, which does not exist, the Measure Service will now suggest to aggregate over the existing metric Sales.

Corrected Issues

The issues listed below have been corrected since the 4.3.14 release.

  • Fixed a bug in the InsideOut query optimization for float aggregations that involve integer values in the body of the rule.

  • The sorting of the lb web-server list -s command has been enhanced to consistently sort the results.

  • Resolved an issue that prevented the usage of non-negative and positive decimal formats for TDX columns.

  • Resolved an issue which occurred when installing a workbook template with a custom default group in a workspace where other templates were already installed.

  • Measure Service:
    • The negate operator will now compile to the appropriate LogiQL type:negate predicate instead of multiplication by negative -1.

    • Resolved an issue where a hierarchy could be chosen non-deterministically for some operations if there is no default hierarchy for a dimension.

  • modeler-js User Interface Components:
    • Resolved an issue with slice synching when opening a view for the first time.

    • Resolved an issue where the updated value of an entity typed measure was not displayed correctly while editing it in deferred calc mode.

    • Resolved an issue where double-clicking on the Navigation button failed to bring back the navigation pane.

    • The level members in the slice drop-down are now sorted by their label instead of their ID.

    • Resolved an issue where an exception occurred when adding a level to the grid that is incompatible with the filter mask.

    • Resolved an issue where under certain conditions an exception occurred when moving a level from the Rows to the Slice panel.

    • It is now possible to re-size a column when there is no level or measure on the x-axis.

Installation and Upgrade information

Installation Instructions

Install LogicBlox 4.3.15 by following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.15/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 8, update 101 or higher
Python 2.7 or higher
CPU compatible with the corei7 architecture
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.

LogicBlox 4.3.14

Release Date: September 2nd 2016

Executive Summary

LogicBlox 4.3.14 moves the database and services further toward supporting self service in two ways:

  • The database includes improved automatic optimization strategies that reduce the need for manual performance tuning of logic.

  • The Measure Service includes its first alpha release of a purely "managed mode", where the declaration of the schema is entirely managed by the service. Users manage the Measure Service model only, and no longer need to manage both the Measure Service model, as well as the database schema.

As this is the first release of these features, we expect to continually improve their effectiveness over the next releases.

LogicBlox 4.3.14 additionally includes various improvements to the LogiQL language, web services, workflow framework, and the cloud-store developer tool.

What's New

Database
  • InsideOut optimizer: Inside-out is a rule rewriting mechanism that transforms a rule into a set of rules that can be more efficiently evaluated. The technique applies to a rule whose joins can be independently, and more efficiently, evaluated as separate rules, with the results of these intermediate rules then aggregated into the final result.

    The theory of Inside-Out is published in and awarded the Best Paper Award by PODS 2016.

    Example 11. 

    An example of where this optimization applies is a common pattern that appears in analytical applications:

    _[class,region,month] = sls_total <- 
      agg<< sls_total = total(sls) >>
        sales[sku, store, week] = sls,
        product_sku_subclass[sku] = subclass, 
        product_subclass_class[subclass] = class,
        location_store_city[store] = city, 
        location_city_region[city] = region,
        time_week_month[week] = month.

    A common optimization rewrite applied manually today to rules such as the above, is to pull out the path computation from sku to class, and store to region, to be separately evaluated. Inside-out optimization obviates the need for such manual tuning.

  • Prefix-join mechanism is now applied more comprehensively to all queries, including those with default-value predicates, and those rules that write into file predicates. We expect some positive performance impacts on application data loads; a full evaluation of the mechanism is in progress and will be released with further prefix-join enhancements in a subsequent release.

LogiQL Language
  • New arithmetic functions int:negate and float:negate predicates. Additionally, {int,float,decimal}:negate can be used as long as at least one variable is bound, regardless of whether the variable is in the key or the value position.

Services Framework
  • lb-web's built-in authentication mechanism now supports case-insensitive usernames, which is now also the default configuration. In previous releases, usernames were treated in a case-sensitive manner (e.g. user Joe was different from user joe). To revert back to the case-sensitive mode, the username_case_sentitive() flag can be pulsed, for example by running the following command from the command line:

    $ lb exec test '+lb:web:credentials:database:username_case_sensitive().' 

    Note

    Note: For any workspace that has two usernames that only differ in case, migrating to this release can raise the following error: Error. Duplicated users found, and case sensitive flag is true.

Measure Service
  • The lb measure-service query command now accepts JSON input via the --in-format json option, and can output JSON via the --out-format json option.

  • It is now possible to configure the maximum number of expressions for which optimized versions and generated logic should be cached. These can be controlled by setting the max_compile_cache and max_optimize_cache fields in lb-measure-service.config.

  • In the editability analysis, visible metrics may now be at intersections incomparable to or less than the metric's base intersection (this feature will be used in a later release to support dimension splitting).

  • When the complete_model field of the MeasureModel protobuf message is set to true, the Measure Service will attempt to manage all predicates referenced in the model. If a predicate backs a level in the measure model that does not exist in the workspace, the Measure Service will generate its predicate declaration; if the predicate is removed from the measure model, it will also be removed from the workspace.

    Note

    Note that this feature is experimental at this moment and will be used in future releases to support some of the self-service related features in applications using modeler-js User Interface Components.

Developer Tools
  • The cloud-store list command now supports listing information on versions of an object.

  • The new command cloud-store du can now be used to provide the size of a bucket or folder.

Workflow
  • The lb workflow documentation generation tool now supports the --input-dir and --input-file options, to specify the location of the .wf files that should be used for generating the documentation.

Corrected Issues

The issues listed below have been corrected since the 4.3.13 release.

  • When using separate compilation, a number of decimal and float built-ins that were not previously recognized are now accepted.

  • Fixed a memory leak due to caching in the Measure Service.

  • The CubiQL REPL now supports the use of the character !.

  • A relabel expression now gets the same default value as the underlying expression, which allows performance optimizations (e.g. in aggregations).

  • modeler-js User Interface Components:
    • Resolved an issue where slice synchronization did not work if the Slice panel was not visible on the grid.

    • Resolved an issue where the Measures shown in Configuration Mode were not limited by the displayableMeasures configuration option.

    • Resolved an issue with the sorting of inline aggregations.

Installation and Upgrade information

Installation Instructions

Installing LogicBlox 4.3.14 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.14/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Upgrade Information

  • The warning FUNC_SEMICOLON_DEPRECATED has been eliminated. The compiler will still recognize the error code just in case there is logic attempting to alter the warning/error status with a pragma.

  • The multi_dim_level_map field of MeasureModel has been deprecated (scheduled to be removed in 4.3.16). Slide messages and the slide field of MeasureModel should now be used instead.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 7, update 11 or higher
Python 2.7 or higher
CPU compatible with the corei7 architecture
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.

LogicBlox 4.3.13

Release Date: August 5th 2016

What's New

modeler-js User Interface Components

See what's new in this release for modeler-js based applications in our LogicBlox 4.3.13 - New Features playlist!

  • End-User Charting: End-users can now turn any grid into a chart by selecting the Chart Mode option, available via the settings cogwheel. Users can choose from a number of chart types through a drop-down, and change the dimensions and measures used for the chart using the same axis configuration panels used for configuring the pivot grid. To return to grid mode, users can de-select the Chart Mode option via the settings cogwheel.

    Example 12. End-User Charting

    The figure below shows the new Chart Mode option that is available in the settings cogwheel:

    While in chart mode, a number of new options are available to users, as also illustrated below:

    • The drop-down box can be used to change the chart type on the fly.
    • The "Export as PNG" button allows users to save the chart in .png format.
    • The "View Source" button allows the user to see the Vega configuration of the chart. This can be very useful for debugging issues.
    • The "Open in Vega Editor" button can be used to inspect as well as edit the chart configuration using the Vega Editor.

    It is possible to configure via the modelingFeatures section of the application's html page, which of these options are visible to users. By default, only the "Export to PNG" and "View Source" options are visible.

     modelingFeatures: {
        ...
        chart: {
          exportAsPng: true,
          viewSource: true,
          openInVegaEditor: false
        },
        ...
    }

    The figure below shows the Vega Editor, where users can make changes to the Vega specification of a chart. This is mostly useful for debugging purposes.

  • Slice Synchronization: This feature replaces the sliceValueMeasure feature for slice synchronization. Whereas slices synchronized using a measure (through sliceValueMeasure) have their choices shared by all users on the same workspace, this new feature allows for slice choices to be separately maintained and synchronized for each individual user. Slice synchronization can be configured through the view JSON, or through the UI by end users.

    Example 13. End-User Slice Synchronization

    The following options are now available to users via the context menu of slice fields, as illustrated in the figure below:

    • Synch All: selecting this option causes the slice values of the selected level to be synchronized over all views that are available via the active navigation panel.
    • Synch All on Page: this option is only available on canvases containing multiple views. It allows users to synchronize all views on the canvas.
    • Synch with ...: this option opens up a pop-up window, displaying all the views with which the user can synchronize the slice value of the selected level. Only views that the user can access via the active navigation panel are listed here. The user can then choose to select either one or multiple views for slice synchronization.

    Once views are synchronized, the user also has the option to disable the synchronization, using the options Unsynch All, Unsynch All on Page, and Unsynch This:

    • Unsynch All: unsynchronizes all views that are synchronized with the initiating view, including from one another.
    • Unsynch All on Page: unsynchronizes the views on the initiating page.
    • Unsynch This: unsynchronizes the initiating view from the other views it is synchronized with, while the other views remain synchronized.

    Note

    Please review the Upgrade Information section on how to upgrade the application's sheet JSON configuration to use this new feature.

    Example 14. Example of the JSON configuration of Quick Slice Synch

    The example below illustrates the JSON configuration for synchronizing the slice selection of sheet1 and sheet2, as well as setting the initial value of the slice.

    // sheet1.json
    {
        "id": "sheet1",
        "title": "Sheet 1",
        "pivotConfig": {
            "axis": {...}
        },
        "signals": {
            "output": [
                {
                    "dataType": "POSITIONS",
                    "emitterId": "Slice@Calendar:day",
                    "eventName": "Slice Change",
                    "name": "<signal_name>",
                    "init": {
                        value: [{
                            "qualifiedName": "Calendar:day",
                            "measure": "SelectedDay" //configuration of the initial value via a measure
                        }]
                    }
                }
            ],
            "input": [
                {
                    "handler": "changeSliceSignalHandler",
                    "listenerId": "Slice@Calendar:day",
                    "signalName": "<signal_name>"
                }
            ]
        }
        ...
    }
    
    // sheet2.json
    {
        "id": "sheet2",
        "title": "Sheet 2",
        "pivotConfig": {
            "axis": {...}
        },
        "signals": {
            "output": [
                {
                    "dataType": "POSITIONS",
                    "emitterId": "Slice@Calendar:day",
                    "eventName": "Slice Change",
                    "name": "<signal_name>"
                }
            ],
            "input": [
                {
                    "handler": "changeSliceSignalHandler",
                    "listenerId": "Slice@Calendar:day",
                    "signalName": "<signal_name>"
                }
            ]
        }
        ...
    }
  • Session Recording Support via FullStory: It is often useful to allow users to record their sessions to provide information about issues they encounter in the application, or to provide information for developers and product owners on how the application is being used. We now offer an easy option for an application to embed session recording capabilities, through FullStory. FullStory, "DVR for all customer interactions", provides access to full fidelity recordings of user sessions, including JavaScript console outputs which can be extremely useful for diagnosing issues.

    To turn on recording, add the following segment to your AuthenticatedModelerController object:

    fullStory: {
        enabled: true,
        startByDefault: true,
        organizationKey: <ORG_KEY>
    },

    <ORG_KEY> should be arranged for your specific application. Reach out to your operations team to obtain on organization key.

    This will add a button to the menu pane of the application, that will allow users to start and stop recording. startByDefault can be set to false, which will allow webpage to be loaded with recording off by default.

  • View Configuration: The services for hosting sheets and canvases have changed. We now have the following services:

    # Upload a list of view configurations (sheets and canvases)
    POST /views
    
    # Get all view configurations
    GET /views
    
    # Get a particular view configuration
    GET /view/?id=abc
    
    # Upload a single view configuration
    POST /view
    
    # Get all view states for a particular user
    GET /viewstates/?user=user1
    
    # Get a single view state for a user
    GET /viewstate/?id=abc&user=user1
    
    # Put a view state for a user
    POST /viewstate/?user=user1

    The default view configuration is now stored and can be retrieved via the /view services. Previously, the /sheet and /canvas service would return either the default configuration, or if a user had ever edited it, the user's configuration. Hence, once a user made an edit, it was not possible to query the default configuration without deleting the user's edit in the database. User's changes are now stored separately in the /viewstate services allowing us to "Reset State" without retracting facts from the database.

    Note

    Please review the Upgrade Information section to learn about the changes that are required to your application.

  • Simplified HTML Configuration: Previously, applications were required to pass in a pageConfig object when constructing the AuthenticatedModelerController in their HTML pages. These were usually copied over from the reference implementation, where developers only changed one attribute, the navigationTreeId. Now it is possible to simply pass in the navigationTreeId as an attribute of the configuration and which will then be used as the default pageConfig to a view with the navigation tree as a sidebar.

    Note

    The change is backwards compatible and applications can continue to send pageConfigs but we recommend switching to the much simpler navigationTreeId instead. Take a look at the Upgrade Information section to learn about the changes that are required to your application to use this feature.

Database
  • The performance of decimal total aggregations has improved by about 25% for the most common use-cases (1, 2 or 3 key argument of type entity or integer). Note that the performance of an entire transaction may differ from 25% depending on the percentage of the workload that consists of total aggregations.

  • The auto_backup_mode is now set to none by default. Enable this setting can cause disk usage and performance issues, and our recommendations is for projects to leave this mode disabled. For projects that desire to experiment with enabling auto_backup_mode, this option can be configured in lb-server.config

Services Framework
  • Implemented a new handler to expose the connectblox remove block command. This can be used in the same way as other connectblox handlers, by declaring a remove_block service to be exposed in a certain URI prefix.

Developer Tools
  • To improve the error reporting of the cloud-store command when incorrect encryption keys are used, the download command now verifies a small checksum of the public key before encrypting. This allows the tool to give a high-level error about the incorrect key usage, as opposed to obscure internal decryption errors. This feature is only used for objects uploaded with this version or later.

  • The list command has internally been revised and the object listing now also shows more information about the objects. At the API-level the list command has changed as well, but as far as we know there are no API users that call list.

  • Cloud-store has improved handling of AWS S3 SlowDown responses. The improved retry code now avoids overwhelming the service when many objects are uploaded at the same time, improving overall throughput.

Workflow
  • Previously all parameters of a main workflow needed to be bound during its installation in a workspace. It is now possible to delay the binding until the time the instance is started, which provides more flexibility to how a workflow is used. It is also now possible to have set variables as main workflow parameters, as well as set literals.

    Example 15. 

    Given a main workflow foo(x, y="y", z*, w*={"1", "2"}) {...} it is now possible to choose to bind parameters when installing the workflow or when starting it. For example:

    • Trying to both install and then start the workflow foo without passing in any parameters causes an error, because both the x and z parameters are unbound:

      $lb workflow install -f file.wf --name foo --main foo
      $lb workflow start --name foo
      error: start command failed:
      	Parameter 'x' must be bound when starting workflow 'foo'.
      	Parameter 'z' must be bound when starting workflow 'foo'.

    • In the example below, we bind the parameters while installing the workflow. The parameters do not need to be bound anymore when starting it:

      $lb workflow install -f file.wf --name foo --main foo --param x="x" --param z="z"
      $lb workflow start --name foo
      Started new instance of workflow 'foo' rooted by process with id '10000000058'.

    • It's also possible to install the workflow without parameters, and only binding them when starting the workflow:

      $lb workflow install -f file.wf --name foo --main foo
      $lb workflow start --name foo --param x="x" --param z="z"
      Started new instance of workflow 'foo' rooted by process with id '10000000092'.

    • Main parameters that have default values can have their values overwritten on install:

      $ lb workflow install -f file.wf --name foo --main foo --param y="new_y"
      $ lb workflow start --name foo --param x="x" --param z="z"
      Started new instance of workflow 'foo' rooted by process with id '10000000063'.

    • However, if no value is passed on install, the default value is used, and cannot be overwritten on start:

      $ lb workflow install -f file.wf --name foo --main foo
      $ lb workflow start --name foo --param x="x" --param z="z" --param y="new_y"
      error: start command failed:
          Parameter 'y' cannot be resolved when starting workflow 'foo'.
          Either the workflow's main method does not expose a parameter with this name,
          or the parameter has a default value, which can only be overwritten when
          installing the workflow.

  • The lb workflow action and purge commands now also support a --timeout option.

Workbook Framework
  • The workbook handler now does not bundle hierarchy files together by default as it previously did, because in certain situations that can lead to very poor performance. This change only impacts TDX-based transfers, often used for backup and restore, but does not affect cross-branch transfers usually used for commit and refresh.

Corrected Issues

The issues listed below have been corrected since the 4.3.12 release.

  • We identified that a low-level prefetching method was causing problems in out-of-core use-cases, in particular for workspaces that are fragmented (with free as well as used pages). This low-level prefetching method has been disabled in this release. Some applications will see improved out-of-core performance, in particular on less performant I/O devices.

  • We identified and fixed a rare low-level data structure problem that could manifest itself with various errors or incorrect results. This fix has not yet been backported to earlier releases. We encourage applications to update to 4.3.13 to obtain this fix.

  • We addressed a problem in the handling of aborted transactions that caused obscure errors (missing pageinfo or invalid pageid, followed by continued push_request errors). This is an important fix that we encourage applications to update for.

  • When trying to install a workflow that has the same name as an already installed workflow (either when using the lb workflow install or lb workflow run commands), the user is now presented with an error.

  • Fixed an issue in the TDX transaction API that could potentially lead to a thread being leaked if the client code would not shutdown the API properly.

  • Measure Service:

    • Admin restart requests now complete in isolation from non-admin requests, to avoid a possible race condition.

    • Resolved a caching issue when clearing all measure service installed logic.

    • Resolved and issue where the use of an average, mode, ambig or count-distinct aggregation anywhere deeper than the first position in a composite aggregation method would cause an exception.

    • The measure service can now be started with an incomplete model that mentions entities for levels that don't exist yet, however the entities still have to be declared manually before the level is used in a query.

Installation and Upgrade information

Installation Instructions

Installing LogicBlox 4.3.13 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.13/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Upgrade Information

Upgrade information for applications using the modeler-js User Interface Components:
  • To upgrade the application sheet JSON configurations to use the new quick slice synchronization, you can simply run the following command:
    $ migrate-modeler 4.3.12 4.3.13
    Note that ModelerTestUtil currently does not support tests for slice synchronization. setSliceValue no longer results in the propagation of the set slice value to other sheets with which that slice is synchronized.
  • Update your html pages to use the simplified HTML configuration:

    Example 16. Simplified HTML Configuration

    Example of the pageConfig object that applications were required to provide prior to the 4.3.13 release:

    var pageConfig = {
        id: "page-with-sidebar",
        views: {
            navigationTree: {
                id: "sidebar",
                module: "ModelerNavigationTree",
                config: {
                    navigationTreeId: "unitplan-reporting"
                }
            }
        },
        layout: {
            shape: {
                columns: "200px,1",
                rows: "1,1px"   // 1px is for the LogView. We actually want to hide it,
                                // and this is the only way we can do that, for now
            },
            viewPositions: {
                navigationTree: { col: 0, row: 0, colspan: 1, rowspan: 1 }
            }
        }
    };
    
    React.render(React.createElement(TargetModeler, {
        pageConfig: pageConfig,
        actionButtons: [{
            actionId: "refresh-all", label: "Retrieve"
        }]
    }), contentEl);

    Starting this release, applications can simply pass in the navigationTreeId as an attribute of the configuration:

    React.render(React.createElement(AuthenticatedModelerController, {
        navigationTreeId: "your-navigation-tree-id",
        actionButtons: [{
            actionId: "refresh-all", label: "Retrieve"
        }]
    }), contentEl);

  • Authorize users the new /view and /viewstate services. For modeler-js applications, these are typically stored in src/config/authorization/:

    • Add the following lines to src/config/authorization/operations.csv, replacing <APPPREFIX> with your application's service prefix:

      /<APPPREFIX>/view|view service
      /<APPPREFIX>/views|view service
      /<APPPREFIX>/viewstate|view state service
      /<APPPREFIX>/viewstates|view states service
      /<APPPREFIX>/navtree|navigation tree service
    • Add the following lines to src/config/authorization/permission_operations.csv:

      general access permission|/<APPPREFIX>/view
      general access permission|/<APPPREFIX>/views
      general access permission|/<APPPREFIX>/viewstate
      general access permission|/<APPPREFIX>/viewstates
      general access permission|/<APPPREFIX>/navtree 

  • To migrate saved user preferences, follow the steps below:

    1. Export current user canvas and sheets preferences using the /user_preferences_canvases and /user_preferences_sheets.
    2. Modify the exported files by replacing CANVASID (or SHEETID) with VIEWID
    3. Import them into the new /user_preferences_viewstates service

  • Update performanceTestUtil to call the tearDown function in modelerTestUtil.destroy. This will clean up memory for each test run.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 7, update 11 or higher
Python 2.7 or higher
CPU compatible with the corei7 architecture
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.

LogicBlox 4.3.12

Release Date: July 1st 2016

What's New

Database
  • Delta predicate revisions: the way that retractions are treated in LogiQL has changed in this release. In brief, retractions from functional predicates are now specified by key only, rather than by key and value. For example, -F[1]=2 becomes -F[1]=_. Please note that this is a breaking language change! The Upgrade Information gives an extensive overview on how to adapt to the new scheme.

    Reason for this change:

    • Under the covers, this has always been the semantics of retractions in LB 4.x. In 4.3.11 and earlier, given an EDB F with F[1]=1, if the user issued an exec -F[1]=2, the F[1]=1 fact will be retracted. By requiring that users now write -F[1]=_, we bring the syntax in line with the underlying semantics of the runtime.

    • We avoid an opportunity for spurious functional dependency violations (FDVs) with conflicting retraction requests. For example, issuing -F[1]=2 and -F[1]=3 in the same transaction in 4.3.11 and earlier would yield an FDV, because the retraction requests were gathered in a predicate which was itself functional. In the revised scheme, retraction requests involve only keys and no FDVs are possible. It is now always true that retracting a fact that does not exist is not an error.

    Note

    Please review the Upgrade Information section on how to adapt to the new scheme.

  • The refmode value associated with an entity value is not supposed to be changed after the initial creation of an entity, but this was not enforced by the compiler until this release. The compiler now immediately gives a compilation error on retractions and upserts of refmode values.

Developer Tools
  • Cloud-store now supports downloading specific versions of a versioned S3 object with the --version-id option.

  • The lb popcount command now features a density option that shows what the percentage of non-default facts for default value predicates.

  • lb-server logging and daemonize changes:
    • The logging file option of lb-server now supports the value '-' to log to stdout (which only makes sense when not starting lb-server a daemon)

    • The lb server start command internally now uses the same initialization notification method as when running services using systemd.

      Note

      This internal change should not impact users, except when the LB_DEPLOYMENT_HOME setting is using a very long path name. The limit on the length of the path is now about 100 characters (the limit of Unix domain sockets). For longer paths an error will be reported.

    • The daemonize option has been removed from lb-server.config and can now only be specified as an option to lb server start.

      Note

      This generally should not impact users, except if users copied the full lb-server.config file to make modifications. This would result in the following error after updating LogicBlox: lb-server: daemonize option is deprecated and must be false. We recommend only overriding the necessary configuration settings to always pick up the right defaults after updating LogicBlox.

Measure Service
  • It is now possible to remove all the caching logic the measure service installs using lb measure-service admin clear-cached-logic. Removing the cache logic is useful when adjusting the measure service model without having to completely rebuild the workspace. This does, however, require that administrative measure service operations have been enabled.

  • The measure service now generates optimized logic for min and max aggregations over default values.

Workflow
  • The lb-workflow driver now supports push notifications. A driver configuration file can be used to set up filters over events that the driver can emit (e.g. driver startup, task start, task end, etc). A notifier can be setup to receive notifications when an event matches the filter. The following notifiers are currently supported: file notifier (where the events are dumped to a file) and Amazon SNS notifier.

Corrected Issues

The issues listed below have been corrected since the 4.3.11 release.

  • Changing the logging level online (using lb server loglevel) did not work when logging to journald.

  • The command lb server start did not correctly handle the logging file option from lb-server.config and would always use the default value. This has been fixed.

  • The command lb-server start did not correctly handle the option --daemonize false. This has been fixed.

  • Resolved an issue where the lb-workflow driver sometimes crashed, due to SimpleTasks not being cleaned up from the driver, causing excessive memory usage.

  • A potential race condition has been resolved in the Measure Service that could arise when using the administrative refresh command to reload measure service model definitions.

Installation and Upgrade information

Installation Instructions

Installing LogicBlox 4.3.12 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.12/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Upgrade Information

How to adapt to the new scheme:

  • For retractions from functional predicates, just replace the value with an underscore.

    Example 17. 

    For example,

    -data[i]=w <- +remove(w), data@prev[i]=w.

    becomes

    -data[i]=_ <- +remove(w), data@prev[i]=w.

  • If a retraction atom -F[x]=y occurs in the body of a rule, and the semantics of the rule requires deriving facts only for specific values of y, then you may need to find a different way of binding the variable. Typically you can then use F@prev[x]=y to bind the variable.

    Example 18. 

    For example,

    +finished_subscription[store, sku, start]=end <-
      -active_subscription[store, sku]=start,
      week_less_or_equal(start, end),
      current_week[]=end.

    (note how start is needed) can be changed to

    +finished_subscription[store, sku, start]=end <-
      -active_subscription[store, sku]=_,
      active_subscription@prev[store, sku]=start,
      week_less_or_equal(start, end),
      current_week[]=end.

  • If replacing -F[x]=y with -F[x]=_ leaves only a single use of the variable y, then to avoid a variable solitary warning (or error, as configured by most projects) you may also need to modify the body to use an _ for an @prev predicate.

    Example 19. 

    For example,

    -effective[sku]=_ <-
      +new_clone(sku),
      effective@prev[sku]=date.

    becomes

    -effective[sku]=_ <-
      +new_clone(sku),
      effective@prev[sku]=_.

    Note that it is not an error to generate retraction requests for facts that do not exist, so depending on the desired semantics the effective@prev atom could also be entirely removed in this case.

  • For retractions from EDB entity types/constructors, do the retraction from the constructor, and rely on auto-retraction rules to retract from the type.

    Example 20. 

    For example,

    a(x), -a_from_string["foo"]=x.

    becomes

    -a_from_string["foo"]=_.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 7, update 11 or higher
Python 2.7 or higher
CPU compatible with the corei7 architecture
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.

LogicBlox 4.3.11

Release Date: June 2nd 2016

What's New

Database
  • This release introduces a new implementation of the meta-engine (used for managing meta data such as predicates and logic rules). This implementation results in significant performance improvements and reduction in the memory consumption in the runtime overall, compared to the previous implementation.

    Key new features are:

    • Greatly improved latency of transactions that execute very large queries. Most measure service queries fall into this category and should benefit significantly.

    • Greatly improved latency of transactions that modify the schema and logic of the database (e.g. adding or removing blocks). Due to the incremental execution graph maintenance, full evaluation of the installed logic is now never necessary on logic additions or removal. Only the affected logic (that is in rules depending on the change) need to be recomputed. The impact is most significant for applications that have many predicates or rules.

    • Several longstanding bugs with removing blocks have been resolved. Removing blocks should now work and we ask users to actively start using this feature and report any problems. Note that it is still a requirement that any dependent blocks be removed manually. E.g. if block A contains declarations of a predicate used in block B, block B must be removed before block A is.

    • Rules for derived-only predicates may now be added or removed in any database lifetime block. In previous versions of 4.x, we required all rules for a single derived-only predicate to be added in the same block.

    • Significantly reduced memory usage. We have measured an improvement of about 20% in most applications, but the gain can be significantly bigger for workbook-based applications.

    • Significant reduction in the time needed to open workspaces, in particular when opening workspaces from EBS snapshots.

  • The lb-server process now uses static linking to include the various LB libraries. This has no impact on the user and in our experiments it improves performance by about 10% depending on the workload.

Developer Tools
  • The logging format has been improved to make it easier to correlate error messages to a workspace, branch or a transaction for concurrent transactions. We have changed the logging format to now include information on workspace, branch, and transaction on every line. As an example of the change:

    2016-05-26 13:54:52,611999+00:00 warning long-running "ExUnRule file:69(5)--79(45)#24BYO45A" (took 16.81 sec)

    is now logged as:

    2016-05-26 13:54:52,611999+00:00 warning [workspace] [txn:317e946be] long-running "ExUnRule file:69(5)--79(45)#24BYO45A" (took 16.81 sec) 

    Important

    Note that this change may break monitoring or log analysis tools!

Workbook Framework
  • Workbook framework performance and memory footprint has been dramatically improved due to reduction in installed block per workbook instantiation, and using more performant logic when logic does need to be installed.

    • The workbook framework no longer installs inactive blocks that are only needed for the TDX-based implementation, if branch transfer implementation is used. This results in improved memory usage and performance.

    • The workbook framework now utilizes a new runtime branch parameter feature, which drastically reduces the amount of logic installed into the database and consequently memory usage and time needed to open a workspace. The branch parameter feature allows the workbook framework to add inactive blocks with unbound branch parameters. The concrete branch to use is only specified when executing the inactive block.

Workflow
  • Implemented an optimization for the LogiQL rules related to restartable processes lifecycle. This makes the workflows more scalable in the presence of a large number of restartables.

  • By default, lb workflow status now avoids printing workflow instances that had no process instances selected. The new --show-empty-instances flag forces the previous behavior, where those instances were printed.

  • New workflow tasks to manipulate strings that represent URLs: lb.string.EncodeURL and lb.string.DecodeURL.

  • The workflow task lb.ReadCsv now accepts file URIs as parameters.

Corrected Issues

The issues listed below have been corrected since the 4.3.10 release.

  • The lb export-workspace previously did not correctly handle pending diskcommit transactions and could export a previous version of the database. This bug could cause the exported workspace to miss schema or data changes executed just before the export. This problem has been fixed.

  • Fixed a memory leak that would very slowly, but steadily cause memory usage to increase when executing many transactions (hours of short-running transactions). The growth is small enough that it is unlikely to have impacted any current users.

  • In very rare cases lines from a CSV file would be dropped when importing the file due to a bug in the implementation of parallel processing. This issue has been resolved.

  • Resolved an issue where in rare cases a deadlock could occur when restarting the measure service.

  • Resolved an issue with the measure service that would prevent an expression and the same expression with a functional annotation from being distinguished during code generation.

  • Internal improvements to the CubiQL optimizer yielding significant performance improvements to lengthy warm-ups and queries that have not yet been cached.

  • It is now possible to use @prev stages on IDB metrics in measure language inverses.

  • Resolved an issue with the workflow code generator that prevented multiple restartable cleanup sections to be installed in a workspace.

  • modeler-js User Interface Components:
    • Resolved an issue where the "clear filter" option was not available via the context menu

Installation and Upgrade information

Installation Instructions

Installing LogicBlox 4.3.11 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.11/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 7, update 11 or higher
Python 2.7 or higher
CPU compatible with the corei7 architecture
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.

LogicBlox 4.3.10

Release Date: May 3rd 2016

What's New

See what's new in this release in our LogicBlox 4.3.10 - New Features playlist!

modeler-js User Interface Components
  • Revised Filtering and Slicing Experience: Filtering and slicing on headers has been revised such that users can easily see in one place all the members that can be filtered, and their relationship to one another.

    Filter and slice windows are easily accessible through the filter icon on every column or row header pill, or through the dropdown for slice pills, respectively. The color coding on members creates a visual indication for the effect of the filter/slice selections:

    • Members that have been directly selected to be in the filter, through user clicks, are displayed as blue.
    • Members that participate in no (hierarchical) relationship to directly selected members, and are thus implicitly excluded from the filter, are displayed in grey. Directly selecting a grey member would invalidate all current choices, as it will necessitate some current selections to be excluded.
    • Members whose siblings (in the same level) are included in the current selection, and whose selection does not require invalidation of current choices, are displayed in green.
    • And finally, members that have not been included or excluded from the filter, directly or indirectly, are displayed in white.

    Filter/slice windows also come with other usability features such as:

    • Keyboard navigation: UP/DOWN, LEFT/RIGHT keys can be used to change focus between the items in the filter window; TAB selects the item in focus, ENTER allows for quick application of changes and closing the window, while ESCAPE closes the window without applying any change.
    • The "Undo" icon allows a user to undo his changes in a single session.
    • The "Pin" icon allows a user to pin the filter/slice window in place as he applies different filtering conditions.

    Example 21. 

    As illustrated in the figure below, when the users selects a state, the filter panel clearly indicates which cities belong to the selected state, so that he can decide to either select one, multiple or all of the stores rolling up to the state:

    Tip

    Due to the redesign of the slice pill, the slice panel now takes up less vertical space. To take advantage of this extra space, you can reduce the height of your slice panel in each sheet configuration. The ideal height is now 39px (in earlier releases we recommended a height of 62px).

    You can run the command below to automatically replace <SLICE_HEIGHT> with "39px" in all of your sheet configuration files (that are stored under src/config/sheets). Most applications use 62px as the <SLICE_HEIGHT>, but it can be different per application. We advise you to carefully review the changes caused by running this script, as the command simply replaces every occurence of <SLICE_HEIGHT> with 39px and could potentially lead to unwanted changes.

    $ cd <PROJECT_DIR>
    $ find src/config/sheets/ -type f -name "*sheet.json" -exec sed -i '' 's/<SLICE_HEIGHT>/39px/g' {} +

  • Improved Axis Configuration: Adding new levels to Rows, Columns or Slice has never been this easy and quick! You can now add new levels directly from the grid, without the need to switch to configuration mode. You can either search for a level or navigate through the Dimension Browser that is being displayed in a pop-up window.

    Example 22. 

    The figure below illustrates how the new pop-up window helps users in selecting a level to be added to the rows. Levels that cannot be added (for example due to the fact that levels from the same dimension but a different hierarchy are already displayed on the grid) are indicated with a grey background color and cannot be selected.
  • Support for displaying view titles: Views of the type SheetView can now have a title that is being displayed at the top of the view.

    Example 23. 

    In the example below the title AdHoc View is configured via the title entry of the view configuration:
    "adhoc-sheet": {
      "id": "adhoc-sheet",
      "title": "AdHoc View",
      "module": "SheetView"
    } 

  • Configuration of sorting of levels: It is now possible for developers to configure the default sorting of levels on a view. To do this, add the sort option inside the pivotCofig setting of the sheet.

    Example 24. 

    In the example below, the level Product:SKU is sorted descending, using it's label:

    ...
    "pivotConfig": {
     ...,
     "sort": {
      "Product:SKU": {
       "qualifiedName": "Product:SKU",
       "sortAttr": "label",
       "sortType": "desc"
      }
     }
    },
    ...

  • Bulk User View Preferences Backup and Restore: Two sets of services are now available to support the bulk backup and restoration of user view preferences:

    • /<APPPREFIX>/user_preferences_sheets: the GET version of this service exports all user sheets in TDX format, and the POST version of this service can be used to import back user sheets in the same TDX format.

    • /<APPPREFIX>/user_preferences_canvases: the GET version of this service exports all user canvases in TDX format, and the POST version of this service can be used to import back user canvases in the same TDX format.

    If used with the workbook framework, these services are only available in the master workspace.

    These services are useful when backing up data in a workspace in preparation for an application or platform upgrade, where the workspace would need to be rebuilt. These services should most likely be invoked together unless there is a specific reason to only back up sheets and not the canvas, or vice versa.

Measure Service
  • RelationBinding messages can now associate a parameter (ParamExpr) with an arbitrary CubiQL expression (MeasureExpr):

    {param: P, expr: E}

    The previous association of a parameter with a literal relation (LiteralTypedRelation):

    {param: P, relation: {intersection: I, type: T, column: C}}

    is deprecated in favor of using a literal expression (LiteralExpr):

    {param: P,
    expr: {kind: LITERAL,
     literal: {kind: GENERAL,
      signature: {intersection: I,
       type: {kind: SINGLETON, type: T}},
      column: C}}}

  • Update syntax: It is now possible to express updates in a textual syntax. This will help make it easier to experiment with spreading, etc. in the REPL directly or via scripts.

    Example 25. 

    In the following example, we simultaneously update M1 at intersections {A2,B2,C0} and {A1,B1A,C0}, and remove at intersection {A2,B2,C2A}:

    do spread
      {("A2_1", "B2_0", "C0_0_C1A_0_C2A_0", 333.0)} : {A.A2, B.B2, C.C0} => decimal
      via ratio into M1
    and spread
      {("A1_1_A2_0", "B1A_0_B2_1", "C0_1_C1A_1_C2A_1", 73.0),
     ("A1_0_A2_1", "B1A_0_B2_0", "C0_0_C1A_0_C2A_0", 60.0)} : {A.A1, B.B1A, C.C0} => decimal
      via ratio into M1
    and remove
      {("A2_0", "B2_0", "C2A_0"),
      ("A2_1", "B2_1", "C2A_1")} : {A.A2, B.B2, C.C2A}
      from M1 

  • Promote expression: promote converts the value of a CubiQL expression into a dimension, leaving the expression position-only:

    promote label in expr

    It isn’t quite the inverse of demote, as the result of demote will always be considered to be set valued, so if a functional expression is promoted that information is lost.

Workflow

  • lb-workflow does not print timestamps in logs anymore under systemd, because systemd already prints a timestamp.

Corrected Issues

The issues listed below have been corrected since the 4.3.9 release.

  • Fixed a correctness problem in decimal max aggregations.

  • Fixed an issue that was resulting in the following error message: "Request for reclaimed pageid".

  • Fixed an issue that prevented submitting concurrent write transactions and branching operations.

  • Fixed a problem in frame rule domain parallelism that caused the following error message: "predicate cardinality does not match that recorded in internal SIP state".

  • modeler-js User Interface Components:
    • When exporting data directly to Microsoft Excel, the id of a level was exported instead of its label.

    • Attempting to copy / export for a second time in a row when there was an existing broken query failed to complete.

    • The automatically generated measures representing hierarchy mappings now have more informative names by following the form "child to parent".

    • When a measure was sorted, the user could face an error when trying to move a level from the Rows to the Slice panel.

    • In-line only roll-ups did not respect the configured sort attribute.

    • Pasting a large number of rows than what fit on the grid resulted in a query that did not terminate.

    • The measure filter panel did not behave correctly with % formatted measures.

    • The measure filter panel did not filter correctly on 0 values for measures that had 0 as their default value.

    • Resolved an issue where min/max input fields of the measure filter panel were not updated correctly for decimal numbers.

    • If more than one level from the same dimension was visible on either the Rows or Columns, the filter dialog failed to show.

    • Resolved an issue where conditional formatting did not work when comparing = 0 for measures with default value 0.

    • The is in range option in conditional formatting now behaves as expected.

    • Resolved an issue where the logout notification did not show show up anymore when a user was idle for a certain amount of time.

Installation and Upgrade information

Installation Instructions

Installing LogicBlox 4.3.10 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.10/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 7, update 11 or higher
Python 2.7 or higher
CPU compatible with the corei7 architecture
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.

LogicBlox 4.3.9

Release Date: April 2nd 2016

What's New

modeler-js User Interface Components

See what's new in this release for modeler-js based applications in our LogicBlox 4.3.9 - New Features playlist!

  • Conditional Formatting: This new feature allows measure values to be conditionally formatted based on comparisons with constants. Conditions apply to measures of any value type: numeric, boolean, string, or level entities. Formatting options include the background color, foreground color, and font styles such as bold, italic, and strike-through.

    The feature is available for end-users, while more expressive configuration options are available to configuration consultants only. End-users can open up the conditional formatting panel by clicking on the settings cogwheel in the bottom-right of a view, where they can add, edit and remove conditional rules. The configuration is stored per user.

    In order to apply some initial (conditional) formatting to a view, a conditionalFormatting configuration section needs to be added to the view's json configuration file.

    Example 26. 

    The figure below shows the conditional formatting panel, which has a rule defined for the measure Sales Units TY stating that all cells need to be rendered with a red background color, and the font needs to be displayed in white and in bold if the value of the measure is smaller or equal to 175,000.

    The backend configuration of this example would look as follows:

    "conditionalFormatting": {
      "TY_Sales_Units": [
        {
          "condition": "<=",
          "value": 175000,
          "style": {
    	"background-color": "red",
    	"font-style": "bold",
    	"color": "white"
          }
        }
      ]
    }

    The backend option allows configuration consultants to specify conditional formatting that applies to measures at certain intersections only, rather than applying to a measure's value at all intersections.

  • Shortcut Buttons View: The new ShortcutButtons view allows developers to configure a group of buttons that can change the configuration of one or more views on the same canvas.

    A shortcut-buttons section in a canvas configuration can specify a partial configuration for another view or views. When button is pressed, the partial configuration overrides the current configuration for the specified view(s) with the provided configuration.

    Example 27. 

    In the example below:

    • module is required to be ShortcutButtons
    • applyToViews attribute specifies a list of views the button's configuration should be applied to when pressed.
    • buttons contains a list of button specifications. label specifies the label of the botton, while config contains the partial configuration that will be applied to the views listed in applyToViews.

      In this example, when the "Enable Mask Filter" button is clicked, the list of mask filters on the mask-filter-sheet will be set to SkuMask. The second button, "Disable Mask Filter", clears the list of mask filters.

    "shortcut-buttons": {
      "module": "ShortcutButtons",
      "applyToViews": ["mask-filter-sheet"],
      "buttons": [
        {
          "label": "Enable Mask Filter",
          "config": {
    	"pivotConfig": {
    	  "maskFilter": ["SkuMask"]
    	}
          }
        },
        {
          "label": "Disable Mask Filter",
          "config": {
    	"pivotConfig": {
    	  "maskFilter": []
    	}
          }
        }
      ]
    } 

    The buttons will appear as a group, whether a button is active depends on whether the current configuration of the sheet(s) already matches the configuration defined for the button.

  • Copy/paste to and from external applications: In previous releases, modeler-js supported copy/paste activities within its own pivot grids. This release introduces the possibility of copying and pasting a rectangular region of data to and from an external application, such as Microsoft Excel.

Services Framework
  • New configuration option in lb-web-server.config to avoid sending the NameIdPolicy element in a SAML authorization request, as sending it could cause issues with certain IDP's.

    # if true, include the NameIDPolicy element in the AuthnRequest. This is an optional field that may
    # cause issues with some IDPs. Default is true.
    #include_name_id_policy=true 
Measure Service
  • Simplification of removal requests: Previously when removing values from a metric with inverses, it was necessary to provide not only the keys, but also the value to be removed, even though the value can be determined from the keys. With this release, the value is no longer necessary.

  • Intersection operators: It is now possible to define intersections in terms of the meet and join of other intersections. See the CompositeIntersection protocol message for details. It is also possible to define an intersection as the restriction of another intersection. See the RestrictionIntersection message for details.

Workflow

  • It is now possible to configure the prefix for the status service and control service at once, using lb:workflow:service:prefix[]; by setting it to app, the services will be hosted at /app/status and /app/control respectively. Furthermore, the LB_WORKFLOW_SERVICES environment variable is used to change the default prefix on the command line (e.g. LB_WORKFLOW_SERVICES=http://localhost:55183/app) to avoid having to pass it to every command. Finally, the status service library was incorporated into the core lb-workflow library, so there's no need to include it in application projects anymore.

  • New workflow tasks to call the new workflow control service:

    • The task lb.control.Purge can be used to delete selected, terminated workflow instances.

    • The task lb.control.Action can be used to send actions to selected workflow processes.

  • The action command now provides feedback on which processes were affected by the executed action.

Corrected Issues

The issues listed below have been corrected since the 4.3.8 release.

  • Resolved an issue in TDX that prevented error files to be generated if the combination of async mode, errors only on error and a file URI were used together.

  • The output of log-level perf has been revised. Unnecessary lines are removed to make the logs significantly smaller and easier to review.

  • Fixed various runtime internal errors.

  • modeler-js User Interface Components:
    • Resolved an issue with the rendering of the settings menu when the user clicked onto the settings cogwheel on a split canvas, where the top view was configured to be very small.

    • Resolved an issue that prevented users from being able to export a view to MS Excel, when a measure was displayed at an aggregate level and the aggregation method none was configured as its default aggregation method.

    • All numeric measures are now exported correctly to MS Excel, including their formatting.

    • The "remove" option now also works correctly for levels on the slice.

    • Fixed a regression where the mouseover formatting option no longer displayed image on mouseover.

  • Measure Service:

    • The update_pred field of the Predicate message is now correctly consulted when defined.

Installation and Upgrade information

Installation Instructions

Installing LogicBlox 4.3.9 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.9/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Upgrade Information

  • Support for TDX plugin has been removed in this release.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 7, update 11 or higher
Python 2.7 or higher
CPU compatible with the corei7 architecture
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.

LogicBlox 4.3.8

Release Date: March 4th 2016

What's New

  • In LogicBlox 4.3.7 and earlier the fixed-point decimal type supported numbers in the range of -10^14 to 10^14 and the precision of the fraction was 0.00001 (10^-5). The limited range and precision was increasingly becoming a problem, for example in banking applications with different currencies, and in planning applications big spreading operations lacked the required precision. To address these problems, LogicBlox 4.3.8 introduces a revised fixed-point decimal type that supports a range of -10^18 to 10^18 and a fractional precision of 10^-18. Internally, decimal values can now use up to 128 bits, but are stored in a compressed format that will often limit the disk usage growth when upgrading applications to LogicBlox 4.3.8. In benchmarking of the new implementation, the performance impact has been limited to single-digit percentages.

  • The changes in the decimal type can potentially break some applications due to the increased precision or small differences we had to make in the implementation of built-ins. An overview of issues you may encounter:

    • In general, the much improved precision most likely causes different values to be computed for any non-trivial application. This may require updating expected test results. While reviewing the test changes needed, we advise adjusting tests to compare expected results only up to a certain precision.

    • decimal:string:convert used to pad the fraction with zeros to five digits (e.g. 1 would be rendered as 1.00000.) This is impractical with a precision of 18 digits, so the new implementation only prints the fractional digits up to the last non-zero digit.

    • When converting binary floating-point values (float) to decimal fixed-point values, the earlier versions of LogicBox 4.x would round the floating-point number to 5 digits due to limited precision of the decimal type. With the increased precision of decimal values in LB 4.3.8 binary floating-point number representation artifacts are now more visible.

      Example 28. 

      For example, float:decimal:convert[123.345f] now results in the decimal value 123.344999999999998863.

      Note

      We recommend consistent usage of either binary floating-point numbers or decimal fixed-point values, depending on the implementation.

    • The previous implementation for parsing strings to decimal values (string:decimal:convert) carefully rounded the input if the fraction in the string had more than 5 digits. In LB 4.3.8 the string value is truncated at 18 digits.

    • The representation of decimal values in the protocol for communication with the lb-server has changed. The decimal_column field is gone and has been replaced with a decimal128_column field. Also, the DecimalColumn is now called Decimal128Column, and it has different fields for the actual values. The Java library for communicating with the lb-server only required internal changes (not interface changes) because it already used BigDecimal (which can represent LogicBlox 128-bit decimal values). The Python library did not hide the internal decimal representation details and users of this library might have to make changes to their code. Because neither the Python or Java library is a formal interface with public documentation we hope that no users have code that is impacted. Please just reach out to us if you have problems upgrading client code.

  • New features for the decimal type:

    • The built-in decimal:int:convert has been reintroduced. It drops the fraction of the input decimal value.

      Example 29. 

      decimal:int:convert[13.8] results in 13, decimal:int:convert[-5.9] in -5.

    • We have extended the decimal rounding built-ins to support more tie-breaking policies. In LB 4.3.7 we only supported round (which rounds half to even) and roundHalfUp. We now support roundHalfAwayFromZero, roundHalfDown, roundHalfToEven, roundHalfToOdd, roundHalfTowardZero, and roundHalfUp. The simple round built-in still defaults to half-to-even. All rounding built-ins come in roundX and roundX2 variants that support rounding the fraction at the n’th digit.

    • The round2 built-in now also supports negative numbers, which rounds the decimal value all the way into the integral part of the number.

      Example 30. 

      round2[n, -2] with -2 rounds the value n to a multiple of hundred.

    Note

    Please refer to the LogicBlox Reference Manual for a complete overview of all the built-in predicates.

Installation and Upgrade information

Installation Instructions

Installing LogicBlox 4.3.8 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.8/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 7, update 11 or higher
Python 2.7 or higher
CPU compatible with the corei7 architecture
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.

LogicBlox 4.3.7

Release Date: March 1st 2016

What's New

Database
  • The performance of frame rules has significantly improved for larger incremental data loads. There was a problem with the parallel evaluation of frame rules, which would be used for larger deltas. As part of this tuning we have also lowered the delta size limit for the usage of parallel evaluation. This will mean better core utilization for some applications.

LogiQL Language
  • Bitwise integer operations:

    • LogiQL now supports "unsigned" hexadecimal and binary literals (0xABCD and 0b1010111).

    • Added support for several new bitwise operations on integers.

    Note

    Please refer to the Reference Manual for an overview of all the integer bit manipulation functions.

modeler-js User Interface Components
  • Vega Charting Plugin: The new Vega Charting plugin allows developers to configure a sheet containing a chart. Chart configuration must be specified using the Vega visualization language. Data used to populate the chart can be specified using the sheet configuration language already supported for modeler-js sheets. Charts can be configured to interact with other visualization components, such as grids. When linked, changing the data in grids will cause chart displays to change and reflect the new data, as well.

    Example 31. 

    The figure below illustrates how a Pareto chart can be displayed on the same canvas as a grid. The grid and the chart interact with each other, allowing the user to immediately see changes in the grid being reflected in the chart.

  • Locking: This feature allows an end user to lock certain values, such that they are not affected by the spreading that may result from other edits. A lock applies to any edit performed by the user, regardless of on which view the edit is performed. Locks persists throughout the user’s browser session. Locks will be reset upon closing of the browser tab.

  • Individual resizing of rows and columns: Resizing can be now be done individually for the innermost headers (i.e. those next to the data cells). The outer headers are resized based on the inner headers. For cross producted headers all instances of the resized header will be resized. To resize all columns, users can hold down the <CTRL> key while resizing.

    Developers can also configure the individual column/row sizes via the pivot configuration.

    Example 32. 

    The example below allows to apply custom height to subclass1 and width for the Sales column. The members object is used to define the innermost headers sizes. The rollups object is used to define sizes for aggregated heading:

    {
        "gridOptions": {
            "display": {
                "Product:subclass": {
                    "members": {
                        "subclass1": {
                            "height": 50
                        }
                    },
                    "rollups": {
                        "Product:class": {
                            "class1": {
                                "height": 60
                            }
                        }
                    },
                },
                "-:Measures": {
                    "members": {
                        "Sales": {
                            "width": 99
                        }
                    }
                }
            }
        }
    }

  • Logout notification over multiple tabs: The logout notification now also works over multiple tabs. The new configuration option ignoreBroadcastingTime allows developers to configure after how many seconds of idle time the processing of broadcasting messages should stop.

    The notification around user inactivity and automatic logout can now be configured as follows:

    modelingFeatures: {
        ...
        idleTimeValues: {
                //in seconds
                idleNotificationTime: 1800,  //in seconds
                idleLogoutTime: 60,		 //in seconds
                ignoreBroadcastingTime: 300  //in seconds
            }
        ...
    }

  • Level mapping measures: The measure model configuration utility distributed with modeler-js now defines all level mappings to be measures. Every level, consequently, is also declared to be an intersection.

    Example 33. 

    For instance, if a Product hierarchy contains levels Product:Sku, which maps to Product:Subclass, then a measure, Product:Sku:subclass, is automatically configured, backed by the predicate Product:Sku:subclass.

    For level Product:Sku the corresponding intersection, Product:Sku, is declared and can be used to define measures.

  • Improved Dimension Browser: The new Dimension Browser is a more compact representation of all the levels in the schema.

  • Warmups: The new lbconfig.modeler python module can be used for generating warmups for both measure rules and views.

    Example 34. 

    To use this new module, the following needs to be added to the config.py script:

    from lbconfig import modeler
    
    modeler.configure()

    This will create targets to copy over all assets, JavaScript libraries as well as scripts from the platform build installation. For warmups it will create the following make targets:

    • rule-queries: generates queries for rule warmup

    • view-queries: generates queries for view warmup

    • warmup-rules: submits the generated queries to warmup rules

    Note

    Application developers are advised to using this module instead of application specific scripts, to be able to automatically benefit from future enhancements.

Services Framework
  • lb-web allows services to declare an inactive block to be executed in the same transaction as a request. It is now possible to declare multiple such inactive blocks, which allows handling code to be better modularized and reused.

  • The raw file-mode is now supported for TDX imports. It was previously only implemented for TDX export. The raw file-mode does no escaping or unquoting whatsoever, but does not support the delimiter in fields.

  • The lb-web server now prints a clear message when startup is done and it is ready to serve requests.

    INFO lb-web         	- lb web-server initialized and ready to accept requests (code: LB_WEB_INIT_COMPLETE)

  • New json_printing option is now available for protobuf services to configure whether strict or loose mode should be used.

  • It is now possible to configure the force_authn flag on the SAML request that lb-web sends to IDPs when a user attempts to log into a SAML realm. This forces the IDP to re-request authentication even if the user has an ongoing session with the IDP.

  • lb-web services normally obtain the username for a service request from request parameters that are imported into the database using a protobuf message. In active logic, this populates the pulse predicate lb:web:auth:username. We earlier introduced an optional setting enable_pulse_auth on services, which directly populates this predicate at stage initial. This makes the username also easily accessible to inactive blocks. This feature is now also available for TDX services (earlier it only was supported by protobuf services).

  • New features to control lb web-server logs:

    • A new log_file config property can be used to determine where to store the logs.

    • The new systemd property (also settable via the IN_SYSTEMD environment variable) determines 1) whether logs will contain timestamps (if in systemd timestamps are redundant so they are omitted) and 2) if explicitly set to false, then even when running as a systemd unit, the specified log_file will be used. This can be convenient for experiments that produce very large logs.

    • The log_elapsed flag makes the logs contain an additional column with the number of milliseconds since startup.

    • UTC is now the default timezone for timestamps in lb-web server and client, including server access logs. There's a special host timezone that can be used to select the default timezone of the running host.

Measure Service
  • It is now possible to use a CubiQL expression as the input to an update rather than a LiteralTypedRelation.

    Note

    Please refer to the the section called “Upgrade Information” for more information on the deprecation of LiteralTypedRelation.

  • Optimizations for default-valued boolean min/max aggregations.

Workflow

  • lb workflow is now integrated into the lb command line.

  • The new command lb workflow purge allows the removal of workflow instances from the workspace.

    Example 35. 

    You can use the following command to remove all workflow instances of the daily workflow that are older than 7 days:

    $ lb workflow purge --name daily --older-than 7days

    Example of removing a workflow instance with a specific ID:

    $ lb workflow purge --root 10000000063 

    It is also possible to remove all workflow instances at once:

    $ lb workflow purge --all

  • All duration values in lb workflow tasks can now be specified as strings that contain the value plus a unit. For backwards compatibility, tasks still accept integers, which will be interpreted in the same unit as before (seconds for most tasks; milliseconds for lb.tdx.* tasks).

    Example 36. 

    The expected syntax is an integer (specifying the length) followed by an optional resolution specification. The supported values are as follows:

    • ms: milliseconds

    • s: seconds

    • m: minutes

    • h: hours

    • d: days

    • M: months

    • y: years

    Examples:

    "5"  	 //5 milliseconds for lb.tdx.* tasks, 5 seconds otherwise
    "50ms"   //50 milliseconds
    "4h"    //4 hours

    Note

    Please refer to the the section called “Upgrade Information” for more information on the deprecation of using integers as the duration values.

  • It is now possible to list workflow instances using lb workflow status --list-roots. This will use the status service to lookup the roots of workflow instances and list them. It can be used together with the --name option to select instances of a particular workflow.

  • lb workflow has now command line completion like all other lb commands.

    Tip

    To enable locally, source [path_to_lb_home]/etc/bash_completion.d/logicblox.sh in your bash profile or similar.

  • It is now possible to pass the --failed-restartables option to the lb workflow status command to get the status for the restartables that have a failed descendant.

  • The new workflow lb.wb.SetUsers can be used to add and remove users that have access to workbooks in one transaction. Previously, these activities had to be performed via two separate tasks.

Workbook Framework

  • Cross-branch Commit/Refresh: This feature supports an alternative method of performing commit/refresh between workbooks. Whereas the default method of commit/refresh currently is through exchanging data between master and workbooks by exporting/importing CSVs, the cross-branch commit/refresh executes cross-branch logic: rules that refer to data in one branch of the workbook, and derive it into another branch.

    Cross-branch commit/refresh represent performance wins, but are only applicable for workbooks that are co-located on the same instance as their parent.

Developer Tools

  • The lb export-workspace command now also works on secondary workspaces.

  • Various improvements to the profileDiskSpace command:

    • The --branch option can be used to show disk usage only for a specific branch of the workspace.

    • The --versions option has been improved to show the total number of reachable pages for every branch (earlier it only showed differences between versions in the listing).

    • The command can now be used online using lb batch-script. It does block write transactions.

  • lb print-rules (introduced in the 4.3.6 release) now also shows delta rules when applied to EDB predicates.

  • All command line arguments that accept a log level now have a better help message with some examples.

  • Several lb commands that accept a --csv parameter to create a CSV file now support a --delimiter parameter to configure the file's delimiter. This includes lb exec, lb query, lb execblock, lb workbook list and list-templates.

  • lb info now supports the --json option to print the information in JSON format.

Corrected Issues

The issues listed below have been corrected since the 4.3.6 release.

  • Resolved an issue introduced in 4.3.6 where some disjunctive rules would not evaluate correctly. The latest 4.3.6 patch releases also included this fix.

  • The lb batch-script command did not correctly use security tokens and required the LB_CONNECTBLOX_ENABLE_ADMIN environment variable to be set. This is not the case anymore.

  • Resolved an issue where lb export-workspace would hang if the lb-server was terminated unexpectedly during the export.

  • lb export-workspace now respects the port configuration in lb-server.config.

  • A transaction abort in earlier releases would print an ERROR line in the log, which complicated monitoring for errors. This error line has been removed.

  • Resolved an issue with executing write transactions on ws vs. ws@master, which are both the same branch. The transactions would occasionally fail with an error that a concurrent write is detected.

  • The composition of configuration files in lb-web-server has been fixed. Previously, configuration sections in lb_deployment would only be able to send new options to handlers, but never change an existing one. Also, dynamically loaded jars and config files would completely overwrite anything from lb_deployment. This has been fixed such that lb_deployment files have precedence over jar config files, and dynamically loaded config files have precedence over lb_deployment.

  • modeler-js User Interface Components:
    • Resolved an issue that could cause a pivot grid to be rendered empty for incorrectly selected slices on initial render when the slice was set via a measure.

    • The scroll position is now correctly maintained when the user switches between views.

    • Resolved an issue that could cause an incorrect sorting behavior when editing a previously sorted column.

    • Resolved an issue that caused pending calculations not to be marked as such when switching between views.
  • Measure Service:

    • Resolved issues with computing updates with respect to locked recalc metrics.

    • Improved handling of aborts and no longer blocks lb-web threads to handle requests.

    • Resolved an issue that could cause a crash stemming from invalid relabel and split expressions.

Installation and Upgrade information

Installation Instructions

Installing LogicBlox 4.3.7 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.7/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Upgrade Information

  • Java 8: LogicBlox requires Java 8 starting with the 4.3.7 release. All Java code is compiled for Java 8, so on older Java versions the components will not run (e.g. the compiler server). If accidentally deployed on an older Java version, then an error "Unsupported major.minor version 52.0" should be reported by the JVM.

  • Workflow:

    • The lb-workflow command has been deprecated in favour of lb workflow.

    • All duration values in lb workflow tasks should now be specified as strings containing the value and their unit (e.g., "300ms", "5s", "2h", etc). The use of integers is discouraged and the compiler and driver will raise deprecation warnings.

  • All uses of QueueTransport and SQS support for transports has been removed: Only TCP is now available.

  • Deprecated Measure Service Protocol Changes: To facilitate work on removing deprecated features from projects, we re-announce in this release previously-announced deprecations, accompanied with migration advice. We aim to remove them in the 4.3.9 release.

    To check for absence of these deprecated features in a project, set fatal_warnings=true in the project's lb-measure-service.config.

    • ad hoc fields for textual CubiQL are deprecated (since 4.3.3): Most string fields that have been variously available to hold CubiQL expressions in textual form are now deprecated in favor of replacement fields holding a MeasureExpr.

      For example, expr_str in ExprSubst message is deprecated in favor of the expr field and measure_str in QueryRequest in favor of measure.

      Since MeasureExpr now has the str string field for textual CubiQL, the transition amounts to moving the content of the deprecated field to the replacement field, after wrapping it as a MeasureExpr.

      Example 37. 

      For example,

      { "query_request": {
                "measure_str": ["1+2", "total Sales"]
              }}

      becomes

            { "query_request": {
                "measure": [{"str": "1+2"}, {"str": "total Sales"}]
              }}

    • map_pred is deprecated in Edge and Path (since 4.3.3): The string field map_pred is deprecated in favor of the Predicate field pred, in the Dimension.Edge and Dimension.Hierarchy.Path messages.

      Projects that use the config library do not need adjustments to deal with this deprecation (the config predicates dimension_hasEdge and levelMap have been adjusted to produce the pred field). Clients that create protobuf directly should change, e.g.,

      path {
        map_pred: "MyPred", 
        //… 
      }
      to 
      path {
        pred: {name: "MyPred"}, 
        //… 
      }

    • pred field in ExprSubst is deprecated (since 4.3.3): The field pred holding a Predicate within an ExprSubst entry in a model has been deprecated. It was used to specify a predicate in the workspace to be used as a replacement of a CubiQL expression, instead of any optimization that measure service would compute for it. The preferred alternative in protobuf is the replacement_expr field that holds a MeasureExpr, with an anonymous Metric defined for the predicate. In the configuration library, the alternative would also require defining a named metric in the model based on the predicate.

      Example 38. 

      For example, if a substitution was defined as

        +measure:config:exprSubst_replacesExpr["total Returns by to Location.region", false]=subst,
        +measure:config:exprSubst(subst) {
          +measure:config:exprSubst_usesPredicateName[]="pdx:sku:region:week:returns"
        }.

      then the new definitions would be

        +measure:config:exprSubst_replacesExpr["total Returns by to Location.region", false]=subst,
        +measure:config:exprSubst(subst) {
          +measure:config:exprSubst_usesExpression[]="TotalReturnsToRegion"
        }.

      with the named metric for the predicate defined as

        +measure:config:metric(TotalReturnsToRegion) {
          +measure:config:metric_hasName[]="TotalReturnsToRegion",
          +measure:config:metric_hasSignature[]= +measure:config:baseSignature(sig){
            +measure:config:baseSignature_singleton("sku,region,week", "FLOAT")},
          +measure:config:metric_usesPredicate[]="pdx:sku:region:week:returns"
        }.

    • metric field in UpdateExpr is deprecated (since 4.3.4): metric field in UpdateExpr is deprecated, as it is now a special case of the target field holding an UpdateExpr.Target message.

      Example 39. 

      For example, an UpdateExpr

        {"kind": "SPREAD", 
         "metric": "Sales", 
         //...
        } 

      should now be written

        {"kind": "SPREAD", 
         "target": {"metric": "Sales"},
         //...
        } 

    • signature field in UpdateExpr is deprecated (since 4.3.7): The field signature in UpdateExpr is now deprecated as it can be inferred from the source or input fields of the enclosing UpdateRequest.

    • source field in UpdateRequest is deprecated (since 4.3.7): The field source in UpdateRequest that holds a LiteralTypedRelation with input data for the update is now deprecated. It is replaced with the new field input, holding a MeasureExpr. It is more general, in allowing the input data to be computable. The same effect as source can be achieved by using a ParamExpr as input, coupled with a RelationBinding in the enclosing Request, to hold a LiteralTypedExpression with the data. A LiteralExpr (rather than a ParamExpr) could be used as well, but in this case the data would be installed as logic. Unless these data are used more than once, this would likely be undesirable.

    • Using LiteralTypedRelations as the input for updates is deprecated: To eliminate the use of LiteralTypedRelations as the input of updates, use a ParamExpr as the input expression.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 7, update 11 or higher
Python 2.7 or higher
CPU compatible with the corei7 architecture
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.

LogicBlox 4.3.6

Release Date: January 8th 2016

What's New

Database
  • Configurable auto backup mode: Revisions of the database are automatically tagged for backup and investigation purposes. The new configuration option auto_backup_mode in lb-server.config can be set to none to disable this feature. The default setting is default, where the server automatically keeps versions.
LogiQL Language
  • datetime:parse now supports the %p directive to recognize am/AM/pm/PM for 12-hour time inputs.
modeler-js User Interface Components

  • Automatic Support for Ordered Levels: Modeler-js based applications now can have sorting logic for ordered levels automatically generated for them. The sort is by default created using the refmode of a level entity. The only declaration required from developers is to set the IsOrdered column in Levels.csv to be true, for levels that should be ordered. The Modeler-js toolset will automatically generate and populate the following predicates for an ordered level L:
    L_first[] = l   -> L(l).
    L_last[] = l    -> L(l).
    L_next[l1] = l2 -> L(l1), L(l2).
    L_index[l] = i  -> L(l), int(i).
    For two ordered levels that have a hierarchical mapping, L mapping to M, the following predicates are generated and populated:
    L:M_first[l] = m    -> L(l), M(m).
    L:M_last[l] = m     -> L(l), M(m).
    L:M_next[m,l1] = l2 -> M(m), L(l1), L(l2).
    If the refmode of entity L does not provide a proper alphanumeric sort order for the entities, developers can additionally provide a transform predicate P, and a transformed primitive type T. Predicate P with the following signature will be declared for the developer:
    P[l] = t -> L(l), T(t).
    The project logic must then implement rules to populate P. The expectation is for P to produce a primitive value for each member of L such that these values can be used for alphanumeric sorting.

    Note

    Note that due to this change, the header format of Levels.csv has changed. Please refer to the Upgrade Information section for more information.

Measure Service

  • A new config option fatal_warnings has been added to the measure service configuration file (lb-measure-service.config). If set to true, the measure service will consider warnings to be errors. This setting can be useful for clearing deprecated features out of a project prior to their removal from the measure service.

Services Framework

  • In order to be consistent with the LogiQL syntax for number literals, some TDX column formats where changed in a backwards incompatible manner. Previously, 0.0+ and >0.0 would format non-negative and positive floats, respectively. Now, these formats are used for decimals; floats are formatted with 0.0f+ and >0.0f. Furthermore, >0 can be used for positive integers.
  • lb-web transactions aborted explicitly with lb:web:abort:error_response now also return the error message in the body of the response.

Developer Tools

  • lb info for a secondary workspace now shows how many pages have been received and how many are expected in the current incoming version. For example:
    pages_in_incoming_version: 81
    pages_received_in_incoming_version: 9 
    This also works when lb start-mirror is still receiving the initial version. The new field state that has been added to lb info's output indicates whether the workspace is ready to process requests. When lb start-mirror is still busy, it will show state: INIT and otherwise state: OPENED.
  • The lb info command now also shows the timestamp of the last committed version (for both primary and secondary workspaces).
  • The new command lb print-block can be used to print the content of a block (both active or inactive).

    Usage:

    lb print-block [-h] [--loglevel LOGLEVEL] [-L] [--cwd [DIR]]
                          WORKSPACE BLOCK
  • The new command lb print-rules allows developers to print the rules for a given predicate name that exist in the database. The command can also be used on EDBs that have delta rules.

    Usage:

    lb print-rules [-h] [--internal] [--loglevel LOGLEVEL] [-L] [--cwd [DIR]]
    			  WORKSPACE PREDICATE

    Note

    Please refer to the Command Reference chapter in the reference manual for an extensive overview of all the LogicBlox commands and their usage.

  • The lb execblock command now also has a --timeout parameter.
  • It is now possible to abort a running transaction using CTRL+C on the 'lb' command. The lb tool now also aborts transactions when the process terminated with the Linux 'timeout' command.
  • Logging enhancements:

    • We now log a clearer message when initializing all services has completed and lb-web-server is up and running:

      INFO lb-web             - lb web-server initialized and ready to accept requests (code: LB_WEB_INIT_COMPLETE)

    • Various improvements to replication related error reporting and logging.

    • The perf logging level now includes logging for the number of changed facts.

Corrected Issues

The issues listed below have been corrected since the 4.3.5 release.

  • Improved performance of copying large S3 buckets using cloud-store copy.

  • Resolved several internal issues in the lb web-server that impacted memory usage.

  • Resolved an issue with server access logs were not respecting the log_timezone setting in lb-web-server.config.

  • Resolved an issue where protobuf messages weren't displayed correctly via the meta service.

  • Write transactions on the secondary workspace now result in an error.

  • Write transactions do not get blocked anymore during initial replication.

  • Resolved an issue where delta rules involving foreign predicates with default values didn't update correctly.

  • Resolved an issue where lb exec produced inconsistent results for queries involving foreign predicates and default values.

  • Resolved an issue that could cause lb server to crash when aborting a transaction.

  • ModelerTestUtil will now fail on queries to canvas when a query error occurs. Previously, ModelerTestUtil was erroneously swallowing these errors.

  • Measure Service:

    • The measure service handler is now fully asynchronous and should respond to abort requests.

    • Selection of hierarchies is now deterministic when the choice of ambiguous.

    • Locked measure language primary metrics will now properly be converted to identity updates if one of their dependencies is edited.

    • Performance improvements for queries and updates involving many large CubiQL expressions.

Installation and Upgrade information

Installation Instructions

Installing LogicBlox 4.3.6 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.6/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Upgrade Information

Upgrade information for applications using the modeler-js User Interface Components:

  • The header format of Levels.csv had to change to support specification of transform predicate and transformed type. Levels.csv now expects two additional columns: OrderTransform, and TransformType.

    Example 40. 

    The header of Levels.csv is now as follows:

    Level,Label,Dimension,ElementType,IsOrdered,OrderAttribute,OrderTransform,TransformedType 
  • Filter Panel removed: The filter panel was removed to make way for position filtering improvements. A new and improved filter panel will be taking its place soon.

Removed Features from the Measure Service

  • CondExpr message is removed: The conditional expression CondExpr is removed. The textual CubiQL had form if e0 then e1 else e2. There is no direct replacement: CondExpr did not find much use and had somewhat confusing semantics. Moreover, in the point-free style of CubiQL, the more natural idiom emerged as filtering some input with several FilterExpr and then combining their results in some way.
  • attr field is removed from AttributeExpr: The attr field (holding an Attribute) in an AttributeExpr was used to represent an "anonymous attribute". That is, it would produce a measure that looked like a measure for an attribute of a level, but without the requirement the attribute be named in the model. Instead, the attr field would directly name the predicate that implemented the attribute. The same result can be obtained as an "anonymous metric" defined by the same predicate.

    Example 41. 

    For example, the anonymous attribute expression

      {"kind": "ATTRIBUTE", 
       "qualified_level": {"dimension": "Location", "hierarchy": "Geographic", 
                           "level": "store"}, 
       "attr": {"predicate": {"name": "pdx:loc:store_id"}, 
                "type": {"kind": "STRING"}}
      }

    gives the same result as the anonymous metric expression

      {"kind": "METRIC", 
       "metric": 
         {"metric": 
           {"signature": 
              {"kind": "BASE", 
               "base_signature": 
                 {"intersection": 
                    {"qualified_level": [{"dimension": "Location", 
                        "hierarchy": "Geographic", "level": "store"}]}, 
                  "type": {"kind": "", type: {"kind": "STRING"}}
                 }
              }, 
            "predicate": {"name": "pdx:loc:store_id"}
           }
         }  
      } 

  • NO_GROUPING kind of aggregation grouping is removed: The NO_GROUPING kind is removed from the AggExpr.Grouping.Kind enumeration. It had the same semantics as having no groupings of any kind within an aggregation expression.
  • Support for the TDX measure service plugin has been disabled.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 7, update 11 or higher
Python 2.7 or higher
CPU compatible with the corei7 architecture
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.

LogicBlox 4.3.5

Release Date: December 2nd 2015

What's New

modeler-js User Interface Components

  • Collaboration: modeler-js components now support multi-user collaboration, where each user will see the results of actions and edits made by other users in the same workbook, within a configurable amount of time (the default is 6 seconds). Cell changes made by other users will appear green briefly, whereas an action triggered by other users will cause a notification window to appear. Figure 1 shows an example of the notification window, where the user is notified that "user1" has triggered the action "Set Timestamp". The action will be disabled for all active users in the workbook until the re-querying of updated data is made.

    Figure 1. 

    Note that if a user is editing the same cell that others are also editing, her cell will not be updated with new values. In this situation, the last writer wins.

    To activate a collaboration feature, a collaboration section has to be defined inside the modelingFeatures section. The collaboration update interval can be configured as follows:

      modelingFeatures: {
        ...
        collaboration: {
            minUpdateInterval: 0, //in seconds
            maxUpdateInterval: 6 //in seconds
        },
        ...
    }
  • Export to Excel: It is now possible to export the contents of a sheet to a Microsoft Excel file. The new Export to Excel option is available in the sheet's config pop-up menu that can be accessed by clicking on the cogwheel located at the bottom right side of a sheet. The export process runs in the background; users receive a notification once the export has completed. The file is automatically saved in the Downloads folder that is configured for the browser.

    Example 42. 

    Figure 2 shows an example sheet with various aggregations (aggregations are being displayed with a light blue background). Figure 3 shows how the corresponding exported file in MS Excel would look like.

    Figure 2. 

    Figure 3. 

  • Logout Notification: After some time of being idle the user will be notified via a pop-up message that her session is about to close, as shown in Figure 4. She can choose from one of two possible options: continue the session or close the session. If neither of these options is selected, the user will be logged out automatically after the timer runs out.

    Figure 4. 

    Developers can configure the time until the notification message, as well as the time until the automatic logout. The default values are 1800 seconds for the idleNotificationTime and 60 seconds for the idleLogoutTime.

    The idle notification time and the idle logout times can be configured as follows:

    modelingFeatures: {
        ...
            idleNotificationTime: 1800, //in seconds
            idleLogoutTime: 60 //in seconds
        ...
    }
  • afterAction Event: After each action is executed, an afterAction event is emitted, that applications can listen to. This feature allows developers to configure JavaScript functions to be executed after an action.
    React.createElement(AuthenticatedModelerController, {
        ...,
        events: {
            afterAction: function(actionConfig, response, getServiceUrl) {
                if (actionConfig.id === "action_info") {
                    if (!logicblox.util.actionResponseHasError(response)) {
                        // do something here
                    }
                }
            }
        }
    }); 
  • Formula Editor: We have improved the formula editor to allow the inspection of rules. Opening the formula editor will display the formulas for all metrics on the visible measures axis that have formulas. If a metric that is used in a formula is not yet on the grid, clicking on its name will automatically add it.

    Below is an example on how to enable the formula editor via the html files. The formula editor can be found under the cogwheel located at the bottom right side of a sheet, if the ruleEditor setting is enabled.

    modelingFeatures: {
        ...
          ruleEditor: true
        ...
    }

    Example 43. 

    In the example below a sheet is being displayed with the EDB measures Sales and Returns and the IDB measure NetSales. The formula editor shows the primary formula as well as the 2 inverse formulas. Via the formula editor, developers can inspect all the formulas as well as update the inverse rules.

    Figure 5. 

Measure Service

  • Support for min/max/sort aggregations on entity-valued expressions. In order for an entity to be used for min/max/sort aggregations, it must have:
    • A predicate that maps the entity to a primitive type.
    • The entity must back a unique level in the measure service meta model, and must have an order attribute using the ordering predicate.

    Example 44. 

    If Day is an entity type, it should have a predicate that orders Day’s

    Day_index[d] = index -> Day(d), int(index).

    The entity Day must back a level Calendar.Day and the predicate Day_index must back the attribute Calendar.Day.order.

Services Framework

  • TDX imports now return an HTTP Header called x-blox-tdx-import-errors that contains "true/false" indicating whether there were import errors. This is only available if allow_partial_import is true (otherwise any error would make the request fail completely). This feature is exposed to lb web-client import and lb-workflow, to indicate via a warning message that there were import errors. The new option --silent to the lb web-client import command can be used to suppress the warning.
  • Improved meta-service to list all HTTP methods accepted by a service.

Developer Tools

  • New command lb batch-script allows dlbatch commands to be executed inside a running lb-server instance.

    Usage:

    usage: lb batch-script [-h] [-t] [-f FILE] [--loglevel LOGLEVEL] [-L]
                           [--cwd [DIR]]
                           WORKSPACE [SCRIPT]

    Example 45. 

    In the example below the showDefiningRules dlbatch command is called via the lb batch-script command:

    lb foo> addblock 'c(x,y) <- a(x), b(y).'
    added block 'block_1Z3FC9NG'
    lb foo> batch-script -t 'showDefiningRules --predicate c'
    // block_1Z3FC9NG:1(1)--1(20)#1Z3FGTPE c(x,y)
    c(v0,v1) <-
       a(v0),
       b(v1).

    The list of dlbatch commands and their usage can be gotten via dlbatch’s underlying help command, i.e., by invoking batch-script ‘help’. For instance:

    lb foo> batch-script 'help summary'
    Usage: summary [options]
      --lifetime arg        Lifetime to summarize (e.g. database, builtin,
                            transaction)
      --kind arg            Kind of objects to list (e.g. predicates or blocks)
      --list                List names of objects only (no detailed description)
      --filename arg        Filename where summary is written to
    
    Summarize high-level workspace state

    Note

    Running this command is only permitted under authenticated mode, and there are a few restrictions:

    • Workspace-related commands like create and destroy are disallowed.
    • Transaction-related commands like transaction, abort, and commit are disallowed. Instead, if the --transactional flag is passed, then the script will be executed inside a (soft-commit write) transaction. This flag is required for commands like addBlock or exec which need to execute inside a transaction.

    Please refer to the Command Reference chapter in the reference manual for an extensive overview of all the LogicBlox commands and their usage.

  • The new command lb list-blocks can be used to list all active and inactive blocks installed in a workspace.

    Usage:

    usage: lb list-blocks [-h] [--inactive | --active] [--loglevel LOGLEVEL] [-L]
                          [--cwd [DIR]]
                          WORKSPACE 
    Please refer to the Command Reference chapter in the reference manual for an extensive overview of all the LogicBlox commands and their usage.
  • The log message about long-running rules is enhanced to include the changed predicate(s).

Corrected Issues

The issues listed below have been corrected since the 4.3.4 release.

  • Corrected a performance issue with default value rules when entity elements are added
  • Corrected an error when using default value values on pulse predicates
  • Corrected a performance issue with many concurrent requests on machines with many cores (e.g. 40)
  • Corrected some issues with regard to incremental maintenance.
  • It is not possible anymore to execute a write transaction on a mirror via the dlbatch command.
  • Trying to export a predicate column with type decimal with format type integer, TDX now returns an exception, including hints on how to to use an import/export function to declare the exact conversion to use (e.g. use decimal:round, decimal:floor or decimal:ceil to get another decimal and then convert it into a string or float).
  • Resolved an issue that prevented browsing the meta service for protobuf services.
  • Measure Service:
    • Resolved an issue with the ambig aggregation over expressions with a non-zero default value.
    • Improved performance of composite spreads and spreads involving several chained transformations.
    • Resolved an issue where install requests were only caching optimized expressions and compiled logic, but not installing logic.
    • Improvements to predicate key ordering in generated logic and prototype support for generating pragmas to force runtime variable order choices.

Installation and Upgrade information

Installation Instructions

Installing LogicBlox 4.3.5 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.5/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 7, update 11 or higher
Python 2.7 or higher
CPU compatible with the corei7 architecture
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.

LogicBlox 4.3.4

Release Date: November 6th 2015

What's New

LogiQL Language
  • predicate@branch support: LogicBlox now supports referring to a predicate on another branch in ad-hoc queries and inactive blocks. This feature makes database branches even more useful.

    Predicates from other branches can be referenced in rule bodies via a predicate@branchname syntax, similar to the syntax for stage-names.

    Example 46. 

    For example, if foo is a branch name, then

      +R[x]=y <- S@foo[x]=z, T@foo[z]=y.

    joins the versions of S and T on branch foo, and inserts the result into the local version of R.

    An arbitrary number of branches can be used for read-only purposes in a single transaction, but only predicates on the local branch of the transaction can be written to.

    Note

    Use of this feature is limited to ad-hoc queries logic and inactive blocks. In the current version, there are also some restrictions on the use of entities: in the example above, if e.g.

    R[x]=y -> E(x), decimal(x),

    where E is an entity type, then we need to add a "guard" to the rule:

    +R[x]=y <- S@foo[x]=z, T@foo[z]=y, E(x). 

    This ensures that only entities existing both on the main branch and on foo are copied over.

    Some useful applications of this feature include:

    • Migration of data to a new version of an application: The rules and schema of the new version are installed in a separate workspace branch, and then rules using predicate@branchname migrate the data.
    • Long-running write transactions, involving e.g. calls to a solver: To avoid blocking write-transactions on the main branch, these transactions can be done in a separate branch, and then predicate@branchname rules can be used to import the results.
    • Comparison of current and past database versions: Branches can be used to "snapshot" the workspace at periodic intervals, and then predicate@branchname rules can be used to compare the versions.

Measure Service

  • Support for updating targets other than metrics. It is now possible to also edit
    • Attributes
    • Level members
    • Mappings between level members
  • +/add can now be used for string concatenation.
  • Ratio and even spreading on integer types will now spread exactly. Previously there would be some loss due to naive division.
  • Table headers in REPL results now display axis labels, in addition to level names.

modeler-js User Interface Components

  • Editability: When in deferred calc mode, the grid will now render cells that are no longer editable as read-only, based on a user’s outstanding edits. Editability is computed at the granularity of intersections, (e.g. (Sku, Store, Day)) rather than positions (e.g. (sku-1, store-3, day-2)). This level of granularity can be too coarse; improving it is part of future work.

    Example 47. 

    Figures 1 and 2 show a simple sheet with 3 measures: Sales, Returns and NetSales at their base intersection (SKU,Store,Day). The measure NetSales is defined via the following primary and inverse rules:

      NetSales = Sales - Returns
      Sales = NetSales + Returns
      Returns = Sales - NetSales  

    In the initial situation as depicted in Figure 1, all 3 measures are editable.

    Figure 2 shows that once the user switches to deferred calc mode and updates Sales(sku1,store1,day1) as well as NetSales(sku11,store1,day1), Returns is not editable anymore at any intersection.

  • Dynamic drop-downs: Drop-down selections can now be configured and filtered through a measure. Based on the content of the measure, the selection in the drop-down can change.

    Example 48. 

    This feature is particularly useful for showing attributes of products, where only certain attributes are relevant for a given product, but not another. For instance, a dynamic drop-down can be used to show only brands relevant for the product class on the same row.

  • Formatted input: Grid cells now accept formatted inputs.

    Example 49. 

    For example, letters, such a M for "million" or B for "billion" can be used to enter large numbers quickly. The supported formatted inputs are listed below.

    User entry Considered as
    .25 0.25
    2,000 2000
    $2000 2000
    2% 0.02
    2k 2000
    2.02k 2020
    2M 2000000
    2B 2000000000
    2T 2000000000000
  • Input validation: Cell inputs are now validated against the accepted cell type.

    Example 50. 

    For instance, entering a non-numeric value into a numeric cell will result in an error presented to the user, asking her to correct her input.

  • Precise integer spreading: Integer spreading requires special handling when the value being spread result in a remainder after integer division evenly or by ratio across all elements at the intersection. Spreading the number 6 evenly across 4 elements is such an example. Precise integer spreading, the default spreading method now used with integer-valued measures, spread the remainder after division across some randomly selected number of elements as evenly as possible, such that the aggregation after the spread is the same as the value being spread.
  • Export CSV feature now exports data formatted as they are displayed in the grid: for instance, numbers are formatted with currency signs if applicable, entities are formatted using their labels.
  • Configuration of displayableMeasures per sheet: displayableMeasures can now be configured at the sheet level, instead of the canvas level. Previously configured displayableMeasures at the canvas level will continue to work, being applied to all sheets in a canvas. This new feature allows users to configure the available measures in the Model Browser per sheet, which can be useful when configuring canvases with multiple sheets.
  • Users can now configure the direction that the cursor should move upon pressing the TAB key. This can be achieved via the "Pivot Settings" window that is available via the cogwheel on the lower-right of the grid.

Developer Tools

  • The option suppress_task_parallelism is now supported in lb-server.config. It used to be only supported as an environment variable.

    Note

    This option is only useful for performance analysis and should not be used for other purposes.

Corrected Issues

The issues listed below have been corrected since the 4.3.3 release.

  • Replication:
    • On the primary database, the command lb info now lists the IP addresses and ports of the active mirrors.
    • Resolved an issue that was causing the primary and secondary database to grow very quickly in size.
    • Resolved an issue where the secondary database would return query results for the latest database version if the primary database server was killed.
    • When executing a transaction on the secondary database we no longer require all pages for later versions of the database to exist in the primary database. This is particularly important when the secondary database is used for very long-running transactions.
    • Improved robustness of UUID checks that prevent a secondary database to receive a mixture of pages from different instances of the primary database. Importing a database into the lb-server now changes its UUID.
  • Default Value:
    • Resolved a semantic problem with default values in delta rules. Delta rules with default values in the body are now also substantially more efficient than the old implementation of default values in LB3.

      Example 51. 

      For example, a delta rule that needs to replace the full content of a predicate f with the content of g (^f[x,y,z] = v <- g[x,y,z] = v. ) now only considers the union of the non-default facts of predicates f and g.

    • We now guarantee that EDB default value predicates never materialize facts that have a default value. This means that it is not possible for a database to become densely populated with default values over extended usage. For IDB aggregations it is still possible that a default value is materialized due to incremental re-computations of the value leading to the default value.
    • Optimized evaluation of equality comparison to constants for boolean-valued default predicates (for example f[x] = true or f[x] = false). When comparing the value for equality to the non-default value, then the facts with default values are efficiently ignored. Earlier developers had to convert the equality to an inequality.
  • Fixed a performance problem in checking for deltas in incremental maintenance of an IDB rule. This check could be expensive in some specific cases where indexes were earlier created online.
  • Resolved an issue that was preventing blocks from being removed from the workspace in some situations.

Installation and Upgrade information

Installation Instructions

Installing LogicBlox 4.3.4 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.4/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Depreacted Features

Measure Service Protobuf interface changes:

  • CondExpr is now deprecated; use FilterExpr instead.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 7, update 11 or higher
Python 2.7 or higher
CPU compatible with the corei7 architecture
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.

LogicBlox 4.3.3

Release Date: October 5th 2015

What's New

Database
  • The lb branch command now takes an optional --overwrite parameter to replace an earlier existing branch. Please refer to the Reference Manual for an overview of all the commands to query and manipulate branches.
LogiQL Language
  • The running total aggregation now provides a mechanism to reset the accumulated total at a specific point in time. Resets can be used, for example, to reset the accumulated total at the beginning of each month. Please refer to the Reference Manual for the details on how to use this feature and extensive examples.
Measure Service

  • Protobuf interface changes:
    • A MeasureExpr can now be defined by a textual CubiQL, using its new str field. This deprecates all other string f ileds for textual CubiQL expressions throughout the protocol.
    • New MAX binary operator; an outer MAX operator is definable as (max(E1,E2) ++ E1 ++ E2).
  • New EditabilityRequest / EditabilityResponse message pair for information about which visible metrics can be still be edited given a set of current edits.
  • Protobuf messages are now validated for structural constraints, with warnings and errors being reported in request responses.
  • Union expressions (CompositeExpr of UNION kind) are now set-valued.
  • The REPL now supports multi-line commands.

modeler-js User Interface Components

modeler-js is a set of components that can be used to declaratively configure HTML5 user interfaces for LogicBlox applications. This first release of modeler-js includes the following components:

  • Pivot table: a component that supports the reading and writing of measure data in a pivot table interface. Data can be displayed at arbitrary levels of aggregation, filtered and/or sorted by level members or measure values.
  • Schema browser: a component that supports browsing and searching of measures, dimensions, hierarchies, and levels, configured using the Measure Service metamodel. The schema browser interacts with the pivot table component, allowing measures and levels to be dragged drom the browser and placed onto the pivot table component for display.

In addition to UI components, modeler-js also includes two set of tools that support the configuration and upgrade of application data models:

  • Applications data model can be configured using a set of csv files. These files are then translated into configurations in the Measure Service metamodel, as well as the modeler-js component metamodel.
  • LogicBlox 3.x Blade applications can use the included blade_to_4x.py to upgrade an existing Blade configuration, partially automatically. The data model and display properties of measures, dimensions, hierarchies, and levels can be upgraded autamotically. The view configurations must be manually upgraded from UISL to the modeler-js JSON configurations.

Corrected Issues

The issues listed below have been corrected since the 4.3.2 release.

  • If the path specified with libpath option to the lb addproject command had an empty string, it expanded to CWD where services were started. This could lead to search of unintended directories and in some cases failure to open file.
  • For data exports using file predicates or TDX, some I/O errors were not detected, this led to truncated exports in some cases. This has been corrected.
  • Resolved an issue where the deletion of a workspace would not terminate under certain circumstances.
  • Various measure service related bugfixes and adjusted optimization rewrites.

Installation and Upgrade information

Installation Instructions

Installing LogicBlox 4.3.3 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.3/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Depreacted Features

Measure Service Protobuf interface changes:

  • String references to LogiQL predicates are deprecated in favor of the Predicate message.
  • ExprSubst should now be backed by a MeasureExpr. Use of Predicate for this purpose is now deprecated.
  • ADD_OUTER operator has been deprecated. Instead of E1 +_outer E2, use (E1 + E2) ++ E1 ++ E2.
  • Binding has been removed; use ExprSubst instead.
  • is_disjunctive_view annotation on Predicate has been removed.
  • ParamTerm has been removed; instead, use ParamExpr defined on the top intersection.
  • ParamBinding has been removed; instead, use RelationBinding at the top intersection.
  • EmptyExpr has been removed; use LiteralExpr with empty data content.
  • Numeric conversion operators (e.g., INT_TO_FLOAT) have been removed; use CastExpr instead.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 7, update 11 or higher
Python 2.7 or higher
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.

LogicBlox 4.3.2

Release Date: September 6th 2015

What's New

Database
  • Tuning intended to improve the latency of ad-hoc queries has reduced the overhead of the meta-engine significantly (around 30%). In real workloads, this has resulted in an overall improvement of 17% in performance. The relative performance win will increase as queries get bigger and data-sets get smaller.
LogiQL Language
  • It is now possible to write e:eq_2(x,y) for equality on the entity e.
  • New arithmetic functions decimal:max and decimal:min.
  • New arithmetic function decimal:round2 rounds to the given number of decimal digits after the decimal point.

    Example 52. 

    [12.1235].round2(2) =>  12.12000
    [12.1235].round2(0) =>  12.00000
    [-12.1235].round2(3) => -12.12400
    [-12.1245].round2(3) => -12.12400 

Please refer to the LogicBlox Reference Manual for a complete overview of all the built-in predicates and their usage.

Measure Service

  • It is now possible to indirectly specify an intersection by referring to the intersection of an expression. In ProtoBuf/JSON this is done by setting the expr field of an Intersection message. In the textual format this is done by writing interof (E) where E is the expression used to compute the intersection from.
  • A new option has been added to the measure service configuration to aid in debugging. By setting query_after_fixpoint to true in the measure service configuration file (lb-measure-service.config), the measure service will also query contents of all intermediate child expressions, and if the appropriate log level is set, the result will be visible in the measure service log. This option is helpful in tracking down why a query result was empty or an update made no change.

Developer Tools

  • cloud-store improvements:
    • The -o option is now optional on download. It is inferred from the source URL if not specified.
  • Logging Enhancements:
    • lb-server perf logging now uses journald severity level debug, not notice.
    • lb-web logging is changed to log failed authentication attempts as info, not errors.
  • lb server start now creates a pid file, similar to lb services start.

Corrected Issues

The issues listed below have been corrected since the 4.3.1 release.

  • Datetime creation did not validate the supported year range of the datetime formatting function that we use, which resulted in a database that contains data that cannot be queried. We now restrict the range of years immediately on data import. Currently, the supported range is 1400 - 6771.
  • cloud-store:
    • Retriable exceptions are now logged more clearly to avoid confusion about whether the S3 operation was successful or not.
    • When uploading many files, we now the number of open file descriptors to avoid running out of the maximum allowed number.
    • Empty files are now consistently supported.

Installation and Upgrade information

Installation Instructions

Installing LogicBlox 4.3.2 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.2/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Upgrade Information

  • Admission queues now must be declared in lb-web-server.config before services can be assigned to them.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 7, update 11 or higher
Python 2.7 or higher
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.

LogicBlox 4.3.1

Release Date: August 10th 2015

What's New

Database
  • Default Values: With LogicBlox 4.3.1 we re-introduce the ability to define default values for predicates. Default values make it possible to efficiently work with sparse data by physically storing only facts with values different from the default value (usually non-zero). Default values automatically handle the complications of sparse data representations in rules. For example, with default values, a computation for netsales from sales and returns is simply expressed as:

    net_sales[sk,st,d] = sales[sk,st,d] - returns[sk,st,d].

    The database will transparently handle the cases where only sales or returns have non-zero values. The evaluation is efficient in the sense that it ignores cases where both predicates logically have a zero value.

    This feature is based on an earlier capability in LogicBlox 3.x. Please refer to the Default Values chapter in the LogicBlox Reference Manual for an extensive overview of how to work with default values.

  • Database Branching: LogicBlox 4.3.1 officially introduces database branching as a novel feature of databases. The feature is similar to branching and tagging in source code management system (like Git and Mercurial), but is unique for a database system. In LogicBlox a database can now be branched in constant time, which means that creating a branch will be very quick, and independent of the size of the database. Branches are isolated from each other and can differ in data as well as schema.

    Branches can be used for what-if analysis or other tasks that need long transactions lasting hours, days, or weeks. It can also be used to execute application upgrades with a simple way to back out of the upgrade procedure. Versions of the database can also be tagged simply as checkpoints for recovery or to determine changes made during a time period. Branches are also great for executing testcases from a testsuite in isolation.

    Given that database branching is such a unique and novel feature in databases, we are looking forward to seeing how the community will use this capability.

    Please refer to the Database Branching chapter in the manual for an extensive overview.

  • Databases can now be imported from read-only files.

  • We have re-introduced a feature to log long-running rules. This is a convenient tool for finding performance issues without having to enable full performance logging, which is usually not appropriate for deployed applications. The limit can be configured in lb-server.config or using an environment variable.

    Please refer to the Performance Monitoring chapter in the LogicBlox Administration Manual for details on how to use this feature.

LogiQL Language
  • Added support for POSIX extended regular expression builtins:
    • string:match:
      string:match(s,pattern) -> string(s), string(pattern).

      True if the string s matches the pattern. A pattern is a regular expression in POSIX extended regular expression syntax.

      Example 53. 

      string:match("226-223-4921", """\d+-\d+-\d+""").  // true
      string:match("226-223-4921", """\l+-\l+-\l+""").  // false 
    • string:matches:
      string:matches[s,regex,n]=subexp -> string(s), string(regex), int(n), string(subexp).

      Matches the string s against the regular expression regex and returns the indexed (n) matching subexpressions.

      Example 54. 

      For example:

      matches(n, subexp) -> int(n), string(subexp).
      matches(n, subexp) <- string:matches["aaabc", "(a*)(b).", n]=subexp.

      results in:

      /--- start of matches facts ---\
        0, "aaabc"
        1, "aaa"
        2, "b"
      \---- end of matches facts ----/ 
    • string:rewrite:
      string:rewrite[s, regex, replace]=r -> string(s), string(regex), string(replace), string(r).

      Rewrite the string 's' using the regular expression regex and the replacement text replace.

      Example 55. 

      string:rewrite["afoobbarc", ".*(foo).*(bar).*", """_\1_\2_"""]="_foo_bar_". 

    Please refer to the LogicBlox Reference Manual for a complete overview of all the built-in predicates.

Services Framework
  • Asynchronous TDX: The Tabular Data Exchange (TDX) transaction service now supports an asynchronous variant, which is indicated by an async=true URL parameter to the transaction start request. In async mode, the requests to parts (e.g. TDX import or export requests) and the commit request all return immediately instead of blocking until the workspace transaction completes. Clients can then GET the transaction's poll URL to check the status of the workspace transaction, and GET the transaction URL to retrieve more detailed information. This functionality is also exposed by the lb-web's Java and Python APIs, and by the TdxImport and TdxExport lb-workflow tasks.

  • Generalized admission control: The admission control support in lb-web has been generalized. Previously the only supported policy was "exclusive writes", in which read-only service requests were executed in parallel but write requests were executed exclusively. We have now introduced a more extensible method where we initially support exclusive writes (the previous default), and executing all requests for services assigned to a queue serially. The configuration of the admission control has been moved to lb-web-server.config. Before, admission control queues used to be declared by simply using them in a service.

    Please refer to the LogicBlox Admin Manual for a complete overview of the feature and examples. Please note the backwards compatibility note in the upgrade instructions.

Measure Service

  • Aggregation fusion optimization that attempts to merge generated aggregations if they are aggregating along the same dimensions over the same source data.
  • Optimized ambig aggregation for both the default and non-default value cases.
  • Optimized the average aggregation for the default value case.
  • The notion of the "signature" has been introduced in several places to encapsulate the intersection and type of an expression, as well as whether it is functional or set-valued. This will replace the use of separate intersection and type fields throughout the protocol.
  • ParamExprs now use the order of the dimensions in their intersection to determine their key-order.

Developer Tools

  • cloud-store improvements:
    • cloud-store now automatically aborts any multipart upload that cannot be completed successfully.

    • The new command cloud-store list-pending-uploads will list all pending uploads whose associated keys start with the specified S3 prefix. The prefix should contain, at least, a bucket name.

    • New command cloud-store abort-pending-uploadsto abort a pending upload, either by id or by date/datetime.

      For more information on cloud-store, please refer to the LogicBlox Admin Manual.

Corrected Issues

The issues listed below have been corrected since the 4.3.0 release.

  • Resolved an issue where workspace disk usage would continue to grow quickly for certain workloads, due to continued allocation of unused pages.

  • We have addressed an issue where long-running applications with diverse and large queries could use a significant amount of memory caching query evaluation infrastructure.

  • Resolved a durability issue.

  • Resolved an issue with regard to the memory usage when running lb export-workspace.

  • Transaction aborts are now logged as "---TXN-ABORTED----", instead of an error, to support more accurate monitoring of logs.

  • The lb popcount command, similarly to the lb predinfo command, does not show internal predicates by default anymore.

  • Measure Service:
    • Ratio/even now uses the default aggregation when calculating the value of the OldFixed/NewFixed fixed positions.

Installation and Upgrade information

Installation Instructions

Installing LogicBlox 4.3.1 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.1/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Upgrade Information

  • Admission queues now must be declared in lb-web-server.config before services can be assigned to them.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 7, update 11 or higher
Python 2.7 or higher
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.

LogicBlox 4.3.0

Release Date: July 9th 2015

What's New

Database
  • Pager Embedding: There is no longer a separate BloxPagerDaemon process. The pager is now embedded in the lb-server process. One immediate advantage for users is that we no longer require /run/shm (or /dev/shm) to be larger than the default configuration on most Linux distributions. The lb-server does still use a bit of shared memory, but only for communicating statistics efficiently to client tools.
  • Port Changes: We have changed the default TCP ports used by the lb-server and lb-compiler. This change became necessary when we found out that sporadic failures to start these services were caused by an unfortunate choice of ports. The previous default ports were in the ephemeral port range, which the kernel arbitrarily uses for the source port of a tcp socket. When starting services frequently enough, eventually this will always fail, and we decided it would be wiser to change the default ports. This change should not impact applications, except if there exists external code for establishing connections to the lb-server.

    Note

    Note that the port change might require revision of monitoring tools.

  • The memory usage for workspaces having large numbers of predicates and installed rules has been reduced.
  • The semantics of subtyping has been tightened up and it is no longer possible to insert a supertype into a subtype entity predicate, or from sibling subtypes into each other. This is a temporary measure that will be revisited in the future.

    Example 56. 

    Use cases for this feature can usually be rewritten to use unary predicates instead of entity predicates for attributes of interest. For instance, instead of writing:

    person(_) -> .
    person_by_name[x]=y -> string(x), person(y).
    lang:constructor(`person_by_name).
    
    male(x) -> person(x).
    female(x) -> person(x).
    parent(x) -> person(x).
    lang:entity(`male).
    lang:entity(`female).
    lang:entity(`parent).
    
    // Bob has no kids
    +person(x), +person_by_name["Bob"]=x, +male(x).
    
    // Alice does
    +person(x), +person_by_name["Alice"]=x, +female(x), +parent(x).

    One would now write:

    person(_) -> .
    person_by_name[x]=y -> string(x), person(y).
    lang:constructor(`person_by_name).
    
    // Not entity declarations, just unary predicates
    male(x) -> person(x).
    female(x) -> person(x).
    parent(x) -> person(x).
    
    // Bob has no kids
    +person(x), +person_by_name["Bob"]=x, +male(x).
    
    // Alice does
    +person(x), +person_by_name["Alice"]=x, +female(x), +parent(x).

Services Framework
  • lb web-client support for stateless authentication: lb web-client now supports stateless authentication via the RSA-SHA protocol already supported by lb web-server. In order to make an authenticated call, users must pass username and keyname, and optionally select the keydir to locate the private key (defaults to the directory configured in lb-web-client.config).
Measure Service

  • Significant revision to the implementation for updates:
    • Introduces a distinction between locks on metrics that can be converted to edits and those that should not (these would tend to be used to implement what were called historical locks).
    • New support for dimension locking, that allows locking by a member of some dimension level. For example, it is now possible to lock edits to not change values relating to a particular sku, across all metrics.
    • Metrics can now have inverses defined in CubiQL. The measure service can also now automatically generate inverses for metrics defined via CubiQL.
  • spread-by-metric is now supported as a built in spreading method.
  • Metrics defined by measure language recalcs now inherit the default value of their base intersection at aggregated intersections.
  • Literal expressions:
    • A literal can now be written in CubiQL by enumerating its tuples and specifying their types, using the new signature construct.

      Example 57. 

      A measure with 2 tuples, defined at the intersection {String, Int} and having decimal values:

      { ("a", 1, 10d), ("b", 2, 20d) } : {String, Int} => decimal  

      Example 58. 

      A position-only measure at the intersection {String, Int}:

      { ("a", 1), ("b", 2) } : {String, Int}   

      Example 59. 

      A measure that allows multiple values per key tuple:

      { ("a", 1, 10d), ("a", 1, 20d) } : {String, Int} => Set(decimal) 

      Example 60. 

      With a label and predefined dimensions in keys, and an entity type in the value:

      {("Colgate 20oz", "Atlanta, GA", "1150118")} : {merchandise:sku, Location.store} => `pdx:cal:week` 

    • In protocol buffers syntax, the relevant new messages are LiteralExpr, BaseSignature, and ValueType. The data within a LiteralExpr is maintained in columns, in the same manner as in LiteralTypedRelation message. In the future, the contexts in the protocol syntax that currently make use of LiteralTypedRelation will be generalized to use the more general MeasureExpr. Since LiteralExpr is a kind of MeasureExpr, it will take place of the current usages of LiteralTypedRelation.
    • Measure signatures, whose introduction was prompted by the needs of literal expressions, are expected to see more use in CubiQL, as the types of measure expressions. They will also replace several uses of Intersection + Type field pairs in protocol buffers that currently serve this purpose.

Developer Tools

  • The lb query command now defaults to read-only, to avoid unintended write transactions.
  • Usability improvements to the lb popcount command:
    • lb popcount now raises a concise error if the predicate in question does not exist
    • Improved option parsing for lb popcount to match lb print and lb predinfo commands
    • lb popcount now only shows the non-default facts in the predicate.

Corrected Issues

The issues listed below have been corrected since the 4.2.1 release.

  • Durability bug fixes: Extensive testing has revealed several durability bugs that have been fixed with this release. Durability problems would manifest as strange errors or crashes upon reopening a workspace.

    Important

    Durability bugs are severe problems and we strongly advise users of a previous LB4 release to update.

  • Out-of-memory bug fixes: Out-of-memory errors could occur when exporting large csv files, or when performing large aggregations which produced a small number of result records (e.g., aggregating into a scalar result predicate). These bugs would usually appear as an lb-server process using ever-increasing amounts of memory, becoming sluggish, and eventually being killed by the Linux kernel (OOM killer).

Installation and Upgrade information

Installation Instructions

Installing LogicBlox 4.3.0 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.3.0/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Upgrade Information

  • Using refmode conversion in LogiQL is now treated as an error. The error may still be disabled, but in 4.3.2 the support for refmode conversion will be completely eliminated.
  • Using deprecated numeric precisions in LogiQL is now treated as an error. The error may still be disabled, but in 4.3.2 the support for numeric precisions will be completely eliminated.

Release Information

Server requirements
Operating System: 64 bit Linux; OSX 10.10+ is supported for local development.
Java Runtime Environment 7, update 11 or higher
Python 2.7 or higher
Client requirements
Applications using modeler-js User Interface Components: Google Chrome
Requirements for applications using non-modeler-js components may vary per application.