LogicBlox 4.1.0

Release Date: April 1st 2014

Executive Summary

LogicBlox 4.1 marks the first 4.x release that is feature compatible with LogicBlox 3.x for applications built on the service-oriented architecture. LogicBlox 4.1 includes full support for two of the most commonly used services: tabular data exchange and measure service.

  • Tabular data exchange in 4.1 is now feature compatible with 3.x. Certain configuration differences exist. Please refer to the the LB4 migration guide for details.
  • The performance issues for the measure service have been addressed in this release, and it has at least comparable performance with LogicBlox 3.x, with more rigorous benchmarks forthcoming with the next releases.

What's New

Language Semantics
  • New arithmetic built-ins: Added built-in support for:
    • float:arccos: to calculate the arccosine of x.
    • float:arcsine: to calculate the arcsine of x.
    • float:arctan: to calculate the arctangent of x.
    Please refer to the LogicBlox Reference Manual for a complete overview of all the built-in predicates.
  • Updates to File Predicates:
    • When using a delimited file predicate, it is now required to specify whether the file predicate will be used for import or for export. This is done through a new setting lang:physical:fileMode['_file]=... where the possible values are import, export and import_detailed. If no file mode is specified, importis assumed.
    • The signature of import delimited file predicates now includes an extra argument of integer type, that is used for specifying the offset (in bytes) at which a record occurs in the file. This new argument comes first and is followed by a semicolon.

      Example 19. 

      Instead of

      _file(SKU,WEEK,STORE,SALES)

      one should now write

      _file(offset; SKU,WEEK,STORE,SALES)

      or simply

      _file(_; SKU,WEEK,STORE,SALES).

    • The signature of import_detailed delimited file predicates extends the signature of import delimited file predicates with two further attributes, that come last, and are used for advanced error reporting.

      Example 20. 

      For example, if _file is import_detailed then instead of

      _file(SKU,WEEK,STORE,SALES)

      we write

      _file(offset; SKU,WEEK,STORE,SALES, errorCode, rawLine)

      where errorCode is an integer that is non-zero if the record could not be parsed correctly, and rawLine is a string containing the entire (unparsed) line of the input file.

    Note

    The signature of "export" delimited file predicates remains unchanged (and, in particular, does not include an offset attribute).

Services Framework

The Tabular Exchange Services are now feature complete with respect to LogicBlox 3.x. LogicBlox 4.1.0 includes support for the following features:

  • Transform Functions: TDX now supports Transform Functions that can be used to transform values prior to importing values to predicates in a workspace or to exporting values to files. The feature can be used to achieve the same effect as the "transforms" feature of TDX on LogicBlox 3.x, but offers more flexibility since developers can define their own functions. Please refer to the LogicBlox Reference Manual for an overview of the transformation functions supported and some examples.
  • Auto primitive and refmode conversions: While generating code for imports and exports, TDX may include primitive and refmode conversions if necessary. This makes it possible, for example, to bind a string column to an int value: (TDX will generate string:int:convert for imports and int:string:convert for exports). Multiple conversions may be necessary. For example, binding a string column to an entity with a float refmode needs a string:float:convert followed by a refmode lookup on imports.
  • New column format "char": Added support for the char TDX column format. This format checks that the values have a single (string or integer) character.
  • Abort on error: A TDX service now by default aborts the transaction on any error. To allow partial imports, the allow_partial_import flag must be set.

    Example 21. 

    service_by_prefix["/a-tdx-service"] = x,
    delim_service(x) {
          delim_file_binding[] = "file-binding",
          allow_partial_import()
    }.
    

    Note

    In LogicBlox 3.x, the default behavior for a TDX service was to accept partial imports. During an import, TDX would accept any correct row, and would discard rows with errors.

  • TDX import now reports back rows that are malformed, i.e., that cannot be parsed by a file predicate according to the specified columns. The whole erroneous line is returned in the CAUSE column, with a MALFORMED_ROW CAUSE_CODE.

    Example 22. 

    For example the following file contains two columns:

    USER|LOCALE
    John|en
    Mary|en|
    Joao|pt
    
    Maria
    "Johannes"|de|zuviel
    

    This file contains the following errors:

    • Mary has 3 columns, thanks to the last |
    • Maria has a single column
    • Johannes has 3 columns

    Note that empty lines are discarded. A POST to the service would return the following file:

    USER|LOCALE|CAUSE|CAUSE_CODE
    ||"Mary|en|"|MALFORMED_ROW
    ||"Maria"|MALFORMED_ROW
    ||"Johanes|de|zuviel"|MALFORMED_ROW
    

  • Un/zip support: TDX now supports GZIP compression when importing or exporting data from a workspace.

    Note

    Please note that if an uncompressed file is passed to lb web-client, it needs to be indicated explicitly that compression is not used with the -n or --no-compression flags.
  • Offset exposed as file header: TDX now exposes a file offset as a special (reserved) column header named TDX_OFFSET, which can then be used in predicate bindings.

    Example 23. 

    In the example below we bind the sales file, which has a single column, to the predicate measures:sales, which is a functional predicate from int to int.

    sales[x] = v -> int(x), int(v).
    
    file_definition_by_name["sales"] = fd,
    file_definition(fd) {
      file_delimiter[] = "|",
      column_headers[] = "SALES",
      column_formats[] = "int"
    },
    file_binding_by_name["sales"] = fb,
    file_binding(fb) {
      file_binding_definition_name[] = "sales",       
      predicate_binding_by_name["measures:sales"] =
        predicate_binding(_) {
          predicate_binding_columns[] = "TDX_OFFSET, SALES"
        }
    }.
    

    Upon import, the offset of the rows in the file are be bound to the TDX_OFFSET column and then used to populate measure:sales. Upon export, the value in the column bound to TDX_OFFSET is ignored (equivalent to measures:sales\[_\] = SALES).

    TDX_OFFSET works as any integer column: it can be bound to an entity, accumulated, be subject of transformation functions, etc. The value of the offset is the number of bytes from the beginning of the file up to the row being imported and, therefore, guaranteed to be unique and monotonically increasing in the file being imported.

Other new features of the LogicBlox Services Framework:

  • Lazy services: Services can now be configured to be lazy, in which case they will only be initialized when first used. This new feature may be used to speed up initialization times, with the drawback that errors are only caught when the service is used.

    Example 24. 

    In the example below the TDX lazy service will only be initialized the first time /sales is used.

    service_by_prefix[ "/sales" ] = x,
    delim_service(x)  {
        delim_file_binding[] = "sales",
        lazy()
    }.
    
  • Support for declaring target workspace in services: It is now possible to have a workspace declare a service that is executed in another workspace. This allows, for example, to speed up the web-server startup time by scanning only the workspaces that declare services, specially if used with lazy services.

    Example 25. 

    The declaration below can be installed in a service-declaration-workspace. When the web-server scans this workspace, it will load the service. However, because it is lazy, it will not immediately initialize the service. When the first request to /sales arrives, the web-server will initialize the service, which includes looking up the sales file binding. The service will then look up sales in the service-host-workspace, and will execute every service request in that workspace.

    service_by_prefix[ "/sales" ] = x,
    delim_service(x) {
        delim_file_binding[] = "sales" ,
        service_host_workspace[] = "service-host-workspace",
        lazy()
        }.
    

Measure Service

  • Protobuf-based update requests: User edits for updates/spreads can now be sent via a protobuf-based format, in addition to TDX.

    Example 26. 

    kind: UPDATE
    update_request {
      expr {
        kind: SPREAD
        metric: "Population"
        inter {
          qualified_level {
            dimension : "BadGuy"
            level : "Foe"
          }
        }
      }
      source {
        intersection {
          qualified_level {
            dimension: "BadGuy"
            level: "Foe"
          }
        }
        type: { kind: FLOAT }
        column {
          string_column {
            value: "Moblin"
            value: "Like Like"
          }
        }
        column {
          float_column {
            value: 42.0
            value: 312.0
          }
        }
      }
    }
    
  • Multi-measure updates: A single update request may now contain more than one update. This is only accessible through the new protobuf-based update interface. Updates can be made to several metrics, including multiple updates to the same metric. However, if multiple edits are made to the same metric, all input intersections must be comparable in the lattice order.
  • Named and composite aggregations and spreads: It is now possible to specify the aggregation and spreading method by name. This is mostly useful for naming a composite aggregation/spreading, but it can also be used to alias primitive aggregation/spreading methods. While composite aggregations allow specifying that a particular primitive aggregation method be used when aggregating up along a given dimension, composite spreads can be used to specify that a particular primitive spreading method be used when spreading down along a given dimension..

    Example 27. Named and composite aggregations

    Named aggregations can be defined via the aggregation field of the MeasureModel message, which is set of AggDef messages. Using our logic configuration library, a primitive aggregation may be aliased via

    +measure:config:aggregation("Sum") {
      +measure:config:aggregation_primitive[]="TOTAL"
    }.
    

    A composite aggregation can be defined like

    +measure:config:aggregation("TotalThenMax") {
      +measure:config:aggregation_composite[0,"Product"]="TOTAL", 
      +measure:config:aggregation_composite[1,"Calendar"]="MAX"
    }.
    

    Note

    Note that the order is important, thus the need to provide the index.

    A named aggregation can then be used anywhere where an aggregation method may be specified. In JSON, for example we could write:

    "method": { "named": "TotalThenMax" },
    

    Example 28. Named and Composite Spreads

    Named spreads can be defined via the spread field of the MeasureModel message, which is set of SpreadDef messages. Using our logic configuration library, a primitive spread may be aliased via

    +measure:config:spread("RatioEven") {
      +measure:config:spread_primitive[]="RATIO"
    }.
    

    A composite spread can be defined like

    +measure:config:spread("RatioThenEven") {
      +measure:config:spread_composite[0,"Product"]="RATIO",
      +measure:config:spread_composite[1,"Calendar"]="EVEN"
    }.
    

    Note

    Similar to the composite aggregations, the order is also important when defining composite spreads, thus the need to provide the index.

    A named spread can then be used anywhere a spread kind may be specified. In JSON, for example we could write

    "spread_kind": { "named": "RatioThenEven" },
    

  • Measure language rule installation: The measure service now understands the measure rule language that was used in Blade applications. It is possible to install such rules into the measure service using an InstallRequest that now has a string field called rules. Alternatively, the command-line measure install tool can be used to install rules like:
    lb measure-service install --rules --uri http://localhost:8080/measure my.rules
    
    Successfully installed rules are persisted in the workspace backing the measure service, so that if the measure service is restarted you do not need to reinstall the rules.

    Note

    The measure service currently uses a more naive strategy for selecting primary formulas than the LogicBlox 3.x measure engine, by simply attempting to pick the first formula of each rule.

  • Recalc metrics: It is now possible to specify that a metric be backed by a recalc rule when requesting it an intersection other than the base.

    Example 29. 

    In the example configuration logic below a recalc metric NetSales is defined by

    +measure:config:metric("NetSales") {
      +measure:config:metric_usesRule[]="NetSales",
      +measure:config:metric_hasIntersection[]="sku,store,week",
      +measure:config:metric_hasType[]="FLOAT"
    }.
    

    The usesRule line tells the measure service to use the first formula in the given rule as the formula to be generated at the requested intersection. Then if NetSales is requested at an intersection other than base, like

    "expr": {
      "kind": "METRIC",
      "metric": { 
        "name": "NetSales",    
        "inter": { 
          "qualified_level": [ 
            { "dimension": "Product", "level": "class" }
            { "dimension": "Location", "level": "region" }
            { "dimension": "Calendar", "level": "month" } ] 
       }
     }
    

    the appropriate recalc logic will be generated as needed.

  • Optional predicate specifications in configuration logic: In the configuration logic, it is now possible to omit the name of the predicate backing a metric if it matches the metric name.

    Example 30. 

    Instead of

    +measure:config:metric("PriceCloseDollars") {
      +measure:config:metric_usesPredicate[] = "PriceCloseDollars",
      +measure:config:metric_hasIntersection[] = "Security,Day",
      +measure:config:metric_hasType[]="DECIMAL"
    }.
    

    it is now possible to simply write

    +measure:config:metric("PriceCloseDollars") {
      +measure:config:metric_hasIntersection[] = "Security,Day",
      +measure:config:metric_hasType[]="DECIMAL"
    }.
    

  • Default aggregation of metrics: If a metric has a default aggregation (or is a recalc rule) it is possible to request it at a specific intersection rather than first having to start at its base intersection.

    Example 31. 

    For example, if the default aggregation method of the Sales metric is TOTAL, instead of writing

    "expr": { "kind": "AGGREGATION",
      "aggregation": {
        "method": { "primitive": "TOTAL" },
        "expr": {
          "kind": "METRIC",
          "metric": { "name": "Sales" }
        },  
        "grouping": [ { "kind": "ALL", "dimension": "Product" } ]
      }
    }
    

    you can now write

    "expr": {
      "kind": "METRIC",
      "metric": { 
        "name": "Sales",    
        "inter": { 
          "qualified_level": [ 
            { "dimension": "Location", "level": "store" }
            { "dimension": "Calendar", "level": "week" } ] 
       }
     }
    

  • Aggregation to intersection: It is now possible to specify directly the intersection of the desired result when aggregating a measure expression.

    Example 32. 

    In earlier releases, to aggregate a measure expression it was necessary to specify a set of grouping operators to indirectly describe the intersection for the result of the aggregation:

    "expr": { "kind": "AGGREGATION",
      "aggregation": {
        "method": { "primitive": "TOTAL" },
        "expr": {
          "kind": "METRIC",
          "metric": { "name": "Sales" }
        },  
        "grouping": [ { "kind": "ALL", "dimension": "Product" } ]
      }
    }
    

    Now it is possible to write the following:

    "expr": { "kind": "AGGREGATION",
      "aggregation": {
        "method": { "primitive": "TOTAL" },
        "expr": {
          "kind": "METRIC",
          "metric": { "name": "Sales" }
        },  
        "inter": { 
          "qualified_level": [ 
            { "dimension": "Location", "level": "store" }
            { "dimension": "Calendar", "level": "week" } ]
        }
      }
    }
    

  • New and changed spreading methods:
    • Measure service EVEN spreading has be renamed to DELTA.

      Example 33. 

        Before DELTA spread After
      P1 4.0   6.0
      P2 2.0   4.0
      Q (Total P1+P2) 6.0 10.0 10.0
    • When using the new EVEN spreading operator, the new value of an aggregated predicate is spread down evenly over the lower level predicates.

      Example 34. 

        Before EVEN spread After
      P1 4.0   5.0
      P2 2.0   5.0
      Q (Total P1+P2) 6.0 10.0 10.

Corrected Issues

The issues listed below have been corrected since the 4.0.8 release.

  • Resolved an issue where compiler-generated ordered entity predicates were not recognized when used in modules.
  • Plugin Logic is now loaded with the same classloader as the handlers in the jar. Previously, lb-web-server used to create a separate classloader for plugins than it was using for handlers packaged in a jar, which caused sharing of static state impossible between a plugin and a handler.
  • Resolved an issue where any error that prevents an import via TDX (such as missing headers) was causing a status 500 to be sent to the client. Now these errors lead to a status 4xx.
  • It is now possible to use the lb web-server import-users command to import users from a delimited file. Please refer to the LogicBlox Reference Manual for more information on this command.
  • Resolved an issue that caused an exception when TDX export involved certain refmode conversions. In particular, this issue could cause the credential services file export to fail.

Installation and Upgrade information

Installation Instructions

Installing LogicBlox 4.1.0 is as simple as following the steps outlined below:

  1. Download the installation package.
  2. Extract the tarball in <YourPreferredInstallDirectory>
  3. Run the following command:
    source <YourPreferredInstallDirectory>/logicblox-4.1.0/etc/profile.d/logicblox.sh
    
    NOTE: this script will set all the necessary environment variables. You might want to add this command to your .bashrc.

Upgrade Information

  • Import file predicates now require an extra attribute: _file(x,y,z) is now _file(offset; x,y,z).
  • Export file predicates now require a pragma: lang:physical:fileMode[`_file] = “export”.
  • Aggregation methods are no longer enums: In order to support named and composite aggregations, the method enum has been converted to a message. Existing uses of aggregation methods need to be rewritten to the new form.

    Example 35. 

    "method": "TOTAL" ,

    becomes

    "method": { "primitive": "TOTAL" } ,
  • Types are no longer enums: In order to ease expansion of types and eventually allow entity-typed metrics and attributes, type has been changed from being an enum to a message. Existing uses of type need to be rewritten to the new form.

    Example 36. 

    "type": "STRING" ,

    becomes

    "type": { "kind": "STRING" } ,
  • Spreading kinds are no longer enum: In order to support named and composite spreads, the SpreadKind enum has been converted to a message. Existing uses of spread kinds need to be rewritten to the new form.

    Example 37. 

    "spread_kind": "RATIO" ,

    becomes

    "spread_kind": { "primitive": "RATIO" } ,
  • Ratio/even spreading semantics have changed What was previously called "even" spreading is now called "delta". See what's new for more details.
  • Dimensions with more than one hierarchy are now required to have a default: In previous releases, specifying a default hierarchy was optional. Now this is required if a dimension has more than one hierarchy.

    Example 38. 

    Example of specifying a default hierarchy, when using the LogiQL measure configuration library:

    +measure:config:dimension(dim) {
      +measure:config:dimension_hasName[]="MyDim",
      +measure:config:dimension_hasDefaultHierarchy[]="MyHierarchy"
    }.
    

Permanently Removed Features

  • Metadata service is no longer supported: The previously deprecated ModelRequest and ModelResponse messages have been removed, and the MetadataService no longer exists. The model may be retrieved via the MODEL_QUERY kind of the Request message.
  • Filter comparisons only use expressions instead of terms or expressions: The previously deprecated term field has been removed from the Comparison message. Filtering is now done exclusively using the expr field of the Comparison message. Because Terms are a subset of MeasureExprs, this only requires a minimal change in queries.

    Example 39. 

    For example, the comparison

    "comparison": [ {
         "op": "EQUALS", 
         "term": {
           "kind": "CONSTANT",
             "constant": { "string_constant": "Atlanta, GA" }
          }
     } ]
    

    would be rewritten to

    "comparison": [ {
         "op": "EQUALS", 
         "expr": {
           "kind": "TERM",
           "term": {
             "kind": "CONSTANT",
               "constant": { "string_constant": "Atlanta, GA" }
            }
         }
    } ]
    

  • Legacy measure expression syntax is no longer supported: The previously deprecated field measure_text has been removed from the Binding and InstallRequest messages.

Release Information

Table 10. 

Server requirements
Operating System: 64 bit Linux
Java Runtime Environment 1.7, update 11 or higher
Python 2.7 or higher