Structuring and Compiling LogicBlox Applications

After following our “LogiQL in 30 minutes” tutorial you may be wondering: “but are LogicBlox applications really developed entirely in a REPL?” The answer to this, of course, is: no. The REPL is a useful tool for learning and testing ideas, but is not generally used to develop full-blown applications. LogicBlox is different to what you may be used to in many ways, but you will be relieved to learn that “real” LogicBlox applications are developed similar to applications using most other technology stacks: by typing code into text files, structured neatly into directories. In this article we will explore how to structure and compile applications, and how to load compiled code into a workspace.

LogicBlox applications typically consist of various components, e.g.:

1. Back-end: code that lives in the LogicBlox database.
2. Web client: the code that lives in the browser and communicates with the back-end using web service (AJAX) calls.
3. Data integration: scripts that pull data from various data sources and import them to your LogicBlox application for processing, usually using LogicBlox’ Data Exchange Services.

In this article we will focus on structuring your application’s back-end code base.

Back-end code is organized into projects, often just one, but sometimes more. A project in turn consists of one or more modules, which you can think of as packages in your application. A module in turn consists of one or more source files (called “blocks”), for instance .logic files containing LogiQL, or .proto files containing protocol buffer definitions.

You compile, run tests and package your application using the lb config tool, which will produce a standard Makefile based on an application specification. When the Makefile is generated you can compile your application by running make, run tests using make check and package using make dist.

Let’s have a look at an example inspired by our ice cream emporium application. For this application we defined a number of predicates:

• cost: tracking the per ice cream cost
• price: tracking the selling price per ice cream
• week_sales: tracking number of sales per ice cream per week
• week_revenue: calculating the revenue per ice cream per week based on sales and price.
• profit: calculating the profit per ice cream (based on cost and price)
• week_profit: calculating the profit per ice cream per week

We will reimplement these predicates, previously added to a workspace in an ad-hoc way as a LogicBlox application. Since our project is small, we will put all predicates into a single module called core. As we expand the example, we will refactor this structure as required.

Directory structure

The full source code of our example can be found in this bitbucket repository. The structure used in this project looks as follows:

• config.py: the LB config configuration file
• application.project: the project file for our application project
• core/: the directory containing the logic for our core module
  • icecream.logic: a LogiQL source file with predicates for tracking ice cream attributes like cost and price.
  • sales.logic: a LogiQL source file with all sales related predicates.
  • revenue.logic: a LogiQL source file with all revenue related predicates.
  • profit.logic: a LogiQL source file with all profit related predicates.
• data/: a directory containing some initial test data
  • init_data.logic: test values for our predicates

Before we look at the .logic files, let’s first have a look at the config.py and application.project files.

Project files

LogicBlox project files have a very simple comma-separated-value format. Here’s the initial version of our project file:

application, projectname
core, module
data/init_data.logic, execute

The first value on each line represents the value of a project entry and the second represents the entry type. This particular project file defines:

1. application is the name of the project;
2. it has a module named core; and
3. upon installing the project into a workspace, the data/init_data.logic file should be “exec”ed (like the exec REPL command) — this file contains some initial sample data. Incidentally, this is not how you should import large amounts of data, but for a couple of values it’s ok.

That wasn’t so hard. Let’s move on to config.py.

config.py

Our project’s config.py file describes the structure of a project, its dependencies, the libraries it consists of, unit tests to be run to verify its correct operation, and the workspaces to be created for testing. As you can probably tell from its file name, config.py is indeed a Python script, but you do not to know much Python to read or extend it. Here’s the config.py for our basic application:

//lang:python
from lbconfig.api import *

lbconfig_package('application', version='0.1',
                 default_targets=['lb-libraries'])

depends_on(logicblox_dep)

lb_library(name='application', srcdir='.')

check_lb_workspace(name='application', libraries=['application'])

In regular English: this config.py describes how to build the LogicBlox application named application version 0.1 and that the default make target to run is lb-libraries (that is: all libraries defined in this file). The package depends on LogicBlox (obviously), and consists of one library named application whose source files are stored in the same directory as the config.py file.

For convenience we’ve added a “check workspace” definition, which will automatically create a fresh workspace for you with the name application and will load the application library into it. This is useful for manually checking that it’s indeed possible to load your predicates into an actual workspace.

Logic

Once upon a time, LogicBlox applications were built simply by writing LogiQL definition after LogiQL definition in plain .logic files, very similar to how we’ve seen in the REPL tutorial:

//lang:logiql
cost[icecream] = c ->
  string(icecream), int(c).

price[icecream] = p ->
  string(icecream), int(p).

While very simple, this approach has a few disadvantages:

1. Potential name clashes. This is not an issue for small projects, but as projects get big (and many LogicBlox applications contain hundreds or even thousands of predicates) it becomes useful to organize predicates into their own namespaces, similar to packages in Java. So that you can have multiple sales predicates, for instance.
2. Compilation times. Adding code to a LogicBlox workspace using addblock is quick if you just have a few predicates, but the speed of an addblock quickly degrades as the number of predicates grows due to the complicated checks that the compiler performs to ensure the code is valid. Therefore, a mechanism was required to not have to recompile and reanalyze the entire code base every time you change a single line.

To solve these issues LogicBlox now has a proper module system with separate compilation. The module system provides namespacing and aliasing of names for convenience. For instance, if you have a module named core, with a block named icecream with the above two predicates in them, these predicates will be named core:icecream:cost and core:icecream:price.

Here’s what a LogiQL source file using the module syntax looks like:

//lang:logiql
block(`icecream) {
  export(`{
    cost[icecream] = c ->
      string(icecream), int(c).

    price[icecream] = p ->
      string(icecream), int(p).

  })
} <-- .

All definitions are enclosed in a block whose name has to match the file name. So in this case, the file name should be icecream.logic. A module can have aliases, exports and clauses. This particular file only has exports and exports two predicates: cost and price. The export section of a file defines its external interface. The export section can only contain predicate signatures, that is: the predicate name, its arguments and types for all arguments. Any other definitions, for instance rules that derive their values from other predicates as well as extra constraints, have to go into the clauses section.

To show how this works, the profit.logic file uses all three sections:

//lang:logiql
block(`profit) {
  alias_all(`core:icecream),
  alias_all(`core:sales),
  alias_all(`core:revenue),
  export(`{
    profit[icecream] = value -> string(icecream), int(value).

    week_profit[icecream, week] = value ->
      string(icecream),
      int(week),
      int(value).

    agg_profit[week] = value -> int(week), int(value).

    agg_revenue[week] = value -> int(week), int(value).
  }),
  clauses(`{
    profit[icecream] =
      price[icecream] - cost[icecream].

    profit[_] = value -> value >= 0.

    week_profit[icecream, week] =
      profit[icecream] * week_sales[icecream, week].

    agg_profit[week] = value <-
       agg<<value=total(p)>> week_profit[_, week] = p.

    agg_revenue[week] = value <-
       agg<<value=total(r)>> week_revenue[_, week] = r.
  })
} <-- .

The alias_all constructs import all predicate names from the core:icecream, core:sales, core:revenue namespaces so that they can be referenced without having to use their fully qualified names. If you prefer, you can also alias them with a custom prefix, e.g.:

//lang:logiql
alias(`core:icecream, `ic),

To be able to refer to them via ic:cost and ic:price.

We also see that the export section contains signatures for all predicates, including the ones whose facts are derived. This is obligatory when using modules. The reason is explicitness: to find out the signature of a predicate, you can simply open its .logic file and look at the export section, without having to consider where its facts are derived from, what the types of those predicate’s arguments are and so on.

Finally, the clauses section contain the implementation of the predicates. For instance, the facts in profit are calculated by subtracting cost from price. Additional constraints — like that the profit on any ice cream should always be zero or more — are also placed here.

Building and testing

So, now we have our project properly structured, how do we compile and test it? This depends on how you installed LogicBlox.

If you chose to go the Vagrant route, you need a Vagrantfile in your project. Our repository already contains one so you should be all set after copying or symlinking a LogicBlox release tarball into the checkout directory. To setup the Vagrant VM, simply run:

$ vagrant up
$ vagrant ssh

You should now get a Linux bash prompt inside of the VM with LogicBlox installed and running.

If you installed LogicBlox natively you’re all set already, just make sure that the lb services are running:

$ lb services status

If not, start them with:

$ lb services start

In either setup, to compile the project, simply run:

$ lb config
$ make

To load your code into a workspace for testing run:

$ make check check-ws-application

You can now open the command-line REPL (very similar tot he Web-based REPL) using:

$ lb

and then opening the test workspace (which we named application) using:

lb> open application

From here we can list all predicates (note that this REPL also lists built-in predicates, not just custom ones) and print their contents. For instance:

lbi application> list
lbi application> print core:profit:agg_profit

To end the REPL session, simply run the exit command:

lbi application> exit

This covers the bare basics of structuring and compiling your LogicBlox projects. To learn more about LogicBlox’ module system, read the reference manual chapter about it. More details about lb config are also available in the reference manual, and a full list of lb config commands is available as well.