Chapter 42. BloxOptimize

BloxOptimize is an extension of LogiQL that can express optimization problems under linear, integer, and quadratic constraints.

42.1. Introduction

BloxOptimize is partitioned into the following components:

  • The BloxOptimize front-end preprocessor translates the source LogiQL file into an intermediate language, in which the linear constraints are represented in a more conventional way, i.e., in algebraic notation.

  • The BloxOptimize middle-end preprocessor translates the intermediate file into a clean LogiQL file, which, given the appropriate trigger, will produce input for the underlying optimizer in the form of a table.

  • Several optimizer handlers. Given the appropriate trigger, one of the handlers will initiate a calculation using the underlying optimizer. Then, it will populate the database with the results. Each supported underlying optimizer has its own handler.

The two preprocessors are included in the distribution (executable bin/bloxopt). The handlers are included under directory libexec.

The following sections explain:

42.2. Source Logic

In this section we describe the source language for BloxOptimize. The source language is LogiQL, extended with certain features that are specific to BloxOptimize. Because of these features, a BloxOptimize source program cannot be compiled directly by LogicBlox: it must first be passed through the BloxOptimize preprocessor (see Section 42.4.2, “Running the preprocessor”).

42.2.1. Working Problem

Here is our working scenario: a nutrition specialist aims at improving efficiency of ordering food, by composing individual food items into meal packages that are both inexpensive and nutritious. Given a set of food products, their prices and nutritional contents, as well as the minimum required amounts for a set of nutrients, she wants to plan meals that meet the nutritional requirements at the lowest possible cost. Let

  • FOOD be the set of food items;

  • NUTR be the set of nutrients of interest;

  • amt(n, f) be the amount of nutrient n in food item f;

  • nutrLow(n) be the minimum required amount of a nutrient n;

  • cost(f) be the cost of food item f;

  • Buy(f) be the amount of food item f that she will decide to buy.

The nutritionist wants to find appropriate values for Buy(f) (for all f ∈ FOOD), so the variables of the problem are all quantities Buy(f). She wants to minimize the cost of the food items bought, thus the expression

f ∈ FOODcost(f)⋅Buy(f)

is going to be the objective function of the problem. She has to abide by the lower bounds for the nutrients of the food items, provided by the function nutrLow, therefore the constraints of her problem are

∀ n ∈ NUTR ∑f ∈ FOOD amt(n, f)⋅Buy(f) ≥ nutrLow(n)

This is a typical LP (linear programming) problem, if we assume that functions amt, nutrLow, and cost are actually known constants. In our case, these parameters of the problem come from the contents of a LogiQL database. By changing the contents of the database, we can therefore create a new LP problem, similar in the sense that the model remains the same. Since it is parameterized by the contents of the database, a LogiQL optimization model is in fact a template for infinitely many optimization problems.

The model in our source language (extended LogiQL) is as follows:

// entities
FOOD(f), FOOD_id(f:s) -> string(s).
NUTR(n), NUTR_id(n:s) -> string(s).

// predicates populated in the database
amt[n, f] = v -> NUTR(n), FOOD(f), float(v).
nutrLow[n] = v -> NUTR(n), float(v).
cost[f] = v -> FOOD(f), float(v).

// unknown predicate
Buy[f] = v -> FOOD(f), float(v).
lang:solver:variable(`Buy).

// objective function and target (minimize)
objective[] = v -> float(v).
objective[] = v <- agg << v=total(z) >>
  FOOD(f), cost[f] = v1, Buy[f] = v2, z = v1*v2.
lang:solver:minimal(`objective).

// more predicate definitions
totalNutr[n] = v -> NUTR(n), float(v).
totalNutr[n] = v <- agg << v = total(z) >>
  NUTR(n), FOOD(f),
  amt[n, f] = v1, Buy[f] = v2, z = v1*v2.

// constraints
NUTR(n), totalNutr[n] = v1, nutrLow[n] = v2 -> v1 >= v2. 

This is the whole LogiQL model. Two BloxOptimize-specific pragmas are introduced here:

  • lang:solver:variable introduces a predicate whose values are the variables of the MIP problem;

  • lang:solver:minimal introduces the objective function and dictates that the function should be minimized. The corresponding maximization problem is introduced by pragma lang:solver:maximal.

Further BloxOptimize-specific features of the front-end language will be discussed below.

42.2.2. LogiQL versus MIP Terminology

Before moving any further with the description of the source language, we must clarify a point of terminology: the word variable means two different things. In the context of MIP (mixed-integer programming), a variable is one of the quantities we solve over. For example, in the above optimization problem, the MIP variables are Buy(f) for all food items f found in the database. If there are n food items in the database, then the optimization problem has n variables. In the context of LogiQL, a variable is an identifier (other than a predicate name) used in a rule or constraint. For example, in the LogiQL constraint

NUTR(n), totalNutr[n] = v1, nutrLow[n] = v2 -> v1 >= v2. 

we use the LogiQL variables n, v1, v2. To avoid the ambiguity, we will refer to variables in the MIP sense simply as variables and to variables in the LogiQL sense as LogiQL variables.

A similar confusion arises with the term constraint. For example, the above LogiQL constraint corresponds to m MIP constraints, where m is the number of nutrients contained in the database. We will follow the same convention here, referring to the constraints in the MIP sense as constraints and to constraints in the LogiQL sense as LogiQL constraints.

42.2.3. Grouping

BloxOptimize allows for the possibility of using indexed objectives to solve parallel independent MIP problems. For example, let us extend the nutrition problem as follows. Our dietitian works on several different diets. Each diet has different needs in terms of nutritional elements, for example an athlete may needs more protein than the average person. Now nutrLow is a function not only of nutrients but also of diets. The total cost, which is our objective function, is now indexed by diets. In fact, if we have n diets, we must solve n MIP problems, one per diet.

The BloxOptimize front-end makes it possible to do that by introducing arguments to the objective and a grouping predicate. A group-by predicate G(x, y, ...) indicates that we want to solve the optimization problem described in our file, independently for every combination of the possible values of x, y, ... . The arity of the grouping predicate is unrestricted. Grouping is specified by using the pragma lang:solver:groupby.

The working example is modified as follows:

// entities
FOOD(f), FOOD_id(f:s) -> string(s).
NUTR(n), NUTR_id(n:s) -> string(s).
DIET(d), DIET_id(d:s) -> string(s).

// predicates populated in the database
amt[n, f] = v -> NUTR(n), FOOD(f), float(v).
nutrLow[n, d] = v -> NUTR(n), DIET(d), float(v).
cost[f] = v -> FOOD(f), float(v).

// unknown predicate
Buy[f, d] = v -> FOOD(f), DIET(d), float(v).
lang:solver:variable(`Buy).

// objective function
objective[d] = v -> DIET(d), float(v).
objective[d] = v <- agg << v=total(z) >>
  FOOD(f), DIET(d),
  cost[f] = v1, Buy[f, d] = v2, z = v1*v2.
lang:solver:minimal(`objective).

// more predicate definitions
totalNutr[n, d] = v -> NUTR(n), float(v).
totalNutr[n, d] = v <- agg << v=total(z) >>
  NUTR(n), FOOD(f), DIET(d),
  amt[n, f] = v1, Buy[f, d] = v2, z = v1*v2.

// constraints
DIET(d), NUTR(n),
totalNutr[n, d] = v1, nutrLow[n, d] = v2 ->
  v1 >= v2.

// grouping
lang:solver:groupby(`DIET). 

42.2.4.  The BloxOptimize Source Language: Conventions and Restrictions

Variable and Dynamic Predicates

A predicate p that appears in a lang:solver:variable pragma is called a variable predicate. If p is a variable predicate, then all possible valuations of the expression p(x,y, ...) are variables of the MIP problem. In the diet example, Buy is a variable predicate. A variable predicate should never occur in the head of a rule.

A predicate is dynamic if it is a variable predicate or if it occurs in the head of a rule whose body contains a dynamic predicate. In the diet example, the dynamic predicates are Buy, totalNutr and objective.

A non-variable dynamic predicate may occur in the head of a rule only once. We say that this rule defines (or is the definition of) that predicate. All the arguments in the head occurrence of the definition of a dynamic predicate must be distinct LogiQL variables. The body of a definition must be either a conjunction of atoms or an agg<<total>> of a conjunction of atoms.

There should be no direct or indirect recursion among the definitions of dynamic predicates. In the diet example, objective depends on totalNutr, which in turn depends on Buy, thus there are no dependency cycles. But if we had introduced a dependency by including objective in the definition of totalNutr, the result would not have constituted a valid MIP model, and the BloxOptimize preprocessor would have reported an error.

In the definition of any dynamic predicate the grouping predicate, if one exists, must occur exactly once, with arguments which should be distinct LogiQL variables. These arguments must also appear among the key arguments of the predicate under definition.

LogiQL Types

Every dynamic predicate should have exactly one value argument of type float. The predicate can also have zero or more key arguments, each of which must be of a finite type: this means the native types string, float, decimal and int are not allowed in key arguments. In the diet example, the predicate Buy has one value argument of type float, and two key arguments, whose types are both finite: FOOD and DIET.

If there exists a grouping predicate, it should have no value arguments and all its key arguments must be of finite types. In the diet example, the grouping predicate DIET has only one key argument, of type DIET.

The objective must have exactly one float value. Its keys must match exactly the type of the grouping predicate (or there should be no keys if there is no grouping predicate). For example, in the diet example, the type of the grouping predicate dictates that the objective predicate must have one key argument of type DIET and one value argument of type float. Typically the objective is a dynamic predicate; otherwise there is nothing to optimize.

Constraints

A LogiQL constraint that contains dynamic variable predicates translates to a set of constraints of the MIP problem. We call such a LogiQL constraint a MIP-LogiQL-constraint.

The body of a MIP-LogiQL-constraint is a conjunction of atom occurrences (arithmetic comparisons count as atoms). If there is a grouping predicate, it must occur in the body, following the same rules as in the definitions.

The arguments in an atom occurrence in the body or head of a MIP-LogiQL-constraint may only be LogiQL variables, wildcards, or constants. For example p(f[x]) is not acceptable. The modeler has to introduce an extra LogiQL variable, as in: p(v), v=f[x]. Notice that we wrote

NUTR(n),totalNutr[n] = v1, nutrLow[n] = v2 -> v1 >= v2. 

instead of

NUTR(n) -> totalNutr[n] >= nutrLow[n]. 

as the latter would introduce illegal arguments in the occurrence of the built-in predicate >=.

The reader may wonder here why our example atom occurrences nutrLow[n] = v2 and v = f[x] are legal. Indeed, it seems that these are occurrences of the built-in predicate = with illegal arguments nutrLow[n] and f[x]. This is not so, because the form v = f[x] in LogiQL is simply a different way of writing f(x;v), which does not involve the predicate = and is legal.

The head of a LogiQL-MIP-constraint may be only a conjunction of mathematical comparisons between LogiQL variables, e.g., v = x, y >= w. The following three comparison operators are allowed: =, >= and <=.

Other Restrictions

The BloxOptimize source language is subject to two more restrictions:

  • The front-end parser does not support hierarchical constructs.

  • There is no support for predicate or block name aliasing in the presence of modules.

42.2.5. Non-Continuous Values

For the time being, the constructs that we have shown suffice to express pure LP problems, i.e., the restriction that some variables range over integers is not yet expressible. One would expect that a variable predicate whose value type is int would introduce IP (integer programming) in the model. This is not so; in fact the LogiQL types of all variables must be declared float. To introduce variables that range over sets other than the reals we use the following BloxOptimize predicates:

bloxopt_binary
The variable ranges over 0 and 1.
bloxopt_integer
The variable ranges over the integers.
bloxopt_real (default)
The variable ranges over the reals.
bloxopt_semicontinuous
A semi-continuous variable is a variable whose value can be either 0, or greater than or equal to some lower bound: cl ≤ x.
bloxopt_semiinteger
A semi-integer variable is an integer variable whose value can be either 0, or be greater than equal to some lower bound: il ≤ x.

To express that a variable ranges over one of the above sets, we use a MIP-LogiQL-constraint whose head is of the form p(v), where v is a LogiQL variable and p is one of the above special predicates. For example, if in the diet example nutrients existed only in discrete quantities, we would write the following LogiQL constraint:

NUTR(n), FOOD(f), amt[n, f] = v -> bloxopt_integer(v). 

The discrete knapsack problem is a more appropriate example of IP. Here is an implementation:

ITEM(i), ITEM_id(i:s) -> string(s).
weight[i] = v -> ITEM(i), float(v).
value[i] = v -> ITEM(i), float(v).
maxWeight[] = v -> float(v).
lang:solver:variable(`Pick).

// LogiQL declaration of variable predicate Pick
Pick[i]= v -> ITEM(i), float(v).

// MIP-LogiQL-constraint that restricts the
// values of Pick(i): an item may either be
// included or not in the knapsack
ITEM(i), Pick[i] = v -> bloxopt_binary(v).

objective[] = v -> float(v).
objective[] = v <- agg << v=total(z) >>
  ITEM(i), Pick[i] = 1.0f, z = value[i].
lang:solver:maximal(`objective).

totalWeight[] = v -> float(v).
totalWeight[] = v <- agg << v=total(z) >>
  ITEM(i), Pick[i] = 1.0f, z = weight[i].

totalWeight[] = v1, maxWeight[] = v2 -> v1 <= v2. 

Notice the two different LogiQL constraints that are highlighted by the comments in the code: one is a LogiQL declaration that gives the LogiQL type of the predicate Pick (notice also that elsewhere the expression Pick[i] appears two times equated with the floating point constant 1.0f instead of the integer 1). There is a separate LogiQL constraint, involving the built-in predicate bloxopt_binary, that informs BloxOptimize that the values of all the expressions Pick[x] can actually only be 0 or 1.

It might seem an overkill to not use LogiQL type declarations to deduce the set over which variables range. However, this design choice makes the language interestingly more expressive. In particular, sometimes, it is useful to have different MIP-types for the variables introduced by the same predicate.

For example, consider a situation where we track the integer quantity of a certain product in a certain store over time. Let

  • inventory[t] be the quantity of the product in the store at time t;

  • incoming[t] be the quantity of the product that is shipped into the store at time t;

  • outgoing[t] be the quantity of the product that is shipped out of the store at time t;

  • demand[t] be the demand for the product at the store at time t.

Typical constraints for such models include the what goes in equals what stays and what goes out property, i.e.,

inventory[t+1] =
    inventory[t] + incoming[t] - outgoing[t] - demand[t]. 

Other constraints include the capacity of the store, how much it costs to ship items from one store to another etc. The optimization problem could be, for example, to find the best integer values that minimize shipping costs given a certain (expected) demand.

The exact details of the problem are not interesting. What is interesting is that, while such an IP problem becomes quickly intractable, its LP version (the same problem without the restriction that the variables be integers) scales many orders of magnitude better with the number of products, stores, days, shipments etc.

To make the problem scalable, a technique that is often used is to make the variables of interest integers only for the first few days. The approximation is good enough to make today’s decisions about product shipments, while, at the same time, the problem remains computationally tractable. Assuming that the time is measured in days and it is modeled as an integer (where 0 means today), we can write constraints like:

outgoing[t] = v -> float(v), int(t).
int(t), 0 <= t < 3, outgoing[t] = v -> bloxopt_integer(v).
int(t), t >= 3, outgoing[t] = v -> bloxopt_real(v). 

Such a control over the ranges of the variables of the problem would be impossible if these were determined by the LogiQL declaration.

42.2.6. Naming Constraints

The BloxOptimize front-end language supports a naming feature. In particular, a LogiQL constraint can be given a name. This name follows the constraint to the intermediate language (presented in Intermediate Language). The name can be used to spot the intermediate-language constraint that corresponds to the LogiQL constraint and vice-versa, thus providing help in debugging.

To name a constraint, we use the keyword bloxopt_constraintName in the body of the constraint. The keyword is used like a predicate with a string literal as a parameter. A constraint may be named at most once. Here is an example of a named constraint:

bloxopt_constraintName("Basic Knapsack Constraint"),
totalWeight[] = v1, maxWeight[] = v2 ->
  v1 <= v2. 

42.3. Intermediate Language

The source file goes through the first phase of processing which produces a version of the problem in an intermediate language. This language is an extension of LogiQL with several domain-specific declarations that describe the optimization problem in a more algebraic style.

42.3.1. Directives

There are three directives that introduce intermediate language declarations: #PROBLEM, #OBJECTIVE, and #OPT. The first directive, #PROBLEM, appears at most once and describes the grouping information. #OBJECTIVE appears exactly once and gives the definition of the objective function directly in terms of the variable predicates (intermediate dynamic predicates are replaced with their definitions). Similary, #OPT expresses a LogiQL constraint only in terms of the variable predicates. These directives utilize a language that supports universal quantification (∀) and summation over domains (∑).

Besides the transformation of the LogiQL constraints and the definition of the objective function, the preprocessor leaves all other rules and constraints of the program untouched. It also introduces some new LogiQL predicate definitions, which are to be used as the domains of the universal quantifications and summations.

As a working example, we will take that of the parameterized diet in page . The preprocessor returns the following intermediate file (note: changes to the formatting were made to fit this document and for readability):

#(PROBLEM groupBy0 :: DIET(_)).
#(OBJECTIVE MIN
  sum _, v1__3, f__4 :: domain1(groupBy0,  _,  _)
  in ^v1__3*Buy[f__4,groupBy0]  ).

FOOD(f), FOOD_id(f : s) -> string(s).
NUTR(n), NUTR_id(n : s) -> string(s).
DIET(d), DIET_id(d : s) -> string(s).
amt[n,f] = v -> float(v), NUTR(n), FOOD(f).
nutrLow[n,d] = v -> float(v), NUTR(n), DIET(d).
cost[f] = v -> FOOD(f), float(v).

Buy[f,d] = v -> float(v), FOOD(f), DIET(d).
lang:solver:variable(`Buy).

objective[d] = v -> DIET(d), float(v).
objective[d] = v <- agg <<v=total(z)>>
             z =  v1 * v2,
             Buy[f,d] = v2,
             cost[f] = v1,
             FOOD(f),
             DIET(d).
lang:solver:minimal(`objective).

totalNutr[n,d] = v -> NUTR(n), float(v).
totalNutr[n,d] = v <- agg <<v=total(z)>>
             z =  v1 * v2,
             Buy[f,d] = v2,
             amt[n,f] = v1,
             DIET(d),
             NUTR(n),
             FOOD(f).

#(OPT
  forall n, _, v2 :: domain7(_,  groupBy0,  _)
  in
   sum _, _, v1__5, f__6 :: domain2(n, groupBy0, _, _)
   in ^v1__5*Buy[f__6,groupBy0]
  >= ^v2  ).

lang:solver:groupby(`DIET).

domain1(groupBy0,v1,f) <-
  cost[f] = v1,
  FOOD(f),
  DIET(groupBy0).
domain2(n,groupBy0,v1,f) <-
  amt[n,f] = v1,
  DIET(groupBy0),
  NUTR(n),
  FOOD(f).
domain7(n,groupBy0,v2) <-
  nutrLow[n,groupBy0] = v2,
  DIET(groupBy0),
  NUTR(n). 

We see that the LogiQL code has remained as is, except for the following changes:

  • #PROBLEM and #OBJECTIVE directives have been generated;

  • an #OPT directive has replaced the only LogiQL constraint;

  • several new predicates were defined: their names begin with domain.

Let us go through the new directives:

  • The #PROBLEM directive informs BloxOptimize what the grouping predicate is and what is its arity n. Furthermore, it introduces n variables with the names groupBy0, .., groupBy(n-1). These variables are meant to appear in the expressions inside the other directives and their scope is global. They act as parameters of the problem. The file directs BloxOptimize to solve all the MIP problems for all possible valuations of the group-by variables. The syntax of the directive is

    #(PROBLEM v::g(_, _, ..., _) )

    where v are the group-by variables separated by commas, g is the name of the group-by predicate, and the number of underscores in the parentheses equals n.

    In our example, we see that the group-by predicate is of arity 1 and that it is called DIET. The special variable groupby0 is to be used globally in the other directives, to stand for the value of any specific diet.

  • The #OBJECTIVE directive expresses the objective function solely in terms of the variable predicates of the problem. The syntax is

    #(OBJECTIVE m e)

    where m is the sense of the problem (either MIN or MAX) and e is an intermediate language expression (ILE). ILEs are explained below.

  • There is one #OPT directive per LogiQL constraint and it expresses the LogiQL constraint solely in terms of the variable predicates of the problem. The syntax is

    #(OPT c)

    where c is is an intermediate language constraint (ILC). ILCs are also explained below.

42.3.2. Syntax for Quantification Domains

To use ∀ and ∑ quantifiers we need a way to introduce bound variables and restrict their domain. To do that, we use a domain predicate d. The predicate d is defined in LogiQL (the definition is planted by the preprocessor). The number of bound variables is equal to the arity n of d, and the variables range over all value assignments that make d true. The syntax is:

x, y, ... :: d(_, _, ...)

where x, y, ... are the bound variables and the number of underscores in the parentheses is equal to n.

More generally, it might be that some parameters of d are fixed. The value of a fixed parameter appears in the appropriate place within the parentheses of d. The corresponding place in the bound variables is replaced by an underscore. For example, consider the following domain restriction:

x, _, _, y :: dom(_, a, 4, _)

This appears in a context where variable a is in scope. It introduces two bound variables x, y which range over all value assignments such that dom(x, a, 4, y) is true.

Let us consider the #OBJECTIVE directive of our running example:

#(OBJECTIVE MIN
  sum _, v1__3, f__4 :: domain1(groupBy0,  _,  _)
  in ^v1__3*Buy[f__4,groupBy0]  ).

The directive says that the problem is to minimize an expression. The expression is a sum over a domain. Two bound variables are introduced: v1__3 and f__4. They range over all value assignments such that domain1(groupBy0, v1__3, f__4).

Let us look at the LogiQL definition of domain1 at the bottom of the file:

domain1(groupBy0,v1,f) <-
  cost[f] = v1,
  FOOD(f),
  DIET(groupBy0). 

We see that the summation is over all v1__3, f__4 such that cost[f__4] = v1__3 and f__4 is a food item.

Let us now look at the body of the summation: ^v1__3*Buy[f__4,groupBy0]. The syntax ^x, where x is an identifier, means the value of the variable x in the intermediate language. Given our domain, we see that the body of the summation is equal to cost[f__4]*Buy[f__4,groupBy0]) for any food item f__4 and diet groupBy0. In conventional mathematical notation, this is written:

f ∈ FOODcost(f)⋅Buy(f, groupBy0)

The fact that the diet variable groupBy0 is not bound by the summation signifies that the problem is to be solved separately for all possible values of groupBy0, that is, all diets.

42.3.3. ILEs and ILCs

From the above, it should be clear that ILEs (Intermediate Language Expressions) are written in the expression sublanguage of LogiQL, with two exceptions:

Similarly, the language for writing ILCs (Intermediate Language Constraints) is the same as the formula sublanguage of LogiQL extended with universal quantification.

The whole point of introducing binding constructs is to get rid of the dynamic predicates in the definition of the objective function and in the LogiQL constraints. For example, the intermediate definition of the objective function in our running example is given in terms of the variable predicate Buy only. Auxiliary dynamic predicates such as totalNutr are not needed anymore.

As an example of an ILC, let us consider the only #OPT directive of our running example. The ILC contains a summation nested inside a universal quantification. Starting from the summation, we see that it ranges over all v1__5, f__6 such that domain2(n,groupBy0,v1__5,f__6) is true. Looking at the definition of domain2, we see these variables' ranges are such that amt[n,f__6] = v1__5 and f__6 is a food item. Looking at the body of the summation, we can translate it in mathematical notation as follows:

f ∈ FOODamt(n, f)⋅Buy(f, groupBy0)

and the whole body of the universal quantification:

f ∈ FOODamt(n, f)⋅Buy(f, groupBy0) ≥ v2

which means: the full amount of nutrient n that we buy in diet groupBy0 has to be greater than v2. We do not yet know where the variables n and v2 are going to be bound.

Let us now turn our attention to the universal quantification, which is the whole ILC. The bound variables are n and v2, and their range is such that domain7(n,groupBy0,v2) is true, or, making the substitution, such that n is a nutrient and v2=nutrLow[n,groupBy0]. The quantification can be expressed with a single bound variable n over all nutrients:

∀ n ∈ NUTR ∑f ∈ FOOD amt(n, f)⋅Buy(f, groupBy0) ≥ nutrLow(n, groupBy0)

42.3.4. Named Constraints

As we explained in Section 42.2, “Source Logic”, the source language allows us to give names to LogiQL constraints. These names appear in the corresponding #OPT directive. For example, if the LogiQL constraint in the diet example were

bloxopt_constraintName("SomeName"),
DIET(d), NUTR(n),
totalNutr[n, d] = v1, nutrLow[n, d] = v2 -> v1 >= v2. 

then the corresponding #OPT construct would read:

#(OPT SomeName as
  forall n, _, v2 :: domain7(_, groupBy0, _)
  in
   sum _, _, v1__5, f__6 :: domain2(n, groupBy0, _, _)
   in ^v1__5*Buy[f__6,groupBy0]
  >= ^v2  ). 

42.4. Use of BloxOptimize

42.4.1. Getting the preprocessor

The preprocessor is distributed with the LB patform as the executable bin/bloxopt.

42.4.2. Running the preprocessor

To produce an intermediate file IF from a source file SF, run:

bloxopt process -iSF -oIF

To produce the final LogiQL file FSF from IF:

bloxopt translate -iIF -oFSF

The file FSF can now be compiled by LogicBlox and added to a database.

If the intermediate preprocessesed program is unnecessary, the command full can be used to produce the final logic in one single step.

bloxopt full -iSF -oFSF

Modules

In the presence of module structure, we must inform the preprocessor that the produced LogiQL file d will be part of a module structure. We do so by passing it the absolute path to d through the -m command line option.

For example, suppose that we want to use the BloxOptimize preprocessor to produce a LogiQL file, model/engine.logic. Our steps are as follows:

  • We create a BloxOptimize source file. Let us call it model/engine.pre. The file should already contain the necessary module structure.

  • We create the intermediate file as follows:

    bloxopt process
      -imodel/engine.pre -omodel/engine.middle
      -mmodel:engine
  • We create the post-processed LogiQL file as follows:

    bloxopt translate
      -imodel/engine.middle -omodel/engine.post
      -mmodel:engine 

A known limitation that the user should keep in mind is that aliasing is not supported by the BloxOptimize preprocessor. Do not use any aliasing directives in files that are going to be preprocessed with bloxopt.

42.4.3. Triggering the (re-)optimization

The LogiQL sources generated by the preprocessor includes a trigger predicate that controls the invocation of the optimization in the database.

optimization_problem_trigger_0[p] = i ->
   problem_0(p), int(i). 

The type problem_0 is automatically generated by BloxOptimize. There is also a problem constructor problem_ctor[x1,...,xn] = problem_0 where x1,...,xn is a set of keys determined by lang:solver:groupby.

Putting it all together, after the data is loaded in the database, the user triggers the optimization for the first time by executing a rule:

+optimization_problem_trigger_0[p] = 1 <-
   problem_0@prev(p). 

The user triggers subsequent optimizations by changing the value of optimization_problem_trigger_0, e.g.,

^optimization_problem_trigger_0[p] = v1 <-
   problem_0@prev(p),
   optimization_problem_trigger_0@prev[p] = v,
   v1 = v + 1. 

42.4.4. Preprocessor commands

bloxopt process

Syntax
bloxopt process [-i INPUT] [-o OUTPUT]
Description

Performs the first step of translation. Produces an intermediate file which is not legal LogiQL, but can be used for visual inspection of the intermediate model for debugging purposes.

Required Arguments
-iFILE or –input=FILE

Name of the input file that contains the source.

-oFILE or –output=FILE

Name of the output file that will contain the intermediate code produced by the bloxopt preprocessor.

bloxopt translate

Syntax
bloxopt translate [-i INPUT] [-o OUTPUT] [-v]
                  [-m MODNAME] [-t PREDNAME] [-r Predname=UserPredName]
                  [–predicate_prefix=UserPrefix]
Description

Performs translation from the intermediate file to LogiQL with optimization code.

This command can also be invoked by its synonym, alt_translate.

Required Arguments
-iFILE or –input=FILE

Name of the input file that contains the intermediate code produced with bloxopt process.

-oFILE or –output=FILE

Name of the output file to be produced by the tool. This will be LogiQL code that can be integrated with the rest of the LogiQL application. If the input LogiQL code contains a module, the resulting output will be a module with the same name and export section. All code added by the preprocessor will be placed in the clauses section of the module. If the input file does not contain a module, the resulting file will not contain any module structures.

Optional Arguments
-v or –verbose

Print verbose progress and diagnostic messages while performing translation.

-mMODNAME or –module_name=MODNAME

The fully-qualified name of the module that the resulting LogiQL file is supposed to have. Failure to provide a module name for an input problem that has a module structure will result in LogiQL code that cannot be compiled.

-tPREDNAME or –trigger=PREDNAME

Override the automatically generated name for the optimization trigger predicate.

-rPredname=UserPredName

Override BloxOptimize-generated predicate name Predname with UserPredName.

–predicate_prefix=UserPrefix

All predicate names generated by BloxOptimize will be prefixed with UserPrefix.

–-skip_presolveUserPrefix

By default,bloxopt generates code which performs some rudimentary presolving steps before sending a problem instance to the solver. For example, it promotes constraints of the form 2*x >= 4 to bounds on the variable like x >= 2. The --skip_presolve option will generate code that does not perform such steps. This is sometimes useful for debugging.

42.4.5. Choosing the Back-end Optimizer

The user of BloxOptimize may select which back-end optimizer to run, by setting environment variable BLOXOPT_OPTIMIZER before running LB services.

Each supported optimizer comes with its own handler. An optimizer handler is a program whose purpose is to mediate the communication of the database with the optimizer. All optimizer handlers are under libexec/logicblox in the distribution. The name of each handler starts with bloxoptimize_handler_ followed by the name of the solver. The program bloxoptimize_handler is the default handler that corresponds to Gurobi.

To choose one of the supported handlers, the user sets BLOXOPT_OPTIMIZER to the name of the optimizer as it appears in the name of its handler. For example, the CBC optimizer is supported by handler bloxoptimize_handler_cbc. If the optimizer is properly installed in the user’s machine, the user must set BLOXOPT_OPTIMIZER to cbc. Leaving the variable unset defaults to Gurobi.

Currently, CBC 2.9.4 and Gurobi 5.6.3 are supported in the default branch.

To retrigger the optimization, either change the value of optimization_problem_trigger_0[p] or retract it and add it again:

-optimization_problem_trigger_0[p] = v <-
   problem_0@prev(p),
   optimization_problem_trigger_0@prev[p] = v.
+optimization_problem_trigger_0[p] = 1 <-
   problem_0@prev(p). 

42.4.6. Optimizer Options

The source language supports setting configuration options for the back-end optimizer. The configuration parameters depend on the specific optimizer. The support in the source language uses pragma lang:solver:parameters. The pragma specifies a configuration predicate that takes a single key argument of type string. The configuration predicate is populated with strings that set the configuration parameters. For example, the following source code can be used to set the timeout of the Gurobi solver to 10 seconds:

lang:solver:parameters(`gurobi_parameters).
gurobi_parameters(str) -> string(str).
gurobi_parameters("TimeLimit 10"). 

42.4.7. Advanced Options

In addition to a solution to a linear or mixed-integer programming problem, the code produced by bloxopt includes three predicates that store information about each problem instance for which a solution has been obtained.

The three predicates store information about the model as a whole (model_attributes), about each column (i.e., variable: column_attributes) and about each constraint (i.e., row: row_attributes).

These predicates are intended for debugging only, and their contents can vary considerably depending on the type of problem (linear, integer, mixed-integer), and the type of solver used (e.g,. Gurobi).

      model_attributes_0[p,name] =  v -> string(v), problem_0(p), string(name).
      column_attributes_0[p,name,col] =  v ->int(col), string(v),problem_0(p), string(name).
      row_attributes_0[p,name,col] =  v -> int(col), string(v), problem_0(p), string(name).

For example, for an integer programming problem, solved with Gurobi 6.5, the predicate contents might look something like this:

      [10000000006] "BoundVio"       "0"
      [10000000006] "BoundVioIndex"  "-1"
      [10000000006] "BoundVioSum"    "0"
      [10000000006] "ConstrVio"      "0"
      [10000000006] "ConstrVioIndex" "-1"
      [10000000006] "ConstrVioSum"   "0"
      [10000000006] "IntVio"         "0"
      [10000000006] "IntVioSum"      "0"
      [10000000006] "IsMIP"          "1"
      [10000000006] "IsQCP"          "0"
      [10000000006] "IsQP"           "0"
      [10000000006] "IterCount"      "54"
      [10000000006] "MIPGap"         "0"
      [10000000006] "ModelName"      "instance__2016_10_28_12_57_21_895022+"
      [10000000006] "ModelSense"     "1"
      [10000000006] "NodeCount"      "37"
      [10000000006] "NumBinVars"     "0"
      [10000000006] "NumConstrs"     "7"
      [10000000006] "NumIntVars"     "9"
      [10000000006] "NumNZs"         "58"
      [10000000006] "NumQCNZs"       "0"
      [10000000006] "NumQConstrs"    "0"
      [10000000006] "NumQNZs"        "0"
      [10000000006] "NumSOS"         "0"
      [10000000006] "NumVars"        "9"
      [10000000006] "ObjBound"       "15.05"
      [10000000006] "ObjBoundC"      "15.05"
      [10000000006] "ObjVal"         "15.05"
      [10000000006] "Runtime"        "0.0114942"
      [10000000006] "SolCount"       "4"
      [10000000006] "Status"         "2"

Furthermore, for each variable predicate, a number of properties predicates is automatically generated, controlled by the bloxopt option --variable_attributes=comma_separated_list.

For example, consider the following command::

    bloxopt translate  -v --debug_verbose  --variable_attributes=RC,VBasis -isuite.middle -osuite.logic

For each predicate X where X is declared as lang:solver:variable, two additional predicates called X_RC, and X_VBasis will also be generated and will store the reduced costs for each variable, as well as whether the variable is in the basis.

Note that the availablity of information in these predicates is highly problem and solver dependent (consult the Gurobi manual, and its column properties for a list).

The default behavior is to create predicates for all variable attributes currently supported by the solver. The command line option --no_variable_attributes turns off generation of all variable attribute prdicates. The --variable_attributes=list option restricts the generation of variable attribute predicate to the list specified.