Chapter 19. Transaction Logic

A LogicBlox database contains data (the contents of predicates, plus some internal information) and "logic" (declarations, rules and constraints, in compiled form).

As is usual in other database systems, changes to the database are performed in units called transactions. A transaction is a series of actions such as deletion of old data, addition of new data and logic, evaluation of rules that update the data, and verification that the data violates no constraints. (The last two kinds of actions are automatically initiated and performed by the system.)

If any of these actions result in an error (e.g., detection of an inconsistency, or an error detected during compilation of new logic), then the transaction is aborted, and the state of the database is not changed: it is as if the transaction never occurred. If there is no error, the state of the database produced by the transaction is known to be consistent, and the transaction may be committed (i.e., the new state of the database may be made to become the permanent one).

Transactions in LogicBlox are somewhat complicated: in particular, each transaction consists of several separate stages. To effectively use the LogicBlox system one must have at least a basic understanding of these complications. This chapter is an introduction to the topic.

In order to make the chapter more self-contained, we begin with a preliminary introduction to the various concepts necessary for appreciating the workings of a transaction. The latter are described in Section 19.4, “Stages”.

Some of the preliminary information is intended to help you experiment with the system. We recommend that you try to construct and execute some simple examples, perhaps basing them loosely on the examples in this manual. Even a little hands-on experimentation can go a long way towards gaining more confidence with the system (as well as towards uncovering gaps and ambiguities in the knowledge that you gained from just reading the manual).

19.1. Preliminaries

19.1.1. The lb tool

There are several direct and indirect ways to load and execute (evaluate) "logic", i.e., a "program" written in LogiQL. For the purposes of this chapter we will assume that the user accesses the LogicBlox server via the interactive mode of a command-line tool called lb (see Chapter 33, LogicBlox Command Reference). The tool is invoked by typing lb in your terminal. It is often most convenient to put lb commands in a file, for example file.lb (the suffix .lb is mandatory). The commands in the file can then be executed by writing the following on the command line:
lb file.lb

The list of all commands can be obtained by invoking lb -h, but since it is long and somewhat confusing, we will briefly describe the most essential ones below.

19.1.2. Workspaces

A workspace is essentially an instance of the LogicBlox database. The user can fill it with data and rules, modify its contents, evaluate commands within the context so created, etc. All these things cannot be done outside a particular workspace, so you will have to create one in order to experiment with LogiQL.

The lb tool provides the following basic commands for manipulating workspaces:

create name

Creates and opens a workspace with the given name. If the workspace is to be treated as just a temporary scratchpad, you can replace name with --unique. (Please note that even then the workspace will persist if it is not explicitly destroyed.)

close

Close the current workspace. Use close --destroy to also delete it.

close --destroy is particularly useful if the current workspace is a temporary scratchpad one, as it will be assigned a long name. (However, be aware that such a workspace will persist if the close command has not been reached: this may happen, for instance, when the execution is aborted because of an error.)

open name

Open an existing workspace with this name.

delete name

Delete the workspace with this name.

The easiest way to learn more about how to use each of these commands is to request help information directly from the tool. For example, to find out about the options for delete we could write

lb delete -h

These are not all the commands that manipulate workspaces. To see a complete list of commands invoke lb -h from the command line.

19.1.3. Blocks

A declaration, rule, fact, etc., is not processed or compiled by itself, but as a part of a larger unit called a block. The division of logic into blocks is carried out by the user, and can be pretty much arbitrary. There is, however, a facility for declaring a predicate as local to the block, i.e., invisible in other blocks (see Section 8.10, “Local predicates”).

Blocks can contain only logic, not lb commands such as close or print.

A little more information about blocks can be found in Section 19.1.6, “Inactive Blocks”.

A block can be thought of as a unit of compilation, but is not, in general, a unit of execution (see Section 19.4.3, “Order of execution”).

19.1.4. Loading and executing logic

Most of the commands listed below take a block as an argument (see Section 19.1.3, “Blocks”). The text of the block (which often consists of a number of lines) can be enclosed in single quotes (i.e., apostrophes: '), or -- equivalently -- put between <doc> and </doc>. We will follow the latter convention in our examples.

The following commands should suffice for running the simplest examples. (As always, use lb command -h for details, and lb -h for the complete list of commands.)

print name

Print the contents of the named predicate.

echo text

Print the rest of the line. Useful, e.g., for making the output of print more self-explanatory.

addblock block

Add the block to the current workspace.

The logic in the block usually has database lifetime (see Section 19.1.5, “The notion of "lifetime"”). See Section 19.1.6, “Inactive Blocks” for the exception.

exec block

Execute the block within the contents of the current workspace near the beginning of the transaction (see Section 19.4, “Stages”).

If the block declares any predicates, they must be local, i.e., their names must begin with underscores. They will not be accessible from outside the block, and will be discarded at the end of the transaction. (See Section 8.10, “Local predicates”.)

All the facts, constraints and rules in the block (if any) must refer to local predicates. Apart from that, the block may contain only delta logic (Section 19.2, “Delta logic”).

The logic in the block has transaction lifetime (see Section 19.1.5, “The notion of "lifetime"”).

query block

A variant of exec that is executed at the very end of a transaction (see Section 19.4, “Stages”). Execution of the block may modify the database, but these modifications are strictly temporary, and disappear after execution is terminated (even if it is terminated by abortion).

The limitations noted for exec apply.

It is often convenient to precede or follow the block argument with one or more arguments of the form --print name to print the contents of the named predicate(s).

The logic in the block has query lifetime (see Section 19.1.5, “The notion of "lifetime"”).

By default, each of these lb commands is treated as a separate transaction. It is, however, easy to enclose several of them in the same transaction: see Example 19.11, “Execution of multiple blocks”.

Example 19.1. A command or query cannot declare a non-local predicate

Suppose the file non-local-in-exec.lb contains the following text (see Section 19.2, “Delta logic” for the meaning of + before a predicate name):

create --unique

addblock <doc>
  p(x) -> int(x).
</doc>

exec <doc>
  q(x) -> int(x).

  +q(x) <- int:range(0, 3, 1, x).

  +p(x) <- q(x).
</doc>

print p

close --destroy

An attempt to execute the file will result in the following:

> lb non-local-in-exec.lb
created workspace 'unique_workspace_2016-02-08-21-50-37'
added block 'block_1Z38MU1A'
block block_1Z3DEKTT: line 1: error: every predicate declared in a command or query must be local: 'q' (code: NON_LOCAL_PREDICATE_DECLARATION)
q(x) -> int(x).
^^^^

1 ERROR 

The error message tells us to rename q to _q. The example will then run as expected:

> lb non-local-in-exec.lb
created workspace 'unique_workspace_2016-02-08-21-50-56'
added block 'block_1Z38MU1A'
0
1
2
3
deleted workspace 'unique_workspace_2016-02-08-21-50-56' 

Please note that _q is local to the block passed to exec, so it cannot be mentioned directly as an argument to the print command. This would result in the following message from the compiler:

error: Could not find predicate _q 

As noted above (in Section 19.1.3, “Blocks”), print _q cannot appear directly in the block passed to exec.

19.1.5. The notion of "lifetime"

Depending on how a block is brought into the system, it (or the logic contained in it) is said to have one of three possible lifetimes:

database lifetime

This term applies to logic that is permanently installed in the workspace and survives the transaction (unless the transaction is aborted).

For example, the lb command addblock installs database-lifetime logic (except when the command is given the additional argument --inactive: see Section 19.1.6, “Inactive Blocks”).

Database-lifetime logic is often referred to by the shorter term installed logic.

Note

"Permanently" installed logic can be removed by an explicit command: for example, the removeblock command in lb.

transaction lifetime

This term applies to logic that is available throughout the transaction, but is not permanently installed in the workspace and does not survive the transaction. Its execution may, however, have lasting effects on database-lifetime predicates.

For example, the lb command exec temporarily installs transaction-lifetime logic.

query lifetime

This term applies to logic that is available only during the execution of queries, is not permanently installed in the workspace and does not survive the transaction.

Unlike transaction-lifetime logic, query-lifetime logic is executed at the very end of a transaction and has access to all its effects; moreover, any modifications to the database performed during the execution of query-lifetime logic are strictly temporary and do not survive the transaction.

For example, the lb command query temporarily installs query-lifetime logic.

See Section 19.4, “Stages” for more details about how and when a transaction handles logic of different lifetimes.

19.1.6. Inactive Blocks

Apart from the "normal" blocks described above, the LogicBlox system supports inactive blocks. These are somewhat similar to "precompiled queries" of SQL.

An inactive block is a block that is processed ahead of time, and stored in compiled form as a persistent part of the database. It can be activated on demand, i.e., executed as transaction-lifetime or query-lifetime logic. Activating the block installs and executes it at the requested lifetime, but it is then discarded at the end of the transaction. The block persists in inactive form as a part of the database, ready to be activated again and again.

To install an inactive block through the lb tool, one can exeute the command lb addblock --name block_name --inactive. Although this is the addblock command, the logic will not have database lifetime, so -- just like in the case of exec and query -- the block can declare only local predicates.

In order to execute an inactive block one can use the command execblock block_name.

We defer an example to the end of the section that introduces delta logic (which must be used in the example). See Example 19.8, “A simple inactive block”.

19.2. Delta logic

Facts (Section 10.1.1, “Atoms as facts”) and IDB rules (Chapter 11, Rules) can be used to populate only intensional (IDB) predicates. Extensional (EDB) predicates (see Section 8.8, “Derivation Types”) are populated by "EDB logic" which is often referred to as deltas, or delta logic. This section is an introduction to the topic of deltas.

Note

It is important to remember that EDB predicates must be manipulated by EDB logic, and IDB predicates cannot be manipulated by EDB logic. The system will raise an error if this rule is violated. The diagnostic message may sometimes be a little confusing: if you write a single IDB rule for an EDB predicate with many delta rules, the IDB rule may take precedence, and the message will begin with something like

error: predicate 'union' is a derived predicate (intensional, IDB) and should not be used in the head of a delta rule (extensional, EDB).

19.2.1. Direct manipulation of EDB predicates

Note

The explicit modification operations presented below are suitable only for relatively minor changes to the database. The reason is that each such operation is internally translated into a so-called frame rule (see Section 19.4.6, “Frame rules”). The frame rule will then be used by the general mechanism for updating predicates during maintenance (see Section 19.4.1, “Maintenance”).

If the number of frame rules becomes too large, efficiency suffers. So if you want to insert more than several hundred tuples, you should use CVS/TDX import instead (see Chapter 27, Data Exchange Services).

A fact such as age("Mary", 7). is a static declaration that predicate age will always contain the tuple ("Mary", 7).

EDB predicates must support deletion of data, so if age is an EDB predicate, we must be able to insert or delete a particular tuple. The notation for this is called delta atoms, and it is quite intuitive:

+age("Mary", 7).

Insert the tuple ("Mary", 7) into predicate age. Do nothing if the tuple is already there.

-age("Mary", 7).

Delete the tuple ("Mary", 7) from predicate age. Do nothing if the tuple is not there.

Please note that the arguments must be fully instantiated, i.e., none of them may contain an unbound variable. (See the section called “Bound variables and their instantiations”.)

If age is a functional predicate, we can also use the forms +age["Mary"]=7. and -age["Mary"]=_.

For a functional predicate it is often convenient to update information associated with a given key (e.g., on Mary's birthday), so we have a third possibility:

^age["Mary"] = 8.

If predicate age contains a tuple whose key is "Mary", delete that tuple. Then insert the tuple ("Mary", 8).

This operation is often called an upsert ("update or insert"). It will amount to a simple insertion if there was no tuple to be deleted.

It is sometimes convenient (e.g., in the compiler's error messages) to use the term "delta" to denote one of the three prefixes introduced above.

Deletion of tuples from a functional predicate

To delete a tuple from a functional predicate you must provide only the keys. For example, to delete information about Mary's age in our running example, we write just

-age["Mary"] = _ . 

An attempt to provide the value would be treated as an error, regardless of whether the value is correct or not. This may be a little surprising at first, but is in fact both logical and convenient.

Deletion of tuples from an entity predicate

Just like insertion, deletion of tuples from an entity predicate must be performed via its associated constructor predicate or refmode predicate.

Example 19.2. Deletion from an entity predicate with a constructor

create --unique

addblock <doc>
  person(p) -> .
  person_by_name[name] = p -> string(name), person(p).
  lang:constructor(`person_by_name).

  person(p), person_by_name[nm] = p <- name(nm).

  name(nm) -> string(nm).
</doc>

exec <doc>
  +name("Jay").
  +name("Jane").
</doc>

echo person:
print person
echo person_by_name:
print person_by_name

exec <doc>
  -name("Jay").
</doc>

echo person:
print person
echo person_by_name:
print person_by_name

close --destroy

The result is:

created workspace 'unique_workspace_2016-06-29-21-40-59'
added block 'block_1Z1C3B7J'
person:
[10000000004]
[10000000005]
person_by_name:
"Jane" [10000000005]
"Jay"  [10000000004]
person:
[10000000005]
person_by_name:
"Jane" [10000000005]
deleted workspace 'unique_workspace_2016-06-29-21-40-59' 

Example 19.3. Deletion from an entity predicate with a refmode

In the example below, -person_has_name(_ : "Jay"). can also be written in the form -person_has_name[_] = "Jay".

create --unique

addblock <doc>
  person(p), person_has_name(p : nm) -> string(nm).
</doc>

exec <doc>
  +person(p), +person_has_name(p : "Jay").
  +person(p), +person_has_name(p : "Jane").
</doc>

echo person:
print person
echo person_has_name:
print person_has_name

exec <doc>
  -person_has_name(_ : "Jay").
</doc>

echo person:
print person
echo person_has_name:
print person_has_name

close --destroy

The result is:

created workspace 'unique_workspace_2016-06-29-21-55-07'
added block 'block_1Z1C3AAW'
person:
[10000000004] "Jay"
[10000000005] "Jane"
person_has_name:
[10000000004] "Jay"  "Jay"
[10000000005] "Jane" "Jane"
person:
[10000000005] "Jane"
person_has_name:
[10000000005] "Jane" "Jane"
deleted workspace 'unique_workspace_2016-06-29-21-55-07'

19.2.2. Delta rules

EDB predicates can also be manipulated by delta rules. A delta rule is similar to an IDB rule, but each of the head atoms is prefixed with a delta, i.e., +, - or ^ (the latter only in the case of functional predicates).

The meaning of these prefixes is as described in Section 19.2.1, “Direct manipulation of EDB predicates”: a head atom in a delta rule generates only insertions, only deletions, or upserts.

The prefixes can also be used for some or all of the atoms in the body. It is convenient to think of +p as a predicate that contains all the tuples that were requested to be inserted into predicate p by the current transaction; -p would contain all the tuples that were requested to be deleted from p by the transaction. A body atom such as ^f[x]=y is treated as equivalent to +f[x]=y. See Section 19.4, “Stages” for more details. (Please note the difference between a request for an insertion or deletion and an actual insertion or deletion. A request for an insertion will not result in an insertion if the tuple is already present in the database; a request for deletion will not result in a deletion if the tuple is not present in the database.)

Note

  • The above is just an approximation of the truth. If the name of a delta atom in a body refers to an entity predicate, a refmode predicate or a constructor predicate, then not every request for insertion will be accessible through that atom: you will not see requests that would result in the creation/addition of an entity that already exists.

  • The runtime system does actually construct so-called delta predicates to keep information about the requests for insertions and deletions. The internal names of these predicates are somewhat different from +p or -p. These delta predicates are pulse predicates. See Section 19.3.1, “Pulse predicates” and the section called “Stage suffixes in auxiliary internal predicates”.

  • If you are thinking about using delta atoms in the body of a rule, you might want to consider whether external diff predicates would not be more appropriate for your application. (See Section 8.11, “External Diff Predicates”.)

Such delta atoms are also allowed in the body of an IDB rule, but only within transaction-lifetime or query-lifetime logic (e.g., within a block that is executed by exec or query). The IDB rule must derive into a local predicate. (See Section 19.1.5, “The notion of "lifetime"” and Section 8.10, “Local predicates”.)

Example 19.4. Simple delta rules

p(x) -> int(x).
q(x) -> int(x).
r(x) -> int(x).

+q(x) <- +p(x).
+r(x) <- -p(x). 

If the rules above are the only rules for q and r, then q will contain the set of all the integers that have ever been inserted into p, while r will contain the set of all the integers that have ever been requested to be deleted from p. (The word ever should be interpreted as since q and r have been declared.)

(If p is an IDB predicate, then q will include all that has been added to p after q has been declared, including explicitly declared facts of p. r will be empty, of course.)

Installed delta rules

A delta rule can appear in a transaction-lifetime or query-lifetime block (e.g., if the block is an argument to exec or query). A delta rule can also be installed as database-lifetime logic (e.g., when it appears in a block that is an argument to addblock).

There is an important requirement that must be satisfied by every database-lifetime delta rule: its body must contain at least one delta atom (we say that the rule is guarded by that delta atom).

The rationale for this is as follows: a rule such as

+p(x) <- q(x). 

actually derives into the delta predicate +p, which is a pulse predicate (see Section 19.3.1, “Pulse predicates”). At the end of a transaction all pulse predicates must be cleared of all contents, which would be inconsistent with the rule if q were not empty. By introducing a "guard" (such as +r(x) in the rule below) we ensure that, as pulse predicates are made empty, the guard will not hold, so the rule will derive nothing and consistency will be preserved.

+p(x) <- q(x), +r(x). 

Example 19.5. Installed delta rules must be guarded

addblock <doc>

p(x) -> int(x).
q(x) -> int(x).
r(x) -> int(x).
s(x) -> int(x).

+p(x) <- q(x), (r(x) ; +s(x)).

</doc> 

The delta rule triggers the following error message:

error: Installed delta rules must be guarded by a delta or pulse predicate in the body of the rule. (code: DELTA_UNGUARDED)
    +p(x) <- q(x), (r(x) ; +s(x)).
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

This is, of course, because a rule with a disjunction is equivalent to several rules (see Example 10.18, “Combining conjunction with disjunction”).

To satisfy the requirement, we must rewrite the rule into one of the following forms:

+p(x) <- q(x), (+r(x) ; +s(x)).

+p(x) <- +q(x), (r(x) ; s(x)).

+p(x) <- +q(x), (+r(x) ; +s(x)).

+p(x) <- +q(x), (r(x) ; +s(x)).

+p(x) <- +q(x), (+r(x) ; s(x)). 

19.2.3. Insertions vs. deletions

If insertions and deletions to the same predicate are carried out at the same stage of a transaction (Section 19.4, “Stages”), then the deletions are always done first.

More precisely: information about insertions and deletions is gathered and compared in advance, and deletions that would be "undone" by insertions are not carried out (without even paying the cost of accessing the contents of the predicate to see whether the relevant tuples are there).

So for example, as far as the effect on predicate q is concerned, the pair of rules

+q(x)  <- +p(x).
-q(x)  <- +p(x). 

has exactly the same effect as any one of the following rules:

+q(x) <- +p(x).
+q(x), -q(x) <- +p(x).
-q(x), +q(x) <- +p(x). 

The effect would not be exactly the same, as the requests for deletion are noted, and may affect the effects of any rule that has -q(x) in its body.

Example 19.6. Insertions "take precedence"

What will be the printout produced by the following?

create --unique

//------------------
addblock <doc>

p(x) -> int(x).

</doc>

//------------------
exec <doc>

+p(x) <- int:range(0, 5, 1, x).
-p(x) <- int:range(0, 5, 2, x).

</doc>

print p

close --destroy 

One might expect p to contain only the odd integers between 0 and 5, but it will in fact contain all the integers between 0 and 5.

19.2.4. Delta logic is more imperative than logical

It must be noted that "delta logic" is a bit of a misnomer. Unlike IDB rules, delta rules have no obvious logical interpretation, at least not a static one, as their effect may very much depend on the particulars of the various changes applied to the contents of the database, rather than on their overall effect.

For example, the presence of an (active) IDB rule such as

p(x) <- q(x).

ensures that the contents of predicate p will always include the contents of predicate q. However, no such conclusion can be drawn from the delta rule

+p(x) <- +q(x).

After all, the user is free to delete some elements from p without removing them from q.

Example 19.7. The difference between IDB rules and EDB (delta) rules

The following file has a mixture of IDB predicates and EDB predicates. In this example, once we declare set1 and set2, the IDB rules provide enough information about the types of the IDB predicates, so we do not have to declare them explicitly. For EDB predicates and delta rules such type inference is not currently supported, so union must be declared.

The EDB predicates set1 and set2 are populated by the first exec command. Recall that these delta rules could not have been installed by the first block, because they are not guarded by delta atoms in the body (see the section called “Installed delta rules”).

The second exec command modifies set1 and set2 via explicit deletions and insertions.

create --unique

//------------------
addblock <doc>

set1(x)  -> int(x).  // intended to be EDB
set2(x)  -> int(x).  // intended to be EDB
union(x) -> int(x).  // intended to be EDB

+union(x) <- +set1(x) ; +set2(x).       // EDB

intersection(x) <- set1(x), set2(x).    // IDB

symdiff(x) <- set1(x), ! set2(x) ; set2(x), ! set1(x).  // IDB

</doc>

//------------------
exec <doc>  // populate set1 and set2

+set1(x) <- int:range(0, 5, 1, x).      // yes, EDB
+set2(x) <- int:range(3, 7, 1, x).      // yes, EDB

</doc>

//------------------
exec <doc>  // modify set1 and set2

-set1(0).
-set1(5).
-set2(5).
+set2(10).

</doc>

//------------------
echo SET1:
print set1
echo SET2:
print set2
echo UNION:
print union
echo INTERSECTION:
print intersection
echo SYMDIFF:
print symdiff

close --destroy 

Execution results in the following printout:

created workspace 'unique_workspace_2016-02-11-20-20-25'
added block 'block_1Z38MVCB'
SET1:
1
2
3
4
SET2:
3
4
6
7
10
UNION:
0
1
2
3
4
5
6
7
10
INTERSECTION:
3
4
SYMDIFF:
1
2
6
7
10
deleted workspace 'unique_workspace_2016-02-11-20-20-25' 

Notice that the rules for intersection and symdiff cause all the changes in set1 and set2 to be correctly tracked. However, the delta rule for union is not fired by deletions, so the predicate contains what we would consider obsolete values. If we wanted union to be an EDB predicate, and yet to always contain the union of the two sets, we would have to add the following two delta rules:

-union(x) <- -set1(x), ! set2(x).    // track deletions
-union(x) <- -set2(x), ! set1(x).    // track deletions

This illustrates quite clearly the advantages of declarative IDB logic. However, there are situations where EDB logic cannot be dispensed with, as already shown in Example 19.4, “Simple delta rules”, where we introduced predicates that log all the changes made to another predicate.

Example 19.8. A simple inactive block

The following example illustrates the use of inactive blocks (see Section 19.1.6, “Inactive Blocks”). It is a simple variant of Example 19.7, “The difference between IDB rules and EDB (delta) rules”.

Suppose that we have two predicates, set1 and set2, that will be modified heavily during a long computation. After that these predicates will remain unchanged, and the rest of the computation will access a predicate that contains their symmetric difference.

In order to avoid unnecessary recomputation of the symmetric difference whenever the sets are modified, we might choose to make symdiff an EDB predicate, and populate it only when it is finally needed. This could be accomplished by declaring the logic that computes symdiff as an inactive block, and activating the block at the right moment. The listing below shows the pattern.

create --unique

//---------------
addblock <doc>

set1(x)    -> int(x).
set2(x)    -> int(x).
symdiff(x) -> int(x).  // symmetric difference

</doc>

//---------------
addblock --name SymDiff --inactive <doc>

+symdiff(x) <- set1(x), ! set2(x) ; set2(x), ! set1(x).

</doc>

//---------------
exec <doc>
+set1(x) <- int:range(0, 5, 1, x).
+set2(x) <- int:range(3, 7, 1, x).
</doc>

echo SYMDIFF 1:
print symdiff

execblock SymDiff    // <----------

echo SYMDIFF 2:
print symdiff

close --destroy 

Execution will yield the result shown below. We see that symdiff will become populated only after the inactive block is activated (by execblock).

created workspace 'unique_workspace_2016-02-15-04-22-51'
added block 'block_1Z38MUEE'
added block 'SymDiff'
SYMDIFF 1:
SYMDIFF 2:
0
1
2
6
7
deleted workspace 'unique_workspace_2016-02-15-04-22-51' 

19.3. Events

19.3.1. Pulse predicates

A pulse predicate is a special kind of EDB predicate whose contents always have transaction lifetime, even though the predicate itself may have database lifetime. In other words, the predicate will always be made empty at the end of a transaction.

Pulse predicates are often used to propagate information about events, and are sometimes referred to as event logic.

For example, a GUI may interact with the LogicBlox database by inserting a tuple into a pulse predicate whenever the user clicks on a button. This may trigger evaluation of some rules, which in turn may use pulse predicates to trigger other computations.

To create a pulse predicate pp one must declare the type of pp in the usual fashion, and then add

lang:pulse(`pp). 

or

lang:isPulse[`pp] = true. 

A pulse predicate pp can usually be accessed only via delta atoms of the form +pp(...) (where the three dots represent arguments, if any). An exception is made for event rules (see Section 19.3.2, “Event Rules”).

As an EDB predicate, a pulse predicate can have its values changed only through delta operations. Moreover, since it is empty prior to the current transaction, there is seldom a need to delete tuples. If pp is a pulse predicate, then the presence of a body atom of the form -pp(...) is probably a programming error.

Internal use of pulse predicates

In Section 19.2.2, “Delta rules” we mentioned "delta predicates" that are created and maintained by the system to keep information about requests for insertions and deletions. The delta predicates are pulse predicates. (See also the section called “Stage suffixes in auxiliary internal predicates”.)

The runtime system also uses a pulse predicate to support checking of constraints. A constraint such as

A(x) -> x > 2. 

is translated to a rule that looks, roughly, like this:

system:constraint_fail(...) <- A(x), !(x > 2), ... . 

A constraint failure can now be detected during the general process of evaluating rules. (See Section 19.4.1, “Maintenance”.)

Example 19.9. Diagnostics for a constraint failure

Consider the following example:

create --unique

addblock <doc>

  age[name] = years -> string(name), int(years).
  age[_] = years -> 0 <= years < 150.

</doc>

exec <doc>

  +age["John"] = 151.

</doc>

close --destroy 

The result of running this is an error with a diagnostic message that begins like this:

Error: Constraint failure(s):
block_1Z38I0AT:2(1)--2(34):
    false <-
      Exists __a::string,__b::int,__c::int,years::int .
         age[__a]=years,
         !(
            Exists ___b3::int .
               int:le_2(__b,___b3),
               int:eq_2(years,___b3),
               int:lt_2(___b3,__c)
         ),
         int:eq_2(__b,#0#),
         int:eq_2(__c,#150#).
(1) __a="John",__b=0,__c=150,years=151 

This is a somewhat user-friendly presentation of the kind of rule that was described above. Since the user is not interested in the pulse predicate system:constraint_fail, it is presented as false.

The variables whose names begin with double underscores are generated internally by the system. Information about the types of variables is provided after the double colons, and the implicit quantifiers are shown.

int:le_2 etc. are standard LogiQL comparison operations which are normally written as the infix operators <= etc. (see Section 10.2, “Comparisons”).

Literal values appear between pound marks (#).

The entire formula can thus be read as

false <-
  Exists a, b, c, years such that
     age[a] = years,
     !(
        Exists b3 such that
           b <= b3,
           years = b3,
           b3 < c
      ),
     b = 0,
     c = 150. 

After additional obvious simplification we get

false <-
     age[a] = years,
     ! (0 <= years, years < 150). 

which is quite close to the appearance of the original constraint.

19.3.2. Event Rules

Event rules provide the programmer with a means of defining delta rules without having to use explicit delta operators. Event rules are closely tied to pulse predicates, as every event rule contains some reference to pulse predicates.

A rule is considered to be an event rule if all of the following conditions hold:

  • the enclosing block has transaction lifetime;
  • the rule contains no delta atoms;
  • all atoms in the head of the rule refer to pulse predicates;
  • at least one atom in the body of the rule refers to a pulse predicate.

Example 19.10. An event rule

If we execute the following, the output will be 5.

create --unique

//--------------
addblock <doc>

p(x) -> int(x).
q(x) -> int(x).

pp(x) -> int(x).
lang:pulse(`pp).

pq(x) -> int(x).
lang:pulse(`pq).

p(x) <- int:range(0, 5, 1, x).

+q(x) <- +pp(x).

</doc>

//--------------
exec <doc>

pp(x) <-  p(x), pq(x).   // <----- an event rule

+pq(5). +pq(6).

</doc>

//--------------
print q

close --destroy 

19.4. Stages

We are now ready to take a closer look at the anatomy of a transaction and how it affects the way we write LogiQL code.

After introducing the important notion of "maintenance" we look at the stages of a transaction, introduce "stage suffixes", and briefly mention a few topics that are more or less closely tied to stages, stage suffixes and delta logic.

19.4.1. Maintenance

Predicates may depend on each other. For example, if a rule refers to p in the head, and to q and r in the body, then p depends on q and r: when the contents of one (or both) of the latter predicates are changed, the contents of p may also have to be changed. One can also consider this rule as dependent on any rule that derives into q and r.

Information about such dependencies (plus some auxiliary information) is expressed in internal data structures maintained by the system. The structures are known as execution graphs, because they allow the runtime system to execute (evaluate) rules more efficiently and correctly:

  • there is no need to evaluate rules whose bodies do not refer to predicates that have been recently updated;
  • it is better to evaluate rules that are depended on (rules for q and r in our example) before evaluating a rule that depends on it (our rule for p).

The dependency graph is not, in general, acyclic, so updates may have to be performed repeatedly until there are no more changes to be made. The resulting stable state of the database is colloquially referred to as the fixpoint.

The term maintenance refers to the process of:

  1. updating the execution graph to reflect additions and removals of logic;
  2. using the updated graph as a guide in the evaluation of rules, until a fixpoint is reached.

In other words, maintenance ensures that the contents of all predicates are consistent with the contents of the other predicates and with all the rules that are currently active.

19.4.2. The Six Stages

The execution of a transaction consists of several successive stages. The figure and table below provide a quick overview. We then give some additional information in the text below.

It should be noted that:

  • each stage (except for START and transaction setup) involves a round of maintenance (and has its own execution graph);
  • only database-lifetime predicates (and the execution graph of stage FINAL) persist across transactions.

Stage Description
Transaction setup

Initialization of datetime:now[] and transaction:id[]. Their values will not change throughout the transaction. This means that all uses of datetime:now[] will result in the same value, even though the system time will progress during the transaction. (See the section called “datetime:now” and the section called “transaction:id[]”.)

START: the preamble

Addition/activation and removal of logic from the workspace, according to the requests (e.g., lb commands) made for this transaction:

  • Addition of:
    • transaction-lifetime, database-lifetime and query-lifetime blocks;
    • inactive blocks.
  • Activation of selected inactive blocks.
  • Removal of database-lifetime blocks.
INITIAL: evaluation of initial logic

Evaluation of transaction-lifetime logic; the transaction makes changes to EDB predicates, and to transaction-lifetime local predicates. (See Section 19.1.5, “The notion of "lifetime"”.)

FINAL: maintenance of installed logic

Maintenance of database-lifetime logic, which involves evaluation of installed rules. This ensures that:

  • IDB predicates are kept up-to-date with the changes made by the transaction;
  • database-lifetime delta rules are applied to the affected EDB predicates.
QUERY: evaluation of query logic

Evaluation of query-lifetime logic. This is quite similar to stage INITIAL, except that

  • logic at stage QUERY has access to the effects of stage FINAL;
  • all the effects of stage QUERY (in particular: changes made to predicates) will disappear when the stage terminates.

It is worth noting that an error in stage QUERY is treated as an error in the transaction: the transaction will abort. So this stage can be used for additional checking of consistency (in a way that is not expressed by the installed constraints).

Cleanup and teardown

The purpose of this stage is to ensure that the system is ready for the next transaction:

  • Removal of transaction-lifetime blocks and predicates.
  • Removal of the contents of pulse predicates (followed by an additional round of maintenance on the execution graph of stage FINAL, in order to reset it to a state corresponding to empty pulse predicates and to ensure that no further changes to non-pulse predicates occur). (See Section 19.3.1, “Pulse predicates”.)
  • Removal of various kinds of auxiliary data, such as information about the contents of predicates at various stages. (See Section 19.4.4, “Stage suffixes”.)
  • Various internal operations that set up the database for the next transaction.

19.4.3. Order of execution

If a transaction executes multiple inactive blocks or transaction-lifetime blocks at stage INITIAL, then the blocks are not necessarily executed in the order in which they have been written. The rules from all the blocks are combined, and an attempt is made to sort them according to their dependencies, in order to make evaluation more efficient; however, cyclic dependencies between rules, even from different blocks, are allowed and cause no problems.

Example 19.11. Execution of multiple blocks

The following example demonstrates that execution of the blocks is not sequential, and that cyclic dependencies between blocks are supported. The example prints 5, which is only possible if the blocks are not executed sequentially.

create --unique

addblock <doc>
  p(x) -> int(x).
  t1(x) -> int(x).
  t2(x) -> int(x).
  lang:pulse(`t1).
  lang:pulse(`t2).
</doc>

//---------------------------------
transaction     // start a new transaction

exec <doc>
  +t2(x) <- +t1(x).
</doc>

exec <doc>
  +t1(5).
  +p(x) <- +t2(x).
</doc>

commit          // commit and end the transaction
//---------------------------------

print p

close --destroy 

Notice that we enclosed two blocks (added by exec commands) within one transaction. Had we relied on the default behaviour (one transaction per block), p would have been empty.

19.4.4. Stage suffixes

LogicBlox allows logic to refer to the state of a predicate during an earlier (or current) stage of a transaction by means of a stage suffix of the form @stage.

Note

The same notation (i.e., extending a predicate name with @name) is used also for referencing predicates from different branches: see Section 41.3, “Branches in LogiQL Rules”.

These stage suffixes have very different meanings for "normal" atoms and for delta atoms. For a unary predicate p:

p@previous(x) or p@prev(x)

refers to the contents of p just before the transaction began.

(The perceptive reader will notice that this appears as @start in the diagram above. That is also the name used internally by the system, so it might appear in some error messages.)

p@initial(x) or p@init(x)

refers to the contents of p just after stage INITIAL, i.e., after evaluating the transaction-lifetime rules that make changes to EDB predicates, but before installed (i.e., database-lifetime) rules are evaluated.

p@final(x)

refers to the contents of p after stage FINAL, i.e., after installed logic rules have been evaluated and a fixpoint has been reached. Since a stage tag cannot refer to a stage that is executed later than the logic in which the tag occurs, @final is only valid in installed rules: it must not be used in a transaction-lifetime rule.

p(x)

an atom without a stage suffix is interpreted as referring to the contents of the predicate after stage FINAL.

+p@initial(x) or +p@init(x)

refers to the insert requests for p in stage INITIAL.

+p@final(x)

refers to the insert requests for p in stage FINAL.

+p(x)

refers to the insert requests for p in this transaction so far:

  • +p@initial(x) in stage INITIAL;
  • (+p@initial(x); +p@final(x)) in stage FINAL.

Example 19.12. Stage suffixes

The script below sets up a transaction (marked by "====== A"), in which f@previous = {"start"}, there is an initial delta +f("initial"), and a final delta +f("final").

create --unique

addblock <doc>
   f(s)           -> string(s).
   insert_to_f(s) -> string(s).

   // This installed rule will create a @final delta:
   +f(s) <- +insert_to_f(s).
</doc>

exec <doc>
   +f("start").
</doc>

echo "======= A"
exec <doc>
   // This will create an @initial delta:
   +f("initial").

   // This will create a @final delta, via the installed rule:
   +insert_to_f("final").
</doc>

close --destroy 

In an installed logic rule, in the transaction marked "====== A", the predicates f, f@previous, etc. would have the following contents:

Stage tag Predicate Deltas
@previous

f@previous = {"start"}

@initial

f@initial = {"start", "initial"}

+f@initial = {"initial"}

@final

f@final = {"start", "initial", "final"}

+f@final = {"final"}

(no stage tag)

f = {"start", "initial", "final"}

+f = {"initial", "final"}

Example 19.13.  Using stage suffixes to obtain information about tuples to be deleted

In the following script we create a predicate in the first transaction, and populate it in the second one.

In the third transaction we want to update the age of Mary and delete all information about people whose age is 6. Unfortunately, we no longer remember who are those people. We can query the database, and the right way to do this is by using a stage suffix, as shown below.

create --unique

addblock <doc>
   age[name] = years -> string(name), int(years).
</doc>

exec <doc>
   +age["Mary"] = 7.
   +age["John"] = 6.
   +age["Jim"]  = 10.
</doc>

print age
echo ------

exec <doc>
   ^age["Mary"] = 8.

   -age[person] = _ <- age@prev[person] = 6.  // <<<<<<<<<<<<<<<
</doc>

print age
echo ------

close --destroy 

The resulting printout looks like this:

created workspace 'unique_workspace_2016-06-24-21-00-21'
added block 'block_1Z1C3A9I'
"Jim"  10
"John" 6
"Mary" 7
------
"Jim"  10
"Mary" 8
------
deleted workspace 'unique_workspace_2016-06-24-21-00-21' 

If we want to just delete "Jim", we don't have to remember his age: it is enough to just write

-age["Jim"] = _.

Indeed, an attempt to specify the age

-age["Jim"] = 10 

would result in an error message.

Stage suffixes in auxiliary internal predicates

(This subsection mentions details of the current implementation of the runtime system. We include it here, because the user will sometimes be exposed to these details in error messages.)

In Section 19.2.2, “Delta rules” we mentioned "delta predicates", i.e., pulse predicates (Section 19.3.1, “Pulse predicates”) that are created and maintained by the system to keep information about requests for insertions and deletions.

In the internal representation of a delta predicate its name refers explicitly to its stage. For example, an atom such as +f[x]=y that appears in a transaction-lifetime rule (which will be evaluated at stage INITIAL) will be represented as f$delta_initial_insert[x]=y.

Such renaming is internally applied also to LogiQL rules which use a combination of delta syntax and stage suffixes. For example, the atom +f@final[x]=y in a query-lifetime rule will be represented as f$delta_final_insert[x]=y.

19.4.5. Ghost Entity Check

A ghost entity is an entity-typed value that occurs in a predicate, but not in the predicate that enumerates values of that entity type. In the following, entity [1] is a ghost entity.

Table 19.1. An example of a ghost entity

Entity band Predicate nameOf
[0] ([0], "The National")
[2] ([1], "The Mountain Goats")
  ([2], "Belle and Sebastian")

In LogicBlox 3.x databases may contain ghost entities and developers are responsible for programming accordingly. LogicBlox 4 includes a compile-time check that issues an error for rules that may lead to the introduction of ghost entities.

It is only necessary to check active delta rules, all other rules are safe. A ghost entity is created when an entity-typed value is inserted by the head of a delta rule, but nothing in the body ensures that this particular value will still exist at transaction end. For example, each of the following rules may lead to creation of ghost entities and is flagged with an error.

// Error
+nameOf[b] = "foo" <- -band(b).

// Error
+nameOf[b] = "foo" <- -myFavoriteBands(b).

// Error
+nameOf[b] = "foo" <- band@initial(b), +someEvent(_).

In most cases, a rule can be made safe by extending its body with an atom ensuring that the added entities are not ghosts. For instance, the following rules are safe:

+nameOf[b] = "foo" <- -myFavoriteBands(b), band(b).

+nameOf[b] = "foo" <- band@initial(b), +someEvent(_), band(b).

+nameOf[b] = "foo" <- +band(b).

The rule

+nameOf[b] = "foo" <- -band(b), band(b).

is safe too, but will never fire.

19.4.6. Frame rules

The LogicBlox system uses frame rules to declaratively specify how changes to the state of the database are handled at different stages of a transaction.

We illustrate the principal ideas by means of a simple example, then list the details. While frame rules are internally generated rules that the user does not write, they can appear in error messages and must be taken into account when analyzing performance, so it is useful to have a basic understanding of how they work.

An example

Consider the following trivial lb script:

create W

addblock --name B <doc>
  age[x] = y -> string(x), int(y).
</doc>

exec <doc>
  +age["Ann"] = 7.
  +age["Bob"] = 6.
</doc>

close --destroy

The delta rule +age["Ann"] = 7 is transformed into a more detailed internal representation that looks like this:

Forall __t0::string,__t1::int .
   age$delta_initial_insert{block_1Z1CY2UJ:1(1)--1(15)#1Z1CZI9D}#0[__t0]=__t1 <-
      string:eq_2(__t0,"Ann"), int:eq_2(__t1,7).

Such rules can be read only with some difficulty, so henceforth we will make them more palatable by editing out quantifiers, type information, redundant parentheses and identifiers of internal blocks. For good measure we will also rename variables, and omit rules whose only function is to consolidate rules that are almost identical, but have different information about blocks encoded in the names of their head atoms. The above rule is the first one shown below: it might be instructive to carefully compare the two versions.

age$delta_initial_insert[x] = y <- string:eq_2(x, "Ann"), int:eq_2(y, 7).
age$delta_initial_insert[x] = y <- string:eq_2(x, "Bob"), int:eq_2(y, 6).

The p$delta_initial_insert predicate is an internal predicate that exists for every EDB predicate p. It is used to collect requests to insert facts into the predicate. The delta rule that derives facts into age$delta_initial_insert does not immediately cause these facts to be in the actual predicate age. Similarly, there is a predicate age$delta_initial_erase that contains the requests for deleting facts.

The request changes in age$delta_initial_insert and age$delta_initial_erase are applied to the actual predicate by a frame rule. This happens separately for every stage where changes can be requested (INITIAL and FINAL). For stage INITIAL, the frame rule for age must take the predicate age@START, consider the delta requests, and create the predicate age@INITIAL, by applying appropriate changes to a (logical) copy of age@START. The frame rule is shown as follows, in the form of a LogiQL rule that derives into an auxiliary predicate whose contents are the changed tuples, each annotated with an insert/erase tag. In reality the rule has an internal implementation that cannot currently be expressed in logic.

age@START..INITIAL[x, delta] = y <-
    age$delta_initial_insert[x] = y,
    delta:eq_2(delta, DELTA_INSERT)
  ; age$delta_initial_erase(x),
    age@START[x] = y,
    delta:eq_2(delta, DELTA_ERASE).

Let us now extend our example by adding one more simple transaction at the end:

exec <doc>
   ^age["Ann"] = 8.
   -age["Bob"] = _.
</doc>

The upsert and the deletion are translated to

age$delta_initial_upsert[x] = y <- string:eq_2(x, "Ann"), int:eq_2(y, 8).

age$delta_initial_erase(x) <- string:eq_2(x, "Bob").

Since there is an upsert, we get also an upsert-erase rule and an upsert-insert rule. The first rule states that if there is an upsert for x at stage INITIAL, and there is a previous value for x (i.e., at stage START), then the latter should be erased at stage INITIAL. The second rule states that the upserted value should be inserted at stage INITIAL.

age$delta_initial_erase(x) <-
   age$delta_initial_upsert[x] = _,
   age@START[x] = _.

age$delta_initial_insert[x] = y <-
   age$delta_initial_upsert[x] = y.

Details

The name of a delta predicate is of the form name$delta_stage_kind, where

  • stage is initial, final or all;

  • kind is insert, erase or upsert.

Additionally, for pulse predicates there are delta atoms of the form name$delta_query_insert.

Stage all corresponds to the combination of stages INITIAL and FINAL. Predicates with this stage are defined by rules such as

f$delta_all_insert[x] = y <-
    f$delta_initial_insert[x] = y
  ; f$delta_final_insert[x] = y.

f$delta_all_erase(x) <-
    f$delta_initial_erase(x)
  ; f$delta_final_erase(x).

If f is a pulse predicate, then the following rule is used at stage QUERY:

f$delta_all_insert[x] = y <-
    f$delta_initial_insert[x] = y
  ; f$delta_final_insert[x]   = y
  ; f$delta_query_insert[x]   = y.

If there is an f$delta_final_upsert predicate, then the following rules are generated:

f$delta_final_insert[x] = y <- f$delta_final_upsert[x] = y.

f$delta_final_erase(x) <- f$final_upsert[x] = _, f@initial[x] = _.

(And similarly for stage INITIAL, as shown in our example above.)

If there are rule(s) with head atoms of the form f$delta_stage_kind[x] = y, the following frame rules are generated to apply the deltas:

  • At stage INITIAL:

    f@START..INITIAL[x, delta] = y  <-
         f$delta_initial_insert[x] = y, delta = DELTA_INSERT
       ; f$delta_initial_erase(x), f@START[x] = y, delta = DELTA_ERASE.

  • At stage FINAL:

    f@INITIAL..FINAL[x, delta] = y <-
         f$delta_final_insert[x] = y, delta = DELTA_INSERT
       ; f$delta_final_erase(x), f@INITIAL[x] = y, delta = DELTA_ERASE.

The above patterns were shown in the simplest versions, where they are applied to a functional predicate with one key argument. If a functional predicate has more key arguments, then these are all used where appropriate, in particular in predicates that end with _erase. Non-functional predicates are treated similarly, except that there will be no upserts and related rules (the ..._erase predicates will contain all the arguments).

The separation between requests for changes and application of the changes has several implications:

  • If a fact is derived into p$delta_initial_insert when it already exists in p, then the delta_initial_insert predicate will contain the new fact, but the actual predicate p will not change (i.e., it is not an error to insert an already existing fact). This means that the author of delta logic that is triggered by +age must be cautious about such re-assertions. For example, +age[x] = y, !age@prev[x] = y may in some cases be required.

  • Similarly, it is not an error to retract a fact that does not exist. This means that the author of delta logic that is triggered by -age should also be cautious about the existence of the fact that is requested to be retracted.

  • While derivation of tuples into delta_initial_insert may succeed, the application of the deltas can still fail. For example, if the current age of Ann is 7, and we attempt to change her age to 8 by using +age["Ann"] = 8 rather than ^age["Ann"] = 8, then when the frame rule is evaluated a functional dependency violation will be reported. This is why a functional dependency violation error points at a frame rule as the source of the error.

Since delta predicates are state specific, and since frame rules are used to apply the change requests at both stage INITIAL and stage FINAL, it is possible to make changes twice in a single transaction, and these changes are allowed to be conflicting. For example, the following logic assures that the author of a document gets an appropriate bonus, no matter what bonus was requested.

create W --overwrite

addblock --name B <doc>
  bonus[x] = y -> string(x), int(y).
  author(x)    -> string(x).

  ^bonus[x] = y * 2 <- +bonus@initial[x] = y, author(x).
</doc>

exec <doc>
  +author("John").
  ^bonus["John"] = 50.
</doc>

print bonus

close --destroy

The existence of the delta_all predicate makes this feature a bit difficult to use. The predicate is lazily generated when needed, and if it is generated, then the example above causes a functional dependency violation on delta_all. This happens, for instance, if we add the following to the contents of addblock:

salary[x]  = y -> string(x), int(y).
payment[x] = y -> string(x), int(y).

^payment[x] = salary[x] + z <- +bonus[x] = z.

We plan to remove the delta_all in the near future and introduce a more convenient and efficient way to use all the actual changes made to a predicate.