LogicBlox Administration Guide


Part I. LogicBlox Administration Guide

Chapter 1. LogicBlox Components and Architecture

The LogicBlox platform is composed of three server programs and the lb utility program. Server programs are shown in orange; utility programs in blue. The server (or daemon) programs are lb-server, lb-web, and lb-compiler.

lb-server

The main database server is lb-server. Within lb-server, workspaces with logic and data are managed. In particular, lb-server manages all active and inactive logic, maintains materialized views, and executes queries. Not only is lb-server executing LogiQL, but it also manages the existence of workspaces and branches within.

lb-compiler

LogiQL code is compiled to a machine representation by the lb-compiler. The lb-server and lb-compiler coordinate this process, so users can exclusively interact with lb-server. The lb-compiler runs as a separate process because it is written in Java.

lb-web

The last server component is lb-web, which exposes services through a standard http interface. Services from lb-web can be accessed directly from the browser or via standard command line utilities such as curl or wget.

lb tool

Users and administrators can interact with all LogicBlox server components via the lb command line tool. All server processes support lb <component> status to obtain general information about the process. Additionally, the lb tool comes with extensive documentation that can be reached by issuing lb -h at the command line.

Chapter 2. Installation & Update

2.1. Obtain Latest Stable Release

LogicBlox releases new versions about once a month. LogicBlox releases are available from download.logicblox.com

Accompanying release notes highlight which aspects of the platform have improved. The platform is generally back-wards compatible, that is applications written for older versions are expected to also work with newer versions of the platform. For the rare cases that special handling is required for applications while updating the platform, these will also be detailed on the release site.

Even though, application code (eg, LogiQL) should be able to run on new versions unmodified, actual workspaces are (generally) not binary compatible between version upgrades. That means a workspace created by an older version can in general not be opened by a newer version, requiring export and import of EDB data. Having binary-compatibility for workspaces is part of LogicBlox's road-map.

2.2. Binary Installation via tar.gz (64-bit Linux)

If you run a 64-bit flavor of Linux, you can install LogicBlox natively on your machine. The easiest way is to use the binary distribution available as a tar.gz.

System requirements

  • A 64-bit flavor of Linux. Verify that you're running 64-bit Linux by running: uname -m which should return x86_64.

  • At least 8GB of available memory.

  • Python 2.7 or newer (but not Python 3.x). On Ubuntu systems, install via

    $ sudo apt-get install python2.7

    To verify the correct version of python is installed issue python --version.

  • Java version 8 or newer. Available from Oracle's Java website It is also possible to use OpenJDK. To install the required Java on Ubuntu systems, use:

    $ sudo apt-get install openjdk-8-jre

    To verify the correct version of Java is installed check that java -version reports a version starting with 1.8.

Installation steps

  • Download the latest LogicBlox tarball from the LogicBlox release page.

  • Extract the tarball into a directory of your choice. A good location is /opt/logicblox. It is also good habit to create a symbolic link to the active version. Assuming the version you downloaded is 4.1.4:

    $ sudo mkdir /opt/logicblox
    $ cd /opt/logicblox
    $ sudo tar -xzf logicblox-4.1.4.tar.gz
    $ sudo ln -s /opt/logicblox/logicblox /opt/logicblox/logicblox-4.1.4

    Tip

    Simply extracting the contents of the tarball "installs" LogicBlox with default configurations. Custom configurations are performed in a separate location to ease the updating procedure. It is possible to unpack the software to a place that is subsequently mounted read-only to prevent tempering.

  • The archive contains a script to setup all the environment variables required to run LogicBlox. You can source this script to set these up automatically. With bash as shell, do:

    $ source /opt/logicblox/logicblox/etc/profile.d/logicblox.sh

    Tip

    For convenience, you may want to add this line to your shell's startup script, e.g. in ~/.bash_profile or ~/.bashrc.

    To verify the steps so far issue lb status, which should report that the lb-server is currently not running:

    $ lb status
    lb server is OFF

2.3. Managing LogicBlox Services

LogicBloxRequires three daemon processes to be running to be operational. Ideally, these are started, monitored and shut down by the operating system rather than the user.

We detail two methods for managing the daemons. The first method is using the lb services command, which is a simple, portable shell script for managing daemons. This should only be used for development and testing purposes, because this method lacks proper daemon management features that tools for specific Linux distributions provide (systemd, upstart). The second, preferred method explains how to use systemd to manage the LogicBlox daemons.

2.3.1. Manually Starting Services

If daemons are managed by systemd then systemctl should be used to stop and start daemons. The following applies if daemons are managed ad-hoc via lb services.

After performing the installation steps, LogicBlox services need to be started before the LogicBlox server can be used. This is done via:

$ lb services start

An overview of the currently running services should now look like:

$ lb services status
lb-compiler              : OK
lb-server                : OK
lb-web-server            : OK

2.3.2. Manually Stopping Services

If daemons are managed by systemd then the appropriate systemctl commands should be used to stop and start daemons. The following applies if daemons are managed ad-hoc via lb services.

The command lb services stop shuts down all LogicBlox daemons. To list currently running daemons, use lb services print.

Daemons can also be terminated and restarted individually. To terminate individual daemons, use the command stop as shown in the list below; to start use the command start:

lb-server

$ lb server start

$ lb server stop

lb-compiler

$ lb compiler-server start

$ lb compiler-server stop

lb-web

$ lb web-server start

$ lb web-server stop

In rare cases, deamons might hang and not respond to stop requests by lb services. It is then safe (in the sense that the database will not be corrupted) to use OS-level kill instructions to shut down daemons. The PID of the respective processes can be obtained from lb services processes. If processes do not react to kill <pid>, then it is also safe to kill via SIGKILL. If any of the daemons are killed currently running transactions can be aborted. Transactions that have been committed via softcommit mode might not have been written to disk yet and can thus be lost. Any transaction that was committed via diskcommit mode is durable on disk. In any case, killing processes (even with -9 / SIGKILL) does not corrupt LogicBlox workspaces.

2.3.3. Managing Services with Systemd

It is highly recommend to use a system service manager for the management of the database daemons. For recent Linux distributions, this is usually systemd. For older Linux distributions, this could still be upstart. The system service manager of the operating system typically implements robust features to track the process id of the daemons, restart daemons on failures, and properly manage the logs produced by the daemons. For systemd, the services are managed using the systemctl command. Systemd works with journald to manage LogicBlox logfiles that can then be read via journalctl. See the Systemd chapter in this manual for details on using LogicBlox services with systemd.

2.4. Testing Installation

The following hello-world-sequence of commands verifies that lb-server and lb-compiler are working as expected:

$ lb create testworkspace
created workspace 'testworkspace'
$ lb addblock --name firstblock testworkspace 'x(i) -> int(i). double(2*i) <- x(i).'
added block 'firstblock'
$ lb exec testworkspace '+x(21).'
$ lb print testworkspace double 
42
$ lb delete testworkspace 
deleted workspace 'testworkspace'

Tip

The command line tool lb comes with extensive tab-completion capabilities. It completes top-level commands such as addblock or create, workspace names, command options, and even predicate names.

To confirm lb-web is running properly, list available services:

$ lb web-server list --services
--------------- ------------- -------------- ----------------- ----------------- ------- -----------
     Prefix      HTTP Method    Workspace          Groups       Disabled Status   Realm   Endpoints
--------------- ------------- -------------- ----------------- ----------------- ------- -----------
 /lb-web/admin   POST          no workspace   lb:web:internal   enabled                   internal
 /login          POST          login                            enabled

2.5. Update Procedure

To update the installation of LogicBlox, stop running services as described in Section 2.3. Then follow the steps for a fresh installation. Here, removing the contents of the old version from /opt/logicblox is safe. All configuration data and workspace data is stored elsewhere. While application code should generally continue to work well on future releases of the LogicBlox platform, actual workspace data stored on disk can generally not be read by different versions of the platform. It is thus necessary to re-deploy and to export and import all data. It is on the future LogicBlox roadmap to address this issue, that is to allow newer versions of the platform to open workspaces created with an older version.

2.6. Trouble Shooting

LogicBlox daemon processes have associated log-files that usually explain the cause of potential errors or problems. For lb services, the logfiles are located under $LB_DEPLOYMENT_HOME/logs/current/. If LB_DEPLOYMENT_HOME is not set, then it defaults to $HOME/lb_deployment. For systemd, consult journalctl.

Lb-web daemon does not start

The logfile for lb-web is lb-web-server.log. A common reason for lb-web not starting is if there are workspaces created with an older version of LogicBlox that are not workspace-binary compatible. If this is the case, removing these workspaces via lb delete <workspacename> would address the problem.

Lb-server does not stop

Stopping the lb-server can sometimes take time as all workspaces will be closed and durably written to disk. Even though transactions are constantly written to disk during normal operation, closing workspaces can sometimes take longer.

However, in rare circumstances lb-server might appear to be idle (no CPU nor I/O performed for a while) and still fail to shut down. These situations are likely caused by software bugs and the LogicBlox team will be thankful for any information that is required to investigate the issue. Information includes stack-traces, log-files and other details about workloads and operations. It is best to contact LogicBlox in these cases. To quickly get operational again, it is OK to use system-level mechanisms to kill lb-server. Besides using SIGTERM, that is kill PID, it is also safe to kill lb-server with the signal SIGKILL, that is kill -9 PID. In both cases, the following will happen:

  • Transactions that are currently in queues will be lost, connections to clients are aborted.

  • Transactions that have not finished committing to disk will likely be lost.

  • The workspace on disk will not be corrupted. In particular, when opened by a fresh lb-server the workspace will be in a state in which all transactions up to a transactions x are committed, whereas transactions that were to be committed after x will not be visible. In particular, for any transaction, either it is fully committed meaning all of its changes have been written to disk, or it never happened meaning none of its changes is visible.

Transaction processing and writing to disk is decoupled meaning that transactions are continuously processed and when no integrity violations appear are continuously applied to the memory-representation of the workspace. In parallel, a committer-component writes the changes to the memory representation to disk, essentially group-committing potentially batched up transactions. The LogicBlox technology ensures that the process of writing changes to disk will always atomically write either all or none of the new transaction data to disk -- even if interrupted at any time.

Transactions can be configured in two modes diskcommit or softcommit. From the engine's point-of-view, they are processed exactly the same way as described above. The only difference is that for diskcommit, the client is notified of the committed transaction when it has been written to disk. Thus, once the client has been notified, even a kill -9 on the server won't prevent the transaction to be applied. If transactions are configured as softcommit then the client is notified as soon as the transaction has been applied to the memory-version of the workspace. It will thus soon after be written to disk. However, sudden failures might prevent this.

In general, both diskcommit and softcommit achieve the same transaction throughput, however transaction latency is generally better with softcommit. We suggest using diskcommit as default and only resorting to softcommit if transactions require very low latencies and it is acceptable that softcommitted transactions are not persisted in the event of failures (power outages, kill -9, etc.).

Chapter 3. Server Configuration

LogicBlox services can be configured by various config files and with command-line parameters. In general, default configurations are provided within config files that are located in the LogicBlox installation directory (eg., /opt/logicblox). These configuration files should not be changed. Instead, the configuration files under $LB_DEPLOYMENT_HOME/config should be used to override default configurations. If $LB_DEPLOYMENT_HOME is not set, it defaults to $HOME/lb_deployment.

In the following we explain the configuration options for the LogicBlox server components in turn.

3.1. lb-server

The configuration files for lb-server are $LOGICBLOX_HOME/config/lb-server.config and $LB_DEPLOYMENT_HOME/config/lb-server.config. The latter overwrites the default settings specified in the former file. Furthermore, command-line options that are provided when launching the server do overwrite settings in configuration files.

The config file is in a standard ini format structured in sections in which properties are assigned specified values. An example configuration is the following:

[server]
port=5518
adminPort=5519

[workspace]
folder=$LB_DEPLOYMENT_HOME/workspaces

[logging]
file=$LB_DEPLOYMENT_HOME/logs/current/lb-server.log
level=info

To illustrate the customization of these settings: if the configuration file $LB_DEPLOYMENT_HOME/config/lb-server.config contains only the following adjustment of the logging level, then only this setting is customized to perf. For all other settings the defaults specified in $LOGICBLOX_HOME/config/lb-server.config are used.

[logging]
level=perf

In the following, we list configurable properties categorized by the section of the file they occur in.

server

port

Specifies the TCP port on which lb-server is listening. Default: 5518.

adminPort

Specifies the TCP port on which lb-server is listening for admin-commands such as lb status. Default: 5519.

unix_domain_socket

Configures lb-server to listen on the specified Unix domain socket instead of on the regular TCP ports. The lb tool can be used to connect to a unix-domain socket via lb -u /path/to/socket. By default, lb-server listens on TCP ports.

suppress_task_parallelism

When suppress_task_parallelism is set to any value, then the server will not evaluate different rules in parallel. We do not recommend that task parallelism is disabled, but it can be useful for performance analysis. Without task parallelism the duration of separate rules can be measured better and the log is easier to understand. Note that disabling task parallelism does not disable domain parallelism, where a single rule is evaluated in parallel over subsets of the data.

suppress_domain_parallelism

When suppress_domain_parallelism is set to any value, then the server will not evaluate individual rules in parallel by subsetting the data and executing the subsets individually. We do not recommend that domain parallelism is disabled, but it can be useful for performance analysis. It can also be useful in certain high-concurrency situations, where server resources are already maximized. Note that disabling domain parallelism does not disable task parallelism, where different rules are evaluated in parallel.

workspace

folder

Specifies the root folder where all workspace (=database) data is stored. Ideally, this should be on an SSD device with plenty of free space. Default: $LB_DEPLOYMENT_HOME/workspaces.

commit_mode

Default transaction commit mode. Allowed values are diskcommit (the commit doesn't return until the transaction has been durably written to disk) and softcommit (the commit may return before the transaction has been written to disk, so in case of a crash the database may roll back to a previous version). The commit mode can be overriden per transaction using lb's --commit-mode option. Default: softcommit.

replication_mode

Whether transactions are replicated synchronously. (See Section 5.4 for information about replication.) If set to synchronous, a disk commit won't return until the transaction has been durably replicated to all connected mirrors. If set to asynchronous, a disk commit may return before the transaction has been replicated.

auto_backup_mode

Versions of the database are automatically tagged for backup and investigation purposes. If set to none, then this facility is disabled. If set to default, then upto 64 database versions are continuously kept according to a strategy that gradually keeps less versions around as the versions get older (a chart of version numbers vs the log of the age of the version will roughly be a straight line). Default: default.

workspace:wsname

You can override the workspace properties listed above (except folder) per workspace by adding a section named workspace:wsname, where wsname is the name of the workspace. For example, the following states that the default commit mode shall be diskcommit for every workspace except foo:

[workspace]
commit_mode=diskcommit

[workspace:foo]
commit_mode=softcommit

logging

file

Specifies the log file location. If file already exists, log data will be appended. This option is not used when systemd is set or inferred to true. The special value - can be used to log to stdout (which only makes sense if the server is not run as a daemon). Default: $LB_DEPLOYMENT/logs/current/lb-server.log.

systemd

If true, log to stderr, use systemd log levels (e.g. <3>) and omit timestamps. If not explicitly set, then the value is inferred from environment variables that are set in the systemd units that we provide.

level

Specifies the default logging level. Allowed logging levels are: error, warning, info, perf, and debug. For a typical deployment scenario, use level=info. To diagnose performance issues, use level=perf. For debugging LogiQL rules, use level=debug.

More fine-grained control can also be achieved by specifying log-scope specific settings. For instance, to use info-level logging by default, but debug-level logging under rule evaluation, one can specify level=info:debug@EvaluateRules.

The log level can also be specified on a per-transaction basis, with the log returned for a transaction at the specified level. The following command will, for example, evaluate the request under the perf level:

lb addblock testworkspace 'x(x)->string(x).' -L --loglevel perf
monitor_rule_time

Specifies the duration in milli-seconds beyond which rules will be logged as long-running. The system will log (at warning level) all rules which take longer to execute. By default, the threshold is 10000 ms. This value can also be adjusted using the environment variable LB_MONITOR_RULE_TIME.

compiler

jvmArgs

Specifies the default JVM arguments for the compiler. Unless jvmServerArgs is set, this is used for both separate compilation and the compiler server. Otherwise, this only applies to separate compilation. Default: -Xmx2000m -Xss2048k.

jvmServerArgs

Specifies the JVM arguments for the lb-compiler server. If this setting is not specified, then jvmArgs is used for the compiler server.

port

Specifies the TCP port on which the compiler is listening. Default: 5520.

adminPort

Specifies the TCP port on which the compiler is listening for admin requests. Default: 5521.

3.2. lb-web

Configuration of lb-web is more complex as it includes security aspects like defining SSL endpoints and configuring user authentication. Available configuration options and suggestions for common configuration scenarios are detailed in the Reference Manual.

3.3. Environment Variables

The following environment variables can be used alongside configuration files to influence aspects of the LogicBlox platform. In general LogicBlox environment variables have the prefix LB_.

3.3.1. Location Of Components

The following variables specify where various components of the LogicBlox platform and dependencies are installed. They are usually set up by sourcing etc/profile.d/logicblox.sh in the LogicBlox directory.

LOGICBLOX_HOME Directory where LogicBlox is installed. For example, /usr/local/logicblox-4.
JAVA_HOME Variable can be used to select a specific Java installation. If not set, the system's standard Java is used.
LB_WEBSERVER_HOME Directory where the lb-web component is installed. Usually, this is in $LOGICBLOX_HOME/lb-web.
LB_WORKFLOW_HOME Directory where the lb-workflow component is installed. Usually, this is in $LOGICBLOX_HOME/lb-workflow.
LB_MEASURE_SERVICE_HOME Directory where the lb-measure service component is installed. Usually, this is in $LOGICBLOX_HOME/measure-service.
LB_WORKBOOK_SERVICE_HOME Directory where the lb-workbook-service component is installed. Usually, this is in $LOGICBLOX_HOME/lb-workbooks.
LB_LIBRARY_PATH Specifies a colon-separated list of directories that are searched for LogiQL libraries. Defaults to $LB_WEBSERVER_HOME/share:$LB_MEASURE_SERVICE_HOME/share.

3.3.2. Configuration Variables

LB_DEPLOYMENT_HOME Directory where workspaces, configuration files, log files, etc. are stored. If not set, this defaults to $HOME/lb_deployment.
LB_LOGDIR Overwrite directory where log-files are written to. Defaults to $LB_DEPLOYMENT_HOME/logs.
LB_TEMPDIR Overwrite directory where temporary files are written to. Defaults to the operating system temporary directory (such as /tmp in Linux).
LB_MEM The LB_MEM variable is used to specify the amount of physical memory to be used by the lb-server for the page buffer pool. The value can be given as a percentage, in megabytes or gigabytes. For example, valid settings are 80% specifying that 80% of the main memory can be used; or 10G or 12500M specifying that 10 GiB or 12500MiB should be used respectively. By default, a heuristic is used which selects little more than half of the available main memory after setting aside 1GB; For machines with more than 32GB of memory, proportionally more memory is used.
LB_PRODUCTION_ENVIRONMENT Should be set to 1 in production environments. Will prevent lb-server from launching lb-compiler if not currently running. In production environments, these devices should be managed by the operating system. This will also suppress logging of some warning messages that are useful only during the development of an application.
LB_CONNECTBLOX_ADMIN When set to 1, workspaces can be deleted via lb and other tools connecting to the lb-server directly without authentication. In production environments, non-trusted clients should not be allowed to directly access the lb-server; this option is thus often enabled.
LB_WEBCLIENT_JVM_ARGS Can be used to configure the JVM arguments of all lb web-client invocations. Example value: -Xmx200m.
LB_WARNING_OPT_LIMIT The runtime may issue warnings called "optional warnings": if the number of such warnings exceeds a limit of ca. 100 per second, then they are suppressed until sufficient time elapses to start showing them again. This environment flag may be set to a value that changes the limit of optional warnings per second. For example, LB_WARNING_OPT_LIMIT=1 will allow only one such warning per second, while LB_WARNING_OPT_LIMIT=0 will suppress them altogether.

Chapter 4. Monitoring and Intervention

4.1. Review Current Executing Transactions

Transactions are generally either in a queue waiting for execution or they are active. LogicBlox processes read-only transactions in parallel utilizing all available cores on the machine, while read-write transactions are serialized.

To list transactions that are currently under processing for a specific workspace such as testworkspace, use:

lb status testworkspace --active

To also list all requested transactions that are currently within some queue, use:

lb status testworkspace --all

Note that it is important to select the workspace for which the detailed status is inquired. To list all transactions that are in progress or in some queue by any workspace use for example:

lb workspaces | xargs -n 1 lb status --all

Additional information, such as general information of how many workspaces are opened by the lb-server can be obtained by adding --debug to the status command. Note that information provided by --debug might change in future releases.

4.2. Logfiles

In the following, we describe logfiles and their default locations for installations that are managed via lb services.

LogicBlox daemons create logfiles in $LB_DEPLOYMENT_HOME/logs. In general, they contain valuable information for trouble-shooting the LogicBlox system. The following gives a brief overview of the individual logfiles and the content they provide.

lb-compiler

The lb-compiler.log logfile contains a status entry once the compiler has been started indicating the TCP ports on which the compiler is listening for connections. Additional entries indicate unexpected problems that should be addressed.

lb-server

The lb-server.log logfile contains information according to the configured log-level. During start-up, all configurations and significant environment variables are printed. At log-level info only major database events such as workspace creation or deletion are logged. With loglevel perf all activity of the lb-server is logged. This includes received requests; started and finished transactions; evaluation of meta-rules that manage the schema-level information stored in workspaces; evaluation of regular LogiQL rules; and any other activities. Furthermore, also details about fixpoint evaluations and optimizer decisions are logged at log-level perf.

Tip

Individual requests can be executed at a log-level that differs from the default log-level for the lb-server. To do so, use the command-line option --loglevel for lb commands. The log for such requests can then also be returned to the lb command by using the option --log.

lb-web

The lb-web component creates multiple logfiles: lb-web-server.log and files such as lb-web-server-access-2014_11_18.log.

The former contains information about the general status of the lb-web. During start-up, all workspaces are scanned to gather all services defined within them. Discovered services are then summarized in the logfile. Additional information in this log-file should be sparse and is generally indicative of some error condition.

The access logfiles, for example lb-web-server-access-2014_11_18.log, log accesses to the lb-web server. Each connection from a client is logged using the standard Combined Log Format a widely used extension of the Common Log Format (NCSA Common log format). Entries in this log format look like:

127.0.0.1 -  -  [18/Nov/2014:17:10:35 -0800] "POST /lb-web/admin HTTP/1.1" 200 132 "-" "Python-urllib/2.7" 12

Each entry is explained below:

127.0.0.1

The IP address of the client.

-

Here, the RFC 1413 identity of the client would be written. The hyphen indicates that this information is not available.

-

Here, the user id of the program sending the request is written if available. As the with the previous field, this information is usually not available, and moreover not reliable.

[18/Nov/2014:17:10:35 -0800]

The date and time when the server finished processing the request.

"POST /lb-web/admin HTTP/1.1"

The request line of the client program, containing the method (e.g., GET or POST) and the full URL.

200

The status code sent back to the client. The code is one of the usual HTTP status codes with a full list in RFC2616 section 10. Success is indicated by a code starting with 2, a redirection starts with 3, an error caused by the client with 4 and an error caused by the server with 5.

132

This is the size of the return message to the client in bytes.

"Python-urllib/2.7"

The string by which the client identifies itself. Here, the client was a python program.

12

The last entry logs the latency for the given request in milliseconds. Here, it took 12 milliseconds from receiving the request to sending the response.

Tip

Standard log-analyzing tools such as GoAccess, or AWStats can be used to analyze lb-web logfiles since they are in the standard "combined log format".

4.3. Performance Monitoring

Performance can be monitored at the lb-server or lb-web level. Database transactions can best be monitored with the perf loglevel following the lb-server logfile. Web-service performance is best monitored by following the access logfile of lb-web. The last column in the lb-web access log is the request duration in milli-seconds.

The database by defaults logs information in the lb-server log on rules that take more than 10 seconds to evaluate. This setting can be changed using the LB_MONITOR_RULE_TIME environment variable or the lb-server configuration option logging.monitor_rule_time. The value is expected in milli-seconds.

4.4. Resource Consumption

4.4.1. Disk Usage of Workspaces

LogicBlox workspaces are stored on disk within the workspace directory, usually within the $LB_DEPLOYMENT_HOME/workspaces directory. Here, workspace names are encoded to allow special symbols to occur in them. Use lb filepath workspacename to obtain the specific directory in which workspace testworkspace is stored. For an overview of which workspaces occupy how much space, use for example the following bash snippet:

$ for ws in `lb workspaces`; do echo -n "$ws : "; du -sh `lb filepath "$ws"`; done
w2 : 1.7M	/home/q/lb_deployment/workspaces/dzI
ws : 52M	/home/q/lb_deployment/workspaces/d3M

It is also possible to obtain fine-grained information about how disk storage is used within a single workspace. Unfortunately, this currently requires exclusive access to the workspace. It is thus necessary to either export the workspace and investigate the copy, or to shut down lb-server while performing the investigation.

For example, given a workspace stored in the directory /tmp/ws that is not concurrently accessed by the lb-server, use the following dlbatch commands to investigate which components contribute to the workspace size:

$ dlbatch
<blox> open /tmp/ws
<blox> profileDiskSpace -S -V

This will print a report similar to the following:

<blox> open /tmp/foo
<blox> profileDiskSpace -S -V   
Visited 1609 pages (52723712 bytes)

          Pages           Bytes Thing
--------------------------------------------------------------------------------
              5         163,840 refcounts
             68       2,228,224 x{block_1Z2F4REW:1(1)--1(32)#1Z2FCC2R}#0: (int)
             84       2,752,512 x$sip: (int)
            663      21,725,184 x{block_1Z2OWOZV:1(1)--1(33)#1Z2P7OYG}#0: (int)
            731      23,953,408 x: (int)
          1,546      50,659,328 oodb::TreeBase
          1,586      51,970,048 objects
          1,609      52,723,712 resource

          Pages           Bytes   Removed pages       New pages Version
--------------------------------------------------------------------------------
              0               0               0               0 2014-11-20 01:39:54,870397+00:00
             78       2,555,904               0              78 2014-11-20 01:40:19,385577+00:00
          1,468      48,103,424              78           1,468 2014-11-20 01:40:36,059848+00:00
<blox> 

To better understand the report, it is useful to know how the database was created:

$ lb create ws
created workspace 'ws'
$ lb addblock ws 'x(i) <- int:range(1,1000000,1,i).'
added block 'block_1Z2F4REW'
$ lb addblock ws 'x(i) <- int:range(1,10000000,1,i).'
added block 'block_1Z2OWOZV'
lb export-workspace ws /tmp/ws
     52,756,480 bytes received
    377,805,823 bytes/s
     86,376,448 pagefile size
     33,619,968 unused space
0.412384s (0.139639s transmit, 0.272745s fsync)
exported workspace 'ws' to directory '/tmp/ws'

We see that the database stores three versions of itself corresponding to the empty database just after creation, the database after the first block block_1Z2F4REW was added and the database after adding the second block.

Note

LogicBlox keeps some old versions of the database to allow investigation of the history or even to revert to an older state. There is a maximum number of such "automatic" backups that are maintained. Once this number has been reached, we remove older versions in a way that backups are more dense for the recent history and more sparse otherwise. For example, we might have 10 of the 64 most-recent versions, 20 of the 128 most-recent versions, 30 of the 256 most-recent versions, etc.

Returning to the disk-profile. We see that 2MB are used for the predicate x in the first block we added; while around 22MB are used for the predicate x in the second block. These two predicates are then merged into the single predicate x using 24MB of space. The entries oodb::TreeBase, objects, and resource count the same data in a different way; they should be ignored in the analysis. The total size of this workspace is around 51MB. Predicates that have $sip in their name represent "sampling and indexing" predicates that are used internally by the LogicBlox engine.

4.4.2. Memory Usage of Daemons

Standard tools such as ps, top, or htop can be used to investigate and monitor the memory usage of LogicBlox daemons. This section discusses the most important information relating to memoru usage of LogicBlox daemons. A more detailed introduction into memory-analysis on Linux systems in general is available in Chapter 9.

From the three daemons (lb-compiler, lb-web-server, lb-server), the only interesting daemon is the lb-server. The lb-compiler and lb-web-server processes are both Java processes that have a configured maximum heap size that does normally not need to be changed. The memory usage of these processes is independent of the size of the database. The lb-server is the daemon that has the database open and the memory configuration of this process is important for the performance of the application. For non-trivial databases, the bulk of a system's resident memory should be used by lb-server.

The resident memory usage of the lb-server consists of:

  • The size of the buffer pool, which is an in-memory cache of pages of the database. Virtually every database system has some type of buffer pool (sometimes called buffer cache). Similar to other databases, the buffer pool should be as large as possible, because this means that the system less frequently has to go to disk to obtain data. The size of the buffer pool should only be constrained by the memory needs of other daemons running on the server. The LB_MEM environment variable configures the size of the buffer pool (see Section 3.3.2).

  • The memory used by the database engine for the database schema, active logic, and query evaluation. Normally, this should be a fairly limited amount of memory, but for databases with an extreme large schema (thousands of predicates and rules), it could amount to a few gigabyte.

Generally, the resident memory usage of the lb-server process should be in the order of the size of the buffer pool. For extreme large schemas, the buffer pool size might have to be reduced to leave enough memory for the non-paged resident memory of the database engine and other daemons.

The total resident memory of the lb-server can be determined using standard Linux tools. For example:

$ top -b -n 1 -p `cat $LB_DEPLOYMENT/logs/current/lb-server.pid` | grep PID -A 2
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 8429 q         20   0 3179116  66808  26212 S   0.0  0.4   0:01.44 lb-server

Here, VIRT shows the total amount of the virtual memory used in KiB. This value can grow very large (beyond the available physical memory and even SWAP space) and is usually not of concern. RES shows the amount of physical memory used in KiB, excluding memory that is swapped to the system swap space or memory that is written back into memory-mapped files.

The generic tools of the operating systems cannot provide a breakdown of the resident memory into buffer pool resident memory, and the remainder used by the database engine. The buffer pool is gradually increased until the configured limit, so it is not sufficient to simply subtract the buffer pool size from the total resident memory. The BloxPagerStat tool can be used to determine how much of the resident memory of lb-server is used by the buffer pool. The portion will not exceed LB_MEM, but could be significantly smaller depending on the size of the database. The remaining resident memory is what the database uses to keep the schema and evaluation infrastructure. The value in bytes can be obtained using:

$ BloxPagerStat -property residentMemory
residentMemory=959479808

4.5. Aborting Transactions

Long-running transactions can be aborted. To do so, first obtain the request identifier using lb status --active workspacename for an active transaction or via lb status --all workspacename for running or queued transactions. To abort the transaction with a request identifier of e.g., 10000000004, issue lb aborttransaction workspacename 10000000004. Aborts are performed asynchronously, that is the aborttransaction command will return immediately even before the transaction is aborted. The return message of the original request will indicate whether the transaction was successfully aborted or had been committed already.

Note that simply issuing CTRL-C on a lb addblock, lb exec or similar, does not abort the transaction but only abort the lb client.

Chapter 5. Backup & Recovery

5.1. Exporting Databases using Hotcopy

LogicBlox supports hotcopy for making copies of the database that are consistent snapshots. With hotcopy the database can be copied while transactions are being processed, potentially modifying the database state. The state of the database copy will be identical to the source database at the moment when the hotcopy process started, even though the copying is taking place in parallel to concurrent write transactions.

Note: before LogicBlox 4.3.6.1 write transactions on the source database would be blocked during hotcopy operations due to an oversight in the implementation.

Similar to other database systems, it is not possible to naively do a file system copy of the database while the database is processing transactions. File system copies are not atomic operations, so the resulting database would not be a consistent snapshot, and likely to be unusable (corrupted).

To hotcopy a database to a backup location use lb export-workspace. For example, to export database test to the local directory /mnt/backup/copy_of_test use:

$ lb export-workspace test /mnt/backup/copy_of_test

The database is copied using a socket connection to the database server, so the database server process does not need permission to write to the target location.

The hotcopy operation can also be executed from a different machine. If the lb-server is running on a machine serv1, then we can hotcopy a database from serv1 directly to the filesystem of another machine, for example remote1. Here, the LogicBlox database software needs to be installed on machine remote1 as well and remote1 needs to be able to communicate with the lb-server running on serv1. To verify that communication is possible, use

user@remote1$ lb --hostname serv1 status

on remote1. If the lb-server on serv1 listens on a non-standard port, the --port option can be used. It might be necessary to use the full-qualified DNS name for serv1 or its IP address. For example, to create a local copy of the database test on serv1 into the directory /mnt/backup/copy_of_testat the machine remote1, use the following command on remote1:

 user@remote1$ lb --hostname serv1 export-workspace test /mnt/backup/copy_of_test

5.2. Importing Databases

To restore a database that had been exported, use lb import-workspace. For example, to import the database located in the directory /mnt/backup/copy_of_test into the lb-server as workspace restored use:

$ lb import-workspace restored /mnt/backup/copy_of_test

The import-workspace operation requires the lb-server process to have read permission to the source database files. Due to the direct usage of the file system by the server, importing databases from a remote server is currently not supported.

LogicBlox databases are stored in the workspaces directory the same way as they are stored after export, albeit under a different directory name. It is thus possible, though not encouraged, to simply copy an exported into the lb-server workspaces directory, but only when lb-server is not running. This could be exploited to directly import a database stored on a remote server without creating a local copy first. Before copying into the lb-server workspaces directory, obtain the directory name where the database will be stored via lb filepath test. Then, create this directory, and copy the database files into it. Once lb-server is started database test will be available.

5.3. Backup Strategies

Cloud Deployments

When the LogicBlox database is deployed on cloud infrastructure such as AWS, EBS snapshots of the device containing the workspaces are a good method for making backups. The snapshotting facility of cloud block devices is incremental, so taking snapshots regularly should be efficient. Note that fully initializing a volume from a snapshot can take many hours though, so if an application is restored to a snapshot, then initially the performance of the appliction can be significantly worse.

As long as the snapshotting facitility used is an atomic operation that creates a consistent snapshot of the file system, then the database can be actively processing transactions when the snapshot procedure is started.

If a striped RAID configuration is used, then it is not possible to obtain a consistent snapshot by snapshotting the striped EBS volumes separately.

Alternative to storing databases on EBS volumes, databases can also be stored on local instance storage. Because instance storage can easily be lost, it is best to combine this with replication.

Non-Cloud Deployments

For large databases, it may not be feasible to use full copies of the database as a backup solution. However, if the database is relatively small, then hotcopy can be used for backing up workspaces. LogicBlox currently does not support incrementally updating copies of the database to a new version.

If the filesystem supports taking atomic snapshots (eg., btrfs) or if the device-layer supports snapshotting in cooperation with the filesystem (eg., lvm +ext3/ext4), then it is also possible to take snapshots of the partition containing the LB_DEPLOYMENT_HOME/workspaces directory. Because of the way LogicBlox organizes data and the commit process, it is safe to take atomic snapshots of the filesystem for backup at any time even when write-transactions are concurrently processed. Note that simply copying the workspace folders is not an atomic operation and can easily create a corrupted workspace.

Tagging Database Versions for Point-in-time Recovery

LogicBlox supports tagging database revisions, which can be used as form of point-in-time recovery. If a database has multiple branches, then all branches have to be tagged separately. Until deleted, the tagged versions will use disk space in the database files. Depending on the application and update pattern, this could grow the database fairly quickly.

Backup vs Replication

Replication (discussed in the next section) is in principle the best method for keeping an up-to-date copy of the database available for recovery or failover. Combined with tagging, this also supports point-in-time recovery. However, if a deployment only uses replication as a recovery strategy, then only a single physical copy of the database is kept. If somehow the primary as well as the secondary databases get corrupted by an operation, then recovery to an earlier point-in-time is not possible and no recovery option is available at all. We recommend to combine replication with a backup strategy that does keep strictly separate physical copies of the database.

5.4. Replication

LogicBlox supports online replication, where transactions committed to a workspace on a primary server are automatically copied to a read-only workspace on one or more secondary servers. The replicated workspaces can answer read-only queries. If the primary server fails, one of the secondary servers can be “promoted” to the primary role.

5.4.1. Overview

This section gives a short tour of the replication facilities. In the examples below, we assume that there are two machines with host names primary and secondary, both running lb-server. The machine secondary must be able to connect to TCP port 5518 on primary. We use the prefix primary$ to denote commands executed on the primary server, and secondary$ for commands on the secondary server.

Note

You can try out replication on a single machine by running multiple lb-server instances. For example, the following starts an instance that listens on port 2000 and stores workspaces underneath /tmp/secondary:

$ lb-server --logfile /dev/stderr -p 2000 --adminPort 2001 --workspaceFolder /tmp/secondary

You can connect to this instance using lb’s --port flag, e.g. lb --port 2000 print test foo.

We start by creating a simple workspace named test on the primary:

primary$ lb create test
primary$ lb addblock test 'foo(x) -> int(x).'
added block 'block_1Z2N9P2J'
primary$ lb exec test '+foo(x) <- int:range(1, 10, 1, x).'

We can now tell the secondary server to start replicating workspace test:

secondary$ lb start-mirror test primary

When this command returns, the secondary will have a complete copy of the latest version of the workspace.

Note

With LogicBlox replication, the secondary servers connect to the primary, not the other way around. This is to make configuration easier: we do not need to change the configuration of the primary as secondaries come and go. If the primary were responsible for connecting to the secondaries, it would always need to have an up-to-date list of secondaries.

However, once the connection is established, it is the primary that pushes new versions of the workspace to the secondaries; secondaries do not need to poll continuously for updates.

The secondary cannot be modified, but it can answer read-only queries, e.g.

secondary$ lb print test foo
1
…
10

Note that lb query needs the flag --readonly:

secondary$ lb query --readonly test '_[x] = x * x <- foo(x).'
/--------------- _ ---------------\
1  1
2  4
…
10 100
\--------------- _ ---------------/

After replication has started, the secondaries automatically receive new versions of the workspace from the primary during or after every transaction. For example:

primary$ lb exec test '+foo(11).'

secondary$ lb query --readonly test '_[] = x <- agg<<x = max(y)>> foo(y).'
11

You can use the command lb info to get information about the status of a workspace. For instance, on the primary, it will show something like this:

primary$ lb info test
commit_version: 4
is_mirror: false
active_mirror_count: 1
…

The line commit_version: 4 shows that there have been four transactions in this workspace; is_mirror: false denotes that this is a primary workspace; and active_mirror_count: 1 indicates that one secondary server is currently replicating this workspace. Here is the output on the secondary:

secondary$ lb info test
commit_version: 4
is_mirror: true
is_mirroring: true
pages_received: 154
versions_received: 4
…

Here, the commit version is also 4, meaning that the secondary is up to date. The line is_mirroring: true shows that the secondary is currently connected to the primary. There are also some connection statistics: the secondary has received 4 versions and 154 database pages (each of which is 32 KiB).

You can temporarily stop replication using lb stop-mirror:

secondary$ lb stop-mirror test

secondary$ lb info test | grep is_mirroring
is_mirroring: false

In this state, the secondary does not receive updates from the primary. It can still answer read-only queries. You can resume replication in the same way you started the initial replication, using lb start-mirror:

secondary$ lb start-mirror test primary

Resumption is incremental: the primary will only send the delta between the most recent versions on the secondary and primary.

The final operation on a secondary workspace is to promote it, turning it into a regular (primary) workspace that can be modified and that can be replicated. This is done as follows:

secondary$ lb promote-mirror test

secondary$ lb info test | grep is_mirror
is_mirror: false

This operation is irreversible: it is not possible to demote a primary.

5.4.2. Synchronous and Asynchronous Replication

LogicBlox support synchronous as well as asynchronous replication. For synchronous replication, a transaction using diskcommit will not complete until the transaction is durably replicated to all connected mirrors.

replication_mode commit_mode
diskcommit softcommit
synchronous Transactions do not complete until they have been written to the primary disk and have been replicated to mirrors. Transactions complete before they have been written to the primary disk. Replication also might still be in progress. This configuration is currently not useful.
asynchronous Transactions do not complete until they have been written to the primary disk. Replication might still be in progress. Transactions complete before they have been written to the primary disk. Replication also might still be in progress.

In addition to the configuration on the primary server, the commit_mode on the secondary server can be used to refine the durability requirements. The commit_mode on the secondary determines when replication is considered complete on a mirror. If the setting is softcommit, then replication is considered complete before the transaction is durably written to disk on the secondary. If the setting is diskcommit, then the changes are written to disk before completion.

Synchronous replication of transactions only starts after an initial full replication of the database has completed. The purposes of this is to not block write transactions for the initial full replication, which could take many hours if the database is large.

5.4.3. Monitoring and Database Information

Progress tracking

Commit version and commit timestamp

5.4.4. Limitations

Replication currently has a number of limitations:

  • There is no automatic failover yet: secondaries are not automatically promoted if the master fails, and there is no election mechanism to select a secondary.

  • Secondaries do not yet automatically reconnect to the primary if the primary is restarted or the connection is interrupted. As a workaround, you can call lb start-mirror periodically from a cron job or similar; this command does nothing if replication is already active.

  • Secondaries are read-only. All changes must be performed on the primary workspaces.

  • While resuming replication is incremental (only the changes, rather than the full workspace, is sent), sending of the initial version is not incremental. Thus, if, during the creation of a new secondary, the initial lb start-mirror command is interrupted, a subsequent call to lb start-mirror must start all over.

  • The TCP connection between the primary and the secondaries is unencrypted. Thus, if they need to communicate across untrusted network links, it is important to use IPsec or a similar technology to establish a secure channel between the machines.

Chapter 6. ACID Properties of Lb-server

This section explains the guarantees the lb-server database provides. We also highlight requirements for filesystem and hard-drives as well as necessary configuration options to achieve the described properties.

6.1. Atomicity

Transactions have an all or nothing property. Changes that are performed within a transaction are either applied and visible to future (or concurrent) transactions in their entirety or not at all. In particular, if a transaction aborts (due to a constraint violation), then non of the changes that were made within this transaction are visible to any other transaction or user of the database. In other words, at any time the observed state of a workspace is the result of a recently successfully committed transaction.

6.2. Consistency

The LogiQL language allows to define a wide variety of rules and constraints that describe how data in various predicates relates to each other. Rules include standard IDB rules, eg. that define returns[x] <- sales[x] - costs[x] and trigger or delta-rules, eg., +employee(x) <- +new_hire(x) to name the most common. Constraints include basic functional constraints, eg. in a functional predicate foo[x]=y there is at most one y for each x, as well as constraints that are explicitly formulated using "right-arrow" rules.

The lb-server ensures that a transaction only successfully commits if all rules and constraints from the active workspace state are satisfied. That means, at the end and the start of any transaction, the database is always in a consistent state. Transactions that would violate integrity constraints are aborted with an integrity violation message. Note that integrity violations are the only cause for aborted transactions. The unique concurrency model of the LogicBlox database never aborts otherwise valid transactions. This is for example, not the case for conventional optimistic concurrency models where transactions can be aborted due to concurrent behavior.

6.3. Isolation

The LogicBlox database guarantees the isolation level SERIALIZABLE. Currently, we support concurrently executing read-only transactions while write-transactions are serialized and executed one after another. Future releases will allow concurrent write-transactions while providing the same isolation level of SERIALIZABLE. In practice this means that application development is simplified: Even though transactions are processed concurrently the state changes between one transaction to the next can fully be explained by a serial execution. Thus, application developers need not to be concerned with potential anomalies that arise due to concurrency.

For example, a long-running read-only transaction T of, say 20 seconds, obtains a consistent state of the database when it is started. Now, during the evaluation of this transactions, other write transactions may modify the database state and even commit new versions--however, these modifications are not visible to T.

6.4. Durability

Transactions to the lb-server can be configured with softcommit or diskcommit. As described in Section 2.6 the two commit-modes differ only in the time when the lb-server reports to the client that the transaction has been finished. While for diskcommit we report SUCCESS only after the transaction has been written to disk, we already notify the client of a successful transaction once it is committed to the memory representation for softcommit transactions. In case of power failures, or kill -9, or similar events it is thus possibly that softcommited transactions are lost.

Note that the transaction throughput is not effected by the diskcommit vs. softcommit setting. We thus recommend using softcommit only for transactions that do not have strict durability requirements.

Hardware and OS Requirements for Durability

The durability mechanism of the LogicBlox database is based on versioned data structures and requires that once fsync has been called and has returned successfully, all outstanding requests to the file have been written to disk to non-volatile memory. It is important that the system administrator verifies that the software stack comprising hard-drive-firmware and kernel software guarantees that these requirements on fsync are satisfied. Many hard drives are for example "lying" to the operating system by using write-caches. Thus, even though the operating system assumes that data has been written to the non-volatile disk memory, the disk itself holds the data in volatile memory.

On Linux, hdparm -I /dev/disk_device can be used to check whether a write-cache is enabled for a certain device. Use hdparm -W to disable disk write-caches. It is also important to verify that all layers on top of the disk support fsync properly. These layers might include raid configurations (eg, hardware as well as software raids), logical volume managers (eg, LVM) and the used file system.

Note

LogicBlox does perform write-ahead logging, but uses persistent data structures to provide its durability guarantees. It is thus not necessary to configure any write-ahead logging related parameters.

LogicBlox is resilient against partial writes, that is the database data cannot become corrupted if any write operation is aborted at any point in time. We require that the underlying mechanism does not reorder writes to a consecutively region of 144 bytes.

Chapter 7. Web Service Configuration

7.1. Introduction

ServiceBlox is a framework for developing and hosting services backed by the LogicBlox database. ServiceBlox services are the interfaces to the LogicBlox database from other application components: user interfaces, data integration tools, or 3rd party applications. For example, a typical service might provide data for a UI component that displays data in charts or tables, or receives input from a web form and makes modifications to the database accordingly. This chapter introduces the implementation and configuration of ServiceBlox services.

ServiceBlox is an extensible framework that comes with a few different types of services that should meet the needs of most applications:

  • Protocol buffer (protobuf) services are HTTP services that are invoked using an HTTP POST request. The request contains a binary protobuf or textual JSON message. The service returns the protobuf or JSON result of the invocation as an HTTP response message. This type of service is similar to other service frameworks that resemble remote procedure calls, such as JSON-based services used in AJAX applications, SOAP and XML-RPC. In ServiceBlox, the schema of the request and responses messages are precisely specified by a protobuf protocol. Optionally, messages can be encoded as a JSON string, to support access from web browsers. The services can be accessed by any HTTP client, including browsers, curl or really any application that understands the HTTP protocol and is able to encode and decode protocol buffers or JSON messages.

  • Tabular data exchange (TDX) services are HTTP services that can be accessed by GET, POST, and PUT requests. TDX is the core service for getting data in and out of the LogicBlox database, in large volume. TDX uses delimited files as the input/output data format. Data is retrieved from the database using GET requests. Using POST requests, data in the database can be updated, and using PUT requests data can be replaced. TDX services are typically used for integration purposes, for example for importing large volumes of sales data or for exporting large volumes of forecast data.

  • Global protobuf services are protobuf services that are implemented by distributing incoming requests to services hosted on other LogicBlox workspaces. The responses from the individual services are merged into a single response of the global service. Global services are useful when data needed for a service is stored in multiple, partitioned workspaces.

  • Proxy services act as a simple proxy for a service hosted on a different machine. Proxy services can be used to require authentication on top of existing unauthenticated services, or can be used to provide access to a distributed service-oriented system on a single host.

  • Custom services are supported as plugins to the ServiceBlox service container. Custom services must provide implementation to a set of ServiceBlox Java interfaces. Custom services have a great deal of flexibility, and are used internally to implement Tabular Data Exchange, Global, and Proxy services. However, they should be used very sparingly as they do complicate the deployment of your LogicBlox-based application. If you find yourself needing a custom service, we recommend that you contact LogicBlox support personnel first to explore all appropriate options before proceeding.

ServiceBlox supports services request/response via HTTP, where the service message as well as the payload are part of a HTTP message. Alternatively, for longer-running services, an asynchronous queue can be used in place of HTTP; for services with large payloads (e.g. importing/exporting a large delimited file), AWS S3 objects can be used to transfer the payload. Support for HTTP, as well as queues and S3, are built into ServiceBlox, and is a matter of configuration to select the right mechanism for a given service.

ServiceBlox supports different authentication methods, where the different methods have benefits when used from the relatively hostile environment of a browser versus non-browser applications running in the controlled environment of a machine.

7.2. General Configuration

7.2.1. Anatomy of Service

At the most basic level, a 'service' in ServiceBlox is made up of the following parts:

  • A URI or URI prefix - This is the part of the URI that a user would enter in their browser after the domain. In other words the /example in the URI http://www.logicblox.com/example.

  • An HTTP method - Accessing /bars with an HTTP GET is a different service than accessing /bars with an HTTP POST. NOTE: Many service handlers will set defaults for which methods are allowed if you do not specify them.

  • A Handler - This is really just identifying the type of the service, by identifying the type of handler that will service requests. For more, information on handlers see the section on Handlers. Typically the user will just specify the name of a built-in handler (e.g. protobuf, delimited-file, etc.).

There are many other configurable options for a service such as the authentication realm, workspace, handler specific options, etc. that will be covered in the sections related to those features.

7.2.2. Service Configuration in Workspaces

Most services are configured in the workspace with which the service will exchange data. By default, when ServiceBlox starts it scans all available workspaces and handles all services for which it finds a specification. This involves opening and executing queries in multiple workspaces, which can be time-consuming. It is possible to control workspace scanning in two important ways. First, the server uses a regular expression to match the names of workspaces to be scanned. This is controlled by the scan_workspaces configuration option, which defaults to .*, essentially matching any workspace. Second, setting the scan_workspaces_on_startup option to false instructs the server to avoid scanning workspaces altogether during startup. The lb web-server command line can then be used to dynamically load and unload services specified in particular workspaces and branches, without restarting ServiceBlox. For example, lb web-server load-services -w myWorkspace scans myWorkspace and loads services from the specifications in this workspace.

Detailed documentation of the schema used to configure a service in a workspace will be provided for different kind of services in the respective section, but in all cases the specification of a service in LogiQL is a rule which declares the existence of an entity whose type is a subtype of lb:web:config:service, sets some attributes of this entity and associates this service with a prefix and HTTP method.

    service_by_prefix_and_method["/time/", "POST"] = x,
    default_protobuf_service(x) {
      disabled_status[]=503,
      protobuf_protocol[] = "time",
      protobuf_request_message[] = "Request",
      protobuf_response_message[] = "Response"
    }.

One setting common to all services is the ability to set a disabled status for this service when it hosts this service and it is put in disabled mode. This will instruct ServiceBlox to return the specified status for this service when it implements this service. This status can then be changed at runtime to either enable the service or switch the disabled status through the lb web-server disable-services and lb web-server enable-services commands.

7.2.3. Static Services / Workspaces

A static service is a service that is hosted by ServiceBlox that is not hosted in a real LogicBlox workspace, but instead is configured via JSON. We recommend defining your services in LogiQL whenever possible.

You can set up a static workspace and services by specifying json files to load in your configuration file.

[workspace:myStaticWorkspace]
static_service = /path/to/static_service.json
some_other_static_service = /path/to/other_static_service.json

The JSON would look like the following:

/path/to/static_service.json

{
    "handler" : "my-handler",
    "prefix"  : "/my-static-service",
    "http_method": "POST",
    "request_protocol"  : "myprotocol",
    "request_message"   : "Request",
    "response_protocol" : "myprotocol",
    "response_message"  : "Response"
}

7.2.4. Service Groups

ServiceBlox supports service groups to provide the flexibility to host different versions of a service in a single URI and to facilitate authentication enforcement. It is possible, for example, to provide authenticated and non-authenticated versions of a service in the same URI, but at different ports. For example, suppose we want to expose the time service in authenticated and non-authenticated versions in the same URI. These would be the steps:

  1. Services can declare the group to which they belong. A group is a simple string that identifies services. By default, services belong to the special group lb-web:no-group. Here, we declare a non-authenticated version of the time service at the /time URI and public group. Also, we declare an authenticated version in the same URI but private group.

    service_by_group["/time", "public"] = x,
    default_protobuf_service(x) {
    protobuf_protocol[] = "time",
    ...
    }.
    service_by_group["/time", "private"] = x,
    default_protobuf_service(x) {
    protobuf_protocol[] = "time",
    ...
    auth_realm[] = "realm_name"
    }.  
  2. Endpoints define how the web-server communicates with clients. There are, for example, TCP endpoints that use TCP sockets, and queue endpoints that use SQS or RabbitMQ queues. They are declared in endpoint sections of the lb-web-server.config. Endpoints can declare the groups that they host and can enforce that all services in those groups must be authenticated. In our example, we create endpoint configurations to host the public and private groups: the clear TCP endpoint hosts the public group; the ssl TCP endpoint hosts the private group. Furthermore, the ssl endpoint declares that it requires authentication, which will make the ServiceBlox server verify that all services that declare to be in the private group indeed have authentication support.

    [tcp:clear]
    port = 8080
    groups = public
    
    [tcp:ssl]
    ssl = true
    port = 8443
    groups = private
    requires_authentication = true
              

With this configuration in place, clients accessing the /time URI via TCP port 8080 will be directed to the non-authenticated version of the service. A client accessing /time via TCP port 8443 will be directed to the authenticated version instead.

To support backwards compatibility, service and endpoint groups are optional. If a service does not declare a group, it automatically belongs to the special group lb-web:no-group. Similarly, if an endpoint does not declare the groups it hosts, it automatically hosts only the group lb-web:no-group.

This is a summary of how the ServiceBlox service interprets the group configuration of services and endpoints.

  • A service can belong to one or more groups; endpoints support at least one group. The special lb-web:no-group group is assigned to services and endpoints that do not declare groups. Endpoints can explicitly list lb-web:no-group next to other groups.

  • For each endpoint, services prefixes must be unambiguous. That is, in the set of services hosted by an endpoint (taken from all groups it hosts), two services cannot declare the same prefix. The ServiceBlox server issues a warning and ignores all services that conflict.

  • If a service belongs to a group that is not supported by any endpoint (including the lb-web:no-group group), there's a warning and the service is not hosted.

  • If a service without authentication belongs to a group hosted by an endpoint that requires authentication, there's a warning and the service is not hosted in that particular endpoint (but it may be hosted on other endpoints).

7.2.5. Admission Control

By default, ServiceBlox services requests as soon as they are submitted, and will issue as many concurrent requests as there are worker threads to service those requests. For a mix of read-only and update services, this can sometimes result in poor performance, depending on the type of concurrency control used by the workspace, and the transaction times resulting from the services. In some cases, it is desirable to have ServiceBlox order the execution of the services, for example, such that read-only requests are run concurrently, while update requests are run exclusively. Besides resulting in better performance from avoiding transaction aborts, this can also result in performance gains from disabling concurrency control on the workspace.

This can be achieved by configuring services to use specific admission queues. All requests to services that belong to a queue are handled according to the queue policy. Admission queues are declared in lb-web-server.config. Configuring a service to use an admission queue is done by setting the queue name on the AdmissionQueue property to the service_parameter option when configuring the service. For instance, the following service configuration shows two services using the admission queue named my_exclusive_writes, with one being read-only while the other can update the state of the workspace.

block(`service_config) {

  alias_all(`lb-web:config:service),
  alias_all(`lb-web:config:service_abbr),
  alias_all(`lb-web:config:protobuf),
  alias_all(`lb-web:config:protobuf_abbr),

  clauses(`{

    service_by_prefix["/readonly-service"] = x,
    service_parameter[x,"AdmissionQueue"] = "my_exclusive_writes",
    default_protobuf_service(x) {
      readonly(),
      protobuf_protocol[] = "time",
      protobuf_encoding[] = "binary",
      protobuf_request_message[] = "Request",
      protobuf_response_message[] = "Response"
    }.

    service_by_prefix["/update-service"] = x,
    service_parameter[x,"AdmissionQueue"] = "my_exclusive_writes",
    default_protobuf_service(x) {
      protobuf_protocol[] = "settime",
      protobuf_request_message[] = "Request",
      protobuf_response_message[] = "Response"
    }.

  })
} <-- . 
       

The policy that governs the admission queue behavior is determined by the HandlerExecutor implementation declared for the queue. ServiceBlox currently provides two implementations: QueuedHandlerExecutor implements an exclusive-writes policy, and SerializerHandlerExecutor implements a serializer policy. The following excerpt from an lb-web-server.config declares two queues using these implementations.

# declaration of 'my_exclusive_writes' queue with exclusive-writes policy
[admission-control:my_exclusive_writes]
classname = com.logicblox.bloxweb.QueuedHandlerExecutor

# declaration of 'my_serializer' queue with serializer policy
[admission-control:my_serializer]
classname = com.logicblox.bloxweb.SerializerHandlerExecutor
       

The serializer policy simply serializes the execution of requests, such that they execute one-by-one, in the order they are received. It also accepts a sleep parameter that causes the queue to wait a number of milliseconds between request executions.

The exclusive-writes policy allows read-only requests to execute concurrently but gives write request an exclusive lock. In detail:

  • Requests are submitted for execution in the order received.

  • Requests for read-only services can run concurrently.

  • Requests for update services wait until all currently running requests are complete.

  • Requests for read-only services are only submitted for execution after any currently running write requests are complete.

7.2.6. Service Configuration Reference

This section presents a reference to the configuration options of the service interface. Each service type has extensive options that are covered in the corresponding sections of the Core Reference Manual.

Service types
delim_service lb-web:config:delim Relational data exchange service hosted by the workspace that contains the configuration of the service.
default_protobuf_service lb-web:config:protobuf Default protobuf services hosted by the workspace that contains the configuration of the service.
global_protobuf_service lb-web:config:global_protobuf Global protobuf service.
exact_proxy lb-web:config:proxy Exact proxy service.
transparent_proxy lb-web:config:proxy Transparent proxy service.

Configuration options applicable to all services
service_prefix string Required Path of the URL where the ServiceBlox service container will make a service available
service_description string Optional Human-readable description of the service. Currently displayed only in the meta-service.
auth_realm string Optional Name of the realm used for authenticating users of this service (see the authentication section for further details)
custom_handler string Optional Name of a custom handler for this service. The handler needs to be configured in the configuration file lb-web-server.config as a section [handler:name].
disabled_status uint[32] Optional Set the ServiceBlox service to be disabled. The value defines the status code that will be returned to clients that try to access the disabled service.
service_parameter string, string Optional Key/value pairs that are passed as parameters to service handlers.
lazy Optional Declares that the service should only be initialized when it receives the first request. By default services are initialized as soon as they are loaded from the workspace. Initialization may be expensive if it requires additional transactions to lookup data, for example, so making the services lazy makes server startup potentially faster by delaying the initialization.
sync_mode string Optional One of: sync, async, strict_sync or strict_async. If none specified, the sync mode will be inherited from the handler's sync_mode. For more details, see the specific section about asynchronous service configuration.
group string Optional The group to which the service belongs. The default is the unnamed group.
ignore_authentication_requirement Optional Will not require authentication even if this service is placed into a group that requires authentication.
service_operation string Optional The operation for this service for use in role-based authorization. If an authenticated user does not have a Role that contains a Permission for this Operation, then they will be denied access. See the authorization module of the credentials service or the reference manual for more information.

Configuration options on workspace services
inactive_block_name, inactive_after_fixpoint_block_name string Optional Name of an inactive block to be executed when serving a request to this service. There are variants to execute the block before or after fixpoint.
readonly Optional Marks a service as read-only, which means that database transactions executed by this service will use a read only setting.
service_host_workspace string Optional The name of the workspace that is the recipient of the service invocation. By default the host workspace is the workspace in which the service is declared. This allows services to be declared by one workspace but executed by another.

7.2.7. Config Files

The ServiceBlox server and client can be configured statically using configuration files. The format of ServiceBlox configuration files follows the HierarchicalINIConfiguration of the Apache Commons Configuration project. In essence, they are INI files with named sections and a global section (the variables defined before the first section is defined). For example, the following simple config file defines a global variable and a section with two local variables:

global_variable = some_value

[section_type:section_name]
section_variable_1 = some_value
section_variable_2 = some_value  

At startup, the initial configuration is created by composing the available configuration files. The ServiceBlox server first loads the default lb-web-server.config file that is included in the LogicBlox distribution. Next, it loads the lb-web-server.config file found in $LB_DEPLOYMENT/config, if it exists. Finally, it loads any configuration file passed as a parameter through the command line.

When a file is loaded, it overrides existing config variables, so it is possible to refine the default configuration. Composition is at the level of variables, but sections are used to identify variables. For example, if the following configuration file is loaded after the previous example, both global_variable and section_variable_1 will be refined, but section_variable_2 will stay the same. Note that the configuration loading process for lb web-client is the same, but using lb-web-client.config instead.

global_variable = new_value

[section_type:section_name]
section_variable_1 = new_value  

ServiceBlox supports some variables to be used in config files. A variable reference is in the form $(VARNAME). The following variables are currently supported and are substituted when a config file is loaded:

CONFIG_DIR The fully qualified file path for the directory containing the config file being loaded.
LB_DEPLOYMENT_HOME The contents of the environment variable.
LOGICBLOX_HOME The contents of the environment variable.
HOME The contents of the environment variable.

The lb web-server tool has commands that allow certain aspects of the configuration to be modified dynamically, that is, after the ServiceBlox server has been started. The install-config command takes a config file in the ServiceBlox config format and composes it with the currently running configuration.

$ lb web-server load --config my.config  

Currently it is only possible to load new handlers and static workspaces (see the Server configuration section for details on the section types that are interpreted by the server). The install-handler command is a sub-set of install-config in that it will only load new handlers and will ignore other sections.

One problem with using these commands is that some sections may need to reference files, such as the class of a custom handler. In order for ServiceBlox to locate these files, they need to have a fully qualified path, which is not portable. ServiceBlox solves this problem with the install-jar command.

$ lb web-server load --jar CustomHandlerRelease.jar 

The jar file contains all files that may need to be referenced. Furthermore, it contains a lb-web-server.config file in its top directory. The ServiceBlox server will open the jar and load this config file. The advantage of this approach is that all paths in the config file are resolved relative to the jar file, so it becomes simple to reference custom handlers, for example, in a portable manner.

7.2.7.1. Server

When the web server first starts up, it reads the internal configuration file which is where all the default values are set. The file also contains comments and commented out options for all available options and is therefore an ideal place to look for most of your configuration questions. We include the text of the file here for your convenience.

############################
#   GLOBAL CONFIGURATION   #
############################

#
#  SERVER STARTUP
#

# Whether lb services should start lb web-server or not
enabled = true

# How long lb services should wait for the web server to start before timing out, in seconds
startup_timeout = 60

# Arguments to the JVM instance
jvm_args = -Xmx3200m -Xss2048k

# The directory where the web server will write memory dumps on Out Of Memory Errors
# Set to empty to disable dumps
jvm_dump_dir = $LB_DEPLOYMENT_HOME/logs/current

# Whether the web server should scan workspaces for services at startup
scan_workspaces_on_startup = true

# A regular expression to match workspaces to scan at startup; only used if
# scan_workspaces_on_startup is true, and has no effect on static workspaces
#   Example: delim-.*|protobuf.* scans only workspaces starting with "delim-"
#   or "protobuf", besides installing static workspaces.
scan_workspaces = .*

# Number of threads to use when scanning workspaces for services
service_scanning_threads = 1

# A list of directories to scan for handler jars. Equivalent to calling lb web-server load -j on
# each jar with the exception that it is not allowed to have configuration conflicts between
# options in the lb-web-server.config files stored inside the jars.
handler_dirs = lib/java/handlers, ../measure-service/lib/java/handlers



#
#  LOGGING
#

# Directory for access logs. If this property is empty, the access log will be printed to stderr.
logdir_access = $(LB_DEPLOYMENT_HOME)/logs/current

# Directory for general log files
logdir = $(LB_DEPLOYMENT_HOME)/logs/current

#
#  DEBUG LOGGING
#
#  The following should not be used in production as they affect performance.
#

# Print all protobuf request and response messages to the log
log_messages = false

# Print all HTTP messages in protobuf format to the log
log_http = false

# Print all connectblox requests to the log
log_connectblox = false

# Log very detailed info when an exception is detected
log_exception_details = false

# Enable or disable logging of rules generated for TDX services
log_delim_file_rules = false



#
#  DEBUG SUPPORT
#

# Set the web server debug flag to log all events (very noisy).
debug = false

# Do not delete temporary files created by TDX.
debug_keep_tdx_files = false



#
#  AUTHENTICATION
#

# Set to true to force authentication to use secure cookies
secure_cookies = false

# Set to true to force authentication to use HttpOnly cookies
httponly_cookies = false

# Session ids are stored here so that user sessions survive restarts of the web server
authentication_cache = $(LB_DEPLOYMENT_HOME)/authentication_cache

# Path of directory where encryption keys are kept
keydir = $(HOME)/.s3lib-keys



#
#  SERVER
#

# Set the maximum size of a form post, to protect against DOS attacks from large forms (4MB)
max_form_content_bytes = 4194304

# Number of threads to use to handle requests
http_server_threads = 250

# Whether or not a client disconnection should abort the transaction it spawned
# Currently only works for protobuf services.
tcp_abort_on_disconnect = true



#
#  QUEUES
#

# Should all messages submitted to the message queues be handled locally (false), or should the web
# server assume that there are other instances handling requests that are not support by this
# web server instance (true)
mq_distributed = false

# Set the global SQS queue to use for queue transports
sqs_endpoint = sqs.us-east-1.amazonaws.com



#
#  HTTP CLIENT
#
# These are configuration options for the HttpClient used internally by the web server when
# connecting to other services (e.g. when acting as a proxy).

# Size of thread pool for HttpClient (shared among all clients)
tcp_client_threads = 100

# Max time the whole exchange can take (in ms)
#tcp_timeout = 320000

# Max time to wait when establishing a connection to the server (in ms)
#tcp_connect_timeout = 75000

# Max time in a connection without processing anything (in ms).
# Processing is defined to be "parsing or generating".
#tcp_idle_timeout = 20000

# Max number of connections httpclient can hold to an address
#tcp_max_connections_per_address = 100



############################
#  CONFIGURATION SECTIONS  #
############################

#
#  S3 CONFIGURATION
#

# The s3:default configuration is currently used for all S3 access from services. It is a separate
# configuration section because we might later allow multiple S3 configurations.
[s3:default]
# Concurrency of general management of S3 downloads.
max_concurrent = 50

# Number of concurrent uploads/downloads for S3
s3_max_concurrent = 10

# Size of chunks for multipart uploads to S3
chunk_size = 5242880

# iam_role = default
# access_key = ...
# secret_key = ...



#
#  TCP ENDPOINTS
#

[tcp:public]
port = 8080
address = 0.0.0.0
# comma-separated list of groups allowed in this endpoint; if omitted, defaults to lb:web:no_group
groups = lb:web:public, lb:web:no-group
# Attempt to use an inherited channel, usually created by systemd or inetd
#inherited = true

# Set setMaxIdleTime on SocketConnector, which is similar to setting
# SO_TIMEOUT on a socket. 0 means no timeout.
tcp_server_max_idle_time = 200000

[tcp:internal]
port = 55183
address = 0.0.0.0
groups = lb:web:internal, lb:web:no-group

# Set setMaxIdleTime on SocketConnector, which is similar to setting
# SO_TIMEOUT on a socket. 0 means no timeout.
tcp_server_max_idle_time = 0

# [tcp:public-ssl]
# ssl = true
# port = 8443
# address = 0.0.0.0
# groups = authenticated, public
# requires_authentication = true
# keystore_path = $(LB_DEPLOYMENT_HOME)/config/keystore
# truststore_path = $(LB_DEPLOYMENT_HOME)/config/keystore
# keystore_password = password
# truststore_password = password
# keymanager_password = password



#
# SQS ENDPOINTS
#

#[sqs:sample]
#request_queue_name = lb-web-sample-request
#response_queue_name = lb-web-sample-response
#access_key = ...
#secret_key = ...



#
# RABBITMQ ENDPOINTS
#

#[rabbit:test]
#request_queue_name = bloxweb-test-request
#response_queue_name = bloxweb-test-response
#endpoint = amqp://localhost:5673



#
#  HANDLERS
#

[handler:default-protobuf]
classname = com.logicblox.bloxweb.DefaultProtoBufHandler


[handler:delimited-file]
classname = com.logicblox.bloxweb.delim.DelimitedFileHandler
tmpdir = /tmp


[handler:dynamic-delimited-file]
classname = com.logicblox.bloxweb.delim.DynamicDelimitedFileHandler
tmpdir = /tmp


[handler:delimited-file-txn]
classname = com.logicblox.bloxweb.delim.DelimTransactionHandler


[handler:transparent-proxy]
classname = com.logicblox.bloxweb.proxy.TransparentProxyHandler


[handler:exact-proxy]
classname = com.logicblox.bloxweb.proxy.ExactProxyHandler


[handler:global-protobuf]
classname = com.logicblox.bloxweb.GlobalProtoBufHandler


[handler:admin]
classname = com.logicblox.bloxweb.AdminHandler


[handler:login-handler]
classname = com.logicblox.bloxweb.authentication.LoginHandler


# The wait handler just waits. It is only useful for performance testing.
[handler:wait]
classname = com.logicblox.bloxweb.WaitHandler



#
#  EXTENSION HANDLERS (will be extracted in the future)
#

# The bcrypt handler converts a clear text password into a hashed password on requests that update
# passwords; otherwise, it behaves like an normal protobuf service.
[handler:lb:web:bcrypt-credentials]
classname = com.logicblox.bloxweb.authentication.BCryptCredentialsHandler


# Handler to report on the currently authenticated user.
[handler:lb:web:current-user]
classname = com.logicblox.bloxweb.authentication.UserInfoHandler


[handler:lb:web:bcrypt-change-password]
classname = com.logicblox.bloxweb.authentication.ChangePasswordHandler


[handler:lb:web:bcrypt-reset-password]
classname = com.logicblox.bloxweb.authentication.ResetPasswordHandler


[handler:lb:web:bcrypt-confirm-reset-password]
classname = com.logicblox.bloxweb.authentication.ConfirmResetPasswordHandler


[handler:lb:web:send-email]
classname = com.logicblox.bloxweb.email.SendEmailHandler



#
#  STATIC WORKSPACES
#

[workspace:login]
service_login = $(CONFIG_DIR)/login_service_config.json
available = true


#[workspace:some-static-service]
#config = configuration.json
#request_protocol = request_protocol.descriptor
#response_protocol = response_protocol.descriptor



#
#  REALM CONFIGURATIONS
#

[realm-config:default-password]
class = com.logicblox.bloxweb.authentication.PasswordBCryptAuthenticationMechanism
stateful = true
realm_option_session_key_prefix = lb-session-key
realm_option_seconds_to_live = 1800
mechanism_option_credential_service = /admin/credentials
user_cache_size = 100
user_cache_seconds_until_expire = 1800


[realm-config:default-signature]
class = com.logicblox.bloxweb.authentication.SignatureAuthenticationMechanism
stateful = false
realm_option_header_regex = ^x-logicblox-.*$
realm_option_required_headers = Date Digest
realm_option_max_request_time_skew = 900000
mechanism_option_allowed_signature_algorithm = SHA512withRSA
mechanism_option_credential_service = /admin/credentials
user_cache_size = 100
user_cache_seconds_until_expire = 1800


[realm-config:default-saml]
class = com.logicblox.bloxweb.authentication.saml.SAMLMechanism
stateful = true
realm_option_session_key_prefix = lb-session-key
realm_option_seconds_to_live = 1800
user_cache_size = 100
user_cache_seconds_until_expire = 1800
    
7.2.7.1.1. Endpoints

Services can be configured to be hosted on different endpoints. Typically, this means setting different TCP ports, addresses or service groups. See the Service Group section above for a common use case of configuring new endpoints with different service groups. See the Transport Methods section for details on setting up queue endpoints.

By default, the lb web server has two TCP endpoints configured. One, running on port 55183, is for internal services. The credentials services run on this endpoint which is only available within the server's local network. The other endpoint, [tcp:public], is what most standard services are served on.

You can also secure endpoints such that only authenticated users can access them. Declaring that it requires authentication will make the ServiceBlox server verify that all services that use that endpoint have authentication support.

7.2.7.1.2. Handlers

Handlers are the specific Java classes that are used to handle all the different types of service requests. ServiceBlox comes with quite a few handlers already installed such as a login handler, tabular data handler, etc. Typically, handlers do not need to be configured unless you are implementing your own custom handler.

If you are implementing your own custom handler, it is recommended that you place a config file in the root of your handler jar. This config file should just contain the configuration for the new handler. Ex:

        [handler:my-handler]
        classname = com.logicblox.bloxweb.myhandler.MyHandler
        

You can then specify that a particular service should use this handler in your service configuration.

        service_by_prefix["/my-service"] = x,
        protobuf_service(x) {
          custom_handler[] = "my-handler",
          ...
        }.
        

7.2.7.1.3. Realms

Realms are a way of grouping users by authentication policy. You can then restrict services to different authentication realms. ServiceBlox supports several types of authentication policies / realms. For more information on authentication and the different types of authentication policies, please see the Authentication section.

ServiceBlox comes with three default realm types installed: default-password, default-signature and default-saml. Each of these can be configured via a configuration file with things such as session timeout and other more policy specific options. For more information about how to configure these different realms, please see the relevant subsection of Authentication.

7.2.7.2. Client

  # This configuration file documents the default configuration options
  # for the lb web-client command-line tool. Do not modify these
  # settings, instead either:
  #
  # a) Make a configuration file
  #    LB_DEPLOYMENT_HOME/config/bloxweb-client.config that customizes
  #    the configuration. In this file, only the custom overrides need
  #    to be configured. This file is automatically loaded when
  #    executing bloxweb-client
  #
  # b) Use the --config option of the
  #    bloxweb-client commands to specify a bloxweb-client.config file.
  #

  # Path of directory where encryption keys are kept.
  keydir = $(HOME)/.s3lib-keys

  # Name of encryption key to use for s3 uploads. By default not configured.
  # keyname = ...

  batch_max_concurrent = 50

  # Number of concurrent uploads/downloads for S3
  s3_max_concurrent = 10

  # Size of chunks for multipart uploads to S3
  s3_chunk_size = 5242880

  # Set the global SQS queue to use for queue transport
  sqs_endpoint = sqs.us-east-1.amazonaws.com

  # The s3:default configuration is used if no --config option is used
  # on the s3 command used in the batch specification. To change the
  # defaults, specify either the access/secret keys or an IAM role in a
  # custom bloxweb-client.config file.
  [s3:default]
  # iam_role = default
  # access_key = ...
  # secret_key = ...

  [tcp:default]
  # TCP configurations can specify a host and port, which will be used
  # for relative URLs. The default tcp configuration would typically not
  # have such a configuration.
  # host = ...
  # port = ...

  

Environment variables:

  • LB_WEBCLIENT_JVM_ARGS can be used to configure the JVM arguments of all lb web-client invocations. Example:

    export LB_WEBCLIENT_JVM_ARGS="-Xmx200m"

7.3. Asynchronous Services

ServiceBlox allows for asynchronous service calls, so long running transactions can be processed in the background, with no need for an open TCP connection between client and server.

To enable asynchronous requests, sync_mode configuration is used at service level, as per below:

      service_by_prefix["/my-service"] = x,
      protobuf_service(x) {
        custom_handler[] = "my-handler",
        sync_mode[] = "async"
        ...
      }.
      

Or syncmode at handler level:

      [handler:my-handler]
      classname = com.logicblox.bloxweb.myhandler.MyHandler
      syncmode=sync
      

Below is the list of acceptable values:

  • strict_sync: no asynchronous calls will be acceptable.
  • sync: service will allow both synchronous and asynchronous requests, synchronous being the default.
  • async: service will allow both synchronous and asynchronous requests, asynchronous being the default.
  • strict_async: only asynchronous calls will be acceptable.

If the handler is configured as strict_sync or strict_async, the service cannot override this value, otherwise, the service configuration will take precedence over the handler. If no mode is defined at service level, the handler configuration will be assumed, and if no mode is defined at handler level, strict_sync will be assumed.

7.4. Authentication

By default ServiceBlox services do not require authentication by users. They can however be reserved to authenticated users if the specification includes an authentication realm which ServiceBlox has access to. Just like a service, an authentication realm is specified in a workspace according to the ServiceBlox protocol.

ServiceBlox comes with two default realm configurations, which are types of realms that can be instantiated in workspaces. The use of realms is advised to avoid potential security problems and help with future compatibility issues. The two realm configurations are default-password, for stateful authentication with usernames and passwords, and default-signature, for stateless authentication with RSA-SHA512 signatures. ServiceBlox also supports SAML Authentication to rely on third parties to authenticate users.

7.4.1. Glossary

  • Authentication Realm: A collection of credentials used to identify users of a service. A realm can be shared by several services.

  • Authentication Realm Configuration: The specification of a set of parameters defining the characteristics of an Authentication Realm.

  • Authentication Mechanism: A Java class implementing methods to manage credentials in an Authentication Realm.

  • Stateful authentication: An authentication scheme where users provide credentials (aka log in) and obtain a session key which will authenticate them with the service.

  • Stateless authentication: An authentication scheme where users present credentials at each request.

7.4.2. Credentials Database and Service

ServiceBlox contains a library that implements local storage of all user related data, such as usernames, passwords, public keys, email addresses and locale. This library can be used in applications by including the library lb_web_credentials in a project.

The lb web-server command-line tool has support for administration of users. It supports the following commands:

$ lb web-server import-users users.dlm
$ lb web-server export-users users.dlm
$ lb web-server set-password john.smith
enter password for user 'john.smith':
$ lb web-server list-users
john.smith
james.clark

The library will cause the workspace that includes it to host the following delimited-file service:

  • /admin/users for files with headers:

    • USER - required

    • DEFAULT_LOCALE - optional

    • EMAIL - optional

    • ACTIVE - optional

    • PASSWORD - optional

    • PUBLIC_KEY - optional

The PASSWORD column is required to be a bcrypt hash of the clear text password. Importing plain text passwords to these services will not work. We enforce the usage of separate bcrypt hashing to discourage transferring and storing files with plain text passwords. Passwords can be hashed using:

$ echo "password" | lb web-client bcrypt
$2a$10$f3yGStD8yx57jXtZixP4bOtVuTjlRCLJl1HnCwy6HuB60cmJN799i

Or by giving the lb web-client bcrypt command a full file:

$ cat passwords.txt
aaa
bbb
$ lb web-client bcrypt -i passwords.txt
$2a$10$2f1UDZiqp1yHSQpwL5ciYe69lKiA.n22pvhPVtmBI/B6vB7JzPzAW
$2a$10$B4lyNNnWhjtBgmk9lOUi8Ofq2N.GrEjOX8N1XZ0A7gECpOCsO3J0m

The credentials database can be configured to treat usernames as case sensitive or insensitive. In case sensitive mode, the library guarantees that usernames are unique in the database; in case insensitive mode, usernames are guaranteed to be unique after being canonicalized in lowercase, such that usernames that only differ in case are not allowed.

Case insensitive authentication is the default configuration. This can be changed by pulsing the username_case_sentitive() flag, for example, from the command line:

$ lb exec test '+lb:web:credentials:database:username_case_sensitive().'

7.4.3. Stateful Authentication using Passwords and Sessions

ServiceBlox comes with a default realm configuration for authentication using passwords.

  • The realm configuration uses a session key that is stored in a cookie that expires after a reasonable amount of time. The default name of the session key is "lb-session-key_" plus the realm name. This value can be configured by the realm declaration.

  • The realm configuration stores passwords using the secure bcrypt hashing algorithm with salt. It uses a service to retrieve the credentials from a workspace. The service is by default /admin/users. This service can currently not be proxied to a different machine, but support for this is planned.

To use the default password realm configuration, configure a service as follows:

block(`service_config) {

  alias_all(`lb:web:config:service),
  alias_all(`lb:web:config:service_abbr),
  alias_all(`lb:web:config:protobuf),
  alias_all(`lb:web:config:protobuf_abbr),
  alias_all(`lb:web:config:auth),
  alias_all(`lb:web:config:auth_abbr),

  clauses(`{

    service_by_prefix["/time"] = x,
    default_protobuf_service(x) {
      ...
      auth_realm[] = "realm_name"
    }.

    realm_by_name["realm_name"] = x,
    realm(x) {
      realm_config[] = "default-password",
      realm_session_key[] = "time_session_key"
    }.
  })

} <-- .

This will declare a realm called "realm_name" using the default password configuration. In order to be authenticated in this realm, a client needs to send a request to the login service of ServiceBlox. The login service is a ServiceBlox service that accepts JSON or protobuf requests defined by the following protocol:

message Request
{
  required string realm = 1;
  optional string username = 2;
  optional string password = 3;
  optional bool logout =4;
}

message Response
{
  required bool success = 1;
  optional string key = 2;
  optional string exception = 3;
}

Upon successful login, a key is returned to the client to be stored in a cookie. Since this realm overwrites the default session key name, the cookie will be named "time_session_key" (instead of the default "lb-session-key_realm_name"). In the server, a security context is created which will allow subsequent requests with this cookie or with an HTTP header of the same name to access services in this realm.

The security context will expire when more than secondsToLive seconds elapse without a request using this security context, or after a call to the login service with the key for the context as a cookie or http-header and the attribute logout set to true.

7.4.4. Stateless Authentication using RSA-SHA

For services that are not accessed from a browser, but instead are used from non-browser applications deployed on some machine, LogicBlox advises to use an authentication mechanism called RSA-SHA512 (for reference, the closely related HMAC-SHA1 and RSA-SHA1 is more commonly used, but these have weaker security properties and key management complications).

Clients of the web service compute a string-to-sign based on a hash of the content of a request and some important aspects of the HTTP headers, such as the HTTP method, query-string, etc. The goal of this string-to-sign is to be specific enough so that the HTTP request cannot be altered by an attacker to send a critically different request to the server with the same signature. Clients of the web service compute a signature using their private key and the string-to-sign. This signature is passed to the web-service in the standard header used for HTTP authentication. Although we do use SSL sockets, the signature does not expose any secret information if intercepted.

On the server-side, the web service computes the same string-to-sign and verifies with the public key of the user that the client originally did use the correct private key to sign the request. We also verify that the date of the request is within a certain skew (to be determined, but probably 15 minutes).

This method of authentication and ensuring message integrity is similar to the authentication methods of many current mainstream, highly security-sensitive web services, notably Google Apps and the AWS S3 REST authentication scheme, called HMAC-SHA1. LogicBlox uses asymmetric cryptography (public/private keys) rather than symmetric cryptography (private keys on both client and server), as is the case in HMAC-SHA1. This reduces the problem of secure key management, because our servers will only contain the public key of the client.

7.4.4.1. Usage

Generate an RSA key-pair as follows:

$ openssl genrsa -out priv.pem 2048
$ openssl rsa -in priv.pem -out pub.pem -pubout
$ openssl pkcs8 -topk8 -inform PEM -outform PEM -in priv.pem -out priv-pkcs.pem -nocrypt

The priv-pkcs.pem key is intended for deployment on the client-side (the server of a client using the ServiceBlox service). The ServiceBlox client library supports reading this private key format directly. The pub.pem file should be deployed on the server hosting the service to authenticate the user.

See protobuf-auth-rsa in lb-web-samples/

7.4.5. Customizing Authentication Realms and Mechanisms

An authentication realm is an instance of an authentication realm configuration, and it is uniquely identified by its name in a ServiceBlox instance. If multiple realms are configured with the same name, a warning is raised and the realms are not hosted. The realm accepts several configuration options, which are listed in the realm configuration section of lb-web-server.config. For example, this code can be used to set the session timeout for a default-password realm to be one hour:

realm_by_name["time_auth_proxy"] = x,
realm(x) {
  realm_config[] = "default-password",
  realm_option["seconds_to_live"] = "3600"
}.

7.4.6. Using Authenticated User Information in Services

The authenticated user for a service call is available in the pulse predicate lb:web:auth:username[] = username.

The service can be configured in two different ways to populate the lb:web:auth:username predicate. The first method (default) relies on the imported HTTP control information (which also includes the URL and headers). In this confgiuration, computation of the lb:web:auth:username predicate happens in active logic, based on HTTP control predicates. This means that the authenticated user is then not available in inactive blocks.

The second method directly populates the predicate at the intial stage of the transaction. This method allows entirely disabling the import of the HTTP control information, which can help with the transaction throughput of services. To use this method, configure the service as follows (replacing protobuf_service with the service type used):

protobuf_service(x) {
  ...
  service_parameter["enable_http_control"] = "false",
  service_parameter["enable_pulse_auth"] = "true"
}.

Security Considerations.  The authenticated user information should only be trusted if the service is actually authenticated. For services that do not have authenticated configured, clients can claim to be any user, because it is based on HTTP header information (x-logicblox-username). The authentication facilities initialize information for a request.

Supported Services.  The authenticated user is available to protobuf as well as data exchange services. If a proxy service is authenticated, then the authenticated user information is forwarded to the target service. Custom service handlers need to make sure to import the HTTP request information to make this information available.

7.4.7. SAML Authentication

SAML is an XML standard for applications to securely communicate authentication and authorization information. The most common application of SAML is browser single sign-on (SSO), which lets web applications use a trusted, external application to manage the information of users and execute the authentication procedure. The web application is commonly called the service provider, or SP. The external application that manages users and executes authentication is called the identity provider, or IDP. The key advantage of single sign-on is that the service provider never gets to see the secret credentials of the user, and user information does not need to be separately maintained for every web application.

ServiceBlox has native support for SAML 2.0 authentication. SAML is really a collection of different authentication methods though, and not all these methods make sense for ServiceBlox. The remainder of this paragraph is to clarify the level of support to SAML experts. ServiceBlox only uses the Authentication Request Protocol. ServiceBlox supports the Redirect (urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect) and POST (urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST) bindings. By default, ServiceBlox submits the authentication request using a HTTP redirect and instructs the identity provider to use a HTTP POST to the assertion consumer service hosted by ServiceBlox.

SAML authentication is enabled by including a [saml:somename] section in the lb-web-server.config file. The SAML section supports the following configuration options:

Required settings
Configuration Description
request Path of URL for triggering the initial authentication request to the identity provider. This is the URL that users need to visit to trigger authentication. For example, if the path is configured to be /sso/request, then on a local machine the user would need to visit http://localhost:8080/sso/request. From this URL they will be redirected to the identity provider. If the user was already authenticated (for example with a cookie for the identity provider), then the identity provider will normally immediately send the user back to the application. Due to the automatic sequence of redirects, this entry-point can be configured as the front-page of the application.
response Path of URL that the identity provider will use to confirm successful authentication. It is important that this setting is consistent with the assertion_consumer_service configuration.
redirect URL to redirect the user to after successful authentication. For testing purposes it can be helpful to use /login/user?realm=test, which shows the current username.
realm Name of the authentication realm that this SAML configuration will authenticate users to. Services that use this realm can be accessed by the user after successful authentication. The realm is separately configured in the workspace as a normal ServiceBlox authentication realm. It has to use the realm configuration default-saml.
meta Path of URL for hosting xml metadata for the service provider (an EntityDescriptor) that can be used by some identity providers to register the service provider.
alias_idp Alias name given to the identity provider. This alias is used to find a certificate (.cer format) for the identity provider. It is only used by ServiceBlox and is not used in the exchange of information with the identity provider.
alias_sp Alias name given to this service provider. This alias is used to find public/private keys (.pem format) for the service provider. It is only used by ServiceBlox and is not used in the exchange of information with the identity provider.
entity_id_idp Identity for the identity provider. Depending on the registration procedure with the entity provider, this can normally be found in the XML metadata for the identity provider, as the entityID attribute of the EntityDescriptor element.
entity_id_sp Identity for the service provider. This is the name used by the identity provider to refer to the service provider, and is typically agreed upon in some way during the registration procedure of the service provider with the identity provider. This can in simple cases be identical to the alias_sp. In some cases identity providers can have specific requirements for the id, such as URIs, in which case the entity_id_sp cannot be the same as the alias_sp.
sso_service URL hosted by the IDP that a user will be redirected to after visiting the request URL explained previously. This can normally be found in the XML metadata of the identity provider. Be careful to select the Location for the redirect binding, for example: <SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://idp.testshib.org/idp/profile/SAML2/Redirect/SSO"/>
assertion_consumer_service Full URL that the IDP will use to confirm successful authentication. The path of the URL should be the same as the response path, unless a proxy is used that rewrites paths.
attribute_map List of pairs, where the first element is the local identifier to use to refer to a certain attribute of a user. The second element is the standard object identifier, which is the recommended way to refer to attributes returned by the IDP to the SP. The local identifier uid is special in ServiceBlox and will be used as the username. This is the only required attribute. See the example configuration below for how to determine what attributes are available.
Optional settings
keydir Path of directory where encryption keys and certificates are kept. Defaults to the global ServiceBlox key directory.
binding_idp Whether to use GET (Redirect) or POST to submit the SAML request to the Identity Provider. Note that this must match the binding exposed by the sso_service endpoint configured above. The default is Redirect.
binding_sp whether to request the IDP to use GET (Redirect) or POST to submit the SAML response to the SP. The default is POST.
force_authn if true, the request sent to the IDP will set the force_authn flag, which forces the IDP to request the user for authentication credentials. The default is false, in which case an older ongoing IDP session may be reused.
include_name_id_policy if true, include the NameIDPolicy element in the AuthnRequest. This is an optional field that may cause issues with some IDPs. The default is true.
log_level

Log level for the SAML libraries. Possible values are:

  • off (default) - no logging.

  • info - print the response attribute map.

  • message - same as info and additionally print all exchanged XML messages.

  • debug - same as message and additionally print decrypted assertions. Do not use in production.

In addition to the configuration options, the following cryptographic keys are needed:

  • alias_idp.cer - Certificate of the IDP. The certificate is used to verify that authentication confirmations really do originate from the intended identity provider. The certificate is not a secret, and is usually available in the XML metadata of the identity provider. This file should contain something similar to:

    -----BEGIN CERTIFICATE-----
    MIIEDjCCAvagAwIBAgIBADANBgkqhkiG9w0BAQUFADBnMQswCQYDVQQGEwJVUzEV
    ...
    8K/qhmFT2nIQi538n6rVYLeWj8Bbnl+ev0peYzxFyF5sQA==
    -----END CERTIFICATE-----

  • alias_sp.pem - Private key for the service provider. This is used to sign the request to the identity provider. The identity provider uses a certificate generated from this key (see next item) to validate that the authentication request does indeed originate from the registered service provider. This file should contain something similar to:

    -----BEGIN PRIVATE KEY-----
    MIICdwIBADANBgkqhkiG9w0BAQEFAASCAmEwggJdAgEAAoGBANXOgiM+NVoAIiZY
    ...
    Opkj9UzXZ+nvzfk=
    -----END PRIVATE KEY-----

  • alias_sp.cer - Certificate file for the service provider. The format is the same as the certificate for the identity provider.

7.4.7.1. Testing SAML with TestShib

This section describes step by step how to use a free online SAML testing service called TestShib with ServiceBlox. Clearly this service should never be used in any actual application, but it is useful as an exercise in deploying SAML, without having to install and configure an identity provider as well.

Choose an Alias.  First, decide on an alias for your web application, which we will refer to as the service provider (SP). This example will use logicblox-abc from now. Do pick a different alias, because the TestShib testing service will use this as the account name, and different users of this guide will interfere!

Private Key.  We begin with generating a private key using the s3lib-keygen tool (which uses OpenSSL). This private key is used by the service provider to sign requests to the identity provider, which can in this way confirm that authentication requests only originate from approved service providers.

$ s3lib-keygen logicblox-abc

Certificate.  While the SAML standard could have chosen to simply provide the public key of the key pair to the identity provider, the standard is using certificates to get additional evidence about the identity of the service provider. Therefore, we next need to create a certificate for the private key. In this example we use a self-signed certificate. Note that the certificate is never used in a browser, so there will not be a problem with the recent trend of severe browser warnings for self-signed certificates. The openssl tool will ask a few pieces of information, which do not matter for this example. Simply hitting enter is sufficient.

$ openssl req -new -x509 -key logicblox-abc.pem -out logicblox-abc.cer -days 1095

If you later want to inspect certificate files (either the one just generated, or the one from the IDP that we will obtain later), then you can use openssl as well:

$ openssl x509 -in logicblox-abc.cer -text -noout

Key Deployment.  Copy the logicblox-abc.pem and logicblox-abc.cer files to your standard key directory, which normally is $HOME/.s3lib-keys.

$ cp logicblox-abc.cer logicblox-abc.pem ~/.s3lib-keys

Collect IDP Metadata.  Before we can configure ServiceBlox we need to collect some information on the identity provider. The TestShib configuration instructions link to a XML document that describes the TestShib services: testshib-providers.xml. The XML document contains two EntityDescriptors: an identity provider (for testing service providers, which is what we are doing here), and a service provider (for testing identity providers, which is not the purpose of this guide). We need to collect three pieces of information:

  • The entityID of the IDP, which can be found in this XML tag: <EntityDescriptor entityID="https://idp.testshib.org/idp/shibboleth">. We will use this value in the configuration of ServiceBlox.

  • The URL to redirect users to when authentication is needed, which can be found in this XML tag: <SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://idp.testshib.org/idp/profile/SAML2/Redirect/SSO"/>. We will use this value in the configuration of ServiceBlox.

  • The certificate of the IDP, which the service provider will use to validate that it is not being tricked by somebody posing to be the IDP. The certificate can be found as the first <KeyDescriptor> in the <IDPSSODescriptor> tag. You need to copy the content of <ds:X509Certificate>...</ds:X509Certificate> and create a file testshib.cer that looks like the following example (which is actually the testshib certificate). Make sure that the file is formatted exactly in this way.

    -----BEGIN CERTIFICATE-----
    MIIEDjCCAvagAwIBAgIBADANBgkqhkiG9w0BAQUFADBnMQswCQYDVQQGEwJVUzEV
    MBMGA1UECBMMUGVubnN5bHZhbmlhMRMwEQYDVQQHEwpQaXR0c2J1cmdoMREwDwYD
    VQQKEwhUZXN0U2hpYjEZMBcGA1UEAxMQaWRwLnRlc3RzaGliLm9yZzAeFw0wNjA4
    MzAyMTEyMjVaFw0xNjA4MjcyMTEyMjVaMGcxCzAJBgNVBAYTAlVTMRUwEwYDVQQI
    EwxQZW5uc3lsdmFuaWExEzARBgNVBAcTClBpdHRzYnVyZ2gxETAPBgNVBAoTCFRl
    c3RTaGliMRkwFwYDVQQDExBpZHAudGVzdHNoaWIub3JnMIIBIjANBgkqhkiG9w0B
    AQEFAAOCAQ8AMIIBCgKCAQEArYkCGuTmJp9eAOSGHwRJo1SNatB5ZOKqDM9ysg7C
    yVTDClcpu93gSP10nH4gkCZOlnESNgttg0r+MqL8tfJC6ybddEFB3YBo8PZajKSe
    3OQ01Ow3yT4I+Wdg1tsTpSge9gEz7SrC07EkYmHuPtd71CHiUaCWDv+xVfUQX0aT
    NPFmDixzUjoYzbGDrtAyCqA8f9CN2txIfJnpHE6q6CmKcoLADS4UrNPlhHSzd614
    kR/JYiks0K4kbRqCQF0Dv0P5Di+rEfefC6glV8ysC8dB5/9nb0yh/ojRuJGmgMWH
    gWk6h0ihjihqiu4jACovUZ7vVOCgSE5Ipn7OIwqd93zp2wIDAQABo4HEMIHBMB0G
    A1UdDgQWBBSsBQ869nh83KqZr5jArr4/7b+QazCBkQYDVR0jBIGJMIGGgBSsBQ86
    9nh83KqZr5jArr4/7b+Qa6FrpGkwZzELMAkGA1UEBhMCVVMxFTATBgNVBAgTDFBl
    bm5zeWx2YW5pYTETMBEGA1UEBxMKUGl0dHNidXJnaDERMA8GA1UEChMIVGVzdFNo
    aWIxGTAXBgNVBAMTEGlkcC50ZXN0c2hpYi5vcmeCAQAwDAYDVR0TBAUwAwEB/zAN
    BgkqhkiG9w0BAQUFAAOCAQEAjR29PhrCbk8qLN5MFfSVk98t3CT9jHZoYxd8QMRL
    I4j7iYQxXiGJTT1FXs1nd4Rha9un+LqTfeMMYqISdDDI6tv8iNpkOAvZZUosVkUo
    93pv1T0RPz35hcHHYq2yee59HJOco2bFlcsH8JBXRSRrJ3Q7Eut+z9uo80JdGNJ4
    /SJy5UorZ8KazGj16lfJhOBXldgrhppQBb0Nq6HKHguqmwRfJ+WkxemZXzhediAj
    Geka8nz8JjwxpUjAiSWYKLtJhGEaTqCYxCCX2Dw+dOTqUzHOZ7WKv4JXPK5G/Uhr
    8K/qhmFT2nIQi538n6rVYLeWj8Bbnl+ev0peYzxFyF5sQA==
    -----END CERTIFICATE-----

    You can check if the certificate is saved correctly by printing the certificate info:

    openssl x509 -in testshib.cer -text -noout

    This should include the following: Issuer: C=US, ST=Pennsylvania, L=Pittsburgh, O=TestShib, CN=idp.testshib.org. Now copy this certificate to the standard key directory as well:

    $ cp testshib.cer ~/.s3lib-keys

Minimal ServiceBlox Configuration.  Now we have all the information needed to configure ServiceBlox. Create a configuration file lb-web-testshib.config (or alternatively put this directly in LB_DEPLOYMENT_HOME/config/lb-web-server.config).

[saml:testshib]
request = /sso/request
response = /sso/response
redirect = /login/user?realm=test
realm = test
meta = /sso/metadata

alias_idp = testshib
alias_sp = logicblox-abc

entity_id_sp = logicblox-abc
entity_id_idp = https://idp.testshib.org/idp/shibboleth

sso_service = https://idp.testshib.org/idp/profile/SAML2/Redirect/SSO
assertion_consumer_service = http://localhost:8080/sso/response

attribute_map = uid urn:oid:0.9.2342.19200300.100.1.1

Except for attribute_map, all the configuration settings have been discussed in the previous steps. The attribute_map is mostly an initial attempt. The urn:oid:0.9.2342.19200300.100.1.1 is the standard object identifier for user identifiers, and is used by all SAML providers. The IDP will offer more information on the user, but these attributes are not formally documented, so we need to trigger an authentication request before we can discover them. Some SAML identity providers will publish this information in their documentation or metadata.

Start ServiceBlox. 

$ lb web-server start lb-web-testshib.config

Register Service Provider with TestShib.  ServiceBlox is now hosting an XML metadata file at the configured meta path. This file can be used to register the service provider with TestShib. Download this file and store it as logicblox-abc.xml. You can do this either using a browser or with the following command.

$ curl http://localhost:8080/sso/metadata > logicblox-abc.xml

On the TestShib website the service provider can now be registered with this XML file. Visit the metadata upload form for this and upload the logicblox-abc.xml file.

First Authentication.  Everything is now setup to attempt the first authentication. Point your browser at http://localhost:8080/sso/request. This will redirect to the TestShib website, where you can login with one of the suggested accounts. Pick the username and password myself. After confirming, the browser will go back to the ServiceBlox-hosted application. Most likely, you will now see an error Could not find realm : test. This indicates that the SAML request was processed correctly, but that there is no authentication realm test. This error is simply because we did not cover hosting an actual service with a realm of the name test. While the error is perhaps unsatisfying, this means that the SAML configuration was successful. Note that SAML supports keeping track of the relay state so that users can be redirected to a specific resource upon authentication. For example, if we initiated our request using http://localhost:8080/sso/request?RelayState=/some/resource, we would have been redirected to /some/resource after authentication succeeds. If the RelayState parameter is not used, ServiceBlox falls back to the redirect configuration attribute as defined above.

Configuring more attributes.  In the lb-web-server.log or the terminal (depending on how you started the LogicBlox web-server), there are two tables printed as the result of this authentication request. The first table corresponds to the actual configured attribute_map, which currently only contains uid.

|------------------------------------------------------------------------------|
| name                           | value                                       |
|------------------------------------------------------------------------------|
| uid                            | myself                                      |
|------------------------------------------------------------------------------|

The second table lists all attributes returned by the IDP.

|---------------------------------------------------------------------------------------------------------------|
| oid                                | friendly name              | value                                       |
|---------------------------------------------------------------------------------------------------------------|
| urn:oid:0.9.2342.19200300.100.1.1  | uid                        | myself                                      |
| urn:oid:1.3.6.1.4.1.5923.1.1.1.1   | eduPersonAffiliation       | Member                                      |
| urn:oid:1.3.6.1.4.1.5923.1.1.1.6   | eduPersonPrincipalName     | myself@testshib.org                         |
| urn:oid:2.5.4.4                    | sn                         | And I                                       |
| urn:oid:1.3.6.1.4.1.5923.1.1.1.9   | eduPersonScopedAffiliation | Member@testshib.org                         |
| urn:oid:2.5.4.42                   | givenName                  | Me Myself                                   |
| urn:oid:1.3.6.1.4.1.5923.1.1.1.7   | eduPersonEntitlement       | urn:mace:dir:entitlement:common-lib-terms   |
| urn:oid:2.5.4.3                    | cn                         | Me Myself And I                             |
| urn:oid:1.3.6.1.4.1.5923.1.1.1.10  | eduPersonTargetedID        | <saml2:NameID Format="urn:oasis:names:tc... |
| urn:oid:2.5.4.20                   | telephoneNumber            | 555-5555                                    |
|---------------------------------------------------------------------------------------------------------------|

Based on this information we can now extend the attribute_map configuration to have more attributes that are relevant to the service provider. For this, modify lb-web-testshib.config to use the following setting for attribute_map:

attribute_map = \
  uid urn:oid:0.9.2342.19200300.100.1.1 \
  cn urn:oid:2.5.4.3 \
  phone urn:oid:2.5.4.20 \
  sn urn:oid:2.5.4.4 \
  givenName urn:oid:2.5.4.42

Note that ServiceBlox currently does not support exposing user attributes to implementations of services, so configuring more attributes is currently actually not very useful, but we expect this to be supported soon. After restarting the LB web server, the next authentication request will now show a more detailed attribute table:

|------------------------------------------------------------------------------|
| name                           | value                                       |
|------------------------------------------------------------------------------|
| uid                            | myself                                      |
| phone                          | 555-5555                                    |
| sn                             | And I                                       |
| cn                             | Me Myself And I                             |
| givenName                      | Me Myself                                   |
|------------------------------------------------------------------------------|

7.4.8. Google Sign-In

Google Sign-In allows the web applications to sign in users using their existing Google account. Similar to SAML mentioned above, the user information does not need to be separately maintained for every web application.

ServiceBlox supports authenticating users with their existing Google accounts into a realm. It can then authorize services for those users just like any other natively defined user. To enable this functionality, create a project in Google Developer Console, add a credential, configure a consent screen and create a client ID. This client ID will be used for configuring the lb-web server and also the client side log-in UI code.

The lb-web server is configured by including a [google-signin:my-app-name] section in the lb-web-server.config file. Replace 'my-app-name' in the section title with project name created in Google Developer Console. The following are the configuration options are available under this section:

Required settings
Configuration Description
client_id This is the client id that Google issues from their Developer Console. It must match the ID used by the client side to generate the ID token.
realm The realm into which the user will be authenticated if Google Sign-In succeeds.
allowed_domains Google sends to lb-web server the authenticated user email. This allows administrators to restrict the domains that can log into the realm. If this is empty or commented out, any domain is allowed; otherwise, only emails belonging to one of these domains is allowed.
Optional settings
Configuration Description
credential_service An absolute url to a credentials service. If defined, new users will be created in this credentials service for users that authenticated successfuly but who did not have an entry. Note that the username will be the user email, and no password will be set.
roles A list with names of user roles. If a credentials service is set (above) and roles are set, then these roles will be assigned to any new users that authenticate via this Google Sign-In configuration. Note that the roles must already exist in the credentials database configured above.

After configuring lb-web server as described above, the application's web client should implement the Google Sign-In in its page as explained at Google Sign-In for Websites. Then, in the sign-in succeeded event handler, invoke the service /google-signin?idtoken= passing the ID token received from Google Sign-In as idtoken parameter.

7.5. Authorization

ServiceBlox supports a flavor Role-Based Authorization Control (RBAC) for securing services.

7.5.1. Definitions

7.5.1.1. Roles

A Role can be thought as a behavior or job function that can be assigned to a user or groups of users. A user can be assigned to many roles. A role can be assigned a set of Permissions. (e.g. "Buyer" or "Admin")

7.5.1.2. Permissions

A Permission is an approval of access to a given function or resource. A permission is assigned to one or many Roles and a permission consists of a set of Operations. (e.g. "Manage Order")

7.5.1.3. Operations

An Operation is specific business function or application function. An Operation is assigned to one or many Permissions. A service can have only one Operation. (e.g. "Cancel Order" or "Create User")

7.5.2. Configuration

You can enable RBAC by assigning your services to operations by specifying the service_operation relation for a service, like so:

service_by_prefix["/sell_product"] = x,
default_protobuf_service(x) {
  protobuf_protocol[] = "sample_authorization",
  protobuf_request_message[] = "SellRequest",
  protobuf_response_message[] = "SellResponse",
  auth_realm[] = "default",
  service_operation[] = "Sell Product"
}.
            

The rest of the configuration is done via data in the workspace. The lb:web:credentials:authorization module defines predicates that map users to roles, roles to permissions, and permissions to operations. The credentials service comes equipped with TDX services for loading this data. The data is then queried out any time the credentials workspace is re-scanned, typically after server startup or calling lb web-server load-services.

Services that fail authorization (because a user does not have a role that has a permission for the operation defined for the service) will return an HTTP status code of 403.

7.5.2.1. File Bindings

file_binding_by_name["lb:web:credentials:authorization:permissions"] = fb,
file_binding(fb) {
  file_binding_definition_name[] = "lb:web:credentials:authorization:permissions",
  file_binding_entity_creation[] = "accumulate",

  predicate_binding_by_name["lb:web:credentials:authorization:permission_description"] =
    predicate_binding(_) {
      predicate_binding_columns[] = "PERMISSION, DESCRIPTION"
    }
}.

file_binding_by_name["lb:web:credentials:authorization:operations"] = fb,
file_binding(fb) {
  file_binding_definition_name[] = "lb:web:credentials:authorization:operations",
  file_binding_entity_creation[] = "accumulate",

  predicate_binding_by_name["lb:web:credentials:authorization:operation_description"] =
    predicate_binding(_) {
      predicate_binding_columns[] = "OPERATION, DESCRIPTION"
    }
}.

file_binding_by_name["lb:web:credentials:authorization:roles"] = fb,
file_binding(fb) {
  file_binding_definition_name[] = "lb:web:credentials:authorization:roles",
  file_binding_entity_creation[] = "accumulate",

  predicate_binding_by_name["lb:web:credentials:authorization:role_description"] =
    predicate_binding(_) {
      predicate_binding_columns[] = "ROLE, DESCRIPTION"
    }
}.

file_binding_by_name["lb:web:credentials:authorization:role_permission_mappings"] = fb,
file_binding(fb) {
  file_binding_definition_name[] = "lb:web:credentials:authorization:role_permission_mappings",
  file_binding_entity_creation[] = "none",

  predicate_binding_by_name["lb:web:credentials:authorization:role_permissions"] =
    predicate_binding(_) {
      predicate_binding_columns[] = "ROLE, PERMISSION"
    }
}.

file_binding_by_name["lb:web:credentials:authorization:permission_operation_mappings"] = fb,
file_binding(fb) {
  file_binding_definition_name[] = "lb:web:credentials:authorization:permission_operation_mappings",
  file_binding_entity_creation[] = "none",

  predicate_binding_by_name["lb:web:credentials:authorization:permission_operations"] =
    predicate_binding(_) {
      predicate_binding_columns[] = "PERMISSION, OPERATION"
    }
}.

file_binding_by_name["lb:web:credentials:authorization:user_role_mappings"] = fb,
file_binding(fb) {
  file_binding_definition_name[] = "lb:web:credentials:authorization:user_role_mappings",
  file_binding_entity_creation[] = "none",

  predicate_binding_by_name["lb:web:credentials:authorization:user_roles"] =
    predicate_binding(_) {
      predicate_binding_columns[] = "USER, ROLE"
    }
    service_by_prefix[authorization_prefix[] + "/permissions"] = x,
delim_service(x) {
  delim_file_binding[] = "lb:web:credentials:authorization:permissions",
  group("lb:web:internal")
}.
            

7.5.2.2. TDX Services

service_by_prefix[authorization_prefix[] + "/permissions"] = x,
delim_service(x) {
  delim_file_binding[] = "lb:web:credentials:authorization:permissions",
  group("lb:web:internal")
}.

service_by_prefix[authorization_prefix[] + "/operations"] = x,
delim_service(x) {
  delim_file_binding[] = "lb:web:credentials:authorization:operations",
  group("lb:web:internal")
}.

service_by_prefix[authorization_prefix[] + "/roles"] = x,
delim_service(x) {
  delim_file_binding[] = "lb:web:credentials:authorization:roles",
  group("lb:web:internal")
}.

service_by_prefix[authorization_prefix[] + "/role_permissions"] = x,
delim_service(x) {
  delim_file_binding[] = "lb:web:credentials:authorization:role_permission_mappings",
  group("lb:web:internal")
}.

service_by_prefix[authorization_prefix[] + "/permission_operations"] = x,
delim_service(x) {
  delim_file_binding[] = "lb:web:credentials:authorization:permission_operation_mappings",
  group("lb:web:internal")
}.

service_by_prefix[authorization_prefix[] + "/user_roles"] = x,
delim_service(x) {
  delim_file_binding[] = "lb:web:credentials:authorization:user_role_mappings",
  group("lb:web:internal")
}.
            

7.6. Aborting Transactions

ServiceBlox supports a mechanism in which you can abort a transaction and report a desired HTTP status code and debug message. The most obvious use case for this feature is to enable developers on the platform to enforce custom authorization schemes.

In order to cause a transaction abort and return a status code and message you must delta in to the lb:web:abort:error_response predicate which defined as:

            
error_response(code, msg) -> int(code), string(msg).
        

This will in turn cause a constraint violation which will abort the transaction in a way that ServiceBlox can interpret and then return the status code and message.

Currently this functionality only works on Protobuf services but TDX services support will be added very soon. In order for the message to be returned to the browser, the server must be in debug mode (this is for security reasons).

7.7. Transport Methods

7.7.1. Using Amazon Simple Storage Service (S3)

Normally, an HTTP request contains a request line, headers, and the content of the request. For POST and PUT requests to tabular data exchange services, the content would be the CSV data. For files that are extremely large, this is not a good idea, because a connection failure would force re-doing the entire upload. Also, a file might already be available in highly redundant distributed storage, which makes it undesirable to send the data directly to a service from the client.

ServiceBlox supports transferring large data files separate from HTTP requests. This makes it possible to have an interface that resembles the REST architecture even when the data files exceed the volume of data one would normally want to transfer over a single socket in a single request.

ServiceBlox uses a custom HTTP header x-blox-content-uri for this. The idea of the HTTP header is that the content is not actually in the body of the HTTP request, but is stored as an S3 object located at the specified URI. The additional HTTP headers of the request or response still apply to the content at this URI (e.g. the content-type header of the HTTP request would specify the content-type of the S3 object). The header x-blox-response-content-uri is used to indicate that a response is expected not as-is in the HTTP response, but at a certain URI. The content-uri support is applicable to all transport mechanisms, so it can be used on TCP requests as well as queue requests.

Users do not need to be aware of the HTTP headers. ServiceBlox client APIs and the lb web-client tool internally use the headers to communicate with the LB web server when importing and exporting delimited files using S3.

ServiceBlox uses a high performance S3 library developed by LogicBlox, called s3lib, to upload and download files to and from S3. The library can also be used through a separate command-line tool called cloud-store. The s3lib library uses encryption keys to encrypt data stored in S3. Because cloud-store, lb web-client and the LB web server use s3lib, it is necessary to set up s3lib encryption keys before using any of those tools with S3. The following section explains how to manage the encryption keys used by s3lib.

7.7.1.1. Managing s3lib keys

The s3lib library uses asymmetric encryption keys (public/private RSA keys) to encrypt and decrypt data that is stored in S3. Data is encrypted locally prior to uploading to S3, and it is decrypted locally after downloading, so it is never transferred in clear text. The asymmetric encryption method is used to simplify key management: the uploader of a file only needs access to the public key (the private key is still needed to decrypt upon download).

Encryption is currently not an optional feature, so before the tools can be used, encryption keys need to be generated and configured. The s3lib library uses a directory of .pem text files to store the encryption keys. On EC2 instances, this directory should be in an in-memory file system to protect the keys from being available on disk. The s3lib tool and ServiceBlox manage encryption keys by a key alias, which is the name of the .pem file in the key directory. It is important that key aliases are unique and consistent across machines, otherwise decryption will fail and data-loss will occur. A key pair can be generated using:

$ cloud-store keygen -n mykey

This command generates a file mykey.pem under ~/.s3lib-keys by default (it can be changed with the --keydir option). The key alias is thus mykey. The file contains the public as well as the private key, in the following format:

-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvS53DLCAPMQv3mBb/G/D
...
-----END PUBLIC KEY-----

-----BEGIN PRIVATE KEY-----
MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQC9LncMsIA8xC/e
...
-----END PRIVATE KEY-----

For upload clients that only need the public key, the private key section can be removed from the file.

The s3lib by default uses the directory ~/.s3lib-keys for encryption keys, but all commands and tools can change the default setting.

7.7.1.2. Using cloud-store

The cloud-store is a command-line tool around s3lib and it can interact with both S3 and Google Cloud Storage (GCS). It has 3 major commands that allow to upload a local file to S3/GCS, to download an S3/GCS file, and to verify whether an S3/GCS file exists. These commands take a command-line option --keydir to change the default setting for the directory where the keys are stored. They also accept a variety of command-line options to configure the retry behavior and influence performance. To review these options, use:

$ cloud-store upload --help
$ cloud-store download --help
$ cloud-store exists --help

The cloud-store needs AWS credentials to access S3. Currently, it uses the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_KEY, or falls back to credentials delivered via the EC2 meta-data service (usually set using EC2 instance roles).

For GCS access, currently, the following environment variables are required: GOOGLE_APPLICATION_CREDENTIALS, GCS_XML_ACCESS_KEY and GCS_XML_SECRET_KEY. The first one should point to a JSON file, generated in Google Developers Console, that defines the GCS service credentials. The other two should be used for accessing the GCS interoperable XML API.

After credentials have been set up and the mykey key has been generated and stored in ~/.s3lib-keys, cloud-store can upload a file as follows:

$ cloud-store upload s3://bucket/AS400.jpg -i AS400.jpg --key mykey # S3
$ cloud-store upload gs://bucket/AS400.jpg -i AS400.jpg --key mykey # GCS

The upload command will encrypt the file using mykey and then will send the encrypted file to the S3/GCS object. The cloud-store also attaches meta-data to the object to identify the key with which it is encrypted. To download the file back:

$ cloud-store download s3://bucket/AS400.jpg -o AS400-2.jpg  # S3
$ cloud-store download gs://bucket/AS400.jpg -o AS400-2.jpg  # GCS

Note that the download command automatically determines what key to use to decrypt from the meta-data attached to the S3/GCS object.

7.7.1.3. Using S3 in ServiceBlox

The following sections describe how to configure the ServiceBlox components that can use S3. In all cases it is assumed that encryption keys were already generated according to the Managing s3lib keys section.

LB web server

The LB web server can be configured to access S3 to retrieve content of requests and to store responses. Just like cloud-store, it needs AWS credentials and needs access to the directory where the keys are stored. These configurations are set with an [s3:default] section in lb-web-server.config. For example, the following section defines the directory holding encryption keys and the credentials to access AWS:

[s3:default]
keydir = ...
access_key = ...
secret_key = ...
#iam_role =
#env =

Note that keydir could have been specified outside the [s3:default] section because variables of the global section are inherited by sections (see details in ServiceBlox Configuration). Also, instead of explicitly specifying the access and secret keys, it is possible to use AWS Identity and Access Management (IAM) by setting the iam_role variable (the value of the variable is ignored and the default role is used). Finally, setting the env variable requests the LB web server to load access and secret keys from environment variables.

After setting AWS credentials and keydir, the LB web server is ready to use S3 on requests. When the server is instructed with the x-blox-content-uri HTTP header to download a file from S3, it will determine the key to decrypt the file from the meta-data stored in the S3 object. It will search for the key in keydir. When the server needs to store response content in S3 due to a x-blox-response-content-uri HTTP header, it expects an additional HTTP header called x-blox-response-content-enckey. This header defines the alias of the key to use to encrypt the file before sending to S3 (the alias is also stored in S3 meta-data).

ServiceBlox Client

The ServiceBlox client accepts the same configuration options as the server, but in the lb-web-client.config file instead. Additionally, it supports the keyname variable in the global section. This variable defines the key alias to use by default for the commands export-delim (to encrypt the exported file) and import-delim (to encrypt bad records if requested; the decryption key for the file to import is determined by meta-data). The key to use in these commands can be overridden with the command line option --key.

ServiceBlox Batch Language

The ServiceBlox client can be used to execute batches specified in the ServiceBlox Batch Language. Since batch specifications are executed with lb web-client, the configurations available in lb-web-client.config are immediately supported. Furthermore, batch specifications can override the values of the config files in two ways:

  • The BatchConfig message can specify the keydir and keyname to be used globally in the batch specification.

  • Most statements that can make use of S3 can individually override the global setting for the key name using the key attribute. Currently, S3Upload, ImportDelim, and ExportDelim can specify the key to use. Note that S3Download uses S3 but the key is determined by meta-data.

7.7.2. Queues

In addition to HTTP over TCP, ServiceBlox also allows you to communicate through the use of queues. Normal TCP communication has a few drawbacks which queues allow you to solve. First, with typical TCP communication, each server needs to know about every other server with which it needs to communicate. In larger deployments with many machines, this can get very complicated to set up and maintain. Second, TCP leaves connections open for the entire duration of the request. This is generally fine if your handling short running requests but during batch operations where requests can take hours, this can be problematic and error prone. If there is a network hiccup entire batches could be aborted.

Queues solve both of these problems. Queues solve the network complexity problem by allowing servers to talk to each other via queues. Typically, there is one queue endpoint that all servers communicate with and no server needs to know anything about any other server. This allows message consumers to be taken offline while producers continue to operate. It also allows consumers to be replaced if there is an error as a new consumer can start up and start handling the back log of messages. Queues solve the long running connection problem by having a request and a response queue. Instead of having one connection per message, you have two total connections, one for putting messages on the queue and one for reading responses off the queue.

ServiceBlox offers built in integration with two queue implementations, each with their own pros and cons. See the introduction paragraph of each section for more information on why you would use one over the other. In general, we recommend using RabbitMQ as it has far better consistency guarantees than SQS (i.e. no zombie messages which can derail batch processes).

7.7.2.1. Using RabbitMQ

RabbitMQ is the recommended queue implementation for use with LogicBlox deployments. The primary differences between it and SQS are:

  • RabbitMQ is much faster and completely consistent (i.e. guarantees only once delivery) whereas SQS can sometimes deliver messages more than once / fail to delete messages.

  • RabbitMQ can be run locally for development and must be deployed to an EC2 server as part of the application's deployment.

7.7.2.1.1. Installing RabbitMQ

To install RabbitMQ on your developer machine, it is recommended you download the generic-unix version of RabbitMQ. The lb-config extension for RabbitMQ is designed to work with this distribution. All you need to do is download the file and unzip it. It does not require any installation and all command to start it will be found in the sbin folder.

The easiest way to start RabbitMQ is to run RABBITMQ_HOME/sbin/rabbitmq-server. Shut it down by pressing Ctrl+C in that terminal.

7.7.2.1.2. Configuring RabbitMQ

RabbitMQ requires very little configuration. Once you have started the server, you add a section to both your lb-web-server.config and lb-web-client.config that specifies request and response queues and an amqp endpoint to the RabbitMQ server. An endpoint looks just like a normal URL except it starts with "amqp://" instead of "http://" and must include a port number. The default port for RabbitMQ is 5672.

An example configuration is shown below:

[rabbitmq:sample]
request_queue_name = lb-web-sample-request
response_queue_name = lb-web-sample-response
endpoint = 'amqp://localhost:5672'

7.7.2.1.3. Testing RabbitMQ

A JSON service now be invoked over RabbitMQ as follows:

$ echo '{"min_price" : 30}' | lb web-client call /search --format --config rabbitmq:sample

7.7.2.2. Using SQS

SQS is a cloud based queue implementation offered by AWS. While SQS is good for distributing messages to workers, it is not necessarily a good fit for the ServiceBlox use case where we want to simulate HTTP traffic over TCP. The reason it is not a good fit is that SQS does not guarantee only once delivery. Unless all your services are completely idempotent, then you can easily get data problems because you will execute some messages multiple times.

7.7.2.2.1. Installing SQS

One nice thing about SQS is that there is no installation, it is completely hosted in the cloud. You do, however, have to set up an AWS account with SQS privileges so you can retrieve your SQS access key and secret key. AWS configuration is outside the scope of this document.

7.7.2.2.2. Configuring SQS

Once you have your access key and secret key, you can configure a pair of SQS queues as an end-point. To do this, add a configuration section to the lb-web-server.config file in LB_DEPLOYMENT_HOME/config.

[sqs:sample]
request_queue_name = lb-web-sample-request
response_queue_name = lb-web-sample-response
access_key = ...
secret_key = ...

To invoke a service via SQS, you can use the lb web-client tool. The lb web-client tool uses a separate configuration located in LB_DEPLOYMENT_HOME/config/lb-web-client.config. The configuration is identical to the configuration for the server:

[sqs:sample]
request_queue_name = lb-web-test-request
response_queue_name = lb-web-test-response
access_key = ...
secret_key = ...

A JSON service now be invoked over SQS as follows:

$ echo '{"min_price" : 30}' | lb web-client call /search --format --config sqs:sample

Configuration options:

AWS credentials are specified as one of the following options:
access_key secret_key Configure access keys and secret keys to use.
iam_role Use IAM instance roles to obtain credentials. This option only works on EC2 instances. The value of the iam_role currently does not matter because EC2 supports only a single IAM role per instance.
env_credentials Use credentials set by environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_KEY. The value of env_credentials does not matter: the setting serves as a flag to enable the use of environment variables for credentials.
Queues can be configured using short names as well as full URLs
request_queue_name response_queue_name The short name of the request and response queues. The queue is required to be in the same account as the credentials used. The endpoint is configured by the global (not section-specific) setting sqs_endpoint. If the queues do not exist, then ServiceBloxserviceblox will attempt to create them.
request_queue_url response_queue_url Full request and response urls. The SQS urls have the form https://sqs.us-east-1.amazonaws.com/accountid/queuename.

7.7.2.3. Deploying Multiple Servers with Queues

There are a few things you should know when you run a multi-server deployment that communicates via queues.

The first is that each ServiceBlox server should typically only have one queue configuration section. A typical mistake is to have the same lb-web-server.config file deployed on each server which contains all the queues. Since the primary reason to have a multi-server deployment is to partition data, having all servers process messages from all queues will cause problems b/c it's quite likely that the server that pulls the message off the queue will not contain the data necessary to process it. Another somewhat comment deployment scenario is to have different services hosted on different servers. Since the entire HTTP message is placed on the queue, including the endpoint, it's likely that the server that processes the message will not contain the service you are trying to invoke. Hence the rule of thumb of only one queue configuration per server.

A second thing to note is that tabular service transactions are currently limited to a single server. This is true for both queue and TCP transports but since you technically can process multiple queues on one server, it's less obvious that your transaction is crossing server boundaries. What this means is that if you have a transaction that includes imports on two queues on different servers then that transaction will never complete. The solution to this problem is to batch your imports into transactions per queue, much the same as you would by IP address or host name in the TCP case.

7.8. CORS Rules

Cross-origin resource sharing (CORS) defines a way for client web applications that are loaded in one domain to interact with resources in a different domain. With CORS support in ServiceBlox, you can build rich client-side web applications and selectively allow cross-origin access to your ServiceBlox services.

For ServiceBlox and LogiQL, CORS essentially allows a rich web application application running in one server, to access services from a ServiceBlox server on a different server.

To enable CORS on ServiceBlox, we create CORS rules. These rules fundamentally declare the origins that will be allowed to access a particular service. The following LogiQL rule declares a CORS rule that, when applied to a URL prefix, will allow remote calls from the origin http://foo.com for GET requests to the service with URL prefix of /foo-service.

cors:rule_by_name["allow_foo_GET"] = r,
  cors:rule(r) {
    cors:allowed_origin("http://foo.com"),
    // prefixes
    cors:prefix("/foo-service"),
    cors:prefix("/bar-service")
  }.

It is also possible to configure origins using wildcards, other HTTP methods, and allow for non-standard headers. Here are the predicates that allow for this:

  • allowed_origin(r, origin): origin can be a string with one wildcard, such as '*foo', 'foo*', or 'foo*bar'. The '*' string will allow all origins;

  • allowed_method(r, method): method should be a string describing an HTTP method ('GET', 'POST', 'PUSH', etc);

  • allowed_header(r, header): header should be the name of an http request header that the client will be allowed to send.

The following rule samples how to use these predicates as well as how to configure a CORS rule for all services starting with /foo by using a wildcard in the prefix:

cors:rule_by_name["allow_PUSH_POST"] = r,
cors:rule(r) {
  cors:allowed_origin("http://*foo"),
  cors:allowed_method("GET"),
  cors:allowed_method("PUSH"),
  cors:allowed_header("Header1"),
  cors:allowed_header("Header2"),
  cors:prefix("/foo/*")
}
  <- .

7.9. Using URL path and parameters in Web Services

Each service is associated with a prefix and ServiceBlox will use this prefix to match a request with a service for this request, however the URL can encode additional information which can be used by web services.

When ServiceBlox handles a request, it pulses a message in the database which contains this information, as encoded in the lb.web.http protobuf protocol:

message Field
{
  required string name = 1;
  repeated string value = 2;
}

message RequestURI
{
  // Full original uri. This contains the query string.
  required string full = 1;

  // Prefix of the service that is invoked.
  required string service_prefix = 2;

  // Best attempt to provide a subset of the path that indicates the
  // specific request made to the service.
  optional string extra_path = 3;

  // Parsed version of the query string
  repeated Field parameter = 4;
}

  

This can be used to implement behavior which depends on the URL.

    +log(datetime:string:convert[now]+":prefix:"+pref)<-    
        datetime:now(now),
        +lb:web:http:Request_uri[req] = uri,
        +lb:web:http:RequestURI_service_prefix[uri] =pref.
    

This rule for instance will insert a fact in a log predicate for each service call, with a string including the date of the call and the prefix which the URL was matched against. Similarly one could log the full URL, the URL parameters ( not that these are a list of Fields, each of which associates a parameter name which a list of parameter values, all encoded as string.)

    // Logging full URL
    +log(datetime:string:convert[now]+":full:"+pref)<-    
        datetime:now(now),
        +lb:web:http:Request_uri[req] = uri,
        +lb:web:http:RequestURI_full[uri] =pref.

    // Logging parameters    
    +log(datetime:string:convert[now]+":param: "+name+":"+int:string:convert[ind]+":"+value)<-    
        datetime:now(now),
        +lb:web:http:Request_uri[req] = uri,
        +lb:web:http:RequestURI_parameter[uri,n] = param,
        +lb:web:http:Field_name[param] = name,
        +lb:web:http:Field_value[param,ind] = value.
 

If the prefix ends with a slash followed by a wildcard character (/*), ServiceBlox will match any URL whose path starts with this prefix. If it does not, ServiceBlox will match any URL whose path is this prefix. For prefix including a wildcard, ServiceBlox will populate the extra_path as the part of the path following the matching prefix. e.g. if a service uses the prefix /time/*, ServiceBlox will match the URL http://localhost/time/EST with this service and populate extra_path with the string "EST".

    +log(datetime:string:convert[now]+":extra_path:"+pref)<-    
        datetime:now(now),
        +lb:web:http:Request_uri[req] = uri,
        +lb:web:http:RequestURI_extra_path[uri] =pref.
 

Chapter 8. Using LogicBlox Services with Systemd

8.1. Introduction

Systemd is an initialization system being standardized on by many Linux distributions (i.e Debian 7 and 8, Ubuntu 15.04 and newer, CentOS 7). It handles bootstrapping the operating system as well as controlling application services. LogicBlox deployments use systemd to control the core lb-server, lb-compiler, and lb-web-server services needed by LogicBlox applications. Systemd provides a robust infrastructure and set of tools for service management, typically replacing adhoc service-specific shell scripts that aren’t nearly as stable or standardized. This chapter describes how systemd can be used to replace the old “lb services” utilities with a more robust and powerful mechanism.

8.2. Basics

Systemd uses configuration files to define service units and resources needed by those services, as well as conditions that start and stop services. Systemd also manages dependencies between service units and their resources. Configuration files are named with the name of the service followed by an extension that indicates the type of resource defined in the file. For example, lb-server.service, lb-server.socket, and lb-server-admin.socket. Files that end in “*.target” are typically used to group a related set of service units or resources. For example, lb.target groups the lb-server, lb-web-server, and lb-compiler services. Service configuration files installed by an application are located in the /lib/systemd/system/ directory. Settings in these files can be overridden by putting files into the /etc/systemd/system/ or /etc/systemd/user/ directories. See systemd documentation for details.

The systemctl command line utility is the main interface used to interact with services. Commonly used systemctl commands include start, stop, restart, and status. For more detail, see the systemctl man page or examples of running LogicBlox services below.

Systemd services can run at root or user level. Root level services are typically used to start services available to all users at boot time while user level services are typically started and stopped manually by a particular user account for more transient use cases. Note that the systemctl utility needs to be run as root to control and query root level services. Manipulating user level services requires the "--user" option to be passed to systemctl.

Various Linux distributions are slowly adopting systemd. The Ubuntu 15.04 release is the first Ubuntu version with full support for systemd. You can find packages that add some systemd support Ubuntu 14, but there have been instances of it not running properly on some machine configurations. Plus it does not contain the systemctl utility which makes it much more difficult to control services. The examples in this document have been tested with Ubuntu 16.04.

8.3. LogicBlox Service Configuration

8.3.1. Config Files

A LogicBlox distribution contains a set of systemd configuration files for LogicBlox lb-server, lb-web-server, and lb-compiler services. The configuration files for these services are in the lib/systemd/ directory of a LogicBlox distribution. The system/ subdirectory contains files for root level services and the user/ subdirectory contains files for user level services. The main difference between the two sets of files is the location of the $LB_DEPLOYMENT_HOME directory. The user/ files assume this has been set outside the systemd configuration files to point to the directory containing LogicBlox database files, log files, and local configuration files. The system/ files set $LB_DEPLOYMENT_HOME to /var/db/logicblox, assuming this directory will contain all LogicBlox files. When installing a LogicBlox binary tarball, both sets of configuration files assume that the LogicBlox distribution is installed in /opt/logicblox, or /opt/logicblox is set up as a symbolic link to the installation. For a nixops deployment, paths in the config files will be automatically set to locations of the appropriate binaries in the nix store.

The set of service configuration files includes:

  • lb-compiler.service : configuration for the lb-compiler service
  • lb-compiler.socket : configuration for the lb-compiler socket used for normal requests
  • lb-compiler-admin.socket : configuration for the lb-compiler socket used for administrative requests
  • lb-server.service : configuration for the lb-server service
  • lb-server.socket : configuration for the lb-server socket used for normal requests
  • lb-server-admin.socket : configuration for the lb-server socket used for administrative requests
  • lb-web-server.service : configuration for the lb-web-server service
  • lb-web-server.socket : configuration for the lb-web-server socket used for normal requests
  • lb-web-server-internal.socket : configuration for the lb-web-server socket used for administrative request
  • lb.target : groups the three LogicBlox services to allow them to be controlled together

8.3.2. Socket Configuration

Starting with release 4.4.8, the LogicBlox services use systemd socket activation. Earlier versions would start all the service processes at once instead of starting them dynamically as requests were made. With socket activated services, systemd creates the initial sockets needed by the services and the service processes are not started until the first request is sent to one of the service’s sockets. If you look into the lb.target file, you will see that it only lists the “*.socket” files in the Wants property instead of the “*.service” files. The ListenStream property in the “*.socket” files defines that socket port waiting to receive requests. These ports are also defined in lb-server.config and lb-web-server.config files (default files are in $LOGICBLOX_HOME/config/). Currently you must manually ensure that port configurations are consistent between the systemd and LogicBlox config files. The deployed files consistently use standard LogicBlox ports. If you change any of the ports, you need to change them for both the LogicBlox and systemd config files.

The lb-server, lb-compiler, and lb-web-server services all use two ports each, one for normal requests and a second for administrative requests. The lb-web-service can be configured to use more than these two ports. Some specific web services can be hosted on different ports, for example. If you configure additional ports for lb-web-server, you need to decide if these ports should participate in socket activation of the lb-web-server service. Normally you would want all ports configured to activate the service. To do so, you will need to create additional “*.socket” files for each port (copy from one of the existing lb-web-server*.socket files) and include them in the Wants property of the lb.target file. These ports also need to be defined in the lb-web-server.config file.

For releases prior to 4.8.0, the lb-web-server service needs to be told that the service is being started by systemd socket activation. This is not necessary with 4.8.0 and later releases. Prior to 4.8.0, add inherited=true to the tcp:public and tcp:internal sections of the lb-web-server.config file. Also set inherited=true for any service-specific ports defined for an application. For example:

[tcp:public]
inherited = true

[tcp:internal]
inherited = true

8.3.3. Timeouts

Systemd has a startup timeout for services, typically defaulted to 90 seconds. If it takes longer than the specified timeout for a service to start, systemd will kill the process and try to restart it. It can sometimes take a bit of time for the LogicBlox lb-web-server process to start if many workspaces exist and have to be scanned for service configurations. To increase the timeout for the lb-web-server service, add a TimeoutStartSec parameter to the [Service] section of your lb-web-server.service configuration file. This parameter's value is a unit-less value in seconds, or a time span value such as "5min 20s". It can also be set to "infinity" to disable the timeout. This should be set to the same value you use for the startup_timeout parameter in lb-web-server.config. Note that both startup_timeout in lb-web-server.config and TimeoutStartSec in lb-web-server.service must be set to large enough values to prevent either systemd or the lb-web-server process itself from timing out.

Commands like the following can be used to see what timeout values are in the lb-web-server systemd configuration files:

# for root services
systemctl cat lb-web-server.service
systemctl show lb-web-server -a | grep -i timeout

# for user services
systemctl --user cat lb-web-server.service
systemctl --user show lb-web-server -a | grep -i timeout

When using nixops to manage application deployments, the systemd.service.lb-web-server.serviceConfig.TimeoutStartSec property can be used to set the systemd TimeoutStartSec parameter for the lb-web-server service.

8.4. Logging

The old “lb services” service management utilities provided by LogicBlox created log files for each service in the $LB_DEPLOYMENT_HOME/logs/current/ directory. By default, LogicBlox systemd services use system journaling managed by journalctl. The lb-server and lb-web-server processes can be configured to write to the old $LB_DEPLOYMENT_HOME/logs/current/ log files by setting the "systemd=true" property in the "[global]" section of the lb-web-server.config file or in the "[logging]" section of the lb-server.config file.

To access the system journaling logs, use the journalctl utility. Note that you need to run the journalctl utility from the same user account that started the systemd services. Log files for root level services can be accessed by running journalctl as root. See the journalctl man page for details, but here are some useful examples to get you started (remove "sudo" from in front of the commands to see logs for user level services).

$ sudo journalctl -f
    # interactively display all logging messages in the 
    # current shell as they arrive
$ sudo journalctl -u lb-server
    # display all log messages from the lb-server service
$ sudo journalctl -u lb-compiler
    # display all log messages from the lb-compiler service
$ sudo journalctl -f -u lb-web
    # interactively display all logging messages for the
    # lb-web-server service in the current shell as they arrive
$ sudo journalctl --since today
    # prints the log for all units, starting today.
$ journalctl -p err
    # prints only the message with priority error.

8.5. Installing LogicBlox Services

8.5.1. Distribution Location

Systemd requires absolute paths to the binaries for services. The config files in a LogicBlox binary tarball distribution assume that the distribution has been unpacked in /opt/logicblox or that a symbolic link has been created that points /opt/logicblox to the unpacked distribution directory. For example:

$ sudo ln -sfn /path/to/unpacked/logicblox/distribution /opt/logicblox

Either create this symbolic link (recommended) or manually edit all the *.service files in lib/systemd/user/ and lib/systemd/system/, changing the ExecStart and ExecStop variables to point to your distribution directory.

8.5.2. Database Location

If you will be running root level services you need to create the /var/db/logicblox directory or create a symbolic link from it to another directory that will contain all LogicBlox database files (this is recommended). You could also choose to edit the LB_DEPLOYMENT_HOME environment variable set in the lib/systemd/system/lb-server.service config file.

For user level services, LogicBlox database files will be created in the directory pointed to by the LB_DEPLOYMENT_HOME environment variable in the shell used to start the services. See the next section for more detail on environment variables.

8.5.3. Environment Variables

Before installing and starting LogicBlox services with systemd, you need to make sure the environment variables from the /opt/logicblox/etc/profile/logicblox.sh file are initialized. You can do this manually for user level services by sourcing the logicblox.sh file and using the import-environment systemctl command to cause systemd to transfer the local environment variables to the user level systemd processes.

$ source /opt/logicblox/etc/profile/logicblox.sh
$ systemctl --user import-environment

This needs to be done in all shells from which you'll be controlling the services or running LogicBlox commands.

To set the environment variables for all users or for root level services, create an /etc/profile.d/logicblox.sh file that contains

  source /opt/logicblox/etc/profile.d/logicblox.sh

and reboot the server or at least start a new shell.

8.5.4. Deploy Service Config Files

Copy configuration files from the LogicBlox distribution into systemd directories and tell systemd to update any cached service configuration information. After any changes are made to service configuration files in /lib/systemd/system/, /etc/systemd/system/, or /etc/systemd/user/ directories, the “systemctl daemon-reload” command must be executed to notify systemd of the changes.

To set up the root level services:

$ sudo cp /opt/logicblox/lib/systemd/system/* /etc/systemd/system/.
$ sudo systemctl daemon-reload
$ sudo systemctl status lb.target lb-compiler.service lb-server.service \
                        lb-web-server.service

For user level services (note the use of the --user systemctl option):

$ sudo cp /opt/logicblox/lib/systemd/user/* /etc/systemd/user/.
$ systemctl --user daemon-reload
$ systemctl --user status lb.target lb-compiler.service lb-server.service \
                          lb-web-server.service

The status command should report that all services are "loaded" but "inactive".

8.5.5. Start the Service

Start a second shell and run journalctl to see any log messages from the services. For root level services

$ sudo journalctl -f

and for user level services

$ journalctl -f

Next start all the LogicBlox services. Note that due to the systemd socket activation used by these services, the status command will report that lb.target is “active” but all others will still be inactive. For root level services:

$ sudo systemctl start lb.target
$ sudo systemctl status lb.target lb-compiler.service lb-server.service \
                        lb-web-server.service

and for user level services:

$ systemctl --user start lb.target
$ systemctl --user status lb.target lb-compiler.service lb-server.service \
                          lb-web-server.service

You should see messages in the journalctl log indicating that systemd has started listening on all lb-server, lb-compiler, and lb-web-server ports. Something like the following:

Aug 18 13:35:27 my-server systemd[1]: Listening on LogicBlox Web Server Daemon Port.
Aug 18 13:35:27 my-server systemd[1]: Listening on LogicBlox Compiler Daemon Port.
Aug 18 13:35:27 my-server systemd[1]: Listening on LogicBlox Compiler Daemon Admin Port.
Aug 18 13:35:27 my-server systemd[1]: Listening on LogicBlox Web Server Internal Daemon Port.
Aug 18 13:35:27 my-server systemd[1]: Listening on LogicBlox ConnectBlox Port.
Aug 18 13:35:27 my-server systemd[1]: Listening on LogicBlox ConnectBlox Admin Port.
Aug 18 13:35:27 my-server systemd[1]: Reached target All LogicBlox Services.

8.5.6. Testing the Services

Since the LogicBlox services use socket activation, the systemctl status command and journalctl logs may report no problems at this point but there still may be configuration problems encountered when individual services are started. Creating a new workspace should start the lb-server and lb-compiler services. You should see messages in the journalctl log indicating that these services have started and the “systemctl status” command should report them as “active”.

# create a new workspace and add a predicate declaration to test 
# lb-server and lb-compiler
$ lb create /test
$ lb addblock /test 'sku(s), sku_id(s:id) -> string(id).'
$ lb list /test

# check service status for root level services
$ sudo systemctl status lb-compiler.service lb-server.service

# or check service status for user level services
$ systemctl --user status lb-compiler.service lb-server.service

To trigger the lb-web-server service to start and test it, use the following command to list all registered LogicBlox web services. You will see messages in the journalctl log when the services starts up and the status command will report “active”.

$ lb web-server list -s

# for root level services
$ sudo systemctl status lb-web-server.service

# or for user level services
$ systemctl --user status lb-web-server.service

If there are any problems starting the services, detailed messages should appear in the journalctl log.

8.5.7. Stopping Services

To stop any LogicBlox services that are running, use the “systemctl stop” command. For root level services:

$ sudo systemctl stop lb.target

and for user level services:

$ systemctl --user stop lb.target

8.5.8. Starting Services at Boot Time

To start the LogicBlox systemd services automatically at boot time, follow the steps above to deploy root level service config files, then execute the following:

$ sudo systemctl enable lb.target
$ sudo ln -sfn /etc/systemd/system/lb.target \
               /etc/systemd/system/multi-user.target.wants/

8.6. Developer Notes

For LogicBlox developers, user level systemd services work much the same as the old lb services scripts. Use these notes to guide you in managing LB services with systemd.

  • Set up all the LB environment variables as you would to run "lb services start".
  • Use the systemctl import-environment command to copy all your local environment variables into the systemd user level service processes.
    $ systemctl --user import-environment
  • Link the systemd config files so systemd can find them. For local lb-database builds, the LB systemd configuration files will be located in a lib/systemd/ subdirectory of the installation prefix for lb-database. See the prefix variable in lb-database/Makefile.conf for the installation location you are using. This prefix will also be used in the systemd config files to locate the binaries used to start and stop the services. For lb-server and lb-compiler services, you will just need to build and install lb-database. Use commands like the following to copy the config files into systemd directories:
    $ sudo mkdir -p /etc/systemd/user
    $ sudo ln -sfn $prefix/logicblox/lib/systemd/user/* /etc/systemd/user/
         # where $prefix is the lb-database prefix in Makefile.conf
    $ systemctl --user daemon-reload
  • Start and stop user level services with systemctl. Logs will be in journalctl. Note that since the services use socket activation, you may not see any errors in the logs until you actually use the service for the first time.
    # start LB services
    $ systemctl --user start lb.target
    
    # stop LB services
    $ systemctl --user start lb.target
  • Use systemctl status command as a replacement for "lb services status".
    $ systemctl --user status lb.target lb-compiler.service lb-server.service \
                              lb-web-server.service
  • Don't forget to run "systemctl --user daemon-reload" any time you make changes to the systemd config files for the services.
  • For my local dev environment, lb-web stuff is deployed in a different directory ($LB_INSTALL_DIR/lb-web) than $LOGICBLOX_HOME . To get lb-web-server to start, I had to comment out this line from the lib/systemd/user/lb-web-server.service file, then reload service configs and restart services (shouldn't have to do this if using LB release package):
    Environment=LB_WEBSERVER_HOME=/home/weshunter/dev/LB4/deploy/logicblox
  • Old lb-web-server.config and lb-server.config files support a “systemd” property that was used in the past to indicate that the services were running within systemd. This property is deprecated and should no longer be used. It has been supplanted with an IN_SYSTEMD environment variable that is set in the systemd service configuration files.
  • Managing Multiple LogicBlox Versions
    • If you're working with multiple locally built LB versions, switching between them should be a simple matter of stopping the systemd services in one shell, switching to another shell, relinking the config files in /etc/systemd/user/, reloading the daemon, using the systemctl import-environment command, and restarting services with systemctl.
      # from one LB environment (shell)
      $ systemctl --user stop lb.target
      
      # switch to another LB environment (shell)
      # assuming logicblox.sh is already sourced in this shell
      $ sudo ln -sfn /my/lb/dist/dir/lib/systemd/user/* /etc/systemd/user/
      $ systemctl --user daemon-reload
      $ systemctl --user import-environment
      $ systemctl --user start lb.target
      
    • If you're working with multiple LB distribution packages, these packages all come with systemd config files that assume binaries are in /opt/logicblox/. You can either manually edit all the *.service files in your distribution directories to change the paths used by the ExecStart variables, or change an /opt/logicblox/ link when you switch environments:
      # from one LB environment (shell)
      $ systemctl --user stop lb.target
      
      # switch to another LB environment (shell)
      # assuming logicblox.sh is already sourced in this shell
      $ sudo ln -sfn /lb/dist/directory /opt/logicblox
      $ sudo ln -sfn /opt/logicblox/lib/systemd/user/* /etc/systemd/user/
      $ systemctl --user daemon-reload
      $ systemctl --user import-environment
      $ systemctl --user start lb.target
      

Part II. Appendix

Chapter 9. Linux Memory Analysis

This chapter is for anyone wishing to monitor memory usage of processes on Linux, and describes some of the issues that are encountered when trying to get a measure of how much memory a process is actually consuming.

9.1. Issues

Shared memory, swapping and memory-mapped files all make it difficult to accurately assess how much memory a process is actually using. Running “top” for instance, usually shows this kind of output:

top - 13:19:53 up 12 min,  3 users,  load average: 0.14, 0.36, 0.34
Tasks: 225 total,   1 running, 224 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.1 sy,  0.0 ni, 99.5 id,  0.0 wa,  0.1 hi,  0.0 si,  0.0 st
KiB Mem:  12297356 total,  3922792 used,  8374564 free,    79796 buffers
KiB Swap:  6152188 total,        0 used,  6152188 free,  2206980 cached

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  933 root      20   0  200336  69916  54876 S   0.8  0.6   0:19.34 Xorg
 4020 wsinno    20   0 6035584  47176  10876 S   0.3  0.4   0:00.59 java
 4144 wsinno    20   0 7457168 123956  13788 S   0.3  1.0   0:03.79 java
 

In this view, VIRT represents the size of the memory address space allocated by the process, which includes the size of any memory mapped file, the size of any shared memory segment that may be shared with other processes, as well as regular chunks of private memory allocated by the process, some of which may be swapped out. RES shows how much memory associated with the process is currently resident, regardless of what type of memory that is. SHR is the total amount of shared memory mapped into the process’ address space, and can be double counted with any other process also mapping the same shared memory.

9.2. Understanding the Memory Model

A process on Linux can allocate memory in several ways, and there are several ways of classifying the chunks of memory mapped into its address space. These can be broken down as shown in the following table:

Anonymous File Backed
Private malloc() ; mmap(PRIVATE, ANON); stack mmap(PRIVATE, FILE)
Shared mmap(SHARED, ANON) requires fork(); shmget mmap(SHARED, FILE)

Anonymous memory can be swapped out by the operating system to the swap partition. The same mechanism is used for file-backed memory, however, it is written directly to the backing file. Thus, anonymous memory can be in one of two states, either "resident" or "swapped".

Note that it is very common to memory map a file stored in a tmpfs filesystem, which behaves more like an anonymous mapping, since the file is stored in a filesystem that is stored in memory, and has it’s own paging to swap, i.e. the difference between truly anonymous memory and tmpfs backed memory is that in the latter case, it is the filesystem layer that takes care of the swapping.

An additional facet of memory usage in Linux is the extensive use of copy-on-write to implement lazy allocation. For instance, if a process is forked, all the memory of the original process is marked as copy-on-write, and the two processes effectively have the same amount of private anonymous memory, which would show up as such in most tools, but the actual amount of RAM resident for both processes is still only the original amount, which thus results in double counting of memory. An undercount can result from the same facility, in that a call to “malloc” will not result in any memory being used until the memory allocated to is actually written.

Shared memory is counted for each process that is using it, which means that summing up the memory usage for all processes leads to "double counting". There is no "owner" of shared memory, i.e. there is no way to say that a particular piece of shared memory should be accounted for by a given process.

Buffers for files are not accounted for in this document, since this transparently managed by the operating system.

9.3. Tools

top

Commonly used tool, provides many metrics including total address-space (VIRT), size of shared memory (SHR), and size of resident memory (RES).

ps

Also shows resident memory (RSS) with option v. Example: top -p PID v.

/proc/[pid]/statm

Provides info about mem usage (in pages). This is useful for automating measurement of values only available in top otherwise, namely shared memory SHR.

smem

Used to provide additional metrics that aren’t typically available - more expensive to run, since doing queries and computations across /proc filesystem. Includes facility to capture raw output of /proc for later analysis (smemcap).

htop

Provides a very wide variety of metrics related to memory, CPU, and I/O.

9.4. Measuring Different Types of Memory

The total address space of a process is indicated by VSZ.

Total private memory, PM equals VSZ - SHR where SHR is the total shared memory.

Total swapped memory whether in system swap file or in memory-mapped files Swap .

Swapped private memory SPM = PM - USS where USS is resident private memory.

Swapped shared memory SSHR = Swap - SPM.

Resident shared memory RSHR = SSHR - SHR.

In table form:

Resident Swapped Total
Private USS SPM PM
Shared RSHR SSHR SHR
Total RSS Swap VSZ

Note that we aren’t able to break down memory based on whether it is anonymous or file backed, we can just measure whether it is swapped or resident.