Chapter 8. Using LogicBlox Services with Systemd

8.1. Introduction

Systemd is an initialization system being standardized on by many Linux distributions (i.e Debian 7 and 8, Ubuntu 15.04 and newer, CentOS 7). It handles bootstrapping the operating system as well as controlling application services. LogicBlox deployments use systemd to control the core lb-server, lb-compiler, and lb-web-server services needed by LogicBlox applications. Systemd provides a robust infrastructure and set of tools for service management, typically replacing adhoc service-specific shell scripts that aren’t nearly as stable or standardized. This chapter describes how systemd can be used to replace the old “lb services” utilities with a more robust and powerful mechanism.

8.2. Basics

Systemd uses configuration files to define service units and resources needed by those services, as well as conditions that start and stop services. Systemd also manages dependencies between service units and their resources. Configuration files are named with the name of the service followed by an extension that indicates the type of resource defined in the file. For example, lb-server.service, lb-server.socket, and lb-server-admin.socket. Files that end in “*.target” are typically used to group a related set of service units or resources. For example, lb.target groups the lb-server, lb-web-server, and lb-compiler services. Service configuration files installed by an application are located in the /lib/systemd/system/ directory. Settings in these files can be overridden by putting files into the /etc/systemd/system/ or /etc/systemd/user/ directories. See systemd documentation for details.

The systemctl command line utility is the main interface used to interact with services. Commonly used systemctl commands include start, stop, restart, and status. For more detail, see the systemctl man page or examples of running LogicBlox services below.

Systemd services can run at root or user level. Root level services are typically used to start services available to all users at boot time while user level services are typically started and stopped manually by a particular user account for more transient use cases. Note that the systemctl utility needs to be run as root to control and query root level services. Manipulating user level services requires the "--user" option to be passed to systemctl.

Various Linux distributions are slowly adopting systemd. The Ubuntu 15.04 release is the first Ubuntu version with full support for systemd. You can find packages that add some systemd support Ubuntu 14, but there have been instances of it not running properly on some machine configurations. Plus it does not contain the systemctl utility which makes it much more difficult to control services. The examples in this document have been tested with Ubuntu 16.04.

8.3. LogicBlox Service Configuration

8.3.1. Config Files

A LogicBlox distribution contains a set of systemd configuration files for LogicBlox lb-server, lb-web-server, and lb-compiler services. The configuration files for these services are in the lib/systemd/ directory of a LogicBlox distribution. The system/ subdirectory contains files for root level services and the user/ subdirectory contains files for user level services. The main difference between the two sets of files is the location of the $LB_DEPLOYMENT_HOME directory. The user/ files assume this has been set outside the systemd configuration files to point to the directory containing LogicBlox database files, log files, and local configuration files. The system/ files set $LB_DEPLOYMENT_HOME to /var/db/logicblox, assuming this directory will contain all LogicBlox files. When installing a LogicBlox binary tarball, both sets of configuration files assume that the LogicBlox distribution is installed in /opt/logicblox, or /opt/logicblox is set up as a symbolic link to the installation. For a nixops deployment, paths in the config files will be automatically set to locations of the appropriate binaries in the nix store.

The set of service configuration files includes:

  • lb-compiler.service : configuration for the lb-compiler service
  • lb-compiler.socket : configuration for the lb-compiler socket used for normal requests
  • lb-compiler-admin.socket : configuration for the lb-compiler socket used for administrative requests
  • lb-server.service : configuration for the lb-server service
  • lb-server.socket : configuration for the lb-server socket used for normal requests
  • lb-server-admin.socket : configuration for the lb-server socket used for administrative requests
  • lb-web-server.service : configuration for the lb-web-server service
  • lb-web-server.socket : configuration for the lb-web-server socket used for normal requests
  • lb-web-server-internal.socket : configuration for the lb-web-server socket used for administrative request
  • lb.target : groups the three LogicBlox services to allow them to be controlled together

8.3.2. Socket Configuration

Starting with release 4.4.8, the LogicBlox services use systemd socket activation. Earlier versions would start all the service processes at once instead of starting them dynamically as requests were made. With socket activated services, systemd creates the initial sockets needed by the services and the service processes are not started until the first request is sent to one of the service’s sockets. If you look into the lb.target file, you will see that it only lists the “*.socket” files in the Wants property instead of the “*.service” files. The ListenStream property in the “*.socket” files defines that socket port waiting to receive requests. These ports are also defined in lb-server.config and lb-web-server.config files (default files are in $LOGICBLOX_HOME/config/). Currently you must manually ensure that port configurations are consistent between the systemd and LogicBlox config files. The deployed files consistently use standard LogicBlox ports. If you change any of the ports, you need to change them for both the LogicBlox and systemd config files.

The lb-server, lb-compiler, and lb-web-server services all use two ports each, one for normal requests and a second for administrative requests. The lb-web-service can be configured to use more than these two ports. Some specific web services can be hosted on different ports, for example. If you configure additional ports for lb-web-server, you need to decide if these ports should participate in socket activation of the lb-web-server service. Normally you would want all ports configured to activate the service. To do so, you will need to create additional “*.socket” files for each port (copy from one of the existing lb-web-server*.socket files) and include them in the Wants property of the lb.target file. These ports also need to be defined in the lb-web-server.config file.

For releases prior to 4.8.0, the lb-web-server service needs to be told that the service is being started by systemd socket activation. This is not necessary with 4.8.0 and later releases. Prior to 4.8.0, add inherited=true to the tcp:public and tcp:internal sections of the lb-web-server.config file. Also set inherited=true for any service-specific ports defined for an application. For example:

[tcp:public]
inherited = true

[tcp:internal]
inherited = true

8.3.3. Timeouts

Systemd has a startup timeout for services, typically defaulted to 90 seconds. If it takes longer than the specified timeout for a service to start, systemd will kill the process and try to restart it. It can sometimes take a bit of time for the LogicBlox lb-web-server process to start if many workspaces exist and have to be scanned for service configurations. To increase the timeout for the lb-web-server service, add a TimeoutStartSec parameter to the [Service] section of your lb-web-server.service configuration file. This parameter's value is a unit-less value in seconds, or a time span value such as "5min 20s". It can also be set to "infinity" to disable the timeout. This should be set to the same value you use for the startup_timeout parameter in lb-web-server.config. Note that both startup_timeout in lb-web-server.config and TimeoutStartSec in lb-web-server.service must be set to large enough values to prevent either systemd or the lb-web-server process itself from timing out.

Commands like the following can be used to see what timeout values are in the lb-web-server systemd configuration files:

# for root services
systemctl cat lb-web-server.service
systemctl show lb-web-server -a | grep -i timeout

# for user services
systemctl --user cat lb-web-server.service
systemctl --user show lb-web-server -a | grep -i timeout

When using nixops to manage application deployments, the systemd.service.lb-web-server.serviceConfig.TimeoutStartSec property can be used to set the systemd TimeoutStartSec parameter for the lb-web-server service.

8.4. Logging

The old “lb services” service management utilities provided by LogicBlox created log files for each service in the $LB_DEPLOYMENT_HOME/logs/current/ directory. By default, LogicBlox systemd services use system journaling managed by journalctl. The lb-server and lb-web-server processes can be configured to write to the old $LB_DEPLOYMENT_HOME/logs/current/ log files by setting the "systemd=true" property in the "[global]" section of the lb-web-server.config file or in the "[logging]" section of the lb-server.config file.

To access the system journaling logs, use the journalctl utility. Note that you need to run the journalctl utility from the same user account that started the systemd services. Log files for root level services can be accessed by running journalctl as root. See the journalctl man page for details, but here are some useful examples to get you started (remove "sudo" from in front of the commands to see logs for user level services).

$ sudo journalctl -f
    # interactively display all logging messages in the 
    # current shell as they arrive
$ sudo journalctl -u lb-server
    # display all log messages from the lb-server service
$ sudo journalctl -u lb-compiler
    # display all log messages from the lb-compiler service
$ sudo journalctl -f -u lb-web
    # interactively display all logging messages for the
    # lb-web-server service in the current shell as they arrive
$ sudo journalctl --since today
    # prints the log for all units, starting today.
$ journalctl -p err
    # prints only the message with priority error.

8.5. Installing LogicBlox Services

8.5.1. Distribution Location

Systemd requires absolute paths to the binaries for services. The config files in a LogicBlox binary tarball distribution assume that the distribution has been unpacked in /opt/logicblox or that a symbolic link has been created that points /opt/logicblox to the unpacked distribution directory. For example:

$ sudo ln -sfn /path/to/unpacked/logicblox/distribution /opt/logicblox

Either create this symbolic link (recommended) or manually edit all the *.service files in lib/systemd/user/ and lib/systemd/system/, changing the ExecStart and ExecStop variables to point to your distribution directory.

8.5.2. Database Location

If you will be running root level services you need to create the /var/db/logicblox directory or create a symbolic link from it to another directory that will contain all LogicBlox database files (this is recommended). You could also choose to edit the LB_DEPLOYMENT_HOME environment variable set in the lib/systemd/system/lb-server.service config file.

For user level services, LogicBlox database files will be created in the directory pointed to by the LB_DEPLOYMENT_HOME environment variable in the shell used to start the services. See the next section for more detail on environment variables.

8.5.3. Environment Variables

Before installing and starting LogicBlox services with systemd, you need to make sure the environment variables from the /opt/logicblox/etc/profile/logicblox.sh file are initialized. You can do this manually for user level services by sourcing the logicblox.sh file and using the import-environment systemctl command to cause systemd to transfer the local environment variables to the user level systemd processes.

$ source /opt/logicblox/etc/profile/logicblox.sh
$ systemctl --user import-environment

This needs to be done in all shells from which you'll be controlling the services or running LogicBlox commands.

To set the environment variables for all users or for root level services, create an /etc/profile.d/logicblox.sh file that contains

  source /opt/logicblox/etc/profile.d/logicblox.sh

and reboot the server or at least start a new shell.

8.5.4. Deploy Service Config Files

Copy configuration files from the LogicBlox distribution into systemd directories and tell systemd to update any cached service configuration information. After any changes are made to service configuration files in /lib/systemd/system/, /etc/systemd/system/, or /etc/systemd/user/ directories, the “systemctl daemon-reload” command must be executed to notify systemd of the changes.

To set up the root level services:

$ sudo cp /opt/logicblox/lib/systemd/system/* /etc/systemd/system/.
$ sudo systemctl daemon-reload
$ sudo systemctl status lb.target lb-compiler.service lb-server.service \
                        lb-web-server.service

For user level services (note the use of the --user systemctl option):

$ sudo cp /opt/logicblox/lib/systemd/user/* /etc/systemd/user/.
$ systemctl --user daemon-reload
$ systemctl --user status lb.target lb-compiler.service lb-server.service \
                          lb-web-server.service

The status command should report that all services are "loaded" but "inactive".

8.5.5. Start the Service

Start a second shell and run journalctl to see any log messages from the services. For root level services

$ sudo journalctl -f

and for user level services

$ journalctl -f

Next start all the LogicBlox services. Note that due to the systemd socket activation used by these services, the status command will report that lb.target is “active” but all others will still be inactive. For root level services:

$ sudo systemctl start lb.target
$ sudo systemctl status lb.target lb-compiler.service lb-server.service \
                        lb-web-server.service

and for user level services:

$ systemctl --user start lb.target
$ systemctl --user status lb.target lb-compiler.service lb-server.service \
                          lb-web-server.service

You should see messages in the journalctl log indicating that systemd has started listening on all lb-server, lb-compiler, and lb-web-server ports. Something like the following:

Aug 18 13:35:27 my-server systemd[1]: Listening on LogicBlox Web Server Daemon Port.
Aug 18 13:35:27 my-server systemd[1]: Listening on LogicBlox Compiler Daemon Port.
Aug 18 13:35:27 my-server systemd[1]: Listening on LogicBlox Compiler Daemon Admin Port.
Aug 18 13:35:27 my-server systemd[1]: Listening on LogicBlox Web Server Internal Daemon Port.
Aug 18 13:35:27 my-server systemd[1]: Listening on LogicBlox ConnectBlox Port.
Aug 18 13:35:27 my-server systemd[1]: Listening on LogicBlox ConnectBlox Admin Port.
Aug 18 13:35:27 my-server systemd[1]: Reached target All LogicBlox Services.

8.5.6. Testing the Services

Since the LogicBlox services use socket activation, the systemctl status command and journalctl logs may report no problems at this point but there still may be configuration problems encountered when individual services are started. Creating a new workspace should start the lb-server and lb-compiler services. You should see messages in the journalctl log indicating that these services have started and the “systemctl status” command should report them as “active”.

# create a new workspace and add a predicate declaration to test 
# lb-server and lb-compiler
$ lb create /test
$ lb addblock /test 'sku(s), sku_id(s:id) -> string(id).'
$ lb list /test

# check service status for root level services
$ sudo systemctl status lb-compiler.service lb-server.service

# or check service status for user level services
$ systemctl --user status lb-compiler.service lb-server.service

To trigger the lb-web-server service to start and test it, use the following command to list all registered LogicBlox web services. You will see messages in the journalctl log when the services starts up and the status command will report “active”.

$ lb web-server list -s

# for root level services
$ sudo systemctl status lb-web-server.service

# or for user level services
$ systemctl --user status lb-web-server.service

If there are any problems starting the services, detailed messages should appear in the journalctl log.

8.5.7. Stopping Services

To stop any LogicBlox services that are running, use the “systemctl stop” command. For root level services:

$ sudo systemctl stop lb.target

and for user level services:

$ systemctl --user stop lb.target

8.5.8. Starting Services at Boot Time

To start the LogicBlox systemd services automatically at boot time, follow the steps above to deploy root level service config files, then execute the following:

$ sudo systemctl enable lb.target
$ sudo ln -sfn /etc/systemd/system/lb.target \
               /etc/systemd/system/multi-user.target.wants/

8.6. Developer Notes

For LogicBlox developers, user level systemd services work much the same as the old lb services scripts. Use these notes to guide you in managing LB services with systemd.

  • Set up all the LB environment variables as you would to run "lb services start".
  • Use the systemctl import-environment command to copy all your local environment variables into the systemd user level service processes.
    $ systemctl --user import-environment
  • Link the systemd config files so systemd can find them. For local lb-database builds, the LB systemd configuration files will be located in a lib/systemd/ subdirectory of the installation prefix for lb-database. See the prefix variable in lb-database/Makefile.conf for the installation location you are using. This prefix will also be used in the systemd config files to locate the binaries used to start and stop the services. For lb-server and lb-compiler services, you will just need to build and install lb-database. Use commands like the following to copy the config files into systemd directories:
    $ sudo mkdir -p /etc/systemd/user
    $ sudo ln -sfn $prefix/logicblox/lib/systemd/user/* /etc/systemd/user/
         # where $prefix is the lb-database prefix in Makefile.conf
    $ systemctl --user daemon-reload
  • Start and stop user level services with systemctl. Logs will be in journalctl. Note that since the services use socket activation, you may not see any errors in the logs until you actually use the service for the first time.
    # start LB services
    $ systemctl --user start lb.target
    
    # stop LB services
    $ systemctl --user start lb.target
  • Use systemctl status command as a replacement for "lb services status".
    $ systemctl --user status lb.target lb-compiler.service lb-server.service \
                              lb-web-server.service
  • Don't forget to run "systemctl --user daemon-reload" any time you make changes to the systemd config files for the services.
  • For my local dev environment, lb-web stuff is deployed in a different directory ($LB_INSTALL_DIR/lb-web) than $LOGICBLOX_HOME . To get lb-web-server to start, I had to comment out this line from the lib/systemd/user/lb-web-server.service file, then reload service configs and restart services (shouldn't have to do this if using LB release package):
    Environment=LB_WEBSERVER_HOME=/home/weshunter/dev/LB4/deploy/logicblox
  • Old lb-web-server.config and lb-server.config files support a “systemd” property that was used in the past to indicate that the services were running within systemd. This property is deprecated and should no longer be used. It has been supplanted with an IN_SYSTEMD environment variable that is set in the systemd service configuration files.
  • Managing Multiple LogicBlox Versions
    • If you're working with multiple locally built LB versions, switching between them should be a simple matter of stopping the systemd services in one shell, switching to another shell, relinking the config files in /etc/systemd/user/, reloading the daemon, using the systemctl import-environment command, and restarting services with systemctl.
      # from one LB environment (shell)
      $ systemctl --user stop lb.target
      
      # switch to another LB environment (shell)
      # assuming logicblox.sh is already sourced in this shell
      $ sudo ln -sfn /my/lb/dist/dir/lib/systemd/user/* /etc/systemd/user/
      $ systemctl --user daemon-reload
      $ systemctl --user import-environment
      $ systemctl --user start lb.target
      
    • If you're working with multiple LB distribution packages, these packages all come with systemd config files that assume binaries are in /opt/logicblox/. You can either manually edit all the *.service files in your distribution directories to change the paths used by the ExecStart variables, or change an /opt/logicblox/ link when you switch environments:
      # from one LB environment (shell)
      $ systemctl --user stop lb.target
      
      # switch to another LB environment (shell)
      # assuming logicblox.sh is already sourced in this shell
      $ sudo ln -sfn /lb/dist/directory /opt/logicblox
      $ sudo ln -sfn /opt/logicblox/lib/systemd/user/* /etc/systemd/user/
      $ systemctl --user daemon-reload
      $ systemctl --user import-environment
      $ systemctl --user start lb.target