2. Installation

Lenses for Apache Kafka is available to download as a platform specific archive. Linux (64bit), macOS and Windows (64bit) platforms are supported. More installation options will be available in the future.

Download links for the software will be sent via email to the individuals who have registered for a multi broker setup. You can request your copy today.

2.1. Requirements

Lenses has a direct dependency on a deployment of Apache Kafka deployment from version 0.10.2 onwards. Whilst earlier versions of Kafka may work, there will be no official support.

3GB of free RAM and about 300MB of disk space is recommended. Lenses will work with less memory but performance may suffer. There is no minimum processor requirement although the embedded JRE runs only on 64bit Operating Systems. More memory and a better CPU might be needed especially for multiple Lenses SQL queries running concurrently.

JVM options can be set via the LENSES_HEAP_OPTS environment variable. For example to increase the heap space:

export LENSES_HEAP_OPTS="-Xmx3g -Xms512m"

It is recommended that Lenses is deployed in the same data center as the Kafka cluster it is monitoring.

2.1.1. Kafka

Lenses has been tested with the latest Kafka releases (i.e. 0.11.0 and 0.10.2). Releases from Apache Kafka and Confluent are both endorsed.

The configured Kafka cluster must be accessible from the host running Lenses. Think of it as a typical producer or consumer, which should have access to all the brokers in the cluster and the Zookeeper ensemble.

Confluent’s, Apache licensed, Schema Registry and Kafka Connect are first class citizens of Lenses, although their presence is optional.

2.1.1.1. Payload Formats

Lenses supports multiple type of Kafka message payloads. Json and Avro support are first class citizens, but Lenses can also work with String, Long, Int or binary content. If Schema Registry is used, Lenses provides easy schema management and more.

2.1.1.2. JMX

To leverage Lenses full monitoring capabilities we advise you to enable JMX for the Kafka components and Zookeeper The procedure is straightforward; it is enough to export the environment variable JMX_PORT=[PORT] for each software instance (broker, zookeeper node, schema registry, connect worker), where [PORT] is the port that should be used for JMX access.

2.2. Setup

Lenses is available as a platform specific archive. There are tar.gz archives for Linux (64bit) and macOS, and a zip archive for Windows (64bit).

Once the archive is downloaded, it should be uncompressed and extracted to a location of your choice. We recommend to avoid white spaces for the installation path.

The installation directory structure will be:

lenses
├── LICENSE.txt
├── NOTICE.txt
├── bin/
├── lenses.conf
├── license.json
├── jre8u131/
├── lib/
├── licenses/
└── logs/

The file lenses.conf contains the configuration. Edit the connection details for the Kafka cluster before using the software. Other configuration options include the user accounts and roles, port, logs directory and more. A complete overview of the configurations options can be found here.

If you are on Linux, Lenses requires a slightly larger open file descriptor limit than typical desktop applications. You can check you current limit with the ulimit command:

$ ulimit -n -S # soft limit
$ ulimit -n -H # hard limit

Typically on Linux the soft limit is 1024 and can be set up to the hard limit by the user. In order to further increase the limit, superuser access is needed. A soft limit of 4096 is recommended for Lenses, though 2048 may be enough. To change the soft limit run:

$ ulimit -S -n 4096

Once the configuration in lenses.conf is complete and the open file descriptor limit has been set in case of Linux platforms, Lenses is ready to run.

To run on Linux or macOS:

$ bin/lenses

Or to explicitly set the configuration file:

$ bin/lenses lenses.conf

To run on Windows:

$ bin/lenses.bat lenses.conf

Once Lenses is running, visit the address:port set in the configuration. The default is http://localhost:9991 and login using the credentials. The default Lenses credentials are admin:admin. Always remember to change the default credentials!

To stop Lenses, press CTRL+C.

[1]The configuration file is using the HOCON format.

2.3. Quick Start

To quick start with Lenses just edit the lenses.conf file and set the following properties to your infrastructure end-points:

lenses {

  # Set the ip:port for Lenses to bind to
  ip = 0.0.0.0
  port = 9991

  # Set up infrastructure end-points

  kafka.brokers        = "PLAINTEXT://host1:9092,PLAINTEXT://host2:9092,PLAINTEXT://host3:9092"
  zookeeper.hosts      = "zkhost1:2181,zkhost2:2181,zkhost3:2181"
  schema.registry.urls = "http://schema-host:8081"
  connect = [
    {dev: "http://kafka-connect-dev-host-or-ip:8082"}
    #, {uat: "http://connect1:8082,http://connect2:8082,http://connect3:8082"}
  ]

  # Set up monitoring end-points

  jmx {
    brokers = "host1:jmx-port,host2:jmx-port,host3:jmx-port"
    schema.registry = "schema-host:jmx-port"
    zookeepers = "zkhost1:jmx-port,zkhost2:jmx-port,zkhost3:jmx-port"
    connect = [ {dev: "kafka-connect-dev-host-or-ip:jmx-port"} ]
  }

}

Then execute bin/lenses and you should be able to loging with admin/admin within moments at http://host-or-ip:9991

Note

We recommend going through Lenses Configuration to figure out how to fully configure your Lenses installation

2.4. Deployment

Lenses currently supports only manual deployment. Future releases will be enhanced with package repositories, containers, etc. Some best practice advice can be offered for installation on a server:

  1. Install Lenses under /opt.

  2. Make sure the directory is owned by the root user.

  3. Create a user specially for Lenses.

  4. Lenses configuration may be placed under /opt/lenses/lenses.conf, /etc/lenses/lenses.conf or passed in at the command line when staring Lenses.

    bin/lenses myconf.conf
    
  5. The configuration file contains information about the Lenses users’ credentials. Please make sure it is only readable by the Lenses user:

    chmod 0600 /path/to/lenses.conf
    chown [lenses-user]:root /path/to/lenses.conf
    
  6. Please adjust the logs configuration to a path for which the running Lenses process has write access to or, to stdout/stderr in case the log management should happen by the process supervisor.

If your server is on a current Linux distribution and it uses the systemd service manager, the script below can be used to start and stop Lenses with the management and supervision facilities of systemd.

[Unit]
Description=Run Landoop Lenses Service

[Service]
Restart=always
User=[LENSES-USER]
Group=[LENSES-GROUP]
LimitNOFILE=4096
WorkingDirectory=/opt/lenses
#Environment=JAVA_OPTS="-Dlogback.configurationFile=file:/etc/lenses/logback..xml"
ExecStart=/opt/lenses/bin/lenses /etc/lenses/lenses.conf

[Install]
WantedBy=multi-user.target

2.4.1. Logging

The logback.xml file can be used to control logging. By default Lenses will log into logs directory. See the logging configuration section for more information.