Lenses Installation

Note

Lenses supports Kafka versions 0.10.2+

Download

The current version of Lenses is v1.1. See Release Notes.

For those upgrading from 1.0, please see the Upgrade Path.

Archives

Lenses for Apache Kafka is available to download as a platform-specific archive.

The supported platforms are:

  • Linux (64bit),
  • macOS and
  • Windows (64bit)

We provide tar.gz archives for Linux (64bit) and macOS, and a zip archive for Windows (64bit).

Request your free trial here. The download link for the software will be sent via email to the individuals who have registered for an enterprise setup.

Containers

For Docker, Kubernetes or Openshift environments Lenses ships in a docker container. Find the details for the docker image here: https://github.com/Landoop/lenses-docker

Development Environment

If you are looking for a Lenses Development Environment (free) just download it from here

System Requirements

Hardware and software requirements may vary depending on the machine and operating system.

The minimum hardware specifications to install and effectively operate Lenses are:

  • 64bit Operating System
  • 4GB of free RAM
  • 300MB of disk space
  • Lenses is not CPU heavy, unless Lenses SQL Processors are used in Lenses IN_PROC mode.

Note: You can reduce the RAM resources below the above-specified minimum for development and testing purposes, however performance may be affected.

JVM

JVM specific options can be set via the LENSES_HEAP_OPTS environment variable. For example to increase the heap space you will have to do the following:

export LENSES_HEAP_OPTS="-Xmx6g -Xms512m"

It is recommended that Lenses is deployed in the same data center as the Kafka cluster linked to it. See more on supported Java Options at the configuration section.

Open File Descriptors

If you are running on a Linux machine, Lenses requires a slightly larger open file descriptor limit than typical desktop applications. You can check your current limit with the ulimit command:

$ ulimit -n -S # soft limit
$ ulimit -n -H # hard limit

Typically on Linux, the soft limit is 1024 and can be set up to the hard limit by the user. In order to change the limit, superuser access is needed. A soft limit of 4096 is recommended for Lenses, though 2048 may be enough. To change the soft limit run:

$ ulimit -S -n 4096

Kafka Support

Versions

Lenses has been tested with the Apache Kafka releases from version 0.10.2 upwards.

Distributions

Distributions from Apache Kafka and Confluent are both endorsed.

Connectivity

Your Apache Kafka cluster must be accessible from the host running Lenses. Think of it as a typical producer or consumer which should have access to all the brokers in the cluster and the Zookeeper ensemble. Apache Kafka Connect and Confluent’s, Apache 2 licensed, Schema Registry are fully supported however their presence is optional.

Message Payloads

Lenses supports multiple types of Kafka message payload. Json and Avro support are first class citizens, but Lenses also works with String, Long, Int or binary content. If Schema Registry is used, Lenses provides easy schema management and more.

JMX

In order to make the most out of the monitoring capabilities, Lenses requires JMX metrics to be enabled for the Kafka Brokers, Zookeepers, Schema Registry, Connect workers. To enable JMX the environment variable needs to be exported for each of the above components:

JMX_PORT=[PORT], where [PORT] is the port that should be used for the JMX access.

See JMX section for more details on enabling JMX

Quick Setup

Once the archive has been downloaded, it should be extracted to a location of your choice. We recommend avoiding white spaces for the installation path.

The directory structure is:

lenses
├── LICENSE.txt
├── NOTICE.txt
├── bin/
├── lenses.conf
├── license.json
├── jre8u131/
├── lib/
├── licenses/
└── logs/

lenses.conf file contains the application specific settings. You will need to edit the connection details for the Kafka cluster before using the software. Other configuration options include the user accounts and roles, port, logs directory and more. A complete overview of the configurations options can be found here.

Example Configuration

# Set the ip:port for Lenses to bind to
lenses.ip = 0.0.0.0
lenses.port = 9991
#lenses.jmx.port = 9992

# License file allowing connecting to up to N brokers
lenses.license.file = "license.json"

# topics created on start-up that Lenses uses to store state
lenses.topics.audits = "_kafka_lenses_audits"
lenses.topics.cluster = "_kafka_lenses_cluster"
lenses.topics.metrics = "_kafka_lenses_metrics"
lenses.topics.profiles = "_kafka_lenses_profiles"
lenses.topics.processors = "_kafka_lenses_processors"

# Set up infrastructure end-points
lenses.kafka.brokers        = "PLAINTEXT://host1:9092,PLAINTEXT://host2:9092,PLAINTEXT://host3:9092"
lenses.zookeeper.hosts      = "zkhost1:2181,zkhost2:2181,zkhost3:2181"
lenses.schema.registry.urls = "http://schema-host:8081"
lenses.connect.clusters     = [{name: "dev", url: "http://kafka-connect-host-or-ip:8083", statuses: "connect-statuses", configs: "connect-configs", offsets: "connect-offsets" }]

# Set up monitoring end-points
lenses.jmx.schema.registry = "schema-host:jmx-port"
lenses.jmx.zookeepers = "zkhost1:jmx-port,zkhost2:jmx-port,zkhost3:jmx-port"
lenses.jmx.connect = [ {dev: "connect1:jmx-port,connect2:jxm-port"} ]

Note

We recommend going through Lenses Configuration to understand how to fully configure your Lenses installation

Run & Deploy

Once the configuration in lenses.conf is complete, Lenses is ready to be run. The bin folder contains the start scripts. Depending on your operating system you have the choice between lenses and lenses.bat. Lenses application requires a startup argument which should contain the path to the configuration file. Here is an example of how to launch Lenses:

To run on Linux or macOS

$ bin/lenses

Or to explicitly set the configuration file:

$ bin/lenses lenses.conf

To run on Windows:

$ bin/lenses.bat lenses.conf

Once the application is running, open your browser and navigate to http://address:port` based on the configuration file. The default is http://localhost:9991. Login using the credentials you have provided, otherwise the default values are admin:admin. We strongly recommend changing the default credentials!

To stop Lenses, press CTRL+C.

The configuration file is using the HOCON format.

Lenses Instances

We recommend running 1 instance of Lenses per Kafka cluster. Running multiple instances against the same infrastructure, (if not configured appropriately) could result in Lenses SQL processors duplicating data.

Manual Deployment

Some best practice advice can be offered for installation on a server:

  1. Install Lenses under /opt

  2. Make sure the directory is owned by the root user

  3. Create a user specially for Lenses

  4. Lenses configuration may be placed under /opt/lenses/lenses.conf, /etc/lenses/lenses.conf or passed in at the command line when staring Lenses.

    bin/lenses myconf.conf
    
  5. The configuration file contains information about the Lenses users’ credentials. Please make sure it is only readable by the Lenses user:

    chmod 0600 /path/to/lenses.conf
    chown [lenses-user]:root /path/to/lenses.conf
    
  6. Please adjust the logs configuration to a path for which the running Lenses process has write access to or, to stdout/stderr in case the log management should happen by the process supervisor. See the logging configuration section for more information.

If your server is on a current Linux distribution and it uses the systemd service manager, the script below can be used to start and stop Lenses with the management and supervision facilities of systemd.

[Unit]
Description=Run Landoop Lenses Service

[Service]
Restart=always
User=[LENSES-USER]
Group=[LENSES-GROUP]
LimitNOFILE=4096
WorkingDirectory=/opt/lenses
#Environment=LENSES_LOG4J_OPTS="-Dlogback.configurationFile=file:/etc/lenses/logback..xml"
ExecStart=/opt/lenses/bin/lenses /etc/lenses/lenses.conf

[Install]
WantedBy=multi-user.target

Upgrade

From 1.0 to 1.1

Deployment wise Lenses 1.1 brings some small differences in the configuration, apart from this it is practically a drop-in replacement for 1.0.

If you are installing through our archives please be careful, you can not just overwrite the old directory because this will lead to multiple versions of various libraries being installed concurrently.

To install Lenses 1.1 on the same location as the current installation, please stop Lenses, then remove the directories lib/ and jre8u131/ and then extract the 1.1 archive. Alternatively extract Lenses 1.1 to a new directory, copy over your lenses.conf and license.json, then remove the old installation and move the new to its place.

The differences in configuration are two:

  1. The option lenses.connect is renamed to lenses.connect.clusters, name and cluster URL are now explicit fields and the topics used by each connect cluster should be set.

    The old format is:

    Lenses 1.0 Connect Cluster Configuration
    lenses.connect = [
       {DEVELOPMENT: "http://dev.connect.worker.1:8083"},
       {PRODUCTION: "http://prod.connect.worker.1:8083,http://prod.connect.worker.2:8083"}
    ]
    

    The new format is:

    Lenses 1.1 Connect Cluster Configuration
    lenses.connect.clusters = [
       {
           name: "DEVELOPMENT",
           url: "http://dev.connect.worker.1:8083",
           statuses: "connect-statuses-dev",
           configs: "connect-configs-dev",
           offsets: "connect-offsets-dev"
       },
       {
           name: "PRODUCTION",
           url: "http://prod.connect.worker.1:8083,http://prod.connect.worker.2:8083",
           statuses: "connect-statuses",
           configs: "connect-configs",
           offsets: "connect-offsets"
       }
    ]
    
  2. Security settings, i.e every option that starts with lenses.security moved to a different file that is set by the new option lenses.secret.file. That way operations can restrict access to secrets (e.g Lenses’ LDAP user credentials) without restricting the developer teams from setting up Lenses to their needs. Furthermore the LDAP configuration options have changed and now also support third-party plugins for user-role retrieval in order to easily tailor Lenses to your specific LDAP setup.

    Old BASIC configuration:

    Lenses 1.0 Security Configuration (inline — lenses.conf)
    ...
    lenses.security.mode = BASIC
    lenses.security.users = [
       {
           "username": "admin",
           "password": "admin",
           "displayname": "Lenses Admin",
           "roles": ["admin", "write", "read"]
       },
       {
           "username": "write",
           "password": "write",
           "displayname": "Write User",
           "roles": ["read", "write"]
       }
    ]
    ...
    

    New BASIC configuration:

    Lenses 1.1 Configuration, Secret File Path (lenses.conf)
    ...
    lenses.secret.file=/path/to/security.conf
    ...
    
    Lenses 1.1 Security Configuration (separate file — security.conf)
    lenses.security.mode = BASIC
    lenses.security.users = [
       {
           "username": "admin",
           "password": "admin",
           "displayname": "Lenses Admin",
           "roles": ["admin", "write", "read"]
       },
       {
           "username": "write",
           "password": "write",
           "displayname": "Write User",
           "roles": ["read", "write"]
       }
    ]
    

    Old LDAP configuration:

    Lenses 1.0 Security Configuration (inline — lenses.conf)
    ...
    security {
      mode = LDAP
      ldap {
        url = your_ldap_url
        base = your_base
        user = your_ldap_user
        password = your_ldap_password
        login.filter = the_login_filter
        memberof.key = the_member_of_ldap_entry
        group.extract.regex = the_group_name_regex_extractor
        person.name.key = the_entry_attribute_containing_the_full_user_name
        roles {
          admin = [ LDAP_GROUPS_ALLOWING_ADMIN ]
          read = [ LDAP_GROUPS_ALLOWING_READ ]
          write = [ LDAP_GROUPS_ALLOWING_WRITE ]
          nodata = [ LDAP_GROUPS_ALLOWING_NODATA ]
        }
      }
    }
    ...
    

    New LDAP configuration:

    Lenses 1.1 Configuration, Secret File Path (lenses.conf)
    ...
    lenses.secret.file=/path/to/security.conf
    ...
    
    Lenses 1.1 Security Configuration (separate file — security.conf)
    lenses.security.mode=LDAP
    lenses.security.ldap.url="ldaps://mycompany.com:636"
    lenses.security.ldap.base="OU=Users,DC=mycompany,DC=com"
    lenses.security.ldap.user="$LDAP_USER"
    lenses.security.ldap.password="$LDAP_USER_PASSWORD"
    lenses.security.ldap.roles.admin=[ "GROUP_ONE" ]
    lenses.security.ldap.roles.write=[ "GROUP_TWO" ]
    lenses.security.ldap.roles.read=[ "GROUP_THREE" ]
    lenses.security.ldap.roles.nodata=[ "GROUP_FOUR" ]
    lenses.security.ldap.filter="(&(objectClass=person)(sAMAccountName=<user>))"
    
    //LDAP roles retriever settings
    lenses.security.ldap.plugin.class="com.landoop.kafka.lenses.security.LdapMemberOfUserGroupPlugin"
    lenses.security.ldap.plugin.group.extract.regex="(?i)CN=(\\w+),ou=ServiceGroups.*"
    lenses.security.ldap.plugin.memberof.key="memberOf"
    lenses.security.ldap.plugin.person.name.key = "sn"
    

Lenses 1.1 brings in many new features which are covered in the relevant parts of the documentation. Please see the Release Notes to learn more.