Upgrade Notes

To see the full list of changes made for this release check the Release Notes. Upgrading from the previous versions requires changes to the existing configuration file. This section covers all the changes to be made to the existing configuration file to make it work with this version.

This is a brief list of the important configuration options that have been altered and you should adjust before starting Lenses 2.0. Please read the full document for the full list and detailed examples for each change.

  • lenses.zookeeper.hosts, changed
  • lenses.jmx.zookeepers, deprecated
  • lenses.zookeeper.chroot, added, no need to set if you don’t use a chroot
  • lenses.zookeeper.security.enabled, added, important if you use SASL authentication (kerberos)
  • lenses.schema.registry.urls, changed
  • lenses.jmx.schema.registry, deprecated
  • lenses.connect.clusters, changed
  • lenses.jmx.connect, deprecated
  • lenses.alert.consumers.lag.threshold, deprecated (in favor of new alerting system)
  • lenses.security.users, changed
  • lenses.security.groups, added (split from lenses.security.users / replaces lenses.security.ldap.roles.*)
  • lenses.security.ldap.roles.*, deprecated (in favor of lenses.security.users)

Zookeeper Connection

In this release, the Zookeeper nodes connection and their JMX endpoints have been merged into one configuration entry. In addition, the (optional) zookeeper chroot entry has moved to its own configuration entry. When utilizing SASL (Kerberos) authentication to the brokers, the connection to zookeeper can optionally be authenticated as well.

Previous:

lenses.zookeeper.hosts="host.1:2181,host.2:2181,host.3:2181"
lenses.jmx.zookeepers="host.1:9585,host.2:9585,host.3:9585"

New:

lenses.zookeeper.hosts=[
  {
    url:"host.1:2181",
    jmx:"host.1:9585"
  },
  {
    url:"host.2:2181",
    jmx:"host.2:9585"
  },
  {
    url:"host.3:2181",
    jmx:"host.3:9585"
  }
]

lenses.zookeeper.chroot= "" # Optional in case a chroot path is in use, without leading slash

Zookeeper may optionally provide authentication via SASL GSSAPI (Kerberos). Zookeeper authentication in Kafka context is used in order to create nodes with (zookeeper) ACLs, which can only be written or deleted by the appropriate Kerberos principal.

By default, Lenses will not authenticate to Zookeeper. The behaviour is driven by a new configuration entry lenses.zookeeper.security.enabled which by default is false. If this setting is enabled and authentication configuration (via jaas.conf) is provided, Lenses will use Zookeeper ACLs when storing Kafka Quotas.

Warning

If lenses.zookeeper.security.enabled is set to true, then Lenses should be configured with the same Kerberos principal as the brokers otherwise the brokers will fail to restart as they will be unable to manipulate the Quota entries with their own Kerberos principal.

Schema Registry Connection

Following the trend mentioned earlier, the same changes have been applied for Schema Registry configuration.

Previous:

lenses.schema.registry.urls="http://host.1:8081,http://host.2:8081"
lenses.jmx.schema.registry="host.1:9582,host.2:9582"

New:

lenses.schema.registry.urls=[
  {
    url:"http://host.1:8081",
    jmx:"host.1:9582"
  },
  {
    url:"http://host.2:8081",
    jmx:"host.2:9582"
  }
]

Connect Configuration

Following the trend the following changes have been applied for the Connect cluster configuration details:

Previous:

lenses.connect.clusters=[
  {
    name: "dev",
    url: "http://host.1:8083,http://host.2:8083,http://host.3:8083",
    statuses: "connect-status-dev",
    configs: "connect-configs-dev",
    offsets: "connect-offsets-dev"
  }
]

lenses.jmx.connect=[
  {
    dev: "host.1:9584,host.2:9584,host.3:9584"
  }
]

New:

lenses.connect.clusters=[
  {
    name: "dev",
    urls: [
      {
        url:"http://host.1:8083",
        jmx:"host.1:9584"
      },
      {
        url:"http://host.2:8083",
        jmx:"host.2:9584"
      },
      {
        url:"http://host.3:8083",
        jmx:"host.3:9584"
      }
    ],
    statuses: "connect-status-dev",
    configs: "connect-configs-dev",
    offsets: "connect-offsets-dev"
  }
]

Security

The concept of user groups has been added with this release. Therefore amendments to lenses.security are required. Here is how to update the configuration when using BASIC authentication:

Previous:

lenses.security.mode = BASIC
lenses.security.users = [
  {"username": "admin", "password": "admin", "displayname": "Lenses Admin", "roles": ["admin", "write", "read"]},
  {"username": "write", "password": "write", "displayname": "Write User", "roles": ["read", "write"]},
  {"username": "read", "password": "read", "displayname": "Read Only", "roles": ["read"]}
  {"username": "nodata", "password": "nodata", "displayname": "No Data", "roles": ["nodata"]}
]

New:

lenses.security.mode=BASIC
lenses.security.groups=[
  {"name": "adminGroup", "roles": ["admin", "write", "read"]},
  {"name": "writeGroup", "roles": ["read", "write"]},
  {"name": "readGroup",  "roles": ["read"]},
  {"name": "nodataGroup",  "roles": ["nodata"]}
]

lenses.security.users=[
  {"username": "admin", "password": "admin", "displayname": "Lenses Admin", "groups": ["adminGroup"]},
  {"username": "write", "password": "write", "displayname": "Write User", "groups": ["writeGroup"]},
  {"username": "read", "password": "read", "displayname": "Read Only", "groups": ["readGroup"]},
  {"username": "nodata", "password": "nodata", "displayname": "No Data", "groups": ["nodataGroup"]}
]

If LDAP security mode is used, here is how to amend the current configuration:

Previous:

lenses.security.mode=LDAP
lenses.security.ldap.url="ldaps://LDAP_HOST:LDAP_HOST"
lenses.security.ldap.base="xxxx"
lenses.security.ldap.password="xxxx"
lenses.security.ldap.user="xxxx"
lenses.security.ldap.filter="(CN=<user>)"
lenses.security.ldap.roles.admin=[ "groupA" ]
lenses.security.ldap.roles.nodata=[ "groupB" ]
lenses.security.ldap.plugin.class="com.landoop.kafka.lenses.security.LdapMemberOfUserGroupPlugin"
lenses.security.ldap.plugin.group.extract.regex="(?i)CN=(\\w+),ou=ServiceGroups.*"

New:

lenses.security.mode=LDAP

lenses.security.groups=[
  {"name": "groupA", "roles": ["admin", "write", "read"]},
  {"name": "groupB",  "roles": ["nodata"]}
]

lenses.security.ldap.url="ldaps://LDAP_HOST:LDAP_HOST"
lenses.security.ldap.base="xxxx"
lenses.security.ldap.password="xxxx"
lenses.security.ldap.user="xxxx"
lenses.security.ldap.filter="(CN=<user>)"
lenses.security.ldap.plugin.class="com.landoop.kafka.lenses.security.LdapMemberOfUserGroupPlugin"
lenses.security.ldap.plugin.group.extract.regex="(?i)CN=(\\w+),ou=ServiceGroups.*"

Deprecated configuration

Part of this release the following configuration entries have been deprecated, so you can remove them from your configuration file.

lenses.alert.consumers.lag.threshold=...

lenses.security.ldap.roles.admin = [...]
lenses.security.ldap.roles.read = [...]
lenses.security.ldap.roles.nodata = [...]

lenses.jmx.schema.registry = ""     // "schema-host:jmx-port"
lenses.jmx.zookeepers = ""          // "zkhost1:jmx-port,zkhost2:jmx-port,zkhost3:jmx-port"
lenses.jmx.connect = []             // [ {dev: "kafka-connect-dev-host-or-ip:jmp-port"} , .. ]

Renamed configuration

If current configuration has overwritten the existing defaults these entries have been renamed:

lenses.kafka.read.max.polling.ms --> lenses.sql.max.time

From 1.0 to 1.1

For those upgrading from version 1.0 please also read

See Upgrade path from 1.0 to 1.1