../../_images/lenses-data-management-platform.png


1. Features Overview

Lenses is a complete Streaming Data Management Platform for Apache Kafka. Lenses delivers the core elements of Kafka along with a web user interface and vital enterprise capabilities thus improving the experience engineering teams, business users, data scientists and administrators have while using Apache Kafka.

  • Flexibility View and store any type of data and manipulate it using batch processing, or interactive SQL
  • Integration Connect to popular data stores using one of the 30+ open source stream-reactor Kafka connectors
  • Security Process and control sensitive data, LDAP, and Kerberos support
  • Scalability Provision SQL Processors and scale them to suit your requirements
  • High Availability Perform real-time operational business tasks with confidence
  • Monitoring Provides infrastructure and application topology KPIs
  • Compatibility Leverage your existing IT infrastructure
  • Auditing Role-based access and auditing

1.1. Lenses SQL Engine

Lenses SQL is a multi-purpose engine, it fully supports Avro and Json payload types and can execute both batch and real-time streaming SQL.

../../_images/lenses-sql-engine.jpg

1.2. Explore data in motion

Lenses provides a Web User Interface to explore Kafka topics in real time or simply browse the topics content while controlling the data returned by filtering on the Kafka message content and its partition/offset/timestamp information.

../../_images/lenses-explore-data-in-motion.png

1.3. Processors Topologies

Create boundless data queries with SQL processors to aggregate, join and/or transform streams. Via the UI you can visualise, monitor the topologies or even scale out the processing. Furthermore the execution plan viewer allows your to optimize for performance.

../../_images/lenses-topologies.png

1.4. Avro Messages

Full support for Avro messages, including decimal type, is available in Lenses SQL Engine. Within financial institution aggregating on data type Doubles is not acceptable. The platform is compatible with schema registry and provides a rich user interface to create, edit and track schemas.

../../_images/lenses-avro-messages.png

1.5. Kafka Connectors

Manage multiple connect clusters as well as manage and monitor connectors workers and their tasks. We have built 20+ Kafka connectors with Lenses SQL support.

Through our Cloudera integration, the Cloudera Custom Service Descriptor, you can manage your Kafka Connect clusters. It allows spinning up new clusters, controlling the number of Connect workers and of course leverages Cloudera Manager integration for monitoring.

Use simple configuration and only SQL to create end-to-end pipelines from all major data sources and sinks including Cassandra, Elastic, InfluxDB, Azure CosmosDB, MQTT, JMS and more. See how British Gas loads 4 billion entries per day into Elastic with our Connector and simple SQL statement.

# Upsert into elastic with auto create and index suffix
UPSERT INTO index SELECT id, string_field FROM topic PK id AUTOCREATE WITHINDEXSUFFIX=_{YYYY-MM-dd} WITHDOCTYPE=lenses
../../_images/lenses-kafka-connectors.png

1.6. Consumers Lag

Monitor Kafka consumer lag in real time and reset commit offsets. Set up alerting to make sure you can scale or take the appropriate action.

../../_images/lenses-consumers-lag.png

1.7. Monitor Services

Monitor the core services and infrastructure via JMX for Kafka Brokers, Zookeeper, Schema Registry and Connect. Prevent and react to potential issues and get insights into your clusters’ performance.

../../_images/lenses-monitoring.png ../../_images/lenses-offline-partitions.png

1.8. Alerts

Set up alerting and prevent issues early.

../../_images/lenses-alerts.png

1.9. Auditing

Track changes to the cluster, topic creation, deletion, modification and processor deployment.

../../_images/lenses-auditing.png

1.10. Security

Lenses supports basic authentication and LDAP. Additionaly, different operator roles are provided to restrict user actions, giving operators a way to provide security on Kafka. Users can manage topic ACLs (Access Control Lists) to restrict readers and writers of data.

../../_images/lenses-acls.png