Lenses supports Avro. Avro is a great choice for a data serialization format,
- It has a direct mapping to and from JSON
- It has a very compact format. The bulk of JSON, repeating every field name with every single record, is what makes JSON inefficient for high-volume usage.
- It is very fast
- It has great bindings for a wide variety of programming languages so you can generate Java objects that make working with event data easier, but it does not require code generation so tools can be written generically for any data stream.
- It has a rich, extensible schema language defined in pure JSON
- It has the best notion of compatibility for evolving your data over time
- It can be used as an on disk format and Parquet has built in support for Avro.
Lenses allows you to manage schemas and their evolution. Navigate to the
Schemas option in the side
menu, this will land you on the schema overview page, which allows you to:
- View and search schemas
- Change the global compatibility levels
- Create new schemas.
Create new schemas¶
To create a new schema select the
New Schema button on the right-hand side. This presents a template
Avro schema in json format. Construct your schema and submit the request. Lenses with validate the
schema before it is deployed.
Both the key and the value of a Kafka message can have a schema. Add the
-key suffix to the subject accordingly.
Edit a schema¶
To edit a schema, select the schema you wish to edit. This will display schema both in JSON and a tabular format. Click the edit button and apply your changes. Before the edits are applied Lenses will validate that the changes match the schema evolution rules.
View schema history¶
Schemas can evolve and Lenses not only allows you to browse the different versions of schemas as
they evolve over time but also can show a Git style diff between versions. For the selected schema
Version button to view the different versions or
View History for the changes between them.
You might want to have control over who makes schemas changes.By default, schema creation, deletion and editing is only allowed to users with WRITE access, such as admin and write.