Hey, everyone!

We just released Quickwit 0.3 and we are excited to share it with Hashnode community.

At Quickwit, we aim to provide developers with the cost-efficient software for log management. Even when it comes to petabytes of logs. Quickwit achieves that through decoupled compute & storage, where logs are stored on an object storage instead of local hardware. There is many more features we have added over the 2 years, so feel free to check them out to see if Quickwit fits your user-case.

But for now, we will dive deeper into Quickwit 0.3.

Five months have passed since our last release, and our team has been working hard to deliver the next version of Quickwit. This new feature-rich version introduces:

• An embedded UI
• Schemaless indexing
• Aggregations (Elasticsearch-compatible API)
• Ingest API (Elasticsearch-compatible)
• Support for Amazon Kinesis

For readers in a hurry, we suggest you go through our revised quick start guide to try 0.3 out now!

If you’re still reading, let’s go through the new functionalities brought by Quickwit 0.3 in greater detail.

Embedded UI

For advanced dashboarding and data exploration, our long-term plan is to offer a data source plugin for Grafana. However, we also wanted to provide our users with an out-of-the-box solution to search and visualize their logs without installing and configuring another third-party software.

We developed this front-end application using Typescript with React, and the interface consists of two sections: Discover and Admin.

Discover

The discover section is dedicated to searching and exploring indexes. This page consists of an index selection panel on the left, a search box and a date range picker at the top, and a data table at the center to inspect matching documents. We are not trying to reinvent the wheel here, and this layout should feel very familiar: select an index, type your query, hit the run button (or use the shortcut cmd/ctrl + enter), and matching hits magically appear a few milliseconds later.

Admin

The admin section is dedicated to administration tasks. The indexes page lists and displays the properties (doc mapping, sources, splits) of the indexes managed by Quickwit. The cluster page lists and shows the properties of each node comprising a cluster. We will enrich this section with additional information and increase the number of functionalities (update or delete an index, stop a node) available for use in the coming releases.

Schemaless indexing

In the wild, many events cannot be mapped to a strict schema. For instance, in structured logging, each line describes an event in the application, and some JSON fields are present in all log lines (e.g. server, timestamp, event_type). However, some of the JSON fields are specific to the event type and are often subject to change.

Quickwit 0.3 makes it possible to define a strict schema for the former and handle the latter in a schemaless manner.

You can read more about how this works in our schemaless guide.

Let your data tell a story with aggregation queries

Aggregations summarize your data as metrics or statistics and help answer questions such as:

• How many errors occurred in the last hour?
• What was the average response latency per service in the previous 15 minutes?

With the addition of aggregations, Quickwit isn’t exclusively for search anymore; it’s now a powerful analytics engine that lets you unlock the hidden value of your business data.

The aggregations API is Elasticsearch-search compatible and documented here.

Two new ways to ingest data with Quickwit

Quickwit 0.3 delivers two new ways to load data into indexes:

• Ingest API
• Amazon Kinesis

Ingest API

Quickwit 0.3 ships with a new HTTP endpoint for ingesting newline-delimited JSON documents in batches:

POST api/v1/<index ID>/ingest
{"timestamp": 1653068019, "message": "Hello, World!"}
{"timestamp": 1653068020, "message": "Bonjour, monde!"}

We take data durability seriously, and records are persisted into a write-ahead log (WAL) before acknowledging any write request.

Elasticsearch-compatible bulk endpoint

Furthermore, we added an Elasticsearch-compatible bulk endpoint enabling log shippers such as FluentD, Logstash, or Vector to support Quickwit natively. As a result, you can start shipping logs to Quickwit today simply by updating one line in your current log collector configuration!

This tutorial provides an example of how to use Vector to ship logs to Quickwit.

Support for Amazon Kinesis

After Apache Kafka, Quickwit 0.3 supports a second distributed message queue: Amazon Kinesis. Amazon Kinesis is a fully managed service that offers an alternative to companies that may not have the in-house knowledge, or resources, to manage and operate a Kafka cluster.

Discover how to ingest data from Amazon Kinesis in this tutorial.

With this new release, there are now four ways to load data into Quickwit:

• Ingest API
• Amazon Kinesis
• Apache Kafka
• Files

Better under the hood

In addition to packing a lot of user-facing features, Quickwit is also getting better under the hood. This release features performance improvements, notably in the doc mapper (the class responsible for parsing raw JSON objects into Tantivy documents). Additionally, we switched our cluster membership algorithm from SWIM to Scuttlebutt, which we described extensively in this blog post.

What’s next?

While you’re enjoying the value delivered by Quickwit 0.3, we are already focused on the next release. Key features that you can expect are:

• Native support for Kubernetes
• Datetime field
• IP address field
• Range queries
• Sort
• Retention policies

We plan to deliver the next version in August. Our roadmap is public, and you can follow our progress in this GitHub project.

In the meantime, we invite you to check out this quick start guide to ingest your first dataset with Quickwit.

Finally, if you have any questions regarding Quickwit or encounter any issues, feel free to start a discussion or open an issue on GitHub or contact us directly on Discord.