Data Streaming With Redis

In the modern-day world, data is generated continuously from social media platforms, websites, and IoT devices. Hence, data streaming is unstoppable. A Stream can be identified as a series of events generated from multiple concurrent sources. For example, a weather station generates weather data at a given location hourly. Each event consists of weather data such as temperature, humidity, wind speed, and direction.

Redis Streams help capture, manage, and make sense of this large amount of streamed data.

Usually, the sources that generate data or write to a Stream are called producers. Each producer adds a new entry to the Redis Stream with a unique event ID where an event consists of one or more field-value pairs similar to a Redis hash. On the other end, consumers are reading from the Stream to generate some meaningful insights. The producers and consumers do not need to know about each other’s implementation details, making Redis Streams more robust.

Redis Stream Entries and Macro Nodes

Redis Stream type is based on widely used radix-tree data structures. Usually, a radix-tree consists of several macro nodes where one macro node can store multiple Stream entries. Since the radix tree is a memory-optimized prefix tree, Redis Streams can be used with low memory consumption. Furthermore, the underlying radix trees enable Streams to access data randomly or within a specified range at higher efficiency.

The XADD command adds a new entry to a macro node in the Stream. When you add an entry, it is immutable. Hence, it is called an append-only data structure.

The XDEL Command

In some cases, maintaining millions of historical Stream data might not be worth it due to data expirations. Therefore, Redis Streams support the deletion of entries from the data structure and free up some memory. We can use the XDEL command to delete an entry from a specified stream.

Syntax:

• stream_key: The Redis key is used to identify a stream uniquely.
• entry_id: One Stream can have several entries which belong to multiple macro nodes. The “entry_id” is the unique identifier to identify an entry.

The XDEL command accepts multiple entry_ids. Hence, you can delete multiple entries per command. This command returns the count of the deleted stream entries. Whenever the given entry_id does not exist in a Stream, the returned count is less than the specified ID count.

Redis Stream Entry Deletion Mechanism

The XDEL command does not dump the Stream’s entries straightaway. Instead, it marks the entry as deleted first. Hence, the Stream entry will keep utilizing the allocated memory. The memory will be released when all the entries belonging to a macro node have been marked deleted. It would force the Redis server to release the memory for that macro node.

Example 1: Delete Old Weather Data Entry From a Stream

Let’s first create a Stream to store weather data in san-Francisco city using the XADD command. Assume that the sensors produce data daily at noon. Hence, we will be adding data for three days in this example.

Let’s assume the first entry is from the last month, and we do not need to keep track of the previous month’s data.

Hence, we can use the XDEL command to delete the first entry with the ID 1656493771190-0.

The XDEL command returned the integer 1, which means only one entry has been deleted from the Stream.

Let’s use the XRANGE command to read the Stream and verify that entry ID 1656493771190-0 has been marked as deleted.

As expected, only the last two entries are available, and the first entry has been deleted.

Example 2: Delete Multiple Weather Entries From a Stream

Let’s say we must delete multiple entries from a Stream simultaneously. The XDEL command supports it. You can specify the entry IDs as shown in the following.

We’ll delete the remaining two entries from the sanfranciscoWeather Stream we created in the previous example. The 1656493789125-0 and 1656493802770-0 are the IDs of the remaining two entries.

As expected, the command return value is 2, which indicates two entries have been deleted.

Let’s read the Stream

The sanfranciscoWeather Stream is empty.

Conclusion

Redis Streams can be identified as a series of events produced by concurrent sources. A Stream can hold several entries where each entry has a unique ID and consists of field-value pairs. Redis Streams are based on radix tree data structure, making them highly efficient and memory-optimized. The entries added to a Stream can be deleted using the Redis XDEL command. This command marks the entry as deleted instead of dumping the entry and releasing memory. It reclaims the memory when all the entries of a macro node are marked as deleted.