Merge pull request #6 from simonprickett/introduce-search

Replaces `SCAN` approach with the Search capability in Redis Stack

Author: Simon Prickett

README.md

@@ -2,7 +2,13 @@
 
 ## Overview 
 
-This repository contains an example application that demonstrates how to capture images from a Raspberry Pi using the camera module, store them in [Redis Hashes](https://redis.io/docs/data-types/hashes/) and render them in a web application.
+This repository contains an example application that demonstrates how to capture images from a Raspberry Pi using the camera module, store them in [Redis Hashes](https://redis.io/docs/data-types/hashes/) and render them in a web application.  
+
+The project started by using an older Raspberry Pi camera module and the Redis `SCAN` command to retrieve all of the data and present it in the front end.  [Release v0.1](https://github.com/simonprickett/redis-pi-camera/releases/tag/v0.0.1) has that code.
+
+I enhanced the code to use the newer Raspberry Pi camera module with autofocus and also added configurable image expiry time in Redis and capture of Lux values from the camera.  Check out [release v0.2](https://github.com/simonprickett/redis-pi-camera/releases/tag/v0.0.2) for that.
+
+In my second live stream for this project, I replaced the `SCAN` command with the [Search capability of Redis Stack](https://redis.io/docs/stack/search/).  [Release 0.3](https://github.com/simonprickett/redis-pi-camera/releases/tag/v0.0.3) contains the code from that stream.
 
 Here's what the front end looks like when a few images have been captured by the Raspberry Pi...
 
@@ -12,9 +18,13 @@ And here's a Raspberry Pi with a camera attached:
 
 ![Raspberry Pi 3 with Camera Module attached](raspberry_pi_3_with_camera_module.jpg)
 
-## Watch the Video
+## Watch the Videos
+
+Watch the first video for a full walkthrough of this project.  [Watch on YouTube](https://www.youtube.com/watch?v=OTDZIK55DX0).
 
-Watch the video for a full walkthrough of this project.  [Watch on YouTube](https://www.youtube.com/watch?v=OTDZIK55DX0).
+In the second episode I started to use Redis Stack's Search capability.  [Watch on YouTube](https://www.youtube.com/watch?v=mcGL6Lk2IXU).
+
+I'm planning a third episode once I can figure out what to do in it :)
 
 ## Components of the Demo
 
@@ -27,9 +37,9 @@ Details of how each component works including how to configure and run it can be
 
 ## Redis
 
-Both components need to be connected to the same Redis instance to talk to each other.  Right now, this demo works on any Redis 5 or higher server.  The enclosed Docker Compose file uses Redis Stack - this is to allow for future enhancements to the application to use Stack's Search capability.
+Both components need to be connected to the same Redis Stack instance to talk to each other. 
 
-If you want to use Docker, start Redis like this:
+If you want to use Docker, start Redis Stack like this:
 
 ```
 docker-compose up -d
@@ -53,9 +63,11 @@ See the RedisInsight section of this document if you're interested in a graphica
 
 You'll need to make sure that both components of the application can connect to your Redis instance.  See details in each component's README.
 
-This project will also work with a free Redis Stack cloud instance from Redis (the company).  To use this, [sign up here](https://redis.com/try-free/) and make a note of your Redis host, port and password.  You'll need those to configure each component.
+This project will also work with a free Redis Stack cloud instance from Redis (the company).  To use this, [sign up here](https://redis.com/try-free/) and make a note of your Redis host, port and password.  You'll need those to configure each component.  When using the free cloud instance, note that you get 30Mb space which will fill up with images quickly.  You can manage this by setting a shorter image expiry time in the capture component's configuration.
+
+## The Redis Data Model, Key Naming Strategy and Indexing
 
-## The Redis Data Model and Key Naming Strategy
+### Data Type and Key Naming Strategy
 
 The application stores each image plus associated metadata in its own [Redis Hash](https://redis.io/docs/data-types/hashes/).  A Hash in Redis can be thought of as a flat map of name/value properties.  Each Hash is stored in its own key in Redis.
 
@@ -85,6 +97,77 @@ Here's a complete example, with the image data truncated for brevity:
 ```
 With the camera that I used ([Raspberry Pi Camera Module 3](https://www.raspberrypi.com/products/camera-module-3/) capturing at 4608x2592 pixels - configurable in `capture.py`) you can expect each Hash to require around 1Mb of RAM in Redis.
 
+### Indexing and Querying
+
+As we saw in the initial live stream video, using `SCAN` allows us to retrieve all of the keys containing image data for display in the front end.  This has a couple of problems:
+
+* We can't do any meaningful filtering or searching on the server side in Redis.
+* `SCAN` is effectively a O(n) time complexity command, so the bigger the dataset the longer it's going to take / the more load it will put on the Redis server.
+
+The Search capability of Redis Stack addresses both of these and gives us a flexible way to index and query our data.  To use it, we first have to create and index with the [`FT.CREATE` command](https://redis.io/commands/ft.create/).  Once created, Redis Stack will monitor changes to keys that match the index criteria and update the index automatically.  We can then write rich queries using the `FT.SEARCH` command.
+
+Here's our index creation command - you'll need to paste this into Redis CLI or RedisInsight and execute it before running the front end:
+
+```
+FT.CREATE idx:images ON HASH PREFIX 1 image: SCHEMA mime_type TAG lux NUMERIC SORTABLE timestamp NUMERIC SORTABLE
+```
+
+This command creates an index on keys in Redis whose key names begin `image:` and which are of type `HASH`.  Where found, the values of fields named `timestamp` and `lux` are indexed as numeric values and the values of fields named `mime_type` are indexed as tags (exact match string values).
+
+Here are some example queries that we can run against this index.
+
+Find the 9 most recent images (most recent first), returning their timestamp, MIME type and lux values:
+
+```
+FT.SEARCH idx:images "*" RETURN 3 timestamp mime_type lux SORTBY timestamp DESC LIMIT 0 9
+```
+
+Truncated example response:
+
+```
+1) "12"
+2) "image:1684427475"
+3) 1) "timestamp"
+   2) "1684427475"
+   3) "mime_type"
+   4) "image/jpeg"
+   5) "lux"
+   6) "85"
+4) "image:1684427251"
+5) 1) "timestamp"
+   2) "1684427251"
+   3) "mime_type"
+   4) "image/jpeg"
+   5) "lux"
+   6) "104"
+6) "image:1684427190"
+7) 1) "timestamp"
+   2) "1684427190"
+   3) "mime_type"
+   4) "image/jpeg"
+   5) "lux"
+   6) "109"
+8) "image:1684427130"
+9) 1) "timestamp"
+   2) "1684427130"
+   3) "mime_type"
+   4) "image/jpeg"
+   5) "lux"
+   6) "93"
+...
+```
+
+Find the 9 most recent images with lux value between 100 and 120, again with the most recent image first:
+
+```
+FT.SEARCH idx:images "@lux:[100 120]" RETURN 3 timestamp mime_type lux SORTBY timestamp DESC LIMIT 0 9
+```
+
+For more information on how to write search queries, check out:
+
+* [Search query syntax documentation](https://redis.io/docs/stack/search/reference/query_syntax/).
+* [RU203 - Querying, Indexing and Full-Text Search](https://university.redis.com/courses/ru203/): A free course at Redis University.
+
 ## (Optional, but Recommended): RedisInsight
 
 RedisInsight is a free graphical management and database browsing tool for Redis. You don't need it to look at how the application stores data in Redis (you can use redis-cli if you prefer) but I'd recommend it as it's much easier to get an overall picture of the state of the database with a graphical tool.  RedisInsight runs as a desktop application on your Mac, Windows or Linux machine.

pi/README.md

@@ -83,13 +83,13 @@ data_to_save["lux"] = int(image_metadata["Lux"])
 
 First, we create the key name we're going to use when storing the Hash.  It's `image:<timestamp>`.
 
-`data_to_save` is a Python dictionary containing the name/value pairs to store in the Redis Hash. This needs to be a flat map of name/value pairs - nested structure isn't allowed in a Redis Hash.  If you want more complex data structure, use the [Redis JSON data type](https://redis.io/docs/stack/json/) in Redis Stack.
+`data_to_save` is a Python dictionary containing the name/value pairs to store in the Redis Hash. This needs to be a flat map of name/value pairs - nested structure isn't allowed in a Redis Hash.  If you want to model a more complex data structure, use the [JSON data type](https://redis.io/docs/stack/json/) in Redis Stack.
 
 Hashes in Redis are schemaless, so if you add extra fields there's no need to change any database schema (if you're looking for one, it doesn't exist!).  You'll just need to modify any application code that reads the Hashes to use new fields.
 
 We store the bytes of the image, the timestamp and the MIME or media type of the image... so that any front end knows what encoding the data in `image_data` is in.
 
-Saving the Hash to Redis is then simply a matter of running the [`HSET` command](https://redis.io/commands/hset/), passing it the key name and dict of name/value pairs to store.  When saving this fata, we also want to set an expiry time for it which we do with the Redis [`EXPIRE` command](https://redis.io/commands/expire/).  The time to live for each hash is a configurable number of seconds, read from the `IMAGE_EXPIRY` environment variable (see later for details).
+Saving the Hash to Redis is then simply a matter of running the [`HSET` command](https://redis.io/commands/hset/), passing it the key name and dict of name/value pairs to store.  When saving this data, we also want to set an expiry time for it which we do with the Redis [`EXPIRE` command](https://redis.io/commands/expire/).  The time to live for each Hash is a configurable number of seconds, read from the `IMAGE_EXPIRY` environment variable (see later for details).
 
 This means that we want to send two commands to Redis.  To save on network bandwidth, let's use a feature of the Redis protocol called a [pipeline](https://redis.io/docs/manual/pipelining/) and send both in the same network round trip:
 
@@ -100,7 +100,7 @@ pipe.expire(redis_key, IMAGE_EXPIRY)
 pipe.execute()
 ```
 
-This sets up the `HSET` and `EXPIRE` commands in a pipeline, which is then sent to Redis using the `execute` function.
+This sets up the `HSET` and `EXPIRE` commands in a pipeline, which is then sent to Redis using the `execute` function.  We don't need the results returned from Redis in this instance, but if we did then we can access them as a List returned by `execute`.
 
 ## Setup