Skip to content

Natural Language and Image-Based Search Support for Recordings #603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
WANGXIAOMIN-HIK wants to merge 11 commits into
base: development
Choose a base branch
from
Open

Natural Language and Image-Based Search Support for Recordings #603

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
ddf91bc
Natural Language and Image-Based Search Support for Recordings
WANGXIAOMIN-HIK Jun 26, 2025
c272ef2
Expand the recording search feature to support multi-recording token …
WANGXIAOMIN-HIK Jun 30, 2025
5b48ba2
#603 update
WANGXIAOMIN-HIK Jul 14, 2025
a1eec95
Correct the spelling mistake : SerachImagebyImage -> SearchImagebyImage
WANGXIAOMIN-HIK Nov 3, 2025
c0b206a
Verander SearchImagebyImage na SearchImageByImage
WANGXIAOMIN-HIK Nov 7, 2025
4509b2c
Update the search functionality, add the TargetImageData parameter, a…
WANGXIAOMIN-HIK Nov 12, 2025
80803e4
Update the TargetImageURI description, remove the NPL prefix to unify…
WANGXIAOMIN-HIK Nov 14, 2025
a606279
change Likelihood score to Likelihood threshold , change FindImageRes…
WANGXIAOMIN-HIK Nov 18, 2025
ec431ae
Change the "Likelihood" field to "CosineSimilarity".
WANGXIAOMIN-HIK Dec 16, 2025
0fd61ed
Change the "CosineSimilarity" field to "Similarity"
WANGXIAOMIN-HIK Jan 26, 2026
2a39b4a
We have made the following changes considering the future scalability…
WANGXIAOMIN-HIK Feb 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
197 changes: 197 additions & 0 deletions doc/RecordingSearch.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1320,6 +1320,195 @@ http://www.onvif.org/ver10/tptz/ZoomSpaces/PositionGenericSpace
</varlistentry>
</variablelist>
</section>
<section>
<title>SearchImageByNL</title>
<para>SearchImageByNL starts a search session, looking for video records based on a natural language description of the content, such as "a person wearing a red hat" or "a blue car". Results from the search are acquired using the GetNLSearchResults request, specifying the search token returned from this request.</para>
<para>The device shall continue searching until one of the following occurs:</para>
<itemizedlist>
<listitem>
<para>The entire time range from <emphasis>StartPoint</emphasis> to <emphasis>EndPoint</emphasis> has been searched through.</para>
</listitem>
<listitem>
<para>The total number of matches has been found, defined by the <emphasis>MaxMatches</emphasis> parameter.</para>
</listitem>
<listitem>
<para>The session has been ended by a client EndSearch request.</para>
</listitem>
<listitem>
<para>The session has been ended because <emphasis>KeepAliveTime</emphasis> since the last request related to this session has expired.</para>
</listitem>
</itemizedlist>
<variablelist role="op">
<varlistentry>
<term>request</term>
<listitem>
<para role="param">StartPoint [xs:dateTime]</para>
<para role="text">The point of time where the search will start.</para>
<para role="param">EndPoint [xs:dateTime]</para>
<para role="text">The point of time where the search will stop.</para>
<para role="param">RecordingToken - optional [tt:RecordingReference]</para>
<para role="text">Token for the recording to search.</para>
<para role="param">Text [xs:string]</para>
<para role="text">Natural language description for the search.</para>
<para role="param">Similarity - optional [xs:float]</para>
<para role="text">It represents the similarity between two vectors, which is used to measure the similarity of the directions of the vectors. The closer the value is to 1, the higher the similarity; the closer the value is to 0, the lower the similarity.</para>
<para role="param">MaxMatches - optional [xs:int]</para>
<para role="text">The search ends after MaxMatches.</para>
<para role="param">KeepAliveTime [xs:duration]</para>
<para role="text">The session timeout after each request concerning this session.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>response</term>
<listitem>
<para role="param">SearchToken [tt:JobToken]</para>
<para role="text">Identifies the search session created by this request.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>faults</term>
<listitem>
<para role="param">env:Receiver - ter:Action - ter:ResourceProblem</para>
<para role="text">Device is unable to create a new search session.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>access class</term>
<listitem>
<para role="access">READ_MEDIA</para>
</listitem>
</varlistentry>
</variablelist>
</section>

<section>
<title>GetNLSearchResults</title>
<para>GetNLSearchResults acquires the results from a natural language search session previously initiated by a SearchImageByNL operation. The response shall not include results already returned in previous requests for the same session.</para>
<variablelist role="op">
<varlistentry>
<term>request</term>
<listitem>
<para role="param">SearchToken [tt:JobToken]</para>
<para role="text">Specifies the search session.</para>
<para role="param">MinResults - optional [xs:int]</para>
<para role="text">Specifies the minimum number of results that should be returned.</para>
<para role="param">MaxResults – optional [xs:int]</para>
<para role="text">Specifies the maximum number of results to return.</para>
<para role="param">WaitTime – optional [xs:duration]</para>
<para role="text">Defines the maximum time to block, waiting for results.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>response</term>
<listitem>
<para role="param">ResultList [tt:FindNLSearchResultList]</para>
<para role="text">A structure containing the search results.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>faults</term>
<listitem>
<para role="param">env:Sender - ter:InvalidArgVal - ter:InvalidToken</para>
<para role="text">The search token is invalid.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>access class</term>
<listitem>
<para role="access">READ_MEDIA</para>
</listitem>
</varlistentry>
</variablelist>
</section>

<section>
<title>SearchImageByImage</title>
<para>SearchImageByImage starts a search session, looking for video records based on a provided image. Results from the search are acquired using the GetImageSearchResults request, specifying the search token returned from this request.</para>
<variablelist role="op">
<varlistentry>
<term>request</term>
<listitem>
<para role="param">StartPoint [xs:dateTime]</para>
<para role="text">The point of time where the search will start.</para>
<para role="param">EndPoint [xs:dateTime]</para>
<para role="text">The point of time where the search will stop.</para>
<para role="param">RecordingToken - optional [tt:RecordingReference]</para>
<para role="text">Token for the recording to search.</para>
<para role="param">TargetImageURI - optional [xs:anyURI]</para>
<para role="text">The URI of the detected target object image. This can be either: - a local image stored in the Target Image repository (LocalStorage format), or - an external image provided by the client for image search or feature matching.</para>
<para role="param">TargetImageData - optional [xs:base64Binary]</para>
<para role="text">Base64-encoded target object image used for visual search.</para>
<para role="param">MaxMatches - optional [xs:int]</para>
<para role="text">The search ends after MaxMatches.</para>
<para role="param">KeepAliveTime [xs:duration]</para>
<para role="text">The session timeout after each request concerning this session.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>response</term>
<listitem>
<para role="param">SearchToken [tt:JobToken]</para>
<para role="text">Identifies the search session created by this request.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>faults</term>
<listitem>
<para role="param">env:Receiver - ter:Action - ter:ResourceProblem</para>
<para role="text">Device is unable to create a new search session.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>access class</term>
<listitem>
<para role="access">READ_MEDIA</para>
</listitem>
</varlistentry>
</variablelist>
</section>
<section>
<title>GetImageSearchResults</title>
<para>GetImageSearchResults acquires the results from an image-based search session previously initiated by a SearchImageByImage operation. The response shall not include results already returned in previous requests for the same session.</para>
<variablelist role="op">
<varlistentry>
<term>request</term>
<listitem>
<para role="param">SearchToken [tt:JobToken]</para>
<para role="text">Specifies the search session.</para>
<para role="param">MinResults - optional [xs:int]</para>
<para role="text">Specifies the minimum number of results that should be returned.</para>
<para role="param">MaxResults – optional [xs:int]</para>
<para role="text">Specifies the maximum number of results to return.</para>
<para role="param">WaitTime – optional [xs:duration]</para>
<para role="text">Defines the maximum time to block, waiting for results.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>response</term>
<listitem>
<para role="param">ResultList [tt:FindObjectImageResultList]</para>
<para role="text">A structure containing the search results.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>faults</term>
<listitem>
<para role="param">env:Sender - ter:InvalidArgVal - ter:InvalidToken</para>
<para role="text">The search token is invalid.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>access class</term>
<listitem>
<para role="access">READ_MEDIA</para>
</listitem>
</varlistentry>
</variablelist>
</section>




<section>
<title>EndSearch</title>
<para>EndSearch stops an ongoing search session, causing any blocking result request to return and the <emphasis>SearchToken</emphasis> to become invalid. If the search was interrupted before completion, the point in time that the search had reached shall be returned. If the search had not yet begun, the <emphasis>StartPoint</emphasis> shall be returned. Note that an error message will occur if the search session has been already completed before this request. If the search was completed the original <emphasis>EndPoint</emphasis> supplied by the Find operation shall be returned. When issuing EndSearch on a FindRecordings request the <emphasis>EndPoint</emphasis> is undefined and shall not be used since the FindRecordings request doesn't have StartPoint/EndPoint. This operation is mandatory to support for a device implementing the recording search service.</para>
Expand Down Expand Up @@ -1387,6 +1576,14 @@ http://www.onvif.org/ver10/tptz/ZoomSpaces/PositionGenericSpace
<term>GeneralStartEvents</term>
<listitem><para>Indicates support for general virtual property events in the FindEvents method</para></listitem>
</varlistentry>
<varlistentry>
<term>NLSearch</term>
<listitem><para>Indicates if the device supports natural language based search for recorded video</para></listitem>
</varlistentry>
<varlistentry>
<term>ImageSearch</term>
<listitem><para>Indicates if the device supports image based search for recorded video</para></listitem>
</varlistentry>
</variablelist>
</section>
<section xml:id="_Ref300655850">
Expand Down
83 changes: 82 additions & 1 deletion wsdl/ver10/schema/onvif.xsd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3565,6 +3565,12 @@ decoding .A decoder shall decode every data it receives (according to its capabi
<xs:sequence>
<xs:element name="XAddr" type="xs:anyURI"/>
<xs:element name="MetadataSearch" type="xs:boolean"/>
<xs:element name="NLSearch" type="xs:boolean" minOccurs="0">
<xs:annotation><xs:documentation>Indicates support for natural language based search.</xs:documentation></xs:annotation>
</xs:element>
<xs:element name="ImageSearch" type="xs:boolean" minOccurs="0">
<xs:annotation><xs:documentation>Indicates support for image based search.</xs:documentation></xs:annotation>
</xs:element>
<xs:any namespace="##any" processContents="lax" minOccurs="0" maxOccurs="unbounded"/> <!-- first Vendor then ONVIF -->
</xs:sequence>
<xs:anyAttribute processContents="lax"/>
Expand Down Expand Up @@ -7621,6 +7627,76 @@ and sample rate. </xs:documentation>
</xs:sequence>
<xs:anyAttribute processContents="lax"/>
</xs:complexType>
<!-- FindObjectImageResultList -->
<xs:complexType name="FindObjectImageResultList">
<xs:sequence>
<xs:element name="SearchState" type="tt:SearchState">
<xs:annotation>
<xs:documentation>The state of the search when the result is returned. Indicates if there can be more results, or if the search is completed.</xs:documentation>
</xs:annotation>
</xs:element>
<xs:element name="Result" type="tt:FindObjectImageResult" minOccurs="0" maxOccurs="unbounded">
<xs:annotation><xs:documentation>A FindObjectImageResult structure for each found set of image matching the search.</xs:documentation></xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>

<!-- FindObjectImageResult -->
<xs:complexType name="FindObjectImageResult">
<xs:sequence>
<xs:element name="ObjectUUID" type="xs:string">
<xs:annotation><xs:documentation>Object unique identifier.</xs:documentation></xs:annotation>
</xs:element>
<xs:element name="TargetImageURI" type="xs:anyURI">
<xs:annotation><xs:documentation>The URI of the detected target object image. This can be either: - a local image stored in the Target Image repository (LocalStorage format), or - an external image provided by the client for image search or feature matching.</xs:documentation></xs:annotation>
</xs:element>
<xs:element name="Time" type="xs:dateTime">
<xs:annotation><xs:documentation>The time when the target was found. Users can use this as a reference point to search records that occurred before and after this specific time point.</xs:documentation></xs:annotation>
</xs:element>
<xs:element name="Similarity" type="xs:float">
<xs:annotation><xs:documentation>It represents the similarity between two vectors, which is used to measure the similarity of the directions of the vectors. The closer the value is to 1, the higher the similarity; the closer the value is to 0, the lower the similarity.</xs:documentation></xs:annotation>
</xs:element>
<xs:element name="RecordingToken" type="tt:RecordingReference">
<xs:annotation><xs:documentation>The recording where this result was found.</xs:documentation></xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>

<!-- FindNLSearchResultList -->
<xs:complexType name="FindNLSearchResultList">
<xs:sequence>
<xs:element name="SearchState" type="tt:SearchState">
<xs:annotation>
<xs:documentation>The state of the search when the result is returned. Indicates if there can be more results, or if the search is completed.</xs:documentation>
</xs:annotation>
</xs:element>
<xs:element name="Result" type="tt:FindNLSearchResult" minOccurs="0" maxOccurs="unbounded">
<xs:annotation><xs:documentation>A FindNLSearchResult structure for each found set of image matching the search.</xs:documentation></xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>

<!-- FindNLSearchResult -->
<xs:complexType name="FindNLSearchResult">
<xs:sequence>
<xs:element name="ObjectUUID" type="xs:string">
<xs:annotation><xs:documentation>Object unique identifier.</xs:documentation></xs:annotation>
</xs:element>
<xs:element name="TargetImageURI" type="xs:anyURI">
<xs:annotation><xs:documentation>The URI of the detected target object image. This can be either: - a local image stored in the Target Image repository (LocalStorage format), or - an external image provided by the client for image search or feature matching.</xs:documentation></xs:annotation>
</xs:element>
<xs:element name="Time" type="xs:dateTime">
<xs:annotation><xs:documentation>The time when the target was found. Users can use this as a reference point to search records that occurred before and after this specific time point.</xs:documentation></xs:annotation>
</xs:element>
<xs:element name="Similarity" type="xs:float">
<xs:annotation><xs:documentation>It represents the similarity between two vectors, which is used to measure the similarity of the directions of the vectors. The closer the value is to 1, the higher the similarity; the closer the value is to 0, the lower the similarity.</xs:documentation></xs:annotation>
</xs:element>
<xs:element name="RecordingToken" type="tt:RecordingReference">
<xs:annotation><xs:documentation>The recording where this result was found.</xs:documentation></xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>

<!--===============================-->
<xs:simpleType name="SearchState">
<xs:restriction base="xs:string">
Expand Down Expand Up @@ -9192,7 +9268,12 @@ and sample rate. </xs:documentation>
<!--=========================================-->
<!-- End, StorageConfiguration -->
<!--=========================================-->

<xs:element name="FindObjectImageResultList" type="tt:FindObjectImageResultList"/>
<xs:element name="FindObjectImageResult" type="tt:FindObjectImageResult"/>

<xs:element name="FindNLSearchResultList" type="tt:FindNLSearchResultList"/>
<xs:element name="FindNLSearchResult" type="tt:FindNLSearchResult"/>

<xs:element name="PolygonOptions" type="tt:PolygonOptions"/>

<xs:complexType name="PolygonOptions">
Expand Down
Loading