Final changes for VECTOR support in Oracle Database 23ai.

Author: anthony-tuininga

doc/src/api_manual/fetch_info.rst

@@ -112,3 +112,19 @@ FetchInfo Attributes
     This read-only attribute returns the type of the column as mandated by the
     Python Database API. The type will be one of the :ref:`database type
     constants <dbtypes>` defined at the module level.
+
+.. attribute:: FetchInfo.vector_dimensions
+
+    This read-only attribute returns the number of dimensions required by
+    vector columns. If the column is not a vector column or allows for any
+    number of dimensions, the value returned is ``None``.
+
+    .. versionadded:: 2.2.0
+
+.. attribute:: FetchInfo.vector_type
+
+    This read-only attribute returns the storage type required by vector
+    columns. If the column is not a vector column or allows for any
+    type of storage, the value returned is ``None``.
+
+    .. versionadded:: 2.2.0

doc/src/api_manual/module.rst

@@ -3147,6 +3147,14 @@ Also see the table :ref:`supporteddbtypes`.
     type VARCHAR2. It will compare equal to the DB API type :data:`STRING`.
 
 
+.. data:: DB_TYPE_VECTOR
+
+    Describes columns, attributes or array elements in a database that are of
+    type VECTOR.
+
+    .. versionadded:: 2.2.0
+
+
 .. data:: DB_TYPE_XMLTYPE
 
     Describes columns, attributes or array elements in a database that are of

doc/src/index.rst

@@ -40,6 +40,7 @@ User Guide
     user_guide/lob_data.rst
     user_guide/json_data_type.rst
     user_guide/xml_data_type.rst
+    user_guide/vector_data_type.rst
     user_guide/soda.rst
     user_guide/aq.rst
     user_guide/cqn.rst

doc/src/release_notes.rst

@@ -34,6 +34,7 @@ Thick Mode Changes
 Common Changes
 ++++++++++++++
 
+#)  Added support for columns of type VECTOR.
 #)  Added support for database type :data:`oracledb.DB_TYPE_INTERVAL_YM` which
     is represented in Python by instances of the new
     :ref:`oracledb.IntervalYM <interval_ym>` class

doc/src/user_guide/appendix_a.rst

@@ -480,6 +480,10 @@ values.
       - :data:`~oracledb.DB_TYPE_OBJECT`
       - No relevant notes
       - OBJECT of specific type
+    * - VECTOR
+      - :data:`~oracledb.DB_TYPE_VECTOR`
+      - No relevant notes
+      -
 
 Binding of contiguous PL/SQL Index-by BINARY_INTEGER arrays of string, number, and date are
 supported in python-oracledb Thin and Thick modes. Use :meth:`Cursor.arrayvar()` to build

doc/src/user_guide/vector_data_type.rst

@@ -0,0 +1,223 @@
+.. _vectors:
+
+*****************
+Using Vector Data
+*****************
+
+Oracle Database 23ai introduced a new data type VECTOR for artificial
+intelligence and machine learning search operations. The vector data type
+is a homogeneous array of 8-bit signed integers, 32-bit floating-point
+numbers, or 64-bit floating-point numbers. With the vector data type, you
+can define the number of dimensions for the data and the storage format
+for each dimension value in the vector.
+
+To create a table with three columns for vector data, for example:
+
+.. code-block:: sql
+
+    CREATE TABLE vector_table (
+        v32 vector(3, float32),
+        v64 vector(3, float64),
+        v8  vector(3, int8)
+    )
+
+In this example, each column can store vector data of three dimensions where
+each dimension value is of the specified storage format. This example is used
+in subsequent sections.
+
+.. _insertvector:
+
+Inserting Vectors
+=================
+
+With python-oracledb, vector data can be inserted using Python arrays
+(``array.array()``). To use Python arrays, import the ``array`` module in your
+code.
+
+Python arrays (``array.array()``) of float (32-bit), double (64-bit), or
+int8_t (8-bit signed integer) are used as bind values when inserting vector
+columns. For example:
+
+.. code-block:: python
+
+    vector_data_32 = array.array("f", [1.625, 1.5, 1.0]) # 32-bit float
+    vector_data_64 = array.array("d", [11.25, 11.75, 11.5]) # 64-bit float
+    vector_data_8 = array.array("b", [1, 2, 3]) # 8-bit signed integer
+
+    cursor.execute(
+        "insert into vector_table (v32, v64, v8) values (:1, :2, :3)",
+        [vector_data_32, vector_data_64, vector_data_8],
+    )
+
+See `vector.py <https://github.com/oracle/python-oracledb/tree/main/
+samples/vector.py>`__ for a runnable example.
+
+If you are using python-oracledb Thick mode with older versions of Oracle
+Client libraries than 23ai, see this
+:ref:`section <vector_thick_mode_old_client>`.
+
+.. _fetchvector:
+
+Fetching Vectors
+================
+
+With python-oracledb, vector columns are fetched as Python arrays
+(``array.array()``). For example:
+
+.. code-block:: python
+
+    cursor.execute("select * from vector_table")
+        for row in cursor:
+            print(row)
+
+This prints an output such as::
+
+    (array("f", [1.625, 1.5, 1.0]), array("d", [11.25, 11.75, 11.5]), array("b", [1, 2, 3]))
+
+The :ref:`FetchInfo <fetchinfoobj>` object that is returned as part of the
+fetched metadata contains attributes :attr:`FetchInfo.vector_dimensions` and
+:attr:`FetchInfo.vector_type` which return the number of dimensions of the
+vector column and the storage format of each dimension value in the vector
+column respectively.
+
+You can convert the vector data fetched from a connection to a Python list by
+using the following :ref:`output type handler <outputtypehandlers>`:
+
+.. code-block:: python
+
+    def output_type_handler(cursor, metadata):
+        if metadata.type_code is oracledb.DB_TYPE_VECTOR:
+            return cursor.var(metadata.type_code, arraysize=cursor.arraysize,
+                              outconverter=list)
+    connection.outputtypehandler = output_type_handler
+    cursor.execute("select * from vector_table")
+    for row in cursor:
+        print(row)
+
+For each vector column, the database will now return a Python list
+representation of each row's value.
+
+If you are using python-oracledb Thick mode with older versions of Oracle
+Client libraries than 23ai, see :ref:`below <vector_thick_mode_old_client>`.
+
+.. _vector_thick_mode_old_client:
+
+Using python-oracledb Thick Mode with Older Versions of Oracle Client Libraries
+===============================================================================
+
+If you are using python-oracledb Thick mode with older versions of Oracle
+Client libraries than 23ai, then you must use strings when inserting vectors.
+For example:
+
+.. code-block:: python
+
+    vector_data_32 = "[1.625, 1.5, 1.0]"
+    vector_data_64 = "[11.25, 11.75, 11.5]"
+    vector_data_8 = "[1, 2, 3]"
+
+    cursor.execute(
+        "insert into vector_table (v32, v64, v8) values (:1, :2, :3)",
+        [vector_data_32, vector_data_64, vector_data_8],
+    )
+
+The vector columns are fetched as Python lists. For example:
+
+.. code-block:: python
+
+    cursor.execute("select * from vector_table")
+    for row in cursor:
+        print(row)
+
+See `vector_string.py <https://github.com/oracle/python-oracledb/tree/main/
+samples/vector_string.py>`__ for a runnable example.
+
+.. _numpyvectors:
+
+Using NumPy
+===========
+
+Vector data can be used with Python's `NumPy <https://numpy.org>`__ package
+types. To use NumPy's ndarray type, install NumPy, for example with
+``pip install numpy``, and import the module in your code.
+
+Inserting Vectors with NumPy
+----------------------------
+
+To insert vectors, you must convert NumPy ndarray types to array types. This
+conversion can be done by using an input type handler. For example:
+
+.. code-block:: python
+
+    def numpy_converter_in(value):
+        if value.dtype == numpy.float64:
+            dtype = "d"
+        elif value.dtype == numpy.float32:
+            dtype = "f"
+        else:
+            dtype = "b"
+        return array.array(dtype, value)
+
+    def input_type_handler(cursor, value, arraysize):
+        if isinstance(value, numpy.ndarray):
+            return cursor.var(
+                oracledb.DB_TYPE_VECTOR,
+                arraysize=arraysize,
+                inconverter=numpy_converter_in,
+            )
+
+Using it in an ``INSERT`` statement:
+
+.. code-block:: python
+
+    vector_data_32 = numpy.array([1.625, 1.5, 1.0])
+    vector_data_64 = numpy.array([11.25, 11.75, 11.5])
+    vector_data_8 = numpy.array([1, 2, 3])
+
+    connection.inputtypehandler = input_type_handler
+
+    cursor.execute(
+        "insert into vector_table (v32, v64, v8) values (:1, :2, :3)",
+        [vector_data_32, vector_data_64, vector_data_8],
+    )
+
+Fetching Vectors with NumPy
+---------------------------
+
+To fetch vector data as an ndarray type, you can convert the array type to
+an ndarray type by using an output type handler. For example:
+
+.. code-block:: python
+
+    def numpy_converter_out(value):
+        if value.typecode == "b":
+            dtype = numpy.int8
+        elif value.typecode == "f":
+            dtype = numpy.float32
+        else:
+            dtype = numpy.float64
+        return numpy.array(value, copy=False, dtype=dtype)
+
+    def output_type_handler(cursor, metadata):
+        if metadata.type_code is oracledb.DB_TYPE_VECTOR:
+            return cursor.var(
+                metadata.type_code,
+                arraysize=cursor.arraysize,
+                outconverter=numpy_converter_out,
+            )
+
+Using it in a query:
+
+.. code-block:: python
+
+    connection.outputtypehandler = output_type_handler
+
+    cursor.execute("select * from vector_table")
+        for row in cursor:
+            print(row)
+
+This prints an output such as::
+
+    (array([1.625, 1.5, 1.0], dtype=float32), array([11.25, 11.75, 11.5], dtype=float64), array([1, 2, 3], dtype=int8))
+
+See `vector_numpy.py <https://github.com/oracle/python-oracledb/tree/main/
+samples/vector_numpy.py>`__ for a runnable example.

samples/create_schema.py

@@ -54,4 +54,8 @@
     sample_env.run_sql_script(
         conn, "create_schema_21", main_user=sample_env.get_main_user()
     )
+if sample_env.get_server_version() >= (23, 4):
+    sample_env.run_sql_script(
+        conn, "create_schema_23", main_user=sample_env.get_main_user()
+    )
 print("Done.")

samples/sql/create_schema_23.sql

@@ -0,0 +1,39 @@
+/*-----------------------------------------------------------------------------
+ * Copyright 2023, Oracle and/or its affiliates.
+ *
+ * This software is dual-licensed to you under the Universal Permissive License
+ * (UPL) 1.0 as shown at https://oss.oracle.com/licenses/upl and Apache License
+ * 2.0 as shown at http://www.apache.org/licenses/LICENSE-2.0. You may choose
+ * either license.*
+ *
+ * If you elect to accept the software under the Apache License, Version 2.0,
+ * the following applies:
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *---------------------------------------------------------------------------*/
+
+/*-----------------------------------------------------------------------------
+ * create_schema_23.sql
+ *
+ * Performs the actual work of creating and populating the schemas with the
+ * database objects used by the python-oracledb samples that require Oracle
+ * Database 23.4 or higher. It is executed by the Python script
+ * create_schema.py.
+ *---------------------------------------------------------------------------*/
+
+create table &main_user..SampleVectorTab (
+    v32 vector(3, float32),
+    v64 vector(3, float64),
+    v8  vector(3, int8)
+)
+/