async_py

A Rust library for calling Python code asynchronously using pyo3 and tokio.

This library provides a PyRunner that runs a Python interpreter in a dedicated background thread. Global variables are preserved within the runner's scope. You can send Python code to this executor from any async Rust task, and it will be executed without blocking your application.

Due to the Python Global Interpreter Lock (GIL), you cannot run Python code simultaneously. If one Python task is performing a computation or a sleep, no other task can access the interpreter.

Using multiple PyRunner instances does not enable simultaneous Python code execution due to the GIL. However, it does allow you to isolate global variables between different runners.

Usage

Add async_py to your Cargo.toml:

[dependencies]
async_py = "0.2.0"
tokio = { version = "1", features = ["full"] }

Example

use async_py::PyRunner;

#[tokio::main]
async fn main() {
    let runner = PyRunner::new();

    let code = r#"
counter = 0
def greet(name):
    global counter
    counter = counter + 1
    s = "" if counter < 2 else "s"
    return f"Hello {name}! Called {counter} time{s} from Python."
"#;
    // Python function "greet" and variable "counter" is added to globals
    runner.run(code).await.unwrap();

    // Calling code
    let result1 = runner.call_function("greet", vec!["World".into()]).await.unwrap();
    println!("{}", result1.as_str().unwrap()); // Prints: Hello World! Called 1 time from Python.
    let result2 = runner.call_function("greet", vec!["World".into()]).await.unwrap();
    println!("{}", result2.as_str().unwrap()); // Prints: Hello World! Called 2 times from Python.
}

Async Python Example

use async_py::PyRunner;

#[tokio::main]
async fn main() {
    let runner = PyRunner::new();
    let code = r#"
import asyncio
counter = 0

async def add_and_sleep(a, b, sleep_time):
    global counter
    await asyncio.sleep(sleep_time)
    counter += 1
    return a + b + counter
"#;

    runner.run(code).await.unwrap();
    let result1 = runner.call_async_function("add_and_sleep", vec![5.into(), 10.into(), 1.into()]);
    let result2 = runner.call_async_function("add_and_sleep", vec![5.into(), 10.into(), 0.1.into()]);
    let (result1, result2) = tokio::join!(result1, result2);
    assert_eq!(result1.unwrap(), Value::Number(17.into()));
    assert_eq!(result2.unwrap(), Value::Number(16.into()));
}

Both function calls are triggered to run async code at the same time. While the first call waits for the sleep, the second can already start and also increment the counter first. Therefore, result1 will wait longer and compute 5 + 10 + 2, while the result2 can compute 5 + 10 + 1.

Each call will use its own event loop. This may not be very efficient and changed later.

Make sure to use call_async_function for async python functions. Using call_function will probably raise an error. call_async_function is not available for RustPython.

Thread Safety

The PyRunner is designed to be safely shared across multiple threads.

You can clone a PyRunner instance and move it into different threads. All commands sent from any clone are funneled through a channel to a single, dedicated OS thread that houses the Python interpreter. This design correctly handles Python's Global Interpreter Lock (GIL) by ensuring that only one Python operation is attempted at any given time within that interpreter instance.

Here is an example of using PyRunner from multiple std::threads:

use async_py::{PyRunner, PyRunnerError};
use std::thread;

fn main() -> Result<(), PyRunnerError> {
    let runner = PyRunner::new();
    runner.run_sync("x = 0")?;

    let mut handles = vec![];

    for i in 0..5 {
        let runner_clone = runner.clone();
        let handle = thread::spawn(move || {
            // Use the _sync versions for non-async contexts
            runner_clone.run_sync(&format!("x += {}", i)).unwrap();
        });
        handles.push(handle);
    }

    for handle in handles {
        handle.join().unwrap();
    }

    let final_x = runner.read_variable_sync("x")?;
    // Expected: 0 + 1 + 2 + 3 + 4 = 10
    assert_eq!(final_x, Value::Number(10.into()));

    runner.stop_sync()?;
    Ok(())
}

Using a venv

It is generally recommended to use a venv to install pip packages. While you cannot switch the interpreter version with this crate, you can use an existing venv to load installed packages.

let runner = PyRunner::new();
runner.set_venv(&Path::new("/path/to/venv")).await?;

rustpython

PyO3 has usually the best compatibility for python packages. But if you want to use python on android, wasm, or if you don't want to use any external library, you can use rustpython. Keep in mind that some essential packages like os are missing on rustpython.

Cargo.toml

[dependencies]
async_py = { features = ["rustpython"] }