Engine Weight Sharing

[aianau]

Hi.
Objective: I am trying to create multiple instances of OAROCR engines to more easily distribute work across threads. Is there a way to share weights across instances?

I saw that ONNX Runtime is able to share weights across models: https://onnxruntime.ai/docs/execution-providers/EP-Context-Design.html#epcontext-with-weight-sharing , but i couldn't find something about it in the oar-ocr guides.

Is there a way to share weights across instances using oar-ocr?

[GreatV]

The ort crate (which oar-ocr uses internally) does support PrepackedWeights for sharing weights across sessions. However:

1. CPU-only limitation: ONNX Runtime only supports prepacked weights for the CPU execution provider
2. Not exposed by oar-ocr: The oar-ocr library doesn't expose this low-level ort configuration

According to microsoft/onnxruntime#15301, the official recommendation is:

"You don't have to create multiple sessions for the same model; it's a huge waste of resources. Create only one session and reuse the same in different run threads."

A single session provides:

• One thread pool
• One memory allocator
• One copy of weights
• Thread-safe concurrent execution

[package]
name = "oarocr-sse"
version = "0.1.0"
edition = "2024"

[dependencies]
anyhow = "1.0.100"
oar-ocr = "0.4.0"
rayon = "1.11.0"

use anyhow::{Context, Result};
use oar_ocr::prelude::*;
use rayon::prelude::*;
use std::{path::PathBuf, sync::Arc, time::Instant};

fn main() -> Result<()> {
    let det_model_path = "ppocrv5_mobile_det.onnx";
    let rec_model_path = "ppocrv5_mobile_rec.onnx";
    let dict_path = "ppocrv5_dict.txt";

    let image_paths: Vec<PathBuf> = std::fs::read_dir("./images")
        .context("missing ./images directory")?
        .filter_map(|e| e.ok())
        .map(|e| e.path())
        .filter(|p| {
            let ext = p.extension().and_then(|x| x.to_str()).unwrap_or("").to_lowercase();
            matches!(ext.as_str(), "jpg" | "jpeg" | "png" | "webp" | "bmp")
        })
        .collect();

    anyhow::ensure!(!image_paths.is_empty(), "put some images into ./images first");

    // Build ONE engine (weights loaded once)
    let ocr = OAROCRBuilder::new(det_model_path, rec_model_path, dict_path).build()?;
    let ocr = Arc::new(ocr);

    let images: Vec<_> = image_paths
        .iter()
        .map(|p| load_image(p).with_context(|| format!("failed to load image: {p:?}")))
        .collect::<Result<Vec<_>>>()?;

    let t0 = Instant::now();

    // Parallel inference: share the same engine
    let outputs: Vec<_> = images
        .par_iter()
        .map(|img| {
            // oar-ocr expects Vec<image> for batch; we do batch=1 per task
            ocr.predict(vec![img.clone()])
        })
        .collect();

    let elapsed = t0.elapsed();

    // print a tiny summary
    let mut ok = 0usize;
    let mut fail = 0usize;

    for out in outputs {
        match out {
            Ok(batch) => {
                ok += 1;
                // show first few lines
                if let Some(first) = batch.get(0) {
                    for tr in first.text_regions.iter().take(3) {
                        if let Some((text, conf)) = tr.text_with_confidence() {
                            println!("{text} ({conf:.2})");
                        }
                    }
                }
            }
            Err(e) => {
                fail += 1;
                eprintln!("predict failed: {e:#}");
            }
        }
    }

    println!(
        "\nDone. ok={ok} fail={fail} total_images={} elapsed={:?} avg={:?}/img",
        images.len(),
        elapsed,
        elapsed / (images.len() as u32)
    );

    Ok(())
}

[aianau]

Thank you very much for the response!