Reproducibility of BBKNN results

[liangyf24]

Hi there,

Thank you very much for developing this wonderful tool for data integration.

However, I have encountered some reproducibility issues in my analysis and would really appreciate your suggestions. When running BBKNN with the code below, I find that the final UMAP embeddings are not reproducible, even though I have set all the random seeds that I can think of.

np.random.seed(202504)
random.seed(202504)
numba.set_num_threads(1)

adata=sc.read_h5ad("~/adata.h5ad")
adata

I run BBKNN using either of the following configurations:
%%time
sc.external.pp.bbknn(adata, batch_key="sample_id",use_rep='X_pca', n_pcs=50,neighbors_within_batch=3,) # (which uses the default Annoy backend)

or

%%time
sc.external.pp.bbknn(adata, batch_key="sample_id",use_rep='X_pca', n_pcs=30, neighbors_within_batch=3, computation="pynndescent", pynndescent_n_neighbors=100, pynndescent_random_state=0)

sc.tl.umap(adata_int_PBMC,random_state=202504)

2 results showed here:

I would like to ask whether this behavior is expected, because I use the same code in another datasets, I did not detect this question and if there are additional settings or recommended practices to achieve fully reproducible results when using BBKNN and UMAP.

Thank you very much for your time and help.

[ktpolanski]

You can try inspecting the actual graph output by the methods to see if there's a difference. I remember when developing the package, annoy behaved consistently between notebook restarts without needing to do any extra seeding. If all else fails, you can swap the backend to cKDTree which is an exact neighbour search and should be fully deterministic.

Also I presume you referring to the object as adata_int_PBMC in the UMAP call is just a lack of adjustment for the minimal example.

[liangyf24]

Thanks for you quickly reply!

I checked the h5ad of my connectivities, X_pca and X_umap. I pretty sure the input files have the same 'X_pca', but after my BBKNN step, the connectivities has been differed. I will try to use the cKDTree.

I have checked the code in another dataset, the output files are similar, only in this data I can not reproduce the results. I hope to know could certain data characteristics or structure affect reproducibility?

I try to subset 100000 cells and the results can be reproduced, but if I subset 500000 cells I can not reproduce the results.