What's not working?

[DrCytometer]

Any general problems, things you couldn't understand from the instructions, stuff that outright failed when you tried to do it? List stuff here with a description of what happened. Other people may have had the same issues and may even have found a solution.

[claude-chew]

I have been having a persistent error when trying to get spectral variants for ID7000 data. I have managed to get it to work for Aurora and Xenith but when calling get.spectral.variants using the following:

variants <- get.spectral.variants( control.dir,
control.file,
asp,
spectra,
af.spectra,
pos.quantile = 0.8,
som.dim = 40,
parallel = TRUE,
verbose = TRUE )

I end up with this error when it reaches one of my reference controls in particular (RY586):

Error in apply(neg.selected, 2, median) :
dim(X) must have a positive length

Any help would be appreciated, I tried using Copilot/GPT5 and its solutions ended up being absolutely useless and convoluted.

I am happy to share more of my code for context.

Thanks,
Claude

[DrCytometer]

Well, one issue is that you're drawing a 40 x 40 SOM grid (som.dim = 40), which is 1600 nodes. That means you're splitting your cells across 1600 clusters, and so you could get an error if one of those nodes has one or fewer cells in it. Have you tried running with the default settings? I can put in some safeguards for this, as I'll be doing some improvements to this step anyway. Also, using som.dim = 40 will perform 1600 unmixing calculations per fluorophore per cell later if you use the "slow" method for per-cell optimization.

The line that's being flagged relates to the negative control, though, suggesting there's no data present there as read by the function. What is your unstained for the RY586? Is it a universal negative (show flow.control$universal.negative)? Are these beads for RY586 or cells? It's possible you're using a combination of factors that I haven't explicitly tested, and that I haven't written it properly. If you're able to share a minimal set of files, say an FCS for a fluorophore that works, the FCS for RY586 and any related unstained FCS files, that would help with troubleshooting.

[claude-chew]

Ok, I had tried playing around with some of the parameters (pos.quantile, som.dim) to get a little further along and so far some combination of setting number of dimensions close to the number of fluorophores seemed to help along with decreasing the pos.quantile value. I will most likely revert to default since I wondered if it was the RY586 control in particular.

I don't have a universal negative bead for reference controls for this data set. I have an unstained cells sample for sample AF extraction.

I'm going to drop the reference control files in the following link as well as my control file and AutoSpectral R markdown if that helps, so that gives a little more context to what I'm trying to do:

https://bcmedu-my.sharepoint.com/:f:/g/personal/u242081_bcm_edu/EjMDaqz2u9lMjVhAZzL5HiQBXAS9784y_WppScUU1mWOYg?e=S2JPrq

Also, I have managed to unmix the data files without using get.spectral.variants but want to take advantage of per-cell fluorophore optimization.

Thanks very much for your help,
Claude

[DrCytometer]

OMIP112_AutoSpectral_OTB.txt

Thanks, that's really helpful. I'll look through this to see if I can figure out what's going wrong. Attached is a .txt file (GitHub won't let me upload R files) with some notes. Reducing the pos.quantile value is likely a really bad idea. I'm going to remove that argument because changing from the default value will almost certainly cause unmixing problems.

[claude-chew]

Thanks for this, this was very helpful!

[DrCytometer]

I have opened this as an "Issue" since it is a specific problem. Also, FYI, the asp$parallel flag is no longer in use. I switched everything over to using parallel arguments in the specific function calls.

[DrCytometer]

There are issues with the control file that I haven't anticipated or written checks for. The large.gate setting is only for cells. The cells bearing the marker will have different scatter profiles if they are larger cells, so there is a large.gate option for granulocyte, monocyte and DC markers. This is irrelevant for bead controls because the bead scatter of the positive events does not change depending on the antigen-specificity of the conjugate. So, don't set large.gate = TRUE for bead controls. I will write an automated check for that. Note that this affects your RY586 control.

Secondly, you have marked the NovaFluor Blue 660 control as is.viability = TRUE. This isn't the viability marker, and the Zombie NIR is no longer marked as is.viability, which happens automatically, so it would appear you've edited this. While this won't cause critical issues, it doesn't help. I will write a check for this as well.

FYI, if you name your single-stained controls with the marker name as well as the fluorophore, AutoSpectral will pick up on this and fill in your marker names in the control file (as best it can).

Also, unmixing of ID7000 files will only work if the controls and samples are run on the same settings. Sony recommends using standardized mode to run controls and then altering the voltages for your samples. If you have done that, the unmixing will be inaccurate because AutoSpectral does not have access to the instrument QC to adjust the spectra in line with detector-dependent sensitivity changes.

[claude-chew]

Ok thanks for that, I had been staring at the control file and trying changes and changing parameters to the code that I went cross eyed and missed those very obvious things.

Yes, I noticed that in the latest version of Spectroflo that it matches fluor and marker much better than even 1-2 iterations ago.

Let me make these changes and try again.

Thanks for your help and apologies for those very obvious, silly mistakes in my control file.

[claude-chew]

And I am an R newbie, so it's likely that I will break code easily!

[DrCytometer]

A patch for this problem has been pushed to dev branch. You will need to install @dev to access it.

I've made some other changes to the spectral variant search. The pos.quantile and sim.threshold arguments have been removed and replaced with fixed values since they should be fixed. I have also done some testing on the som.dim variable, which determines the number of variant spectra produced per fluorophore. The unmixing optimization appears to work better with fewer, higher quality options, so I have set the default to 7 rather than 10. This halves the number of variants produced, cutting unmixing time by a factor of three or so. som.dim can be reduced further to 5, if desired, with similar results in less time. Finally, I have added a denoising step to reduce the impact of low-level autofluorescence variability on variant spectra. This will likely reduce any discontinuities in the data when using the "fast" per-cell fluorophore optimization. All of this requires a bit more testing with additional datasets, which is why the changes are in dev branch.

[claude-chew]

Thanks I'll install development version and try it

[claude-chew]

Thanks, I installed this and ran it, it did complete the get.spectral.variants function without and issues, it only gave me this one particular error message for Zombie NIR and did not generate a spectral plot for it:

"Error for fluorophore Zombie NIR: number of items to replace is not a multiple of replacement length"

[DrCytometer]

That should be fixed with the latest push to dev branch.

[DrCytometer]

Great! Thanks for the feedback. That likely means it only came up with one or no variants for that fluorophore, and that I have been sloppy about keeping the result as a matrix. It should be fine to proceed with the unmixing anyway--it should just skip optimization for Zombie NIR.

[claude-chew]

This is running dev branch and parallel = TRUE. Should I try running sequential?

[DrCytometer]

That is unlikely to solve it. There is another error in dev that I did not catch earlier because it only manifests infrequently. I have a remnant that subsets data that is now already subsetted. I’ll push the patch later.

Are you getting all fluorophores assessed when running parallel? It is dropping some for me currently on some datasets, most likely due to not exporting something to the clusters properly. I recommend sequential for now, but if it gives you all of them, great.

Apologies for this. It is, at least, a lot faster now. I am new to all of this, so I am probably setting up the package development wrong.

[claude-chew]

No worries, I am still very very new too, I'm glad that this is an R pipeline that actually works and is actively maintained, so thank you for that!

[claude-chew]

I've re-run the whole dev version for Aurora, ID7000, A8 data and it is much faster, so far not getting any errors.

[DrCytometer]

Try now with dev. I've pushed a patch for the issue causing the subsetting error in get.spectral.variants().

Yes, dev should be about twice as fast now, even without parallelization. I've put some benchmarks in the dev README from running on my laptop, which is not a good set up for this.

[claude-chew]

Ok, everything went swimmingly with Aurora, A8, ID7000 data. Now I am hitting this error on the clean.controls step for Xenith data:

Error in clean.controls(flow.control, asp) :
'list' object cannot be coerced to type 'logical'

What is happening here?

[claude-chew]

The Xenith data does have a universal bead negative control. I was able to just run the clean.controls as if there was not a universal negative so I worked around it for now.

[claude-chew]

Another set of Xenith data, now am getting this, have double checked my export, control file, not sure what it is flagging with inconsistent parameter names?

"Inconsistencies found in parameter names across FCS files.
Please inspect or this will likely fail."

[claude-chew]

Nevermind, it's an export issue from the Xenith software, am working through it with Thermo

[DrCytometer]

This should now be fixed in the latest push to dev branch.

[claude-chew]

Yes, thanks very much!

[SamGG]

Hi,
I downloaded the Mendeley archive called "AutoSpectral: Full Workflow Example" at https://data.mendeley.com/datasets/ch5dnspd79/1.

I opened various unmixing results available in the "AutoSpectral_unmixed" folder. Please find below the plot (FlowJo defaults) of "D4 Spleen_Set1 OLS with AF extraction.fcs" (left) and "D4 Spleen_Set1 AutoSpectral per-cell AF extraction.fcs" (right). No cell filtering was performed.

On the right plot (per-cell AF extraction), I don't understand "cissor" or "density jump" effect I see on the CD4-CD45+ population at the center and at zero. Is it a side effect of the dist0 minimization? Should I care?

[SamGG]

These are my words to express my feeling. The density is not continuous, what I call a "jump".

[DrCytometer]

This is an effect of the dist0 minimization. It occurs primarily on small fluorophore data sets where an essentially optimal solution has been found. In the paper (please read), use of the dist0 (fluorophore signal) minimization is recommended for large data sets, which is where the residual is much smaller and less informative.

More generally, you are looking at an incomplete step in the unmixing. The workflow should encompass both single cell AF extraction and per-cell fluorophore optimization, both for accuracy and to avoid (hopefully) discontinuities. I did not know how to code that in an efficient enough way, and also wanted people to be able to see and assess the impacts of the different parts on the unmixing, so I wrote the code to allow you to see the intermediate unmixing step. I am hoping that the next version will be fast enough to allow for a default where we don't need to stop part way through.

This particular data set is designed to allow maximum space for autofluorescence signal in a small fluorophore panel. The impact of single cell AF extraction should be lower here than in larger panels.

[SamGG]

Thanks for your answer. I don't understand every part of it, but I hope others do. Thanks for improving the software.
IMHO, it's important to get a view of each step in order to better which part is important and which is less.

[SamGG]

BTW, if you can point a public dataset that you consider as relevant, could you add thelink here?

[DrCytometer]

If you want to explore complex autofluorescence extraction, the messy tissue data set is a good one. You'll want to unmix both unstained and stained samples. If you want a cleaner data set with minimal autofluorescence but high parameters, you can try the ID7000 40C data.

[DrCytometer]

Yeah, I can't keep up with this. I have to take a break to do my job. Sorry.

[claude-chew]

Totally understand, thanks very much for all of your help and updates to this, it has been a game changer in unmixing for me.