Workspace ignore_missing_files

[ksebby]

I suggest changing the Workspace ignore_missing_files parameter to load_missing_file_data or something similar.

Currently:
ignore_missing_files = False (the default) does in fact ignore the missing samples--the gate data and sample ids are not loaded. Although there is a warning message.

ignore_missing_files = True does not ignore the missing files and loads the gate data and sample ids of missing FCS files.

I expected the opposite behavior from the name. This one tripped me up for a while...

[whitews]

Hi Karl,

Yeah, I remember implementing this out of necessity and not liking that kwarg name either. I agree it is unclear. I'd like to keep the names as concise as possible. How about allow_missing_files? Would that be clearer? Open to other ideas, just brainstorming.

Thanks for the input,
Scott

[ksebby]

I didn't like having 4 words either, but I couldn't think of something clear that is shorter. I like having 'load' in there because that is what the flag is controlling--whether samples that are in a FlowJo workspace are loaded into a fk.Workspace when the FCS file is not found and is consistent with the language in the docstring. allow_missing_files makes me think that an error would be thrown if it is False and an FCS file wasn't found. Maybe just load_missing?

FYI. My use case is for a workspace validator. We have had problems with 2 kinds of copying mistakes--bound to happen with many FCS files.

1. FCS files can get missed when dragging them into a FJ workspace.
2. People do the gating on their own computer in FJ, then drag the .wsp file with all the FCS files to a shared folder to generate a standardized report. Sometimes an FCS file gets missed when moving to the shared folder.

Now this is caught. We have a list of expected sample_ids for different assays and users get a message right away when an expected sample is completely missing or if it is in the workspace but the FCS file isn't there.

[whitews]

Hey Karl,

After thinking some more, you've convinced me. The only reason I have against your proposed name is the length and there's already another Workspace constructor kwarg that is just as long. I'd rather have a clearer name than a shorter one. We'll plan to go ahead with the load_missing_file_data name.

I'm converting this discussion to an issue and adding it to the 1.3.0 milestone. It'll have to go to 1.3 since it is a breaking API change. I have tentatively planned for that version to be released around April 2025. However, we can go ahead and make the change on the develop branch. Let me know if you want to submit a PR off that branch for some more contib cred 😉.

One additional thought reading your use cases. Should there be a Workspace attribute like missing_samples that holds a list of sample IDs that were referenced in the wsp but were not loaded? Or should that kind of bookkeeping be left to the user?

-Scott

[whitews]

Well, GitHub seems to have removed the ability to convert discussions to issues. I had to create a new issue here: #230.

[ksebby]

Thanks Scott!

Should there be a Workspace attribute like missing_samples that holds a list of sample IDs that were referenced in the wsp but were not loaded?

This would be a nice convenience, but not necessary. It can be done easily enough by getting sample id sets of all samples and loaded samples and taking the difference.