Open
Conversation
I'm a little dissapointed to see that in the newly regenerated Fig 2 I get an R^2 of .44 instead of the expected .49. Not sure whats going on here because as far as I can tell this should be the code exaclty as Monika had left it, with only superficial changes such as wrapping things in a function and pointing to the correct filenams. But its pretty close. I plan to use this as a jumping off point to reimpement the minumum changes I want to change how e do normalizaiton, interpolation and smoothing.
and some of the other analyses that I don't think are particularly relevant..
I've added the photobleachign correction code. I am photobleaching following the logic outlined in Xiaowen Chen's Phys Rev E paper 2019. This is slightly different then what Jeff did in the 3dBrain MATLAB code. In that code he fit the exponential and then subtracted it off. Here I divide it off, but rescale the trace to match the original. Note: in its current state, the pipeline is not utilizing the photocorrected traces, but now at least I have the machinery to create it.
Now I can run the analysis all the way through and generate fig 2. Performance however is extremely low, probably because by using the raw traces I am skipping over some outlier removal that Jeff must have been taking care of. Also turned of the PCARaw analysis which seems to be unused and was throwing an error.
Plan is to use a Hampel filter but replacing the outliers with NaNs. See: https://towardsdatascience.com/outlier-detection-with-hampel-filter-85ddf523c73d
added a nanoutliers funciotn that uses a modified hampel filter to replace 3sigma outliers in a specificed framewindow size (here 3.3s, or 20 volumes) with NaNs. The functions run without error. but i haven't fully tested the results. Note that the function is super slow and seems to seriously slow down the analysis.
…tion Code should be functionally the same as the state Monika left it for the latest manuscript. Note fig 2v3 still generates 0.44 R2 instaed of 0.49 as expected (but still consistent with how I recieved the code in early june).
Made minimum changes necessary to run metaRunAnalysis and regenerate the analysis for AML32 moving. And also to run dataQualityCheck. Code should otherwise be as Monika left it in June.
Visually inspected the output on real data. Looks good. I added some visualization stuff.
Previously the photobleaching correction was failing by producing traces that blew up towards the latter half of the trace. The bug was an algebra error. To fit the exponent, at Francesco's suggestion, I rescaled the origional traces so that it was on the range of zero to one. This dramatically improves fitting performance and efficiency. So then I had to rescale the fitting parameters back. Previously I had been only scaling the a scaler in y=a exp(-bx)+ c. But of course c had been scaled to. So now I also rescale c. And that fixes everything. ADDITONAL: fixed a bug about using max instead of nanmax in the fitting routines.
moving photobleaching correiton and outlier detectiong and naning from Jeff's matlab code to python appears to decrease velocity decoding performance of our marquee dataset in fig 2v3. (but strangely decoding performance of turning is preserved). In previous commits, photobleaching correciton had been done according to Xiaowen's method in the Phys Rev E paper (division not subtraction). Visual inspection shows no obvious difference between the old matlab based photoblecahign correciotn and the new division approach in python. If anything, the new approach does a better job preserving variance across time. Remains unresolved why performance suffers. One hypothesis: maybe I am currently being less agressive about nan'ing entire columns then Jeff's matlab pipeline? Need to check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.