bp_names not considered during analysis

[DMBurrito]

I am training my model using custom body parts and changed the project_bp_names.csv file to reflect these changes. These body part names were used by the software for outlier correction successfully.

However, from the feature extraction step onwards, the output files, i.e. all csv files in folders features_extracted, targets_inserted, validation, and so on all use the default names for the 1 animal, 8 body parts option I originally selected when creating the project.

Is it possible to correct this error?

Thank you

[sronilsson]

Hey @DMBurrito - thanks for reporting. I think it may be related to a setting in the project_config.ini, under pose settings, but away from the computer and can't remember the section or option on the top of my head...

Could you share the project_config.ini with me here, and I can take a look and get back to you tomorrow?

[DMBurrito]

Hi @sronilsson, Thank you for your swift reply.
Github would not let me attach a .ini file but so I made a txt file with the content of project_config.ini

project_config.txt

I also attached the project_bp_names.csv.

project_bp_names.csv

Thank you so much

[sronilsson]

Thanks @DMBurrito - in the project_config.ini, there is a pose_estimation_body_parts setting that is set to 8 in your case:

8 represents the default setting for 1 animal and 8 body-parts as you say. When SimBA sees that 8 during feature extraction, it may revert whatever body-part names you have set manually.

To fix it, change that 8 to user_defined, then save the project_config.ini, and rerun it in SimBA. Does your columns still get overwritten?

NOTE: SimBA looks at thepose_estimation_body_parts setting, and depending on what it says, extract different feature sets. If the setting is user_defined, SimBA can't make any assumption about what body-parts or species or camera angle that the user have, have extracts a smaller more generic feature set which may affect performance of the model.

[DMBurrito]

Thank you for your help.

Then it might be better to use user-defined pose configuration and restart from scratch. I am not using a mouse as a model and the default labels might not work for my animal. Is there a way to set the species and camera angle information somewhere? I do not see the option while creating a custom pose_config

One more question if I may. When attempting to label my model when creating a user_defined pose configuration, is there a way to change the size of the circles that locate the points. I am using a rather small organism and the circles are too close and overlapping.

[sronilsson]

Hey @DMBurrito

Sorry for the slow response on this. Yes if it's not a rodent, it is better to use a custom configuration. What species are you using?

I don't have a way of setting the species and camera angle in the GUI, the features will mainly be computed based on the distances between
the different body-parts and the sizes of the animals if you go for the custom option. There are however MANY calculations that are accessible in SimBA through the command line (listed
HERE, which can be called through GUI after writing some code (described HERE, and I've had to dip into when use-case is edge-case. Happy to help you use them in SimBA GUI if the default are not working because of species/setup.

Can you check if the circles are still very large in the latest version of SimBA (4.2.2) after running pip install simba-uw-tf-dev -U? If not looking great, can you send me a screengrab how it looks? I have set the default circles to be smaller in relation to the resolution (no more than 100th of the image height/width). However, if it doesn't work, I should make it a user-definable parameter. But to be honest, it doesn't matter so much, you can click anywhere, its just a visualization representing the configuration and is not used anywhere in the code - just an image representing the pose configuration for user reference.

[DMBurrito]

Thank you for your reply!

I forgot to mention but changing from 8 to user_defined solved my previous issue so thank you for that.

As for the current issue. I am using the nymphs of a species of stinkbug called Riptortus pedestris. They look similar to ants at that point. My videos include only one individual at a time and I am trying to train the model to recognize when they are moving, resting, and feeding.
I am currently going over the links you sent to figure out what to change to fit my organism, thank you again for your guidance.

As for the circles, I updated to 4.2.2 as you suggested, however the circles are still too big for my organism. Since it's just visualization I will try moving forward with these settings but I really appreciate you considering making it a user-definable parameter. Due to my organisms size, for instance, I have to change the labels size from the default 12 to a value of 2 in DeepLabCut during the labeling stage so the current label size in SimBA GUI ends up overlapping and completely covering the animal. I took a screenshot during the labeling process, this is when 7 of my 8 body parts are labeled.

[sronilsson]

That is so damn cool. I have many stinkbugs here at my place in North Carolina, find them crawling on me and the keyboard all the time, please do something about that :)

Just some general thoughts on the pose but I am not a domain expert on these bugs so take it with grain of salt!: When working with non-shape shifting animals like insects, the relationships (distances, position) between the body-parts on the main body doesn't vary too much. For example, if I only know the positions of the most anterior/posterior/center parts of the animal, I could infer the other body-parts positions fairly accurately. Or, if I know the acceleration/movement of the head/tail, the other body-parts movement/acceleration will rarely deviate much from the known movement/acceleration of the head/tail. This mean that tracking many body-parts on the animals main body/hull doesn't give much more information than just tracking 2-3 body-parts. Resting/movement you can score pretty accuratly with what you have (although with a lot of redundancy e.g., you can do it with fewer main body body-parts). However, if feeding somehow depends on leg movements, and you have no leg movement data, it may be tricky to tell it apart from resting.

I've been in situations where pose-estimated keypoints are not good representations of for insect movements, and I've had to compute these lines like in the docs and below, and then compute how the lines move and intersects with other things to get good classifiers. Not saying thats the way to go, but it can be a good way to align what the human sees when they score behavior to what the computer sees when it scores behavior. But, again with grain of salt as I don't know what a stinkbug looks like when it eats.

[DMBurrito]

I'll see what I can you about that! ;)

Thank you for these pointers. When going over the literature you recommended I gathered that, like you said, as a non-shape shifting organism the changes in body shape will not provide much information. Actually the way I structured the model was to have a tracker at the beginning and end of each body section as well as a central one for the abdomen and thorax, and finally one on each eye. I thought the left and right eye would be useful when tracking the turns of the insect. As for the selection of the other points, although the shape of the body doesn't change much, they frequently arch themselves backwards for grooming and such so I thought that being able to record angular change between body parts might be relevant. Especially because besides resting and feeding, my 3rd and only really complex behavior is feeding and from our observations the best determinant of feeding is not the legs as might be the case in other insects, but rather a up and down bobbing of the head and thorax while the rest of the body remains static. It looks somewhat similar to a crane (the heavy machinery, not bird) with the abdomen staying firmly on the ground and the head and thorax stiffening in a straight line and moving up and down at the hinge between abdomen and thorax.
For the record I thought about labeling the proboscis but it's hidden too often to be useful due to our camera angle.

Based on your experience do you think this is something that SimBa could readily detect? I can also reduce the number of labeled body points if you think that would improve the behavior detection ability.
Also I am using a ROI to determine the feeding area to simplify feeding detection when up and down motion occurs within the ROI, however the food source was placed randomly in each recording, could this affect the performance?

Edit: I forgot to include the Feature extraction code I am currently writing, I started from the feature_extractor_user_defined.py and modified based on my needs. My current issue is that it apparently runs the feature extraction script correctly but fails to output any file that I could use to train the machine model. At first I thought it just was failing to find the correct output directory, but after debugging for a while it sees the right output directory but just doesn't output anything.

riptortus_feature_extractor_user_defined_2.0.py

[sronilsson]

Got it thanks for that - you def know this better than I and have thought more carefully about this than me and all seems right.

The shifting locations of the ROI won’t matter - no raw spatial coordinate data is fed into the model, just the distances between the animal and the ROI(s), and if the animal is inside the ROI(s)or not.

Generally you can detect any behavior as long as human can observe it and you have good tracking. You seem to have a good understanding of the behavior you are trying to measure, like what signals are critical, like angular diffs.

As I mentioned before, when you use a user_defined pose-configuration, SimBA won’t know to measure things like angular differences in sliding windows unless you tell it to. It only calculates movements and distances between body-parts. It may be enough signal in those measure to get the behaviors detected. However, it’s possible to tailor the features to be best correlated with you annotations, i.e., to compute features that align with what human looks at to score these behaviors, like anglers diffs across time. This approach has been done in SimBA when working with ants, termites, hornets and cavefish for example. Details how to do this is HERE, but would be happy help put script together with your expertise should you bump into any blockers...

[sronilsson]

PS. Ah just saw your edit let me take a look.

[sronilsson]

I took a look but I can't see where data_df is defined, it imports from from simba.feature_extractors import FeatureSubset which isn't a class.

Instead of feature_extractor_user_defined.py, maybe take a look at simba.feature_extractors.cave_fish_featurizer.py, simba.feature_extractors.aggression_feature_extractor.py, or simba.feature_extractors.mitra_feature_extractor.py. If you template according to that, which a class with 3 methods: init, run, and save. If you have a first template code according to those, I can clean it up.

[DMBurrito]

Thank you so much for the pointers! Truth be told I am not that proficient at python so your help is greatly appreciated.
I also apologize for taking so long to answer, I am currently working in Japan and we have a 14 hours time difference if I am not mistaken!

After going over the cave fish featurizer, it seems most of the data it collects and computes is suitable for my project so I just changed the BP names throughout the code. Due to the size and shape of my enclosures (petri dishes) I tried removing the distance to border measurements, however this broke the script so I left it in for the moment. I also tried to add the code from angle_3pt since the fish code already calculated the angle between 2 points, but I think the angle between 3 could be valuable.

Currently the script based on the fish featurizer works, but the one with the added 3 points angle calculation does not.
I'm thinking maybe I am not calling it correctly or there may be an error with where I set the definition. Additionally, the analysis yields very low probabilities so I'm still trying to improve its ability to learn.

Any help is appreciated!

Riptortus_featurizer_3_plus_angles.py

[sronilsson]

Thanks @DMBurrito - no worries about time difference, a friend just messaged me holidaying in Shibuya, BUSY and beautiful was the verdict. I have some 3pt angle calculators e.g., here or here we can call that are quick. I will try and update this later today or tomorrow.

I will try and reproduce your project from the info I have. However, if you have time, could you share a complete minimal project that you have? E.g., the whole project compressed but yank the videos and any large files. That will help me avoid situation where the updated code work for me but then not for you. If not possible, no worries and I try a SimBA project that I assume has all the info your project has.

[sronilsson]

@DMBurrito - in the meantime, you can try with this updated feature extractor attached.

Open your SImBA project, select the path to this file (whichever directory it is in, and click to extract features):

riptortus_featurizer.py

[DMBurrito]

@sronilsson Thank you so much! I will try the attached extractor!

For the minimal project, should I keep or remove the frames? Also is there an e-mail where I could send it to you? I will compress the project directory with the large files removed as you mentioned and send it to you.