Skip to content

[New Model] Add TimeSformer model#18908

Merged
NielsRogge merged 99 commits intohuggingface:mainfrom
fcakyon:timesformer
Dec 2, 2022
Merged

[New Model] Add TimeSformer model#18908
NielsRogge merged 99 commits intohuggingface:mainfrom
fcakyon:timesformer

Conversation

@fcakyon
Copy link
Contributor

@fcakyon fcakyon commented Sep 6, 2022 •
edited
Loading

What does this PR do?

Fixes #18724

  • Create a working environment for successful inference with the original source code
  • Create a debugging script for the original source code
  • Separate original model from original preprocessing pipeline
  • Test the original model with transformers/VideoMAEFeatureExtractor preprocessing pipeline
  • Port TimeSformer to HuggingFace/transformers
  • Adds tests for transformers/TimeSformer implementation
  • Update variable names to be more explicit

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

@NielsRogge

@fcakyon fcakyon changed the title add timesformer add TimeSformer model Sep 6, 2022
@fcakyon fcakyon marked this pull request as draft September 6, 2022 19:40
@fcakyon fcakyon changed the title add TimeSformer model [WIP] add TimeSformer model Sep 6, 2022
fcakyon added 26 commits September 6, 2022 23:06
@fcakyon
apply fix-copies
5319515
@fcakyon
reformat style
9a28a8d
@fcakyon
revert back some incoorect style updates
1a5a80d
@fcakyon
init timesformer
1486457
@fcakyon
apply fix-copies
308d155
@fcakyon
reformat style
f1b565c
@fcakyon
revert back some incoorect style updates
485352f
@fcakyon
Merge branch 'timesformer' of https://github.com/fcakyon/transformers …
2da62c2 
…into timesformer
@fcakyon
update timseformer doc
18e026e
@fcakyon
Merge branch 'main' of https://github.com/fcakyon/transformers into t…
8ae5693 
…imesformer
@fcakyon
add some functions and classes
2e565fa
@fcakyon
Merge branch 'main' of https://github.com/fcakyon/transformers into t…
e19d62d 
…imesformer
@fcakyon
add new config params
5233173
@fcakyon
implement multiple classes
9e38cab
@fcakyon
update TimeSformerLayer
08ce823
@fcakyon
update TimeSformerModel, TimeSformerPreTrainedModel, TimeSformerEncoder
e0f682d
@fcakyon
several fixes
f7ee70c
@fcakyon
Merge branch 'main' of https://github.com/fcakyon/transformers into t…
a713c20 
…imesformer
@fcakyon
reformat
27abdcd
@fcakyon
temporary update
6d19175
@fcakyon
Merge branch 'main' of https://github.com/fcakyon/transformers into t…
d1447aa 
…imesformer
@fcakyon
fix some typos
71c3c8c
@fcakyon
fix weight converter
3266bd9
@fcakyon
more fixes
cffad1b
@fcakyon
fix a typo
707faf7
@fcakyon
Merge branch 'main' of https://github.com/fcakyon/transformers into t…
a349fbe 
…imesformer
@fcakyon fcakyon requested a review from NielsRogge November 20, 2022 17:40
NielsRogge
NielsRogge approved these changes Nov 21, 2022
View reviewed changes
Copy link
Contributor

@NielsRogge NielsRogge left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your work, looks great to me. Pinging @amyeroberts for the feature extractor part; I guess we can't replace it with VideoProcessor for the moment?

@NielsRogge NielsRogge requested a review from sgugger November 21, 2022 08:17
@fcakyon fcakyon changed the title [WIP] add TimeSformer model add TimeSformer model Nov 21, 2022
@fcakyon fcakyon changed the title add TimeSformer model [New Model] Add TimeSformer model Nov 21, 2022
@fcakyon
Copy link
Contributor Author

fcakyon commented Nov 21, 2022

Thanks for your work, looks great to me.

Thanks for all the constructive feedback!

sgugger
sgugger approved these changes Nov 21, 2022
View reviewed changes
Copy link
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for adding this new model! I have a couple of nits and the checkpoints should be put in the right place, but good to merge otherwise!

NielsRogge
NielsRogge reviewed Nov 28, 2022
View reviewed changes
@fcakyon
Copy link
Contributor Author

fcakyon commented Dec 1, 2022 •
edited
Loading

Hello @NielsRogge, thanks a lot for all the help you have provided. I have opened multiple PRs for the config and model files of all timesformer variants:
https://huggingface.co/facebook/timesformer-base-finetuned-k400/discussions/1
https://huggingface.co/facebook/timesformer-hr-finetuned-ssv2/discussions/1
https://huggingface.co/facebook/timesformer-hr-finetuned-k600/discussions/1
https://huggingface.co/facebook/timesformer-hr-finetuned-k400/discussions/1
https://huggingface.co/facebook/timesformer-base-finetuned-k600/discussions/2
https://huggingface.co/facebook/timesformer-base-finetuned-ssv2/discussions/2

Is there anything I should do about this PR?

@sgugger
Copy link
Collaborator

sgugger commented Dec 1, 2022

Looks great on my side and ready to merge! Will let @NielsRogge double-check on last time and merge if he's happy :-)

@NielsRogge NielsRogge merged commit cc3d0e1 into huggingface:main Dec 2, 2022
@NielsRogge
Copy link
Contributor

NielsRogge commented Dec 2, 2022

Thanks for all your work!

Feel free to share on social media, we'll amplify ;)

@fcakyon
Copy link
Contributor Author

fcakyon commented Dec 2, 2022

Thanks a lot @NielsRogge, will share it after preparing a space :)

@fcakyon fcakyon deleted the timesformer branch December 2, 2022 14:16
@fcakyon
Copy link
Contributor Author

fcakyon commented Dec 4, 2022 •
edited
Loading

Thanks for all your work!

Feel free to share on social media, we'll amplify ;)

I have shared it on Twitter and Linkedin with the space demo link 🚀

@fcakyon fcakyon mentioned this pull request Dec 5, 2022
Open
2 tasks
mpierrau pushed a commit to mpierrau/transformers that referenced this pull request Dec 15, 2022
@fcakyon
[New Model] Add TimeSformer model (huggingface#18908)
62ff7e5 
* init timesformer

* apply fix-copies

* reformat style

* revert back some incoorect style updates

* init timesformer

* apply fix-copies

* reformat style

* revert back some incoorect style updates

* update timseformer doc

* add some functions and classes

* add new config params

* implement multiple classes

* update TimeSformerLayer

* update TimeSformerModel, TimeSformerPreTrainedModel, TimeSformerEncoder

* several fixes

* reformat

* temporary update

* fix some typos

* fix weight converter

* more fixes

* fix a typo

* fix typo

* remove redundant params

* fix for latest hf-hub

* merge fix

* fix some checks

* video classification works with einops

* add paper info to docs

* merge fix

* remove redundant line

* remove redundant docstring

* update config

* fix some typos

* fix converter

* update some test constants

* refactor einops functions

* reformat

* fix a comment

* remove redundat imports

* reformat

* fix a typo

* remove comment

* remove unused imports

* remove redundant doc line

* reformat

* add missing line

* fix docs

* fix timesformer auto feat ext

* add unittests

* reformat

* fix docs

* some fixes and updates

* fix readme

* fix modeling

* fix readme

* update index

* revert _toctree.yml changes

* update timseformer.mdx

* update drop_path_prob to drop_path_rate

* add dosctring for drop_path_rate

* update TimeSformerPatchEmbed naming

* remove to_2tuple

* explicit use of nn.functional

* reformat

* many updates from review comments

* fix a typo

* reformat

* remove assert, better variable name

* make variable names more explicit

* add some adapted from

* more explicit variable names

* remove redundant docstring

* fix initilaization

* move permute inside embedding

* update class names

* remove unused imports

* add test for video classification

* update PretrainedModel with PreTrainedModel

* remove double permute

* update based on sylvain's review

* aply auto fix

* update image_processing_auto for timesformer

* update hub urls

* reformat

* remove duplicate import

* update doc link
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@NielsRogge NielsRogge NielsRogge approved these changes

+2 more reviewers

@amyeroberts amyeroberts amyeroberts left review comments

@sgugger sgugger sgugger approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

add timesformer model

6 participants

@fcakyon @HuggingFaceDocBuilderDev @Darktex @sgugger @NielsRogge @amyeroberts

Comments