[ConvNetJS] Implement DropConnect for regularizing large fully-connected layers #3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ted layers Summary: This diff implements DropConnect, following the paper in http://cs.nyu.edu/~wanli/dropc/dropc.pdf. "When training with Dropout, a randomly selected subset of activations are set to zero within each layer. DropConnect instead sets a randomly selected subset of *weights* within the network to zero. Each unit thus receives input from a random subset of units in the previous layer." DropConnect has been shown to achieve significant improvements in model performance over DropOut. Test Plan: I'm not sure on how to proceed here - do you want me to add a demo demonstrating DropOut vs DropConnect on an otherwise identical dataset/architecture? I'd like to add unit tests but it's unclear how to add them into this project. I verified that existing demonstrations (MNIST, autoencoder, etc) work, and that adding a drop-connect layer allows training to continue as expected. Reviewers: CC:
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This diff implements DropConnect, following the paper in
http://cs.nyu.edu/~wanli/dropc/dropc.pdf.
"When training with Dropout, a randomly selected subset of activations
are set to zero within each layer. DropConnect instead sets a randomly
selected subset of weights within the network to zero. Each unit thus
receives input from a random subset of units in the previous layer."
DropConnect has been shown to achieve significant improvements in model
performance over DropOut.
Test Plan
I'm not sure on how to proceed here - do you want me to add a demo
demonstrating DropOut vs DropConnect on an otherwise identical
dataset/architecture?
I'd like to add unit tests but it's unclear how to add them into this
project.
I verified that existing demonstrations (MNIST, autoencoder, etc) work,
and that adding a drop-connect layer allows training to continue as expected.