[Feature] Support SWD and MS-SSIM metrics by LeoXing1996 · Pull Request #82 · open-mmlab/mmeval

LeoXing1996 · 2023-01-18T08:18:51Z

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Support SWD and MS-SSIM.

Modification

As title.

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repos?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMCls.
The documentation has been modified accordingly, like docstring or example tutorials.

CLAassistant · 2023-01-18T08:18:56Z

All committers have signed the CLA.

C1rN09

Generally good! Please also update api docs.

mmeval/metrics/ms_ssim.py

tests/test_metrics/test_swd.py

mmeval/metrics/ms_ssim.py

mmeval/metrics/swd.py

mmeval/metrics/ms_ssim.py

mmeval/metrics/swd.py

C1rN09

LGTM with 1 minor comment

mmeval/metrics/ms_ssim.py

.pre-commit-config.yaml

mmeval/metrics/ms_ssim.py

zhouzaida · 2023-03-15T08:29:42Z

Hi, have you verified the PR in mmediting?

zhouzaida · 2023-03-15T08:36:01Z

mmeval/metrics/ms_ssim.py

+            tuple: Pair containing the mean SSIM and contrast sensitivity
+                between `img1` and `img2`.


Suggested change

tuple: Pair containing the mean SSIM and contrast sensitivity

between `img1` and `img2`.

tuple: Pair containing the mean SSIM and contrast sensitivity

between `img1` and `img2`.

zhouzaida · 2023-03-15T08:53:00Z

mmeval/metrics/swd.py

+    """SWD (Sliced Wasserstein distance) metric. We calculate the SWD of two
+    sets of images in the following way. In every 'feed', we obtain the
+    Laplacian pyramids of every images and extract patches from the Laplacian
+    pyramids as descriptors. In 'summary', we normalize these descriptors along
+    channel, and reshape them so that we can use these descriptors to represent
+    the distribution of real/fake images. And we can calculate the sliced
+    Wasserstein distance of the real and fake descriptors as the SWD of the
+    real and fake images. Note that, as with the official implementation, we
+    multiply the result by 10 to prevent the value from being too small and to
+    facilitate comparison.


Suggested change

"""SWD (Sliced Wasserstein distance) metric. We calculate the SWD of two

sets of images in the following way. In every 'feed', we obtain the

Laplacian pyramids of every images and extract patches from the Laplacian

pyramids as descriptors. In 'summary', we normalize these descriptors along

channel, and reshape them so that we can use these descriptors to represent

the distribution of real/fake images. And we can calculate the sliced

Wasserstein distance of the real and fake descriptors as the SWD of the

real and fake images. Note that, as with the official implementation, we

multiply the result by 10 to prevent the value from being too small and to

facilitate comparison.

"""SWD (Sliced Wasserstein distance) metric.

We calculate the SWD of two sets of images in the following way. In every

'feed', we obtain the Laplacian pyramids of every image and extract patches

from the Laplacian pyramids as descriptors. In 'summary', we normalize

these descriptors along channel, and reshape them so that we can use

these descriptors to represent the distribution of real/fake images. And

we can calculate the sliced Wasserstein distance of the real and fake

descriptors as the SWD of the real and fake images. Note that, as with the

official implementation, we multiply the result by 10 to prevent the value

from being too small and to facilitate comparison.

zhouzaida · 2023-03-15T08:56:12Z

mmeval/metrics/swd.py

+                                              self.gaussian_k)
+
+        # init list to save fake desp and real desp
+        if self._results == []:


Suggested change

if self._results == []:

if not self._results:

zhouzaida · 2023-03-15T08:57:48Z

mmeval/metrics/swd.py

+        del real_descs
+        del fake_descs


Is it necessary to delete them immediately?

zhouzaida · 2023-03-15T08:58:34Z

mmeval/metrics/swd.py

+        self.dir_repeats = 4  # times of sampling directions
+        self.dirs_per_repeat = 128  # number of directions per sampling
+
+        self.resolution = resolution


This variable is not used.

zhouzaida · 2023-03-15T08:59:10Z

mmeval/metrics/swd.py

+        while resolution >= 16 and len(self._resolutions) < 4:
+            self._resolutions.append(resolution)
+            resolution //= 2


What will happen if resolution is less than 16?

zhouzaida · 2023-03-15T09:01:22Z

mmeval/metrics/swd.py

+
+        Returns: Dict[str, float]: The computed SWD metric.
+        """
+        results_fake, results_real = results


Suggested change

results_fake, results_real = results

fake_results, real_results = results

LeoXing1996 requested a review from ice-tong January 18, 2023 08:18

LeoXing1996 self-assigned this Jan 18, 2023

LeoXing1996 removed their assignment Jan 18, 2023

LeoXing1996 mentioned this pull request Jan 18, 2023

[Feature] Adopt mmeval's swd and ms-ssim open-mmlab/mmagic#1614

Closed

7 tasks

C1rN09 reviewed Feb 13, 2023

View reviewed changes

mmeval/metrics/ms_ssim.py Outdated Show resolved Hide resolved

mmeval/metrics/ms_ssim.py Show resolved Hide resolved

C1rN09 reviewed Feb 21, 2023

View reviewed changes

tests/test_metrics/test_swd.py Show resolved Hide resolved

mmeval/metrics/ms_ssim.py Outdated Show resolved Hide resolved

mmeval/metrics/swd.py Outdated Show resolved Hide resolved

mmeval/metrics/swd.py Outdated Show resolved Hide resolved

LeoXing1996 force-pushed the leoxing/add-swd-and-ms-ssim branch from 6c17aa5 to dc0e23f Compare March 2, 2023 05:58

LeoXing1996 requested a review from zhouzaida as a code owner March 2, 2023 05:58

LeoXing1996 requested a review from C1rN09 March 2, 2023 06:27

C1rN09 reviewed Mar 8, 2023

View reviewed changes

LeoXing1996 force-pushed the leoxing/add-swd-and-ms-ssim branch from dc0e23f to fb3387e Compare March 8, 2023 11:11

LeoXing1996 requested a review from C1rN09 March 8, 2023 11:41

LeoXing1996 force-pushed the leoxing/add-swd-and-ms-ssim branch from b84f63f to cc9d889 Compare March 9, 2023 05:13

C1rN09 approved these changes Mar 9, 2023

View reviewed changes

mmeval/metrics/ms_ssim.py Show resolved Hide resolved

LeoXing1996 requested a review from C1rN09 March 10, 2023 05:34

zhouzaida reviewed Mar 14, 2023

View reviewed changes

.pre-commit-config.yaml Outdated Show resolved Hide resolved

LeoXing1996 added 11 commits March 14, 2023 20:08

support ms-ssim

60a2037

update docstring and compulate logic of ms-ssim

04ba8d6

support swd metric

63a9b0d

revise unit test of ms-ssim

204f41c

revise docstring for SWD and MS-SSIM

80501b5

add more unit test for SWD

ea7696a

add shape checking for MS-SSIM input and revise docstring as comment

2988a63

support groundtruths input in MS-SSIM for reconstructive tasks

fb7824d

do not support groundtruths=None is MS-SSIM

5c035cf

revise unit test of swd

a965fe2

move word ignore setting to setup.cfg

1f4675b

LeoXing1996 force-pushed the leoxing/add-swd-and-ms-ssim branch from 0095367 to 1f4675b Compare March 14, 2023 12:10

LeoXing1996 requested a review from zhouzaida March 14, 2023 12:11

zhouzaida reviewed Mar 14, 2023

View reviewed changes

mmeval/metrics/ms_ssim.py Outdated Show resolved Hide resolved

zhouzaida reviewed Mar 14, 2023

View reviewed changes

mmeval/metrics/ms_ssim.py Outdated Show resolved Hide resolved

zhouzaida reviewed Mar 14, 2023

View reviewed changes

mmeval/metrics/ms_ssim.py Outdated Show resolved Hide resolved

revise ms-ssim as comment

c9bc955

LeoXing1996 requested a review from zhouzaida March 14, 2023 13:12

zhouzaida reviewed Mar 15, 2023

View reviewed changes

		tuple: Pair containing the mean SSIM and contrast sensitivity
		between `img1` and `img2`.

	results_fake, results_real = results
	fake_results, real_results = results

Conversation

LeoXing1996 commented Jan 18, 2023

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

Uh oh!

CLAassistant commented Jan 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

C1rN09 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

C1rN09 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhouzaida commented Mar 15, 2023

Uh oh!

zhouzaida Mar 15, 2023

Choose a reason for hiding this comment

Uh oh!

zhouzaida Mar 15, 2023

Choose a reason for hiding this comment

Uh oh!

zhouzaida Mar 15, 2023

Choose a reason for hiding this comment

Uh oh!

zhouzaida Mar 15, 2023

Choose a reason for hiding this comment

Uh oh!

zhouzaida Mar 15, 2023

Choose a reason for hiding this comment

Uh oh!

zhouzaida Mar 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhouzaida Mar 15, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CLAassistant commented Jan 18, 2023 •

edited

Loading

C1rN09 left a comment •

edited

Loading

zhouzaida Mar 15, 2023 •

edited

Loading