Skip to content

Add rgwdedup ut#4

Open
ssdohammer-sl wants to merge 12 commits intoreleasefrom
add-rgwdedup-ut
Open

Add rgwdedup ut#4
ssdohammer-sl wants to merge 12 commits intoreleasefrom
add-rgwdedup-ut

Conversation

@ssdohammer-sl
Copy link
Collaborator

@ssdohammer-sl ssdohammer-sl commented Jul 7, 2023

This pull request includes followings.

  • Add unit tests
  • Fix cold object name against possible same name
  • Improve RGWFPManager fp_map entry eviction

Contribution Guidelines

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows

@ssdohammer-sl ssdohammer-sl force-pushed the release branch 6 times, most recently from 249913f to 80f9929 Compare August 9, 2023 08:34
@yangdaegon yangdaegon force-pushed the release branch 2 times, most recently from 4a9510b to a021274 Compare August 10, 2023 06:08
@ssdohammer-sl ssdohammer-sl force-pushed the release branch 2 times, most recently from 747a251 to d10384a Compare August 13, 2023 17:30
@ssdohammer-sl ssdohammer-sl force-pushed the release branch 9 times, most recently from a3687fb to fe41720 Compare August 21, 2023 07:21
@ssdohammer-sl
rgw: add RGWDedup and RGWDedupManager
2a14f97 
This commit adds codes of RGWDedup and RGWDedupManager.
RGWDedup is a function that enables deduplication into RGW layer.
 - RGWDedup: an instance that controlls RGWDedupManager during its lifecycle
 - DedupManager: a thread that manages whole deduplication routine

Signed-off-by: Sungmin Lee <sung_min.lee@samsung.com>
@yangdaegon @ssdohammer-sl
rgw: add RGWFPManager
1404277 
RGWFPManager is a component of RGWDedup that stores chunks information.

Signed-off-by: daegon.yang <daegon.yang@samsung.com>
Signed-off-by: Sungmin Lee <sung_min.lee@samsung.com>
yangdaegon and others added 10 commits August 26, 2023 15:15
@yangdaegon @ssdohammer-sl
rgw: add RGWDedupWorker
53bde58 
RGWDedupWorker does chunking RADOS objects and stores chunks' fingerprints in FPManager.
If the fingerprint is already stored in the FPManager, chunk dedup is performed.

This commit implements RGWDedupWorker's dedup logic.

Signed-off-by: daegon.yang <daegon.yang@samsung.com>
Signed-off-by: Sungmin Lee <sung_min.lee@samsung.com>
@ssdohammer-sl
rgw: add RGWChunkScrubWorker
9eedc69 
RGWChunkScrubWorker finds reference mismatches between metadata objects and chunk objects and fix them.
It compares all the chunk objects’ references and referenced metadata objects’ chunk information.
During scrub processing, if any reference mismatch is found, the workers fix it.

Signed-off-by: Sungmin Lee <sung_min.lee@samsung.com>
@yangdaegon @ssdohammer-sl
rgw: add RGWDedup perfcounter for monitoring dashboard
c402fb5 
Signed-off-by: daegon.yang <daegon.yang@samsung.com>
@ssdohammer-sl
rgw: enable multi-RGWDedup object sharding
943375c 
In an environment where multiple-RGW daemons are running in a zone,
RGWDedup also supports to operate in a scalable manner.

Signed-off-by: Sungmin Lee <sung_min.lee@samsung.com>
Signed-off-by: daegon.yang <daegon.yang@samsung.com>
@ssdohammer-sl
rgw: do not allow dedup when the chunk has too many references
835175a 
Too many references in a chunk object causes chunk related operation slow.
This commit constrains chunk dedup if the number of references of the chunk object
exceeds max_chunk_ref_size.

Signed-off-by: Sungmin Lee <sung_min.lee@samsung.com>
@ssdohammer-sl
osd: fix manifest object not to be promoted when references_chunk called
d84650a 
When a cls_cas_references_chunk() is called on a chunked metadata object,
it makes the object's chunks be promoted in maybe_handle_manifest_detail().
It happens, for instance, while doing a chunk scrub job.

However, this operation doesn't need to get evicted data.
It only needs metadata information that already exists in metadata object.
To prevent this object promotion, this commit adds an exception handling for this.

Signed-off-by: Sungmin Lee <sung_min.lee@samsung.com>
@ssdohammer-sl
rgw: add RGWDedupWorker unit test
939de11 
Signed-off-by: Sungmin Lee <sung_min.lee@samsung.com>
@ssdohammer-sl
rgw: add RGWFPManager unit test
f02ed3c 
Signed-off-by: Sungmin Lee <sung_min.lee@samsung.com>
@ssdohammer-sl
rgw: add RGWChunkScrubWorker unit test
ed2b022 
Signed-off-by: Sungmin Lee <sung_min.lee@samsung.com>
@ssdohammer-sl
rgw: add RGWDedupManager unit test
b1c59a8 
Signed-off-by: Sungmin Lee <sung_min.lee@samsung.com>
@yangdaegon
Copy link
Member

yangdaegon commented Aug 30, 2023

TODO: rebase to release branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yangdaegon yangdaegon Awaiting requested review from yangdaegon

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ssdohammer-sl @yangdaegon