Skip to content

Memory integration#135

Merged
marcoSanti merged 33 commits intocapio-v2from
memory-integration
May 12, 2025
Merged

Memory integration#135
marcoSanti merged 33 commits intocapio-v2from
memory-integration

Conversation

@marcoSanti
Copy link
Copy Markdown
Member

This merge brings back the option to store the files inodes inside the CAPIO main memory. The mechanism can be used in two different ways:

  • either start the capio_server binary with --mem-only flag, which will store all the files inside the main memory automatically
  • Add in the CAPIO-CL configuration file a section called "storage", in which two vector objects (memory and fs) contains paths, globs or aliases. Those files will be treated accordingly.

This commit also adds tests to check capio server storage service, as well as bandwidth in both read and write.

Marco Edoardo Santimaria added 23 commits April 24, 2025 08:37
… fixed an issue for initialization of map of files to handle in memory
This commit fixes the read operation when file is stored in memory.
TODO: fix as soon as possible the ReadRequestCacheMEM class, as right
now it might contain bugs somewhere that are not clear. Also some
varibles can be removed
Caches now work independently of size. Increased posix read cache line
size to 32K of data
Since caches are used to improve performance for small size IO
operation, when a read() operation that targets a size greater than the
size of the read cache line size is issued, the cache is being bypassed
and the read operation occurs directly from the shared memory segment.
@marcoSanti marcoSanti force-pushed the memory-integration branch from d668929 to f7bc6bf Compare May 6, 2025 08:00
@marcoSanti marcoSanti force-pushed the memory-integration branch from 84570bd to c9ab329 Compare May 6, 2025 09:43
@marcoSanti marcoSanti force-pushed the memory-integration branch from a969efc to 2718483 Compare May 6, 2025 11:42
Marco Edoardo Santimaria added 2 commits May 6, 2025 13:47
@marcoSanti marcoSanti force-pushed the memory-integration branch 20 times, most recently from 5dc7178 to 18d4ef6 Compare May 12, 2025 12:18
@marcoSanti marcoSanti force-pushed the memory-integration branch from 18d4ef6 to 1f88af1 Compare May 12, 2025 12:21
@codecov
Copy link
Copy Markdown

codecov bot commented May 12, 2025

Codecov Report

Attention: Patch coverage is 64.53901% with 100 lines in your changes missing coverage. Please review.

Project coverage is 51.41%. Comparing base (4a24887) to head (e2b0460).
Report is 35 commits behind head on capio-v2.

Files with missing lines Patch % Lines
src/posix/utils/cache/read_request_cache_mem.hpp 32.30% 18 Missing and 26 partials ⚠️
src/server/client-manager/handlers/read.hpp 31.81% 2 Missing and 13 partials ⚠️
src/server/file-manager/file_manager_impl.hpp 10.00% 1 Missing and 8 partials ⚠️
src/server/capio_server.cpp 57.14% 0 Missing and 6 partials ⚠️
...rver/storage-service/CapioFile/CapioMemoryFile.hpp 84.00% 2 Missing and 2 partials ⚠️
src/posix/handlers/open.hpp 50.00% 2 Missing and 1 partial ⚠️
src/posix/utils/filesystem.hpp 66.66% 1 Missing and 2 partials ⚠️
...c/server/storage-service/capio_storage_service.hpp 87.50% 2 Missing and 1 partial ⚠️
src/posix/handlers/read.hpp 92.59% 1 Missing and 1 partial ⚠️
src/posix/utils/cache.hpp 50.00% 0 Missing and 2 partials ⚠️
... and 6 more
Additional details and impacted files
@@             Coverage Diff              @@
##           capio-v2     #135      +/-   ##
============================================
+ Coverage     47.46%   51.41%   +3.94%     
============================================
  Files            71       71              
  Lines          3339     3637     +298     
  Branches       1913     2099     +186     
============================================
+ Hits           1585     1870     +285     
+ Misses          865      844      -21     
- Partials        889      923      +34     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@marcoSanti marcoSanti merged commit dd4d8aa into capio-v2 May 12, 2025
28 checks passed
@marcoSanti marcoSanti deleted the memory-integration branch May 12, 2025 13:01
marcoSanti pushed a commit that referenced this pull request Jun 22, 2025
- Fixed mechanism to send memory storage regex
- Added flag to store everything in memory
- Added request for memory file in init_client()
- Fixed an issue in return value for posix in memory write handler, and fixed an issue for initialization of map of files to handle in memory
- Fixed write in memory for simple test
- Fix on writev
- Fix on cache wrong bytes. TODO: fix wrong read size
- Completed IO operations on single machine
- Added test for read and write in memory file. broken when reaching 1.3MB of data
- Bugfix on offset calculation in capio memory file
- Fixed reads on different mem pages
- Fixed custom cache line size issues
- Caches now work independently of size. Increased posix read cache line
  size to 32K of data
- Improved test with bandwidth data
- Performance improvement for data transfer: Since caches are used to improve
  performance for small size IO operation, when a read() operation that targets
  a size greater than the size of the read cache line size is issued, the cache is being bypassed
  and the read operation occurs directly from the shared memory segment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant