Overhaul BeeSerde and msg buffers #15
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Found a bug where I accidentally always sent the whole 4 MiB buffer instead of only as many byte as needed on outgoing stream/tcp requests. This might be the whole issue, please retry. |
iamjoemccormick
requested changes
Jul 10, 2025
There was a problem hiding this comment.
todo(blocking): I'm not sure where exactly, but using v8.0.0 meta/storage servers this appears to break quotas.
https://github.com/ThinkParQ/beegfs-rs/pull/185#discussion_r2031905565
Found a bug where I accidentally always sent the whole 4 MiB buffer instead of only as many byte as needed on outgoing stream/tcp requests. This might be the whole issue, please retry.
Unfortunately after rebuilding+restarting the mgmtd I get the exact same errors as before. In case it helps this is my mgmtd.toml quota config:
quota-enable = true
quota-enforce = true
quota-user-system-ids-min = 0
And my quota defaults, limits, and usage:
beegfs quota list-defaults
ID ALIAS TARGETS MIRRORS USER_SPACE_LIMIT USER_INODE_LIMIT GROUP_SPACE_LIMIT GROUP_INODE_LIMIT
s:1 default s:103,s:104,s:203,s:204 s:1 ∞ ∞ ∞ ∞
s:2 archive s:101,s:201 ∞ ∞ ∞ ∞
s:3 mirrored s:102,s:202 s:2 ∞ ∞ ∞ ∞
beegfs quota list-limits --uids=all --gids=all
NAME ID TYPE POOL SPACE INODE
_apt 100 user default 1.00TiB -
systemd-network 101 user default 1.00TiB -
systemd-resolve 102 user default 1.00TiB -
docker 999 group default 10.00GiB -
joe 1000 user default 1.00TiB -
joe 1000 group default - 10.00M
tux 1001 user default 1.00TiB 100.00M
tux 1001 group default 2.00GiB 100.00M
beegfs quota list-usage --uids=all --gids=all --page-size=1000
NAME ID TYPE POOL SPACE INODE
root 0 user default 105.61GiB/∞ 2.27M/∞
root 0 user archive 52.80GiB/∞ 1.14M/∞
root 0 user mirrored 52.80GiB/∞ 1.14M/∞
root 0 group default 99.96GiB/∞ 2.83M/∞
root 0 group archive 49.98GiB/∞ 1.42M/∞
root 0 group mirrored 49.98GiB/∞ 1.42M/∞
daemon 1 group default 16.00KiB/∞ 8.00/∞
daemon 1 group archive 8.00KiB/∞ 4.00/∞
daemon 1 group mirrored 8.00KiB/∞ 4.00/∞
adm 4 group default 61.50MiB/∞ 256.00/∞
adm 4 group archive 30.75MiB/∞ 128.00/∞
adm 4 group mirrored 30.75MiB/∞ 128.00/∞
tty 5 group default 80.00KiB/∞ 4.00/∞
tty 5 group archive 40.00KiB/∞ 2.00/∞
tty 5 group mirrored 40.00KiB/∞ 2.00/∞
man 6 user default 9.39MiB/∞ 708.00/∞
man 6 user archive 4.70MiB/∞ 354.00/∞
man 6 user mirrored 4.70MiB/∞ 354.00/∞
mail 8 group default 48.00KiB/∞ 12.00/∞
mail 8 group archive 24.00KiB/∞ 6.00/∞
mail 8 group mirrored 24.00KiB/∞ 6.00/∞
man 12 group default 9.44MiB/∞ 720.00/∞
man 12 group archive 4.72MiB/∞ 360.00/∞
man 12 group mirrored 4.72MiB/∞ 360.00/∞
shadow 42 group default 1.91MiB/∞ 140.00/∞
shadow 42 group archive 976.00KiB/∞ 70.00/∞
shadow 42 group mirrored 976.00KiB/∞ 70.00/∞
utmp 43 group default 4.78MiB/∞ 48.00/∞
utmp 43 group archive 2.39MiB/∞ 24.00/∞
utmp 43 group mirrored 2.39MiB/∞ 24.00/∞
staff 50 group default 128.00KiB/∞ 32.00/∞
staff 50 group archive 64.00KiB/∞ 16.00/∞
staff 50 group mirrored 64.00KiB/∞ 16.00/∞
_apt 100 user default 224.00KiB/1.00TiB 48.00/∞
_apt 100 user archive 112.00KiB/∞ 24.00/∞
_apt 100 user mirrored 112.00KiB/∞ 24.00/∞
systemd-network 101 user default 144.00KiB/1.00TiB 36.00/∞
systemd-network 101 user archive 72.00KiB/∞ 18.00/∞
systemd-network 101 user mirrored 72.00KiB/∞ 18.00/∞
systemd-journal 101 group default 9.91GiB/∞ 308.00/∞
systemd-journal 101 group archive 4.95GiB/∞ 154.00/∞
systemd-journal 101 group mirrored 4.95GiB/∞ 154.00/∞
systemd-resolve 102 user default 64.00KiB/1.00TiB 16.00/∞
systemd-resolve 102 user archive 32.00KiB/∞ 8.00/∞
systemd-resolve 102 user mirrored 32.00KiB/∞ 8.00/∞
systemd-network 102 group default 128.00KiB/∞ 32.00/∞
systemd-network 102 group archive 64.00KiB/∞ 16.00/∞
systemd-network 102 group mirrored 64.00KiB/∞ 16.00/∞
systemd-timesync 104 user default 48.00KiB/∞ 20.00/∞
systemd-timesync 104 user archive 24.00KiB/∞ 10.00/∞
systemd-timesync 104 user mirrored 24.00KiB/∞ 10.00/∞
messagebus 104 group default 224.00KiB/∞ 8.00/∞
messagebus 104 group archive 112.00KiB/∞ 4.00/∞
messagebus 104 group mirrored 112.00KiB/∞ 4.00/∞
pollinate 105 user default 16.00KiB/∞ 8.00/∞
pollinate 105 user archive 8.00KiB/∞ 4.00/∞
pollinate 105 user mirrored 8.00KiB/∞ 4.00/∞
systemd-timesync 105 group default 16.00KiB/∞ 8.00/∞
systemd-timesync 105 group archive 8.00KiB/∞ 4.00/∞
systemd-timesync 105 group mirrored 8.00KiB/∞ 4.00/∞
input 106 group default 64.00KiB/∞ 16.00/∞
input 106 group archive 32.00KiB/∞ 8.00/∞
input 106 group mirrored 32.00KiB/∞ 8.00/∞
syslog 107 user default 97.58MiB/∞ 68.00/∞
syslog 107 user archive 48.79MiB/∞ 34.00/∞
syslog 107 user mirrored 48.79MiB/∞ 34.00/∞
kvm 108 group default 1.05MiB/∞ 4.00/∞
kvm 108 group archive 536.00KiB/∞ 2.00/∞
kvm 108 group mirrored 536.00KiB/∞ 2.00/∞
tss 110 user default 16.00KiB/∞ 4.00/∞
tss 110 user archive 8.00KiB/∞ 2.00/∞
tss 110 user mirrored 8.00KiB/∞ 2.00/∞
lxd 110 group default 16.00KiB/∞ 12.00/∞
lxd 110 group archive 8.00KiB/∞ 6.00/∞
lxd 110 group mirrored 8.00KiB/∞ 6.00/∞
landscape 111 user default 32.00KiB/∞ 12.00/∞
landscape 111 user archive 16.00KiB/∞ 6.00/∞
landscape 111 user mirrored 16.00KiB/∞ 6.00/∞
_ssh 111 group default 1.05MiB/∞ 4.00/∞
_ssh 111 group archive 536.00KiB/∞ 2.00/∞
_ssh 111 group mirrored 536.00KiB/∞ 2.00/∞
fwupd-refresh 112 user default 16.00KiB/∞ 4.00/∞
fwupd-refresh 112 user archive 8.00KiB/∞ 2.00/∞
fwupd-refresh 112 user mirrored 8.00KiB/∞ 2.00/∞
crontab 112 group default 160.00KiB/∞ 8.00/∞
crontab 112 group archive 80.00KiB/∞ 4.00/∞
crontab 112 group mirrored 80.00KiB/∞ 4.00/∞
syslog 113 group default 16.00KiB/∞ 4.00/∞
syslog 113 group archive 8.00KiB/∞ 2.00/∞
syslog 113 group mirrored 8.00KiB/∞ 2.00/∞
tss 116 group default 16.00KiB/∞ 4.00/∞
tss 116 group archive 8.00KiB/∞ 2.00/∞
tss 116 group mirrored 8.00KiB/∞ 2.00/∞
landscape 117 group default 48.00KiB/∞ 16.00/∞
landscape 117 group archive 24.00KiB/∞ 8.00/∞
landscape 117 group mirrored 24.00KiB/∞ 8.00/∞
fwupd-refresh 118 group default 16.00KiB/∞ 4.00/∞
fwupd-refresh 118 group archive 8.00KiB/∞ 2.00/∞
fwupd-refresh 118 group mirrored 8.00KiB/∞ 2.00/∞
beegfs 998 group default 32.00KiB/∞ 8.00/∞
beegfs 998 group archive 16.00KiB/∞ 4.00/∞
beegfs 998 group mirrored 16.00KiB/∞ 4.00/∞
lxd 999 user default 16.00KiB/∞ 4.00/∞
lxd 999 user archive 8.00KiB/∞ 2.00/∞
lxd 999 user mirrored 8.00KiB/∞ 2.00/∞
docker 999 group default 53.61MiB/10.00GiB 288.00/∞
docker 999 group archive 26.80MiB/∞ 144.00/∞
docker 999 group mirrored 26.80MiB/∞ 144.00/∞
joe 1000 user default 148.22GiB/1.00TiB 2.36M/∞
joe 1000 user archive 74.11GiB/∞ 1.18M/∞
joe 1000 user mirrored 74.11GiB/∞ 1.18M/∞
joe 1000 group default 151.35GiB/∞ 3.08M/10.00M
joe 1000 group archive 75.68GiB/∞ 1.54M/∞
joe 1000 group mirrored 75.68GiB/∞ 1.54M/∞
tux 1001 user default 80.00KiB/1.00TiB 20.00/100.00M
tux 1001 user archive 40.00KiB/∞ 10.00/∞
tux 1001 user mirrored 40.00KiB/∞ 10.00/∞
tux 1001 group default 8.00GiB/2.00GiB 8.00/100.00M
tux 1001 group archive 4.00GiB/∞ 4.00/∞
tux 1001 group mirrored 4.00GiB/∞ 4.00/∞
tux2 1002 user default 80.00KiB/∞ 20.00/∞
tux2 1002 user archive 40.00KiB/∞ 10.00/∞
tux2 1002 user mirrored 40.00KiB/∞ 10.00/∞
beegfsquotas 2000 group default 8.00GiB/∞ 8.00/∞
beegfsquotas 2000 group archive 4.00GiB/∞ 4.00/∞
beegfsquotas 2000 group mirrored 4.00GiB/∞ 4.00/∞
nogroup 65534 group default 32.00KiB/∞ 8.00/∞
nogroup 65534 group archive 16.00KiB/∞ 4.00/∞
nogroup 65534 group mirrored 16.00KiB/∞ 4.00/∞
|
Found and fixed the bug. Used the wrong buffer when sending the authentication BeeMsg. I had authentication disabled, so my tests didn't trigger it. It has nothing to do with quota itself, it's just that quota is one of the few cases where management opens an outgoing tcp connection. |
|
After discussing with @rustybee42 I'm tagging in @jayapsrivastava since I think it would be good for her to also be familiar with this code. In particular this makes the code more reusable by external projects, and is a prerequisite for any potential future RDMA support from our Rust code, both of which are likely of interest to her. Edit: I verified the issue I observed earlier is now resolved. |
* Serializer now requires a preallocated buffer in form of a &mut [u8]. This moves the responsibility to allocate the required memory to the user. Before it used a dynamically growing buffer (which would reallocate internally if needed). This change is crucial for future RDMA support. * Since we now no longer use `BytesMut`, remove the bytes crate. * The buffers handled by the store have a fixed size of 4 MiB. This is apparently the maximum size of a BeeMsg (see `WORKER_BUF(IN|OUT)_SIZE` in `Worker.h`). C++ server code also uses these fixed size. * Generalize `msg_feature_flags` to a `Header` that can be modified from within the `Serializable` implementation and can be read out from within the `Deserializable` implementation. * Collect all BeeMsg (de)serialization functions in one module and provide functions for header, body and both combined. The split is required because depending on where the data comes from / goes to different actions need to be taken. This also provides an easy interface for potential external users to handle BeeMsges. * Remove the MsgBuf struct, instead just pass a `&mut [u8]` into the dispatcher. * Add documentation * Various small code cleanups in BeeSerde and other locations
rustybee42
force-pushed
the
rb/bee-serde-overhaul
branch
from
738859a to
811704c
Compare
…e logging Found by test_empty_udp_packets_server integration test. The test needs to be changed as well to adapt to the corrected log output (it was not really correct before)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pulled in from https://github.com/ThinkParQ/beegfs-rs/pull/185
&mut [u8]. This moves the responsibility to allocate the required memory to the user. Before it used a dynamically growing buffer (which would reallocate internally if needed). Not doing that is crucial for future RDMA support where we basically just get a pointer to a buffer, without having control over it.BytesMut, remove the bytes crate.BeeMsg(seeWORKER_BUF(IN|OUT)_SIZEin Worker.h). C++ server code also uses these fixed size.msg_feature_flagsto a Header that can be modified from within the Serializable implementation and can be read out from within theDeserializableimplementation.BeeMsg(de)serialization functions in one module and provide functions for header, body and both combined. The split is required because depending on where the data comes from / goes to different actions need to be taken. This also provides an easy interface for potential external users to handle BeeMsges.MsgBufstruct, instead just pass a&mut [u8]into the dispatcher.Closes #9