Compact CNFS buffers

[Julien-Elie]

A suggestion from Nigel Reed in news.software.nntp to remove cancelled or expired articles from CNFS buffers, as well as articles in newsgroups no longer in the active file. Only still readable articles remain in what would be called a compact CNFS buffer. It would save disk space.

Maybe a new CNFSv5 kind of buffers with a storage.conf option to enable the feature.

[jrehmer]

If a new CNFS version is introduced, please consider adding support for compression. If one uses CNFS on a ZFS filesystem with compression (enabled by default in all OSes I'm aware of that use ZFS) you can get over 3x compression:

$ cnfsstat
Class BINARY for groups matching "*.bina*"
 Buffer BINARY1, size:   977 GBytes, position:  8.97 GBytes  0.01 cycles
  Newest: 2025-10-22  6:22:02,    5 days, 18:40:43 ago

Class TEST for groups matching "*.jobs,*.jobs.*,*.test,*.testing"
 Buffer TEST1, size:   977 GBytes, position:  83.8 GBytes  0.09 cycles
  Newest: 2025-10-22 16:42:06,    5 days,  8:20:39 ago

Class OTHER for groups matching "junk,control.*"
 Buffer OTHER1, size:   977 GBytes, position:  3.52 GBytes  0.00 cycles
  Newest: 2025-10-22  6:22:02,    5 days, 18:40:43 ago

Class TEXT for groups matching "*"
 Buffer TEXT1, size:   977 GBytes, position:  30.5 MBytes  1.00 cycles
  Newest: 2025-10-05  5:39:17,   22 days, 19:23:28 ago
 Buffer TEXT2, size:   977 GBytes, position:  30.5 MBytes  1.00 cycles
  Newest: 2025-10-07  1:05:11,   20 days, 23:57:34 ago
 Buffer TEXT3, size:   977 GBytes, position:  30.5 MBytes  1.00 cycles
  Newest: 2025-10-09 14:32:24,   18 days, 10:30:21 ago
 Buffer TEXT4, size:   977 GBytes, position:  30.5 MBytes  1.00 cycles
  Newest: 2025-10-18 17:15:36,    9 days,  7:47:09 ago
 Buffer TEXT5, size:   977 GBytes, position:   916 GBytes  0.94 cycles
  Newest: 2025-10-22 16:42:06,    5 days,  8:20:39 ago

There is approximately 5TB of news articles on this server, but on disk consume 1.35TB of space on ZFS with lz4 compression (the total size here includes several 1TB empty buffers not represented in the cnfsstat output above):

$ du -h cycbuffs/
1.35T   cycbuffs

$ du -Ah cycbuffs/
10.5T   cycbuffs

[Julien-Elie]

Good idea!
Compression (with for instance zlib) could even be extended to all our storage methods! Quite a bit of work though.