Add ZSTD compression

[dbeinder]

Picking up on #16

The first two commits integrate zstd compression into NSIS

• Tested on MSVC 14 (Visual Studio 2022)
• In my testing, zstd at level 9-19 beats zlib & bzip2 in compression ratio and speed
• For the absolute best compression, lzma is still the best option, but everywhere else on the speed/size spectrum zstd seems to offer the best tradeoff with extremely wide tunability: https://facebook.github.io/zstd/
• Zstd stubs are 90kiB, this could be reduced to 75kiB with minimal compile options
  • Decompression speed is >200MB/s on my machine, but this could probably be improved
  • Zstd decompression is always single-threaded, unless exehead is extended to decompressing multiple files in parallel
  • Zstd requires between 2-10MB memory depending on zstd level (1-19)
• Compression in makensis is multithreaded
  • Compression speed at level 17 is 20MB/s on my machine
  • Without further changes, multithreading only helps for larger files or when using solid mode
  • Zstd compression uses 20MB per thread at level 9, and about 150MB per thread at level 19
• I have made some minimal changes to zstd and added a diff similar to 7zip
  • One change is to use fallback byteswap functions instead of calling into the CRT
  • All other changes are #ifndef EXEHEAD to strip unused features and reduce code size of the stubs
• The remaining uglyness is necessary to link zstd without a CRT into the stub
  • Zstd requires memcpy, memset, memmove, malloc, calloc, free, _allmul, and __allshl
  • For now, I have put simple implementations into exehead/crt_replacement.c
• Documentation updated, zstd License (BSD) added
  • Documented SetCompressionLevel script command to control compression level for zstd (and zlib)
• MakeNsisW & Zip2Exe updated

The newest 5 commits make everything Windows 95 compatible. Getting zstd to compile on VC6 would be a lot of work, and those changes would make updating zstd impractical - on the other hand, getting a EXE compiled by modern MSVC to run on Win95 is relatively easy:

• For the stubs, patch the PE header to allow running on Win9x and avoid calls to ConditionVariable kernel API
• For the Math plugin, instead of linking to a modern W95-incompatible CRT, embed libm and xprintf
• Moving to modern MSVC only increases the size of the existing stubs by 1-2kiB

[tbertels]

See also https://sourceforge.net/p/nsis/patches/317/

[renodr]

I just tried applying this to a copy of NSIS 3.11 I have and it seems to have several major conflicts, especially in the Contrib/zip2exe changes. One example is that the compression selector logic is more generic as of 0b7a3fe

[COD3HUNT3R]

Any update for this?

[janisozaur]

what's this file for?

[dbeinder]

To document source changes from the zstd release - so they can be rebased when updating the zstd version. The existing 7z implementation also has a diff: https://github.com/NSIS-Dev/nsis/blob/master/Source/7zip/sdk.diff

[janisozaur]

This is... impractical. I'd rather leave versioning stuff to the VCS. Can't there be a commit importing external library and then showing what's changed there by means of subsequent commits?

[dbeinder]

Ideally zstd should be a git submodule, so rebasing to newer versions of zstd would be straightforward.

However the actual development still happens on sourceforge, and they use svn.

I'm not involved with the project, you have to raise it with the maintainers on sourceforge, this is just a mirror.

[janisozaur]

It appears there are a lot of new files not relevant to zstd itself - mostly in mathlib? Some diff files that I wouldn't expect to see in such PR, asm inlines, commented out new lines for scons, it seems zstd is compiled in multiple times (for tests and for the main executable), reserved identifiers are abused... Some of these might make sense, but certainly requires good inspection.

[dbeinder]

Oh, I know...

Everything "as far as I remember", things may have changed since I implemented this patch over 5 years ago.

The problem is that nsis still targets Windows 95. Which means either getting MSVC6.SP5 (released 2001) to compile the ZSTD source code OR getting the output of a modern MSVC to run on Windows 95. I've taken the second approach.

Fundamentally, exehead is compiled without dependency on a C runtime (MS CRT, /nodefaultlib) (and modern CRT wouldn't run on Win95 anyway), so any 'reserved identifiers' must be reimplemented if they are used.

The biggest headache is the Math plugin which makes heavy use of the C runtime - so if you want to preserve that, you need to add a whole mathlib into the codebase.

My recommendation is to drop Win95 support, at least for the Math plugin. As you say, it feels wrong to add 50k LOC of near unreadable floating point math, just to maintain compatibility with a 30 year old OS.

I don't have time for it these days - if you do want to take this on, you need to have a conversation with the maintainers over on sourceforge: https://sourceforge.net/p/nsis/patches/317/

[janisozaur]

That clears things up a bit, thanks.

I'd love to see zstd support added, but why keep win95 support? I can see the thrill of getting something working on such obsolete system, but to burden the vast majority of users on fairly modern systems with over 30 years of baggage in a quite widely used software? Is there any justification provided or is it just the bragging rights? Is there a plan to sunset support for those systems?

I don't see much movement in this repo, is it still alive? PRs not getting for over 5 years (#16) is not a good sign either.

Just some things I'd like to have cleared up before trying to work on this.

[renodr]

For our use case (LegacyUpdate), dropping Windows 95 support would be bad if it also drops support for Windows NT 4.0. We provide a program that allows the retro computing community as well as industrial systems to be able to reconnect to Windows Update currently on Windows 2000 and later, but eventually including NT 4.0. @kirb can chime in some more here as well regarding our use case

[janisozaur]

Isn't the current nsis 3.11 enough for that? I don't suggest invalidating past versions. Even if zstd became available, would it actually affect you, would you realistically recreate all your redistributables with that?

I've also checked https://legacyupdate.net/stats and there is not one entry for windows 95, 98, ME nor nt 4.0. LegacyUpdate does not provide new software, the catalogue of available updates to win9x is closed.

No new software comes out supporting those systems and it's getting harder to target windows 7, just yesterday I learned that after switching a project to VS2026 we can no longer support anything pre-win10. We still maintain a fork targeting windows 5.1, because having this code upstream is not something we want to burden our users and other developers with.

[kirb]

@janisozaur My concern would be how the codebase will change should early OSes be dropped. We build our project with modern mingw-w64 13/gcc 15.2 on Linux, including our own build of NSIS (Debian’s exehead crashes on 486/Pentium since it defaults to -mcpu pentium2). We can maintain a fork if needed, but I would fear that legacy code paths (there are quite a lot) will be tidied up, making it increasingly difficult to apply patches cleanly. It’s still desirable to benefit from modern tools where possible, and improvements from NSIS upstream are still valuable. We’d also like to use zstd for its better performance.

I’m also aware of other semi-active projects using NSIS to target Win9x, such as dotnet9x, though we probably rely a lot more heavily on NSIS than them.

With that said, a compromise to release a primary build on a newer toolchain, while still maintaining a special build still using the Server 2003 PSDK without zstd support, could be fine. My suggestion would be a Windows 7 or 8.1-era DDK, which I believe should target Windows 2000 RTM and later. Visual Studio is problematic because it would want to dynamically or statically link a versioned build of msvcrt, when the user may not yet have it installed.

[janisozaur]

So do I read your comment right:

1. Include zstd and try targeting windows 2000+, which should remove the need to import half the universe
2. Have a zstd-less "special build" for older targets

?

Let's assume this is the case and we all agree to that and decide to move forward. But I don't see any member or contributor tag next to your comments - are the repo owners active on github or is it only SVN? Do we have to still convince them or is your approval enough (assuming you're the only active users of those old targets)?

[janisozaur]

Also - what is exehead? I see it in the repo, but it lacks any readme or other explanation

[kirb]

It’s not a decision I can make - just a potential suggestion if anyone wants to bring it up on SourceForge to help this get merged (this repo is a mirror of the SVN repo over there), since it seems like a safe/tidy compromise. I haven’t yet tested whether linking msvcrt.dll is sufficient to make libzstd work though.

exehead is the binary that gets output containing the NSIS runtime, which has the compiled script and installation files merged into it.