Crashing on Windows

[LostRuins]

I'm trying to build this on windows, and it does compile successfully, however, the CLI crashes when trying to load any model. Any idea why?

[danielzgtg]

Can you capture a stack trace?

[LostRuins]

Sure, these are my full repro steps. I am using w64devkit as my build environment

git clone https://github.com/mmwillet/TTS.cpp.git --recurse-submodules
cd TTS.cpp
cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Debug

We need to add a quick fix to support M_PI first, so I added this

#define _USE_MATH_DEFINES
#include <cmath>

to the top of util.cpp and kokoro_model.cpp

make

The build completes successfully. I can see tts-cli.exe inside the bin directory. The server build fails but we don't need that now.

Now, I download your Kokoro_no_espeak.gguf from your huggingface.

And then I run tts-cli.exe --model-path C:\dev\Kokoro_no_espeak.gguf --prompt hello
It prints out a single line
ggml_gallocr_reserve_n: reallocating CPU buffer from size 0.00 MiB to 18.00 MiB
and then crashes.

Running it with gdb, we can see a segfault occurred.

Reading symbols from tts-cli.exe...
(gdb) run
Starting program: C:\dev\TTS.cpp\bin\tts-cli.exe --model-path C:\dev\Kokoro_no_espeak.gguf --prompt hello
[New Thread 18316.0x4ac0]
[New Thread 18316.0x279c]
ggml_gallocr_reserve_n: reallocating CPU buffer from size 0.00 MiB to 18.00 MiB

Thread 1 received signal SIGSEGV, Segmentation fault.
0x00007ff6274c77cc in ggml_hash_insert (hash_set=0x18a06619090, key=0x18a12f2b1e0) at C:/dev/TTS.cpp/ggml/src/ggml-impl.h:234
234         size_t h = ggml_hash(key) % hash_set->size;

Running bt to get the backtrace

(gdb) bt
#0  0x00007ff6274c77cc in ggml_hash_insert (hash_set=0x18a06619090, key=0x18a12f2b1e0)
    at C:/dev/TTS.cpp/ggml/src/ggml-impl.h:234
#1  0x00007ff6274d5462 in ggml_visit_parents (cgraph=0x18a06619060, node=0x18a12f2b1e0)
    at C:/dev/TTS.cpp/ggml/src/ggml.c:5725
#2  0x00007ff6274d54d8 in ggml_visit_parents (cgraph=0x18a06619060, node=0x18a12f2b6f0)
    at C:/dev/TTS.cpp/ggml/src/ggml.c:5735
#3  0x00007ff6274d54d8 in ggml_visit_parents (cgraph=0x18a06619060, node=0x18a12f2b8a0)
    at C:/dev/TTS.cpp/ggml/src/ggml.c:5735
#4  0x00007ff6274d54d8 in ggml_visit_parents (cgraph=0x18a06619060, node=0x18a12f2cce0)
    at C:/dev/TTS.cpp/ggml/src/ggml.c:5735
#5  0x00007ff6274d54d8 in ggml_visit_parents (cgraph=0x18a06619060, node=0x18a12f2ce90)
    at C:/dev/TTS.cpp/ggml/src/ggml.c:5735
#6  0x00007ff6274d54d8 in ggml_visit_parents (cgraph=0x18a06619060, node=0x18a12f2d040)
    at C:/dev/TTS.cpp/ggml/src/ggml.c:5735
#7  0x00007ff6274d54d8 in ggml_visit_parents (cgraph=0x18a06619060, node=0x18a12f2d1f0)
    at C:/dev/TTS.cpp/ggml/src/ggml.c:5735
#8  0x00007ff6274d54d8 in ggml_visit_parents (cgraph=0x18a06619060, node=0x18a12f2df70)
    at C:/dev/TTS.cpp/ggml/src/ggml.c:5735
#9  0x00007ff6274d54d8 in ggml_visit_parents (cgraph=0x18a06619060, node=0x18a12f2e120)
    at C:/dev/TTS.cpp/ggml/src/ggml.c:5735
#10 0x00007ff6274d54d8 in ggml_visit_parents (cgraph=0x18a06619060, node=0x18a12f2e2d0)
    at C:/dev/TTS.cpp/ggml/src/ggml.c:5735
#11 0x00007ff6274d54d8 in ggml_visit_parents (cgraph=0x18a06619060, node=0x18a12f2e480)
    at C:/dev/TTS.cpp/ggml/src/ggml.c:5735
#12 0x00007ff6274d54d8 in ggml_visit_parents (cgraph=0x18a06619060, node=0x18a12f2f8c0)
    at C:/dev/TTS.cpp/ggml/src/ggml.c:5735
#13 0x00007ff6274d54d8 in ggml_visit_parents (cgraph=0x18a06619060, node=0x18a12f2fa70)
    at C:/dev/TTS.cpp/ggml/src/ggml.c:5735
#14 0x00007ff6274d54d8 in ggml_visit_parents (cgraph=0x18a06619060, node=0x18a12f2fc20)
--Type <RET> for more, q to quit, c to continue without paging--

it goes on and on like that in that same function.

Hope this helps.
Edit: for consistency since I ran it a few times with a different prompt, same results.

[danielzgtg]

In #106, I made it that that function recurs no more than 100 frames deep. Does it fix the problem? Does it occur for other models?

[LostRuins]

Hi @danielzgtg , thanks for the response. It gets a little bit further now, but still crashes.

gdb --args tts-cli.exe --model-path Kokoro_no_espeak.gguf --prompt hello
Reading symbols from tts-cli.exe...
(gdb) run
Starting program: C:\dev\TTS.cpp\bin\tts-cli.exe --model-path Kokoro_no_espeak.gguf --prompt hello
[New Thread 15244.0x74b8]
[New Thread 15244.0x4984]
ggml_gallocr_reserve_n: reallocating CPU buffer from size 0.00 MiB to 18.00 MiB
ggml_gallocr_reserve_n: reallocating CPU buffer from size 0.00 MiB to 9077.51 MiB
[Thread 15244.0x74b8 exited with code 0]
[Thread 15244.0x4984 exited with code 0]
ggml_gallocr_needs_realloc: graph has different number of nodes
ggml_gallocr_alloc_graph: reallocating buffers automatically
[New Thread 15244.0x6fec]
[New Thread 15244.0x291c]
[New Thread 15244.0x28a4]
[New Thread 15244.0x7270]
[New Thread 15244.0x7048]
[New Thread 15244.0x6408]
[New Thread 15244.0x666c]
[New Thread 15244.0x642c]
[New Thread 15244.0x6ab4]
[New Thread 15244.0xfdc]
[New Thread 15244.0x48a4]
[New Thread 15244.0x4e98]
[New Thread 15244.0x34c4]
[New Thread 15244.0x6704]
[New Thread 15244.0x7760]
[New Thread 15244.0xa34]
[New Thread 15244.0x7288]
[New Thread 15244.0x5a80]
[New Thread 15244.0x5184]
[New Thread 15244.0x3f34]
[New Thread 15244.0x5860]
[New Thread 15244.0x1c4]
[New Thread 15244.0x50c0]
[New Thread 15244.0x6928]
[New Thread 15244.0x66a0]
[New Thread 15244.0x70a4]
[New Thread 15244.0x5d20]
[New Thread 15244.0x28d0]
[New Thread 15244.0x52ec]
[New Thread 15244.0x46a8]
[New Thread 15244.0x30ec]

Thread 12 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 15244.0x6ab4]
0x00007ff7fc2ae7be in ggml_vec_soft_max_f32 (n=28, y=0x1bbcbe18910, x=0x1b7d484c4b0, max=1.27257073) at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:2106
2106            __m256 val = ggml_v_expf(_mm256_sub_ps(_mm256_loadu_ps(x + i),
(gdb) bt
#0  0x00007ff7fc2ae7be in ggml_vec_soft_max_f32 (n=28, y=0x1bbcbe18910, x=0x1b7d484c4b0, max=1.27257073)
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:2106
#1  0x00007ff7fc2ca690 in ggml_compute_forward_soft_max_f32 (params=0x47c91ffd60, dst=0x1b988d96b30)
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:9442
#2  0x00007ff7fc2ca808 in ggml_compute_forward_soft_max (params=0x47c91ffd60, dst=0x1b988d96b30)
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:9466
#3  0x00007ff7fc2d84c7 in ggml_compute_forward (params=0x47c91ffd60, tensor=0x1b988d96b30)
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:13116
#4  0x00007ff7fc2d9c26 in ggml_graph_compute_thread (data=0x1b7f12a2e50)
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:14089
#5  0x00007ff7fc2da56f in ggml_graph_compute._omp_fn.0 ()
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:14383
#6  0x00007ff7fc34cab5 in gomp_thread_start ()
#7  0x00007ff7fc369bc3 in pthread_create_wrapper ()
#8  0x00007fff4cd9af5a in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#9  0x00007fff4cd9b02c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#10 0x00007fff4d4e7374 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#11 0x00007fff4d7dcc91 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#12 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

As for other models, I tried your Parler_TTS_mini_Q5.gguf and also got a different crash.

Reading symbols from tts-cli.exe...
(gdb) run
Starting program: C:\dev\TTS.cpp\bin\tts-cli.exe --model-path C:\dev\Parler_TTS_mini_Q5.gguf --prompt hello
[New Thread 30044.0x6cf8]
[New Thread 30044.0x7334]
[New Thread 30044.0x1ad0]
ggml_gallocr_reserve_n: reallocating CPU buffer from size 0.00 MiB to 1.15 MiB
[New Thread 30044.0x7340]
[New Thread 30044.0x5ed8]
[New Thread 30044.0x59f0]
[New Thread 30044.0x7140]
[New Thread 30044.0x5dc0]
[New Thread 30044.0x6868]
[New Thread 30044.0x6e54]
[New Thread 30044.0x58a8]
[New Thread 30044.0x58f0]
[New Thread 30044.0x73ec]
[New Thread 30044.0x2b0c]
[New Thread 30044.0x75bc]
[New Thread 30044.0x6170]
[New Thread 30044.0x3b88]
[New Thread 30044.0x62a4]
[New Thread 30044.0x46d0]
[New Thread 30044.0x56ac]
[New Thread 30044.0x733c]
[New Thread 30044.0x5dfc]
[New Thread 30044.0x7498]
[New Thread 30044.0x5f18]
[New Thread 30044.0x2168]
[New Thread 30044.0x6ff0]
[New Thread 30044.0x3e1c]
[New Thread 30044.0x5c3c]
[New Thread 30044.0x6064]
[New Thread 30044.0x6700]
[New Thread 30044.0x62d4]
[New Thread 30044.0x4b68]
[New Thread 30044.0x288c]
[New Thread 30044.0x7018]
ggml_gallocr_reserve_n: reallocating CPU buffer from size 0.00 MiB to 5079.38 MiB
ggml_gallocr_reserve_n: reallocating CPU buffer from size 0.00 MiB to 1136.12 MiB

Thread 27 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 30044.0x6ff0]
0x00007ff692d0664f in bytes_from_nibbles_32 (rsi=0x21d3e4a5146 "S\230É\200Æ\036«O@o!_ì\037>\030R!ù\377ü\a\201ï\\\004\n~\016+\017-\036;;C\002\060r%") at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu-quants.c:95
95          return _mm256_and_si256(lowMask, bytes);
(gdb) bt
#0  0x00007ff692d0664f in bytes_from_nibbles_32 (
    rsi=0x21d3e4a5146 "S\230É\200Æ\036«O@o!_ì\037>\030R!ù\377ü\a\201ï\\\004\n~\016+\017-\036;;C\002\060r%")
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu-quants.c:95
#1  0x00007ff692d0aabc in ggml_vec_dot_q5_0_q8_0 (n=1024, s=0x4c2dfff850, bs=0, vx=0x21d3e4a5140, bx=0,
    vy=0x21d34a1a500, by=0, nrc=1) at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu-quants.c:2756
#2  0x00007ff692ce2f1e in ggml_compute_forward_mul_mat_one_chunk (params=0x4c2dfffd80, dst=0x21d765a1c40,
    type=GGML_TYPE_Q5_0, num_rows_per_vec_dot=1, ir0_start=736, ir0_end=768, ir1_start=0, ir1_end=2)
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:7600
#3  0x00007ff692ce3edf in ggml_compute_forward_mul_mat (params=0x4c2dfffd80, dst=0x21d765a1c40)
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:7820
#4  0x00007ff692cf8323 in ggml_compute_forward (params=0x4c2dfffd80, tensor=0x21d765a1c40)
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:13036
#5  0x00007ff692cf9c26 in ggml_graph_compute_thread (data=0x21d346b8a70)
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:14089
#6  0x00007ff692cfa56f in ggml_graph_compute._omp_fn.0 ()
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:14383
#7  0x00007ff692d6cab5 in gomp_thread_start ()
#8  0x00007ff692d89bc3 in pthread_create_wrapper ()
#9  0x00007fff4cd9af5a in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#10 0x00007fff4cd9b02c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007fff4d4e7374 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#12 0x00007fff4d7dcc91 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#13 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

In both cases it seems to be related to some CPU intrinsic. I disabled the GGML_OPENMP and retried, similar error.

Thread 6 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 9864.0x6de4]
0x00007ff628c5e7be in ggml_vec_soft_max_f32 (n=8, y=0x1b6c9faa280, x=0x1b2e65a0140, max=32.3373451) at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:2106
2106            __m256 val = ggml_v_expf(_mm256_sub_ps(_mm256_loadu_ps(x + i),
(gdb) bt
#0  0x00007ff628c5e7be in ggml_vec_soft_max_f32 (n=8, y=0x1b6c9faa280, x=0x1b2e65a0140, max=32.3373451)
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:2106
#1  0x00007ff628c7a73b in ggml_compute_forward_soft_max_f32 (params=0x13e77ff820, dst=0x1b486f57b30)
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:9442
#2  0x00007ff628c7a8b3 in ggml_compute_forward_soft_max (params=0x13e77ff820, dst=0x1b486f57b30)
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:9466
#3  0x00007ff628c88572 in ggml_compute_forward (params=0x13e77ff820, tensor=0x1b486f57b30)
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:13116
#4  0x00007ff628c89e7e in ggml_graph_compute_thread (data=0x1b2e752b400)
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:14089
#5  0x00007ff628c8a226 in ggml_graph_compute_secondary_thread (data=0x1b2e752b400)
    at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:14209
#6  0x00007fff4d4e7374 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#7  0x00007fff4d7dcc91 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#8  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

[LostRuins]

I disabled ALL CPU intrinsics and got it to work by setting GGML_NATIVE to off! But it is super slow, as expected with 0 acceleration.

[danielzgtg]

Can you tell me your CPU model from msinfo32? Does it work if you leave GGML_NATIVE disabled but enable GGML_AVX2?

Now I'm suspecting hardware-caused memory corruption. Can you run https://www.maxon.net/en/downloads/cinebench-2024-downloads and see if that crashes?

[LostRuins]

My Processor is a 13th Gen Intel(R) Core(TM) i9-13980HX, 2200 Mhz, 24 Core(s), 32 Logical Processor(s)
This is in a stock laptop MSI Titan GT77HX

My PC handles AVX2 fine, I run llama.cpp, stable-diffusion.cpp and whisper.cpp using the same compile environment, all fully accelerated with all available CPU intrinsics. So I would speculate that any issues come from the modifications to ggml library and/or invalid/unaligned arguments passed into these functions.

I tried your suggest and built with, first setting GGML_AVX2=ON alone, the compile fails

In file included from d:\mainapplications\mingw\w64devkit\lib\gcc\x86_64-w64-mingw32\12.2.0\include\immintrin.h:105,
                 from d:\mainapplications\mingw\w64devkit\lib\gcc\x86_64-w64-mingw32\12.2.0\include\x86intrin.h:32,
                 from d:\mainapplications\mingw\w64devkit\x86_64-w64-mingw32\include\intrin.h:69,
                 from c:\dev\tts.cpp\ggml\src\ggml-cpu\ggml-cpu-impl.h:326,
                 from C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu-aarch64.c:11:
d:\mainapplications\mingw\w64devkit\lib\gcc\x86_64-w64-mingw32\12.2.0\include\fmaintrin.h: In function 'ggml_gemv_q4_0_8x8_q8_0':
d:\mainapplications\mingw\w64devkit\lib\gcc\x86_64-w64-mingw32\12.2.0\include\fmaintrin.h:63:1: error: inlining failed in call to 'always_inline' '_mm256_fmadd_ps': target specific option mismatch
   63 | _mm256_fmadd_ps (__m256 __A, __m256 __B, __m256 __C)
      | ^~~~~~~~~~~~~~~
C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu-aarch64.c:895:27: note: called from here
  895 |                 acc_row = _mm256_fmadd_ps(_mm256_cvtepi32_ps(iacc), _mm256_mul_ps(col_scale_f32, row_scale_f32), acc_row);
      |                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
d:\mainapplications\mingw\w64devkit\lib\gcc\x86_64-w64-mingw32\12.2.0\include\fmaintrin.h:63:1: error: inlining failed in call to 'always_inline' '_mm256_fmadd_ps': target specific option mismatch
   63 | _mm256_fmadd_ps (__m256 __A, __m256 __B, __m256 __C)
      | ^~~~~~~~~~~~~~~
C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu-aarch64.c:895:27: note: called from here
  895 |                 acc_row = _mm256_fmadd_ps(_mm256_cvtepi32_ps(iacc), _mm256_mul_ps(col_scale_f32, row_scale_f32), acc_row);
      |                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [ggml/src/ggml-cpu/CMakeFiles/ggml-cpu.dir/build.make:107: ggml/src/ggml-cpu/CMakeFiles/ggml-cpu.dir/ggml-cpu-aarch64.c.obj] Error 1
make[1]: *** [CMakeFiles/Makefile2:308: ggml/src/ggml-cpu/CMakeFiles/ggml-cpu.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

I then added -DCMAKE_C_FLAGS="-mavx2 -mfma" -DCMAKE_CXX_FLAGS="-mavx2 -mfma" and the compile does succeed.

However, I get the same crash again, always the same function at the same line in ggml_vec_soft_max_f32 (n=8, y=0x27e47e98820, x=0x27a3f9f06e0, max=4.87640572) at C:/dev/TTS.cpp/ggml/src/ggml-cpu/ggml-cpu.c:2106

[danielzgtg]

Using the CMake debug button in VS Code on Windows, I found another bug. Does merging both #110 and #106 fix the problem?

I tested with

{
    "files.associations": {
        "array": "cpp"
    },
    "cmake.debugConfig": {
        "args": [
            "-nt",
            "4",
            "-p",
            "Hello",
            "-mp",
            "C:\\Users\\Home\\Downloads\\Kokoro_no_espeak.gguf"
        ]
    }
}

Does this short prompt work?

[LostRuins]

Yes! It seems to be working now! After applying both PRs, I was able to produce a native executable build which worked!

I did notice a few mispronounced words, but I assume that's due to the model rather than the implementation.

It's much faster now, too.
Thanks for the fix!