Segfault when running normally

[rexendevar]

I'm getting an unexplained segmentation fault while running even basic example snippets and tests. I feel like I must have misbuilt it but I do not know enough about building to fix it myself.

$make -j base.en
I ccache not found. Consider installing it for faster compilation.
I whisper.cpp build info: 
I UNAME_S:   Linux
I UNAME_P:   x86_64
I UNAME_M:   x86_64
I CFLAGS:    -Iggml/include -Iggml/src -Iinclude -Isrc -Iexamples -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_OPENMP  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -fopenmp -Wdouble-promotion 
I CXXFLAGS:  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread -fopenmp  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -Iggml/include -Iggml/src -Iinclude -Isrc -Iexamples -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_OPENMP 
I NVCCFLAGS: -std=c++11 -O3 
I LDFLAGS:    
I CC:        cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
I CXX:       c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

bash ./models/download-ggml-model.sh base.en
Downloading ggml model base.en from 'https://huggingface.co/ggerganov/whisper.cpp' ...
ggml-base.en.bin    100%[===================>] 141.11M  6.52MB/s    in 21s     
Done! Model 'base.en' saved in '/home/flynn/ScriptsandCompiling/whisper.cpp/models/ggml-base.en.bin'
You can now use it like this:

  $ ./main -m /home/flynn/ScriptsandCompiling/whisper.cpp/models/ggml-base.en.bin -f samples/jfk.wav


===============================================
Running base.en on all samples in ./samples ...
===============================================

----------------------------------------------
[+] Running base.en on samples/jfk.wav ... (run 'ffplay samples/jfk.wav' to listen)
----------------------------------------------

whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-base.en.bin'
whisper_init_with_params_no_state: use gpu    = 1
whisper_init_with_params_no_state: flash attn = 0
whisper_init_with_params_no_state: gpu_device = 0
whisper_init_with_params_no_state: dtw        = 0
whisper_init_with_params_no_state: backends   = 1
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51864
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 512
whisper_model_load: n_audio_head  = 8
whisper_model_load: n_audio_layer = 6
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 512
whisper_model_load: n_text_head   = 8
whisper_model_load: n_text_layer  = 6
whisper_model_load: n_mels        = 80
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 2 (base)
whisper_model_load: adding 1607 extra tokens
whisper_model_load: n_langs       = 99
whisper_model_load:      CPU total size =   147.37 MB
whisper_model_load: model size    =  147.37 MB
whisper_init_state: kv self size  =    6.29 MB
whisper_init_state: kv cross size =   18.87 MB
whisper_init_state: kv pad  size  =    3.15 MB
Segmentation fault

I'm fairly sure this isn't enough information to solve the problem so please let me know what else I need to provide.

[rexendevar]

Just tried again with no-gpu as well as decreasing the number of threads to 1. I also restarted my computer and deleted/rebuilt/reinstalled both main and the model in question. I would love to look at debug mode but the man page doesn't mention any syntax I can use so I don't know how.

[ggerganov]

Yes, I can reproduce on my linux box. The fault occurs when I build without the CUDA backend. It works when the CUDA backend is enabled.

@slaren Could you help me debug this? I suspect it is related to not using the backend registry properly yet, but I'm looking for a quick fix before refactoring for the new changes.

Here is a repro:

# verify this seg faults
make -j base.en

WHISPER_DEBUG=1 make -j && gdb --args ./main -m models/ggml-base.en.bin -f samples/jfk.wav

Stacktrace info

whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-base.en.bin'
whisper_init_with_params_no_state: use gpu    = 1
whisper_init_with_params_no_state: flash attn = 0
whisper_init_with_params_no_state: gpu_device = 0
whisper_init_with_params_no_state: dtw        = 0
register_backend: registered backend CPU (1 devices)
register_device: registered device CPU (AMD Ryzen 9 5950X 16-Core Processor)
whisper_init_with_params_no_state: backends   = 1
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51864
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 512
whisper_model_load: n_audio_head  = 8
whisper_model_load: n_audio_layer = 6
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 512
whisper_model_load: n_text_head   = 8
whisper_model_load: n_text_layer  = 6
whisper_model_load: n_mels        = 80
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 2 (base)
whisper_model_load: adding 1607 extra tokens
whisper_model_load: n_langs       = 99
whisper_model_load:      CPU total size =   147.37 MB
whisper_model_load: model size    =  147.37 MB
whisper_init_state: kv self size  =    6.29 MB
whisper_init_state: kv cross size =   18.87 MB
whisper_init_state: kv pad  size  =    3.15 MB
main: ggml/src/ggml-backend.cpp:1297: void ggml_backend_sched_split_graph(ggml_backend_sched_t, ggml_cgraph*): Assertion `node_backend_id != -1' failed.

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140737352357824) at ./nptl/pthread_kill.c:44
44	./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737352357824) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737352357824) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737352357824, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7842476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff78287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff782871b in __assert_fail_base (fmt=0x7ffff79dd130 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x5555556e5048 "node_backend_id != -1", file=0x5555556e49fa "ggml/src/ggml-backend.cpp", line=1297, function=<optimized out>) at ./assert/assert.c:92
#6  0x00007ffff7839e96 in __GI___assert_fail (assertion=0x5555556e5048 "node_backend_id != -1", file=0x5555556e49fa "ggml/src/ggml-backend.cpp", line=1297, function=0x5555556e5000 "void ggml_backend_sched_split_graph(ggml_backend_sched_t, ggml_cgraph*)") at ./assert/assert.c:101
#7  0x00005555555b4135 in ggml_backend_sched_split_graph (sched=0x55555607c5a0, graph=0x7fffee87f030) at ggml/src/ggml-backend.cpp:1297
#8  0x00005555555b6909 in ggml_backend_sched_alloc_graph (sched=0x55555607c5a0, graph=0x7fffee87f030) at ggml/src/ggml-backend.cpp:1745
#9  0x0000555555604d67 in whisper_sched_graph_init(whisper_sched &, std::vector<ggml_backend*, std::allocator<ggml_backend*> >, std::function<ggml_cgraph*()> &&) (allocr=..., backends=std::vector of length 1, capacity 1 = {...}, get_graph=...) at src/whisper.cpp:554
#10 0x00005555556113d8 in whisper_init_state (ctx=0x555555817420) at src/whisper.cpp:3428
#11 0x00005555556122f3 in whisper_init_from_file_with_params (path_model=0x555555814b80 "models/ggml-base.en.bin", params=...) at src/whisper.cpp:3696
#12 0x00005555556d9b7d in main (argc=5, argv=0x7fffffffdf98) at examples/main/main.cpp:1008
(gdb) frame 7
#7  0x00005555555b4135 in ggml_backend_sched_split_graph (sched=0x55555607c5a0, graph=0x7fffee87f030) at ggml/src/ggml-backend.cpp:1297
1297	           assert(node_backend_id != -1); // all nodes should be assigned by now
(gdb) 

It crashes when initializing the scheduler for the first "conv" graph:

https://github.com/ggerganov/whisper.cpp/blob/f02b40bcb4849139fd7de2558aee64c737ab678e/src/whisper.cpp#L1932-L1992

[slaren]

This should fix it, but it is also already fixed in ggml master.

diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cpp
index 0b8ebac..fd60eef 100644
--- a/ggml/src/ggml-backend.cpp
+++ b/ggml/src/ggml-backend.cpp
@@ -2566,7 +2566,7 @@ static bool ggml_backend_cpu_device_supports_op(ggml_backend_dev_t dev, const st
                 op->type != GGML_TYPE_IQ1_S   &&
                 op->type != GGML_TYPE_IQ1_M; // missing type_traits.from_float
         case GGML_OP_MUL_MAT:
-            return op->src[1]->type == GGML_TYPE_F32;// FIXME || op->src[1]->type == ggml_get_type_traits(op->src[0]->type)->vec_dot_type;
+            return op->src[1]->type == GGML_TYPE_F32 || op->src[1]->type == ggml_get_type_traits_cpu(op->src[0]->type)->vec_dot_type;
         case GGML_OP_ROPE_BACK:
             return op->src[2] == NULL && (op->op_params[2] & 4) == 0;
         case GGML_OP_IM2COL_BACK:

[rexendevar]

What's the easiest way for a noob like me to implement this and build properly? Is it already merged so I can just download again and retry?

[ggerganov]

Yes, it's merged. Just pull the latest source code and try again. It should work.

[rexendevar]

Problem solved 👍