[BUG] ** crash sw::redis::GuardedAsyncConnection::GuardedAsyncConnection(const sw::redis::AsyncConnectionPoolSPtr&): Assertion `!_connection->broken()' failed

[amosyxy]

Describe the bug
app some times crash as below stack.
redis-server work in cluster mode
3 master and 3 slave

To Reproduce
do robuness test for each redis server node, stop and start one by one
do some times, app will crash outputing below assert error
asyncredis_benchmark_each_thread_asynccluster: /root/redis/redis-plus-plus/src/sw/redis++/async_connection.cpp:587: sw::redis::GuardedAsyncConnection::GuardedAsyncConnection(const sw::redis::AsyncConnectionPoolSPtr&): Assertion `!_connection->broken()' failed.

sw::redis::GuardedAsyncConnection::GuardedAsyncConnection(const sw::redis::AsyncConnectionPoolSPtr&): Assertion `!_connection->broken()' failed

Expected behavior
no crash and app client can recover after redis server recovered
and app client and still handle traffic

Environment:

• OS:
  gcc version 13.3.0 (Ubuntu 13.3.0-6ubuntu224.04)
  root@iZuf6gqbr8mboxi4muxirlZ:/redis/redis-plus-plus/build#
  root@iZuf6gqbr8mboxi4muxirlZ:~/redis/hiredis# git branch

• (HEAD detached at v1.2.0)
  root@iZuf6gqbr8mboxi4muxirlZ:~/redis/redis-plus-plus# git branch
• (HEAD detached at 1.3.13)

root@iZuf6gqbr8mboxi4muxirlZ:~/redis/redis_client# gdb -c /var/lib/apport/coredump/core._root_redis_redis_client_asyncredis_benchmark_each_thread_asynccluster.0.b5646083-5f84-4223-b807-5a0246c64d88.140988.9729700 ./asyncredis_benchmark_each_thread_asynccluster
GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
https://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./asyncredis_benchmark_each_thread_asynccluster...

This GDB supports auto-downloading debuginfo from the following URLs:
https://debuginfod.ubuntu.com
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
Downloading separate debug info for /root/redis/redis_client/asyncredis_benchmark_each_thread_asynccluster
(No debugging symbols found in ./asyncredis_benchmark_each_thread_asynccluster)

warning: Can't open file anon_inode:[io_uring] which was expanded to anon_inode:[io_uring] during file-backed mapping note processing
[New LWP 180979]
[New LWP 140988]
[New LWP 140992]
[New LWP 140993]
[New LWP 140991]
Downloading separate debug info for /lib/x86_64-linux-gnu/libuv.so.1
Downloading separate debug info for system-supplied DSO at 0x7ffdbfefb000
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./asyncredis_benchmark_each_thread_asynccluster -h 127.0.0.1 -p 7001 -n 100000'.
Program terminated with signal SIGABRT, Aborted.
Download failed: Invalid argument. Continuing without source file ./nptl/./nptl/pthread_kill.c.
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=) at ./nptl/pthread_kill.c:44

warning: 44 ./nptl/pthread_kill.c: No such file or directory
[Current thread is 1 (Thread 0x74d22ea006c0 (LWP 180979))]
(gdb) bt
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x000074d235e4527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x000074d235e288ff in __GI_abort () at ./stdlib/abort.c:79
#5 0x000074d235e2881b in __assert_fail_base (fmt=0x74d235fd01e8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=assertion@entry=0x74d2367e332d "!_connection->broken()",
file=file@entry=0x74d2367e2d30 "/root/redis/redis-plus-plus/src/sw/redis++/async_connection.cpp", line=line@entry=587,
function=function@entry=0x74d2367e32c8 "sw::redis::GuardedAsyncConnection::GuardedAsyncConnection(const sw::redis::AsyncConnectionPoolSPtr&)") at ./assert/assert.c:96
#6 0x000074d235e3b517 in __assert_fail (assertion=0x74d2367e332d "!_connection->broken()",
file=0x74d2367e2d30 "/root/redis/redis-plus-plus/src/sw/redis++/async_connection.cpp", line=587,
function=0x74d2367e32c8 "sw::redis::GuardedAsyncConnection::GuardedAsyncConnection(const sw::redis::AsyncConnectionPoolSPtr&)") at ./assert/assert.c:105
#7 0x000074d2367a1c26 in sw::redis::GuardedAsyncConnection::GuardedAsyncConnection (this=0x74d22e9fbf30,
pool=std::shared_ptrsw::redis::AsyncConnectionPool (use count 3, weak count 1) = {...})
at /root/redis/redis-plus-plus/src/sw/redis++/async_connection.cpp:587
#8 0x0000615f448e5659 in void sw::redis::AsyncRedisCluster::_callback_command_with_parser<long long, sw::redis::DefaultResultParser, RedisBenchmark::run_del_benchmark_thread(int, int, int)::{lambda(std::future&&)#1}, sw::redis::FormattedCommand ()(std::basic_string_view<char, std::char_traits > const&), std::basic_string_view<char, std::char_traits > const&>(RedisBenchmark::run_del_benchmark_thread(int, int, int)::{lambda(std::future&&)#1}&&, sw::redis::FormattedCommand ()(std::basic_string_view<char, std::char_traits > const&), std::basic_string_view<char, std::char_traits > const&, std::basic_string_view<char, std::char_traits > const&) ()
#9 0x0000615f448e2e72 in void sw::redis::AsyncRedisCluster::_callback_generic_command<long long, RedisBenchmark::run_del_benchmark_thread(int, int, int)::{lambda(std::future&&)#1}, sw::redis::FormattedCommand ()(std::basic_string_view<char, std::char_traits > const&), std::basic_string_view<char, std::char_traits > const&>(RedisBenchmark::run_del_benchmark_thread(int, int, int)::{lambda(std::future&&)#1}&&, sw::redis::FormattedCommand ()(std::basic_string_view<char, std::char_traits > const&), std::basic_string_view<char, std::char_traits > const&, std::basic_string_view<char, std::char_traits > const&) ()
#10 0x0000615f448df988 in void sw::redis::AsyncRedisCluster::_callback_generic_command<long long, RedisBenchmark::run_del_benchmark_thread(int, int, int)::{lambda(std::future&&)#1}, sw::redis::FormattedCommand ()(std::basic_string_view<char, std::char_traits > const&)>(RedisBenchmark::run_del_benchmark_thread(int, int, int)::{lambda(std::future&&)#1}&&, sw::redis::FormattedCommand ()(std::basic_string_view<char, std::char_traits > const&), std::integral_constant<bool, true>, std::basic_string_view<char, std::char_traits > const&) ()
#11 0x0000615f448db369 in void sw::redis::AsyncRedisCluster::_callback_fmt_command<long long, RedisBenchmark::run_del_benchmark_thread(int, int, int)::{lambda(std::future&&)#1}, sw::redis::FormattedCommand ()(std::basic_string_view<char, std::char_traits > const&), std::basic_string_view<char, std::char_traits > const&>(RedisBenchmark::run_del_benchmark_thread(int, int, int)::{lambda(std::future&&)#1}&&, sw::redis::FormattedCommand ()(std::basic_string_view<char, std::char_traits > const&), std::basic_string_view<char, std::char_traits > const&) ()
#12 0x0000615f448d705f in std::enable_if<std::is_invocable<std::decay<RedisBenchmark::run_del_benchmark_thread(int, int, int)::{lambda(std::future&&)#1}>::type, std::future&&>::value, void>::type sw::redis::AsyncRedisCluster::del<RedisBenchmark::run_del_benchmark_thread(int, int, int)::{lambda(std::future&&)#1}>(std::basic_string_view<char, std::char_traits > const&, RedisBenchmark::run_del_benchmark_thread(int, int, int)::{lambda(std::future&&)#1}&&) ()
#13 0x0000615f448d2999 in RedisBenchmark::run_del_benchmark_thread(int, int, int) ()
#14 0x0000615f448efb1c in void std::__invoke_impl<void, void (RedisBenchmark::)(int, int, int), RedisBenchmark, int, int, int>(std::__invoke_memfun_deref, void (RedisBenchmark::&&)(int, int, int), RedisBenchmark&&, int&&, int&&, int&&) ()
#15 0x0000615f448ef162 in std::__invoke_result<void (RedisBenchmark::)(int, int, int), RedisBenchmark, int, int, int>::type std::__invoke<void (RedisBenchmark::)(int, int, int), RedisBenchmark, int, int, int>(void (RedisBenchmark::&&)(int, int, int), RedisBenchmark&&, int&&, int&&, int&&) ()
#16 0x0000615f448eeb57 in void std::thread::_Invoker<std::tuple<void (RedisBenchmark::)(int, int, int), RedisBenchmark, int, int, int> >::_M_invoke<0ul, 1ul, 2ul, 3ul, 4ul>(std::_Index_tuple<0ul, 1ul, 2ul, 3ul, 4ul>) ()
#17 0x0000615f448ed172 in std::thread::_Invoker<std::tuple<void (RedisBenchmark::)(int, int, int), RedisBenchmark, int, int, int> >::operator()() ()
#18 0x0000615f448eb130 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (RedisBenchmark::)(int, int, int), RedisBenchmark, int, int, int> > >::_M_run() ()
#19 0x000074d2362ecdb4 in std::execute_native_thread_routine (__p=0x615f4912bf50)
at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:104
#20 0x000074d235e9caa4 in start_thread (arg=) at ./nptl/pthread_create.c:447
#21 0x000074d235f29c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
(gdb)

[amosyxy]

redis-cpp-cpp client should not crash
and can recover to noram status after redis-server cluster come back normal

[amosyxy]

can some one help fix this crash issue

[amosyxy]

can we fix it as below:

root@iZuf6gqbr8mboxi4muxirlZ:~/redis/redis-plus-plus# git diff ./
diff --git a/src/sw/redis++/async_connection.cpp b/src/sw/redis++/async_connection.cpp
index f19b2f3..8be59f4 100644
--- a/src/sw/redis++/async_connection.cpp
+++ b/src/sw/redis++/async_connection.cpp
@@ -584,7 +584,11 @@ AsyncConnection::AsyncContextUPtr AsyncConnection::_connect(const ConnectionOpti

GuardedAsyncConnection::GuardedAsyncConnection(const AsyncConnectionPoolSPtr &pool) :
_pool(pool), _connection(_pool->fetch()) {

• assert(!_connection->broken());

• //assert(!_connection->broken());
• if(_connection and _connection->broken())
• {

•   throw Error("connection broken in GuardedAsyncConnection");
  

• }
  }

GuardedAsyncConnection::GuardedAsyncConnection() {
root@iZuf6gqbr8mboxi4muxirlZ:/redis/redis-plus-plus#

[sewenew]

Sorry, but I cannot reproduce the problem.

Can you give a minimum compilable code snippet that can reproduce the problem?

Regards

[sewenew]

@amosyxy I reproduced your problem, and fixed it by removing the assertion. The connection fetched from the pool might be broken, since the loop thread might change the state of the connection. However, we can safely send command with the broken connection, since the event callback will handle it properly.

The fixed code is on dev branch. Could please test it? I'll also do some more tests and merge it in master branch.

Thanks for reporting the bug!

Regards

[sewenew]

The code has been merge into master branch. Thanks again!

Regards