22 Debugging a Seastar program

22.1 Debugging ignored exceptions

If a future resolves with an exception, and the application neglects to handle that exception or to explicitly ignore it, the application may have missed an important problem. This is likely to be an application bug.

Therefore, Seastar prints a warning message to the log if a future is destroyed when it stores an exception that hasn’t been handled.

For example, consider this code:

#include <seastar/core/future.hh>
#include <seastar/core/sleep.hh>
#include <seastar/core/app-template.hh>

class myexception {};

seastar::future<> g() {
    return seastar::make_exception_future<>(myexception());
}

seastar::future<> f() {
    g();
    return seastar::sleep(std::chrono::seconds(1));
}

int main(int argc, char** argv) {
    seastar::app_template app;
    app.run(argc, argv, f);
}

Here, the main function f() calls g(), but doesn’t do anything with the future it returns. But this future resolves with an exception, and this exception is silently ignored. So Seastar prints this warning message about the ignored exception:

WARN  2020-03-31 11:08:09,208 [shard 0] seastar - Exceptional future ignored: myexception, backtrace:   /lib64/libasan.so.5+0x6ce7f
  0x1a64193
  0x1a6265f
  0xf326cc
  0xeaf1a0
  0xeaffe4
  0xead7be
  0xeb5917
  0xee2477
  0xec312e
  0xec8fcf
  0xeec765
  0xee1b29
  0xed9fab
  0xec27c8
  0xec867f
  0xf00acc
  0xef179d
  0xef1824
  0xef18b5
  0xee4827
  0xee470f
  0xf00f81
  0xebac29
  0xeb9095
  0xeb9174
  0xeb925a
  0xeb9964
  0xebef89
  0x10f74c3
  0x10fb439
  0x11005fd
  0xec4f08
  0xec2f43
  0xec3461
  0xeadabe
  /lib64/libc.so.6+0x271a2
  0xead52d

This message says that an exceptional future was ignored, and that the type of the exception was “myexception”. The type of the exception is usually not enough to pinpoint where the problem happened, so the warning message also includes the backtrace - the call chain - leading to where the exceptional future was destroyed. The backtrace is given as a list of addresses, where code in other shared libraries is written as a shared library plus offset (when ASLR is enabled, the shared libraries are mapped in a different address each time).

Seastar includes a utility, seastar-addr2line, for translating these addresses into readable backtraces including exact method names, source files and line numbers. This utility needs the unstripped executable. Typically, a stripped executable is used for production, but an unstripped copy is kept separately to be used in debugging - including seastar-addr2line.

To decode the backtrace, we run

seastar-addr2line -e a.out

And then paste the list of addresses in the warning message, and conclude with a control-D (it’s also possible, if you want, to put the list of addresses in the seastar-addr2line command line). The result looks like this:

void seastar::backtrace<seastar::current_backtrace()::{lambda(seastar::frame)#1}>(seastar::current_backtrace()::{lambda(seastar::frame)#1}&&) at include/seastar/util/backtrace.hh:56
seastar::current_backtrace() at src/util/backtrace.cc:84
seastar::report_failed_future(std::__exception_ptr::exception_ptr const&) at src/core/future.cc:116
seastar::future_state_base::~future_state_base() at include/seastar/core/future.hh:335
seastar::future_state<>::~future_state() at include/seastar/core/future.hh:414
 (inlined by) seastar::future<>::~future() at include/seastar/core/future.hh:990
f() at test.cc:12
std::_Function_handler<seastar::future<> (), seastar::future<> (*)()>::_M_invoke(std::_Any_data const&) at /usr/include/c++/9/bits/std_function.h:286
std::function<seastar::future<> ()>::operator()() const at /usr/include/c++/9/bits/std_function.h:690
seastar::app_template::run(int, char**, std::function<seastar::future<> ()>&&)::{lambda()#1}::operator()() const at src/core/app-template.cc:131
std::_Function_handler<seastar::future<int> (), seastar::app_template::run(int, char**, std::function<seastar::future<> ()>&&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /usr/include/c++/9/bits/std_function.h:286
std::function<seastar::future<int> ()>::operator()() const at /usr/include/c++/9/bits/std_function.h:690
seastar::future<int> seastar::futurize<seastar::future<int> >::invoke<std::function<seastar::future<int> ()>&>(std::function<seastar::future<int> ()>&) at include/seastar/core/future.hh:1670
auto seastar::futurize_invoke<std::function<seastar::future<int> ()>&>(std::function<seastar::future<int> ()>&) at include/seastar/core/future.hh:1754
seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&)::{lambda()#1}::operator()() at src/core/app-template.cc:120 (discriminator 4)
std::_Function_handler<void (), seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /usr/include/c++/9/bits/std_function.h:300
std::function<void ()>::operator()() const at /usr/include/c++/9/bits/std_function.h:690
seastar::apply_helper<std::function<void ()>&, std::tuple<>&&, std::integer_sequence<unsigned long> >::apply(std::function<void ()>&, std::tuple<>&&) at include/seastar/core/apply.hh:36
auto seastar::apply<std::function<void ()>&>(std::function<void ()>&, std::tuple<>&&) at include/seastar/core/apply.hh:44
seastar::future<> seastar::futurize<void>::apply<std::function<void ()>&>(std::function<void ()>&, std::tuple<>&&) at include/seastar/core/future.hh:1634
auto seastar::futurize_apply<std::function<void ()>&>(std::function<void ()>&, std::tuple<>&&) at include/seastar/core/future.hh:1766
seastar::future<>::then<std::function<void ()>, seastar::future<> >(std::function<void ()>&&)::{lambda()#1}::operator()() at include/seastar/core/future.hh:1191
seastar::noncopyable_function<seastar::future<> ()>::direct_vtable_for<seastar::future<>::then<std::function<void ()>, seastar::future<> >(std::function<void ()>&&)::{lambda()#1}>::call(seastar::noncopyable_function<seastar::future<> ()> const*) at include/seastar/util/noncopyable_function.hh:101
seastar::noncopyable_function<seastar::future<> ()>::operator()() const at include/seastar/util/noncopyable_function.hh:184
seastar::apply_helper<seastar::noncopyable_function<seastar::future<> ()>, std::tuple<>&&, std::integer_sequence<unsigned long> >::apply(seastar::noncopyable_function<seastar::future<> ()>&&, std::tuple<>&&) at include/seastar/core/apply.hh:36
auto seastar::apply<seastar::noncopyable_function<seastar::future<> ()>>(seastar::noncopyable_function<seastar::future<> ()>&&, std::tuple<>&&) at include/seastar/core/apply.hh:44
seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> ()>>(seastar::noncopyable_function<seastar::future<> ()>&&, std::tuple<>&&) at include/seastar/core/future.hh:1660
seastar::future<>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, seastar::future_state<>&&)#1}::operator()(seastar::internal::promise_base_with_type<>, seastar::future_state<>) at include/seastar/core/future.hh:1213
seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, seastar::future_state<>&&)#1}>::run_and_dispose() at include/seastar/core/future.hh:509
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at src/core/reactor.cc:2124
seastar::reactor::run_some_tasks() at src/core/reactor.cc:2539 (discriminator 2)
seastar::reactor::run() at src/core/reactor.cc:2694
seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at src/core/app-template.cc:199 (discriminator 1)
seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) at src/core/app-template.cc:115 (discriminator 2)
seastar::app_template::run(int, char**, std::function<seastar::future<> ()>&&) at src/core/app-template.cc:130 (discriminator 2)
main at test.cc:19 (discriminator 1)
__libc_start_main at /usr/src/debug/glibc-2.30-34-g994e529a37/csu/../csu/libc-start.c:308
_start at ??:?

Most of the lines at the bottom of this backtrace are not interesting, and just showing the internal details of how Seastar ended up running the main function f(). The only interesting part is the first few lines:

seastar::report_failed_future(std::__exception_ptr::exception_ptr const&) at src/core/future.cc:116
seastar::future_state_base::~future_state_base() at include/seastar/core/future.hh:335
seastar::future_state<>::~future_state() at include/seastar/core/future.hh:414
 (inlined by) seastar::future<>::~future() at include/seastar/core/future.hh:990
f() at test.cc:12

Here we see that the warning message was printed by the seastar::report_failed_future() function which was called when destroying a future (future<>::~future) that had not been handled. The future’s destructor was called in line 11 of our test code (26.cc), which is indeed the line where we called g() and ignored its result. This backtrace gives us an accurate understanding of where our code destroyed an exceptional future without handling it first, which is usually helpful in solving these kinds of bugs. Note that this technique does not tell us where the exception was first created, nor what code passed around the exceptional future before it was destroyed - we just learn where the future was destroyed. To learn where the exception was originally thrown, see the next section:

22.2 Finding where an exception was thrown

Sometimes an application logs an exception, and we want to know where in the code the exception was originally thrown. Unlike languages like Java, C++ does not have a builtin method of attaching a backtrace to every exception. So Seastar provides functions which allow adding to an exception the backtrace recorded when throwing it.

For example, in the following code we throw and catch an std::runtime_error normally:

#include <seastar/core/future.hh>
#include <seastar/util/log.hh>
#include <exception>
#include <iostream>

seastar::future<> g() {
    return seastar::make_exception_future<>(std::runtime_error("hello"));
}

seastar::future<> f() {
    return g().handle_exception([](std::exception_ptr e) {
        std::cerr << "Exception: " << e << "\n";
    });
}

The output is

Exception: std::runtime_error (hello)

From this output, we have no way of knowing that the exception was thrown in g(). We can solve this if we use make_exception_future_with_backtrace instead of make_exception_future:

#include <util/backtrace.hh>
seastar::future<> g() {
    return seastar::make_exception_future_with_backtrace<>(std::runtime_error("hello"));
}

Now the output looks like

Exception: seastar::internal::backtraced<std::runtime_error> (hello Backtrace:   0x678bd3
  0x677204
  0x67736b
  0x678cd5
  0x4f923c
  0x4f9c38
  0x4ff4d0
...
)

Which, as above, can be converted to a human-readable backtrace by using the seastar-addr2line script.

In addition to seastar::make_exception_future_with_backtrace(), Seastar also provides a function throw_with_backtrace(), to throw an exception instead of returning an exceptional future. For example:

    seastar::throw_with_backtrace<std::runtime_error>("hello");

In the current implementation, both make_exception_future_with_backtrace and throw_with_backtrace require that the original exception type (in the above example, std::runtime_error) is a subclass of the std::exception class. The original exception provides a what() string, and the wrapped exception adds the backtrace to this string, as demonstrated above. Moreover, the wrapped exception type is a subclass of the original exception type, which allows catch(...) code to continue filtering by the exception original type - despite the addition of the backtrace.

22.3 Debugging with gdb

handle SIGUSR1 pass noprint
handle SIGALRM pass noprint