Asynchronous Programming with Seastar

Nadav Har’El - nyh@ScyllaDB.com

Avi Kivity - avi@ScyllaDB.com

Back to table of contents. Previous: 22 Promise objects.

23 Memory allocation in Seastar

23.1 Per-thread memory allocation

Seastar requires that applications be sharded, i.e., that code running on different threads operate on different objects in memory. We already saw in Seastar memory how Seastar takes over a given amount of memory (often, most of the machine’s memory) and divides it equally between the different threads. Modern multi-socket machines have non-uniform memory access (NUMA), meaning that some parts of memory are closer to some of the cores, and Seastar takes this knowledge into account when dividing the memory between threads. Currently, the division of memory between threads is static, and equal - the threads are expected to experience roughly equal amount of load and require roughly equal amounts of memory.

To achieve this per-thread allocation, Seastar redefines the C library functions malloc(), free(), and their numerous relatives — calloc(), realloc(), posix_memalign(), memalign(), malloc_usable_size(), and malloc_trim(). It also redefines the C++ memory allocation functions, operator new, operator delete, and all their variants (including array versions, the C++14 delete taking a size, and the C++17 variants taking required alignment).

It is important to remember that Seastar’s different threads can see memory allocated by other threads, but they are nontheless strongly discouraged from actually doing this. Sharing data objects between threads on modern multi-core machines results in stiff performance penalties from locks, memory barriers, and cache-line bouncing. Rather, Seastar encourages applications to avoid sharing objects between threads when possible (by sharding — each thread owns a subset of the objects), and when threads do need to interact they do so with explicit message passing, with submit_to(), as we shall see later.

23.2 Foreign pointers

An object allocated on one thread will be owned by this thread, and eventually should be freed by the same thread. Freeing memory on the wrong thread is strongly discouraged, but is currently supported (albeit slowly) to support library code beyond Seastar’s control. For example, std::exception_ptr allocates memory; So if we invoke an asynchronous operation on a remote thread and this operation returns an exception, when we free the returned std::exception_ptr this will happen on the “wrong” core. So Seastar allows it, but inefficiently.

In most cases objects should spend their entire life on a single thread and be used only by this thread. But in some cases we want to reassign ownership of an object which started its life on one thread, to a different thread. This can be done using a seastar::foreign_ptr<>. A pointer, or smart pointer, to an object is wrapped in a seastar::foreign_ptr<P>. This wrapper can then be moved into code running in a different thread (e.g., using submit_to()).

The most common use-case is a seastar::foreign_ptr<std::unique_ptr<T>>. The thread receiving this foreign_ptr will get exclusive use of the object, and when it destroys this wrapper, it will go back to the original thread to destroy the object. Note that the object is not only freed on the original shard - it is also destroyed (i.e., its destructor is run) there. This is often important when the object’s destructor needs to access other state which belongs to the original shard - e.g., unlink itself from a container.

Although foreign_ptr ensures that the object’s destructor automatically runs on the object’s home thread, it does not absolve the user from worrying where to run the object’s other methods. Some simple methods, e.g., methods which just read from the object’s fields, can be run on the receiving thread. However, other methods may need to access other data owned by the object’s home shard, or need to prevent concurrent operations. Even if we’re sure that object is now used exclusively by the receiving thread, such methods must still be run, explicitly, on the home thread:

    // fp is some foreign_ptr<>
    return smp::submit_to(fp.get_owner_shard(), [p=fp.get()]
        { return p->some_method(); });

So seastar::foreign_ptr<> not only has functional benefits (namely, to run the destructor on the home shard), it also has documentational benefits - it warns the programmer to watch out every time the object is used, that this is a foreign pointer, and if we want to do anything non-trivial with the pointed object, we may need to do it on the home shard.

Above, we discussed the case of transferring ownership of an object to a another shard, via seastar::foreign_ptr<std::unique_ptr<T>>. However, sometimes the sender does not want to relinquish ownership of the object. Sometimes, it wants the remote thread to operate on its object and return with the object intact. Sometimes, it wants to send the same object to multiple shards. In such cases, seastar::foreign_ptr<seastar::lw_shared_ptr<T>> is useful. The user needs to watch out, of course, not to operate on the same object from multiple threads concurrently. If this cannot be ensured by program logic alone, some methods of serialization must be used - such as running the operations on the home shard withsubmit_to()` as described above.

Normally, a seastar::foreign_ptr cannot not be copied - only moved. However, when it holds a smart pointer that can be copied (namely, a shared_ptr), one may want to make an additional copy of that pointer and create a second foreign_ptr. Doing this is inefficient and asynchronous (it requires communicating with the original owner of the object to create the copies), so a method future<foreign_ptr> copy() needs to be explicitly used instead of the normal copy constructor. # Seastar::thread Seastar’s programming model, using futures and continuations, is very powerful and efficient. However, as we’ve already seen in examples above, it is also relatively verbose: Every time that we need to wait before proceeding with a computation, we need to write another continuation. We also need to worry about passing the data between the different continuations (using techniques like those described in the Lifetime management section). Simple flow-control constructs such as loops also become more involved using continuations. For example, consider this simple classical synchronous code:

    std::cout << "Hi.\n";
    for (int i = 1; i < 4; i++) {
        sleep(1);
        std::cout << i << "\n";
    }

In Seastar, using futures and continuations, we need to write something like this:

    std::cout << "Hi.\n";
    return seastar::do_for_each(boost::counting_iterator<int>(1),
        boost::counting_iterator<int>(4), [] (int i) {
        return seastar::sleep(std::chrono::seconds(1)).then([i] {
            std::cout << i << "\n";
        });
    });

But Seastar also allows, via seastar::thread, to write code which looks more like synchronous code. A seastar::thread provides an execution environment where blocking is tolerated; You can issue an asyncrhonous function, and wait for it in the same function, rather then establishing a callback to be called with future<>::then():

    seastar::thread th([] {
        std::cout << "Hi.\n";
        for (int i = 1; i < 4; i++) {
            seastar::sleep(std::chrono::seconds(1)).get();
            std::cout << i << "\n";
        }
    });

A seastar::thread is not a separate operating system thread. It still uses continuations, which are scheduled on Seastar’s single thread (per core). It works as follows:

The seastar::thread allocates a 128KB stack, and runs the given function until the it blocks on the call to a future’s get() method. Outside a seastar::thread context, get() may only be called on a future which is already available. But inside a thread, calling get() on a future which is not yet available stops running the thread function, and schedules a continuation for this future, which continues to run the thread’s function (on the same saved stack) when the future becomes available.

Just like normal Seastar continuations, seastar::threads always run on the same core they were launched on. They are also cooperative: they are never preempted except when seastar::future::get() blocks or on explict calls to seastar::thread::yield().

It is worth reiterating that a seastar::thread is not a POSIX thread, and it can only block on Seastar futures, not on blocking system calls. The above example used seastar::sleep(), not the sleep() system call. The seastar::thread’s function can throw and catch exceptions normally. Remember that get() will throw an exception if the future resolves with an exception.

In addition to seastar::future::get(), we also have seastar::future::wait() to wait without fetching the future’s result. This can sometimes be useful when you want to avoid throwing an exception when the future failed (as get() does). For example:

    future<char> getchar();
    int try_getchar() noexcept { // run this in seastar::thread context
        future fut = get_char();
        fut.wait();
        if (fut.failed()) {
            return -1;
        } else {
            // Here we already know that get() will return immediately,
            // and will not throw.
            return fut.get();
        }
    }

23.3 Starting and ending a seastar::thread

After we created a seastar::thread object, we need wait until it ends, using its join() method. We also need to keep that object alive until join() completes. A complete example using seastar::thread will therefore look like this:

#include <seastar/core/sleep.hh>
#include <seastar/core/thread.hh>
seastar::future<> f() {
    seastar::thread th([] {
        std::cout << "Hi.\n";
        for (int i = 1; i < 4; i++) {
            seastar::sleep(std::chrono::seconds(1)).get();
            std::cout << i << "\n";
        }
    });
    return do_with(std::move(th), [] (auto& th) {
        return th.join();
    });
}

The seastar::async() function provides a convenient shortcut for creating a seastar::thread and returning a future which resolves when the thread completes:

#include <seastar/core/sleep.hh>
#include <seastar/core/thread.hh>
seastar::future<> f() {
    return seastar::async([] {
        std::cout << "Hi.\n";
        for (int i = 1; i < 4; i++) {
            seastar::sleep(std::chrono::seconds(1)).get();
            std::cout << i << "\n";
        }
    });
}

seastar::async()’s lambda may return a value, and seastar::async() returns it when it completes. For example:

seastar::future<seastar::sstring> read_file(sstring file_name) {
    return seastar::async([file_name] () {  // lambda executed in a thread
        file f = seastar::open_file_dma(file_name).get0();  // get0() call "blocks"
        auto buf = f.dma_read(0, 512).get0();  // "block" again
        return seastar::sstring(buf.get(), buf.size());
    });
};

While seastar::threads and seastar::async() make programming more convenient, they also add overhead beyond that of programming directly with continuations. Most notably, each seastar::thread requires additional memory for its stack. It is therefore not a good idea to use a seastar::thread to handle a highly concurrent operation. For example, if you need to handle 10,000 concurrent requests, do not use a seastar::thread to handle each — use futures and continuations. But if you are writing code where you know that only a few instances will ever run concurrently, e.g., a background cleanup operation in your application, seastar::thread is a good match. seastar::thread is also great for code which doesn’t care about performance — such as test code.