Background on http frontends

-civetweb frontend

--thread per connection, requires a lot of threads
---qos or priority queuing would block frontend threads

-beast frontend

--boost::beast for http parsing
--boost::asio for async networking/io
--async for accepting connections and reading headers
---good model for qos - can queue requests without blocking threads
--synchronous call to process_request()
---thread per request, still need lots of threads

goal: scale requests independently of threads

-why boost::asio

--doesn't impose a threading model. io_service object is a reactor, call run() from any thread
--mature library, basis for C++ std::net library in Networking TS [1]
--the Extensible Asynchronous Model [2] provides several options for async primitives (callbacks, futures, coroutines)
--boost::asio::spawn() stackful coroutines: "enables programs to implement asynchronous logic in a synchronous manner" [3 http://www.boost.org/doc/libs/1_65_1/doc/html/boost_asio/reference/spawn.html]

proposed librados interfaces for asio [4 https://github.com/ceph/ceph/pull/19054]

--header-only wrapper over librados c++ api
--conform to the Extensible Asynchronous Model, so support the same primitives - see unit tests for examples
--deeper Objecter integration work in progress by Adam Emerson [5]
--gives radosgw a unified interface for async operations over http and rados

async process_request() [6]

-add optional yield_context* argument to process_request()

-beast frontend passes one, civetweb passes nullptr

-any librados calls use new interface when given a yield_context

-requires the yield_context* to be passed everywhere in between

--but we can stash it in req_state to make it available to all ops

-getting started with the easy stuff

--rgw_get_system_obj()
--reading user objects for authentication
--reading bucket/bucket instance objects (common to most s3/swift ops)

-this process leaves a lot of gaps. for example, rgw_get_system_obj() is in tons of call paths without access to a yield_context

--(either outside process_request(), or just aren't hooked up yet)
--just passing 'nullptr' makes it impossible to differentiate the yield_context argument from its 4 other arguments that default to nullptr!
--that makes it impossible to reason about which call paths could run asynchronously

-measurable progress towards full asynchrony

--new vocabulary type 'optional_yield_context' with 'null_yield' for empty value
--null_yield designates a call site that is definitely synchronous
--makes it easy to audit the code and find the pieces that still need conversion

-fighting regression once we're close

--have librados calls log warnings when called synchronously from a beast frontend thread (using a thread_local flag)
--scan those logs in teuthology runs to flag failures

and then?

-vastly reduce the number of frontend threads for beast

-consolidate other background threads

remaining work:

- RGWGetObj waits on AioCompletions - use AioThrottle from PutObj instead

- replace librados IoCtx::operate() calls with rgw_rados_operate() and optional_yield_context

- thread optional_yield_context all the way from beast frontend to rgw_rados_operate() calls
- some cls client calls use IoCtx::operate() directly

- block_while_resharding() sleeps on a condition variable

- no async interface for pool object listings with IoCtx.nobjects_begin()

- libcurl http requests for auth (Keystone and OPA)

[1] "C++ Technical Specification - Extensions for Networking"
http://cplusplus.github.io/networking-ts/draft.pdf
[2] "Library Foundations for Asynchronous Operations"
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n3896.pdf
[3] Reference: boost::asio::spawn
http://www.boost.org/doc/libs/1_65_1/doc/html/boost_asio/reference/spawn.html
[4] "librados: add async interfaces for use with boost::asio"
https://github.com/ceph/ceph/pull/19054
[5] "osdc/Objecter: Boost.Asio (I object!)"
https://github.com/ceph/ceph/pull/16715
[6] work in progress branch:
https://github.com/cbodley/ceph/commits/wip-rgw-async-process-171120