Skip to content
hgryoo · Docs

CUBRID Thread Manager — High-Concurrency Connection/Worker Pool and Logical-Wait-Aware Concurrency Control (CBRD-26177 + CBRD-26662)

Contents:

This document tracks the redesign that replaces CUBRID’s legacy server threading — one polling thread per accepted connection plus a max_clients-sized cubthread::worker_pool with all dispatch through a per-pool mutex. That legacy baseline is documented separately in the sibling doc cubrid-thread-worker-pool.md; anything there describing “a thread per connection” or the task_group/task_worker sizing is superseded by the design below. The work landed for the guava release in two phases:

  • Phase 1 — Connection/Worker Pool Redesign (EPIC CBRD-26177, Resolved). Replaces the front half with a small bounded set of epoll-driven connection workers, adds a coordinator that balances connections across them and dynamically scales their count, bounds per-tick I/O via send/recv budgets, and rotates context allocation through per-worker freelists so the hot path no longer touches new/delete. The back-end task-worker pool is retained, resized by two interim tunables (task_group, task_worker).

  • Phase 2 — Logical-Wait-Aware Concurrency Control (EPIC CBRD-26662, implemented in PR #7323, open against develop). Retires the fixed task_group/task_worker sizing in favour of an elastic worker pool whose true concurrency is bounded by per-core slots rather than by a fixed thread count. A worker that enters a logical wait (a transaction lock wait, a CSS job-queue idle, or a blocking PL call) surrenders its slot so a different worker can run in its place; the freed slot is redistributed by a 50 ms slot daemon. The two knobs become max_request_concurrency (active-task cap) and max_request_worker (thread overcommit cap), both tunable at runtime.

Phase 1 is the front-end story (how bytes reach a task worker); Phase 2 is the back-end story (how many task workers may run at once, and what happens when one blocks). They share the same guava release train and the same “no perfmon on the hot path” discipline (below), so this single document carries both.

Theoretical Background

Section titled “Theoretical Background”

The connection front-end of a database server has to multiplex many TCP sockets onto a finite number of CPU cores. Three architectures have dominated the literature, each with a distinct mapping between sockets, threads, and event-loop iterations.

Thread-per-connection (one-thread-per-client). Each accepted socket gets a dedicated kernel thread that calls read()/write() directly. The model is simple — no event-loop bookkeeping, no demultiplexing — and is the design Stevens describes in UNIX Network Programming, Vol. 1 (3rd ed., §16.5 “TCP Concurrent Server, One Child per Client”) as the canonical Unix server. It scales until the kernel’s thread-switch overhead dominates: at C10K and beyond, the working set of stacks blows the L1/L2 caches, the scheduler’s runqueue grows linearly with idle threads, and any shared mutex between threads serializes the entire server. Database Internals (Petrov, 2019, §5.3 “Concurrent Execution”) summarises the lesson: “If you want to scale to tens of thousands of concurrent connections, having one thread per connection becomes impractical.”

Reactor (event-driven). A small fixed pool of event-loop threads each blocks on a multiplexer (select/poll/epoll/kqueue) and dispatches ready sockets synchronously. The reference work is Pai, Druschel, and Zwaenepoel, “Flash: An Efficient and Portable Web Server” (USENIX 1999) which demonstrated that a single asymmetric event loop using non-blocking I/O could match or beat threaded servers at an order-of-magnitude lower memory cost. The crucial mechanical refinement is edge-triggered epoll: with EPOLLET, the kernel reports a readiness transition exactly once, and the user-space loop is responsible for draining the socket until EAGAIN. Edge-triggering eliminates wake-up storms but forces the loop to bound how much it drains per fd — otherwise a single fat connection can starve the others. This is the head-of-line blocking problem inside an event-loop worker.

Proactor (asynchronous I/O). The kernel signals completion, not readiness — Windows IOCP, Linux io_uring, POSIX AIO. Conceptually superior for write-heavy workloads but operationally heavier and not yet the default for database front-ends. CUBRID’s redesign deliberately chose reactor + edge trigger; proactor is out of scope.

Admission control via budgets. Welsh, Culler, and Brewer’s SEDA (“SEDA: An Architecture for Well-Conditioned, Scalable Internet Services,” SOSP 2001) framed the front-end as a sequence of stages connected by bounded queues, with each stage applying its own admission policy. The empirical observation is that latency under saturation degrades far less when each stage caps the work it will absorb in a single tick. CUBRID’s recv_budget_per_connection and send_budget_per_connection (CBRD-26392) are the SEDA admission gate applied to a single epoll tick: a fat reader that would happily drain a megabyte must instead yield after 16 KB, register itself in an “exhausted” list, and let the worker round-robin back to it on the next iteration.

Pool sizing — Little’s law. Given an arrival rate λ (requests/sec) and an average per-request service time S (sec), the average number of in-flight requests is L = λ · S. A pool with fewer than L workers will queue indefinitely; a pool with significantly more workers wastes CPU on context switching and blocks on internal critical sections. Database Internals (§5.3) notes that real systems usually pick a small multiple of physical cores and tune empirically, because S varies with the workload. CBRD-26424 (score-based assignment) and CBRD-26636 (Worker count sweep) implement exactly this empirical loop: measure throughput at several task_worker sizes, pick the local maximum.

Atomic-free monitoring. Naïve performance counters use std::atomic<uint64_t>::fetch_add per event. Under high load the cache-line of the counter pings between cores; at hundreds of thousands of events per second per worker the contention itself becomes the bottleneck the counter was meant to measure. The established workaround is thread-local accumulation with lazy aggregation: each worker increments a private counter and the monitor reader sums them. CBRD-26191 demonstrates the gain on YCSB (workload-a: 58 K → 60 K ops; workload-b: 70 K → 73 K ops) by removing only the atomic instructions on the hot path. Connection worker statistics in this redesign follow the same rule — statistics::metrics<> is a plain uint64_t[] per worker, summed by the coordinator on a 1-second timer.

Bounded concurrency vs. worker count — the logical-wait problem. Phase 1 answers how many threads should exist; Phase 2 answers a different question — how many threads should be allowed to run at once. The two are not the same number. Little’s law (L = λ · S) sets the number of in-flight requests, but the service time S of a transaction is not pure CPU: it includes logical waits — time spent blocked on a row/page lock, on a condition variable, or on a nested call to another server (a PL/Java stored procedure). A worker parked in a logical wait consumes a pool slot without consuming a core. Database System Concepts (§18 “Concurrency Control”) frames the lock wait as the dominant non-CPU component of S under contention. The consequence for pool sizing is sharp: if the pool is deliberately kept small — because CBRD-26636 measured that a lean pool (≈4–6 × cores) maximises throughput by minimising critical-section and cache contention — then a handful of workers stuck in lock waits can occupy the entire pool, and the server appears frozen even though every core is idle. This is the classic thread-starvation deadlock: fewer worker threads than the peak number of simultaneously-blocked units of work.

The admission-control escape: decouple “active” from “resident”. The established fix is to separate a concurrency limit (how many units of work may be actively executing) from the thread count (how many OS threads are resident). MySQL’s Enterprise Thread Pool is the closest prior art: each thread group runs a stall-detection timer, and when it detects that all of a group’s threads are blocked with work still queued, it wakes or spawns an extra thread so the group makes progress — capping active threads in the common case but overcommitting when blocking demands it. SQL Server’s SOS scheduler achieves the same end cooperatively: a worker that blocks yields its scheduler to a runnable sibling. CUBRID Phase 2 adopts the MySQL shape but inverts the token: instead of counting running threads, it hands each active worker a slot, caps the slot count at max_request_concurrency, and makes a worker return its slot when it enters a logical wait so the slot — not a new thread — is what gets handed to the next unit of work. Threads may overcommit up to max_request_worker, but only max_request_concurrency of them hold a slot and run at any instant. The slot is thus a runtime-adjustable admission gate placed exactly at the boundary between “has CPU work to do” and “is waiting on something logical”.

Common DBMS Design

Section titled “Common DBMS Design”

The shared design space for connection front-ends has narrowed since the C10K era. Almost every modern engine sits at one of four points on the threads × event loop matrix.

PostgreSQL — process per connection. postmaster forks a postgres backend process per accepted connection. The model gives strong isolation (a crashing backend can be restarted without killing peers) at the cost of high per-connection memory (≥10 MB). The PostgreSQL community has consistently rejected proposals to replace the model in core; instead, the project recommends external poolers such as PgBouncer for high-concurrency workloads. There is no equivalent of CUBRID’s “one CPU-pinned event loop per N connections” inside PostgreSQL itself.

MySQL — thread-per-connection by default; thread pool plugin optional. The default Connection_handler_manager runs one-thread-per-connection, giving each TCP session a dedicated pthread. The Enterprise Thread Pool plugin replaces this with a fixed number of thread groups (typically equal to core count) plus a small admission queue per group. The plugin exists exactly because the unbounded thread-per-connection model collapses past a few hundred concurrent sessions on the same workloads CUBRID measured in CBRD-26152. CUBRID’s redesign moves into this same architectural neighbourhood — bounded connection workers, group-based task dispatch, admission via budgets — without making it a plugin.

Oracle — dedicated server vs. shared server (DRCP). The default mode is dedicated-server (process per session). Shared-server mode multiplexes many sessions onto a small pool of server processes via a dispatcher that owns the listening socket and passes requests through queues. Database Resident Connection Pooling (DRCP) generalises this so multiple application servers share the same backend pool. CUBRID’s coordinator has the same arbitration role as the Oracle dispatcher, but with finer per-worker statistics and an auto-scaling rule.

SQL Server — SOSScheduler (cooperative). SQL Server’s SOS scheduler runs a fixed number of worker threads (≈ logical core count) and switches them cooperatively at well-defined yield points inside the engine. Connections are attached to schedulers rather than owning a thread of their own. The CUBRID redesign is closer to this model than to PostgreSQL’s: connection workers are CPU-pinned, fixed in count within a min/max range, and process many sessions per loop iteration.

Where legacy CUBRID sat. Before CBRD-26177 the server ran a polling thread per connection (each css_master_thread-spawned session looped on its own socket) plus a cubthread::worker_pool of size max_clients — see cubrid-thread-worker-pool.md for the detailed walkthrough. With max_clients set to 2000 the engine genuinely held ≥4000 threads at full saturation. Each polling thread contended for the worker-pool’s per-core mutex on every job dispatch; CBRD-26152 measured the result on YCSB-a as monotonically decreasing throughput as concurrency rose, with CPU spending the extra cycles in mutex idle rather than user code.

Where the redesign sits. With CBRD-26177 the front becomes a small set (min_connection_worker … max_connection_worker, defaults 4 … cores/2) of epoll-driven cubconn::connection::worker threads each pinned to a core; the back stays a cubthread::worker_pool sized by task_group × task_worker (renamed from thread_core_count × the old worker count). A single cubconn::connection::coordinator thread, also pinned, brokers new-client placement, rebalancing, and auto-scaling. The hot path (connection worker → task push → task worker pop) no longer takes a shared mutex except briefly for css_conn_entry::cmutex / rmutex, both of which are per-connection.

Handling a worker blocked on a lock (Phase 2 comparison). The four engines above also diverge on what happens when a busy worker blocks on a lock, which is precisely the gap CBRD-26662 closes. PostgreSQL and Oracle dedicated-server sidestep the question — a blocked backend is its own process, so a blocked lock-waiter costs a process, not a shared pool slot. MySQL’s thread pool detects the stall and adds a thread to the group. SQL Server’s SOS scheduler yields the blocked worker’s scheduler to a runnable sibling. CUBRID Phase 2 lands between MySQL and SQL Server: like SQL Server it lets the blocked worker’s entitlement to run pass to another unit of work, and like MySQL it will overcommit threads (up to max_request_worker) when blocking demands it — but it expresses both through one artifact, the concurrency slot, rather than through group-local thread spawning or a cooperative scheduler. The distinguishing detail is where the hand-off is wired: not inside the lock manager (which is left untouched) but at the generic thread-suspension primitive in thread_entry.cpp that the lock manager, page-buffer latch, and CSS job queue all funnel through. The slot is surrendered for the two reasons that dominate back-end blocking — a transaction lock wait and a CSS job-queue idle — with the stored-procedure/PL path handled by an explicit release: one narrow integration site rather than edits scattered across each subsystem.

Motivation

Section titled “Motivation”

Phase 1 — CPU idle under concurrency (CBRD-26152 + CBRD-26177)

Section titled “Phase 1 — CPU idle under concurrency (CBRD-26152 + CBRD-26177)”

CBRD-26152 — “[Survey] 동시성 증가에 따른 CPU idle 증가 원인 조사” (“Survey of why CPU idle rises when concurrency increases”) — is the empirical study that motivated the redesign. Yechan Hong ran YCSB workload-b (read 95%, update 5%) with the client/CAS cap at 2000 and swept thread counts from 200 to 1000. The unexpected finding was quoted directly in the ticket:

“스레드의 개수가 200개에서 1000개로 증가하였지만, 오히려 iowait가 아닌 CPU idle이 증가하고 있다.” (As the thread count increased from 200 to 1000, CPU idle — not iowait — increased.)

If the bottleneck were disk, more threads would have shown up as iowait. CPU idle rising under load instead pointed at internal synchronization: threads arriving at the worker-pool dispatch mutex faster than the holder could release it, then the kernel parking them, leaving cores genuinely idle.

CBRD-26177 names two structural causes:

“각 connection 스레드들이 모두 따로 polling하고 cub_server는 이론 상 max_clients × 2 이상의 thread를 가지게 되므로 자원 및 관리 관점에서 비효율적이다.” (Each connection thread polls independently, and cub_server theoretically holds at least max_clients × 2 threads, which is inefficient from both a resource and management perspective.)

“동시성이 점차 높아질수록 각각이 core의 mutex를 잡고 job을 할당 받으려고 하므로 이 contention은 CPU가 idle에 있게 하는 주요 병목 지점이 된다.” (As concurrency rises, each thread contends for a core’s mutex to be assigned a job; this contention is the main bottleneck that keeps the CPU idle.)

The resulting goals were:

  1. Replace per-connection polling with a small bounded set of epoll-driven connection workers — eliminate excessive poll() calls (Acceptance Criterion 1 of CBRD-26177).
  2. Make throughput monotonic in concurrency — additional clients should not degrade the rate (Acceptance Criterion 2).
  3. Add admission-style backpressure inside each worker (CBRD-26392) so a single fat connection cannot starve its peers.
  4. Add load-aware placement and dynamic resizing (CBRD-26406, CBRD-26407, CBRD-26424) so the engine self-tunes between idle and saturated regimes.
  5. Strip atomics off the monitoring hot path (CBRD-26191).

CBRD-26177 also issued a hard directive that shaped every subsequent ticket and shapes this document:

“connection worker는 매우 동시성이 높은 hot-path이므로 perfmon 계열의 모니터링 코드를 추가해서는 안된다. 심각한 성능 저하를 일으킬 수 있다.” (The connection worker is a very high-concurrency hot path, so perfmon-class monitoring code must not be added. It can cause serious performance degradation.)

This is the single most important constraint to keep in mind when reading the source: anything that smells like a global atomic counter or a perfmon_inc_stat() call on the worker tick is a regression.

Phase 2 — Lock waits starve a lean worker pool (CBRD-26636 + CBRD-26662)

Section titled “Phase 2 — Lock waits starve a lean worker pool (CBRD-26636 + CBRD-26662)”

Phase 1 removed the front-end bottleneck, which surfaced a second one in the back end. CBRD-26636 — “[성능 실험] Worker 개수에 따른 성능 추이” — swept the task_worker count and found that a lean pool (≈ 4–6 × cores) consistently beat task_worker = max_clients on read-heavy YCSB, because each additional worker adds critical-section entries and resource contention:

“각 Worker들은 내부적으로 잦은 Critical section 진입과 Resource contention을 가지고 있어 너무 많으면 성능 저하를 유발합니다. 반면, 너무 적은 개수로 설정되면 자원을 활용하지 못해 성능 저하를 일으키고, 잠금이나 다른 task 대기로 인해 작업이 밀려, 사용자 입장에서는 DBMS가 멈춘 것처럼 보일 수 있다.” (Each worker internally does frequent critical-section entry and resource contention, so too many degrade performance. But too few under-utilise resources, and work backs up behind lock waits or other task waits — so from the user’s point of view the DBMS looks frozen.)

CBRD-26636 names the two failure modes of a lean pool concretely:

  1. A single heavy task (a long-running unit of work, distinct from a long transaction) occupies a worker and shrinks the effective pool.
  2. Workers entering a lock wait reduce the number of running workers, so throughput collapses toward the “DB is frozen” regime.

There is a third, structural instance of the same problem in the stored-procedure path. When a PL/SP method calls a nested method, the caller’s worker blocks until the callee returns; deep nesting can park many workers on nested calls and hang the whole pool. Legacy CUBRID worked around this with a dedicated temporary worker spawned per nested call (CBRD-26683).

CBRD-26662 — “Logical-Wait-Aware Concurrency Control” — attacks all three with one idea, quoting the EPIC:

“Task worker가 논리적 대기로 인해 멈추게 되면 새 Task worker를 띄운다. 동시에 수행되는 Transaction (=Active한 Task worker)의 개수는 high_concurrency 이하여야 한다.” (When a task worker stalls on a logical wait, start a new task worker. The number of concurrently-executing transactions — active task workers — must stay at or below high_concurrency.)

So the goal is to hold active concurrency at a lean, throughput-optimal level while never letting a lock wait shrink the set of runnable workers: a blocked worker yields its entitlement, a fresh or idle worker picks up queued work, and the temporary-worker hack for SP nesting becomes unnecessary because a nested-call wait now yields a slot like any other logical wait. (The EPIC’s working name high_concurrency became the parameter max_request_concurrency in the implementation — see Phase 2 below.)

CUBRID’s Approach

Section titled “CUBRID’s Approach”

The work is presented in two phases. Phase 1 (CBRD-26177) is the front-end redesign — how bytes get from a socket to a task worker — and is fully resolved in the guava branch. Phase 2 (CBRD-26662) is the back-end redesign — how many task workers may run at once and what happens when one blocks on a logical wait — and is implemented in PR #7323, open against develop. Phase 2 keeps every front-end mechanism of Phase 1 unchanged; it only replaces the fixed task_group/task_worker back-end pool with an elastic, slot-bounded one.

CUBRID’s Approach: Phase 1 — Connection/Worker Pool Redesign (CBRD-26177)

Section titled “CUBRID’s Approach: Phase 1 — Connection/Worker Pool Redesign (CBRD-26177)”

Status: Resolved (guava). Front-end redesign.

Phase 1 is best understood as three figures, mirroring the diagram pages of the EPIC: the AS-IS baseline, the TO-BE state after CBRD-26212/26255, and the post-CBRD-26407 state after the coordinator is added.

Architecture diagrams

Section titled “Architecture diagrams”

AS-IS (legacy). Each accepted client got a dedicated polling thread. Each polling thread, on every iteration, would push a task into the shared cubthread::worker_pool of size max_clients. The push acquired a per-core mutex; with hundreds of polling threads the mutex was contended on every dispatch.

Figure 1 — Legacy (AS-IS) architecture

Figure 1 — Legacy (AS-IS) architecture. N polling threads each acquire a shared per-core mutex before pushing to the task pool, causing contention that scales with active client count.

TO-BE (CBRD-26212 + CBRD-26255). A small bounded set of connection_worker threads each runs an epoll_wait loop with edge-triggered I/O over many client sockets. Each connection worker is CPU-pinned. When a complete request arrives, the connection worker calls css_push_server_task into the back-end task pool. The number of connection workers is controlled by min_connection_worker/max_connection_worker; the task pool is sized by task_group × task_worker.

Figure 2 — TO-BE (CBRD-26212) architecture

Figure 2 — TO-BE (CBRD-26212) architecture. A bounded set of CPU-pinned connection_worker threads each runs epoll_wait over many client sockets; completed requests are pushed via css_push_server_task into a task_group × task_worker back-end pool.

Post-CBRD-26407 (coordinator + freelist). A single coordinator thread, pinned to core 0, owns placement (new-client → worker), rebalancing (move existing connections between workers when load skews), and auto-scaling (hibernate/awaken workers within min..max). Workers send statistics to it on a slow timer; the coordinator broadcasts control messages back. Inside each worker, contexts are claimed from a per-pool freelist instead of new/delete-allocated each time.

Figure 3 — Post-CBRD-26407 coordinator architecture

Figure 3 — Post-CBRD-26407 coordinator architecture. A single coordinator thread pinned to core 0 owns placement, rebalancing, and auto-scaling; workers claim and retire contexts from pool::freelist; the Unix-domain controller socket allows external SCALE_UP / SCALE_DOWN / CLIENT_MOVE commands.

Connection worker (CBRD-26212)

Section titled “Connection worker (CBRD-26212)”

The connection worker is implemented as cubconn::connection::worker in connection_worker.{cpp,hpp}. It owns:

  • a Linux epoll instance (cubsocket::epoll m_events);
  • two file descriptors registered into that epoll: an eventfd (m_eventfd) for inter-thread wakeups and a timerfd (m_timerfd) for periodic work (hibernation check, statistics push, HA close-all);
  • two per-worker message queues (IMMEDIATE, LAZY) implemented with tbb::concurrent_queue<message> and an atomic size counter;
  • the live set of context * it owns (m_context), and a deferred removal queue (m_removed_context);
  • two budget knobs (m_recv_budget, m_send_budget) and an exhausted-context map (m_exhausted);
  • an atomic-free statistics::metrics<statistics::worker> m_stats for self-reporting to the coordinator.

Figure 4 — Connection worker anatomy

Figure 4 — Connection worker anatomy. One CPU-pinned thread runs a cubsocket::epoll reactor; the client sockets, the inter-thread m_eventfd doorbell, and the periodic m_timerfd are all registered into it. Ready fds flow through handle_reception / handle_transmission, gated by the send/recv budgets, with over-budget fds parked in m_exhausted and re-driven on the next tick; completed requests are pushed via css_push_server_task. The worker owns its IMMEDIATE/LAZY message queues, its live and deferred-removal context sets, and its atomic-free stats.

The constructor wires the epoll, registers the eventfd/timerfd, installs three timer handlers, and spawns the worker thread:

// worker::worker — src/connection/connection_worker.cpp
m_recv_budget = static_cast<size_t> (prm_get_integer_value (PRM_ID_CSS_RECV_BUDGET_PER_CONNECTION));
m_send_budget = static_cast<size_t> (prm_get_integer_value (PRM_ID_CSS_SEND_BUDGET_PER_CONNECTION));
m_exhausted.reserve (128);


m_eventfd = eventfd (0, EFD_NONBLOCK | EFD_CLOEXEC);
m_timerfd = timerfd_create (CLOCK_MONOTONIC, TFD_NONBLOCK | TFD_CLOEXEC);
// ... eventfd_register both into m_events ...


eventfd_addtimer (timer_type::HIBERNATE,    timer_latency::MEDIUM_LATENCY, &worker::hibernate_check);
eventfd_addtimer (timer_type::STATISTICS,   timer_latency::MEDIUM_LATENCY, &worker::statistics_metrics_to_coordinator);
eventfd_addtimer (timer_type::HA,           timer_latency::HIGH_LATENCY,   &worker::ha_close_all_connections);


m_thread = std::thread (&worker::attach, this);

worker::attach is the thread entry point; it calls initialize → run → finalize. initialize pins the thread to its assigned core via os::resources::cpu::setaffinity (m_core), claims a cubthread::entry, and sets the thread name to "connections" (a name that, as we shall see, leaks into the task pool in CBRD-26617).

The main loop is the textbook reactor:

// worker::run — src/connection/connection_worker.cpp
while (!m_stop)
  {
    nfds = m_events.wait (events.data (), events.size (),
                          m_exhausted.empty () ? TIMEOUT_INFINITE : TIMEOUT_NOWAIT);
    // ...
    for (i = 0; i < nfds; i++)
      {
        ctx = reinterpret_cast<context *> (events[i].data.ptr);
        if ((events[i].events & (EPOLLHUP | EPOLLRDHUP | EPOLLERR)) && ...)
          {
            this->handle_hangup_or_error (ctx, events[i].events & EPOLLERR);
            continue;
          }
        if (events[i].events & EPOLLIN)
          {
            if (ctx->m_conn->fd == m_eventfd) { eventfds[0] = true; continue; }
            if (ctx->m_conn->fd == m_timerfd) { eventfds[1] = true; continue; }
            status = this->handle_reception (ctx, false);
            // ...
          }
        if (events[i].events & EPOLLOUT)
          status = this->handle_transmission (ctx, false);
      }


    if (m_exhausted.size () > 0) handle_exhausted ();
    if (eventfds[0] || eventfds[1]) eventfd_handler (eventfds);
  }

Note the timeout switch: when there are exhausted contexts to re-drive (see Send/recv budgets below) the loop polls with TIMEOUT_NOWAIT so it can immediately revisit them, otherwise it blocks indefinitely on epoll_wait. The eventfd is the single inter-thread doorbell — any outside producer (the coordinator, another connection worker handing off, a task worker returning a buffer) writes 1 into m_eventfd and the worker drains its in-process queue once the loop wakes.

The connection::context (connection_context.hpp) is the per-client object the worker owns. It contains the css_conn_entry *m_conn, a worker index, a unique 64-bit id, a receive state machine (HEADER → DATA → ERROR), the receiver and transmitter, and an inline statistics::metrics<statistics::context>. A complete request (header + optional data) is parsed inside worker::handle_reception → handle_packet → handle_header_packet or handle_data_packet, and the task push into the back-end pool happens at handle_command_header_packet (when the request has no following data) or handle_data_packet (after the data arrives):

// worker::push_task_into_worker_pool — src/connection/connection_worker.cpp
void worker::push_task_into_worker_pool (context *ctx)
{
  /* push new task into worker pool */
  css_push_server_task (*ctx->m_conn);
}

That single call is the entire interface between the new front and the legacy back. css_push_server_task (in server_support.c) wraps the connection in a css_server_task and routes it to the cubthread worker pool with push_task_on_core (..., conn_ref.idx, conn_ref.in_method) — the core hash being the connection index, exactly as in the legacy design, so a long-running session keeps affinity for the same back-end core.

Connection lifecycle (close path) is driven by worker::handle_connection_close. It serialises against ctx->m_conn->cmutex, drains any in-flight task workers via net_server_active_workers, retries (re-enqueues a SHUTDOWN_CLIENT on the LAZY queue) if back-end workers are still active, and on success removes the fd from epoll, marks the context m_removed = true, and pushes it into m_removed_context. The actual context return to the pool is deferred to purge_stale_contexts, which sends a single RETURN_TO_POOL message to the coordinator with the batched list — so the freelist is touched once per loop tick, not once per closed connection.

Connection pool (CBRD-26255)

Section titled “Connection pool (CBRD-26255)”

The pool (cubconn::connection::pool in connection_pool.{cpp,hpp}) is the owner of workers, coordinator, and context freelist. It exists for the lifetime of the server and is held by cub_server as a single instance.

The freelist itself is a singly-linked stack of pool::freelist nodes, each of which embeds the actual context as its first member so that reinterpret_cast<freelist *> (ctx) recovers the node. The trick replaces the legacy “new context per connection” allocation pattern:

// pool::freelist — src/connection/connection_pool.hpp
struct freelist
{
  /* THIS MUST BE THE FIRST */
  context m_context;
  freelist *m_next;


  freelist (std::size_t capacity) : m_context (capacity), m_next (nullptr) {}
  ~freelist () = default;
};
// pool::claim_context / retire_context — src/connection/connection_pool.cpp
context *pool::claim_context ()
{
  freelist *head;
  assert (m_mutex_holder == std::this_thread::get_id ());


  head = m_freelist.m_head;
  if (head)
    {
      m_freelist.m_head = m_freelist.m_head->m_next;
    }
  else
    {
      head = new freelist (32 * 1024);
    }
  m_freelist.m_claim++;


  return &head->m_context;
}


void pool::retire_context (context *ctx)
{
  freelist *head;
  // ...
  head = reinterpret_cast<freelist *> (ctx);
  head->m_context.reset ();
  if (m_freelist.m_claim > m_freelist.m_max)
    delete head;            /* over-cap: actually free */
  else
    {
      head->m_next = m_freelist.m_head;
      m_freelist.m_head = head;
    }
  m_freelist.m_claim--;
}

Figure 5 — Context freelist embedding and claim/retire

Figure 5 — Context freelist. Each pool::freelist node embeds the context as its first member, so reinterpret_cast<freelist *> (ctx) recovers the node with no allocation header. claim_context pops the stack head (or news a node), and retire_context resets and pushes it back — or frees it when the live count is over the cap. Both run single-threaded on the coordinator under pool::m_mutex, so contexts need no per-object atomics.

The freelist is only manipulated by code holding pool::m_mutex. The coordinator’s handle_message_queue_new_client (which calls claim_context) and handle_message_queue_return_to_pool (which calls retire_context) both run on the coordinator thread, and the coordinator holds the pool lock for its entire lifetime (see coordinator::initialize → m_parent->lock_resource ()). This is the design choice that makes context allocation single-threaded without ever needing per-context atomics.

pool::initialize is wired via pool::initialize_topology, which maps the requested max_connection_workers onto an actual NUMA core layout via os::resources::cpu::effective () and may additionally serialise NIC RX/TX IRQ to those cores via os::resources::net::map_nic_to_index (cores). CBRD-26255 also provides this NIC-pinning, which is the source of the warning log messages discussed in the ticket comments (warning: NIC channel configuration failed) — they are non-fatal, surfacing only when the binary lacks CAP_NET_ADMIN or runs in a virtualised environment.

The shutdown sequence uses a thread_watcher (a bare condvar plus int active) to count down workers as they exit, and pool::finalize_workers waits up to css_get_shutdown_timeout() for m_watcher->active == 1 (only the coordinator left), then pool::finalize_coordinator waits for active == 0. Failure to reach those states triggers _exit(0) after a 10 s try-lock loop in try_to_lock_resource — a deliberate hard exit because the alternative is to wait forever for a thread holding state nothing else can clean up.

Send/recv budgets (CBRD-26392)

Section titled “Send/recv budgets (CBRD-26392)”

The budget mechanism is the single most subtle part of the design. Without it, edge-triggered epoll plus a draining reader would let a single client with backlog monopolise its worker: once EPOLLIN fires, the reader is contractually obliged to drain until EAGAIN; if the peer keeps writing, that drain loop never returns. CBRD-26392 caps the drain per epoll tick.

Quoting the ticket directly:

“하나의 connection worker는 여러 connection들을 관리한다. 이때 하나의 긴 송수신을 수행하게 되면 다른 송수신들이 계속 blocked되며 response가 지연되게 된다. 이때 한 번에 송수신할 수 있는 양을 제한하여 전체 지연을 안정화한다.” (One connection worker manages many connections. If a single long send or receive runs, the other I/Os remain blocked and their response is delayed. Bound the amount that can be sent or received at once to stabilise the overall latency.)

Defaults: 16 KB receive, 32 KB send (see system_parameter.c). Both can be set as low as 0 (no limit) or as high as 1 GB.

The implementation lives partly in receiver::drain / transmitter::fill (their second argument is a size_t limit = 0 budget) and partly in worker::handle_reception / worker::handle_transmission / worker::handle_exhausted_add_context / worker::handle_exhausted (connection_worker.cpp).

// worker::handle_reception — src/connection/connection_worker.cpp
io_status = ctx->m_recv.m_receiver.drain (ctx->m_conn->fd, m_recv_budget);
if (io_status == result::PeerReset || io_status == result::Error) { /* close */ }


assert (io_status == result::Pending || io_status == result::BudgetExhausted);


if (!in_exhausted && io_status == result::BudgetExhausted)
  {
    handle_exhausted_add_context (ctx, EPOLLIN);
  }
// worker::handle_transmission — src/connection/connection_worker.cpp
status = ctx->m_send.m_transmitter.fill (ctx->m_conn->fd, m_send_budget);
// ...
else if (!in_exhausted && status == result::BudgetExhausted)
  {
    handle_exhausted_add_context (ctx, EPOLLOUT);
  }

When a context exhausts its budget, it lands in m_exhausted keyed by context id. The main loop notices the non-empty exhausted map and switches epoll_wait to TIMEOUT_NOWAIT, then re-drives those contexts via handle_exhausted after serving the current epoll batch. The prepared flag in exhausted_context is the deferral guard: the first time a context is added it is marked !prepared and skipped; only on the second visit does the worker re-drain it. This ensures every other ready fd in the current epoll batch gets serviced before the budget-exceeded context is revisited.

The flow control finite-state machine for one fd:

stateDiagram-v2
  [*] --> Idle
  Idle --> Reading : EPOLLIN \n handle_reception
  Reading --> Idle : drain Pending \n EAGAIN
  Reading --> Exhausted : drain BudgetExhausted \n add to m_exhausted, EPOLLIN
  Exhausted --> Reading : revisit on next loop \n prepared flag
  Idle --> Writing : EPOLLOUT \n handle_transmission
  Writing --> Idle : fill Ok
  Writing --> Exhausted : fill BudgetExhausted \n add to m_exhausted, EPOLLOUT
  Reading --> Closing : ClosedConnection or PeerReset
  Writing --> Closing : ClosedConnection or PeerReset
  Closing --> [*] : handle_connection_close

Figure 6 — Per-fd flow-control FSM. BudgetExhausted (voluntary yield with bytes remaining) is distinct from Pending (kernel buffer empty); an exhausted context is re-queued with a prepared flag so it is revisited in the same epoll loop iteration.

Note that result::BudgetExhausted is a distinct enum value from result::Pending — the difference being that Pending means “the kernel has no more bytes for me right now” (back-off naturally until next epoll edge) while BudgetExhausted means “I have more bytes available but I’m yielding voluntarily” (must come back this loop or the next).

Auto scaling (CBRD-26406)

Section titled “Auto scaling (CBRD-26406)”

CBRD-26406 wires the mechanism for connection-rebalancing and worker-count scaling; the policy lives in CBRD-26424 (score-based selection, below). The mechanism is simple in shape: workers report statistics on a 1-second timer, the coordinator’s 5-second REBALANCING timer compares per-worker scores and asks the heaviest worker to hand off one of its connections to the lightest, the coordinator’s 60-second SCALING timer drives the auto-scaling state machine.

The scaling_status enum has only two states:

  • STABLE — current count is “good enough”, no measurement in progress.
  • TRIAL — sweep through count candidate sizes recording their throughput score, then pick the best.

At each SCALING tick:

// coordinator::statistics_scaling — src/connection/coordinator.cpp
if (m_scaling_statistics.status == scaling_status::STABLE)
  { this->scale_trial (); return true; }


assert (m_scaling_statistics.status == scaling_status::TRIAL);


bytes_inout = 0;
for (i = 0; i < m_max_worker; i++)
  {
    bytes_inout += m_statistics[i].m_sum.get (statistics::context::BYTES_IN_TOTAL);
    bytes_inout += m_statistics[i].m_sum.get (statistics::context::BYTES_OUT_TOTAL);
  }
m_scaling_statistics.history.push_back (
{ m_current_worker,
  VAL_TO_SCORE (50, 1000, bytes_inout) + m_task_statistics.completed.first * 2 });
m_scaling_statistics.count--;


if (m_scaling_statistics.count == 0)
  {
    selected = this->scale_selection ();      /* pick max-score scale */
    if (selected < m_current_worker)      this->scale_down ();
    else if (selected > m_current_worker) this->scale_up ();
    /* else stable */
  }
else
  {
    if (m_scaling_statistics.direction == scaling_direction::DOWN)
      this->scale_down ();
    else
      this->scale_up ();
  }

scale_trial clears the history, alternates the trial direction relative to the previous one (so consecutive trials don’t drift uni-directionally), and sets count to the auto_scaling_window_size parameter — the hyper-parameter that trades trial length for sensitivity. The default of 4 means each trial collects 4 samples (one per SCALING tick = 60 s) before deciding.

Sliding-window mechanism:

sequenceDiagram
  participant T as SCALING timer (60s)
  participant C as coordinator
  participant H as history (window_size = 4)

  Note over C: status = STABLE
  T->>C: tick
  C->>C: scale_trial()
  Note over C: direction = DOWN (or UP)<br/>count = 4<br/>status = TRIAL

  loop count = 4
    T->>C: tick
    C->>H: push_back({ current_worker, score })
    C->>C: scale_down() or scale_up()
  end

  T->>C: tick
  C->>H: push_back({ current_worker, score })
  C->>C: selected = scale_selection()
  alt selected != current
    C->>C: scale_down() or scale_up() to reach selected
  end
  Note over C: status = STABLE again

Figure 7 — Auto-scaling sliding-window sequence. Each SCALING tick (60 s) pushes a (current_worker, score) sample into a window_size=4 history; after the trial window scale_selection picks the highest-scoring worker count and adjusts m_current_worker accordingly.

scale_selection picks any sample within 95% of the maximum score, then chooses uniformly among them — a small Boltzmann-style randomisation to avoid getting stuck at a flat local maximum (see CBRD-26424 commentary on the dual local maxima observed in small-machine measurements).

scale_up flips the next-in-line hibernating worker out of HIBERNATING by sending an AWAKEN lazy message to it and incrementing m_current_worker. scale_down does the reverse in two phases: scale_down itself migrates every connection of the draining worker via transfer_connection and parks the coordinator status as DRAINING; scale_down_finish is the actual hibernation, called from handle_message_queue_statistics only once the draining worker reports an empty context list. This two-phase shutdown is necessary because worker shutdown is asynchronous and the coordinator must not allow a worker to be re-targeted by statistics_find_score_extremes while it is still serving connections.

Coordinator + context freelist (CBRD-26407)

Section titled “Coordinator + context freelist (CBRD-26407)”

The coordinator (cubconn::connection::coordinator in coordinator.{cpp,hpp}) is structurally the same shape as a worker — pinned thread, epoll instance, eventfd + timerfd, single-producer-single-consumer (TBB) queue — but it owns three distinct timers and an external Unix-domain control socket.

// coordinator::coordinator — src/connection/coordinator.cpp
m_controller.open ("/tmp/cub_server_" + std::to_string (getpid ()) + "_coordinator.sock",
                   SOCK_NONBLOCK | SOCK_CLOEXEC);
m_ctrlfd = m_controller.get_fd ();
m_eventfd = eventfd (0, EFD_NONBLOCK | EFD_CLOEXEC);
m_timerfd = timerfd_create (CLOCK_MONOTONIC, TFD_NONBLOCK | TFD_CLOEXEC);


eventfd_register (m_eventfd);
eventfd_register (m_timerfd);
eventfd_register (m_ctrlfd);


eventfd_addtimer (timer_type::STATISTICS,   timer_latency::LOW_LATENCY,    &coordinator::statistics_update);
eventfd_addtimer (timer_type::REBALANCING,  timer_latency::MEDIUM_LATENCY, &coordinator::statistics_rebalancing);
eventfd_addtimer (timer_type::SCALING,      timer_latency::HIGH_LATENCY,   &coordinator::statistics_scaling);

The three timer latencies are 1 second / 5 seconds / 60 seconds respectively (see the timer_latency enum in coordinator.hpp).

The control socket exposes administrative commands:

  • SHOW_STATS — print per-worker EWMA throughput and queue depth (statistics_print) to stdout.
  • SCALE_UP / SCALE_DOWN — force one step of the auto-scaling state machine.
  • CLIENT_MOVE — manually transfer one connection by id from worker from to worker to.

This is an out-of-band debugging interface; nothing in the data path uses it. Sending a control_recv struct via SOCK_DGRAM/SOCK_NONBLOCK triggers a reply with a single OK/NOK byte. The directive from CBRD-26177 (“no perfmon on the hot path”) means there is no SHOW server-side equivalent through the standard server channel — the controller is intentionally a side door, not a performance counter.

coordinator::handle_message_queue_new_client is where the placement policy lands. Note that it calls the same EWMA-driven score-extremes function used by rebalancing:

// coordinator::handle_message_queue_new_client — src/connection/coordinator.cpp
std::tie (worker, std::ignore) = statistics_find_score_extremes ();


m_statistics[worker].m_contexts.emplace (id,
                                         std::pair</*EWMA*/, /*prev*/>{ });
m_statistics[worker].m_client_num++;


request.type = connection::worker::message_type::NEW_CLIENT;
request.ctx = m_parent->claim_context ();
request.ctx->m_worker = worker;
request.ctx->m_id = id++;
request.conn = item.conn;
workers[worker]->enqueue (queue_type::IMMEDIATE, std::move (request));
workers[worker]->notify ();
this->statistics_update_score (worker);

— so every new client is immediately routed to the worker with the lowest current score, and that score is updated on the spot to bias the next placement.

The context migration protocol (used by both rebalancing and scale-down) is a four-step handshake between coordinator and two workers:

sequenceDiagram
  participant C as coordinator
  participant Wf as worker[from]
  participant Wt as worker[to]

  C->>C: m_migrating.insert(id)
  C->>Wf: HANDOFF_CLIENT(id, worker_ptr=Wt, worker_index=to)
  Wf->>Wf: locate ctx, remove from epoll/m_context
  Wf->>Wt: TAKEOVER_CLIENT(ctx)
  Wt->>Wt: register ctx in epoll (EPOLLIN | maybe EPOLLOUT)
  Wf->>C: HANDOFF_REPLY(transferred=true, id, from, to)
  C->>C: m_migrating.erase(id)<br/>fix m_statistics[from/to]

Figure 8 — Context migration handshake. Coordinator sends HANDOFF_CLIENT to worker[from]; worker[from] removes the context from its epoll and sends TAKEOVER_CLIENT to worker[to]; HANDOFF_REPLY closes the four-step transfer; m_migrating prevents double-targeting.

m_migrating prevents a connection from being targeted twice in flight. If the worker discovers the context is already gone (the client closed concurrently with the migration), the reply carries transferred=false and the coordinator reverts the projected stats. This is the single concurrency invariant the design relies on: a context is only ever owned by exactly one connection worker at a time, with ownership transferred via explicit message. No locks are required around the context itself — only the conn entry’s cmutex, briefly, for adapter field updates.

The context freelist (described above under CBRD-26255) was finalised in this same ticket. The CBRD-26407 description states the goal directly:

“context는 생성마다 Physical Memory와 Virtual Memory를 할당받고 이를 mapping하므로 이 과정을 생략하도록 한다.” (Each context creation allocates physical and virtual memory and maps the two, so skip this process.)

By pre-warming the freelist with max_connections * 1.1 preallocated freelist nodes (each 32 KB capacity), the runtime hot path is a pointer swap, not a mmap/page-fault sequence.

Score-based connection assignment (CBRD-26424)

Section titled “Score-based connection assignment (CBRD-26424)”

The coordinator’s score function combines three signals into a single comparable scalar per worker:

// coordinator::statistics_update_score — src/connection/coordinator.cpp
m_statistics[worker].m_score =
  1 * static_cast<double> (m_statistics[worker].m_client_num) / 1
  + EVAL_WORKER  (EWMA(MQ_COMPLETED), EWMA(BLOCKED_RMUTEX))
  + EVAL_CONTEXT (EWMA(BYTES_IN_TOTAL) + EWMA(BYTES_OUT_TOTAL),
                  EWMA(RECV_BUDGET_HIT) + EWMA(SEND_BUDGET_HIT));

with the weight macros

#define VAL_TO_SCORE(w, m, s) ((w) * static_cast<double> (s) / (m))
#define EVAL_WORKER(mq, rmutex)  (VAL_TO_SCORE (25, 3.5, (mq))    + VAL_TO_SCORE (500, 1, (rmutex)))
#define EVAL_CONTEXT(bytes, bgt) (VAL_TO_SCORE (50, 1000, (bytes)) + VAL_TO_SCORE (10, 1, (bgt)))

Concretely the weights mean: bytes-of-traffic count for 50 × 1/1000 (≈ 1 unit per kilobyte); rmutex blocked microseconds count for 500 × 1 (≈ 500 units per microsecond blocked); MQ completions count for 25 × 1/3.5 (≈ 7 units per completion). Budget-hit events (i.e., contexts that hit the recv/send budget cap) are weighted at 10 — because a high budget-hit count means the worker is repeatedly running into its admission cap and would benefit from an extra peer to share load. CBRD-26424’s commentary explains the dual local maxima visible in measured throughput curves: small machines exhibit a non-monotonic relationship between worker count and throughput because of NUMA / RX-TX / HT-sibling interactions, and a naïve hill-climber gets stuck. The randomised top-5% selection in scale_selection is the escape hatch.

EWMA aggregation uses α = 0.06 (EWMA_ALPHA):

// coordinator::statistics_EWMA — src/connection/coordinator.hpp
acc = acc * (1 - alpha) + (current - prev) * (alpha / (time_delta * 1e-6));
prev = current;

The division by time_delta * 1e-6 normalises to microseconds, so the EWMA is a smoothed rate (events per microsecond) rather than a raw delta. With α = 0.06 and a 1 s sampling interval the effective half-life is roughly 11 samples (≈ 11 s); aged samples contribute less than 1 % after about a minute.

Atomic-free stats (CBRD-26191)

Section titled “Atomic-free stats (CBRD-26191)”

The statistics::metrics<T, VT = uint64_t> template (connection_statistics.hpp) is a fixed-size VT[STATS_COUNT] with add / sub / get / set / reset operations. There is no std::atomic anywhere — every increment is a plain memory write, because every increment is performed by exactly one thread (the worker that owns the metric). Aggregation across workers happens once per second, when the worker copies its metric block into a coordinator::message::statistics payload and the coordinator does a per-worker EWMA update inside its own single-threaded handler:

// worker::statistics_metrics_to_coordinator — src/connection/connection_worker.cpp
message.type = coordinator::message_type::STATISTICS;
message.statistics.cpu_time_ns = get_time_ns (CLOCK_THREAD_CPUTIME_ID);
message.statistics.time_ns     = get_time_ns (CLOCK_MONOTONIC);
message.statistics.worker.first  = m_index;
message.statistics.worker.second = m_stats;        /* copy */
message.statistics.contexts.reserve (m_context.size ());
for (context *ctx : m_context)
  message.statistics.contexts.emplace_back (ctx->m_id, ctx->m_stats);  /* copy */
m_coordinator->enqueue (std::move (message));

The bulk copy is cheap because m_stats is a fixed array (≈ 88 bytes) and the per-context array is at most a few hundred entries of 56 bytes each. The copy moves ownership across the single-producer-single-consumer queue without crossing any cache line that the worker is concurrently writing to. Crucially, this design exists to uphold the CBRD-26177 directive (“no perfmon on the hot path”): the worker never increments a shared counter, never spins on a lock, never executes a memory barrier in the dispatch loop.

CBRD-26191 measured the wider goal — strip atomics from server-wide monitoring — on YCSB:

workloadbeforeaftergain
workloada58 464.2860 646.59+3.7%
workloadb70 009.9972 976.31+4.2%
update44 158.6645 128.96+2.2%
mix9 440.8210 115.33+7.1%

The connection-side metrics design follows the same template at the new layer.

TCP keepalive tunables

Section titled “TCP keepalive tunables”

CBRD-26177 promised three new per-socket keepalive parameters: tcp_keepalive_idle (start probing after N seconds idle), tcp_keepalive_interval (interval between probes), tcp_keepalive_count (consecutive failures = dead). The defaults are 300 s / 300 s / 3 with a high cap at 1 year of seconds. They are registered in system_parameter.c alongside the existing tcp_keepalive boolean and are intended to be applied by the socket-setup helper (tcp.c::css_sockopt) which already calls setsockopt (SOL_SOCKET, SO_KEEPALIVE, ...) when tcp_keepalive is set; the three new knobs feed TCP_KEEPIDLE, TCP_KEEPINTVL, TCP_KEEPCNT respectively for fine-grained tuning of dead-peer detection. The CUBRIDMAN-333 manual update covers the documentation rollout.

Task worker pool sizing — task_group / task_worker (interim, superseded by Phase 2)

Section titled “Task worker pool sizing — task_group / task_worker (interim, superseded by Phase 2)”

Interim mechanism. The task_group / task_worker sizing below is Phase 1’s back-end tuning knob and is what shipped with the CBRD-26177 merge. Phase 2 (CBRD-26662) retires it: task_worker is renamed to max_request_worker, task_group is demoted to a deprecated no-op, and the real cap on simultaneously-running transactions moves to the new max_request_concurrency. Read this subsection for the as-merged Phase 1 behaviour; read Phase 2 below for what replaces it.

The back-end pool in Phase 1 is cubthread::worker_pool in thread_worker_pool_impl.{hpp,cpp}. Its sizing is controlled by two parameters that replace the legacy thread_core_count/thread_worker_count pair:

  • task_group (renamed from thread_core_count) — number of cores in the worker pool. Each “core” in CUBRID terminology is a sub-pool with its own queue, owned by one worker_pool::core.
  • task_worker — total number of worker threads across all groups. Default at server startup: css_get_max_connections () (i.e., effectively the legacy max_clients), normalised down if it exceeds the system core count.

The auto-tuning code clamps task_group ≤ system core count and task_group ≤ task_worker (system_parameter.c boot sysprm tuning block):

/* sysprm_tune_client_parameters — src/base/system_parameter.c */
task_worker_prm = GET_PRM (PRM_ID_TASK_WORKER);
if (PRM_GET_INT (task_worker_prm->value) < 0)
  {
    /* the value of task worker is default. */
    sprintf (newval, "%d", task_worker);   /* css_get_max_connections() */
    (void) prm_set (task_worker_prm, newval, false);
  }


task_group_prm = GET_PRM (PRM_ID_TASK_GROUP);
if (PRM_GET_INT (task_group_prm->value) > system_cpu_count)
  {
    sprintf (newval, "%d", system_cpu_count);
    (void) prm_set (task_group_prm, newval, false);
  }
if (PRM_GET_INT (task_group_prm->value) > PRM_GET_INT (task_worker_prm->value))
  {
    sprintf (newval, "%d", PRM_GET_INT (task_worker_prm->value));
    (void) prm_set (task_group_prm, newval, false);
  }

The semantic shift is that task_worker is now interpreted as the total worker budget and task_group controls partitioning. The legacy thread_core_count was loosely “number of cores” with no policy; the new naming makes the intent explicit, and the coordinator’s task-completion EWMA (m_task_statistics.completed) uses css_get_task_stats from server_support.c to read the pool’s running totals into the score.

CBRD-26636 (“[성능 실험] Worker 개수에 따른 성능 추이”) found that task_worker ≈ 4–6 × cores consistently outperformed task_worker = max_clients on read-heavy YCSB workloads, but at the cost of a deadlock risk when task_worker < max_clients and many workers wait on a long lock. That risk motivates CBRD-26662 (see Phase 2 below).

CUBRID’s Approach: Phase 2 — Logical-Wait-Aware Concurrency Control (CBRD-26662)

Section titled “CUBRID’s Approach: Phase 2 — Logical-Wait-Aware Concurrency Control (CBRD-26662)”

Status: implemented in PR #7323 (feature/worker_pool_elastic, 21 commits, 45 files, +4054/−801), open against develop; test cases pending before merge. Back-end redesign.

Phase 2 replaces the fixed task_group × task_worker request pool with an elastic worker pool whose true concurrency is governed by a pool of slots rather than by a fixed thread count. The design realises the theory above: a worker must hold a slot to run, the slot count is capped at max_request_concurrency, and a worker that enters a logical wait surrenders its slot so another unit of work can run. Threads may overcommit up to max_request_worker.

The most important structural fact — and the reason the change is far smaller than “rewrite the lock manager” — is that the lock manager, page-buffer, and CSS queue are not modified. All three already suspend a thread through the same primitive in thread_entry.cpp, and Phase 2 injects the slot release/reacquire there, keyed off the existing suspended_reason — so one narrow site handles the slot hand-off for a transaction lock wait (soft) and a CSS job-queue idle (immediate), rather than edits scattered across each subsystem. A page-buffer latch wait funnels through the same primitive but is not slot-managed today — it only records a trace timestamp.

Three JIRA-to-code naming clarifications, because the ticket text and the shipped code diverge:

  • The EPIC’s high_concurrency and sub-task CBRD-26688’s max_transaction_concurrency both became max_request_concurrency in the code; the worker cap max_transaction_worker became max_request_worker.
  • CBRD-26688 says “remove task_group and task_worker”. In the code only task_worker is removed (renamed to max_request_worker); task_group is retained as a deprecated no-op (PRM_DEPRECATED, hard-coded to 1) so existing cubrid.conf files do not fail.
  • CBRD-26688’s constraint line reads max_transaction_worker <= max_transaction_concurrency, which is inverted; the code enforces the sensible direction, max_request_worker >= max_request_concurrency (workers are the overcommit ceiling above the active cap).

Elastic worker pool (CBRD-26684)

Section titled “Elastic worker pool (CBRD-26684)”

The elastic pool is cubthread::worker_pool_elastic<Stats> in thread_worker_pool_elastic.hpp, a subclass of the reworked worker_pool_impl<Stats>. It keeps the Phase 1 structure — one core_elastic per CPU core, each with its own task queue — but each core now also owns a concurrency_slot_pool m_slots, and the two sizing numbers become pool-wide atomics:

// worker_pool_elastic / core_elastic — src/thread/thread_worker_pool_elastic.hpp
std::atomic<std::size_t> &m_current_worker;   // shared across all cores
std::atomic<std::size_t> &m_max_worker;       // global overcommit cap
// global. this is not hard cap. allow to make worker overcommit

That the worker counters are global (shared by reference across every core_elastic) rather than per-core is the CBRD-26977 hardening change bundled into the same PR — it makes max_request_worker a whole-server ceiling, not a per-core one.

Dispatch is slot-gated. When work arrives or a slot frees up, core_elastic::adjust_workers drains the queue only as far as available slots permit:

// core_elastic::adjust_workers — src/thread/thread_worker_pool_elastic.hpp
while (!m_task_queue.empty () && m_slots.available_slots () > 0)
  {
    worker_elastic *w = get_or_make_available_worker ();   // reuse idle, else spawn
    if (w == nullptr) break;                               // hit m_max_worker
    std::unique_ptr<concurrency_slot> slot = m_slots.try_acquire_slot (ulock);
    // pop one task, hand it the slot
    w->try_execute_task_with_slot (std::move (task), std::move (slot));
  }

get_or_make_available_worker reuses an idle worker if one exists; otherwise it reserve_available_worker() (a CAS bounded by the global m_max_worker) and allocate_worker() → new worker_elastic() → start_thread(). This is the only place a fresh worker thread is born, and it happens precisely when there is queued work, a free slot, and no idle worker to run it — i.e. when a running worker has surrendered its slot to a logical wait.

A worker_elastic moves its slot into the thread entry for the duration of the task and returns it afterward:

// worker_elastic::execute_current_task — src/thread/thread_worker_pool_elastic.hpp
m_context_p->m_slot = std::move (m_slot);           // entry now holds the slot
m_wrapped_task->execute (*m_context_p);             // run the task
// ... on completion, the slot is returned to its pool ...
m_slot = std::move (m_context_p->m_slot);
concurrency_slot::return_to_pool (std::move (m_slot));   // return_slot drives this on the completion path

Workers self-retire when the pool has shrunk: worker_elastic::run calls get_retire_if_excess(this) after each task, which tears the worker down once m_workers.size () > m_max_concurrency and no snapshot reader is iterating the worker list.

Figure 9 — Elastic request pool

Figure 9 — Elastic request pool. Each core_elastic owns a task queue, a concurrency_slot_pool, and a variable set of worker_elastic threads. A worker must acquire a slot to run; the number of slots caps active concurrency while threads may overcommit up to the pool-wide m_max_worker. The 50 ms slot daemon rebalances slots across cores.

Concurrency slots and the per-core slot pool (CBRD-26685 / 26689)

Section titled “Concurrency slots and the per-core slot pool (CBRD-26685 / 26689)”

A concurrency_slot (in concurrency_slot.hpp) is a movable token. It remembers the core that created it and the core currently using it — slots migrate between cores — plus a soft-wait flag and timestamp:

// concurrency_slot — src/thread/concurrency_slot.hpp
class concurrency_slot
{
    concurrency_slot_pool *const m_owner_pool;   // core that created the slot
    concurrency_slot_pool       *m_holder_pool;  // core currently using it
    bool                         m_wait;         // soft-wait flag
    std::chrono::steady_clock::time_point m_wait_since;
  public:
    void reset ();
    void start_waiting ();
    void stop_waiting ();
    bool has_wait_expired (now) const;           // threshold: 50 ms
    static void return_to_pool (std::unique_ptr<concurrency_slot> slot);
};

The per-core pool concurrency_slot_pool is guarded by the owning core’s mutex (there is no separate slot lock — it piggy-backs on the core lock, so acquiring a slot and popping a task are one critical section):

// concurrency_slot_pool — src/thread/concurrency_slot.hpp
class concurrency_slot_pool : public concurrency_slot_subscriber
{
    std::mutex                                        *m_mutex;   // the core mutex
    std::queue<std::unique_ptr<concurrency_slot>>      m_available_slots;
    std::list<cubthread::entry *>                      m_wait_queue;  // threads waiting for a slot
    std::size_t                                        m_slot_count;   // slots this core owns
    std::size_t                                        m_target_count; // desired count
    bool                                               m_surplus;
    std::chrono::steady_clock::time_point              m_surplus_since;
    static constexpr std::size_t SLOT_SURPLUS_THRESHOLD = 2;
};

Slot management is entirely per-core, on the assumption that tasks distribute evenly across cores, so capping concurrency per-core caps it globally (CBRD-26685’s stated rationale). The key operations:

  • try_acquire_slot — non-blocking; pop from m_available_slots or return nullptr.

  • acquire_slot(entry *thread_p) — blocking. If a slot is free, assign it to thread_p->m_slot; otherwise enqueue thread_p on m_wait_queue, set its resume status to THREAD_CONCURRENCY_SLOT_SUSPENDED, and pthread_cond_wait until a slot is handed over. Crucially it saves and restores the thread’s original resume status, because a slot wait can nest inside a lock wait and must not clobber the lock’s wake result:

    // concurrency_slot_pool::acquire_slot — src/thread/concurrency_slot.cpp (condensed)
    if (!m_available_slots.empty ())
      {
        thread_p->m_slot = std::move (m_available_slots.front ());
        m_available_slots.pop ();
        thread_p->m_slot->set_holder_pool (this);
        return;
      }
    m_wait_queue.push_back (thread_p);
    auto saved = thread_p->resume_status;                       // nested wait guard
    thread_p->resume_status = THREAD_CONCURRENCY_SLOT_SUSPENDED;
    while (thread_p->resume_status == THREAD_CONCURRENCY_SLOT_SUSPENDED)
      pthread_cond_wait (&thread_p->wakeup_cond, &thread_p->th_entry_lock);
    thread_p->resume_status = saved;

  • release_slot(slot) — reset() the slot; if this core owns it and is over target, destroy it; otherwise store it in m_available_slots or, if a thread is waiting, hand it straight to the front waiter and wake it with thread_wakeup_already_had_mutex (waiter, THREAD_CONCURRENCY_SLOT_RESUMED).

  • return_slot(slot, force) — the completion path; returns the slot to a pool when there is queued work, a waiter, an over-target condition, or force, and otherwise settles the slot back toward its owner core via concurrency_slot::return_to_pool.

  • get_score() = -available_slots.size () + wait_queue.size () * 2.0f — the demand rank the daemon sorts on (higher = hungrier).

  • get_runtime_stats(total, target, busy) — feeds statdump; busy = m_slot_count − m_available_slots.size ().

Figure 10 — Per-core concurrency slot pool

Figure 10 — Per-core concurrency slot pool. m_available_slots holds free slot tokens; m_wait_queue holds threads suspended with THREAD_CONCURRENCY_SLOT_SUSPENDED while they wait for one. Each concurrency_slot records the core that created it (m_owner_pool) and the core using it now (m_holder_pool), so slots migrate between cores; get_score() ranks a core’s demand for the daemon. Everything runs under the core m_mutex.

stateDiagram-v2
  [*] --> Available : pool initialize \n one per max_request_concurrency
  Available --> Running : try_acquire_slot \n worker takes a task
  Running --> Available : task done \n return_slot
  Running --> Available : THREAD_CSS_QUEUE_SUSPENDED \n idle worker returns slot at once
  Running --> SoftWait : THREAD_LOCK_SUSPENDED \n start_waiting, keeps slot
  SoftWait --> Running : lock granted within 50 ms \n stop_waiting
  SoftWait --> Reclaimed : daemon sees has_wait_expired \n steal_from_entries_if_excess
  Reclaimed --> Available : distribute_slots to a hungry core

Figure 11 — Slot lifecycle. A slot is Available in a per-core pool, Running while a worker executes, and enters SoftWait on a lock suspension. If the lock is granted within 50 ms the worker keeps its slot; otherwise the daemon reclaims it and redistributes it. A CSS-queue idle returns the slot immediately. The worker whose slot was reclaimed must re-acquire one before resuming (Figure 12).

The slot daemon — soft/hard wait and inter-core transfer (CBRD-26686 / 26691)

Section titled “The slot daemon — soft/hard wait and inter-core transfer (CBRD-26686 / 26691)”

The redistributor is a single daemon, concurrency_slot_daemon, registered with a 50 ms looper and named "concurrency":

concurrency_slot.cpp
REGISTER_DAEMON (concurrency_slot_daemon);
looper loop = looper (std::chrono::milliseconds (50));
cubthread::get_manager ()->create_daemon (loop, new concurrency_slot_daemon_task (), "concurrency");

Its execute runs a five-step pass over every registered core pool (cores register themselves with the daemon via the publisher/subscriber pair concurrency_slot_publisher / concurrency_slot_subscriber):

// concurrency_slot_daemon_task::execute — src/thread/concurrency_slot.cpp (steps)
1. check_and_propagate_parameters ();                 // apply runtime param changes
2. steal_from_entries_if_excess (slots, identifier);  // reclaim slots from expired lock-waiters
3. steal_from_cores_if_excess   (slots, subs);        // reclaim per-core surplus
4. distribute_slots             (slots, subs);        // give slots to hungriest cores
5. wakeup_workers               (identifier, subs);   // core_elastic::adjust_workers

The soft-wait / hard-wait distinction lives in steps 2 and the suspension code. When a worker blocks on a lock it does not release its slot immediately (a soft wait): it just records the wait start time. Only if the wait outlasts the 50 ms threshold does the daemon promote it to a hard wait by stealing the slot:

// steal_from_entries_if_excess — via pool->map_running_contexts (func)
if (thread_ref.m_status == cubthread::entry::status::TS_WAIT
    && thread_ref.m_slot != nullptr
    && thread_ref.m_slot->has_wait_expired (now))       // held > 50 ms
  {
    steal thread_ref.m_slot;                            // hard wait: reclaim
  }

This two-tier scheme avoids churn: a lock wait that resolves quickly (the common case) never touches the daemon or gives up its slot, so short waits cost nothing; only genuinely long waits release capacity to other work.

Step 3 handles inter-core slot transfer. A core that has held a surplus of ≥ SLOT_SURPLUS_THRESHOLD (2) available slots for longer than 2000 ms hands the excess back:

// concurrency_slot_pool::borrow_surplus_slots — src/thread/concurrency_slot.cpp
if (m_surplus
    && now - m_surplus_since > std::chrono::milliseconds (2000)
    && m_available_slots.size () >= SLOT_SURPLUS_THRESHOLD)
  {
    hand back the surplus slots;    // returned to owners / hungry cores
  }

Step 4 sorts the pools by get_score() (descending demand) and round-robins the reclaimed slots to the hungriest cores; a borrowed slot is returned to its owner core once the borrowing work finishes, via the m_owner_pool / m_holder_pool bookkeeping on the slot.

sequenceDiagram
  participant W as worker (entry)
  participant SP as core slot pool
  participant D as slot daemon (50 ms)
  participant PEER as hungry core

  W->>SP: blocks on lock → THREAD_LOCK_SUSPENDED
  W->>W: thread_prepare_suspension<br/>start_waiting() — soft wait, keeps slot
  W->>W: pthread_cond_wait (suspended)
  Note over D: every 50 ms
  D->>W: has_wait_expired(50 ms)? yes<br/>steal m_slot (hard wait)
  D->>PEER: distribute_slots → release_slot
  PEER->>PEER: wake a waiter / spawn worker
  Note over W: later, lock granted
  W->>W: thread_prepare_resumption<br/>slot was stolen
  W->>SP: acquire_slot (blocking) — charged to slot_waits
  SP-->>W: slot granted → resume

Figure 12 — Lock-wait slot hand-off. A lock-blocked worker keeps its slot for up to 50 ms (soft wait). If the daemon finds the wait expired it reclaims the slot (hard wait) and redistributes it to a hungry core, which spins or wakes a worker. When the original worker is finally granted its lock it must re-acquire a slot before it can resume; that re-acquire time is accounted to event_stats.slot_waits.

Wiring logical waits — the suspension layer, locks, and SP (CBRD-26687 / 26683)

Section titled “Wiring logical waits — the suspension layer, locks, and SP (CBRD-26687 / 26683)”

The integration point is src/thread/thread_entry.cpp. The two existing entry points thread_suspend_wakeup_and_unlock_entry and thread_suspend_timeout_wakeup_and_unlock_entry were refactored to funnel through two new static helpers, thread_prepare_suspension and thread_prepare_resumption, which is where the slot logic lives. On suspend, the behaviour branches on the existing suspended_reason:

// thread_prepare_suspension — src/thread/thread_entry.cpp (condensed)
if (thread_p->m_slot != nullptr)
  switch (suspended_reason)
    {
    case THREAD_CSS_QUEUE_SUSPENDED:            // worker idling on the job queue
      holder = thread_p->m_slot->get_holder_pool ();
      slot   = std::move (thread_p->m_slot);
      thread_p->m_slot = nullptr;
      concurrency_slot::return_to_pool (std::move (slot));   // release immediately
      break;
    case THREAD_LOCK_SUSPENDED:                 // waiting to acquire a transaction lock
      holder = thread_p->m_slot->get_holder_pool ();
      thread_p->start_waiting ();               // soft wait; daemon may reclaim after 50 ms
      break;
    }

On resume, if a holder was recorded (asserting the reason was a CSS-queue or lock suspension), the worker’s fate depends on whether its slot survived:

// thread_prepare_resumption — src/thread/thread_entry.cpp (condensed)
if (timedout || resume_status == THREAD_RESUME_DUE_TO_INTERRUPT
              || resume_status == THREAD_RESUME_DUE_TO_SHUTDOWN)
  {
    if (thread_p->m_slot) release_slot (std::move (thread_p->m_slot));   // giving up
  }
else if (thread_p->m_slot != nullptr)
  {
    thread_p->stop_waiting ();                  // wait resolved < 50 ms; slot never stolen
  }
else
  {
    // slot was stolen by the daemon — must re-acquire, blocking
    static_cast<concurrency_slot_pool *> (holder)->acquire_slot (thread_p);
    // wait time charged to thread_p->event_stats.slot_waits
  }

Two supporting facts make this work. First, new resume statuses were added to thread_entry.hpp — THREAD_CONCURRENCY_SLOT_SUSPENDED = 25 and THREAD_CONCURRENCY_SLOT_RESUMED = 26 — along with the std::unique_ptr<cubthread::concurrency_slot> m_slot member (“only set for workers from an elastic worker pool”) and a struct timeval slot_waits inside event_stat. Second, network_sr.c adds THREAD_CONCURRENCY_SLOT_SUSPENDED to the net_server_wakeup_workers wake-eligible switch so a slot-suspended worker can be interrupted like any other.

Stored procedures (CBRD-26687 / temp-worker removal CBRD-26683). The SP path is the explicit (non-suspension) logical wait: when a PL method calls a nested method it blocks on a receive from the PL/Java server. Legacy CUBRID spawned a temporary worker per nested call to avoid parking the pool; Phase 2 deletes that mechanism entirely (execute_on_core(task, core_hash, bool is_temp) → execute_on_core(task, core_hash), and execute_task(task, is_temp) → execute_task(task)). Instead the SP call releases its slot around the blocking receive and reacquires afterward, using the explicit helpers thread_concurrency_slot_release / thread_concurrency_slot_acquire from thread_manager.hpp:

// pl_execution_stack_context.hpp — around a blocking PL receive
auto *holder = thread_concurrency_slot_release (m_thread_p);   // give up slot now
int error = conn->receive_buffer (b, &interrupt_func, 500);    // wait on PL server
thread_concurrency_slot_acquire (m_thread_p, holder);          // reacquire to resume

The difference from the lock path is the timing of the release: a lock wait is a soft wait (keep the slot, let the daemon reclaim only if it drags on), because most lock waits are short; a nested SP call is known up front to block on a different task, so it releases the slot immediately. Both funnel to the same concurrency_slot_pool.

Parameters: max_request_concurrency / max_request_worker (CBRD-26688 / 26690 / 26977)

Section titled “Parameters: max_request_concurrency / max_request_worker (CBRD-26688 / 26690 / 26977)”

task_worker is renamed to max_request_worker, task_group is deprecated to a no-op, and max_request_concurrency is added. Both new parameters are PRM_FOR_SERVER | PRM_USER_CHANGE, i.e. changeable at runtime with SET SYSTEM PARAMETERS:

ParameterEnum idDefault (server)RangeRuntimeMeaning
max_request_concurrencyPRM_ID_MAX_REQUEST_CONCURRENCYsystem_core_count() * 3[core*3 … CSS_MAX_CLIENT_COUNT]yesmax simultaneously-running transactions (total slots)
max_request_workerPRM_ID_MAX_REQUEST_WORKERCSS_MAX_CLIENT_COUNT[… CSS_MAX_CLIENT_COUNT]yesmax worker threads (overcommit ceiling)
task_groupPRM_ID_TASK_GROUP1 (deprecated no-op)—noretained only so old configs still load

The × 3 default is CBRD-26636’s empirical pick (a source comment reads /* adjusted based on CBRD-26636 */), where core × 2 and core × 3 gave the best throughput plateau. When the values are not user-set, prm_tune_parameters derives them from the client limit:

/* prm_tune_parameters — src/base/system_parameter.c (SERVER_MODE, condensed) */
client_limit = std::min (max_clients, CSS_MAX_CLIENT_COUNT);
factor = number_of_right_shifts (client_limit - 1);      /* clamped >= 3 */
max_value = std::min (std::min (cpu * factor, cpu * 16),
                      std::min (client_limit, CSS_MAX_CLIENT_COUNT / 2));
max_request_concurrency = std::max (cpu, max_value);
max_request_worker = std::min (max_request_concurrency * 2
                               + PRM_REQUEST_WORKER_ELASTIC_HEADROOM /* 32 */,
                               CSS_MAX_CLIENT_COUNT);
/* invariant enforced: max_request_worker >= max_request_concurrency */

Runtime changes (CBRD-26690). Because system parameters are polled rather than event-driven, the slot daemon re-checks them every tick in check_and_propagate_parameters, and on a change calls css_set_max_concurrency_and_workers → worker_pool_elastic::adjust_runtime_parameter, then writes the clamped values back with prm_set_integer_value:

  • Raising max_request_concurrency splits the new total evenly across cores (quotient + remainder) and supplies each core’s slot pool with more slots; lowering it lets the surplus slots be destroyed on return (applied lazily).
  • Raising max_request_worker lifts the global overcommit cap; lowering it blocks new worker creation and lets completed workers retire down toward the pool target.

cgroup awareness (CBRD-26977). The same PR makes the worker cap a global atomic (above) and teaches system_core_count() to read the CPU quota from cgroup v1 with a v2 fallback (cgroup.cpp gains quota_v1 / effective_v1 and a quota() dispatcher), so “core count” — which seeds all of the defaults above — reflects a container’s CPU limit rather than the host’s physical cores.

Observability: statdump and SHOW JOB QUEUES (CBRD-26692)

Section titled “Observability: statdump and SHOW JOB QUEUES (CBRD-26692)”

Because the CBRD-26177 “no perfmon on the hot path” rule still holds, concurrency is observed through peek statistics computed on demand, not hot-path counters. Six new single-peek PSTAT metrics are added (perf_monitor.{c,h}), filled by css_get_thread_runtime_stats which sums each core’s concurrency_slot_pool::get_runtime_stats:

$ cubrid statdump demodb | grep Num_request
Num_request_concurrency_total  = 100   # slots that exist
Num_request_concurrency_target = 100   # desired slot count
Num_request_concurrency_busy   =   1   # slots currently held
Num_request_worker_total       = 100   # worker threads alive
Num_request_worker_target      = 100   # desired worker count
Num_request_worker_busy        =   1   # workers currently running a task

The same six values re-column SHOW JOB QUEUES (its scan column count went 4 → 7 in show_meta.c / server_support.c, and the scan mapper css_wp_core_job_scan_mapper became a template over the worker-pool core type). Together with the per-thread event_stats.slot_waits accounting, these are the operator-facing signals for tuning max_request_concurrency against a real workload.

Source Walkthrough

Section titled “Source Walkthrough”

Symbols are grouped by subsystem. CBRD-* annotations attribute each symbol to its driving ticket where one is identifiable. The first groups (epoll, connection::*, worker_pool sizing, keepalive) are Phase 1 (CBRD-26177); the later groups (elastic pool, concurrency slot, slot daemon, suspension integration) are Phase 2 (CBRD-26662, PR #7323).

Figure 13 — File and subsystem map

Figure 13 — File and subsystem map. The connection reactor lives in src/connection/ (Phase 1, CBRD-26177): connection_pool owns the connection_worker epoll threads and the coordinator, with per-connection state in connection_context / connection_statistics / controller. server_support.c is the bridge — css_push_server_task hands a completed request from the reactor into the back-end task pool under src/thread/ (the thread_worker_pool interface, thread_worker_pool_impl, thread_manager, thread_entry). Phase 2 (CBRD-26662, PR #7323, dashed green) layers thread_worker_pool_elastic and concurrency_slot on the same base pool. src/base/epoll underpins the reactor and src/base/system_parameter feeds both layers; the wiring files integrate suspension (network_sr.c, pl_execution_stack_context.hpp) and observability (show_meta.c).

epoll wrapper (CBRD-26212)

Section titled “epoll wrapper (CBRD-26212)”
  • cubsocket::epoll (class, src/base/epoll.hpp) — RAII wrapper over epoll_create1/epoll_ctl/epoll_wait. Constructor opens an EPOLL_CLOEXEC instance; destructor closes it.
  • cubsocket::epoll::wait — thin shim over epoll_wait.
  • cubsocket::epoll::add_descriptor — EPOLL_CTL_ADD with optional void *ptr payload (used to thread context pointers through events[i].data.ptr).
  • cubsocket::epoll::modify_descriptor — EPOLL_CTL_MOD, used to add/remove EPOLLOUT when the transmitter queues pending data.
  • cubsocket::epoll::remove_descriptor — EPOLL_CTL_DEL.
  • cubsocket::nonblocking (parent class, nonblocking.hpp) — defines the result enum (Ok, Pending, BudgetExhausted, PeerReset, Error, ClosedConnection, Skewed, Aborted) that every receiver/transmitter/worker call returns.

connection::worker (CBRD-26212 / 26392 / 26406 / 26407 / 26617)

Section titled “connection::worker (CBRD-26212 / 26392 / 26406 / 26407 / 26617)”
  • cubconn::connection::worker — class definition in connection_worker.hpp. Members include m_parent (pool), m_coordinator, m_watcher, the per-thread state (m_thread, m_core, m_status, m_stop, m_entry), the context set (m_context, m_removed_context), the epoll (m_events), the eventfd/timerfd (m_eventfd, m_timerfd), the timer table (m_timer_handler), the dual-priority message queues (m_queue[IMMEDIATE/LAZY], m_queue_size[]), the budget knobs and exhausted map (m_recv_budget, m_send_budget, m_exhausted), and the worker-side metrics (m_stats).
  • worker::worker — constructor; reads system parameters, installs three timers, spawns the thread.
  • worker::attach — thread entry; calls initialize → run → finalize.
  • worker::initialize — sets affinity, claims thread entry, sets pthread name "connections" (the name leak CBRD-26617 caught).
  • worker::run — main reactor loop.
  • worker::finalize — drain still-open contexts, retire thread entry, signal watcher.
  • worker::enqueue / worker::notify / worker::enqueue_and_notify — outside-thread interface.
  • worker::push_task_into_worker_pool — single-line bridge to css_push_server_task (the back-end pool).
  • worker::handle_reception / worker::handle_transmission — per-fd I/O drivers; honour m_recv_budget/m_send_budget and emit BudgetExhausted. (CBRD-26392)
  • worker::handle_exhausted_add_context / worker::handle_exhausted — exhausted-fd revisitation queue. (CBRD-26392)
  • worker::handle_message_queue_new_client — bind a fresh context to a fd; register in epoll with EPOLLET|EPOLLIN|EPOLLRDHUP.
  • worker::handle_message_queue_handoff_client / worker::handle_message_queue_takeover_client — the two halves of the migration handshake. (CBRD-26406 / CBRD-26407)
  • worker::handle_message_queue_send_packet / worker::handle_message_queue_release_packet — task workers shipping bytes back to a connection use these messages instead of writing the socket directly. Sending may add EPOLLOUT to the fd if the transmitter buffers data.
  • worker::handle_message_queue_shutdown_client — close- connection request from outside; calls handle_connection_close.
  • worker::handle_message_queue_hibernate / worker::handle_message_queue_awaken — auto-scaling state transitions.
  • worker::handle_connection_close — six-step close protocol with retry-via-LAZY-queue when back-end task workers still hold the conn.
  • worker::statistics_metrics_to_coordinator — every MEDIUM tick (1 s default), copy m_stats plus per-context metrics into a coordinator::message::STATISTICS. (CBRD-26191)
  • worker::hibernate_check — every MEDIUM tick, if status is HIBERNATING and m_context.empty(), stop the timer.
  • worker::ha_close_all_connections — every HIGH tick, if css_ha_server_state () == HA_SERVER_STATE_TO_BE_STANDBY, forcibly close all idle connections — the HA mode-change path that interacts with CBRD-26523.

connection::pool (CBRD-26255 / 26407)

Section titled “connection::pool (CBRD-26255 / 26407)”
  • cubconn::connection::pool::freelist — the singly-linked context cache node.
  • pool::initialize / pool::finalize — top-level bring-up / tear-down; called by the executable wire-up.
  • pool::initialize_topology — interrogates os::resources::cpu::effective () and (where capable) os::resources::net::map_nic_to_index ().
  • pool::initialize_freelist — pre-allocate max_connections * 1.1 freelist nodes.
  • pool::initialize_workers — create max_connection_workers pinned workers and pre-warm by sending each a START message on both queues.
  • pool::initialize_coordinator / pool::start_coordinator / pool::finalize_coordinator — coordinator lifecycle.
  • pool::dispatch — accept hand-off; called by master_connector once a TCP connection has completed CUBRID handshake. Sends a NEW_CLIENT to the coordinator.
  • pool::claim_context / pool::retire_context — freelist API; require m_mutex held by the calling thread.
  • pool::lock_resource / pool::release_resource / pool::try_to_lock_resource — the pool-wide mutex used by the coordinator for the duration of its lifetime.

connection::coordinator (CBRD-26406 / 26407 / 26424)

Section titled “connection::coordinator (CBRD-26406 / 26407 / 26424)”
  • cubconn::connection::coordinator — class definition in coordinator.hpp. Members include m_parent, m_watcher, the controller (Unix-domain socket m_controller, m_ctrlfd), the message queue (m_queue, m_queue_size), the worker count tracking (m_max_worker, m_min_worker, m_current_worker), the migration-in-flight set (m_migrating), the scaling bookkeeping (m_scaling, m_scaling_statistics), and per-worker statistics (m_statistics).
  • coordinator::coordinator — opens the controller socket, registers fds into epoll, installs three timers, spawns thread.
  • coordinator::run — main reactor loop.
  • coordinator::initialize — pin to core 0 (or the first effective core), claim thread entry, set name "coordinator", take the pool lock for life.
  • coordinator::handle_message_queue_new_client — placement: pick min-score worker, allocate context, forward NEW_CLIENT. (CBRD-26424)
  • coordinator::handle_message_queue_return_to_pool — bulk return from a worker’s m_removed_context; clears per-context stats and calls pool::retire_context.
  • coordinator::handle_message_queue_handoff_reply — finalise migration; revert stats on transferred=false.
  • coordinator::handle_message_queue_statistics — per-worker stats arrival; runs EWMA update via statistics_update_connection, then statistics_update_score; if the reporting worker is the current draining_worker and reports empty contexts, calls scale_down_finish. (CBRD-26424)
  • coordinator::handle_message_queue_shutdown — flip m_stop true.
  • coordinator::transfer_connection — guarded by m_migrating; sends HANDOFF_CLIENT to the source worker.
  • coordinator::scale_up — AWAKEN next worker, bump m_current_worker. (CBRD-26406)
  • coordinator::scale_down / coordinator::scale_down_finish — drain target worker’s connections, then HIBERNATE. (CBRD-26406)
  • coordinator::scale_trial / coordinator::scale_selection / coordinator::statistics_scaling — the auto-scaling state machine. (CBRD-26406 / CBRD-26424)
  • coordinator::statistics_rebalancing — every MEDIUM tick (5 s), find score extremes, transfer one context if the gap exceeds 20 % of the high score. (CBRD-26424)
  • coordinator::statistics_EWMA — α = 0.06, microsecond-normalised, used for both worker and context metrics.
  • coordinator::statistics_find_score_extremes — linear scan over m_statistics[0..m_current_worker) returning (min_index, max_index).
  • coordinator::statistics_update_score — applies the EVAL_WORKER + EVAL_CONTEXT + client_num formula.
  • coordinator::statistics_print — controller-driven console dump of per-worker score, EWMA, byte counts.
  • coordinator::handle_controller / coordinator::handle_controller_request — dispatch the four control-socket commands.

connection::context, controller, statistics

Section titled “connection::context, controller, statistics”
  • cubconn::connection::context — per-client state (worker index, id, ignore guard, recv state machine, receiver, transmitter, blocker shared_ptr, per-context metrics). 32 KB inline send/recv buffer.
  • cubconn::connection::context::reset — reset for reuse via the freelist.
  • cubconn::thread_watcher — mutex + cv + int active used for ordered shutdown.
  • cubconn::message_blocker — single-shot mutex + cv + bool done used for blocking enqueue_and_notify callers.
  • cubconn::connection::controller<RX,TX> — templated Unix-domain datagram socket wrapper (controller.hpp).
  • cubconn::statistics::context / cubconn::statistics::worker — enums of metric keys (connection_statistics.hpp).
  • cubconn::statistics::metrics<T,VT> — fixed-size array of counters; supports +=, - (returns metrics<T,double>), * (scaling), add, sub, get, set, reset, copy_from. No atomics. (CBRD-26191)

task worker pool changes (Phase 1)

Section titled “task worker pool changes (Phase 1)”

Phase 1 back-end symbols. Phase 2 (CBRD-26662) replaces this pool for the request path with worker_pool_elastic — see the elastic-worker-pool group below.

  • cubthread::worker_pool (thread_worker_pool.hpp) — abstract interface (Phase 2 renames get_worker_count → get_pool_size and drops the is_temp parameter).
  • cubthread::worker_pool::core — in Phase 1 sized by task_group; in Phase 2 the request pool has one core per CPU core.
  • cubthread::worker_pool::execute / execute_on_core — entry points called from css_push_server_task.
  • cubthread::worker_pool_task_capper (thread_worker_pool_taskcap.{hpp,cpp}) — the legacy admission-cap wrapper retained for HA daemons; m_tasks_available = m_max_tasks = worker_pool->get_worker_count ().
  • css_push_server_task (server_support.c) — the hot-path handoff; partitions by static_cast<size_t> (conn_ref.idx) so a connection always lands on the same task-pool core.
  • css_get_task_stats (server_support.c) — fills stats[3] = { requested, started, completed } from the pool’s internal counters; consumed by coordinator::statistics_update_task.

system parameters

Section titled “system parameters”
  • PRM_ID_TCP_KEEPALIVE_IDLE / PRM_ID_TCP_KEEPALIVE_INTERVAL / PRM_ID_TCP_KEEPALIVE_COUNT — keepalive tunables.
  • PRM_ID_TASK_GROUP (renamed from thread_core_count in Phase 1; deprecated to a 1-valued no-op in Phase 2 — PRM_DEPRECATED).
  • PRM_ID_TASK_WORKER — removed in Phase 2, renamed to PRM_ID_MAX_REQUEST_WORKER (see Phase 2 symbols).
  • PRM_ID_CSS_MAX_CONNECTION_WORKER / PRM_ID_CSS_MIN_CONNECTION_WORKER (Phase 2 retunes the server default to max (system_core_count () / 2, 1)).
  • PRM_ID_CSS_AUTO_SCALING_WINDOW_SIZE.
  • PRM_ID_CSS_RECV_BUDGET_PER_CONNECTION / PRM_ID_CSS_SEND_BUDGET_PER_CONNECTION.

elastic worker pool (CBRD-26684 / 26683)

Section titled “elastic worker pool (CBRD-26684 / 26683)”
  • cubthread::worker_pool_elastic<Stats> (thread_worker_pool_elastic.hpp) — subclass of worker_pool_impl<Stats>; hosts one core_elastic per CPU core.
  • worker_pool_elastic::adjust_runtime_parameter (max_concurrency, max_worker) — split max_concurrency evenly across cores, apply to each core’s slot pool and worker set.
  • worker_pool_elastic::core_elastic — per-core unit; owns the task queue, a concurrency_slot_pool m_slots, and the worker_elastic list; carries the pool-wide std::atomic<std::size_t> &m_current_worker / &m_max_worker.
  • core_elastic::adjust_workers — slot-gated dispatch loop (while !queue.empty && slots available).
  • core_elastic::get_or_make_available_worker / reserve_available_worker (CAS on m_max_worker) / allocate_worker — worker reuse-or-spawn.
  • core_elastic::get_task_and_slot_or_become_available — atomic fetch-task-and-slot or register worker idle.
  • core_elastic::get_runtime_stats / has_queued_task / adjust_runtime_parameter.
  • worker_elastic — the elastic worker; execute_current_task (moves slot into entry::m_slot for the task), get_new_task, run, get_retire_if_excess.
  • cubthread::stats_t (enum class : bool { on, off }) / cubthread::pool_t (enum class : uint8_t { basic, elastic }) in thread_compat.hpp.
  • worker_pool_type<Stats, Type> alias + thread_create_worker_pool<...> (Args&&...) (thread_manager.hpp) — replaces the old worker_pool_type/stats_worker_pool_type typedefs and the two inline factories.
  • Removed temporary-worker surface (CBRD-26683): execute_on_core (task, core_hash, bool is_temp) → execute_on_core (task, core_hash); execute_task (task, bool is_temp) → execute_task (task); get_worker_count () → get_pool_size (); push_task_on_core loses its method_mode argument.

concurrency slot + per-core slot pool (CBRD-26685 / 26689)

Section titled “concurrency slot + per-core slot pool (CBRD-26685 / 26689)”
  • cubthread::concurrency_slot (concurrency_slot.hpp) — the token; m_owner_pool, m_holder_pool, m_wait, m_wait_since; reset, start_waiting, stop_waiting, has_wait_expired (now) (50 ms), return_to_pool.
  • cubthread::concurrency_slot_pool : concurrency_slot_subscriber — per-core pool guarded by the core mutex; m_available_slots (queue), m_wait_queue (std::list<entry *>), m_slot_count, m_target_count, m_surplus, SLOT_SURPLUS_THRESHOLD = 2.
  • concurrency_slot_pool::initialize / adjust_concurrency / try_acquire_slot / acquire_slot (blocking; nested-wait resume-status save/restore) / release_slot / return_slot / borrow_surplus_slots (2000 ms) / needs_slot / get_score / get_runtime_stats / check_surplus_slots / wakeup_workers / has_queued_task.

slot daemon + pub/sub (CBRD-26686 / 26691 / 26690)

Section titled “slot daemon + pub/sub (CBRD-26686 / 26691 / 26690)”
  • cubthread::concurrency_slot_subscriber — a core pool registers itself on activate (identifier) (identifier = owning worker_pool *).
  • cubthread::concurrency_slot_publisher — traverse (func) over m_subscribers under m_mutex.
  • cubthread::concurrency_slot_daemon : concurrency_slot_publisher — singleton; initialize / finalize / get_publisher / create_daemon (50 ms looper, name "concurrency").
  • concurrency_slot_daemon_task : entry_task (in .cpp) — execute = check_and_propagate_parameters → steal_from_entries_if_excess → steal_from_cores_if_excess → distribute_slots → wakeup_workers; plus has_slot_demand, tune_parameters (clamp [system_core_count(), CSS_MAX_CLIENT_COUNT], worker ≥ concurrency).

thread suspension integration (CBRD-26687)

Section titled “thread suspension integration (CBRD-26687)”
  • thread_prepare_suspension / thread_prepare_resumption (thread_entry.cpp, static) — slot release/soft-wait on suspend, slot keep/reacquire on resume; called from thread_suspend_wakeup_and_unlock_entry and thread_suspend_timeout_wakeup_and_unlock_entry.
  • cubthread::entry::m_slot (std::unique_ptr<concurrency_slot>), entry::start_waiting / stop_waiting, event_stat::slot_waits (thread_entry.hpp).
  • thread_resume_suspend_status::THREAD_CONCURRENCY_SLOT_SUSPENDED (25) / THREAD_CONCURRENCY_SLOT_RESUMED (26); labels added to thread_resume_status_to_string.
  • thread_concurrency_slot_release / thread_concurrency_slot_acquire (thread_manager.hpp) — explicit release/acquire around the SP blocking receive in pl_execution_stack_context.hpp.
  • net_server_wakeup_workers (network_sr.c) — accepts THREAD_CONCURRENCY_SLOT_SUSPENDED.

Phase 2 system parameters, stats, server wiring (CBRD-26688 / 26692 / 26977)

Section titled “Phase 2 system parameters, stats, server wiring (CBRD-26688 / 26692 / 26977)”
  • PRM_ID_MAX_REQUEST_CONCURRENCY (max_request_concurrency) — new; default system_core_count () * 3; PRM_USER_CHANGE.
  • PRM_ID_MAX_REQUEST_WORKER (max_request_worker) — renamed from PRM_ID_TASK_WORKER; PRM_USER_CHANGE.
  • PRM_REQUEST_WORKER_ELASTIC_HEADROOM (= 32); prm_tune_parameters rewrite deriving both from the client limit.
  • PRM_ID_PAGE_LATCH_TIMEOUT → PRM_ID_PAGE_LATCH_TIMEOUT_IN_MSECS (renamed + ×1000 rescale; incidental to this PR).
  • css_set_max_concurrency_and_workers / css_get_thread_runtime_stats (server_support.c) — runtime apply
    • peek-stats accessor; css_Server_request_worker_pool retyped to worker_pool_type<stats_t::on, pool_t::elastic>; concurrency_slot_daemon::initialize/finalize in css_init.
  • PSTAT_REQUEST_CONCURRENCY_TOTAL / _TARGET / _BUSY, PSTAT_REQUEST_WORKER_TOTAL / _TARGET / _BUSY (perf_monitor.{c,h}) — statdump Num_request_concurrency_* / Num_request_worker_*.
  • metadata_of_job_queues (show_meta.c) / css_wp_core_job_scan_mapper (templated) — SHOW JOB QUEUES 7-column layout.
  • cubthread::system_core_count / cgroup::cpu::quota (cgroup.cpp, resources.cpp) — cgroup v1/v2 CPU-quota aware core count (CBRD-26977).

Position hints (as of 2026-04-30)

Section titled “Position hints (as of 2026-04-30)”
SymbolFileLine
cubsocket::epoll (class)src/base/epoll.hpp42
cubsocket::epoll::epollsrc/base/epoll.cpp37
cubsocket::epoll::waitsrc/base/epoll.cpp54
cubsocket::epoll::add_descriptorsrc/base/epoll.cpp59
cubsocket::epoll::modify_descriptorsrc/base/epoll.cpp80
cubsocket::epoll::remove_descriptorsrc/base/epoll.cpp101
cubconn::connection::worker (class)src/connection/connection_worker.hpp52
worker::message_type (enum)src/connection/connection_worker.hpp106
worker::workersrc/connection/connection_worker.cpp75
worker::attachsrc/connection/connection_worker.cpp2107
worker::initializesrc/connection/connection_worker.cpp1943
worker::finalizesrc/connection/connection_worker.cpp1975
worker::runsrc/connection/connection_worker.cpp2007
worker::enqueuesrc/connection/connection_worker.cpp160
worker::notifysrc/connection/connection_worker.cpp182
worker::enqueue_and_notifysrc/connection/connection_worker.cpp218
worker::push_task_into_worker_poolsrc/connection/connection_worker.cpp288
worker::purge_stale_contextssrc/connection/connection_worker.cpp294
worker::handle_connection_closesrc/connection/connection_worker.cpp386
worker::statistics_metrics_to_coordinatorsrc/connection/connection_worker.cpp562
worker::hibernate_checksrc/connection/connection_worker.cpp584
worker::ha_close_all_connectionssrc/connection/connection_worker.cpp606
worker::handle_message_queue_new_clientsrc/connection/connection_worker.cpp1016
worker::handle_message_queue_handoff_clientsrc/connection/connection_worker.cpp1079
worker::handle_message_queue_takeover_clientsrc/connection/connection_worker.cpp1160
worker::handle_message_queue_shutdown_clientsrc/connection/connection_worker.cpp1227
worker::handle_message_queuesrc/connection/connection_worker.cpp1356
worker::handle_receptionsrc/connection/connection_worker.cpp1694
worker::handle_transmissionsrc/connection/connection_worker.cpp1782
worker::handle_exhausted_add_contextsrc/connection/connection_worker.cpp1837
worker::handle_exhaustedsrc/connection/connection_worker.cpp1854
cubconn::connection::pool (class)src/connection/connection_pool.hpp39
pool::freelistsrc/connection/connection_pool.hpp42
pool::initializesrc/connection/connection_pool.cpp62
pool::finalizesrc/connection/connection_pool.cpp89
pool::dispatchsrc/connection/connection_pool.cpp109
pool::claim_contextsrc/connection/connection_pool.cpp140
pool::retire_contextsrc/connection/connection_pool.cpp160
pool::initialize_freelistsrc/connection/connection_pool.cpp213
pool::initialize_topologysrc/connection/connection_pool.cpp249
pool::initialize_workerssrc/connection/connection_pool.cpp269
pool::finalize_workerssrc/connection/connection_pool.cpp314
pool::initialize_coordinatorsrc/connection/connection_pool.cpp353
pool::start_coordinatorsrc/connection/connection_pool.cpp376
cubconn::connection::coordinator (class)src/connection/coordinator.hpp41
coordinator::coordinatorsrc/connection/coordinator.cpp57
coordinator::initializesrc/connection/coordinator.cpp1192
coordinator::runsrc/connection/coordinator.cpp1240
coordinator::transfer_connectionsrc/connection/coordinator.cpp237
coordinator::scale_upsrc/connection/coordinator.cpp281
coordinator::scale_downsrc/connection/coordinator.cpp348
coordinator::scale_down_finishsrc/connection/coordinator.cpp317
coordinator::scale_trialsrc/connection/coordinator.cpp378
coordinator::scale_selectionsrc/connection/coordinator.cpp415
coordinator::statistics_find_score_extremessrc/connection/coordinator.cpp460
coordinator::statistics_update_scoresrc/connection/coordinator.cpp482
coordinator::statistics_update_connectionsrc/connection/coordinator.cpp502
coordinator::statistics_update_tasksrc/connection/coordinator.cpp545
coordinator::statistics_rebalancingsrc/connection/coordinator.cpp586
coordinator::statistics_scalingsrc/connection/coordinator.cpp629
coordinator::handle_message_queue_new_clientsrc/connection/coordinator.cpp934
coordinator::handle_message_queue_return_to_poolsrc/connection/coordinator.cpp970
coordinator::handle_message_queue_handoff_replysrc/connection/coordinator.cpp992
coordinator::handle_message_queue_statisticssrc/connection/coordinator.cpp1032
coordinator::handle_controller_requestsrc/connection/coordinator.cpp1110
cubconn::connection::contextsrc/connection/connection_context.hpp141
cubconn::statistics::metricssrc/connection/connection_statistics.hpp111
cubconn::connection::controller (template)src/connection/controller.hpp43
cubthread::worker_poolsrc/thread/thread_worker_pool.hpp54
cubthread::worker_pool_task_cappersrc/thread/thread_worker_pool_taskcap.hpp30
css_push_server_tasksrc/connection/server_support.c2354
css_get_task_statssrc/connection/server_support.c2647
REGISTER_CONNECTION (macro)src/thread/thread_manager.hpp496
PRM_ID_TCP_KEEPALIVE_IDLE (param row)src/base/system_parameter.c5161
PRM_ID_TASK_WORKER (param row)src/base/system_parameter.c5197
PRM_ID_CSS_MAX_CONNECTION_WORKER (param row)src/base/system_parameter.c5209
PRM_ID_CSS_AUTO_SCALING_WINDOW_SIZE (param row)src/base/system_parameter.c5243
PRM_ID_CSS_RECV_BUDGET_PER_CONNECTION (param row)src/base/system_parameter.c5259
PRM_ID_CSS_SEND_BUDGET_PER_CONNECTION (param row)src/base/system_parameter.c5271

Phase 2 position hints (PR #7323 — line numbers omitted until merge). Because CBRD-26662 is still an open PR against develop and the branch is not checked out locally, absolute line numbers are not stable; the table below anchors Phase 2 symbols to their files only. Refresh with line numbers when the branch merges.

SymbolFile
cubthread::worker_pool_elastic / core_elastic / worker_elasticsrc/thread/thread_worker_pool_elastic.hpp
cubthread::concurrency_slot / concurrency_slot_poolsrc/thread/concurrency_slot.hpp
concurrency_slot_daemon / concurrency_slot_daemon_tasksrc/thread/concurrency_slot.cpp
stats_t / pool_tsrc/thread/thread_compat.hpp
worker_pool_type<Stats,Type> / thread_create_worker_pool / thread_concurrency_slot_release / thread_concurrency_slot_acquiresrc/thread/thread_manager.hpp
thread_prepare_suspension / thread_prepare_resumptionsrc/thread/thread_entry.cpp
entry::m_slot / event_stat::slot_waits / THREAD_CONCURRENCY_SLOT_SUSPENDED / _RESUMEDsrc/thread/thread_entry.hpp
css_set_max_concurrency_and_workers / css_get_thread_runtime_statssrc/connection/server_support.c
PL slot release/acquire around receive_buffersrc/sp/pl_execution_stack_context.hpp
PRM_ID_MAX_REQUEST_CONCURRENCY / PRM_ID_MAX_REQUEST_WORKER / PRM_REQUEST_WORKER_ELASTIC_HEADROOMsrc/base/system_parameter.c
PSTAT_REQUEST_CONCURRENCY_* / PSTAT_REQUEST_WORKER_*src/base/perf_monitor.{c,h}
cgroup::cpu::quota (v1/v2)src/base/cgroup.cpp

Cross-check Notes

Section titled “Cross-check Notes”

Sibling doc — cubrid-thread-worker-pool.md. The legacy doc describes (a) css_master_thread accept loop, (b) one polling thread per accepted connection, (c) the cubthread::worker_pool and its core::worker machinery, (d) css_push_server_task as the dispatch point. Of those, (c) and (d) are still live and current. (a) is unchanged at the master-thread accept layer, but the handover point is now pool::dispatch (forwarding NEW_CLIENT to the coordinator) instead of “spawn a polling thread for this fd”. (b) is replaced: any reference in the legacy doc to “each connection has a thread” is no longer accurate. Look-up symbols that moved domains:

  • Polling/recv-loop logic in legacy was scattered across per-connection threads driven by css_internal_request_handler; now lives in cubconn::connection::worker::handle_reception and friends.
  • Connection-close protocol in legacy was a synchronous css_close_socket from the polling thread; now is worker::handle_connection_close with retry-via-LAZY-queue and a separate freelist return phase.
  • Stats in legacy were per-worker cubperf::stat_value arrays read with the worker pool’s get_stats; for the connection side, those readings no longer exist as counters at all (CBRD-26177 directive). Use the coordinator’s controller socket (SHOW_STATS) for diagnostics.
  • Admission control in legacy was worker_pool_task_capper for HA daemons only; in NG, every connection worker enforces a per-tick byte budget. The capper class is still in tree but is not on the connection-worker path.

Sibling doc — cubrid-server-session.md. Server session state lookup happens during request processing inside the task worker (after css_push_server_task lands in the back-end pool). The connection worker does not look up sessions; it only parses the network protocol. The session_p field on css_conn_entry is read on the task side (see css_server_task::execute in server_support.c). This is unchanged from the legacy doc and the redesign does not move it.

Regressions tracked under the EPIC.

  • CBRD-26586 — parallel query uses only one CPU after worker timeout. Root cause confirmed by Hong Yechan to be the interaction between thread_worker_timeout_seconds and affinity inheritance: when the connection worker creates a task worker (because the task pool let a thread expire), the new pthread inherits the connection worker’s CPU affinity, pinning all back-end work to the connection worker’s core. Fix: do not inherit affinity for newly-spawned task workers. Workaround until fix lands: set thread_worker_timeout_seconds high so back-end threads are not recycled.
  • CBRD-26617 — task worker thread name inherits “connections”. Same mechanism (attribute inheritance from the spawning thread). Confusing in core dumps because the thread name is used to label core.<name>... files, so a task-worker crash produced core.connections.*. Fix: set thread name when the task pool spawns a worker.
  • CBRD-26544 — schema_type_str synonym enum coredump. Pre-existing on develop; surfaced under the new build because CCI’s enum and its string array drifted out of sync. Fixed in the same merge window.
  • CBRD-26523 — HA test cases cbrd_21506_02, cbrd_22705_02 fail. Diagnosed as a pre-existing HA timing bug (logwr/copylog interaction on tid:0 system commits) that the redesign exposes because the new connection structure speeds up state transitions. Not a redesign regression; rerouted to CBRD-26576 for the actual fix.

HA-shell test set after merge. CBRD-26255 comments record a separate batch of HA shell-test failures (bug_bts_5212, bug_bts_9047, cbrd_22207, cbrd_23854, etc.) all attributed to timing changes — the redesign genuinely is faster, and that exposes test scripts whose sleeps and grep filters were calibrated to the legacy speed. The fixes were a mix of test-script timing tweaks and one genuine bug (-353 Resource temporarily unavailable under ulimit -n constraint, fixed by raising the FD limit and documenting the new minimum).

The CBRD-26177 “no perfmon” directive. Repeated here because it is the most likely thing to be broken by a future contributor:

“connection worker는 매우 동시성이 높은 hot-path이므로 perfmon 계열의 모니터링 코드를 추가해서는 안된다. 심각한 성능 저하를 일으킬 수 있다.”

Practical implications when reading or editing the code:

  1. Do not add perfmon_inc_stat or any global atomic increment to worker::run, worker::handle_reception, worker::handle_transmission, worker::handle_packet, the message-queue handlers, or any of their callees.
  2. Do add metrics to statistics::metrics<> instances on the worker (they are private uint64_t[]); the coordinator already sums them.
  3. The controller socket (SHOW_STATS) is the supported read-out path; statistics_print is the renderer.
  4. Per-context counters belong on context::m_stats, and their aggregation via the coordinator’s statistics_update_connection is already wired.

Phase 1 → Phase 2 status. Phase 1 (CBRD-26177) delivered “high throughput at high concurrency” but exposed the follow-on weakness CBRD-26636 measured: a lean back-end pool stalls when a few workers enter lock waits. Phase 2 (CBRD-26662) closes that with the slot mechanism documented above and is implemented in PR #7323 (feature/worker_pool_elastic), open against develop; per the assignee only test cases remain before merge. Consequences for anyone reading the tree:

  • The canonical back-end knobs are now max_request_concurrency (active-transaction cap) and max_request_worker (thread overcommit cap), not task_group / task_worker. task_worker is renamed away; task_group survives only as a deprecated 1-valued no-op so old cubrid.conf files still load.
  • Naming drift to watch for when cross-referencing tickets: the EPIC says high_concurrency, sub-task CBRD-26688 says max_transaction_concurrency / max_transaction_worker, but the shipped parameters are max_request_concurrency / max_request_worker. CBRD-26688’s constraint text (worker <= concurrency) is inverted relative to the code (worker >= concurrency).
  • The lock manager, page-buffer, and CSS queue were not modified; do not look there for the slot logic. It lives in thread_entry.cpp (thread_prepare_suspension / thread_prepare_resumption) and concurrency_slot.{hpp,cpp}.
  • PR #7323 also carries the CBRD-26977 hardening series (global worker cap, cgroup-v1/v2-aware core count) and several other changes that appear orthogonal to concurrency control — aggregate value-slot sharing in xasl_generation.c, page-buffer fix/unfix counter sharding in page_buffer.c, a connection_* NET_HEADER refactor, and a flashback/CDC guard in log_manager.c — none of which are part of the concurrency-slot mechanism (likely co-developed on the branch, though a few may have arrived via a develop merge).

Open Questions

Section titled “Open Questions”

  1. Affinity-aware connection placement. The coordinator picks the minimum-score worker. When a connection is pgxc-style stateful (HA replication, CDC consumer, log-writer slave), is there value in pinning it to a fixed worker for the connection lifetime? The current transfer_connection will re-balance even long-lived sessions; the only opt-out is is_wait_required returning false for cdc_Gl.conn.fd in worker::is_wait_required. A first-class “affinity-pinned connection” flag would close the gap.

  2. HA replication’s connection model. The connection worker honours HA_SERVER_STATE_TO_BE_STANDBY by force-closing non-active contexts (ha_close_all_connections). What happens during the opposite transition (standby → master), when a fresh batch of clients reconnects en masse and the coordinator has to allocate many contexts in a burst? The freelist is sized to max_connections * 1.1, so it should absorb the burst, but the coordinator is single-threaded on handle_message_queue_new_client. Concrete bound on the new-connection rate the coordinator can sustain is not measured.

  3. Score-function weights. The macros EVAL_WORKER (25, 3.5, …) + (500, 1, …) and EVAL_CONTEXT (50, 1000, …) + (10, 1, …) are tuned constants. CBRD-26424 acknowledged this is empirical. What is the sensitivity surface? Could a runtime-tunable weight set obviate auto_scaling_window_size by letting operators bias the score toward latency or throughput?

  4. Verification gap from CBRD-26421. The task explicitly stated that connection-worker rebalancing and dynamic scaling are not covered by automated tests because the connection pool’s internal state is not exposed through any user-visible interface. The controller socket is for debugging only. A read-only SHOW STATS SQL or DBA-RPC view would close the test gap. Phase 2 partly answers this on the back-end side — the Num_request_concurrency_* / Num_request_worker_* statdump counters and the re-columned SHOW JOB QUEUES (CBRD-26692) expose slot/worker occupancy — but the connection-worker (front-end) rebalancing state still has no user-visible view.

  5. std::nothrow vs. STL exceptions (CBRD-26412). The ticket’s resolution is essentially “we cannot guard exhaustively because STL throws and the codebase uses STL”. Some hot-path allocations (pool::freelist (32 * 1024), m_context.reserve (256), m_exhausted.reserve (128)) still throw on OOM. What’s the failure semantic the operator should expect — server crash, dropped connection, or graceful degradation? Today it is the first.

  6. Send/recv budget defaults. 16 KB / 32 KB are reasonable for OLTP but are likely small for bulk-load and CDC streaming. Is there a per-connection-class override path short of editing cubrid.conf?

  7. Per-core slot balance assumption (CBRD-26685). The slot pool manages concurrency per core on the stated assumption that “tasks distribute evenly across cores”, so a per-core cap approximates a global cap. Phase 1 routes a task to a core by conn_ref.idx hash, which is even in aggregate but not instantaneously — a burst of heavy tasks hashing to one core can locally exhaust its slots while peers idle. The slot daemon’s 2000 ms surplus-transfer smooths this over seconds; is that fast enough for latency-sensitive bursts, or should redistribution be demand-triggered rather than timer-triggered?

  8. Hard-wait threshold and surplus timings. The soft→hard wait promotion fires at a fixed 50 ms (concurrency_slot::has_wait_expired) and the inter-core surplus transfer at 2000 ms with SLOT_SURPLUS_THRESHOLD = 2. These are compile-time constants. Under a workload whose typical lock wait sits just above 50 ms, the daemon will churn slots (steal, then the waiter reacquires moments later). Should the threshold adapt to the observed lock-wait distribution?

  9. Overcommit ceiling vs. thread cost. max_request_worker defaults to max_request_concurrency * 2 + 32 and is a soft, non-hard cap (“allow to make worker overcommit”). Each parked worker still owns a cubthread::entry and a stack. What is the memory/scheduling cost at the ceiling on a many-core box, and is the + 32 headroom (PRM_REQUEST_WORKER_ELASTIC_HEADROOM) the right constant, or should it scale with core count?

  10. SP “execute up to some point without a slot” boundary (CBRD-26687). The ticket notes that for the SP path, reception itself is not tied to a code critical section, so a worker is allowed to run some way past the slot boundary before it must hold a slot again. Where exactly is that boundary drawn in pl_execution_stack_context, and can it admit more than max_request_concurrency genuinely-CPU-active workers in the SP case?

  11. Test-case status. CBRD-26662’s sub-tasks are all marked “regression test 로 검증 (verify by regression test after merge)”, and PR #7323 is open with test cases still pending. Until the regression suite lands, the concurrency invariant (“active transactions ≤ max_request_concurrency”) is asserted by design, not yet by an automated test.

Sources

Section titled “Sources”

Source paths

Section titled “Source paths”
  • src/connection/connection_worker.cpp (≈ 58 KB)
  • src/connection/connection_worker.hpp (≈ 10 KB)
  • src/connection/connection_pool.cpp (≈ 10 KB)
  • src/connection/connection_pool.hpp (≈ 3 KB)
  • src/connection/coordinator.cpp (≈ 35 KB)
  • src/connection/coordinator.hpp (≈ 10 KB)
  • src/connection/controller.hpp
  • src/connection/connection_context.hpp
  • src/connection/connection_statistics.hpp
  • src/connection/connection_support.{cpp,hpp}
  • src/connection/server_support.c — css_push_server_task (line 2354), css_get_task_stats (line 2647)
  • src/connection/tcp.c — setsockopt SO_KEEPALIVE (line 203)
  • src/base/epoll.{cpp,hpp}
  • src/thread/thread_worker_pool.hpp — abstract pool interface (line 54)
  • src/thread/thread_worker_pool_impl.{cpp,hpp} — pool implementation
  • src/thread/thread_worker_pool_taskcap.{cpp,hpp} — legacy admission cap
  • src/thread/thread_manager.hpp — REGISTER_CONNECTION (line 496)
  • src/base/system_parameter.{c,h} — param IDs and rows for tcp_keepalive_*, task_group, task_worker, min/max_connection_worker, auto_scaling_window_size, recv/send_budget_per_connection
  • src/executables/server.c — cubconn::connection::pool connections; (line 557)

Phase 2 (CBRD-26662, PR #7323) — line numbers are diff-relative until the branch merges:

  • src/thread/concurrency_slot.hpp (+235) / concurrency_slot.cpp (+925) — slot, per-core slot pool, pub/sub, slot daemon.
  • src/thread/thread_worker_pool_elastic.hpp (+831) — worker_pool_elastic / core_elastic / worker_elastic.
  • src/thread/thread_worker_pool_impl.{hpp,cpp} — reworked base pool (available-worker list, snapshot guard, get_pool_size).
  • src/thread/thread_worker_pool.hpp / thread_compat.hpp / thread_manager.{hpp,cpp} — interface, stats_t/pool_t, the worker_pool_type<> alias + factory, thread_concurrency_slot_*.
  • src/thread/thread_entry.{cpp,hpp} — thread_prepare_suspension/_resumption, m_slot, slot_waits, new resume statuses.
  • src/communication/network_sr.c — wake-eligible switch.
  • src/sp/pl_execution_stack_context.hpp — SP slot release/acquire.
  • src/base/system_parameter.{c,h} — max_request_concurrency / max_request_worker, task_group deprecation, prm_tune_parameters.
  • src/base/perf_monitor.{c,h} — Num_request_* peek stats.
  • src/base/cgroup.{cpp,hpp} / resources.cpp — cgroup-aware core count (CBRD-26977).
  • src/parser/show_meta.c + src/connection/server_support.c — SHOW JOB QUEUES 7-column layout; css_set_max_concurrency_and_workers, css_get_thread_runtime_stats.
  • Basic-pool migrations to the new alias: src/query/vacuum.c, src/loaddb/load_worker_manager.cpp, src/storage/btree_load.c, src/transaction/log_page_buffer.c, src/query/parallel/px_worker_manager_global.hpp, src/transaction/log_recovery_redo_parallel.hpp.

JIRA tickets

Section titled “JIRA tickets”

Phase 2 (Logical-Wait-Aware Concurrency Control):

Textbook references

Section titled “Textbook references”
  • Silberschatz, Korth, Sudarshan. Database System Concepts, 6th ed. — Ch. 13 “Storage and File Structure” (buffer basics, framing of front-end vs back-end).
  • Petrov, Alex. Database Internals (O’Reilly, 2019). §5.3 “Concurrent Execution” — pool sizing intuition, C10K framing.
  • Stevens, W. Richard. UNIX Network Programming, Vol. 1, 3rd ed. — §16.5 “TCP Concurrent Server, One Child per Client” (the model the redesign moves away from).
  • Pai, V., P. Druschel, W. Zwaenepoel. Flash: An Efficient and Portable Web Server. USENIX 1999. (event-driven asymmetric multi-process design — direct ancestor of the reactor pattern in this redesign).
  • Welsh, M., D. Culler, E. Brewer. SEDA: An Architecture for Well-Conditioned, Scalable Internet Services. SOSP 2001. (admission control via bounded stage queues — the intellectual basis for recv_budget_per_connection / send_budget_per_connection).
  • Linux kernel docs — epoll(7), eventfd(2), timerfd_create(2). The EPOLLET (edge-triggered) semantics are mandatory background reading for anyone modifying worker::run.
  • Silberschatz, Korth, Sudarshan. Database System Concepts — Ch. 18 “Concurrency Control” (lock waits as the dominant non-CPU component of transaction service time — the framing for Phase 2’s slot admission).
  • MySQL Reference Manual — “MySQL Enterprise Thread Pool” and the thread-group stall detection mechanism (the closest prior art: cap active threads, overcommit on stall — Phase 2 adopts the shape but tokenises it as a slot).
  • Microsoft SQL Server docs — “Thread and Task Architecture Guide” (SOS scheduler, cooperative yielding at defined points — the yield-your-entitlement analogue to slot hand-off).
  • Linux kernel docs — cgroups(7) (cpu.max / cpu.cfs_quota_us) — the CPU-quota source system_core_count() reads under CBRD-26977 so defaults track a container’s limit.