Skip to content
hgryoo · Docs

CUBRID Broker — CAS Process Pool, Connection Routing, and the Client-Facing Front-End

Contents:

Theoretical Background

Section titled “Theoretical Background”

A relational database engine has to expose itself to the network, but the moment it does so, three orthogonal forces start to pull on the design. The engine wants long-lived, expensive in-process state — a parser, an XASL cache, a per-session prepared-statement registry, an authenticated transaction descriptor — that should not be torn down on every TCP disconnect. The driver wants the simplest possible model: open a socket, send SQL, read rows, close the socket. The operator wants a horizontal cap on how many of these heavyweight contexts the host actually has to feed, regardless of how many drivers happen to be banging on the door at peak. Database Internals (Petrov, ch. 13 “Database Architectures”) frames the resulting tension as the choice between dedicated worker contexts (one per client) and shared worker contexts (clients multiplexed onto a smaller pool), with a connection broker sitting between the two and turning many cheap TCP sockets into a few expensive engine handles.

Three independent design choices shape every concrete broker implementation, and frame the rest of this document:

  1. Process model — threads, processes, or both. A broker can run the pooled workers as threads inside one address space (low context-switch cost, but a worker fault takes the whole pool down, and a stuck worker may starve a shared malloc arena), as independent processes (paid an extra fork/IPC cost per handoff, but each worker is a fault domain of its own and can exit without poisoning anything), or as a hybrid (e.g., one I/O thread per accepted socket fanning out into a process pool). CUBRID picks pure processes — cub_broker parent forks a fixed array of cub_cas children at startup, and they live for the broker’s lifetime unless explicitly recycled. The price (a fork+execle once at startup, an SCM_RIGHTS socket pass per client) buys CUBRID a clean fault-isolation story, an OS-level resource cap (appl_server_max_num), and the ability to attach gdb to one CAS without freezing the rest.

  2. Pooling granularity — session pooling vs. transaction pooling. With session pooling a client gets its own dedicated worker context for the whole TCP session (driver connect() to driver close()), and the worker is reusable only after the client disconnects. With transaction pooling the worker context is held only for the duration of an open transaction; between transactions the client is unbound and any free worker can pick up its next statement. Transaction pooling gives much higher worker reuse but requires every session-scoped piece of state — prepared statements, SET variables, temp tables — to either live behind a session id the worker can recover, or be redone on every transaction. CUBRID lands closer to session pooling: a CAS that has accepted a client holds it for as long as the driver wants, with optional reset between transactions controlled by KEEP_CONNECTION (ON / AUTO). When AUTO is set and the client is idle past time_to_kill, the broker can mark the CAS for harvest while the driver is between transactions; when the driver comes back, the broker reassigns. The session-id continuity comes from the server session table (see cubrid-server-session.md), not from the broker.

  3. Where the broker proxies the wire. The broker is a level-7 proxy on the SQL protocol (it understands PING, QC query-cancel, the connect handshake, and routes the rest), so it has to make a choice about whether the client socket stays on the broker for the lifetime of the request or is handed off to a worker for direct talk. Handing off is faster (one less memcpy hop per packet, no broker bottleneck) but requires either having multiple listeners (one TCP port per worker — wasteful and hard to firewall) or transferring an open file descriptor between two unrelated processes. POSIX gives exactly one primitive for the latter: an SCM_RIGHTS ancillary message on a AF_UNIX SOCK_STREAM socket, in which the kernel duplicates the descriptor into the receiver’s table during recvmsg. This is the standard Linux idiom (used by nginx, pgbouncer in some modes, and the Postgres pg_dump parallel infrastructure). CUBRID uses it: after accepting a TCP socket on the public broker port and choosing a free CAS, the broker send_fd()’s the kernel descriptor to the CAS over a pre-existing AF_UNIX rendezvous channel, then drops its copy. From that point on, broker and CAS speak only through shared memory; the client’s bytes flow CAS↔driver directly, and the broker can be debugged or restarted without dropping live queries (in principle — see “Open Questions”).

After these three choices are named — process pool, session-bias pooling, fd-passing handoff — the rest of CUBRID’s broker code is a direct consequence.

Common DBMS Design

Section titled “Common DBMS Design”

Below the textbook framing, every relational engine that needs more than one client at a time evolves a similar shape. They are not in any of the canonical RPC papers; they are the engineering vocabulary that lives between the abstract pool and the source.

A bare engine without a broker. The simplest setup is no broker: each client TCP connection forks (PostgreSQL classic) or spawns a thread (MySQL classic) inside the engine itself. PostgreSQL’s postmaster process accepts on the public TCP port and forks a backend per client; the backend exec’s nothing — it inherits the postmaster’s address space and runs the SQL pipeline directly. MySQL spawns a THD thread per client. Both pay one OS-level worker per connected driver. This is fine for tens of clients; it does not scale to thousands of mostly-idle JDBC pools, which is why connection brokers were invented.

pgbouncer (PostgreSQL). Stand-alone process; pure session or transaction pooling. A driver connects to pgbouncer’s port; pgbouncer either holds a server-side connection per driver (session mode), per active transaction (transaction mode), or per SQL statement (statement mode, rarely used). It is a libevent-based single-process proxy: every byte goes through pgbouncer’s memory twice (read from driver, write to server, and back). The strength is statelessness — pgbouncer can be restarted without dropping the back-end Postgres workers; the weakness is the proxy hop on the hot path.

pgpool-II (PostgreSQL). Adds replication-aware load balancing and read/write split on top of pooling. Larger feature surface, larger blast radius. The operator pays for a more capable broker in exchange for a more complex one.

Oracle’s “shared servers” / MTS. No external process — the broker is inside the engine. A small pool of “dispatchers” owns the TCP sockets and a larger pool of “shared servers” actually runs the SQL; clients are queued from dispatcher to shared server on every request, statement-pool style. Dedicated-server mode (one server per session) is the default; shared servers are an opt-in. Oracle Connection Manager (CMAN) sits a level above this as an Oracle-aware TCP proxy with ACL and protocol-level firewall features.

MySQL Router / ProxySQL. External proxies that understand the MySQL protocol; ProxySQL adds query rewriting and routing rules. The shape is pgbouncer’s, decorated with more L7 logic.

CUBRID’s broker — process pool with fd-handoff. CUBRID’s position in this design space is a crisp mix of pgbouncer’s externality (separate process, separate port, ACL, SQL log collection) and Oracle MTS’s intra-engine pooling (the workers are real engine processes, not generic byte proxies). The handoff via SCM_RIGHTS removes the data-path hop that pgbouncer and ProxySQL pay; the broker stays in the control plane, the CAS owns the data plane.

Theory ↔ CUBRID mapping

Section titled “Theory ↔ CUBRID mapping”
Theoretical conceptCUBRID name
Broker parent processcub_broker (binary built from broker.c)
Pooled worker processcub_cas (binary built from cas.c + cas_common_main.c)
Per-broker config recordT_BROKER_INFO in broker_config.h, populated by broker_config_read
Per-CAS runtime recordT_APPL_SERVER_INFO in broker_shm.h:290
Process-tree shared stateSysV shm segment containing T_SHM_BROKER, T_SHM_APPL_SERVER, optional T_SHM_PROXY
Public TCP listenersock_fd bound to getenv(PORT_NUMBER_ENV_STR) in broker.c::init_env
Receiver thread (accept loop)receiver_thr_f in broker.c
Job queue (max-heap by priority + insertion time)T_MAX_HEAP_NODE job_queue[JOB_QUEUE_MAX_SIZE+1] inside T_SHM_APPL_SERVER
Dispatcher threaddispatch_thr_f (non-shard) / shard_dispatch_thr_f (shard) in broker.c
Idle-CAS chooserfind_idle_cas in broker.c
Auto-grow chooserfind_add_as_index + broker_add_new_cas in broker.c
Auto-shrink chooserfind_drop_as_index + broker_drop_one_cas_by_time_to_kill in broker.c
Per-CAS rendezvous socketAF_UNIX socket at path from ut_get_as_port_name (one per CAS index)
Client fd handoffsend_fd (broker_send_fd.c) on broker side; recv_fd (broker_recv_fd.c) inside net_connect_client
CAS main loopcas_main_loop in cas_common_main.c, called from cas_main/shard_cas_main
Per-CAS request dispatch tablestatic T_SERVER_FUNC server_fn_table[] in cas.c
CAS↔server protocolCSS protocol — see cubrid-network-protocol.md (CAS is a client of the server)
Connection-state machineenum t_con_status (CON_STATUS_OUT_TRAN, _IN_TRAN, _CLOSE, _CLOSE_AND_CONNECT)
UTS (worker) state machineUTS_STATUS_IDLE, _BUSY, _RESTART, _START, _CON_WAIT, _STOP macros in broker_shm.h
Access controlaccess_control_check_right in broker_acl.c + per-broker [%name] sections in cubrid_broker.acl
Monitoringbroker_monitor.c (broker_monitor admin tool, reads shm read-only)
Admin interfacebroker_admin_pub.c (drives cubrid broker {start,stop,on,off,reset,…})
SQL log harvestingbroker_log_top.c, broker_log_replay.c, broker_log_converter.c
Shard / proxy variantT_SHM_PROXY + proxy_* daemons in broker.c; one extra hop CAS↔proxy↔broker
ODBC gateway variantcas_cgw.c, cas_cgw_execute.c, cas_cgw_odbc.c — same broker, different CAS binary

CUBRID’s Approach

Section titled “CUBRID’s Approach”

The broker has six moving parts: the broker parent process that listens on the public port, the CAS pool that does the actual SQL work, the shared-memory segment that wires the two together without locks, the connection lifecycle that takes a fresh TCP syn through to a SQL-ready CAS handle, the access-control + SQL log + monitoring out-of-band facilities, and the SHARD/proxy and CGW (ODBC gateway) variants that reshape the same skeleton for different deployment shapes.

Process tree

Section titled “Process tree”
flowchart TB
  subgraph host["Single host"]
    direction TB
    BR["cub_broker (parent)<br/>broker.c::main<br/>listens on TCP :broker_port"]
    subgraph THR["broker threads"]
      RX["receiver_thr_f<br/>accept() + protocol-prefix sniff"]
      DSP["dispatch_thr_f<br/>pull from job_queue<br/>find_idle_cas → send_fd"]
      MON["cas_monitor_thr_f"]
      PSZ["psize_check_thr_f"]
      HC["hang_check_thr_f"]
      SVM["server_monitor_thr_f"]
    end
    subgraph CASES["forked CAS children (N = appl_server_max_num)"]
      direction LR
      C0["cub_cas[0]<br/>cas.c::cas_main_loop"]
      C1["cub_cas[1]"]
      Cn["…cub_cas[N-1]"]
    end
    SHM[["SysV shm:<br/>T_SHM_BROKER<br/>T_SHM_APPL_SERVER<br/>(job_queue, as_info[])"]]
    SRV[["cub_server<br/>(separate process,<br/>different port)"]]
  end
  CL["JDBC / CCI / ODBC<br/>driver process"]

  CL -- "1: connect TCP" --> BR
  BR --- THR
  BR -- "2: SCM_RIGHTS<br/>over AF_UNIX" --> C0
  BR -.-> C1
  BR -.-> Cn
  CL -. "3: client fd reused<br/>direct CSS over inherited TCP" .-> C0
  C0 -- "4: CSS protocol" --> SRV
  RX <-.-> SHM
  DSP <-.-> SHM
  MON <-.-> SHM
  C0 <-.-> SHM

The broker parent has one TCP listening socket sock_fd and a small fixed set of worker threads. The CAS children are forked once at boot (in broker_admin_pub.c::as_activate, called from br_activate), one per CAS slot up to appl_server_max_num. Their lifetime is decoupled from individual driver connections: a CAS handles many clients sequentially, and only restarts when the broker requests it (rolling restart, oversize, hang).

Shared memory layout

Section titled “Shared memory layout”

The broker’s coordination is almost entirely through SysV shm, because the children are independent processes that have to see the parent’s accounting decisions (which CAS is busy, which one is being drained, when did each one start, how many requests has it served). Lock-free access is enforced by who writes what rather than by mutexes:

flowchart LR
  subgraph SHM["SysV shm segment (broker_shm_id)"]
    direction TB
    BRINFO["T_SHM_BROKER<br/>&mdash; magic, num_broker, my_ip_addr,<br/>access_control_file, br_info[1]"]
    APPLSHM["T_SHM_APPL_SERVER<br/>&mdash; broker_name, log dirs,<br/>job_queue[JOB_QUEUE_MAX_SIZE+1],<br/>as_info[APPL_SERVER_NUM_LIMIT],<br/>access_info[ACL_MAX_ITEM_COUNT],<br/>unusable_databases[][]"]
    PROXSHM["T_SHM_PROXY (shard only)<br/>&mdash; proxy_info[MAX_PROXY_NUM],<br/>shard_user, shard_conn, shard_key"]
  end
  BR["cub_broker<br/>(writer for almost everything)"]
  CAS["cub_cas[i]<br/>(writes only as_info[i].*)"]
  ADM["cubrid broker admin tools<br/>(broker_admin_pub.c)"]
  MON["broker_monitor<br/>(read-only)"]

  BR -- "writes job_queue, br_info,<br/>service_flag, num_appl_server" --> SHM
  CAS -- "writes uts_status, con_status,<br/>num_request, last_access_time,<br/>session_id of self slot" --> SHM
  ADM -- "writes config-derived fields,<br/>num_appl_server changes" --> SHM
  MON -. "read-only via SHM_MODE_MONITOR" .- SHM

The contract: the broker writes the job queue and the service flags; each CAS writes its own slot’s status fields; admin tooling rewrites configuration when a cubrid broker reset lands. Because each writer touches a disjoint subset of fields, almost no mutex is needed; the few that exist are scoped narrowly: broker_shm_mutex (broker-internal slot accounting), run_appl_mutex (serialises fork+execle so the new CAS’s pid slot is not raced), clt_table_mutex + clt_table_cond (the producer/consumer rendezvous between receiver_thr_f and dispatch_thr_f), and a per-CAS POSIX semaphore as_info->con_status_sem for the broker↔CAS handshake on the “is this connection still live?” question.

The shm key is derived deterministically (DEFAULT_SHM_KEY = 0x3f5d1c0a plus the broker’s own appl_server_shm_id from config), so child processes attach by a value they get from the environment (APPL_SERVER_SHM_KEY_STR, AS_ID_ENV_STR) and a magic field check in uw_shm_open rejects stale segments.

Connection lifecycle

Section titled “Connection lifecycle”

The journey from client connect() to a SQL-ready CAS handle is the broker’s central choreography. The state machine is:

stateDiagram-v2
  [*] --> SERVICE_OFF: CAS slot configured but unstarted
  SERVICE_OFF --> SERVICE_ON: broker_add_new_cas \n fork+execle cub_cas

  SERVICE_ON --> UTS_IDLE: child registers \n service_ready_flag=TRUE

  UTS_IDLE --> UTS_BUSY: dispatch_thr_f picks slot \n send_fd handoff

  UTS_BUSY --> CON_OUT_TRAN: cas_main_loop accept \n cas_db_connect ok
  CON_OUT_TRAN --> CON_IN_TRAN: process_request first stmt
  CON_IN_TRAN --> CON_OUT_TRAN: end_tran(commit/rollback)
  CON_OUT_TRAN --> CON_CLOSE: KEEP_CON_AUTO + idle past time_to_kill
  CON_OUT_TRAN --> CON_CLOSE_AND_CONNECT: explicit con_close + reconnect

  CON_CLOSE --> UTS_IDLE: cas_cleanup_session \n ux_database_shutdown
  UTS_BUSY --> UTS_RESTART: cas_monitor_worker \n fn_status hung past hang_timeout
  UTS_RESTART --> SERVICE_ON: restart_appl_server(): kill+fork

  SERVICE_ON --> SERVICE_OFF_ACK: drop request from operator
  SERVICE_OFF_ACK --> SERVICE_OFF: stop_appl_server()

The two state vectors — service_flag (operator-visible: ON / OFF / OFF_ACK / UNKNOWN) and uts_status (transient: IDLE / BUSY / RESTART / START / CON_WAIT / STOP) — are written by different actors and combine to drive admission. The broker’s find_idle_cas walks as_info[0..N-1] looking for service_flag==SERVICE_ON && uts_status==UTS_STATUS_IDLE.

Step 1: receiver thread accepts the TCP socket and sniffs

Section titled “Step 1: receiver thread accepts the TCP socket and sniffs”
// receiver_thr_f — src/broker/broker.c
clt_sock_fd = accept (sock_fd, (struct sockaddr *) &clt_sock_addr,
                      &clt_sock_addr_len);
if (IS_INVALID_SOCKET (clt_sock_fd)) continue;


// ... condensed: TCP_NODELAY, keepalive, optional reject-while-hung ...


read_len = read_nbytes_from_client (clt_sock_fd, cas_req_header,
                                    SRV_CON_CLIENT_INFO_SIZE);


if (strncmp (cas_req_header, "PING", 4) == 0)        { /* health check */ }
if (strncmp (cas_req_header, "ST",   2) == 0)        { /* status query */ }
else if (strncmp (cas_req_header, "QC", 2) == 0
         || strncmp (cas_req_header, "CANCEL", 6) == 0
         || strncmp (cas_req_header, "X1", 2) == 0)  { /* query cancel */ }


// Otherwise: regular driver handshake.
// Validate magic, SSL flag, cas_client_type, version,
// run uw_acl_check on source IP, push to job_queue.

The receiver thread is the only place in the entire broker that actually inspects the client’s first bytes. The 30-byte cas_req_header (SRV_CON_CLIENT_INFO_SIZE) carries the magic string CUBRK00 (or CUBRKS00 for SSL), the client type (CCI/JDBC/ODBC/…), the protocol version, a function flag byte (capability bits like BROKER_RENEWED_ERROR_CODE), and padding. The same prefix is used as a control channel for PING, ST (status), QC/CANCEL/X1 (query cancel) so the broker can answer them inline without burning a CAS. Query cancel works by finding the CAS slot whose pid matches and sending SIGUSR1 — which the CAS has already wired to query_cancel (registered in cas_common_main.c::cas_main_init).

A successful regular client gets boxed into a T_MAX_HEAP_NODE:

// receiver_thr_f — src/broker/broker.c
new_job.id = job_count;
new_job.clt_sock_fd = clt_sock_fd;
new_job.recv_time = time (NULL);
new_job.priority = 0;
new_job.cas_client_type = cas_client_type;
new_job.port = ntohs (clt_sock_addr.sin_port);
memcpy (new_job.ip_addr, &(clt_sock_addr.sin_addr), 4);
strcpy (new_job.prg_name, cas_client_type_str[(int) cas_client_type]);
new_job.clt_version = client_version;
memcpy (new_job.driver_info, cas_req_header, SRV_CON_CLIENT_INFO_SIZE);


pthread_mutex_lock (&clt_table_mutex);
max_heap_insert (job_queue, job_queue_size, &new_job);
pthread_cond_signal (&clt_table_cond);
pthread_mutex_unlock (&clt_table_mutex);

The job queue is a max-heap (broker_max_heap.c) on (priority, insert_time_inverse), which means a long-waiting job’s priority gets bumped by max_heap_incr_priority between iterations and eventually beats a fresh arrival. Capacity is shm_appl->job_queue_size, configured per broker (default 1024). If the heap is full the receiver returns CAS_ER_FREE_SERVER to the driver and closes the socket — back-pressure rather than unbounded queueing.

Step 2: dispatcher pops the job and rendezvous-binds a CAS

Section titled “Step 2: dispatcher pops the job and rendezvous-binds a CAS”

The dispatcher thread pops from the same heap. It blocks on a condition variable when the heap is empty:

// dispatch_thr_f — src/broker/broker.c
pthread_mutex_lock (&clt_table_mutex);
if (max_heap_delete (job_queue, &cur_job) < 0) {
  // ... wait up to 30ms on clt_table_cond ...
}
hold_job = 1;
max_heap_incr_priority (job_queue);
pthread_mutex_unlock (&clt_table_mutex);


retry:
while (1) {
  as_index = find_idle_cas ();
  if (as_index < 0) {
    if (broker_add_new_cas ()) continue;     // auto-grow up to max
    else                       SLEEP_MILISEC (0, 30);
  } else break;
}


shm_appl->as_info[as_index].num_connect_requests++;
shm_appl->as_info[as_index].clt_version  = cur_job.clt_version;
memcpy (shm_appl->as_info[as_index].driver_info, cur_job.driver_info,
        SRV_CON_CLIENT_INFO_SIZE);
shm_appl->as_info[as_index].cas_client_type = cur_job.cas_client_type;
memcpy (shm_appl->as_info[as_index].cas_clt_ip, cur_job.ip_addr, 4);
shm_appl->as_info[as_index].cas_clt_port    = cur_job.port;


srv_sock_fd = connect_srv (shm_br->br_info[br_index].name, as_index);
// ... handshake on con_status, then ...
ret_val = send_fd (srv_sock_fd, cur_job.clt_sock_fd, ip_addr,
                   cur_job.driver_info);

Two important choices live in this snippet:

  • Idle-CAS scan is linear, not pooled. find_idle_cas walks as_info from index 0 forward and returns the first slot where service_flag == SERVICE_ON && uts_status == UTS_STATUS_IDLE and the per-CAS keep-alive predicate passes. There is no free-list, because the natural free-list is the slot table itself and contention is bounded by appl_server_max_num. The inner-loop cost is real but pre-paid: appl_server_max_num is typically tens to low hundreds. Larger pools would justify a free-list; CUBRID’s deployment shape does not.

  • Per-CAS rendezvous socket. connect_srv opens an AF_UNIX SOCK_STREAM socket whose path comes from ut_get_as_port_name(path, br_name, as_index, …) (typically under $CUBRID_TMP/). Each CAS is listening on that path; the broker connects a fresh control connection per handoff, does a 4-byte con_status ping-pong to make sure the CAS is alive and willing, calls send_fd, reads back a uts_status ack, and closes the rendezvous socket. The client’s TCP fd has now been duplicated into the CAS’s table. Both sides drop their copies of the original handoff socket; the CAS will use the passed fd as its client_sock_fd for the rest of the session.

Step 3: CAS receives the fd and runs cas_main_loop

Section titled “Step 3: CAS receives the fd and runs cas_main_loop”
// cas_main_loop — src/broker/cas_common_main.c (condensed)
for (;;) {
  ssl_client = false;
  // ...
  br_sock_fd = net_connect_client (srv_sock_fd);  // recv_fd from broker
  if (IS_INVALID_SOCKET (br_sock_fd)) goto finish_cas;


  req_info.client_version = as_info->clt_version;
  memcpy (req_info.driver_info, as_info->driver_info,
          SRV_CON_CLIENT_INFO_SIZE);


  // accept the now-passed fd as our client
  if (cas_accept_client (br_sock_fd, &client_sock_fd, &client_ip_addr) < 0)
    goto finish_cas;


  cas_log_open (broker_name);
  // ... read DB connection blob, parse via cas_parse_db_info ...


  err_code = cas_handle_db_connection (client_sock_fd, &req_info,
                                       &conn_info, cas_info,
                                       client_ip_addr, ops,
                                       is_new_connection);
  if (err_code < 0) { cas_finish_session (...); goto finish_cas; }


  // request loop
  while (fn_ret == FN_KEEP_CONN) {
    fn_ret = ops->process_request (client_sock_fd, &net_buf,
                                   &req_info, srv_sock_fd);
    cas_log_error_handler_clear ();
    as_info->last_access_time = time (NULL);
    if (as_info->con_status == CON_STATUS_OUT_TRAN
        && hm_srv_handle_get_current_count () >= shm_appl->max_prepared_stmt_count)
      fn_ret = FN_CLOSE_CONN;
  }


  ops->cleanup_session ();
finish_cas:
  cas_log_close (true);
  CLOSE_SOCKET (client_sock_fd);
  // and around the loop: maybe restart_is_needed, maybe just go IDLE
}

The clean-room observation here is that cas_main_loop is the only loop the CAS actually has. Per-driver state — DB connection to cub_server, prepared-statement registry, req_info — lives in the heap of the CAS process for the duration of the inner while (fn_ret == FN_KEEP_CONN). When the driver disconnects, cleanup_session either rolls back the open transaction and ends the server-side session (ux_end_session, which goes to xsession_end_session over the wire — see cubrid-server-session.md) or, if KEEP_SESS was returned, hands the session id back to the next driver via the server’s session table, not the broker’s.

process_request indexes server_fn_table[] by func_code (one of the 44 CAS_FC_* codes — END_TRAN, PREPARE, EXECUTE, FETCH, …). Each fn_* is the CAS-side stub that translates a driver request into one or more xsession_*/xqmgr_*/xtran_* calls on cub_server via the CSS protocol. The CAS is therefore both a server (to the driver) and a client (to cub_server).

Step 4: SQL flow end-to-end

Section titled “Step 4: SQL flow end-to-end”
sequenceDiagram
  autonumber
  participant Drv as JDBC/CCI driver
  participant Br as cub_broker
  participant Cas as cub_cas[i]
  participant Srv as cub_server

  Drv->>Br: TCP connect to broker_port
  Drv->>Br: 30-byte SRV_CON_CLIENT_INFO header (CUBRK00 + version)
  Br->>Br: validate magic, ACL, capacity → push to job_queue
  Br->>Cas: connect AF_UNIX rendezvous + send_fd(client_sock_fd)
  Cas->>Cas: cas_accept_client, cas_parse_db_info
  Cas->>Srv: NET_SERVER_PING_WITH_HANDSHAKE (CSS) [cub_server boot path]
  Cas->>Srv: NET_SERVER_BO_REGISTER_CLIENT (db_user, db_pass, capabilities)
  Srv-->>Cas: SESSION_ID, transaction_id, server caps
  Cas-->>Drv: CAS connection reply (cas_send_connect_reply_to_driver)
  loop per driver request
    Drv->>Cas: MSG_HEADER + func_code + args (CAS protocol)
    Cas->>Srv: NET_SERVER_QM_QUERY_PREPARE / _EXECUTE / etc.
    Srv-->>Cas: CSS reply
    Cas-->>Drv: net_buf reply
  end
  Drv-->>Cas: CON_CLOSE or TCP close
  Cas->>Srv: NET_SERVER_BO_UNREGISTER_CLIENT
  Cas->>Cas: ux_end_session, set uts_status = IDLE

The doubled-protocol shape — CAS protocol on the driver side, CSS protocol on the server side — is what the broker buys you. The driver speaks a stable, narrowly-versioned, 44-opcode protocol optimised for connection setup and statement execution; the CAS speaks a wider, internal CSS protocol with hundreds of opcodes (see cubrid-network-protocol.md’s enum net_server_request). Driver and server can evolve independently as long as the CAS keeps both ends honest.

Auto-grow and auto-shrink

Section titled “Auto-grow and auto-shrink”

Whenever find_idle_cas returns -1, the dispatcher calls broker_add_new_cas:

// broker_add_new_cas — src/broker/broker.c
cur_appl_server_num = shm_br->br_info[br_index].appl_server_num;
if (cur_appl_server_num >= shm_br->br_info[br_index].appl_server_max_num)
  return false;


add_as_index = find_add_as_index ();
if (add_as_index < 0) return false;


pid = run_appl_server (&(shm_appl->as_info[add_as_index]),
                       br_index, add_as_index);
if (pid <= 0) return false;


pthread_mutex_lock (&broker_shm_mutex);
shm_appl->as_info[add_as_index].pid          = pid;
shm_appl->as_info[add_as_index].uts_status   = UTS_STATUS_IDLE;
shm_appl->as_info[add_as_index].service_flag = SERVICE_ON;
(shm_br->br_info[br_index].appl_server_num)++;
pthread_mutex_unlock (&broker_shm_mutex);
return true;

The mirror-image is broker_drop_one_cas_by_time_to_kill, called from the parent’s idle loop in main every 100 ms when auto_add_appl_server == ON. It subtracts current busy CAS from queued waiters and, if the slack is positive, scans for a CAS that is idle, out-of-transaction, and last-accessed past time_to_kill. The chosen CAS gets con_status = CON_CLOSE and the CAS’s own loop, on its next net_read_*, sees the close and shuts down cleanly. The shrink path explicitly avoids reaping a busy CAS even if its idle timer has elapsed — the broker waits until the CAS naturally returns to OUT_TRAN.

The min_appl_server floor is enforced by the same function:

// broker_drop_one_cas_by_time_to_kill — src/broker/broker.c
if (cur_appl_server_num <= shm_br->br_info[br_index].appl_server_min_num
    || wait_job_cnt > 0)
  return false;

So the CAS pool oscillates between min and max, with max acting as a hard back-pressure ceiling and min as a warm-pool guarantee.

Access control (ACL)

Section titled “Access control (ACL)”

The broker has its own ACL layer, distinct from server-side authentication. A separate cubrid_broker.acl file (path is access_control_file in T_SHM_BROKER) carries per-broker sections:

[%query_editor]
demodb:dba:dba_ips.txt
demodb:public:public_ips.txt


[%broker1]
*:*:all_ips.txt

Each entry binds a (dbname, dbuser, ip-list-file) triple to a broker section. broker_acl.c::access_control_read_config_file parses the file at startup and on cubrid broker acl reload, pushing the result into shm_appl->access_info[]. Each CAS caches its own copy and refreshes it on a CHN-bump (shm_as_p->acl_chn increments whenever the broker reloads):

// access_control_check_right — src/broker/broker_acl.c
if (access_info_changed != shm_as_p->acl_chn) {
  uw_sem_wait (&shm_as_p->acl_sem);
  memcpy (access_info, shm_as_p->access_info, sizeof (access_info));
  num_access_info     = shm_as_p->num_access_info;
  access_info_changed = shm_as_p->acl_chn;
  uw_sem_post (&shm_as_p->acl_sem);
}
return access_control_check_right_internal (shm_as_p, dbname,
                                            dbuser, address);

The ACL check itself is run twice per connection: once cheaply at the broker layer in receiver_thr_f against v3_acl (a coarse per-broker IP filter loaded by uw_acl_make from shm_br->br_info[br_index].acl_file), and once finely at the CAS layer inside access_control_check_right against the (dbname, dbuser, ip) triple after the driver has supplied DB credentials. The two-stage check means a hostile client whose IP is on the broker-wide blacklist is rejected before any CAS is woken up; a client whose IP is allowed at the broker layer but not for a specific (dbname, dbuser) gets a clean error after auth.

The internal IP match supports CIDR-like wildcards encoded as a prefix length in ip_info->address_list[i*5] (the leading byte is the number of significant octets, 0..4); 0 means “match any”.

SQL log harvesting

Section titled “SQL log harvesting”

Every CAS writes its own SQL log to log/broker/sql_log/<broker>_<as>.sql.log (governed by sql_log_max_size and rotated by the CAS itself in cas_log.c). These files are large and per-process; a deployment typically wants three derived views:

  • broker_log_top (broker_log_top.c) — slow-query report. It scans one or more SQL log files, groups by SQL signature, and emits top queries by elapsed time, count, or fetched rows. Multi-threaded via thr_main.
  • broker_log_replay (broker_log_replay.c) — re-runs the recorded SQL against a target database; useful for capacity testing and for replaying a production trace into a staging cluster.
  • broker_log_converter (broker_log_converter.c) — reformats the SQL log into other shapes (CSV, time-merged streams).

These tools open the log files directly — they do not go through the broker. The broker’s job is to make the logs exist; analysis is offline.

Monitoring

Section titled “Monitoring”

broker_monitor.c is the user-visible broker_monitor admin tool (cubrid broker status). It opens the broker’s shm in SHM_MODE_MONITOR (kernel SHM_RDONLY) and renders the per-CAS state table to a curses-style screen. Because shm is RO, the monitor can never accidentally corrupt broker state; it pays for this by potentially seeing torn reads of multi-byte fields under contention, which is acceptable for a UI.

Per-broker rolled-up counters (num_requests_received, num_transactions_processed, num_queries_processed, num_long_queries, …) are written by each CAS into its T_APPL_SERVER_INFO slot and the monitor sums them across as_info[].

broker_admin_pub.c is the write side: it implements the operator commands (start, stop, on, off, restart, reset, acl, reload, add, drop) by manipulating the same shm and signalling the broker. Most operations boil down to flipping service_flag on individual as_info[] slots and letting the broker’s monitor threads observe the change.

Hang detection

Section titled “Hang detection”

The broker’s hang_check_thr_f runs only when monitor_hang_flag is set. Its model is conservative: it samples each CAS’s claimed_alive_time (a heartbeat the CAS bumps when it enters a long server call) and last_access_time; if too many CASes have not advanced their claim within monitor_hang_interval, the broker sets reject_client_flag, which causes receiver_thr_f to refuse new TCP accept calls (short-circuit CLOSE_SOCKET (clt_sock_fd); continue; in the accept loop). The signal is intentionally coarse — the broker prefers shedding new load over killing arbitrary live CASes. Operator intervention is expected after a hang event.

SHARD / proxy variant

Section titled “SHARD / proxy variant”

When br_info_p->shard_flag == ON, an extra layer slides in between broker and CAS: a small fixed pool of cub_proxy processes, configured in T_SHM_PROXY. The dispatch path becomes:

flowchart LR
  CL["JDBC driver"] --> BR["cub_broker"]
  BR -- "send_fd" --> PX1["cub_proxy[0]"]
  BR -. "send_fd" .-> PX2["cub_proxy[1..M-1]"]
  PX1 --> CASGRP["cub_cas[*] for shard 0"]
  PX1 --> CASGRP2["cub_cas[*] for shard 1"]

The broker does not know about shards directly; it picks any proxy via broker_find_available_proxy and sends the client fd to it. The proxy parses the SQL hint (/*+ SHARD_KEY('foo') */), computes a shard id via the configured library (the shard_key_function_name in T_SHM_PROXY), and routes to the right CAS pool. CASes in shard mode are bound to a specific shard_id at fork time (as_info->shard_cas_id, as_info->proxy_id), and they connect back to the proxy (net_connect_proxy in cas_network.c) rather than waiting for a broker handoff:

// shard_cas_main — src/broker/cas.c
proxy_sock_fd = net_connect_proxy ();
if (IS_INVALID_SOCKET (proxy_sock_fd)) {
  SLEEP_SEC (1); goto conn_proxy_retry;
}
error = cas_register_to_proxy (proxy_sock_fd);

This inversion of control (CAS connects to proxy) lets the proxy maintain a long-lived, ordered list of available CASes per shard and dispatch incoming statements to them by hint without an extra fd-passing roundtrip. The handoff cost is amortised across many clients sharing one CAS pool.

CGW (ODBC gateway) variant

Section titled “CGW (ODBC gateway) variant”

FOR_ODBC_GATEWAY builds a different binary that reuses the same broker skeleton but ships a different CAS:

  • cas_cgw.c — main loop, registered through cas_main_loop with a different CAS_MAIN_OPS (different db_connect, different process_request).
  • cas_cgw_execute.c — implements the same CAS_FC_* opcodes by calling ODBC instead of CUBRID’s own dbi.h.
  • cas_cgw_odbc.c — the actual ODBC wrapper layer.
  • cas_cgw_function.c — the CGW server function table (same shape as cas.c::server_fn_table[] but pointing at CGW implementations).

From the broker’s point of view a CGW CAS is interchangeable with a regular CAS — same shm slot layout, same handoff protocol, same SQL log files. The only on-disk difference is appl_server_name (CAS_CGW instead of CAS) in the broker config. This means a single CUBRID broker instance can front multiple “databases” where some are real cub_server instances and others are remote SQL Server / Oracle / MySQL endpoints reached through ODBC, all with identical client-facing CCI/JDBC semantics. That is the specific case the gateway product targets.

Failover

Section titled “Failover”

Failover is not the broker’s job — the broker is single-host. A CAS’s database_host and db_connection_file (a list of candidate hosts) are passed through to dbi’s connect path, and the driver inside the CAS iterates the candidate list when the primary is unreachable (connect_order in shm controls whether this is sequential or random; replica_only_flag and access_mode further constrain candidates by HA role). Any HA re-routing CUBRID does happens server-side in the heartbeat / master process (see cubrid-heartbeat.md); the broker’s only contribution is server_monitor_thr_f, which periodically asks each registered db_host for its state via the master and flips unusable_databases[] so future driver connects are short-circuited away from a dead replica.

Source Walkthrough

Section titled “Source Walkthrough”

Below is the reading order, grouped by subsystem. The position-hints table at the end of this section pairs each symbol with (file, line) as observed when the doc was last updated:.

Broker process (broker.c)

Section titled “Broker process (broker.c)”
  • Process bootstrap — main (parses argv, reads shm, installs signals), broker_init_shm (attach the broker’s own shm), init_env (TCP socket+bind+listen on PORT_NUMBER_ENV_STR), init_proxy_env (shard-only).
  • Thread fan-out — THREAD_BEGIN(receiver_thr_f, …) and siblings in main. Threads: receiver_thr_f, dispatch_thr_f, shard_dispatch_thr_f, psize_check_thr_f, cas_monitor_thr_f, hang_check_thr_f, proxy_monitor_thr_f, proxy_listener_thr_f, server_monitor_thr_f.
  • Job queue — T_MAX_HEAP_NODE (in broker_max_heap.h), max_heap_insert, max_heap_delete, max_heap_incr_priority (defined in broker_max_heap.c).
  • CAS lifecycle — broker_add_new_cas, broker_drop_one_cas_by_time_to_kill, find_idle_cas, find_drop_as_index, find_add_as_index, run_appl_server (the actual fork+execle), stop_appl_server, restart_appl_server.
  • Handoff — connect_srv (broker side, opens AF_UNIX to a specific CAS), send_fd (in broker_send_fd.c).
  • Health & monitoring — cas_monitor_worker, psize_check_worker, proxy_check_worker, proxy_monitor_worker, connect_to_master_for_server_monitor, get_server_state_from_master, insert_db_server_check_list.
  • Inline pre-dispatch protocol — receiver-thread PING/ST/QC/CANCEL/X1 handling.

CAS process (cas.c + cas_common_main.c)

Section titled “CAS process (cas.c + cas_common_main.c)”
  • CAS bootstrap — cas_init, cas_init_shm (attaches the same broker shm by APPL_SERVER_SHM_KEY_STR env, indexes as_info by AS_ID_ENV_STR).
  • Two main loops — cas_main (calls cas_main_loop with non-shard ops vector) and shard_cas_main (different loop, proxy-driven).
  • Common loop — cas_main_loop (in cas_common_main.c), cas_main_init, cas_accept_client, cas_parse_db_info, cas_handle_db_connection, cas_finish_session.
  • Per-request dispatch — process_request (CAS-side; reads msg header, decodes args, indexes server_fn_table[func_code-1]). Function table entries: fn_end_tran, fn_prepare, fn_execute, fn_fetch, fn_schema_info, fn_oid_get, fn_collection, fn_savepoint, fn_lob_*, fn_check_cas, fn_get_row_count, fn_get_last_insert_id, fn_set_cas_change_mode, etc. (44 total in enum t_cas_func_code).
  • DB connection — cas_db_connect, cas_post_db_connect, set_db_connection_info, clear_db_connection_info, need_database_reconnect, set_db_parameter.
  • Session-id surface — cas_make_session_for_driver, cas_set_session_id, cas_send_connect_reply_to_driver.
  • Signal handling — cas_sig_handler, query_cancel, set_hang_check_time, unset_hang_check_time, check_server_alive.
  • Network helpers — net_connect_client (recv_fd from broker), net_connect_proxy (shard CAS connects out to proxy), net_read_header_keep_con_on, net_read_int_keep_con_auto, net_read_process (shard variant).

Shared memory (broker_shm.c + broker_shm.h)

Section titled “Shared memory (broker_shm.c + broker_shm.h)”
  • Top-level structs — T_SHM_BROKER, T_SHM_APPL_SERVER, T_SHM_PROXY, T_BROKER_INFO, T_APPL_SERVER_INFO, T_PROXY_INFO, T_SHARD_INFO, T_SHARD_CONN_INFO, T_SHARD_USER, T_SHARD_KEY, T_SHARD_KEY_RANGE, T_CLIENT_INFO, T_DB_SERVER.
  • State enums and macros — enum t_con_status, T_BROKER_SERVICE_STATUS, UTS_STATUS_*, CON_STATUS_LOCK_*, CON_STATUS_LOCK, CON_STATUS_UNLOCK, MAGIC_NUMBER, SHM_APPL_SERVER/SHM_BROKER/SHM_PROXY, T_SHM_MODE.
  • Attach/detach — uw_shm_open, uw_shm_create, uw_shm_destroy, uw_shm_detach. Note uw_shm_open checks magic == MAGIC_NUMBER to reject stale segments.
  • Initialisation — broker_shm_initialize_shm_broker, broker_shm_initialize_shm_as, broker_shm_set_as_info, shard_shm_set_shard_conn_info.
  • Sync primitives — uw_sem_init, uw_sem_wait, uw_sem_post, uw_sem_destroy (POSIX wrapper; Windows version is named-semaphore based).

Configuration (broker_config.c + broker_config.h)

Section titled “Configuration (broker_config.c + broker_config.h)”
  • Top-level entry — broker_config_read (reads cubrid_broker.conf, populates T_BROKER_INFO[]), broker_config_dump, dir_repath, conf_get_value_table_n.
  • Tables — tbl_appl_server (CAS vs CAS_CGW), tbl_on_off, tbl_allow_deny, tbl_sql_log_mode, tbl_keep_connection, tbl_access_mode, tbl_connect_order, tbl_proxy_log_mode.
  • Section name — SECTION_NAME is "broker" in normal build, "gateway" when FOR_ODBC_GATEWAY.

Access control (broker_acl.c + broker_access_list.c)

Section titled “Access control (broker_acl.c + broker_access_list.c)”
  • Setup — access_control_set_shm (called once per broker init), access_control_read_config_file (parses cubrid_broker.acl into shm_appl->access_info[]).
  • Runtime check — access_control_check_right (entry point used by CAS), access_control_check_right_internal, access_control_check_ip, record_ip_access_time.
  • Format check — is_invalid_acl_entry (validates [%broker] headers and db:user:files lines).
  • IP file parse — access_control_read_ip_info (octet-by-octet with * wildcards encoded as a prefix length).
  • broker_access_list.c — coarse per-broker IP filter (uw_acl_make, uw_acl_check) used in receiver_thr_f.

Admin (broker_admin_pub.c + broker_admin.c + broker_admin_so.c)

Section titled “Admin (broker_admin_pub.c + broker_admin.c + broker_admin_so.c)”
  • Public command surface — admin_start_cmd, admin_stop_cmd, admin_on_cmd, admin_off_cmd, admin_reset_cmd, admin_info_cmd, admin_drop_cmd, admin_add_cmd, admin_acl_load_cmd, admin_acl_status_cmd, admin_acl_reload_cmd, admin_get_broker_status_from_shm, admin_get_proxy_conf, admin_get_shard_conf, etc.
  • Per-CAS lifecycle helpers — as_activate, as_inactivate, proxy_activate, proxy_inactivate, proxy_activate_internal, make_env, free_env.
  • broker_admin.c — thin cubrid utility entry that parses argv and calls the above. broker_admin_so.c is the same logic exposed as a shared object for cubridmanager.

SQL logging (broker_log_*.c)

Section titled “SQL logging (broker_log_*.c)”
  • broker_log_top.c — main, log_top_query, log_top, log_execute, read_multi_line_sql, read_bind_value, read_execute_end_msg, search_offset, organize_query_string, thr_main. Multi-threaded scanner that aggregates by SQL.
  • broker_log_top_tran.c — same shape but groups by transaction.
  • broker_log_replay.c — replay_main, replay_one_log_file (re-runs SQL).
  • broker_log_converter.c — format conversion utilities.
  • broker_log_util.c + broker_log_sql_list.c + broker_log_time.c — shared parsers for the SQL log line format used by all of the above.

Monitoring (broker_monitor.c)

Section titled “Monitoring (broker_monitor.c)”
  • Entry — main, get_args, print_usage.
  • Display — appl_monitor, brief_monitor, metadata_monitor, client_monitor, unusable_databases_monitor, print_appl_header, print_monitor_header, print_monitor_items, appl_info_display, set_monitor_items, print_value, time_format, time2str, ip2str.
  • Curses-or-not — function pointers tgoto_func_t, tgetent_func_t, tgetstr_func_t, tputs_func_t, tgetnum_func_t. The tool runs in either curses mode or plain stdout; the latter is what most CI uses.

CGW (cas_cgw*.c)

Section titled “CGW (cas_cgw*.c)”
  • cas_cgw.c — alternate CAS main; same cas_main_loop, different CAS_MAIN_OPS vector.
  • cas_cgw_execute.c — opcode handlers that call cas_cgw_function.c ODBC wrappers instead of dbi.h.
  • cas_cgw_odbc.c — direct ODBC API surface (SQLAllocHandle, SQLConnect, SQLPrepare, …).

Misc support

Section titled “Misc support”
  • broker_send_fd.c / broker_recv_fd.c — send_fd / recv_fd SCM_RIGHTS helpers (Linux/macOS); on Windows the broker uses an in-process port handoff via WSADuplicateSocket.
  • broker_proxy_conn.c — proxy connection table used by shard mode (broker_init_proxy_conn, broker_find_available_proxy, broker_delete_proxy_conn_by_fd, broker_destroy_proxy_conn).
  • broker_changer.c — cubrid broker_changer utility for changing broker config at runtime.
  • broker_tester.c — cubrid broker_tester health check CLI.
  • broker_process_size.c — getsize (RSS lookup) used to drive psize_check_thr_f and the auto-restart on oversize.
  • broker_filename.c — path conventions for SQL logs, ACL files, AF_UNIX sockets (ut_get_as_port_name).
  • broker_max_heap.c — generic max-heap used for job_queue.
  • broker_list.c — small linked-list helper used in Windows shm tracking.
  • broker_wsa_init.c — Windows WSA bootstrap.

Position hints (as of updated:)

Section titled “Position hints (as of updated:)”
SymbolFileApprox line
main (broker)src/broker/broker.c461
receiver_thr_fsrc/broker/broker.c784
dispatch_thr_fsrc/broker/broker.c1156
shard_dispatch_thr_fsrc/broker/broker.c1080
cas_monitor_thr_fsrc/broker/broker.c1912
cas_monitor_workersrc/broker/broker.c1797
broker_add_new_cassrc/broker/broker.c346
broker_drop_one_cas_by_time_to_killsrc/broker/broker.c392
find_idle_cassrc/broker/broker.c2700
find_drop_as_indexsrc/broker/broker.c2786
find_add_as_indexsrc/broker/broker.c2847
run_appl_serversrc/broker/broker.c1505
stop_appl_serversrc/broker/broker.c1619
restart_appl_serversrc/broker/broker.c1646
connect_srvsrc/broker/broker.c1726
init_envsrc/broker/broker.c1370
cleanupsrc/broker/broker.c715
shard_broker_processsrc/broker/broker.c653
main (cas)src/broker/cas.c209
cas_mainsrc/broker/cas.c262
shard_cas_mainsrc/broker/cas.c470
cas_initsrc/broker/cas.c735
cas_init_shmsrc/broker/cas.c764
process_requestsrc/broker/cas.c836
cas_register_to_proxysrc/broker/cas.c1468
net_read_processsrc/broker/cas.c1338
cas_db_connectsrc/broker/cas.c375
cas_post_db_connectsrc/broker/cas.c410
cas_make_session_for_driversrc/broker/cas.c280
cas_set_session_idsrc/broker/cas.c296
cas_send_connect_reply_to_driversrc/broker/cas.c321
cas_main_loopsrc/broker/cas_common_main.c79
cas_main_initsrc/broker/cas_common_main.c425
cas_accept_clientsrc/broker/cas_common_main.c981
cas_parse_db_infosrc/broker/cas_common_main.c1033
cas_handle_db_connectionsrc/broker/cas_common_main.c1119
cas_finish_sessionsrc/broker/cas_common_main.c1204
cas_sig_handlersrc/broker/cas_common_main.c522
query_cancelsrc/broker/cas_common_main.c781
restart_is_neededsrc/broker/cas_common_main.c813
set_hang_check_timesrc/broker/cas_common_main.c716
check_server_alivesrc/broker/cas_common_main.c741
net_read_header_keep_con_onsrc/broker/cas_common_main.c882
net_read_int_keep_con_autosrc/broker/cas_common_main.c1226
T_SHM_BROKERsrc/broker/broker_shm.h654
T_SHM_APPL_SERVERsrc/broker/broker_shm.h558
T_APPL_SERVER_INFOsrc/broker/broker_shm.h291
T_PROXY_INFOsrc/broker/broker_shm.h458
T_SHM_PROXYsrc/broker/broker_shm.h525
enum t_con_statussrc/broker/broker_shm.h187
uw_shm_open (POSIX)src/broker/broker_shm.c142
uw_shm_create (POSIX)src/broker/broker_shm.c337
broker_shm_initialize_shm_brokersrc/broker/broker_shm.c419
broker_shm_initialize_shm_assrc/broker/broker_shm.c481
access_control_set_shmsrc/broker/broker_acl.c62
access_control_read_config_filesrc/broker/broker_acl.c122
access_control_check_rightsrc/broker/broker_acl.c469
access_control_check_right_internalsrc/broker/broker_acl.c495
access_control_check_ipsrc/broker/broker_acl.c554
is_invalid_acl_entrysrc/broker/broker_acl.c326
access_control_read_ip_infosrc/broker/broker_acl.c370
log_top_querysrc/broker/broker_log_top.c347
log_topsrc/broker/broker_log_top.c471
log_executesrc/broker/broker_log_top.c717
thr_main (log_top)src/broker/broker_log_top.c448
appl_monitorsrc/broker/broker_monitor.c1316 (approx; long file)
brief_monitorsrc/broker/broker_monitor.c1395 (approx)
print_appl_headersrc/broker/broker_monitor.c768 (approx)
as_activatesrc/broker/broker_admin_pub.c1846 (approx)
as_inactivatesrc/broker/broker_admin_pub.c1995 (approx)
proxy_activatesrc/broker/broker_admin_pub.c2400 (approx)
broker_config_readsrc/broker/broker_config.c1100 (approx)

Lines marked “approx” are for files larger than my read window; the symbol name is the canonical anchor.

Cross-check Notes

Section titled “Cross-check Notes”

Broker SHM vs. server-side session table (cubrid-server-session.md)

Section titled “Broker SHM vs. server-side session table (cubrid-server-session.md)”

The broker has its own idea of “who is the client” that lives in T_APPL_SERVER_INFO (cas_clt_ip, cas_clt_port, clt_version, driver_info, cas_client_type). The server has another idea that lives in SESSION_STATE keyed by SESSION_ID (see cubrid-server-session.md). The CAS is the bridge: it carries a SESSION_ID it received from cub_server at first connect and caches it in as_info->session_id. When a driver reconnects with the same session id (KEEP_CON_AUTO + driver retry), the CAS hands that id back into xsession_check_session and keeps the prepared-statement cache. Crucially, the broker itself does not track sessions — it tracks slots; cross-CAS session migration is not supported (a session id is sticky to the CAS that opened it because the prepared-statement cache lives in CAS heap, not in shm).

Broker CSS vs. driver CAS protocol (cubrid-network-protocol.md)

Section titled “Broker CSS vs. driver CAS protocol (cubrid-network-protocol.md)”

The CSS protocol covered in cubrid-network-protocol.md is what the CAS speaks when talking to cub_server. The 30-byte SRV_CON_CLIENT_INFO header + MSG_HEADER framing the driver speaks to the broker/CAS is a different, narrower protocol (the “CAS protocol”, versions CAS_PROTO_VER_*). The broker’s receiver_thr_f is the only place that parses that driver header; everything past that is CAS-internal.

In particular, the enum net_server_request from network.h is not visible at the driver-broker boundary; only the 44 CAS_FC_* opcodes are. Drivers should never need a code change for new CSS opcodes, only for new CAS opcodes — which is why the CAS protocol moves much more slowly than the CSS one.

Broker vs. heartbeat (cubrid-heartbeat.md)

Section titled “Broker vs. heartbeat (cubrid-heartbeat.md)”

server_monitor_thr_f calls connect_to_master_for_server_monitor and get_server_state_from_master, which speak the master/heartbeat protocol described in cubrid-heartbeat.md. The broker is a read-only heartbeat consumer: it asks the master for each configured db_name@db_host, learns whether that server is REGISTERED_AND_ACTIVE / _STANDBY / DEAD, and updates its own unusable_databases[] list. Failover itself (which cub_server becomes active) is heartbeat’s decision; the broker just observes the result and avoids steering new sessions toward a known-bad host.

Drift / open

Section titled “Drift / open”
  • The broker’s T_SHM_BROKER::magic field is checked by uw_shm_open, but there is no monotonic version field, so a CAS built from a different commit than the broker can attach cleanly if the magic happens to match. In practice CUBRID ships broker and CAS together, so this is a packaging-layer guarantee, not a runtime-checked one.
  • WIN_FW is a legacy Windows in-process forking model (service_thr_f, process_cas_request, read_from_cas_client). It is gated by an old build flag and not exercised in standard Linux builds; the live path is the POSIX fd-passing variant.
  • _EDU_ build (#ifdef _EDU_) ships an EDU_KEY constant that gates one-day-trial behaviour. It is unused in the open-source build.

Open Questions

Section titled “Open Questions”
  • TLS termination location. The broker honours use_SSL per broker section; if set, IS_SSL_CLIENT(req_info.driver_info) routes to cas_init_ssl inside the CAS, and the TLS handshake happens after the broker has already done send_fd. That means the broker never sees the cleartext — good for security but it also means the broker cannot do L7 routing decisions based on the SQL itself. Is there appetite for terminating TLS at the broker and re-encrypting to the CAS, or for L7 routing in the proxy?
  • HTTP/REST gateway. None of the source under src/broker/ speaks HTTP. A client wanting to consume CUBRID over REST must today go through CCI/JDBC + an external gateway. CGW is ODBC-only.
  • Async query support. process_request is strictly request-then-response per func_code. There is no opcode for server-pushed cursor advancement or async result-set chunk delivery. The CAS protocol predates the Postgres-style SUSPENDED portal model. A possible evolution path is a new CAS_FC_* opcode for streaming, but it would require driver changes (fetch would need to be reentrant on the same request id).
  • Cross-CAS session migration. As noted in Cross-check Notes, prepared-statement caches live in CAS heap. Could the cache be moved into shm so that a driver reconnect can reattach to a different CAS holding the same session id? This would let the broker do graceful CAS recycling without the driver noticing. The cost is making the cache shm-safe, which interacts uncomfortably with XASL_NODE deserialised pointers.
  • Shard rebalancing while online. The shard layout in T_SHM_PROXY::shm_shard_conn is loaded once at proxy start and not designed for live reshape. cubrid broker shard reload exists but only re-reads metadata; moving a key range from one shard to another while traffic flows is not supported by the broker today (and would require the engine to have moved the rows first).
  • Job-queue priority semantics. max_heap_incr_priority is called on every dequeue, so old jobs eventually beat new ones, but the bump is uniform — there is no per-broker user-tunable priority class. A future rev could expose priority classes (e.g., interactive vs. batch) on the [%broker] config section.

Sources

Section titled “Sources”
  • Code (paths under /data/hgryoo/references/cubrid):
    • src/broker/broker.c
    • src/broker/cas.c
    • src/broker/cas_common_main.c + cas_common_main.h
    • src/broker/cas_common_execute.c + cas_common_function.c
    • src/broker/cas_common_vars.c
    • src/broker/cas_network.c + cas_network.h
    • src/broker/broker_shm.c + broker_shm.h
    • src/broker/broker_config.c + broker_config.h
    • src/broker/broker_acl.c + broker_acl.h
    • src/broker/broker_access_list.c + broker_access_list.h
    • src/broker/broker_admin_pub.c + broker_admin_pub.h
    • src/broker/broker_admin.c + broker_admin_so.c
    • src/broker/broker_log_top.c + broker_log_top_tran.c
    • src/broker/broker_log_replay.c + broker_log_converter.c
    • src/broker/broker_log_util.c + broker_log_sql_list.c
    • src/broker/broker_monitor.c
    • src/broker/broker_proxy_conn.c
    • src/broker/broker_send_fd.c + broker_recv_fd.c
    • src/broker/broker_max_heap.c + broker_list.c
    • src/broker/broker_process_size.c + broker_process_info.c
    • src/broker/broker_filename.c + broker_util.c
    • src/broker/cas_cgw.c + cas_cgw_execute.c + cas_cgw_function.c + cas_cgw_odbc.c
    • src/communication/network.h (CSS protocol opcode enum referenced by CAS upstream calls)
    • src/connection/connection_defs.h (CSS_CONN_ENTRY used by connect_to_master_for_server_monitor)
  • Cross-references inside this repo:
    • cubrid-network-protocol.md (CSS protocol the CAS uses to reach cub_server)
    • cubrid-server-session.md (the server-side session that as_info->session_id indexes into)
    • cubrid-heartbeat.md (the master/HA layer that server_monitor_thr_f consults)
  • Theoretical references:
    • Petrov, Database Internals, ch. 13 “Database Architectures” — connection pooling, shared vs. dedicated worker contexts.
    • Birrell & Nelson, Implementing Remote Procedure Calls, ACM TOCS 1984 — the framing/dispatch decomposition the broker→CAS→server pipeline instantiates.
    • pgbouncer documentation (transaction- vs. session-pooling semantics) — the closest comparable to CUBRID’s broker, but with byte-proxying instead of fd-handoff.
    • pgpool-II documentation (load balancing + failover atop pooling).
    • Oracle “Shared Server” (formerly MTS) documentation — intra-engine pool of dispatchers + shared servers; closest architectural cousin to broker + CAS, although CUBRID’s workers are external processes rather than threads sharing SGA.
    • Oracle Connection Manager (CMAN) documentation — outside-engine proxy with ACL/firewall, comparable to the v3_acl / access_control layer.