CUBRID Heartbeat — Code-Level Deep Dive

Where this document fits: The high-level analysis cubrid-heartbeat.md covers design intent and theoretical background. This document traces every branch and field at the code level. Each chapter is self-contained, but reading in order follows the full lifecycle of an HA node — from join and heartbeat exchange through failure detection to failover.

Contents:

Ch	Title	Status
1	Data-Structure Map	✅
2	Initialization and Memory Management	✅
3	The Job Queue and Worker Threads	✅
4	Heartbeat Exchange on the Wire	✅
5	Score Computation and Local Leader Election	✅
6	Failover and the Ping Witness Check	✅
7	Failback when a Peer Reclaims Master	✅
8	Demote on Local Resource Failure	✅
9	The Resource Side and Process Supervision	✅
10	Operator Commands Process Bridge and Shutdown Paths	✅

---

Chapter 1: Data-Structure Map

This chapter answers: what in-memory state does the heartbeat module hold, and how does every field earn its place? Pure storage cartography — the why CUBRID elects locally narrative lives in cubrid-heartbeat.md (“Local leader election”). State splits into a cluster family (rooted at hb_Cluster), a resource family (rooted at hb_Resource), a wire family (HBP_HEADER, HBP_PROC_REGISTER), and shared list/job machinery (HB_LIST, HB_JOB, HB_JOB_ENTRY, HB_JOB_ARG). Cluster/resource/job types live in src/executables/master_heartbeat.h; wire types and HB_NODE_STATE live in src/connection/heartbeat.h. Each field table fuses Role / why into one cell; layout-trivial structs are tabled without a re-quoted declaration, and every symbol is anchored in the trailing position-hint table regardless.

1.1 The four globals and how they fan out

Four extern pointers in master_heartbeat.h root all state: hb_Cluster (membership + election), hb_Resource (supervised processes), and the two timer queues cluster_Jobs / resource_Jobs.

flowchart TD
  C["hb_Cluster<br/>HB_CLUSTER"]
  C -->|nodes| N1["HB_NODE_ENTRY<br/>(intrusive list)"]
  C -->|myself / master cursors| N1
  C -->|ping_hosts| P1["HB_PING_HOST_ENTRY"]
  C -->|ui_nodes| U1["HB_UI_NODE_ENTRY"]
  R["hb_Resource<br/>HB_RESOURCE"]
  R -->|procs| PR["HB_PROC_ENTRY"]
  PR -->|conn| CONN["CSS_CONN_ENTRY"]
  CJ["cluster_Jobs<br/>HB_JOB"]
  CJ -->|jobs| JE1["HB_JOB_ENTRY<br/>(sorted by expire)"]
  JE1 -->|arg| JA["HB_JOB_ARG (union)"]
  RJ["resource_Jobs<br/>HB_JOB"]
  RJ -->|jobs| JE2["HB_JOB_ENTRY"]
  JE2 -->|arg| JA

Figure 1-1. The four globals and the lists they own. myself and master are cursors into the same nodes list, not separate allocations.

Invariant — myself and master alias into nodes. Both point at a node already linked in nodes (master is NULL when none is known), never independently allocated; freeing a node under a live cursor dereferences freed memory in later score/failover decisions.

1.2 HB_CLUSTER — the membership and election root

// hb_cluster -- src/executables/master_heartbeat.h
struct hb_cluster {
  pthread_mutex_t lock;
  SOCKET sfd;
  HB_NODE_STATE_TYPE state;
  char group_id[HB_MAX_GROUP_ID_LEN];
  char host_name[CUB_MAXHOSTNAMELEN];
  int num_nodes; HB_NODE_ENTRY *nodes;
  HB_NODE_ENTRY *myself; HB_NODE_ENTRY *master;   /* cursors into nodes */
  bool shutdown, hide_to_demote, is_isolated, is_ping_check_enabled;
  HB_PING_HOST_ENTRY *ping_hosts; int num_ping_hosts;
  int ping_timeout;                 /* TCP ping only */
  HB_UI_NODE_ENTRY *ui_nodes; int num_ui_nodes;
};

Field	Role / why
lock	struct mutex; serializes workers + reader.
sfd	UDP socket, send+receive; -1 until init.
state	my HB_NODE_STATE; failover flips it.
group_id	HA group (HB_MAX_GROUP_ID_LEN=64); foreign-group packets rejected.
host_name	my hostname; recognise self, stamp orig_host_name.
num_nodes	cached nodes length.
nodes	HB_NODE_ENTRY list head; membership (peers + self).
myself	self cursor into nodes; own score, no search.
master	master cursor into nodes or NULL; failover/failback rewrite it.
shutdown	cluster thread stop flag.
hide_to_demote	suppress demote log noise during a demote.
is_isolated	reach no peer; gates failover.
is_ping_check_enabled	run ping check; off when no ping hosts.
ping_hosts	ping-witness list head (external IPs).
num_ping_hosts	cached ping_hosts length.
ping_timeout	per-ping timeout, TCP only.
ui_nodes	unidentified-node list head; rate-limits reject logging.
num_ui_nodes	cached ui_nodes length.

Invariant — num_nodes equals the nodes length, under lock. Every insert/remove adjusts both in one critical section; drift makes the score loop skip a peer or run off the end.

1.3 HB_NODE_ENTRY — one configured peer

struct hb_node_entry (master_heartbeat.h): the §1.8 next/prev links, then host_name, priority (unsigned short), state, score (short), heartbeat_gap (short), last_recv_hbtime (struct timeval).

Field	Role / why
next / prev	intrusive links into nodes (§1.8).
host_name	peer hostname; match key for incoming orig_host_name.
priority	election weight, lower = stronger; HB_REPLICA_PRIORITY=0x7FFF = never-master.
state	peer’s last-known HB_NODE_STATE; feeds election.
score	election score; set Ch 5 from HB_NODE_SCORE_*, smallest wins.
heartbeat_gap	missed beats; down past HB_DEFAULT_MAX_HEARTBEAT_GAP=5.
last_recv_hbtime	last beat time; gap/isolation vs now, {0,0} = never heard.

1.4 HB_PING_HOST_ENTRY — an external ping witness

struct hb_ping_host_entry (master_heartbeat.h): §1.8 links, then host_name, port (int), ping_result (int).

Field	Role / why
next / prev	intrusive links into ping_hosts (§1.8).
host_name	witness IP/hostname; probe target.
port	TCP port, TCP ping only; ICMP ignores it.
ping_result	last outcome; an HB_PING_RESULT (§1.12).

1.5 HB_UI_NODE_ENTRY — a cached unidentified peer

A heartbeat that fails validation is cached here so the error log is rate-limited instead of spamming on every stray packet. struct hb_ui_node_entry (master_heartbeat.h): §1.8 links, then host_name, group_id, saddr (struct sockaddr_in), last_recv_time (struct timeval), v_result (int).

Field	Role / why
next / prev	intrusive links into ui_nodes (§1.8).
host_name	claimed hostname; “already logged?” key.
group_id	claimed group; part of the match key.
saddr	source sockaddr_in; detects HB_VALID_IP_ADDR_MISMATCH.
last_recv_time	last seen; cache window + eviction (..._CACHE_/..._CLEANUP_TIME_...).
v_result	validation verdict; an HB_VALID_RESULT (§1.13).

1.6 HB_RESOURCE — the locally-supervised-process root

struct hb_resource (master_heartbeat.h): lock (pthread_mutex_t), state (HB_NODE_STATE_TYPE, the mode), num_procs (int), procs (HB_PROC_ENTRY *), shutdown (bool).

Field	Role / why
lock	struct mutex; workers + command handlers mutate procs.
state	resource-side role; mirrors cluster verdict, failover propagates changemode.
num_procs	cached procs length; max HB_MAX_NUM_RESOURCE_PROC=16.
procs	HB_PROC_ENTRY list head; one per registered process.
shutdown	resource thread stop flag.

Invariant — HB_RESOURCE::state tracks HB_CLUSTER::state. The resource side does not elect; it mirrors the cluster verdict, and Ch 8 pushes a matching changemode to every proc on demote. Divergence leaves a node cluster-master while its server still runs as standby.

1.7 HB_PROC_ENTRY — one supervised process

The densest struct: a state machine plus restart/confirm/changemode bookkeeping.

// HB_PROC_ENTRY -- src/executables/master_heartbeat.h
struct HB_PROC_ENTRY {
  HB_PROC_ENTRY *next; HB_PROC_ENTRY **prev;
  unsigned char state;              /* HB_PROC_STATE */
  unsigned char type;               /* HB_PROC_TYPE */
  int sfd; int pid;
  char exec_path[HB_MAX_SZ_PROC_EXEC_PATH];
  char args[HB_MAX_SZ_PROC_ARGS];
  struct timeval frtime, rtime, dtime, ktime, stime;  /* first-reg/reg/dereg/kill/start */
  unsigned short changemode_rid; unsigned short changemode_gap;
  LOG_LSA prev_eof; LOG_LSA curr_eof;
  bool is_curr_eof_received;
  CSS_CONN_ENTRY *conn;
  bool being_shutdown; bool server_hang;
};

Field	Role / why
next / prev	intrusive links into procs (§1.8).
state	HB_PROC_STATE (§1.11); unsigned char matches broker.c SERVER_STATE.
type	HB_PROC_TYPE (§1.12); selects supervision policy.
sfd	socket fd; key for hb_return_proc_state_by_fd.
pid	OS pid; TERM/KILL, start/death confirm.
exec_path	executable path (128); re-spawn.
args	argv (HB_MAX_SZ_PROC_ARGS=1024); re-spawn + dereg-by-args key.
frtime	first-registered time; restart-too-fast anchor.
rtime	most-recent registered time.
dtime	deregistered time; {0,0} = still registered.
ktime	kill/shutdown time; confirm-dereg anchor.
stime	start time; start-confirm retries.
changemode_rid	in-flight changemode request id; correlates reply.
changemode_gap	changemode failures; escalate TERM (..._TO_TERM=12), KILL (=24).
prev_eof	prior-poll end-of-log LSA; hang detection.
curr_eof	latest end-of-log LSA (a LOG_LSA, §1.7.1).
is_curr_eof_received	curr_eof reply arrived; guards hang check.
conn	the CSS_CONN_ENTRY; send changemode/get-eof.
being_shutdown	shutdown in progress; suppresses restart.
server_hang	hung flag; set when EOF stalls, read by hb_is_hang_process.

1.7.1 LOG_LSA recap

prev_eof/curr_eof are LOG_LSA (struct log_lsa, log_lsa.hpp): packed pageid:48+offset:16, treated as opaque — advancing means alive, frozen arms server_hang. Full LSA semantics belong to recovery/log.

1.8 The intrusive-list idiom — HB_LIST

HB_NODE_ENTRY, HB_PING_HOST_ENTRY, HB_UI_NODE_ENTRY, HB_PROC_ENTRY, and HB_JOB_ENTRY embed the same next/prev pair, generically HB_LIST:

// hb_list -- src/executables/master_heartbeat.h
struct hb_list {
  HB_LIST *next;
  HB_LIST **prev;     /* address of the predecessor's `next` slot */
};

prev is pointer-to-pointer holding the address of the slot pointing at this node (&head for the first element, else &predecessor->next) — the Linux hlist idiom, so unlink is *prev = next; if (next) next->prev = prev; with no head special case.

Invariant — *(entry->prev) == entry for every linked node. If broken, an unlink writes through a dangling prev and corrupts an unrelated next.

1.9 The timer-job machinery — HB_JOB, HB_JOB_ENTRY, HB_JOB_ARG

Both sides schedule future work through one deadline-ordered queue (worker execution is Ch 3). HB_JOB_FUNC is typedef void (*)(HB_JOB_ARG *).

struct hb_job (the queue): lock (pthread_mutex_t), num_jobs (unsigned short), jobs (HB_JOB_ENTRY *), job_funcs (HB_JOB_FUNC *), shutdown (bool):

HB_JOB field	Role / why
lock	queue mutex; worker pops while schedulers push.
num_jobs	cached count.
jobs	HB_JOB_ENTRY head, sorted by expire; soonest at head.
job_funcs	dispatch table, job type -> handler; cluster/resource differ.
shutdown	stop the worker.

struct hb_job_entry (one job): §1.8 next/prev, then:

HB_JOB_ENTRY field	Role / why
type (unsigned int)	an HB_CLUSTER_JOB/HB_RESOURCE_JOB (§1.14); selects handler.
expire (timeval)	absolute deadline; worker sleeps until head’s expire.
func (HB_JOB_FUNC)	the handler; entry is self-contained.
arg (HB_JOB_ARG *)	heap arg the handler owns and frees.

Invariant — jobs is sorted ascending by expire. The worker inspects only the head; a misordered insert fires failover confirmations late.

// hb_cluster_job_arg / hb_resource_job_arg / hb_job_arg -- master_heartbeat.h
struct hb_cluster_job_arg { unsigned int ping_check_count; unsigned int retries; };
struct hb_resource_job_arg {
  int pid; int sfd;
  char args[HB_MAX_SZ_PROC_ARGS];
  unsigned int retries; unsigned int max_retries;
  struct timeval ftime, ltime;      /* first / last job execution time */
};
union hb_job_arg {
  HB_CLUSTER_JOB_ARG cluster_job_arg;
  HB_RESOURCE_JOB_ARG resource_job_arg;
};

HB_JOB_ARG is a union: a job is cluster or resource, never both, so the two shapes overlay one allocation; the active arm is implied by which queue holds the job (and HB_JOB_ENTRY::type).

HB_CLUSTER_JOB_ARG	Role / why		HB_RESOURCE_JOB_ARG	Role / why
ping_check_count	ping rounds run vs HB_MAX_PING_CHECK=3		pid / sfd	target process / connection fd
retries	generic retry counter		args	re-spawn argv
			retries / max_retries	retries so far / ceiling
			ftime / ltime	first run (backoff) / last run

1.10 The wire family — HBP_HEADER and HBP_PROC_REGISTER

These are serialized onto the wire, so byte layout matters.

// hbp_header -- src/connection/heartbeat.h
struct hbp_header {
  unsigned char type;               /* HBP_CLUSTER_MESSAGE */
#if defined(HPUX) || defined(_AIX) || defined(sparc)
  char r:1;                         /* is request? -- MSB-first hosts */
  char reserved:7;
#else
  char reserved:7;                  /* LSB-first hosts (x86): order flipped */
  char r:1;
#endif
  unsigned short len; unsigned int seq;
  char group_id[HB_MAX_GROUP_ID_LEN];
  char orig_host_name[CUB_MAXHOSTNAMELEN];
  char dest_host_name[CUB_MAXHOSTNAMELEN];
};

Field	Role / why
type	message kind; an HBP_CLUSTER_MESSAGE (§1.13).
r:1	request-vs-reply bit; shares a byte with reserved.
reserved:7	flag-byte padding; future flags.
len	body length after the header.
seq	sequence number; detect duplicate/old beats.
group_id	sender HA group; validated vs HB_CLUSTER::group_id.
orig_host_name	sender hostname; match key into nodes.
dest_host_name	recipient; ignore beats not addressed here.

Invariant — the r:1/reserved:7 bit order is endian-conditional. The HPUX || _AIX || sparc arm puts r first; every other (little-endian) host puts reserved first. Both encode the same wire byte; dropping the #if makes big- and little-endian nodes read the request bit from opposite ends.

// hbp_proc_register -- src/connection/heartbeat.h
struct hbp_proc_register {
  int pid; int type;                /* type = HB_PROC_TYPE */
  char exec_path[HB_MAX_SZ_PROC_EXEC_PATH];
  char args[HB_MAX_SZ_PROC_ARGS];
};

Field	Role / why
pid	registering pid; becomes HB_PROC_ENTRY::pid.
type	process kind (HB_PROC_TYPE); becomes HB_PROC_ENTRY::type.
exec_path	path; copied for re-spawn.
args	argv; copied, dereg-by-args key.

HBP_PROC_REGISTER is the body a child sends to register; the master copies it into a fresh HB_PROC_ENTRY (Ch 9). It travels over the resource connection (CSS_CONN_ENTRY), not the UDP socket that carries HBP_HEADER.

1.11 Node and process state enums

HB_NODE_STATE (typedef HB_NODE_STATE_TYPE) is shared by HB_CLUSTER::state, HB_NODE_ENTRY::state, and HB_RESOURCE::state:

Value	Meaning		Value	Meaning
HB_NSTATE_UNKNOWN (0)	undetermined		HB_NSTATE_TO_BE_SLAVE (3)	demotion in flight
HB_NSTATE_SLAVE (1)	standby		HB_NSTATE_MASTER (4)	active master
HB_NSTATE_TO_BE_MASTER (2)	promotion in flight		HB_NSTATE_REPLICA (5)	never-master replica
			HB_NSTATE_MAX (6)	sentinel

HB_PROC_STATE is the per-process machine in HB_PROC_ENTRY::state:

Value	Meaning		Value	Meaning
HB_PSTATE_UNKNOWN (0)	uninitialized		HB_PSTATE_REGISTERED (5)	registered
HB_PSTATE_DEAD (1)	not running		HB_PSTATE_REGISTERED_AND_STANDBY	alias = 5
HB_PSTATE_DEREGISTERED (2)	cleanly deregistered		HB_PSTATE_REGISTERED_AND_TO_BE_STANDBY (6)	demotion pending
HB_PSTATE_STARTED (3)	spawned, awaiting reg		HB_PSTATE_REGISTERED_AND_ACTIVE (7)	serving as master
HB_PSTATE_NOT_REGISTERED (4)	running, not yet reg		HB_PSTATE_REGISTERED_AND_TO_BE_ACTIVE (8)	promotion pending
			HB_PSTATE_MAX (9)	sentinel

Watch the REGISTERED == REGISTERED_AND_STANDBY alias. A freshly registered process is a standby, so both names share value 5: a switch cannot list both (duplicate case), and they are indistinguishable by value. The header notes the numbering mirrors broker.c’s SERVER_STATE.

1.12 Process-type and ping-result enums

HB_PROC_TYPE (hb_proc_type) and HB_PING_RESULT (note HB_PING_UNKNOWN is -1):

HB_PROC_TYPE	Meaning		HB_PING_RESULT	Meaning
HB_PTYPE_SERVER (0)	DB server		HB_PING_UNKNOWN (-1)	not yet probed
HB_PTYPE_COPYLOGDB (1)	log copier		HB_PING_SUCCESS (0)	reachable
HB_PTYPE_APPLYLOGDB (2)	log applier		HB_PING_USELESS_HOST (1)	skipped (e.g. self)
HB_PTYPE_MAX (3)	sentinel		HB_PING_SYS_ERR (2)	local probe error
			HB_PING_FAILURE (3)	unreachable

1.13 Validation-result and cluster-message enums

HB_VALID_RESULT (in HB_UI_NODE_ENTRY::v_result) and HBP_CLUSTER_MESSAGE (in HBP_HEADER::type):

HB_VALID_RESULT	Meaning		HBP_CLUSTER_MESSAGE	Meaning
HB_VALID_NO_ERROR (0)	valid		HBP_CLUSTER_HEARTBEAT (0)	only message type today
HB_VALID_UNIDENTIFIED_NODE (1)	sender not in nodes		HBP_CLUSTER_MSG_MAX (1)	sentinel
HB_VALID_GROUP_NAME_MISMATCH (2)	wrong group_id
HB_VALID_IP_ADDR_MISMATCH (3)	hostname/IP disagree
HB_VALID_CANNOT_RESOLVE_HOST (4)	DNS failure

1.14 Job-type enums and the score macros

HB_CLUSTER_JOB	Job		HB_RESOURCE_JOB	Job
HB_CJOB_INIT (0)	one-shot init		HB_RJOB_PROC_START (0)	spawn a process
HB_CJOB_HEARTBEAT (1)	send/recv a beat (Ch 4)		HB_RJOB_PROC_DEREG (1)	deregister a process
HB_CJOB_CALC_SCORE (2)	recompute, elect (Ch 5)		HB_RJOB_CONFIRM_START (2)	confirm a spawn
HB_CJOB_CHECK_PING (3)	ping check (Ch 6)		HB_RJOB_CONFIRM_DEREG (3)	confirm a dereg
HB_CJOB_FAILOVER (4)	promote (Ch 6)		HB_RJOB_CHANGE_MODE (4)	push changemode (Ch 8)
HB_CJOB_FAILBACK (5)	yield master (Ch 7)		HB_RJOB_DEMOTE_START_SHUTDOWN (5)	begin demote-shutdown
HB_CJOB_CHECK_VALID_PING_SERVER (6)	revalidate ping hosts		HB_RJOB_DEMOTE_CONFIRM_SHUTDOWN (6)	confirm demote-shutdown
HB_CJOB_DEMOTE (7)	demote (Ch 8)		HB_RJOB_CLEANUP_ALL (7)	tear down (Ch 10)
HB_CJOB_MAX (8)	sentinel		HB_RJOB_CONFIRM_CLEANUP_ALL (8)	confirm teardown
			HB_RJOB_MAX (9)	sentinel

The score macros written into HB_NODE_ENTRY::score during election (Ch 5):

// score macros -- src/executables/master_heartbeat.h
#define HB_REPLICA_PRIORITY         0x7FFF
#define HB_NODE_SCORE_MASTER        0x8000   /* read as a NEGATIVE short */
#define HB_NODE_SCORE_TO_BE_MASTER  0xF000
#define HB_NODE_SCORE_SLAVE         0x0000
#define HB_NODE_SCORE_UNKNOWN       0x7FFF

0x8000 deliberately reads negative. HB_NODE_ENTRY::score is a signed short: 0x8000 = -32768 and 0xF000 = -4096 (both negative), while SLAVE (0) and UNKNOWN (0x7FFF = +32767) are non-negative. Election picks the smallest score, so master/to-be-master sort ahead of every slave with no special case. As unsigned short, 0x8000 would read +32768 and the master would lose every election. Ch 5 depends on this sign trick.

1.15 Chapter summary — key takeaways

1. Four globals root everything: hb_Cluster, hb_Resource, cluster_Jobs, resource_Jobs; every other struct hangs off one of these.
2. myself/master are cursors into nodes, not allocations (master may be NULL).
3. Five structs share one intrusive-list idiom with a pointer-to-pointer prev (*(entry->prev) == entry), so unlinking needs no head special case.
4. HB_PROC_ENTRY is a state machine plus restart/confirm/changemode bookkeeping — five timevals, a changemode pair, two LOG_LSA EOF snapshots.
5. HB_JOB_ARG is a queue-discriminated union; jobs is kept sorted by expire.
6. HBP_HEADER’s request bit is endian-conditional via #if, so both host classes agree on the wire byte.
7. Two numeric traps: HB_PSTATE_REGISTERED == HB_PSTATE_REGISTERED_AND_STANDBY (alias 5), and HB_NODE_SCORE_MASTER=0x8000 read as a negative signed short so masters sort first in “smallest wins”.

Chapter 2: Initialization and Memory Management

How do the three heartbeat globals (hb_Cluster, hb_Resource, and the two HB_JOB queues) get allocated and populated before a single heartbeat packet crosses the wire, and in what order? Struct layouts are in Chapter 1; here we trace the bring-up code — every branch and error path — stopping the instant hb_thread_initialize is reached (thread spawning is Chapter 3). For the why of nodes, priorities, replicas, and ping witnesses, see the companion cubrid-heartbeat.md (“Cluster membership”, “Split-brain and the ping witness”); we do not re-derive that theory.

2.1 The driver: hb_master_init

hb_master_init stands up heartbeat as a strict five-step pipeline guarded by a shared goto error_return block.

// hb_master_init -- src/executables/master_heartbeat.c
hb_enable_er_log ();
sysprm_reload_and_init (NULL, NULL);                /* <- re-read cubrid.conf so HA params are fresh */
error = hb_cluster_initialize (prm_get_string_value (PRM_ID_HA_NODE_LIST),
                               prm_get_string_value (PRM_ID_HA_REPLICA_LIST));
if (error != NO_ERROR) goto error_return;           /* <- step 1 */
error = hb_cluster_job_initialize ();   if (...) goto error_return;   /* <- step 2 */
error = hb_resource_initialize ();      if (...) goto error_return;   /* <- step 3 */
error = hb_resource_job_initialize ();  if (...) goto error_return;   /* <- step 4 */
error = hb_thread_initialize ();        if (...) goto error_return;   /* <- step 5: Chapter 3 */
hb_Deactivate_immediately = false;                  /* <- arm: deactivation no longer a no-op */
return NO_ERROR;

Ordering is load-bearing: each global must exist before its job queue refers to it. Any non-NO_ERROR return jumps to cleanup.

Invariant — partial-init cleanup is keyed on shutdown == false. The cleanup block tears down exactly the globals that finished:

// hb_master_init (error_return) -- src/executables/master_heartbeat.c
error_return:
  if (hb_Cluster && hb_Cluster->shutdown == false)   hb_cluster_cleanup ();
  if (cluster_Jobs && cluster_Jobs->shutdown == false) hb_cluster_job_shutdown ();
  if (hb_Resource && hb_Resource->shutdown == false)  hb_resource_cleanup ();
  if (resource_Jobs && resource_Jobs->shutdown == false) hb_resource_job_shutdown ();
  return error;

Each initializer sets shutdown = false as its last act before success, so the guard is true only for finished globals. If violated a step-3 failure would tear down a queue whose worker never spawned, or double-free a freed global.

flowchart TD
  A["hb_master_init"] --> C["hb_cluster_initialize"]
  C -->|error| Z["error_return:\nshutdown==false cleanup"]
  C -->|NO_ERROR| D["hb_cluster_job_initialize"]
  D -->|error| Z
  D -->|NO_ERROR| E["hb_resource_initialize"]
  E -->|error| Z
  E -->|NO_ERROR| F["hb_resource_job_initialize"]
  F -->|error| Z
  F -->|NO_ERROR| G["hb_thread_initialize\n=> Chapter 3"]
  G -->|error| Z
  G -->|NO_ERROR| H["hb_Deactivate_immediately = false"]

_{Figure 2-1. The five-step pipeline and its shared cleanup funnel.}

2.2 Step 1 — hb_cluster_initialize: the hb_Cluster global

The heaviest initializer — allocate hb_Cluster, learn the hostname, pick the state, build node and ping-host lists, open the UDP socket.

// hb_cluster_initialize -- src/executables/master_heartbeat.c
if (hb_Cluster == NULL) {
    hb_Cluster = (HB_CLUSTER *) malloc (sizeof (HB_CLUSTER));
    if (hb_Cluster == NULL) { ...; return ER_OUT_OF_VIRTUAL_MEMORY; }  /* <- nothing to unlock yet */
    pthread_mutex_init (&hb_Cluster->lock, NULL);  /* <- mutex init only on first alloc */
}
if (GETHOSTNAME (host_name, ...)) { ...; return ER_BO_UNABLE_TO_FIND_HOSTNAME; } /* <- pre-lock early return */
/* ... pthread_mutex_lock; scalars default; host_name copied ... */
if (HA_GET_MODE () == HA_MODE_REPLICA) hb_Cluster->state = HB_NSTATE_REPLICA; /* <- never elects */
else                                   hb_Cluster->state = HB_NSTATE_SLAVE;   /* <- everyone boots SLAVE */

The malloc is idempotent (guarded by == NULL), so a reactivation reuses the existing struct and its mutex. Both early returns (malloc-fail, GETHOSTNAME) happen before pthread_mutex_lock so neither leaks a held lock; from here every error path unlocks. Under the lock the scalars default — shutdown, hide_to_demote, is_isolated to false; is_ping_check_enabled to true; sfd to INVALID_SOCKET; master/myself/nodes to NULL — then the state pick above.

Invariant — a freshly-initialized node is never MASTER. It boots SLAVE (or REPLICA) with master/myself NULL; mastership is only assumed later via the election job (Chapter 5). If violated two nodes booting into MASTER split-brain before the first heartbeat round.

hb_Cluster->num_nodes = hb_cluster_load_group_and_node_list (...) follows (§2.6); if (hb_Cluster->num_nodes < 1) is fatal — unlock and return ER_PRM_BAD_VALUE. The ping-host load then tries ICMP first, falling back to TCP only on zero ICMP hosts:

// hb_cluster_initialize -- src/executables/master_heartbeat.c
hb_Cluster->num_ping_hosts = hb_cluster_load_ping_host_list (prm_get_string_value (PRM_ID_HA_PING_HOSTS));
if (hb_cluster_check_valid_ping_server () == false) { pthread_mutex_unlock (...); return ER_FAILED; }
if (hb_Cluster->num_ping_hosts == 0) {         /* <- no ICMP hosts -> try TCP */
    hb_Cluster->num_ping_hosts = hb_cluster_load_tcp_ping_host_list (prm_get_string_value (PRM_ID_HA_TCP_PING_HOSTS));
    hb_Cluster->ping_timeout   = prm_get_integer_value (PRM_ID_HA_PING_TIMEOUT) * 1000; /* secs -> msecs */
    if (hb_cluster_check_valid_ping_server () == false) { pthread_mutex_unlock (...); return ER_FAILED; }
}
/* ... sfd = socket(AF_INET, SOCK_DGRAM, 0); on <0 unlock, return ERR_CSS_TCP_DATAGRAM_SOCKET ... */
/* ... bind to INADDR_ANY:ha_port_id; on <0 unlock, return ERR_CSS_TCP_DATAGRAM_BIND ... */
pthread_mutex_unlock (&hb_Cluster->lock);  return NO_ERROR; /* <- success: single tail unlock */

hb_cluster_check_valid_ping_server returns true when no ping hosts are configured (a witness-less setup is valid); it fails only when hosts are configured but none respond (Chapter 6). The socket open has two failure branches (socket() < 0, bind() < 0), both unlocking, then one tail unlock on success. The wire protocol over this socket is Chapter 4.

2.3 Step 2 — hb_cluster_job_initialize

Allocate (idempotently, same == NULL malloc + pthread_mutex_init idiom as §2.2), reset the queue head under the lock, point job_funcs at the static dispatch array, queue the first job:

// hb_cluster_job_initialize -- src/executables/master_heartbeat.c
/* ... idempotent malloc cluster_Jobs + pthread_mutex_init on first alloc; lock ... */
cluster_Jobs->shutdown = false;  cluster_Jobs->num_jobs = 0;  cluster_Jobs->jobs = NULL;
cluster_Jobs->job_funcs = &hb_cluster_jobs[0];   /* <- index -> function dispatch table; unlock */
error = hb_cluster_job_queue (HB_CJOB_INIT, NULL, HB_JOB_TIMER_IMMEDIATELY);
if (error != NO_ERROR) { assert (false); return ER_FAILED; }   /* <- queue failure is a logic bug */
return NO_ERROR;

job_funcs is bound to a file-scope dispatch table mapping a job-type enum to a handler — hb_cluster_jobs[] has hb_cluster_job_init at index 0 (HB_CJOB_INIT), then _heartbeat, _calc_score, _check_ping, _failover, _failback, _check_valid_ping_server, _demote, and a NULL sentinel. The seed HB_CJOB_INIT (enum 0) is queued HB_JOB_TIMER_IMMEDIATELY but sits dormant until a worker exists (Chapter 3).

2.4 Steps 3 & 4 — the resource side

hb_resource_initialize follows the §2.3 malloc-and-reset pattern exactly, booting hb_Resource->state = HB_NSTATE_SLAVE with procs = NULL and num_procs = 0 (no supervised processes yet). hb_resource_job_initialize is the §2.3 queue initializer with &hb_resource_jobs[0] bound instead, and one load-bearing difference in the seed job:

// hb_resource_job_initialize -- src/executables/master_heartbeat.c
error = hb_resource_job_queue (HB_RJOB_CHANGE_MODE, NULL,
                               prm_get_integer_value (PRM_ID_HA_INIT_TIMER_IN_MSECS) +
                               prm_get_integer_value (PRM_ID_HA_FAILOVER_WAIT_TIME_IN_MSECS));
if (error != NO_ERROR) { assert (false); return ER_FAILED; }

The seed is HB_RJOB_CHANGE_MODE (enum 4, indexing hb_resource_jobs[4]) and the timer is not immediate — it fires ha_init_timer_in_msecs + ha_failover_wait_time_in_msecs later, letting the cluster settle a leader before the resource side commits the local process tree to a role.

2.5 The intrusive list primitives and per-entry allocators

Every dynamic collection (nodes, ping hosts, supervised processes) is the same intrusive doubly-linked list. Each entry embeds an HB_LIST header first, with prev a pointer-to-pointer so insertion and unlink are head-agnostic:

// struct hb_list -- src/executables/master_heartbeat.h
struct hb_list { HB_LIST *next; HB_LIST **prev; };   /* prev points at the slot that points to us */

All three primitives are branch-complete below: hb_list_add pushes at the head (its if (n->next) guard back-points the old head); hb_list_remove splices via if (n->prev) / inner if (*(n->prev)); hb_list_move re-homes a whole head, with if (*dest_pp) skipped when the source was empty.

// hb_list_add -- src/executables/master_heartbeat.c
n->next = *(p);
if (n->next) n->next->prev = &(n->next);   /* <- old head back-points into new node */
n->prev = p;  *(p) = n;                    /* <- new node back-points at head slot */

// hb_list_remove -- src/executables/master_heartbeat.c
if (n->prev) { *(n->prev) = n->next;       /* <- slot that pointed to n now points past it */
    if (*(n->prev)) n->next->prev = n->prev; }
n->next = NULL;  n->prev = NULL;           /* <- detached node has no dangling links */

// hb_list_move -- src/executables/master_heartbeat.c
*dest_pp = *source_pp;
if (*dest_pp) (*dest_pp)->prev = dest_pp;  /* <- re-home back-ptr; skipped if source empty */
*source_pp = NULL;

Invariant — prev always points at the pointer that points to the entry. For the first entry that slot is the list head (&hb_Cluster->nodes); for an interior entry it is &(predecessor->next). This lets hb_list_remove splice without special-casing the head. If violated removing the first node leaves hb_Cluster->nodes pointing at freed memory.

The per-entry allocators malloc, default-fill, and hb_list_add onto the right head. hb_alloc_new_proc is the proc-list one:

// hb_alloc_new_proc -- src/executables/master_heartbeat.c
p = (HB_PROC_ENTRY *) malloc (sizeof (HB_PROC_ENTRY));
if (p) {
    memset (p, 0, sizeof (HB_PROC_ENTRY));     /* <- zero first, then override non-zero defaults */
    p->state = HB_PSTATE_UNKNOWN;
    LSA_SET_NULL (&p->prev_eof);  LSA_SET_NULL (&p->curr_eof);  /* <- log addresses start null */
    hb_list_add ((HB_LIST **) &hb_Resource->procs, (HB_LIST *) p);
}
return (p);

hb_alloc_new_proc is not called during bring-up — the proc list starts empty (§2.4); entries appear only when a managed process registers (Chapter 9). It is shown here because it shares the allocator idiom.

flowchart LR
  subgraph Cluster["hb_Cluster"]
    N0["nodes (head slot)"] --> N1["HB_NODE_ENTRY"] --> N2["HB_NODE_ENTRY"]
    P0["ping_hosts (head slot)"] --> P1["HB_PING_HOST_ENTRY"]
  end
  R0["hb_Resource->procs (head slot)"] --> R1["HB_PROC_ENTRY"]

_{Figure 2-2. All three collections are the same intrusive list rooted in a global; head insertion via hb_list_add.}

2.6 Building the node table: hb_cluster_load_group_and_node_list and hb_add_node_to_cluster

ha_node_list has the form group_id@host1:host2:host3. The loader tokenises on @ first to peel off the group id, then on ,: for hosts, using one priority counter as both token index and assigned priority.

// hb_cluster_load_group_and_node_list -- src/executables/master_heartbeat.c
if (ha_node_list == NULL || ha_node_list[0] == '\0') { ...; goto error; }  /* <- empty list fatal */
for (priority = 0, p = strtok_r (tmp_string, "@", &savep); p;
     priority++, p = strtok_r (NULL, " ,:", &savep)) {
    if (priority == 0) strncpy_bufsize (hb_Cluster->group_id, p);  /* <- token 0 is group id */
    else {
        node = hb_add_node_to_cluster (p, (priority));   /* <- priority = 1, 2, 3, ... by order */
        if (node && are_hostnames_equal (node->host_name, hb_Cluster->host_name)) {
            if (hb_Cluster->state == HB_NSTATE_REPLICA) { ...; goto error; } /* <- replica must NOT be here */
            else hb_Cluster->myself = node;              /* <- found myself */
        }
    }
}
if (hb_Cluster->state != HB_NSTATE_REPLICA && hb_Cluster->myself == NULL) { ...; goto error; }
num_nodes = priority;

This is why two nodes can never share a priority. Priority is assigned strictly by config order (first host 1, second 2, third 3); the counter is monotonic, so config-position-to-priority is a bijection. The election (Chapter 5) relies on this: ties are impossible, so the lowest-priority survivor is always unique.

hb_add_node_to_cluster guards a NULL token, mallocs the entry, applies the localhost rewrite, and stamps defaults:

// hb_add_node_to_cluster -- src/executables/master_heartbeat.c
if (host_name == NULL) return NULL;            /* <- NULL token yields NULL entry */
p = (HB_NODE_ENTRY *) malloc (sizeof (HB_NODE_ENTRY));
if (p) {
    /* localhost -> hb_Cluster->host_name via are_hostnames_equal, else strncpy host_name as-is */
    p->priority = priority;  p->state = HB_NSTATE_UNKNOWN; /* <- peers UNKNOWN until first heartbeat */
    /* ... score, heartbeat_gap, last_recv_hbtime, next/prev each set to 0 individually ... */
    hb_list_add ((HB_LIST **) &hb_Cluster->nodes, (HB_LIST *) p);
}
return (p);

A NULL host_name returns NULL; the caller’s if (node) skips it (likewise if malloc fails). Unlike hb_alloc_new_proc (§2.5), this allocator does not memset — it assigns each field individually.

Before the replica loop, ha_replica_list is copied into tmp_string (or tmp_string is emptied when the param is NULL); if the node is in REPLICA state but the replica list is empty, that is fatal — if (hb_Cluster->state == HB_NSTATE_REPLICA && tmp_string[0] == '\0') goto error. The replica pass then runs over ha_replica_list, identical in shape to the node pass but with two extra checks: its priority == 0 token must equal hb_Cluster->group_id — if (strcmp (hb_Cluster->group_id, p) != 0) goto error — and each host is added with hb_add_node_to_cluster (p, HB_REPLICA_PRIORITY). The sentinel HB_REPLICA_PRIORITY (0x7FFF) sits far above any ordinary node, so a replica never wins election. The role-validation matrix (the empty-list guard plus the myself-placement goto error branches):

hb_Cluster->state	Replica-list empty	In node list	In replica list
SLAVE / would-be MASTER	allowed	must contain myself; else goto error	must not; else goto error
REPLICA	fatal goto error	must not; else goto error	must contain myself; else goto error

On success the function returns num_nodes + priority (node count plus replica count). Any goto error sets ER_PRM_BAD_VALUE and returns ER_FAILED (negative), caught by §2.2’s num_nodes < 1 guard.

Hard cap. The header defines HB_MAX_NUM_NODES = 8. The loop does not enforce it (the limit is checked when nodes are matched against incoming heartbeats), but fixed-size buffers downstream assume at most eight nodes — any extension must respect this ceiling.

2.7 Building the ping-host list: ICMP and TCP loaders

hb_cluster_load_ping_host_list parses ha_ping_hosts (space/comma/colon delimited) and appends each via hb_add_ping_host. Its branches: if (ha_ping_host_list == NULL) return 0 (unset param is valid — zero hosts); per token, if (host_p == NULL) break ends the loop; if (strcmp (host_p, "0.0.0.0") == 0) logs and skips the placeholder (not counted); else hb_add_ping_host (host_p); num_hosts++.

hb_add_ping_host has the same two branches as hb_add_node_to_cluster (§2.6) — a if (host_name == NULL) return NULL guard, then an if (p) malloc-success body — differing only in the defaults stored: p->port = -1 (ICMP has no port) and p->ping_result = HB_PING_UNKNOWN, then hb_list_add onto hb_Cluster->ping_hosts.

hb_cluster_load_tcp_ping_host_list is reached only when zero ICMP hosts were configured (§2.2). It tokenises on ,, splits each token at : into host and port (validated via hb_port_str_to_num), and applies the same 0.0.0.0 exclusion. Its distinguishing branches are the malformed-input breaks:

// hb_cluster_load_tcp_ping_host_list -- src/executables/master_heartbeat.c
port_p = strstr (host_p, ":");
if (port_p == NULL || port_p == host_p) break;     /* <- no ':' or empty host -> stop parsing */
else { *port_p = '\0'; port_p++; port = hb_port_str_to_num (port_p);
       if (port == -1) break; }                    /* <- bad port -> stop parsing */

A malformed host:port breaks out of parsing entirely, so anything after it is silently dropped. hb_add_tcp_ping_host has the same two branches as hb_add_ping_host, differing in exactly one line: it stores p->port = port instead of -1. Both push onto the same hb_Cluster->ping_hosts head, but ICMP and TCP entries are never mixed since the TCP fallback runs only when ICMP gave zero.

2.8 Chapter summary — key takeaways

1. hb_master_init is a strict five-step pipeline (cluster, cluster jobs, resource, resource jobs, threads) with a shared error_return funnel that tears down only globals whose shutdown field is false.
2. hb_cluster_initialize does the heavy lifting under hb_Cluster->lock: idempotent malloc, GETHOSTNAME, the SLAVE-vs-REPLICA pick from HA_GET_MODE, node/ping-host loading, and the INADDR_ANY UDP socket on ha_port_id. Every post-lock error path unlocks; the two pre-lock early returns do not.
3. A node always boots non-MASTER (master/myself NULL); mastership is assumed only later via the election job, preventing boot-time split-brain.
4. Each job queue binds job_funcs to a static array and seeds one job — cluster HB_CJOB_INIT immediately, resource HB_RJOB_CHANGE_MODE on an init + failover_wait delay so the cluster settles a leader first; neither runs until a worker exists.
5. All three collections are one intrusive doubly-linked list with a pointer-to-pointer prev; the primitives are head-agnostic, and the allocators all NULL-guard, malloc, default-fill, hb_list_add.
6. Priority is assigned 1, 2, 3 by config order via the loop’s monotonic counter — why ties are impossible. A REPLICA-mode node with an empty ha_replica_list is fatal; replicas get the 0x7FFF sentinel, appear only in ha_replica_list (whose group id must equal ha_node_list’s), and HB_MAX_NUM_NODES = 8 caps the table.
7. Ping witnesses prefer ICMP, falling back to TCP only on zero ICMP hosts; both loaders exclude 0.0.0.0 and break on malformed TCP host:port.

Chapter 3: The Job Queue and Worker Threads

Chapter 1 laid out the structs; Chapter 2 built and zeroed them. The next question is: what machinery actually runs the FSM? The answer is one generic timer-ordered job queue (HB_JOB), instantiated twice (cluster_Jobs, resource_Jobs), drained by four detached pthreads. Every behaviour in Chapters 5–10 is a function pointer parked in one of these queues with an expire time. For the meaning of the cluster/resource split and the FSM, see the companion cubrid-heartbeat.md (“Two planes”, “The job model”); this chapter dissects queue mechanics and thread bodies, not that theory.

3.1 The two queue instances and the dispatch tables

HB_JOB (every field in Chapter 1) bundles a mutex, an expire-sorted intrusive entry list, a dispatch-table pointer, and a shutdown flag. Two file-scope instances exist — cluster_Jobs and resource_Jobs, defined NULL until hb_*_initialize allocates them (Chapter 2). Each instance’s job_funcs points at one of two enum-indexed dispatch tables; a queued job carries only a type (unsigned int), which the table turns into a function pointer at enqueue time.

HB_JOB field	Role	Why
lock	per-queue mutex	Serialises every list mutation; separate from the data locks (3.9) and never nested with them.
num_jobs	unsigned short	Vestigial — set to 0 in hb_*_job_initialize and never touched again. No enqueue/dequeue updates it; do not read it as a live depth.
jobs	head of the sorted HB_JOB_ENTRY list	The expire-ascending queue itself; hb_job_dequeue peeks only this head.
job_funcs	pointer to the dispatch table	Binds the instance to hb_cluster_jobs[] or hb_resource_jobs[]; resolves type to a handler.
shutdown	latch flag	Set by hb_job_shutdown; once true, dequeue/reorder become no-ops and the matching worker loop exits.

A queued job is one HB_JOB_ENTRY:

HB_JOB_ENTRY field	Role	Why
next	forward link	Singly-walked forward; insertion and dequeue both traverse this.
prev	HB_JOB_ENTRY **, address of predecessor’s next slot	Enables O(1) splice-out without back-walking (*(n->prev) = n->next); the reason the list is “doubly”-linked despite never being walked backward.
type	unsigned int, the enum ordinal	Doubles as the dispatch-table index at enqueue time and the match key for reorder.
expire	absolute struct timeval deadline	The sort key; now + msec computed once at enqueue.
func	resolved HB_JOB_FUNC	Cached at enqueue (job_funcs[type]) so dispatch is a bare call, no re-lookup.
arg	HB_JOB_ARG * payload	Handler-owned; the worker frees the entry, the handler frees arg.

INVARIANT — num_jobs is not a live count. It is zeroed at init and never incremented or decremented anywhere. Any future code that branches on num_jobs to decide “is the queue empty / full” is reading a permanent 0. Test jobs->jobs == NULL for emptiness, not num_jobs.

// hb_cluster_jobs[] -- src/executables/master_heartbeat.c
static HB_JOB_FUNC hb_cluster_jobs[] = {
  hb_cluster_job_init,        /* HB_CJOB_INIT = 0 */   hb_cluster_job_heartbeat,   /* = 1 */
  hb_cluster_job_calc_score,  /* = 2 */                hb_cluster_job_check_ping,  /* = 3 */
  hb_cluster_job_failover,    /* = 4 */                hb_cluster_job_failback,    /* = 5 */
  hb_cluster_job_check_valid_ping_server, /* = 6 */    hb_cluster_job_demote,      /* = 7 */
  NULL  /* sentinel at index HB_CJOB_MAX */
};

hb_resource_jobs[] has the identical shape: nine handlers in HB_RJOB_* order (hb_resource_job_proc_start … …_confirm_cleanup_all), then NULL at HB_RJOB_MAX. Order must match the enum ordinals, because the type is the array index (new_job->func = jobs->job_funcs[job_type]).

INVARIANT — table order equals enum order; no compile-time guard. Nothing static-asserts ARRAY_SIZE(hb_cluster_jobs) == HB_CJOB_MAX + 1; the only protection is the runtime >= HB_*_MAX wrapper check (3.6) plus the NULL sentinel. Insert a new HB_CJOB_* mid-enum without its function pointer at the same slot and every later job silently dispatches to the wrong handler — no crash. Edit enum and table in one diff. The sentinel only catches job_type == HB_*_MAX, not mis-alignment.

flowchart LR
  J["HB_JOB instance\n(cluster_Jobs / resource_Jobs)"] -->|job_funcs| T["dispatch table\nhb_cluster_jobs[] / hb_resource_jobs[]"]
  J -->|jobs| L["sorted dbl-linked\nHB_JOB_ENTRY list"]

Figure 3-1. Each HB_JOB instance binds a sorted entry list to a dispatch table; cluster_Jobs and resource_Jobs are structurally identical but fully independent — separate lists, tables, mutexes.

3.2 hb_job_queue — insertion sorted by expire time

The single enqueue primitive: allocate, compute the absolute deadline, walk to the sorted slot.

// hb_job_queue -- src/executables/master_heartbeat.c  (condensed)
  new_job = malloc (sizeof (HB_JOB_ENTRY));
  if (new_job == NULL) return ER_OUT_OF_VIRTUAL_MEMORY;   /* <- branch: malloc fail -> ER */
  gettimeofday (&now, NULL); hb_add_timeval (&now, msec); /* now := wall clock + msec == deadline */
  new_job->type = job_type;
  new_job->func = jobs->job_funcs[job_type];              /* <- enum-to-fnptr resolved HERE */
  new_job->arg  = arg; memcpy (&new_job->expire, &now, sizeof (struct timeval));
  pthread_mutex_lock (&jobs->lock);
  for (job = &(jobs->jobs); *job; job = &((*job)->next))
    {
      if (hb_compare_timeval (&((*job)->expire), &now) <= 0) continue;  /* <- expires earlier: scan on */
      break;                                              /* <- first later entry: insert before it */
    }
  hb_list_add ((HB_LIST **) job, (HB_LIST *) new_job);    /* loop-exhaust -> appends at tail */
  pthread_mutex_unlock (&jobs->lock);  return NO_ERROR;

hb_add_timeval does not normalize tv_usec carry past 1e6; for the small msec used here it stays below 1e6, so comparisons against a fresh gettimeofday are unperturbed in practice (observation, not guaranteed). prev/next are set by hb_list_add per the entry-field table above.

INVARIANT — jobs->jobs is always sorted ascending by expire. hb_job_queue and hb_job_set_expire_and_reorder are the only insert paths, both walking to the correct slot, so hb_job_dequeue can inspect only the head. Break the order and a far-future head would block every ready job behind it. The sort is the scheduler.

3.3 hb_job_dequeue — pop only when the head has expired

Workers call this in a tight loop; it returns the head only if its deadline has passed, else NULL.

// hb_job_dequeue -- src/executables/master_heartbeat.c  (condensed)
  gettimeofday (&now, NULL); pthread_mutex_lock (&jobs->lock);
  if (jobs->shutdown == true) { unlock; return NULL; }   /* <- branch 1: shutting down */
  job = jobs->jobs;
  if (job == NULL)            { unlock; return NULL; }   /* <- branch 2: empty queue */
  if (hb_compare_timeval (&now, &job->expire) >= 0)
    hb_list_remove ((HB_LIST *) job);                    /* <- branch 3a: now >= expire -> detach head */
  else                       { unlock; return NULL; }    /* <- branch 3b: head not ready */
  pthread_mutex_unlock (&jobs->lock);  return job;       /* caller owns job, must free_and_init it */

Because the list is sorted (3.2), checking only the head suffices; hb_list_remove splices it in O(1) via the prev idiom (3.1 table). The worker that gets a non-NULL entry owns it and must free_and_init it; hb_job_dequeue detaches but never frees.

3.4 hb_job_set_expire_and_reorder — the bump-to-immediate primitive

The only way to reschedule an already-queued job. Its real use (Chapter 5) is yanking the periodic HB_CJOB_CALC_SCORE job to the front so a recompute happens now, after a heartbeat changed the cluster picture.

// hb_job_set_expire_and_reorder -- src/executables/master_heartbeat.c  (condensed)
  gettimeofday (&now, NULL); hb_add_timeval (&now, msec); /* msec usually HB_JOB_TIMER_IMMEDIATELY (0) */
  pthread_mutex_lock (&jobs->lock);
  if (jobs->shutdown == true) { unlock; return; }         /* <- branch 1: shutting down -> no-op */
  for (job = &(jobs->jobs); *job; job = &((*job)->next))
    if ((*job)->type == job_type) { target_job = *job; break; }  /* FIRST entry of this type */
  if (target_job == NULL) { unlock; return; }             /* <- branch 2: not queued -> no-op */
  memcpy (&target_job->expire, &now, sizeof (struct timeval));
  hb_list_remove ((HB_LIST *) target_job);                /* detach, then re-walk to sorted slot */
  for (job = &(jobs->jobs); *job; job = &((*job)->next))  /*   (same insertion loop as 3.2) */
    if (hb_compare_timeval (&((*job)->expire), &(target_job->expire)) > 0) break;
  hb_list_add ((HB_LIST **) job, (HB_LIST *) target_job); /* reinsert; sort invariant preserved */
  pthread_mutex_unlock (&jobs->lock);

The target-not-found no-op is normal — a reorder can race a handler that already re-armed the job. Remove-then-reinsert (not in-place mutation) keeps the 3.2 sort intact, and touching only the first matching type suffices because the FSM keeps at most one of each periodic type alive.

3.5 hb_job_shutdown — drain and latch

// hb_job_shutdown -- src/executables/master_heartbeat.c  (condensed; under jobs->lock)
  for (job = jobs->jobs; job; job = job_next)
    { job_next = job->next; hb_list_remove (job); free_and_init (job); }  /* <- save next FIRST */
  jobs->shutdown = true;                      /* <- latch: dequeue/reorder now no-op */

After the latch, hb_job_dequeue and hb_job_set_expire_and_reorder short-circuit, and the two queue-drain workers fall through their guards: cluster_Jobs->shutdown == false and resource_Jobs->shutdown == false. Note the scope: this latch stops only those two of the four threads. The reader keys on a different flag, hb_Cluster->shutdown, and the disk-failure thread on hb_Resource->shutdown — neither is touched by hb_job_shutdown; both are set elsewhere during teardown (Chapter 10).

3.6 The typed wrappers and the HB_*_MAX bound check

Callers never touch hb_job_* directly. They go through six trivial wrappers that bind the right HB_JOB instance and bound-check the type. hb_cluster_job_queue is the template:

// hb_cluster_job_queue -- src/executables/master_heartbeat.c  (condensed)
  if (job_type >= HB_CJOB_MAX) return ER_FAILED;  /* <- the ONLY guard against out-of-range index */
  return hb_job_queue (cluster_Jobs, job_type, arg, msec);

hb_cluster_job_dequeue is just hb_job_dequeue (cluster_Jobs) (no bound check — it takes no type); hb_cluster_job_set_expire_and_reorder does the same >= HB_CJOB_MAX check then delegates. The resource trio (hb_resource_job_queue / _dequeue / _set_expire_and_reorder) is identical with resource_Jobs and HB_RJOB_MAX. Because the check is >=, the maximal valid index is HB_*_MAX - 1; the NULL sentinel at HB_*_MAX is unreachable through a valid call and is only a defensive tail marker.

3.7 hb_cluster_job_init — the bootstrap job

The first thing dequeued on the cluster side is HB_CJOB_INIT, queued by hb_cluster_job_initialize (Chapter 2). Its handler seeds the three recurring cluster jobs:

// hb_cluster_job_init -- src/executables/master_heartbeat.c  (condensed; each queue call assert'd NO_ERROR)
  hb_cluster_job_queue (HB_CJOB_HEARTBEAT, NULL, HB_JOB_TIMER_IMMEDIATELY);
  hb_cluster_job_queue (HB_CJOB_CHECK_VALID_PING_SERVER, NULL, HB_JOB_TIMER_IMMEDIATELY);
  hb_cluster_job_queue (HB_CJOB_CALC_SCORE, NULL, prm_get_integer_value (PRM_ID_HA_INIT_TIMER_IN_MSECS));
  if (arg) free_and_init (arg);              /* <- INIT queued with NULL arg, so defensive */

HB_JOB_TIMER_IMMEDIATELY is (0), so heartbeat and ping-server-validity fire on the very next worker pass; CALC_SCORE is deferred by PRM_ID_HA_INIT_TIMER_IN_MSECS so the first score lands after a few heartbeats. Each handler re-queues itself (or is re-bumped via 3.4) — Chapters 4 and 5. The canonical pattern: a job’s last act is to re-arm itself.

3.8 The four threads and hb_thread_initialize

hb_thread_initialize spawns four threads, all detached (PTHREAD_CREATE_DETACHED, system scope), sharing one pthread_attr_t. No join handle is kept.

// hb_thread_initialize -- src/executables/master_heartbeat.c  (condensed; each step returns its own ER_CSS_* on failure)
pthread_attr_setdetachstate (&thread_attr, PTHREAD_CREATE_DETACHED);   /* detached: no join */
pthread_attr_setscope (&thread_attr, PTHREAD_SCOPE_SYSTEM);            /* + bump stack if too small */
pthread_create (&cluster_worker_th,     &thread_attr, hb_thread_cluster_reader,     NULL);
pthread_create (&cluster_worker_th,     &thread_attr, hb_thread_cluster_worker,     NULL);  /* <- same local reused */
pthread_create (&resource_worker_th,    &thread_attr, hb_thread_resource_worker,    NULL);
pthread_create (&check_disk_failure_th, &thread_attr, hb_thread_check_disk_failure, NULL);
return NO_ERROR;

A harmless quirk: reader and cluster-worker are both created into the same local cluster_worker_th (one of three TID locals). Detached and never re-read, the overwrite is irrelevant. Any pthread_create failure returns immediately, leaving earlier threads running.

flowchart TD
  HTI["hb_thread_initialize"] --> R["hb_thread_cluster_reader\nrecvfrom on hb_Cluster->sfd"]
  HTI --> CW["hb_thread_cluster_worker\ndrains cluster_Jobs"]
  HTI --> RW["hb_thread_resource_worker\ndrains resource_Jobs"]
  HTI --> DF["hb_thread_check_disk_failure\n100ms tick, demote-on-disk-fail"]
  R -->|hb_cluster_receive_heartbeat\nmay queue CJOBs| CQ["cluster_Jobs"]
  CW -->|dequeue + dispatch| CQ
  RW -->|dequeue + dispatch| RQ["resource_Jobs"]

Figure 3-2. Four detached threads. The reader feeds the cluster queue indirectly via receive handling; the two workers drain their queues; the disk-failure thread is timer-driven, not queue-driven.

hb_thread_cluster_reader is the only thread doing I/O on the heartbeat UDP socket. It caches sfd = hb_Cluster->sfd once, then polls with a 1 ms timeout:

// hb_thread_cluster_reader -- src/executables/master_heartbeat.c  (condensed)
sfd = hb_Cluster->sfd;
while (hb_Cluster->shutdown == false)
  {
    po[0].fd = sfd; po[0].events = POLLIN;
    if (poll (po, 1, 1) <= 0) continue;                  /* <- timeout(0) or poll error(-1): re-poll */
    if ((po[0].revents & POLLIN) && sfd == hb_Cluster->sfd)  /* <- re-check sfd unchanged */
      {
        len = recvfrom (sfd, aligned_buffer, HB_BUFFER_SZ, 0, &from, &from_len);
        if (len > 0)                                     /* <- 0 or negative len dropped silently */
          hb_cluster_receive_heartbeat (aligned_buffer, len, &from, from_len);
      }
  }

The sfd == hb_Cluster->sfd re-check guards against the socket being swapped under the thread. The buffer is HB_BUFFER_SZ (4096) plus MAX_ALIGNMENT slack, PTR_ALIGNed once outside the loop. Decode and any resulting hb_cluster_job_queue calls happen inside hb_cluster_receive_heartbeat (Chapter 4).

hb_thread_cluster_worker and hb_thread_resource_worker share one shape — the queue-drain loop:

// hb_thread_cluster_worker -- src/executables/master_heartbeat.c  (resource_worker is identical with resource_Jobs)
while (cluster_Jobs->shutdown == false)
  {
    while ((job = hb_cluster_job_dequeue ()) != NULL)
      { job->func (job->arg); free_and_init (job); }     /* <- run, then free the entry */
    SLEEP_MILISEC (0, 10);                                /* <- nothing ready: sleep 10ms */
  }

Inner loop drains every ready job (dequeue returns NULL when the head isn’t expired, the queue is empty, or shutdown is latched). Outer loop sleeps 10 ms — the FSM’s scheduling granularity, so a job armed “immediately” runs within ~10 ms. The worker frees the entry; the handler owns and frees job->arg.

hb_thread_check_disk_failure wakes every HB_DISK_FAILURE_CHECK_TIMER_IN_MSECS (100 ms), counting down toward PRM_ID_HA_CHECK_DISK_FAILURE_INTERVAL_IN_SECS, then takes all three locks (anchor → cluster → resource) and demotes on server-log-growth failure. That demote logic is deferred to Chapter 8. Here, note only that this is the one queue-independent actor, it loops on hb_Resource->shutdown (not the queue latch), and it is why the three-lock ordering (3.9) exists.

3.9 The two-mutex synchronisation contract

Two data-protecting mutexes, plus one borrowed from the connection layer:

Mutex	Protects	Held by
hb_Cluster->lock	cluster node list, states, scores, is_isolated, sfd	cluster jobs, receive path, disk-failure thread
hb_Resource->lock	resource process list, hb_Resource->state	resource jobs, disk-failure thread
css_Master_socket_anchor_lock	the master’s socket anchor (connection layer)	taken outside the HB locks where process I/O crosses into the master

The queue mutexes (cluster_Jobs->lock, resource_Jobs->lock) are separate from the data mutexes, taken only inside the hb_job_* primitives; they never nest with the data locks. The data locks are what every later chapter contends for.

INVARIANT — lock ordering is anchor, then hb_Cluster->lock, then hb_Resource->lock. hb_thread_check_disk_failure acquires them in exactly that order and releases in reverse. Any code needing more than one must follow the same order or the two heartbeat planes can deadlock against the disk-failure thread. The cluster and resource workers normally touch only their own plane’s data lock, so they run truly in parallel; deadlock risk appears only on cross-plane paths (failover, demote, disk-failure). Chapters 6–8 lean on this.

3.10 Chapter summary — key takeaways

1. One generic queue, two instances. HB_JOB (mutex + expire-sorted intrusive doubly-linked list + dispatch-table pointer + shutdown flag, Chapter 1) is instantiated as cluster_Jobs and resource_Jobs; everything the FSM does is a job in one of them.
2. Enum-indexed tables, no static assert. hb_cluster_jobs[] / hb_resource_jobs[] must stay aligned with the HB_CJOB_* / HB_RJOB_* enums; the only guard is the >= HB_*_MAX wrapper check plus a NULL sentinel. Mis-alignment dispatches the wrong handler silently.
3. Sort-on-insert, peek-on-dequeue. hb_job_queue and the reorder primitive keep the list ascending by expire; hb_job_dequeue returns the head only when now >= head->expire. The sort is the timer.
4. hb_job_set_expire_and_reorder is the accelerate primitive. It rewrites the first matching job’s expire (usually IMMEDIATELY) and re-sorts it to the front — how CALC_SCORE gets bumped. Silent no-op if shutting down or no such job exists.
5. Jobs re-arm themselves. hb_cluster_job_init seeds heartbeat / ping-validity / calc-score, and each periodic handler re-queues its successor. This self-perpetuation is the FSM clock.
6. Four detached threads, no joins. A reader (one recvfrom per POLLIN), two queue-drain workers (10 ms granularity), one 100 ms disk-failure ticker. hb_job_shutdown’s jobs->shutdown latch stops only the two workers; the reader and disk-failure threads exit on the separate hb_Cluster->shutdown / hb_Resource->shutdown flags set during teardown (Chapter 10).
7. Two data mutexes, strict ordering. hb_Cluster->lock and hb_Resource->lock (plus the borrowed anchor lock) protect the two planes; the canonical order anchor → cluster → resource, set by the disk-failure thread, is the deadlock-avoidance contract every cross-plane chapter (6–8) must honour.

Chapter 4: Heartbeat Exchange on the Wire

This chapter answers one question: how does a node put a heartbeat packet on the wire, and exactly how does each inbound packet mutate the peer table? It is the wire-level companion to cubrid-heartbeat.md (its “UDP gossip” and “split-brain avoidance” sections explain why); here we trace how, branch by branch. The transport is connectionless UDP on PRM_ID_HA_PORT_ID. Everything runs under the single big lock hb_Cluster->lock; we produce the two staleness signals Chapter 5 consumes and do not compute scores.

4.1 The wire format: HBP_HEADER

Every cluster datagram begins with a fixed HBP_HEADER (declared in heartbeat.h). The body for HBP_CLUSTER_HEARTBEAT is exactly one packed int — the sender’s node state.

// struct hbp_header -- src/connection/heartbeat.h
struct hbp_header
{
  unsigned char type;
#if defined(HPUX) || defined(_AIX) || defined(sparc)
  char r:1;            /* is request? */     /* <- on big-endian platforms r comes first */
  char reserved:7;
#else
  char reserved:7;     /* both r:1 and reserved:7 pack into one byte; */
  char r:1;            /* physical order is endian/platform-gated */
#endif
  unsigned short len;
  unsigned int seq;
  char group_id[HB_MAX_GROUP_ID_LEN];
  char orig_host_name[CUB_MAXHOSTNAMELEN];
  char dest_host_name[CUB_MAXHOSTNAMELEN];
};

The field semantics drive every branch below, so here is the full role table.

Field	Role	Why it exists
type	Message discriminator; HBP_CLUSTER_HEARTBEAT is the only value	Receiver switches on it; anything else hits default log-and-drop
r	Request bit. 1 = “please respond”; 0 = this is the response	One primitive serves both directions; gates the reply that prevents split-brain
reserved	Padding filling the bitfield byte	Keeps the header byte-stable across compilers
len	Body length, net byte order; always OR_INT_SIZE for cluster HB	Length check rejects truncated/oversized datagrams
seq	Sequence number, net byte order	Reserved; cluster HBs always pass 0; no dedup today
group_id	Cluster identity string	Two clusters on one subnet must not cross-pollinate
orig_host_name	Sender hostname	Key into the peer table and the unidentified-node table
dest_host_name	Intended recipient hostname	First validation gate; a datagram not for me is dropped

Invariant — the body is always OR_INT_SIZE for HBP_CLUSTER_HEARTBEAT. The sender packs exactly one int with len = OR_INT_SIZE; the receiver recomputes sizeof(*hbp_header) + htons(hbp_header->len) and drops the packet if it disagrees with the byte count actually received — intended fail-closed.

flowchart LR
  subgraph Outbound
    JOB["hb_cluster_job_heartbeat<br/>re-queues ~0.5s"]
    ALL["hb_cluster_request_heartbeat_to_all<br/>gap++ per peer"]
    REQ["hb_cluster_send_heartbeat_req<br/>r=1"]
    INT["hb_cluster_send_heartbeat_internal<br/>or_pack_int(state)"]
    JOB --> ALL --> REQ --> INT
  end
  subgraph Inbound
    RDR["hb_thread_cluster_reader<br/>recvfrom loop"]
    RCV["hb_cluster_receive_heartbeat"]
    RESP["hb_cluster_send_heartbeat_resp<br/>r=0"]
    RDR --> RCV
    RCV -->|r==1| RESP
  end
  INT -. UDP datagram .-> RDR
  RESP -. UDP datagram .-> RDR

Figure 4-1. The two halves of the gossip path. A request (r=1) provokes a response (r=0); both share hb_cluster_send_heartbeat_internal.

4.2 Outbound: the self-re-queuing job

hb_cluster_job_heartbeat is the metronome — it broadcasts, then re-arms.

// hb_cluster_job_heartbeat -- src/executables/master_heartbeat.c
  pthread_mutex_lock (&hb_Cluster->lock);
  if (hb_Cluster->hide_to_demote == false)      /* <- gate: skip sends while demoting */
    hb_cluster_request_heartbeat_to_all ();
  pthread_mutex_unlock (&hb_Cluster->lock);
  error = hb_cluster_job_queue (HB_CJOB_HEARTBEAT, NULL,
                                prm_get_integer_value (PRM_ID_HA_HEARTBEAT_INTERVAL_IN_MSECS));
  // ... assert (error == NO_ERROR); if (arg) free_and_init (arg); ...

Three branches: (1) hide_to_demote == false → broadcast; (2) hide_to_demote == true → skip the send (the “going quiet” state while demoting, Chapter 8 — it keeps re-queuing but stops emitting); (3) arg != NULL → free the job argument (defensive; heartbeat jobs are queued with NULL). The re-queue interval is PRM_ID_HA_HEARTBEAT_INTERVAL_IN_MSECS (default 500 ms — HB_DEFAULT_HEARTBEAT_INTERVAL_IN_MSECS).

Invariant — the heartbeat job always re-queues. hb_cluster_job_queue sits after the unlock and outside every if, so even when hide_to_demote suppresses sends the job re-arms. Skip it and the cluster stops gossiping; every peer marks this node dead once its heartbeat_gap climbs past PRM_ID_HA_MAX_HEARTBEAT_GAP (default HB_DEFAULT_MAX_HEARTBEAT_GAP = 5).

The broadcast loop

// hb_cluster_request_heartbeat_to_all -- src/executables/master_heartbeat.c
  if (hb_Cluster == NULL) { ... return; }       /* defensive: pre-init / post-shutdown */
  for (node = hb_Cluster->nodes; node; node = node->next)
    {
      if (are_hostnames_equal (hb_Cluster->host_name, node->host_name))
        continue;                               /* <- never heartbeat myself */
      hb_cluster_send_heartbeat_req (node->host_name);
      node->heartbeat_gap++;                    /* <- signal #1, raised optimistically */
    }

Two branches: the self entry is skipped; every other peer gets a request and an unconditional heartbeat_gap++, bumped before we know whether it answers (the decrement happens in the receiver only on a valid reply). So heartbeat_gap is a running count of outstanding, unanswered heartbeats; sendto’s return is ignored — an un-sendable peer is exactly the one we want the gap to mark stale.

4.3 Outbound: the three send primitives

_req and _resp are thin one-line wrappers over _internal, differing only in the is_req flag. _req first resolves dest_host_name to a fresh sockaddr_in via hb_hostname_n_port_to_sockaddr, early-returning on DNS/parse failure (abandoning that peer for the round); _resp reuses the from address the datagram arrived on, so it never does a DNS lookup. _internal builds the buffer, packs the one int of state, and sends — the load-bearing lines:

// hb_cluster_send_heartbeat_internal -- src/executables/master_heartbeat.c
  // ... memset buffer; hbp_header = (HBP_HEADER *) &buffer[0] ...
  error_code = hb_set_net_header (hbp_header, HBP_CLUSTER_HEARTBEAT, is_req,
                                  OR_INT_SIZE, 0 /* seq */, dest_host_name);
  if (error_code != NO_ERROR)                   /* <- myself==NULL: cluster unhealthy, bail */
    return error_code;
  p = or_pack_int ((char *) (hbp_header + 1), hb_Cluster->state);  /* <- the ONE int of payload */
  hb_len = sizeof (HBP_HEADER) + OR_INT_SIZE;
  if (hb_Cluster->sfd == INVALID_SOCKET)        /* socket not open */
    return ERR_CSS_TCP_DATAGRAM_SOCKET;
  send_len = sendto (hb_Cluster->sfd, &buffer[0], hb_len, 0, (struct sockaddr *) saddr, saddr_len);
  if (send_len <= 0)                            /* kernel refused / interface down */
    return ER_FAILED;
  return NO_ERROR;

Three error branches — (a) myself == NULL; (b) INVALID_SOCKET; (c) sendto <= 0 — all propagate to the caller, which on the broadcast path discards the error and lets heartbeat_gap climb. The payload is only hb_Cluster->state.

Filling the header: hb_set_net_header

Its single guard is myself == NULL (no self-entry → cannot fill orig_host_name); otherwise it stamps type/r/len/seq and the identity strings:

// hb_set_net_header -- src/executables/master_heartbeat.c
  if (hb_Cluster->myself == NULL)
    return ER_FAILED;                           /* <- only failure branch */
  header->type = type;                          /* HBP_CLUSTER_HEARTBEAT */
  header->r = (is_req) ? 1 : 0;                 /* the request/response bit */
  header->len = htons (len);                    /* network byte order */
  header->seq = htonl (seq);
  // ... strncpy_bufsize group_id / dest_host_name / myself->host_name -> orig_host_name ...

len/seq go to network byte order; host-name strings are copied raw, so the receiver compares them with plain strcmp / are_hostnames_equal.

4.4 Inbound: hb_cluster_receive_heartbeat, branch by branch

The reader thread loops on recvfrom and calls this per datagram: a chain of validation gates then the state mutation. Figure 4-2 walks every branch; the prose covers each load-bearing condition.

flowchart TD
  A["recvfrom datagram"] --> B{"shutdown?"}
  B -->|yes| Z1["unlock, return"]
  B -->|no| C{"dest_host_name == me?"}
  C -->|no| Z2["log mismatch, unlock, return"]
  C -->|yes| D{"len == header+htons(len)?"}
  D -->|no| Z3["log size mismatch, unlock, return"]
  D -->|yes| E{"type == HBP_CLUSTER_HEARTBEAT?"}
  E -->|no| Z4["default: log unknown, break"]
  E -->|yes| F["or_unpack_int -> state"]
  F --> G{"state in [UNKNOWN, MAX)?"}
  G -->|no| Z5["log unknown state, unlock, return"]
  G -->|yes| H{"hb_is_heartbeat_valid == NO_ERROR?"}
  H -->|no| UI["unidentified-node bookkeeping"]
  H -->|yes| J{"group_id == my group?"}
  UI --> J
  J -->|no| Z6["unlock, return"]
  J -->|yes| K{"r == 1 and not hide_to_demote?"}
  K -->|yes| RESP["send heartbeat resp"]
  K -->|no| L
  RESP --> L["hb_return_node_by_name_except_me"]
  L --> M{"node found?"}
  M -->|no| Z7["log unknown host, break"]
  M -->|yes| N{"old==MASTER and new!=MASTER?"}
  N -->|yes| O["is_state_changed=true, save hb_Master_host_name"]
  N -->|no| P
  O --> P["state=new; gap=MAX(0,gap-1); stamp last_recv_hbtime"]
  P --> Q{"is_state_changed?"}
  Q -->|yes| R["reorder CALC_SCORE to IMMEDIATELY"]
  Q -->|no| Z8["return"]

Figure 4-2. Every branch of hb_cluster_receive_heartbeat. Left rejects; right accepts and mutates.

The first four reject gates share one shape — test, MASTER_ER_LOG_DEBUG, pthread_mutex_unlock, return (the type gate breaks through the switch to the same bottom unlock). Their guards:

• Gate 0 — shutdown: if (hb_Cluster->shutdown) — nothing processed in teardown.
• Gate 1 — destination: if (!are_hostnames_equal (hb_Cluster->host_name, hbp_header->dest_host_name)) — drops broadcast/misroute packets not for me.
• Gate 2 — length: if (len != (int)(sizeof (*hbp_header) + htons (hbp_header->len))) — enforces the §4.1 body-size invariant on receive.
• Gate 3 — type: switch (hbp_header->type); only HBP_CLUSTER_HEARTBEAT is handled, default logs “unknown heartbeat message” and breaks.
• Gate 4 — state range: in the case, or_unpack_int reads the body into hb_state, then if (hb_state < HB_NSTATE_UNKNOWN || hb_state >= HB_NSTATE_MAX) unlocks and returns — dropping a corrupt or incompatible state value.

Gate 5 — validity + unidentified-node bookkeeping. hb_is_heartbeat_valid (§4.6) returns HB_VALID_NO_ERROR or one of four rejection reasons. On rejection the code records the offender rather than dropping silently:

  rv = hb_is_heartbeat_valid (hbp_header->orig_host_name, hbp_header->group_id, from);
  if (rv != HB_VALID_NO_ERROR)
    {
      ui_node = hb_return_ui_node (hbp_header->orig_host_name, hbp_header->group_id, *from);
      if (ui_node && ui_node->v_result != rv)   /* reason changed -> stale entry */
        { hb_remove_ui_node (ui_node); ui_node = NULL; }
      if (ui_node == NULL)                       /* first sighting (or reason changed) */
        {
          MASTER_ER_SET (..., ER_HB_NODE_EVENT, 1, error_string);
          (void) hb_add_ui_node (hbp_header->orig_host_name, hbp_header->group_id, *from, rv);
        }
      else
        gettimeofday (&ui_node->last_recv_time, NULL);  /* known offender: just refresh */
    }

Three sub-branches: (a) no entry → log once + add; (b) entry, different v_result → remove and re-add (log fires again with the new reason); (c) entry, same reason → silently refresh last_recv_time. Sub-branch (c) rate-limits — a spamming peer produces one ER_HB_NODE_EVENT, not one per packet.

Gate 6 — group match (hard drop). if (strcmp (hbp_header->group_id, hb_Cluster->group_id)) unlocks and returns, deliberately duplicating the group check inside hb_is_heartbeat_valid: a different-group packet may have been logged above but must never reach the mutation code.

Response branch — split-brain avoidance. if (hbp_header->r && hb_Cluster->hide_to_demote == false) calls hb_cluster_send_heartbeat_resp. The hide_to_demote guard mirrors §4.2’s outbound gate — a demoting node neither initiates nor answers, so peers stop counting it alive (rationale in the companion doc).

The mutation — peer lookup, state, gap, timestamp.

  node = hb_return_node_by_name_except_me (hbp_header->orig_host_name);
  if (node)
    {
      if (node->state == HB_NSTATE_MASTER && node->state != hb_state)
        {
          /* current master demoted: save its name for the failover error message */
          is_state_changed = true;
          snprintf (hb_Master_host_name, strlen (node->host_name) + 1, node->host_name);
        }
      node->state = hb_state;                                   /* adopt peer's reported state */
      node->heartbeat_gap = MAX (0, (node->heartbeat_gap - 1));  /* <- signal #1, lowered */
      gettimeofday (&node->last_recv_hbtime, NULL);             /* <- signal #2, stamped */
    }
  else
    MASTER_ER_LOG_DEBUG (..., "receive heartbeat have unknown host_name. ...");

Two branches. (1) node == NULL: not a configured peer; Gate 5 already flagged it HB_VALID_UNIDENTIFIED_NODE. (2) node found: the nested old==MASTER && new!=MASTER test is the only place is_state_changed is raised in the receive path; then, unconditionally, the three mutations above (adopt state, floor-decrement gap, stamp timestamp) fire.

Invariant — counters mutate only under hb_Cluster->lock, and heartbeat_gap is floored at 0. The receiver holds the lock from Gate 0 to the bottom unlock and the sender across the whole broadcast, so the per-peer gap++ and gap = MAX(0, gap-1) never interleave and lose a tick. The floor stops a chatty peer (more replies than counted requests) driving gap negative and masking later silence — a negative gap would take extra ticks to climb back above the PRM_ID_HA_MAX_HEARTBEAT_GAP threshold Chapter 5 checks.

The tail — deferred score recomputation. After unlocking, only when a master demotion was observed:

  pthread_mutex_unlock (&hb_Cluster->lock);
  if (is_state_changed == true)
    hb_cluster_job_set_expire_and_reorder (HB_CJOB_CALC_SCORE, HB_JOB_TIMER_IMMEDIATELY);

The receiver yanks HB_CJOB_CALC_SCORE forward to HB_JOB_TIMER_IMMEDIATELY (== 0) — the hand-off to Chapter 5, done after releasing the lock to avoid holding it across the job-queue call.

4.5 Dual staleness signals — what Chapter 5 consumes

Signal	Field	Raised by	Lowered by	Meaning
Outstanding-request count	heartbeat_gap	sender, gap++ per broadcast	receiver, MAX(0, gap-1) per valid reply	Heartbeats sent without a matching answer; the score path declares the peer HB_NSTATE_UNKNOWN once it exceeds PRM_ID_HA_MAX_HEARTBEAT_GAP
Last-heard time	last_recv_hbtime	—	receiver, stamped now per valid reply	Wall-clock recency; the score path uses PRM_ID_HA_CALC_SCORE_INTERVAL_IN_MSECS, while hb_cluster_is_received_heartbeat_from_all uses the tighter PRM_ID_HA_HEARTBEAT_INTERVAL_IN_MSECS

Redundant on purpose: the gap counts events and survives clock jumps; the timestamp measures elapsed wall time and survives suppressed sends (e.g. hide_to_demote, where the gap stops climbing but real silence still needs detecting). A peer is healthy only while both stay below threshold.

4.6 The validation and lookup helpers

hb_is_heartbeat_valid returns the first failing reason, checked in order — four rejections plus success:

// hb_is_heartbeat_valid -- src/executables/master_heartbeat.c
  if (hb_return_node_by_name_except_me (host_name) == NULL)
    return HB_VALID_UNIDENTIFIED_NODE;          /* not a configured peer */
  if (strcmp (group_id, hb_Cluster->group_id) != 0)
    return HB_VALID_GROUP_NAME_MISMATCH;        /* wrong cluster */
  // ... hb_hostname_to_sin_addr fail -> HB_VALID_CANNOT_RESOLVE_HOST ...
  if (memcmp (&sin_addr, &from->sin_addr, sizeof (struct in_addr)) != 0)
    return HB_VALID_IP_ADDR_MISMATCH;           /* <- anti-spoof: name must resolve to source IP */
  return HB_VALID_NO_ERROR;

The IP check is the anti-spoof guard. The enum maps to HB_VALID_*_STR via hb_valid_result_string for the log line. hb_return_node_by_name and _except_me are the two peer-table lookups; _except_me adds || are_hostnames_equal (name, hb_Cluster->host_name) to its continue, so it never returns the self-entry. The receiver uses _except_me everywhere — a node must never process its own heartbeat as a peer’s.

The unidentified-node table (hb_Cluster->ui_nodes, a doubly-linked HB_UI_NODE_ENTRY list) has three managers:

• hb_return_ui_node — linear search on the triple (host_name, group_id, source IP); all three must match.
• hb_add_ui_node — asserts a valid rejection code, dedups via hb_return_ui_node, else mallocs, fills the triple plus v_result, stamps last_recv_time, links, bumps num_ui_nodes; malloc-fail returns NULL.
• hb_cleanup_ui_nodes — the GC: removes via hb_remove_ui_node any entry older than HB_UI_NODE_CLEANUP_TIME_IN_MSECS (one hour).

Invariant — num_ui_nodes equals the ui_nodes list length. Add bumps it, hb_remove_ui_node unlinks/frees/decrements and clamps at 0 with an assert(0) guard. Drift would make the operator-facing count (Chapter 10) lie; the clamp prevents an underflow wrapping to a huge value.

4.7 Chapter summary — key takeaways

1. The wire payload is one packed int — hb_Cluster->state. Gap, timestamps, and scores are all derived locally; nothing about liveness is on the wire.
2. heartbeat_gap and last_recv_hbtime are the two staleness signals (§4.5), both lowered/stamped only on a valid reply; Chapter 5 consumes both.
3. The receiver is a chain of fail-closed gates — destination, length, type, state range, validity, group — all checked before any mutation; group and validity failures additionally logged as unidentified-node events.
4. r and hide_to_demote implement split-brain avoidance: a request is answered (r == 0) unless the node is demoting, when it stays silent both directions.
5. A peer losing master status is the one event that pre-empts election — the old==MASTER && new!=MASTER branch sets is_state_changed and reorders HB_CJOB_CALC_SCORE to fire immediately.
6. Unidentified hosts are logged once per reason, not per packet, via the ui_node dedup (host+group+IP, refreshed in place, GC’d after an hour).
7. Counters mutate only under hb_Cluster->lock, and the metronome never stops — the send job re-queues outside all guards even while suppressed.

Chapter 5: Score Computation and Local Leader Election

A fresh peer table (Ch.4 wrote last_recv_hbtime, heartbeat_gap, and per-peer state for every HB_NODE_ENTRY) is raw observation. This chapter answers: how does a node collapse that table into a single elected master, and decide whether to act on the verdict? hb_cluster_calc_score is the arithmetic stage (age out stale peers, score each state, pick the minimum); hb_cluster_job_calc_score is the decision hub (read the verdict against this node’s own state, queue exactly one transition job or re-arm). The score model and election concept are in the companion (cubrid-heartbeat.md, “Score-based election”); here we trace every branch and bit. The chapter ends the instant a transition job lands on the queue; the jobs themselves — HB_CJOB_CHECK_PING, HB_CJOB_FAILOVER, HB_CJOB_FAILBACK — are Ch.6–8.

Both figures use flowchart, not stateDiagram-v2, so <br/> and parentheses inside node labels are permitted (KB Mermaid rule 5); no transition label uses them.

5.1 The score bitmask and why MASTER wins a minimum search

Election is a minimum-score search: the node with the smallest score becomes hb_Cluster->master. The four HB_NODE_SCORE_* constants in master_heartbeat.h are the four hex values in the table below.

The final score is node->priority | HB_NODE_SCORE_*, where priority is a small positive unsigned short (1 for the configured master, ascending down ha_node_list; HB_REPLICA_PRIORITY = 0x7FFF for replicas). Because score is a signed short (short score; in HB_NODE_ENTRY), OR-ing the high bit 0x8000 sets the sign bit, so the value reads as negative and sorts below every slave and unknown.

Score constant	Hex	As signed short	Sorts
HB_NODE_SCORE_MASTER (MASTER, TO_BE_SLAVE)	0x8000	0x8000 = −32768 (SHRT_MIN); 0x8000|1 = −32767	lowest — wins
HB_NODE_SCORE_TO_BE_MASTER	0xF000	−4096	low, but above MASTER
HB_NODE_SCORE_SLAVE	0x0000	0 + priority	middle
HB_NODE_SCORE_UNKNOWN (UNKNOWN, REPLICA)	0x7FFF	+32767	highest — never wins

Invariant — the elected master is the numerically smallest score, and a live MASTER always beats a SLAVE. Enforced by the sign-bit encoding: only MASTER/TO_BE_SLAVE/TO_BE_MASTER set bit 15, so a SLAVE (non-negative) can never undercut them. Declaring score as unsigned short would make 0x8000 the largest value, so the master is never elected — a one-keyword change that silently inverts the HA cluster.

Why TO_BE_SLAVE shares the MASTER bit (0x8000). A HB_NSTATE_TO_BE_SLAVE node is a former master mid-demotion (Ch.8); until it finishes it must out-rank a real slave so no second master is elected prematurely. HB_NODE_SCORE_TO_BE_MASTER = 0xF000 is the opposite — a slave mid-promotion (Ch.6): negative (beats slaves/unknowns) but larger than 0x8000, so a genuine MASTER still beats it.

5.2 hb_cluster_calc_score — staleness aging, then the score switch

This function walks hb_Cluster->nodes once. Before scoring a peer it forcibly demotes any stale peer to HB_NSTATE_UNKNOWN, then the switch maps state to score:

// hb_cluster_calc_score -- src/executables/master_heartbeat.c
int num_master = 0;
short min_score = HB_NODE_SCORE_UNKNOWN;          /* start at the worst possible */
hb_Is_master_node_isolated = false;
// ... NULL guard elided: if (hb_Cluster == NULL) return ER_FAILED; ...
hb_Cluster->myself->state = hb_Cluster->state;    /* sync own entry from cluster state */
gettimeofday (&now, NULL);
for (node = hb_Cluster->nodes; node; node = node->next)
  {
    if (node->heartbeat_gap > prm_get_integer_value (PRM_ID_HA_MAX_HEARTBEAT_GAP)        /* count path */
        || (!HB_IS_INITIALIZED_TIME (node->last_recv_hbtime)              /* <- never-heard skipped */
            && HB_GET_ELAPSED_TIME (now, node->last_recv_hbtime)                         /* time path */
               > prm_get_integer_value (PRM_ID_HA_CALC_SCORE_INTERVAL_IN_MSECS)))
      {
        if (hb_Cluster->myself != node && node->state == HB_NSTATE_MASTER)
          { hb_Is_master_node_isolated = true; /* save node->host_name into hb_Master_host_name */ }
        node->heartbeat_gap = 0;                                         /* reset staleness counters */
        node->last_recv_hbtime.tv_sec = node->last_recv_hbtime.tv_usec = 0;
        node->state = HB_NSTATE_UNKNOWN;                                 /* <- demote stale peer */
      }
    switch (node->state)
      {
      case HB_NSTATE_MASTER:
      case HB_NSTATE_TO_BE_SLAVE:    node->score = node->priority | HB_NODE_SCORE_MASTER;       break; /* 0x8000 */
      case HB_NSTATE_TO_BE_MASTER:   node->score = node->priority | HB_NODE_SCORE_TO_BE_MASTER; break; /* 0xF000 */
      case HB_NSTATE_SLAVE:          node->score = node->priority | HB_NODE_SCORE_SLAVE;        break; /* 0x0000 */
      case HB_NSTATE_REPLICA:
      case HB_NSTATE_UNKNOWN:
      default:                       node->score = node->priority | HB_NODE_SCORE_UNKNOWN;      break; /* 0x7FFF */
      }
    if (node->score < min_score)
      { hb_Cluster->master = node; min_score = node->score; }            /* <- running argmin */
    if (node->score < (short) HB_NODE_SCORE_TO_BE_MASTER)                /* < 0xF000 as short == MASTER bit */
      num_master++;
  }
return num_master;

The elided NULL guard returns ER_FAILED, treated by the caller as a non-positive num_master.

The two-armed staleness test. A peer is reset to UNKNOWN if either the count path heartbeat_gap > ha_max_heartbeat_gap (gap bumped on send, zeroed on receive, per Ch.4) catches a peer that stopped replying while we polled, or the time path elapsed > ha_calc_score_interval catches a stalled send loop where heartbeat_gap never grew but real time passed. The HB_IS_INITIALIZED_TIME guard ({0,0}) makes a never-heard peer skip the time clause, so it fails only via its mounting heartbeat_gap, never a bogus “elapsed since 1970” value. When that stale peer is the current MASTER, the hb_Is_master_node_isolated/hb_Master_host_name globals are set — read only later for the failover diagnostic (§5.4), never affecting the arithmetic.

The switch and num_master. The switch is total (default + explicit REPLICA/UNKNOWN degrade any state to UNKNOWN); the reduction leaves hb_Cluster->master at the lowest score. num_master counts a node only if its score is strictly below (short)0xF000 (−4096) — true only for the 0x8000-class (scores ≤ −32767), never a 0xF000 TO_BE_MASTER; the caller uses num_master > 1 as its split-brain detector.

Invariant — num_master counts only settled-master-class nodes, never in-flight promotions. Enforced by the strict < against (short)0xF000. If the comparison were <= or used 0x8000, an in-progress TO_BE_MASTER promotion would inflate num_master, and §5.4’s num_master > 1 branch would fire a spurious FAILBACK against a node merely becoming master — tearing down a legitimate failover mid-flight.

flowchart TD
  A["start: min_score = 0x7FFF, num_master = 0"] --> B["myself->state = cluster->state"]
  B --> C{"for each node in nodes"}
  C -->|stale: gap > max OR elapsed > interval| D{"stale peer == MASTER and not myself?"}
  D -->|yes| E["hb_Is_master_node_isolated = true, save host_name"]
  D -->|no| F["reset gap, last_recv_hbtime; state = UNKNOWN"]
  E --> F
  C -->|fresh| G["score = priority | bitmask by state"]
  F --> G
  G --> H{"score < min_score?"}
  H -->|yes| I["master = node; min_score = score"]
  H -->|no| J["keep current master"]
  I --> K{"score < (short)0xF000?"}
  J --> K
  K -->|yes| L["num_master++"]
  K -->|no| M[" "]
  L --> C
  M --> C
  C -->|done| N["return num_master"]

Figure 5-1. hb_cluster_calc_score: per-peer staleness aging, scoring, running argmin, and the master counter — every branch.

5.3 The two helpers: hb_cluster_is_isolated and hb_cluster_is_received_heartbeat_from_all

hb_cluster_is_isolated answers “am I cut off from every real peer?”:

// hb_cluster_is_isolated -- src/executables/master_heartbeat.c
for (node = hb_Cluster->nodes; node; node = node->next)
  {
    if (node->state == HB_NSTATE_REPLICA)
      continue;                                            /* <- replicas never count */
    if (hb_Cluster->myself != node && node->state != HB_NSTATE_UNKNOWN)
      return false;                                        /* a live non-replica peer exists */
  }
return true;

Three branches: REPLICA skipped (read-only appliers, never master-eligible); myself skipped (own entry would mask isolation); any other non-UNKNOWN peer returns false. Only if every non-replica, non-self peer is UNKNOWN does it return true.

Invariant — isolation means every non-replica peer was aged to UNKNOWN by hb_cluster_calc_score this same cycle. The caller runs calc_score immediately before is_isolated (§5.4), so the UNKNOWN states read are this cycle’s verdict; calling is_isolated without a preceding calc_score could declare a healthy node isolated.

hb_cluster_is_received_heartbeat_from_all is a stricter real-time freshness check used only to tune the failover delay:

// hb_cluster_is_received_heartbeat_from_all -- src/executables/master_heartbeat.c
heartbeat_confirm_time = prm_get_integer_value (PRM_ID_HA_HEARTBEAT_INTERVAL_IN_MSECS);
gettimeofday (&now, NULL);
for (node = hb_Cluster->nodes; node; node = node->next)
  {
    if (hb_Cluster->myself != node
        && HB_GET_ELAPSED_TIME (now, node->last_recv_hbtime) > heartbeat_confirm_time)
      return false;                                        /* one peer is even one interval stale */
  }
return true;

It returns true only if every non-self peer was heard within one ha_heartbeat_interval. Unlike is_isolated it does not skip replicas and does not guard {0,0} — a never-heard peer’s elapsed time exceeds heartbeat_confirm_time, correctly yielding false. This tightest predicate exists purely to pick a fast (500 ms) versus conservative failover wait in §5.4.

5.4 hb_cluster_job_calc_score — the decision hub

The CALC_SCORE job body runs under hb_Cluster->lock, calls the two stages above, then branches on this node’s own hb_Cluster->state, emitting at most one transition job (only the fall-through re-arms CALC_SCORE):

// hb_cluster_job_calc_score -- src/executables/master_heartbeat.c   (lock/unlock, free_and_init, malloc-arg setup condensed)
num_master = hb_cluster_calc_score ();
hb_Cluster->is_isolated = hb_cluster_is_isolated ();

if (hb_Cluster->state == HB_NSTATE_REPLICA || hb_Cluster->hide_to_demote == true)
  goto calc_end;                                /* Branch 0: replicas / demoting never elect */

if (hb_Cluster->state == HB_NSTATE_MASTER && hb_Cluster->is_isolated == true)
  {                                             /* Branch 1: isolated master -> ping witness */
    if (malloc ok) hb_cluster_job_queue (HB_CJOB_CHECK_PING, job_arg, HB_JOB_TIMER_IMMEDIATELY);
    return;                                     /* malloc fail: NOT queued, stay master; NO re-queue */
  }

if ((num_master > 1) && hb_Cluster->master && hb_Cluster->myself   /* Branch 2: split-brain loser */
    && hb_Cluster->myself->state == HB_NSTATE_MASTER
    && hb_Cluster->master->priority != hb_Cluster->myself->priority)        /* <- loser test */
  {
    hb_cluster_job_queue (HB_CJOB_FAILBACK, NULL, HB_JOB_TIMER_IMMEDIATELY);
    return;
  }

if ((hb_Cluster->state == HB_NSTATE_SLAVE) && hb_Cluster->master && hb_Cluster->myself  /* Branch 3: winner */
    && hb_Cluster->master->priority == hb_Cluster->myself->priority)        /* <- winner == me */
  {
    hb_Cluster->state = HB_NSTATE_TO_BE_MASTER;        /* <- the election write */
    hb_cluster_request_heartbeat_to_all ();
    // ... pick diagnostic from hb_Is_master_node_isolated / hb_Master_host_name ...
    if (malloc ok)
      hb_cluster_job_queue (HB_CJOB_CHECK_PING, job_arg, HB_JOB_TIMER_WAIT_100_MILLISECOND);
    else                                        /* malloc failed: skip ping, pick FAILOVER delay */
      {
        SLEEP_MILISEC (0, HB_JOB_TIMER_WAIT_100_MILLISECOND);
        failover_wait_time = hb_cluster_is_received_heartbeat_from_all ()
          ? HB_JOB_TIMER_WAIT_500_MILLISECOND                              /* fast: all fresh */
          : prm_get_integer_value (PRM_ID_HA_FAILOVER_WAIT_TIME_IN_MSECS); /* conservative */
        hb_cluster_job_queue (HB_CJOB_FAILOVER, NULL, failover_wait_time);
      }
    return;
  }

calc_end:                                       /* Branch 4: steady state */
hb_cluster_job_queue (HB_CJOB_CALC_SCORE, NULL,            /* <- only path that re-arms */
                      prm_get_integer_value (PRM_ID_HA_CALC_SCORE_INTERVAL_IN_MSECS));

The inline /* Branch N */ comments and Figure 5-2 enumerate the mechanics; the load-bearing rationale per branch:

• Branch 0 — REPLICA and hide_to_demote (a master mid-step-down, Ch.8) compute for observability only.
• Branch 1 — the malloc-failure path is a deliberate fail-safe: an isolated master stays master rather than demote under memory pressure.
• Branch 2 (loser) — the priority != conjunct is the loser test; the winner is the tied master with the smallest priority (lower = higher rank). Equal priorities ⇒ I am the winner and fall through.
• Branch 3 (winner) — the mirror priority == guard means the argmin winner is my own entry. The commit hb_Cluster->state = HB_NSTATE_TO_BE_MASTER is the chapter’s most consequential line; the diagnostic reads §5.2’s globals (hb_Is_master_node_isolated ⇒ lost-heartbeat, else hb_Master_host_name ⇒ “server process problem”, Ch.8); the malloc-failure arm degrades to direct FAILOVER tuned by hb_cluster_is_received_heartbeat_from_all (§5.3).
• Branch 4 — the only path that re-queues CALC_SCORE; a slave’s is_isolated is stored but never triggers here.

flowchart TD
  A["lock; num_master = calc_score(); is_isolated = is_isolated()"] --> B{"state == REPLICA<br/>or hide_to_demote?"}
  B -->|yes| Z["goto calc_end"]
  B -->|no| C{"state == MASTER<br/>and is_isolated?"}
  C -->|yes| C1["queue CHECK_PING immediately; return (no re-queue)"]
  C -->|no| D{"num_master &gt; 1<br/>and I am MASTER<br/>and master.prio != my.prio?"}
  D -->|yes| D1["queue FAILBACK; return"]
  D -->|no| E{"state == SLAVE<br/>and master.prio == my.prio?"}
  E -->|yes| E1["state = TO_BE_MASTER; request_heartbeat_to_all"]
  E1 --> E2{"malloc job_arg ok?"}
  E2 -->|yes| E3["queue CHECK_PING +100ms; return"]
  E2 -->|no| E4["sleep 100ms; pick wait via received_from_all; queue FAILOVER; return"]
  E -->|no| Z
  Z --> ZZ["calc_end: unlock; re-queue CALC_SCORE; return"]

Figure 5-2. hb_cluster_job_calc_score: the full branch tree from score verdict to a single queued transition job.

5.5 Chapter summary — key takeaways

1. Election is a signed-short minimum search. score = priority | HB_NODE_SCORE_*; OR-ing 0x8000 makes a master’s score negative so it wins score < min_score. Declaring score unsigned would invert the entire HA verdict.
2. hb_cluster_calc_score ages stale peers to UNKNOWN via a two-armed test (heartbeat_gap > ha_max_heartbeat_gap OR elapsed > ha_calc_score_interval, the HB_IS_INITIALIZED_TIME guard sparing never-heard peers), then runs a single-pass argmin into hb_Cluster->master and counts num_master.
3. num_master counts only 0x8000-class nodes (score < (short)0xF000), so a TO_BE_MASTER promotion in flight is not counted — this keeps the split-brain detector from firing against a legitimate failover.
4. TO_BE_SLAVE shares the master bit, TO_BE_MASTER uses 0xF000 so a demoting ex-master still out-ranks a slave while a promoting slave still loses to a genuine master.
5. hb_cluster_job_calc_score emits at most one action, keyed on this node’s own state; only the steady-state fall-through re-arms CALC_SCORE, so the periodic loop and the transition pipeline never double-schedule.
6. Priority equality is the winner test, inequality the loser test. The global argmin winner having my priority means it is my entry. Priority encodes configured rank (lower = higher); isolation forces action only for a MASTER.

Chapter 6: Failover and the Ping Witness Check

This chapter answers one question: how does a slave that won the local election (Chapter 5) actually become master, and what stops a partitioned slave from promoting when the master is still alive across a broken link?

The companion high-level doc (cubrid-heartbeat.md, “Failover and split-brain avoidance”) explains why a quorum-free two-node HA cluster needs an external witness; this chapter traces the code path. We assume the Chapter 5 score model (hb_cluster_calc_score, the priority/master fields) and the Chapter 3 job queue, and do not re-derive them.

Promotion is a three-state walk — SLAVE to TO_BE_MASTER to MASTER — guarded by two independent checks at two different times:

1. The ping witness check (hb_cluster_job_check_ping): before committing, ask external hosts “can you still reach me?” If no witness answers, this node is the isolated one — abort back to SLAVE.
2. The failover double-check (hb_cluster_job_failover): after a deliberate wait, recompute the score once more. If a slow-but-alive master reasserted itself meanwhile, abort back to SLAVE.

These guards defend orthogonal failures — a self-partitioned slave vs. a slow-but-alive master — so wrongful promotion requires both signals to fail at once: the ping check must not abort (the witness was reachable, or — in the witness-less / no-verdict case — gave no proof of isolation) and the master must stay silent through the whole wait. The witness-less path below is the documented availability-over-safety escape hatch.

stateDiagram-v2
    [*] --> SLAVE
    SLAVE --> TO_BE_MASTER : calc_score: master.priority == myself.priority \n queue CHECK_PING after 100ms
    TO_BE_MASTER --> SLAVE : CHECK_PING hosts answered, none reachable \n positive proof of isolation, abort
    TO_BE_MASTER --> TO_BE_MASTER : CHECK_PING still ambiguous \n retry up to HB_MAX_PING_CHECK
    TO_BE_MASTER --> WAIT : CHECK_PING passes or no usable verdict \n queue FAILOVER after failover_wait_time
    WAIT --> MASTER : FAILOVER recheck still highest \n set both planes MASTER, queue CHANGE_MODE
    WAIT --> SLAVE : FAILOVER a new master appeared \n abort
    MASTER --> [*]

Figure 6-1. The promotion state walk. WAIT is not a stored state — it is the interval during which the queued FAILOVER job is pending; hb_Cluster->state stays TO_BE_MASTER throughout.

The walk is entered in hb_cluster_job_calc_score (Chapter 5): when a SLAVE finds master->priority == myself->priority (it became the highest-priority survivor), it sets hb_Cluster->state = HB_NSTATE_TO_BE_MASTER, zeroes ping_check_count, and queues HB_CJOB_CHECK_PING after 100 ms — where this chapter begins.

6.1 The witness check: hb_cluster_job_check_ping branch by branch

This function is shared by two callers with opposite intent: an isolated master decides whether to failback (abdicate), a to-be-master slave whether to failover. hb_Cluster->state disambiguates them at every branch. Setup: clst_arg carries ping_check_count across retries; ping_try_count counts hosts giving a definite verdict; ping_success records whether any host was reachable.

// hb_cluster_job_check_ping -- src/executables/master_heartbeat.c
HB_CLUSTER_JOB_ARG *clst_arg = (arg) ? &(arg->cluster_job_arg) : NULL;
if (clst_arg == NULL || hb_Cluster->num_ping_hosts == 0 || hb_Cluster->is_ping_check_enabled == false)
  {
    /* If Ping Host is either empty or marked invalid, MASTER->MASTER, SLAVE->MASTER. */
    if (hb_Cluster->state == HB_NSTATE_MASTER)
      goto ping_check_cancel;    /* <- master, no usable witness: keep mastership */
    /* SLAVE: no goto, no else-block; skip past the whole if/else to the decision tail */
  }

Branch A — no usable witness (num_ping_hosts == 0 or is_ping_check_enabled == false). With no host to ask, the cluster cannot tell “master died” from “I am partitioned” and resolves toward promotion: a MASTER goto ping_check_cancels and keeps mastership; a to-be-master SLAVE takes neither the goto nor the else block — it skips all ping I/O and the retry loop entirely, falling past the whole if/else to the decision tail where it proceeds to failover.

INVARIANT — without a witness, the cluster prefers availability over partition-safety. The source comment is blunt: “It may cause split-brain problem.” Both a stale master and a promoting slave keep/take mastership, so a partition can yield two masters — but a witness-less two-node cluster has no other tie-breaker, and an unavailable cluster is judged worse than a transient split-brain (later reconciled by the Chapter 5 calc_score split-brain branch and FAILBACK).

Branch B — witnesses configured and enabled. The else block pings every host in hb_Cluster->ping_hosts (ICMP hb_check_ping or TCP hb_check_tcp_ping), then splits on state:

// hb_cluster_job_check_ping -- src/executables/master_heartbeat.c
for (ping_host = hb_Cluster->ping_hosts; ping_host; ping_host = ping_host->next)
  {
    // ... ping_result = hb_check_ping(...) or hb_check_tcp_ping(...) ...
    ping_host->ping_result = ping_result;
    if (ping_result == HB_PING_SUCCESS)
      { ping_try_count++; ping_success = true; break; }   /* <- one reachable witness is enough */
    else if (ping_result == HB_PING_FAILURE)
      { ping_try_count++; }                               /* <- definite unreachable; counts */
    /* HB_PING_USELESS_HOST / SYS_ERR / UNKNOWN: counted neither way, skip */
  }

Only HB_PING_SUCCESS/HB_PING_FAILURE increment ping_try_count; a misconfigured/errored/unknown host gives no evidence — so ping_try_count == 0 means “no usable answer”, distinct from “every host unreachable”. The verdict then goto ping_check_cancels on a mirror-image condition: a master cancels failback when ping_try_count == 0 || ping_success == true (reached anything or learned nothing — stays master); a slave cancels failover only on positive proof of isolation, ping_try_count > 0 && ping_success == false (a host answered and none were reachable — witnesses up, but this node cannot reach them).

Branch C — retry loop. If neither cancel fired, the picture is still ambiguous; the count is bumped and the job re-queued immediately:

// hb_cluster_job_check_ping -- src/executables/master_heartbeat.c
if ((++clst_arg->ping_check_count) < HB_MAX_PING_CHECK)   /* HB_MAX_PING_CHECK == 3 */
  {
    pthread_mutex_unlock (&hb_Cluster->lock);
    error = hb_cluster_job_queue (HB_CJOB_CHECK_PING, arg, HB_JOB_TIMER_IMMEDIATELY);
    return;                    /* <- same arg reused; ping_check_count persists across requeue */
  }

INVARIANT — at most HB_MAX_PING_CHECK (3) ping rounds before a decision is forced. ping_check_count lives in the heap HB_JOB_ARG (zeroed by the caller), is incremented only here, and the same arg is threaded through every requeue unfreed — so the count accumulates instead of resetting and the loop is bounded.

Branch D — decision tail (loop exhausted, or slave fell through Branch A). Once ping_check_count reaches 3 — or a slave arrived directly from the witness-less Branch A — the node commits and frees arg:

// hb_cluster_job_check_ping -- src/executables/master_heartbeat.c
if (hb_Cluster->state == HB_NSTATE_MASTER)
  error = hb_cluster_job_queue (HB_CJOB_FAILBACK, NULL, HB_JOB_TIMER_IMMEDIATELY);   /* master -> Ch 7 */
else
  {
    failover_wait_time = hb_cluster_is_received_heartbeat_from_all ()
      ? HB_JOB_TIMER_WAIT_500_MILLISECOND                            /* all peers seen */
      : prm_get_integer_value (PRM_ID_HA_FAILOVER_WAIT_TIME_IN_MSECS); /* a node silent: wait longer */
    error = hb_cluster_job_queue (HB_CJOB_FAILOVER, NULL, failover_wait_time);
  }
if (arg) free_and_init (arg);   /* <- terminal path frees the carried arg */

The master defers to FAILBACK (Chapter 7); the slave queues FAILOVER delayed. The longer wait when a node is still silent (hb_cluster_is_received_heartbeat_from_all == false) is the first half of the slow-master defense — time for a sluggish-but-alive master to send one more heartbeat. The re-check in 6.3 is the second half.

Branch E — ping_check_cancel label. Reached by the gotos above. It logs a diagnostic keyed on the same state/ping_try_count distinction, and crucially only the slave branch mutates state — hb_Cluster->state = HB_NSTATE_SLAVE reverts the in-flight TO_BE_MASTER and aborts failover, while the master leaves its state alone. Both then re-arm the CALC_SCORE cycle and free_and_init (arg).

6.2 Keeping witnesses honest: hb_cluster_check_valid_ping_server and its job

The 6.1 check trusts is_ping_check_enabled, maintained by a slow background sweep so a dead witness cannot block failover. hb_cluster_check_valid_ping_server is the pure probe:

// hb_cluster_check_valid_ping_server -- src/executables/master_heartbeat.c
if (hb_Cluster->num_ping_hosts == 0)
  return true;                 /* <- "no ping host configured" counts as valid (see NOTE in source) */
for (ping_host = hb_Cluster->ping_hosts; ping_host; ping_host = ping_host->next)
  // ... ping_host->ping_result = hb_check_ping/tcp_ping(...); if SUCCESS -> valid_ping_host_exists = true ...
return valid_ping_host_exists;   /* <- no break in the loop: pings ALL hosts to refresh every ping_result */

Unlike 6.1 it does not break on first success — it pings every host so each ping_host->ping_result is freshly stamped for diagnostic dumps, returning true if any host is reachable (and, per its NOTE, when none is configured). The job wrapper hb_cluster_job_check_valid_ping_server is the only place is_ping_check_enabled is toggled:

// hb_cluster_job_check_valid_ping_server -- src/executables/master_heartbeat.c
int check_interval = HB_DEFAULT_CHECK_VALID_PING_SERVER_INTERVAL_IN_MSECS;   /* 1 hour */
if (hb_Cluster->num_ping_hosts == 0)
  goto check_end;              /* <- nothing to validate; reschedule at the 1h default */
valid_ping_host_exists = hb_cluster_check_valid_ping_server ();
if (valid_ping_host_exists == false && hb_cluster_is_isolated () == false)
  {
    check_interval = HB_TEMP_CHECK_VALID_PING_SERVER_INTERVAL_IN_MSECS;   /* 5 min, faster recheck */
    hb_Cluster->is_ping_check_enabled = false;   /* <- disable: witnesses unreachable but NOT isolated */
  }
else if (valid_ping_host_exists == true)
  {
    hb_Cluster->is_ping_check_enabled = true;    /* <- re-enable when a witness comes back */
  }
check_end:
  pthread_mutex_unlock (&hb_Cluster->lock);
  error = hb_cluster_job_queue (HB_CJOB_CHECK_VALID_PING_SERVER, NULL, check_interval);

The disable condition is precise: valid_ping_host_exists == false and hb_cluster_is_isolated() == false — witnesses unreachable while cluster heartbeat still flows means the witness is broken, not the network. Disabling the flag then makes 6.1 fall into Branch A (promote) rather than Branch E (abort), so a broken witness cannot wrongly abort a legitimate failover. Cadence: 5 min while disabled, 1 h once a witness returns. This job only flips the flag and reschedules.

INVARIANT — at runtime is_ping_check_enabled is toggled only by hb_cluster_job_check_valid_ping_server, under hb_Cluster->lock. (The one other write is the one-time = true initialization in hb_cluster_initialize, before the job loop runs; the config/info paths never write the flag directly — they only reorder this job to run immediately when they observe a mismatch.) Every reader (6.1) takes the same lock, so the flag is never read mid-flip; separating the slow sweep from the fast decision path keeps a transient ping miss from being mistaken for a dead witness.

6.3 Committing the promotion: hb_cluster_job_failover

FAILOVER runs only after the Branch D delay. Its first act is a fresh hb_cluster_calc_score over heartbeats that arrived during the wait:

// hb_cluster_job_failover -- src/executables/master_heartbeat.c
rv = pthread_mutex_lock (&hb_Cluster->lock);
num_master = hb_cluster_calc_score ();   /* <- SECOND recompute, after the wait window */
if (hb_Cluster->master && hb_Cluster->myself
    && hb_Cluster->master->priority == hb_Cluster->myself->priority)
  {                                          /* "[Failover] [Success] ... promoted to master" */
    hb_Cluster->state = HB_NSTATE_MASTER;
    hb_Resource->state = HB_NSTATE_MASTER;   /* <- BOTH planes flip together */
    error = hb_resource_job_set_expire_and_reorder (HB_RJOB_CHANGE_MODE, HB_JOB_TIMER_IMMEDIATELY);
  }
else                                         /* "[Failover] [Cancelled] New master ... found" */
  {
    hb_Cluster->state = HB_NSTATE_SLAVE;     /* <- abort: someone else reasserted mastership */
  }

The confirm branch fires when master->priority == myself->priority still holds after the wait and second recompute. It flips both hb_Cluster->state (heartbeat plane) and hb_Resource->state (resource plane — the supervised CUBRID processes, Chapter 9) to HB_NSTATE_MASTER, then queues HB_RJOB_CHANGE_MODE immediately via hb_resource_job_set_expire_and_reorder. That job switches the server/replication processes into active master mode (Chapter 9/10); without it the state fields are cosmetic and the processes stay a replica.

INVARIANT — the cluster and resource planes flip to MASTER atomically under hb_Cluster->lock, with CHANGE_MODE queued in the same critical section. Both assignments and the queue precede the unlock, so no observer sees hb_Cluster->state == MASTER while hb_Resource->state is still SLAVE — no half-promoted ghost claiming mastership on the wire without promoting its processes.

The abort branch fires when the equality no longer holds — hb_cluster_calc_score now points hb_Cluster->master at another node, because while this slave waited a merely-slow master sent heartbeats again and the recompute re-elected it. The node reverts to HB_NSTATE_SLAVE, never touches the resource plane, queues no CHANGE_MODE — the local processes are never promoted.

After either branch the function logs node/ping-host info, re-broadcasts its new state via hb_cluster_request_heartbeat_to_all, unlocks, and re-arms the CALC_SCORE cycle (the WAIT --> MASTER / WAIT --> SLAVE edges in Figure 6-1).

6.4 Chapter summary — key takeaways

1. Promotion is a three-state walk SLAVE to TO_BE_MASTER to MASTER, started by hb_cluster_job_calc_score (Ch 5) and finished by hb_cluster_job_failover; TO_BE_MASTER is the gate held while the checks run.
2. hb_cluster_job_check_ping is shared by master (failback) and slave (failover); hb_Cluster->state disambiguates every branch. The master cancels if it reaches anything; the slave aborts only on positive proof of isolation (ping_try_count > 0 && !ping_success).
3. The retry loop is bounded by HB_MAX_PING_CHECK (3) via ping_check_count in the heap HB_JOB_ARG, threaded unchanged through every requeue and freed only at terminal paths.
4. The witness-less path (num_ping_hosts == 0 or !is_ping_check_enabled) prefers availability over partition-safety — masters keep their role, slaves promote — accepting possible split-brain.
5. is_ping_check_enabled is toggled only by hb_cluster_job_check_valid_ping_server (1h healthy / 5min disabled), which disables witness checking precisely when witnesses are unreachable but cluster heartbeat still flows.
6. hb_cluster_job_failover recomputes the score a SECOND time after failover_wait_time; the confirm branch flips hb_Cluster->state and hb_Resource->state to MASTER atomically under lock and queues HB_RJOB_CHANGE_MODE, while the abort branch reverts to SLAVE if a slow master reasserted itself. The two guards are orthogonal, so wrongful promotion needs both to fail.

Chapter 7: Failback when a Peer Reclaims Master

When a split-brain heals, exactly one of the two masters must step down. Chapters 5 and 6 explain who decides; this chapter answers the mechanical question: once a node knows it is the split-brain loser, how does it walk itself from MASTER back to SLAVE, and what happens to the cub_server processes it was hosting? The transition is the job hb_cluster_job_failback, deliberately unconditional — by the time it runs the decision is final. For the conceptual framing, see the companion’s “Failover, failback, and split-brain” section.

7.1 Where failback is decided, and why the job is unconditional

hb_cluster_job_failback has no “should I really demote?” guard — the decision lives entirely in its caller. The job’s contract is I am a master that has lost; clean up, which is why the body opens by writing the new state, not testing the old one.

Invariant — failback is the master-loser-only cleanup path. hb_cluster_job_failback runs iff the node held HB_NSTATE_MASTER when the caller decided against it. Enforcement is structural: every hb_cluster_job_queue (HB_CJOB_FAILBACK, ...) call site is guarded by a hb_Cluster->myself->state == HB_NSTATE_MASTER test. The two call sites are the CHECK_PING witness loser-branch (Chapter 6) and the split-brain detection near line 867, guarded by (num_master > 1) && hb_Cluster->myself->state == HB_NSTATE_MASTER && hb_Cluster->master->priority != hb_Cluster->myself->priority (this node sees another master of differing priority) before queuing FAILBACK at line 894. If the invariant broke — failback queued on a slave — the function would gratuitously SIGTERM that slave’s cub_server processes: an availability hole with no role change.

7.2 The state write: SLAVE directly, no TO_BE_SLAVE

The first thing failback does, under the cluster lock, is set the node to SLAVE outright, mirror it onto its node entry, and announce it:

// hb_cluster_job_failback -- src/executables/master_heartbeat.c
  pthread_mutex_lock (&hb_Cluster->lock);

  hb_Cluster->state = HB_NSTATE_SLAVE;            /* <- no TO_BE_SLAVE step */
  hb_Cluster->myself->state = hb_Cluster->state;  /* keep self-node mirror in sync */

  hb_cluster_request_heartbeat_to_all ();         /* announce SLAVE on the wire now */

Note what is absent: no transit through HB_NSTATE_TO_BE_SLAVE. Contrast failover (Chapter 6), where a slave promotes through HB_NSTATE_TO_BE_MASTER — an intermediate holding the node in committed-but-not-yet-active limbo while resources spin up. Failback has no symmetric limbo because demotion is destructive and immediate. hb_Cluster->myself->state is mirrored alongside hb_Cluster->state so the node’s own nodes-list entry agrees with the scalar role field; Chapter 5’s score computation reads node entries, so an out-of-sync myself would corrupt CALC_SCORE.

After the announce, failback emits diagnostics with no control-flow effect on the demotion: an ER_HB_NODE_EVENT success message (HA_FAILBACK_SUCCESS_STRING), a nodes-info dump, and — only when hb_Cluster->num_ping_hosts > 0 — a ping-host info line whose inner branch picks hb_help_sprint_ping_host_info if PRM_ID_HA_PING_HOSTS is set, else hb_help_sprint_tcp_ping_host_info. Both arms only format a log string; this is the only if/else pair in the function besides the §7.5 kill-mode branches. The state enum HB_NODE_STATE_TYPE (src/connection/heartbeat.h, see Chapter 1) defines TO_BE_SLAVE=3, conspicuously never assigned by any code path.

Invariant — HB_NSTATE_TO_BE_SLAVE is never assigned; demotion has no warm-up state. Grepping the tree, there is no state = HB_NSTATE_TO_BE_SLAVE assignment anywhere — not to hb_Cluster->state, not to hb_Resource->state, and not constructed for an outgoing heartbeat. The only role values ever written are MASTER, TO_BE_MASTER, SLAVE, REPLICA, and UNKNOWN. Two consequences follow. First, hb_resource_job_change_mode guards a change-mode broadcast on hb_Resource->state == HB_NSTATE_TO_BE_SLAVE — but since that field is never set to TO_BE_SLAVE, that arm of the guard is effectively dead: change-mode requests fire only from the MASTER arm. Second, the heartbeat receive path can stamp a peer’s HB_NODE_ENTRY.state to TO_BE_SLAVE (node->state = hb_state after unpacking the peer’s role), but because the sender packs its own hb_Cluster->state (§7.3) and never holds TO_BE_SLAVE, no heartbeat ever carries that value, so the peer-mirror path is dead in practice too. The enum value exists; the running code never reaches it. Failback’s jump straight to SLAVE is therefore not skipping a transit state — there is no transit state to skip on the demotion side, only on the promotion side (TO_BE_MASTER).

7.3 Announcing the demotion: hb_cluster_request_heartbeat_to_all

Before releasing the cluster lock, failback broadcasts its new SLAVE state so the winning peer learns the partition resolved:

// hb_cluster_request_heartbeat_to_all -- src/executables/master_heartbeat.c
  if (hb_Cluster == NULL)
    { MASTER_ER_LOG_DEBUG (ARG_FILE_LINE, "hb_Cluster is null. \n"); return; }  /* pre-init / shutdown race */

  for (node = hb_Cluster->nodes; node; node = node->next)
    {
      if (are_hostnames_equal (hb_Cluster->host_name, node->host_name))
        continue;                              /* skip myself; don't heartbeat the loopback */

      hb_cluster_send_heartbeat_req (node->host_name);
      node->heartbeat_gap++;                   /* optimistic gap bump, decremented on reply */
    }

Every branch: the hb_Cluster == NULL guard logs and returns rather than dereferencing null during an init/shutdown race; the are_hostnames_equal(...) check skips the self entry; otherwise hb_cluster_send_heartbeat_req ships a request whose payload is built by hb_cluster_send_heartbeat_internal, which packs hb_Cluster->state (now SLAVE) via or_pack_int and sendtos it, and node->heartbeat_gap++ records an outstanding request the receive path (Chapter 4) decrements on reply. Because the wire payload is the local hb_Cluster->state, a peer never receives TO_BE_SLAVE from this node (§7.2). A peer tracking this node as MASTER sees the flip to SLAVE and stops contesting. This is the generic “publish my current state” primitive, not failback-specific.

7.4 Lock ordering: drop the cluster lock before the kill loop

A load-bearing detail: failback releases hb_Cluster->lock before acquiring hb_Resource->lock to do the killing.

// hb_cluster_job_failback (lock handoff) -- src/executables/master_heartbeat.c
  pthread_mutex_unlock (&hb_Cluster->lock);    /* <- cluster lock dropped first */

  pthread_mutex_lock (&hb_Resource->lock);     /* then take the resource lock */
  hb_Resource->state = HB_NSTATE_SLAVE;
  // ... walk hb_Resource->procs, collect server pids ...

Invariant — the cluster and resource locks are never nested cluster-then-resource across the kill. The cluster mutex guards the node/state view, the resource mutex the process table. The subsequent hb_kill_process waits for servers to die (its header comment warns it blocks up to a minute), so holding hb_Cluster->lock across that wait would stall the heartbeat paths and the score job, manufacturing the very gaps that look like node failure. Dropping the cluster lock first keeps the heartbeat machinery live while teardown proceeds under the resource lock alone.

7.5 Walking the process table and collecting server pids

Under the resource lock, hb_Resource->state becomes SLAVE, then procs is walked to gather every HB_PTYPE_SERVER pid:

// hb_cluster_job_failback (collect loop) -- src/executables/master_heartbeat.c
  proc = hb_Resource->procs;
  while (proc)
    {
      if (proc->type != HB_PTYPE_SERVER)
        { proc = proc->next; continue; }       /* (1) non-server helpers untouched */

      if (emergency_kill_enabled == false)
        {
          size = sizeof (pid_t) * (count + 1);
          pids = (pid_t *) realloc (pids, size);
          if (pids == NULL)                    /* (2) OOM -> switch to inline SIGKILL */
            {
              MASTER_ER_SET (..., ER_OUT_OF_VIRTUAL_MEMORY, 1, size);
              emergency_kill_enabled = true;
              proc = hb_Resource->procs;       /* restart the walk from head */
              continue;
            }
          pids[count++] = proc->pid;           /* (3) normal: buffer the pid */
        }
      else
        {
          assert (proc->pid > 0);
          if (proc->pid > 0)
            kill (proc->pid, SIGKILL);         /* (4) emergency: kill in place */
        }
      proc = proc->next;
    }
  pthread_mutex_unlock (&hb_Resource->lock);

The four branches: (1) non-server helper types (copy/apply-log) are skipped — only cub_server instances are demoted here, the helpers follow the slave role via their own change-mode path; (2) on realloc failure the demotion does not abort (leaving master-side servers running) but flips emergency_kill_enabled, resets proc to the head, and re-walks so every server hits branch (4); (3) the normal path buffers the pid, no signal yet; (4) with pids unusable each server is SIGKILL-ed inline, the runtime if (proc->pid > 0) re-check ensuring a corrupt zero pid never becomes kill(0, ...) (a process-group signal).

After the walk, the resource lock is released and the graceful kill runs outside it: if (emergency_kill_enabled == false) hb_kill_process (pids, count); runs the SIGTERM-then-SIGKILL sequence on the buffered pids, skipped when the emergency path already SIGKILL-ed everything inline. Cleanup and the CALC_SCORE requeue follow (§7.8). Figure 7-1 shows the full control flow.

flowchart TD
  C["under hb_Cluster lock:\nstate=SLAVE, myself=SLAVE,\nannounce, unlock cluster"] --> F["under hb_Resource lock:\nhb_Resource->state = SLAVE"]
  F --> G{"walk procs:\nproc->type == SERVER?"}
  G -- "no" --> H["proc = proc->next"]
  H --> G
  G -- "yes, emergency off" --> I["realloc pids"]
  I -- "ok" --> J["pids[count++] = pid"]
  J --> H
  I -- "NULL" --> K["emergency_kill_enabled = true\nproc = head, restart walk"]
  K --> G
  G -- "yes, emergency on" --> L["kill(pid, SIGKILL) inline"]
  L --> H
  G -- "list end" --> M["unlock hb_Resource"]
  M --> N{"emergency off?"}
  N -- "yes" --> O["hb_kill_process(pids, count)"]
  N -- "no" --> P["skip; already SIGKILLed"]
  O --> Q["free pids, queue CALC_SCORE,\nfree arg, return"]
  P --> Q

7.6 hb_kill_process: SIGTERM, wait, escalate to SIGKILL

hb_kill_process is the graceful-then-forceful killer: it retries SIGTERM up to 20 times at 3-second intervals (~1 min), then SIGKILLs survivors:

// hb_kill_process -- src/executables/master_heartbeat.c
  int signum = SIGTERM;                        /* max_retries = 20, wait = 3s */
  for (i = 0; i < max_retries; i++)
    {
      finished = true;
      for (j = 0; j < count; j++)
        {
          if (pids[j] > 0)
            {
              error = kill (pids[j], signum);
              if (error && errno == ESRCH)     /* process already gone */
                pids[j] = 0;                   /* mark slot dead, stop signalling it */
              else
                finished = false;              /* still alive (or signal landed) */
            }
        }
      if (finished == true)
        return;                                /* all reaped -> done early */
      signum = 0;                              /* <- later passes only probe, don't re-TERM */
      SLEEP_MILISEC (wait_time_in_secs, 0);
    }

  for (j = 0; j < count; j++)                  /* timeout -> force kill survivors */
    if (pids[j] > 0)
      kill (pids[j], SIGKILL);

Branch-complete reading: a pids[j] <= 0 slot is skipped; a kill failing with ESRCH zeroes that slot permanently; any other outcome (success or non-ESRCH error) clears finished so the loop sleeps and re-checks; if finished survives a full inner pass the function returns immediately. Crucially signum = 0 after the first pass turns every later pass into a signal-less kill(pid, 0) existence probe — so the real SIGTERM is delivered exactly once and the server is not hammered while it flushes. After 20 passes (~60 s) with survivors, the trailing loop SIGKILLs them.

Invariant — no master-side cub_server survives failback. No path out of hb_kill_process leaves a pids[j] > 0 process running: every server either exits under SIGTERM or is removed by the trailing SIGKILL sweep. A leftover master-mode server could accept writes and re-open the split-brain failback meant to close.

7.7 Why kill-and-restart, not in-place mode change

The most surprising choice is that failback kills the cub_server rather than telling it “you are now a slave.” The reason is configuration: a CUBRID server loads its HA role and matching in-process settings (log applier/copier wiring, replication direction, buffer roles) at startup, and there is no live path that safely retargets a running master into a slave.

So failback does only half the job — removing the old master-mode servers. The restart with slave-side configuration happens through the resource-side death handler: when a killed cub_server drops its connection, hb_cleanup_conn_and_start_process marks the proc HB_PSTATE_DEAD and queues hb_resource_job_queue (HB_RJOB_PROC_START, ...) (Chapter 9). By then hb_Resource->state is already HB_NSTATE_SLAVE (§7.5), so HB_RJOB_PROC_START re-fork/execs the server reading the slave-side HA configuration. The net effect — old master dies, fresh slave spawns — is a role change by process replacement, not mutation. (The handler’s anti-flap demotion via HB_RJOB_DEMOTE_START_SHUTDOWN is the resource-driven path in Chapter 8.)

7.8 Re-queuing CALC_SCORE and cleanup

The final actions (§7.5) close the loop: HB_CJOB_CALC_SCORE is re-queued and the job’s arg is freed (a no-op here, since failback was launched with NULL, but it keeps the job ABI uniform). After failback returns the node is a fully demoted slave: cluster state SLAVE, resource state SLAVE, no master-mode servers alive, slave-mode servers respawning, the score job ticking again. The winning peer, having received the SLAVE heartbeat, is the sole master; split-brain is closed.

7.9 Chapter summary — key takeaways

1. Failback is unconditional by contract. Both call sites — the CHECK_PING witness loser-branch (Chapter 6) and the num_master > 1 split-brain detection at line 894 — queue HB_CJOB_FAILBACK only when hb_Cluster->myself->state == HB_NSTATE_MASTER. The job is cleanup, not decision.
2. Demotion writes SLAVE directly — there is no TO_BE_SLAVE warm-up. Unlike promotion’s TO_BE_MASTER step, demotion is immediate. HB_NSTATE_TO_BE_SLAVE is never assigned anywhere in the tree; the hb_resource_job_change_mode guard and the peer-mirror receive path that mention it are both dead in practice because no node ever holds or broadcasts that value.
3. The cluster lock is dropped before the kill loop. hb_kill_process blocks up to a minute, so holding hb_Cluster->lock across it would stall heartbeats and manufacture false node-failure gaps.
4. Only HB_PTYPE_SERVER procs are killed. The walk skips copy/apply-log helpers; under realloc failure it re-walks with inline SIGKILL so demotion completes under memory pressure. The post-announce ping-host logging branch is diagnostic-only.
5. hb_kill_process escalates SIGTERM to SIGKILL. First pass sends SIGTERM, later passes kill(pid, 0) poll only, and after ~60 s survivors are SIGKILL-ed — no master server outlives failback.
6. Servers are killed-and-restarted, not mode-changed. HA role is fixed at startup, so the safe retarget is process replacement: failback kills, the death handler (hb_cleanup_conn_and_start_process -> HB_RJOB_PROC_START) respawns with slave config because hb_Resource->state is SLAVE.
7. Failback re-queues CALC_SCORE. The demoted node rejoins the election cadence as a slave; the winning peer, seeing the SLAVE heartbeat, becomes sole master, closing the split-brain.

Chapter 8: Demote on Local Resource Failure

This chapter answers one question: when the local CUBRID server hangs while this node is master, how does the node step aside so a peer can take over — without two nodes briefly fighting over the role? The high-level companion (cubrid-heartbeat.md, “Failover, failback, demote”) gives the why; here we trace every branch.

The path runs in two worker contexts. The detector (hb_thread_check_disk_failure) probes the server’s log EOF and, on a wedge, flips the resource role to SLAVE and queues teardown. The teardown (hb_resource_job_demote_start_shutdown -> _confirm_shutdown) drains the server processes, then hands off to the cluster step-aside (hb_cluster_job_demote), which advertises UNKNOWN, drops to SLAVE, and waits for a peer to claim master. Throughout the cluster half, hide_to_demote (Chapter 1’s HB_CLUSTER) is the linchpin: while set, the node neither broadcasts heartbeats nor counts itself in score computation, so it cannot re-elect itself during the give-up window.

8.1 The detector: hb_thread_check_disk_failure

The disk-failure thread loops on a timer until hb_Resource->shutdown. Every PRM_ID_HA_CHECK_DISK_FAILURE_INTERVAL_IN_SECS seconds it grabs the anchor, cluster, and resource locks (nested in that order for deadlock-freedom against the job-queue paths) and runs the guarded probe.

// hb_thread_check_disk_failure -- src/executables/master_heartbeat.c
if (interval > 0 && remaining_time_msecs <= 0)   /* <- 0 disables the check entirely */
  {
    /* ... lock anchor, cluster, resource ... */
    if (hb_Cluster->is_isolated == false && hb_Resource->state == HB_NSTATE_MASTER)
      {                                          /* <- only an attached, serving master demotes itself */
        if (hb_resource_check_server_log_grow () == false)
          {
            MASTER_ER_SET (... HA_FAILBACK_DIAG_STRING " ... lost its role due to server process problem");
            MASTER_ER_SET (... HA_FAILBACK_SUCCESS_STRING " ... successfully demoted to slave"); /* <- detector emits the demote diagnostics itself */
            hb_disable_er_log (HB_NOLOG_DEMOTE_ON_DISK_FAIL, NULL); /* <- go quiet: logging may block on the bad disk */
            hb_Resource->state = HB_NSTATE_SLAVE;
            /* ... unlock resource, cluster, anchor ... */
            error = hb_resource_job_queue (HB_RJOB_DEMOTE_START_SHUTDOWN, NULL, HB_JOB_TIMER_IMMEDIATELY);
            continue;                            /* <- re-test guard; state is now SLAVE so probe won't re-fire */
          }
      }
    if (hb_Resource->state == HB_NSTATE_MASTER)
      hb_resource_send_get_eof ();               /* <- arm the NEXT interval's probe */
    /* ... unlock all three ... */
    remaining_time_msecs = interval * 1000;
  }
SLEEP_MILISEC (0, HB_DISK_FAILURE_CHECK_TIMER_IN_MSECS); /* 100 ms tick; decrement remaining when interval>0 */

Beyond the inline comments: interval <= 0 skips the whole body. An isolated node never demotes — dropping the only serving node is worse than a wedge, and isolation is the failover thread’s concern (Chapter 6); control falls to the second if, where a still-MASTER node arms the next probe.

Invariant — the probe verdict and the demote action are atomic against the resource state. The flip to SLAVE happens under all three locks before they are released; continue re-enters with state already SLAVE, skipping the send_get_eof arm. The in-lock flip plus continue guarantee exactly one teardown per wedge — a concurrent path could otherwise queue a second.

8.2 The EOF probe: send, receive, and the grow predicate

The wedge test is a two-interval comparison: each interval the detector requests the server’s log EOF and reads back what arrived since last time. An alive server advances its EOF; a wedged one repeats the same LSA or never answers. hb_resource_send_get_eof fans SERVER_GET_EOF to every active server, marking each reply not-yet-received.

// hb_resource_send_get_eof -- src/executables/master_heartbeat.c
if (hb_Resource->state != HB_NSTATE_MASTER)
  return;                                        /* <- only a master probes its servers */
for (proc = hb_Resource->procs; proc; proc = proc->next)
  if (proc->state == HB_PSTATE_REGISTERED_AND_ACTIVE)
    {
      css_send_heartbeat_request (proc->conn, SERVER_GET_EOF);
      proc->is_curr_eof_received = false;        /* <- set true only when a reply lands */
    }

hb_resource_receive_get_eof is the connection-thread callback that lands the reply, unpacking the LSA into proc->curr_eof.

// hb_resource_receive_get_eof -- src/executables/master_heartbeat.c
if (css_receive_heartbeat_data (conn, reply, ...) != NO_ERRORS)
  return;                                        /* <- transport error: curr_eof stays stale -> looks wedged */
/* ... lock resource ... */
proc = hb_return_proc_by_fd (conn->fd);
if (proc == NULL) { /* unlock; return */ }       /* <- reply for a process we no longer track */
if (proc->state == HB_PSTATE_REGISTERED_AND_ACTIVE)
  {
    or_unpack_log_lsa (reply, &proc->curr_eof);
    proc->is_curr_eof_received = true;           /* <- distinguishes "stale value" from "no answer" */
  }

hb_resource_check_server_log_grow is the predicate the detector calls, comparing curr_eof against prev_eof for each active, non-hung server.

// hb_resource_check_server_log_grow -- src/executables/master_heartbeat.c
for (proc = hb_Resource->procs; proc; proc = proc->next)
  {
    if (proc->type != HB_PTYPE_SERVER || proc->state != HB_PSTATE_REGISTERED_AND_ACTIVE
        || proc->server_hang == true)
      continue;                                  /* <- skip non-servers, non-active, already-flagged */
    if (LSA_ISNULL (&proc->curr_eof) == true)
      continue;                                  /* <- never received a first EOF; can't judge yet */
    if (LSA_GT (&proc->curr_eof, &proc->prev_eof) == true)
      LSA_COPY (&proc->prev_eof, &proc->curr_eof); /* <- advanced: healthy, slide window forward */
    else
      {
        proc->server_hang = true;                /* <- did NOT advance: declare this server wedged */
        dead_cnt++;
        if (proc->is_curr_eof_received) { /* syslog "no change to eof" */ }
        else                            { /* syslog "no response to eof request" */ }
      }
  }
if (dead_cnt > 0)
  return false;                                  /* <- at least one server wedged -> caller demotes */
return true;

The is_curr_eof_received sub-branch is cosmetic only — it changes the syslog wording, not the verdict. One wedged server makes dead_cnt > 0 and returns false.

Invariant — prev_eof only ever moves forward, never back. The LSA_GT / LSA_COPY pair treats an out-of-order older LSA as “did not advance” (false-wedge at worst), never a regression. A backward-moving prev_eof would let a briefly-lagging server reset the bar and mask a real future wedge.

8.3 Resource-side teardown: start shutdown then confirm

HB_RJOB_DEMOTE_START_SHUTDOWN runs hb_resource_job_demote_start_shutdown: ask each server to stop, then schedule a confirmation job.

// hb_resource_job_demote_start_shutdown -- src/executables/master_heartbeat.c
hb_resource_demote_start_shutdown_server_proc ();      /* under anchor+resource lock */
job_arg = malloc (sizeof (HB_JOB_ARG));
if (job_arg == NULL) { /* css_master_cleanup(SIGTERM); return; */ } /* <- can't continue */
proc_arg = &(job_arg->resource_job_arg);         /* <- confirm-job arg lives inside the fresh job_arg */
proc_arg->retries = 0;
proc_arg->max_retries = prm_get_integer_value (PRM_ID_HA_MAX_PROCESS_DEREG_CONFIRM);
error = hb_resource_job_queue (HB_RJOB_DEMOTE_CONFIRM_SHUTDOWN, job_arg, /* interval */);

hb_resource_demote_start_shutdown_server_proc issues the per-server shutdown — three branches: skip non-active, SIGKILL the wedged one, ask the rest gracefully.

// hb_resource_demote_start_shutdown_server_proc -- src/executables/master_heartbeat.c
for (proc = hb_Resource->procs; proc; proc = proc->next)
  {
    if (proc->state != HB_PSTATE_REGISTERED_AND_TO_BE_ACTIVE
        && proc->state != HB_PSTATE_REGISTERED_AND_ACTIVE)
      continue;                                  /* <- leave non-active processes alone */
    if (proc->server_hang)
      {                                          /* <- the wedged one: don't ask politely */
        if (proc->pid > 0 && (kill (proc->pid, 0) == 0 || errno != ESRCH))
          kill (proc->pid, SIGKILL);             /* <- SIGKILL; can't drain. kill(,0) guards recycled PID */
        continue;
      }
    sock_entq = css_return_entry_by_conn (proc->conn, &css_Master_socket_anchor);
    if (sock_entq != NULL && sock_entq->name != NULL)
      {
        css_process_start_shutdown (sock_entq, 0, buffer); /* <- graceful request */
        proc->being_shutdown = true;             /* <- mark so confirm knows we asked */
      }
  }

hb_resource_job_demote_confirm_shutdown then polls with a retry cap.

// hb_resource_job_demote_confirm_shutdown -- src/executables/master_heartbeat.c
if (++(proc_arg->retries) > proc_arg->max_retries)
  {
    hb_resource_demote_kill_server_proc ();      /* <- patience exhausted: force-kill survivors */
    goto demote_confirm_shutdown_end;
  }
if (hb_resource_demote_confirm_shutdown_server_proc () == false)
  {                                              /* <- some server still active: requeue and wait */
    error = hb_resource_job_queue (HB_RJOB_DEMOTE_CONFIRM_SHUTDOWN, arg, /* ... interval ... */);
    return;                                       /* <- NOTE: keeps arg; do not free on requeue */
  }
demote_confirm_shutdown_end:
  job_arg = malloc (sizeof (HB_JOB_ARG));
  if (job_arg == NULL) { /* free arg; css_master_cleanup(SIGTERM); return; */ }
  clst_arg = &(job_arg->cluster_job_arg);         /* <- the handoff arg is the FRESH job_arg, not the inbound arg */
  clst_arg->ping_check_count = 0;
  clst_arg->retries = 0;                          /* <- retries==0 -> 8.4 takes its first-pass branch */
  error = hb_cluster_job_queue (HB_CJOB_DEMOTE, job_arg, HB_JOB_TIMER_IMMEDIATELY);
  /* free arg */

hb_resource_demote_confirm_shutdown_server_proc skips the hung server (already SIGKILL-ed) and returns true only when no other active/to-be-active server remains; hb_resource_demote_kill_server_proc SIGKILLs any survivor on the patience-exhausted path. The cluster handoff is built on a fresh job_arg with retries reset to 0, so 8.4 enters its first-pass branch; the inbound arg is freed after the queue. The requeue path retains the same arg and returns without freeing; a failed handoff malloc calls css_master_cleanup (SIGTERM) — no safe way to continue.

Invariant — arg is freed exactly once. On requeue it is retained (passed straight back into the queue); on every terminating branch it is freed. Freeing on requeue would be a use-after-free; not freeing on a terminating branch would leak.

8.4 Cluster-side step-aside: hb_cluster_job_demote

The heart of the chapter: the job advertises that the node is leaving the master role, drops to SLAVE, then waits for a peer to claim master, re-queuing once per second under hide_to_demote.

// hb_cluster_job_demote -- src/executables/master_heartbeat.c
/* lock cluster */
if (clst_arg->retries == 0)                        /* <- first invocation only */
  {
    assert (hb_Cluster->state == HB_NSTATE_MASTER);
    assert (hb_Resource->state == HB_NSTATE_SLAVE); /* <- resource already demoted in 8.1 */
    hb_Cluster->state = HB_NSTATE_UNKNOWN;          /* <- "I'm not master" so a peer steps up */
    hb_cluster_request_heartbeat_to_all ();
    MASTER_ER_SET (... "Waiting for a new node to be elected as master");
  }
hb_Cluster->hide_to_demote = true;                  /* <- silence broadcast + score EVERY iteration */
hb_Cluster->state = HB_NSTATE_SLAVE;
hb_Cluster->myself->state = hb_Cluster->state;
if (hb_Cluster->is_isolated == true || ++(clst_arg->retries) > HB_MAX_WAIT_FOR_NEW_MASTER)
  {                                                 /* <- give up: isolated OR waited 60 iterations */
    MASTER_ER_SET (... "Failed to find a new master node and it changes its role back to master again");
    hb_Cluster->hide_to_demote = false;             /* <- re-enable participation; score may re-promote */
    /* unlock; free arg; return */
  }
for (node = hb_Cluster->nodes; node; node = node->next)
  if (node->state == HB_NSTATE_MASTER)              /* <- a peer took over */
    {
      assert (node != hb_Cluster->myself);
      hb_Cluster->hide_to_demote = false;           /* <- clean exit: we are now a real slave */
      MASTER_ER_SET (... HA_FAILBACK_SUCCESS_STRING " ... successfully demoted to slave");
      /* unlock; free arg; return */
    }
/* unlock */
error = hb_cluster_job_queue (HB_CJOB_DEMOTE, arg, HB_JOB_TIMER_WAIT_A_SECOND); /* <- 1s, try again */

Branch inventory: a malformed job (arg/clst_arg == NULL) logs and returns. The first pass (retries == 0) asserts MASTER/SLAVE, sets UNKNOWN, and broadcasts so peers race to take over. Every pass then sets hide_to_demote = true and forces SLAVE — re-asserted each time because the job drops the lock and sleeps a second between runs, so a concurrent hb_cluster_job_calc_score never sees a SLAVE that still scores. The two terminating branches are mutually exclusive and both clear the flag and free arg: expiry (is_isolated, or ++retries > HB_MAX_WAIT_FOR_NEW_MASTER = 60) and peer-found (some node->state == MASTER). Otherwise requeue with HB_JOB_TIMER_WAIT_A_SECOND (1000 ms).

Invariant — exactly one of hide_to_demote’s two clear sites runs per demote. While set, the flag suppresses score participation (hb_cluster_job_calc_score, Chapter 5), the periodic-broadcast gate (hb_cluster_job_heartbeat), and the split-brain response in the receive path — the node is invisible to its own election machinery. Left set forever it would be a silent slave that never participates again; cleared too early it could re-elect itself mid-demote and create the very contention this path prevents.

Figure 8-1 — the wait-for-new-master loop.

stateDiagram-v2
  [*] --> First: retries==0
  First --> Waiting: UNKNOWN broadcast; hide_to_demote=true; SLAVE
  Waiting --> Expired: is_isolated OR retries GT 60
  Waiting --> PeerFound: some node.state==MASTER
  Waiting --> Waiting: requeue 1s, retries++
  Expired --> [*]: flag cleared, back to master via score
  PeerFound --> [*]: flag cleared, settled as SLAVE

8.5 The expiry branch and its open question

The expiry branch only clears hide_to_demote and sets SLAVE; the actual re-promotion is left to the next hb_cluster_job_calc_score pass, which — flag clear, no peer master — elects this node master again.

Open question (flagged for the maintainer). Expiry restores the cluster role toward master but does not revive the resource side: the 8.3 teardown has already killed or stopped the local server, so the node can become cluster-master again with no running server. Whether that is intended (operator intervenes) or a gap turns on Chapter 9’s process-revival logic, not resolvable from hb_cluster_job_demote alone.

8.6 Chapter summary — key takeaways

1. Two contexts, one demote. The disk-failure thread detects the wedge and tears down the resource side; hb_cluster_job_demote then steps the node aside — chained by HB_RJOB_DEMOTE_START_SHUTDOWN -> _CONFIRM_SHUTDOWN -> HB_CJOB_DEMOTE.
2. The wedge test is a forward-only EOF comparison. A server is hung when curr_eof fails to exceed the monotonic prev_eof, so a lagging reply never resets the baseline.
3. The detector guard is narrow: only a non-isolated MASTER demotes; the SLAVE flip happens under all three locks, and continue guarantees exactly one teardown per wedge.
4. Teardown is graceful-then-forceful. Active servers get css_process_start_shutdown, the hung one is SIGKILL-ed at once, and confirm force-kills survivors after max_retries.
5. hide_to_demote makes the step-aside safe. While set it silences broadcast, the split-brain response, and score participation, so the node cannot re-elect itself; re-asserted every iteration, cleared on exactly one of two branches.
6. The wait is bounded at HB_MAX_WAIT_FOR_NEW_MASTER (60) one-second iterations: peer-found ends it cleanly; expiry/isolation clears the flag and lets score re-promote.
7. Open question: the expiry branch can return the node to master without a running server. Confirm Chapter 9’s process-revival first.

Chapter 9: The Resource Side and Process Supervision

The cluster-side FSM of Chapters 4-8 decides what role this node should play; it never forks, kills, or talks to a cub_server. That lives on the resource side: a second job queue (resource_Jobs, Ch.3) whose worker drains HB_RJOB_* jobs and drives a per-process FSM. This chapter answers: how does cub_master actually start, register, mode-switch, and stop the cub_server, copylogdb, and applylogdb processes a node-state change drives?

The subsystem hangs off the global hb_Resource (HB_RESOURCE, Ch.1): a mutex, a node state mirror, num_procs, and a singly-walked list procs of HB_PROC_ENTRY. Every field of that slot is load-bearing on the supervision path:

Field	Role	Why
next / prev	HB_LIST links	procs is a doubly-linked list; hb_alloc_new_proc prepends, hb_remove_proc unlinks
state	HB_PROC_STATE FSM cursor	the trust/role spine of §9.1; every job branches on it
type	HB_PROC_TYPE (SERVER/COPYLOGDB/APPLYLOGDB)	servers take the socket-shutdown and change-mode paths; logwriters take SIGTERM
sfd	socket fd	key for hb_return_proc_by_fd; reset to INVALID_SOCKET on conn loss
pid	child pid	key for hb_return_proc_by_pid; target of kill liveness probes and signals
exec_path	binary to execv	the program a restart re-launches; written once on first register
args	server name + log path	the identity key (hb_return_proc_by_args); survives crash/restart
frtime	first-registered time	gates the HB_PROC_RECOVERY_DELAY_TIME young-restart throttle (§9.4)
rtime	last-registered time	start of the flapping window measured in §9.7
dtime	deregistered time	bookkeeping for the dereg path
ktime	shutdown/kill-detected time	end of the flapping window (§9.7 ktime - rtime)
stime	start time	stamped by the parent after fork (§9.4)
changemode_rid	reserved request id	declared, never read or written — vestigial (Cross-check Notes)
changemode_gap	missed-changemode counter	read to escalate SIGTERM/SIGKILL (§9.6); never incremented on this tree
prev_eof / curr_eof	replication EOF LSAs	progress tracking for log processes; reset on restart (§9.7)
is_curr_eof_received	EOF-seen flag	paired with the EOF LSAs; reset on restart
conn	CSS_CONN_ENTRY *	the live socket connection; NULL once the conn drops
being_shutdown	graceful-teardown flag	makes proc_start wait for the old instance to die (§9.4)
server_hang	hang-detected flag	lets the demote path SIGKILL immediately; reset on register/restart

Shared defensive guards (not repeated per function below). Each resource job (hb_resource_job_proc_start, _proc_dereg, _confirm_start, _confirm_dereg) opens with if (arg == NULL || proc_arg == NULL) return;. hb_resource_job_change_mode takes a NULL arg by design and has no such guard. Each handler (hb_register_new_process, hb_is_registered_process, hb_resource_receive_changemode, hb_cleanup_conn_and_start_process) opens with if (hb_Resource == NULL) return ...; before taking hb_Resource->lock.

9.1 The process FSM and its lookups

HB_PROC_ENTRY.state (HB_PROC_STATE) is the spine. The enum is ordered so HB_PSTATE_REGISTERED (5) is the trust boundary: >= 5 is supervised, below is transient. HB_PSTATE_REGISTERED_AND_STANDBY aliases HB_PSTATE_REGISTERED (both 5) — a freshly-registered server is standby by definition. Transient values in order: DEAD (1), DEREGISTERED (2), STARTED (3, forked+execv’d but child not yet re-registered), NOT_REGISTERED (4, restart child reporting back, unconfirmed). Above the boundary: REGISTERED_AND_TO_BE_STANDBY (6), REGISTERED_AND_ACTIVE (7), REGISTERED_AND_TO_BE_ACTIVE (8).

Three lookup helpers index procs by a different key each; all share one “lock-held, linear walk, first match wins” loop, differing only in the comparison: hb_return_proc_by_args matches strcmp (proc->args, args) == 0, hb_return_proc_by_pid matches proc->pid == pid, hb_return_proc_by_fd matches proc->sfd == sfd; each returns the matching HB_PROC_ENTRY * or NULL. args (server name + log path) survives crash-and-restart, so start/confirm/register look up by args and a restart reuses the same slot; pid serves the dereg path (operator named a pid); sfd serves socket-event paths (kernel hands us only the fd).

Invariant — args uniquely identifies a live slot. Enforced by hb_register_new_process looking up by args before allocating, then reusing the found entry. If two slots shared args, a restart would re-register the wrong one and num_procs would drift.

hb_is_registered_process is the read-only probe the connection layer uses: take the lock, return false if hb_Resource->shutdown is set, else return proc != NULL from a by-args lookup.

9.2 Registration — hb_register_new_process

The HBP_PROC_REGISTER handler. A process sends a fixed-layout register packet (pid, type, exec_path, args); the master creates a fresh slot or re-attaches an existing slot to the new connection. After the shutdown check (tear down the conn if set), it looks up by args and picks an intent proc_state: not found -> allocate via hb_alloc_new_proc (or drop conn on OOM), set HB_PSTATE_REGISTERED, stamp frtime; found in HB_PSTATE_STARTED -> HB_PSTATE_NOT_REGISTERED (a restart re-attach); found in any other state -> HB_PSTATE_UNKNOWN (an imposter claiming live args).

proc_state	Trigger	Meaning
HB_PSTATE_REGISTERED	slot not found	first ever registration; frtime stamped
HB_PSTATE_NOT_REGISTERED	found HB_PSTATE_STARTED	master just forked this slot (§9.4); child reports back
HB_PSTATE_UNKNOWN	found any other state	a process claims args already wired to a live conn — failure

The acceptance gate admits exactly two cases — a genuine first register, or a restart whose reported pid both matches the pid we forked and is still alive:

// hb_register_new_process (acceptance gate) -- src/executables/master_heartbeat.c
  if ((proc_state == HB_PSTATE_REGISTERED)
      || (proc_state == HB_PSTATE_NOT_REGISTERED
          && proc->pid == (int) ntohl (hbp_proc_register->pid)
          && !(kill (proc->pid, 0) && errno == ESRCH)))      /* <- pid alive check */
    {
      proc->state = proc_state; proc->sfd = conn->fd; proc->conn = conn;
      gettimeofday (&proc->rtime, NULL);
      proc->changemode_gap = 0; proc->server_hang = false;
      if (proc->state == HB_PSTATE_REGISTERED)               /* first-register only */
        { proc->pid = ntohl (...); proc->type = ntohl (...);
          if (proc->type == HB_PTYPE_SERVER) proc->state = HB_PSTATE_REGISTERED_AND_STANDBY;
          memcpy (proc->exec_path, ...); memcpy (proc->args, ...);
          hb_Resource->num_procs++; }                        /* <- bumped ONLY on first register */
      ...; return;
    }
  ...; css_remove_entry_by_conn (conn, ...);                 /* else: log FAILURE, tear down */

pid/type/exec_path/args/num_procs are written only on first register; a re-attach already owns those and only refreshes sfd/conn/rtime/changemode_gap/server_hang. The else fall-through (imposter, or stale/dead pid) tears down the connection.

Invariant — num_procs counts first-registrations, not re-attachments. Incremented in exactly one place (this branch) and decremented in exactly two (_proc_dereg, _confirm_dereg, §9.5). A re-attach must not bump it or the count diverges from the list.

9.3 The lifecycle FSM as a whole

stateDiagram-v2
  [*] --> STARTED: proc_start fork+execv
  STARTED --> NOT_REGISTERED: child re-registers \n hb_register_new_process
  NOT_REGISTERED --> REGISTERED: confirm_start, non-server
  NOT_REGISTERED --> REGISTERED_AND_STANDBY: confirm_start, server
  [*] --> REGISTERED: first register, non-server
  [*] --> REGISTERED_AND_STANDBY: first register, server
  REGISTERED_AND_STANDBY --> REGISTERED_AND_TO_BE_ACTIVE: changemode resp TO_BE_ACTIVE
  REGISTERED_AND_TO_BE_ACTIVE --> REGISTERED_AND_ACTIVE: changemode resp ACTIVE
  REGISTERED_AND_ACTIVE --> REGISTERED_AND_TO_BE_STANDBY: demote, changemode resp
  REGISTERED_AND_TO_BE_STANDBY --> REGISTERED_AND_STANDBY: changemode resp STANDBY
  REGISTERED_AND_STANDBY --> DEAD: conn lost, cleanup
  DEAD --> STARTED: proc_start restart
  REGISTERED_AND_STANDBY --> DEREGISTERED: operator dereg
  DEREGISTERED --> [*]: confirm_dereg removes slot

Figure 9-1: the HB_PROC_STATE lifecycle. Left column (start/restart) §9.4; right (mode bridge) §9.6; bottom (dereg) §9.5.

9.4 Starting and confirming — hb_resource_job_proc_start / _confirm_start

HB_RJOB_PROC_START is queued whenever a supervised process must be (re)launched — most often by hb_cleanup_conn_and_start_process (§9.7) after a crash. The job arg carries args, pid, retry counters, ftime. After looking up by args, three pre-fork branches fire: (1) slot gone or HB_PSTATE_DEREGISTERED -> free_and_init(arg) and abort; (2) proc->being_shutdown -> if the old pid is gone (pid <= 0 or kill(pid,0) gives ESRCH) clear the flag and proceed, else re-queue PROC_START in 1 s (old instance still dying); (3) HB_GET_ELAPSED_TIME(now, proc->frtime) < HB_PROC_RECOVERY_DELAY_TIME (30 000 ms) -> re-queue in 1 s, throttling a process that flaps right after boot. Past the guards comes the fork:

// hb_resource_job_proc_start (fork) -- src/executables/master_heartbeat.c
  args = strdup (proc->args); hb_proc_make_arg (argv, args);   /* tokenise args into argv[] */
  pid = fork ();
  if (pid < 0) { hb_resource_job_queue (HB_RJOB_PROC_START, arg, ...); free (args); return; }  /* retry */
  else if (pid == 0)                                          /* child */
    { error = execv (proc->exec_path, argv);                  /* no return on success */
      ...; css_master_cleanup (SIGTERM); return; }            /* execv FAILED: must die, not run on */
  else                                                        /* parent */
    { proc->pid = pid; proc->state = HB_PSTATE_STARTED;       /* awaits re-register */
      gettimeofday (&proc->stime, NULL); free (args); }
  hb_resource_job_queue (HB_RJOB_CONFIRM_START, arg, prm_get_integer_value (PRM_ID_HA_PROCESS_START_CONFIRM_INTERVAL_IN_MSECS));

Four branches: fork failure (retry), child (execv or die), parent (STARTED), then queue confirm.

Invariant — a child that fails to execv must die. Post-fork the child shares the parent’s master role; execv returning means it failed, so the child calls css_master_cleanup (SIGTERM) rather than fall through, preventing two processes both believing they are cub_master.

hb_proc_make_arg splits args on " \t\n" with strtok_r into successive argv[] slots; no bounds check, no terminating NULL of its own — the execv terminator is the caller’s zero-initialised argv[HB_MAX_NUM_PROC_ARGV]. It mutates args in place, hence the strdup’d copy.

hb_resource_job_confirm_start verifies the child re-registered and is alive, retrying up to max_retries. After the gone-slot abort it walks five branches:

1. retries exhausted, master + server + not isolated — a master cannot run without its cub_server, so it self-demotes: hb_Resource->state = HB_NSTATE_SLAVE, reset retries, re-queue CONFIRM_START, queue HB_RJOB_DEMOTE_START_SHUTDOWN immediately (Ch.8), return.
2. retries exhausted otherwise — reset retries, re-queue CONFIRM_START, keep watching.
3. kill(proc->pid, 0) fails with ESRCH — pid gone, re-queue PROC_START to relaunch.
4. kill fails otherwise — re-queue CONFIRM_START, keep watching.
5. alive and proc->state == HB_PSTATE_NOT_REGISTERED — the child re-registered (§9.2); promote to REGISTERED_AND_STANDBY (server) or REGISTERED (logwriter), set retry = false; if still retrying re-queue, else free_and_init(arg).

Branch 1 is load-bearing: it is the resource-side trigger for the demote path Ch.8 owns.

9.5 Deregistering — hb_resource_job_proc_dereg / _confirm_dereg

Operator-driven teardown (Ch.10 bridges the command in). The slot is set to HB_PSTATE_DEREGISTERED before these jobs run; both refuse any other state. _proc_dereg takes two locks (css_Master_socket_anchor_lock then hb_Resource->lock) because a server must reach into the socket layer. Its branches after the by-pid lookup: unknown pid -> free+return; state != DEREGISTERED -> free+return; server -> fetch the socket entry, and if it has none goto the confirm queue, else send css_process_start_shutdown (graceful) then queue CONFIRM_DEREG; non-server already dead (pid <= 0 or kill(pid, SIGTERM) gives ESRCH) -> decrement num_procs, hb_remove_proc, free, return with no confirm job; non-server alive -> the SIGTERM just sent lands, fall through to queue CONFIRM_DEREG.

// hb_resource_job_proc_dereg (server vs logwriter) -- src/executables/master_heartbeat.c
  if (proc->type == HB_PTYPE_SERVER)
    { sock_entq = css_return_entry_by_conn (proc->conn, &css_Master_socket_anchor);
      if (sock_entq == NULL || sock_entq->name == NULL) goto hb_resource_job_proc_dereg_end;
      css_process_start_shutdown (sock_entq, 0, buffer); }   /* graceful server shutdown msg */
  else
    { if (proc->pid <= 0 || (kill (proc->pid, SIGTERM) && errno == ESRCH))
        { hb_Resource->num_procs--; hb_remove_proc (proc); proc = NULL;   /* already gone: remove now */
          free_and_init (arg); return; } }                   /* NO confirm job */
hb_resource_job_proc_dereg_end:
  hb_resource_job_queue (HB_RJOB_CONFIRM_DEREG, arg, prm_get_integer_value (PRM_ID_HA_PROCESS_DEREG_CONFIRM_INTERVAL_IN_MSECS));

hb_resource_job_confirm_dereg is the escalation loop. After the same unknown-pid / wrong-state aborts: if kill(proc->pid, 0) fails with ESRCH the process is gone (retry = false); else if ++retries > max_retries it escalates with kill(proc->pid, SIGKILL) and stops retrying; otherwise it re-queues CONFIRM_DEREG. When retry is cleared it decrements num_procs, hb_remove_procs the slot, and frees the arg. So SIGTERM (in _proc_dereg) then SIGKILL (here, after max_retries) gives a bounded clean-exit window before force-kill. hb_remove_proc (hb_list_remove + free_and_init) is the only place a slot leaves the list outside startup teardown.

Invariant — a slot is freed exactly once, with num_procs decremented alongside. Both removal sites pair num_procs-- with hb_remove_proc(proc); proc = NULL; the proc = NULL guards against use-after-free.

9.6 The mode bridge — change-mode send and receive

How a node-state decision reaches a running cub_server. hb_resource_job_change_mode (HB_RJOB_CHANGE_MODE, self-perpetuating periodic) walks every procs entry, skips non-HB_PTYPE_SERVER, and for a server whose role lags the node role calls hb_resource_send_changemode, then re-queues itself after PRM_ID_HA_CHANGEMODE_INTERVAL_IN_MSECS. The lag test: a MASTER node nudges any server still REGISTERED_AND_STANDBY or REGISTERED_AND_TO_BE_ACTIVE; a TO_BE_SLAVE node nudges any server still REGISTERED_AND_ACTIVE or REGISTERED_AND_TO_BE_STANDBY. Other node states match nothing and send nothing. Because the job re-arms, the nudge repeats every HA_CHANGEMODE_INTERVAL until the server confirms.

hb_resource_send_changemode is one tick: it reads changemode_gap to decide whether the server has gone deaf, then sends the request:

// hb_resource_send_changemode -- src/executables/master_heartbeat.c
  if (proc->conn == NULL) return ER_FAILED;
  if (proc->changemode_gap == HB_MAX_CHANGEMODE_DIFF_TO_TERM)      sig = SIGTERM;  /* 12 */
  else if (proc->changemode_gap >= HB_MAX_CHANGEMODE_DIFF_TO_KILL) sig = SIGKILL;  /* 24 */
  if (sig)                                                   /* server ignored us too many times */
    { if (proc->pid > 0 && (kill (proc->pid, 0) == 0 || errno != ESRCH)) kill (proc->pid, sig);
      return ER_FAILED; }
  switch (hb_Resource->state)                                /* pick target state */
    { case HB_NSTATE_MASTER:      state = HA_SERVER_STATE_ACTIVE;  break;
      case HB_NSTATE_TO_BE_SLAVE: state = HA_SERVER_STATE_STANDBY; break;
      default:                    return ER_FAILED; }        /* SLAVE etc: nothing to request */
  css_send_heartbeat_request (proc->conn, SERVER_CHANGE_HA_MODE);   /* + htonl(state) data; errors -> ER_FAILED */

changemode_gap is reset to 0 on register (§9.2) and on every changemode reply, and read here to escalate at == 12 / >= 24. No code path in this revision increments it (Cross-check Notes), so it is pinned at 0 and sig never fires — the send path always runs: pick the target HA_SERVER_STATE from the node state and send SERVER_CHANGE_HA_MODE plus the network-order state word, returning ER_FAILED on any conn/send failure or on a node state with nothing to request.

hb_resource_receive_changemode is the SERVER_CHANGE_HA_MODE response handler — it reads the server’s actual HA_SERVER_STATE off the wire, looks the proc up by socket fd (hb_return_proc_by_fd(sfd), not by request id), aborts if the proc is gone or DEREGISTERED, then writes the mapped HB_PSTATE_* into proc->state, taking both hb_Cluster->lock and hb_Resource->lock. The switch maps ACTIVE -> REGISTERED_AND_ACTIVE, TO_BE_ACTIVE -> REGISTERED_AND_TO_BE_ACTIVE, TO_BE_STANDBY -> REGISTERED_AND_TO_BE_STANDBY, and STANDBY -> REGISTERED_AND_STANDBY plus forcing both hb_Cluster->state and hb_Resource->state to HB_NSTATE_SLAVE (the server confirming a demote drags the node to slave). All cases then reset proc->changemode_gap = 0. This is where TO_BE_ACTIVE finalises to ACTIVE: the master keeps sending ACTIVE (§9.6 nudge), the server passes through TO_BE_ACTIVE and finally reports ACTIVE, and only then does the slot reach REGISTERED_AND_ACTIVE.

9.7 Connection loss and restart — hb_cleanup_conn_and_start_process

When a supervised process’s socket closes, the connection layer calls this (non-static, exported). It bridges “a process died” to “queue a restart” and owns the demote-on-repeated-failure decision Ch.8 references:

// hb_cleanup_conn_and_start_process -- src/executables/master_heartbeat.c
  css_remove_entry_by_conn (conn, &css_Master_socket_anchor);
  if (hb_Resource == NULL) return;
  proc = hb_return_proc_by_fd (sfd);
  if (proc == NULL) return;                                  /* fd not ours */
  proc->conn = NULL; proc->sfd = INVALID_SOCKET;
  if (proc->state < HB_PSTATE_REGISTERED) return;            /* never fully registered: leave for a job to reap */
  gettimeofday (&proc->ktime, NULL);
  if (hb_Resource->state == HB_NSTATE_MASTER && proc->type == HB_PTYPE_SERVER
      && hb_Cluster->is_isolated == false
      && HB_GET_ELAPSED_TIME (proc->ktime, proc->rtime)
         < prm_get_integer_value (PRM_ID_HA_UNACCEPTABLE_PROC_RESTART_TIMEDIFF_IN_MSECS))
    { hb_Resource->state = HB_NSTATE_SLAVE;                  /* died too soon after register: flapping -> demote */
      hb_resource_job_queue (HB_RJOB_DEMOTE_START_SHUTDOWN, NULL, HB_JOB_TIMER_IMMEDIATELY); }
  job_arg = (HB_JOB_ARG *) malloc (sizeof (HB_JOB_ARG));
  if (job_arg == NULL) return;                               /* OOM: no restart queued */
  proc_arg->pid = proc->pid; memcpy (proc_arg->args, proc->args, ...);   /* identity for restart lookup */
  proc_arg->retries = 0; proc_arg->max_retries = prm_get_integer_value (PRM_ID_HA_MAX_PROCESS_START_CONFIRM);
  proc->state = HB_PSTATE_DEAD;                              /* mark dead; clears for restart */
  proc->server_hang = false; proc->is_curr_eof_received = false; LSA_SET_NULL (&proc->prev_eof); ...
  hb_resource_job_queue (HB_RJOB_PROC_START, job_arg, HB_JOB_TIMER_WAIT_A_SECOND);

Branches: not-our-fd (return), not-yet-REGISTERED (clear conn, leave the slot for a job to reap), job_arg OOM (no restart), fully-registered (main path). The flapping guard — distinct from the §9.4 frtime/30 s guard — measures ktime - rtime: if a master’s server died sooner than HA_UNACCEPTABLE_PROC_RESTART_TIMEDIFF after its last register, the master demotes (Ch.8). Otherwise it builds a restart HB_JOB_ARG keyed on args, marks the slot HB_PSTATE_DEAD, resets EOF/hang bookkeeping, and queues HB_RJOB_PROC_START.

This closes the loop: §9.7 detects death and queues a start; §9.4 forks and confirms; §9.2 re-attaches on re-register; §9.6 mode-switches it back to the role the cluster FSM (Ch.5-8) wants.

Cross-check Notes

• changemode_gap is never incremented in this source revision. Repo-wide the field has exactly two write sites — hb_register_new_process and hb_resource_receive_changemode, both setting it to 0 — and two read sites in hb_resource_send_changemode. With no increment anywhere, the value stays 0, so the SIGTERM-at-12 / SIGKILL-at-24 escalation cannot fire as written; the protocol-silence watchdog is dead code on this tree.
• changemode_rid is declared but unused. Reserved in HB_PROC_ENTRY (master_heartbeat.h) for a changemode request id, but no code reads or writes it — replies are correlated purely by socket fd via hb_return_proc_by_fd. Vestigial.

Open Questions

• Where was changemode_gap meant to be incremented? The thresholds and read sites imply a per-tick increment in hb_resource_job_change_mode or hb_resource_send_changemode that was removed or never landed on this branch. Whether the live watchdog exists in an older/sibling release is unresolved.
• Was changemode_rid intended to replace the fd-based correlation? It would matter only if change-mode replies become asynchronous or multiplexed; today it is dead weight.

9.8 Chapter summary — key takeaways

1. args is identity, pid/sfd are handles. hb_return_proc_by_args makes a restart reuse its slot; _by_pid and _by_fd serve dereg and socket-event paths, preventing leaked or duplicated HB_PROC_ENTRY slots.
2. hb_register_new_process has exactly two acceptance cases — first register (allocate, stamp frtime, bump num_procs, servers enter STANDBY) and a restart re-attach whose pid matches and is alive. num_procs is bumped only on first register; everything else tears down the connection.
3. Start is fork+execv with two throttles. HB_PROC_RECOVERY_DELAY_TIME (30 s, from frtime) gates young restarts; the child must css_master_cleanup on execv failure so no half-formed master clone survives. _confirm_start retries to max_retries, and a master that cannot keep its server up self-demotes (Ch.8).
4. Dereg escalates SIGTERM then SIGKILL. _proc_dereg sends a graceful shutdown to servers / SIGTERM to logwriters; _confirm_dereg force-SIGKILLs after max_retries. Both removal sites pair num_procs-- with hb_remove_proc.
5. The mode bridge is a periodic nudge plus a response handler. hb_resource_job_change_mode re-arms and calls hb_resource_send_changemode for every server whose role lags the node role; hb_resource_receive_changemode writes the reported HA_SERVER_STATE back into proc->state, finalising TO_BE_ACTIVE to ACTIVE and dragging the node to SLAVE when the server confirms STANDBY.
6. The changemode_gap watchdog is vestigial in this revision. hb_resource_send_changemode reads it to escalate to SIGTERM (12) / SIGKILL (24), but no code path increments it, so it stays 0 and the escalation never fires. changemode_rid is likewise declared but unused — replies correlate by socket fd. Both are recorded as drift in Cross-check Notes.
7. hb_cleanup_conn_and_start_process is the death-to-restart bridge. It demotes a master whose server died within HA_UNACCEPTABLE_PROC_RESTART_TIMEDIFF of its last register (flapping), otherwise marks the slot HB_PSTATE_DEAD and queues HB_RJOB_PROC_START keyed on args, closing the supervision loop.

Chapter 10: Operator Commands Process Bridge and Shutdown Paths

Everything that is not the steady-state lifecycle: how an operator drives the feature on/off, how a managed process attaches to cub_master and learns the master died, and how every layer tears down. Four actors:

• cubrid hb … (util_service.c) — CLI front end; never talks to the master, spawns cub_commdb per sub-step.
• cub_commdb (commdb.c) — short-lived courier: one connection, one opcode, print reply, exit.
• cub_master request loop (master_request.c) — maps the opcode to hb_* entry points in master_heartbeat.c.
• the managed process (connection/heartbeat.c, CS_MODE library) — a reader thread that self-terminates when the master connection drops.

Theory and the steady-state state machine live in the companion cubrid-heartbeat.md; this chapter assumes the node states (HB_NSTATE_*), process states (HB_PSTATE_*), and the job queue (Chapter 3).

10.1 The operator surface: process_heartbeat

cubrid heartbeat <verb> (alias hb) lands in process_heartbeat: a guard plus a switch (command_type). Two early returns gate everything — ha_mode_in_common == HA_MODE_OFF returns ER_FAILED (“not HA”), and a failed util_make_ha_conf returns ER_FAILED (parse fail) — before the switch dispatches each command_type to its handler. The operator-facing arms are START/STOP/DEREGISTER/STATUS/LIST/RELOAD -> their process_heartbeat_* handler; the same switch also carries internal arms (SC_COPYLOGDB/SC_APPLYLOGDB -> process_heartbeat_util, REPLICATION -> process_heartbeat_replication) out of scope for this chapter. default yields ER_GENERIC_ERROR; on Windows the whole body #else-compiles to ER_FAILED (POSIX-only). Each non-start verb checks css_does_master_exist, builds a const char *args[], and proc_execute (UTIL_COMMDB_NAME, …). status fires 3–4 commdb spawns in sequence (node, process, ping-host, verbose admin); deregister rejects argc < 1 -> COMMDB_HA_DEREG_BY_PID; reload carries no payload (COMMDB_HA_RELOAD).

flowchart TD
  cli["cubrid hb <verb>"] --> ph["process_heartbeat<br/>guard + switch"]
  ph -->|"STOP"| pstop["process_heartbeat_stop"]
  ph -->|"STATUS/LIST"| pstat["process_heartbeat_status"]
  ph -->|"RELOAD"| prel["process_heartbeat_reload"]
  ph -->|"DEREGISTER"| pdereg["process_heartbeat_deregister"]
  pstop --> ushd["us_hb_deactivate<br/>4-phase orchestration"]
  ushd -.->|"proc_execute"| commdb["cub_commdb<br/>(one opcode per spawn)"]
  pstat -.->|"x3-4"| commdb
  prel -.-> commdb
  pdereg -.-> commdb
  commdb -.->|"TCP request"| master["cub_master request loop"]

Figure 10-1. The operator front end never touches the master directly; cub_commdb is the one-shot courier.

10.2 The deactivate orchestration: a four-phase handshake

cubrid hb stop (no -i <dbname>) deactivates the whole feature. us_hb_deactivate reuses one argv slot and rewrites the trailing opcode four times, spawning cub_commdb per phase.

// us_hb_deactivate -- src/executables/util_service.c
  if (immediate_stop) args[opt_idx++] = COMMDB_HB_DEACT_IMMEDIATELY;     /* <- BEFORE opt_idx frozen */
  args[opt_idx]=DEACT_STOP_ALL;          if (proc_execute(...)) return;                  /* ph1 */
  args[opt_idx]=DEACT_CONFIRM_STOP_ALL;  while (proc_execute(...)) { if(==EXIT_FAILURE) return; sleep(1);} /* ph2 poll */
  args[opt_idx]=DEACTIVATE;              if (proc_execute(...)) return;                  /* ph3 */
  args[opt_idx]=DEACT_CONFIRM_NO_SERVER; while (proc_execute(...)) { if(==EXIT_FAILURE) return; sleep(1);} /* ph4 poll */

The -i flag is inserted at args[opt_idx++] before opt_idx is frozen, so it stays in the vector for all four phases (only the master’s STOP_ALL handler reads it; harmless on 2–4, where only args[opt_idx] is rewritten).

Invariant — deactivation is a confirm-loop, not fire-and-forget. Phases 2 and 4 busy-poll until success, distinguishing EXIT_FAILURE (abort) from any other non-zero (not-yet: sleep(1), retry); drop that and a transient “still draining” reply aborts the whole stop, leaving the cluster half-torn-down. These loops are why cubrid hb stop can visibly hang while servers flush.

Master side of each opcode:

commdb opcode	master handler (master_request.c)	effect
DEACT_STOP_ALL	css_process_deact_stop_all	sets hb_Deactivate_immediately, hb_prepare_deactivate_heartbeat (queues HB_RJOB_CLEANUP_ALL)
DEACT_CONFIRM_STOP_ALL	css_process_deact_confirm_stop_all	SUCCESS once hb_is_deactivation_ready() (no proc has a conn)
DEACTIVATE_HEARTBEAT	css_process_deactivate_heartbeat	hb_deactivate_heartbeat
DEACT_CONFIRM_NO_SERVER	css_process_deact_confirm_no_server	SUCCESS once hb_get_deactivating_server_count()==0
ACTIVATE_HEARTBEAT	css_process_activate_heartbeat	hb_activate_heartbeat
RECONFIG_HEARTBEAT	css_process_reconfig_heartbeat	hb_reconfig_heartbeat -> hb_reload_config

Only the two mutating deactivate handlers gate on origin: css_process_deact_stop_all and css_process_deactivate_heartbeat each run hb_check_request_eligibility(conn->fd) up front. HB_HC_FAILED (getpeername failed) / HB_HC_UNAUTHORIZED (peer IP not in the node list) -> HA_REQUEST_FAILURE; HB_HC_ELIGIBLE_REMOTE (peer IP matches a node) is allowed but logged via hb_disable_er_log; only HB_HC_ELIGIBLE_LOCAL (AF_UNIX socket) is silent — this authorizes a remote cubrid hb stop -h <peer>. The two confirm handlers (css_process_deact_confirm_stop_all, css_process_deact_confirm_no_server) run no eligibility check at all; css_process_activate_heartbeat and css_process_reconfig_heartbeat gate only on HA_DISABLED(), not on origin.

10.2.1 Inside cub_commdb: the courier handlers

The three opcode-only couriers share one skeleton — send request, receive string reply, print-or-fail — but differ in arity and result test:

handler	request	replies	NO_ERROR when	ER_FAILED when
process_reconfig_heartbeat	RECONFIG_HEARTBEAT, no args	1	reply non-empty (size>0 && [0]!='\0')	reply empty
process_activate_heartbeat	ACTIVATE_HEARTBEAT, no args	1	strncmp(reply, HA_REQUEST_SUCCESS, size-1)==0	otherwise
process_deactivate_heartbeat	DEACTIVATE_HEARTBEAT, no args	2 (msg, result)	result == HA_REQUEST_SUCCESS	otherwise

process_reconfig_heartbeat discriminates by content (non-empty reply is printf’d and = success; empty = ER_FAILED). process_activate_heartbeat discriminates by sentinel, so an “already activated” message not matching the token is reported as failure even though the master treats it as benign. process_deactivate_heartbeat reads two replies off one rid: a message string (printed if non-empty), then a result string tested for success. All buffers are free_and_init’d on every arm. The -i flag never reaches these three — only process_deact_stop_all sends commdb_Arg_deact_immediately via send_request_one_arg; the rest are send_request_no_args.

10.3 css_process_deact_stop_all and the immediately flag

// css_process_deact_stop_all -- src/executables/master_request.c
  if (hb_is_deactivation_started () == false) {           /* <- idempotent: only first wins */
      hb_start_deactivate_server_info ();
      hb_Deactivate_immediately = (deact_immediately && *((bool*)deact_immediately)) ? true : false;
      if (hb_prepare_deactivate_heartbeat () != NO_ERROR) goto error_return; }
  css_send_data (conn, request_id, HA_REQUEST_SUCCESS, HA_REQUEST_RESULT_SIZE);

hb_prepare_deactivate_heartbeat flips hb_Resource->shutdown = true under the resource lock (returns NO_ERROR early if already set, so a second STOP_ALL is harmless) and queues one HB_RJOB_CLEANUP_ALL at HB_JOB_TIMER_IMMEDIATELY.

Invariant — hb_Deactivate_immediately chooses politeness vs. force. false: cleanup records every connected cub_server PID into hb_Deactivate_info.server_pid_list and sends graceful shutdowns, letting servers flush. true: that bookkeeping is skipped and the confirm job SIGKILLs survivors on the first pass. Global, set once per cycle, reset to false in hb_master_init.

HB_DEACTIVATE_INFO (static hb_Deactivate_info) threads the phases:

Field	Role	Why it exists
server_pid_list	int* of cub_server PIDs captured at cleanup start	phase 4 polls them with kill(pid,0) to know each DB server exited
server_count	count of valid entries	bounds the poll loop; 0 = no servers to wait for
info_started	bool latch: deactivation in progress	makes STOP_ALL idempotent; blocks a racing hb_activate_heartbeat

10.4 hb_activate_heartbeat, hb_deactivate_heartbeat, hb_reload_config

hb_activate_heartbeat — four guarded branches:

guard	action	rationale
hb_Cluster == NULL	return ER_FAILED	never initialized
hb_Deactivate_info.info_started == true	return ER_FAILED	re-init mid-teardown corrupts lists
hb_Is_activated == true	return NO_ERROR	benign: operator gets a notice, not a failure
else	hb_master_init(); on error ER_FAILED, else hb_Is_activated=true	rebuild singletons; resets hb_Deactivate_immediately=false

hb_deactivate_heartbeat — the symmetric off switch (phase 3):

// hb_deactivate_heartbeat -- src/executables/master_heartbeat.c
  if (hb_Cluster == NULL) return ER_FAILED;
  if (hb_Is_activated == false) return NO_ERROR;                         /* already off, benign */
  if (hb_Resource && resource_Jobs) hb_resource_shutdown_and_cleanup (); /* resource FIRST */
  if (hb_Cluster  && cluster_Jobs)  hb_cluster_shutdown_and_cleanup ();  /* cluster SECOND */
  hb_Is_activated = false;

Resource is torn down before cluster: the resource side owns the live process connections, the cluster side owns peer state — draining processes first lets the cluster’s goodbye heartbeats go out after local DB servers are already quiescing. Both calls are conditional on the jobs queue still existing, so a partially-initialized module deactivates cleanly.

hb_reload_config — the only verb that mutates live topology without a stop/start: backup-mutate-restore under hb_Cluster->lock.

// hb_reload_config -- src/executables/master_heartbeat.c   (condensed)
  sysprm_reload_and_init (NULL, NULL);                                          /* re-read cubrid.conf */
  if (prm_get_string_value (PRM_ID_HA_NODE_LIST) == NULL) return ER_FAILED;
  lock; hb_list_move (&old_ping_hosts,...); hb_list_move (&old_nodes,...);      /* backup + null out */
        old_myself = ...->myself; old_master = ...->master;
  if (hb_cluster_check_valid_ping_server () == false) goto reconfig_error;      /* fail 1 */
  if (...->num_ping_hosts == 0 && hb_cluster_load_tcp_ping_host_list (...),     /* ICMP empty: TCP ping */
      hb_cluster_check_valid_ping_server () == false) { unlock; return ER_FAILED; }  /* fail 2: early */
  ...->num_nodes = hb_cluster_load_group_and_node_list (...);
  if (...->num_nodes < 1 || (master && hb_return_node_by_name(master->host_name)==NULL))
    goto reconfig_error;                                                        /* fail 3 */
  for (new_node...) for (old_node...) if (are_hostnames_equal(...))             /* carry state forward by host */
    { ...->master = new_node if matches old_master; copy state/score/heartbeat_gap/last_recv_hbtime;
      old_node->host_name[0]='\0'; }                                           /* mark "kept", no goodbye */
  free old_ping_hosts + old_nodes; unlock; return NO_ERROR;
reconfig_error: /* swap old_* back into hb_Cluster, free half-built new lists, unlock, return error */

Invariant — reload is atomic to peers: the new topology is fully installed or the old one fully restored. goto reconfig_error (fail 1 = bad ICMP ping server, fail 3 = empty/invalid node list) swaps the old_* lists back and frees the half-built new lists. The TCP-ping failure (fail 2) returns early without a full restore — only ping_hosts (already backed up) was touched there. Runtime score/state is carried forward by hostname so adding a ping host does not reset election progress; were this not atomic, a typo’d ha_node_list would leave the master pointing at freed memory.

10.5 hb_deregister_by_pid / _by_args and the cleanup jobs

A managed process is evicted by PID (cubrid hb deregister <pid>) or by args (internal). hb_deregister_by_pid/_by_args are thin wrappers: bail if hb_Resource == NULL; under the lock look the proc up (hb_return_proc_by_pid / hb_return_proc_by_args) and on miss unlock and return (“cannot find process”); else call hb_deregister_process (builds the job arg, flips proc->state), unlock, and queue HB_RJOB_PROC_DEREG at IMMEDIATELY (a queue failure assert(false)s and frees the arg).

The shared hb_deregister_process rejects a proc whose state is outside [HB_PSTATE_DEAD, HB_PSTATE_MAX) or pid < 0, stamps proc->dtime, allocates an HB_JOB_ARG (PID, args, retries=0, max_retries = PRM_ID_HA_MAX_PROCESS_DEREG_CONFIRM), and flips proc->state = HB_PSTATE_DEREGISTERED before returning. The async HB_RJOB_PROC_DEREG job confirms the exit (Chapter 9).

hb_resource_job_cleanup_all (the HB_RJOB_CLEANUP_ALL worker) locks css_Master_socket_anchor_lock then hb_Resource->lock (this fixed order); if hb_Deactivate_immediately == false it callocs server_pid_list and records each proc->conn && proc->type==HB_PTYPE_SERVER PID into it (server_count); calls hb_resource_shutdown_all_ha_procs (10.7); mallocs the confirm HB_JOB_ARG (returning on OOM with both locks released) setting retries=0 and max_retries = PRM_ID_HA_MAX_PROCESS_DEREG_CONFIRM; then unlocks both and queues HB_RJOB_CONFIRM_CLEANUP_ALL at WAIT_500_MS.

It chains into hb_resource_job_confirm_cleanup_all, which re-queues until all processes are gone. Every branch:

branch	condition	action
null guard	`arg == NULL
deadline / force	++retries > max_retries OR hb_Deactivate_immediately == true	loop all procs: SIGKILL any alive, hb_remove_proc; assert(num_procs==0); goto end
normal: non-server	proc->type != HB_PTYPE_SERVER	SIGKILL + hb_remove_proc now (stateless replica)
normal: server alive	HB_PTYPE_SERVER, kill(pid,0)==0	leave in place; count num_connected_rsc if conn != NULL
normal: server dead	HB_PTYPE_SERVER, kill -> ESRCH	hb_remove_proc, continue
done	num_procs == 0 OR num_connected_rsc == 0	goto end, free arg, ER_SET “ready to deactivate”
loop	otherwise	unlock, re-queue at …DEREG_CONFIRM_INTERVAL…

Invariant — cub_server processes are killed last and gently. In the normal pass copylogdb/applylogdb are SIGKILL’d and removed immediately, but a HB_PTYPE_SERVER is removed only once already dead (kill(pid,0) -> ESRCH); servers exit on their own after flushing. Only when the retry budget is exhausted, or -i was given, does the force branch SIGKILL the survivors — every proc having first passed through hb_resource_shutdown_all_ha_procs.

stateDiagram-v2
  [*] --> CleanupAll: STOP_ALL queues HB_RJOB_CLEANUP_ALL
  CleanupAll --> Confirm: record server pids \n shutdown_all_ha_procs \n queue CONFIRM
  Confirm --> Confirm: retries <= max \n kill non-servers \n wait for servers \n re-queue
  Confirm --> Done: num_procs == 0 \n or num_connected == 0
  Confirm --> ForceKill: retries > max \n or deactivate immediately
  ForceKill --> Done: SIGKILL survivors \n remove all procs
  Done --> [*]: ER_SET ready to deactivate

Figure 10-2. The cleanup-all / confirm-cleanup-all loop, with the graceful drain and forced-kill exits.

10.6 Module teardown: cluster and resource shutdown-and-cleanup

The two public entry points are two lines each: hb_resource_shutdown_and_cleanup = hb_resource_job_shutdown() then hb_resource_cleanup(); hb_cluster_shutdown_and_cleanup = hb_cluster_job_shutdown() then hb_cluster_cleanup().

Ordering invariant — shut the job queue before freeing the data it touches. hb_*_job_shutdown drains and stops the worker thread (Chapter 3) so no job runs after hb_*_cleanup frees procs/nodes; reversed, an in-flight HB_CJOB_* job dereferences a freed list.

hb_resource_cleanup locks, re-calls hb_resource_shutdown_all_ha_procs (idempotent — every conn already NULL), SIGKILLs any proc that still has a live conn && pid > 0, frees the list via hb_remove_all_procs, resets procs=NULL, num_procs=0, state=HB_NSTATE_UNKNOWN, shutdown=true.

hb_cluster_cleanup is the goodbye: under the lock it sets state = HB_NSTATE_UNKNOWN, loops nodes sending one final hb_cluster_send_heartbeat_req to every peer (skipping itself) while bumping heartbeat_gap — peers see the gap climb and start failover promptly. It then frees nodes, clears master/myself, closes sfd, frees ping hosts, and frees the unidentified-node diagnostics list:

// hb_cluster_cleanup -- src/executables/master_heartbeat.c
  hb_cluster_remove_all_ui_nodes (hb_Cluster->ui_nodes);   /* unauthorized-host diagnostics */
  hb_Cluster->ui_nodes = NULL;  hb_Cluster->num_ui_nodes = 0;

The ui_nodes list (HB_UI_NODE_ENTRY) holds hosts that sent heartbeats but failed the node-list / group-id authorization check; purely diagnostic (cubrid hb status -v), dropped wholesale with no per-node ceremony.

10.7 hb_resource_shutdown_all_ha_procs: closing vs. shutting down

The per-process disconnect step shared by cleanup-all and final cleanup. It branches on type:

// hb_resource_shutdown_all_ha_procs -- src/executables/master_heartbeat.c
  for (proc...) {
    if (proc->conn) {
      if (proc->type != HB_PTYPE_SERVER) {                 /* copy/apply: yank the socket */
          css_remove_entry_by_conn (proc->conn, &css_Master_socket_anchor);
          proc->conn=NULL; proc->sfd=INVALID_SOCKET; }
      else {                                                /* server: ask it to shut down */
          sock_ent = css_return_entry_by_conn (proc->conn, &css_Master_socket_anchor);
          if (sock_ent && sock_ent->name) css_process_start_shutdown (sock_ent, 0, buffer);
          else { proc->conn=NULL; proc->sfd=INVALID_SOCKET; } } }
    else MASTER_ER_LOG_DEBUG (... "invalid socket-queue entry" ...);
    proc->state = HB_PSTATE_DEREGISTERED; }                /* <- unconditional, EVERY proc */

copylogdb/applylogdb have their socket-queue entry removed and conn nulled — stateless replicas, safe to drop. A cub_server keeps its entry but receives a graceful css_process_start_shutdown so it flushes and exits on its own; only if the entry is missing does the master null the conn directly. The last line runs on every proc, satisfying the HB_PSTATE_DEREGISTERED precondition the confirm job (10.5) relies on.

10.8 The process-side bridge: hb_process_init and the suicide reader

A managed process (copylogdb/applylogdb, CS_MODE) calls hb_process_init once — a thread-safe run-once latch:

// hb_process_init -- src/connection/heartbeat.c
  static std::atomic<bool> is_first{true}; static std::mutex init_mtx;
  if (is_first.load(acquire)==false) return NO_ERROR;                    /* fast path */
  std::lock_guard<std::mutex> lock(init_mtx);
  if (is_first.load(relaxed)==false) return NO_ERROR;                    /* double-check */
  if (hb_Exec_path[0]=='\0' || *(hb_Argv)==0) return ER_FAILED;          /* not set up */
  hb_Conn = hb_connect_to_master (server_name, log_path, type); sleep(1);
  if ((error = hb_register_to_master (hb_Conn, type))) return error;     /* send registration */
  if ((error = hb_create_master_reader ())) return error;               /* spawn reader */
  is_first.store (false, release);

hb_connect_to_master packs a name ($=copylogdb, %=applylogdb via hb_pack_server_name) and dials prm_get_master_port_id(). hb_register_to_master sends SERVER_REGISTER_HA_PROCESS plus an HBP_PROC_REGISTER payload (PID, type, exec path, argv) from hb_make_set_hbp_register; on any send error it frees the payload and returns ER_FAILED. hb_create_master_reader spins a detached pthread running hb_thread_master_reader:

// hb_thread_master_reader -- src/connection/heartbeat.c
  if (hb_process_master_request () != NO_ERROR) { hb_process_term (); sleep(1); kill (getpid(), SIGTERM); }

Invariant — a managed process that loses the master must die. hb_process_master_request always returns ER_FAILED (no success exit), so the reader always self-kills. An orphaned copylogdb/applylogdb must not keep replicating against a failed-over master — it exits and is respawned by the new master.

// hb_process_master_request -- src/connection/heartbeat.c
  while (false == hb_Proc_shutdown) {
    po[0].fd = hb_Conn->fd; po[0].events = POLLIN;
    r = css_platform_independent_poll (po, 1, PRM_ID_TCP_CONNECTION_TIMEOUT*1000);
    switch (r) {
      case 0: break;                                       /* timeout: loop, master alive */
      case -1: if (!IS_INVALID_SOCKET(hb_Conn->fd) && fcntl(hb_Conn->fd,F_GETFL,status)<0)
                 hb_Proc_shutdown=true; break;             /* socket truly broken (guards EINTR) */
      default: if (hb_process_master_request_info(hb_Conn)!=NO_ERROR) hb_Proc_shutdown=true; break; } } /* closed */
  return (ER_FAILED);

hb_process_master_request_info does css_receive_heartbeat_request and ignores the content — the request is a liveness probe. A clean read keeps looping; a read error sets hb_Proc_shutdown, exits, triggers suicide. r==0 is benign; r==-1 shuts down only if fcntl confirms a dead fd.

flowchart TD
  init["hb_process_init<br/>(once, latched)"] --> conn["hb_connect_to_master"]
  conn --> reg["hb_register_to_master<br/>SERVER_REGISTER_HA_PROCESS"]
  reg --> spawn["hb_create_master_reader<br/>detached pthread"]
  spawn --> loop["hb_process_master_request<br/>poll loop"]
  loop -->|"poll timeout r=0"| loop
  loop -->|"readable, read OK"| loop
  loop -->|"read error / socket dead"| term["hb_process_term<br/>close conn"]
  term --> suicide["kill(getpid, SIGTERM)<br/>process exits"]

Figure 10-3. Register once, then a reader thread that turns master disconnect into self-termination.

hb_deregister_from_master is the graceful counterpart on clean shutdown: it sends SERVER_DEREGISTER_HA_PROCESS plus its PID over hb_Conn, returning ER_FAILED if hb_Conn is gone or either send fails. The four wire primitives — css_send_heartbeat_request/_data, css_receive_heartbeat_request/_data — are framed send/css_readn wrappers returning CONNECTION_CLOSED on a NULL/invalid socket, ERROR_ON_WRITE/ERROR_ON_READ on a short transfer, NO_ERRORS only on exact size; a request is one network-order int, data an opaque blob.

10.9 The replica special case

A replica is configured via ha_replica_list and parsed with HB_REPLICA_PRIORITY (0x7FFF — the worst priority, in master_heartbeat.h). Two rules follow:

• Never elected master. In hb_cluster_job_calc_score (Chapter 5) — the periodic job that calls the score-summing helper hb_cluster_calc_score — the guard right after the sum is if (hb_Cluster->state == HB_NSTATE_REPLICA || hb_Cluster->hide_to_demote == true) goto calc_end;, skipping the master-isolation, split-brain, and failover transitions. The worst priority is the belt-and-suspenders backstop if it were ever scored.
• Doesn’t count toward isolation. hb_cluster_is_isolated continues past any HB_NSTATE_REPLICA node, so a lone master plus a replica is still “isolated”.

For shutdown the replica is an ordinary node — same hb_cluster_cleanup path, sends goodbye heartbeats like any peer.

10.10 Chapter summary — key takeaways

1. The operator never touches cub_master directly. process_heartbeat only spawns cub_commdb; its couriers (process_activate_heartbeat / process_deactivate_heartbeat / process_reconfig_heartbeat) send one opcode and print the reply, which master_request.c maps to the hb_* functions.
2. Deactivation is a four-phase confirm handshake (us_hb_deactivate): STOP_ALL -> poll CONFIRM_STOP_ALL -> DEACTIVATE -> poll CONFIRM_NO_SERVER; the busy-poll loops separate EXIT_FAILURE (abort) from “not yet” (retry), which is why a clean stop can visibly hang.
3. hb_Deactivate_immediately picks force vs. politeness. Polite mode waits up to HA_MAX_PROCESS_DEREG_CONFIRM intervals for servers to flush; -i (read only by the STOP_ALL handler) SIGKILLs survivors on the first confirm pass.
4. hb_reload_config is an atomic backup-mutate-restore transaction that carries per-node score/state forward by hostname and, on any of three validation failures, restores the old lists so peers never see a half-applied topology.
5. Teardown ordering is fixed: stop the job queue, then free its data. Resource teardown precedes cluster; hb_resource_shutdown_all_ha_procs drops copy/apply sockets immediately but only requests cub_server shutdown, killing it last and only under deadline/force.
6. A managed process self-terminates on master loss. hb_process_init registers once and spawns a detached reader; hb_process_master_request always returns ER_FAILED, so the reader inevitably runs hb_process_term + kill(getpid(), SIGTERM).
7. A replica is a permanent non-candidate. Worst priority HB_REPLICA_PRIORITY (0x7FFF), short-circuited out of hb_cluster_job_calc_score via the HB_NSTATE_REPLICA guard before any state transition, and skipped by hb_cluster_is_isolated; it still tears down like any peer.

Position hints as of this revision

The following are line numbers as observed on 2026-06-23; symbols are the canonical anchor and line numbers are hints that decay.

Symbol	File	Line
PRM_ID_HA_HEARTBEAT_INTERVAL_IN_MSECS	src/base/system_parameter.h	226
PRM_ID_HA_CALC_SCORE_INTERVAL_IN_MSECS	src/base/system_parameter.h	227
PRM_ID_HA_MAX_HEARTBEAT_GAP	src/base/system_parameter.h	235
SERVER_CHANGE_HA_MODE	src/connection/connection_defs.h	155
css_send_heartbeat_request	src/connection/heartbeat.c	160
css_send_heartbeat_data	src/connection/heartbeat.c	187
css_receive_heartbeat_request	src/connection/heartbeat.c	211
css_receive_heartbeat_data	src/connection/heartbeat.c	239
hb_make_set_hbp_register	src/connection/heartbeat.c	262
hb_register_to_master	src/connection/heartbeat.c	298
hb_thread_master_reader	src/connection/heartbeat.c	347
hb_deregister_from_master	src/connection/heartbeat.c	378
hb_process_master_request_info	src/connection/heartbeat.c	411
hb_process_master_request	src/connection/heartbeat.c	456
hb_connect_to_master	src/connection/heartbeat.c	588
hb_create_master_reader	src/connection/heartbeat.c	618
hb_process_init	src/connection/heartbeat.c	691
hb_process_term	src/connection/heartbeat.c	755
HB_DEFAULT_HEARTBEAT_INTERVAL_IN_MSECS	src/connection/heartbeat.h	36
HB_DEFAULT_CHECK_VALID_PING_SERVER_INTERVAL_IN_MSECS	src/connection/heartbeat.h	38
HB_TEMP_CHECK_VALID_PING_SERVER_INTERVAL_IN_MSECS	src/connection/heartbeat.h	39
HB_DEFAULT_MAX_HEARTBEAT_GAP	src/connection/heartbeat.h	47
HB_JOB_TIMER_IMMEDIATELY	src/connection/heartbeat.h	50
HB_JOB_TIMER_WAIT_A_SECOND	src/connection/heartbeat.h	51
HB_JOB_TIMER_WAIT_500_MILLISECOND	src/connection/heartbeat.h	52
HB_DISK_FAILURE_CHECK_TIMER_IN_MSECS	src/connection/heartbeat.h	54
hb_proc_type	src/connection/heartbeat.h	60
HB_PTYPE_SERVER	src/connection/heartbeat.h	62
HBP_CLUSTER_MESSAGE	src/connection/heartbeat.h	73
HBP_CLUSTER_HEARTBEAT	src/connection/heartbeat.h	75
HB_MAX_GROUP_ID_LEN	src/connection/heartbeat.h	79
HB_MAX_NUM_PROC_ARGV	src/connection/heartbeat.h	81
HB_NODE_STATE	src/connection/heartbeat.h	86
HB_NSTATE_UNKNOWN	src/connection/heartbeat.h	88
HB_NSTATE_SLAVE	src/connection/heartbeat.h	89
HB_NSTATE_TO_BE_MASTER	src/connection/heartbeat.h	90
HB_NSTATE_TO_BE_SLAVE	src/connection/heartbeat.h	91
HB_NSTATE_MASTER	src/connection/heartbeat.h	92
HB_NSTATE_REPLICA	src/connection/heartbeat.h	93
hbp_header	src/connection/heartbeat.h	114
HBP_PROC_REGISTER	src/connection/heartbeat.h	137
hbp_proc_register	src/connection/heartbeat.h	138
process_reconfig_heartbeat	src/executables/commdb.c	814
process_deactivate_heartbeat	src/executables/commdb.c	852
process_deact_confirm_no_server	src/executables/commdb.c	895
process_deact_confirm_stop_all	src/executables/commdb.c	927
process_deact_stop_all	src/executables/commdb.c	959
process_activate_heartbeat	src/executables/commdb.c	997
cluster_Jobs	src/executables/master_heartbeat.c	246
resource_Jobs	src/executables/master_heartbeat.c	247
hb_Deactivate_immediately	src/executables/master_heartbeat.c	248
hb_Deactivate_info	src/executables/master_heartbeat.c	252
hb_cluster_jobs	src/executables/master_heartbeat.c	259
hb_resource_jobs	src/executables/master_heartbeat.c	272
hb_list_add	src/executables/master_heartbeat.c	351
hb_list_remove	src/executables/master_heartbeat.c	368
hb_list_move	src/executables/master_heartbeat.c	389
hb_add_timeval	src/executables/master_heartbeat.c	412
hb_compare_timeval	src/executables/master_heartbeat.c	431
hb_job_queue	src/executables/master_heartbeat.c	517
hb_job_dequeue	src/executables/master_heartbeat.c	568
hb_job_set_expire_and_reorder	src/executables/master_heartbeat.c	614
hb_job_shutdown	src/executables/master_heartbeat.c	679
hb_cluster_job_init	src/executables/master_heartbeat.c	708
hb_cluster_job_heartbeat	src/executables/master_heartbeat.c	734
hb_cluster_is_isolated	src/executables/master_heartbeat.c	762
hb_cluster_is_received_heartbeat_from_all	src/executables/master_heartbeat.c	785
hb_cluster_job_calc_score	src/executables/master_heartbeat.c	811
hb_cluster_job_calc_score	src/executables/master_heartbeat.c	812
hb_cluster_job_check_ping	src/executables/master_heartbeat.c	992
hb_cluster_job_failover	src/executables/master_heartbeat.c	1163
hb_cluster_job_demote	src/executables/master_heartbeat.c	1236
hb_cluster_job_failback	src/executables/master_heartbeat.c	1351
hb_cluster_check_valid_ping_server	src/executables/master_heartbeat.c	1463
hb_cluster_job_check_valid_ping_server	src/executables/master_heartbeat.c	1500
hb_cluster_calc_score	src/executables/master_heartbeat.c	1556
hb_cluster_request_heartbeat_to_all	src/executables/master_heartbeat.c	1646
hb_cluster_send_heartbeat_req	src/executables/master_heartbeat.c	1677
hb_cluster_send_heartbeat_resp	src/executables/master_heartbeat.c	1696
hb_cluster_send_heartbeat_internal	src/executables/master_heartbeat.c	1702
hb_cluster_receive_heartbeat	src/executables/master_heartbeat.c	1750
hb_set_net_header	src/executables/master_heartbeat.c	1914
hb_cluster_job_dequeue	src/executables/master_heartbeat.c	2061
hb_cluster_job_queue	src/executables/master_heartbeat.c	2075
hb_cluster_job_set_expire_and_reorder	src/executables/master_heartbeat.c	2094
hb_add_node_to_cluster	src/executables/master_heartbeat.c	2130
hb_add_ping_host	src/executables/master_heartbeat.c	2210
hb_add_tcp_ping_host	src/executables/master_heartbeat.c	2246
hb_cluster_load_ping_host_list	src/executables/master_heartbeat.c	2318
hb_port_str_to_num	src/executables/master_heartbeat.c	2362
hb_cluster_load_tcp_ping_host_list	src/executables/master_heartbeat.c	2422
hb_return_node_by_name	src/executables/master_heartbeat.c	2493
hb_return_node_by_name_except_me	src/executables/master_heartbeat.c	2517
hb_is_heartbeat_valid	src/executables/master_heartbeat.c	2535
hb_valid_result_string	src/executables/master_heartbeat.c	2572
hb_return_ui_node	src/executables/master_heartbeat.c	2597
hb_add_ui_node	src/executables/master_heartbeat.c	2629
hb_remove_ui_node	src/executables/master_heartbeat.c	2666
hb_cleanup_ui_nodes	src/executables/master_heartbeat.c	2686
hb_cluster_load_group_and_node_list	src/executables/master_heartbeat.c	2730
hb_resource_job_confirm_cleanup_all	src/executables/master_heartbeat.c	2870
hb_resource_job_cleanup_all	src/executables/master_heartbeat.c	3003
hb_resource_job_proc_start	src/executables/master_heartbeat.c	3071
hb_resource_job_proc_dereg	src/executables/master_heartbeat.c	3203
hb_resource_demote_start_shutdown_server_proc	src/executables/master_heartbeat.c	3307
hb_resource_demote_confirm_shutdown_server_proc	src/executables/master_heartbeat.c	3356
hb_resource_demote_kill_server_proc	src/executables/master_heartbeat.c	3384
hb_resource_job_demote_confirm_shutdown	src/executables/master_heartbeat.c	3416
hb_resource_job_demote_start_shutdown	src/executables/master_heartbeat.c	3494
hb_resource_job_confirm_start	src/executables/master_heartbeat.c	3552
hb_resource_job_confirm_dereg	src/executables/master_heartbeat.c	3702
hb_resource_job_change_mode	src/executables/master_heartbeat.c	3791
hb_resource_job_dequeue	src/executables/master_heartbeat.c	3858
hb_resource_job_queue	src/executables/master_heartbeat.c	3872
hb_resource_job_set_expire_and_reorder	src/executables/master_heartbeat.c	3891
hb_alloc_new_proc	src/executables/master_heartbeat.c	3925
hb_remove_proc	src/executables/master_heartbeat.c	3957
hb_return_proc_by_args	src/executables/master_heartbeat.c	3992
hb_return_proc_by_pid	src/executables/master_heartbeat.c	4014
hb_return_proc_by_fd	src/executables/master_heartbeat.c	4037
hb_proc_make_arg	src/executables/master_heartbeat.c	4061
hb_cleanup_conn_and_start_process	src/executables/master_heartbeat.c	4089
hb_is_registered_process	src/executables/master_heartbeat.c	4203
hb_register_new_process	src/executables/master_heartbeat.c	4238
hb_resource_send_changemode	src/executables/master_heartbeat.c	4356
hb_resource_receive_changemode	src/executables/master_heartbeat.c	4444
hb_resource_check_server_log_grow	src/executables/master_heartbeat.c	4518
hb_resource_send_get_eof	src/executables/master_heartbeat.c	4577
hb_resource_receive_get_eof	src/executables/master_heartbeat.c	4605
hb_thread_cluster_worker	src/executables/master_heartbeat.c	4659
hb_thread_cluster_reader	src/executables/master_heartbeat.c	4704
hb_thread_resource_worker	src/executables/master_heartbeat.c	4769
hb_thread_check_disk_failure	src/executables/master_heartbeat.c	4811
hb_cluster_job_initialize	src/executables/master_heartbeat.c	4912
hb_cluster_initialize	src/executables/master_heartbeat.c	4952
hb_resource_initialize	src/executables/master_heartbeat.c	5072
hb_resource_job_initialize	src/executables/master_heartbeat.c	5104
hb_thread_initialize	src/executables/master_heartbeat.c	5146
hb_master_init	src/executables/master_heartbeat.c	5250
hb_resource_shutdown_all_ha_procs	src/executables/master_heartbeat.c	5343
hb_resource_cleanup	src/executables/master_heartbeat.c	5401
hb_resource_shutdown_and_cleanup	src/executables/master_heartbeat.c	5433
hb_cluster_cleanup	src/executables/master_heartbeat.c	5446
hb_cluster_shutdown_and_cleanup	src/executables/master_heartbeat.c	5496
hb_reload_config	src/executables/master_heartbeat.c	5576
hb_kill_process	src/executables/master_heartbeat.c	6242
hb_deregister_by_pid	src/executables/master_heartbeat.c	6346
hb_deregister_by_args	src/executables/master_heartbeat.c	6396
hb_deregister_process	src/executables/master_heartbeat.c	6441
hb_prepare_deactivate_heartbeat	src/executables/master_heartbeat.c	6515
hb_deactivate_heartbeat	src/executables/master_heartbeat.c	6557
hb_activate_heartbeat	src/executables/master_heartbeat.c	6599
hb_enable_er_log	src/executables/master_heartbeat.c	6723
hb_start_deactivate_server_info	src/executables/master_heartbeat.c	7167
hb_is_deactivation_ready	src/executables/master_heartbeat.c	7187
hb_get_deactivating_server_count	src/executables/master_heartbeat.c	7213
HB_PING_RESULT	src/executables/master_heartbeat.h	45
HB_PING_SUCCESS	src/executables/master_heartbeat.h	48
HB_PING_FAILURE	src/executables/master_heartbeat.h	51
HB_CLUSTER_JOB	src/executables/master_heartbeat.h	62
HB_CJOB_INIT	src/executables/master_heartbeat.h	64
HB_CLUSTER_JOB	src/executables/master_heartbeat.h	64
HB_RESOURCE_JOB	src/executables/master_heartbeat.h	76
HB_RESOURCE_JOB	src/executables/master_heartbeat.h	77
HB_RJOB_CHANGE_MODE	src/executables/master_heartbeat.h	82
HB_PROC_STATE	src/executables/master_heartbeat.h	93
HB_REPLICA_PRIORITY	src/executables/master_heartbeat.h	119
HB_NODE_SCORE_MASTER	src/executables/master_heartbeat.h	122
HB_NODE_SCORE_TO_BE_MASTER	src/executables/master_heartbeat.h	123
HB_NODE_SCORE_SLAVE	src/executables/master_heartbeat.h	124
HB_NODE_SCORE_UNKNOWN	src/executables/master_heartbeat.h	125
HB_BUFFER_SZ	src/executables/master_heartbeat.h	127
HB_MAX_NUM_NODES	src/executables/master_heartbeat.h	128
HB_MAX_PING_CHECK	src/executables/master_heartbeat.h	130
HB_MAX_WAIT_FOR_NEW_MASTER	src/executables/master_heartbeat.h	131
HB_MAX_CHANGEMODE_DIFF_TO_TERM	src/executables/master_heartbeat.h	132
HB_MAX_CHANGEMODE_DIFF_TO_KILL	src/executables/master_heartbeat.h	133
HB_NOLOG_DEMOTE_ON_DISK_FAIL	src/executables/master_heartbeat.h	155
HB_VALID_RESULT	src/executables/master_heartbeat.h	161
HB_VALID_NO_ERROR	src/executables/master_heartbeat.h	163
HB_GET_ELAPSED_TIME	src/executables/master_heartbeat.h	176
HB_IS_INITIALIZED_TIME	src/executables/master_heartbeat.h	180
HB_PROC_RECOVERY_DELAY_TIME	src/executables/master_heartbeat.h	183
HB_UI_NODE_CLEANUP_TIME_IN_MSECS	src/executables/master_heartbeat.h	185
hb_list	src/executables/master_heartbeat.h	191
hb_node_entry	src/executables/master_heartbeat.h	200
hb_ping_host_entry	src/executables/master_heartbeat.h	216
hb_ui_node_entry	src/executables/master_heartbeat.h	228
hb_cluster	src/executables/master_heartbeat.h	242
is_ping_check_enabled	src/executables/master_heartbeat.h	261
num_ping_hosts	src/executables/master_heartbeat.h	264
HB_PROC_ENTRY	src/executables/master_heartbeat.h	272
changemode_rid	src/executables/master_heartbeat.h	292
changemode_gap	src/executables/master_heartbeat.h	293
hb_resource	src/executables/master_heartbeat.h	307
hb_cluster_job_arg	src/executables/master_heartbeat.h	321
hb_resource_job_arg	src/executables/master_heartbeat.h	329
hb_job_arg	src/executables/master_heartbeat.h	345
struct hb_job_entry	src/executables/master_heartbeat.h	354
hb_job_entry	src/executables/master_heartbeat.h	355
hb_job	src/executables/master_heartbeat.h	370
struct hb_job	src/executables/master_heartbeat.h	370
hb_Cluster	src/executables/master_heartbeat.h	382
css_process_deactivate_heartbeat	src/executables/master_request.c	1527
css_process_deact_confirm_no_server	src/executables/master_request.c	1625
css_process_deact_stop_all	src/executables/master_request.c	1691
css_process_activate_heartbeat	src/executables/master_request.c	1782
us_hb_deactivate	src/executables/util_service.c	3948
process_heartbeat_stop	src/executables/util_service.c	4636
process_heartbeat_deregister	src/executables/util_service.c	4718
process_heartbeat_status	src/executables/util_service.c	4773
process_heartbeat_reload	src/executables/util_service.c	4881
process_heartbeat	src/executables/util_service.c	5069
log_lsa	src/transaction/log_lsa.hpp	35

Sources

• cubrid-heartbeat.md — the high-level companion. See also cubrid-ha-replication.md (what failover protects) and cubrid-master-process.md.
• Raw analyses under raw/code-analysis/cubrid/distributed/heartbeat/.
• Code: src/executables/master_heartbeat.{c,h}, src/connection/heartbeat.{c,h}, util_service.c, commdb.c.
• Methodology: knowledge/methodology/code-analysis-detail-doc.md.