CUBRID OOS (Out-of-Line Storage) — Code-Level Deep Dive

Where this document fits: The high-level analysis cubrid-oos.md covers design intent and theoretical background. This document traces every branch and field at the code level. Each chapter is self-contained, but reading in order follows the full lifecycle of a single out-of-line column value inside the kernel.

In development: OOS lives on the feat/oos branch (@ fa26eff3a, M1 done / M2 in progress). Symbols are the stable anchor; line numbers in the position-hint table decay quickly as the branch moves.

Contents:

Ch	Title	Status
1	Data Structure Map	✅
2	Initialization and Memory	✅
3	The Demotion Decision and the Inline Stub	✅
4	INSERT of a Single Chunk Value	✅
5	Bestspace Placement	✅
6	Multi Chunk Chains and HA Replication	✅
7	SELECT and Re Inlining	✅
8	The Physical Delete Primitive and Recovery	✅
9	UPDATE DELETE and Eager Non MVCC Cleanup	✅
10	Vacuum Reclamation	✅
11	Stats Diagnostics and Edge Paths	✅

---

Chapter 1: Data Structure Map

This chapter inventories every struct, flag, and identifier that, taken together, is an externalized column in CUBRID’s Out-Of-row Storage (OOS) subsystem. Later chapters cite these fields by name. For the why of OOS — when a value is externalized, the demotion threshold, the storage tradeoff — read cubrid-oos.md (“What problem OOS solves”, “The demotion decision”); this chapter does not re-derive that theory.

An OOS value is a distributed representation spanning three layers: a 16-byte inline stub inside the heap record, a chain of chunk records in a dedicated OOS file, and a per-file statistics header that steers placement. The identifiers below are how those layers point at each other.

1.1 The identity flags — how the engine knows a column is externalized

Two independent flag bits, at two different scopes, mark externalization. Confusing them is the most common reading error, so pin them down first.

// OR_MVCC_FLAG_HAS_OOS -- src/base/object_representation_constants.h
#define OR_MVCC_FLAG_HAS_OOS                       0x08  // 0b00001000

OR_MVCC_FLAG_HAS_OOS is a per-row bit in the MVCC flag nibble of the record header. It is a fast-path gate answering “does this record contain any externalized column?” without a walk of the variable-offset table (VOT). It sits beside OR_MVCC_FLAG_VALID_INSID 0x01, OR_MVCC_FLAG_VALID_DELID 0x02, and OR_MVCC_FLAG_VALID_PREV_VERSION 0x04 under OR_MVCC_FLAG_MASK 0x1f. The header comment flags the placement honestly: “OOS is not related to MVCC, move to another place when POC ends.” For now it rides in the MVCC nibble.

// OR_VAR_BIT_OOS -- src/base/object_representation.h
#define OR_VAR_BIT_OOS 0x1
#define OR_VAR_BIT_LAST_ELEMENT 0x2
#define OR_VAR_FLAG_MASK 0x3
#define OR_SET_VAR_OOS(length) ((int) (length) | OR_VAR_BIT_OOS)
#define OR_SET_VAR_LAST_ELEMENT(length) ((int) (length) | OR_VAR_BIT_LAST_ELEMENT)
#define OR_GET_VAR_FLAG(length) ((int) (length) & OR_VAR_FLAG_MASK)
#define OR_GET_VAR_OFFSET(length) ((int) (length) & (~OR_VAR_FLAG_MASK)) /* strip flags, recover offset */
#define OR_IS_OOS(length) (OR_GET_VAR_FLAG (length) & OR_VAR_BIT_OOS)
#define OR_IS_LAST_ELEMENT(length) (OR_GET_VAR_FLAG (length) & OR_VAR_BIT_LAST_ELEMENT)

OR_VAR_BIT_OOS is a per-VOT-entry bit. Each variable attribute has a VOT entry; the low bits of that offset word carry flags. OR_VAR_BIT_OOS (0x1) says “this column is out-of-row, so the bytes at its offset are a 16-byte stub, not the value.” Its sibling OR_VAR_BIT_LAST_ELEMENT (0x2) is the VOT sentinel that closes the table. Both live under the 2-bit OR_VAR_FLAG_MASK (0x3); OR_GET_VAR_OFFSET masks them off to recover the true offset. The two bits are orthogonal — one entry can be both OOS and the last element. Setters OR_SET_VAR_OOS / OR_SET_VAR_LAST_ELEMENT OR the bits in; both are bitwise-OR macros, so they never clear the sibling bit.

The relationship is hierarchical: OR_MVCC_FLAG_HAS_OOS on the row promises at least one VOT entry carries OR_VAR_BIT_OOS. heap_oos_parse_vot enforces it — after walking to the sentinel, n_var <= 0 triggers an assert_release with “OOS flag set without variable attributes” (the corrupt-record path in Chapter 7).

The OOS file itself is a first-class file type:

// FILE_TYPE -- src/storage/file_manager.h
typedef enum { FILE_TRACKER, FILE_HEAP, /* ... condensed ... */ FILE_TEMP, FILE_OOS, FILE_UNKNOWN_TYPE, FILE_LAST = FILE_UNKNOWN_TYPE } FILE_TYPE;

FILE_OOS is the type stamped on the per-class overflow file, distinct from FILE_HEAP and the legacy FILE_MULTIPAGE_OBJECT_HEAP. It is the second-to-last enumerator, just before FILE_UNKNOWN_TYPE. The file_manager and vacuum branch on it.

INVARIANT (one-to-one ownership). Exactly one heap record references any given OOS head OID. Chunks are not shared, deduplicated, or reference-counted — the inline stub holds the only pointer to its head chunk. Why it matters: every reclamation path (Ch 9 eager cleanup, Ch 10 vacuum REMOVE and forward-walk) deletes chunks unconditionally on owner death, with no “is anyone else pointing here?” check, because none can be. What breaks if violated: a second referrer would observe a dangling OID after the first owner frees the chunk; oos_read would land on a deallocated page. The probe oos_chunk_exists lets the single owner re-check its own chunk across recovery — not to support sharing.

1.2 The inline stub and the chunk header — the on-disk skeleton

The 16 bytes left behind in the heap record are the stub:

// OR_OOS_INLINE_SIZE -- src/base/object_representation.h
/* OOS inline size: OOS OID (8 bytes) + OOS length (8 bytes) */
#define OR_OOS_INLINE_SIZE (OR_OID_SIZE + OR_BIGINT_SIZE)   /* 8 + 8 = 16 */

heap_oos_read_blobs (Ch 7) reads it: or_get_oid consumes the first 8 bytes into the head-chunk OID, or_get_bigint consumes the next 8 into the full reconstituted length. The length lives inline so the reader sizes its destination buffer before touching the OOS file — oos_read writes exactly dest.size() bytes and never discovers the length itself. The read also bounds-checks: a value offset plus OR_OOS_INLINE_SIZE past src_length triggers assert_release “OOS inline slot extends past record bounds”.

Stub field layout (16 bytes):

Bytes	Field	Role	Why it exists
0–7	OID	head-chunk OID in the OOS file	the single pointer into the chunk chain (per the 1.1 invariant)
8–15	BIGINT length	total reconstituted value length	sizes dest before the first oos_read; the value oos_get_length returns

The first 8 bytes are an ordinary CUBRID OID (OR_OID_SIZE), and the disk byte order is fixed by the OR_OID_* offset constants — it is not the in-memory struct’s declaration order. The in-memory OID is declared { int pageid; short slotid; short volid; } (db_identifier in src/compat/dbtype_def.h), but OR_GET_OID / OR_PUT_OID serialize it in this on-disk order:

Head-chunk OID layout (8 bytes, bytes 0–7 of the stub):

Bytes	Field	Constant	Role
0–3	pageid (INT, 4B)	OR_OID_PAGEID = 0	page of the head chunk inside the OOS file
4–5	slotid (SHORT, 2B)	OR_OID_SLOTID = 4	slot index of the head chunk on that page
6–7	volid (SHORT, 2B)	OR_OID_VOLID = 6	volume the OOS page lives on

So the full 16-byte stub is pageid(4) | slotid(2) | volid(2) | full_length(8 BIGINT). The trailing full_length is the BIGINT half — it is not part of the OID; the OID is only the leading 8 bytes. (The task-level shorthand “volid2/pageid4/slotid2/full_length8” lists the four physical fields by size; the byte order on disk is pageid-first, per OR_OID_PAGEID = 0.)

On the OOS-file side, every chunk record begins with a fixed header:

// oos_record_header -- src/storage/oos_file.hpp
struct oos_record_header
{
  int total_data_length;  /* total length of user data across all chunks (excluding OOS headers) */
  int chunk_index;    /* 0-based index of this chunk in the chain */
  OID next_chunk_oid;    /* OID of next chunk, or NULL OID if this is the last */
};
using OOS_RECORD_HEADER = struct oos_record_header;
#define OOS_RECORD_HEADER_SIZE ((int) sizeof (OOS_RECORD_HEADER))

oos_record_header fields:

Field	Role	Why it exists
total_data_length	total user-data bytes across the whole chain, excluding per-chunk headers	redundant with the stub’s BIGINT but OOS-side, so oos_get_length answers without the heap
chunk_index	0-based position of this chunk in its chain	recovery/read validate ordering, detect a truncated chain
next_chunk_oid	OID of the next chunk, NULL on the last	the chain link; NULL terminates the walk (Ch 6)

OOS_RECORD_HEADER_SIZE is the prefix every chunk reserves before payload; usable capacity per slot is slot_size - OOS_RECORD_HEADER_SIZE.

Two documentation-only aliases ride alongside, with no compile-time distinction from their underlying types:

// OOS_RECDES -- src/storage/oos_file.hpp
using OOS_RECDES = RECDES;          /* a RECDES whose first OOS_RECORD_HEADER_SIZE bytes are the header */
using oos_buffer = cubbase::span<char>;  /* caller-owned payload span; size() is authoritative */

oos_buffer is the payload contract: oos_insert only reads it, oos_read only writes it, and .size() is authoritative in both directions. The header comment notes the alias also dodges the .c-file formatter mangling a raw cubbase::span<char> template argument.

Figure 1-1. The pointer skeleton: per-row flag gates per-entry flag gates stub; the stub’s OID points at the head chunk; each chunk’s next_chunk_oid links the chain until NULL.

1.3 The placement statistics — oos_bestspace and oos_hdr_stats

To place a chunk without scanning every page, the OOS file caches free-space estimates in its header. The atom is one (page, free-bytes) pair:

// oos_bestspace -- src/storage/oos_file.hpp
typedef struct oos_bestspace OOS_BESTSPACE;
struct oos_bestspace { VPID vpid; int freespace; };

oos_bestspace fields:

Field	Role	Why it exists
vpid	volume+page id of a candidate page	the page a chunk might go on
freespace	estimated free bytes on that page	ranks candidates; compared against chunk size needed

These pack into the header-resident statistics block, each ring capped at OOS_NUM_BEST_SPACESTATS (10):

// oos_hdr_stats -- src/storage/oos_file.hpp
struct oos_hdr_stats
{
  VFID oos_vfid;
  struct {
    int num_pages; int num_recs; float recs_sumlen;
    int num_other_high_best; int num_high_best; int num_substitutions;
    int num_second_best; int head_second_best; int tail_second_best; int head;
    VPID full_search_vpid;
    VPID second_best[OOS_NUM_BEST_SPACESTATS];
    OOS_BESTSPACE best[OOS_NUM_BEST_SPACESTATS];
  } estimates;
  int reserve0_for_future; int reserve1_for_future;
};

oos_hdr_stats fields:

Field	Role	Why it exists
oos_vfid	the file this header belongs to	self-identification; sanity-checked on fix
estimates.num_pages	physical user pages in the OOS file	density denominator; exposed via stats (Ch 11)
estimates.num_recs	live OOS records (chunks)	density numerator; surfaced as num_recs in OOS_STATS_INFO
estimates.recs_sumlen	running sum of live record body bytes (float)	average-record-size estimate; surfaced as recs_sumlen
estimates.num_other_high_best	high-free pages known but not in best[]	signals headroom beyond the cached ring
estimates.num_high_best	best[] slots above the high-free threshold	how many cached candidates are “good” now
estimates.num_substitutions	times a best[] slot was replaced	churn signal; throttles re-scans
estimates.num_second_best	valid entries in second_best[]	size of the backlog feeding best[]
estimates.head_second_best	head index of the second_best[] ring	producer cursor (OOS_STATS_NEXT_BEST_INDEX)
estimates.tail_second_best	tail index of the second_best[] ring	consumer cursor
estimates.head	head index into the best[] ring	rotates the page tried first
estimates.full_search_vpid	where the last exhaustive scan stopped	resume point so full search is incremental, not from page 0
estimates.second_best[10]	ring of VPIDs that might become best	feeder pool promoted into best[] on demand
estimates.best[10]	ring of OOS_BESTSPACE (page + free) candidates	the hot set the placement fast path consults (Ch 5)
reserve0_for_future / reserve1_for_future	unused padding ints	on-disk format stability; room to grow without a layout break

The two ring-index macros make the buffers circular:

// OOS_STATS_NEXT_BEST_INDEX -- src/storage/oos_file.hpp
#define OOS_STATS_NEXT_BEST_INDEX(i) (((i) + 1) % OOS_NUM_BEST_SPACESTATS)
#define OOS_STATS_PREV_BEST_INDEX(i) (((i) == 0) ? (OOS_NUM_BEST_SPACESTATS - 1) : ((i) - 1))

INVARIANT (estimates are hints, never authority). Every estimates field is an approximation that may lag the true page free-space; best[i].freespace is what the header thinks, not what the page has. How the code stays safe: placement (Ch 5) re-fixes the candidate page and re-checks real free space before committing, so a stale-high estimate costs a wasted page fix, never corruption. What breaks if trusted: writing to a page the header claims has room but is actually full would overflow the slot. The estimates only order candidates and bound the search; they are validated, not trusted.

The placement search returns a tri-state, separating “no room found” from “the search failed”:

// OOS_FINDSPACE -- src/storage/oos_file.hpp
typedef enum { OOS_FINDSPACE_FOUND = 0, OOS_FINDSPACE_NOTFOUND, OOS_FINDSPACE_ERROR } OOS_FINDSPACE;

OOS_FINDSPACE_NOTFOUND means “search succeeded, allocate a fresh page”; OOS_FINDSPACE_ERROR means “fixing/reading failed, propagate.” Collapsing them would mask I/O errors or trigger spurious allocation.

flowchart TD
  hdr["oos_hdr_stats.estimates"]
  hdr --> best["best[0..9] of OOS_BESTSPACE"]
  hdr --> sb["second_best[0..9] of VPID"]
  best --> bp0["page A vpid+freespace"]
  best --> bp1["page B vpid+freespace"]
  sb -.promote.-> best
  bp0 -.re-validate before write.-> pageA["real page A"]

Figure 1-2. The header’s best[] ring caches page+free candidates; second_best[] feeds it; every candidate is re-validated against the real page before a write.

1.4 The statistics surface — oos_stats_info and db_oos_stats

Two near-twin structs expose OOS metrics, one server-side and one client-side, kept separate because the client struct must be C-linkable and OID/VFID-free.

// oos_stats_info -- src/storage/oos_file.hpp
struct oos_stats_info {
  int has_oos_file; VFID oos_vfid; int num_user_pages; int page_size;
  int num_recs; INT64 recs_sumlen;
};
using OOS_STATS_INFO = struct oos_stats_info;

oos_stats_info fields (server-side):

Field	Role	Why it exists
has_oos_file	0 = no OOS file, 1 otherwise	short-circuits callers before touching oos_vfid
oos_vfid	the OOS file id	server-internal; not exported to the client
num_user_pages	physical user pages in the OOS file	capacity metric
page_size	DB_PAGESIZE	capacity-math denominator
num_recs	live OOS records, from OOS_HDR_STATS	population metric
recs_sumlen	sum of live record body bytes (INT64)	total externalized bytes

// db_oos_stats -- src/compat/db_oos.h
struct db_oos_stats {
  int has_oos_file; int oos_vfid_volid; int oos_vfid_fileid;
  int num_user_pages; int page_size; int num_recs; int64_t recs_sumlen;
};

db_oos_stats fields (client-side):

Field	Role	Why it exists
has_oos_file	0/1 as above	client gate
oos_vfid_volid	volume id of the OOS file	VFID split into plain ints — client header avoids the server VFID type
oos_vfid_fileid	file id of the OOS file	second half of the decomposed VFID
num_user_pages	physical user pages	mirrors server field
page_size	DB_PAGESIZE	mirrors server field
num_recs	live OOS records (slots on pages)	mirrors server field
recs_sumlen	sum of body bytes (int64_t)	mirrors server INT64 field

The only structural difference: oos_stats_info carries a VFID oos_vfid; db_oos_stats decomposes it into oos_vfid_volid + oos_vfid_fileid so db_oos.h stays free of server-only types and C-linkable. Chapter 11 dissects the xoos_get_stats_by_class_oid → oos_get_stats_by_vfid → db_get_oos_stats marshalling.

1.5 The transient working structs — expand state and the vacuum memo

HEAP_OOS_EXPAND_STATE is the scratch bag threaded through the re-inlining helpers (Ch 7). It is declared zero-initialized on the stack of heap_record_replace_oos_oids (HEAP_OOS_EXPAND_STATE state = { };), filled by heap_oos_parse_vot / heap_oos_read_blobs / heap_oos_compute_layout, then passed by pointer into heap_oos_build_record; it is never persisted.

// HEAP_OOS_EXPAND_STATE -- src/storage/heap_oos.cpp
struct HEAP_OOS_EXPAND_STATE
{
  const char *src; int src_length; int src_offset_size; int src_header_size;
  int n_var; int new_length; int src_vot_bytes; int dst_vot_bytes; int fixed_bitmap_bytes;
  std::vector<int> vot_raw;
  std::vector<std::vector<char>> oos_blobs;
};

HEAP_OOS_EXPAND_STATE fields:

Field	Role	Why it exists
src	pointer to the source (stub-bearing) record bytes	input read from (a snapshot copy, since rec->data may move)
src_length	length of the source record	bounds-check ceiling for every offset
src_offset_size	width of each source VOT entry (1/2/4)	drives the OR_BYTE/SHORT/INT switch in heap_oos_parse_vot
src_header_size	source record header byte count	base offset added to each VOT-relative offset
n_var	index of the LAST_ELEMENT sentinel (= number of variable attrs)	loop bound; <= 0 while flag set ⇒ corrupt record
new_length	length of the rebuilt (re-inlined) record	output sizing for the destination buffer
src_vot_bytes	byte size of the source VOT	input layout accounting
dst_vot_bytes	byte size of the destination VOT (4-byte BIG_VAR_OFFSET_SIZE)	output layout; destination always uses 4-byte offsets
fixed_bitmap_bytes	bytes between VOT end and first value (fixed attrs + null bitmap)	copied verbatim into output; < 0 ⇒ corrupt
vot_raw	parsed VOT entries, flag bits intact	per-entry OR_IS_OOS / OR_IS_LAST_ELEMENT / offset decisions
oos_blobs	per-entry reconstituted payloads (vector<vector<char>>, sized to n_var)	each OOS column’s full bytes after oos_read, ready to splice inline

The vacuum side carries a far smaller working struct — a one-entry memo:

// vacuum_oos_vfid_memo -- src/query/vacuum_oos.hpp
typedef struct vacuum_oos_vfid_memo VACUUM_OOS_VFID_MEMO;
struct vacuum_oos_vfid_memo
{
  bool valid = false;  /* false until the first successful lookup */
  VFID heap_vfid;  /* key; meaningful only when valid */
  VFID oos_vfid;  /* value; meaningful only when valid (VFID_NULL = "no OOS file") */
};

vacuum_oos_vfid_memo fields:

Field	Role	Why it exists
valid	false until the first successful lookup	distinguishes “never resolved” from “resolved to VFID_NULL”
heap_vfid	cached lookup key	the heap whose OOS file was last resolved
oos_vfid	cached result; VFID_NULL = “heap has no OOS file”	elides repeated file_descriptor_get + page fixes across a run of records for one heap

INVARIANT (memo is per-worker, per-block, never shared). VACUUM_OOS_VFID_MEMO is declared on the stack of vacuum_process_log_block, giving it per-worker-per-block lifetime, so the lookup needs no synchronization. Why: a bulk UPDATE emits a run of consecutive RVHF_UPDATE_NOTIFY_VACUUM records for one heap; a one-slot memo collapses that run’s repeated VFID resolution. What breaks if violated: a static would race — vacuum runs across multiple worker threads, and a shared static would corrupt the cached key/value. A transient lookup failure is never memoized, so a later record retries cleanly; only valid == true results are cached.

The lookup that populates the memo returns a tri-state mirroring the placement enum’s spirit:

// VACUUM_OOS_VFID_LOOKUP_RESULT -- src/query/vacuum_oos.cpp
typedef enum {
  VACUUM_OOS_VFID_FOUND,  /* found a real OOS file; its id is in out_oos_vfid */
  VACUUM_OOS_VFID_NONE,    /* the heap simply has no OOS file (normal) */
  VACUUM_OOS_VFID_ERROR    /* lookup failed; error left set, answer not cached */
} VACUUM_OOS_VFID_LOOKUP_RESULT;

FOUND caches and proceeds; NONE is the normal “nothing to reclaim” exit (also cached, as oos_vfid = VFID_NULL); ERROR leaves the engine error set and deliberately does not cache, so the next record retries — the OOS bytes stay on disk as a small, logged leak rather than failing the whole vacuum block. Chapter 10 traces how vacuum_oos_vfid_lookup maps each branch onto the memo.

stateDiagram-v2
  [*] --> Unresolved
  Unresolved --> Cached_real : FOUND caches oos_vfid
  Unresolved --> Cached_null : NONE caches VFID_NULL
  Unresolved --> Unresolved : ERROR not cached, retry next record
  Cached_real --> Cached_real : same heap_vfid reuses memo
  Cached_null --> Cached_null : same heap_vfid reuses memo

Figure 1-3. The memo state machine: only FOUND and NONE transition to a cached state; ERROR leaves the memo unresolved so the next record retries.

1.6 Chapter summary — key takeaways

1. Externalization is marked at two scopes. OR_MVCC_FLAG_HAS_OOS (0x08) is the per-row fast-path gate; OR_VAR_BIT_OOS (0x1) is the per-VOT-entry bit naming which column is out-of-row. The row flag promises at least one entry carries the VOT bit.
2. The stub is 16 bytes: OID (8) + BIGINT length (8), sized by OR_OOS_INLINE_SIZE. The leading OID serializes on disk as pageid(4) | slotid(2) | volid(2) (OR_OID_PAGEID/SLOTID/VOLID), not in struct-declaration order. The inline length lets a reader size its buffer before touching the OOS file, because oos_read writes exactly dest.size() bytes and never discovers the length.
3. oos_record_header chains chunks. next_chunk_oid links each to the next and is NULL on the last; total_data_length and chunk_index let recovery validate the chain without the heap.
4. One OOS OID is owned by exactly one heap record — no sharing, no refcounting. Every reclamation path deletes chunks unconditionally on owner death; a second referrer would dangle.
5. oos_hdr_stats.estimates is a hint cache, never authority. best[]/second_best[] are circular candidate rings ordered by oos_bestspace.freespace; placement re-validates the real page before writing, so a stale estimate costs a wasted fix, not corruption.
6. Stats are split server/client. oos_stats_info carries a VFID; db_oos_stats decomposes it into two ints so the client header stays C-linkable and server-type-free.
7. The transient structs scope tightly. HEAP_OOS_EXPAND_STATE is stack-local scratch declared in heap_record_replace_oos_oids; VACUUM_OOS_VFID_MEMO is a per-worker-per-block one-slot cache (never a static, to avoid a thread race), and its VACUUM_OOS_VFID_LOOKUP_RESULT tri-state caches FOUND/NONE but never ERROR.

Chapter 2: Initialization and Memory

The reader question this chapter answers: before any column can be externalized, what brings the per-class OOS file and the process-wide bestspace cache into existence, and when? The high-level companion (cubrid-oos.md, sections “Per-class file pairing” and “Bestspace”) states the design intent; this chapter traces the bootstrap branch by branch. Two levels of initialization exist and they are independent of each other:

• Process level (eager, once per server): the global in-memory bestspace cache oos_Bestspace, built at heap-manager boot. This is purely an in-RAM acceleration structure; it survives no restart and is never logged.
• File level (lazy, once per class on first externalization): the per-class FILE_OOS file, created the first time a column actually needs to spill out-of-line. A class that never externalizes a column never gets an OOS file, and a missing OOS file is not an error — it is the normal initial state.

The two structures this chapter touches are dissected field-by-field in Chapter 1; here is only the subset whose fields are load-bearing for the bootstrap paths below.

Field	Role	Why it matters here
oos_stats_bestspace_cache.vpid_ht	hash of entries keyed by page (best.vpid)	insert-time “free space on a known page” lookup; the table finalize maps the free callback over
oos_stats_bestspace_cache.vfid_ht	hash of the same entries keyed by file (vfid)	drop-time bulk eviction “forget every page of this file”; destroyed without per-entry callback to avoid double-free
oos_stats_bestspace_cache.free_list / free_list_count	recycle pool of detached OOS_STATS_ENTRY	lets oos_stats_entry_free recycle instead of free; finalize drains it in a second pass
oos_stats_bestspace_cache.bestspace_mutex	guards all four fields above	inited in branch B, destroyed last in finalize
heap_hdr_stats.oos_vfid	per-class persisted VFID of the OOS file	this is where the OOS file is stored/found for a class; VFID_ISNULL means “no file yet”
oos_hdr_stats.estimates.best[10] / second_best[10]	in-file bestspace hints	oos_create_file NULL-sets all 20 so the first sync scan reads them as empty

2.1 Process-level bootstrap — oos_bestspace_initialize

The cache lives in a single statically-allocated struct oos_Bestspace_cache_area and is reached through the file-static pointer oos_Bestspace. The cache is brought up by heap_manager_initialize (in heap_file.c), right after the heap’s own bestspace cache, and torn down symmetrically in heap_manager_finalize:

// heap_manager_initialize -- src/storage/heap_file.c
  ret = heap_stats_bestspace_initialize ();   /* heap's own cache first */
  if (ret != NO_ERROR) { return ret; }
  ret = oos_bestspace_initialize ();          /* <- OOS cache piggybacks on heap boot */
  if (ret != NO_ERROR) { return ret; }

So the OOS cache shares the heap manager’s lifetime exactly — there is no separate OOS subsystem boot. Note the finalize order is reversed: heap_manager_finalize tears down the OOS cache (oos_bestspace_finalize) before the heap’s own (heap_stats_bestspace_finalize), the standard LIFO teardown. oos_bestspace_initialize itself has four branches:

// oos_bestspace_initialize -- src/storage/oos_file.cpp
  if (oos_Bestspace != NULL)            /* branch A: double-init guard */
    {
      assert (false);                   /* <- a bug if it fires in a debug build */
      (void) oos_bestspace_finalize (); /* recover in release: tear down old, re-init */
    }
  oos_Bestspace = &oos_Bestspace_cache_area;   /* point at the static area */
  if (pthread_mutex_init (&oos_Bestspace->bestspace_mutex, NULL) != 0)  /* branch B */
    { er_set_with_oserror (...); return ER_CSS_PTHREAD_MUTEX_INIT; }
  oos_Bestspace->num_stats_entries = 0;
  oos_Bestspace->free_list_count = 0;
  oos_Bestspace->free_list = NULL;
  oos_Bestspace->vpid_ht = mht_create ("OOS best-space vpid hash table",
                                       OOS_BESTSPACE_CACHE_CAPACITY,        /* 1000 */
                                       oos_hash_vpid, oos_compare_vpid);
  if (oos_Bestspace->vpid_ht == NULL)   /* branch C: OOM on first table */
    { er_set (... ER_OUT_OF_VIRTUAL_MEMORY ...); return ER_OUT_OF_VIRTUAL_MEMORY; }
  oos_Bestspace->vfid_ht = mht_create ("OOS best-space vfid hash table",
                                       OOS_BESTSPACE_CACHE_CAPACITY,
                                       oos_hash_vfid, oos_compare_vfid);
  if (oos_Bestspace->vfid_ht == NULL)   /* branch D: OOM on second table */
    {
      mht_destroy (oos_Bestspace->vpid_ht);   /* <- roll back the first table */
      oos_Bestspace->vpid_ht = NULL;
      er_set (...); return ER_OUT_OF_VIRTUAL_MEMORY;
    }
  return NO_ERROR;

Invariant — the two hash tables are all-or-nothing. After a successful return, both vpid_ht and vfid_ht are non-NULL and sized to OOS_BESTSPACE_CACHE_CAPACITY (1000). Branch D enforces this on the OOM path: if the second mht_create fails it destroys the first table and nulls the pointer, so the caller never sees a half-built cache with one live table. If this invariant were violated, every later mht_put/mht_get against the missing table would dereference NULL. Branch B leaves both pointers NULL (they were never touched), which is also a consistent — if empty — state.

Why two tables for the same entries? Each OOS_STATS_ENTRY is indexed two ways simultaneously: by its page (vpid_ht, keyed on best.vpid) for the insert-time lookup “give me free space on a known page”, and by its file (vfid_ht, keyed on vfid) for the drop-time bulk eviction “forget every page belonging to this file”. The entry is one allocation shared between both tables; eviction must remove it from both (Chapter 5 covers oos_stats_del_bestspace_by_vfid).

flowchart TD
  A["oos_bestspace_initialize"] --> B{"oos_Bestspace != NULL?"}
  B -- "yes (re-entry bug)" --> C["assert false; finalize old"]
  B -- "no" --> D["point at static cache area"]
  C --> D
  D --> E{"pthread_mutex_init ok?"}
  E -- "no" --> F["return ER_CSS_PTHREAD_MUTEX_INIT"]
  E -- "yes" --> G["zero counters; mht_create vpid_ht"]
  G --> H{"vpid_ht != NULL?"}
  H -- "no" --> I["return ER_OUT_OF_VIRTUAL_MEMORY"]
  H -- "yes" --> J["mht_create vfid_ht"]
  J --> K{"vfid_ht != NULL?"}
  K -- "no" --> L["mht_destroy vpid_ht; return OOM"]
  K -- "yes" --> M["return NO_ERROR"]

Figure 2-1: oos_bestspace_initialize branch flow.

The hash/compare callbacks are trivial but worth pinning because they define the key identity used everywhere else. The VPID hash folds the 1-byte volid into the high byte of the 4-byte pageid before the modulo; the VFID hash does the same with fileid. Equality is delegated to the standard VPID_EQ / VFID_EQ macros:

// oos_hash_vpid -- src/storage/oos_file.cpp
  return ((vpid->pageid | ((unsigned int) vpid->volid) << 24) % htsize);
// oos_compare_vpid -- src/storage/oos_file.cpp
  return VPID_EQ (vpid1, vpid2);   /* <- mht "compare" returns nonzero on EQUAL */

Note the mht convention: oos_compare_* returns the equality result, not a C strcmp-style ordering — VPID_EQ is true (nonzero) when the keys match. Getting this backwards would make every lookup miss.

2.2 Process-level teardown — oos_bestspace_finalize and oos_stats_entry_free

oos_bestspace_finalize reverses initialize and is idempotent:

// oos_bestspace_finalize -- src/storage/oos_file.cpp
  if (oos_Bestspace == NULL) { return NO_ERROR; }   /* branch A: never inited / already gone */
  if (oos_Bestspace->vpid_ht != NULL)
    {
      (void) mht_map_no_key (NULL, oos_Bestspace->vpid_ht, oos_stats_entry_free, NULL);  /* <- recycle every live entry */
      mht_destroy (oos_Bestspace->vpid_ht);
      oos_Bestspace->vpid_ht = NULL;
    }
  if (oos_Bestspace->vfid_ht != NULL)
    {
      mht_destroy (oos_Bestspace->vfid_ht);   /* NO map_no_key here — see note */
      oos_Bestspace->vfid_ht = NULL;
    }
  OOS_STATS_ENTRY *ent = oos_Bestspace->free_list;   /* drain the recycle pool */
  while (ent != NULL) { OOS_STATS_ENTRY *next = ent->next; free_and_init (ent); ent = next; }
  oos_Bestspace->free_list = NULL;
  oos_Bestspace->free_list_count = 0;
  oos_Bestspace->num_stats_entries = 0;
  pthread_mutex_destroy (&oos_Bestspace->bestspace_mutex);
  oos_Bestspace = NULL;

Invariant — every live entry is freed exactly once, despite living in two tables. The walk runs oos_stats_entry_free only over vpid_ht (via mht_map_no_key). The vfid_ht is destroyed without a per-entry callback because both tables point at the same OOS_STATS_ENTRY objects; mapping the free callback over both would double-free. The single map-then-destroy on vpid_ht, plain destroy on vfid_ht, is what keeps the count at one.

There is a subtlety in oos_stats_entry_free that matters here. As a free-list recycler it normally does not free memory — it pushes the entry onto free_list for reuse:

// oos_stats_entry_free -- src/storage/oos_file.cpp
  if (ent != NULL)
    {
      if (oos_Bestspace->free_list_count < OOS_BESTSPACE_CACHE_CAPACITY)   /* room to recycle */
        { ent->next = oos_Bestspace->free_list; oos_Bestspace->free_list = ent; oos_Bestspace->free_list_count++; }
      else
        { free_and_init (ent); }   /* pool full: actually release */
    }
  return NO_ERROR;

So during finalize the mht_map_no_key pass moves live entries onto free_list (or frees them once the pool tops out at 1000), and the explicit while loop that follows then frees whatever the recycler parked on free_list. Net effect: every byte is released, in two passes. This dual role is why oos_stats_entry_free reads oos_Bestspace directly — it is both the per-entry eviction callback during normal operation (Chapters 5, 9, 10) and the finalize sweeper.

2.3 File-level pairing — heap_oos_find_vfid

A class’s OOS file does not exist until a column first needs it. The pairing point is heap_oos_find_vfid (in heap_file.c), which resolves a heap’s HFID to its OOS VFID, optionally creating the file. It deliberately mirrors heap_ovf_find_vfid (the long-standing overflow-file pairing). The oos_vfid is persisted in the heap header record HEAP_HDR_STATS.oos_vfid — that field is the canonical answer to “where is a class’s OOS file recorded.” In the HEAP_HDR_STATS struct it sits right after next_vpid (the field order is class_oid, ovf_vfid, next_vpid, oos_vfid, unfill_space), so the new OOS pointer was appended next to the long-standing ovf_vfid/next_vpid pair rather than reordering the header.

The header contract is stated up front in the function comment and is the single most important thing to internalize:

A false return ALWAYS means a real error (er_errid() is set); it never means “no OOS file”. Callers using docreate == false must inspect VFID_ISNULL(oos_vfid) to tell whether an OOS file exists.

The function fixes the heap header page, peeks the header record, and branches on whether the file already exists:

// heap_oos_find_vfid -- src/storage/heap_file.c
  success = true;
  VFID_SET_NULL (oos_vfid);                       /* <- default: "no file" */
  mode = (docreate == true ? PGBUF_LATCH_WRITE : PGBUF_LATCH_READ);  /* write latch only if we may stamp */
  addr_hdr.pgptr = pgbuf_fix (thread_p, &vpid, OLD_PAGE, mode, PGBUF_UNCONDITIONAL_LATCH);
  if (addr_hdr.pgptr == NULL) { goto exit_on_error; }
  if (spage_get_record (thread_p, addr_hdr.pgptr, HEAP_HEADER_AND_CHAIN_SLOTID, &hdr_recdes, PEEK) != S_SUCCESS)
    { goto exit_on_error; }
  heap_hdr = (HEAP_HDR_STATS *) hdr_recdes.data;
  if (VFID_ISNULL (&heap_hdr->oos_vfid))          /* no file recorded yet */
    {
      if (docreate == true)  { /* ... create + stamp, see below ... */ }
      else                   { goto end; }        /* <- branch: leave oos_vfid NULL, NOT an error */
    }
  else
    {
      VFID_COPY (oos_vfid, &heap_hdr->oos_vfid);  /* file exists: just hand it back */
    }
  goto end;

The latch mode is chosen up front: a read-only resolution (docreate == false) takes a PGBUF_LATCH_READ, while a creating resolution takes PGBUF_LATCH_WRITE because it is about to modify the header record. The create path is wrapped in a top system operation so the file creation, TDE application, and header stamp commit or abort as one unit:

// heap_oos_find_vfid -- src/storage/heap_file.c (docreate == true, VFID was NULL)
  TDE_ALGORITHM tde_algo = TDE_ALGORITHM_NONE;
  log_sysop_start (thread_p);                                  /* TOP SYSTEM OPERATION */
  if (oos_create_file (thread_p, *oos_vfid) != NO_ERROR)       /* path A: create failed */
    { log_sysop_abort (thread_p); goto exit_on_error; }
  if (heap_get_class_tde_algorithm (thread_p, &heap_hdr->class_oid, &tde_algo) != NO_ERROR)  /* path B */
    { log_sysop_abort (thread_p); goto exit_on_error; }
  if (file_apply_tde_algorithm (thread_p, oos_vfid, tde_algo) != NO_ERROR)  /* path C */
    { log_sysop_abort (thread_p); goto exit_on_error; }
  log_append_undo_data (thread_p, RVHF_STATS, &addr_hdr, sizeof (*heap_hdr), heap_hdr);  /* undo: old hdr (NULL vfid) */
  VFID_COPY (&heap_hdr->oos_vfid, oos_vfid);                   /* stamp the new vfid into the header */
  log_append_redo_data (thread_p, RVHF_STATS, &addr_hdr, sizeof (*heap_hdr), heap_hdr);  /* redo: new hdr */
  pgbuf_set_dirty (thread_p, addr_hdr.pgptr, DONT_FREE);
  log_sysop_commit (thread_p);

Invariant — the heap header’s oos_vfid and the on-disk OOS file are atomically consistent. The undo image captures the header before the stamp (VFID still NULL), the redo image captures it after. If the transaction rolls back, RVHF_STATS undo restores the NULL VFID, and the file created inside the same sysop is undone by the sysop’s own file-creation undo. Crash recovery sees either “no file, NULL header” or “file present, header points at it” — never a stamped header pointing at a file that was never created, nor an orphan file the header doesn’t reference. The TDE algorithm is applied inside the sysop too (path C), so an encrypted class never gets a half-encrypted OOS file.

flowchart TD
  A["heap_oos_find_vfid"] --> B["VFID_SET_NULL(oos_vfid)"]
  B --> C["pgbuf_fix header (READ or WRITE)"]
  C --> D{"fix + peek ok?"}
  D -- "no" --> E["exit_on_error: success=false"]
  D -- "yes" --> F{"heap_hdr.oos_vfid NULL?"}
  F -- "no" --> G["VFID_COPY existing -> oos_vfid"]
  F -- "yes" --> H{"docreate?"}
  H -- "false" --> I["end: oos_vfid stays NULL, success=true"]
  H -- "true" --> J["log_sysop_start"]
  J --> K{"oos_create_file ok?"}
  K -- "no" --> L["sysop_abort; exit_on_error"]
  K -- "yes" --> M{"get + apply TDE ok?"}
  M -- "no" --> L
  M -- "yes" --> N["undo old hdr; stamp vfid; redo new hdr; set_dirty"]
  N --> O["log_sysop_commit"]
  G --> P["end: unfix header; return success"]
  I --> P
  O --> P
  E --> P

Figure 2-2: heap_oos_find_vfid — every branch including the not-an-error NULL path.

The cleanup label end: always unfixes the header page if it was fixed (if (addr_hdr.pgptr) pgbuf_unfix_and_init (...)), so both the success and exit_on_error paths release the latch. The function returns success (true unless an error set it false).

2.4 File creation — oos_create_file

oos_create_file builds the physical artifact: a numerable FILE_OOS file plus a sticky first page that permanently holds the OOS_HDR_STATS bestspace header at slot 0.

// oos_create_file -- src/storage/oos_file.cpp
  tablespace.initial_size    = DB_PAGESIZE;                          /* one page to start */
  tablespace.expand_ratio    = (float) 0.01;
  tablespace.expand_min_size = DISK_SECTOR_NPAGES * DB_PAGESIZE;
  tablespace.expand_max_size = DISK_SECTOR_NPAGES * DB_PAGESIZE * 1024;
  err = file_create (thread_p, FILE_OOS, &tablespace, &des,
                     false /* is_temp */, true /* is_numerable */, &oos_vfid);  /* <- numerable */
  if (err != NO_ERROR)
    { oos_error (...); assert_release_error (er_errid () != NO_ERROR); assert (false); return err; }  /* branch A */

The file is numerable (is_numerable == true): pages can be addressed by ordinal, which the bestspace sync scan relies on (Chapter 5). It is not temporary, so it is WAL-protected and survives restart. After the file exists, a sysop wraps allocation of the sticky header page so a failure rolls back cleanly:

// oos_create_file -- src/storage/oos_file.cpp
  PAGE_TYPE page_type = PAGE_OOS;
  log_sysop_start (thread_p);
  err = file_alloc_sticky_first_page (thread_p, &oos_vfid, oos_vpid_init_new,
                                      &page_type, &hdr_vpid, &hdr_page);
  if (err != NO_ERROR || hdr_page == NULL)        /* branch B */
    { oos_error (...); log_sysop_abort (thread_p); return (err != NO_ERROR) ? err : ER_FAILED; }

The header page is sticky — file_alloc_sticky_first_page pins it as the file’s permanent first page so the bestspace header is always findable at a fixed location and is never reclaimed by vacuum. The page is initialized through the shared oos_vpid_init_new callback (2.5). Next the header record is constructed in a stack buffer, zeroed, then its embedded best-space hint arrays are explicitly NULL-set:

// oos_create_file -- src/storage/oos_file.cpp
  OOS_HDR_STATS hdr_stats;
  memset (&hdr_stats, 0, sizeof (OOS_HDR_STATS));
  hdr_stats.oos_vfid = oos_vfid;                              /* self-reference */
  VPID_SET_NULL (&hdr_stats.estimates.full_search_vpid);
  for (int i = 0; i < OOS_NUM_BEST_SPACESTATS; i++)           /* 10 slots */
    { VPID_SET_NULL (&hdr_stats.estimates.best[i].vpid);
      VPID_SET_NULL (&hdr_stats.estimates.second_best[i]); }
  RECDES hdr_recdes;
  hdr_recdes.area_size = hdr_recdes.length = sizeof (OOS_HDR_STATS);
  hdr_recdes.type = REC_HOME;
  hdr_recdes.data = (char *) &hdr_stats;
  PGSLOTID slotid;
  int sp_status = spage_insert (thread_p, hdr_page, &hdr_recdes, &slotid);
  if (sp_status != SP_SUCCESS)                                /* branch C */
    { oos_error (...); pgbuf_unfix_and_init (thread_p, hdr_page); log_sysop_abort (thread_p);
      er_set (... ER_GENERIC_ERROR ...); return ER_GENERIC_ERROR; }
  assert (slotid == 0);                                       /* header MUST land at slot 0 */

Invariant — the header record occupies slot 0 of the sticky first page. The page is freshly initialized and empty, so the first spage_insert deterministically returns slotid == 0, asserted explicitly. Every reader (e.g. oos_get_header_stats_ptr, Chapter 1) hard-codes slot 0 to fetch the header. memset-zero plus the explicit VPID_SET_NULL loop matters because a NULL VPID is not all-zero bytes in every build — the loop guarantees the 10 best[] and 10 second_best[] hints start as genuine NULL VPIDs, so the first sync scan treats them as empty rather than as bogus page hints.

The insert is then logged and the sysop committed:

// oos_create_file -- src/storage/oos_file.cpp
  log_addr.vfid = &oos_vfid; log_addr.pgptr = hdr_page; log_addr.offset = slotid;
  log_append_undoredo_recdes (thread_p, RVOOS_INSERT, &log_addr, NULL, &hdr_recdes);  /* undo NULL, redo the header */
  pgbuf_set_dirty (thread_p, hdr_page, FREE);    /* FREE: unfix as part of set_dirty */
  log_sysop_commit (thread_p);
  return NO_ERROR;

Note pgbuf_set_dirty(..., FREE) unfixes the header page here, whereas the page-init callback used DONT_FREE (2.5) because the caller still needed the page. The branch-B/branch-C error paths abort the sysop inside the outer top-sysop opened by heap_oos_find_vfid (2.3), so a creation failure unwinds both nested operations.

2.5 Page initialization — oos_vpid_init_new and oos_file_alloc_new

oos_vpid_init_new is the per-page initializer handed to both file_alloc_sticky_first_page (header page) and file_alloc (data pages). It establishes a PAGE_OOS-typed, ANCHORED slotted page:

// oos_vpid_init_new -- src/storage/oos_file.cpp
  PAGE_TYPE ptype = * (PAGE_TYPE *) args;        /* always PAGE_OOS in practice */
  pgbuf_set_page_ptype (thread_p, page, ptype);
  spage_initialize (thread_p, page, ANCHORED, OOS_ALIGNMENT, false);  /* ANCHORED = slot ids stay fixed */
  log_append_undoredo_data2 (thread_p, RVPGBUF_NEW_PAGE, NULL, page, (PGLENGTH) ptype, 0,
                             SPAGE_HEADER_SIZE, NULL, (SPAGE_HEADER *) page);  /* ptype carried in redo */
  pgbuf_set_dirty (thread_p, page, DONT_FREE);   /* DONT_FREE: caller keeps the fix */
  return err;

Two design choices are load-bearing. ANCHORED slot ownership means a slot id never moves once assigned — essential because OOS OIDs in heap records and in multi-chunk next_chunk_oid links (Chapter 6) embed (vpid, slotid) permanently; a non-anchored page that renumbered slots on compaction would invalidate every stored OID. OOS_ALIGNMENT is MAX_ALIGNMENT, matching the alignment the OOS record header assumes. The RVPGBUF_NEW_PAGE log record stashes ptype as its redo length so pgbuf_rv_new_page_redo can restore the page type during recovery.

oos_file_alloc_new is the data-page allocator (used by the insert path in later chapters, not by oos_create_file, which uses the sticky-first-page variant). It wraps file_alloc in its own sysop and returns an auto-unfixing page handle:

// oos_file_alloc_new -- src/storage/oos_file.cpp  (returns auto_unfix_page_ptr)
  PAGE_TYPE page_type = PAGE_OOS;
  log_sysop_start (thread_p);
  err = file_alloc (thread_p, &oos_vfid, oos_vpid_init_new, &page_type, &vpid_out, nullptr);
  if (err != NO_ERROR)                            /* branch: alloc failed */
    { oos_error (...); assert_release_error (er_errid () != NO_ERROR); assert (false);
      log_sysop_abort (thread_p); return nullptr; }
  log_sysop_commit (thread_p);
  return pgbuf_fix_auto_unfix (thread_p, &vpid_out, OLD_PAGE, PGBUF_LATCH_WRITE, PGBUF_UNCONDITIONAL_LATCH);

The return type is auto_unfix_page_ptr (a std::unique_ptr<char, page_auto_unfix> from page_buffer_util.hpp): on the failure branch it returns nullptr, and on success it re-fixes the freshly allocated page with a write latch wrapped in an RAII handle that unfixes on scope exit. Allocating a page is a self-contained, immediately-durable sysop — the page exists regardless of whether the enclosing transaction later commits, which is acceptable because an unused OOS data page is reclaimable by vacuum (Chapter 10).

graph TD
  CF["oos_create_file"] -->|"file_alloc_sticky_first_page"| INIT["oos_vpid_init_new"]
  AN["oos_file_alloc_new"] -->|"file_alloc"| INIT
  INIT --> P["PAGE_OOS, ANCHORED slotted page, logged via RVPGBUF_NEW_PAGE"]
  CF --> SLOT0["spage_insert OOS_HDR_STATS at slot 0"]

Figure 2-3: who initializes pages, and how the header lands at slot 0.

2.6 The FILE_OOS file-type registration in file_manager

oos_create_file calls file_create with the dedicated file type FILE_OOS, which is a first-class member of the FILE_TYPE enum in file_manager.h. Registering a new type is more than naming it — the type has to be threaded through the file-manager switch statements that special-case behavior. Two of those switches matter for correctness; the rest are deliberately unfinished, which is the central feat/oos caveat of this chapter.

The one place FILE_OOS is fully wired is the durable-file VFID-uniqueness guard in file_create_with_npages / file_perm_alloc’s reservation step, which groups OOS with the other permanent, recovery-visible types:

// file_create internals -- src/storage/file_manager.c (~line 3448)
  if (file_type == FILE_BTREE || file_type == FILE_HEAP || file_type == FILE_HEAP_REUSE_SLOTS
      || file_type == FILE_OOS)
    {
      /* we need to consider dropped files in vacuum's list. If we create a file with a
       * duplicate VFID, we can run into problems. */
      ...
    }

OOS belongs in that set because, like heap and btree files, an OOS file is permanent (not temp), survives restart, and its VFID can collide with a still-pending dropped-file entry in vacuum’s list. The human-readable name is registered too: file_type_to_string returns "OUT_OF_LINE_OVERFLOW_STORAGE" for FILE_OOS.

Invariant (in-development) — OOS files are NOT yet wired into the file-manager’s class-OID / tracker-protection / spacedb paths; those arms are intentional assert(false) stubs. Several switch (... type ...) arms that exist for FILE_HEAP/FILE_BTREE carry only an assert(false); break; for FILE_OOS:

// file_header_dump_descriptor -- src/storage/file_manager.c
    case FILE_OOS:
      {
        assert (false);   /* <- reached only if OOS enters this path; not yet expected */
        break;
      }

Five such arms exist. Four are plain assert(false) stubs: (1) the descriptor printing in file_header_dump_descriptor, and three inside file_tracker_get_and_protect — (2) the desired_type accept filter, (3) the item->type lock-protection switch, and (4) the class-OID extraction switch used when iterating files for a class. The fifth lives in the spacedb file-type classification and is slightly different — an assert_release(false) followed by an explicit workaround:

// file_tracker_item_spacedb -- src/storage/file_manager.c
    case FILE_OOS:
      assert_release (false);
      // TODO: spacedb_ftype = SPACEDB_OOS_FILE;
      spacedb_ftype = SPACEDB_HEAP_FILE;   /* workaround: no SPACEDB_OOS_FILE enum yet */
      break;

The comment there explains the kludge: a dedicated SPACEDB_OOS_FILE category was not added, and leaving spacedb_ftype uninitialized broke the CI build, so OOS is temporarily counted as a heap file in spacedb output. The implication for a reader is concrete: in feat/oos an OOS file is created, stamped into the heap header, and read/written by the OOS layer, but it does not participate in the generic file-iteration-by-class machinery and is mis-reported by spacedb — which is consistent with the teardown stub in 2.7 (oos_remove_page) and the incomplete vacuum integration of Chapter 10. A debug build will abort if some future caller routes an OOS file through the four assert(false) switches before they are filled in.

2.7 Teardown stubs — oos_remove_file and oos_remove_page

Two teardown primitives exist. oos_remove_file is now wired into the heap drop paths — both xheap_destroy and xheap_destroy_newly_created call heap_oos_find_vfid (..., docreate == false) and then oos_remove_file when the returned VFID is non-NULL — but the page-level reclaim hook (oos_remove_page) remains unwired pending vacuum.

oos_remove_file evicts the file’s bestspace entries from the cache and then defers the physical file destroy to commit time:

// oos_remove_file -- src/storage/oos_file.cpp
  (void) oos_stats_del_bestspace_by_vfid (thread_p, &oos_vfid);  /* drop cache entries via vfid_ht */
  file_postpone_destroy (thread_p, &oos_vfid);                   /* <- deferred: runs at transaction commit */
  return NO_ERROR;

file_postpone_destroy registers the destroy as a postpone action so the file is only actually removed once the dropping transaction commits — until then a rollback can keep it. The cache eviction is the vfid_ht payoff from 2.1: one call forgets every page of the file at once.

oos_remove_page is the page-level dealloc, explicitly commented as the future vacuum entry point and not yet wired into a caller:

// oos_remove_page -- src/storage/oos_file.cpp
  // TODO: will be called by vacuum when OOS vacuum is implemented
  int err = file_dealloc (thread_p, &oos_vfid, &vpid, FILE_OOS);
  if (err != NO_ERROR) { oos_error (...); return err; }
  return NO_ERROR;

Chapter 10 (Vacuum Reclamation) revisits why this is still a stub: OOS page reclamation is gated on the vacuum integration that is incomplete in feat/oos — the same incompleteness that leaves the file_manager arms in 2.6 as assert(false).

2.8 Chapter summary — key takeaways

1. Two independent bootstraps. The process-wide bestspace cache (oos_Bestspace) is built eagerly at heap-manager boot; the per-class FILE_OOS file is built lazily on first column externalization. Neither depends on the other, and finalize tears them down LIFO (OOS before heap).
2. The cache’s two tables are an all-or-nothing pair keyed on VPID (page lookup) and VFID (file-wide eviction) over shared entries — which is why finalize maps the free callback over vpid_ht only and plain-destroys vfid_ht, avoiding a double-free.
3. oos_stats_entry_free is a recycler, not a deallocator, under normal operation — it parks entries on a 1000-slot free_list; finalize drains that list in a second explicit pass so all memory is released.
4. A class’s OOS file is found via HEAP_HDR_STATS.oos_vfid, and a missing OOS file is the normal initial state, never an error. heap_oos_find_vfid returns false only on a genuine error; callers with docreate == false must test VFID_ISNULL to detect “no file.”
5. File creation and header stamping are atomic. The create path runs inside a top system operation with RVHF_STATS undo/redo around the VFID stamp and TDE applied in the same sysop, so recovery never sees an orphan file or a dangling header reference.
6. The bestspace header is permanently pinned at slot 0 of a sticky first page of a numerable, non-temporary FILE_OOS file; the slotid == 0 assertion is what every header reader depends on.
7. FILE_OOS is a registered file type but only partly integrated. It joins the durable-file VFID-uniqueness guard and the type-name table, yet five file_manager switch arms — descriptor print, tracker desired_type filter, tracker lock protection, class-OID extraction (all assert(false)), and spacedb classification (assert_release(false) + a “count as heap” workaround) — are unfinished stubs, matching oos_remove_page’s unwired vacuum hook and the incomplete feat/oos vacuum path of Chapter 10.

Chapter 3: The Demotion Decision and the Inline Stub

A row arrives at the heap layer as a HEAP_CACHE_ATTRINFO — an in-memory bag of DB_VALUEs with no opinion yet about where any column will physically live. Before that row can be serialized into a RECDES and slotted onto a page, the engine must answer one question: will the whole thing fit, and if not, which columns get evicted to the OOS file? That decision lives in exactly one place — heap_attrinfo_determine_disk_layout — traced here branch by branch. For the “why externalize at all” framing (the page-size ceiling, the contrast with classic TEXT/CLOB overflow, the PG-TOAST analogy), see the companion cubrid-oos.md, CUBRID’s Approach and its Trigger heuristic differs from the design intent cross-check note; that theory is not repeated here.

The function’s contract is narrow: it is a pure sizing pass. It mutates no page, allocates no OID, writes no bytes. It returns the predicted disk size and, through two out-parameters (oos_columns and has_oos), the demotion verdict that the later serializer (Ch 4) obeys.

3.1 The three size components

The function opens by reducing the row to three integers.

// heap_attrinfo_determine_disk_layout -- src/storage/heap_file.c
  std::vector<int> column_size (attr_info->num_values);
  int payload_size, header_size;
  int mvcc_extra;

  *has_oos = false;

  /* calcuate the entire size of columns */
  payload_size = heap_attrinfo_get_record_payload_size (attr_info, &column_size);
  header_size  = heap_attrinfo_get_record_header_size (attr_info, payload_size, is_mvcc_class, offset_size_ptr);
  mvcc_extra   = is_mvcc_class ? OR_MVCC_MAX_HEADER_SIZE - OR_MVCC_INSERT_HEADER_SIZE : 0;

Each term is computed by a dedicated helper, and the side effect of column_size is what makes the later candidate loop possible — it is the per-column byte ledger.

payload_size comes from heap_attrinfo_get_record_payload_size. It walks every value and fills column_size[i] with that column’s on-disk byte count, branching on whether the column is fixed:

// heap_attrinfo_get_record_payload_size -- src/storage/heap_file.c
      if (value->last_attrepr->is_fixed != 0)
        {
          (*column_size)[i] = tp_domain_disk_size (value->last_attrepr->domain); /* <- fixed: domain dictates width */
          size += (*column_size)[i];
        }
      else
        {
          (*column_size)[i] = pr_data_writeval_disk_size (&value->dbvalue);      /* <- variable: actual value width */
          size += (*column_size)[i];
        }

This is_fixed fork does not itself decide eligibility — both branches add to size and populate column_size[i]. What it does is record the flag that later excludes CHAR: a fixed CHAR(n) reports a domain-driven width via tp_domain_disk_size, while a VARCHAR/STRING/BLOB-style column reports the actual serialized width of its value. The same is_fixed flag is the gate in the candidate loop (§3.3), so a fixed column can never be demoted no matter how large its domain. The exclusion happens there, not here; this helper only sizes.

header_size comes from heap_attrinfo_get_record_header_size. The base is OR_MVCC_INSERT_HEADER_SIZE (16) for MVCC classes or OR_NON_MVCC_HEADER_SIZE (8) otherwise, plus the variable-offset table (VOT) and the bound-bit bytes. Critically, this helper escalates the offset width as the record grows — it starts at OR_BYTE_SIZE (1 byte per offset) and bumps to OR_SHORT_SIZE (2) then OR_INT_SIZE (4) bytes when header_size + payload_size crosses OR_MAX_BYTE (127) or OR_MAX_SHORT (32767):

// heap_attrinfo_get_record_header_size -- src/storage/heap_file.c
  *offset_size_ptr = OR_BYTE_SIZE;
  header_size = is_mvcc_class ? OR_MVCC_INSERT_HEADER_SIZE : OR_NON_MVCC_HEADER_SIZE;
  header_size += OR_VAR_TABLE_SIZE_INTERNAL (attr_info->last_classrepr->n_variable, *offset_size_ptr);
  header_size += OR_BOUND_BIT_BYTES (attr_info->last_classrepr->n_attributes - attr_info->last_classrepr->n_variable);

  if (*offset_size_ptr == OR_BYTE_SIZE && header_size + payload_size > OR_MAX_BYTE)
    { /* ... rebuild VOT at OR_SHORT_SIZE ... */ }
  if (*offset_size_ptr == OR_SHORT_SIZE && header_size + payload_size > OR_MAX_SHORT)
    { /* ... rebuild VOT at OR_INT_SIZE ... */ }

This is why the header must be recomputed after demotion (§3.5): shrinking payload_size can drop the offset width back down, which in turn shrinks the header again. The two feed back into each other.

mvcc_extra is the headroom reservation. An MVCC row is first written with only OR_MVCC_INSERT_HEADER_SIZE (16: repid/chn + INSID), but it may later grow in place to OR_MVCC_MAX_HEADER_SIZE (32) when a DELID and a prev-version LSA are stamped on update. The difference, 32 - 16 = 16 bytes, is reserved now so a future in-place header growth never overflows the page. For non-MVCC classes there is no later growth, so mvcc_extra = 0.

Invariant — sizing must account for the row’s maximum footprint, not its current one. The threshold test uses header_size + payload_size + mvcc_extra, never just header_size + payload_size. If mvcc_extra were omitted, a row could be admitted at exactly DB_PAGESIZE/4 and then overflow the slot when an MVCC update later expands its header by 16 bytes. The reservation is what lets later in-place DELID stamping (Ch 9) be a no-relocation operation.

3.2 The threshold test — DB_PAGESIZE/4

With the three components in hand, the whole demotion machinery is gated behind one comparison:

// heap_attrinfo_determine_disk_layout -- src/storage/heap_file.c
  /* TODO: change the statistics */
  /* push the largest variable column to OOS one by one until the heap record
   * fits within DB_PAGESIZE/4 (PG TOAST style), instead of pushing every eligible column */
  if (header_size + payload_size + mvcc_extra > DB_PAGESIZE / 4)
    {
      /* ... build candidates, sort, externalize-largest-first ... */
    }
  return header_size + payload_size;

If the row already fits within a quarter page, the entire if body is skipped: *has_oos stays false, *oos_columns is left untouched (all-false), and the function returns header_size + payload_size. This early-skip is the common case; most rows never touch the OOS path.

The return value deliberately excludes mvcc_extra. That term exists only to make the admission decision conservative; the value the caller stores for the record-descriptor length is the actual serialized inline size. The reserved 16 bytes are accounted for elsewhere by the MVCC header machinery, not double-counted into the returned size.

Condition — the live trigger is header_size + payload_size + mvcc_extra > DB_PAGESIZE / 4. Verified against heap_attrinfo_determine_disk_layout at this revision. Note this is not the rule stated in the OOS design epic. Per cubrid-oos.md’s Trigger heuristic differs from the design intent note, the epic specifies row size > DB_PAGESIZE/8 (2 KB at a 16 KB page) AND column size > 512 B. The shipped code uses a flat DB_PAGESIZE/4 row test and a per-column eligibility floor of OR_OOS_INLINE_SIZE (16 B), not 512 B. The two are not equivalent: at a 16 KB page the epic’s row rule fires at ~2048 bytes while the code fires at 4096 bytes, so the implementation externalizes less aggressively and keeps larger rows inline. The /* TODO: change the statistics */ comment immediately above the test is the live acknowledgment that the threshold (and the cost model that should drive it) is provisional. Treat DB_PAGESIZE / 4 as the single source of truth in the code and the epic’s /8 + 512B formula as stale.

3.3 Building the candidate set

Once the row is over budget, the function collects the columns that are worth externalizing into a vector of {size, attr-index} pairs:

// heap_attrinfo_determine_disk_layout -- src/storage/heap_file.c
      std::vector<std::pair<int, int>> oos_candidates;  /* {column_size, attr index} */

      for (i = 0; i < attr_info->num_values; i++)
        {
          /* a variable column is OOS-eligible only if externalizing it shrinks the inline record:
           * its value must be larger than the OOS stub (OID + length) it is replaced with */
          if (!attr_info->values[i].last_attrepr->is_fixed && column_size[i] > OR_OOS_INLINE_SIZE)
            {
              oos_candidates.emplace_back (column_size[i], i);
            }
        }

Two conditions, both mandatory (&&):

1. !is_fixed — the column must be variable-length. This is the single, decisive CHAR exclusion (§3.1 only recorded the flag; the eligibility decision is made here). A fixed CHAR(n), INT, DATE, etc. can never be demoted; OOS is purely a variable-length-payload mechanism. Even a CHAR(8000) whose domain width dwarfs the threshold is ineligible — its bytes stay inline and the row simply spills toward the offset-width escalation in §3.1 instead.
2. column_size[i] > OR_OOS_INLINE_SIZE — the value must be strictly larger than the 16-byte stub that will replace it. Demoting a value that is 16 bytes or smaller would grow the inline record (you remove column_size[i] bytes but add back 16), so such columns are filtered out. This guarantees every demotion makes net forward progress toward the fit.

Columns failing either test are never added, so the loop in §3.4 only ever shrinks the record.

3.4 Externalize-largest-first

The candidates are sorted descending by size, then drained until the record fits:

// heap_attrinfo_determine_disk_layout -- src/storage/heap_file.c
      std::sort (oos_candidates.begin (), oos_candidates.end (), std::greater<std::pair<int, int>> ());

      for (auto & cand : oos_candidates)
        {
          if (header_size + payload_size + mvcc_extra <= DB_PAGESIZE / 4)
            {
              break;                          /* <- fits now: stop, leave the rest inline */
            }
          (*oos_columns)[cand.second] = true; /* <- mark this attr index for demotion */
          payload_size -= cand.first;         /* <- remove the value's inline bytes ... */
          payload_size += OR_OOS_INLINE_SIZE; /* <- ... and add the 16-byte stub */
          *has_oos = true;
        }

Because std::pair sorts lexicographically and the first element is the size, std::greater orders by size descending (ties broken by descending attr index — immaterial to correctness). Evicting the largest column first removes the most inline bytes per demotion, so the loop converges in the fewest demotions — the same greedy heuristic PostgreSQL TOAST uses.

The fit re-check sits at the top of each iteration, so it runs before the first eviction too. That ordering matters: the §3.2 guard already proved the record is over budget, so the first iteration’s check is guaranteed false and at least one column is always demoted once the body is entered. On every subsequent iteration the check can short-circuit — the instant payload_size has shrunk enough, the loop breaks and every remaining candidate stays inline. That is the entire point of the PG-TOAST style: demote the minimum number of columns, not all eligible ones. Each demotion subtracts the column’s full inline width (cand.first), adds OR_OOS_INLINE_SIZE for the replacing stub, and sets *has_oos; *oos_columns records which attr index was demoted.

Invariant — payload_size after the loop equals exactly what the serializer will emit. Each iteration’s -= cand.first; += OR_OOS_INLINE_SIZE mirrors, byte for byte, what Ch 4’s serializer does when it writes a stub in place of a value. If the two ever diverged (e.g. the loop subtracted a stale size, or the serializer wrote a differently-sized stub), the RECDES.length reserved here would not match the bytes actually written, corrupting the slot. The shared constant OR_OOS_INLINE_SIZE is what keeps the sizing pass and the writing pass in lockstep.

The exhaustion case is silent and intentional: if the loop runs out of candidates while the record is still over DB_PAGESIZE/4 (e.g. mostly fixed columns), there is no error — the function returns the best-effort reduced size. The slotted-page layer’s hard ceiling is the page size itself, not DB_PAGESIZE/4; the quarter-page test is a placement target, not a correctness limit. A row that cannot be squeezed under a quarter page but still fits a whole page is accepted as-is.

flowchart TD
  A["enter: compute payload_size,<br/>header_size, mvcc_extra"] --> B{"hdr + payload + mvcc_extra<br/>&gt; DB_PAGESIZE/4 ?"}
  B -- no --> Z["return hdr + payload<br/>(has_oos = false)"]
  B -- yes --> C["build oos_candidates:<br/>!is_fixed AND size &gt; 16"]
  C --> D["sort candidates<br/>descending by size"]
  D --> E{"next candidate?"}
  E -- none left --> H["recompute header_size<br/>from new payload_size"]
  E -- yes --> F{"hdr + payload + mvcc_extra<br/>&lt;= DB_PAGESIZE/4 ?"}
  F -- yes, fits --> H
  F -- no --> G["mark oos_columns[idx]=true<br/>payload -= size; payload += 16<br/>has_oos = true"]
  G --> E
  H --> Z2["return hdr + payload"]

Figure 3-1. Branch-complete control flow of heap_attrinfo_determine_disk_layout.

3.5 Recomputing the header

After the loop the header is rebuilt from the now-smaller payload:

// heap_attrinfo_determine_disk_layout -- src/storage/heap_file.c
      /* re-calculate the header size */
      header_size = heap_attrinfo_get_record_header_size (attr_info, payload_size, is_mvcc_class, offset_size_ptr);

  return header_size + payload_size;

This second call is not cosmetic. Recall from §3.1 that the helper chooses the VOT offset width (1/2/4 bytes) from header_size + payload_size, escalating at OR_MAX_BYTE (127) and OR_MAX_SHORT (32767). A pre-demotion payload of, say, 40000 bytes is above OR_MAX_SHORT, so it forced 4-byte offsets; after demoting a 38000-byte column the payload drops to a few thousand bytes — back below OR_MAX_SHORT, so the offset width falls to 2 bytes (and below OR_MAX_BYTE it would fall to 1), shrinking the header further. The recompute also re-writes *offset_size_ptr so the caller serializes the VOT at the correct width. The final return — header_size + payload_size — is the authoritative inline size the caller reserves in the record descriptor.

The VOT entry count is unchanged by demotion: a demoted column still occupies its offset slot. What changes is the kind of that entry, the subject of §3.6.

3.6 What the stub is, and the two markers

The verdict produced by this function is realized later (Ch 4) as a 16-byte inline stub written where the value used to be. Its size is the constant this whole chapter pivots on:

// OR_OOS_INLINE_SIZE -- src/base/object_representation.h
/* OOS inline size: OOS OID (8 bytes) + OOS length (8 bytes) */
#define OR_OOS_INLINE_SIZE (OR_OID_SIZE + OR_BIGINT_SIZE)   /* 8 + 8 = 16 */

So the stub is 8 bytes of OOS OID + 8 bytes of BIGINT length = 16 bytes. The OID points at the externalized chunk (or chain head) in the OOS file; the BIGINT carries the full logical length so a reader can size its buffer without first touching the OOS file. (Chunk-chain mechanics and how the length is consumed are Ch 6-7.)

Two independent markers tell the rest of the engine that demotion happened, at two granularities. Both are stamped by the serializer in Ch 4, in two different helpers.

Per-column marker — OR_SET_VAR_OOS on the VOT entry. When the serializer (heap_attrinfo_transform_variable_to_disk) writes a demoted column’s offset-table entry, it tags that entry’s low bit so readers know “this offset points at a 16-byte stub, not an ordinary inline value”:

// OR_SET_VAR_OOS / OR_VAR_BIT_OOS -- src/base/object_representation.h
#define OR_VAR_BIT_OOS 0x1
#define OR_SET_VAR_OOS(length) ((int) (length) | OR_VAR_BIT_OOS)

This is feasible because VOT offsets are INT_ALIGNMENT-aligned, leaving the low 2 bits free for flags (OR_VAR_BIT_OOS = 0x1 and OR_VAR_BIT_LAST_ELEMENT = 0x2, masked together by OR_VAR_FLAG_MASK = 0x3). The helper applies the tag precisely on the demoted column, then writes the 16-byte stub itself:

// heap_attrinfo_transform_variable_to_disk -- src/storage/heap_file.c
      length = CAST_BUFLEN (*ptr_varvals - buf->buffer - header_size);
      if (is_oos)
        {
          assert (dbvalue != NULL && db_value_is_null (dbvalue) != true);
          /* see Implementation in CBRD-26352 for details on why this design is possible. */
          /* use 2-bit of offset value as flags */
          length = OR_SET_VAR_OOS (length);     /* <- tag THIS offset entry as an OOS stub */
        }
      or_put_offset_internal (buf, length, offset_size);
  /* ... then, when is_oos, the stub itself: */
      buf->ptr = *ptr_varvals;
      or_put_oid (buf, oos_oid);                /* 8 bytes: the OOS OID  */
      or_put_bigint (buf, oos_length);          /* 8 bytes: the logical length */

A reader recovers the flag with OR_IS_OOS(length) and the clean offset with OR_GET_VAR_OFFSET(length) (masking off the low 2 bits with ~OR_VAR_FLAG_MASK). A NULL value is never OOS — the assert enforces that demotion only happens to a non-NULL value, which has payload bytes to externalize.

Per-row marker — OR_MVCC_FLAG_HAS_OOS in the record header. Independently, the row’s repid/flag word is stamped (in heap_attrinfo_transform_header_to_disk) so a scan can cheaply skip the OOS-aware re-inlining path (Ch 7) for rows that have no externalized columns at all:

// heap_attrinfo_transform_header_to_disk -- src/storage/heap_file.c
  if (has_oos)
    {
      repid_bits |= (OR_MVCC_FLAG_HAS_OOS << OR_MVCC_FLAG_SHIFT_BITS);
    }

with the flag defined as a distinct bit alongside the INSID/DELID flags (OR_MVCC_FLAG_SHIFT_BITS = 24 shifts it into the repid/flag word’s high byte):

// OR_MVCC_FLAG_HAS_OOS -- src/base/object_representation_constants.h
#define OR_MVCC_FLAG_VALID_INSID          0x01
#define OR_MVCC_FLAG_VALID_DELID          0x02
#define OR_MVCC_FLAG_VALID_PREV_VERSION   0x04
// TODO: OOS is not related to MVCC, move to another place when POC ends
#define OR_MVCC_FLAG_HAS_OOS              0x08  // 0b00001000

The two markers are complementary, neither redundant. The per-row flag is the fast gate — (mvcc_flags & OR_MVCC_FLAG_HAS_OOS) != 0 decides in one bitwise test whether the row needs OOS handling at all. The per-column bit then identifies exactly which offset entries are stubs, so a row with one demoted column out of ten pays no re-inlining cost on the other nine. The *has_oos set in this chapter’s loop is the boolean that flows into the per-row flag; the *oos_columns bitmap is what drives the per-column OR_SET_VAR_OOS tagging. (The source comment marks OR_MVCC_FLAG_HAS_OOS as not MVCC-related and slated to move out of the MVCC flag block once the feature leaves POC — the bit’s position may drift, but its role does not.)

3.7 Chapter summary — key takeaways

1. heap_attrinfo_determine_disk_layout is a pure sizing pass: it computes payload_size, header_size, and mvcc_extra, decides demotions, and returns the predicted inline size, but writes no bytes and allocates no OID. The actual stub writing is Ch 4.
2. The demotion trigger is a single test, header_size + payload_size + mvcc_extra > DB_PAGESIZE/4. mvcc_extra (= 16 for MVCC classes, the OR_MVCC_MAX_HEADER_SIZE - OR_MVCC_INSERT_HEADER_SIZE reservation) makes admission conservative so a later in-place DELID stamp never overflows the slot.
3. CHAR / fixed columns are never demoted. The decisive exclusion is the !is_fixed candidate gate (§3.3); the is_fixed fork in payload sizing (§3.1) only records the flag and picks the size formula. OOS is strictly a variable-length-payload mechanism.
4. Eligibility also requires column_size[i] > OR_OOS_INLINE_SIZE: a value must be larger than the 16-byte stub that replaces it, guaranteeing every demotion is net-shrinking.
5. The externalize loop is largest-first and minimal: candidates are sorted descending and drained only until the record fits, re-checking before each eviction, so it demotes the fewest columns possible (PG-TOAST style) and leaves the rest inline. Header size is recomputed afterward because shrinking the payload can drop the VOT offset width (it escalates at OR_MAX_BYTE = 127 and OR_MAX_SHORT = 32767).
6. The stub is OR_OOS_INLINE_SIZE = OR_OID_SIZE (8) + OR_BIGINT_SIZE (8) = 16 bytes: an OOS OID plus a BIGINT logical length. The loop’s payload -= size; payload += 16 keeps the sizing pass byte-aligned with the serializer.
7. Two markers record the verdict, written by two different Ch 4 helpers: per-column OR_SET_VAR_OOS (low bit of the VOT offset entry, set in heap_attrinfo_transform_variable_to_disk) identifies which offsets are stubs, and per-row OR_MVCC_FLAG_HAS_OOS (0x08, set in heap_attrinfo_transform_header_to_disk) is the fast gate that tells a scan whether to bother with OOS handling at all. The shipped DB_PAGESIZE/4 threshold diverges from the epic’s DB_PAGESIZE/8 row rule, flagged live by the /* TODO: change the statistics */ comment.

Chapter 4: INSERT of a Single Chunk Value

This chapter answers one question: once the demotion decision (Ch.3) has flagged a column as out-of-space (OOS), how is that column’s serialized value written into the per-class OOS file, and how does the resulting head-chunk OID flow back into the 16-byte inline stub the heap record carries? Scope is the single-chunk case — a value whose serialized form fits one page. The multi-chunk split (oos_insert_across_pages) and its replication discipline are deferred to Chapter 6; page selection (oos_find_best_page) is a black box opened in Chapter 5. For the inline-stub layout (OR_OOS_INLINE_SIZE = 16) and the demotion heuristic, see the high-level companion’s INSERT and The trigger heuristic sections.

4.1 The two-layer split: heap orchestration vs. storage primitive

The path has a clean seam. The heap layer owns the loop over columns, the scratch serialization buffer, and the bookkeeping vectors the disk-transform and replication paths later consume. The storage layer (oos_insert) owns nothing about columns — it takes an opaque byte span and returns an OID.

flowchart TD
  A["heap_attrinfo_transform_to_disk_internal\nhas_oos == true"] --> B["heap_attrinfo_insert_to_oos"]
  B --> C["resolve OOS VFID\nheap_oos_find_vfid docreate=true"]
  C --> D["clear tdes oos_insert_lsa_queue\nclear thread_p oos_oids"]
  D --> E["per-column loop"]
  E -->|oos_columns i| F["serialize dbvalue\nheap_attrinfo_dbvalue_to_recdes"]
  F --> G["oos_insert\nsrc = recdes.data, recdes.length"]
  G --> H["push oid to thread_p oos_oids\nstore oid + length into vectors"]
  H --> E
  E -->|done| I["S_SUCCESS\ncolumns now externalized"]

Figure 4-1: Top-level INSERT externalization. The vectors oos_oids / oos_lengths are written here and consumed later by heap_attrinfo_transform_columns_to_disk, which writes the OID+length stub into the record’s variable-offset slot.

A structural detail in the caller heap_attrinfo_transform_to_disk_internal: the if (has_oos) block calls heap_attrinfo_insert_to_oos once, before the do { ... } while (status == S_DOESNT_FIT) retry loop that builds the heap record (and calls heap_attrinfo_transform_columns_to_disk to write the stubs). Externalization is therefore not repeated when the in-page record build overflows and retries — the OIDs are minted once, and the loop only re-serializes the (now small) inline stubs.

4.2 heap_attrinfo_insert_to_oos — resolve, reset, loop

The signature is bracketed by // *INDENT-OFF* / *INDENT-ON* because its std::vector template arguments would be mangled by the C formatter. It opens by aiming recdes.data at a stack scratch buffer, then does three things.

Step 1 — resolve the OOS file with docreate = true:

// heap_attrinfo_insert_to_oos -- src/storage/heap_file.c
char recbuf[IO_MAX_PAGE_SIZE + MAX_ALIGNMENT];   /* <- stack scratch for serialized value */
recdes.area_size = IO_MAX_PAGE_SIZE;
recdes.data = PTR_ALIGN (recbuf, MAX_ALIGNMENT);
recdes.type = REC_HOME;
// ... condensed ...
if (heap_get_class_info (thread_p, &attr_info->class_oid, &oos_hfid, NULL, NULL) != NO_ERROR)
  return S_ERROR;                       /* <- branch: class lookup failed, nothing allocated */
if (!heap_oos_find_vfid (thread_p, &oos_hfid, &oos_vfid, true))   /* docreate=true */
  return S_ERROR;                       /* <- branch: file create/lookup failed */

docreate = true is load-bearing. heap_oos_find_vfid peeks the heap header’s oos_vfid; if it is VFID_ISNULL, the create branch wraps oos_create_file + TDE application + the RVHF_STATS undo/redo of the header in a top system operation (log_sysop_start / log_sysop_commit). So the very first externalized column on a heap pays a one-time file-creation cost that survives transaction abort independently of the row insert. With docreate == false (the read paths, Ch.7) a null oos_vfid returns with success == true and is not an error.

Invariant — a false return always means a real error. heap_oos_find_vfid sets VFID_SET_NULL (oos_vfid) up front and returns true even when no file exists (read path). The contract: false iff er_errid() is set. The insert path passes docreate = true, so a null VFID can never come back as success here — any failure to create forces S_ERROR. If a caller treated false as “no file” it would silently drop externalized data.

Step 2 — reset per-transaction tracking before the loop:

// heap_attrinfo_insert_to_oos -- src/storage/heap_file.c
tdes->oos_insert_lsa_queue.clear ();   /* <- replication LSA boundary queue */
thread_p->oos_oids.clear ();           /* <- per-thread head-OID list for the repl log */

The LOG_TDES is fetched via LOG_FIND_THREAD_TRAN_INDEX / LOG_FIND_TDES; a null tdes is a fatal ER_LOG_UNKNOWN_TRANINDEX returning S_ERROR. Clearing both establishes a clean slate for this row’s externalization: for the single-chunk case, each inserted column pushes exactly one OID and the WAL machinery pushes exactly one matching LSA (see 4.7, step 7). The replication applier later drains the two in lockstep.

Step 3 — the per-column loop externalizes only flagged columns:

// heap_attrinfo_insert_to_oos -- src/storage/heap_file.c
for (i = 0; i < attr_info->num_values; i++)
  {
    if ((*oos_columns)[i])           /* <- branch: skip non-OOS columns entirely */
      {
        assert (!attr_info->values[i].last_attrepr->is_fixed);   /* <- fixed cols never demote */
        if (heap_attrinfo_dbvalue_to_recdes (thread_p, &attr_info->values[i], attr_info->class_oid,
                                             lob_create_flag, &recdes) != S_SUCCESS)
          goto error_oos;            /* <- branch: serialization failed */
        if (oos_insert (thread_p, oos_vfid, oos_buffer (recdes.data, (size_t) recdes.length), oos_oid)
            != NO_ERROR)
          goto error_oos;            /* <- branch: storage insert failed */

        thread_p->oos_oids.push_back (oos_oid);          /* for replication log */
        (*oos_oids)[i] = oos_oid;                        /* <- head-OID for column i */
        (*oos_lengths)[i] = (DB_BIGINT) recdes.length;   /* <- full payload length */
        if (recdes.data != PTR_ALIGN (recbuf, MAX_ALIGNMENT))   /* <- malloc fallback was taken */
          {
            free_and_init (recdes.data);
            recdes.area_size = IO_MAX_PAGE_SIZE;
            recdes.data = PTR_ALIGN (recbuf, MAX_ALIGNMENT);    /* <- restore stack scratch */
          }
      }
  }
return S_SUCCESS;
error_oos:
  if (recdes.data != PTR_ALIGN (recbuf, MAX_ALIGNMENT))
    free_and_init (recdes.data);   /* <- malloc'd this iteration; free before bailing */
  return S_ERROR;

Branch accounting: a skipped column keeps its initialized oos_oids[i]/oos_lengths[i] and is written inline by the disk transform; serialization or insert failure jumps to error_oos. Prior columns inserted before a mid-loop failure are not unwound here — each oos_insert logged its own undo, so they are reclaimed when the transaction aborts (the source comment is explicit that both failure branches must reach the cleanup).

Invariant — the scratch buffer is restored to the stack pointer after every iteration. heap_attrinfo_dbvalue_to_recdes may malloc a larger area and overwrite recdes.data (4.3). After consuming the insert, the loop tests recdes.data != PTR_ALIGN (recbuf, ...); if a heap buffer was taken it frees it and re-points recdes.data at the stack scratch with area_size reset. Skipping the restore would leave the next column comparing a stale heap pointer against the stack base — it would never re-grow and would leak or write past a too-small area. The error_oos label repeats the free so a mid-iteration bail does not leak.

4.3 heap_attrinfo_dbvalue_to_recdes — serialize, with malloc fallback

This helper turns one DB_VALUE into raw on-disk bytes. The OOS-relevant mechanics are the sizing and the fallback allocation:

// heap_attrinfo_dbvalue_to_recdes -- src/storage/heap_file.c
length = pr_type->get_disk_size_of_value (dbvalue);
if (length > recdes->area_size)        /* <- value larger than the stack scratch */
  {
    recdes->area_size = length;
    recdes->data = (char *) malloc (length);   /* <- fallback heap buffer; freed by caller */
  }
buf.ptr = buf.buffer = recdes->data;
buf.endptr = recdes->data + length;
pr_type->data_writeval (&buf, dbvalue);
recdes->length = length;               /* <- serialized byte count */
return S_SUCCESS;

The length > recdes->area_size branch is exactly what the caller’s malloc-fallback cleanup in 4.2 pairs with. A LOB pre-pass (lob_create_flag == LOB_FLAG_INCLUDE_LOB with DB_TYPE_BLOB/DB_TYPE_CLOB) copies the ELO and flips value->state to HEAP_WRITTEN_LOB_ATTRVALUE to avoid double-copying across record-build retries; that is orthogonal to single-chunk sizing. On exit recdes.length is the authoritative payload size that becomes oos_lengths[i] and, via the header, the chain’s total_data_length.

4.4 oos_insert — entry guards and the single-vs-multi branch

oos_insert is the column-agnostic storage entry point: it sees only oos_buffer src (a cubbase::span<char>) and an out OID &.

// oos_insert -- src/storage/oos_file.cpp
assert (src.data () != nullptr);
assert (src.size () > 0);
/* Guards the narrowing cast below against wrap-around from a corrupt caller. */
if (src.data () == nullptr || src.size () == 0 || src.size () > (std::size_t) INT_MAX)
  {
    oos_error ("oos_insert rejected invalid src (data=%p, size=%zu)", src.data (), src.size ());
    er_set (ER_ERROR_SEVERITY, ARG_FILE_LINE, ER_GENERIC_ERROR, 0);
    return ER_GENERIC_ERROR;          /* <- branch: reject null / zero / oversized */
  }
const int src_len = static_cast<int> (src.size ());
if (src_len <= oos_get_max_chunk_size_within_page ())
  {
    const OOS_RECORD_HEADER header{src_len, 0, OID_INITIALIZER};  /* <- single chunk: idx 0, no next */
    err = oos_insert_within_page (thread_p, oos_vfid, src, header, oid);
  }
else
  {
    err = oos_insert_across_pages (thread_p, oos_vfid, src, oid);  /* <- Chapter 6 */
  }
return err;

Three guards fold into one if, each defending the static_cast<int> that follows: a null span; a zero-byte payload (the caller’s debug assert rules it out, but oos_insert is a public storage API and re-checks in release); and src.size () > INT_MAX, which would wrap the cast negative and defeat every downstream size comparison. All three return ER_GENERIC_ERROR via er_set so callers observe a real error.

The single-chunk branch is the entire scope of this chapter. When src_len <= oos_get_max_chunk_size_within_page (), the header is built inline as {src_len, 0, OID_INITIALIZER}: total length equals this chunk’s length, chunk index 0 (head), and next_chunk_oid null (no continuation). The else branch routes oversized values to Chapter 6.

OOS_RECORD_HEADER (declared in src/storage/oos_file.hpp as oos_record_header) is the on-page prefix every chunk carries. Its three fields, as used here:

Field	Role	Why it exists
total_data_length	Total user-data bytes across all chunks, excluding every chunk’s header. Single-chunk: equals this chunk’s src_len.	Lets the read path size the destination buffer and cross-validate the chain (oos_read_across_pages checks each chunk repeats the same value).
chunk_index	0-based position in the chain. Always 0 for a single-chunk value (the head).	Lets the applier detect a missing/reordered chunk; the head (index 0) is the OID stored in the heap stub.
next_chunk_oid	OID of the next chunk, or the null OID for the last/only chunk. Always OID_INITIALIZER here.	Forms the singly-linked chain the read path walks; null terminates the walk.

4.5 oos_get_max_chunk_size_within_page — the fit threshold

The single-vs-multi decision pivots on this one value:

// oos_get_max_chunk_size_within_page -- src/storage/oos_file.cpp
STATIC_INLINE __attribute__ ((ALWAYS_INLINE)) int
oos_get_max_chunk_size_within_page ()
{
  const int actual_upper_limit = DB_ALIGN_BELOW (spage_max_record_size (), OOS_ALIGNMENT);
  return actual_upper_limit - (int) sizeof (OOS_RECORD_HEADER);
}

It is the largest slot record a page can hold (spage_max_record_size ()), rounded down to OOS_ALIGNMENT (= MAX_ALIGNMENT), minus the header. Subtracting sizeof (OOS_RECORD_HEADER) makes the comparison in oos_insert an apples-to-apples payload comparison: a value of exactly this size, plus its header, exactly fills the aligned slot. A TODO notes the value is constant and could be memoized at boot; functionally it recomputes each call.

4.6 oos_prepend_header — assembling [header | payload]

oos_insert_within_page does not write the user’s span directly. It first builds a combined record whose first OOS_RECORD_HEADER_SIZE bytes are the chain header:

// oos_prepend_header -- src/storage/oos_file.cpp
const int src_len = static_cast<int> (src.size ());
err = recdes_allocate_data_area (&oos_recdes, src_len + OOS_RECORD_HEADER_SIZE);
if (err != NO_ERROR)
  {
    oos_error ("recdes_allocate_data_area failed in oos_prepend_header");
    assert_release_error (er_errid () != NO_ERROR);
    assert (false);
    return err;                      /* <- branch: allocation failed */
  }
oos_recdes.type = REC_HOME;
oos_recdes.length = src_len + OOS_RECORD_HEADER_SIZE;
std::memcpy (oos_recdes.data, &oos_header, OOS_RECORD_HEADER_SIZE);                /* <- header first */
std::memcpy (oos_recdes.data + OOS_RECORD_HEADER_SIZE, src.data (), src.size ());  /* <- then payload */
return NO_ERROR;

A fresh data area of src_len + OOS_RECORD_HEADER_SIZE is allocated into oos_recdes (an OOS_RECDES, alias for RECDES). The header lands at offset 0, the payload immediately after. Because the area is freshly allocated, the on-page record is a copy — it never aliases the caller’s recdes.data from 4.2, which is what lets heap_attrinfo_insert_to_oos recycle its scratch buffer across columns. The single failure branch propagates the allocation error after asserting one was set.

Invariant — the on-page record is exactly header_size + payload, header at offset 0. Every read path (oos_read_within_page, the vacuum and stats walkers) memcpys the first OOS_RECORD_HEADER_SIZE bytes back into an OOS_RECORD_HEADER and treats the remainder as payload. If the prepend ever wrote payload first or under-allocated, oos_read_within_page’s assert (oos_recdes.length >= OOS_RECORD_HEADER_SIZE) would fire and chain traversal would read a garbage next_chunk_oid.

4.7 oos_insert_within_page — the single-page workhorse

This is where the record reaches the disk page and the OID is minted.

flowchart TD
  A["oos_insert_within_page\nsrc, header, oid"] --> B["required_length =\nsrc_len + OOS_RECORD_HEADER_SIZE"]
  B --> C["oos_find_best_page\nrequired_length -> page_ptr, vpid"]
  C --> D["oos_prepend_header\nbuild header|payload into oos_recdes"]
  D -->|err| E["return err\nnothing allocated to free"]
  D -->|ok| F["scope_exit: free oos_recdes on any return"]
  F --> G["spage_insert\npage_ptr, oos_recdes -> slotid"]
  G -->|sp_status != SP_SUCCESS| H["er_set ER_GENERIC_ERROR\nreturn, scope_exit frees"]
  G -->|SP_SUCCESS| I["oid = vpid.volid, vpid.pageid, slotid"]
  I --> J["oos_log_insert_physical\nRVOOS_INSERT undoredo: undo data NULL, redo recdes"]
  J --> K["freespace = spage_max_space_for_new_record"]
  K --> L["oos_stats_add_bestspace\nvfid, vpid, freespace"]
  L --> M["return NO_ERROR\nscope_exit frees oos_recdes"]

Figure 4-2: oos_insert_within_page control flow. The scope_exit guard frees the prepend buffer on every exit edge after it is registered, including the spage_insert failure path.

// oos_insert_within_page -- src/storage/oos_file.cpp
const int src_len = static_cast<int> (src.size ());
assert (src_len <= oos_get_max_chunk_size_within_page ());
int required_length = src_len + OOS_RECORD_HEADER_SIZE;
assert (required_length <= DB_ALIGN_BELOW (spage_max_record_size (), OOS_ALIGNMENT));

auto auto_page_ptr = oos_find_best_page (thread_p, oos_vfid, required_length, vpid);  /* <- Chapter 5 */

OOS_RECDES oos_recdes{};
{
  err = oos_prepend_header (src, header, oos_recdes);
  if (err != NO_ERROR)
    {
      // ... condensed: oos_error + assert ...
      return err;                    /* <- branch: prepend failed; nothing inserted, nothing to free */
    }
  scope_exit defer_oos_recdes_free ([&]() { recdes_free_data_area (&oos_recdes); });  /* <- frees on every path */

  PGSLOTID slotid = NULL_SLOTID;
  PAGE_PTR page_ptr = auto_page_ptr.get();
  int sp_status = spage_insert (thread_p, page_ptr, &oos_recdes, &slotid);
  if (sp_status != SP_SUCCESS)
    {
      oos_error ("spage_insert failed with status %d", sp_status);
      er_set (ER_ERROR_SEVERITY, ARG_FILE_LINE, ER_GENERIC_ERROR, 0);
      return ER_GENERIC_ERROR;       /* <- branch: page insert rejected; scope_exit still frees */
    }
  assert (slotid != NULL_SLOTID);

  oid.pageid = vpid.pageid;          /* <- mint the OID from page VPID + slot */
  oid.slotid = slotid;
  oid.volid = vpid.volid;

  oos_log_insert_physical (thread_p, page_ptr, const_cast<VFID *> (&oos_vfid), &oid, &oos_recdes);

  int freespace_after = spage_max_space_for_new_record (thread_p, page_ptr);
  (void) oos_stats_add_bestspace (thread_p, &oos_vfid, &vpid, freespace_after);  /* <- refresh cache */
}
assert (oos_recdes.data == nullptr);  /* <- scope_exit must have nulled it */
return NO_ERROR;

Branch-by-branch:

1. Size assertions — debug guards only; oos_insert already routed oversized payloads away, so both hold by construction.
2. Page acquisition. oos_find_best_page returns a smart pointer holding a write-latched page plus its vpid, guaranteed to have room for required_length. Black box — Chapter 5.
3. Build the record. oos_prepend_header allocates oos_recdes. On failure it returns before scope_exit is registered, so there is nothing to free.
4. Register the free guard. Once the buffer exists, scope_exit defer_oos_recdes_free runs recdes_free_data_area on every subsequent return — the spage_insert failure path and the success path alike. The trailing assert (oos_recdes.data == nullptr) proves it fired.
5. spage_insert copies oos_recdes into a free slot and returns slotid. Any status other than SP_SUCCESS → er_set (ER_GENERIC_ERROR) and return; the page is unlatched by auto_page_ptr’s destructor and the buffer by scope_exit.
6. Mint the OID from the page’s vpid (volid, pageid) and the fresh slotid. This becomes oos_oid in 4.2, flows into oos_oids[i], and is written into the 16-byte inline stub by the disk transform.

Invariant — the OID returned for the stub is {vpid.volid, vpid.pageid, slotid} of the head chunk. For a single-chunk value the head is the whole record, so this OID alone locates the value. The stub stores this OID plus the recdes.length captured in 4.2; the SELECT path (Ch.7) sizes its read buffer from that length and starts its chain walk at this OID. Assembling the OID from anything but the page actually inserted into would leave the stub dangling.

7. WAL the insert. oos_log_insert_physical emits an RVOOS_INSERT record so the insert is redo-recoverable after a crash and reversible on abort:

// oos_log_insert_physical -- src/storage/oos_file.cpp
LOG_DATA_ADDR log_addr;
log_addr.vfid = vfid_p;
log_addr.offset = oid_p->slotid;       /* <- slot is the in-page address */
log_addr.pgptr = page_p;
log_append_undoredo_recdes (thread_p, RVOOS_INSERT, &log_addr, NULL, recdes_p);

The call is log_append_undoredo_recdes, but the undo data is NULL — only the redo half carries the [header | payload] bytes. The inverse of an insert is a delete, which needs only the slot, and log_addr.offset = oid_p->slotid already records it. The recovery table wires this up: {RVOOS_INSERT, "RVOOS_INSERT", oos_rv_redo_delete, oos_rv_redo_insert, NULL, NULL} (in src/transaction/recovery.c) registers oos_rv_redo_delete as the undo function (it spage_deletes the slot and evicts the stale bestspace entry) and oos_rv_redo_insert as the redo. So a single oos_insert whose transaction aborts is reversed slot-by-slot through this record — Chapter 8 covers both handlers and the symmetric RVOOS_DELETE.

This append also drives replication, and it does so in the single-chunk path too — not only in Chapter 6’s multi-chunk case. When log_manager.c appends an RVOOS_INSERT it auto-pushes tdes->tail_lsa onto tdes->oos_insert_lsa_queue unless tdes->oos_suppress_insert_lsa_queueing is set — and that flag is set only inside oos_insert_across_pages. So a single-chunk insert pushes exactly one LSA here, pairing with the single head OID pushed in 4.2 (thread_p->oos_oids). The replication applier later pops the two together. Chapter 6’s discipline is the multi-chunk extension of this same mechanism (it suppresses the per-chunk auto-push and enqueues one boundary LSA + the tail-chunk LSA by hand).

Invariant — for one single-chunk insert, the LSA queue and the OID list each gain exactly one entry, and they are paired. 4.2 clears both at row start; this step pushes one LSA, and 4.2’s loop pushes one OID. A mismatch (e.g. suppressing the LSA push but still pushing the OID) would desynchronize the applier’s pop loop in replication.c.

8. Refresh the bestspace cache. After the insert consumes space, spage_max_space_for_new_record reads the page’s remaining capacity and oos_stats_add_bestspace writes it back. The same spage_max_space_for_new_record is used here and in the lookup (oos_stats_find_page_in_bestspace, Ch.5), deliberately, so the cached free-space value uses identical accounting on the write-back and read-back sides — otherwise a page could be cached as having room it cannot grant. The (void) cast discards the return: a failed cache update only degrades future page-search quality, so it never aborts the insert.

4.8 What the caller does with the result

Back in heap_attrinfo_insert_to_oos, the minted oos_oid and recdes.length are committed to thread_p->oos_oids (replication), (*oos_oids)[i], and (*oos_lengths)[i]. These vectors flow to heap_attrinfo_transform_columns_to_disk, which writes the 16-byte stub (OID + 8-byte length) into the record’s variable-offset slot in place of the original value — see the companion’s INSERT section for the stub layout. From the storage layer’s view the job ends at step 8: the value is on a page, recoverable, addressable by OID, with the bestspace cache reflecting consumed space.

4.9 Chapter summary — key takeaways

1. Two layers, one seam. heap_attrinfo_insert_to_oos owns the column loop, the scratch buffer, and the oos_oids / oos_lengths / thread_p->oos_oids bookkeeping; oos_insert owns only span-in, OID-out. The seam lets the heap layer recycle one stack buffer across all columns, and the loop runs once before the record-build retry loop.
2. docreate = true lazily materializes the OOS file inside a top system operation on the first externalized column, and the heap_oos_find_vfid contract guarantees a false return is always a real error — never “no file.”
3. The scratch buffer is recycled with a malloc fallback. heap_attrinfo_dbvalue_to_recdes may swap recdes.data for a heap buffer when the value exceeds the stack scratch; the loop (and error_oos) free it and re-point at the stack base every iteration, or the next column corrupts.
4. oos_insert guards then branches on one threshold. Null / zero / > INT_MAX spans are rejected; src_len <= oos_get_max_chunk_size_within_page () selects the single-chunk path with header {src_len, 0, OID_INITIALIZER}, everything larger goes to Chapter 6.
5. oos_prepend_header builds an independent [header | payload] copy. The on-page record never aliases the caller’s buffer, and the header always sits at offset 0 — the invariant every read path relies on.
6. oos_insert_within_page is scope_exit-disciplined. The prepend buffer is freed on every return edge after registration, including the spage_insert failure; the head-chunk OID is minted as {vpid.volid, vpid.pageid, slotid} and is exactly what lands in the heap stub.
7. WAL, replication, and bestspace are the side effects of a successful insert. oos_log_insert_physical logs RVOOS_INSERT with the full redo recdes and null undo data — abort-reversal still works because the recovery table registers oos_rv_redo_delete as the undo (deletes the slot recorded in log_addr.offset). The same append auto-pushes one LSA onto oos_insert_lsa_queue even in the single-chunk path (suppression is multi-chunk only), pairing with the single OID; and oos_stats_add_bestspace refreshes the free-space cache using the same accounting as the lookup side.

Chapter 5: Bestspace Placement

Chapter 4 treated the page-selection step inside oos_insert_within_page as a black box: it called oos_find_best_page, got back a write-latched page with room for the chunk, and inserted. This chapter opens that box. The reader question is: how does the inserter find a page with room without scanning the whole file, and how does it avoid latch contention while doing so?

The high-level companion’s ### Bestspace section in cubrid-oos.md gives the design intent (a heap-style multi-entry estimate in the sticky header plus a 1000-entry global in-memory hash cache, queried by a 3-tier search). The whole mechanism is a deliberate clone of the heap manager’s bestspace machinery — the analogous code (heap_stats_find_page_in_bestspace, heap_stats_sync_bestspace, the HEAP_BESTSPACE cache) is dissected in cubrid-heap-manager-detail.md §§9.2/9.4/9.7, and the field-by-field parallel is worth reading alongside this chapter. This chapter does not repeat that rationale; it traces every branch of the OOS driver and its two workhorses, plus the mutation helpers that keep the estimate fresh.

5.1 The two-layer estimate

Two stores answer the question “which page has room”:

• Header best[] — oos_hdr->estimates.best[OOS_NUM_BEST_SPACESTATS] (OOS_NUM_BEST_SPACESTATS = 10) lives in the slot-0 record of the sticky first page. Durable across restarts, but its freespace values are hints that decay as pages fill.
• Global hash cache — oos_Bestspace, a process-wide OOS_STATS_BESTSPACE_CACHE keyed two ways (vfid_ht groups entries by file, vpid_ht keys by page) under one bestspace_mutex. Capacity OOS_BESTSPACE_CACHE_CAPACITY = 1000.

oos_bestspace is the placement atom — one (page, free-bytes) pair, fields vpid and freespace — and oos_hdr_stats.estimates is the durable bookkeeping block this chapter mutates. Note the two rings differ in element type: best[OOS_NUM_BEST_SPACESTATS] is an array of OOS_BESTSPACE (carries free bytes), while second_best[OOS_NUM_BEST_SPACESTATS] is a bare VPID[] (page id only — the freespace is re-measured if such a page is ever consulted). The fields actually read or written by the placement path:

Field	Role	Why it matters here
oos_bestspace.vpid	candidate page id	the page a chunk might land on; re-validated in Phase C
oos_bestspace.freespace	estimated free bytes on that page	ranked against needed_space; corrected in place when stale
estimates.best[10]	OOS_BESTSPACE[] — the hot ring of (vpid, freespace) candidates	Phase B scans it; idx_badspace reports its weakest slot
estimates.second_best[10]	VPID[] — feeder ring of evicted-but-roomy page ids (no freespace)	parks pages displaced from best[] (§5.6)
estimates.num_pages	pages counted by the last scan	denominator of the sync-decision ratio (§5.2)
estimates.num_recs	live OOS records counted	density input; refreshed by sync
estimates.recs_sumlen	running sum of body bytes (float)	average-size input; refreshed by sync
estimates.num_other_high_best	roomy pages known beyond best[]	numerator of the ratio; “is a rescan worth it?”
estimates.num_high_best	count of currently-good best[] slots	set to best_count after each sync; partial-scan delta base
estimates.num_substitutions	best[] eviction counter	gates the 1-in-1000 second_best[] sampling (§5.6)
estimates.num_second_best	valid entries in the second_best[] ring	bounds the ring; [0, 10] invariant (§5.6)
estimates.head_second_best	ring consumer cursor	where get_second_best pops
estimates.tail_second_best	ring producer cursor	where put_second_best writes
estimates.head	head index into best[]	reserved rotation cursor; not touched by any code in this chapter
estimates.full_search_vpid	page the last scan stopped on	intended resume point — currently a stubbed TODO (§5.4)

Figure 5-1 shows how the placement driver sits above both stores.

flowchart TD
  A["oos_insert_within_page (Ch4)"] --> B["oos_find_best_page<br/>3-tier driver"]
  B -->|fix sticky header| H["OOS_HDR_STATS.estimates<br/>(slot 0 of first page)"]
  B --> C["oos_stats_find_page_in_bestspace<br/>Phase A hash + Phase B best[]"]
  C -->|reads/prunes| G["oos_Bestspace<br/>global hash cache"]
  C -->|reads| H
  B -->|escalate| S["oos_stats_sync_bestspace<br/>scan pages, refill"]
  S -->|writes| G
  B -->|last resort| N["oos_file_alloc_new"]
  H -.mutated by.- U["oos_stats_update / _internal"]
  G -.mutated by.- M["oos_stats_add_bestspace<br/>del_by_vpid / del_by_vfid"]

Figure 5-1. The placement driver above the two-layer estimate.

Invariant — every hint write is non-logged. Both the header best[] and the global cache are reconstructable by rescanning the file (oos_stats_sync_bestspace). The code therefore never WAL-logs a hint change: header writes go through log_skip_logging + pgbuf_set_dirty(..., DONT_FREE), and the global cache is pure process memory. If a crash loses the latest hints, correctness is unaffected — only the first post-restart insert pays an extra scan. Logging a hint would bloat the redo stream for data the recovery manager can already rebuild.

5.2 oos_find_best_page — the 3-tier driver

The signature is oos_find_best_page (thread_p, oos_vfid, rec_length, vpid) returning an auto_unfix_page_ptr. Tiers: (1) consult the two estimates via oos_stats_find_page_in_bestspace; (2) on miss, escalate to oos_stats_sync_bestspace; (3) on continued miss, oos_file_alloc_new.

Header acquisition — three fall-through-to-alloc paths. Before any search the driver fixes the sticky first page. Three distinct failures each fall straight through to oos_file_alloc_new: no header VPID (file_get_sticky_first_page fails or returns NULL), the header pgbuf_fix returns NULL, or slot 0 is unreadable (oos_get_header_stats_ptr returns NULL, in which case the header is unfixed first).

The header is fixed with an unconditional WRITE latch — the driver will mutate best[] and must hold it across the search (except during sync, below). total_space = rec_length + sizeof (SPAGE_SLOT) budgets the slot-directory entry alongside the record body, matching what spage_max_space_for_new_record reports.

The while (try_find < 2) loop. Each pass calls oos_stats_find_page_in_bestspace (..., oos_hdr->estimates.best, &idx_badspace, total_space, &vpid, &found_page) and branches on the three-valued result: OOS_FINDSPACE_FOUND breaks with found_page set; OOS_FINDSPACE_ERROR unfixes the header and returns nullptr (an interrupt propagating to Chapter 4); OOS_FINDSPACE_NOTFOUND falls through to the sync decision.

The sync decision — the num_other_high_best / num_pages ratio. On NOTFOUND:

// oos_find_best_page -- src/storage/oos_file.cpp
float ratio = 0;
if (oos_hdr->estimates.num_pages > 0)
  ratio = (float) oos_hdr->estimates.num_other_high_best
          / (float) oos_hdr->estimates.num_pages;

if (ratio >= OOS_BESTSPACE_SYNC_THRESHOLD || try_find == 0)   /* 0.1f, or always pass 0 */
  { ...sync... }
else
  break;        /* not worth scanning; leave loop, will alloc new */

num_other_high_best counts pages known to have good space beyond the 10 in best[]. A high ratio means many roomy pages exist that we have not cached, so a scan will likely surface one. try_find == 0 forces a sync on the first miss regardless of ratio, guaranteeing at least one rescan attempt before giving up. If neither holds, the loop breaks and the driver allocates rather than scan a file that is genuinely near-full.

Releasing the header latch before sync. Holding the header WRITE latch across the scan (which fixes many data pages) would serialize all concurrent inserters. So the driver drops it first — log_skip_logging + pgbuf_set_dirty (..., DONT_FREE) flush its pending best[] mutations non-logged, then pgbuf_unfix_and_init and oos_hdr = NULL. It then memsets a stack OOS_HDR_STATS tmp_hdr to zero and calls oos_stats_sync_bestspace (..., &tmp_hdr, &hdr_vpid, false) — a throwaway header. Sync’s writes into that struct are discarded; it only needs to populate the global hash cache (which it does directly), so dropping the latch is safe — the scan never touches the real best[]. A modifier extending sync to mutate the header must not rely on tmp_hdr surviving.

Re-acquiring the header after sync. Two more fall-throughs appear on re-fix: if pgbuf_fix fails with ER_INTERRUPTED the driver returns nullptr; any other fix failure or an unreadable slot 0 falls through to oos_file_alloc_new. On success try_find++ and the loop runs once more — the global cache is now warmer, so Phase A is far more likely to hit.

The found_page re-fix race window. After the loop, if a candidate was found, the driver converts it from the conditional WRITE latch oos_stats_find_page_in_bestspace left held into an auto_unfix page. It deliberately unfixes then re-fixes unconditionally — and that gap is a race window another inserter can exploit to fill the page:

// oos_find_best_page -- src/storage/oos_file.cpp
pgbuf_unfix_and_init (thread_p, found_page);          /* drop conditional latch */
auto result_page = pgbuf_fix_auto_unfix (thread_p, &vpid, OLD_PAGE,
                   PGBUF_LATCH_WRITE, PGBUF_UNCONDITIONAL_LATCH);
if (result_page == nullptr)
  { if (er_errid () == ER_INTERRUPTED) return nullptr;  /* interrupt: bail */
    /* benign latch failure -> fall through to alloc */ }
else
  { int actual_free = spage_max_space_for_new_record (thread_p, result_page.get ());
    if (actual_free >= total_space)
      return result_page;                              /* still has room: done */
    result_page.reset ();   /* raced away; fall through to alloc */ }

The re-check after the unconditional re-fix is the guard against the race: if another thread filled the page in the gap, actual_free < total_space, the page is reset, and control falls to the closing branch — pgbuf_unfix_and_init_after_check (hdr_page) (a no-op if the header was already unfixed) then oos_file_alloc_new, whose fresh page is measured and seeded into the global cache via oos_stats_add_bestspace so the next inserter hits it in Phase A. Figure 5-2 traces the whole driver.

flowchart TD
  S["fix sticky header<br/>(WRITE, uncond)"] -->|no vpid / fix fail / slot0 bad| ALC["oos_file_alloc_new"]
  S -->|ok| L{"try_find < 2?"}
  L -->|find_page_in_bestspace| R{"result?"}
  R -->|FOUND| FP["found_page set,<br/>exit loop"]
  R -->|ERROR| ERR["unfix hdr,<br/>return nullptr"]
  R -->|NOTFOUND| RT{"ratio >= 0.1<br/>OR try_find==0?"}
  RT -->|no| FP
  RT -->|yes| SY["drop hdr latch (non-logged)<br/>sync_bestspace into tmp_hdr<br/>re-fix hdr"]
  SY -->|refix fail, interrupted| ERR
  SY -->|refix fail, other| ALC
  SY -->|ok, try_find++| L
  FP -->|found_page != NULL| RF["unfix cond latch<br/>re-fix uncond"]
  RF -->|interrupted| ERR
  RF -->|ok, actual_free >= need| DONE["return page"]
  RF -->|raced / latch fail| ALC
  FP -->|found_page == NULL| ALC
  ALC --> SEED["seed new page into cache,<br/>return"]

Figure 5-2. Every branch of the 3-tier driver.

5.3 oos_stats_find_page_in_bestspace — the searcher

This is the function the driver loops over. It returns an OOS_FINDSPACE and, on success, hands back a conditionally WRITE-latched page via *out_pgptr. The outer loop runs while (found == OOS_FINDSPACE_NOTFOUND).

Zero-wait setup. xlogtb_reset_wait_msecs (thread_p, LK_FORCE_ZERO_WAIT) forces zero lock-wait so a busy page never blocks the searcher; the prior value is saved in old_wait_msecs and restored at the single exit. The header VPID is fetched (file_get_sticky_first_page, NULL-set on failure) only so the search can skip it — the header holds the stats record, never chunk data.

Phase A — global hash scan. Under bestspace_mutex, walk the vfid_ht bucket for this file via mht_get2 (which iterates entries sharing the key), bounded by the notfound_cnt < BEST_PAGE_SEARCH_MAX_COUNT (= 100) budget. For each entry: if its best.freespace >= needed_space and it is not the header VPID, take it as candidate_vpid, set found_in_hash = true, and break; if it is the header VPID, drop it from both hash tables and continue; otherwise the entry is stale (too little space) and is destructively pruned from both tables (mht_rem2 on vfid_ht, mht_rem on vpid_ht, free, num_stats_entries--, notfound_cnt++). So the cache self-cleans on read.

Phase B — best[] scan. Only if Phase A produced no candidate and a bestspace array was passed. best_array_index is incremented first, then the loop scans forward (so successive driver passes resume past the slot they last tried), taking the first non-NULL slot whose freespace >= needed_space that is not the header VPID. If both phases leave candidate_vpid NULL the outer loop breaks with NOTFOUND.

Phase C — conditional fix + actual-free recheck. The candidate VPID is only a hint; the page must be fixed and re-measured:

// oos_stats_find_page_in_bestspace -- src/storage/oos_file.cpp
*out_pgptr = pgbuf_fix (thread_p, &candidate_vpid, OLD_PAGE,
                        PGBUF_LATCH_WRITE, PGBUF_CONDITIONAL_LATCH);
if (*out_pgptr == NULL)                                /* busy or error */
  {
    int err = er_errid ();
    if (err == ER_INTERRUPTED) { found = OOS_FINDSPACE_ERROR; break; }  /* propagate */
    if (err != NO_ERROR) { oos_trace (...); er_clear (); }              /* skip-on-busy */
    notfound_cnt++; continue;
  }
int actual_free = spage_max_space_for_new_record (thread_p, *out_pgptr);
if (actual_free >= needed_space)
  {
    *out_vpid = candidate_vpid; found = OOS_FINDSPACE_FOUND;
    if (found_in_hash)
      (void) oos_stats_add_bestspace (thread_p, vfid, &candidate_vpid, actual_free); /* refresh */
  }
else                                                    /* stale hint */
  {
    pgbuf_unfix_and_init (thread_p, *out_pgptr); *out_pgptr = NULL;
    if (found_in_hash) (void) oos_stats_del_bestspace_by_vpid (thread_p, &candidate_vpid);
    if (best_array_index >= 0 && best_array_index < OOS_NUM_BEST_SPACESTATS)
      bestspace[best_array_index].freespace = actual_free;  /* correct the hint in place */
    notfound_cnt++;
  }

This is the heart of skip-on-busy contention avoidance: a CONDITIONAL_LATCH that fails (page write-latched by another inserter) is not waited on — err is cleared, notfound_cnt++, and the loop tries the next candidate. Only ER_INTERRUPTED escalates to OOS_FINDSPACE_ERROR. A stale hint (page real but too full) drops the cache entry and corrects the best[] slot’s freespace in place so the next caller skips it.

Invariant — a FOUND page is returned still WRITE-latched. When the function returns OOS_FINDSPACE_FOUND, *out_pgptr holds a conditional WRITE latch the caller now owns. oos_find_best_page relies on this to immediately re-fix unconditionally (§5.2). If the searcher ever returned FOUND with *out_pgptr == NULL, the driver would dereference a freed/unlatched page. The code enforces it by setting found = FOUND only inside the actual_free >= needed_space branch where the page is still fixed.

idx_badspace computation. Before restoring wait mode the function reports which best[] slot is the worst (or first empty) so the caller’s eventual oos_stats_update knows where to insert. First NULL slot wins; otherwise the smallest-freespace slot.

5.4 oos_stats_sync_bestspace — the rescan

Called by the driver on escalation. It walks the file via the numerable iterator (file_numerable_find_nth) and refills the cache; scan_all selects partial vs full.

• Guard: file_get_num_user_pages; if it fails or total_pages <= 1 (only the header), return 0 immediately.
• Iteration cap: full scan walks total_pages; partial scan walks total_pages * 0.2, clamped to [10, oos_Find_best_page_limit] (oos_Find_best_page_limit = 100). This bounds the latch-and-measure cost.
• Resume point — a known TODO: the code intends to resume from estimates.full_search_vpid but currently just restarts at start_idx = 1 (page 0 is the header). The source comment is explicit: /* TODO: ideally find the index of full_search_vpid; for now start from 1 */. A modifier should not assume resume works yet.

Per page, under a conditional READ latch (skip-on-busy: er_clear () and continue), it measures spage_max_space_for_new_record and spage_collect_statistics, accumulates num_pages / num_recs / recs_sumlen, then:

// oos_stats_sync_bestspace -- src/storage/oos_file.cpp
if (free_space > OOS_DROP_FREE_SPACE)                  /* 0.3 * DB_PAGESIZE */
  {
    (void) oos_stats_add_bestspace (thread_p, vfid, &scan_vpid, free_space);
    if (best_count < OOS_NUM_BEST_SPACESTATS)          /* room in best[] */
      { ...fill first NULL slot; best_count++; num_high_best++... }
    else
      num_other_high_best++;                           /* roomy but best[] full */
  }
oos_hdr->estimates.full_search_vpid = scan_vpid;       /* resume hint (into tmp_hdr here) */

OOS_DROP_FREE_SPACE is the “worth tracking” floor. Full-scan cleanup (only when scan_all) nulls any best[] slot whose stale freespace is now <= OOS_DROP_FREE_SPACE. The estimate-update branch then splits on scan_all || estimates.num_pages <= num_pages — i.e. a full scan, or a partial scan that happened to observe at least as many pages as the stored estimate:

• That branch (reset): overwrite num_other_high_best, num_pages, num_recs, recs_sumlen outright with the freshly counted values.
• Otherwise (partial scan that saw fewer pages): preserve cumulative knowledge heap-style. Subtract the previously stored estimates.num_high_best from num_other_high_best, then floor it at the freshly counted num_other_high_best; and only adopt the new num_pages / num_recs / recs_sumlen when num_recs or recs_sumlen actually grew.

Finally estimates.num_high_best = best_count, and the function returns num_high_best + num_other_high_best (the locally counted values). In the driver’s escalation path these writes land in the throwaway tmp_hdr; only the global-cache add calls persist — see §5.2.

5.5 oos_stats_update / oos_stats_update_internal — post-op refresh

Called after a delete/insert frees or consumes space on a page (Chapters 4, 8, 9). oos_stats_update measures the page’s current free space via spage_get_free_space_without_saving, then:

1. If freespace > prev_freespace (space was freed), push it into the global cache via oos_stats_add_bestspace — a deleter advertising room.
2. If freespace > OOS_DROP_FREE_SPACE, try to update header best[] — fixing the header under a PGBUF_CONDITIONAL_LATCH WRITE latch with defer-on-busy: if pgbuf_fix returns NULL the function er_clear ()s and returns (the hint is optional, never worth waiting for). On success it calls oos_stats_update_internal, then flushes the change non-logged (log_skip_logging + pgbuf_set_dirty (..., DONT_FREE)) and unfixes.

oos_stats_update_internal decides placement within best[]:

// oos_stats_update_internal -- src/storage/oos_file.cpp
for (idx = 0; idx < OOS_NUM_BEST_SPACESTATS; idx++)
  {
    if (VPID_ISNULL (&best[idx].vpid)) { worst_idx = idx; break; }      /* empty slot */
    if (VPID_EQ (&best[idx].vpid, page_vpid))
      { best[idx].freespace = freespace; return; }                       /* already tracked */
    if (best[idx].freespace < worst_freespace) { worst_freespace = ...; worst_idx = idx; }
  }
if (worst_idx >= 0)
  {
    if (!VPID_ISNULL (&best[worst_idx].vpid)
        && best[worst_idx].freespace > OOS_DROP_FREE_SPACE)
      { oos_stats_put_second_best (oos_hdr, &best[worst_idx].vpid);     /* evict, don't lose */
        oos_hdr->estimates.num_other_high_best++; }
    best[worst_idx].vpid = *page_vpid; best[worst_idx].freespace = freespace;
  }

Three outcomes: already-present (update in place, return), an empty slot (claim it), or evict-the-worst. Note worst_freespace is seeded with the incoming freespace — so an entry only displaces an existing slot when the new page is roomier than the slot’s current occupant. An evicted-but-still-good page is not discarded; it is parked in the second_best[] ring and counted into num_other_high_best.

5.6 second_best[] ring, header reader, and cache mutators

oos_get_header_stats_ptr is the one-liner every header reader uses: peek slot 0 and cast the record data to OOS_HDR_STATS *. It assert(false)s and returns NULL if slot 0 is unreadable — the trigger for the “slot0 bad” fall-throughs above.

oos_stats_put_second_best — a sampling ring. It only records one in every 1000 evictions (++num_substitutions % 1000 == 0), writing at tail_second_best and advancing it with OOS_STATS_NEXT_BEST_INDEX ((i+1) % 10). While below capacity it bumps num_second_best; once full (== OOS_NUM_BEST_SPACESTATS) it advances head_second_best instead, overwriting the oldest. The hpp also defines a companion OOS_STATS_PREV_BEST_INDEX ((i==0) ? 9 : i-1) for symmetric backward stepping, but no function in this chapter calls it — it is a reserved helper, like estimates.head.

oos_stats_get_second_best pops from head_second_best, decrements num_second_best, and returns false (setting *vpid NULL) when empty.

Invariant — the ring’s num_second_best stays in [0, 10] and head/tail never collide ambiguously. put increments the count only while below capacity and otherwise advances head; get decrements only while non-empty. Both index moves go through OOS_STATS_NEXT_BEST_INDEX, so they wrap identically. If num_second_best were allowed to exceed OOS_NUM_BEST_SPACESTATS, get would hand back a slot the producer is about to overwrite — a torn VPID.

Global cache mutators. oos_stats_add_bestspace is the single insert/update path into the global cache. Under bestspace_mutex: if the VPID already exists in vpid_ht, just overwrite freespace; if at OOS_BESTSPACE_CACHE_CAPACITY, refuse (return NULL — caller treats as best-effort); otherwise take an entry from the free_list or malloc, populate it, mht_put into vpid_ht then mht_put_new into vfid_ht, rolling back the first if the second fails. oos_stats_del_bestspace_by_vpid removes one page from both tables; oos_stats_del_bestspace_by_vfid loops mht_get(vfid_ht) removing every entry for a whole file (used on file destroy / drop-table). All three are pure process memory — never logged, consistent with §5.1.

5.7 Chapter summary — key takeaways

1. oos_find_best_page is a 3-tier driver — query the two estimates (oos_stats_find_page_in_bestspace), escalate to a bounded rescan (oos_stats_sync_bestspace), and only then oos_file_alloc_new. Three header acquisition failures plus two post-sync re-fix failures all fall through to alloc; only ER_INTERRUPTED returns nullptr.
2. Latch discipline avoids contention at every step. The searcher runs under LK_FORCE_ZERO_WAIT and fixes candidates with PGBUF_CONDITIONAL_LATCH, skipping busy pages instead of waiting; oos_stats_update defers on a busy header; and the driver drops the header WRITE latch before the sync scan.
3. The num_other_high_best / num_pages ratio gates rescans. A scan happens only when the ratio meets OOS_BESTSPACE_SYNC_THRESHOLD (0.1) or it is the first miss (try_find == 0) — cheap inserts on near-full files skip straight to allocation.
4. Hints are always re-validated before use. A cached/best[] VPID is only a hint; Phase C re-fixes and re-measures with spage_max_space_for_new_record, and the driver re-checks again after the conditional-to-unconditional re-fix to close the race window where another thread fills the page.
5. The estimate self-cleans on read and on scan. Phase A destructively prunes stale/header/too-small cache entries; stale best[] slots get their freespace corrected in place or nulled on a full scan; evicted-but-roomy pages are sampled into the second_best[] ring.
6. Every hint write is non-logged (log_skip_logging + DONT_FREE for the header; pure memory for the global cache) because both stores are fully reconstructable by oos_stats_sync_bestspace. A crash costs at most one extra scan, never correctness.
7. Known rough edges for modifiers: oos_stats_sync_bestspace’s full_search_vpid resume point is a stubbed TODO (always restarts at index 1), oos_stats_put_second_best only samples one eviction in 1000, and estimates.head is a reserved best[] rotation cursor no code in this chapter touches — all intentional approximations/reservations, not bugs.

Chapter 6: Multi Chunk Chains and HA Replication

A value too large for one OOS page is split across a chain of chunks. This chapter answers the two questions a reader about to touch the multi-chunk path must answer: what oos_insert_across_pages writes to disk, in what order, and why — and how an HA slave rebuilds a byte-identical chain from the replication stream. The single-chunk primitive (oos_insert_within_page) and its bestspace placement were dissected in Chapter 4 and Chapter 5; here they are black boxes that “insert one chunk and return its OID”. The chain header oos_record_header, the inline heap stub, and the read-side walk (oos_read_across_pages) belong to Chapter 1 and Chapter 7 — cross-linked, not re-derived. For framing see the Multi-chunk chain and WAL and logging sections in cubrid-oos.md, and the master/slave mechanics in cubrid-ha-replication.md.

6.1 Where the across-pages path begins

oos_insert (the public entry, Chapter 4) picks the path by comparing the payload to the largest chunk that fits a page:

// oos_insert -- src/storage/oos_file.cpp
const int src_len = static_cast<int> (src.size ());
if (src_len <= oos_get_max_chunk_size_within_page ())
  {
    const OOS_RECORD_HEADER header{src_len, 0, OID_INITIALIZER};  /* single chunk: idx 0, next = NULL */
    err = oos_insert_within_page (thread_p, oos_vfid, src, header, oid);
  }
else
  {
    err = oos_insert_across_pages (thread_p, oos_vfid, src, oid);  /* this chapter */
  }

oos_get_max_chunk_size_within_page is the budget that defines a chunk. It takes the slotted-page maximum record size, aligns it down to OOS_ALIGNMENT (= MAX_ALIGNMENT), then subtracts the chain header so the returned value is the payload room in one chunk:

// oos_get_max_chunk_size_within_page -- src/storage/oos_file.cpp
const int actual_upper_limit = DB_ALIGN_BELOW (spage_max_record_size (), OOS_ALIGNMENT);
return actual_upper_limit - (int)sizeof (OOS_RECORD_HEADER);   /* header eats into every chunk */

So a chunk holds at most max_chunk_size payload bytes plus one OOS_RECORD_HEADER. The function carries two TODOs (a known spage_max_record_size rounding bug, declared out of scope; and a wish to precompute the value once in oos_boot()), but it is constant for a given page size.

6.2 Sizing the chain — required_page_nums

The first job inside oos_insert_across_pages is to compute the chunk count and assert the across-pages path was entered for a reason:

// oos_insert_across_pages -- src/storage/oos_file.cpp
const int max_chunk_size = oos_get_max_chunk_size_within_page ();
const int total_data_length = static_cast<int> (src.size ());
/* expressions below never add to total_data_length, which can sit near INT_MAX */
assert (total_data_length > max_chunk_size - OOS_RECORD_HEADER_SIZE);

int required_page_nums = total_data_length / max_chunk_size;
if (total_data_length % max_chunk_size != 0)
  {
    ++required_page_nums;          /* ceil division: a partial tail chunk still needs a page */
  }
assert (required_page_nums > 1);   /* across-pages means at least 2 chunks */

The arithmetic is deliberately phrased so it never adds to total_data_length. The in-source comment makes the rationale explicit: a caller may legitimately pass a value near INT_MAX (the oos_insert entry guard rejects only > INT_MAX), so both the entry assert and the loop’s i * max_chunk_size offset are written without ever incrementing total_data_length, which would signed-overflow.

INVARIANT — across-pages implies required_page_nums > 1. The ceiling division plus assert (required_page_nums > 1) guarantee the loop runs at least twice. The caller routes here only when src_len > max_chunk_size, and max_chunk_size already excludes the header, so the routing condition is strictly stronger than the in-function entry assert. If a single-chunk value ever reached this path, the chain would be built with a head whose next_chunk_oid is NULL but whose chunk_index bookkeeping still expected a tail — oos_read_across_pages (Chapter 7) would then trip on the index sequence. The single-chunk fast path exists precisely so this never happens.

6.3 The reverse-order fill loop

The defining design choice is that chunks are inserted tail first, head last (i = N-1 down to 0). The reason is mechanical: each chunk’s header carries next_chunk_oid, the OID of the next chunk in read order. Writing back-to-front means the next chunk’s OID is already in hand before the current chunk is written, so no chunk is ever patched after the fact.

The chain that lands on disk, in read order, is:

Figure 6-1. The on-disk chunk chain in read order. Only the head-chunk OID is stored in the heap; every chunk repeats total_data_length; the tail chunk’s next_chunk_oid is the NULL OID.

INVARIANT — one OID, one record: the heap stub stores the HEAD-chunk OID, never an interior or tail OID. oos_insert_across_pages returns next_chunk_oid after the final (i == 0) iteration — the head chunk’s OID — and that single OID is what heap_attrinfo_insert_to_oos writes into the inline heap stub (Chapter 1). The N physical chunks are reachable only by walking next_chunk_oid forward from that one head OID, so from the heap’s point of view a 1-chunk and an N-chunk value are indistinguishable: one OID names one logical record. The read side (oos_read_across_pages) and the slave applier both rely on this — each is handed exactly the head OID and rebuilds the rest by chain-walking.

The control flow of the whole function:

flowchart TD
  E2["size, ceil-div required_page_nums, assert > 1"] --> E3{"track_repl ?"}
  E3 -->|"yes"| E4["append LOG_DUMMY_OOS_RECORD<br/>dummy_lsa = tail_lsa<br/>suppress flag = true"]
  E3 -->|"no"| E5
  E4 --> E5["arm scope_exit, clears suppress flag on ANY exit"]
  E5 --> L1["loop i = N-1 .. 0: subspan chunk, tail auto-clamps<br/>header total_data_length, i, next_chunk_oid"]
  L1 --> L4["oos_insert_within_page to current_chunk_oid"]
  L4 -->|"error"| LE["return error_code, caller MUST abort"]
  L4 -->|"ok, i == N-1"| L6["tail_chunk_lsa = tail_lsa"]
  L6 --> L7["next_chunk_oid = current_chunk_oid; --i"]
  L4 -->|"ok, i > 0"| L7
  L7 --> L1
  L4 -->|"ok, i == 0"| D0["assert total_inserted_length == total_data_length"]
  D0 -->|"track_repl"| D2["push dummy_lsa, tail_chunk_lsa; oos_oids.push_back NULL"]
  D0 -->|"either"| D3["oid = next_chunk_oid, the HEAD OID; return NO_ERROR"]
  D2 --> D3

Figure 6-2. Branch-complete control flow of oos_insert_across_pages.

The loop body itself:

// oos_insert_across_pages -- src/storage/oos_file.cpp
int total_inserted_length = 0;
OID next_chunk_oid = OID_INITIALIZER;   // the last chunk has null OID as next_chunk_oid
// ... condensed: replication preamble (6.4) ...
for (int i = required_page_nums - 1; i >= 0; --i)   // tail (N-1) down to head (0)
  {
    // subspan clamps count to the remaining size, so the tail chunk is automatic.
    oos_buffer chunk = src.subspan (static_cast<std::size_t> (i * max_chunk_size),
                                    static_cast<std::size_t> (max_chunk_size));
    total_inserted_length += static_cast<int> (chunk.size ());

    // Keep total_data_length in each chunk so the applier can validate before reassembly.
    OOS_RECORD_HEADER header{total_data_length, i, next_chunk_oid};

    OID current_chunk_oid;
    error_code = oos_insert_within_page (thread_p, oos_vfid, chunk, header, current_chunk_oid);
    if (error_code != NO_ERROR)
      {
        oos_error ("could not insert chunk index=%d of length %zu.", i, chunk.size ());
        assert_release_error (er_errid () != NO_ERROR);
        return error_code;        // partial chunks reclaimed by transaction abort, not here
      }

    if (track_repl && i == required_page_nums - 1)   // the FIRST iteration is the tail
      {
        LSA_COPY (&tail_chunk_lsa, &tdes->tail_lsa);  // capture tail's RVOOS_INSERT LSA
      }

    next_chunk_oid = current_chunk_oid;   // becomes the NEXT chunk's next_chunk_oid
  }
assert (total_inserted_length == total_data_length);

Three subtleties a modifier must respect:

1. The tail chunk’s shorter length is automatic. subspan(offset, count) clamps count to the bytes remaining (max_len = _size - offset; see span<T>::subspan in span.hpp, whose default count is (size_type)-1). For the highest index i = N-1 the offset i * max_chunk_size leaves fewer than max_chunk_size bytes, so the returned chunk is exactly the remainder. No tail special-casing is written; the span does it.
2. next_chunk_oid is correct by construction. It starts as the NULL OID (the tail’s “next”) and at the end of each iteration becomes the OID just written. Because the loop walks down, the next iteration — a lower-index, earlier chunk — sees the OID of the chunk that follows it in read order. This is the entire reason for the reverse loop.
3. Every chunk stamps total_data_length, not its own length. The header’s first field is the whole value’s length, repeated in every chunk. The read side and the slave applier both use it to validate the assembled value before trusting the chain (6.6, Chapter 7). chunk_index (i) is the per-chunk identity.

INVARIANT — total_inserted_length == total_data_length on success. Enforced by the closing assert. Each iteration adds chunk.size(); because subspan clamps the tail and the loop covers exactly indices 0..N-1, the sum must equal src.size(). If a future change to the chunking math broke coverage, this assert fires before a truncated or overlapping chain reaches disk.

Error path. If any oos_insert_within_page fails the function returns the error immediately. Already-written chunks are not rolled back here — each oos_insert_within_page logged its own RVOOS_INSERT with full undo data (Chapter 4/8), so transaction abort replays those undos in reverse and the partial chain disappears. The in-source comment states the caller must not continue the transaction after this error. The scope_exit (6.4) still runs on this path, clearing the suppression flag.

6.4 The HA replication preamble and oos_needs_repl_tracking

A multi-chunk value is one logical column value but N physical RVOOS_INSERT log records. If the slave naively replayed each as an independent OOS insert it would allocate N unrelated OOS OIDs instead of one chain. The master prevents this by bracketing the run with a boundary marker and enqueueing only the head OID for replication.

The whole apparatus is gated by oos_needs_repl_tracking — master only:

// oos_needs_repl_tracking -- src/storage/oos_file.cpp
static bool
oos_needs_repl_tracking (THREAD_ENTRY *thread_p)
{
  return !LOG_CHECK_LOG_APPLIER (thread_p) && log_does_allow_replication () == true;
}

LOG_CHECK_LOG_APPLIER is true when this thread is the slave’s log applier; the slave must not emit its own OOS markers while replaying replicated data. log_does_allow_replication() is false on a standalone server. Only a replication-enabled master that is not itself an applier returns true. This is the same two-condition gate locator_sr.c uses before emitting any OOS repl log, kept as a helper because the OOS path also needs it inside oos_file.cpp.

When tracking is on, the preamble runs before the loop:

// oos_insert_across_pages -- src/storage/oos_file.cpp
track_repl = oos_needs_repl_tracking (thread_p);
if (track_repl)
  {
    const int tran_index = LOG_FIND_THREAD_TRAN_INDEX (thread_p);
    tdes = LOG_FIND_TDES (tran_index);
    if (tdes == NULL)
      {
        er_set (ER_FATAL_ERROR_SEVERITY, ARG_FILE_LINE, ER_LOG_UNKNOWN_TRANINDEX, 1, tran_index);
        return ER_LOG_UNKNOWN_TRANINDEX;     // no tdes: cannot track; bail before any chunk
      }
    log_append_empty_record (thread_p, LOG_DUMMY_OOS_RECORD, NULL);  // boundary marker, no payload
    LSA_COPY (&dummy_lsa, &tdes->tail_lsa);             // remember WHERE the marker landed
    tdes->oos_suppress_insert_lsa_queueing = true;      // intermediate chunks: do NOT auto-queue
  }

scope_exit clear_oos_repl_state ([&]()
{
  if (track_repl && tdes != NULL)
    {
      tdes->oos_suppress_insert_lsa_queueing = false;   // ALWAYS cleared, every exit path
    }
});

What each piece does:

• LOG_DUMMY_OOS_RECORD (= 53 in log_record.hpp) is an empty WAL record — log_append_empty_record writes a header with no data. Its sole purpose is to give the slave a marker that says “the next OOS insert is the head of a multi-chunk record, not an independent single-chunk insert.” dummy_lsa captures the marker’s LSA from tdes->tail_lsa immediately after appending.

• tdes->oos_suppress_insert_lsa_queueing = true turns off the automatic per-RVOOS_INSERT LSA enqueue. That auto-push lives in two structurally-identical sites — log_append_undoredo_crumbs and log_append_redo_crumbs in log_manager.c:

  // log_append_undoredo_crumbs (RVOOS_INSERT auto-queue site) -- src/transaction/log_manager.c
  else if (rcvindex == RVOOS_INSERT && !tdes->oos_suppress_insert_lsa_queueing)
    {
      tdes->oos_insert_lsa_queue.push (tdes->tail_lsa);   // skipped for chunks while flag is set
      assert (tdes->is_active_worker_transaction ());
    }

  With the flag set, none of the N chunk inserts pushes its LSA; the function enqueues exactly the two LSAs it wants (6.5) by hand.

• The scope_exit guarantees the suppression flag is cleared on every return path — the mid-loop error return, the normal return, or a stack unwind. This keeps the flag from leaking into the next log append.

INVARIANT — oos_suppress_insert_lsa_queueing is true for exactly the duration of one multi-chunk insert. Set just before the loop, cleared by scope_exit on any exit. If it leaked true, subsequent single-chunk OOS inserts in the same transaction would silently fail to enqueue their LSAs and the slave would miss those values. The scope_exit — not a plain assignment after the loop — is what makes this hold even on the error return in 6.3.

6.5 Capturing the tail LSA and the final hand-off

The applier needs the tail chunk’s LSA: the master logs tail-first, so the tail’s RVOOS_INSERT is the first chunk record after the marker, and it is the WAL anchor the slave rebuilds from. Inside the loop, the very first iteration is i == required_page_nums - 1 — the tail — and there tail_chunk_lsa is captured from tdes->tail_lsa right after the tail chunk’s RVOOS_INSERT is appended.

After the loop, when tracking is on, the function enqueues its two LSAs and pushes a NULL OID placeholder:

// oos_insert_across_pages -- src/storage/oos_file.cpp
if (track_repl)
  {
    tdes->oos_insert_lsa_queue.push (dummy_lsa);        // pairs with the dummy marker
    tdes->oos_insert_lsa_queue.push (tail_chunk_lsa);   // pairs with the head-OID insert
    thread_p->oos_oids.push_back (oid_Null_oid);        // placeholder; caller pushes real head OID
  }
oid = next_chunk_oid;   // after the loop next_chunk_oid holds the HEAD (idx 0) chunk OID
return NO_ERROR;

The out-parameter oid is next_chunk_oid, which after the final iteration (i == 0) holds the head chunk’s OID — exactly what goes into the heap stub (Chapter 1).

The asymmetry is deliberate and is the crux of the protocol: this function pushes oid_Null_oid into thread_p->oos_oids, not the real head OID. The immediate caller, heap_attrinfo_insert_to_oos (heap_file.c), pushes the real head OID right after oos_insert returns:

// heap_attrinfo_insert_to_oos -- src/storage/heap_file.c
if (oos_insert (thread_p, oos_vfid, oos_buffer (recdes.data, (size_t) recdes.length), oos_oid) != NO_ERROR)
  {
    goto error_oos;
  }
thread_p->oos_oids.push_back (oos_oid);   /* for replication log */
(*oos_oids)[i] = oos_oid;                  // also recorded in the per-column out-vector

oos_push_oos_oid is the standalone accessor for the same operation (thread_p->oos_oids.push_back (*oid)), used where a caller holds an OID outside this inline append. heap_attrinfo_insert_to_oos resets both tdes->oos_insert_lsa_queue and thread_p->oos_oids at the top of its per-row work, so the vectors describe exactly one row’s OOS inserts.

So for one multi-chunk value the per-transaction state ends up:

oos_oids             : [ ..., oid_Null_oid, real_head_oid ]
oos_insert_lsa_queue : [ ..., dummy_lsa,    tail_chunk_lsa ]

The two structures are positionally paired. A single-chunk value contributes one non-null OID and one LSA (from the auto-queue, since suppression was never set); a multi-chunk value contributes the (null, real) OID pair and the (dummy, tail) LSA pair. This pairing is documented verbatim in the header comment above oos_insert_across_pages.

6.6 From queued state to RVREPL items, and slave reassembly

At replication-log emission time (in locator_sr.c, after the primary-key heap insert), the master walks thread_p->oos_oids and chooses the item type per entry by testing the OID for NULL:

// oos repl emission loop (INSERT) -- src/transaction/locator_sr.c
for (int i = 0; i < (int) thread_p->oos_oids.size (); i++)
  {
    LOG_RCVINDEX oos_repl_rcvindex =
      OID_ISNULL (&thread_p->oos_oids[i]) ? RVREPL_DUMMY_OOS_RECORD : RVREPL_OOS_INSERT;   // null -> marker
    error_code = repl_log_insert (thread_p, class_oid, &thread_p->oos_oids[i],
                                  LOG_REPLICATION_DATA, oos_repl_rcvindex, key_dbvalue, REPL_INFO_TYPE_RBR_NORMAL);
    // ... condensed: error check ...
  }

repl_log_insert (via replication.c) attaches the LSA by popping the queue in lock-step for exactly these two item types:

// repl_log_insert oos lsa attach -- src/transaction/replication.c
case RVREPL_OOS_INSERT:
case RVREPL_DUMMY_OOS_RECORD:
  if (!tdes->oos_insert_lsa_queue.is_empty ())
    {
      LOG_LSA oos_lsa = tdes->oos_insert_lsa_queue.pop ();   // FIFO: dummy first, then tail
      LSA_COPY (&repl_rec->lsa, &oos_lsa);
    }
  else
    {
      assert (false);   // queue/vector desync = bug
    }
  break;

Because the queue was pushed [dummy_lsa, tail_chunk_lsa] and the OID vector holds [null, real] in the same order, the FIFO pop pairs dummy_lsa with the RVREPL_DUMMY_OOS_RECORD item and tail_chunk_lsa with the RVREPL_OOS_INSERT item. The net emission for one multi-chunk value:

flowchart LR
  M0["multi-chunk value<br/>on master"] --> M1["RVREPL_DUMMY_OOS_RECORD<br/>lsa = dummy_lsa<br/>oid = NULL"]
  M1 --> M2["RVREPL_OOS_INSERT<br/>lsa = tail_chunk_lsa<br/>oid = real head OID"]
  M2 --> M3["heap row repl item"]
  M3 -.->|"ship to slave"| S0
  S0["la_apply_dummy_oos_log<br/>need_oos_rebuild = true"] --> S1["la_apply_oos_insert_log<br/>rebuild_oos = true"]
  S1 --> S2["la_rebuild_oos_recdes from target_lsa<br/>reassemble N chunks<br/>validate total_data_length"]
  S2 --> S3["one OOS chain on slave<br/>identical to master"]

Figure 6-3. One multi-chunk value yields one DUMMY + one INSERT repl item; the slave rebuilds a single chain.

On the slave (log_applier.c), la_apply_dummy_oos_log sets apply->need_oos_rebuild = true and errors if it was already set — two markers without an intervening insert is corruption. The next RVREPL_OOS_INSERT item runs la_apply_oos_insert_log with rebuild_oos = apply->need_oos_rebuild, which calls la_rebuild_oos_recdes (&item->target_lsa, recdes, &oos_head_oid). That walk reassembles one OOS RECDES from the WAL chunk records anchored at target_lsa (the master’s tail-chunk LSA), enforcing three checks all keyed on the repeated total_data_length: every chunk’s value must agree (else mismatched multi-chunk OOS total size), the running body sum must not exceed it (body length exceeds total size), and the final sum must equal it (incomplete multi-chunk OOS record) before the merged RECDES is built. It also captures the master’s chunk_index == 0 head OID (oos_head_oid) — but only as the sql.log cache key, not for slot allocation; the slave assigns its own slotids afresh. apply->need_oos_rebuild is reset to false at the end: label of la_apply_oos_insert_log. A single-chunk value, by contrast, arrives as a lone RVREPL_OOS_INSERT with need_oos_rebuild false and is read directly from target_lsa with no rebuild. The slave therefore sees exactly one logical OOS-insert per value regardless of physical chunk count — the chapter’s headline result.

The same (null, real) / (dummy, tail) pairing is reused on the UPDATE replication path (the second thread_p->oos_oids walk in locator_sr.c, the line-for-line identical OID_ISNULL ternary), so an UPDATE that rewrites a large column replicates identically — the RVREPL_OOS_INSERT items precede the heap row item in both INSERT and UPDATE, as the applier comment in la_apply_oos_insert_log states explicitly.

6.7 The chunk-header-spec TODO

oos_insert carries a standing TODO directly above the within-page / across-pages branch:

// oos_insert -- src/storage/oos_file.cpp
// TODO: Once the OOS_RECORD_HEADER spec is finalized (first segment header and rest segment header),
// review whether it is possible to generate the segment headers inside the oos_insert_within_page() and
// oos_insert_across_pages() functions.

The intent is that the chain might eventually distinguish a first-segment header (head chunk, carrying chain-wide metadata) from a rest-segment header (interior/tail chunks). At this revision every chunk uses the same oos_record_header layout (total_data_length, chunk_index, next_chunk_oid) — see Chapter 1 — and the header is built by the caller (oos_insert for single-chunk, oos_insert_across_pages for the chain) and handed to oos_insert_within_page. A modifier should treat the header format as not yet frozen: if the spec splits into two header kinds, the construction sites in 6.1 and 6.3 and the validation in oos_read_across_pages (Chapter 7) and la_rebuild_oos_recdes all move together. (Design ref: Notion CBRD-26628.)

6.8 Chapter summary — key takeaways

1. The across-pages path is a black-box loop over the single-chunk primitive. oos_insert_across_pages computes required_page_nums from oos_get_max_chunk_size_within_page (page max, aligned down, minus the chain header), asserts > 1, and calls oos_insert_within_page once per chunk — it adds no new disk-layout logic of its own.
2. The reverse loop (i = N-1 .. 0) exists so next_chunk_oid is always known before a chunk is written. Walking tail-to-head lets each earlier chunk’s header embed the OID of the chunk that follows it in read order, with no post-write patching. subspan auto-clamps the tail chunk’s length.
3. One OID, one record: only the head-chunk OID reaches the heap. Every chunk repeats total_data_length; only chunk_index differs. The head OID names the whole chain, and both oos_read_across_pages (Chapter 7) and the slave applier chain-walk from it. The closing assert (total_inserted_length == total_data_length) is the master-side coverage check.
4. HA tracking is master-only and bracketed. oos_needs_repl_tracking gates everything on !LOG_CHECK_LOG_APPLIER && log_does_allow_replication(). A LOG_DUMMY_OOS_RECORD marker is appended first (its LSA saved as dummy_lsa), oos_suppress_insert_lsa_queueing turns off the per-chunk auto-queue in log_append_undoredo_crumbs / log_append_redo_crumbs, and a scope_exit guarantees the flag is cleared on every exit including the error return.
5. Only two LSAs and a NULL-OID placeholder are enqueued by hand. After the loop the function pushes dummy_lsa then tail_chunk_lsa onto oos_insert_lsa_queue and oid_Null_oid onto oos_oids; the caller (heap_attrinfo_insert_to_oos) then appends the real head OID, yielding the positionally paired vectors oos_oids=[…,null,real] / queue=[…,dummy,tail].
6. The master emits one RVREPL_DUMMY_OOS_RECORD + one RVREPL_OOS_INSERT per value. locator_sr.c picks the item type by OID_ISNULL, and replication.c pops the queue FIFO to attach dummy_lsa to the marker and tail_chunk_lsa to the real-OID insert. The INSERT and UPDATE emission loops are identical.
7. The slave reassembles one chain per value. la_apply_dummy_oos_log raises need_oos_rebuild; the following la_apply_oos_insert_log calls la_rebuild_oos_recdes, which validates the per-chunk total_data_length (agreement, no overflow, exact total) before building the merged RECDES — so a 1-chunk and an N-chunk value look identical (one logical insert) to the applier. The chunk-header format is still governed by an open spec TODO in oos_insert.

Chapter 7: SELECT and Re Inlining

The reader question: on a read, how is the externalized value fetched and spliced back so the record looks as if OOS were never used? This chapter traces the read-side expansion end to end — from the fetch wrapper that opts in, through the four-phase rebuild in heap_oos.cpp, down into the chain walk in oos_file.cpp — and closes with the two int-returning read helpers (oos_get_length and the attrinfo opt-out path) and how their failures propagate. For the inverse (INSERT) path see Chapter 4; for the inline-stub layout and the two flag bits (HAS_OOS, IS_OOS) see the high-level companion cubrid-oos.md, sections Two flags and The inline stub — this chapter assumes both.

The whole reconstruction obeys one design promise: the output record is byte-for-byte indistinguishable from one that never used OOS. It is built using only oos_read plus the on-record variable-offset table (VOT) — never the class representation — so it is schema-change safe and cheap.

7.1 Opting in: heap_get_visible_version_expand_oos and the two dispatch points

The scan machinery never expands OOS by default. A read that needs the real bytes must route through a fetch wrapper that sets one context flag:

// heap_get_visible_version_expand_oos -- src/storage/heap_file.c
heap_init_get_context (thread_p, &context, oid, class_oid, recdes, scan_cache, ispeeking, old_chn);
context.expand_oos = true;                         /* <- the only difference vs the plain wrapper */
scan = heap_get_visible_version_internal (thread_p, &context, false);

expand_oos is a bool on HEAP_GET_CONTEXT (heap_file.h), commented “if true, replace inline OOS OID slots with actual values”. The non-expanding sibling heap_get_visible_version leaves it at its heap_init_get_context default (false), so the inline stub survives the fetch untouched — that is what the scan/index paths and the eager-cleanup paths in Chapters 8–9 rely on.

A live TODO (CBRD-26847) sits above the wrapper: many callers were migrated mechanically from heap_get_visible_version and may not actually need expansion. Treat the call sites as not yet audited.

The flag is consumed inside heap_get_record_data_when_all_ready, which has two dispatch points that call heap_record_replace_oos_oids after the raw spage_get_record — one per data-bearing record type that holds inline columns:

// heap_get_record_data_when_all_ready -- src/storage/heap_file.c
switch (context->record_type)
  {
  case REC_RELOCATION:                              /* <- forward (REC_NEWHOME) data lives on fwd page */
    // ... condensed: allocate recdes area, spage_get_record from fwd_page_watcher (COPY) ...
    scan = spage_get_record (thread_p, context->fwd_page_watcher.pgptr, context->forward_oid.slotid,
                             context->recdes_p, COPY);
    if (scan != S_SUCCESS) { return scan; }
    return heap_record_replace_oos_oids (thread_p, context);   /* <- dispatch #1 */

  case REC_BIGONE:
    return heap_get_bigone_content (...);            /* <- no OOS expansion: overflow is a different path */
  case REC_HOME:
    // ... condensed: allocate recdes area when COPY ...
    scan = spage_get_record (thread_p, context->home_page_watcher.pgptr, context->oid_p->slotid,
                             context->recdes_p, context->ispeeking);
    if (scan != S_SUCCESS) { return scan; }
    return heap_record_replace_oos_oids (thread_p, context);   /* <- dispatch #2 */
  default:
    break;
  }
return S_ERROR;                                      /* <- only data-bearing types are expected to arrive */

REC_BIGONE (the classic OVF overflow record) is not expanded — OOS and OVF are disjoint mechanisms (companion Theoretical Background). The default fall-through returns S_ERROR.

flowchart TD
  W["heap_get_visible_version_expand_oos\nsets context.expand_oos = true"] --> GVI["heap_get_visible_version_internal"]
  GVI --> ALL["heap_get_record_data_when_all_ready"]
  ALL --> T{"record_type"}
  T -- REC_RELOCATION --> R1["spage_get_record from fwd page COPY"] --> RR["heap_record_replace_oos_oids"]
  T -- REC_HOME --> R2["spage_get_record from home page"] --> RR
  T -- REC_BIGONE --> BIG["heap_get_bigone_content\nno OOS expansion"]
  T -- default --> ERR["S_ERROR"]

Figure 7-1. From the fetch wrapper to the two OOS dispatch points.

7.2 heap_record_replace_oos_oids: the gate and the snapshot

This is the read-side entry. It has three early exits before any work, then the four-phase body wrapped in a try:

// heap_record_replace_oos_oids -- src/storage/heap_oos.cpp
RECDES *rec = context->recdes_p;
if (!context->expand_oos)
  { return S_SUCCESS; }                          /* <- opt-out: caller handles OOS itself */
assert (rec != NULL && rec->data != NULL && rec->length > 0);
if (rec == NULL || rec->data == NULL || rec->length <= 0)
  { er_set (...); return S_ERROR; }              /* <- release-build guard mirroring the assert */
if (!heap_recdes_contains_oos (rec))
  { return S_SUCCESS; }                          /* <- HAS_OOS not set: record already final */
try {
    std::vector<char> src_buf (rec->data, rec->data + rec->length);  /* <- snapshot BEFORE realloc */
    HEAP_OOS_EXPAND_STATE state = { };
    state.src             = src_buf.data ();
    state.src_length      = (int) src_buf.size ();
    state.src_offset_size = OR_GET_OFFSET_SIZE (state.src);   /* <- honor the record's own offset width */
    state.src_header_size = OR_HEADER_SIZE ((char *) state.src);
    if (heap_oos_parse_vot (&state)       != NO_ERROR) { return S_ERROR; }   /* phase 1 */
    state.oos_blobs.resize (state.n_var);
    if (heap_oos_read_blobs (thread_p, &state) != NO_ERROR) { return S_ERROR; }  /* phase 2 */
    if (heap_oos_compute_layout (&state)  != NO_ERROR) { return S_ERROR; }   /* phase 3 */
    return heap_oos_build_record (thread_p, context, &state);                /* phase 4 */
} catch (std::bad_alloc &) {
    er_set (..., ER_OUT_OF_VIRTUAL_MEMORY, 1, (size_t) rec->length);
    return S_ERROR;
}

The two cheap exits — !expand_oos and !heap_recdes_contains_oos — keep the common case (no OOS, or caller opts out) to a single flag test. heap_recdes_contains_oos is the short-circuit:

// heap_recdes_contains_oos -- src/storage/heap_file.c
int flag = (INT32) OR_GET_MVCC_FLAG (record->data);
return flag & OR_MVCC_FLAG_HAS_OOS;              /* <- the HAS_OOS bit of the MVCC header flags */

INVARIANT — the source bytes are snapshotted before any reallocation. src_buf is a full copy of rec->data taken before phase 4 may realloc or reassign rec->data via the scan cache. Every phase reads only from state.src (= src_buf.data()), and phase 4 writes only to the (possibly new) rec->data. If the snapshot were skipped, assign_recdes_to_area could free the buffer the rebuild is still reading, producing a use-after-free whose corruption is data-dependent and nearly impossible to reproduce. The std::vector also gives RAII cleanup on the bad_alloc path. The mirror invariant is enforced on the destination in phase 4: the dst offset size is always forced to 4 bytes regardless of src_offset_size.

The HEAP_OOS_EXPAND_STATE struct threads all cross-phase data. Its read-side roles, condensed:

Field	Role on the read path
src / src_length	Pointer into the snapshot and its byte length; the bounds-check ceiling for every VOT/slot read
src_offset_size	Width (1/2/4 B) of each source VOT entry from OR_GET_OFFSET_SIZE; the parser must match it
src_header_size	OR_HEADER_SIZE of the source; base for all VOT and value offsets
n_var	Index of the LAST_ELEMENT sentinel = count of variable attributes; loop bound for every per-variable phase
new_length	Total length of the rebuilt record; drives the realloc decision and final rec->length
src_vot_bytes / dst_vot_bytes	Source VOT size at src_offset_size; destination VOT size at 4-byte width
fixed_bitmap_bytes	Fixed attrs + bound-bit bitmap between VOT and first variable value; copied verbatim
vot_raw	Every raw VOT entry incl. flag bits, sentinel last; one parse reused by all phases
oos_blobs	Per-index expanded bytes, non-empty only at OOS indices; holds the fetched value between phase 2 and phase 4

7.3 Phase 1 — heap_oos_parse_vot: walk the VOT, find the sentinel

The parser reads the source VOT one entry at a time, honoring src_offset_size, collecting each raw entry (flag bits intact), and stopping at the OR_IS_LAST_ELEMENT sentinel:

// heap_oos_parse_vot -- src/storage/heap_oos.cpp
const char *src_vot = (const char *) OR_GET_OBJECT_VAR_TABLE (state->src);
const int capacity = (state->src_length - state->src_header_size) / state->src_offset_size;
state->vot_raw.reserve (capacity + 1);
state->n_var = -1;
for (int i = 0; i <= capacity; ++i) {
    if (i == capacity) {                          /* <- ran off the end without a sentinel */
        assert_release (false && "VOT sentinel (LAST_ELEMENT) not found within record bounds");
        er_set (...); return ER_FAILED;
    }
    int raw;
    const char *ep = src_vot + i * state->src_offset_size;
    switch (state->src_offset_size) {             /* <- decode at the record's own width */
      case OR_BYTE_SIZE:  raw = OR_GET_BYTE (ep);  break;
      case OR_SHORT_SIZE: raw = OR_GET_SHORT (ep); break;
      case OR_INT_SIZE:   raw = OR_GET_INT (ep);   break;
      default: assert_release (false); er_set (...); return ER_FAILED;   /* <- impossible width */
    }
    state->vot_raw.push_back (raw);               /* <- keep flag bits, do NOT mask them off */
    if (OR_IS_LAST_ELEMENT (raw)) { state->n_var = i; break; }
}
if (state->n_var <= 0) {                           /* <- HAS_OOS set but zero variable attrs */
    assert_release (false && "OOS flag set without variable attributes");
    er_set (...); return ER_FAILED;
}
return NO_ERROR;

Branch inventory: the i == capacity arm (VOT has no sentinel inside its byte bounds), the default switch arm (offset width not 1/2/4), and n_var <= 0 (HAS_OOS set but no variable region) are all assert_release + ER_GENERIC_ERROR — they cannot happen on a well-formed record, so they are treated as corruption, not ordinary errors. vot_raw ends up holding n_var + 1 entries: indices 0 .. n_var-1 are real attributes, index n_var is the sentinel (used in phases 3/4 as the “next offset” upper bound).

7.4 Phase 2 — heap_oos_read_blobs: decode each stub and fetch

For every variable index, the loop skips non-OOS entries and, for OOS ones, bounds-checks the 16-byte inline slot, decodes OID + length, validates the length, sizes the destination, and calls oos_read:

// heap_oos_read_blobs -- src/storage/heap_oos.cpp
for (int i = 0; i < state->n_var; ++i) {
    if (!OR_IS_OOS (state->vot_raw[i])) { continue; }            /* <- inline column, untouched */
    const int value_offset = state->src_header_size + OR_GET_VAR_OFFSET (state->vot_raw[i]);
    if (value_offset + OR_OOS_INLINE_SIZE > state->src_length) { /* <- 16B slot past record end */
        assert_release (false && "OOS inline slot extends past record bounds");
        er_set (...); return ER_FAILED;
    }
    OID oos_oid = OID_INITIALIZER; DB_BIGINT oos_len = 0; int rc = NO_ERROR;
    OR_BUF buf;
    or_init (&buf, (char *) state->src + value_offset, OR_OOS_INLINE_SIZE);
    if (or_get_oid (&buf, &oos_oid) != NO_ERROR || OID_ISNULL (&oos_oid)) {   /* <- bad/null OID */
        assert_release (false && "failed to read OOS OID from inline slot");
        er_set (...); return ER_FAILED;
    }
    oos_len = or_get_bigint (&buf, &rc);
    if (rc != NO_ERROR || oos_len <= 0 || oos_len > (DB_BIGINT) DB_MAX_STRING_LENGTH) {  /* <- length sanity */
        assert_release (false && "invalid OOS inline length");
        er_set (...); return ER_FAILED;
    }
    state->oos_blobs[i].resize ((std::size_t) oos_len);          /* <- size the dest from the stub */
    if (oos_read (thread_p, oos_oid, oos_buffer (state->oos_blobs[i].data (), (std::size_t) oos_len)) != NO_ERROR) {
        oos_error ("oos_read failed for OID %d|%d|%d", OID_AS_ARGS (&oos_oid));
        er_set (...); return ER_FAILED;
    }
}
return NO_ERROR;

The inline slot is read with an OR_BUF over exactly OR_OOS_INLINE_SIZE (OR_OID_SIZE + OR_BIGINT_SIZE = 16) bytes: or_get_oid consumes the 8-byte OID, or_get_bigint the 8-byte length — the exact layout the INSERT side wrote (companion The inline stub). The length validation (0, DB_MAX_STRING_LENGTH] rejects zero, negative, and absurd values before resize, so a corrupt stub cannot make the engine attempt a multi-gigabyte allocation. The destination is sized to oos_len exactly — this is the read-side half of the buffer-sizing contract that oos_read then cross-checks against the chain header (§7.7). Branches: four corruption asserts (slot OOB, bad OID, bad length, oos_read failure) plus the continue for inline columns; the oos_read failure is the one that fires in normal-but-broken operation (e.g. a missing OOS page) and propagates a real error.

7.5 Phase 3 — heap_oos_compute_layout: 4-byte offsets and the INT_MAX guard

This phase computes the destination geometry without writing anything. The destination VOT is always 4-byte (BIG_VAR_OFFSET_SIZE):

// heap_oos_compute_layout -- src/storage/heap_oos.cpp
const int dst_offset_size = BIG_VAR_OFFSET_SIZE;
state->src_vot_bytes = OR_VAR_TABLE_SIZE_INTERNAL (state->n_var, state->src_offset_size);
state->dst_vot_bytes = OR_VAR_TABLE_SIZE_INTERNAL (state->n_var, dst_offset_size);
state->fixed_bitmap_bytes = OR_GET_VAR_OFFSET (state->vot_raw[0]) - state->src_vot_bytes;
if (state->fixed_bitmap_bytes < 0) { assert_release (false); er_set (...); return ER_FAILED; }
int64_t new_values_bytes = 0;
for (int i = 0; i < state->n_var; ++i) {
    const int this_off = OR_GET_VAR_OFFSET (state->vot_raw[i]);
    const int next_off = OR_GET_VAR_OFFSET (state->vot_raw[i + 1]);     /* <- sentinel supplies last next_off */
    const int src_val_len = next_off - this_off;
    if (src_val_len < 0 || state->src_header_size + next_off > state->src_length) {  /* <- offsets disordered/past */
        assert_release (false && "VOT offsets out of order or past record");
        er_set (...); return ER_FAILED;
    }
    if (OR_IS_OOS (state->vot_raw[i])) {
        assert_release (src_val_len == OR_OOS_INLINE_SIZE);            /* <- OOS slot must be exactly 16B */
        new_values_bytes += (int64_t) state->oos_blobs[i].size ();    /* <- expanded length replaces 16B */
    } else {
        new_values_bytes += src_val_len;                              /* <- inline column carried over */
    }
}
const int64_t new_length_64 =
        (int64_t) state->src_header_size + state->dst_vot_bytes + state->fixed_bitmap_bytes + new_values_bytes;
if (new_length_64 > (int64_t) INT_MAX) {                              /* <- RECDES.length is a 4-byte int */
    assert_release (false && "OOS-expanded record size exceeds INT_MAX");
    er_set (...); return ER_FAILED;
}
state->new_length = (int) new_length_64;

fixed_bitmap_bytes is derived as the gap between the source VOT end and the first variable value’s offset — i.e. the fixed attributes plus the bound-bit bitmap, which expansion never touches. The running total is accumulated in int64_t precisely so the INT_MAX guard is meaningful: RECDES.length is a signed 4-byte int, capping one expanded record at ~2 GB (companion Cross-check Notes). Forcing 4-byte destination offsets means the new VOT can address arbitrarily large expansions regardless of how compact the source was. Branches: fixed_bitmap_bytes < 0 (VOT overlaps fixed region — corruption), per-entry order/bounds check, the OOS-slot-size assert, and the INT_MAX overflow guard.

7.6 Phase 4 — heap_oos_build_record: assemble, clear HAS_OOS, force 4-byte offsets

The final phase materializes the record. First the realloc decision:

// heap_oos_build_record -- src/storage/heap_oos.cpp
const bool need_realloc = (context->ispeeking == PEEK) || (rec->area_size < state->new_length);
if (need_realloc) {
    if (context->scan_cache == NULL) {            /* <- no buffer to grow into */
        rec->length = - (state->new_length);      /* <- negative length: caller must re-call with space */
        return S_DOESNT_FIT;
    }
    context->scan_cache->assign_recdes_to_area (*rec, (size_t) state->new_length);  /* <- may free old rec->data */
    if (rec->data == NULL || rec->area_size < state->new_length) {
        er_set (..., ER_OUT_OF_VIRTUAL_MEMORY, 1, (size_t) state->new_length);
        return S_ERROR;
    }
    context->ispeeking = COPY;                    /* <- expanded record is a copy, never a peek */
}
char *dst = rec->data;

Three branch outcomes: (a) the area already fits and ispeeking != PEEK — no realloc; (b) realloc needed but no scan cache — emit S_DOESNT_FIT with a negative rec->length so the caller retries with a sized buffer; (c) realloc via assign_recdes_to_area, which is exactly why §7.2 snapshotted the source first — assign_recdes_to_area may free the original rec->data. A PEEK is forced to COPY because an expanded record cannot alias a page.

Then the header copy and the flag rewrite:

// heap_oos_build_record -- src/storage/heap_oos.cpp
std::memcpy (dst, state->src, state->src_header_size);            /* <- header verbatim first */
unsigned int repid_bits = (unsigned int) OR_GET_INT (dst + OR_REP_OFFSET);
repid_bits &= ~ ((unsigned int) OR_MVCC_FLAG_HAS_OOS << OR_MVCC_FLAG_SHIFT_BITS);  /* <- clear HAS_OOS */
repid_bits &= ~ (unsigned int) OR_OFFSET_SIZE_FLAG;              /* <- clear old offset-size bits */
repid_bits |= OR_OFFSET_SIZE_4BYTE;                              /* <- force 4-byte offsets */
OR_PUT_INT (dst + OR_REP_OFFSET, (int) repid_bits);

INVARIANT — the rebuilt record carries no OOS trace. HAS_OOS is cleared and the offset-size bits are rewritten to 4-byte. The output is now a normal record: a second heap_recdes_contains_oos would return false, so a re-expansion is a no-op, and any downstream code that reads OR_GET_OFFSET_SIZE sees a consistent 4-byte VOT matching what phase 3 computed. If HAS_OOS were left set, the record could be re-expanded (the 16-byte slots are gone — a guaranteed corruption read); if the offset-size bits disagreed with the VOT width, every offset read downstream would be mis-scaled.

The VOT is rewritten cumulatively, then trailing alignment padding zeroed, then fixed+bitmap copied unchanged, then each value inlined in place:

// heap_oos_build_record -- src/storage/heap_oos.cpp
const int dst_first_value_rel = state->dst_vot_bytes + state->fixed_bitmap_bytes;
char *dst_vot = dst + state->src_header_size;
int cumulative = 0;
for (int i = 0; i < state->n_var; ++i) {
    OR_PUT_INT (dst_vot + i * dst_offset_size, dst_first_value_rel + cumulative);
    const int val_len = (OR_IS_OOS (state->vot_raw[i])
                         ? (int) state->oos_blobs[i].size ()                       /* <- expanded length */
                         : (OR_GET_VAR_OFFSET (state->vot_raw[i + 1]) - OR_GET_VAR_OFFSET (state->vot_raw[i])));
    cumulative += val_len;
}
OR_PUT_INT (dst_vot + state->n_var * dst_offset_size,
            OR_SET_VAR_LAST_ELEMENT (dst_first_value_rel + cumulative));          /* <- sentinel, no OOS bit */
// ... condensed: memset any padding past the (n_var+1) entries ...
if (state->fixed_bitmap_bytes > 0)
    std::memcpy (dst + state->src_header_size + state->dst_vot_bytes,
                 state->src + state->src_header_size + state->src_vot_bytes, state->fixed_bitmap_bytes);
int dst_pos = state->src_header_size + state->dst_vot_bytes + state->fixed_bitmap_bytes;
for (int i = 0; i < state->n_var; ++i) {
    if (OR_IS_OOS (state->vot_raw[i])) {
        std::memcpy (dst + dst_pos, state->oos_blobs[i].data (), state->oos_blobs[i].size ());  /* <- inline blob */
        dst_pos += (int) state->oos_blobs[i].size ();
    } else {
        const int src_off = state->src_header_size + OR_GET_VAR_OFFSET (state->vot_raw[i]);
        const int len = OR_GET_VAR_OFFSET (state->vot_raw[i + 1]) - OR_GET_VAR_OFFSET (state->vot_raw[i]);
        std::memcpy (dst + dst_pos, state->src + src_off, len);                   /* <- carry inline column */
        dst_pos += len;
    }
}
assert (dst_pos == state->new_length);            /* <- write cursor must land exactly on new_length */
rec->length = state->new_length;
return S_SUCCESS;

The new VOT offsets are cumulative: each entry points at where its value begins, and the rewritten sentinel (OR_SET_VAR_LAST_ELEMENT) carries the LAST bit but never the OOS bit, so the rebuilt record’s VOT terminates cleanly and flags no column as externalized. The OOS rewrite is in place positionally — the blob lands exactly where its 16-byte stub used to be in attribute order — so attribute identity is preserved. The closing assert (dst_pos == state->new_length) ties phases 3 and 4 together: if the layout math and the copy loop ever disagreed, this fires.

flowchart TD
  A["phase 4 entry"] --> B{"need_realloc?\nPEEK or area too small"}
  B -- no --> H["dst = rec->data"]
  B -- yes --> C{"scan_cache == NULL?"}
  C -- yes --> D["rec->length = -new_length\nreturn S_DOESNT_FIT"]
  C -- no --> E["assign_recdes_to_area"]
  E --> F{"alloc ok?"}
  F -- no --> G["S_ERROR ER_OUT_OF_VIRTUAL_MEMORY"]
  F -- yes --> H
  H --> I["copy header, clear HAS_OOS, force 4B offsets"]
  I --> J["rewrite VOT cumulatively, sentinel"]
  J --> K["copy fixed+bitmap unchanged"]
  K --> L["inline each blob / carry inline cols"]
  L --> M["assert dst_pos == new_length\nrec->length = new_length\nS_SUCCESS"]

Figure 7-2. heap_oos_build_record branch map.

7.7 oos_read: the chain walk that phase 2 calls

Each oos_read in phase 2 reassembles one column’s value from its chunk chain. The public entry validates the head, cross-checks the total length, then walks any tail:

// oos_read -- src/storage/oos_file.cpp
assert (dest.data () != nullptr && dest.size () > 0);
const int expected_length = static_cast<int> (dest.size ());        /* <- the stub's full_length */
cubbase::byte_span_writer writer (dest);
OOS_RECORD_HEADER first_header;
int err = oos_read_within_page (thread_p, oid, writer, first_header);
if (err != NO_ERROR) { return err; }
assert (first_header.chunk_index == 0);
if (first_header.chunk_index != 0) {                                /* <- stub OID points mid-chain = corruption */
    oos_error (...); er_set (...); return ER_GENERIC_ERROR;
}
if (first_header.total_data_length != expected_length) {            /* <- stub length vs chain header */
    oos_error (...); er_set (...); return ER_GENERIC_ERROR;
}
if (!OID_ISNULL (&first_header.next_chunk_oid)) {                   /* <- multi-chunk: follow the tail */
    err = oos_read_across_pages (thread_p, first_header.next_chunk_oid, expected_length, writer);
    if (err != NO_ERROR) { return err; }
}
if (!writer.full ()) {                                              /* <- wrote fewer bytes than dest.size() */
    oos_error (...); er_set (...); return ER_GENERIC_ERROR;
}
return NO_ERROR;

expected_length is dest.size(), i.e. the oos_len phase 2 read from the stub. The three correctness checks are: (1) chunk_index == 0 — the caller’s OID must be the chain head, not an interior chunk; (2) total_data_length == expected_length — the chain’s self-declared total must match the inline stub’s length, catching a stub/chain skew; (3) writer.full() — after the whole walk, exactly dest.size() bytes must have been written, no more, no fewer. The single-chunk case skips oos_read_across_pages entirely (the next_chunk_oid is NULL).

INVARIANT — dest.size() is the single source of truth for read length. Phase 2 sized dest from the inline stub’s 8-byte length; oos_read refuses to accept any chain whose header total_data_length differs from it and refuses to return success unless the writer is exactly full(). The stub length, the chain header, and the bytes actually appended are thus triangulated — no path can hand back a value of the wrong size.

oos_read_within_page fetches one chunk’s payload through the bounded byte_span_writer:

// oos_read_within_page -- src/storage/oos_file.cpp
PAGE_PTR page_ptr = pgbuf_fix (thread_p, &vpid, OLD_PAGE, PGBUF_LATCH_READ, PGBUF_UNCONDITIONAL_LATCH);
if (page_ptr == nullptr) { oos_error (...); assert_release_error (er_errid () != NO_ERROR); return er_errid (); }
// ... condensed: scope_exit unfixes the page ...
OOS_RECDES oos_recdes;
SCAN_CODE code = spage_get_record (thread_p, page_ptr, slotid, &oos_recdes, PEEK);
if (code != S_SUCCESS) {                                            /* <- slot gone; force an errid */
    if (er_errid () == NO_ERROR) { er_set (...); }
    return er_errid ();
}
assert (oos_recdes.length >= OOS_RECORD_HEADER_SIZE);
if (oos_recdes.length < OOS_RECORD_HEADER_SIZE) { oos_error (...); er_set (...); return ER_GENERIC_ERROR; }  /* <- too short */
std::memcpy (&header_out, oos_recdes.data, OOS_RECORD_HEADER_SIZE); /* <- hand the chain header back */
const int payload_len = oos_recdes.length - OOS_RECORD_HEADER_SIZE;
if (!writer.append (oos_recdes.data + OOS_RECORD_HEADER_SIZE, (std::size_t) payload_len)) {  /* <- overflow guard */
    oos_error (...); er_set (...); return ER_GENERIC_ERROR;
}
return NO_ERROR;

The page is read-latched unconditionally (a reader can wait), unfixed by a scope_exit. The slot must be at least OOS_RECORD_HEADER_SIZE; the header is copied out so the caller can chase next_chunk_oid. writer.append bounds-checks every chunk against the caller-owned destination span and returns false on overrun without advancing — so a chunk claiming more bytes than dest can hold is rejected, not overflowed. Note that the spage_get_record failure path normalizes a S_…-but-no-errid case into a real error so the caller never sees NO_ERROR after a failed fetch.

oos_read_across_pages walks from chunk index 1, asserting the chain stays consistent and rejecting an empty chunk:

// oos_read_across_pages -- src/storage/oos_file.cpp
int idx = 1; OID current = next_oid;
while (!OID_ISNULL (&current)) {
    OOS_RECORD_HEADER header;
    const std::size_t before = writer.written ();
    int err = oos_read_within_page (thread_p, current, writer, header);
    if (err != NO_ERROR) { return err; }
    assert (idx == header.chunk_index);
    assert (header.total_data_length == total_data_length);
    if (idx != header.chunk_index || header.total_data_length != total_data_length) {   /* <- chain skew */
        oos_error (...); er_set (...); return ER_GENERIC_ERROR;
    }
    if (writer.written () == before) {                              /* <- 0-byte chunk: cyclic next would loop */
        oos_error (...); er_set (...); return ER_GENERIC_ERROR;
    }
    current = header.next_chunk_oid; idx++;
}
return NO_ERROR;

Two invariants are enforced per chunk: chunk_index must equal the monotonically increasing idx (so chunks arrive in order), and every chunk’s total_data_length must equal the head’s (so all chunks agree on the whole value’s size). The writer.written() == before check rejects a 0-byte chunk — important because a chunk that contributes nothing combined with a cyclic next_chunk_oid would loop forever; rejecting empty chunks bounds the walk. The loop terminates on a NULL next_chunk_oid, after which oos_read’s writer.full() check confirms the assembled length.

flowchart TD
  R["oos_read(oid, dest)"] --> RW["oos_read_within_page (head)"]
  RW --> H0{"chunk_index == 0?"}
  H0 -- no --> E1["ER_GENERIC_ERROR"]
  H0 -- yes --> TL{"total_data_length == dest.size()?"}
  TL -- no --> E2["ER_GENERIC_ERROR"]
  TL -- yes --> NX{"next_chunk_oid NULL?"}
  NX -- yes --> FULL{"writer.full()?"}
  NX -- no --> AP["oos_read_across_pages from idx=1"]
  AP --> CK{"per chunk: idx match,\ntotal match, non-empty?"}
  CK -- no --> E3["ER_GENERIC_ERROR"]
  CK -- yes, more chunks --> AP
  CK -- yes, NULL next --> FULL
  FULL -- no --> E4["ER_GENERIC_ERROR"]
  FULL -- yes --> OK["NO_ERROR, dest filled"]

Figure 7-3. oos_read head validation plus chain walk.

7.8 oos_get_length and the int-return failure-propagation contract

oos_read returns int error codes, not SCAN_CODE; phase 2 maps any non-NO_ERROR to its own ER_FAILED and the dispatch chain turns that into S_ERROR. Three sibling read helpers also return plain int — oos_get_length, heap_attrvalue_read_oos_inline, and heap_midxkey_get_oos_extra_size — and each surfaces failure differently; the last is the area touched by CBRD-26769.

oos_get_length — the length probe (no assembly). Where phase 2 reads the length from the inline stub, callers that hold only an OOS head OID (notably test code — see the oos_file.hpp note “or oos_get_length in tests”) can fetch just the chain total without walking the whole value:

// oos_get_length -- src/storage/oos_file.cpp
PAGE_PTR page_ptr = pgbuf_fix (thread_p, &vpid, OLD_PAGE, PGBUF_LATCH_READ, PGBUF_UNCONDITIONAL_LATCH);
if (page_ptr == nullptr) {
    oos_error (...); assert_release_error (er_errid () != NO_ERROR); assert (false);
    return -1;                                                      /* <- sentinel, NOT an error code */
}
// ... scope_exit unfixes the page ...
SCAN_CODE code = spage_get_record (thread_p, page_ptr, slotid, &oos_recdes, PEEK);
if (code != S_SUCCESS) { oos_error (...); er_set (...); return -1; }
assert (oos_recdes.length >= OOS_RECORD_HEADER_SIZE);
OOS_RECORD_HEADER header;
std::memcpy (&header, oos_recdes.data, sizeof (OOS_RECORD_HEADER));
return header.total_data_length;                                    /* <- only the head chunk is read */

Only the head chunk is fixed and read; total_data_length is returned directly. The expansion path does not call this — it already has the length inline — but it is the canonical “how long is this value?” probe and it reads exactly the same OOS_RECORD_HEADER.total_data_length that §7.7 cross-checks against dest.size().

INVARIANT — oos_get_length reports failure as -1, not as an error code. Unlike oos_read, whose int is NO_ERROR/error-code, this function returns the length on success and the sentinel -1 on a fixed/fetch failure (with er_set already done). A caller must test < 0 and must never feed the result into a buffer size without that check — a -1 cast to a size would be catastrophic. This is exactly the failure-propagation discipline CBRD-26769 hardened on the heap side.

The attrinfo opt-out reader. The !expand_oos early exit in §7.2 means the other reader, heap_attrvalue_read_oos_inline (heap_file.c), owns OOS for callers like heap_attrinfo_read_dbvalues. That helper also returns int, but unlike the midxkey probe below it does not use a sentinel — every corrupt-inline condition is raised as ER_HEAP_OOS_BAD_INLINE_HEADER and returned directly:

// heap_attrvalue_read_oos_inline -- src/storage/heap_file.c
buf.ptr = raw->data;  buf.endptr = recdes->data + recdes->length;
/* Cases 1-3: header too short / bad-or-null OID / length outside (0, INT_MAX] -- all the same error. */
if (buf.endptr - buf.ptr < OR_OID_SIZE + OR_BIGINT_SIZE)
  { er_set (..., ER_HEAP_OOS_BAD_INLINE_HEADER, 3, ...); raw->data = NULL; *oos_owned_buffer = false; return ER_HEAP_OOS_BAD_INLINE_HEADER; }
or_get_oid (&buf, &oos_oid);  oos_len = or_get_bigint (&buf, &rc);
if (rc != NO_ERROR || OID_ISNULL (&oos_oid))               { /* er_set */ return ER_HEAP_OOS_BAD_INLINE_HEADER; }
if (oos_len <= 0 || oos_len > (DB_BIGINT) INT_MAX)         { /* er_set */ return ER_HEAP_OOS_BAD_INLINE_HEADER; }
// ... condensed: point raw->data at oos_scratch if it fits, else recdes_allocate_data_area (Case 4) ...
/* Case 5: oos_read failed -- it already er_set; free the heap buffer and propagate. */
error = oos_read (thread_p, oos_oid, oos_buffer (raw->data, (std::size_t) oos_len));
if (error != NO_ERROR) { ASSERT_ERROR (); if (raw->data != oos_scratch) recdes_free_data_area (raw); raw->data = NULL; return error; }

Five distinct failure points: the three header checks (Cases 1–3) all raise ER_HEAP_OOS_BAD_INLINE_HEADER; Case 4 is the allocation failure; Case 5 forwards oos_read’s own code. A fast path points raw->data at the caller-supplied oos_scratch when the value fits — no per-row heap alloc — and *oos_owned_buffer tells the caller whether it must recdes_free_data_area.

CBRD-26769 — int-return propagation on the midxkey-sizing path. A separate helper, heap_midxkey_get_oos_extra_size (heap_file.c), pre-computes how much larger a midxkey grows once an OOS attribute is expanded. It returns a plain int size, so CBRD-26769 had to make its corrupt-inline handling a 0 sentinel rather than an error code:

// heap_midxkey_get_oos_extra_size (CBRD-26769) -- src/storage/heap_file.c
/* CBRD-26769: validate the inline header exactly as heap_attrvalue_read_oos_inline does.
 * A corrupt header must never be cast into a midxkey size: a negative/huge (int) length
 * would mis-size midxkey.buf and let the legitimate columns overrun it before the read path
 * raises ER_HEAP_OOS_BAD_INLINE_HEADER.  Return 0 so the buffer is sized from recdes->length
 * alone; the corruption is then surfaced when the value is actually read. */
if (buf.endptr - buf.ptr < OR_OID_SIZE + OR_BIGINT_SIZE) { return 0; }
or_get_oid (&buf, &oos_oid);
DB_BIGINT length = or_get_bigint (&buf, &rc);
if (rc != NO_ERROR || OID_ISNULL (&oos_oid) || length <= 0 || length > (DB_BIGINT) INT_MAX) { return 0; }
return (int) length;

The lesson the expansion path inherits: an inline length is never trusted blindly. On the expand_oos path the distrust is the (0, DB_MAX_STRING_LENGTH] check plus oos_read’s header cross-check (§7.4, §7.7); on the attrinfo path it is the explicit ER_HEAP_OOS_BAD_INLINE_HEADER cases above; on the midxkey-sizing path it is the return 0-then-size-from-recdes->length pattern that defers the corruption to the actual read. Three different return disciplines, one rule — a corrupt inline header becomes a bounded, propagated failure, never a wild allocation.

A fourth read-side helper, oos_chunk_exists, exists in oos_file.cpp but is not on the SELECT path: it is a read-only “is this chunk still here?” probe for the idempotent vacuum/delete callers (Chapters 8–9), distinguishing a legitimately deallocated chunk (NO_ERROR, *out_exists = false) from a real I/O failure (propagated). It is named here only so the read-side inventory of oos_file.cpp is complete.

7.9 Chapter summary — key takeaways

1. Expansion is opt-in. Only heap_get_visible_version_expand_oos sets context.expand_oos; the scan and eager-cleanup paths leave the inline stub intact. The flag is consumed at the REC_HOME and REC_RELOCATION dispatch points in heap_get_record_data_when_all_ready; REC_BIGONE (OVF) is never OOS-expanded.
2. Two cheap gates, then one snapshot. heap_record_replace_oos_oids exits on !expand_oos and on !heap_recdes_contains_oos before any work, then copies the source bytes into src_buf before phase 4 may realloc — guaranteeing the rebuild never reads freed memory.
3. Four phases, all from the VOT. parse_vot (find the sentinel, keep flag bits) → read_blobs (decode each 16-byte stub, validate length in (0, DB_MAX_STRING_LENGTH], oos_read into a sized buffer) → compute_layout (4-byte dst offsets, int64_t accumulation, INT_MAX guard) → build_record (assemble). No class representation is consulted, so the path is schema-change safe.
4. The output erases every OOS trace. build_record copies the header verbatim, clears OR_MVCC_FLAG_HAS_OOS, forces OR_OFFSET_SIZE_4BYTE, and rewrites the VOT cumulatively with a clean sentinel — a re-expansion would be a no-op and downstream offset reads stay consistent.
5. oos_read triangulates the length. It enforces chunk_index == 0, total_data_length == dest.size(), and a final writer.full(); the chain walk asserts monotonic chunk_index, matching total_data_length, and rejects 0-byte chunks to bound a cyclic chain. dest.size() (the inline stub length) is the one source of truth for the read size.
6. byte_span_writer is the overflow firewall. Every chunk’s payload is appended through a bounded cursor that refuses to overrun the caller-owned destination span, so a corrupt chunk length cannot smash the expansion buffer.
7. The int-return helpers propagate, never paper over. oos_get_length returns -1 (sentinel, after er_set) on a fetch failure and must be < 0-checked before use; heap_attrvalue_read_oos_inline (the attrinfo opt-out reader) raises ER_HEAP_OOS_BAD_INLINE_HEADER on any corrupt header; the CBRD-26769 heap_midxkey_get_oos_extra_size sizing helper returns 0 on a corrupt header so the buffer is sized from recdes->length and the corruption surfaces only when the value is later read. Three return disciplines, one rule — an inline length is never trusted blindly.
8. Most asserts are corruption tripwires. Across all functions the assert_release checks (missing sentinel, OOB slot, bad OID/length, disordered offsets, wrong chunk index) cannot fire on a well-formed record; they convert silent corruption into a loud, debuggable failure.

Chapter 8: The Physical Delete Primitive and Recovery

This chapter answers one question: what is the single low-level operation that removes an OOS record, and why is it crash- and abort-safe without a system operation (sysop)? Both reclamation drivers covered later — eager non-MVCC cleanup (Chapter 9) and vacuum reclamation (Chapter 10) — call this primitive; we dissect it here, before the policy that decides when to invoke it.

For the OOS chain model (head OID, next_chunk_oid linkage, single- vs multi-chunk records) and the WAL/MVCC theory, see cubrid-oos.md. The general WAL undo/redo replay machinery (RV_fun[] dispatch, undo vs redo passes) is in cubrid-recovery-manager-detail.md. This chapter re-derives neither; it traces the OOS delete code branch-by-branch and the OOS-specific recovery replay paths.

8.1 The call funnel: oos_delete to oos_delete_chain

oos_delete is the public entry point — a thin shim that logs the argument OID for tracing and delegates to the internal workhorse oos_delete_chain. No validation, no sysop open, no lock acquisition.

// oos_delete -- src/storage/oos_file.cpp
int
oos_delete (THREAD_ENTRY *thread_p, const VFID &oos_vfid, const OID &oid)
{
  oos_debug ("arguments: oid={vol=%d,page=%d,slot=%d}", OID_AS_ARGS (&oid));

  return oos_delete_chain (thread_p, oos_vfid, oid);   /* <- all real work is in the chain walker */
}

The separation is deliberate: oos_delete is the stable name callers bind to (the only delete symbol exported in oos_file.hpp), while oos_delete_chain is static and free to change. Both reclamation drivers reach chunk removal through this single funnel, so the crash/abort-safety argument is established once, here.

flowchart TD
  C9["Ch.9 eager cleanup"] --> D["oos_delete()"]
  C10["Ch.10 vacuum"] --> D
  D --> CH["oos_delete_chain()<br/>walk next_chunk_oid"]
  CH --> LOG["oos_log_delete_physical()<br/>RVOOS_DELETE undo only"]
  CH --> SP["spage_delete()"]
  CH --> ST["oos_stats_update()<br/>free-space cache update"]

Figure 8-1. Both reclamation drivers funnel through oos_delete into the chain walker.

8.2 oos_delete_chain: the branch-complete chain walker

oos_delete_chain follows next_chunk_oid from the head OID, deleting each chunk slot in turn. The whole body is one while (!OID_ISNULL (&current_oid)) loop; a single-chunk record is simply a chain of length one whose head carries next_chunk_oid == NULL_OID.

Per iteration the function does a fixed sequence of steps, with an early-return error path after each fallible one.

// oos_delete_chain -- src/storage/oos_file.cpp
OID current_oid = oid;
while (!OID_ISNULL (&current_oid))
  {
    VPID vpid = {current_oid.pageid, current_oid.volid};
    PAGE_PTR page_ptr = pgbuf_fix (thread_p, &vpid, OLD_PAGE,
                                   PGBUF_LATCH_WRITE, PGBUF_UNCONDITIONAL_LATCH);   /* (1) write latch; branch A */
    if (page_ptr == nullptr) { ASSERT_ERROR_AND_SET (error); /* oos_error */ return error; }
    scope_exit page_unfixer ([&] () { pgbuf_unfix_and_init_after_check (thread_p, page_ptr); });  /* RAII unfix */

    OOS_RECDES oos_recdes = RECDES_INITIALIZER;
    SCAN_CODE code = spage_get_record (thread_p, page_ptr, current_oid.slotid, &oos_recdes, PEEK); /* (2) PEEK */
    if (code != S_SUCCESS) { ASSERT_ERROR_AND_SET (error); /* oos_error */ return error; }          /* branch B */
    if (oos_recdes.length < (int) sizeof (OOS_RECORD_HEADER))                      /* (3) corruption guard, branch C */
      {
        assert_release (false);
        er_set (ER_ERROR_SEVERITY, ARG_FILE_LINE, ER_GENERIC_ERROR, 0);
        return ER_GENERIC_ERROR;
      }

    OOS_RECORD_HEADER header;
    std::memcpy (&header, oos_recdes.data, sizeof (OOS_RECORD_HEADER));
    OID next_chunk_oid = header.next_chunk_oid;                                    /* (4) read NEXT before delete */
    oos_log_delete_physical (thread_p, page_ptr, const_cast<VFID *> (&oos_vfid),
                             current_oid.slotid, &oos_recdes);                     /* (5) WAL undo append */
    PGSLOTID deleted_slotid = spage_delete (thread_p, page_ptr, current_oid.slotid);
    if (deleted_slotid == NULL_SLOTID) { ASSERT_ERROR_AND_SET (error); return error; }  /* branch D */
    pgbuf_set_dirty (thread_p, page_ptr, DONT_FREE);                              /* (6) mark dirty, keep fixed */
    oos_stats_update (thread_p, page_ptr, &oos_vfid, 0);                          /* (7) free-space cache update */
    current_oid = next_chunk_oid;                                                 /* advance; NULL ends loop */
  }
return NO_ERROR;

The ordering of step (4) before step (5)/(6) is the single most important detail in the function:

Invariant — read-next-before-delete. next_chunk_oid is copied out of the chunk header before spage_delete reclaims the slot. The code enforces this by doing the std::memcpy into the stack header and saving next_chunk_oid on the line above the oos_log_delete_physical/spage_delete pair. If violated (e.g. reading header.next_chunk_oid after the delete), the walker would dereference a freed slot’s payload, lose the tail of the chain, and leak every downstream chunk — the chunks would become permanently unreachable orphans because nothing else records the linkage.

The four fallible steps each early-return: branch A pgbuf_fix returns nullptr; branch B spage_get_record peek is not S_SUCCESS; branch C the length guard (8.3); branch D spage_delete returns NULL_SLOTID. A/B/D all ASSERT_ERROR_AND_SET then return; C calls assert_release and er_set then returns ER_GENERIC_ERROR. Steps (4) header-copy, (5) WAL append, (6) dirty, and (7) stats update always run. Two design choices deserve a callout:

• Unconditional write latch. PGBUF_LATCH_WRITE + PGBUF_UNCONDITIONAL_LATCH: the walker mutates the slot, needs an exclusive latch, and is willing to block — no back-off to a conditional latch. The caller is presumed to hold a row-level lock that serializes deleters of this chain (see 8.9).
• PEEK not COPY. oos_recdes.data points directly into the page buffer and is handed straight to oos_log_delete_physical as the undo image. The peek must not outlive the latch: scope_exit page_unfixer guarantees unfix on every exit path, and the WAL append (step 5) consumes the peek before spage_delete frees the slot.

Invariant — log-before-delete (WAL). oos_log_delete_physical (step 5) runs before spage_delete on every iteration. The undo record carrying the full pre-image reaches the log buffer before the slot is reclaimed in the page. If violated (delete first, log after), a crash between the two steps would lose the slot with no undo image to restore it.

stateDiagram-v2
  [*] --> CheckOID
  CheckOID --> Done: current_oid NULL
  CheckOID --> Fix: not NULL
  Fix --> Err: nullptr, branch A
  Fix --> Peek: latched
  Peek --> Err: not S_SUCCESS, branch B
  Peek --> Guard: got record
  Guard --> Err: short record, branch C
  Guard --> Del: length ok, save next then WAL
  Del --> Err: NULL_SLOTID, branch D
  Del --> CheckOID: slot removed, advance to next
  Err --> [*]: caller MUST abort txn
  Done --> [*]: NO_ERROR

Figure 8-2. State machine of one oos_delete_chain iteration with all four error branches (A-D).

8.3 The corruption guard and why it is assert_release

Branch C is not a normal runtime condition — a well-formed OOS chunk always begins with a full OOS_RECORD_HEADER (the oos_record_header struct: total_data_length, chunk_index, next_chunk_oid). The guard oos_recdes.length < sizeof(OOS_RECORD_HEADER) catches a structurally impossible record. It fires assert_release (false) — which traps in debug builds and logs (without aborting) in release — then sets ER_GENERIC_ERROR at ER_ERROR_SEVERITY and returns rather than reading past the buffer in the following std::memcpy. It is OOS’s own structural sanity check (not a storage-layer return code), placed between the peek and the header copy so the memcpy of sizeof(OOS_RECORD_HEADER) bytes is always in-bounds. Note the severity: the forward guard degrades at ER_ERROR_SEVERITY (a recoverable transaction error that the caller aborts on), whereas the recovery-time equivalents in 8.6 escalate to ER_FATAL_ERROR_SEVERITY — there is no graceful degradation once recovery is replaying.

8.4 oos_log_delete_physical: the undo-carrying WAL record

This helper is what makes the no-sysop design work. It builds a LOG_DATA_ADDR (.offset = slotid, the slot about to be deleted) and appends a record of type RVOOS_DELETE, passing the peeked descriptor as the undo data and NULL redo. Contrast the insert side, which logs the record as redo under RVOOS_INSERT with NULL undo:

// oos_log_delete_physical -- src/storage/oos_file.cpp
  log_append_undoredo_recdes (thread_p, RVOOS_DELETE, &log_addr, recdes_p, NULL);   /* undo = full pre-delete record */
// oos_log_insert_physical -- src/storage/oos_file.cpp
  log_append_undoredo_recdes (thread_p, RVOOS_INSERT, &log_addr, NULL, recdes_p);   /* redo = the record */

The delete undo image is the complete chunk (header plus payload), so it can be re-inserted verbatim at the same slotid during rollback. The mirror symmetry of these two appends — delete logs the record as undo, insert as redo — is exactly what the recovery dispatch table exploits (8.6).

8.5 No sysop: why abort and crash recovery are safe

oos_delete opens no system operation. A sysop would make the whole multi-chunk delete a single atomic recovery unit. OOS deliberately does not do this; instead each chunk delete carries its own full undo data and consistency rides the standard transaction undo path (spelled out in the oos_delete header comment):

• Transaction abort. Undo records replay in reverse order. Each RVOOS_DELETE undo replays as a re-insert (8.6) at the chunk’s original slotid. Deletes are logged head-to-tail, so undo restores them tail-to-head and the chain is fully reconstructed.
• Crash recovery. An uncommitted transaction caught mid-delete has the same undo records; recovery’s undo pass replays them, restoring every removed chunk.
• Error mid-chain. If branch A/B/C/D fires after some chunks were deleted, those deletes are durable undo records. The function returns the error and the caller MUST abort the transaction.

Invariant — caller-aborts-on-error. oos_delete returning non-NO_ERROR is a hard contract that the caller abort the transaction. If violated (caller commits anyway) the deleted chunks become permanent while the tail chunks are orphaned. The source comment says this is acceptable only because storage-layer errors always propagate up to a transaction abort.

A second consequence of the no-sysop design — and the property the idempotency probe in 8.7 exists to support:

Invariant — delete is idempotent. Because each chunk delete is a self-contained logged slot removal with no enclosing sysop, the operation is safe to replay or retry after a crash: a chunk that is already gone simply trips S_DOESNT_EXIST. Vacuum (Ch.10) relies on this — after a crash it may re-walk a chain some of whose chunks a committed run already removed. The probe oos_chunk_exists lets such a retry skip the already-gone OIDs instead of failing.

8.6 The WAL handlers and the cross-wired recovery table

Two handlers replay OOS slot operations during recovery: oos_rv_redo_insert (re-inserts a record) and oos_rv_redo_delete (removes a slot). The interesting part is how they wire into the recovery dispatch table RV_fun[], where each entry names a separate function for the undo column and the redo column.

// RV_fun[] -- src/transaction/recovery.c
  {RVOOS_INSERT,
   "RVOOS_INSERT",
   oos_rv_redo_delete,
   oos_rv_redo_insert,
   NULL,
   NULL},
  {RVOOS_DELETE,
   "RVOOS_DELETE",
   oos_rv_redo_insert,
   oos_rv_redo_delete,
   NULL,
   NULL},

In RV_fun[] the third field is the undo function and the fourth is the redo function. So an RVOOS_DELETE record, when undone (rollback / crash recovery of an incomplete txn), runs oos_rv_redo_insert and re-creates the slot from the undo image — precisely how the chain is restored; symmetrically RVOOS_INSERT undo runs oos_rv_redo_delete. Two physical primitives, reused in all four directions. (The general undo-vs-redo pass mechanics are in cubrid-recovery-manager-detail.md.)

// oos_rv_redo_insert -- src/storage/oos_file.cpp
  slotid = rcv->offset;
  recdes.type = * (INT16 *) (rcv->data);                       /* <- record type prefix */
  recdes.data = (char *) (rcv->data) + sizeof (recdes.type);
  recdes.area_size = recdes.length = rcv->length - sizeof (recdes.type);
  assert (recdes.type == REC_HOME);
  sp_success = spage_insert_for_recovery (thread_p, rcv->pgptr, slotid, &recdes);  /* <- exact slotid restore */
  pgbuf_set_dirty (thread_p, rcv->pgptr, DONT_FREE);
  if (sp_success != SP_SUCCESS)
    {
      if (sp_success != SP_ERROR)
        { er_set (ER_FATAL_ERROR_SEVERITY, ARG_FILE_LINE, ER_GENERIC_ERROR, 0); }  /* <- non-ERROR failure is fatal */
      assert (er_errid () != NO_ERROR);
      return er_errid ();
    }
  return NO_ERROR;

spage_insert_for_recovery (not the ordinary spage_insert) forces the record back into the exact original slotid carried in rcv->offset, preserving the OID identity that the rest of the chain’s next_chunk_oid pointers still reference. A result that is not SP_SUCCESS and not even SP_ERROR escalates to ER_FATAL_ERROR_SEVERITY — during recovery there is no graceful degradation.

// oos_rv_redo_delete -- src/storage/oos_file.cpp
  slotid = rcv->offset;
  PGSLOTID deleted_slotid = spage_delete (thread_p, rcv->pgptr, slotid);
  if (deleted_slotid == NULL_SLOTID)
    { assert (false); er_set (ER_FATAL_ERROR_SEVERITY, ARG_FILE_LINE, ER_GENERIC_ERROR, 0); return er_errid (); }
  pgbuf_set_dirty (thread_p, rcv->pgptr, DONT_FREE);

  VPID page_vpid;                                              /* <- evict stale bestspace hint on rollback */
  pgbuf_get_vpid (rcv->pgptr, &page_vpid);
  (void) oos_stats_del_bestspace_by_vpid (thread_p, &page_vpid);
  return NO_ERROR;

oos_rv_redo_delete mirrors the forward spage_delete and additionally evicts the page’s bestspace cache entry via oos_stats_del_bestspace_by_vpid. Note the asymmetry with the forward path: forward oos_delete_chain calls oos_stats_update with prev_freespace = 0 so the post-delete free space always exceeds the supplied estimate and the page is added to the cache, whereas the rollback redo-delete just removes the stale entry — recovery invalidates the optimistic bestspace cache rather than maintaining it.

flowchart LR
  subgraph forward["forward op (oos_delete_chain)"]
    F1["RVOOS_DELETE logged<br/>undo=full record"]
  end
  subgraph undo["UNDO column"]
    U1["oos_rv_redo_insert<br/>spage_insert_for_recovery"]
  end
  subgraph redo["REDO column"]
    R1["oos_rv_redo_delete<br/>spage_delete"]
  end
  F1 -->|rollback / crash undo| U1
  F1 -->|roll-forward redo| R1

Figure 8-3. An RVOOS_DELETE record reuses the two primitives in opposite directions for undo vs redo.

8.7 oos_chunk_exists: the idempotency probe

Reclamation drivers that may retry (vacuum’s forward-walk block retry; eager cleanup that races a committed sysop) need to ask “is this chunk already gone?” without tripping the S_DOESNT_EXIST hard error inside oos_delete_chain. oos_chunk_exists is the read-only probe for that — the concrete enabler of the idempotency invariant in 8.5.

// oos_chunk_exists -- src/storage/oos_file.cpp
  *out_exists = false;
  ... VPID from oid ...
  PAGE_PTR page_ptr = NULL;
  int error_code = pgbuf_fix_if_not_deallocated (thread_p, &vpid, PGBUF_LATCH_READ,
                                                 PGBUF_UNCONDITIONAL_LATCH, &page_ptr);
  if (error_code != NO_ERROR) { ASSERT_ERROR (); return error_code; }   /* branch 1: I/O/interrupt -> propagate */
  if (page_ptr == NULL) { return NO_ERROR; }                            /* branch 2: page deallocated -> gone */

  RECDES probe = RECDES_INITIALIZER;
  SCAN_CODE code = spage_get_record (thread_p, page_ptr, oid.slotid, &probe, PEEK);
  pgbuf_unfix_and_init (thread_p, page_ptr);
  if (code == S_SUCCESS) { *out_exists = true; return NO_ERROR; }       /* branch 3: slot present -> exists=true */
  if (code == S_DOESNT_EXIST) { return NO_ERROR; }                      /* branch 4: slot removed -> gone */
  ASSERT_ERROR ();
  int errid = er_errid ();
  return errid != NO_ERROR ? errid : ER_FAILED;                         /* branch 5: S_ERROR -> propagate */

The function is read-latched and uses pgbuf_fix_if_not_deallocated, which (unlike plain pgbuf_fix) returns NO_ERROR with page_ptr == NULL when the page has been legitimately freed rather than erroring. The five inline-labelled branches are exhaustive: only branch 2 (page deallocated) and branch 4 (S_DOESNT_EXIST, slot removed but page alive) report NO_ERROR && *out_exists = false. Branch 3 is the live hit (exists=true); branches 1 (fix error) and 5 (S_ERROR) propagate the error. “Already gone” is thus narrowly exactly branches 2 and 4.

Invariant — gone-is-not-an-error, but-only-twice. A retrying caller may treat NO_ERROR && !*out_exists as “skip this OID safely”. The code enforces the narrow definition by reporting NO_ERROR for exactly branches 2 and 4 and propagating everything else. If violated (e.g. collapsing S_ERROR into “gone”), a retry loop would silently swallow real corruption or I/O faults and skip records it should have failed on.

8.8 Page and file reclaim: oos_remove_page and oos_remove_file

Page deallocation is intentionally not done in the delete primitive. oos_delete_chain only removes slots; it never frees a page even when its last slot leaves. Inline dealloc would require a sysop (file-level dealloc is not undo-friendly the way slot delete is), which is what this design avoids — the oos_delete header comment states empty pages “will be reclaimed by vacuum after the transaction commits.” Dealloc lives in a separate, still-unwired function oos_remove_page, marked as future vacuum machinery (Chapter 10):

// oos_remove_page -- src/storage/oos_file.cpp
// TODO: will be called by vacuum when OOS vacuum is implemented
int
oos_remove_page (THREAD_ENTRY *thread_p, const VFID &oos_vfid, const VPID &vpid)
{
  int err = file_dealloc (thread_p, &oos_vfid, &vpid, FILE_OOS);
  if (err != NO_ERROR)
    { oos_error ("file_dealloc failed for vpid={...}", ...); return err; }   /* only branch: dealloc-failed propagate */
  return NO_ERROR;
}

oos_remove_page is a one-call wrapper over file_dealloc(..., FILE_OOS) with a single dealloc-failed error branch. It does no slot iteration and no bestspace bookkeeping — by the time vacuum calls it the page is presumed already empty.

Whole-file reclaim — dropping the entire OOS file when its owning table is dropped — is oos_remove_file, and it is two steps, not one:

// oos_remove_file -- src/storage/oos_file.cpp
int
oos_remove_file (THREAD_ENTRY *thread_p, const VFID &oos_vfid)
{
  (void) oos_stats_del_bestspace_by_vfid (thread_p, &oos_vfid);   /* drop in-memory free-space cache for this file */
  file_postpone_destroy (thread_p, &oos_vfid);                    /* schedule physical file destroy at commit */
  return NO_ERROR;
}

oos_stats_del_bestspace_by_vfid evicts every cached bestspace hint for the file from the in-memory cache (the file is about to vanish, so the hints would dangle), and file_postpone_destroy schedules the actual file destruction as a postpone action — it runs after the transaction commits, so a rollback before commit leaves the file intact. The split between oos_remove_page (slot-level dealloc, vacuum) and oos_remove_file (whole-file destroy, drop-table) keeps the per-record delete path in 8.2 sysop-free while routing the heavyweight reclaim through the standard postpone/commit machinery.

8.9 The row-level-lock concurrency TODO

Immediately above oos_delete_chain the source carries an unresolved concurrency note:

// oos_delete_chain (preceding comment) -- src/storage/oos_file.cpp
// TODO: concurrency — this function assumes the caller holds a row-level lock (e.g., X_LOCK from heap layer)
//       to prevent concurrent deletion of the same OOS chain. Verify this assumption when wiring callers.

The write-latch-per-page strategy in 8.2 serializes a single page but does not prevent two transactions from both walking the same chain (they latch each page in turn, not the whole chain atomically). Correctness therefore rests on an external assumption — the caller holds the heap-layer row lock for the owning record — recorded as a TODO rather than enforced in code. A reader wiring the reclamation drivers must preserve it.

8.10 Chapter summary — key takeaways

1. One funnel. Every OOS deletion (eager cleanup, vacuum) goes through oos_delete, which delegates verbatim to the static walker oos_delete_chain.
2. The walker is a next_chunk_oid loop with four error branches (fix, peek, length-guard, delete) and the load-bearing rule that next_chunk_oid is copied out of the header before the slot is deleted — otherwise the chain tail leaks.
3. No sysop, by design. Each chunk delete is a standalone RVOOS_DELETE undo record carrying the full pre-image, so abort and crash recovery restore the chain by replaying undo in reverse via spage_insert_for_recovery. The same property makes delete idempotent, so vacuum can re-walk a chain after a crash and skip already-gone chunks.
4. Caller-aborts-on-error is a hard contract; a mid-chain error committed past leaves permanent partial deletes and orphaned tail chunks. The forward corruption guard degrades at ER_ERROR_SEVERITY; the recovery-time equivalents escalate to ER_FATAL_ERROR_SEVERITY.
5. The recovery table RV_fun[] is cross-wired: RVOOS_DELETE undo runs oos_rv_redo_insert and redo runs oos_rv_redo_delete (vice versa for RVOOS_INSERT); two primitives reused in all four directions.
6. Reclaim is split out into oos_remove_page (a still-unwired file_dealloc(..., FILE_OOS) wrapper destined for vacuum, Ch.10) and oos_remove_file (drop-table reclaim: oos_stats_del_bestspace_by_vfid then file_postpone_destroy). The delete primitive only removes slots, keeping it sysop-free.
7. oos_chunk_exists is the idempotency probe with a narrow “gone” definition — page deallocated or slot removed both yield NO_ERROR && !exists; all other failures propagate. Whole-chain serialization still rests on the external row-lock TODO.

Chapter 9: UPDATE DELETE and Eager Non MVCC Cleanup

When a row changes or dies, some of the OOS records it pointed at become garbage. This chapter answers which OOS records become garbage and who deletes them on the code paths that have no vacuum behind them. For the reclamation policy — when an OOS chain becomes dead, why MVCC versions linger — see cubrid-oos.md (UPDATE / DELETE and reclamation); for the surrounding heap machinery (REC_HOME vs REC_RELOCATION, the forwarding OID, physical update sub-paths) see cubrid-heap-manager-detail.md. This chapter traces the four heap call sites and the cleanup primitive without re-deriving either. Vacuum’s half of the policy (the MVCC paths) is Chapter 10.

9.1 The reclamation policy split

Reclamation is split by a single boolean, is_mvcc_op, computed at the top of heap_update / heap_delete from HEAP_UPDATE_IS_MVCC_OP:

// HEAP_UPDATE_IS_MVCC_OP -- src/storage/heap_file.c
#if defined (SERVER_MODE)
#define HEAP_UPDATE_IS_MVCC_OP(is_mvcc_class, update_style) \
    ((is_mvcc_class) && (!HEAP_IS_UPDATE_INPLACE (update_style)) ? (true) : (false))
#else
#define HEAP_UPDATE_IS_MVCC_OP(is_mvcc_class, update_style) (false)   /* <- SA_MODE: never MVCC */
#endif

Two facts fall out of this macro that govern the whole chapter:

• In SA_MODE the macro is hard-wired to false. A standalone process has no concurrent readers and no vacuum daemon, so an old OOS record nobody deletes here is leaked forever. Cleanup must be eager.
• In SERVER_MODE the macro is still false when the class is not MVCC-enabled (!is_mvcc_class — system catalog classes) or the update is in-place. So !is_mvcc_op is not a synonym for “SA_MODE”; it also fires server-side on catalog classes. The diagnostic strings still say "SA_MODE eager OOS cleanup", a historical misnomer — the gate is broader than the label.

Mode	is_mvcc_op	Who reclaims old OOS	Where
SA_MODE (any class)	false	eager, inline	this chapter
SERVER_MODE, MVCC-disabled class (catalog)	false	eager, inline	this chapter
SERVER_MODE, MVCC class, normal update/delete	true	vacuum, deferred	Chapter 10

The MVCC branch does nothing about old OOS at delete/update time: it keeps the old record alive so a concurrent reader holding the old version’s snapshot can still dereference its inline OOS OID (see Chapter 7’s re-inlining path), and it tags the WAL record so vacuum’s forward-walk finds the pre-image later. Those tags (RVHF_DELETE_NEWHOME_NOTIFY_VACUUM, RVHF_UPDATE_NOTIFY_VACUUM) are set in the same functions but on the is_mvcc_op == true side; one of them is shown in 9.5.

INVARIANT (one-OID-per-record). Every OOS OID is referenced by exactly one heap record; OOS records are never shared across rows. This is the foundation that makes eager deletion correct, and the code enforces it on the producer side: M1 UPDATE of a changed OOS column always allocates a fresh OOS OID rather than reusing the old one (the Chapter 4 write path calls oos_insert, never an in-place rewrite). So when heap_oos_delete_unreferenced decides an old OID is “unreferenced by the new image”, it can call oos_delete unconditionally with no reference count — nobody else points at it. Violate the invariant (two rows sharing one OID) and this code would delete a chain still live for another row.

9.2 Struct relationships at the four call sites

No new struct is introduced in this chapter. The cleanup primitive reads from a RECDES (the old and, for UPDATE, the new heap-record image) and from HEAP_OPERATION_CONTEXT (for hfid and the heap oid, used to resolve the OOS file and for diagnostics). Figure 9-1 shows how the four guarded call sites feed the one primitive.

flowchart TB
  subgraph DELETE
    dh["heap_delete_home<br/>old=home_recdes (REC_HOME)<br/>new=NULL"]
    dr["heap_delete_relocation<br/>old=forward_recdes (REC_NEWHOME)<br/>new=NULL"]
  end
  subgraph UPDATE
    uh["heap_update_home<br/>old=home_recdes (REC_HOME)<br/>new=context-&gt;recdes_p"]
    ur["heap_update_relocation<br/>old=forward_recdes (REC_NEWHOME)<br/>new=context-&gt;recdes_p"]
  end
  dh --> G{"!is_mvcc_op<br/>&& type matches<br/>&& heap_recdes_contains_oos"}
  dr --> G
  uh --> G
  ur --> G
  G -->|yes| F["heap_oos_delete_unreferenced"]
  G -->|no| skip["skip: MVCC defers to vacuum,<br/>or record has no OOS"]
  F --> ggo["heap_recdes_get_oos_oids (old)"]
  F --> ggn["heap_recdes_get_oos_oids (new) if UPDATE"]
  F --> vfid["heap_oos_find_vfid (resolve OOS VFID)"]
  F --> del["oos_delete per surviving OID"]

The crucial detail is which record is passed as old_recdes. For a REC_HOME row the data (and its inline OOS OIDs) lives in the home slot, so old_recdes is context->home_recdes. For a relocated row the home slot holds only an 8-byte forwarding OID; the actual attribute bytes and OOS slots live on the forward REC_NEWHOME page. So the relocation call sites pass forward_recdes, which they have just spage_get_record-ed from the forward page (the delete site with PEEK, the update site with COPY into a stack buffer — either way the forward bytes are in hand before the slot is destroyed). Passing the home slot there would find no VOT and no OOS.

9.3 heap_oos_delete_unreferenced — the eager primitive, every branch

This is the only function that actually issues oos_delete on the non-vacuum paths. Its contract: delete every OOS OID referenced by old_recdes and not referenced by new_recdes. new_recdes == NULL means “nothing is referenced anymore” (DELETE) so all old OIDs go; a non-NULL new_recdes (UPDATE) means take the set difference.

// heap_oos_delete_unreferenced -- src/storage/heap_oos.cpp
std::vector<OID> old_oos_oids;
std::vector<OID> new_oos_oids;
VFID oos_vfid;
int error_code;

error_code = heap_recdes_get_oos_oids (old_recdes, old_oos_oids);
if (error_code != NO_ERROR)
  {
    ASSERT_ERROR ();
    er_log_debug (...);            /* <- strict failure: flag was set, extraction must succeed */
    return error_code;
  }
if (old_oos_oids.empty ())
  {
    return NO_ERROR;               /* <- nothing tagged OOS in old image; nothing to do */
  }

if (new_recdes != NULL)            /* <- UPDATE only; DELETE skips, leaving new_oos_oids empty */
  {
    error_code = heap_recdes_get_oos_oids (new_recdes, new_oos_oids);
    if (error_code != NO_ERROR) { ASSERT_ERROR (); er_log_debug (...); return error_code; }
  }

if (!heap_oos_find_vfid (thread_p, &context->hfid, &oos_vfid, false))   /* <- false = do not create */
  {
    er_log_debug (...);
    assert_release (false);        /* <- OOS flag set but no OOS file => corruption */
    return ER_FAILED;
  }

for (const OID &old_oid : old_oos_oids)
  {
    if (oos_oid_in_vector (new_oos_oids, &old_oid))
      {
        continue;                  /* <- same physical OOS in both images: keep it (M3 reuse) */
      }
    error_code = oos_delete (thread_p, oos_vfid, old_oid);
    if (error_code != NO_ERROR) { ASSERT_ERROR (); er_log_debug (...); return error_code; }
  }
return NO_ERROR;

Branch-complete walkthrough:

1. Extract old OIDs. On error, log with the op_ctx tag and heap identifiers, and return; the caller must abort (9.6).
2. Old image has no OOS OIDs (old_oos_oids.empty ()): return NO_ERROR. The call site already checked heap_recdes_contains_oos, so an empty result is unexpected but benign here. (Contrast the in-function assert inside heap_recdes_get_oos_oids, which does treat “flag set but zero OIDs” as corruption; see 9.4.)
3. UPDATE vs DELETE fork on new_recdes. If non-NULL (UPDATE), extract the new image’s OIDs — no heap_recdes_contains_oos guard needed, since heap_recdes_get_oos_oids returns NO_ERROR with an empty vector when the new record has no OOS flag (exactly the “delete everything old” behavior). If new_recdes == NULL (DELETE), new_oos_oids stays empty, so every old OID fails the membership test and is deleted.
4. Resolve the OOS file. heap_oos_find_vfid (..., false) (create flag false). False return means the header says OOS but the per-class OOS file is missing — real corruption: log, assert_release (false), ER_FAILED.
5. Per-OID set difference. For each old_oid: if oos_oid_in_vector (new_oos_oids, &old_oid), the same physical chain is referenced before and after — continue, keeping it (the M3 unchanged-column reuse hook; in M1 the producer always reallocates so this branch never fires, but the code is already correct for the day it does). Otherwise oos_delete; on its error, log the full identifier triple and return — a mid-loop failure leaves earlier chunks deleted, hence the abort contract.
6. Success: return NO_ERROR.

Figure 9-2 makes the DELETE-vs-UPDATE asymmetry explicit:

flowchart TD
  A["enter"] --> B["get old OIDs"]
  B -->|err| E1["log + return err"]
  B -->|empty| R0["return NO_ERROR"]
  B -->|non-empty| C{"new_recdes == NULL?"}
  C -->|yes DELETE| D0["new_oos_oids stays empty"]
  C -->|no UPDATE| D1["get new OIDs"]
  D1 -->|err| E1
  D0 --> V["heap_oos_find_vfid create=false"]
  D1 --> V
  V -->|not found| E2["log + assert + ER_FAILED"]
  V -->|found| L["for each old_oid"]
  L --> M{"old_oid in new_oos_oids?"}
  M -->|yes| K["keep, continue"]
  M -->|no| X["oos_delete"]
  X -->|err| E1
  X -->|ok| L
  L -->|done| R1["return NO_ERROR"]

9.4 heap_recdes_get_oos_oids — walking the VOT for OOS entries

This collector (in heap_file.c) bridges the on-disk record format and the OID vector the primitive operates on. It walks the variable-offset table (VOT); for each entry whose offset carries the OOS flag bit it reads the inline OID (just OR_OID_SIZE bytes — the inline stub also carries a length, but only the OID is needed to address the chain for deletion).

// heap_recdes_get_oos_oids -- src/storage/heap_file.c
oos_oids.clear ();
if (!heap_recdes_contains_oos (recdes))
  return NO_ERROR;                  /* <- header flag clear: empty result, success */
const int offset_size = OR_GET_OFFSET_SIZE (recdes->data);
const int max_var_count = (recdes->length - OR_HEADER_SIZE (recdes->data)) / offset_size;

for (int index = 0; index <= max_var_count; ++index)
  {
    if (index == max_var_count)
      { assert_release (false /*LAST_ELEMENT not found in bounds*/); return ER_FAILED; }
    int offset;  /* read per offset_size: BYTE/SHORT/INT; default -> assert_release(false), ER_FAILED */

    if (OR_IS_OOS (offset))
      {
        OID oid = OID_INITIALIZER;
        const char *oid_ptr = (char *) recdes->data + OR_VAR_OFFSET (recdes->data, index);
        if (oid_ptr + OR_OID_SIZE > (char *) recdes->data + recdes->length)
          { assert (false); return ER_FAILED; }   /* <- OID read would overrun record */
        OR_BUF buf; or_init (&buf, (char *) oid_ptr, OR_OID_SIZE);
        if (or_get_oid (&buf, &oid) != NO_ERROR) { assert (false); return ER_FAILED; }
        if (OID_ISNULL (&oid)) { assert (false); return ER_FAILED; }  /* <- null in OOS slot */
        oos_oids.emplace_back (oid);                                  /* oos_debug() in NDEBUG-off builds */
      }
    if (OR_IS_LAST_ELEMENT (offset))
      {
        if (oos_oids.empty ())
          { assert (false); return ER_FAILED; }    /* <- flag said OOS, VOT had none */
        return NO_ERROR;            /* <- normal exit: reached sentinel */
      }
  }
assert (false /*unreachable: there must be last element*/);   /* <- loop never falls through */
return ER_FAILED;

Branch coverage:

• Header flag clear (!heap_recdes_contains_oos): return NO_ERROR, empty vector — the leg the UPDATE path relies on for the new image (9.3 step 3).
• Sentinel not found before max_var_count entries: the VOT lacks its OR_IS_LAST_ELEMENT terminator within bounds — assert_release, ER_FAILED. max_var_count is a loose upper bound (length - header) / offset_size; for legacy records lacking the last-element flag it may include alignment/fixed-attribute bytes (the source comment flags those as “not fully supported yet”), so it is a backstop, not an exact count.
• offset_size switch has BYTE / SHORT / INT cases and a default that assert_release (false), ER_FAILED.
• Entry is OOS (OR_IS_OOS (offset)): read the inline OID. Three guards fail strictly — bounds overrun, or_get_oid failure, null OID in an OOS slot — because the header flag already promised a real OID.
• Entry is the sentinel (OR_IS_LAST_ELEMENT): no OID collected despite the flag is an inconsistency (assert (false), ER_FAILED); otherwise return the collected vector (debug builds dump it via oos_debug).
• Loop fall-through (post-loop): every path inside returns or continues, so the trailing assert (false) / ER_FAILED is a defensive backstop that should never execute.

OR_IS_OOS and OR_IS_LAST_ELEMENT are independent bits on the same offset word — an entry can be both, firing the collect and the return in one iteration. Chapter 1 and cubrid-oos.md’s ### Multi-chunk chain own the bit encoding; this chapter does not re-derive it.

INVARIANT (flag implies extractable OID). If heap_recdes_contains_oos is true, walking the VOT must yield at least one non-null OID inside record bounds. Enforced by the assert/assert_release strict-failure exits above. If violated (flag set but no real OID), the eager path returns ER_FAILED rather than silently leaking, and the caller aborts.

9.5 The four heap call sites

All four guards share the shape !is_mvcc_op && <record-type matches> && heap_recdes_contains_oos (<recdes>). The record-type clause and which recdes is passed differ by site. heap_recdes_contains_oos is a one-line flag test:

// heap_recdes_contains_oos -- src/storage/heap_file.c
int flag = (INT32) OR_GET_MVCC_FLAG (record->data);
return flag & OR_MVCC_FLAG_HAS_OOS;

DELETE sites pass new_recdes == NULL (delete all old OOS) and the old image of the dying record. heap_delete_home runs in the non-MVCC else branch passing context->home_recdes (REC_HOME); heap_delete_relocation passes the re-fetched forward_recdes (REC_NEWHOME, since the home slot is just a forwarding OID). Both run before heap_log_delete_physical / heap_delete_physical — once the slot is gone the record bytes and inline OOS OIDs are gone too:

// heap_delete_home (non-mvcc else branch) -- src/storage/heap_file.c
if (!is_mvcc_op && context->record_type == REC_HOME && heap_recdes_contains_oos (&context->home_recdes))
  {
    error_code = heap_oos_delete_unreferenced (thread_p, context, &context->home_recdes, NULL, "delete home");
    if (error_code != NO_ERROR) { ASSERT_ERROR (); return error_code; }
  }
// heap_delete_relocation (non-mvcc branch) -- src/storage/heap_file.c
if (!is_mvcc_op && forward_recdes.type == REC_NEWHOME && heap_recdes_contains_oos (&forward_recdes))
  {
    rc = heap_oos_delete_unreferenced (thread_p, context, &forward_recdes, NULL, "delete relocation");
    if (rc != NO_ERROR) { ASSERT_ERROR (); return rc; }
  }

UPDATE — heap_update_home (old home type REC_HOME). new_recdes is non-NULL — context->recdes_p, the new image — so the primitive computes a set difference. It runs after heap_update_physical; the old OIDs are still extractable because they come from the in-memory context->home_recdes captured earlier, not the now-overwritten slot:

// heap_update_home -- src/storage/heap_file.c
if (!is_mvcc_op && context->home_recdes.type == REC_HOME && heap_recdes_contains_oos (&context->home_recdes))
  {
    error_code = heap_oos_delete_unreferenced (thread_p, context, &context->home_recdes, context->recdes_p,
                                               "update home");
    if (error_code != NO_ERROR) { ASSERT_ERROR (); goto exit; }
  }

UPDATE — heap_update_relocation (old forward type REC_NEWHOME). old_recdes is forward_recdes, new_recdes is context->recdes_p. This guard sits before the four update sub-paths (relocation-to-home, to-bigone, to-relocation, in-place) because all four overwrite or remove the forward slot, so the old forward’s OOS becomes unreachable regardless of sub-path:

// heap_update_relocation -- src/storage/heap_file.c
if (!is_mvcc_op && forward_recdes.type == REC_NEWHOME && heap_recdes_contains_oos (&forward_recdes))
  {
    rc = heap_oos_delete_unreferenced (thread_p, context, &forward_recdes, context->recdes_p, "update relocation");
    if (rc != NO_ERROR) { ASSERT_ERROR (); goto exit; }
  }

For contrast, the MVCC half of heap_update_relocation does not delete; it only re-tags the WAL so vacuum can find the pre-image later. That same forward_recdes / heap_recdes_contains_oos test appears with the opposite is_mvcc_op polarity at the remove_old_forward log call:

// heap_update_relocation (mvcc remove_old_forward) -- src/storage/heap_file.c
LOG_RCVINDEX delete_rcvindex = RVHF_DELETE;
if (is_mvcc_op && forward_recdes.type == REC_NEWHOME && heap_recdes_contains_oos (&forward_recdes))
  {
    delete_rcvindex = RVHF_DELETE_NEWHOME_NOTIFY_VACUUM;   /* <- vacuum forward-walk reclaims old OOS */
  }
heap_log_delete_physical (..., &forward_recdes, true, &prev_version_lsa, delete_rcvindex);

heap_update_home’s MVCC counterpart tags its undo RVHF_UPDATE_NOTIFY_VACUUM similarly. Those notify-vacuum tags are Chapter 10’s entry points; here they are shown only to make the polarity split visible — eager delete on !is_mvcc_op, notify-vacuum on is_mvcc_op.

9.6 Why unconditional deletion is safe, and the abort contract

Two properties combine to make the eager path correct:

1. The one-OID-per-record invariant (9.1) means an OID dropped from the new image is referenced by nobody else. No reference count is consulted; oos_delete is unconditional except for the explicit “present in both images” keep.
2. oos_delete logs each chunk individually with RVOOS_DELETE carrying the full record as undo data, using no enclosing sysop. Abort or crash recovery replays those undo records in reverse and restores every chunk to its original slot. (Chapter 8 dissects this primitive; for the chain model it operates on, see cubrid-oos.md.)

INVARIANT (abort-on-error). A caller that gets a non-NO_ERROR return from heap_oos_delete_unreferenced must abort the transaction. Enforced structurally by every call site checking the return and bailing (return / goto exit), and by the storage-layer convention that errors propagate up to transaction abort. If a caller ignored the error and committed mid-chain, the partially deleted OOS chunks would become permanent while the surviving heap record still references the (now broken) chain — exactly the corruption the per-chunk undo records exist to prevent. This is why the cleanup runs at delete/update time and not asynchronously: it shares the live transaction’s undo so rollback is automatic.

oos_oid_in_vector is the only remaining helper — a deliberate linear scan, justified because the vectors are tiny (one OOS OID per OOS column, single digits in practice):

// oos_oid_in_vector -- src/storage/oos_util.cpp
for (const OID &candidate : oids)
  if (OID_EQ (&candidate, oid))
    return true;
return false;

9.7 Chapter summary — key takeaways

1. One boolean decides everything. is_mvcc_op (from HEAP_UPDATE_IS_MVCC_OP) routes old-OOS reclamation: !is_mvcc_op reclaims eagerly here; is_mvcc_op defers to vacuum (Chapter 10). !is_mvcc_op covers SA_MODE and SERVER_MODE on MVCC-disabled catalog classes, despite the "SA_MODE" diagnostic label.
2. The fresh-OID rule is what makes eager deletion sound. M1 UPDATE always allocates a new OOS OID for a changed column, so the one-OID-per-record invariant holds and heap_oos_delete_unreferenced can delete unconditionally without reference counting.
3. DELETE deletes all, UPDATE deletes the set difference. new_recdes == NULL (DELETE) drops every old OID; a non-NULL new_recdes (UPDATE) keeps OIDs present in both images via oos_oid_in_vector — the M3 reuse hook, dormant in M1.
4. Four call sites, one primitive, with the right recdes. heap_delete_home/heap_update_home pass context->home_recdes (REC_HOME); heap_delete_relocation/heap_update_relocation pass the freshly re-fetched forward_recdes (REC_NEWHOME), because relocated data lives on the forward page. All four are guarded by !is_mvcc_op && <type> && heap_recdes_contains_oos.
5. Extraction is strict. heap_recdes_get_oos_oids walks the VOT and asserts on every inconsistency (no sentinel, null OID, bounds overrun, flag-set-but-empty, unreachable fall-through) — a header that claims OOS must yield extractable OIDs, or the operation fails and the transaction aborts.
6. Correctness rests on the abort contract. oos_delete logs per chunk with no sysop; any error from the eager path obliges the caller to abort so the undo records replay and restore partially deleted chains. Every call site honors this with return/goto exit.
7. The MVCC polarity is visible in the same functions. Where the non-MVCC branch deletes, the MVCC branch instead tags the WAL (RVHF_DELETE_NEWHOME_NOTIFY_VACUUM, RVHF_UPDATE_NOTIFY_VACUUM) so vacuum’s forward-walk reclaims the pre-image later — the hand-off to Chapter 10.

Chapter 10: Vacuum Reclamation

On the MVCC path an UPDATE or DELETE never frees the old row’s OOS chunks inline — the old version is still visible to in-flight snapshots, so the chunks must outlive the operation. The eager non-MVCC cleanup (Chapter 9) only fires when the row was never MVCC-visible to anyone else; for everything else the OOS bytes are deliberately abandoned at modification time and become vacuum’s responsibility.

For the MVCC theory behind why the old version survives, see cubrid-oos.md (§ “UPDATE / DELETE and reclamation”); for where these hooks sit inside the worker loop — vacuum_process_log_block (which owns the memo), vacuum_heap_record, and the shutdown-only assert in vacuum_finished_block_vacuum — see cubrid-vacuum-detail.md (Chapters 8–9). This chapter dissects only the reclamation code in src/query/vacuum_oos.cpp, of which there are two entry points, reflecting two ways vacuum learns an OOS chunk is dead:

• REMOVE path — vacuum_heap_oos_delete_within_sysop. Vacuum is physically removing a dead heap record (REC_HOME / REC_NEWHOME) that carries OOS references. The body is in front of us; we delete its chunks inside the sysop heap-vacuum already opened.
• Forward-walk path — vacuum_forward_walk_reclaim_oos. The dead version no longer exists as a live slot; it survives only as an undo pre-image in the log. Vacuum reconstructs the old image and frees the OOS it pointed to inside a sysop it opens itself.

Both share one philosophy: a failed OOS reclamation must never fail the vacuum block. Giving up costs a small, logged, bounded leak (a few orphaned chunks on disk); failing the block trips the shutdown-only assert in vacuum_finished_block_vacuum and can wedge vacuum entirely.

10.1 The VFID lookup memo and the FOUND/NONE/ERROR tri-state

Before either path deletes anything it must know which OOS file the heap uses. A heap and its OOS file are 1:1 and the mapping never changes once the heap exists, so the answer is cacheable. The cache is VACUUM_OOS_VFID_MEMO, a single-slot struct declared on the stack in vacuum_process_log_block.

// vacuum_oos_vfid_memo -- src/query/vacuum_oos.hpp
struct vacuum_oos_vfid_memo
{
  bool valid = false;    /* false until the first successful lookup */
  VFID heap_vfid;    /* key; meaningful only when valid */
  VFID oos_vfid;    /* value; meaningful only when valid (VFID_NULL = "no OOS file") */
};

Field	Role	Why it exists
valid	Gate. false until the first successful (FOUND or NONE) lookup populates the slot.	A zero-initialized memo must not look like a cached answer; valid distinguishes “never looked up” from “looked up, no OOS file”.
heap_vfid	Cache key — the heap the cached answer is about; compared with VFID_EQ against the requested heap.	A worker block can touch more than one heap; the key prevents serving heap A’s OOS file when asked about heap B.
oos_vfid	Cache value — the heap’s OOS file id. May itself be VFID_NULL = “this heap legitimately has no OOS file.”	Lets the memo cache the negative answer too, so a heap with no OOS file is not re-probed on every record.

Invariant — single-slot, per-worker-per-block, no synchronization. The memo lives on vacuum_process_log_block’s stack, so each worker thread owns its own copy, destroyed when the block finishes. The header is emphatic: do NOT replace with a static — vacuum runs across multiple worker threads and a shared static would race with no lock. One slot suffices because a bulk UPDATE emits a run of consecutive RVHF_UPDATE_NOTIFY_VACUUM records for the same heap; one slot collapses that whole run into one real lookup plus N cheap VFID_EQ hits.

Invariant — transient errors are never memoized. Only FOUND and NONE write the slot. An ERROR leaves valid/oos_vfid untouched so the next record retries from scratch. Violating this would cache a bogus VFID_NULL after a one-off page-read hiccup and silently leak every subsequent record’s OOS for the rest of the block.

vacuum_oos_vfid_lookup returns an explicit enum precisely so callers can tell a missing OOS file apart from a failed lookup:

// VACUUM_OOS_VFID_LOOKUP_RESULT -- src/query/vacuum_oos.cpp
typedef enum
{
  VACUUM_OOS_VFID_FOUND,  /* found a real OOS file; its id is in out_oos_vfid */
  VACUUM_OOS_VFID_NONE,    /* the heap simply has no OOS file (normal) */
  VACUUM_OOS_VFID_ERROR    /* lookup failed; error left set, answer not cached */
} VACUUM_OOS_VFID_LOOKUP_RESULT;

flowchart TD
  A["vacuum_oos_vfid_lookup(memo, heap_vfid)"] --> B{"memo.valid and VFID_EQ key?"}
  B -- yes --> C{"VFID_ISNULL cached value?"}
  C -- yes --> R_NONE["return NONE, cached negative"]
  C -- no --> R_FOUND["return FOUND, cached hit"]
  B -- no --> F{"file_descriptor_get OK?"}
  F -- no --> R_ERR1["return ERROR, error left set, not cached"]
  F -- yes --> G{"HFID_IS_NULL?"}
  G -- yes --> R_ERR2["return ERROR, not cached"]
  G -- no --> H{"heap_oos_find_vfid true?"}
  H -- no --> I{"er_errid != NO_ERROR?"}
  I -- yes --> R_ERR3["return ERROR, real read failure, not cached"]
  I -- no --> J["honest no-OOS-file, cache VFID_NULL"]
  H -- yes --> K["write memo: valid, key, value"]
  J --> K
  K --> L{"VFID_ISNULL value?"}
  L -- yes --> R_NONE2["return NONE"]
  L -- no --> R_FOUND2["return FOUND"]

Figure 10-1: Branch-complete control flow of vacuum_oos_vfid_lookup — memo hit (NONE/FOUND), the two early ERRORs, the false-return disambiguation, and the never-memoized ERROR.

The subtle branch is the heap_oos_find_vfid false return: it can mean a real read failure or the honest “no OOS file id in the heap header” case. The er_errid () != NO_ERROR test disambiguates them.

// vacuum_oos_vfid_lookup -- src/query/vacuum_oos.cpp
if (!heap_oos_find_vfid (thread_p, &hfid, out_oos_vfid, false))
  {
    VFID_SET_NULL (out_oos_vfid);
    if (er_errid () != NO_ERROR)
      {
  /* A real failure happened while reading the heap header ... Do not cache it. */
  return VACUUM_OOS_VFID_ERROR;
      }
    /* No error was raised, so this is the honest "no OOS file" case ... cache that VFID_NULL. */
  }

memo->valid = true;
VFID_COPY (&memo->heap_vfid, heap_vfid);
VFID_COPY (&memo->oos_vfid, out_oos_vfid);

All three return VACUUM_OOS_VFID_ERROR branches (after file_descriptor_get, after HFID_IS_NULL, and the real-failure branch above) leave the database error set on purpose — the comment is explicit that we do not er_clear() here, because the caller wants to log it. Only after the false-return disambiguation are the three memo fields written together, so a zero-initialized or ERROR-aborted memo never looks like a cached answer.

10.2 The forward-walk path: reconstructing a dead pre-image

vacuum_forward_walk_reclaim_oos is the harder path. The dead version is not a live slot; it exists only as the undo pre-image of an RVHF_UPDATE_NOTIFY_VACUUM (update_old_home) or RVHF_DELETE_NEWHOME_NOTIFY_VACUUM (remove_old_forward) log record. The raw undo bytes are an INT16 record type followed by the pre-image body.

flowchart TD
  A["vacuum_forward_walk_reclaim_oos(undo_data, size, heap_vfid, memo)"] --> B{"undo_data NULL or size too small?"}
  B -- yes --> R0["return, nothing to reclaim"]
  B -- no --> C["build undo_recdes: type, data+2, len-2"]
  C --> D{"REC_HOME/REC_NEWHOME and contains_oos?"}
  D -- no --> R1["return, forwarding ptr or no OOS"]
  D -- yes --> E["db_private_alloc stable_copy"]
  E --> F{"alloc NULL?"}
  F -- yes --> R2["log bounded leak, er_clear, return"]
  F -- no --> G["memcpy, parse_recdes.data = stable_copy"]
  G --> H["vacuum_oos_vfid_lookup"]
  H --> I{"lookup_result"}
  I -- ERROR --> J["log bounded leak, er_clear, do NOT propagate"]
  I -- FOUND --> K["heap_recdes_get_oos_oids then delete_atomic"]
  K --> L{"both OK?"}
  L -- no --> M["log bounded leak, er_clear, do NOT propagate"]
  L -- yes --> N["deleted"]
  I -- NONE --> O["log invariant-violation, abort"]
  J --> Z["db_private_free_and_init stable_copy"]
  M --> Z
  N --> Z
  O -.->|process dies| X(("abort"))

Figure 10-2: Branch-complete control flow of vacuum_forward_walk_reclaim_oos. Every non-aborting outcome converges on db_private_free_and_init; only the NONE branch leaves via abort().

Branch 1 — too small. If undo_data == NULL or undo_data_size <= sizeof(INT16), there is no room for a pre-image body; return immediately.

Branch 2 — wrong record type or no OOS. The undo image is wrapped in a RECDES and gated by a combined check:

// vacuum_forward_walk_reclaim_oos -- src/query/vacuum_oos.cpp
RECDES undo_recdes;
undo_recdes.type = * (INT16 *) undo_data;
undo_recdes.data = undo_data + sizeof (INT16);
undo_recdes.length = undo_data_size - (int) sizeof (INT16);

if (! ((undo_recdes.type == REC_HOME || undo_recdes.type == REC_NEWHOME)
       && heap_recdes_contains_oos (&undo_recdes)))
  {
    return;
  }

This guard is load-bearing. heap_update_bigone / update_old_home log an old REC_BIGONE / REC_RELOCATION slot, which is just an 8-byte OID — not a row record. If such an image reached heap_recdes_contains_oos, it would read the OID’s pageid as if it were an MVCC header; a pageid with bit 27 set would masquerade as the “has OOS” flag, and heap_recdes_get_oos_oids would chase a garbage reference list into assert_release. So only real row records pass — the same REC_HOME / REC_NEWHOME guard the eager-delete paths use.

Branch 3 — the stable, aligned copy. The most subtle defensive move in the file. Before fixing any page, the pre-image is copied out of the log buffer:

// vacuum_forward_walk_reclaim_oos -- src/query/vacuum_oos.cpp
RECDES parse_recdes = undo_recdes;
char *stable_copy = (char *) db_private_alloc (thread_p, undo_recdes.length);
if (stable_copy == NULL)
  {
    vacuum_er_log_error (VACUUM_ER_LOG_HEAP,
       "forward-walk oos cleanup: failed to allocate %d bytes ... (bounded leak) ...",
       undo_recdes.length, VFID_AS_ARGS (heap_vfid));
    er_clear ();
    return;
  }
memcpy (stable_copy, undo_recdes.data, undo_recdes.length);
parse_recdes.data = stable_copy;    /* parse from the copy, never the log page */

One copy fixes two hazards:

1. Page rotation. undo_recdes.data usually points straight into the worker’s current log page. The page reads inside vacuum_oos_vfid_lookup (next step) can generate log activity that swaps that page out. The comment records a live observation: the flags byte at this address changed from 0x69 to 0x00 across the lookup. Parsing the rotated bytes would quietly find nothing.
2. Alignment. The image starts at undo_data + sizeof(INT16) — a 2-byte-misaligned pointer. The OR_BUF readers inside heap_recdes_get_oos_oids assert on an unaligned pointer in debug builds; db_private_alloc returns aligned memory. The alloc-failure branch logs a bounded leak and er_clear()s; it does not fail the block.

Branch 4 — the lookup tri-state. With a stable copy in hand, vacuum_oos_vfid_lookup is called and its three outcomes split the flow:

• ERROR → log a bounded-leak message, er_clear(), fall through to free the copy. The comment is blunt: DO NOT propagate; the block must complete.
• FOUND → extract the OID list with heap_recdes_get_oos_oids and, on success, hand it (by std::move) to vacuum_forward_walk_oos_delete_atomic. Failure from either step is logged as a bounded leak and er_clear()ed; the atomic helper has already aborted its own sysop, so no partial delete survives.
• NONE → the live abort().

// vacuum_forward_walk_reclaim_oos -- src/query/vacuum_oos.cpp  (NONE branch)
else
  {
    /* VACUUM_OOS_VFID_NONE. TODO(do not review, remove before develop merge): the guard above already
     * confirmed this undo image is REC_HOME/REC_NEWHOME and heap_recdes_contains_oos, so reaching here
     * means the record's OOS flag is set but its heap has no OOS file - an invariant violation ...
     * CI runs the release build where assert_release only logs, so abort() to crash and surface the bug. */
    vacuum_er_log_error (VACUUM_ER_LOG_HEAP,
       "forward-walk oos cleanup: undo image has OOS flag but heap has no OOS file; ...",
       VFID_AS_ARGS (heap_vfid));
    abort ();
  }

Caution — live abort() and a temporary include. This abort() is intentional but temporary. Once Branch 2 confirmed the image is a row record and heap_recdes_contains_oos returned true, a NONE result means the record claims OOS but its heap has no OOS file — a true invariant violation. Because CI runs the release build (NDEBUG), assert_release only logs a notification that er_clear() would then wipe, so the bug would never surface; the abort() forces a crash to catch it. It is paired with the temporary #include <cstdlib> (commented /* abort - TODO: remove before develop merge (temporary CI crash) */) and the TODO(do not review, remove before develop merge) marker. A reader preparing this for the develop merge must strip both the abort() and the temporary include — in production this branch must degrade to a bounded leak, not a process kill.

db_private_free_and_init (thread_p, stable_copy) is reached on every non-aborting path (ERROR, FOUND-success, FOUND-failure), so the snapshot buffer is freed exactly once. The NONE branch never reaches it, but abort() tears the process down anyway.

10.3 The atomic multi-chunk delete: ownership, sort, idempotence

vacuum_forward_walk_oos_delete_atomic is where the forward-walk deletes happen. Its signature is the first clue — the OID vector is taken by value:

// vacuum_forward_walk_oos_delete_atomic -- src/query/vacuum_oos.cpp
static int
vacuum_forward_walk_oos_delete_atomic (THREAD_ENTRY *thread_p, const VFID *oos_vfid, std::vector<OID> oos_oids)
{
  int error_code = NO_ERROR;
  std::sort (oos_oids.begin (), oos_oids.end (),
       [] (const OID &a, const OID &b) { return oid_compare (&a, &b) < 0; });

Invariant — the helper owns the OID vector outright. Passed by value so the function can sort in place and, more importantly, so the OIDs do not live in the log buffer that oos_delete may rotate — the same page-rotation hazard as the stable copy in 10.2, applied to the OID list. If the vector aliased the caller’s undo_data, a mid-loop page swap could change the OIDs being iterated.

The std::sort by oid_compare orders OIDs by (volid, pageid, slotid). Deleting in that order keeps back-to-back oos_delete calls touching nearby pages so a just-loaded OOS page stays hot in the buffer pool — the same locality discipline the heap scan uses. It also sets up an unrealized TODO(perf): oos_delete re-pgbuf_fixes and unfixes the OOS page on every call, so OIDs sharing a page are fixed once per OID rather than once per page; the sort makes the eventual page-grouping optimization a local change.

flowchart TD
  A["sort oos_oids by volid,pageid,slotid"] --> B["log_sysop_start"]
  B --> C{"more OIDs?"}
  C -- no --> H{"error_code == NO_ERROR?"}
  C -- yes --> D["oos_chunk_exists"]
  D --> E{"error?"}
  E -- yes --> F["break"]
  E -- no --> G{"exists?"}
  G -- no --> C2["continue, already gone"]
  C2 --> C
  G -- yes --> I["oos_delete"]
  I --> J{"error?"}
  J -- yes --> F
  J -- no --> C
  F --> H
  H -- yes --> K["log_sysop_commit"]
  H -- no --> L["log_sysop_abort"]
  K --> M["return error_code"]
  L --> M

Figure 10-3: Branch-complete control flow of vacuum_forward_walk_oos_delete_atomic — the self-owned sysop, the two break-to-abort failure exits, and the !exists idempotent-skip continue.

Invariant — all deletes commit in exactly one self-owned sysop. This helper runs with no enclosing sysop, so it opens its own with log_sysop_start and closes it with log_sysop_commit on full success or log_sysop_abort on any error. The whole multi-chunk delete is therefore all-or-nothing for crash recovery — the deliberate counterpart to the REMOVE path (10.4): same rule, “OOS deletes happen in exactly one sysop,” but a different nesting level, so one helper must start a sysop and the other must not.

The loop is built for idempotent retry. A vacuum block can be retried; an earlier forward-walk in the same block may have already committed its deletes, so a given OID’s chunk can already be gone. The oos_chunk_exists probe handles exactly this:

// vacuum_forward_walk_oos_delete_atomic -- src/query/vacuum_oos.cpp  (loop body)
bool exists;
error_code = oos_chunk_exists (thread_p, oid, &exists);
if (error_code != NO_ERROR)
  {
    break;      /* real failure (I/O, interrupt): abort the sysop */
  }
if (!exists)
  {
    continue;      /* already deleted on a prior block attempt: skip, not fail */
  }
error_code = oos_delete (thread_p, *oos_vfid, oid);
if (error_code != NO_ERROR)
  {
    break;      /* real delete failure: abort the sysop */
  }

The three loop exits are distinct: a oos_chunk_exists error and a oos_delete error both break to the abort, while !exists continues — making double-deletion a no-op rather than an error. Without oos_chunk_exists, a block retry would send oos_delete after an already-freed slot and surface a spurious failure the upper layer would log as a leak it does not actually have.

10.4 The REMOVE path: deleting within the caller’s sysop

vacuum_heap_oos_delete_within_sysop is the simpler, in-place path. Its only caller, vacuum_heap_record, is physically vacuuming a dead heap slot whose record body still carries OOS references; that record is in front of us, so there is no undo reconstruction and no VFID lookup (the caller already resolved the OOS file, possibly via vacuum_oos_find_vfid_for_heap_record).

// vacuum_heap_oos_delete_within_sysop -- src/query/vacuum_oos.cpp
int
vacuum_heap_oos_delete_within_sysop (THREAD_ENTRY *thread_p, const VFID *oos_vfid, const RECDES *record)
{
  assert (!VFID_ISNULL (oos_vfid));
  std::vector<OID> oos_oids;
  int error_code = heap_recdes_get_oos_oids (record, oos_oids);
  if (error_code != NO_ERROR)
    {
      assert_release (false);    /* malformed OOS reference list */
      return error_code;
    }
  for (const OID &oos_oid : oos_oids)
    {
      error_code = oos_delete (thread_p, *oos_vfid, oos_oid);
      if (error_code != NO_ERROR)
  {
    vacuum_er_log_error (VACUUM_ER_LOG_HEAP, "Failed to delete OOS record %d|%d|%d.",
             oos_oid.volid, oos_oid.pageid, oos_oid.slotid);
    return error_code;    /* propagate to caller's sysop */
  }
    }
  return NO_ERROR;
}

Invariant — runs inside the caller’s sysop; must NOT open its own. The header is explicit: the caller (vacuum_heap_record) has already opened a sysop, and the heap-slot vacuum record and these OOS deletes must commit in the same sysop. If they split, a crash between them could leave the heap slot vacuumed while its OOS chunks still look referenced, or the reverse — a torn state recovery cannot repair. So unlike the forward-walk helper, this function starts no sysop and propagates errors: the caller’s sysop will abort, rolling the heap-slot vacuum back too, and the two stay consistent.

The contrast in error handling between the two delete primitives follows from the nesting. The forward-walk path swallows errors into a bounded leak because there is no enclosing transaction to roll back into — it is opportunistic cleanup of pre-images. The REMOVE path returns the error because it is part of an atomic heap-record vacuum; failing loudly is correct there.

Aspect	vacuum_heap_oos_delete_within_sysop (REMOVE)	vacuum_forward_walk_oos_delete_atomic (forward-walk)
Sysop	Uses caller’s existing sysop; opens none	Opens and commits/aborts its own
OID list	Borrowed const RECDES *record, parsed once	Owned std::vector<OID> by value
Sort	None (small, same-record list)	Sorts by (volid, pageid, slotid) for locality
Idempotence	None — record is deleted exactly once	oos_chunk_exists guard for block-retry safety
On error	Propagates so caller’s sysop aborts	Logs bounded leak, er_clear, never propagates

Both still carry the same TODO(perf) about oos_delete re-fixing the page per OID.

10.5 The first-touch VFID resolver and its temporary abort

vacuum_oos_find_vfid_for_heap_record is the lazy resolver used by the heap-vacuum loop (not the forward walk, which uses the memo). It is called per record with the heap’s hfid, the record, its slotid, and record_type (the last two for logging only); it short-circuits when the OOS file is already known or the record has no OOS:

// vacuum_oos_find_vfid_for_heap_record -- src/query/vacuum_oos.cpp
if (!VFID_ISNULL (oos_vfid) || !heap_recdes_contains_oos (record))
  {
    return NO_ERROR;      /* already resolved, or no OOS on this record */
  }
if (heap_oos_find_vfid (thread_p, hfid, oos_vfid, false))
  {
    return NO_ERROR;      /* resolved successfully */
  }

If both fall through, the record claims OOS but its heap has no OOS file. The comment establishes this is a real invariant violation, not a not-yet-created file: the OOS file is always created (docreate=true) before the flag is committed, so a missing file means a wrongly-set flag, a dropped file, or odd recovery ordering. Returning ER_FAILED here would restart a release-build spin in the vacuum_heap_page loop (it clears the error and retries the same page without advancing — see commit 1bf7dda05), so the function logs a diagnostic dump (repid_and_flags, mvcc_flags, offset_size) and is designed to degrade to a bounded leak.

But like the forward-walk NONE branch, it currently force-crashes:

// vacuum_oos_find_vfid_for_heap_record -- src/query/vacuum_oos.cpp  (tail)
/* TODO(do not review, remove before develop merge): force a hard crash in CI. ... */
abort ();        /* TEMPORARY: surface the bug in release CI */
assert_release (false);      /* debug: catch the bad flag the first time */
er_clear ();
VFID_SET_NULL (oos_vfid);
return NO_ERROR;

Caution — dead code below a live abort(). The abort() makes assert_release(false), er_clear(), VFID_SET_NULL, and return NO_ERROR unreachable today. The intended steady-state behavior is exactly that code: in debug builds assert_release catches the bad flag; in release builds it only logs (then er_clear() wipes it), oos_vfid is nulled so the caller skips OOS cleanup, and NO_ERROR lets vacuum advance. A reader preparing this for develop must remove the abort() (and the <cstdlib> include if the forward-walk one is also removed) so the bounded-leak path becomes live. This is the second of the two TODO(remove before develop merge) aborts in the file; both share the same rationale (release CI runs NDEBUG, so asserts are silent) and both must go before merge.

10.6 Chapter summary — key takeaways

1. Two reclamation entry points exist because there are two ways a dead OOS becomes visible to vacuum: the REMOVE path (vacuum_heap_oos_delete_within_sysop) handles a dead heap record still in front of us; the forward-walk path (vacuum_forward_walk_reclaim_oos) handles an old version that survives only as an undo pre-image. The MVCC theory is in cubrid-oos.md; the surrounding worker loop is in cubrid-vacuum-detail.md (Chapters 8–9).
2. The sysop nesting is the central correctness distinction. The REMOVE primitive runs inside the caller’s existing sysop and propagates errors so the heap-slot vacuum and OOS deletes commit or abort together; the forward-walk primitive opens its own sysop and swallows errors into a bounded leak. Same rule, opposite nesting.
3. VACUUM_OOS_VFID_MEMO is a deliberately single-slot, per-worker-per-block, stack-resident cache with a FOUND/NONE/ERROR tri-state. NONE (oos_vfid == VFID_NULL) is a cached negative answer; ERROR is never cached, so a transient page-read hiccup cannot poison the rest of the block. It must never become a static (would race across workers).
4. The forward-walk path makes two defensive copies against log-page rotation: the pre-image is memcpyed into an aligned db_private_alloc buffer before any page is fixed, and the OID vector is taken by value so it does not alias the log buffer. A live observation (flags byte 0x69 to 0x00 across a lookup) motivated this; the alignment copy also avoids an OR_BUF debug assert.
5. Defensive discipline favors a bounded, logged leak over failing the block. Every error path in the forward walk logs and er_clear()s rather than propagating, because failing a vacuum block trips a shutdown-only assert and can wedge vacuum. oos_chunk_exists makes the atomic delete idempotent so a block retry skips already-freed chunks instead of erroring.
6. Two live abort() calls are temporary CI crash-bait (the vacuum_forward_walk_reclaim_oos NONE branch and the tail of vacuum_oos_find_vfid_for_heap_record), paired with the temporary #include <cstdlib> and TODO(remove before develop merge) markers. They exist because release CI runs NDEBUG where assert_release is silent. Both aborts, and the steady-state bounded-leak code they currently shadow, must be reconciled before the develop merge.
7. An unrealized TODO(perf) recurs in both delete primitives: oos_delete re-pgbuf_fixes the OOS page per OID. The forward-walk sort by (volid, pageid, slotid) already orders OIDs into page groups, making a future single-fix-per-page optimization a localized change.

Chapter 11: Stats Diagnostics and Edge Paths

The first ten chapters traced the per-value lifecycle. This chapter covers everything that surrounds it — the read-only diagnostics a maintainer queries, the standalone logger that is deliberately not WAL, the debug-only VOT auditor that must never be deleted, the per-transaction OID collector the read path leans on, and the in-flux edges where the module is knowingly incomplete. None mutate a live OOS value, but every one is a tripwire: edit the lifecycle code without knowing they exist and you break the QA harness, silence a divergence assert, or expand an OOS unnecessarily. For the “what is OOS” framing, see the companion cubrid-oos.md (Two flags, The inline stub, Client stats); this chapter does not re-derive it.

11.1 The stats surface: two structs, one round trip

A maintainer verifying OOS runs a session command that reads counters off the OOS file: physical page count, live record count, and the sum of live on-page record bytes. Two structs carry those counters — server-side (oos_stats_info) and client-side (db_oos_stats) — and the network layer copies field-for-field between them.

oos_stats_info — src/storage/oos_file.hpp, the server-side collector’s output.

Field	Role	Why it exists
has_oos_file	0 if the class has no OOS file, 1 otherwise	Distinguishes “OOS file exists but empty” (num_recs == 0) from “class never created one”. A demotion-free class never allocates a VFID.
oos_vfid	The OOS file’s VFID{volid, fileid}	Lets the caller cross-check which physical file was measured; VFID_SET_NULL when no file.
num_user_pages	Physical user pages allocated to the file	Comes from file_get_num_user_pages, the allocation map — not from walking records.
page_size	DB_PAGESIZE	Echoed so the client can compute occupancy without a second round trip.
num_recs	Live OOS record count (int)	The struct comment phrases this as “tracked by OOS_HDR_STATS”, but the collector recomputes it by walking slots (see §11.3) rather than trusting the lazy header estimate.
recs_sumlen	Sum of live on-page record lengths (INT64)	INT64 because the sum can exceed 2 GB even when each record’s RECDES.length is a 4-byte int. Note the implementation accumulates slot_recdes.length — the whole on-page slot, header included — not just the user-payload body (see the §11.3 cross-check).

db_oos_stats — src/compat/db_oos.h, the C client mirror. It carries the same six counters re-stamped from the reply, with two differences driven by the client/server boundary: the server VFID is flattened into two plain ints (oos_vfid_volid / oos_vfid_fileid) so the client header need not depend on the server VFID type, and recs_sumlen is int64_t (matching the server INT64) — the only non-int reply field. has_oos_file, num_user_pages, page_size, and num_recs are verbatim echoes.

flowchart TD
  A["db_get_oos_stats class_name -- db_admin.c"] --> B["db_find_class + ws_identifier -> class_oid"]
  B --> C["oos_get_stats_by_class_oid class_oid DB_OOS_STATS -- network_interface_cl.c"]
  C -->|CS_MODE| D["or_pack_oid -> NET_SERVER_OOS_STATS -> unpack 7 ints + 1 int64"]
  C -->|SA_MODE| E["enter_server -> xoos_get_stats_by_class_oid OOS_STATS_INFO"]
  D --> F["xoos_get_stats_by_class_oid on server"]
  E --> F
  F --> G["oos_get_stats_by_vfid -- walks every page/slot"]

Figure 11-1. The stats round trip. The client API is a thin marshaller; all counting happens server-side in oos_get_stats_by_vfid.

The client entry db_get_oos_stats resolves the class name and delegates. The stats surface is a file-level diagnostic that never touches a heap record.

// db_get_oos_stats -- src/compat/db_admin.c
if (class_name == NULL || stats == NULL)
  return ER_OBJ_INVALID_ARGUMENTS;                 /* <- defensive null-arg guard */
class_op = db_find_class (class_name);
if (class_op == NULL) { ASSERT_ERROR (); return er_errid (); }   /* <- class not found */
class_oid = ws_identifier (class_op);
if (class_oid == NULL || OID_ISNULL (class_oid))
  { er_set (..., ER_LC_UNKNOWN_CLASSNAME, 1, class_name); return ER_LC_UNKNOWN_CLASSNAME; }
return oos_get_stats_by_class_oid (class_oid, stats);   /* <- client stub, marshals to server */

The client stub oos_get_stats_by_class_oid (in network_interface_cl.c) has two compile-time arms. Under CS_MODE it packs the OID, fires NET_SERVER_OOS_STATS, then unpacks a leading status int followed by six more ints plus one int64 (has_oos_file, volid, fileid, num_user_pages, page_size, num_recs, then recs_sumlen) into the db_oos_stats; the volid/fileid ints are restamped into oos_vfid_volid/oos_vfid_fileid. Under the non-CS_MODE arm (standalone) it calls xoos_get_stats_by_class_oid directly and copies the OOS_STATS_INFO fields across, narrowing oos_vfid.volid/oos_vfid.fileid to int. The reply buffer is sized OR_INT_SIZE * 7 + OR_BIGINT_SIZE — if you add a field to oos_stats_info, you must widen both the reply buffer and the unpack sequence or the client silently reads garbage.

11.2 xoos_get_stats_by_class_oid — every branch

This is the server entry. It resolves a class OID down to an OOS VFID, then hands off to the per-file walker. Its job is entirely about distinguishing the several legitimate “no OOS file” outcomes from real errors.

// xoos_get_stats_by_class_oid -- src/storage/oos_file.cpp
if (class_oid == NULL || out == NULL || OID_ISNULL (class_oid))
  return ER_OBJ_INVALID_ARGUMENTS;                 /* <- branch 1: bad args */
memset (out, 0, sizeof (*out));
out->page_size = DB_PAGESIZE;
VFID_SET_NULL (&out->oos_vfid);
HFID hfid; HFID_SET_NULL (&hfid);
if (heap_get_class_info (thread_p, class_oid, &hfid, NULL, NULL) != NO_ERROR)
  { ASSERT_ERROR (); return er_errid (); }          /* <- 2: catalog read error */
if (HFID_IS_NULL (&hfid))
  return NO_ERROR;                                 /* <- 3: empty class, has_oos_file stays 0 */
VFID oos_vfid; VFID_SET_NULL (&oos_vfid);
if (!heap_oos_find_vfid (thread_p, &hfid, &oos_vfid, false)) {
  int errid = er_errid ();
  if (errid != NO_ERROR) return errid;             /* <- 4a: real read error */
  return NO_ERROR;                                 /* <- 4b: legit no-OOS (docreate=false) */
}
if (VFID_ISNULL (&oos_vfid)) return NO_ERROR;      /* <- 5: found-but-null */
return oos_get_stats_by_vfid (thread_p, oos_vfid, out);   /* <- 6: hand off */

The non-obvious branch is 4a/4b: heap_oos_find_vfid returns false for two distinct reasons — a genuine read error (which sets er_errid) and the perfectly normal case of a class whose heap header simply has no OOS VFID (docreate=false means “do not create one if absent”). The code reads er_errid() to disambiguate. Get this wrong and either real I/O errors are swallowed (4a treated as 4b) or every demotion-free class returns a spurious error (4b treated as 4a).

Invariant: has_oos_file == 0 ⟺ no VFID was measured. Every early-return path (branches 3, 4b, 5) leaves out at its memset-zeroed state with only page_size set, so has_oos_file stays 0. Only oos_get_stats_by_vfid (branch 6) sets it to 1. A client that sees has_oos_file == 1 but num_recs == 0 is reading a real, currently-empty file; a client that sees has_oos_file == 0 knows no file was even located. If a future edit set has_oos_file = 1 before branch 6, this distinction collapses.

11.3 oos_get_stats_by_vfid — the slot walk and its two corrections

This is the actual counter. It is exposed separately (not static) precisely so unit-test heaps that have a VFID but no catalogued class can be measured without going through heap_get_class_info.

flowchart TD
  A["out==NULL or VFID null?"] -->|yes| B["return ER_OBJ_INVALID_ARGUMENTS"]
  A -->|no| C["memset out, set has_oos_file=1, page_size, oos_vfid"]
  C --> D["file_get_num_user_pages -> num_user_pages"]
  D -->|err| E["ASSERT_ERROR; return er_errid"]
  D --> F["file_get_sticky_first_page -> hdr_vpid"]
  F -->|err| E
  F --> G["loop i in 0..num_user_pages"]
  G --> H["file_numerable_find_nth -> scan_vpid"]
  H -->|err or null vpid| I["er_clear; continue"]
  H --> J["scan_vpid == hdr_vpid?"]
  J -->|yes| K["continue -- skip header page"]
  J -->|no| L["pgbuf_fix CONDITIONAL"]
  L -->|NULL busy| M["er_clear; continue -- accept undercount"]
  L --> N["spage_next_record loop: total_recs++, total_sumlen += slot length"]
  N --> O["unfix page"]
  O --> G
  G -->|done| P["out->num_recs, out->recs_sumlen; return NO_ERROR"]

Figure 11-2. oos_get_stats_by_vfid branch map. Every skip path (continue) is a deliberate best-effort tolerance, never a hard error.

Four branches deserve a maintainer’s attention, all on the tolerance side:

// oos_get_stats_by_vfid -- src/storage/oos_file.cpp
if (file_numerable_find_nth (thread_p, &oos_vfid, i, false, NULL, NULL, &scan_vpid) != NO_ERROR
    || VPID_ISNULL (&scan_vpid))
  { er_clear (); continue; }                         /* <- hole in numerable map: skip */
if (!VPID_ISNULL (&hdr_vpid) && VPID_EQ (&scan_vpid, &hdr_vpid))
  continue;                                          /* <- header page has no user records */
page_ptr = pgbuf_fix (..., PGBUF_LATCH_READ, PGBUF_CONDITIONAL_LATCH);
if (page_ptr == NULL) { er_clear (); continue; }     /* <- page busy: accept slight undercount */
PGSLOTID slotid = -1; RECDES slot_recdes;
while (spage_next_record (page_ptr, &slotid, &slot_recdes, PEEK) == S_SUCCESS)
  { total_recs++; total_sumlen += slot_recdes.length; }   /* <- counts slot 0 too */

The slotid = -1 seed encodes the two corrections that distinguish this walker from a naive spage_collect_statistics call:

1. spage_collect_statistics skips slot 0 (assuming a heap page where slot 0 holds the header record). OOS data pages keep real records starting at slot 0, so the canned helper undercounts by one per page. Seeding from slotid = -1 makes spage_next_record visit slot 0.
2. The bestspace hint is not used. OOS_HDR_STATS.estimates is a lazy best-space cache updated only by the sync scan (Chapter 5), wrong for a live count. The walker reads physical slots — slower but accurate. Accumulators are INT64; out->num_recs narrows to int, while recs_sumlen keeps full 64-bit width.

Cross-check: recs_sumlen is the sum of full on-page slot lengths, not user-payload bytes. The struct comment calls it “sum of live OOS record body bytes”, but the loop accumulates slot_recdes.length, which is the entire on-page record including the OOS_RECORD_HEADER prepended by oos_prepend_header (Chapters 4/6). The reported sum therefore overstates the raw user payload by OOS_RECORD_HEADER_SIZE per chunk. Treat recs_sumlen as a storage-occupancy figure, not as Σ total_data_length.

Invariant: stats are a lower bound under contention, never an over-count. The PGBUF_CONDITIONAL_LATCH plus continue-on-busy means a page being written concurrently is simply omitted. num_recs and recs_sumlen can therefore read slightly low on a busy file but never high or torn. A maintainer must not present these as exact under concurrent load; they are a diagnostic, not an accounting source of truth.

11.4 oos_get_length — test-only header reader vs. the production stub

oos_get_length answers “how long is the full payload behind this OID?” by fixing the head chunk’s page and reading total_data_length out of its OOS_RECORD_HEADER. It exists for tests and assertions; production code does not call it on the hot path — production reads the same length from the inline stub embedded in the heap record (companion The inline stub), avoiding a page fix entirely.

// oos_get_length -- src/storage/oos_file.cpp
PAGE_PTR page_ptr = pgbuf_fix (thread_p, &vpid, OLD_PAGE, PGBUF_LATCH_READ, PGBUF_UNCONDITIONAL_LATCH);
if (page_ptr == nullptr)
  { oos_error (...); assert_release_error (er_errid () != NO_ERROR); assert (false); return -1; } /* <- fix fail */
auto page_unfixer = make_scope_exit ([&]() { pgbuf_unfix_and_init_after_check (thread_p, page_ptr); });
SCAN_CODE code = spage_get_record (thread_p, page_ptr, slotid, &oos_recdes, PEEK);
if (code != S_SUCCESS)
  { oos_error (...); er_set (..., ER_GENERIC_ERROR, 0); return -1; }   /* <- slot gone */
assert (oos_recdes.length >= OOS_RECORD_HEADER_SIZE);
OOS_RECORD_HEADER header;
std::memcpy (&header, oos_recdes.data, sizeof (OOS_RECORD_HEADER));
return header.total_data_length;   /* <- length stored in the head chunk header (Ch 4/6) */

Three branches: fix failure (return -1, asserts an error is set), record-fetch failure (return -1 after er_set), and success. The make_scope_exit guarantees the page is unfixed on every return after the fix succeeds — including the spage_get_record failure path, which would otherwise leak a latch. total_data_length is the chain-wide payload length stamped into every chunk header (Chapters 4 and 6), so reading it from the head chunk is sufficient.

11.5 oos_push_oos_oid — the per-transaction OID collector

The read path records the OOS OIDs it dereferences so a later phase can find them. That place is a plain std::vector<OID> on the thread entry, and oos_push_oos_oid is its one-line append:

// oos_push_oos_oid -- src/storage/oos_file.cpp
void
oos_push_oos_oid (THREAD_ENTRY *thread_p, const OID *oid)
{
  thread_p->oos_oids.push_back (*oid);
}

The companion oos_oid_in_vector (oos_util.cpp) is its read-side counterpart: “have we already collected this OID?” via an OID_EQ sweep, deliberately linear because the vector is small by design (one transaction’s freshly-touched OOS columns, not a whole table). Its source comment states the rationale outright: “linear scan; vector is small by design”.

// oos_oid_in_vector -- src/storage/oos_util.cpp
for (const OID &candidate : oids)
  { if (OID_EQ (&candidate, oid)) return true; }
return false;

11.6 oos_log_internal — diagnostics that are deliberately not WAL

oos_log.hpp is a self-contained logging shim writing to $CUBRID/log/oos.log. It is the single most important “this is not what you think” file in the module: it is not write-ahead logging. WAL goes through the recovery manager (companion WAL and logging, and Chapter 8’s physical-delete recovery). oos_log is printf-to-a-file for humans and the HA QA harness. Confusing the two is the classic OOS onboarding mistake.

// oos_log_internal -- src/storage/oos_log.hpp
if (static_cast<int> (level) < static_cast<int> (oos_log_get_level()))
  return;                                  /* <- level filter: below threshold, drop */
// ... format "[HH:MM:SS] OOS [LEVEL](func:line): " header + vsnprintf body ...
FILE *logfp = oos_log_get_file();          /* <- magic-static fopen("a") of $CUBRID/log/oos.log */
bool file_written = false;
if (logfp != nullptr) { fputs header; fputs body; fputc '\n'; fflush; file_written = true; }
static const bool stderr_enabled = (std::getenv ("CUBRID_OOS_LOG_STDERR") != nullptr); /* <- presence-only */
const bool force_stderr_fallback =
    !file_written && static_cast<int>(level) >= static_cast<int>(OosLogLevel::WARN);  /* <- never drop WARN/ERROR */
if (stderr_enabled || force_stderr_fallback) { fputs ...stderr...; fflush(stderr); }

Four behaviours a maintainer must internalize:

• Level filter first. Messages below oos_current_level (default DEBUG, runtime-settable via oos_log_set_level) return before any strftime/vsnprintf cost. Levels run TRACE < DEBUG < INFO < WARN < ERROR < FATAL (enum 0..5).
• The file sink is a C++11 magic static. oos_log_get_file does one fopen(path, "a") on first call (path from envvar_logdir_file(..., "oos.log")), thread-safe by the §6.7/4 guarantee with no explicit mutex. If the dir is unwritable, fopen returns nullptr and file_written stays false.
• stderr is opt-in and presence-only. CUBRID_OOS_LOG_STDERR enables stderr regardless of value (even "0" or ""); to disable, unset it. Default-off because writing to an inherited tty corrupts a raw-mode csql/readline session or cgdb UI. The check is a static const bool, read once per process.
• WARN/ERROR are never silently dropped. The force_stderr_fallback branch routes WARN+ to stderr when the file sink failed (!file_written).

Invariant: oos_error and oos_warn are release-active; oos_trace/oos_debug/oos_info compile to nothing under NDEBUG. The three low levels are gated behind #if !defined(NDEBUG), expanding to do {} while (0) in release; oos_error/oos_warn are defined unconditionally. A release-build maintainer sees no TRACE/DEBUG/INFO output at any runtime level — and any side effect inside an oos_debug(...) argument vanishes in release. Never put a load-bearing expression inside a debug-level log call.

Because none of this participates in recovery, an oos.log line is not replayed, not flushed at commit, and not guaranteed across a crash. It is purely observational.

11.7 heap_recdes_compute_oos_flag_debug — the VOT auditor that must not be deleted

This debug-only function (oos_util.cpp, guarded by #if !defined(NDEBUG)) walks a heap record’s variable-offset table (VOT) and recomputes whether the record holds an OOS column, so an assert can compare it against the OR_MVCC_FLAG_HAS_OOS bit the upstream builder stamped. Its sole caller is itself inside #if !defined(NDEBUG) (heap_update_adjust_recdes_header, heap_file.c), so release builds carry no reference to the symbol — which is why both the .cpp definition and the .hpp declaration carry a blunt DO NOT REMOVE THIS: dead-code sweeps will flag it.

// heap_recdes_compute_oos_flag_debug -- src/storage/oos_util.cpp
if (recdes == NULL || recdes->data == NULL) return false;          /* <- branch: nothing to walk */
const int offset_size = OR_GET_OFFSET_SIZE (recdes->data);
const int header_size = OR_HEADER_SIZE (recdes->data);
const int max_var_count = (recdes->length - header_size) / offset_size;
if (max_var_count > 0) {                                            /* <- first-entry sanity gate */
  int first = /* OR_GET_BYTE/SHORT/INT per offset_size */;
  int clean = first & ~OR_VAR_FLAG_MASK;
  if (clean < 0 || clean > recdes->length - header_size) return false; /* <- not a real VOT */
}
for (int index = 0; index < max_var_count; ++index) {
  int offset = /* read per offset_size; default: assert(false); return false */;
  if (OR_IS_OOS (offset)) has_oos = true;
  if (OR_IS_LAST_ELEMENT (offset)) return has_oos;   /* <- modern VOT: trust result */
}
return false;   /* <- no LAST_ELEMENT sentinel: legacy VOT, refuse to guess */

The branch structure is its whole point. A heap can hold non-object-instance records (class records, root records) whose bytes, read as a VOT, are arbitrary. The first-entry sanity gate validates that index 0’s masked offset falls in [0, recdes->length - header_size] (offsets are end-of-header-relative); outside that range it returns false rather than walk garbage. (The doc-comment loosely says [0, recdes->length], but the code subtracts header_size — the correct bound.) Only index 0 needs this; once it validates, the loop self-terminates on OR_IS_LAST_ELEMENT, with max_var_count as a backstop.

Invariant: the debug auditor and the live builder must agree on HAS_OOS. The caller asserts walked_has_oos == has_oos (guarded by classrepr->n_variable > 0). If a future recdes-producing path forgets to set or clear OR_MVCC_FLAG_HAS_OOS, this assert fires in debug builds. The production path trusts the bit blindly; this walker is the only thing that catches a stale bit before it corrupts a SELECT’s scancache (companion Two flags). Delete it and divergence goes undetected until a user sees wrong data.

11.8 Legacy-record handling and the max_var_count over-count

Both VOT walkers — the debug heap_recdes_compute_oos_flag_debug above and the production heap_recdes_get_oos_oids (heap_file.c) — share a documented limitation: records lacking the OR_VAR_BIT_LAST_ELEMENT sentinel are not fully supported. That flag is 0x2 in the variable-length flag bits (OR_VAR_FLAG_MASK = 0x3, sharing the field with OR_VAR_BIT_OOS = 0x1).

// heap_recdes_get_oos_oids -- src/storage/heap_file.c
/* NOTE: This upper bound may include VOT alignment padding and fixed-attribute bytes
 * for legacy records that lack the OR_VAR_BIT_LAST_ELEMENT flag. ... */
const int max_var_count = (recdes->length - OR_HEADER_SIZE (recdes->data)) / offset_size;
for (int index = 0; index <= max_var_count; ++index) {
  if (index == max_var_count) { assert_release (false && "LAST_ELEMENT flag not found ..."); return ER_FAILED; }
  // ... read offset; if OR_IS_OOS extract OID (bounds-checked); if OR_IS_LAST_ELEMENT break ...
}

Two concrete hazards for a maintainer:

• max_var_count is an over-count. It is (length - header) / offset_size, which on a legacy VOT includes alignment padding and fixed-attribute bytes — not just real variable offsets. On a modern record the loop never reaches it because OR_IS_LAST_ELEMENT breaks first. The production walker treats reaching max_var_count as a hard ER_FAILED via assert_release; the debug walker treats it as a quiet return false. The loop bounds differ accordingly: production iterates index <= max_var_count (to hit the sentinel-miss guard), the debug walker index < max_var_count.
• Old odd offsets false-positive OR_IS_OOS. OR_IS_OOS(offset) is OR_GET_VAR_FLAG(offset) & OR_VAR_BIT_OOS — effectively offset & 0x1. On a legacy VOT predating the flag scheme, an odd offset looks like an OOS column. The debug auditor’s final return false (no LAST_ELEMENT seen) refuses to interpret a legacy VOT rather than emit a false OOS OID; the production walker additionally bounds-checks each extracted OID against recdes->length.

This is why the companion’s Legacy-record handling note calls old-format VOTs “not fully supported yet”: the LAST_ELEMENT addition is forward-only, and mixed old/new records on one page are an open correctness gap.

11.9 In-flux edges a maintainer must know before touching the module

The knowingly incomplete surfaces — none a bug to fix in passing; each a tracked milestone (companion Milestone status, Not yet implemented / in flux).

Edge	Where	What it means for you
oos_remove_page not wired	oos_file.cpp, // TODO: will be called by vacuum when OOS vacuum is implemented	Vacuum (Chapter 10) frees slots, not pages. Empty data pages stay allocated; num_user_pages does not shrink. Do not assume page reclamation exists.
RECDES.length ~2 GB cap	RECDES.length is a 4-byte int; oos_insert rejects src.size() > INT_MAX up front	A single expanded record cannot exceed ~2 GB. oos_insert guards the narrowing src_len cast against src.size() > (std::size_t) INT_MAX before doing anything.
No payload compression in M1	(absent — TOAST has pglz/lz4)	OOS stores raw bytes. Compression is an open PR (CBRD-26881 / #7258). recs_sumlen is raw, uncompressed bytes.
Expand-OOS over-expansion audit	heap_get_visible_version_expand_oos, CBRD-26847	Many call sites were migrated mechanically and may expand OOS unnecessarily (re-inlining when not needed — Chapter 7). An audit is pending; do not trust that every expand site is minimal.
CDC / flashback cannot resolve an OOS OID	(epic rough-edge list)	Change-data-capture and flashback see the inline stub’s OID but cannot dereference the out-of-row payload. Any CDC-adjacent work must treat OOS columns as unresolvable for now.

The oos_remove_page body itself is trivial and correct — it just calls file_dealloc with FILE_OOS and logs via oos_error on failure — but it has no release caller. That, plus the unused-but-retained debug auditor of §11.7 and the never-WAL logger of §11.6, form the trio of “exists, compiles, looks live, but is off the production hot path” symbols. Knowing which is which is the difference between a safe edit and a silent regression.

11.10 Chapter summary — key takeaways

1. The stats surface is a thin marshaller over one server walker: db_get_oos_stats → oos_get_stats_by_class_oid (CS / non-CS arms) → xoos_get_stats_by_class_oid → oos_get_stats_by_vfid. Adding an oos_stats_info field means widening the OR_INT_SIZE*7 + OR_BIGINT_SIZE reply buffer and the unpack sequence in lockstep.
2. has_oos_file == 0 ⟺ no VFID was measured. Every early-return branch leaves the zeroed struct; only oos_get_stats_by_vfid sets the flag. heap_oos_find_vfid’s false is overloaded (real error vs. legitimate no-OOS), disambiguated by er_errid().
3. oos_get_stats_by_vfid counts physical slots, not the lazy bestspace hint, and under-counts under contention. It walks from slotid = -1 to include slot 0 (which spage_collect_statistics skips) and continues past busy pages — never an over-count, never a torn read. recs_sumlen sums full slot lengths (OOS_RECORD_HEADER included), so it is a storage-occupancy figure, not the raw payload sum.
4. The two helpers are minimal by design. oos_get_length is test/assert-only (production reads total_data_length from the inline stub); its make_scope_exit unfixes the page on every post-fix return. oos_push_oos_oid is the per-transaction collector (thread_p->oos_oids.push_back); oos_oid_in_vector is its deliberately-linear membership check.
5. oos_log is not WAL. It is printf-to-$CUBRID/log/oos.log, level-filtered, stderr opt-in via CUBRID_OOS_LOG_STDERR. oos_error/oos_warn are release-active; oos_trace/oos_debug/oos_info compile to do {} while (0) under NDEBUG — never put load-bearing expressions inside them.
6. heap_recdes_compute_oos_flag_debug must not be deleted. It is the debug-only assert catching OR_MVCC_FLAG_HAS_OOS divergence; release builds carry no reference, so dead-code sweeps flag it. Its first-entry sanity gate and “no LAST_ELEMENT ⇒ return false” guard protect against walking non-VOT records.
7. Legacy VOTs and several in-flux edges are knowingly incomplete: max_var_count over-counts on records lacking OR_VAR_BIT_LAST_ELEMENT, odd legacy offsets false-positive OR_IS_OOS, oos_remove_page has no release caller, records cap at ~2 GB, M1 has no compression, the expand-OOS audit (CBRD-26847) is pending, and CDC/flashback cannot resolve an OOS OID.

Position hints as of this revision

The following are line numbers as observed on 2026-06-22; symbols are the canonical anchor and line numbers are hints that decay.

Symbol	File	Line
byte_span_writer	src/base/byte_span_writer.hpp	38
OR_GET_OID	src/base/object_representation.h	274
OR_VAR_BIT_OOS	src/base/object_representation.h	441
OR_VAR_BIT_LAST_ELEMENT	src/base/object_representation.h	442
OR_VAR_FLAG_MASK	src/base/object_representation.h	443
OR_SET_VAR_OOS	src/base/object_representation.h	445
OR_SET_VAR_LAST_ELEMENT	src/base/object_representation.h	446
OR_GET_VAR_FLAG	src/base/object_representation.h	448
OR_GET_VAR_OFFSET	src/base/object_representation.h	449
OR_IS_OOS	src/base/object_representation.h	451
OR_IS_LAST_ELEMENT	src/base/object_representation.h	452
OR_OOS_INLINE_SIZE	src/base/object_representation.h	455
OR_BIGINT_SIZE	src/base/object_representation_constants.h	45
OR_OID_SIZE	src/base/object_representation_constants.h	67
OR_OID_PAGEID	src/base/object_representation_constants.h	68
OR_OID_SLOTID	src/base/object_representation_constants.h	69
OR_OID_VOLID	src/base/object_representation_constants.h	70
OR_MAX_BYTE	src/base/object_representation_constants.h	130
OR_MAX_SHORT	src/base/object_representation_constants.h	134
OR_MVCC_MAX_HEADER_SIZE	src/base/object_representation_constants.h	142
OR_MVCC_INSERT_HEADER_SIZE	src/base/object_representation_constants.h	148
OR_NON_MVCC_HEADER_SIZE	src/base/object_representation_constants.h	150
OR_MVCC_FLAG_SHIFT_BITS	src/base/object_representation_constants.h	165
OR_MVCC_FLAG_HAS_OOS	src/base/object_representation_constants.h	178
subspan	src/base/span.hpp	112
oos_get_stats_by_class_oid	src/communication/network_interface_cl.c	9928
db_get_oos_stats	src/compat/db_admin.c	1581
db_oos_stats	src/compat/db_oos.h	36
VACUUM_OOS_VFID_LOOKUP_RESULT	src/query/vacuum_oos.cpp	51
VACUUM_OOS_VFID_LOOKUP_RESULT	src/query/vacuum_oos.cpp	56
vacuum_oos_vfid_lookup	src/query/vacuum_oos.cpp	85
vacuum_oos_vfid_lookup	src/query/vacuum_oos.cpp	86
heap_oos_find_vfid	src/query/vacuum_oos.cpp	116
vacuum_forward_walk_oos_delete_atomic	src/query/vacuum_oos.cpp	153
oos_chunk_exists	src/query/vacuum_oos.cpp	179
oos_delete	src/query/vacuum_oos.cpp	188
vacuum_forward_walk_reclaim_oos	src/query/vacuum_oos.cpp	222
heap_recdes_contains_oos	src/query/vacuum_oos.cpp	245
heap_recdes_get_oos_oids	src/query/vacuum_oos.cpp	286
vacuum_oos_find_vfid_for_heap_record	src/query/vacuum_oos.cpp	338
vacuum_heap_oos_delete_within_sysop	src/query/vacuum_oos.cpp	399
VACUUM_OOS_VFID_MEMO	src/query/vacuum_oos.hpp	43
vacuum_oos_vfid_memo	src/query/vacuum_oos.hpp	44
FILE_OOS (stub: file_header_dump_descriptor)	src/storage/file_manager.c	1431
FILE_OOS (type name string)	src/storage/file_manager.c	3062
FILE_OOS (vfid-uniqueness guard)	src/storage/file_manager.c	3448
FILE_OOS (stub: tracker desired_type filter)	src/storage/file_manager.c	10903
FILE_OOS (stub: tracker lock protection)	src/storage/file_manager.c	10931
FILE_OOS (stub: tracker class-OID extraction)	src/storage/file_manager.c	10975
FILE_OOS (stub: spacedb classification)	src/storage/file_manager.c	12235
FILE_OOS	src/storage/file_manager.h	53
FILE_OOS (enum)	src/storage/file_manager.h	53
HEAP_UPDATE_IS_MVCC_OP	src/storage/heap_file.c	160
heap_hdr_stats.oos_vfid	src/storage/heap_file.c	206
heap_manager_initialize	src/storage/heap_file.c	5150
oos_bestspace_initialize (call site)	src/storage/heap_file.c	5180
heap_manager_finalize	src/storage/heap_file.c	5197
heap_get_record_data_when_all_ready	src/storage/heap_file.c	7859
heap_attrvalue_read_oos_inline	src/storage/heap_file.c	10490
heap_midxkey_get_oos_extra_size	src/storage/heap_file.c	10899
heap_attrinfo_get_record_payload_size	src/storage/heap_file.c	12106
heap_attrinfo_get_record_header_size	src/storage/heap_file.c	12143
heap_attrinfo_determine_disk_layout	src/storage/heap_file.c	12183
heap_oos_find_vfid	src/storage/heap_file.c	12258
heap_oos_find_vfid	src/storage/heap_file.c	12259
heap_attrinfo_dbvalue_to_recdes	src/storage/heap_file.c	12357
heap_attrinfo_insert_to_oos	src/storage/heap_file.c	12435
heap_attrinfo_transform_header_to_disk	src/storage/heap_file.c	12573
heap_attrinfo_transform_variable_to_disk	src/storage/heap_file.c	12761
heap_attrinfo_transform_to_disk_internal	src/storage/heap_file.c	13008
heap_update_adjust_recdes_header	src/storage/heap_file.c	21508
heap_delete_relocation	src/storage/heap_file.c	22461
heap_delete_relocation (eager OOS cleanup call site)	src/storage/heap_file.c	22927
heap_delete_home	src/storage/heap_file.c	22977
heap_delete_home (eager OOS cleanup call site)	src/storage/heap_file.c	23273
heap_update_relocation	src/storage/heap_file.c	23629
heap_update_relocation (eager OOS cleanup call site)	src/storage/heap_file.c	23707
heap_update_relocation (mvcc remove_old_forward tag)	src/storage/heap_file.c	23868
heap_update_home	src/storage/heap_file.c	23985
heap_update_home (eager OOS cleanup call site)	src/storage/heap_file.c	24192
heap_get_visible_version_expand_oos	src/storage/heap_file.c	26450
heap_recdes_contains_oos	src/storage/heap_file.c	27772
heap_recdes_get_oos_oids	src/storage/heap_file.c	27779
expand_oos	src/storage/heap_file.h	384
HEAP_OOS_EXPAND_STATE	src/storage/heap_oos.cpp	48
heap_oos_parse_vot	src/storage/heap_oos.cpp	68
heap_oos_parse_vot	src/storage/heap_oos.cpp	69
heap_oos_read_blobs	src/storage/heap_oos.cpp	128
heap_oos_read_blobs	src/storage/heap_oos.cpp	129
heap_oos_compute_layout	src/storage/heap_oos.cpp	183
heap_oos_compute_layout	src/storage/heap_oos.cpp	184
heap_oos_build_record	src/storage/heap_oos.cpp	241
heap_oos_build_record	src/storage/heap_oos.cpp	242
heap_record_replace_oos_oids	src/storage/heap_oos.cpp	337
heap_record_replace_oos_oids	src/storage/heap_oos.cpp	338
heap_oos_delete_unreferenced	src/storage/heap_oos.cpp	424
oid_Null_oid	src/storage/oid.h	209
OOS_ALIGNMENT	src/storage/oos_file.cpp	84
OOS_BESTSPACE_CACHE_CAPACITY	src/storage/oos_file.cpp	86
OOS_DROP_FREE_SPACE	src/storage/oos_file.cpp	87
OOS_BESTSPACE_SYNC_THRESHOLD	src/storage/oos_file.cpp	88
BEST_PAGE_SEARCH_MAX_COUNT	src/storage/oos_file.cpp	89
oos_stats_bestspace_cache	src/storage/oos_file.cpp	104
oos_Bestspace_cache_area	src/storage/oos_file.cpp	114
oos_Bestspace	src/storage/oos_file.cpp	116
oos_Find_best_page_limit	src/storage/oos_file.cpp	118
oos_hash_vpid	src/storage/oos_file.cpp	125
oos_compare_vpid	src/storage/oos_file.cpp	132
oos_hash_vfid	src/storage/oos_file.cpp	140
oos_compare_vfid	src/storage/oos_file.cpp	147
oos_stats_entry_free	src/storage/oos_file.cpp	159
oos_bestspace_initialize	src/storage/oos_file.cpp	184
oos_bestspace_finalize	src/storage/oos_file.cpp	228
oos_stats_add_bestspace	src/storage/oos_file.cpp	271
oos_stats_del_bestspace_by_vpid	src/storage/oos_file.cpp	337
oos_stats_del_bestspace_by_vfid	src/storage/oos_file.cpp	363
oos_get_header_stats_ptr	src/storage/oos_file.cpp	395
oos_stats_put_second_best	src/storage/oos_file.cpp	411
oos_stats_get_second_best	src/storage/oos_file.cpp	433
oos_stats_find_page_in_bestspace	src/storage/oos_file.cpp	454
oos_stats_sync_bestspace	src/storage/oos_file.cpp	634
oos_stats_update_internal	src/storage/oos_file.cpp	808
oos_stats_update	src/storage/oos_file.cpp	853
oos_create_file	src/storage/oos_file.cpp	909
oos_remove_file	src/storage/oos_file.cpp	995
oos_remove_file	src/storage/oos_file.cpp	996
oos_remove_page	src/storage/oos_file.cpp	1007
oos_remove_page	src/storage/oos_file.cpp	1008
oos_prepend_header	src/storage/oos_file.cpp	1022
oos_insert	src/storage/oos_file.cpp	1048
oos_insert_across_pages	src/storage/oos_file.cpp	1109
oos_insert_within_page	src/storage/oos_file.cpp	1205
oos_read_within_page	src/storage/oos_file.cpp	1269
oos_read_across_pages	src/storage/oos_file.cpp	1325
oos_read	src/storage/oos_file.cpp	1372
oos_file_alloc_new	src/storage/oos_file.cpp	1431
oos_file_alloc_new	src/storage/oos_file.cpp	1432
oos_find_best_page	src/storage/oos_file.cpp	1460
oos_vpid_init_new	src/storage/oos_file.cpp	1622
oos_log_insert_physical	src/storage/oos_file.cpp	1649
oos_log_delete_physical	src/storage/oos_file.cpp	1669
oos_delete_chain	src/storage/oos_file.cpp	1696
oos_chunk_exists	src/storage/oos_file.cpp	1779
oos_delete	src/storage/oos_file.cpp	1851
oos_get_max_chunk_size_within_page	src/storage/oos_file.cpp	1861
oos_needs_repl_tracking	src/storage/oos_file.cpp	1882
oos_rv_redo_delete	src/storage/oos_file.cpp	1888
oos_rv_redo_insert	src/storage/oos_file.cpp	1911
oos_get_length	src/storage/oos_file.cpp	1942
oos_push_oos_oid	src/storage/oos_file.cpp	1981
xoos_get_stats_by_class_oid	src/storage/oos_file.cpp	1991
oos_get_stats_by_vfid	src/storage/oos_file.cpp	2043
oos_record_header	src/storage/oos_file.hpp	26
OOS_RECORD_HEADER	src/storage/oos_file.hpp	26
OOS_RECORD_HEADER_SIZE	src/storage/oos_file.hpp	34
OOS_RECDES	src/storage/oos_file.hpp	38
oos_buffer	src/storage/oos_file.hpp	43
OOS_NUM_BEST_SPACESTATS	src/storage/oos_file.hpp	45
OOS_STATS_NEXT_BEST_INDEX	src/storage/oos_file.hpp	47
OOS_STATS_PREV_BEST_INDEX	src/storage/oos_file.hpp	49
oos_bestspace	src/storage/oos_file.hpp	53
oos_bestspace (struct)	src/storage/oos_file.hpp	53
OOS_HDR_STATS	src/storage/oos_file.hpp	59
oos_hdr_stats	src/storage/oos_file.hpp	60
oos_hdr_stats (struct)	src/storage/oos_file.hpp	60
OOS_FINDSPACE	src/storage/oos_file.hpp	101
oos_stats_info	src/storage/oos_file.hpp	111
oos_push_oos_oid	src/storage/oos_file.hpp	130
oos_log_get_file	src/storage/oos_log.hpp	86
oos_log_internal	src/storage/oos_log.hpp	97
oos_error	src/storage/oos_log.hpp	168
oos_warn	src/storage/oos_log.hpp	171
oos_trace	src/storage/oos_log.hpp	175
oos_oid_in_vector	src/storage/oos_util.cpp	34
oos_oid_in_vector	src/storage/oos_util.cpp	35
heap_recdes_compute_oos_flag_debug	src/storage/oos_util.cpp	83
oos repl emission loop (INSERT)	src/transaction/locator_sr.c	8144
oos repl emission loop (UPDATE)	src/transaction/locator_sr.c	8947
la_rebuild_oos_recdes	src/transaction/log_applier.c	4872
la_apply_dummy_oos_log	src/transaction/log_applier.c	6019
la_apply_oos_insert_log	src/transaction/log_applier.c	6044
oos_insert_lsa_queue	src/transaction/log_impl.h	529
oos_suppress_insert_lsa_queueing	src/transaction/log_impl.h	530
RVOOS_INSERT auto-push to oos_insert_lsa_queue	src/transaction/log_manager.c	2213
log_append_undoredo_crumbs (RVOOS_INSERT auto-queue)	src/transaction/log_manager.c	2213
log_append_redo_crumbs (RVOOS_INSERT auto-queue)	src/transaction/log_manager.c	2482
log_append_empty_record	src/transaction/log_manager.c	3150
LOG_DUMMY_OOS_RECORD	src/transaction/log_record.hpp	139
RVOOS_INSERT recovery entry	src/transaction/recovery.c	845
RV_fun[] RVOOS entries	src/transaction/recovery.c	845
RVOOS_INSERT	src/transaction/recovery.h	188
RVOOS_DELETE	src/transaction/recovery.h	189
RVREPL_OOS_INSERT	src/transaction/recovery.h	190
RVREPL_DUMMY_OOS_RECORD	src/transaction/recovery.h	198
repl_log_insert oos lsa attach	src/transaction/replication.c	459

Sources

• cubrid-oos.md — the high-level companion (design intent, JIRA epic, milestones).
• JIRA epic: http://jira.cubrid.org/browse/CBRD-26357 (M1 CBRD-26584, M2 CBRD-26583).
• Design docs: Notion CBRD-26628 multi-chunk insert; git CBRD-26668 vacuum explanation.
• Code (feat/oos): src/storage/oos_file.{cpp,hpp}, oos_log.hpp, oos_util.{cpp,hpp}, heap_oos.{cpp,hpp}, src/query/vacuum_oos.{cpp,hpp}, src/compat/db_oos.h.
• Methodology: knowledge/methodology/code-analysis-detail-doc.md.