CUBRID Overflow File — Code-Level Deep Dive

Where this document fits: The high-level analysis cubrid-overflow-file.md covers design intent and theoretical background. This document traces every branch and field at the code level. Each chapter is self-contained, but reading in order follows a single oversized record from its first overflow page chain through update, read, and reclamation.

Contents:

Ch	Title	Status
1	Data Structure Map	✅
2	File Provisioning and the Owning References	✅
3	Creating a Chain with overflow_insert	✅
4	Reading and Traversing a Chain	✅
5	Updating a Chain in Place with overflow_update	✅
6	Destroying and Reclaiming a Chain	✅
7	The B+Tree OID List Variant	✅
8	Crash Recovery and Edge Paths	✅

---

Chapter 1: Data Structure Map

The overflow file manager (src/storage/overflow_file.c / .h) stores a single logical record too large for one page as a private singly-linked chain of pages. Every entry point reinterprets a fixed-size page buffer (PAGE_PTR, DB_PAGESIZE bytes) as one of exactly two C structs, chosen only by position in the chain: the head page is an overflow_first_part, every other page an overflow_rest_part. There is no separate on-disk page-header struct, slot directory, or checksum field in this module — the page is the struct, cast directly from the page buffer pointer.

This chapter answers one question: what is on each overflow page, and how do the two formats differ field by field? It does not re-derive why a DBMS uses overflow pages, length-on-first-page, or unlocked private chains — that framing lives in the high-level companion under Common DBMS Design.

1.1 The two page formats, field by field

Both structs are declared in src/storage/overflow_file.h, short enough to quote whole:

// overflow_first_part -- src/storage/overflow_file.h
struct overflow_first_part
{
  VPID next_vpid;
  int length;
  char data[1];      /* Really more than one */
};

// overflow_rest_part -- src/storage/overflow_file.h
struct overflow_rest_part
{
  VPID next_vpid;
  char data[1];      /* Really more than one */
};

VPID (a { PAGEID pageid; VOLID volid; } pair, from src/storage/storage_common.h) is the on-disk page address — the “next pointer” of the linked list. data[1] is the C trailing-array idiom: declared with one element so the struct has a defined offsetof(...,data), but the page extends data to the end of the DB_PAGESIZE buffer — the comment /* Really more than one */ acknowledges this. Code never indexes data[] past element 0; it takes (char *) ...->data and memcpys a run of bytes.

overflow_first_part:

Field	Role	Why it exists
next_vpid	VPID of the second page of this chain, or NULL-VPID if the whole record fits on one page.	The linked-list pointer. The only way to reach the rest of the record.
length	Total byte length of the entire logical record (sum across all pages), not the bytes on this page.	The chain has no terminal sentinel for content; readers need the total up-front to size the output buffer and to know when to stop copying. See companion ### Length stored on the first page.
data[1]	Start of the payload byte stream; first DB_PAGESIZE - offsetof(...,data) bytes of the record live here.	The actual record content. The [1] is the trailing-array idiom; the real extent is the rest of the page.

overflow_rest_part:

Field	Role	Why it exists
next_vpid	VPID of the next page, or NULL-VPID on the last page of the chain.	The linked-list pointer. NULL here is the chain terminator.
data[1]	Continuation of the payload byte stream picked up where the previous page’s data left off.	More record content. No length — see below.

1.2 Why only the first page carries length

length is a per-record quantity, not per-page, so it is stored once, on the only page whose address the outside world knows (the head VPID). Every reader fixes the head first and reads length before walking — overflow_get_length casts the head to OVERFLOW_FIRST_PART * and returns ->length directly, never touching a rest_part. A rest_part never needs to report length: the walk terminates on VPID_ISNULL(&next_vpid), and the byte count is bounded by the head’s length. Putting length on every page would waste sizeof(int) per page and add a redundant-state consistency burden. See companion ### Length stored on the first page.

1.3 The header-size difference and per-page capacity

Because overflow_rest_part omits the int length field, its header is sizeof(int) (4 bytes) smaller than the first page’s. The code never hard-codes header sizes; it derives per-page payload capacity as DB_PAGESIZE - offsetof(...):

• First page: DB_PAGESIZE - offsetof(OVERFLOW_FIRST_PART, data) = DB_PAGESIZE - (sizeof(VPID) + sizeof(int)).
• Rest page: DB_PAGESIZE - offsetof(OVERFLOW_REST_PART, data) = DB_PAGESIZE - sizeof(VPID).

A rest page therefore holds sizeof(int) (4) more payload bytes than the first. The same asymmetry drives overflow_insert’s page-count estimate: it strips the smaller (first-page) capacity off the record once, and if anything remains divides it by the larger (rest-page) capacity:

// overflow_insert -- src/storage/overflow_file.c
length = recdes->length - (DB_PAGESIZE - (int) offsetof (OVERFLOW_FIRST_PART, data));
if (length > 0)
  {
    i = DB_PAGESIZE - offsetof (OVERFLOW_REST_PART, data);
    npages = 1 + CEIL_PTVDIV (length, i);
  }
else
  {
    npages = 1;
  }

Here length after the subtraction is the residual byte count beyond the head; i is the rest-page capacity; npages is the head plus ceil(residual / i), or 1 when the record fits in one page.

Invariant — header sizes are derived, never literal. Every payload-capacity and page-count computation reads offsetof(OVERFLOW_FIRST_PART, data) or offsetof(OVERFLOW_REST_PART, data), never a constant — keeping the 4-byte asymmetry correct under alignment or field changes. Replace an offsetof with a hard-coded number and the page-count estimate could under-count: the assert (length == 0) ending the copy loop fires (debug) or the record silently truncates (release).

1.4 The chain invariant and why pages are unlocked

The structural contract:

Invariant — the head VPID is the sole external handle; data[] is one contiguous byte stream stitched across pages; the chain is private. The only address ever stored in a containing record (heap relocation-overflow record, or B+Tree leaf/overflow OID slot) is ovf_vpid, the head page. Every other page is reachable only by following next_vpid from the head; no page is shared between two logical records. Because the chain is private and reachable only through an already-locked containing record, the overflow pages themselves are never locked — they are fixed with page-buffer latches (pgbuf_fix) for physical consistency, not lock-managed. The overflow_insert header comment states this verbatim (“Overflow pages are not locked … since they are not shared … and its address is only know by accessing the relocation overflow record data which has been appropriately locked”). See companion ### Pages not individually locked. If violated — two records pointing into one chain — deleting one frees pages still referenced by the other, corrupting the survivor.

The “contiguous byte stream” half makes the two structs interoperable: a reader does not care that byte k falls on a first_part and byte k+1 on a rest_part. overflow_get_nbytes advances copyfrom to the end of the current page’s data, fixes next_vpid, and resets copyfrom to the new page’s data — seamless because both structs end in data[].

The relationship between the two structs and the chain:

flowchart LR
  ref["containing record\nholds head VPID only"] --> H
  subgraph chain["one private overflow chain"]
    H["overflow_first_part\nnext_vpid -> P1\nlength = total bytes\ndata[ ...payload... ]"]
    P1["overflow_rest_part\nnext_vpid -> P2\ndata[ ...payload... ]"]
    P2["overflow_rest_part\nnext_vpid = NULL-VPID\ndata[ ...payload... ]"]
  end
  H --> P1 --> P2

Figure 1-1. Struct relationship: one overflow_first_part head (carrying length) followed by zero or more overflow_rest_part pages, terminated by a NULL next_vpid. Only the head VPID is externally known.

1.5 overflow_next_vpid — how position selects the format

Nothing on a page says whether it is a first or rest part; position does — is this page’s VPID the head VPID? overflow_next_vpid is the only helper that branches on this:

// overflow_next_vpid -- src/storage/overflow_file.c
if (VPID_EQ (ovf_vpid, vpid))
  {
    *vpid = ((OVERFLOW_FIRST_PART *) pgptr)->next_vpid;
  }
else
  {
    *vpid = ((OVERFLOW_REST_PART *) pgptr)->next_vpid;
  }

The VPID_EQ (ovf_vpid, vpid) test is the head/rest discriminant: when the page’s own VPID equals the head VPID it casts to OVERFLOW_FIRST_PART * (the format carrying length ahead of data), otherwise to OVERFLOW_REST_PART *. Both structs place next_vpid first, so reading the link alone is correct under either cast — but any field after it (length vs data) requires the right cast, which is why the discriminant exists. Readers that touch data/length reproduce the same VPID_EQ(addr_vpid_ptr, ovf_vpid) test rather than calling this helper.

1.6 OVERFLOW_DO_FUNC — the traversal discriminator

Applying a side effect to each page of a chain is centralized in overflow_traverse, parameterized by an enum so one walker serves both whole-chain operations:

// OVERFLOW_DO_FUNC -- src/storage/overflow_file.c
typedef enum
{
  OVERFLOW_DO_DELETE,
  OVERFLOW_DO_FLUSH
} OVERFLOW_DO_FUNC;

The two members carry the per-page side effect: OVERFLOW_DO_DELETE frees every page as it is visited (the destructive walk behind overflow_delete), OVERFLOW_DO_FLUSH WAL-flushes every page (the read-only walk behind overflow_flush). overflow_traverse switches on this per page: OVERFLOW_DO_DELETE first guards ovf_vfid != NULL (if (ovf_vfid == NULL) goto exit_on_error;, since you cannot deallocate without the file id) then calls overflow_delete_internal; OVERFLOW_DO_FLUSH calls overflow_flush_internal; the default: arm is a no-op break. Callers differ in one argument — overflow_delete passes a real ovf_vfid, overflow_flush passes NULL — so the enum alone distinguishes the two walks. Per-branch mechanics are Chapter 6’s subject.

1.7 The file_type assertion and the page-type tag

Two independent type tags guard this subsystem at different layers; do not confuse them.

File type — FILE_TYPE (from src/storage/file_manager.h). overflow_insert accepts a chain only inside one of three file kinds, enforced by an assertion:

// overflow_insert -- src/storage/overflow_file.c
assert (file_type == FILE_TEMP  /* sort files */
        || file_type == FILE_BTREE_OVERFLOW_KEY  /* b-tree overflow key */
        || file_type == FILE_MULTIPAGE_OBJECT_HEAP /* heap overflow file */ );

FILE_TYPE value	Who uses it	Logging behavior in overflow_insert
FILE_MULTIPAGE_OBJECT_HEAP	Heap records too big for a heap page.	Logged (redo of each new page + LOG_DUMMY_OVF_RECORD on the head).
FILE_BTREE_OVERFLOW_KEY	B+Tree keys too big to inline in a leaf.	Logged, same as heap.
FILE_TEMP	Sort / query temporary spill files.	Not logged; allocation uses file_init_temp_page_type. The file_type != FILE_TEMP guards skip all log_append_* calls.

overflow_update narrows this to file_type == FILE_MULTIPAGE_OBJECT_HEAP only, with a source comment warning it has not been audited for the other two types (Chapter 5).

Page type — PAGE_OVERFLOW (from the PAGE_TYPE enum in src/storage/storage_common.h). Regardless of file, every chain page is tagged PAGE_OVERFLOW in the page buffer. overflow_insert sets PAGE_TYPE ptype = PAGE_OVERFLOW; and passes &ptype to file_alloc_multiple; every other entry point asserts the tag after fixing a page:

// repeated guard across overflow_file.c
#if !defined (NDEBUG)
  (void) pgbuf_check_page_ptype (thread_p, addr.pgptr, PAGE_OVERFLOW);
#endif /* !NDEBUG */

Invariant — every fixed overflow page is PAGE_OVERFLOW. Insert stamps it at allocation; recovery re-stamps it (overflow_rv_page_update_redo calls pgbuf_set_page_ptype(..., PAGE_OVERFLOW) before replaying content); every read/update/delete/flush path debug-asserts it via pgbuf_check_page_ptype. A non-overflow page reached through a next_vpid trips the assert in debug builds — the chain is corrupted. FILE_TYPE and PAGE_TYPE are orthogonal: all three legal file types produce PAGE_OVERFLOW pages.

1.8 Byte-level layout of one chain

Byte map of a record spanning three pages. V = sizeof(VPID), L = sizeof(int), P = DB_PAGESIZE.

  head page (overflow_first_part)              offset
  +-----------------------------------------+
  | next_vpid : VPID -> page 2              |  0
  | length    : int  = TOTAL record bytes   |  V
  | data[]    : payload 0 .. P-V-L-1        |  V+L = offsetof(FIRST,data)
  +-----------------------------------------+  P
                | next_vpid
                v
  rest page 2 (overflow_rest_part)
  +-----------------------------------------+
  | next_vpid : VPID -> page 3              |  0
  | data[]    : next P-V bytes (L more room)|  V   = offsetof(REST,data)
  +-----------------------------------------+  P
                | next_vpid
                v
  rest page 3 (overflow_rest_part)  [LAST]
  +-----------------------------------------+
  | next_vpid : NULL-VPID (VPID_ISNULL)     |  0   <- chain terminator
  | data[]    : final bytes; tail = slack   |  V
  +-----------------------------------------+  P

  sum of all three data[] runs == head.length   (read-stop condition)

Figure 1-2. Pointer/layout map of a three-page chain. The head reserves V+L; each rest page reserves only V, giving it L (=4) more payload room. The last page’s next_vpid is NULL and its data[] tail may be unused — length on the head, not page fullness, defines where the record ends.

The final page’s unused tail is real slack: overflow_get_capacity reports it as *ovf_free_space, and overflow_update exploits it to grow a record in place when new content still fits the existing page span (Chapter 5).

1.9 Chapter summary — key takeaways

1. Two formats, position-selected. A page is an overflow_first_part iff its VPID equals the externally-known head VPID, else an overflow_rest_part. No flag on the page — overflow_next_vpid’s VPID_EQ test is the discriminant.
2. length lives only on the head and means the total record byte count, not per-page bytes. Readers (overflow_get_length, overflow_get_nbytes, overflow_get_capacity) read it once from the head before walking.
3. Rest pages hold 4 more payload bytes than the head by omitting int length; the difference is sizeof(int) = offsetof(FIRST,data) - offsetof(REST,data). All capacity math uses offsetof, never literals — an enforced invariant.
4. data[1] is the trailing-array idiom: declared length 1, extended to page end; code uses (char *) ...->data plus memcpy, never data[i] for i > 0.
5. The chain is private and unlocked. The head VPID is the sole external handle; other pages are reached only via next_vpid; no page is shared; pages get page-buffer latches, no lock-manager locks.
6. OVERFLOW_DO_FUNC (OVERFLOW_DO_DELETE / OVERFLOW_DO_FLUSH) is the single discriminator turning the shared overflow_traverse walker into a destructive or a read-only walk.
7. Two orthogonal type tags guard the subsystem: FILE_TYPE (one of FILE_TEMP, FILE_BTREE_OVERFLOW_KEY, FILE_MULTIPAGE_OBJECT_HEAP, also deciding whether inserts are logged) and PAGE_OVERFLOW (every overflow page, any file type, asserted on every fix).

Chapter 2: File Provisioning and the Owning References

A chain needs two things before it exists: a file (VFID) to allocate pages from, and a reference at the home object pointing at the head page. This chapter traces where the file comes from and what field holds the head VPID — for the heap big-record path and the two B+Tree paths — stopping where the file exists and the reference slot is ready (the byte-level insert loop is Chapter 3). The high-level companion (cubrid-overflow-file.md) covers why there is one file per heap and per B+Tree and why pages are not individually locked; this chapter works at the branch level.

2.1 Two provisioning models — lazy vs. lazy-with-preallocation

Both create the overflow file lazily — it does not exist until the first record spills — but differ in the ways this chapter dissects:

Aspect	Heap (heap_ovf_find_vfid)	B+Tree (btree_create_overflow_key_file)
File type	FILE_MULTIPAGE_OBJECT_HEAP	FILE_BTREE_OVERFLOW_KEY
Pages preallocated	1	at least 3
Where the VFID is stored	HEAP_HDR_STATS.ovf_vfid (header slot HEAP_HEADER_AND_CHAIN_SLOTID)	BTID_INT::ovfid + root header ovfid
Recovery record	RVHF_STATS (undo + redo)	RVBT_UPDATE_OVFID (undoredo)
Descriptor arm	FILE_OVF_HEAP_DES	FILE_OVF_BTREE_DES
Creator vs find-only	heap_ovf_insert (docreate=true); heap_ovf_update/heap_ovf_delete (docreate=false)	leaf-insert path on key overflow; no find function — presence read from BTID_INT::ovfid

Every heap big record uses the heap’s one FILE_MULTIPAGE_OBJECT_HEAP; the B+Tree’s two overflow uses are split across two files (§2.5).

flowchart LR
  HHDR["HEAP_HDR_STATS.ovf_vfid"] -- names --> HFILE["FILE_MULTIPAGE_OBJECT_HEAP\ndesc FILE_OVF_HEAP_DES"]
  BROOT["Root header ovfid"] -- persisted --> BINT["BTID_INT::ovfid\nin-mem mirror"]
  BINT -- names --> BFILE["FILE_BTREE_OVERFLOW_KEY\ndesc FILE_OVF_BTREE_DES"]

Figure 2-1 — Owning-file references. The heap stores its VFID in the heap header record; the B+Tree stores it in the root header and mirrors it to in-memory BTID_INT::ovfid.

2.2 heap_ovf_find_vfid — the heap’s find-or-create

heap_ovf_find_vfid is the single entry point for every heap caller needing the overflow file. Its docreate argument switches between “create if missing” (insert) and “look up only” (update/delete); the header latch follows it — write for a creator, read for a reader.

// heap_ovf_find_vfid -- src/storage/heap_file.c
  mode = (docreate == true ? PGBUF_LATCH_WRITE : PGBUF_LATCH_READ);
  addr_hdr.pgptr = pgbuf_fix (thread_p, &vpid, OLD_PAGE, mode, latch_cond);
  if (addr_hdr.pgptr == NULL)
    return NULL;                         /* <- branch A: header fix failed */
  // ... condensed: pgbuf_check_page_ptype (PAGE_HEAP) ...
  if (spage_get_record (..., HEAP_HEADER_AND_CHAIN_SLOTID, &hdr_recdes, PEEK) != S_SUCCESS)
    { pgbuf_unfix_and_init (thread_p, addr_hdr.pgptr); return NULL; }   /* <- branch B: header unreadable */
  heap_hdr = (HEAP_HDR_STATS *) hdr_recdes.data;

The header page sits at a well-known location (hfid->vfid / hfid->hpgid). The record is peeked, so heap_hdr is a live pointer and any mutation below is an in-place edit that RVHF_STATS protects.

Branch-complete flow:

flowchart TB
  A["fix header page\nmode = docreate ? WRITE : READ"] --> B{pgptr == NULL?}
  B -- yes --> R1["return NULL"]
  B -- no --> D{spage_get_record\npeek HEAP_HDR_STATS ok?}
  D -- no --> R2["unfix; return NULL"]
  D -- yes --> E{VFID_ISNULL\novf_vfid?}
  E -- "no, exists" --> F["VFID_COPY out\nfrom heap_hdr->ovf_vfid"]
  E -- "yes, missing" --> G{docreate?}
  G -- "false" --> H["ovf_vfid = NULL\ncaller sees no file"]
  G -- "true" --> J["log_sysop_start\nfile_create_with_npages MOH, 1"]
  J --> K{create AND tde lookup\nAND tde apply all ok?}
  K -- "no, any step" --> X["log_sysop_abort\novf_vfid = NULL; goto exit"]
  K -- yes --> P["log_append_undo RVHF_STATS\nVFID_COPY into heap_hdr\nlog_append_redo RVHF_STATS\npgbuf_set_dirty; log_sysop_commit"]
  F --> Z["exit: unfix; return ovf_vfid"]
  H --> Z
  P --> Z
  X --> Z

Figure 2-2 — Every branch of heap_ovf_find_vfid. The K-node collapses the three independent failure points (create, TDE lookup, TDE apply); each aborts the sysop and returns NULL.

The non-creating outcomes (VFID_ISNULL false, or docreate == false) are the steady state: once the file exists, every later call hits VFID_COPY. The creation arm is bracketed as a top system operation:

// heap_ovf_find_vfid -- src/storage/heap_file.c (create arm, condensed)
  log_sysop_start (thread_p);
  HFID_COPY (&des.heap_overflow.hfid, hfid);          /* <- descriptor: owning heap */
  des.heap_overflow.class_oid = heap_hdr->class_oid;  /* <- + class, for TDE */
  if (file_create_with_npages (..., FILE_MULTIPAGE_OBJECT_HEAP, 1, &des, ovf_vfid) != NO_ERROR)
    { log_sysop_abort (thread_p); ovf_vfid = NULL; goto exit; }   /* <- rolled back */
  // ... TDE lookup + apply, each with its own abort+goto ...
  log_append_undo_data (thread_p, RVHF_STATS, &addr_hdr, sizeof (*heap_hdr), heap_hdr);
  VFID_COPY (&heap_hdr->ovf_vfid, ovf_vfid);          /* <- the actual header mutation */
  log_append_redo_data (thread_p, RVHF_STATS, &addr_hdr, sizeof (*heap_hdr), heap_hdr);
  pgbuf_set_dirty (thread_p, addr_hdr.pgptr, DONT_FREE);
  log_sysop_commit (thread_p);

Ordering matters: the VFID_COPY sits between the two log calls, so undo captures the whole HEAP_HDR_STATS before the copy (null VFID) and redo after it (new VFID) — the entire struct is logged because RVHF_STATS replays it wholesale. TDE is applied before the header points at the file, in the same sysop.

Invariant: the heap-header VFID write and the file creation are atomic to the outer transaction. Enforced by the log_sysop_start / log_sysop_commit bracket around file_create_with_npages plus the RVHF_STATS undo/redo pair. If violated — file created but header write lost — the heap would carry an orphan FILE_MULTIPAGE_OBJECT_HEAP, and the next insert would create a second one.

2.3 The home reference for a heap big record — REC_BIGONE

heap_ovf_find_vfid returns only the file. The chain’s head page, produced by overflow_insert (Chapter 3) as a VPID, is converted by heap_ovf_insert into the home reference:

// heap_ovf_insert -- src/storage/heap_file.c
  if (heap_ovf_find_vfid (thread_p, hfid, &ovf_vfid, true, ...) == NULL
      || overflow_insert (thread_p, &ovf_vfid, &ovf_vpid, recdes, FILE_MULTIPAGE_OBJECT_HEAP) != NO_ERROR)
    return NULL;                         /* <- find/create or insert failed */
  ovf_oid->pageid = ovf_vpid.pageid;
  ovf_oid->volid  = ovf_vpid.volid;
  ovf_oid->slotid = NULL_SLOTID;         /* Irrelevant */

The home slot body is an OID, but only volid and pageid are meaningful — addressing the first overflow page — while slotid is forced to NULL_SLOTID. The overflow record has no slot directory (its bytes are stitched from each page’s data[]), so /* Irrelevant */ warns readers off dereferencing it.

Invariant: a REC_BIGONE home OID always carries slotid == NULL_SLOTID. Enforced by this single assignment site and relied on wherever the big-record path fixes the head page by VPID; a non-null slotid is the tell of a corrupted forwarding pointer. heap_ovf_insert only fills ovf_oid; the heap insert layer writes the REC_BIGONE record (read/update chapters).

2.4 btree_create_overflow_key_file — the B+Tree’s create

The B+Tree has no find-style function: the presence test is a direct VFID_ISNULL (&btid_int->ovfid), and creation is a dedicated function that fills BTID_INT::ovfid:

// btree_create_overflow_key_file -- src/storage/btree.c
  VFID_SET_NULL (&btid->ovfid);
  des.btree_key_overflow.btid = *btid->sys_btid;            /* structure copy */
  des.btree_key_overflow.class_oid = btid->topclass_oid;   // ... assert class_oid not null ...
  error_code = file_create_with_npages (thread_p, FILE_BTREE_OVERFLOW_KEY, 3, &des, &btid->ovfid);
  if (error_code != NO_ERROR)
    return error_code;                   /* <- branch A: create failed, ovfid stays NULL */
  error_code = heap_get_class_tde_algorithm (thread_p, &btid->topclass_oid, &tde_algo);
  if (error_code != NO_ERROR)
    { VFID_SET_NULL (&btid->ovfid); return error_code; }   /* <- branch B: roll back VFID */
  error_code = file_apply_tde_algorithm (thread_p, &btid->ovfid, tde_algo);
  if (error_code != NO_ERROR)
    { VFID_SET_NULL (&btid->ovfid); return error_code; }   /* <- branch C: roll back VFID */
  return error_code;                     /* NO_ERROR: ovfid now valid */

Branch coverage and differences from the heap version:

• VFID_SET_NULL at entry ensures any sub-step failure (branches A, B, C) leaves the caller a null ovfid, never a half-set VFID. The function fills only in-memory BTID_INT::ovfid and opens no log_sysop_start; the durable root write lives in the caller.
• At least 3 pages preallocated versus 1 for the heap, amortizing follow-on overflow-key allocations against one file. TDE inheritance mirrors the heap (heap_get_class_tde_algorithm).

The caller — the leaf-insert path — makes it durable and safe:

// btree leaf-insert path -- src/storage/btree.c (condensed; root is write latched)
  if (VFID_ISNULL (&btid_int->ovfid))                  /* <- re-check under root WRITE latch */
    {
      log_sysop_start (thread_p);
      error_code = btree_create_overflow_key_file (thread_p, btid_int);
      if (error_code != NO_ERROR)
        { log_sysop_abort (thread_p); goto error; }
      log_append_undoredo_data2 (thread_p, RVBT_UPDATE_OVFID, ..., *root_page, HEADER,
                                 sizeof (VFID), sizeof (VFID), &root_header->ovfid, &btid_int->ovfid);
      VFID_COPY (&root_header->ovfid, &btid_int->ovfid);  /* <- persist into root header */
      pgbuf_set_dirty (thread_p, *root_page, DONT_FREE);
      log_sysop_commit (thread_p);
    }

Two points the heap path does not have:

• The null-check is repeated under the root write latch — source comment “Check that overflow key file was not created by another.” Without it two concurrent inserters each create a file; the latch makes the loser observe the winner’s VFID.
• RVBT_UPDATE_OVFID is an undoredo2 scoped to the root HEADER slot, logging only the VFID field (old=undo, new=redo), not the whole header as RVHF_STATS does. The sysop lives in the caller, so the create-failure path aborts it at the call site.

Invariant: at most one FILE_BTREE_OVERFLOW_KEY per B+Tree, and root header ovfid equals BTID_INT::ovfid once created. Enforced by the under-latch re-check and by the RVBT_UPDATE_OVFID undoredo tying the root write to the file-create sysop. Violation leaks the file and splits the overflow-key chains.

2.5 Page-type discipline — PAGE_OVERFLOW vs PAGE_BTREE

The three chain kinds split across two page types; page type — not file — is the disk-level discriminator:

Chain kind	Owning file	Page type	Head reference field	Leaf flag
Heap big record	FILE_MULTIPAGE_OBJECT_HEAP	PAGE_OVERFLOW	REC_BIGONE OID body {volid, pageid}	n/a
B+Tree overflow key	FILE_BTREE_OVERFLOW_KEY (ovfid)	PAGE_OVERFLOW	head VPID embedded in leaf/non-leaf record	BTREE_LEAF_RECORD_OVERFLOW_KEY (0x4000)
B+Tree overflow OID list	pages from the tree file via btree_get_new_page	PAGE_BTREE	LEAF_REC.ovfl (a VPID)	BTREE_LEAF_RECORD_OVERFLOW_OIDS (0x2000)

Footnote: LEAF_REC is { VPID ovfl; short key_len; }, ovfl being VPID_SET_NULL with no OID-list overflow; the overflow-key reference instead reuses the record body in place of the inlined key bytes.

The overflow-key path runs through overflow_insert, which sets PAGE_TYPE ptype = PAGE_OVERFLOW, passes it to file_alloc_multiple as the per-page initializer, and asserts pgbuf_check_page_ptype (..., PAGE_OVERFLOW) on each. The overflow-OID-list path never touches overflow_file.c: its pages come from btree_start_overflow_page → btree_get_new_page, whose initializer is btree_initialize_new_page:

// btree_initialize_new_page -- src/storage/btree.c
  pgbuf_set_page_ptype (thread_p, page, PAGE_BTREE);   /* <- not PAGE_OVERFLOW */
  spage_initialize (thread_p, page, UNANCHORED_KEEP_SEQUENCE, BTREE_MAX_ALIGN, ...);  /* slotted */

Critically, btree_get_new_page allocates from btid->sys_btid->vfid — the main tree file, not ovfid. So an OID-list overflow page is “just another B+Tree page”: slotted, PAGE_BTREE (Chapter 7). The overflow-key chain is byte-identical to a heap big record: PAGE_OVERFLOW, no slot directory, OVERFLOW_*_PART headers.

Invariant: page type partitions the two B+Tree overflow uses. A page reached via LEAF_REC.ovfl must check out PAGE_BTREE; one via an overflow-key VPID must check out PAGE_OVERFLOW. The pgbuf_check_page_ptype asserts enforce it; confusing them feeds an OVERFLOW_FIRST_PART stream to slotted-page access.

All three head references are forward pointers into the chain head, never the chain itself; the internal links (next_vpid / BTREE_OVERFLOW_HEADER.next_vpid) are Chapters 3, 4, 7.

2.6 Chapter summary — key takeaways

1. Lazy provisioning, stored at the home object’s header. The heap keeps its VFID in HEAP_HDR_STATS.ovf_vfid; the B+Tree keeps it in the root header ovfid mirrored to BTID_INT::ovfid.
2. heap_ovf_find_vfid is find-or-create in one function. docreate=true (only from heap_ovf_insert) creates a 1-page FILE_MULTIPAGE_OBJECT_HEAP under a sysop, applies class TDE, logs RVHF_STATS undo/redo; docreate=false reads the existing VFID.
3. btree_create_overflow_key_file only fills BTID_INT::ovfid; the caller persists it. It preallocates ≥3 pages and applies class TDE, nulling the VFID on failure; the caller re-checks VFID_ISNULL under the root write latch and logs the root write with RVBT_UPDATE_OVFID.
4. The heap home reference is a REC_BIGONE OID with slotid = NULL_SLOTID. Only {volid, pageid} matter — the overflow record has no slot directory.
5. Page type, not file, is the disk-level discriminator. Heap big-record and B+Tree overflow-key chains use PAGE_OVERFLOW; overflow-OID-list chains use PAGE_BTREE via btree_get_new_page / btree_initialize_new_page.
6. Each chain kind has a distinct owning file. B+Tree overflow-key chains live in BTID_INT::ovfid (FILE_BTREE_OVERFLOW_KEY), while overflow-OID-list chains live in the main tree file (btid->sys_btid->vfid). Leaf flags BTREE_LEAF_RECORD_OVERFLOW_KEY (0x4000) vs BTREE_LEAF_RECORD_OVERFLOW_OIDS (0x2000) say which to follow.
7. All VFID writes are crash-atomic. Both subsystems wrap file-create plus the header/VFID write in a sysop, so a crash leaves either no file and no reference, or both — never a leaked file.

Chapter 3: Creating a Chain with overflow_insert

This chapter answers one question: given a multi-page RECDES and a VFID, how does CUBRID build the entire overflow chain — page allocation, payload distribution, write-ahead logging — atomically in a single forward pass? The function is overflow_insert. Its two callers, heap_ovf_insert and btree_store_overflow_key, only assemble the RECDES, pick a FILE_TYPE, and translate the first-page VPID into a caller-friendly address. We dissect all three, every branch.

For the chain layout (OVERFLOW_FIRST_PART / OVERFLOW_REST_PART) and the “why overflow exists” theory, see Chapter 1 and the companion cubrid-overflow-file.md. We focus on the write path.

3.1 The two structs the writer fills (recap)

overflow_insert writes two on-page header shapes — OVERFLOW_FIRST_PART (page 0) and OVERFLOW_REST_PART (pages 1..n-1). Chapter 1 covers every field; the write path touches only three: next_vpid (both — forward link from vpids[i+1], NULL on the last page and on page 0 when npages == 1), length (first only — total byte count of the entire record so the reader sizes its buffer without walking the chain, Ch 4), and data[1] (both — flexible array memcpy’d after the header, where offsetof(...,data) is the per-page payload capacity).

Because OVERFLOW_REST_PART drops length, a rest page’s payload capacity exceeds the first page’s by exactly sizeof(int) — the asymmetry the 3.2 math hinges on.

3.2 Up-front exact page-count math

Before touching disk, overflow_insert computes the exact page count — not a guess corrected later, as the trailing assert(length == 0) (3.7) proves.

// overflow_insert -- src/storage/overflow_file.c
length = recdes->length - (DB_PAGESIZE - (int) offsetof (OVERFLOW_FIRST_PART, data));
if (length > 0)
  {
    i = DB_PAGESIZE - offsetof (OVERFLOW_REST_PART, data);   /* rest-page payload capacity */
    npages = 1 + CEIL_PTVDIV (length, i);                    /* 1 first page + ceil(rest/cap) */
  }
else
  {
    npages = 1;                                              /* whole record fits in page 0 */
  }

length is re-purposed as “bytes that do not fit on page 0”: if <= 0 the npages == 1 path runs; otherwise CEIL_PTVDIV rounds the remainder up over rest-page capacity and 1 + re-adds page 0.

Invariant — the page count is exact, not an upper bound. overflow_insert allocates exactly npages and the loop must consume the record to the byte (assert(length == 0), 3.7). Collapse the two offsetof values and the loop either leaves bytes unwritten or indexes vpids[npages] — a page never allocated.

3.3 The VPID array: stack buffer vs malloc, and the OOM branch

overflow_insert needs npages + 1 VPIDs — one per page plus a sentinel slot. To avoid a heap allocation in the common case, it keeps a stack buffer and only mallocs for large chains.

// overflow_insert -- src/storage/overflow_file.c
#define OVERFLOW_ALLOCVPID_ARRAY_SIZE 64       /* (top of file) */
VPID vpids_buffer[OVERFLOW_ALLOCVPID_ARRAY_SIZE + 1];   /* +1 for the sentinel slot */
// ...
if (npages > OVERFLOW_ALLOCVPID_ARRAY_SIZE)
  {
    vpids = (VPID *) malloc ((npages + 1) * sizeof (VPID));
    if (vpids == NULL)
      {
        er_set (ER_ERROR_SEVERITY, ARG_FILE_LINE, ER_OUT_OF_VIRTUAL_MEMORY, 1, (npages + 1) * sizeof (VPID));
        return ER_OUT_OF_VIRTUAL_MEMORY;       /* <- direct return: nothing acquired yet */
      }
  }
else
  {
    vpids = vpids_buffer;                       /* point at the stack buffer */
  }

• npages > 64 malloc OOM → er_set then a plain return, not goto exit_on_error: no sysop started, no page fixed, nothing to abort or free.
• npages <= 64 → vpids aliases vpids_buffer; if (vpids != vpids_buffer) is the sole free signal. The threshold is a space/speed tradeoff — most records fit under 64 pages.

The debug-only (!NDEBUG) pre-fill and post-alloc asserts. NDEBUG-off builds run for (i...) VPID_SET_NULL(&vpids[i]) to null every slot, then after file_alloc_multiple a symmetric loop asserts !VPID_ISNULL(&vpids[i]). That null-fill is debug-only, but the sentinel VPID_SET_NULL(&vpids[npages]) runs in every build: the loop writes it into the last page’s next_vpid (vpids[i+1] reads vpids[npages] when i == npages-1), terminating the chain. Without it the last forward link is garbage and Chapter 4’s traversal never stops.

3.4 Opening the system operation and allocating all pages at once

// overflow_insert -- src/storage/overflow_file.c
log_sysop_start (thread_p);
is_sysop_started = true;
error_code = file_alloc_multiple (thread_p, ovf_vfid,
                                  file_type != FILE_TEMP ? file_init_page_type : file_init_temp_page_type,
                                  &ptype, npages, vpids);
if (error_code != NO_ERROR)
  { ASSERT_ERROR (); goto exit_on_error; }
*ovf_vpid = vpids[0];                            /* head VPID returned to caller */

log_sysop_start opens a nested system operation; success attaches to the outer transaction, any failure aborts (both in 3.7), making chain creation atomic. file_alloc_multiple grabs all npages at once (filling vpids[0..npages-1]) with an initializer that branches on file_type: file_init_temp_page_type for FILE_TEMP, file_init_page_type otherwise — both stamp PAGE_OVERFLOW via &ptype; on failure, goto exit_on_error. The sole output *ovf_vpid = vpids[0] is the chain head.

3.5 The per-page copy loop — first-part vs rest-part branch

A single for loop fixes each page, writes its header + payload slice, logs it, frees it. The only structural branch is i == 0 (first part) vs else (rest).

// overflow_insert -- src/storage/overflow_file.c
data = recdes->data;
length = recdes->length;                        /* re-used: now the running remainder */
for (i = 0; i < npages; i++)
  {
    addr.pgptr = pgbuf_fix (thread_p, &vpids[i], OLD_PAGE, PGBUF_LATCH_WRITE, PGBUF_UNCONDITIONAL_LATCH);
    if (addr.pgptr == NULL)
      { ASSERT_ERROR_AND_SET (error_code); goto exit_on_error; }   /* fix failed: abort whole chain */
    // ... !NDEBUG pgbuf_check_page_ptype (addr.pgptr, PAGE_OVERFLOW) ...
    if (i == 0)
      {
        first_part = (OVERFLOW_FIRST_PART *) addr.pgptr;
        first_part->next_vpid = vpids[i + 1];   /* link to page 1 (or sentinel NULL) */
        first_part->length = length;            /* TOTAL length, written once */
        copyto = (char *) first_part->data;
        copy_length = DB_PAGESIZE - offsetof (OVERFLOW_FIRST_PART, data);
        if (length < copy_length) copy_length = length;            /* last/only page: clamp */
        if (file_type != FILE_TEMP)
          log_append_empty_record (thread_p, LOG_DUMMY_OVF_RECORD, &addr);  /* first page only */
      }
    else
      {                                          /* rest part: identical, minus the length field */
        rest_parts = (OVERFLOW_REST_PART *) addr.pgptr;
        rest_parts->next_vpid = vpids[i + 1];   /* sentinel NULL on last page */
        copyto = (char *) rest_parts->data;
        copy_length = DB_PAGESIZE - offsetof (OVERFLOW_REST_PART, data);
        if (length < copy_length) copy_length = length;
      }
    memcpy (copyto, data, copy_length);
    // ... redo logging (3.6) ...
    data += copy_length;                        /* advance source cursor */
    length -= copy_length;                      /* shrink remainder */
    pgbuf_set_dirty_and_free (thread_p, addr.pgptr);
  }

Each page is fixed-then-freed within one iteration, so at most one is held; on pgbuf_fix failure the 3.7 abort unwinds the rest. The length < copy_length clamp fires on the final partial page (and the sole page when npages == 1).

flowchart TD
  FIX["pgbuf_fix vpids[i] WRITE latch"]
  FIX -->|NULL| ERR["ASSERT_ERROR_AND_SET; goto exit_on_error"]
  FIX -->|ok| Q{"i == 0 ?"}
  Q -->|yes| FIRST["first_part: next_vpid, length, first-part cap, clamp; LOG_DUMMY_OVF_RECORD if not TEMP"]
  Q -->|no| REST["rest_parts: next_vpid, rest-part cap, clamp"]
  FIRST --> CP["memcpy copy_length"]
  REST --> CP
  CP --> REDO{"not TEMP and no_logging != true ?"}
  REDO -->|yes| LOGR["log_append_redo_data RVOVF_NEWPAGE_INSERT"]
  REDO -->|no| ADV["data += copy_length; length -= copy_length"]
  LOGR --> ADV
  ADV --> FREE["pgbuf_set_dirty_and_free"]
  FREE --> NEXT{"i+1 < npages ?"}
  NEXT -->|yes| FIX
  NEXT -->|no| DONE["assert length == 0"]

Figure 3-1. One pass of the copy loop: fix-failure, first/rest header split, redo-logging branches.

3.6 Logging: the dummy first-page marker and per-page redo

Two log records are emitted (both visible in the 3.5 excerpt):

• LOG_DUMMY_OVF_RECORD (log_append_empty_record, i == 0 branch) — an empty record, once, page 0 only, for non-FILE_TEMP chains. It anchors the chain head in the log so recovery/vacuum can correlate it with the owning record (Chapter 8). Temp files are never recovered, so they skip it.
• RVOVF_NEWPAGE_INSERT (log_append_redo_data, after memcpy) — per page, gated on both file_type != FILE_TEMP and thread_p->no_logging != true. Its logged length copy_length + CAST_BUFLEN (copyto - addr.pgptr) covers the whole written prefix (header and data), so one redo rebuilds the page; no_logging lets bulk paths suppress redo while still allocating real pages.

Invariant — temp chains never log; persistent chains always log redo unless no_logging. The gating must stay matched to the 3.4 page-init branch: a dummy record on a temp file makes recovery follow a chain that did not survive the crash; skipping redo on a persistent chain leaves its pages stale.

3.7 Success and failure epilogues

The two exits mirror each other around the sysop bracket:

// overflow_insert -- src/storage/overflow_file.c
assert (length == 0);                           /* 3.2's "exact" invariant, proven here */
#if defined (CUBRID_DEBUG)
  if (length > 0)
    { assert (false); er_log_debug (ARG_FILE_LINE, "ovf_insert: ** SYSTEM ERROR ... # of pages ... incorrect");
      error_code = ER_FAILED; goto exit_on_error; }   /* recoverable miscalc, not a hard assert */
#endif
  log_sysop_attach_to_outer (thread_p);         /* SUCCESS: undone with the outer op, not standalone */
  if (vpids != vpids_buffer) free_and_init (vpids);
  return NO_ERROR;

exit_on_error:
  if (is_sysop_started) log_sysop_abort (thread_p);   /* roll back every page alloc + write */
  if (vpids != vpids_buffer) free_and_init (vpids);
  return error_code;

On success, a non-zero length means the 3.2 math and the loop disagreed; only CUBRID_DEBUG escalates it to a recoverable ER_FAILED rollback rather than the bare assert. log_sysop_attach_to_outer attaches the sysop to the enclosing transaction’s undo chain (not a standalone sysop_commit), so the chain is undone with the outer operation. On the error path, is_sysop_started guards the abort — no goto reaches the label before log_sysop_start (the early malloc-OOM returns to avoid it), so every arrival aborts the sysop, undoing file_alloc_multiple and discarding the redo. Both exits share the vpids != vpids_buffer free guard.

Invariant — all-or-nothing chain creation. Either overflow_insert returns NO_ERROR with a fully written, logged, outer-attached chain, or an error with zero surviving pages — enforced by the sysop bracket: committing before the loop finishes would strand allocated-but-empty pages with no owner on a mid-loop crash.

3.8 The two wrappers that feed it

Two thin wrappers prepare overflow_insert’s arguments. heap_ovf_insert (heap multi-page objects) resolves the heap’s overflow VFID, calls overflow_insert with FILE_MULTIPAGE_OBJECT_HEAP, then converts the head VPID into an OID:

// heap_ovf_insert -- src/storage/heap_file.c
if (heap_ovf_find_vfid (thread_p, hfid, &ovf_vfid, true, PGBUF_UNCONDITIONAL_LATCH) == NULL
    || overflow_insert (thread_p, &ovf_vfid, &ovf_vpid, recdes, FILE_MULTIPAGE_OBJECT_HEAP) != NO_ERROR)
  return NULL;                                  /* no overflow file, or insert failed */
ovf_oid->pageid = ovf_vpid.pageid; ovf_oid->volid = ovf_vpid.volid;
ovf_oid->slotid = NULL_SLOTID;                  /* overflow OIDs have no slot */
return ovf_oid;

The || collapses two failure branches (the true argument creates the overflow file if absent). On success the head VPID fills ovf_oid’s pageid/volid with slotid = NULL_SLOTID — addressed by page, not slot (Chapter 2).

btree_store_overflow_key (oversized index keys). Serializes a DB_VALUE key into a RECDES, then calls overflow_insert with FILE_BTREE_OVERFLOW_KEY:

// btree_store_overflow_key -- src/storage/btree.c
if (src_type != dst_type)                       /* key domain differs from index domain */
  { key_ptr = &new_key;
    status = tp_value_cast (key, key_ptr, tp_domain, false);
    if (status != DOMAIN_COMPATIBLE) { assert (false); goto exit_on_error; }
    size = btree_get_disk_size_of_key (key_ptr); }
rec.data = (char *) db_private_alloc (thread_p, size);   /* OOM -> er_set + goto exit_on_error */
// ... or_init + pr_type->index_writeval (failure -> goto exit_on_error); rec.length = serialized len ...
if (overflow_insert (thread_p, &overflow_file_vfid, first_overflow_page_vpid, &rec, FILE_BTREE_OVERFLOW_KEY) != NO_ERROR)
  { ASSERT_ERROR (); goto exit_on_error; }

The node type pre-selects tp_domain; a domain mismatch casts the key (non-DOMAIN_COMPATIBLE is a hard assert(false)); db_private_alloc OOM, index_writeval failure, and overflow_insert failure each goto exit_on_error (freeing rec.data/key_ptr, normalizing to ER_FAILED). Unlike heap, btree keeps the raw VPID — the index already owns a chain.

3.9 Chapter summary — key takeaways

1. Page count is computed exactly, up front: npages = 1 + CEIL_PTVDIV(rest_bytes, rest_capacity) or 1. The first page holds sizeof(int) fewer payload bytes (it carries length) — a load-bearing asymmetry, proven exact by assert(length == 0).
2. A 64-slot stack buffer avoids malloc in the common case. Only npages > OVERFLOW_ALLOCVPID_ARRAY_SIZE mallocs (direct-return OOM); vpids != vpids_buffer is the sole free signal; the vpids[npages] sentinel terminates the last next_vpid in every build.
3. The chain is built inside one system operation — log_sysop_start … file_alloc_multiple … loop … log_sysop_attach_to_outer on success / log_sysop_abort on error: all-or-nothing.
4. The loop’s only structural branch is first-part vs rest-part. Page 0 writes next_vpid + total length + payload; rest pages drop length. Each iteration fixes, writes, logs, and frees one page — a single-page latch footprint.
5. Logging is file-type gated; the wrappers only marshal arguments. LOG_DUMMY_OVF_RECORD once on page 0, RVOVF_NEWPAGE_INSERT per page, both only for non-temp files (redo also gated on no_logging). heap_ovf_insert picks FILE_MULTIPAGE_OBJECT_HEAP and returns an OID with NULL_SLOTID; btree_store_overflow_key picks FILE_BTREE_OVERFLOW_KEY, head VPID kept raw.

Chapter 4: Reading and Traversing a Chain

Chapter 3 built a chain: a head OVERFLOW_FIRST_PART carrying the total length, then zero or more OVERFLOW_REST_PART pages linked by next_vpid until NULL. This chapter answers: given the head VPID, how does a caller read all or part of the chain, and how is the walk kept safe? The read surface is small — three public entries, a diagnostic walk, a flush walk, two internals — and none takes a lock: the home reference’s row/key lock is the access gate (companion §“Pages not individually locked”).

4.1 The two structs the reader walks over

No new struct is introduced; reading consumes what Chapter 1 defined (overflow_first_part = {VPID next_vpid; int length; char data[1];}, overflow_rest_part = {VPID next_vpid; char data[1];}). For full per-field rationale see Chapter 1; the read-role of each field is:

Field	Role	Why
overflow_first_part.next_vpid	Link to page 2, followed once the head’s data[] is consumed	Read as offset-0 of the head only because VPID_EQ says so; NULL here means a one-page record
overflow_first_part.length	Authoritative total byte count, read O(1) up front	Head-only, so a length query never walks; drives the byte budget (4.4) and page-count walk (4.9)
overflow_first_part.data[]	First payload slice	Its leading OR_MVCC_MAX_HEADER_SIZE bytes are the MVCC header the snapshot gate reads
overflow_rest_part.next_vpid	Link to the next rest page	Same offset-0 position; NULL marks chain end — a NULL seen while bytes remain is corruption (4.4)
overflow_rest_part.data[]	Payload continuation	Rest pages omit length, so each carries sizeof(int) more payload than the head

Invariant (head/rest discrimination by identity). A page is the head of this read iff its VPID equals the caller’s ovf_vpid — no “is-head” flag, only a VPID_EQ compare (overflow_next_vpid, 4.2). Pass a rest-page VPID and the code misreads the four length bytes as the start of data[], silently mis-walking — so every entry takes the first-page VPID from the home REC_BIGONE / leaf reference.

flowchart LR
  HEAD["head page == ovf_vpid\nnext_vpid -> R1\nlength = total\ndata[ MVCC hdr then body ]"]
  R1["rest page\nnext_vpid -> R2\ndata[ body ]"]
  R2["rest page\nnext_vpid = NULL\ndata[ body end ]"]
  HEAD -->|next_vpid| R1 -->|next_vpid| R2

Figure 4-1 — What a reader sees: the head is distinguished only by equality with the caller’s VPID, and length lives there alone.

4.2 overflow_next_vpid — the head-vs-rest selector

// overflow_next_vpid -- src/storage/overflow_file.c
if (VPID_EQ (ovf_vpid, vpid))      /* <- still the original head? */
  *vpid = ((OVERFLOW_FIRST_PART *) pgptr)->next_vpid;   /* link sits after length */
else
  *vpid = ((OVERFLOW_REST_PART *) pgptr)->next_vpid;    /* rest layout, offset 0 */

Two branches, no error path. Only overflow_traverse calls it (it re-walks from ovf_vpid and must ask “still on the head?” each iteration); overflow_get_nbytes and overflow_get_capacity hard-code the discrimination inline — head before the loop, loop body touches only rest pages.

4.3 overflow_get_length — O(1) head-only read

// overflow_get_length -- src/storage/overflow_file.c
pgptr = pgbuf_fix (thread_p, ovf_vpid, OLD_PAGE, PGBUF_LATCH_READ, PGBUF_UNCONDITIONAL_LATCH);
if (pgptr == NULL) return -1;                       /* <- only failure branch */
length = ((OVERFLOW_FIRST_PART *) pgptr)->length;   /* head-only field */
pgbuf_unfix_and_init (thread_p, pgptr);
return length;

One -1 error exit, one success path, PGBUF_LATCH_READ since the field is read-only. The -1 sentinel is what btree_load_overflow_key (4.7) checks before sizing its buffer.

4.4 overflow_get_nbytes — the partial read, branch by branch

The heart of the chapter: read max_nbytes bytes from start_offset, apply the MVCC gate, report a buffer-too-small hint, copy across pages. overflow_get (4.5) and heap_ovf_get (4.6) wrap it.

Step 1 — fix the head with pgbuf_fix; one branch, fix failure → S_ERROR.

Step 2 — the MVCC gate (heap path only), run only with a snapshot; heap_get_mvcc_rec_header_from_overflow decodes the head’s data[] by pointing a RECDES at overflow_get_first_page_data (pgptr) (the real accessor, returning ((OVERFLOW_FIRST_PART *) pgptr)->data) then or_mvcc_get_header.

// overflow_get_nbytes -- src/storage/overflow_file.c
if (mvcc_snapshot != NULL)
  {
    heap_get_mvcc_rec_header_from_overflow (pgptr, &mvcc_header, NULL);  /* decode head data[] */
    if (mvcc_snapshot->snapshot_fnc (thread_p, &mvcc_header, mvcc_snapshot) == TOO_OLD_FOR_SNAPSHOT)
      {
        /* not satisfied ONLY on TOO_OLD; TOO_NEW (e.g. just-updated record locked at select) is accepted */
        pgbuf_unfix_and_init (thread_p, pgptr);
        return S_SNAPSHOT_NOT_SATISFIED;
      }
  }

Of the three verdicts only TOO_OLD_FOR_SNAPSHOT aborts; TOO_NEW and satisfied fall through so a SELECT ... FOR UPDATE against a just-updated big record still sees the candidate to lock it. mvcc_snapshot == NULL (overflow-key path, 4.7) skips the block.

Steps 3-4 — byte budget, then buffer-size verdict.

// overflow_get_nbytes -- src/storage/overflow_file.c
*remaining_length = first_part->length;
if (max_nbytes < 0)                              max_nbytes = *remaining_length - start_offset;  /* "rest from offset" */
else if (max_nbytes > *remaining_length - start_offset) max_nbytes = *remaining_length - start_offset; /* clamp */
if (max_nbytes < 0) { max_nbytes = 0; *remaining_length = 0; }   /* offset past record end -> empty */
else                  *remaining_length -= max_nbytes;
// ... condensed ...
if (max_nbytes > recdes->area_size)                                                  /* too small */
  { pgbuf_unfix_and_init (...); recdes->length = -max_nbytes; return S_DOESNT_FIT; }  /* neg = needed bytes */
else if (max_nbytes == 0)                                                            /* empty / past-end */
  { pgbuf_unfix_and_init (...); recdes->length = 0; return S_SUCCESS; }
recdes->length = max_nbytes;

All four budget/verdict branches are annotated inline; the load-bearing detail is that S_DOESNT_FIT carries the needed size as a negative recdes->length (the caller retries on it — 4.6).

Step 5 — the seek-then-copy loop, starting on the head’s data[].

// overflow_get_nbytes -- src/storage/overflow_file.c (loop, condensed)
data = recdes->data; copyfrom = (char *) first_part->data; next_vpid = first_part->next_vpid;
while (max_nbytes > 0)
  {
    if (start_offset > 0)   /* seek: copy_length = min(start_offset, bytes-left-on-page); advance copyfrom */
      { ... ; start_offset -= copy_length; copyfrom += copy_length; }
    if (start_offset == 0)  /* copy: copy_length = min(max_nbytes, bytes-left-on-page), page-bounded */
      { ... ; if (copy_length > 0) { memcpy (data, copyfrom, copy_length); data += copy_length; max_nbytes -= copy_length; } }
    pgbuf_unfix_and_init (thread_p, pgptr);   /* <- unfix BEFORE fixing the next page */
    if (max_nbytes > 0)
      {
        if (VPID_ISNULL (&next_vpid))         /* corruption guard, see callout */
          { er_set (..., ER_HEAP_OVFADDRESS_CORRUPTED, 3, ovf_vpid->volid, ovf_vpid->pageid, NULL_SLOTID); return S_ERROR; }
        pgptr = pgbuf_fix (thread_p, &next_vpid, OLD_PAGE, PGBUF_LATCH_READ, PGBUF_UNCONDITIONAL_LATCH);
        if (pgptr == NULL) { recdes->length = 0; return S_ERROR; }
        rest_parts = (OVERFLOW_REST_PART *) pgptr;   /* every later page is a rest page */
        copyfrom = (char *) rest_parts->data; next_vpid = rest_parts->next_vpid;
      }
  }
return S_SUCCESS;

Each ... clamps copy_length to the bytes left on the fixed page; seek and copy chain within one iteration (seek-to-page-end, then fall through).

Invariant (one page latched at a time; chain ≥ length bytes). Each page is pgbuf_unfix_and_init’d before the next is fixed — bounding the read-latch footprint and avoiding latch-ordering deadlocks, with the home reference’s lock serializing structural change across the unfix-N/fix-(N+1) window (companion Open Question 5). And a NULL next_vpid while max_nbytes > 0 means the chain ended before the advertised length was delivered, so the code raises ER_HEAP_OVFADDRESS_CORRUPTED — the dual of the writers’ obligation to keep length and page count consistent.

4.5 overflow_get — the (0, -1) wrapper

A one-line wrapper: overflow_get returns overflow_get_nbytes (..., 0, -1, &remaining_length, mvcc_snapshot) — start_offset = 0, max_nbytes = -1 (“everything”), throwaway remaining_length. It is the whole-record read behind heap_ovf_get (4.6) and btree_load_overflow_key (4.7).

4.6 The heap read callers

heap_ovf_get turns the REC_BIGONE OID into a VPID and adds a CHN (cache-coherency-number) fast path:

// heap_ovf_get -- src/storage/heap_file.c (condensed)
ovf_vpid.pageid = ovf_oid->pageid; ovf_vpid.volid = ovf_oid->volid;
if (chn != NULL_CHN)                  /* peek only the MVCC-header prefix to compare CHN */
  {
    scan = overflow_get_nbytes (thread_p, &ovf_vpid, recdes, 0, OR_MVCC_MAX_HEADER_SIZE, &rest_length, mvcc_snapshot);
    if (scan == S_SUCCESS && chn == or_chn (recdes))
      return S_SUCCESS_CHN_UPTODATE;   /* cached copy current; skip full read */
  }
return overflow_get (thread_p, &ovf_vpid, recdes, mvcc_snapshot);   /* full read */

Only S_SUCCESS + CHN match short-circuits to S_SUCCESS_CHN_UPTODATE; anything else (short-read S_SNAPSHOT_NOT_SATISFIED / S_DOESNT_FIT, or NULL_CHN) falls through to the full overflow_get. heap_get_bigone_content owns the S_DOESNT_FIT retry loop:

// heap_get_bigone_content -- src/storage/heap_file.c (condensed)
if (scan_cache != NULL && (ispeeking == PEEK || recdes->data == NULL || ...))   /* reusable area available */
  {
    scan_cache->assign_recdes_to_area (*recdes);
    while ((scan = heap_ovf_get (thread_p, forward_oid, recdes, NULL_CHN, NULL)) == S_DOESNT_FIT)
      scan_cache->assign_recdes_to_area (*recdes, (size_t) (-recdes->length));  /* grow to needed size, retry */
    if (scan != S_SUCCESS) recdes->data = NULL;
  }
else
  scan = heap_ovf_get (thread_p, forward_oid, recdes, NULL_CHN, NULL);   /* caller-owned buffer, no retry */
return scan;

Two branches, annotated inline: the reusable-area branch loops on S_DOESNT_FIT, the caller-buffer branch does not retry. Both pass NULL_CHN and mvcc_snapshot = NULL — the snapshot was already checked upstream by the visible-version walk that reached REC_BIGONE (companion §“Walk: heap read of a big record”).

4.7 btree_load_overflow_key — size, allocate, read, always copy

The only non-heap consumer:

// btree_load_overflow_key -- src/storage/btree.c (condensed)
rec.area_size = overflow_get_length (thread_p, first_overflow_page_vpid);
if (rec.area_size == -1) return ER_FAILED;                       /* length-query failure */
rec.data = (char *) db_private_alloc (thread_p, rec.area_size);  /* NULL -> ER_OUT_OF_VIRTUAL_MEMORY */
if (overflow_get (thread_p, first_overflow_page_vpid, &rec, NULL) != S_SUCCESS) goto exit_on_error; /* NULL snapshot */
or_init (&buf, rec.data, rec.length);
pr_type->index_readval (&buf, key, btid->key_type, -1, true /*copy*/, NULL, 0);   /* always copy overflow keys */

Two non-obvious choices. mvcc_snapshot = NULL is correct, not a shortcut: an overflow key carries no MVCC header — visibility of the OID list attached to the key decides reachability. And index_readval (..., /*copy=*/true, ...) always copies, never peeks, because overflow_get unfixes the page before returning (overflow keys cannot peek under the leaf latch the way inline keys do); pre-sizing keeps this overflow_get from ever returning S_DOESNT_FIT.

4.8 overflow_traverse and overflow_flush — the worker skeleton

Delete (Chapter 6) and flush share overflow_traverse; here, only the OVERFLOW_DO_FLUSH arm.

// overflow_traverse -- src/storage/overflow_file.c (condensed)
next_vpid = *ovf_vpid;
while (!(VPID_ISNULL (&next_vpid)))
  {
    pgptr = pgbuf_fix (thread_p, &next_vpid, OLD_PAGE, PGBUF_LATCH_WRITE, PGBUF_UNCONDITIONAL_LATCH);
    if (pgptr == NULL) goto exit_on_error;
    vpid = next_vpid;
    overflow_next_vpid (ovf_vpid, &next_vpid, pgptr);   /* read link BEFORE acting on page */
    switch (func)
      {
      case OVERFLOW_DO_DELETE: ... /* needs ovf_vfid != NULL; deallocates page -- Chapter 6 */
      case OVERFLOW_DO_FLUSH:
        if (overflow_flush_internal (thread_p, pgptr) != NO_ERROR) goto exit_on_error;
        break;
      default: break;
      }
  }
return ovf_vpid;

The walk reads the next link via overflow_next_vpid before it hands the page to the callback — essential for delete (the page is about to be freed), uniform for both arms. overflow_flush_internal is one branch (pgbuf_flush_with_wal, WAL: log before page); overflow_flush is the void wrapper overflow_traverse (..., NULL, ovf_vpid, OVERFLOW_DO_FLUSH), fixing pages PGBUF_LATCH_WRITE even to flush, the DELETE arm’s ovf_vfid != NULL guard never firing. The exit_on_error: /* TODO: suspect pgbuf_unfix */ is a known rough edge: the last fixed page may leak its fix (companion Cross-check).

4.9 overflow_get_capacity — the full diagnostic walk

Capacity walks every page driven by length (not by chasing next_vpid to NULL), reporting size, page count, header overhead, and trailing free space:

// overflow_get_capacity -- src/storage/overflow_file.c (condensed)
*ovf_size = remain_length = first_part->length;          /* size output = total length */
*ovf_num_pages = *ovf_overhead = *ovf_free_space = 0;    /* other three outputs start zeroed */
hdr_length = offsetof (OVERFLOW_FIRST_PART, data); next_vpid = first_part->next_vpid;  /* head header larger */
while (remain_length > 0)
  {
    if (remain_length > DB_PAGESIZE)
      remain_length -= DB_PAGESIZE - hdr_length;          /* not last page: subtract payload capacity */
    else
      { *ovf_free_space = DB_PAGESIZE - remain_length; remain_length = 0; }  /* last page: tail is free */
    *ovf_num_pages += 1; *ovf_overhead += hdr_length;
    if (remain_length > 0)   /* unfix; same VPID_ISNULL guard + fix as 4.4; then: */
      { hdr_length = offsetof (OVERFLOW_REST_PART, data); next_vpid = ((OVERFLOW_REST_PART *) pgptr)->next_vpid; }
  }
pgbuf_unfix_and_init (thread_p, pgptr);

The three branches (not-last / last / next-page fix) are annotated inline; the next-page fix shares 4.4’s VPID_ISNULL guard and fix-failure exit, hdr_length switches head→rest on iteration two so overhead stays exact, and exit_on_error zeroes all four outputs so a partial walk never reports half-counted capacity. Like overflow_get_nbytes, capacity trusts length over the link chain.

4.10 Chapter summary — key takeaways

1. length lives on the head alone — overflow_get_length is one fix-read-unfix; every other reader reads length once up front.
2. Head vs rest is VPID identity (VPID_EQ), not a flag; inline readers handle the head pre-loop, then every looped page is a rest.
3. The MVCC gate aborts only on TOO_OLD_FOR_SNAPSHOT — TOO_NEW lets SELECT ... FOR UPDATE lock a fresh big record; skipped when mvcc_snapshot == NULL (overflow keys).
4. Buffer-too-small is out-of-band — S_DOESNT_FIT + negative recdes->length; heap_get_bigone_content loops to grow, btree_load_overflow_key avoids it by sizing first.
5. One page latched at a time (unfix-before-fix), relying on the home reference’s lock for chain stability.
6. A NULL next_vpid before length bytes are consumed is corruption — both readers raise ER_HEAP_OVFADDRESS_CORRUPTED.
7. overflow_flush is the benign arm of overflow_traverse (pgbuf_flush_with_wal per page, no VFID); DELETE is Chapter 6.

Chapter 5: Updating a Chain in Place with overflow_update

The reader question: when the new value is rewritten longer or shorter than the one stored, how is the existing chain grown or shrunk page by page — without tearing it down and rebuilding it the way overflow_insert (Chapter 3) and overflow_delete (Chapter 6) would?

The answer is overflow_update: it walks the chain head-to-tail, overwrites each page’s payload in place, and touches the topology (next_vpid links, allocation) only at the transition point where the new length runs out or over. This chapter dissects every branch. The OVERFLOW_FIRST_PART / OVERFLOW_REST_PART fields are in Chapter 1.

5.1 Why only the heap calls this — the FILE_MULTIPAGE_OBJECT_HEAP assert

overflow_update opens with two asserts fencing its callers:

// overflow_update -- src/storage/overflow_file.c
assert (ovf_vfid != NULL && !VFID_ISNULL (ovf_vfid));
/* used only for heap for now... this doesn't consider temporary files. -- author's comment, condensed */
assert (file_type == FILE_MULTIPAGE_OBJECT_HEAP);   /* <- in-place update is a heap-only path */

The B+Tree OID-list variant (Chapter 7) never updates in place; key/OID changes are delete + reinsert. Only the heap’s REC_BIGONE records reach overflow_update, via heap_ovf_update. The assert encodes the contract: the function does not handle temporary files (which need a different page-init function) and has only been audited for the permanent heap-overflow type. The structural consequence, invariant throughout, is that the head page is never freed — see 5.5.

5.2 heap_ovf_update — the thin wrapper

heap_ovf_update adapts the heap’s OID world to VFID/VPID:

// heap_ovf_update -- src/storage/heap_file.c
if (heap_ovf_find_vfid (thread_p, hfid, &ovf_vfid, false, PGBUF_UNCONDITIONAL_LATCH) == NULL)
  return NULL;                                  /* <- could not resolve overflow file VFID */
ovf_vpid.pageid = ovf_oid->pageid;
ovf_vpid.volid = ovf_oid->volid;                /* <- head VPID lives in the OID */
if (overflow_update (thread_p, &ovf_vfid, &ovf_vpid, recdes, FILE_MULTIPAGE_OBJECT_HEAP) != NO_ERROR)
  { ASSERT_ERROR (); return NULL; }             /* <- propagate failure as NULL */
else
  return ovf_oid;                               /* <- OID unchanged: head never moved */

Note the false argument to heap_ovf_find_vfid (do not create the file — it must already exist) and that on success the same ovf_oid is returned: the head being immutable, the relocation record never needs rewriting. The caller is heap_update_physical’s REC_BIGONE -> REC_BIGONE branch, reached only when heap_is_big_length still holds for the new image.

The chain topology (REC_BIGONE slot -> head OVERFLOW_FIRST_PART -> OVERFLOW_REST_PART … -> NULL tail) is the Chapter 1 data-structure map; in-place update mutates each page’s data[] and selectively rewrites next_vpid while the head VPID in the REC_BIGONE slot never changes.

5.3 The main loop skeleton and per-iteration state

The body is one while (length > 0) loop bracketed by a sysop:

// overflow_update -- src/storage/overflow_file.c
next_vpid = *ovf_vpid;        /* <- start at the head */
data = recdes->data;
length = recdes->length;      /* <- bytes still to write */
log_sysop_start (thread_p);   /* <- atomic bracket; see 5.6 */

while (length > 0)
  {
    addr.pgptr = pgbuf_fix (thread_p, &next_vpid, OLD_PAGE, PGBUF_LATCH_WRITE, PGBUF_UNCONDITIONAL_LATCH);
    if (addr.pgptr == NULL) { ASSERT_ERROR_AND_SET (error_code); goto exit_on_error; }
    addr_vpid_ptr = pgbuf_get_vpid_ptr (addr.pgptr);
    /* ... first-page branch vs rest-page branch ... */
  }

Four locals carry state across iterations and select the branch:

Local	Role	Why it exists
length	new-image bytes still unwritten	loop guard; 0 triggers the terminate/shrink path
old_length	old-image bytes not yet covered by an undo before-image	drives how much before-image to log per page
isnewpage	true once this update has had to file_alloc a page	tells the rest-branch to NULL-terminate the new page
rest_parts	non-NULL once past the head	links via first_part vs rest_parts at grow/terminate

The before-image split rule (referenced throughout 5.4). old_length is read once, on the head, from first_part->length. On each page still carrying old payload the code logs an RVOVF_PAGE_UPDATE undo and decrements old_length by what that page held — a full DB_PAGESIZE if the rest overflows, else the exact hdr_length + old_length (then old_length = 0). Logging the old byte count is what lets undo restore the original page exactly, and tracking it per page lets a shorter new image still restore the longer old image. Both 5.4 branches reuse this split verbatim; the head additionally seeds old_length.

5.4 Inside the loop: the two page branches

VPID_EQ (addr_vpid_ptr, ovf_vpid) decides whether the page just fixed is the head.

5.4.1 First-page branch

// overflow_update (first-page branch) -- src/storage/overflow_file.c
first_part = (OVERFLOW_FIRST_PART *) addr.pgptr;
old_length = first_part->length;              /* <- whole old image length, read once (seeds 5.3 rule) */
copyto = (char *) first_part->data;
next_vpid = first_part->next_vpid;            /* <- remember existing successor */
hdr_length = offsetof (OVERFLOW_FIRST_PART, data);

/* copy_length = min(length, DB_PAGESIZE - hdr_length): fill page if more follows, else exact fit */
if ((length + hdr_length) > DB_PAGESIZE) copy_length = DB_PAGESIZE - hdr_length;
else                                     copy_length = length;

/* Before-image, logged unconditionally (head always held payload) -- the 5.3 split: */
if (hdr_length + old_length > DB_PAGESIZE)        /* old image spills past the head */
  { log_append_undo_data (thread_p, RVOVF_PAGE_UPDATE, &addr, DB_PAGESIZE, addr.pgptr);
    old_length -= DB_PAGESIZE - hdr_length; }      /* <- one full page of old payload consumed */
else  /* old image fits on the head: log hdr_length+old_length bytes, then old_length = 0 */
  { log_append_undo_data (thread_p, RVOVF_PAGE_UPDATE, &addr, hdr_length + old_length, addr.pgptr);
    old_length = 0; }

first_part->length = length;                  /* <- store the NEW total length on the head */
log_append_empty_record (thread_p, LOG_DUMMY_OVF_RECORD, &addr);

first_part->length becomes the new total length (only the head carries length). The trailing LOG_DUMMY_OVF_RECORD re-emits the marker recovery/scan uses to recognize a freshly-stamped overflow head — Chapter 8 covers why it matters on redo.

5.4.2 Rest-page branch

// overflow_update (rest-page branch) -- src/storage/overflow_file.c
rest_parts = (OVERFLOW_REST_PART *) addr.pgptr;
copyto = (char *) rest_parts->data;
if (isnewpage == true)
  {
    VPID_SET_NULL (&next_vpid);
    rest_parts->next_vpid = next_vpid;        /* <- a page WE just allocated terminates the chain */
  }
else
  next_vpid = rest_parts->next_vpid;          /* <- pre-existing page: keep following the old links */

hdr_length = offsetof (OVERFLOW_REST_PART, data);
/* copy_length computed exactly as in 5.4.1 (fill page vs fits) */

if (old_length > 0)                           /* <- only log undo if old payload still reaches here */
  {
    /* ... identical before-image split as 5.3 / 5.4.1 ... */
  }

isnewpage drives the grow path: a page just allocated by 5.4.3 has garbage in its next_vpid, so this branch NULL-terminates it (a crash mid-grow then cannot point into an uninitialized link); a pre-existing page reads next_vpid to follow the old topology. The if (old_length > 0) guard is the shrink counterpart — once every old byte is covered by earlier before-images, tail pages about to be deallocated (5.4.4) log no undo; the link opcodes and file_dealloc carry it.

5.4.3 Shared payload copy, then grow-or-terminate

After either branch sets copyto/copy_length/hdr_length:

// overflow_update (shared) -- src/storage/overflow_file.c
memcpy (copyto, data, copy_length);   /* <- overwrite payload in place */
data += copy_length;  length -= copy_length;
if (thread_p->no_logging != true)
  log_append_redo_data (thread_p, RVOVF_PAGE_UPDATE, &addr, copy_length + hdr_length, (char *) addr.pgptr);

The redo logs copy_length + hdr_length bytes — header plus new payload — so it reconstructs the post-update page from the after-image. (no_logging is the bulk-load fast path; it suppresses redo only.) The fork then decides chain topology:

// overflow_update (grow path) -- src/storage/overflow_file.c
if (length > 0)
  {
    if (VPID_ISNULL (&next_vpid))         /* <- ran out of existing pages: GROW */
      {
        error_code = file_alloc (thread_p, ovf_vfid, file_init_page_type, &ptype, &next_vpid, NULL);
        if (error_code != NO_ERROR)
          { ASSERT_ERROR (); pgbuf_set_dirty_and_free (thread_p, addr.pgptr); goto exit_on_error; }

        log_append_undoredo_data (thread_p, RVOVF_NEWPAGE_LINK, &addr, 0, sizeof (next_vpid), NULL, &next_vpid);
        isnewpage = true;                 /* <- next iteration NULL-terminates the new page */

        if (rest_parts == NULL)
          first_part->next_vpid = next_vpid;   /* <- still on head: link grows from first_part */
        else
          rest_parts->next_vpid = next_vpid;   /* <- link grows from current rest page */
      }
    pgbuf_set_dirty_and_free (thread_p, addr.pgptr);
  }

Grow fires only when payload remains and next_vpid is NULL. The RVOVF_NEWPAGE_LINK undoredo has a 0-length undo — undo of a new link is “no link”, and file_alloc’s own records reclaim the page on rollback. The new VPID is linked from first_part (still on the head) or rest_parts; if next_vpid was non-NULL, allocation is skipped and we advance to the existing successor. On file_alloc failure the page is set dirty and freed before goto exit_on_error (no latch leak); the sysop abort then rolls back (5.6).

5.4.4 Terminate-and-shrink path

// overflow_update (terminate/shrink) -- src/storage/overflow_file.c
else  /* length == 0 : no more payload */
  {
    VPID_SET_NULL (&tmp_vpid);
    log_append_undoredo_data (thread_p, RVOVF_CHANGE_LINK, &addr, sizeof (next_vpid), sizeof (next_vpid),
                              &next_vpid, &tmp_vpid);   /* <- undo restores old next_vpid; redo sets NULL */

    if (rest_parts == NULL)
      VPID_SET_NULL (&first_part->next_vpid);   /* <- single-page result: head's link cut */
    else
      VPID_SET_NULL (&rest_parts->next_vpid);   /* <- cut after the last written rest page */
    pgbuf_set_dirty_and_free (thread_p, addr.pgptr);

    while (!(VPID_ISNULL (&next_vpid)))          /* <- free every surplus tail page */
      {
        addr.pgptr = pgbuf_fix (...);             // ... fix; NULL -> exit_on_error ...
        tmp_vpid = next_vpid;
        rest_parts = (OVERFLOW_REST_PART *) addr.pgptr;
        next_vpid = rest_parts->next_vpid;        /* <- read successor BEFORE unfixing */
        pgbuf_unfix_and_init (thread_p, addr.pgptr);
        error_code = file_dealloc (thread_p, ovf_vfid, &tmp_vpid, file_type);
        if (error_code != NO_ERROR) { ASSERT_ERROR (); goto exit_on_error; }
      }
    break;                                        /* <- leave the while(length>0) loop */
  }

This branch runs once, at length == 0. RVOVF_CHANGE_LINK logs the old next_vpid as undo and NULL as redo; the link is NULLed on the last written page (the head if rest_parts is NULL — value fit in one page — else the last rest page). The tail-dealloc loop then frees each surplus page, reading next_vpid before unfixing so the chain survives the free, exiting at NULL; break leaves the outer loop. Any failure jumps to the sysop abort.

Figure 5-1 traces the function.

flowchart TD
  S["sysop start; start at head"] --> L{"length > 0 ?"}
  L -- no --> ATT["attach to outer; return NO_ERROR"]
  L -- yes --> FIX["fix next_vpid WRITE"]
  FIX -- NULL --> ERR["exit_on_error"]
  FIX --> HEAD{"VPID_EQ head ?"}
  HEAD -- yes --> FP["first-page branch"]
  HEAD -- no --> RP["rest-page branch"]
  FP --> CP["memcpy; redo after-image"]
  RP --> CP
  CP --> M{"length > 0 ?"}
  M -- yes --> G{"next_vpid NULL ?"}
  G -- yes --> ALLOC["grow: file_alloc; link"]
  G -- no --> DF["reuse successor; dirty+free; loop"]
  ALLOC --> DF
  DF --> L
  M -- no --> TERM["terminate: cut link"]
  TERM --> TL{"next_vpid NULL ?"}
  TL -- no --> DEA["shrink: free tail page"]
  DEA --> TL
  TL -- yes --> ATT
  ERR --> AB["sysop abort; return error"]

Figure 5-1. overflow_update control flow — every branch and error exit.

5.5 The head-never-removed invariant

Invariant: a heap overflow chain always retains at least its head page (OVERFLOW_FIRST_PART) for the value’s lifetime. overflow_update enforces it structurally: the head is the first page fixed, always receives the leading copy_length bytes, and the 5.4.4 tail-dealloc loop frees pages only after the last written page (starting from next_vpid, never the head). Even when the image collapses to a few bytes, the head survives with next_vpid NULLed. If a shrink could free the head, the REC_BIGONE OID would dangle and every reader resolving it (Chapter 4) would fault — keeping the head pinned is what lets heap_ovf_update return the unchanged ovf_oid.

5.6 The system-operation bracket

The update is one nested atomic unit:

// overflow_update -- src/storage/overflow_file.c
log_sysop_start (thread_p);
/* ... while loop, allocations, deallocations ... */
log_sysop_attach_to_outer (thread_p);  return NO_ERROR;   /* success: splice into enclosing txn */
exit_on_error:
log_sysop_abort (thread_p);  return error_code;           /* failure: undo writes, alloc, dealloc */

On success the sysop is attached to the outer transaction, not committed standalone — every page rewrite, link change, and file_alloc/file_dealloc becomes part of the surrounding heap update. On any error (pgbuf_fix NULL, file_alloc/file_dealloc failure) goto exit_on_error runs log_sysop_abort, replaying the undo logged so far and leaving the chain exactly as before. Chapter 8 details how each opcode named in 5.4 applies at recovery.

5.7 Chapter summary — key takeaways

1. In-place update is heap-only. assert (file_type == FILE_MULTIPAGE_OBJECT_HEAP) fences out everyone else; B+Tree value changes are delete+reinsert.
2. The walk overwrites payload via memcpy page by page, logging a per-page RVOVF_PAGE_UPDATE undo (before-image) and redo (after-image); only the head carries length.
3. old_length drives the undo via the single split rule in 5.3 (seeded on the head, decremented per page); if (old_length > 0) suppresses undo on doomed tail pages, so a shorter image still restores the original.
4. Grow allocates a page (file_alloc + RVOVF_NEWPAGE_LINK) only when payload remains and next_vpid is NULL, sets isnewpage, and links via first_part or rest_parts.
5. Shrink/terminate fires once at length == 0: RVOVF_CHANGE_LINK cuts the trailing link, then a tail loop fixes, reads-ahead, unfixes, and file_deallocs each surplus page.
6. The head is never freed (5.5), so the REC_BIGONE OID stays valid and heap_ovf_update returns the unchanged OID.
7. The whole operation is one sysop: log_sysop_attach_to_outer on success, log_sysop_abort on every error path — atomic with the enclosing transaction.

Chapter 6: Destroying and Reclaiming a Chain

When a big-record value is finally gone, how is every page of its overflow chain freed, and who pulls the trigger under MVCC? The mechanical reclamation is short — overflow_delete → overflow_traverse(OVERFLOW_DO_DELETE) → per-page file_dealloc. The interesting part is the caller side: an MVCC delete of a REC_BIGONE does not free the chain; it stamps a delete-MVCCID into the head overflow page and leaves the home heap slot untouched, so the chain survives until vacuum (or a non-MVCC delete) decides nobody can see the row. For chain layout see Chapter 1; construction Chapter 3; the per-page-no-lock rationale in the companion (cubrid-overflow-file.md).

6.1 The reclamation core — overflow_delete and its traversal

overflow_delete hands off to the shared walk-and-callback worker overflow_traverse with the OVERFLOW_DO_DELETE discriminator (also used for OVERFLOW_DO_FLUSH). It fixes each page, reads the next link before acting on the current page, then dispatches:

// overflow_traverse -- src/storage/overflow_file.c (condensed)
next_vpid = *ovf_vpid;
while (!(VPID_ISNULL (&next_vpid)))
  {
    pgptr = pgbuf_fix (thread_p, &next_vpid, OLD_PAGE, PGBUF_LATCH_WRITE, PGBUF_UNCONDITIONAL_LATCH);
    if (pgptr == NULL)
      { goto exit_on_error; }                            /* <- fix failed: return NULL */
    vpid = next_vpid;
    overflow_next_vpid (ovf_vpid, &next_vpid, pgptr);    /* <- read link BEFORE freeing */
    switch (func)
      {
      case OVERFLOW_DO_DELETE:
        if (ovf_vfid == NULL)                            /* assert: dealloc needs the VFID */
          { goto exit_on_error; }
        if (overflow_delete_internal (thread_p, ovf_vfid, &vpid, pgptr) != NO_ERROR)
          { goto exit_on_error; }
        break;
      case OVERFLOW_DO_FLUSH: /* ... condensed ... */ break;
      default: break;
      }
  }
return ovf_vpid;

The real switch (func) has three arms — OVERFLOW_DO_DELETE, OVERFLOW_DO_FLUSH (the flush walk calls overflow_flush_internal), and a default: break; no-op — but the delete walk exercises only the first. Three distinct error exits funnel to exit_on_error → NULL (Figure 6-1): pgbuf_fix NULL (bail; abort is harmless because the free is postponed to commit, §6.2), the ovf_vfid == NULL assert (cannot dealloc without the owning file — per-branch because overflow_flush deliberately passes NULL), and overflow_delete_internal failure (propagate). The fourth outcome — the NULL next_vpid loop tail — is not an error exit; it falls out of the while and return ovf_vpid as the success sentinel. exit_on_error itself does no unfix (a flagged /* TODO: suspect pgbuf_unfix */), so a page fixed by the loop body leaks on the pgbuf_fix-success error paths.

Invariant — “read the next-page link before you free the current page.” overflow_traverse calls overflow_next_vpid before overflow_delete_internal unfixes and deallocs. Reversed, the walk would dereference a freed page for the next link — a use-after-free corrupting the chain tail.

(overflow_next_vpid is type-aware — casts to OVERFLOW_FIRST_PART at the head, OVERFLOW_REST_PART otherwise — but both carry next_vpid at the same leading offset, so the cast is cosmetic.)

flowchart TD
  A["overflow_delete(ovf_vfid, ovf_vpid)"] --> B["overflow_traverse(..., OVERFLOW_DO_DELETE)"]
  B --> C{"next_vpid is NULL?"}
  C -- "yes" --> Z["return ovf_vpid (success)"]
  C -- "no" --> D["pgbuf_fix(next_vpid, WRITE)"]
  D --> E{"pgptr == NULL?"}
  E -- "yes" --> X["exit_on_error -> return NULL"]
  E -- "no" --> F["vpid := next_vpid\noverflow_next_vpid -> advance next_vpid"]
  F --> G{"ovf_vfid == NULL?"}
  G -- "yes" --> X
  G -- "no" --> H["overflow_delete_internal(ovf_vfid, vpid, pgptr)"]
  H --> I{"NO_ERROR?"}
  I -- "no" --> X
  I -- "yes" --> C

Figure 6-1 — overflow_delete traversal: fix page, harvest next-link, dealloc; error branches funnel to exit_on_error.

6.2 Per-page reclamation — overflow_delete_internal and the FILE_UNKNOWN_TYPE TODO

The per-page worker is tiny — unfix, then file_dealloc:

// overflow_delete_internal -- src/storage/overflow_file.c
  pgbuf_unfix_and_init (thread_p, pgptr);          /* <- release latch+pin before dealloc */
  /* TODO: clarify file_type */
  ret = file_dealloc (thread_p, ovf_vfid, vpid, FILE_UNKNOWN_TYPE);   /* <- hint = UNKNOWN */
  if (ret != NO_ERROR)
    { goto exit_on_error; }
  return ret;
exit_on_error:
  return (ret == NO_ERROR && (ret = er_errid ()) == NO_ERROR) ? ER_FAILED : ret;

The page is unfixed before file_dealloc. The next link is already harvested, so no write-latch contention — safe because the chain is private to the record, whose higher-level lock blocks readers (companion, no per-page locks).

The FILE_UNKNOWN_TYPE hint and its TODO. The third argument is a hint. Passing FILE_UNKNOWN_TYPE makes file_dealloc fix the file header to read the real type (file_type_hint = fhead->type). The guard is #if defined (NDEBUG) wrapping only the if condition, so the header-fix branch’s reach is build-dependent:

// file_dealloc -- src/storage/file_manager.c
#if defined (NDEBUG)
  if (file_type_hint == FILE_UNKNOWN_TYPE
      || file_type_hint == FILE_EXTENDIBLE_HASH || file_type_hint == FILE_EXTENDIBLE_HASH_DIRECTORY)
#endif /* NDEBUG */
    {
      /* this should be avoided in release */
      page_fhead = pgbuf_fix (thread_p, &vpid_fhead, OLD_PAGE, PGBUF_LATCH_WRITE, PGBUF_UNCONDITIONAL_LATCH);
      fhead = (FILE_HEADER *) page_fhead;
      assert (file_type_hint == FILE_UNKNOWN_TYPE || file_type_hint == fhead->type /* ... */);
      file_type_hint = fhead->type;   /* <- hint replaced by the real type */
    }

In release (NDEBUG) the branch is gated — it fires only for FILE_UNKNOWN_TYPE and the two extendible-hash types. In debug the #if removes the condition, so the branch is unconditional — every file_dealloc reads the header to assert the hint. Either way, an overflow dealloc always passes FILE_UNKNOWN_TYPE, so the header fix always happens here. The TODO is thus a correctness-neutral cleanup: the caller knows the type (heap FILE_MULTIPAGE_OBJECT_HEAP, B+Tree-key FILE_BTREE_OVERFLOW_KEY) but does not thread it down. See companion Open Questions §2.

Why no per-page RVOVF_* undo/redo is emitted. Destruction emits no overflow-specific log record; recovery is file_dealloc’s — an RVFL_DEALLOC postpone for all files, plus RVFL_USER_PAGE_MARK_DELETE only in a separate numerable/user-page-table branch. Because RVFL_DEALLOC is a postpone, a permanent-file page is freed only at run postpone (after commit), so no abort needs it back — hence §6.1’s error paths return NULL mid-walk with no rollback (nothing was physically freed). An RVOVF_* record would be redundant.

6.3 The heap MVCC asymmetry — an MVCC delete does NOT free the chain

heap_delete_bigone splits on is_mvcc_op; under MVCC it never calls heap_ovf_delete — it reads the MVCC header from the overflow page, stamps the delete-MVCCID back, and leaves the home slot a live REC_BIGONE:

// heap_delete_bigone -- src/storage/heap_file.c (is_mvcc_op branch, condensed)
overflow_oid = *((OID *) context->home_recdes.data);          /* home slot holds only the OID */
/* ... fix overflow page ... */
if (heap_get_mvcc_rec_header_from_overflow (..., &overflow_header, NULL) != NO_ERROR)
  { return ER_FAILED; }
heap_mvcc_log_delete (thread_p, &log_addr, RVHF_MVCC_DELETE_OVERFLOW);
heap_delete_adjust_header (&overflow_header, mvcc_id, false);  /* <- stamp DELID */
heap_set_mvcc_rec_header_on_overflow (..., &overflow_header);  /* write overflow page */
heap_mvcc_log_home_no_change (thread_p, &log_addr);            /* <- home slot UNCHANGED */

heap_delete_adjust_header is the stamp (MVCC_SET_FLAG_BITS (..., OR_MVCC_FLAG_VALID_DELID) then MVCC_SET_DELID); its third arg need_mvcc_header_max_size is false because the header is already at max size (§6.4). The symmetric peek heap_get_mvcc_rec_header_from_overflow reads the first OR_MVCC_MAX_HEADER_SIZE bytes of the head page’s data[].

Invariant — “an MVCC delete of a big record mutates only the head overflow page; the home heap slot stays a live REC_BIGONE.” Enforced by the is_mvcc_op branch: the only writes are heap_set_mvcc_rec_header_on_overflow on the overflow page plus a no-change record (heap_mvcc_log_home_no_change) advancing the home page’s max-MVCCID / vacuum-status so vacuum revisits. Freeing the chain here would let a reader whose snapshot predates mvcc_id fix a freed overflow page — a visibility failure.

6.4 The forced-max-header invariant — heap_set_mvcc_rec_header_on_overflow

An in-place MVCC delete works only if stamping a DELID never changes the header’s byte size, so CUBRID forces the head’s MVCC header to maximum size always:

// heap_set_mvcc_rec_header_on_overflow -- src/storage/heap_file.c
  ovf_recdes.data = overflow_get_first_page_data (ovf_page);
  ovf_recdes.area_size = ovf_recdes.length = OR_HEADER_SIZE (ovf_recdes.data);
  assert (ovf_recdes.length == OR_MVCC_MAX_HEADER_SIZE);          /* <- size is fixed */
  if (!MVCC_IS_FLAG_SET (mvcc_header, OR_MVCC_FLAG_VALID_INSID))  /* pad missing INSID */
    { MVCC_SET_FLAG_BITS (mvcc_header, OR_MVCC_FLAG_VALID_INSID); MVCC_SET_INSID (mvcc_header, MVCCID_ALL_VISIBLE); }
  if (!MVCC_IS_FLAG_SET (mvcc_header, OR_MVCC_FLAG_VALID_DELID))  /* pad missing DELID */
    { MVCC_SET_FLAG_BITS (mvcc_header, OR_MVCC_FLAG_VALID_DELID); MVCC_SET_DELID (mvcc_header, MVCCID_NULL); }
  return or_mvcc_set_header (&ovf_recdes, mvcc_header);

Invariant — “the head overflow page’s MVCC header is always OR_MVCC_MAX_HEADER_SIZE.” Enforced by padding a missing INSID with MVCCID_ALL_VISIBLE and a missing DELID with MVCCID_NULL, then asserting the size. This lets §6.3 stamp a DELID in place with no body shift; a short header would corrupt the body or mis-parse on read-back.

6.5 The actual free — non-MVCC delete and vacuum

The else (non-MVCC) branch — reached by a non-MVCC-class delete or by vacuum once the §6.3 delete-MVCCID has aged past every snapshot — reclaims the home slot first, then the chain:

// heap_delete_bigone -- src/storage/heap_file.c (non-MVCC branch, condensed)
bool is_reusable = heap_is_reusable_oid (context->file_type);
heap_log_delete_physical (..., &context->home_recdes, is_reusable, NULL);   /* home-slot log */
rc = heap_delete_physical (thread_p, &context->hfid, ..., &context->oid);   /* free home slot */
if (rc != NO_ERROR)
  { return rc; }
if (heap_ovf_delete (thread_p, &context->hfid, &overflow_oid, NULL) == NULL)   /* <- frees the chain */
  { return ER_FAILED; }

heap_log_delete_physical logs an RVHF_DELETE undoredo plus — gated on is_reusable — an RVHF_MARK_REUSABLE_SLOT postpone (the home-slot fate switch; takeaway 6). heap_ovf_delete, called with ovf_vfid_p == NULL, then resolves the overflow file before delegating:

// heap_ovf_delete -- src/storage/heap_file.c
  if (ovf_vfid_p == NULL || VFID_ISNULL (ovf_vfid_p))     /* <- caller passed NULL */
    {
      ovf_vfid_p = (ovf_vfid_p != NULL) ? ovf_vfid_p : &ovf_vfid;
      if (heap_ovf_find_vfid (thread_p, hfid, ovf_vfid_p, false /*docreate*/, ...) == NULL)
        { return NULL; }                                  /* <- could not find ovf file */
    }
  ovf_vpid.pageid = ovf_oid->pageid; ovf_vpid.volid = ovf_oid->volid;  /* OID page = head VPID */
  if (overflow_delete (thread_p, ovf_vfid_p, &ovf_vpid) == NULL)
    { return NULL; }
  return ovf_oid;

docreate = false — destruction never creates a file; the ovf_oid’s (volid, pageid) is the head chain VPID. Branches: (1) ovf_vfid_p non-NULL — skip the lookup (vacuum’s cached-VFID path); (2) NULL/NULL-valued — resolve via heap_ovf_find_vfid, NULL on failure; (3)/(4) propagate the overflow_delete NULL/success.

stateDiagram-v2
  [*] --> Live: REC_BIGONE inserted, chain allocated
  Live --> DelStamped: MVCC delete\nstamp DELID on head page\nhome slot unchanged
  DelStamped --> Reclaimed: vacuum or non-MVCC delete\nheap_ovf_delete -> overflow_delete\nhome slot -> REC_MARKDELETED or REC_DELETED_WILL_REUSE
  Live --> Reclaimed: non-MVCC class delete\ndirect heap_ovf_delete
  Reclaimed --> [*]: pages file_dealloc'd at run-postpone

Figure 6-2 — Lifecycle at delete time. An MVCC delete only moves the row to DelStamped (chain intact); the chain is freed only at Reclaimed.

6.6 The B+Tree overflow-key delete — btree_delete_overflow_key

The B+Tree overflow-key path (a long key spilled into a FILE_BTREE_OVERFLOW_KEY chain) reuses overflow_delete with no MVCC delay (key bytes are MVCC-invariant). btree_delete_overflow_key extracts the embedded head VPID and frees the chain but does not touch the slot (“will not delete the btree slot containing the key”):

// btree_delete_overflow_key -- src/storage/btree.c (condensed)
if (spage_get_record (thread_p, page_ptr, slot_id, &rec, PEEK) != S_SUCCESS)  /* peek the rec */
  { goto exit_on_error; }
if (node_type == BTREE_LEAF_NODE)
  {
    int mvccids_size = 0;                            /* +OR_MVCCID_SIZE per INSID/DELID flag */
    if (btree_record_object_is_flagged (rec.data, BTREE_OID_HAS_MVCC_INSID)) mvccids_size += OR_MVCCID_SIZE;
    if (btree_record_object_is_flagged (rec.data, BTREE_OID_HAS_MVCC_DELID)) mvccids_size += OR_MVCCID_SIZE;
    if (btree_leaf_is_flaged (&rec, BTREE_LEAF_RECORD_CLASS_OID))
      { start_ptr = rec.data + (2 * OR_OID_SIZE) + mvccids_size; }  /* OID + class OID */
    else
      { start_ptr = rec.data + OR_OID_SIZE + mvccids_size; }        /* OID only */
  }
else
  { start_ptr = rec.data + NON_LEAF_RECORD_SIZE; }   /* non-leaf: fixed prefix */
or_init (&buf, start_ptr, DISK_VPID_SIZE);
page_vpid.pageid = or_get_int (&buf, &rc);
if (rc == NO_ERROR) { page_vpid.volid = or_get_short (&buf, &rc); }
if (rc != NO_ERROR) { goto exit_on_error; }
if (overflow_delete (thread_p, &(btid->ovfid), &page_vpid) == NULL)   /* <- free the chain */
  { goto exit_on_error; }
return NO_ERROR;

The leaf/non-leaf branching (annotated inline) only locates the head VPID inside the variable-shape record; the three failure points (spage_get_record, or_get_int/or_get_short, overflow_delete) all goto exit_on_error, which returns rc (or ER_FAILED). As a FILE_BTREE_OVERFLOW_KEY chain on PAGE_OVERFLOW pages (VFID btid->ovfid), its destruction core is identical to the heap case (§6.1/§6.2).

6.7 Contrast — OID-list reclamation is not this path

The two B+Tree overflow shapes share the btid->ovfid file but diverge at delete time. The overflow-key chain (this chapter) is freed wholesale by overflow_delete. The overflow-OID-list chain (Chapter 7) is never freed by overflow_delete — on PAGE_BTREE slotted pages it is reclaimed OID-by-OID (btree_overflow_remove_object), with emptied pages individually unlinked and file_dealloc’d. Only key-chain pages flow through this chapter.

6.8 Chapter summary — key takeaways

1. Reclamation is a thin walk: overflow_delete → overflow_traverse(OVERFLOW_DO_DELETE) → overflow_delete_internal (pgbuf_unfix_and_init then file_dealloc), harvesting next_vpid before freeing each page; all error branches return NULL.
2. No per-page RVOVF_* record on delete — recovery is delegated to file_dealloc (RVFL_DEALLOC postpone for all files, plus RVFL_USER_PAGE_MARK_DELETE only in its numerable-file branch), so mid-walk failure needs no rollback.
3. The FILE_UNKNOWN_TYPE hint is a correctness-neutral TODO — it forces file_dealloc to read the real type from the header; the caller knows the type but does not pass it.
4. An MVCC delete of a REC_BIGONE does NOT free the chain: the is_mvcc_op branch only stamps a DELID on the head overflow page (heap_get_mvcc_rec_header_from_overflow → heap_delete_adjust_header → heap_set_mvcc_rec_header_on_overflow) and logs a no-change record on the home slot — the chain must outlive the logical delete.
5. The overflow head’s MVCC header is forced to OR_MVCC_MAX_HEADER_SIZE (absent INSID/DELID padded), so an in-place DELID stamp never shifts the row body — the enabler of takeaway 4.
6. The chain is freed only by vacuum or a non-MVCC delete: that branch reclaims the home slot first (heap_delete_physical → REC_MARKDELETED, or REC_DELETED_WILL_REUSE via an RVHF_MARK_REUSABLE_SLOT postpone when heap_is_reusable_oid) then calls heap_ovf_delete.
7. The B+Tree overflow-key delete has no MVCC delay: btree_delete_overflow_key extracts the head VPID and calls overflow_delete at once, leaving the slot to the caller — unlike OID-list reclamation (Chapter 7), which never uses overflow_delete.

Chapter 7: The B+Tree OID List Variant

The B+tree index needs the same idea as overflow_file.c — one logical record too big for a slot spills over a chain of pages — but for a sorted list of OIDs sharing one index key. CUBRID does not reuse overflow_file.c; it re-implements the chain in btree.c / btree_load.c on ordinary PAGE_BTREE slotted pages. For the on-disk OID encoding see the companion cubrid-overflow-file.md, “B+tree OID overflow”; the reasons the variant exists are collected in §7.10.

Invariant (page type domain). Every overflow OID page satisfies pgbuf_check_page_ptype(...) == PAGE_BTREE, asserted in debug builds at each entry point (inside #if !defined(NDEBUG)). Under NDEBUG the check compiles out: debug asserts a wrong type, release would misread raw bytes as a slotted OID record and corrupt the list.

7.1 On-page layout and BTREE_OVERFLOW_HEADER

Slot 0/HEADER is the BTREE_OVERFLOW_HEADER; slot 1 is one OID record, sorted ascending at BTREE_OBJECT_FIXED_SIZE stride.

graph LR
  leaf["leaf record<br/>OVERFLOW_OIDS<br/>link=first vpid"]
  o1["page A HEAD<br/>slot0 next=B"]
  o2["page B<br/>slot0 next=C"]
  o3["page C TAIL<br/>slot0 next=NULL"]
  leaf --> o1 --> o2 --> o3

Figure 7-1. One key’s overflow chain; the leaf links to the HEAD, each page’s slot-0 header holds the singly-linked next_vpid.

// btree_overflow_header -- src/storage/btree_load.h
struct btree_overflow_header
{
  VPID next_vpid;
};

Field	Role	Why it exists
next_vpid	VPID of the next page in this key’s chain; VPID_ISNULL at the tail.	The chain is singly linked HEAD->tail; the leaf stores only the HEAD, every later hop reads this field. No back-pointer, no length count — one VPID is the minimum to walk and to splice.

Invariant (single OID record at slot 1). Object data lives only in slot 1; slot 0 is the header. Readers hardcode spage_get_record(..., 1, ...) and HEADER; a stray third slot would break the binary-search stride math.

Invariant (fixed stride, mandatory full MVCC header). Each object is BTREE_OBJECT_FIXED_SIZE(btid) bytes — OID, optional class OID, a full OR_MVCC_MAX_HEADER_SIZE header (leaf records may elide it). The binary search indexes data + size * mid, so a short object would land mid-record; insert enforces the stride via BTREE_MVCC_INFO_SET_FIXED_SIZE(...).

7.2 Header accessors in btree_load.c

The only sanctioned accessors for slot 0.

// btree_get_overflow_header -- src/storage/btree_load.c
#if !defined(NDEBUG)
  (void) pgbuf_check_page_ptype (thread_p, page_ptr, PAGE_BTREE); /* <- debug-only domain check */
#endif
  if (spage_get_record (thread_p, page_ptr, HEADER, &header_record, PEEK) != S_SUCCESS)
    { assert_release (false); return NULL; }      /* <- HEADER slot must exist */
  return (BTREE_OVERFLOW_HEADER *) header_record.data; /* <- PEEK: aliases page buffer */

PEEK aliases the page buffer, so btree_get_next_overflow_vpid calls btree_get_overflow_header (returning ER_FAILED on a NULL header) and then copies ovf_header->next_vpid into the caller’s *vpid by value — so the caller may safely unfix the page afterward.

btree_init_overflow_header installs slot 0 and logs nothing itself — the caller folds it into one redo record with the first object. Sole failure branch:

// btree_init_overflow_header -- src/storage/btree_load.c
// ... condensed: memcpy ovf_header into rec, rec.length = sizeof(...), rec.type = REC_HOME ...
if (spage_insert_at (thread_p, page_ptr, HEADER, &rec) != SP_SUCCESS)
  { return ER_FAILED; }                            /* <- only error path */

7.3 Creating the chain head: btree_start_overflow_page

The new page becomes the HEAD: its next_vpid is set to the current first-overflow VPID; the caller (§7.4) rewrites the leaf link. Every branch is annotated inline below — btree_get_new_page NULL, init != NO_ERROR (returns without unfixing), spage_insert_at failure, then the redo-only log + dirty:

// btree_start_overflow_page -- src/storage/btree.c
*new_page_ptr = btree_get_new_page (thread_p, btid_int, new_vpid, near_vpid);
if (*new_page_ptr == NULL) { ASSERT_ERROR_AND_SET (error_code); return error_code; }
VPID_COPY (&ovf_header_info.next_vpid, first_overflow_vpid);  /* <- HEAD points at old first */
error_code = btree_init_overflow_header (thread_p, *new_page_ptr, &ovf_header_info);
// ... condensed: build rec, BTREE_OVERFLOW_NODE; init!=NO_ERROR returns WITHOUT unfixing ...
if (spage_insert_at (thread_p, *new_page_ptr, 1, &rec) != SP_SUCCESS)  /* <- slot 1 = objects */
  { assert_release (false); return ER_FAILED; }
// ... condensed: redo buffer = first_overflow_vpid + rec.data (link restored by redo too) ...
log_append_redo_data (thread_p, RVBT_RECORD_MODIFY_NO_UNDO, &addr, rv_redo_data_length, rv_redo_data);
pgbuf_set_dirty (thread_p, *new_page_ptr, DONT_FREE);

Redo-only: rollback undoes the allocation wholesale via the surrounding sysop (§7.4), so nothing physical needs undoing.

7.4 Linking the head in, and the leaked-page note

The caller btree_key_append_object_as_new_overflow rewrites the leaf link under one system operation. The three load-bearing lines are the sysop start, the source’s own leaked-page comment, and the leaf-rewrite log call:

// btree_key_append_object_as_new_overflow -- src/storage/btree.c
if (!insert_helper->is_system_op_started) { log_sysop_start (thread_p); /* ... set flag ... */ }
/* Note that this page may be leaked if server crashes before changing the link in leaf page. */
// ... condensed: btree_start_overflow_page, btree_leaf_record_change_overflow_link, spage_update leaf ...
log_append_undoredo_data (thread_p, RVBT_RECORD_MODIFY_UNDOREDO, &insert_helper->leaf_addr, ...);

Invariant (chain reachability vs. the leaked-page window). A page joins the live chain only once a predecessor links to it: the HEAD’s next_vpid is set first, the leaf link second, and the sysop ties allocation and leaf-link rewrite into one atomic unit, so recovery has both or neither. The only residual leak is the crash-before-sysop-end case the source comment names — a harmless, type-correct, unreferenced page. The error: label aborts the sysop, unfixing ovfl_page.

7.5 Finding free space: btree_find_free_overflow_oids_page

A linear WRITE-latched first-fit scan with three branches, annotated inline:

// btree_find_free_overflow_oids_page -- src/storage/btree.c
int space_needed = BTREE_OBJECT_FIXED_SIZE (btid);
ovfl_vpid = *first_ovfl_vpid;                       /* empty -> loop skipped, *overflow_page NULL */
while (!VPID_ISNULL (&ovfl_vpid))
  {
    *overflow_page = pgbuf_fix (...);
    if (*overflow_page == NULL) { ASSERT_ERROR_AND_SET (error_code); return error_code; }
    if (spage_max_space_for_new_record (thread_p, *overflow_page) > space_needed)
      { return NO_ERROR; }                          /* <- hit: leave page FIXED for caller */
    btree_get_next_overflow_vpid (thread_p, *overflow_page, &ovfl_vpid);
    pgbuf_unfix_and_init (thread_p, *overflow_page); /* <- no room: release before next hop */
  }
return NO_ERROR;                                    /* <- miss: *overflow_page == NULL, full chain */

A hit returns with the page fixed; a miss returns *overflow_page == NULL, telling the caller to start a new HEAD.

7.6 Searching: btree_find_oid_and_its_page and btree_find_oid_from_ovfl

btree_find_oid_and_its_page checks the leaf first (hit -> *found_page = leaf), short-circuits on a null link, then walks the chain while tracking the previous page. Every branch is annotated inline below; the after-loop fixup (assign *found_page/*prev_page on a hit, or unfix on miss) is elided:

// btree_find_oid_and_its_page -- src/storage/btree.c
if (*offset_to_object != NOT_FOUND) { *found_page = leaf_page; return NO_ERROR; } /* leaf hit */
if (VPID_ISNULL (&leaf_rec_info->ovfl)) { return NO_ERROR; }   /* <- no chain */
do {
    overflow_page = pgbuf_fix (...);                            /* NULL -> goto error */
    error_code = btree_find_oid_from_ovfl (...);               /* err -> goto error */
    if (*offset_to_object != NOT_FOUND) { break; }             /* <- hit: keep page fixed */
    if (prev_overflow_page != NULL) { pgbuf_unfix_and_init (thread_p, prev_overflow_page); }
    prev_overflow_page = overflow_page; overflow_page = NULL;  /* <- advance prev pointer */
    error_code = btree_get_next_overflow_vpid (thread_p, prev_overflow_page, &overflow_vpid);
    if (error_code != NO_ERROR) { ASSERT_ERROR (); goto error; }
} while (!VPID_ISNULL (&overflow_vpid));
// ... condensed: assign *found_page / *prev_page, or unfix on not-found ...

Invariant (prev_page contract). On a chain hit with prev_page requested, the function returns the immediately preceding page — another overflow page, or the leaf when the hit is in the first overflow page — exactly what the unlink path (§7.8) needs. thread_p->read_ovfl_pages_count is bumped for vacuum.

btree_find_oid_from_ovfl is a per-page binary search over sorted slot 1: PEEK slot 1 (failure -> assert_release); first/last-object OID_LT/OID_GT early-outs; then a while (min <= max) search over [1 .. num_oids-2]. The two load-bearing details are the stride math and the equal-OID delegation:

// btree_find_oid_from_ovfl -- src/storage/btree.c
size = BTREE_OBJECT_FIXED_SIZE (btid_int);          /* <- fixed stride is the search invariant */
oid_ptr = ovf_record.data + (size * mid);           /* <- stride math relies on fixed size */
if (OID_EQ (oid, &inst_oid))                         /* <- equal OID: scan MVCC run, else +/- mid */
  { return btree_seq_find_oid_from_ovfl (thread_p, btid_int, oid, &ovf_record, oid_ptr, ...); }
// ... else OID_GT -> min = mid+1; OID_LT -> max = mid-1; fall through -> NO_ERROR (not found) ...

7.7 Rewriting a link: btree_modify_overflow_link

When the unlinked page’s predecessor is another overflow page, its next_vpid is rewritten — the only place a live overflow header is mutated. It uses RVBT_RECORD_MODIFY_UNDOREDO (in-place change to a surviving page).

// btree_modify_overflow_link -- src/storage/btree.c
assert (delete_helper->is_system_op_started);        /* <- caller must own a sysop */
overflow_header_record.data = rv_undo_data;
if (spage_get_record (thread_p, ovfl_page, HEADER, &overflow_header_record, COPY) != S_SUCCESS)
  { assert_release (false); return ER_FAILED; }      /* <- error branch 1: undo-image read */
VPID_COPY (&ovf_header_info.next_vpid, next_ovfl_vpid);   /* <- new link skips removed page */
overflow_header_record.data = (char *) &ovf_header_info;  // ... condensed: length set ...
if (spage_update (thread_p, ovfl_page, HEADER, &overflow_header_record) != SP_SUCCESS)
  { assert_release (false); return ER_FAILED; }      /* <- error branch 2: update */
ovf_addr.offset = HEADER;                            /* <- slot 0, not slot 1: header rewrite */
BTREE_RV_SET_OVERFLOW_NODE (&ovf_addr);
LOG_RV_RECORD_SET_MODIFY_MODE (&ovf_addr, LOG_RV_RECORD_UPDATE_ALL);  /* <- whole header replaced */
log_append_undoredo_data (thread_p, RVBT_RECORD_MODIFY_UNDOREDO, &ovf_addr, ...);

Undo = old header (copied first), redo = new header, the whole 1-field record (LOG_RV_RECORD_UPDATE_ALL), addressed at slot HEADER. Two error branches: the undo-image read (spage_get_record COPY) and the spage_update — both assert_release(false) and return ER_FAILED. The leaf-predecessor variant is btree_modify_leaf_ovfl_vpid, also RVBT_RECORD_MODIFY_UNDOREDO.

7.8 Shrinking: btree_overflow_remove_object

Two top-level branches keyed on whether the page would become empty.

flowchart TD
  R["spage_get_record slot1 COPY"] --> Q{length == FIXED_SIZE?}
  Q -->|yes: last object| GN{get_next err?}
  GN -->|yes| RET["return err; no sysop yet"]
  GN -->|no| U["unfix; sysop_start"]
  U --> DA["file_dealloc page"]
  DA --> DAE{dealloc error?}
  DAE -->|yes| ERR["goto error: sysop_end if started"]
  DAE -->|no| LK{prev_page == leaf?}
  LK -->|yes| LM["btree_modify_leaf_ovfl_vpid -> next"]
  LK -->|no| OM["btree_modify_overflow_link -> next"]
  LM --> SE["btree_delete_sysop_end if started"]
  OM --> SE
  Q -->|no: more objects| RR["btree_record_remove_object slot1"]
  RR --> OK["NO_ERROR"]
  SE --> OK

Figure 7-2. btree_overflow_remove_object branch map. Note the early bare return after get_next_overflow_vpid fails: it fires before any sysop is opened, so it does NOT route through the error: label that the later dealloc/link failures use.

The empty branch’s distinguishing lines are the early bare return and the unfix-before-dealloc ordering:

// btree_overflow_remove_object -- src/storage/btree.c
if (overflow_record.length == BTREE_OBJECT_FIXED_SIZE (btid_int))
  {
    assert (offset_to_object == 0);                  /* <- single object lives at offset 0 */
    error_code = btree_get_next_overflow_vpid (thread_p, *overflow_page, &next_overflow_vpid);
    if (error_code != NO_ERROR) { ASSERT_ERROR (); return error_code; } /* <- bare return: no sysop yet */
    pgbuf_get_vpid (*overflow_page, &overflow_vpid);
    pgbuf_unfix_and_init (thread_p, *overflow_page); /* <- unfix BEFORE dealloc */
    // ... condensed: sysop_start if needed; file_dealloc (goto error on fail) ...
    if (prev_page == leaf_page)                      /* <- §7.7: leaf vs overflow predecessor */
      { /* btree_modify_leaf_ovfl_vpid */ } else { /* btree_modify_overflow_link */ }
    // ... condensed: goto error on fail; btree_delete_sysop_end if !save_system_op_started ...
  }
else  // ... condensed: non-empty page splices one object via btree_record_remove_object on slot 1 ...

Invariant (deallocate under a sysop, unfix first). Deallocation must be in a sysop or the page leaks on rollback (the source says so); the code starts one if needed, unfixes before file_dealloc, and runs the unlink in the same sysop — making “object removed + page freed + predecessor relinked” atomic. The error: label ends the sysop only if this function opened it (!save_system_op_started), avoiding a double-end; the early get_next failure returns before any sysop exists, so it must NOT touch error:.

7.9 Replacing in place: btree_overflow_record_replace_object

The overflow-side replace of the unique-index “swap first leaf object with last overflow object” delete dance — in place, under the caller’s sysop, single branch:

// btree_overflow_record_replace_object -- src/storage/btree.c
assert (delete_helper->is_system_op_started);        /* <- must run inside a sysop */
// ... condensed: overflow_addr -> slot 1; BTREE_RV_SET_OVERFLOW_NODE; btree_record_replace_object ...
if (spage_update (thread_p, overflow_page, 1, overflow_record) != SP_SUCCESS)
  { assert_release (false); return ER_FAILED; }       /* <- only error branch */
log_append_undoredo_data (thread_p, RVBT_RECORD_MODIFY_UNDOREDO, &overflow_addr, ...);

Both objects are fixed-size, so length is unchanged and sorted order is preserved by construction (the swap inherits the last object’s slot). FI_TEST points bracket the update for half-applied recovery testing.

7.10 Why RVBT_* not RVOVF_*

Three properties force the variant: (1) pages are PAGE_BTREE, allocated via the index’s own file (btid_int->sys_btid->vfid), not a FILE_OVERFLOW; (2) a two-slot spage allows in-place spage_update, impossible for the contiguous OVERFLOW_REST_PART byte format; (3) every mutation logs RVBT_RECORD_MODIFY_NO_UNDO, ..._UNDOREDO, or RVBT_GET_NEWPAGE — records carrying a BTREE_RV_OVERFLOW_FLAG-tagged addr.offset, replayed by the B+tree record-modify apply routine that understands the fixed-size OID encoding and B+tree logical undo. RVOVF_* understands only opaque bytes in a FILE_OVERFLOW file, so reusing it here would replay a structurally wrong page — and the overflow edit and leaf-link edit replay under one RVBT_* family.

7.11 Chapter summary — key takeaways

1. The OID-list chain is a deliberate re-implementation on PAGE_BTREE slotted pages in the index’s own file, not a reuse of overflow_file.c.
2. BTREE_OVERFLOW_HEADER is a single next_vpid; the chain is singly linked HEAD->tail with the leaf storing only the HEAD link.
3. New pages prepend at the HEAD under one sysop — page next_vpid first, leaf link second — with a documented harmless leak if a crash precedes sysop-end.
4. Free-space search is first-fit; object search is a fixed-stride binary search.
5. btree_find_oid_and_its_page returns the predecessor page, letting delete choose btree_modify_leaf_ovfl_vpid vs btree_modify_overflow_link.
6. Shrink has two branches: dealloc-and-unlink under a sysop for a last object, or splice via btree_record_remove_object for a surviving record; the early get_next failure returns before any sysop opens.
7. All edits log RVBT_* with a BTREE_RV_OVERFLOW_FLAG-tagged address — never RVOVF_*.

Chapter 8: Crash Recovery and Edge Paths

The lifecycle chapters (3, 5, 6) showed what the mutators write. This chapter answers: if the server dies between any two of those log appends, which record replays the chain back to a consistent state, and which branches never go through the normal lifecycle?

Four recovery indices carry the whole subsystem (four handlers, two dump helpers); a fifth record, the zero-length LOG_DUMMY_OVF_RECORD, carries no recovery action but anchors HA replication and vacuum head identification. For undo, redo, physical logging, and the sysop bracket, see the high-level companion cubrid-overflow-file.md.

8.1 The dispatch table: four indices, four handlers

Recovery is table-driven: each rcvindex is one row in RV_fun[] in recovery.c, shaped { rcvindex, name, undofun, redofun, undo_dump, redo_dump }:

// RV_fun[] -- src/transaction/recovery.c
  {RVOVF_NEWPAGE_INSERT, "RVOVF_NEWPAGE_INSERT",
   NULL,                            /* <- no undo: insert is redo-only */
   overflow_rv_newpage_insert_redo, NULL, log_rv_dump_hexa},
  {RVOVF_NEWPAGE_LINK, "RVOVF_NEWPAGE_LINK",
   overflow_rv_newpage_link_undo,   /* <- undo: cut the new link to NULL */
   overflow_rv_link,                /* <- redo: re-establish the link */
   overflow_rv_link_dump, overflow_rv_link_dump},
  {RVOVF_PAGE_UPDATE, "RVOVF_PAGE_UPDATE",
   log_rv_copy_char,                /* <- undo: restore the before-image */
   overflow_rv_page_update_redo,    /* <- redo: set ptype then after-image */
   overflow_rv_page_dump, overflow_rv_page_dump},
  {RVOVF_CHANGE_LINK, "RVOVF_CHANGE_LINK",
   overflow_rv_link, overflow_rv_link,   /* <- undo and redo are the SAME fn */
   overflow_rv_link_dump, overflow_rv_link_dump},

Two design facts jump out of the table itself:

• RVOVF_NEWPAGE_INSERT has a NULL undo slot — insert never undoes a page write; the sysop deallocates the pages on abort, so there is nothing to un-write.
• RVOVF_CHANGE_LINK uses the same function for undo and redo: a symmetric record (old VPID for undo, new for redo), and the handler copies whichever recovery hands it into next_vpid.

flowchart TD
  subgraph idx["recovery indices (recovery.h 100..103, values 54..57)"]
    A["RVOVF_NEWPAGE_INSERT"]
    B["RVOVF_NEWPAGE_LINK"]
    C["RVOVF_PAGE_UPDATE"]
    D["RVOVF_CHANGE_LINK"]
  end
  A -->|redo| H1["overflow_rv_newpage_insert_redo<br/>= log_rv_copy_char"]
  B -->|undo| H2["overflow_rv_newpage_link_undo<br/>next_vpid := NULL"]
  B -->|redo| H3["overflow_rv_link<br/>next_vpid := data"]
  C -->|undo| H4["log_rv_copy_char<br/>before-image"]
  C -->|redo| H5["overflow_rv_page_update_redo<br/>set ptype + after-image"]
  D -->|undo / redo| H3

Figure 8-1. The four indices and their handlers; overflow_rv_link is shared by three of the four redo/undo slots.

8.2 RVOVF_NEWPAGE_INSERT — redo-only full-page replay

The handler is a one-liner over the generic log_rv_copy_char, which does memcpy (rcv->pgptr + rcv->offset, rcv->data, rcv->length) then pgbuf_set_dirty; rcv->offset is always 0 here, so the record lands at the page top:

// overflow_rv_newpage_insert_redo -- src/storage/overflow_file.c
int overflow_rv_newpage_insert_redo (THREAD_ENTRY * thread_p, LOG_RCV * rcv)
{ return log_rv_copy_char (thread_p, rcv); }

Emit site (Chapter 3). overflow_insert logs this once per page after stamping next_vpid, length, and the payload, via log_append_redo_data (…, RVOVF_NEWPAGE_INSERT, &addr, copy_length + (copyto - addr.pgptr), addr.pgptr) — the source pointer is the page top (addr.pgptr), not copyto. So the redo image is the header (with the already-set next_vpid) plus the payload — one record reconstitutes the whole prefix in a single memcpy. Why redo-only: content and link share one atomic record, so there is no half-written page to undo; the sysop (§8.8) reclaims pages on abort.

8.3 RVOVF_NEWPAGE_LINK — undo cuts the link, redo re-establishes it

This index appears only in overflow_update (Ch 5), when the rewrite runs out of pages and must extend the chain. After file_alloc yields a fresh next_vpid, it logs log_append_undoredo_data (…, RVOVF_NEWPAGE_LINK, &addr, 0, sizeof(VPID), NULL, &next_vpid). The record is asymmetric — undo length 0, redo length sizeof(VPID) — the signature of “this created a forward pointer that did not exist before.”

// overflow_rv_link -- src/storage/overflow_file.c   (redo: install the pointer)
{ ((OVERFLOW_REST_PART *) rcv->pgptr)->next_vpid = *(VPID *) rcv->data;  /* <- splice in */
  pgbuf_set_dirty (thread_p, rcv->pgptr, DONT_FREE); ... }

// overflow_rv_newpage_link_undo -- src/storage/overflow_file.c   (undo: sever)
{ VPID_SET_NULL (&((OVERFLOW_REST_PART *) rcv->pgptr)->next_vpid);   /* <- ignores rcv->data */
  pgbuf_set_dirty (thread_p, rcv->pgptr, DONT_FREE); ... }

The undo handler ignores rcv->data (hence undo length 0): undo always means “there was no forward link before.” Both cast OVERFLOW_REST_PART even for the first page, since next_vpid sits at offset 0 in both structs (Ch 1).

Invariant — a reachable page is never a dangling source. A predecessor’s next_vpid points at a fully recoverable successor or is NULL; never at a page recovery leaves un-allocated. Enforced by ordering: the page is allocated (logged separately) before RVOVF_NEWPAGE_LINK, both in the same sysop. On undo, the link is NULLed and file undo reclaims the successor — in either replay order no predecessor points at a reclaimed page. If violated, overflow_traverse (Ch 4) would walk into a freed page and read garbage as a next_vpid.

8.4 RVOVF_CHANGE_LINK — symmetric link rewrite for shrink

When overflow_update finishes copying with pages to spare, it shrinks the chain by NULL-ing the last needed page’s link and deallocating the tail (Ch 5, Ch 6). With tmp_vpid set NULL, it logs log_append_undoredo_data (…, RVOVF_CHANGE_LINK, &addr, sizeof(VPID), sizeof(VPID), &next_vpid, &tmp_vpid) — both images are full VPIDs: the old forward link (the doomed tail) for undo, the new one (NULL) for redo. The operation is “replace one VPID with another,” so undo and redo share overflow_rv_link — undo copies the old VPID back, redo copies NULL in (hence §8.1 lists it in both slots).

stateDiagram-v2
  direction LR
  [*] --> Linked: page points at tail
  Linked --> Unlinked: redo CHANGE_LINK \n next_vpid set to NULL
  Unlinked --> Linked: undo CHANGE_LINK \n next_vpid restored from data

Figure 8-2. RVOVF_CHANGE_LINK is reversible in place; the record carries both endpoints, so either direction is a copy into next_vpid.

Contrast with NEWPAGE_LINK. Both rewrite next_vpid but are not interchangeable: NEWPAGE_LINK extends and undoes to NULL (not in the record); CHANGE_LINK shrinks and undoes to the old value (in the record). They coincide only in redo.

8.5 RVOVF_PAGE_UPDATE — before/after images with a ptype guard

In-place content rewrite (Ch 5) logs an undo before-image and a redo after-image per overwritten page:

// overflow_update -- src/storage/overflow_file.c  (first-part branch)
if (hdr_length + old_length > DB_PAGESIZE)   /* <- old payload spans full page */
  log_append_undo_data (..., RVOVF_PAGE_UPDATE, &addr, DB_PAGESIZE, addr.pgptr);
else                                         /* <- only the used prefix */
  log_append_undo_data (..., RVOVF_PAGE_UPDATE, &addr, hdr_length + old_length, addr.pgptr);
// ... memcpy of new content, then ...
log_append_redo_data (..., RVOVF_PAGE_UPDATE, &addr, copy_length + hdr_length, addr.pgptr);

The undo length is DB_PAGESIZE when the old payload filled the page, else the used prefix only — never trailing garbage. The redo length copy_length + hdr_length is header plus new slab.

// overflow_rv_page_update_redo -- src/storage/overflow_file.c
{ (void) pgbuf_set_page_ptype (thread_p, rcv->pgptr, PAGE_OVERFLOW);  /* <- re-stamp */
  return log_rv_copy_char (thread_p, rcv); }                          /* <- then bytes */

The undo handler is the bare log_rv_copy_char (§8.1 undo slot): the before-image needs no ptype fix, since the page already carried the type when it was taken.

Why the redo re-stamps PAGE_OVERFLOW. Recovery may apply the after-image to a page whose pgbuf page-type was reset during reallocation or never persisted; pgbuf_set_page_ptype makes redo self-sufficient.

8.6 The dump helpers

Dump helpers render the recovery payload as text when the log is dumped, never mutating state.

// overflow_rv_page_dump -- src/storage/overflow_file.c
void overflow_rv_page_dump (FILE * fp, int length, void *data)
{ OVERFLOW_REST_PART *rest_parts = (OVERFLOW_REST_PART *) data;
  int hdr_length = offsetof (OVERFLOW_REST_PART, data);   /* <- skip link header */
  fprintf (fp, "Overflow Link to Volid = %d|Pageid = %d\n",
           rest_parts->next_vpid.volid, rest_parts->next_vpid.pageid);
  log_rv_dump_char (fp, length - hdr_length, (char *) data + hdr_length); }  /* <- ascii payload */

overflow_rv_link_dump (fp, length_ignore, data) ignores length (a link record is always one VPID) and fprintfs the VPID; it serves both link indices, undo and redo. overflow_rv_page_dump prints the header next_vpid then ascii-dumps the remaining length - hdr_length bytes — the RVOVF_PAGE_UPDATE redo layout. RVOVF_NEWPAGE_INSERT instead uses the generic log_rv_dump_hexa (§8.1): its image starts at the page top and is hex-dumped.

8.7 LOG_DUMMY_OVF_RECORD — a zero-length anchor, not a recovery action

LOG_DUMMY_OVF_RECORD (“indicator of the first part of an overflow record”) is not in RV_fun[] — no handler, no undo/redo. Both mutators emit it via log_append_empty_record (zero-length body) only on the first page: overflow_insert’s i == 0 branch guards it with file_type != FILE_TEMP, while overflow_update emits it in its first-part branch.

It is a recognizable LSN landmark the per-page records lack: every page emits a near-identical RVOVF_NEWPAGE_INSERT / RVOVF_PAGE_UPDATE, so a log consumer cannot tell where one overflow object ends and the next starts. Emitted only on the first page, it marks the object’s head VPID — the anchor a vacuum or HA/CDC consumer needs (the CDC walker cdc_get_overflow_recdes is one delimiter user).

8.8 The sysop bracket: why no half-built or orphaned chain survives

Every mutator brackets its allocations and writes in a system operation: log_sysop_start, then file_alloc_multiple (handed file_init_page_type, or file_init_temp_page_type for FILE_TEMP), then per-page memcpy + RVOVF_NEWPAGE_INSERT, ending in log_sysop_attach_to_outer on success — or log_sysop_abort at exit_on_error. The bracket gives an all-or-nothing guarantee the individual records cannot:

• Before attach → undo runs log_sysop_abort’s compensations, deallocating every page; no RVOVF_NEWPAGE_INSERT redo image survives.
• After attach, before parent commit → writes belong to the parent tx; an abort triggers file-layer undo to reclaim the pages.
• After parent commit → redo replays each self-contained record in order; the chain assembles correctly whatever was already flushed.

overflow_update adds RVOVF_NEWPAGE_LINK / RVOVF_CHANGE_LINK on the same bracket; their undo handlers (§8.3, §8.4) restore the predecessor’s next_vpid so a partially-resized chain rolls back to its pre-update topology — physical images plus sysop deallocation close every crash window.

8.9 Edge paths: temp files, no_logging, MVCC peek, and the debug dumper

FILE_TEMP and no_logging suppress RVOVF logging. Temp overflow files are volatile, so the emit sites are guarded: overflow_insert wraps its RVOVF_NEWPAGE_INSERT append in if (file_type != FILE_TEMP && thread_p->no_logging != true), while overflow_update guards the content redo by no_logging != true alone. When logging is suppressed the handlers are never reached for that page — there is no record to replay.

Temp pages get their type at allocation, not via redo. With no redo path, the type must be right from birth: overflow_insert’s file_alloc_multiple is handed file_init_temp_page_type instead of file_init_page_type for FILE_TEMP (via file_type != FILE_TEMP ? file_init_page_type : file_init_temp_page_type, §8.8), since no record re-stamps it. The fork lives only here: overflow_update opens with assert (file_type == FILE_MULTIPAGE_OBJECT_HEAP) and never runs for FILE_TEMP, so its extend-branch file_alloc is hard-wired to file_init_page_type with no temp alternative.

MVCC peek via overflow_get_first_page_data. Readers holding the first page fixed who want only the inline payload — without Chapter 4’s traversal — call this accessor: after assert (page_ptr != NULL) it returns ((OVERFLOW_FIRST_PART *) page_ptr)->data, the address past the link+length header. That lets MVCC code peek at the leading bytes (the record header carrying MVCC ids) in place — no fix, latch, or validation beyond the non-NULL assert.

overflow_dump is CUBRID_DEBUG-only. Its three exit-affecting branches are two pgbuf_fix-failure early returns and one VPID_ISNULL corruption check. The initial fix at the head VPID and the per-page fix after advancing both bail with return ((ret = er_errid ()) == NO_ERROR) ? ER_FAILED : ret; — a fix failure aborts the dump (propagating the page-buffer error or ER_FAILED) rather than walking a NULL page. The non-obvious branch is the VPID_ISNULL check inside the remain_length > 0 loop, after the current page is unfixed:

// overflow_dump -- src/storage/overflow_file.c   (#if defined (CUBRID_DEBUG))
  while (remain_length > 0)
    { remain_length -= dump_length;
      if (remain_length > 0)
        { pgbuf_unfix_and_init (thread_p, pgptr);     /* <- release current before checking */
          if (VPID_ISNULL (&next_vpid))   /* <- chain ends early but bytes remain */
            { er_set (..., ER_HEAP_OVFADDRESS_CORRUPTED, ...); return ...; }
          pgptr = pgbuf_fix (..., &next_vpid, ...);     /* <- advance to next page */
          if (pgptr == NULL)              /* <- second fix-failure early return */
            { return ((ret = er_errid ()) == NO_ERROR) ? ER_FAILED : ret; }
          next_vpid = ((OVERFLOW_REST_PART *) pgptr)->next_vpid; } }

It catches the corruption case where the declared length demands more bytes than the links supply, raising the error rather than dereferencing a NULL next pointer — the same length-versus-link consistency Chapter 4 relies on.

8.10 Chapter summary — key takeaways

1. Four indices, four handlers: insert (redo-only full-page copy), NEWPAGE_LINK (undo NULLs / redo installs the link), PAGE_UPDATE (undo before-image / redo re-stamps ptype then after-image), CHANGE_LINK (symmetric VPID swap, same handler both ways).
2. overflow_rv_link is shared by RVOVF_NEWPAGE_LINK redo and both RVOVF_CHANGE_LINK slots; the extend/shrink asymmetry lives only in the undo side (overflow_rv_newpage_link_undo forces NULL, ignoring rcv->data). PAGE_UPDATE redo is the only path re-asserting PAGE_OVERFLOW; insert and FILE_TEMP pages get their type from file_init_page_type / file_init_temp_page_type at allocation — and that temp/non-temp fork lives solely in overflow_insert, since overflow_update asserts FILE_MULTIPAGE_OBJECT_HEAP and never sees a temp file.
3. No half-built or orphaned chain survives a crash: content+link are atomic per record and the operation is wrapped in a sysop whose abort deallocates the pages. FILE_TEMP / no_logging skip RVOVF appends entirely.
4. LOG_DUMMY_OVF_RECORD is a zero-length LSN landmark for the object head; overflow_get_first_page_data is a latch-free MVCC peek; overflow_dump (CUBRID_DEBUG only) aborts on either pgbuf_fix failure and flags a length-versus-link mismatch as corruption.

Position hints as of this revision

The following are line numbers as observed on 2026-06-21; symbols are the canonical anchor and line numbers are hints that decay.

Symbol	File	Line
CEIL_PTVDIV	src/base/memory_alloc.h	57
BTREE_LEAF_RECORD_OVERFLOW_OIDS	src/storage/btree.c	115
BTREE_LEAF_RECORD_OVERFLOW_KEY	src/storage/btree.c	117
BTREE_RV_SET_OVERFLOW_NODE	src/storage/btree.c	971
btree_create_overflow_key_file	src/storage/btree.c	1975
btree_store_overflow_key	src/storage/btree.c	2021
btree_load_overflow_key	src/storage/btree.c	2131
btree_delete_overflow_key	src/storage/btree.c	2202
btree_start_overflow_page	src/storage/btree.c	3973
btree_get_new_page	src/storage/btree.c	5135
btree_initialize_new_page	src/storage/btree.c	5165
btree_modify_leaf_ovfl_vpid	src/storage/btree.c	9977
btree_modify_overflow_link	src/storage/btree.c	10054
btree_key_append_object_as_new_overflow	src/storage/btree.c	11330
btree_key_append_object_to_overflow	src/storage/btree.c	11452
btree_find_free_overflow_oids_page	src/storage/btree.c	11579
btree_find_oid_and_its_page	src/storage/btree.c	11646
btree_find_oid_from_ovfl	src/storage/btree.c	12037
btree_seq_find_oid_from_ovfl	src/storage/btree.c	12210
btree_overflow_remove_object	src/storage/btree.c	32344
btree_overflow_record_replace_object	src/storage/btree.c	33340
struct non_leaf_rec	src/storage/btree.h	103
LEAF_REC	src/storage/btree.h	110
ovfid	src/storage/btree.h	129
btree_get_overflow_header	src/storage/btree_load.c	340
btree_init_overflow_header	src/storage/btree_load.c	576
btree_get_next_overflow_vpid	src/storage/btree_load.c	675
btree_overflow_header	src/storage/btree_load.h	245
file_init_page_type	src/storage/file_manager.c	5349
file_init_temp_page_type	src/storage/file_manager.c	5365
file_dealloc	src/storage/file_manager.c	6116
FILE_MULTIPAGE_OBJECT_HEAP	src/storage/file_manager.h	43
FILE_BTREE_OVERFLOW_KEY	src/storage/file_manager.h	45
FILE_TEMP	src/storage/file_manager.h	52
struct file_ovf_heap_des	src/storage/file_manager.h	90
struct file_ovf_btree_des	src/storage/file_manager.h	106
file_create_with_npages	src/storage/file_manager.h	187
heap_ovf_find_vfid	src/storage/heap_file.c	6462
heap_ovf_insert	src/storage/heap_file.c	6569
heap_ovf_update	src/storage/heap_file.c	6597
heap_ovf_delete	src/storage/heap_file.c	6632
heap_ovf_get	src/storage/heap_file.c	6717
heap_get_mvcc_rec_header_from_overflow	src/storage/heap_file.c	19541
heap_set_mvcc_rec_header_on_overflow	src/storage/heap_file.c	19567
heap_get_bigone_content	src/storage/heap_file.c	19610
heap_delete_adjust_header	src/storage/heap_file.c	21290
heap_delete_bigone	src/storage/heap_file.c	21389
heap_delete_physical	src/storage/heap_file.c	22388
heap_log_delete_physical	src/storage/heap_file.c	22428
HEAP_HEADER_AND_CHAIN_SLOTID	src/storage/heap_file.h	62
OVERFLOW_ALLOCVPID_ARRAY_SIZE	src/storage/overflow_file.c	43
OVERFLOW_DO_FUNC	src/storage/overflow_file.c	45
OVERFLOW_DO_DELETE	src/storage/overflow_file.c	47
overflow_insert	src/storage/overflow_file.c	81
overflow_insert	src/storage/overflow_file.c	82
overflow_next_vpid	src/storage/overflow_file.c	275
overflow_next_vpid	src/storage/overflow_file.c	276
overflow_traverse	src/storage/overflow_file.c	296
overflow_traverse	src/storage/overflow_file.c	297
overflow_update	src/storage/overflow_file.c	374
log_sysop_start	src/storage/overflow_file.c	407
log_sysop_attach_to_outer	src/storage/overflow_file.c	594
log_sysop_abort	src/storage/overflow_file.c	600
overflow_delete_internal	src/storage/overflow_file.c	613
overflow_delete_internal	src/storage/overflow_file.c	614
overflow_delete	src/storage/overflow_file.c	645
overflow_flush_internal	src/storage/overflow_file.c	656
overflow_flush	src/storage/overflow_file.c	678
overflow_get_length	src/storage/overflow_file.c	691
overflow_get_length	src/storage/overflow_file.c	692
overflow_get_nbytes	src/storage/overflow_file.c	738
overflow_get_nbytes	src/storage/overflow_file.c	739
overflow_get	src/storage/overflow_file.c	917
overflow_get_capacity	src/storage/overflow_file.c	934
overflow_get_capacity	src/storage/overflow_file.c	935
overflow_dump	src/storage/overflow_file.c	1039
overflow_rv_newpage_insert_redo	src/storage/overflow_file.c	1113
overflow_rv_newpage_link_undo	src/storage/overflow_file.c	1125
overflow_rv_link	src/storage/overflow_file.c	1145
overflow_rv_link_dump	src/storage/overflow_file.c	1165
overflow_rv_page_update_redo	src/storage/overflow_file.c	1178
overflow_rv_page_update_redo	src/storage/overflow_file.c	1179
overflow_rv_page_dump	src/storage/overflow_file.c	1193
overflow_get_first_page_data	src/storage/overflow_file.c	1217
overflow_get_first_page_data	src/storage/overflow_file.c	1218
OVERFLOW_FIRST_PART	src/storage/overflow_file.h	37
overflow_first_part	src/storage/overflow_file.h	38
struct overflow_first_part	src/storage/overflow_file.h	38
OVERFLOW_REST_PART	src/storage/overflow_file.h	45
overflow_rest_part	src/storage/overflow_file.h	46
struct overflow_rest_part	src/storage/overflow_file.h	46
PAGE_OVERFLOW	src/storage/storage_common.h	158
log_append_empty_record	src/transaction/log_manager.c	3138
log_rv_copy_char	src/transaction/log_manager.c	9087
LOG_DUMMY_OVF_RECORD	src/transaction/log_record.hpp	124
RV_fun	src/transaction/recovery.c	54
RVFL_DEALLOC	src/transaction/recovery.h	57
RVFL_USER_PAGE_MARK_DELETE	src/transaction/recovery.h	63
RVHF_STATS	src/transaction/recovery.h	80
RVOVF_NEWPAGE_INSERT	src/transaction/recovery.h	100
RVOVF_NEWPAGE_LINK	src/transaction/recovery.h	101
RVOVF_PAGE_UPDATE	src/transaction/recovery.h	102
RVOVF_CHANGE_LINK	src/transaction/recovery.h	103
RVBT_UPDATE_OVFID	src/transaction/recovery.h	123

Sources

• cubrid-overflow-file.md — the high-level companion. See also cubrid-heap-manager-detail.md (REC_BIGONE) and cubrid-btree-detail.md (overflow-OID pages).
• Code: src/storage/overflow_file.{c,h}; callers in src/storage/heap_file.c and src/storage/btree.c.
• Methodology: knowledge/methodology/code-analysis-detail-doc.md.