CUBRID OOS (Out-of-Line Storage) — Large-Column Off-Page Storage

IN-DEVELOPMENT FEATURE. This document analyzes the OOS (Out-of-Line Storage) feature on the feat/oos branch as observed at commit fa26eff3a (105 commits ahead of develop). Milestone M1 is done (CRUD, WAL, HA replication, covered index); M2 is in progress (bestspace reuse, compaction, drop-table file destroy, vacuum integration); M3–M4 are planned. Symbol names are the stable anchor; the line numbers in the position-hint table and the APIs described here will drift as the branch evolves. Several abort() calls and TODO(... remove before develop merge) markers are still live in the source — see Cross-check Notes. Treat this as a snapshot, not a spec.

CUBRID’s heap stores every column of a row contiguously inside a single slotted-page record. That is good for SELECT * but pathological for wide rows: reading id from a table whose rows carry a 1 MB CLOB-shaped VARCHAR still pulls the whole record (and overflow pages) into the buffer pool. OOS addresses this by moving only the large columns into a separate per-class file, leaving a fixed-size pointer where the value used to be.

Theoretical Background

Row-store engines lay a tuple out as a contiguous byte string on a page: a header, a variable-offset table, the fixed-width columns, then the variable-width columns. This is optimal when queries touch most of the row, but it has two failure modes once individual attributes grow large:

1. Read amplification. A page holds PAGESIZE / avg_row_size rows. If one column is huge, avg_row_size balloons, rows-per-page collapses, and a query that needs only the small columns still faults in pages dominated by bytes it never reads. Index-only (“covered”) scans help, but any heap touch pays the full record cost.
2. Page-size ceilings. A value larger than a page cannot live inline at all; the engine must spill it somewhere and leave a reference behind.

The standard answer is off-page / out-of-line storage: keep small columns inline, move large ones to a side structure, and store a small fixed-size pointer (plus, usually, the original length) in the record. The pattern has three recurring design choices — what triggers externalization, what the pointer looks like, and how the off-page bytes are reclaimed when the row changes or dies.

Comparable systems:

• PostgreSQL TOAST — values past a per-tuple target (TOAST_TUPLE_THRESHOLD, derived from page size) are compressed (pglz / lz4) and, if still large, pushed to a per-table pg_toast_* relation in ~2 KB chunks. The inline pointer (varatt_external) is roughly 18 bytes and records the toast OID and both raw and stored lengths. Chunks are reassembled on read.
• MySQL/InnoDB off-page — VARCHAR/BLOB/TEXT overflow is moved to off-page extents; the inline part keeps a ~20-byte pointer (space id, page number, offset, length) to the first overflow page, chained for large values.
• CUBRID’s existing overflow (OVF) files — long strings already spill into FILE_MULTIPAGE_OBJECT_HEAP overflow records. OOS is a different, finer-grained mechanism aimed at column-level externalization with a smaller inline footprint and per-column control, sitting alongside OVF rather than replacing it.

The conceptual through-line: trade a guaranteed-cheap pointer dereference for a guaranteed-small inline record. The engineering cost is all in the bookkeeping — the trigger heuristic, pointer integrity, and especially reclamation under MVCC, where an “old” pointer may still be the only thing a concurrent reader can see.

CUBRID’s Approach

OOS (the source consistently expands it as Out-of-row Storage / Out-of-row Overflow Storage; the JIRA epic uses Out-of-Line Storage) externalizes large variable-length columns into a per-class OOS file of type FILE_OOS, leaving a fixed 16-byte inline stub in the heap record. The epic is CBRD-26357.

Per-class file pairing

Each table is 1 heap file + 1 OOS file, created lazily. The heap header (HEAP_HDR_STATS) gains an oos_vfid field. The first time a row needs to externalize a column, heap_oos_find_vfid(..., docreate=true) creates the OOS file, applies the class’s TDE algorithm to it, and stamps the VFID into the heap header under a top system operation:

// heap_oos_find_vfid -- src/storage/heap_file.c
if (VFID_ISNULL (&heap_hdr->oos_vfid))
  {
    if (docreate == true)
      {
        log_sysop_start (thread_p);
        if (oos_create_file (thread_p, *oos_vfid) != NO_ERROR) { /* abort */ }
        // ... condensed: apply TDE to the new OOS file ...
        log_append_undo_data (thread_p, RVHF_STATS, &addr_hdr, sizeof (*heap_hdr), heap_hdr);
        VFID_COPY (&heap_hdr->oos_vfid, oos_vfid);
        log_append_redo_data (thread_p, RVHF_STATS, &addr_hdr, sizeof (*heap_hdr), heap_hdr);
        log_sysop_commit (thread_p);
      }
    // else: oos_vfid stays NULL — "no OOS file" is not an error
  }

oos_create_file allocates a numerable FILE_OOS file and a sticky first page that holds the OOS_HDR_STATS bestspace header at slot 0.

Two flags: HAS_OOS (per-row) and IS_OOS (per-column)

OOS uses two independent flag bits:

• OR_MVCC_FLAG_HAS_OOS = 0x08 — bit 3 of the MVCC header flags. Set on a heap record that has at least one externalized column. It is the cheap “does this record need OOS expansion?” test, read by heap_recdes_contains_oos. (A TODO in the header notes OOS is not actually MVCC-related and the bit should eventually move.)

  // OR_MVCC_FLAG_HAS_OOS -- src/base/object_representation_constants.h
  // TODO: OOS is not related to MVCC, move to another place when POC ends
  #define OR_MVCC_FLAG_HAS_OOS   0x08  // 0b00001000

• OR_VAR_BIT_OOS = 0x1 — the low bit of each variable-offset-table (VOT) entry. The VOT already steals the two low bits of every offset for flags (OR_VAR_FLAG_MASK = 0x3): bit 0 marks an OOS column, bit 1 marks the last element. So a single VOT walk both finds the OOS columns and terminates.

  // OR_IS_OOS / OR_OOS_INLINE_SIZE -- src/base/object_representation.h
  #define OR_VAR_BIT_OOS 0x1
  #define OR_GET_VAR_OFFSET(length) ((int) (length) & (~OR_VAR_FLAG_MASK))
  #define OR_IS_OOS(length) (OR_GET_VAR_FLAG (length) & OR_VAR_BIT_OOS)
  /* OOS inline size: OOS OID (8 bytes) + OOS length (8 bytes) */
  #define OR_OOS_INLINE_SIZE (OR_OID_SIZE + OR_BIGINT_SIZE)

The inline stub: 16 bytes

Where the column value used to be, the record carries OR_OOS_INLINE_SIZE = 16 bytes: an 8-byte CUBRID OID (the head-chunk locator) followed by an 8-byte BIGINT holding the full original value length. The OID itself decomposes the way every CUBRID OID does — volid (2B), pageid (4B), slotid (2B) — and the trailing length lets the reader size its destination buffer without first fetching the chunk.

The design-epic phrasing “16-byte OOS OID” is the whole stub (OID + length), not the OID alone. The OID portion is 8 bytes; the length is the other 8.

The trigger heuristic

Externalization is decided in heap_attrinfo_determine_disk_layout while computing the on-disk layout. As observed at this revision the rule is PostgreSQL-TOAST-style, not the /8 + 512B rule stated in the original design intent (see Cross-check Notes):

// heap_attrinfo_determine_disk_layout -- src/storage/heap_file.c
/* push the largest variable column to OOS one by one until the heap record
 * fits within DB_PAGESIZE/4 (PG TOAST style) ... */
if (header_size + payload_size + mvcc_extra > DB_PAGESIZE / 4)
  {
    // collect every variable, non-fixed column whose size > OR_OOS_INLINE_SIZE
    // sort candidates by size descending, then:
    for (auto & cand : oos_candidates)
      {
        if (header_size + payload_size + mvcc_extra <= DB_PAGESIZE / 4) break;
        (*oos_columns)[cand.second] = true;        // mark this column OOS
        payload_size -= cand.first;                // remove inline value
        payload_size += OR_OOS_INLINE_SIZE;        // add the 16-byte stub
        *has_oos = true;
      }
  }

So: if the assembled record would exceed DB_PAGESIZE/4, the engine externalizes the largest variable column first, re-checks, and repeats until the record fits — only externalizing as many columns as needed, and only columns larger than the 16-byte stub (otherwise it would not shrink).

INSERT

For each marked column, the value is serialized to a scratch RECDES, then oos_insert writes it to the OOS file and returns the head-chunk OID, which is written into the inline stub. The per-row HAS_OOS bit and per-column IS_OOS bits are set during the disk transform.

flowchart TD
  I0["INSERT row"] --> I1["determine_disk_layout:\nrecord > DB_PAGESIZE/4 ?"]
  I1 -- no --> I9["write fully inline record"]
  I1 -- yes --> I2["mark largest var columns IS_OOS\nset HAS_OOS"]
  I2 --> I3["heap_attrinfo_insert_to_oos:\nfor each OOS column"]
  I3 --> I4["serialize column to RECDES"]
  I4 --> I5["oos_insert -> head-chunk OID"]
  I5 --> I6["write 16B stub OID + length\ninto VOT slot"]
  I6 --> I7["push OID to thread oos_oids\nfor replication log"]
  I7 --> I8["spage_insert heap record"]

oos_insert picks single-chunk vs multi-chunk by comparing the payload to the max chunk that fits one page:

// oos_insert -- src/storage/oos_file.cpp
if (src_len <= oos_get_max_chunk_size_within_page ())
  {
    const OOS_RECORD_HEADER header{src_len, 0, OID_INITIALIZER};
    err = oos_insert_within_page (thread_p, oos_vfid, src, header, oid);
  }
else
  {
    err = oos_insert_across_pages (thread_p, oos_vfid, src, oid);
  }

oos_insert_within_page prepends an OOS_RECORD_HEADER, finds a target page via bestspace, spage_inserts, logs RVOOS_INSERT (undo/redo), and updates the bestspace cache:

// oos_insert_within_page -- src/storage/oos_file.cpp
auto auto_page_ptr = oos_find_best_page (thread_p, oos_vfid, required_length, vpid);
// ... condensed: oos_prepend_header builds [OOS_RECORD_HEADER | payload] into oos_recdes ...
int sp_status = spage_insert (thread_p, page_ptr, &oos_recdes, &slotid);
// ... condensed: error check ...
oid.pageid = vpid.pageid;
oid.slotid = slotid;
oid.volid  = vpid.volid;
oos_log_insert_physical (thread_p, page_ptr, const_cast<VFID *> (&oos_vfid), &oid, &oos_recdes);
int freespace_after = spage_max_space_for_new_record (thread_p, page_ptr);
(void) oos_stats_add_bestspace (thread_p, &oos_vfid, &vpid, freespace_after);

The returned oid (volid/pageid/slotid of the slot just written) is what ends up in the heap stub for a single-chunk value, or as a chunk’s next_chunk_oid for a multi-chunk chain. oos_prepend_header allocates a fresh data area sized payload + OOS_RECORD_HEADER_SIZE and is freed by a scope_exit on the way out — the bytes are copied into the page by spage_insert, so the heap-side stub never aliases this buffer.

SELECT

The scan path never expands OOS. Code that needs the real bytes goes through heap_get_visible_version_expand_oos, which sets context.expand_oos = true; the record fetch then calls heap_record_replace_oos_oids, which rebuilds a record that looks as if OOS were never used.

flowchart TD
  S0["fetch visible version\nexpand_oos = true"] --> S1["heap_record_replace_oos_oids"]
  S1 --> S2{"HAS_OOS set?"}
  S2 -- no --> S9["return record unchanged"]
  S2 -- yes --> S3["parse VOT, find IS_OOS entries"]
  S3 --> S4["for each: read 16B stub\nOID + full_length"]
  S4 --> S5["oos_read OID into a sized buffer"]
  S5 --> S6["compute expanded layout\nVOT rewritten to 4-byte offsets"]
  S6 --> S7["build_record: inline blobs in place,\nclear HAS_OOS, reset offset-size bits"]
  S7 --> S8["return expanded record"]

The reconstruction is schema-change safe: it uses only oos_read plus the on-record VOT, never the class representation. The output VOT is always written with 4-byte offsets (BIG_VAR_OFFSET_SIZE) so arbitrarily large expansions fit. The rebuild copies the header verbatim, then clears the HAS_OOS flag and forces the 4-byte offset size so the expanded record is indistinguishable from one that never used OOS:

// heap_oos_build_record -- src/storage/heap_oos.cpp
std::memcpy (dst, state->src, state->src_header_size);
unsigned int repid_bits = (unsigned int) OR_GET_INT (dst + OR_REP_OFFSET);
repid_bits &= ~ ((unsigned int) OR_MVCC_FLAG_HAS_OOS << OR_MVCC_FLAG_SHIFT_BITS);  // clear HAS_OOS
repid_bits &= ~ (unsigned int) OR_OFFSET_SIZE_FLAG;
repid_bits |= OR_OFFSET_SIZE_4BYTE;                                                // force 4B offsets
OR_PUT_INT (dst + OR_REP_OFFSET, (int) repid_bits);
// ... condensed: rewrite VOT with new cumulative offsets, OOS entries sized to blob length ...
// ... condensed: copy fixed+bitmap unchanged; inline each OOS blob in place of its 16B slot ...

Reading sizes the destination from the stub’s stored length and cross-checks it against the chain header:

// heap_oos_read_blobs -- src/storage/heap_oos.cpp
OR_BUF buf;
or_init (&buf, (char *) state->src + value_offset, OR_OOS_INLINE_SIZE);
if (or_get_oid (&buf, &oos_oid) != NO_ERROR || OID_ISNULL (&oos_oid)) { /* fail */ }
oos_len = or_get_bigint (&buf, &rc);
// ... condensed: validate oos_len in (0, DB_MAX_STRING_LENGTH] ...
state->oos_blobs[i].resize ((std::size_t) oos_len);
if (oos_read (thread_p, oos_oid, oos_buffer (state->oos_blobs[i].data (), (std::size_t) oos_len)) != NO_ERROR)
  { /* fail */ }

Multi-chunk chain

A value larger than one OOS page is split into a chain of chunks. Each chunk’s record begins with an OOS_RECORD_HEADER:

// oos_record_header -- src/storage/oos_file.hpp
struct oos_record_header
{
  int total_data_length;  /* total user-data length across all chunks (excludes OOS headers) */
  int chunk_index;        /* 0-based index of this chunk in the chain */
  OID next_chunk_oid;     /* OID of next chunk, or NULL OID if this is the last */
};

The head chunk is index 0; its OID is the one stored in the heap stub. oos_insert_across_pages inserts chunks in reverse order (tail first) so each chunk’s next_chunk_oid is already known when it is written, and every chunk carries total_data_length so the slave applier can validate the whole value before reassembly.

oos_read_across_pages follows next_chunk_oid from index 1, asserting the index increments and total_data_length matches the head, and rejecting a 0-byte chunk (which would let a cyclic next_chunk_oid loop forever).

UPDATE / DELETE and reclamation

This is where OOS interacts with MVCC. The governing rule is an invariant:

INVARIANT: one OOS OID is referenced by EXACTLY ONE heap record — OOS records are never shared across rows. OOS OIDs are freshly allocated per heap record. This is precisely what lets vacuum (and the eager path) delete an old OID unconditionally without reference counting: if a row version no longer references an OID, nobody else does.

• UPDATE (M1): always allocates a new OOS OID for the changed column. The old OOS record is kept alive so concurrent MVCC readers that still see the old row version can dereference it; vacuum reclaims it once the old version is dead. (M3 plans OID reuse when an OOS column is unchanged.)
• DELETE: the heap record is logically deleted; OOS records are left untouched and reclaimed later by vacuum.
• Non-MVCC paths (!is_mvcc_op — SA_MODE, and SERVER_MODE on MVCC-disabled catalog classes) have no readers and no vacuum, so cleanup is eager: heap_oos_delete_unreferenced deletes the OOS OIDs that the old record referenced and the new record does not.

stateDiagram-v2
  [*] --> Live : INSERT, oos_insert
  Live --> Updated : UPDATE M1, allocate NEW oos oid
  Updated --> OldDead : old version becomes invisible
  Live --> RowDeleted : DELETE, heap untouched
  OldDead --> Reclaimed : vacuum oos_delete
  RowDeleted --> Reclaimed : vacuum oos_delete
  Live --> EagerGone : non-mvcc delete or update, eager oos_delete
  Reclaimed --> [*]
  EagerGone --> [*]

Vacuum reclaims via two paths in vacuum_oos.cpp:

1. REMOVE path — vacuum_heap_oos_delete_within_sysop runs inside the caller’s existing sysop and deletes the OOS referenced by a heap record being vacuumed.
2. Forward-walk path — vacuum_forward_walk_reclaim_oos inspects undo images of RVHF_UPDATE_NOTIFY_VACUUM / RVHF_DELETE_NEWHOME_NOTIFY_VACUUM records (the pre-images of UPDATE/DELETE), and deletes OOS referenced only by those dead old versions, inside its own sysop (vacuum_forward_walk_oos_delete_atomic).

A single-slot VACUUM_OOS_VFID_MEMO caches the most-recently-resolved heap-VFID → OOS-VFID mapping so a bulk UPDATE’s run of consecutive records on the same heap skips repeated file/header lookups. The forward-walk path is defensive throughout: lookups distinguish FOUND / NONE / ERROR, errors are logged and cleared (a bounded, logged leak) rather than failing the whole vacuum block, and oos_chunk_exists makes the delete idempotent on block retry.

Bestspace

To place new OOS records without scanning the whole file, OOS_HDR_STATS.estimates keeps a heap-style multi-entry bestspace estimate (header record at slot 0 of the sticky first page) plus a global in-memory hash cache (oos_Bestspace). oos_find_best_page runs a 3-tier search mirroring the heap manager:

flowchart TD
  B0["need page for rec_length"] --> B1["Tier 1: global hash cache\noos_stats_find_page_in_bestspace"]
  B1 -- hit --> BOK["fix page, recheck free space"]
  B1 -- miss --> B2["Tier 2: header best[] array"]
  B2 -- hit --> BOK
  B2 -- miss --> B3["Tier 3: sync scan pages\noos_stats_sync_bestspace, refill hints"]
  B3 -- still none --> B4["oos_file_alloc_new"]
  BOK --> BDONE["return page"]
  B4 --> BDONE

best[] holds up to OOS_NUM_BEST_SPACESTATS (10) candidate pages; a second_best[] ring buffer holds evicted-but-still-good pages. Bestspace updates are non-logged (log_skip_logging) because hints are reconstructable and need not survive crash exactly. Sync uses conditional latches and zero-wait mode to avoid contention.

The hash-cache lookup illustrates the contention-avoidance discipline: it fixes candidate pages with a conditional write latch and zero-wait mode, skipping any busy page rather than blocking, and re-verifies actual free space (spage_max_space_for_new_record) before committing — the cached estimate is only a hint and may be stale:

// oos_stats_find_page_in_bestspace -- src/storage/oos_file.cpp
old_wait_msecs = xlogtb_reset_wait_msecs (thread_p, LK_FORCE_ZERO_WAIT);
// ... condensed: Phase A scans global hash, Phase B scans best[] for a candidate_vpid ...
*out_pgptr = pgbuf_fix (thread_p, &candidate_vpid, OLD_PAGE,
                        PGBUF_LATCH_WRITE, PGBUF_CONDITIONAL_LATCH);
if (*out_pgptr == NULL) { /* busy or interrupted: skip / error, continue */ }
int actual_free = spage_max_space_for_new_record (thread_p, *out_pgptr);
if (actual_free >= needed_space) { *out_vpid = candidate_vpid; found = OOS_FINDSPACE_FOUND; }
// ... condensed: else unfix, drop the stale hint, try next ...

Because each insert constants its target via the cache and corrects the estimate on miss, the file fills denser pages first without a full scan, and oos_find_best_page only escalates to oos_stats_sync_bestspace (a bounded page scan) or oos_file_alloc_new when the hints are exhausted. The header best[] lives in the sticky-first-page record; the global hash cache (oos_Bestspace, capacity OOS_BESTSPACE_CACHE_CAPACITY = 1000 entries across all OOS files) is process-wide and rebuilt on restart.

WAL and logging

OOS physical operations are WAL-logged with dedicated redo handlers: RVOOS_INSERT (oos_rv_redo_insert → spage_insert_for_recovery) and RVOOS_DELETE (oos_rv_redo_delete → spage_delete). oos_delete uses no sysop: each chunk delete is individually logged with full undo data, so transaction abort or crash recovery restores the chain in reverse.

For HA, multi-chunk inserts need the slave to reassemble the same chain. The master logs the chunks in reverse (tail first, head last), wraps the run in a LOG_DUMMY_OOS_RECORD boundary marker, and suppresses the per-crumb auto-queueing of intermediate chunks so only the head OID is enqueued for replication:

// oos_insert_across_pages -- src/storage/oos_file.cpp
track_repl = oos_needs_repl_tracking (thread_p);   // master only, replication allowed
if (track_repl)
  {
    log_append_empty_record (thread_p, LOG_DUMMY_OOS_RECORD, NULL);
    LSA_COPY (&dummy_lsa, &tdes->tail_lsa);
    tdes->oos_suppress_insert_lsa_queueing = true;  // intermediate chunks not enqueued
  }
// ... condensed: reverse loop inserts chunk N-1 .. 0; capture tail_chunk_lsa at the tail ...
if (track_repl)
  {
    tdes->oos_insert_lsa_queue.push (dummy_lsa);
    tdes->oos_insert_lsa_queue.push (tail_chunk_lsa);
    thread_p->oos_oids.push_back (oid_Null_oid);     // caller pushes the real head OID after return
  }

The replication path then emits one RVREPL_DUMMY_OOS_RECORD for the null OID followed by one RVREPL_OOS_INSERT for the real head OID — so the slave applier sees exactly one logical OOS-insert per value regardless of how many physical chunks it spanned. oos_needs_repl_tracking gates this to the master (!LOG_CHECK_LOG_APPLIER && log_does_allow_replication()); the slave must not emit its own markers while replaying.

Note oos_log.hpp is a separate debug/diagnostic logging facility ($CUBRID/log/oos.log), unrelated to WAL — it is a header-only printf-style logger whose oos_error/oos_warn macros stay active in release builds for QA and replication diagnostics, with oos_trace/oos_debug/oos_info compiled to no-ops under NDEBUG.

Source Walkthrough

Symbols grouped by subsystem. Functions are the stable anchor; the table at the end pairs each with a (file, line) hint scoped to this revision.

OOS file core — src/storage/oos_file.{cpp,hpp}

• oos_record_header / OOS_RECORD_HEADER_SIZE — the per-chunk chain header (total_data_length, chunk_index, next_chunk_oid).
• oos_buffer — alias for cubbase::span<char>; the caller-owned byte span used by oos_insert (read-from) and oos_read (write-into).
• oos_create_file / oos_remove_file / oos_remove_page — file lifecycle. oos_create_file makes the FILE_OOS file + sticky header page; oos_remove_file defers destroy via file_postpone_destroy; oos_remove_page (file_dealloc) is marked “called by vacuum when OOS vacuum is implemented”.
• oos_insert — public entry; routes to within-page or across-pages.
• oos_insert_within_page — single-chunk insert: oos_prepend_header, oos_find_best_page, spage_insert, oos_log_insert_physical, bestspace update.
• oos_insert_across_pages — multi-chunk insert (reverse order) with HA replication boundary tracking.
• oos_read — public read; validates head chunk (chunk_index == 0), cross-checks total_data_length against the caller’s buffer size, then walks the chain.
• oos_read_within_page / oos_read_across_pages — single-chunk fetch via byte_span_writer; chain walk with consistency asserts.
• oos_delete / oos_delete_chain — delete a (possibly multi-chunk) record; per-chunk RVOOS_DELETE undo, no sysop.
• oos_chunk_exists — idempotency probe (page deallocated or slot removed both report “gone” with NO_ERROR); used by vacuum block retry.
• oos_get_length — reads total_data_length from a head chunk’s header (used by tests; production reads the inline stub length).
• oos_get_max_chunk_size_within_page — spage_max_record_size aligned down, minus the chain header.
• oos_rv_redo_insert / oos_rv_redo_delete — WAL redo handlers.
• oos_needs_repl_tracking — master-only replication gate.
• oos_push_oos_oid — appends an OID to thread_p->oos_oids for the replication log.

Bestspace (all in oos_file.cpp):

• oos_bestspace, oos_hdr_stats, OOS_NUM_BEST_SPACESTATS, OOS_FINDSPACE — the estimate structures and the find-result enum.
• oos_bestspace_initialize / oos_bestspace_finalize — global hash cache (oos_Bestspace, vpid_ht + vfid_ht) lifecycle.
• oos_stats_add_bestspace / oos_stats_del_bestspace_by_vpid / oos_stats_del_bestspace_by_vfid — cache entry management.
• oos_find_best_page — the 3-tier search driver.
• oos_stats_find_page_in_bestspace — hash-then-best[] lookup with conditional latch + actual-free recheck.
• oos_stats_sync_bestspace — scan pages to refill hints (partial or full).
• oos_stats_update / oos_stats_update_internal — refresh hints after delete/insert.
• oos_stats_put_second_best / oos_stats_get_second_best — the second_best[] ring buffer.
• oos_file_alloc_new / oos_vpid_init_new — allocate + init a new OOS data page (PAGE_OOS, anchored slotted page).
• xoos_get_stats_by_class_oid / oos_get_stats_by_vfid — server-side stats (page count, live record count, body-byte sum) backing the client API.

Heap integration — src/storage/heap_oos.{cpp,hpp} + heap_file.c

• heap_record_replace_oos_oids — SELECT-side expansion entry; gated on context.expand_oos and heap_recdes_contains_oos.
• HEAP_OOS_EXPAND_STATE + heap_oos_parse_vot / heap_oos_read_blobs / heap_oos_compute_layout / heap_oos_build_record — the four-phase expansion: parse VOT, read blobs via oos_read, compute output layout (4-byte offsets), assemble the expanded record and clear HAS_OOS.
• heap_oos_delete_unreferenced — eager (non-MVCC) cleanup: delete OOS OIDs in old_recdes not in new_recdes (NULL new = delete all).
• heap_oos_find_vfid — find or (with docreate) create the heap’s OOS file; applies TDE.
• heap_attrinfo_determine_disk_layout — the trigger heuristic (DB_PAGESIZE/4, largest-column-first).
• heap_attrinfo_insert_to_oos — per-column INSERT-side externalization.
• heap_recdes_contains_oos — tests OR_MVCC_FLAG_HAS_OOS.
• heap_recdes_get_oos_oids — walk the VOT, collect every OR_IS_OOS OID.
• heap_get_visible_version_expand_oos — the OOS-expanding fetch wrapper (CBRD-26847 flags it for a caller audit).

Vacuum — src/query/vacuum_oos.{cpp,hpp}

• vacuum_oos_vfid_memo / VACUUM_OOS_VFID_LOOKUP_RESULT — the per-worker-per-block one-entry memo and the FOUND/NONE/ERROR tri-state.
• vacuum_oos_vfid_lookup — heap-VFID → OOS-VFID resolution with caching; never caches transient errors.
• vacuum_forward_walk_reclaim_oos — undo-image-driven reclamation (copies the undo image to a stable, aligned buffer first).
• vacuum_forward_walk_oos_delete_atomic — owns its OID vector by value, sorts by (volid, pageid, slotid), deletes inside its own sysop.
• vacuum_heap_oos_delete_within_sysop — REMOVE-path delete inside the caller’s sysop.
• vacuum_oos_find_vfid_for_heap_record — first-need lookup for the heap-record vacuum path.

Flags & file type

• OR_MVCC_FLAG_HAS_OOS (object_representation_constants.h) — per-row bit 0x08.
• OR_VAR_BIT_OOS / OR_IS_OOS / OR_SET_VAR_OOS / OR_OOS_INLINE_SIZE (object_representation.h) — per-column VOT flag and 16-byte stub size.
• FILE_OOS (file_manager.h) — the new file type.

Client stats — src/compat/db_oos.h

• db_oos_stats / db_get_oos_stats — client-side struct and call exposing has_oos_file, the OOS VFID, user-page count, live record count, and body-byte sum (mirrors server-side oos_stats_info).

Logging — src/storage/oos_log.hpp

• oos_log::OosLogLevel / oos_log_internal + the oos_error, oos_warn, oos_trace, oos_debug, oos_info macros — a header-only diagnostic logger to $CUBRID/log/oos.log (oos_error/oos_warn active in release; the rest debug-only). Not WAL.

Position hints as of this revision

Line numbers observed on feat/oos @ fa26eff3a, 2026-06-21. Paths are the canonical repo paths.

Symbol	File	Line
oos_record_header	src/storage/oos_file.hpp	26
OOS_RECORD_HEADER_SIZE	src/storage/oos_file.hpp	34
oos_buffer (alias)	src/storage/oos_file.hpp	43
OOS_NUM_BEST_SPACESTATS	src/storage/oos_file.hpp	45
oos_bestspace	src/storage/oos_file.hpp	53
oos_hdr_stats	src/storage/oos_file.hpp	60
OOS_FINDSPACE (enum)	src/storage/oos_file.hpp	101
oos_bestspace_initialize	src/storage/oos_file.cpp	184
oos_stats_find_page_in_bestspace	src/storage/oos_file.cpp	454
oos_stats_sync_bestspace	src/storage/oos_file.cpp	634
oos_create_file	src/storage/oos_file.cpp	910
oos_remove_file	src/storage/oos_file.cpp	996
oos_insert	src/storage/oos_file.cpp	1048
oos_insert_across_pages	src/storage/oos_file.cpp	1109
oos_insert_within_page	src/storage/oos_file.cpp	1205
oos_read	src/storage/oos_file.cpp	1372
oos_find_best_page	src/storage/oos_file.cpp	1460
oos_delete_chain	src/storage/oos_file.cpp	1696
oos_chunk_exists	src/storage/oos_file.cpp	1779
oos_delete	src/storage/oos_file.cpp	1851
oos_get_max_chunk_size_within_page	src/storage/oos_file.cpp	1861
oos_rv_redo_insert	src/storage/oos_file.cpp	1911
oos_get_length	src/storage/oos_file.cpp	1942
xoos_get_stats_by_class_oid	src/storage/oos_file.cpp	1991
oos_get_stats_by_vfid	src/storage/oos_file.cpp	2043
oos_oid_in_vector	src/storage/oos_util.cpp	35
heap_recdes_compute_oos_flag_debug	src/storage/oos_util.cpp	83
HEAP_OOS_EXPAND_STATE	src/storage/heap_oos.cpp	48
heap_oos_parse_vot	src/storage/heap_oos.cpp	69
heap_oos_read_blobs	src/storage/heap_oos.cpp	129
heap_oos_compute_layout	src/storage/heap_oos.cpp	184
heap_oos_build_record	src/storage/heap_oos.cpp	242
heap_record_replace_oos_oids	src/storage/heap_oos.cpp	338
heap_oos_delete_unreferenced	src/storage/heap_oos.cpp	425
vacuum_oos_vfid_memo	src/query/vacuum_oos.hpp	44
vacuum_oos_vfid_lookup	src/query/vacuum_oos.cpp	86
vacuum_forward_walk_oos_delete_atomic	src/query/vacuum_oos.cpp	154
vacuum_forward_walk_reclaim_oos	src/query/vacuum_oos.cpp	223
vacuum_oos_find_vfid_for_heap_record	src/query/vacuum_oos.cpp	339
vacuum_heap_oos_delete_within_sysop	src/query/vacuum_oos.cpp	400
OR_MVCC_FLAG_HAS_OOS	src/base/object_representation_constants.h	178
OR_VAR_BIT_OOS	src/base/object_representation.h	441
OR_IS_OOS	src/base/object_representation.h	451
OR_OOS_INLINE_SIZE	src/base/object_representation.h	455
FILE_OOS	src/storage/file_manager.h	53
heap_attrinfo_determine_disk_layout	src/storage/heap_file.c	12183
heap_oos_find_vfid	src/storage/heap_file.c	12259
heap_attrinfo_insert_to_oos	src/storage/heap_file.c	12435
heap_get_visible_version_expand_oos	src/storage/heap_file.c	26450
heap_recdes_contains_oos	src/storage/heap_file.c	27772
heap_recdes_get_oos_oids	src/storage/heap_file.c	27779
db_get_oos_stats	src/compat/db_oos.h	47
oos_log_internal	src/storage/oos_log.hpp	97
oos_error (macro)	src/storage/oos_log.hpp	168

Cross-check Notes

This is an active feature branch; both the design intent and the line numbers above will drift. Specific discrepancies and rough edges observed at fa26eff3a:

Trigger heuristic differs from the design intent

The original epic states the trigger as row size > DB_PAGESIZE/8 (2 KB at a 16 KB page) AND column size > 512 B. The code at this revision uses a different, simpler rule: row size > DB_PAGESIZE/4 AND column size > OR_OOS_INLINE_SIZE (16 B), externalizing the largest column first and stopping as soon as the record fits (heap_attrinfo_determine_disk_layout, commented “PG TOAST style”). A TODO: change the statistics sits directly above it. Trust the code, not the epic, for the live threshold.

Milestone status

Milestone	JIRA	Status	Scope
M1	CBRD-26584	DONE (~2026-02)	CRUD, WAL logging, HA replication, covered index. Limits: no oos_file_destroy, UPDATE always allocates a new OID, vacuum not wired.
M2	CBRD-26583	IN PROGRESS	bestspace reuse (CBRD-26658), spage_compact compaction (CBRD-26536), drop-table oos_file_destroy (CBRD-26608), vacuum integration (CBRD-26668), develop merge + CI.
M3	—	PLANNED	OOS OID reuse on unchanged UPDATE columns, PEEK mode, across-page compaction.
M4	—	TBD	ordered page-fix for deadlock avoidance, monitoring.

Not yet implemented / in flux

• oos_remove_page carries // TODO: will be called by vacuum when OOS vacuum is implemented — empty-page deallocation is not yet wired; vacuum deletes slots but does not free pages.
• Live abort() calls in vacuum. vacuum_oos.cpp deliberately calls abort() in two places — vacuum_forward_walk_reclaim_oos (the VACUUM_OOS_VFID_NONE branch) and vacuum_oos_find_vfid_for_heap_record — each marked TODO(do not review, remove before develop merge): force a hard crash in CI. These intentionally crash release-build CI to surface an invariant violation (OOS flag set but heap has no OOS file) instead of leaking silently. They MUST be removed before merge; until then the branch will hard-crash on that condition.
• <cstdlib> include in vacuum_oos.cpp is annotated /* abort - TODO: remove before develop merge (temporary CI crash) */.
• heap_get_visible_version_expand_oos over-expansion — CBRD-26847 notes many call sites were migrated mechanically and may expand OOS unnecessarily; an audit is pending.

Known bugs / rough edges (from the epic)

• unloaddb runs 1.6–1.7× slower with OOS (CBRD-26458).
• UPDATE triggers a triple oos_read (CBRD-26516).
• CDC flashback cannot resolve an OOS OID.
• Unnecessary OOS replication log emitted in locator_add_or_remove_index.
• RECDES.length is a 4-byte int, capping a single expanded record at ~2 GB; heap_oos_compute_layout explicitly guards new_length against INT_MAX.
• No payload compression in M1 (unlike TOAST’s pglz/lz4); compression is an open PR (CBRD-26881 / #7258).

Legacy-record handling

heap_recdes_get_oos_oids and heap_recdes_compute_oos_flag_debug note that records lacking OR_VAR_BIT_LAST_ELEMENT (old-format VOTs) are “not fully supported yet” — the upper-bound max_var_count may include padding/fixed bytes, and old odd offsets could false-positive OR_IS_OOS. The debug auditor returns false for any VOT without a LAST_ELEMENT sentinel.

Open Questions

From the 2026-03-05 design discussion, still undecided:

• One stored record per OOS column, or merge multiple OOS columns of a row into one stored OOS record? Undecided. The current code stores each column individually (heap_attrinfo_insert_to_oos loops per column).
• OOS page latch contention — pages may be split later to reduce contention (an M4 concern; oos_find_best_page already uses conditional zero-wait latches to mitigate).
• OVF + OOS coexistence test cases are still needed.
• Should CHAR (not only VARCHAR) be OOS-eligible? The eligibility check is !last_attrepr->is_fixed, so fixed-width CHAR is currently excluded.

Found in the code (TODO/FIXME/XXX) at this revision:

• oos_insert — TODO: Once the OOS_RECORD_HEADER spec is finalized (first segment header and rest segment header), review whether the segment headers can be generated inside oos_insert_within_page / across_pages (the chunk-header format may still change).
• oos_get_max_chunk_size_within_page — TODO: fix bug for spage_max_record_size returning incorrect size (declared out of scope for OOS), and TODO: make it a constant initialized once in oos_boot().
• oos_delete_chain — TODO: concurrency — this function assumes the caller holds a row-level lock (e.g. X_LOCK) to prevent concurrent deletion of the same OOS chain. Verify this assumption when wiring callers.
• oos_stats_sync_bestspace — TODO: ideally find the index of full_search_vpid; for now start from 1 (resume point is approximate).
• vacuum_* — two TODO(perf) notes that oos_delete re-fixes the OOS page per OID; OIDs are already page-sorted, so per-page batching is a future optimization.
• The abort() / develop-merge TODOs listed under Cross-check Notes.

Broader unresolved: how OOS coexists with CDC/flashback (currently cannot resolve an OOS OID), and whether the OR_MVCC_FLAG_HAS_OOS bit should be relocated out of the MVCC header (the constant itself carries TODO: OOS is not related to MVCC, move ... when POC ends).

Sources

JIRA

• Epic: http://jira.cubrid.org/browse/CBRD-26357 — milestones CBRD-26584 (M1), CBRD-26583 (M2); survey CBRD-26198. Linked rough-edge issues: CBRD-26458 (unloaddb slowdown), CBRD-26516 (UPDATE triple read), CBRD-26847 (expand-OOS caller audit), CBRD-26881 (payload compression), CBRD-26658 (bestspace reuse), CBRD-26536 (spage_compact compaction), CBRD-26608 (drop-table destroy), CBRD-26668 (vacuum integration), CBRD-26628 (multi-chunk insert replication).

Design / explanation docs

• Notion — multi-chunk insert replication (CBRD-26628): https://glaze-girl-1d0.notion.site/CBRD-26628-OOS-multi-chunk-insert-34bfe9311b558092a524d46229b496e2
• Git — vacuum cleanup code-review explanation (CBRD-26668): https://github.com/vimkim/my-cubrid-docs/blob/main/cbrd-26668/CBRD-26668-code-review-explanation.md
• Notion — related INTERNAL LOB design: https://glaze-girl-1d0.notion.site/INTERNAL-LOB-2b1fe9311b55808692e2ed1774c60fde

Branch / PRs

• Branch feat/oos (105 commits ahead of develop), analyzed at fa26eff3a. Key PRs: #7097 (oos_read/oos_insert → oos_buffer refactor), #7158 (demote largest variable column to OOS), #7296 (inline-OOS read failure propagation), #7295 (TDE on OOS files), #7169 / #7168 (vacuum reclaim OOS chunks), #7258 (OOS payload compression, open), #6864 (CI tracking draft).

Code

• src/storage/oos_file.cpp, src/storage/oos_file.hpp
• src/storage/oos_log.hpp
• src/storage/oos_util.cpp, src/storage/oos_util.hpp
• src/storage/heap_oos.cpp, src/storage/heap_oos.hpp
• src/query/vacuum_oos.cpp, src/query/vacuum_oos.hpp
• src/compat/db_oos.h
• Touchpoints: src/base/object_representation_constants.h (OR_MVCC_FLAG_HAS_OOS), src/base/object_representation.h (OR_IS_OOS, OR_OOS_INLINE_SIZE), src/storage/file_manager.h (FILE_OOS), src/storage/heap_file.c (trigger, find_vfid, insert, expand, contains/get OIDs call sites).