The constructs of an SoA table
Every SoA table, whatever its layout, is read as the same five constructs. They are the blocks of the extraction schema.
schedule_propertiesSchedule property definitions from table header rows. These define the temporal grid structure.
schedule_gridCell values for schedule properties across columns. The temporal dimension of the table.
activitiesActivity definitions from table body rows. Includes indentation_level for hierarchy reconstruction - parent-child relationships derived in resolved layer.
activity_scheduleScheduling data at activity × timepoint intersections. The core content of the SoA table.
annotationsFootnotes, legends, and other annotations with explicit marker_locations. These encode study logic that doesn't fit the 2D table presentation.
The same constructs through three schemas
Each schema answers one question and adds one kind of precision. Nothing is replaced on the way: an id in a later file always points to the entry it came from, down to the position on the printed page.
schedule_properties[]row_position property_name property_type hierarchical_levelOne entry per header row.
schedule_properties[]property_id parent_property_id linked_annotation_idsEach row gets an id and its parent.
property_hierarchy[]property_id hierarchical_level property_name property_typeOne hierarchy across the tables.
schedule_grid[]row_position column_position cell_value merged_cell_rangeOne entry per header cell.
schedule_columns[]column_id column_position column_values composite_labelHeader cells read downwards become columns.
timeline_segments.<segment>[]xcol_id segment property_values source_columnsColumns of all tables aligned into segments (main, domain, track, subsidiary).
activities[]row_position source_page activity_name activity_name_source.indentation_levelOne entry per body row, with the page it was read from.
activities[]activity_id parent_activity_id hierarchy_level is_section_headerEach row gets an id; the parent is derived from the indentation level.
unified_activities[]xact_id source_refs match_status near_matchesRows of several tables matched into one activity. The source rows stay listed.
activity_schedule[]row_position column_position cell_value methodOne entry per non-empty body cell.
activity_schedule[]activity_id column_id cell_value cell_value_typePositions become id references.
schedule_matrix[]xact_id xcol_id consolidated_value source_valuesOne cell per unified activity and unified column. The value of each source table is kept.
annotations[]annotation_marker annotation_type annotation_text marker_locationsMarker, text, and every place the marker is printed.
annotations[]annotation_id referenced_elements annotation_scope document_referencesBound to the elements it applies to.
unified_annotations[]xannot_id source_occurrences referenced_xacts referenced_xcolsIdentical text deduplicated. Every occurrence is kept.
How corrections work
Review finds things. A reviewer does not edit a file; the reviewer writes an entry in a sidecar, and a program applies it. There are two sidecars, one before Layer 2 and one before Layer 3.
Table corrections written by a human
id | The entry's own id. |
target | Which block of the extraction it acts on. schedule_propertiesschedule_gridactivitiesactivity_scheduleannotationsreview_itemstable_metadata |
op | What it does. addsetremoveconfirmadd_itemremove_item |
match | Key fields that locate the entry, e.g. row and column position. |
set field value | What to write. |
reason by at | Why, who, when. |
source_ref | Where on the printed page the correction is grounded. |
review_item | The judgement call this entry answers, when it answers one. |
A table without a sidecar passes through untouched.
Match decisions written by a human
id | The entry's own id. |
op | What is decided about a pair of rows. keepsplitmergerefines |
source target | The two source rows, named by table_number + activity_id, with activity_name as a check. |
reason by at | Why, who, when. |
That two rows are the same activity is a human statement, not a pipeline result. An entry whose premise no longer holds stops consolidation.
The extraction states its open judgement calls as data
review_items[]: id, severity, call_made, alternative
A correction answers by naming the item
review_item with op confirm: the call stands. With any other op: the alternative was taken. An item is decided exactly when an entry names it.
USDM Instantiation
work in progressNot a fourth layer. Layers 1 to 3 stop at USDM-ready data. Turning one consolidated SoA into a USDM v4 document is a separate step that has only just started to be explored.
The idea of a study shell
- USDM has no standalone serialization of an SoA. A ScheduleTimeline hangs under a StudyDesign, under a StudyVersion, under a Study.
- A StudyDesign mandates arms, cells, elements, a population, eligibility criteria and a model. An SoA contains none of these.
- The shell is that surrounding structure, built from a manifest of a few facts about the protocol: identifier, title, sponsor, epoch list (usdm-manifest v1.0). What is not stated is emitted as an object flagged not stated.
- The shell and the SoA meet at exactly one attribute,
ScheduledInstance.epochId. The epoch list is the one part of the shell the SoA really needs.
The SoA lifting
- The consolidated table is rendered as USDM schedule objects: activities and scheduled instances in a timeline and, where the table supports it, timings and encounters.
- Which column property carries the epoch axis is an interpretation. It is stated outside the Layer 1-3 schemas and logged as a decision.
- Each build is checked against a pinned usdm-rdf release.
What a full-fledged instantiation would need. SoA patterns, to interpret what the timeline and the notes express, and potentially some kind of shared nomenclature. Neither is part of soa2usdm today.