Differences

This shows you the differences between two versions of the page.

--- manipulating_tables_and_lookup_tables [2026/07/28 19:56]
hermann
+++ manipulating_tables_and_lookup_tables [2026/08/18 03:09] (current)
hermann [Printing a table with any number of key columns]
@@ Line 298: / Line 298: @@
 ==== SetTableCellValue ====
-Writes a single cell — the write counterpart to ''GetTableValue'', though its ''column'' and ''keys'' arguments come in the opposite order.
+Writes a single cell — the write counterpart to ''GetTableValue''.
 ^ Port ^ Direction ^ Type ^ Required? ^ Description ^
@@ Line 495: / Line 495: @@
 doubledResults := % [ %sourceLookup[line] * 2 ] "CityId" "DoubledPopulation" sourceLookup;
 </code>
+One difference in shape: the loop accumulates into a Table, since ''emptyResults'' was declared as one, while ''%'' always returns a Lookup Table. Where the Table form is the one wanted, a ''Table'' carrier converts it — ''asTable := Table doubledResults'' — one Real key and one Real value fitting both types equally well.
+===== Table Manager =====
+The accumulator pattern in [[#storing_results_across_a_loop|Storing results across a loop]] uses a Mux to carry a growing table from one iteration to the next -- which, as that section's closing note explains, forces the loop to run sequentially, one iteration at a time. **Table Manager** solves the same class of problem -- building a table with an extra key column from results computed across a loop -- without that cost: each iteration registers its own sub-table independently, with no Mux and nothing threaded between iterations, so the loop producing those partial results stays eligible to run in parallel, per [[basic_data_flow|Basic Data Flow]]. A single [[#mergesubtables|MergeSubTables]] call combines everything afterward.
+A container and two functors do the core work; a fourth covers a narrower case:
+==== TableManager ====
+A container functor. It creates the manager instance and exposes it to whatever runs inside it.
+^ Port ^ Direction ^ Type ^ Description ^
+| ''tableManager'' | Internal Output | TableManager | The manager instance. Auto-binds to the ''tableManager'' input of [[#subtable|SubTable]] and [[#mergesubtables|MergeSubTables]] calls nested anywhere inside this container, including inside further nested containers such as a [[For]] loop or a [[Group]]. |
+The container itself takes no inputs and defines no regular outputs.
+==== SubTable ====
+Registers one sub-table under a name and a key.
+^ Port ^ Direction ^ Type ^ Required? ^ Description ^
+| ''tableName'' | Input | Name | Yes | Identifies which set of sub-tables this one joins. |
+| ''keys'' | Input | Tuple | Yes | The key identifying this sub-table within the set. |
+| ''subTable'' | Input | Table | Yes | The sub-table itself. |
+| ''emptySubTableIsAllowed'' | Input | Boolean | No | Whether an empty sub-table is accepted rather than raising an error. Default ''.yes''. |
+| ''tableManager'' | Input | TableManager | Yes | Auto-binds to the enclosing [[#tablemanager|TableManager]] container; normally left unconnected. |
+| ''object'' | Output | Table | -- | The same sub-table passed in, returned unchanged so it can still be referenced or connected onward. |
+Every ''SubTable'' call sharing a ''tableName'' is checked against the first one registered under that name: the ''keys'' tuple must have the same element types, and ''subTable'' must have the same columns. A mismatch, or registering the same ''keys'' twice under the same ''tableName'', raises an error.
+==== MergeSubTables ====
+Combines every sub-table registered under a name into a base table, then discards them from the manager.
+^ Port ^ Direction ^ Type ^ Required? ^ Description ^
+| ''baseTable'' | Input | Table | Yes | The table the combined sub-tables are inserted into. |
+| ''tableName'' | Input | Name | Yes | Which set of sub-tables to merge. |
+| ''allowNamesWithNoAssociatedSubTables'' | Input | Boolean | No | If ''.no'', merging a name nothing was ever registered under raises an error; if ''.yes'', it's silently skipped. Default ''.no''. |
+| ''tableManager'' | Input | TableManager | Yes | Auto-binds to the enclosing [[#tablemanager|TableManager]] container. |
+| ''table'' | Output | Table | -- | ''baseTable'' with every sub-table registered under ''tableName'' inserted, each keyed by the Tuple it was registered with. |
+Each sub-table's ''keys'' tuple can hold more than one element; every element becomes its own leading key column on ''baseTable''. A ''tableName'' can only be merged once -- the merge removes its sub-tables from the manager as it runs.
+> **Warning:** ''MergeSubTables'' has no data dependency on the ''SubTable'' calls that feed it -- its ''baseTable'' input, its literal ''tableName'', and its auto-bound manager are all available from the moment the container starts, and none of them depend on the loop that registered the sub-tables. Placing it after a loop in the script, even inside the same ''TableManager'' container, does **not** guarantee it runs after that loop finishes; the two have no connection, so the engine is free to run them in either order or at the same time. Force the ordering explicitly with a [[Group]] gated by the loop's ''sequenceOutput'', as in the example below -- see [[basic_data_flow#ordering_without_data_sequence_connections|Ordering without data: sequence connections]] and [[ego_script#sequence_ports|Sequence ports]].
+==== TableManagerValue ====
+An explicit pass-through for a TableManager instance, for the rare case where auto-bind doesn't reach -- for instance, threading a manager through a submodel boundary as a named port rather than relying on ambient container nesting. Within a single model, auto-bind on ''SubTable'' and ''MergeSubTables'' makes this unnecessary.
+^ Port ^ Direction ^ Type ^ Required? ^ Description ^
+| ''tableManager'' | Input | TableManager | No | Auto-binds to the enclosing [[#tablemanager|TableManager]] container. |
+| ''tableManager'' | Output | TableManager | -- | The same instance, under an explicit name. |
+==== Example ====
+Building a two-key table from per-year sub-tables computed inside a loop, with the merge correctly sequenced after the loop:
+<code>
+emptyPriceTable := Table [ "Year*#real", "City*#string", "Price#real" ];
+TableManager {{
+    // forDone carries no data of its own; it exists purely so the Group
+    // below can force itself to wait for every iteration of this loop.
+    forDone := For 2004 2007 {{
+        year := Step;
+        // Stand-in for a per-year computation; in practice this might come
+        // from a database query, a Calculate Table Values call, or another
+        // model entirely. A Table literal only accepts literal cell values,
+        // so a real per-year computation would build this row by row with
+        // AddTupleValue and AddTableRow instead, the same as any table
+        // whose contents aren't known until the model runs.
+        yearPrices := Table [
+            "City*", "Price",
+            "Boston", 4000,
+            "Chelsea", 200
+        ];
+        _ := SubTable "prices" year yearPrices;
+    }};
+    // Group's sequenceInput, fed from the loop's sequenceOutput, forces
+    // this Group -- and MergeSubTables inside it -- to wait for the whole
+    // loop to finish, without threading a Mux through its iterations.
+    _ := Group forDone {{
+        priceTable := MergeSubTables emptyPriceTable "prices";
+    }};
+}};
+</code>
+''priceTable'' is usable here, once the Group it was produced inside — and through its ''sequenceInput'', the loop that Group was forced to wait for — has finished.
+The loop's own iterations are still eligible to run in parallel with each other: nothing inside it is a Mux, nothing inside it is consumed from outside it, and it feeds no submodel output port -- see [[basic_data_flow|Basic Data Flow]]. Only the loop as a whole is sequenced against the merge, which is why this avoids the cost the Mux-accumulator pattern above pays on every single iteration.
+> **Note:** ''year'' is passed to ''SubTable'' directly, not wrapped in brackets as ''[year]''. Bracket syntax is for Tuple **literals**; a connected Real value converts to a one-element Tuple automatically on connection, the same pattern used by ''GetTableFromKey'' in [[#sub-tables|Sub-Tables]] above.
 ===== Printing a table with a generic format =====
@@ Line 502: / Line 600: @@
 [[Get Table Row]], [[Get Tuple Size]], and [[Get Tuple Value]] make the column-count side of this possible: a row read as a Tuple can be measured with ''GetTupleSize'', and each of its elements retrieved by position, rather than by column name, with ''GetTupleValue''. Each retrieved element has type ''TableValue'' — a generic type able to hold either a Real or a String.
-[[String]] and [[Real Value]] each convert a ''TableValue'' the same conditional way: the conversion only succeeds if the ''TableValue'' actually holds that type, and raises an error otherwise — converting an already-typed, known ''RealValue'' or String, by contrast, never fails. Attempting the String conversion wrapped in [[Skip On Error]] turns that failure into a type test: ''SkipOnError'' catches the error rather than aborting the model, and its ''executionCompletedSucessfully'' output reports whether the ''TableValue'' really was a String. [[If Then]] and [[If Not Then]] branch on that result and take the matching path. The first uses the converted String directly. The second has already ruled String out, so it extracts a ''RealValue'' instead — safe now, since only Real remains possible — and converts //that// known value to a string. A [[String Junction]] then recombines the two branches into one result, since only one of them ever actually runs.
+[[String]] and [[Real Value]] each convert a ''TableValue'' the same conditional way: the conversion only succeeds if the ''TableValue'' actually holds that type, and raises an error otherwise — converting an already-typed, known ''RealValue'' or String, by contrast, never fails. Wrapping each attempt in its own [[Skip On Error]] turns that failure into "produced nothing" instead of aborting the model, so only the attempt that actually matches the ''TableValue'''s real type ever produces a display string; the other attempt simply fails silently and produces nothing. A [[String Junction]] then picks whichever one exists.
+> **Note:** A junction always tries its first port before its second — see [[ego_script#carrying_and_selecting_values_across_iterations|Carrying and selecting values across iterations]] for this rule stated generally, since it applies identically to every Junction functor (''String'', ''Table'', ''Value'', ''Lookup Table'', ''Map'', ''Categorical Map'', ''Folder''). It makes no practical difference here, though: the two attempts below are mutually exclusive by construction, so only one of ''stringDisplay''/''realDisplay'' is ever available for the junction to pick regardless of port order.
 <code>
@@ Line 528: / Line 628: @@
             displayColumn := $ [ $columnIndex - 1 ];
-            // Attempt to read the cell as a String; SkipOnError catches the
+            // Try the cell as a String; SkipOnError leaves stringDisplay
-            // failure if it's actually a Real, rather than aborting.
+            // unproduced if cellValue is actually a Real, rather than
-            attempt := SkipOnError .yes {{
+            // aborting the model.
-                asString := String cellValue;
+            _ := SkipOnError .yes {{
-            }};
+                stringDisplay := String cellValue;
-            _ := IfThen attempt {{
-                stringDisplay := String asString;
             }};
-            _ := IfNotThen attempt {{
+            // Try the cell as a Real instead; SkipOnError leaves
-                // cellValue was a Real instead; RealValue extracts it, and
+            // realDisplay unproduced if cellValue is actually a String.
-                // String — safe here, since converting a known Real never
+            _ := SkipOnError .yes {{
-                // fails — turns it into a display string.
                 asReal := RealValue cellValue;
                 realDisplay := String asReal;
             }};
-            // Only one of stringDisplay/realDisplay was actually produced;
+            // Exactly one of the two attempts above ever succeeds;
-            // StringJunction picks whichever one has a value.
+            // StringJunction picks whichever one was produced.
             cellDisplay := StringJunction stringDisplay realDisplay;
@@ Line 693: / Line 789: @@
 Four muxes are threaded through this loop, at two nesting levels. Inside a station, two separate ''MuxValue'' passes sum that station's own daily values — one for the average, a second for the squared deviations from it, since the standard deviation formula needs the average already known before it can compute those deviations. Outside them, two ''MuxLookupTable'' accumulators carry the growing results across stations, one per statistic. [[Set Lookup Table Value]] is what actually inserts each computed value into its growing table — the write counterpart to ''GetLookupTableValue''. ''RealValue cellValue'' is used directly here, without the ''SkipOnError'' type test from the generic printer — this example assumes every data column is genuinely numeric, which is reasonable for daily measurements, unlike the printer, which had to handle either type. The second pass's variable names all carry a ''dev'' marker (''devDayIndex'', ''devCellValue'', ''devDailyValue'', ''devRunningSum'', ''nextDevRunningSum'') to keep them clearly distinct from the first pass's, even though each ''For'' loop is its own separate container.
-Those muxes disqualify this loop from parallel execution — see [[basic_data_flow#the_one_exception_feedback_in_loops|The one exception: feedback in loops]] — consistent with every mux-based accumulator already documented on this page.
 ==== Looping column by column ====
@@ Line 719: / Line 813: @@
     runningSums := MuxLookupTable initialSums nextRunningSums;
     // Adds this day's value for each station to that station's
-    // running total, computed for every station in one call.
+    // running total, computed for every station in one call. The [2]
+    // names dayColumn's single data column explicitly; it can be left
+    // out — [[line]] alone reads the first data column by default.
     nextRunningSums := % [ column + %dayColumn[[line][2]] ] "StationId" "Sum" runningSums;
 }};
@@ Line 743: / Line 839: @@
 </code>
-''firstStationId'' picks an arbitrary station just to measure the row shape once, the same ''GetTableRow''/''GetTupleSize'' technique as above, minus 1 for the key — every row has the same number of columns, so any one station works. ''initialSums'' starts every station at zero via a single-call ''%'', evaluated once per key of ''allStationIds'' — the base lookup table supplying the station keys — with no other operand referenced. Inside the first loop, ''%dayColumn<nowiki>[[line][2]]</nowiki>'' reads the current day's value for the row currently being computed — the multi-column operator, since ''dayColumn'' is a Table, not a Lookup Table — while ''column'' reads that same row's running total from ''runningSums'', the base table the whole expression is evaluated against.
+''firstStationId'' picks an arbitrary station just to measure the row shape once, the same ''GetTableRow''/''GetTupleSize'' technique as above, minus 1 for the key — every row has the same number of columns, so any one station works. ''initialSums'' starts every station at zero via a single-call ''%'', evaluated once per key of ''allStationIds'' — the base lookup table supplying the station keys — with no other operand referenced. Inside the first loop, ''%dayColumn<nowiki>[[line][2]]</nowiki>'' reads the current day's value for the row currently being computed — the multi-column operator, since ''dayColumn'' is a Table, not a Lookup Table — while ''column'' reads that same row's running total from ''runningSums'', the base table the whole expression is evaluated against. Converting first is possible — ''LookupTable dayColumn'' — and the Lookup Table operator is cheaper per query, but the conversion walks the whole column before any query happens; for a table consulted once per pass, as here, that up-front cost can outweigh what it saves.
 The second loop repeats this shape for the sum of squared deviations, referencing ''%averages[line]'' — the already-computed per-station mean from the first loop, looked up by key — the same chaining technique already established in [[#computing_new_columns_for_a_table|Computing new columns for a table]]. Its variables all carry a ''dev'' marker (''devDayNumber'', ''devRawColumnIndex'', ''devDayColumn'', ''devRunningSums'', ''nextDevRunningSums'') to keep them clearly distinct from the first loop's, even though each ''For'' loop is its own separate container. Two muxes thread through this version instead of the four the row-by-row approach needed — one ''MuxLookupTable'' carrying the running sums day to day, a second carrying the running squared-deviation sums — and both sit at the same level; there's still no inner mux within either pass, since ''%'' handles summing across all stations for a given day in one call rather than needing an accumulator of its own.
@@ Line 871: / Line 967: @@
 </code>
-Unlike every example in the previous section, neither of these needs a loop, a mux, or the parallelism cost that comes with one — ''GetTableFromKey'' does the equivalent of what a whole ''For'' loop with ''GetTableColumn'' did before, in a single call. Neither ''stationSubTable''/''daySubTable'' going into ''ExtractLookupTableAttributes'', nor ''attrs'' going into the ''tX["NAME"]'' operator right after, needs an explicit ''LookupTable'' carrier — the first is a direct connection between two ports, and the second reads straight from the Lookup Table that operator is built for.
+Unlike every example in the previous section, neither of these needs a loop or a mux — ''GetTableFromKey'' does the equivalent of what a whole ''For'' loop with ''GetTableColumn'' did before, in a single call. Neither ''stationSubTable''/''daySubTable'' going into ''ExtractLookupTableAttributes'', nor ''attrs'' going into the ''tX["NAME"]'' operator right after, needs an explicit ''LookupTable'' carrier — the first is a direct connection between two ports, and the second reads straight from the Lookup Table that operator is built for.
 This "no loop" advantage is specific to querying a single, known key. Computing the same statistic for **every** station, or **every** day, still needs a loop over that set of keys — though only one level of it, since each pass resolves its whole aggregate in a single ''ExtractLookupTableAttributes'' call, with no inner accumulator the way the wide-format loops needed one.
@@ Line 930: / Line 1026: @@
 </code>
-Both follow the same shape as the single-key examples above, just wrapped in a ''ForEach'' over every key and accumulated with ''MuxLookupTable'' — one accumulator per statistic — the same accumulator pattern used throughout this page, disqualifying these loops from parallel execution the same way.
+Both follow the same shape as the single-key examples above, just wrapped in a ''ForEach'' over every key and accumulated with ''MuxLookupTable'' — one accumulator per statistic — the same accumulator pattern used throughout this page.
 ===== Printing a table with any number of key columns =====
@@ Line 949: / Line 1045: @@
 ];
-result := CalculatePythonExpression (String $"(
+result := CalculatePythonExpression $"(
 inputTable = dinamica.inputs['t1']
 header = inputTable[0]
 rows = inputTable[1:]
-# A column name may carry a "#type" suffix after its "*" key marker, so
+# header holds plain column names only, so nothing needs stripping here.
-# the marker is stripped from the name part alone and the suffix put back
-# untouched.
-def dropKeyMarker(name):
-    columnName, separator, columnType = name.partition('#')
-    return columnName.rstrip('*') + separator + columnType
 # Every original column, key or not, becomes a plain data column; a single
 # sequential integer becomes the new (and only) key.
-newHeader = ['Id*'] + [dropKeyMarker(name) for name in header]
+newHeader = ['Id*'] + list(header)
 newRows = [[i + 1] + list(row) for i, row in enumerate(rows)]
 newTable = [newHeader] + newRows
 dinamica.outputs['reshapedTable'] = dinamica.prepareTable(newTable, 1)
-)") {{
+)" {{
     NumberTable myTable 1;
 }};
@@ Line 990: / Line 1080: @@
             // Same generic Real/String printing technique as the earlier
-            // example: attempt String first, SkipOnError catches the
+            // example: try both conversions independently, each guarded by
-            // failure if it's actually a Real, RealValue then converts it.
+            // its own SkipOnError, and let StringJunction pick whichever
-            attempt := SkipOnError .yes {{
+            // one was actually produced.
-                asString := String cellValue;
+            _ := SkipOnError .yes {{
+                stringDisplay := String cellValue;
             }};
-            _ := IfThen attempt {{
+            _ := SkipOnError .yes {{
-                stringDisplay := String asString;
-            }};
-            _ := IfNotThen attempt {{
                 asReal := RealValue cellValue;
                 realDisplay := String asReal;
@@ Line 1019: / Line 1106: @@
 </code>
-The original key columns' names lose their ''*'' marker as the new header is built, since they're plain data now — the marker comes off the name alone, so an explicit ''#type'' suffix survives it — and the new ''Id'' column carries the marker instead, as the only key. Everything after the Python call is the same single-key iteration and generic value-printing already established — nothing about it needs to know how many keys the original table had.
+''dinamica.inputs'' passes column names alone — no ''*'' key markers and no ''#type'' annotations — so the Python side has nothing to strip, and no way to tell which of the original columns were keys. Losing that distinction costs nothing here, since the reshape turns every original column into a plain data column regardless. The outgoing header marks ''Id'' instead, and ''prepareTable'''s second argument declares that one column as the key. Everything after the Python call is the same single-key iteration and generic value-printing already established — nothing about it needs to know how many keys the original table had.
 ===== Choosing between Tables and Lookup Tables =====
@@ Line 1026: / Line 1113: @@
 Reach for a general Table instead when a single Real value per Real key isn't enough — when a row needs more than one associated value, or a String value, or a String key, or when rows need to be indexed by more than one key at once. A String key column costs more than the type change alone: iterating one means mapping each placeholder index back to its actual key inside the loop, so prefer a Real key column wherever the data allows.