Differences

This shows you the differences between two versions of the page.

--- manipulating_tables_and_lookup_tables [2026/07/28 21:33]
hermann
+++ manipulating_tables_and_lookup_tables [2026/08/18 03:09] (current)
hermann [Printing a table with any number of key columns]
@@ Line 497: / Line 497: @@
 One difference in shape: the loop accumulates into a Table, since ''emptyResults'' was declared as one, while ''%'' always returns a Lookup Table. Where the Table form is the one wanted, a ''Table'' carrier converts it — ''asTable := Table doubledResults'' — one Real key and one Real value fitting both types equally well.
+===== Table Manager =====
+The accumulator pattern in [[#storing_results_across_a_loop|Storing results across a loop]] uses a Mux to carry a growing table from one iteration to the next -- which, as that section's closing note explains, forces the loop to run sequentially, one iteration at a time. **Table Manager** solves the same class of problem -- building a table with an extra key column from results computed across a loop -- without that cost: each iteration registers its own sub-table independently, with no Mux and nothing threaded between iterations, so the loop producing those partial results stays eligible to run in parallel, per [[basic_data_flow|Basic Data Flow]]. A single [[#mergesubtables|MergeSubTables]] call combines everything afterward.
+A container and two functors do the core work; a fourth covers a narrower case:
+==== TableManager ====
+A container functor. It creates the manager instance and exposes it to whatever runs inside it.
+^ Port ^ Direction ^ Type ^ Description ^
+| ''tableManager'' | Internal Output | TableManager | The manager instance. Auto-binds to the ''tableManager'' input of [[#subtable|SubTable]] and [[#mergesubtables|MergeSubTables]] calls nested anywhere inside this container, including inside further nested containers such as a [[For]] loop or a [[Group]]. |
+The container itself takes no inputs and defines no regular outputs.
+==== SubTable ====
+Registers one sub-table under a name and a key.
+^ Port ^ Direction ^ Type ^ Required? ^ Description ^
+| ''tableName'' | Input | Name | Yes | Identifies which set of sub-tables this one joins. |
+| ''keys'' | Input | Tuple | Yes | The key identifying this sub-table within the set. |
+| ''subTable'' | Input | Table | Yes | The sub-table itself. |
+| ''emptySubTableIsAllowed'' | Input | Boolean | No | Whether an empty sub-table is accepted rather than raising an error. Default ''.yes''. |
+| ''tableManager'' | Input | TableManager | Yes | Auto-binds to the enclosing [[#tablemanager|TableManager]] container; normally left unconnected. |
+| ''object'' | Output | Table | -- | The same sub-table passed in, returned unchanged so it can still be referenced or connected onward. |
+Every ''SubTable'' call sharing a ''tableName'' is checked against the first one registered under that name: the ''keys'' tuple must have the same element types, and ''subTable'' must have the same columns. A mismatch, or registering the same ''keys'' twice under the same ''tableName'', raises an error.
+==== MergeSubTables ====
+Combines every sub-table registered under a name into a base table, then discards them from the manager.
+^ Port ^ Direction ^ Type ^ Required? ^ Description ^
+| ''baseTable'' | Input | Table | Yes | The table the combined sub-tables are inserted into. |
+| ''tableName'' | Input | Name | Yes | Which set of sub-tables to merge. |
+| ''allowNamesWithNoAssociatedSubTables'' | Input | Boolean | No | If ''.no'', merging a name nothing was ever registered under raises an error; if ''.yes'', it's silently skipped. Default ''.no''. |
+| ''tableManager'' | Input | TableManager | Yes | Auto-binds to the enclosing [[#tablemanager|TableManager]] container. |
+| ''table'' | Output | Table | -- | ''baseTable'' with every sub-table registered under ''tableName'' inserted, each keyed by the Tuple it was registered with. |
+Each sub-table's ''keys'' tuple can hold more than one element; every element becomes its own leading key column on ''baseTable''. A ''tableName'' can only be merged once -- the merge removes its sub-tables from the manager as it runs.
+> **Warning:** ''MergeSubTables'' has no data dependency on the ''SubTable'' calls that feed it -- its ''baseTable'' input, its literal ''tableName'', and its auto-bound manager are all available from the moment the container starts, and none of them depend on the loop that registered the sub-tables. Placing it after a loop in the script, even inside the same ''TableManager'' container, does **not** guarantee it runs after that loop finishes; the two have no connection, so the engine is free to run them in either order or at the same time. Force the ordering explicitly with a [[Group]] gated by the loop's ''sequenceOutput'', as in the example below -- see [[basic_data_flow#ordering_without_data_sequence_connections|Ordering without data: sequence connections]] and [[ego_script#sequence_ports|Sequence ports]].
+==== TableManagerValue ====
+An explicit pass-through for a TableManager instance, for the rare case where auto-bind doesn't reach -- for instance, threading a manager through a submodel boundary as a named port rather than relying on ambient container nesting. Within a single model, auto-bind on ''SubTable'' and ''MergeSubTables'' makes this unnecessary.
+^ Port ^ Direction ^ Type ^ Required? ^ Description ^
+| ''tableManager'' | Input | TableManager | No | Auto-binds to the enclosing [[#tablemanager|TableManager]] container. |
+| ''tableManager'' | Output | TableManager | -- | The same instance, under an explicit name. |
+==== Example ====
+Building a two-key table from per-year sub-tables computed inside a loop, with the merge correctly sequenced after the loop:
+<code>
+emptyPriceTable := Table [ "Year*#real", "City*#string", "Price#real" ];
+TableManager {{
+    // forDone carries no data of its own; it exists purely so the Group
+    // below can force itself to wait for every iteration of this loop.
+    forDone := For 2004 2007 {{
+        year := Step;
+        // Stand-in for a per-year computation; in practice this might come
+        // from a database query, a Calculate Table Values call, or another
+        // model entirely. A Table literal only accepts literal cell values,
+        // so a real per-year computation would build this row by row with
+        // AddTupleValue and AddTableRow instead, the same as any table
+        // whose contents aren't known until the model runs.
+        yearPrices := Table [
+            "City*", "Price",
+            "Boston", 4000,
+            "Chelsea", 200
+        ];
+        _ := SubTable "prices" year yearPrices;
+    }};
+    // Group's sequenceInput, fed from the loop's sequenceOutput, forces
+    // this Group -- and MergeSubTables inside it -- to wait for the whole
+    // loop to finish, without threading a Mux through its iterations.
+    _ := Group forDone {{
+        priceTable := MergeSubTables emptyPriceTable "prices";
+    }};
+}};
+</code>
+''priceTable'' is usable here, once the Group it was produced inside — and through its ''sequenceInput'', the loop that Group was forced to wait for — has finished.
+The loop's own iterations are still eligible to run in parallel with each other: nothing inside it is a Mux, nothing inside it is consumed from outside it, and it feeds no submodel output port -- see [[basic_data_flow|Basic Data Flow]]. Only the loop as a whole is sequenced against the merge, which is why this avoids the cost the Mux-accumulator pattern above pays on every single iteration.
+> **Note:** ''year'' is passed to ''SubTable'' directly, not wrapped in brackets as ''[year]''. Bracket syntax is for Tuple **literals**; a connected Real value converts to a one-element Tuple automatically on connection, the same pattern used by ''GetTableFromKey'' in [[#sub-tables|Sub-Tables]] above.
 ===== Printing a table with a generic format =====
@@ Line 504: / Line 600: @@
 [[Get Table Row]], [[Get Tuple Size]], and [[Get Tuple Value]] make the column-count side of this possible: a row read as a Tuple can be measured with ''GetTupleSize'', and each of its elements retrieved by position, rather than by column name, with ''GetTupleValue''. Each retrieved element has type ''TableValue'' — a generic type able to hold either a Real or a String.
-[[String]] and [[Real Value]] each convert a ''TableValue'' the same conditional way: the conversion only succeeds if the ''TableValue'' actually holds that type, and raises an error otherwise — converting an already-typed, known ''RealValue'' or String, by contrast, never fails. Attempting the String conversion wrapped in [[Skip On Error]] turns that failure into a type test: ''SkipOnError'' catches the error rather than aborting the model, and its ''executionCompletedSucessfully'' output reports whether the ''TableValue'' really was a String. [[If Then]] and [[If Not Then]] branch on that result and take the matching path. The first uses the converted String directly. The second has already ruled String out, so it extracts a ''RealValue'' instead — safe now, since only Real remains possible — and converts //that// known value to a string. A [[String Junction]] then recombines the two branches into one result, since only one of them ever actually runs.
+[[String]] and [[Real Value]] each convert a ''TableValue'' the same conditional way: the conversion only succeeds if the ''TableValue'' actually holds that type, and raises an error otherwise — converting an already-typed, known ''RealValue'' or String, by contrast, never fails. Wrapping each attempt in its own [[Skip On Error]] turns that failure into "produced nothing" instead of aborting the model, so only the attempt that actually matches the ''TableValue'''s real type ever produces a display string; the other attempt simply fails silently and produces nothing. A [[String Junction]] then picks whichever one exists.
+> **Note:** A junction always tries its first port before its second — see [[ego_script#carrying_and_selecting_values_across_iterations|Carrying and selecting values across iterations]] for this rule stated generally, since it applies identically to every Junction functor (''String'', ''Table'', ''Value'', ''Lookup Table'', ''Map'', ''Categorical Map'', ''Folder''). It makes no practical difference here, though: the two attempts below are mutually exclusive by construction, so only one of ''stringDisplay''/''realDisplay'' is ever available for the junction to pick regardless of port order.
 <code>
@@ Line 530: / Line 628: @@
             displayColumn := $ [ $columnIndex - 1 ];
-            // Attempt to read the cell as a String; SkipOnError catches the
+            // Try the cell as a String; SkipOnError leaves stringDisplay
-            // failure if it's actually a Real, rather than aborting.
+            // unproduced if cellValue is actually a Real, rather than
-            attempt := SkipOnError .yes {{
+            // aborting the model.
-                asString := String cellValue;
+            _ := SkipOnError .yes {{
-            }};
+                stringDisplay := String cellValue;
-            _ := IfThen attempt {{
-                stringDisplay := String asString;
             }};
-            _ := IfNotThen attempt {{
+            // Try the cell as a Real instead; SkipOnError leaves
-                // cellValue was a Real instead; RealValue extracts it, and
+            // realDisplay unproduced if cellValue is actually a String.
-                // String — safe here, since converting a known Real never
+            _ := SkipOnError .yes {{
-                // fails — turns it into a display string.
                 asReal := RealValue cellValue;
                 realDisplay := String asReal;
             }};
-            // Only one of stringDisplay/realDisplay was actually produced;
+            // Exactly one of the two attempts above ever succeeds;
-            // StringJunction picks whichever one has a value.
+            // StringJunction picks whichever one was produced.
             cellDisplay := StringJunction stringDisplay realDisplay;
@@ Line 951: / Line 1045: @@
 ];
-result := CalculatePythonExpression (String $"(
+result := CalculatePythonExpression $"(
 inputTable = dinamica.inputs['t1']
 header = inputTable[0]
 rows = inputTable[1:]
-# A column name may carry a "#type" suffix after its "*" key marker, so
+# header holds plain column names only, so nothing needs stripping here.
-# the marker is stripped from the name part alone and the suffix put back
-# untouched.
-def dropKeyMarker(name):
-    columnName, separator, columnType = name.partition('#')
-    return columnName.rstrip('*') + separator + columnType
 # Every original column, key or not, becomes a plain data column; a single
 # sequential integer becomes the new (and only) key.
-newHeader = ['Id*'] + [dropKeyMarker(name) for name in header]
+newHeader = ['Id*'] + list(header)
 newRows = [[i + 1] + list(row) for i, row in enumerate(rows)]
 newTable = [newHeader] + newRows
 dinamica.outputs['reshapedTable'] = dinamica.prepareTable(newTable, 1)
-)") {{
+)" {{
     NumberTable myTable 1;
 }};
@@ Line 992: / Line 1080: @@
             // Same generic Real/String printing technique as the earlier
-            // example: attempt String first, SkipOnError catches the
+            // example: try both conversions independently, each guarded by
-            // failure if it's actually a Real, RealValue then converts it.
+            // its own SkipOnError, and let StringJunction pick whichever
-            attempt := SkipOnError .yes {{
+            // one was actually produced.
-                asString := String cellValue;
+            _ := SkipOnError .yes {{
+                stringDisplay := String cellValue;
             }};
-            _ := IfThen attempt {{
+            _ := SkipOnError .yes {{
-                stringDisplay := String asString;
-            }};
-            _ := IfNotThen attempt {{
                 asReal := RealValue cellValue;
                 realDisplay := String asReal;
@@ Line 1021: / Line 1106: @@
 </code>
-The original key columns' names lose their ''*'' marker as the new header is built, since they're plain data now — the marker comes off the name alone, so an explicit ''#type'' suffix survives it — and the new ''Id'' column carries the marker instead, as the only key. Everything after the Python call is the same single-key iteration and generic value-printing already established — nothing about it needs to know how many keys the original table had.
+''dinamica.inputs'' passes column names alone — no ''*'' key markers and no ''#type'' annotations — so the Python side has nothing to strip, and no way to tell which of the original columns were keys. Losing that distinction costs nothing here, since the reshape turns every original column into a plain data column regardless. The outgoing header marks ''Id'' instead, and ''prepareTable'''s second argument declares that one column as the key. Everything after the Python call is the same single-key iteration and generic value-printing already established — nothing about it needs to know how many keys the original table had.
 ===== Choosing between Tables and Lookup Tables =====
@@ Line 1028: / Line 1113: @@
 Reach for a general Table instead when a single Real value per Real key isn't enough — when a row needs more than one associated value, or a String value, or a String key, or when rows need to be indexed by more than one key at once. A String key column costs more than the type change alone: iterating one means mapping each placeholder index back to its actual key inside the loop, so prefer a Real key column wherever the data allows.