====== What is new in stable version 8.5.0 ====== (Compared to stable version [[what_is_new_8_3_0|8.3.0]]) The recent updates to the core system and the GUI (Graphical User Interface) significantly enhance both the **overall performance** and the **user experience** of the application. ===== Performance Improvements ===== Many core system updates focus on **optimizing resource utilization**, **speeding up operations**, and **improving efficiency**: ==== Compilation and Build Efficiency ==== * Compilation warnings and errors have been fixed. * Support for C++20 has been added. * Scripts for configuring the build, compiling source files, and installing libraries and utilities have been added. * Performance flags have been added to Windows builds in CMake. * Memory errors caused by compiled expressions have been fixed. ==== Memory Management and Data Handling ==== * A memory leak on Distributed has been fixed. * Unnecessary data copying has been reduced or eliminated in parallel constructs and during data transfer to viewers. * Basic data types used for port data transfer have been refactored and optimized, eliminating the need for casting, improving construction and destruction efficiency, and simplifying inheritance relationships. * The way data is copied during transfer between data holders has been changed to prevent multiple copies. ==== Algorithm and Data Structure Optimizations ==== * The **`CalcPatchLabelMap` execution has been massively optimized** through a complete re-implementation, maintaining the same basic algorithm but improving execution time and performance. * `std::map` has been replaced with `std::unordered_map` where possible. * The `stringstream` use has been replaced with the `fmt` library for category name creation, leading to faster execution. * `std::unordered_set` is now used instead of `std::set` when dealing with task graphs. * `DFF::toString` implementations for float and double have been replaced with `fmt::format`, and direct stream writing using `fmt::format` is now utilized for faster operations. * `LookupTable` functions have been optimized. * `ExtractCategoricalMapAttributes` has been optimized by parallelizing the insertion of attributes into the table. * Optimizations have been added for copying subtables as-is and moving column chunks instead of copying rows. * A fast-double-parser is now used for parsing doubles when reading CSV tables. * The `CalcHistogram` functor has been improved with better floating-point support and performance. * The use of **`gtl::btree_map` has been experimentally introduced to replace `std::map`** in several core data structures, leveraging its potential for better cache-friendliness and performance in: * `LookupTable`. * `Categorization`. * `SparseRasterImage`. * `Properties`. ==== Concurrency and Parallelism ==== * Tasks added to task groups can now be dispatched immediately. * `Extract Categorical Map Attributes` now computes mode, median, average/variance/stddev concurrently. * Subtable merging can now utilize tasks, and table normalization from the UI uses tasks. ==== Resource Management and System Interactions ==== * An event indicating a port received new data was removed to prevent unnecessary performance degradation. * Atomic normalization was removed from tables as it was unused. * Searching for modules that hurt MSVC performance on Windows has been disabled. * Optional `mimalloc` support has been added to Windows builds. * Python scripts are now executed as independent sub-processes, which includes improvements like allowing `dinamica` python to detect termination requests and making Linux system processes use default environment variables. * A custom `AtomicMutex/Lock` (SpinLock) was added to Table to allow locking/unlocking between different threads. ===== User Experience Improvements ===== Both core and GUI updates contribute to a **more intuitive, stable, and feature-rich user experience**: ==== Application Stability and Reliability ==== * Numerous bug fixes address issues such as: * Errors with compiled expressions and invalid data to `RunRemotely`. * Incorrect file path conversions. * Python assertions related to UTF-8 conversion. * Regression bugs. * Issues with parallel constructs. * Memory leaks in the GUI, especially in table editors/viewers, are prevented. * Crashes when visualizing complex image trees. * An inconsistency where a viewer could retain outdated content has been fixed. * A temporary workaround was implemented for a potential crash when joining subtables through the task system. * A **code fix was required to prevent crashes when using `gtl::btree_map` with `SparseRasterImage`**. * Error messages have been improved for clarity and grammar. * The application version has been updated to 8.4.0 and then to 8.5.0. * A custom `AtomicMutex/Lock` (SpinLock) has been added to `Table` for inter-thread locking. ==== Enhanced Functionality and Features ==== * New resampling methods (weighted-sum and root-mean-square) have been added to `TransformMapFunctor`. * Functionality to verify coordinator availability is now exposed to Java and available in the GUI. * Python script execution improvements include detecting termination requests and adding `dinamica`'s Python path to the process. An advanced optional input port for packages has been added to `CalculatePythonExpression`. * The `GetRegionInfo` functor has been introduced for accessing region details. * The ability to expand all submodels in the selected script has been implemented. * An action to import the currently selected submodel has been added. * New features include adding `getTemporaryPathFreeSpace` and allowing agent IDs and temporary space to requirement expressions for `RunRemotely`. * Dynamic debugging support has been added to CMake. * The application now allows tables to have any name format on their column titles. ==== Improved User Interface and Navigation ==== * Updates to the update manager provide better selection between stable and development releases, clearer messages, and fixes for update notifications. * A temporary blinking border has been added around editors and options dialogs where validation errors occur or settings are edited via shortcuts, drawing user attention. * Information about ports is always displayed in the functor editor window, even if the port cannot be edited. The message displayed by port editors when a port cannot be edited has been clarified. * The design of the "Previous-Next History" has been improved with consistent layout, adjusted panel sizes, and standardized borders. * A filter to display only selected submodels has been added. * The Explorer panel's rendering artifacts have been fixed. * Tooltips have been added to UI elements like the `showSelectedItemsButton` and the option to control redundant carrier removal. * Table loading now uses `LazyPresentationPanel` and existing indices to speed up display. * The GUI is prevented from hanging when the log is filled with messages. ==== Streamlined Carrier and Submodel Management ==== * Redundant carrier utilities have been centralized in the C++ codebase, with a new method to remove all redundant carriers from the script. * A flag in `ScriptRunner` allows automatic removal of redundant carriers, with console support. * An action to remove all redundant carriers from the script has been added to the GUI. * The carrier removal process has been simplified by merging the redundancy check into a single dialog with clear options (Remove and Readjust, Just Remove, or Cancel). * The default behavior for redundant carrier removal before model execution has been changed to `false` for safety, preventing unintended semantic alterations or visualization issues. * Submodel pendencies are now ignored if all derived pendencies causing them are ignored. * Selection is no longer cleared after submodel installation, and `CheckBoxSelectionModel` selection handling has been refactored for better clarity. ==== Enhanced Debugging and Diagnostics ==== * Information about the functor corresponding to each running task is always shown in the execution summary tooltip. * An option to force the generation of core dumps when the application crashes has been added. * Improved Python error reporting functions. ---------- ====== Full List of GUI Updates ====== ===== General GUI Improvements ===== * **Formatting fixes** have been applied throughout the code. * The `isAutoUpdateSupported()` method was removed as auto-update is now supported across all operating systems. * The function reporting if the application is 64 bits was removed, as the application is now always 64 bits. * The `Test::runScript` function's documentation for the `optimizeScript` flag has been fixed. * The GUI is now prevented from hanging when the log is constantly filled with messages. ===== Update Manager Enhancements ===== * The **update manager has been improved** to better select between stable and development releases. * **Messages have been clarified** within the update manager. * The pending-update icon no longer loses its tooltip under certain circumstances. * Minor issues related to update notifications showing on the system tray have been fixed. ===== Java Export and Integration Improvements ===== * How functions from the **ProxyFunctorUtils, CellTypeInfo, LogTagInfo, ScriptUtils, SubmodelUtils, NameUtils, and StringUtils namespaces are exported to Java** has been changed to emulate classes with static members. * Some Java exported functions were moved from the `SubmodelUtils` and `ScriptUtils` namespaces to the `Script` class in Java. * Functions related to **projection** are now exported as static members of the `Projection` class in Java. * Functions related to the **comparison of versions as strings** are exported as static members of the `Version` class in Java. * Functionality to **verify coordinator availability** has been added and exposed to Java. * The Java `dff.LogMessage` now uses `char[]` instead of `string` to speed up logging. ===== Application Stability and Reliability ===== * A bug causing **rendering artifacts in the Explorer panel** has been fixed. * **Selection is no longer cleared after submodel installation**. * The handling of selection in `CheckBoxSelectionModel` has been refactored using a `TreeSelectionListener`. * Validation logic from `attemptConnection` was added to `canReadjustConnections` to prevent suggesting invalid connection readjustments. * A bug where closing the dialog still confirmed the action of expanding the submodel definition has been fixed. * An inconsistency where a viewer could retain outdated content if its equivalent peer did not have data during the last model execution has been fixed. * A bug that could trigger an assertion when visualizing a port while submodel expansion and redundant carrier removal were enabled simultaneously has been fixed. * **Grammar in error messages** has been improved, replacing "can not" with "cannot". ===== User Interface and Navigation Enhancements ===== * A **filter to display only selected submodels** has been added. * A **temporary blinking border** has been added around the editor when validation errors occur, or around relevant options in the OptionsDialog when settings are edited via shortcuts. * A method was defined to clarify the behavior when a functor is highlighted as responsible for an error. * Component placement has been fixed, removing unnecessary uses of `fill` and `weightx`. * The presentation of information for the `STB_IMAGE_WRITE` library in the About window has been fixed. * Information about a port is **always displayed in the functor editor window**, even if the port cannot be edited. * The message displayed by port editors when a port cannot be edited has been clarified. * The **design of the "Previous-Next History" has been improved** with consistent layout, adjusted panel sizes, standardized border thickness for selected/unselected items, and a JScrollPane that always shows a horizontal scrollbar. * Tooltips have been added to UI elements, such as the `showSelectedItemsButton` and the option to control the removal of redundant carriers. ===== Submodel and Carrier Management Streamlining ===== * Submodel pendencies are now treated as ignored when all derived pendencies causing them are ignored. * An action to **expand all submodels in the selected script** has been implemented. * An action to **import the currently selected submodel** has been added, and its name made more consistent. * Inconsistencies in the definition of accelerator keys for Submodel menu items have been fixed. * The order of "Sign In" and "Sign Out" items in the Submodel menu has been fixed. * The carrier readjustment logic has been fixed by using `dff.ScriptUtils.isCarrierRemovable` and direct `createConnection` calls. * An **action to remove all redundant carriers from the script** has been added, utilizing DFF redundant-carrier utilities. * The interface refresh logic after connection adjustments has been improved. * `reconnectAllFunctors()` was introduced to refresh interface connections after a removal. * A constraint was added to suggest redundant carrier connection adjustments only when removing a single functor, with more complex cases handled by the dedicated utility. * The **carrier removal process has been simplified** by merging the redundancy check into a single dialog with clear options: "Remove and Readjust", "Just Remove", or "Cancel". * The `removeRedundantCarriers` flag handling has been included in the interface. * The **default behavior for redundant carrier removal before model execution has been changed to `false`** for safety, preventing unintended semantic alterations or visualization issues. * The command-line flag controlling the removal of redundant carriers now enables the feature instead of disabling it. ===== Table Viewer and Editor Improvements ===== * `LazyPresentationPanel` is now used for **loading tables**, with task-based normalization when needed. * Pointers that could possibly leak memory on Table editors/viewers are now cleared. * **Displaying tables is faster** by using existing indices instead of copying them again. ===== Debugging and System Features ===== * An **option to force the generation of core dumps** when the application crashes has been added. * Information about the **functor corresponding to each running task is always shown in the execution summary tooltip**. * The Python environment reset now attempts to restart the application if Python libraries are in use. * Previously missing details about the licensing options of certain libraries used by the application have been included. ====== Full List of Core Updates ====== ===== General Core Improvements ===== * **Compilation warnings and errors** have been fixed, including specific fixes for GCC and C4100 warnings. * **Formatting** has been applied throughout the code. * The application version has been **updated to 8.4.0** and later **8.5.0**. * The **copyright year** has been updated. * The function reporting if the application is 64 bits was **removed**, as the application is now always 64 bits. * The **SWIG version has been updated to 4.3.0**. * **C++20 support** has been added. * The `dinamica_dependencies_windows.zip` has been updated. * Unused includes have been removed. * `stringstream` usage has been replaced by the **`fmt` library** for category names, and `DFF::toString` float/double implementations now use `fmt::format` for speed. A regression caused by the `stringstream` removal was fixed. * The use of `std::map` has been largely switched to **`std::unordered_map`** where possible, particularly for task graphs. * **Performance flags** have been added to Windows builds on CMake. * **Mimalloc support** has been added to Windows builds, with an optional CMake flag. * Previously missing **licensing details** for certain libraries used by the application have been included. * Grammar in **error messages** has been improved, replacing "can not" with "cannot". ===== Java Export and Integration Improvements ===== * Functions from **ProxyFunctorUtils**, **CellTypeInfo**, **LogTagInfo**, **ScriptUtils**, **SubmodelUtils**, **NameUtils**, and **StringUtils** namespaces are now exported to Java to **emulate classes with static members**. * Some Java exported functions were **moved from `SubmodelUtils` and `ScriptUtils` namespaces to the `Script` class** in Java. * **Projection-related functions** are now exported as static members of the `Projection` class in Java. * **Version comparison functions** (for strings) are exported as static members of the `Version` class in Java. * **Functionality to verify coordinator availability** has been added and exposed to Java. * The Java `dff.LogMessage` now uses `char[]` instead of `string` to **speed up logging**. * **Normalization functions** have been exported to Java. ===== Performance and Optimization ===== * **Massive optimization of `CalcPatchLabelMap` execution** by completely re-implementing the functor for better performance, including a new flag for sequential labels. The test case for `CalcPatchLabelMap` was also changed to ensure identical results on computers with different core counts. * Tasks added to task groups can now be **dispatched immediately**. * `ExtractCategoricalMapAttributes` execution now has **extra parallelism**, with mode, median, average/variance/stddev computed concurrently. It was further optimized by **parallelizing the insertion of attributes into the table**. * **Displaying tables is faster** by using existing indices instead of copying them again. * `LookupTable` functions have been **optimized**. * The event indicating that a port received new data was **removed** due to unnecessary performance degradation. * The way data is copied when transferred between data holders has been changed to **avoid performing the copy during `getData()`**, preventing some copies from being performed more than once. * **Table optimizations** are ongoing. * **Basic data types used for port data transfer have been refactored and optimized**, eliminating the need for casting, improving construction/destruction efficiency, and simplifying inheritance. * Functors (e.g., Muxes) now **forward input data directly** instead of creating new instances, reducing overhead. * **`std::map` has been replaced with `gtl::btree_map`** in `LookupTable`, `Categorization`, `SparseRasterImage`, and `Properties` experimentally for performance. * Table normalization from the UI now **uses tasks**. * **Table's existing indices** are used for `getSortedIndicesImpl()`. * `fast-double-parser` is now used for **parsing doubles when reading CSV tables**. * Direct writing to stream using `fmt::format` is used instead of `DFF::toString` for **faster output**. * The `CalcHistogram` functor has been **improved for better floating-point support and performance**. * The search for modules that was hurting MSVC performance on Windows has been **disabled**. ===== Stability and Bug Fixes ===== * **Compiled expressions** no longer cause memory errors. * An error message issued when passing invalid data to a `RunRemotely` was fixed. * A bug where **transferring a Folder to `RunRemotely`** would trigger an assertion has been fixed, and test cases were added. * Variable renaming was performed to prevent declaration hiding and fix C4456 warnings. * The conversion of some filenames to string resulting in relative paths instead of full ones has been fixed. * **Patcher and Expander** now use the random number generator from the current worker. * A **memory leak on Distributed** (the context was never destroyed) has been fixed. * Linux test problems were fixed. * A fix was added to **GDAL projections** on the Linux setup script. * A problem where **Python could throw an assertion** when strings couldn't be converted to UTF-8 has been fixed. * A problem where **tables with a single column could be read with one less row** has been fixed. * A bug where the **workdir property for bundle scripts** was not properly propagated to local submodels has been fixed. * A compilation error related to `TableDescriptionColumn` not having a default constructor in Java binding generation has been fixed. * `RunRemotely` now **clears both the socket and context** after running. * **Python not outputting directly to the console** has been fixed, along with UTF-8 string support for outputs. * **Windows External process** now identifies and prints UTF-8 outputs properly. * Matrices are now **written in scripts to ensure transitions are always sorted**. * **Functor storage by tasks** is now truly protected against concurrent writes by workers and reads by the graphical interface. * A bug related to **parallel constructs using `FOR_EACH_ELEMENT`** (container being copied instead of using a pointer) has been fixed. * **System process management of pipes** has been fixed, and `Calculate Python Expression` can now catch and report unhandled exceptions. * A **duplicate const in `calcDistanceMap2` function declaration** has been fixed. * **Bugs in the subtable implementation** have been fixed, including an optimization to move column chunks instead of copying rows. * **Unreadable tables that don't have a new line** are now fixed by appending a `\n` character after reading from disk. * A **crash when visualizing images formed by a complex tree of images** has been fixed. * **Invalid validation when retrieving Region Managers** has been fixed. * **Compatibility with `vcpkg 2025.04.09` and `msvc 17.14`** has been fixed. * **Forgotten Table locks** have been fixed. * **Forgotten GTL dependencies** were fixed. * GDAL initialization was fixed to prevent throwing an exception for a check. * A problem with a **default value not specified on `Table::addColumn`** has been fixed. * Comments in `Type.h` were fixed. * An inconsistency where a viewer could retain outdated content if its equivalent peer did not have data during the last model execution has been fixed. * The behavior when **transferring data to the viewers** was changed to avoid the possibility of data being transferred twice. Map viewers were updated to ensure deferred maps can still be visualized. * A bug where **copying a temporary image** would always incorrectly assume the first layer was being copied has been fixed. * Properly defined **copy constructors and assignment operators for maps** ensure cloning instances of `MapData` and `CategoricalMapData` preserves their contents and behavior. * The **categorization assignment mechanism for region masks** has been redesigned to avoid failures in edge cases. * A bug where `TEST_CHECK_EQUAL()` would **abort subsequent tests on failure** has been fixed. * Unnecessary parameter assignment to port "shapeAttributeName" in functor `RasterizeShape` was removed. * Image validation that could **incorrectly reject valid Lat/Long images covering Earth's poles** has been removed. * A bug causing models created before version 8.x to **ignore the `keepNonRegionCells` flag** when reading models using `RegionalizeCategoricalMap` has been fixed. * The documentation for the `optimizeScript` flag in the `Test::runScript` function was fixed. * A bug that could trigger an assertion when **visualizing a port while submodel expansion and redundant carrier removal were enabled simultaneously** has been fixed, with improved robustness when handling carriers. * A **forgotten include for `linux statvfs`** has been added. * A **potential crash when joining subtables through the task system** is being temporarily worked around. ===== Table and Data Handling Enhancements ===== * **Tables now allow any name format on their column titles**. * **Custom tabs** have been added when editing the `AddTableColumn` input ports. * Assertions and adjusted parameter order in `ExtractCategoricalMapAttributesFunctor` ensure consistency. * `CalculatePython` port data fetching and `getInputPortValue2` documentation were improved. * Initial work on **merging subtables using sub-table indices**. * Optimization to **copy subtables as-is** if they exist. * **Tasks are used for subtable merging** where possible, and a bug where inserted subtable rows would get the wrong line number in the table index has been fixed. * A hash for Tuples of any size (using `boost::hash_combine`) was added. * **Subtable implementation bugs** have been fixed, and an optimization to move column chunks instead of copying rows has been added. * The table is **normalized before obtaining its indices**. * The table writer **normalizes the table before writing it**. * **Table writing uses the table indices** for the order of keys. * A custom `AtomicMutex/Lock (SpinLock)` has been added to Table to allow locking/unlocking between different threads. These were renamed to `TaskMutex/TaskLock` with guards to ensure proper usage. Recursive support was removed from `TaskMutex`. * **GTL is used for the map implementation of Table**. * Large parts of Table were rewritten to **use `*Impl` functions and lock just once**. * A **formatter for `Tuple/TupleView`** has been added. * Tasks on the table constructor have been **disabled until the `ScriptReader` lock is fixed**. * `StringUtils::trim` has been added and used for **trimming table parser read strings**. * The construction of `Dinamica::None` was refactored to be explicit. * **`DFF::getDataValue()`** was introduced to uniformly extract the internal value from a `BaseData` instance. * **Data converters** (String to Code, Code to String and String to Date) that can be automatically defined were removed. * **Categorization has been added to all maps representing region masks**. * **Refactoring of region mask definition** ensures it uses the right values for categorization classes. ===== Debugging and System Features ===== * Error reporting on tests has been fixed. * Console arguments that print information (version, list-accelerators, list-functors) **no longer require an additional argument**. * Threads corresponding to each worker have been **assigned a description name**. * **Additional information about the current step** is now shown as progress for the cost calculation. * Python scripts from the "Calculate Python Expression" are now executed as **independent sub-processes**. * Dinamica Python can now **detect termination requests** and lock if pip is running. * **Python map tests** and Dinamica module Python dependencies/embedded Python integration have been removed. * **Thread naming on Linux** and Linux SystemProcess waiting times have been fixed. * Linux system process now uses **default environment variables** if none are specified. * Assertions have been placed on **`zeromq` context termination**. * Scripts have been added to **configure the build, compile source files, and install libraries and utilities** all at once. * **Dinamica's Python path and subfolder Scripts** have been added to the Python process. * An **`advanced optional input port` for packages** to be installed using PIP has been added to `CalculatePythonExpression`, and its documentation improved. * **Python error reporting functions** have been improved. * An **option to force the generation of core dumps** when the application crashes has been added. * **Dynamic debugging support** has been added to CMake. * A **script to compare test times with a reference** has been added, reporting changes due to regression or improvement, with options for limits and filtering. * The `Distributed/TooManyAgentsAndRunRemotely` test was **removed from fast tests** due to crashes. * Options to filter the tests to be analyzed by execution time and to select which results to display were added. * Initial implementation of **run remotely requirements**. * **`getTemporaryPathFreeSpace`** has been added. * `std::atomic` has been **replaced by `std::atomic_flag` in distributed** components. * **Agent IDs and tempspacegb** have been added to requirement expressions. * A **timeout limit to `RunRemotely`** has been added. * Safety in `RegionManager` has been improved by **refactoring code to correctly distinguish between pointers and references**. * The `GetRegionInfo` functor has been introduced for **accessing region details**, including direct access to the mask image. * Test cases for functor `GetTableKeys` were improved. * Test cases were added to test the behavior when running another real world model under different configurations. * A **forgotten include for `linux statvfs`** has been added. * The Python environment reset now attempts to **restart the application** if Python libraries are in use. * `ProgressReporter` and related functions were **refactored to be available for the Base library** too. * Error messages related to native expression compilation were updated. ===== Redundant Carrier and Submodel Management ===== * **Carrier necessity logic** has been refactored into a reusable utility method. * The **check for carrier necessity** has been extended to support multiple output-port connections, removing the single-peer limitation. * **Redundant carrier utilities** have been centralized in the C++ codebase, and a method to remove all redundant carriers from the script has been created with unit tests. * The **command-line flag controlling the removal of redundant carriers** now enables the feature instead of disabling it. * An **action to remove all redundant carriers from the script** was added. * A **flag in `ScriptRunner`** allows automatic removal of redundant carriers when `expandSubmodel` is active, with console support. * The **interface refresh logic after connection adjustments** has been improved. * `reconnectAllFunctors()` was introduced to **refresh interface connections after a removal**. * A **constraint was added to suggest redundant carrier connection adjustments** only when removing a single functor, with more complex cases handled by the dedicated utility. * The **carrier removal process has been simplified** by merging the redundancy check into a single dialog with clear options: "Remove and Readjust", "Just Remove", or "Cancel". * The `removeRedundantCarriers` flag handling has been **included in the interface**. (Generated using Google NotebookLM)