Pimcore: Data Director version 2.5.0 is available
The first Data Director update after the launch of Pimcore X again provides users of the Export and Import Bundle with a number of improvements and new features.
Learn in detail which innovations our developers have made in the latest version of the Data Director and from which improvements you can benefit with version 2.5.0:
Pimcore 10 compatibility
Version 2.5.0 is fully compatible with Pimcore 10 - with the small limitation that not all libraries used are compatible with Pimcore 8. For Pimcore 10, you currently need to install Data Director with composer update blackbit/data-director --with-dependencies --ignore-platform-reqs. However, we are working on PHP 8 compatibility - also with regard to the libraries used.
Grid filtered exports
It is now possible to use the grid view (folder view) for filtering and then export the objects selected in it. This way you can perform ad-hoc filtered exports without having to enter a SQL condition. To start the export, there is a new option "Data Director Export" in the drop-down menu of the "CSV Export" button.
In the following modal window you can select the dataport to be used for the export (only compatible exports for the selected grid data object class will be displayed).
Refactor (automatic) exports for API usage
Previously, raw data for automatic exports was updated for each previously executed SQL condition. This resulted in poor performance when saving objects. This has been completely refactored: For automatic exports, raw data is now updated only for the configured dataport SQL condition (and for each language). The updated raw data is then copied into previously executed custom SQL condition blanks (so that the export data is already prepared when an export with this SQL condition is executed again). This way, the raw data only needs to be extracted once (for each language) and not again and again for all custom SQL conditions.
Also important for API usage, the SQL condition from the query parameter of REST API calls for Pimcore-based dataports now extends the SQL condition in the dataport settings instead of overriding it (to avoid retrieving unpublished items, for example).
Restrict items for Pimcore-based dataports to those for which the requesting user has "view" permission.
For incremental exports, the change timestamp of the last successful export is remembered (write to the object property, analogous to imports). When the export is triggered again for this object, its current change timestamp (incl. potentially inherited fields) is compared with this property, and if the current change timestamp is not newer, the raw data of the object is not extracted -> thus not exported. -> this also allows automatic incremental export for all potentially changed objects at once (was previously split to n processes of single object exports).
Data Director 2.5.0: Performance optimizations
- Queues of automatic data ports are now automatically processed in parallel. This ensures that a data port with only a few queues does not have to wait for the queues of other data ports to be processed.
- Reuse of raw data if there is already a raw data item with current hash that has the same modification date as the current object.
- When relational fields are used as key fields for imports, the associated object IDs were previously resolved to avoid a query like "WHERE relationalField LIKE '%,123,%'" - instead, the much faster query in the form "o_id IN (345,13,58)" is now executed.
- Faster operations: Conditions in SQL conditions of Pimcore-based data ports with relational fields like "WHERE relationalField LIKE '%,123,%'" have been replaced with "WHERE o_id IN (SELECT src_id FROM object_relations_* ...)" to allow them to use index.
- Raw data extraction: INSERT INTO use ... ON DUPLICATE KEY UPDATE instead of REPLACE INTO
- Unnecessary locks in the Pimcore parser have been removed.
- Fixed: Data query selectors were evaluated twice if they did not return a Concrete object
- relationCache is also used for data query selectors that return data from the queried object.
- 600% performance increase for image gallery / (extended) many-to-many relation asset mapping imports.
- When checking for isModified() during an import, first sort fields by whether they are mapped or not. This way, mapped fields are checked first because it is more likely that their values have been modified.
Raw data extraction
- Add option to skip versioning for asset imports
- Support for glob expressions in combination with asset folders as import resource, e.g. /import/*.csv when /import is a Pimcore asset folder.
- index is recommended if callback function uses data query selector
- History panel does not automatically reload when dataport runs are filtered (searched) or when not currently displaying page 1
- Added template for adding metadata to assets
History and manual import panel
- Automatically focus SQL condition field when manually launching Pimcore-based dataports from the Pimcore backend.
- Retain SQL condition from previous run to make it easier to run the same import/export multiple times (e.g., when setting up/testing dataports).
Other changes in version 2.5.0
- Support extended many-to-many relationship as key field (stored differently in database than (extended) many-to-many object relationship)
- Enlarged "fieldNo" column so that 3-digit numbers are not truncated
add callback function template to generate absolute asset/miniature URL
- Bugfix: fixed wrong method to update raw items
- Bugfix: inheritance optimized: more efficient check for parent objects if values were actually changed before saving them
- More accurate field value comparison when checking for isModified, so that a change from 0 to '' is detected
- Element type and class name in serializer for relational fields is provided
- Bugfix: overlapping imports: already processed raw data elements are not processed again
- Use of an error-tolerant JSON decoder to not abort the whole import in case of erroneous UTF-8 errors
- Bugfix: attribute mapping preview overlooked configured asset source folder (actual import worked correctly)
- Preventing multiple mapping of same asset to image gallery and many-many relation
- Prevention of multiple parallel requests to update history panel
- Bugfix: when using a relative folder path as import resource, deleting the asset file worked now
- Forced imports no longer check whether the item is currently locked for editing
- Faster showing and hiding of columns when searching in the Dataport preview window
- Bugfix: Starting exports by right-clicking on an object in the object tree is possible again
- Bugfix: raw data import is assigned the correct logger object and logs appear in import run logs
- Use of application_logs table (if used in Data Director) to find worst log levels
- maximum runtime when searching the history panel logs avoids timeout even if some items have been found
- Users starting imports are documented; allows better traceability of changes
- Bugfix: Excel import with column index fixed.
- Raw data from currently exported dataport resources are no longer deleted
- Manually uploaded data is integrated into default dataport resource
- No stack trace to versions is written to the database
- Custom logic is used to compare image gallery fields for changes
- Existing meta column data for extended many-to-many asset relationships is retained to properly code URLs for importing assets
- Pimcore documents matching the URL path for REST API requests are no longer loaded
- Added support for like search (wildcard search) in selector for data queries
- Support for adding items to multiselect fields (instead of always providing all options for selection), now works the same way as for relations, image galleries and other fields with multiselection
- Bugfix: not all raw data was processed on overlapping imports
- Data port runs are marked as "aborted" if an uncaught exception occurs or the process is aborted - manually via CLI or automatically by operating system
- Support access to localized fields in SQL condition of Pimcore-based dataports, e.g. name#en='abc'
- Support of access to object brick fields in SQL condition of Pimcore-based dataports, e.g. brickName.fieldName=123
- Bugfix: variable $params['rawItemData'] preview in attribute mapping for complex data
- Bugfix: Demo data for complex XML data
- Bugfix: XML parsing: multi-value attributes returned  for child nodes without value
- INSERT ... IN DUPLICATE KEY for queue items instead of REPLACE INTO to avoid deleting queue items not currently processed
- Support for accessing element fields of ObjectMetadata, ElementMetadata, Hotspotimage objects without adding "element:" to data query selector
- Bugfix: Serializer for documents (for Pimcore 4 document structure)
- Existing assets for image galleries / m2m relations etc. are only detected via md5 hash if the files are stored locally.
- Show warning if no key fields are specified
- Support calling service class methods from a data query selector, e.g. "field:@service_name::method".
- API keys / allowed dataports of other users are now shown only to admins
Locking of all fields is prevented when a new dataport is created by a non-admin user
- Error when generating auto-complete SQL condition is prevented
- Added special field "__updated" for file and URL based dataport types
- Use of the Pimcore user's language for exports when a locale was not explicitly requested
- Support for '' / "" syntax (virtual fields with quotes) to better copy callback functions into reformatting IDEs without messing up the code
- Field collections are better serialized
- Null values are included in serializers
- Added Importer::translate() method to call DeepL / AWS Translate translations for complex field values
- Change validation for field collections with localized fields reactivated
- Support for exporting all assigned bricks with the Data Director "brickFieldContainer" of the source data class
For more info, detailed questions or a consultation around the Data Director Bundle, feel free to contact David Gottschalk anytime.