Skip to content

Performance improvements dealing with populating from a table on file

Hotovec-Ellis, Alicia Jean requested to merge develop into main

In the course of working with some extremely large .h5 files, I encountered some performance issues.

First, I learned the hard way that going through and constantly checking membership in a very long list is painfully slow (see line 237 in main's redpy/table.py). Removing items from the table using the exclude parameter in redpy.table.populate_from_table() should now be many orders of magnitude faster for very large tables.

Second, I figured out how to copy tables directly from one .h5 file to another, which should speed up cases (e.g., in redpy-extend-table) where many tables are not modified. Two new private functions were created to facilitate this behavior.

Edited by Hotovec-Ellis, Alicia Jean

Merge request reports