Performance improvements dealing with populating from a table on file
In the course of working with some extremely large .h5
files, I encountered some performance issues.
First, I learned the hard way that going through and constantly checking membership in a very long list is painfully slow (see line 237 in main
's redpy/table.py
). Removing items from the table using the exclude
parameter in redpy.table.populate_from_table()
should now be many orders of magnitude faster for very large tables.
Second, I figured out how to copy tables directly from one .h5
file to another, which should speed up cases (e.g., in redpy-extend-table
) where many tables are not modified. Two new private functions were created to facilitate this behavior.
Edited by Hotovec-Ellis, Alicia Jean