Improve performance in cfdm.read by caching any array values retrieved from disk  #313

Activity

1. davidhassell

   added

   performanceRelating to speed and memory performanceRelating to speed and memory performance

   on Jan 7, 2025
2. sadielbartholomew commented on Jan 7, 2025

   

   sadielbartholomew

   on Jan 7, 2025

   Member

   More actions

   I'm a bit confused by your description @davidhassell, so just to check - from re-reading a bit my interpretation is that you mean we already cache first and last values for the array of data variables, but we can do this more often that we currently do? Would appreciate a clarification - on the following PR or here. Thanks.
3. davidhassell commented on Jan 7, 2025

   

   davidhassell

   on Jan 7, 2025

   ContributorAuthor

   More actions

   Hi Sadie - sorry - the word "cache " is doing some heavy lifting, here! I'll try to re-word (although you're already correct!)

   Selected array values (typically the first and last in the array) are read from disk and cached inside the Data objects of the returned constructs. A given netCDF variable might end up providing the data for multiple constructs, in which case the selected values end up getting read from disk multiple times. Therefore we need a new cache of these values in NetCDFRead that means we're not going to disk more often than we need to. I imagine that this new cache will look something like:

   self.read_vars['_cached_data_elements'] = {
       <netCDF variable name>: <selected array values>,  
       <netCDF variable name>: <selected array values>,  
       ...
   }

   Then, each time we want to get the selected array values, we get them from the NetCDFRead cache, only going to disk if they're not there.
4. sadielbartholomew commented on Jan 7, 2025

   

   sadielbartholomew

   on Jan 7, 2025

   Last edited by sadielbartholomew

   Member

   More actions

   Ah I see - thanks for clarifying, I guess I'll be reviewing the PR eventually so it's out of more than just curiosity 🙂

   😄React with 😄1Reacted by David Hassell
5. davidhassell commented on Apr 7, 2025

   

   davidhassell

   on Apr 7, 2025

   ContributorAuthor

   More actions

   PR now up - empirical evidence on my laptop shows that this change speeds up the units tests by ~5%

   🚀React with 🚀1Reacted by Sadie L. Bartholomew
6. davidhassell commented on Jun 5, 2025

   

   davidhassell

   on Jun 5, 2025

   ContributorAuthor

   More actions

   PR now up - empirical evidence on my laptop shows that this change speeds up the units tests by ~5%

   Not been able to repeat this, and further investigation shows that the new cache is not accessed in the unit tests, anyway.

   That said, there is at least one case when the cache does get accessed - when there is a scalar coordinate variable shared between two data variables, so we might as well leave it all in for that.
7. davidhassell

   closed this as completedin #334on Jun 5, 2025
8. davidhassell

   added

   dataset readRelating to reading datasetsRelating to reading datasets

   on Jun 5, 2025
9. davidhassell

   added this to the NEXTVERSION milestone on Jun 5, 2025