Further documentation on creating fields and writing fields to file #53

Activity

1. sadielbartholomew

   added

   documentationImprovements or additions to documentationImprovements or additions to documentation

   on Jun 19, 2020
2. davidhassell

   changed the title [-]Further documentation on writing fields to file[/-] [+]Further documentation on creating fields and writing fields to file[/+] on Jun 22, 2020
3. kurtsansom commented on Sep 15, 2020

   

   kurtsansom

   on Sep 15, 2020

   More actions

   This would be a great help!

   Lets say I already have a compliant file, can I extract the schema in a sensible in order to populate a new file with similar but potentially different data. An example like this would be really helpful.

   Is there a way to create a "skeleton" of the file and then populate it with data?
4. sadielbartholomew commented on Sep 18, 2020

   

   sadielbartholomew

   on Sep 18, 2020

   MemberAuthor

   More actions

   Hi @kurtsansom, thanks for registering that you would also find such further documentation useful. It helps us to know multiple people would benefit from some improvement, as in this case, so we can make it a higher priority.

   Your particular questions are interesting because (if I understand your requests correctly) both are indeed possible and also documented quite well, just not with the cfdm library directly, rather in cf-python, our data analysis library which extends cfdm in functionality and scope. So, if you are willing to use cf-python instead of cfdm standalone (see here for the installation guide), I can point you directly to documentation which covers in detail how to do what you wish to, programmatically:

   Lets say I already have a compliant file, can I extract the schema in a sensible in order to populate a new file with similar but potentially different data.

   Is there a way to create a "skeleton" of the file and then populate it with data?

   Yes, this can be achieved by two different approaches in cf-python:

   1. reading in the compliant file (with cf.read) to produce a corresponding list of fields;
   2. then either:
      • overwriting the data of these read-in fields (or copies of them if preferred) using the set_data methods on the metadata constructs or entire field; or
      • running the creation_commands method on each read-in field to produce the exact sequence of calls required to generate the fields from scratch (see this sub-section with an example), then running those but with the arguments to the set_data call changed as applicable to set different data on the fields;
   3. finally writing the fields from step (2) out to file via the cf.write function).

   An example like this would be really helpful.

   We do have detailed examples for each the three stages I list above in our documentation, but not all three stages at once to create an end-to-end example for what you specifically want to do. Now I have pointed out relevant sections, do you think it could still be helpful to have an end-to-end example, or do you think what we already have, as covered separately, is sufficient? I am asking because I personally think it could be "overkill" to have an additional section, but what is obvious to me about how to link various functionality together might not be obvious to others.

   In general, for documentation on using cf-python to write out new files modified in a CF-compliant way relative to existing ones, I suggest consulting the following sections which detail the two steps necessary:

   1. first to create the appropriate field construct (which does not have to be done by reading in existing fields, as in my above list, but can be done from scratch), see this section for overall approaches and examples;
   2. then to write the field construct to file see this section or the cf.write API reference.

   If instead you want or need to use cfdm and not cf-python, note that in the near future we plan to port the so-called 'creation commands' which enable such constructions from cf-python into cfdm, such that they can be used in both libraries and therefore will be available directly in cfdm too. Of course we will update the documentation accordingly to cover the extended capability.

   I hope this helps. If it is not quite what you were asking for, or you have further questions, please let us know.

   👍React with 👍1Reacted by Kurt Sansom
5. davidhassell commented on Sep 23, 2020

   

   davidhassell

   on Sep 23, 2020

   Contributor

   More actions

   The " creation commands" methods will be available in cfdm at the next release (1.8.7.0). They'll behave exactly the same as they do in cf-python (i.e. https://ncas-cms.github.io/cf-python/tutorial.html#command-modification)

   Thanks.

   👍React with 👍2Reacted by Sadie L. Bartholomew and Kurt Sansom
6. davidhassell

   closed this as completedon Sep 30, 2020
7. kurtsansom commented on Nov 17, 2020

   

   kurtsansom

   on Nov 17, 2020

   More actions

   @sadielbartholomew
   This is really helpful information. I am trying to figure out the cfdm to better understand what I need to modify.

   I think I was thinking along the lines of the metadata as well. pyart, uses something like a simple dictionary of dictionaries to define the attributes that map to the actual variables in the code. Or something like ncdump except it generates a dictionary with the fields and attributes.

   add_attrs = {
       'latitude': {
           'long_name': 'Latitude',
           'standard_name': 'Latitude',
           'units': 'degrees_north',
           'comment': 'Latitude of instrument, using WGS84.',
       },
   }
   
   

   👍React with 👍1Reacted by Sadie L. Bartholomew
8. davidhassell commented on Nov 19, 2020

   

   davidhassell

   on Nov 19, 2020

   Contributor

   More actions

   Hi @kurtsansom, creating such a dictionary should be straightforward, using the properties method of the constructs, for example:

   In [8]: import cfdm
   
   In [9]: f = cfdm.example_field(0)
   
   In [10]: f.properties()
   Out[10]: 
   {'Conventions': 'CF-1.8',
    'project': 'research',
    'standard_name': 'specific_humidity',
    'units': '1'}
   
   In [11]: for c in f.constructs.values():
       ...:     try:
       ...:         print(repr(c), c.properties())
       ...:     except AttributeError:
       ...:         pass
       ...:     
   <DimensionCoordinate: latitude(5) degrees_north> {'units': 'degrees_north', 'standard_name': 'latitude'}
   <DimensionCoordinate: longitude(8) degrees_east> {'units': 'degrees_east', 'standard_name': 'longitude'}
   <DimensionCoordinate: time(1) days since 2018-12-01 > {'units': 'days since 2018-12-01', 'standard_name': 'time'}
   

   Does that help? Do let me know if I've missed anything.
9. davidhassell

   reopened this on Nov 19, 2020