Skip to content

Updates Catalog Model

Gene Trantham requested to merge gt-23-catalog-model into develop

Creates a new pydantic model to ingest the updated data catalog from Mike Johnson.

I had originally put a lot of validation logic around parsing/validating URLs and other things. After talking with Mike, I have come to the conclusion that much of that is not necessary. He validates URLs via an automated pipeline as they are added to the database on his end. So... we're going to trust that has been done and simplify the data model here.

I do retain a couple of validators as a way of enforcing reasonable defaults. The CRS is an example of this. If the CRS is blank in the data catalog, I set it to 4326.

A number of fields are now not-optional. Notably, the min and max X and Y coordinates are required. The MODIS datasets, however, do not set these values... so those will not validate (yet) in this model.

Note, also, that this new catalog contains many datasets which are not OpenDAP.

Merge request reports