iclus Conversion to Zarr

Checklist for Workflow associated with dataset conversion:

Dataset Name: Integrated Climate and Land-Use Scenarios v1.3

https://cida.usgs.gov/thredds/catalog.html?dataset=cida.usgs.gov/ICLUS/is https://cida.usgs.gov/thredds/catalog.html?dataset=cida.usgs.gov/ICLUS/hd https://cida.usgs.gov/thredds/catalog.html?dataset=cida.usgs.gov/ICLUS/hc

https://cida.usgs.gov/thredds/catalog/demo/thredds/iclus/catalog.html

Large spatial domain limited number of time steps. e.g.

Dataset {
    Float64 geoX[x = 4615];
    Float64 geoY[y = 2904];
    Int32 crs;
    Int32 time[time = 22];
    Float32 impervious_surface_is_iclus_a1[time = 22][y = 2904][x = 4615];
    Float32 impervious_surface_is_iclus_a2[time = 22][y = 2904][x = 4615];
    Float32 impervious_surface_is_iclus_b1[time = 22][y = 2904][x = 4615];
    Float32 impervious_surface_is_iclus_b2[time = 22][y = 2904][x = 4615];
    Float32 impervious_surface_is_iclus_bc[time = 22][y = 2904][x = 4615];
} iclus/is;

Recommend straight OPeNDAP conversion.

---

• Identify Source Data location and access (check the dataset spreadsheet)

• Collect ownership information (Who do we ask questions of if we have problems?)

• Create new workflow notebook from template; stash in the ./workflows folder tree in an appropriate spot.
  • Identify landing spot on S3 (currently somewhere in: https://s3.console.aws.amazon.com/s3/buckets/nhgf-development?prefix=workspace/&region=us-west-2)
  • Calculate chunking, layout, compression, etc
  • Run notebook
  • Read test (pattern to be determined by the dataset)
• Create STAC catalog entry;
  • Verify all metadata
  • Create entry
• Reportage
  • add notebook and the dask performance report to the repo
  • Calculate summary statistics on output (compression ratio, total size)
  • Save STAC JSON snippet to repo
• Merge and close the issue.