TopoWx2017 Conversion to Zarr
Checklist for Workflow associated with dataset conversion:
Dataset Name: TopoWx: Topoclimatic Daily Air Temperature Dataset for the Conterminous United States
https://cida.usgs.gov/thredds/catalog.html?dataset=cida.usgs.gov/topowx
https://cida.usgs.gov/thredds/catalog.html?dataset=cida.usgs.gov/topowx_monthly
https://cida.usgs.gov/thredds/catalog.html?dataset=cida.usgs.gov/topowx_normals
https://cida.usgs.gov/thredds/catalog/demo/thredds/TopoWx2017/catalog.html
Daily is 25k time steps. Other domains are much smaller.
Suggest aggregating to ZARR from OPeNDAP to avoid reproducing ncml aggregation.
Float64 lat[lat = 3250];
Float64 lon[lon = 7000];
Int16 crs;
Float64 time[time = 25203];
-
Identify Source Data location and access (check the dataset spreadsheet) -
-
Collect ownership information (Who do we ask questions of if we have problems?) -
-
Create new workflow notebook from template; stash in the ./workflows
folder tree in an appropriate spot.-
Identify landing spot on S3 (currently somewhere in: https://s3.console.aws.amazon.com/s3/buckets/nhgf-development?prefix=workspace/®ion=us-west-2) -
Calculate chunking, layout, compression, etc -
Run notebook -
Read test (pattern to be determined by the dataset)
-
-
Reportage -
add notebook and the dask performance report to the repo -
Calculate summary statistics on output (compression ratio, total size) -
Save STAC JSON snippet to repo
-
-
Merge and close the issue.
Edited by Blodgett, David L.