Refactor Plan

Once work on the FY24 Referenace and Derived Modeling Fabrics data release has concluded, a refactor of the workflows to use targets will be undertaken.

Key goals include:

1. leverage multiple targets projects to separate download, preprocess, refactor, aggregate, and post-process steps.

graph LR
  download --> refactor
  download --> preprocess
  preprocess --> refactor
  preprocess --> aggregate
  refactor --> aggregate
  aggregate --> post-process

The mainstems workflow follows this pattern. https://code.usgs.gov/wma/nhgf/mainstems

2. greatly limit the amount of untested code within the work flow. This will be accomplished through naive characterization testing based on existing process results. How best to do this characterization testing is a planning task.

3. leverage renv to pin versions of all dependencies but also allow use of bespoke local environments for development. How to setup renv for development or reproducible pipeline use is a planning task.

4. have high-level characterization testing that runs in CI and clear documentation of each component of the overall workflow. This will be accomplished with a custom gitlab ci job and creation of a pkgdown site for the overall project that lives with the R package in the project root.

5. ... to be planned

assigned to @dblodgett

marked this issue as related to #154 (closed)

Thinking about 00_AK_prep and 00_gage_info -- Guessing AK_prep can be retired and replaced with whatever we come up with as our unified base fabric but we'll want to convert gage_info into a targets workflow. @abock -- do you still use gage_info at all or was it more or less a one-time thing that doesn't really get rerun?

Spending some time thinking about how to best do this, I think it is going to be to work out targets pipelines that render notebooks at the end.

https://books.ropensci.org/targets/literate-programming.html is the pertinent documentation.

Our workflows have a lot of side affects in the form of logs or cached review files. Those are going to be a bit tedious to work with, but are important and I may just convert them to explicit outputs. Need to work through things step by step here.

mentioned in issue #154 (closed)

mentioned in commit d6e55009

mentioned in commit 4cc570bd

mentioned in commit 945259ff

mentioned in commit cd5f7063

mentioned in commit 3a87c02a

mentioned in commit 2e871fb5

mentioned in commit bf49008a

mentioned in commit 6206532f

mentioned in commit 02e0ade4

mentioned in commit 0a900d39

mentioned in commit 180309e9

closed