ciCalculations on GFN elist crashes R
Hello! Thanks for the great package.
I have an example that basically breaks my R (USGS high performance resources it leads to a signal 11 segmentation fault and on my desktop the R studio throws up the "R session Aborted" window with the bomb icon). After much troubleshooting I can make a reproducible example with the attached eList file. I'm seeing this issue with 3 out of ~2000 eLists, so it is not a very prevalent issue but still is enough to be troublesome. I can't really figure out where it is running into problems or what about this particular file makes it fail (or how to catch this error more elegantly).
It basically just crashes midway through doing the ci calculations without much rhyme or reason. Here's a screenshot of the console printout at the moment of failure, which is about midway through the bootstrapped replicates:
If I really had to guess it seems like maybe on the sixth bootstrap iteration the model fails to converge. For some reason when this is an SFN model it can keep going but when it is a GFN model then it gets hung up and fails spectacularly. Any ideas?
File to use for reprex: eList_01444800_PO4.rds
#Load Files
library(EGRET)
#> Warning: package 'EGRET' was built under R version 4.1.2
library(EGRETci)
#> Warning: package 'EGRETci' was built under R version 4.1.1
library(tidyverse)
#> Warning: package 'ggplot2' was built under R version 4.1.2
#> Warning: package 'tibble' was built under R version 4.1.3
#> Warning: package 'tidyr' was built under R version 4.1.3
#> Warning: package 'readr' was built under R version 4.1.3
#> Warning: package 'dplyr' was built under R version 4.1.3
#Bring in attached file from wherever it is saved
eList<-readRDS("eList_01444800_PO4.rds")
#This eList has a lot more years of discharge data than water quality data.
#We'll trim them to the same record
eList$INFO$start_wy<-min(eList$Sample$waterYear)
eList$INFO$end_wy<-max(eList$Sample$waterYear)
trendMinStart <- min(eList$Sample$waterYear)
trendMaxEnd <- max(eList$Sample$waterYear)
#For now, trim Q data to WQ Data water year
eList$Daily<-eList$Daily%>%filter(waterYear>=min(eList$Sample$waterYear))
#Remove any duplicates in the daily data or sample data
eList[["Daily"]]<-eList[["Daily"]][!duplicated(eList[["Daily"]]%>%select(Date,Q)),]
eList[["Sample"]]<-eList[["Sample"]][!duplicated(eList[["Sample"]]),]
#Run SFN analysis on the elist
eList_modelled_SFN <- modelEstimation(eList,
windowY = eList$INFO$windowY,
windowQ = eList$INFO$windowQ,
windowS = eList$INFO$windowS,
minNumObs = eList$INFO$minNumObs,
minNumUncen = eList$INFO$minNumUncen,
edgeAdjust = T,
verbose = F)
#Run GFN analysis on the elist
eList_modelled_GFN <- runSeries(eList_modelled_SFN,
windowSide = 7,
flowBreak = F,
windowY = eList$INFO$windowY,
windowQ = eList$INFO$windowQ,
windowS = eList$INFO$windowS,
minNumObs = eList$INFO$minNumObs,
minNumUncen = eList$INFO$minNumUncen,
edgeAdjust = T,
verbose = F)
#CI calculations on SFN - this works
ci_values_SFN = ciCalculations(eList = eList_modelled_SFN,
nBoot = 10, #small for testing
blockLength= 200,
widthCI = 90)
#>
#> Running the EGRET modelEstimation function first to have that as a baseline for the Confidence Bands
#> first step running estCrossVal may take about 1 minute
#> estCrossVal % complete:
#> 0 1 2 3 4 5 6 7 8 9 10
#> 11 12 13 14 15 16 17 18 19 20
#> 21 22 23 24 25 26 27 28 29 30
#> 31 32 33 34 35 36 37 38 39 40
#> 41 42 43 44 45 46 47 48 49 50
#> 51 52 53 54 55 56 57 58 59 60
#> 61 62 63 64 65 66 67 68 69 70
#> 71 72 73 74 75 76 77 78 79 80
#> 81 82 83 84 85 86 87 88 89 90
#> 91 92 93 94 95 96 97 98 99
#> Next step running estSurfaces with survival regression:
#> Survival regression (% complete):
#> 0 1 2 3 4 5 6 7 8 9 10
#> 11 12 13 14 15 16 17 18 19 20
#> 21 22 23 24 25 26 27 28 29 30
#> 31 32 33 34 35 36 37 38 39 40
#> 41 42 43 44 45 46 47 48 49 50
#> 51 52 53 54 55 56 57 58 59 60
#> 61 62 63 64 65 66 67 68 69 70
#> 71 72 73 74 75 76 77 78 79 80
#> 81 82 83 84 85 86 87 88 89 90
#> 91 92 93 94 95 96 97 98 99
#> Survival regression: Done
#>
#> There are 45 duplicated Sample dates.
#> Survival regression (% complete):
#> 0 1 2 3 4 5 6 7 8 9 10
#> 11 12 13 14 15 16 17 18 19 20
#> 21 22 23 24 25 26 27 28 29 30
#> 31 32 33 34 35 36 37 38 39 40
#> 41 42 43 44 45 46 47 48 49 50
#> 51 52 53 54 55 56 57 58 59 60
#> 61 62 63 64 65 66 67 68 69 70
#> 71 72 73 74 75 76 77 78 79 80
#> 81 82 83 84 85 86 87 88 89 90
#> 91 92 93 94 95 96 97 98 99
#> Survival regression: Done
#>
#> There are 45 duplicated Sample dates.
#>
#> There are 38 duplicated Sample dates.
#> Survival regression (% complete):
#> 0 1 2 3 4 5 6 7 8 9 10
#> 11 12 13 14 15 16 17 18 19 20
#> 21 22 23 24 25 26 27 28 29 30
#> 31 32 33 34 35 36 37 38 39 40
#> 41 42 43 44 45 46 47 48 49 50
#> 51 52 53 54 55 56 57 58 59 60
#> 61 62 63 64 65 66 67 68 69 70
#> 71 72 73 74 75 76 77 78 79 80
#> 81 82 83 84 85 86 87 88 89 90
#> 91 92 93 94 95 96 97 98 99
#> Survival regression: Done
#>
#> There are 38 duplicated Sample dates.
#>
#> There are 26 duplicated Sample dates.
#> Survival regression (% complete):
#> 0 1 2 3 4 5 6 7 8 9 10
#> 11 12 13 14 15 16 17 18 19 20
#> 21 22 23 24 25 26 27 28 29 30
#> 31 32 33 34 35 36 37 38 39 40
#> 41 42 43 44 45 46 47 48 49 50
#> 51 52 53 54 55 56 57 58 59 60
#> 61 62 63 64 65 66 67 68 69 70
#> 71 72 73 74 75 76 77 78 79 80
#> 81 82 83 84 85 86 87 88 89 90
#> 91 92 93 94 95 96 97 98 99
#> Survival regression: Done
#>
#> There are 26 duplicated Sample dates.
#>
#> There are 45 duplicated Sample dates.
#> Survival regression (% complete):
#> 0 1 2 3 4 5 6 7 8 9 10
#> 11 12 13 14 15 16 17 18 19 20
#> 21 22 23 24 25 26 27 28 29 30
#> 31 32 33 34 35 36 37 38 39 40
#> 41 42 43 44 45 46 47 48 49 50
#> 51 52 53 54 55 56 57 58 59 60
#> 61 62
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> 63 64 65 66 67
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> 68 69 70
#> 71 72
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> 73
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> 74 75 76 77 78
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> 79 80
#> 81 82 83
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> 84 85 86 87 88
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> 89 90
#> 91 92 93
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> 94
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> 95 96 97 98
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> 99
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#> Warning in survreg.fit(X, Y, weights, offset, init = init, controlvals =
#> control, : Ran out of iterations and did not converge
#>
#> Survival regression: Done
#>
#> There are 45 duplicated Sample dates.
#>
#> There are 34 duplicated Sample dates.
#> Survival regression (% complete):
#> 0 1 2 3 4 5 6 7 8 9 10
#> 11 12 13 14 15 16 17 18 19 20
#> 21 22 23 24 25 26 27 28 29 30
#> 31 32 33 34 35 36 37 38 39 40
#> 41 42 43 44 45 46 47 48 49 50
#> 51 52 53 54 55 56 57 58 59 60
#> 61 62 63 64 65 66 67 68 69 70
#> 71 72 73 74 75 76 77 78 79 80
#> 81 82 83 84 85 86 87 88 89 90
#> 91 92 93 94 95 96 97 98 99
#> Survival regression: Done
#>
#> There are 34 duplicated Sample dates.
#>
#> There are 38 duplicated Sample dates.
#> Survival regression (% complete):
#> 0 1 2 3 4 5 6 7 8 9 10
#> 11 12 13 14 15 16 17 18 19 20
#> 21 22 23 24 25 26 27 28 29 30
#> 31 32 33 34 35 36 37 38 39 40
#> 41 42 43 44 45 46 47 48 49 50
#> 51 52 53 54 55 56 57 58 59 60
#> 61 62 63 64 65 66 67 68 69 70
#> 71 72 73 74 75 76 77 78 79 80
#> 81 82 83 84 85 86 87 88 89 90
#> 91 92 93 94 95 96 97 98 99
#> Survival regression: Done
#>
#> There are 38 duplicated Sample dates.
#>
#> There are 34 duplicated Sample dates.
#> Survival regression (% complete):
#> 0 1 2 3 4 5 6 7 8 9 10
#> 11 12 13 14 15 16 17 18 19 20
#> 21 22 23 24 25 26 27 28 29 30
#> 31 32 33 34 35 36 37 38 39 40
#> 41 42 43 44 45 46 47 48 49 50
#> 51 52 53 54 55 56 57 58 59 60
#> 61 62 63 64 65 66 67 68 69 70
#> 71 72 73 74 75 76 77 78 79 80
#> 81 82 83 84 85 86 87 88 89 90
#> 91 92 93 94 95 96 97 98 99
#> Survival regression: Done
#>
#> There are 34 duplicated Sample dates.
#>
#> There are 35 duplicated Sample dates.
#> Survival regression (% complete):
#> 0 1 2 3 4 5 6 7 8 9 10
#> 11 12 13 14 15 16 17 18 19 20
#> 21 22 23 24 25 26 27 28 29 30
#> 31 32 33 34 35 36 37 38 39 40
#> 41 42 43 44 45 46 47 48 49 50
#> 51 52 53 54 55 56 57 58 59 60
#> 61 62 63 64 65 66 67 68 69 70
#> 71 72 73 74 75 76 77 78 79 80
#> 81 82 83 84 85 86 87 88 89 90
#> 91 92 93 94 95 96 97 98 99
#> Survival regression: Done
#>
#> There are 35 duplicated Sample dates.
#>
#> There are 43 duplicated Sample dates.
#> Survival regression (% complete):
#> 0 1 2 3 4 5 6 7 8 9 10
#> 11 12 13 14 15 16 17 18 19 20
#> 21 22 23 24 25 26 27 28 29 30
#> 31 32 33 34 35 36 37 38 39 40
#> 41 42 43 44 45 46 47 48 49 50
#> 51 52 53 54 55 56 57 58 59 60
#> 61 62 63 64 65 66 67 68 69 70
#> 71 72 73 74 75 76 77 78 79 80
#> 81 82 83 84 85 86 87 88 89 90
#> 91 92 93 94 95 96 97 98 99
#> Survival regression: Done
#>
#> There are 43 duplicated Sample dates.
#>
#> There are 41 duplicated Sample dates.
#> Survival regression (% complete):
#> 0 1 2 3 4 5 6 7 8 9 10
#> 11 12 13 14 15 16 17 18 19 20
#> 21 22 23 24 25 26 27 28 29 30
#> 31 32 33 34 35 36 37 38 39 40
#> 41 42 43 44 45 46 47 48 49 50
#> 51 52 53 54 55 56 57 58 59 60
#> 61 62 63 64 65 66 67 68 69 70
#> 71 72 73 74 75 76 77 78 79 80
#> 81 82 83 84 85 86 87 88 89 90
#> 91 92 93 94 95 96 97 98 99
#> Survival regression: Done
#>
#> There are 41 duplicated Sample dates.
#CI calculations on GFN:
#this blows up my R session so bad I can't even include it in the reprex so
# I need to comment it out
#ci_values_GFN = ciCalculations(eList = eList_modelled_GFN,
# nBoot = 10, #small for testing
# blockLength= 200,
# widthCI = 90)
#Print session info
sessionInfo()
#> R version 4.1.0 (2021-05-18)
#> Platform: x86_64-w64-mingw32/x64 (64-bit)
#> Running under: Windows 10 x64 (build 19045)
#>
#> Matrix products: default
#>
#> Random number generation:
#> RNG: Mersenne-Twister
#> Normal: Inversion
#> Sample: Rounding
#>
#> locale:
#> [1] LC_COLLATE=English_United States.1252
#> [2] LC_CTYPE=English_United States.1252
#> [3] LC_MONETARY=English_United States.1252
#> [4] LC_NUMERIC=C
#> [5] LC_TIME=English_United States.1252
#>
#> attached base packages:
#> [1] stats graphics grDevices utils datasets methods base
#>
#> other attached packages:
#> [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.8 purrr_0.3.4
#> [5] readr_2.1.3 tidyr_1.2.0 tibble_3.2.1 ggplot2_3.3.5
#> [9] tidyverse_1.3.1 EGRETci_2.0.4 EGRET_3.0.7
#>
#> loaded via a namespace (and not attached):
#> [1] lubridate_1.8.0 lattice_0.20-45 assertthat_0.2.1 digest_0.6.29
#> [5] utf8_1.2.2 R6_2.5.1 cellranger_1.1.0 backports_1.4.1
#> [9] reprex_2.0.1 spam_2.8-0 evaluate_0.15 httr_1.4.4
#> [13] pillar_1.8.1 rlang_1.1.0 readxl_1.4.0 rstudioapi_0.13
#> [17] Matrix_1.4-1 rmarkdown_2.20 splines_4.1.0 munsell_0.5.0
#> [21] broom_0.7.12 compiler_4.1.0 modelr_0.1.8 xfun_0.37
#> [25] pkgconfig_2.0.3 htmltools_0.5.2 tidyselect_1.2.0 gridExtra_2.3
#> [29] binom_1.1-1 viridisLite_0.4.0 fansi_1.0.3 crayon_1.5.2
#> [33] tzdb_0.3.0 dbplyr_2.1.1 withr_2.5.0 grid_4.1.0
#> [37] jsonlite_1.8.0 gtable_0.3.0 lifecycle_1.0.3 DBI_1.1.3
#> [41] magrittr_2.0.3 scales_1.1.1 cli_3.6.0 stringi_1.7.6
#> [45] viridis_0.6.2 fs_1.5.2 xml2_1.3.3 ellipsis_0.3.2
#> [49] generics_0.1.3 vctrs_0.6.0 tools_4.1.0 glue_1.6.2
#> [53] maps_3.4.0 hms_1.1.2 fields_13.3 fastmap_1.1.0
#> [57] survival_3.3-1 yaml_2.3.5 colorspace_2.0-3 dotCall64_1.0-1
#> [61] rvest_1.0.2 knitr_1.42 haven_2.4.3
Created on 2023-06-08 by the reprex package (v2.0.1)