Problem parsing NWIS data

Created by: lawinslow

Is there a reason that readNWISpeak fails when retrieving this data

readNWISpeak("07154400")

Data seem fine from NWIS

http://nwis.waterdata.usgs.gov/usa/nwis/peak/?site_no=07154400&range_selection=date_range&format=rdb

Hmm...the issue seems to be that first line of the data, the date is "1953-00-00" (I haven't seen that before).

You could grab the data, and not convert it to dates like this:

url <- constructNWISURL("07154400", NA, "", "", "peak")
data <- importRDB1(url, asDateTime = FALSE)

Actually, there are several dates that are YYYY-00-00

If that's a 'thing' for peak data, we can write it into the function...Does that mean mean of water year? Mean of calendar year? Something else?

Created by: lawinslow

Those are for the floods when we don't know the day or month of occurrence. They should have a code for "day or month of occurrence unknown".

Yup, you can see the comments:

comment(data)

that those dates have peak_cd with "B" (Month or Day of occurrence is unknown or not exact)

So what would be ideal? Grab the data, put NA's for the date's if peak_cd contains "B"? We know the year, so that's more than NA. But R doesn't let you do as.Date("2005"). POSIXct seems to also force it to a specific date too:

as.POSIXct("2005", format="%Y")
[1] "2005-03-31 CST"

Created by: lawinslow

Well, ideally we would like to keep the year information but leave the month and day blank. But since that isn't an option in R, second choice would be to force a nonsense date like December 32. If that's not possible (which I'm guessing it won't be) the 3rd choice would be to force an unlikely date that is consistent and that we would recognize for all these instances, like January 1.

Created by: jiwalker-usgs

What about something crazy like ts(value, start=jan1, end=dec31, frequency=1)

Created by: lawinslow

But these are instantaneous values. There is no start or end date. Annual peak, happens once per year.
However for this particular analysis that I'm doing, the peaks that have unknown months and days won't be useful anyway, I can leave those data points out for the purpose of this analysis.

I'm fixing some things on dataRetrieval today. Is this a lingering issue? Does dataRetrieval do what you what considering what NWIS gives out? (ie, can I close?)

Created by: lawinslow

Not sure, my reaction is no. That function shouldn't error out with data that we now know those dates can be expected from NWIS. Maybe it would be better if those yyyy-00-00 were removed and a warning thrown?

That, or don't convert to Date object. Which do you think will cause less pain? Potentially lots of warnings (I have no idea)

Or, check that all dates are legal, if so, make a date, if not leave as character?

Created by: lawinslow

My vote would be always keep date, just toss warning if any match the "[0-9]*-00-00" using grepl and drop those values. Most people will probably just want well-formatted data. Those die-hards can write their own parser for 00-00 dates if they want 'em.

okey dokey smokey

Argg..some of them are narrowed down to the month.

Created by: lawinslow

ha, wow, joy

Created by: lawinslow

Nice.