Water Data For The Nation UI

This repo contains the components of Water Data For The Nation

• wdfn-server: A Flask web application that is used to create server-rendered pages for USGS water data
• assets: Client-side Javascript, CSS, images, etc.

The application requires python and node. The versions needed for each tool is found in pyproject.toml and assets/package.json, respectively. The python tool, poetry, is used to manage dependencies and is also required.

We are now using the formatters, black and prettier to format the code in this repo. Both of these tools are automatically installed when using the make commands to set up your local development environment. In addition, the make commands install pre-commit which is used to ensure that files are formatted before committing. We recommend that developers set up their IDE of choice to use the formatting tools automatically. Details can be found in the documentation for both tools.

Install dependencies

Installing the python and node dependencies when on VPN will likely require setting the following environment variables to a certificate bundles that includes all certs for your environment including the DOIRoot cert: REQUESTS_CA_BUNDLE, NPM_CONFIG_CAFILE, and NODE_EXTRA_CA_CERTS.

Using code.usgs.gov npm package registry

The packages, @wma/wdfn-vue-components and @wma/wdfn-viz, have been published on code.usgs.gov and is a dependency of this project. To install these packages you will need to set up a personal access token on code.usgs.gov. The personal access token should have "api" scope. See Authenticate to the Package Registry. We recommend that you add the _authToken npm setting in your user's .npmrc. Add the following to ~/.npmrc, substituting your personal access token. Your personal access token should not be put in the repo's .npmrc. That contains the location of the wma scoped registry.

//code.usgs.gov/api/v4/packages/npm/:_authToken=<your personal access token>

In addition, if someone has deleted the version that you are using and replaced it with a new build, you will likely get an EINTEGRITY error when using npm install. If this happens, uninstall @wma/wdfn-vue-components and then reinstall as follows:

npm uninstall @wma/wdfn-vue-components
npm install @wma/wdfn-vue-components --save-exact

Installing dependencies using make

The repository contains a make target to configure a local development environment for both python and node:

make devenv

To manually configure your environment, please see the READMEs of wdfn-server and assets.

Development server

We have found that the optimal way to run this system for local development is to run the server, either from your IDE or on the command line. The assets are served using whitenoise from the Flask web server, but you can use npm run watch to update the build as files change.

See the specific project READMEs for additional information.

Special Note - 'lookup files'

The waterdataui application uses what we call 'lookup files' to provide the application with information related to things like state and county codes as well as information about agency names and hydrological units. The data for these files is gathered from various web resources by an Amazon Web Services (AWS) Lambda function.

For local development, example lookup files are stored in the repository in the folder wdfn-server/data and they are not automatically updated. Please note, that the files in the wdfn-server/data directory are there for local development; they are not, necessarily, the files used in the deployed versions of the application.

The sample files can be updated to match what is used on deployment by manually downloading a copy of the new files to replace the files in your wdfn-server/data directory. These new files can be committed to Git and updated in the repository.

Run tests

To run all project tests:

make test

Optionally, you can run the Python and JavaScript tests separately.

# from the root directory
# To start the Python tests:
poetry run python -m pytest 

# from the assets directory
npm run test:watch # (runs the tests and stays active waiting for changes)
# or
npm run test # (runs the JavaScript tests once, produces a coverage report and shuts down)

If you want to run just a single test file, you can use "--" to add to the npm script. For example:

npm run test:watch -- src/scripts/monitoring-location/selectors/daily-value-time_series-selector.test.js

Production build

This command builds the assets much like when using npm run watch but will also run a gulp script to hash the assets. To test this, you will need to set the instance/config.py variable, ASSET_MANIFEST_PATH to you assets/dist/rev-manifest.json file.

make build

Clean targets

This command removes the build artifacts and also removes the assets/node_modules. The poetry virtualenv is retained however. If you wish to do an install from scratch, you will need to use poetry to remove the project's env.

make cleanenv

Run a Local Version on a Mobile Device Using Docker

Preliminary Steps

• Make sure Docker is installed on your system (process varies with Operating System)
• Install a 'tunneling' application. LocalTunnel, an npm app is no longer supported and is unreliable. Another free option is http://localhost.run/
• Disconnect from Virtual Private Network (VPN) or USGS network
• Stop any processes using port 5050 (or be prepared to use a different port)

Build Docker Images from Context in DockerFiles

# From the project root directory run
docker-compose build web_server
# (builds a Docker Image for application

Run the Flask Server as a Process in a Docker Container

docker-compose run web_webserver
# (starts the Flask server in Docker container on port 5050

Is it working?

When the container is running, the terminal should look something like . . .

[2020-12-04 14:07:25 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2020-12-04 14:07:25 +0000] [1] [INFO] Listening at: http://0.0.0.0:5050 (1)
[2020-12-04 14:07:25 +0000] [1] [INFO] Using worker: sync
[2020-12-04 14:07:25 +0000] [7] [INFO] Booting worker with pid: 7
[2020-12-04 14:07:25 +0000] [8] [INFO] Booting worker with pid: 8
[2020-12-04 14:07:25 +0000] [9] [INFO] Booting worker with pid: 9
[2020-12-04 14:07:25 +0000] [10] [INFO] Booting worker with pid: 10

You should also be able to visit the site on port 5050 (localhost:5050) on a local browser.

You can then use a local tunneling application and follow its instructions on how to connect a mobile device that is using the same network as your computer.

Clean up

When testing is finished, you will want to turn off the Docker container

# run 
docker ps # (shows all Docker containers on your system)
then run
docker stop <the container id> 
# (the container ID will match up an image shown in the 'docker ps' command named 'waterdataui' 
# (which is what you named it earlier))

# If you don't want the images on your system, you can then remove them with 
docker rmi <image name>

Limitations

For each change, you will need to rebuild the Docker image and restart the process -- which isn't optimal.

So this is best reserved for testing implementations that involve touching the screen of the device, especially with multiple touch points. Such implementations cannot be tested with normal emulations.

Running pa11yci locally

You can also use the docker-compose scripts to run pa11yci. You can run pa11y_ci by executing:

docker-compose up pa11y_ci

This will run pa11yci in a docker container using the URLs in pa11y.config.json. If you are using docker-compose to start the web_server you will not need to change the URLs. If you wish to run against a locally running flask server or against a deployed tier, just change the URL hosts in pa11y.config.json. Just be sure to revert the changes so that the URLs are correct for gitlab CI.