Merge branch 'dockerize' into 'master'

Dockerize

See merge request !352

docs/develop.md

@@ -111,3 +111,6 @@ https://black.readthedocs.io/en/stable/the_black_code_style
   Resolve any rebase conflicts.
   If you have already pushed this branch to your fork, you *may* need to force push
   because branch history has changed.
+
+## Local Development
+After you get everything installed, you can find further instructions on local development [here](../localdev/local_development.md).

localdev/cleanup-docker.sh

+#!/bin/sh -e
+
+# script to completely clean up all docker resources
+docker stop $(docker ps -aq)
+
+docker rm -vf $(docker ps -aq)
+
+docker rmi -f $(docker images -aq)
+
+# this command will ask if you are sure if you want to continue. the benefit of this command is to free up space by removing cached objects, 
+# but it does mean that the next time you run docker-compose up, it will take longer.
+docker system prune --volumes

localdev/docker-compose.yml

+version: '3'
+services:
+  database:
+    image: mysql:5.7
+    ports:
+      - '3306:3306'
+    environment:
+      - MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD
+      - MYSQL_DATABASE=geomag_operations
+    volumes:
+      - ./init.sql:/docker-entrypoint-initdb.d/init.sql
+    platform: linux/amd64
+
+  edge:
+    build: ./geomag-edge
+    ports:
+      - '7981:7981'
+      - '2060:2060'
+      - '7974:7974'
+      - '2061:2061'
+    volumes:
+      - ./edge-data:/data
+  
+  web:
+    build:
+      context: ../.
+      dockerfile: ./Dockerfile
+    command: bash -c "cd /data && python create_db.py && python test_metadata.py && uvicorn --reload geomagio.api:app --host 0.0.0.0 --workers 2"
+    ports:
+      - '8000:8000'
+    extra_hosts:
+      - host.docker.internal:host-gateway
+    environment:
+      - DATA_HOST=host.docker.internal
+      - OPENID_CLIENT_ID=$OPENID_CLIENT_ID
+      - OPENID_CLIENT_SECRET=$OPENID_CLIENT_SECRET
+      - DATABASE_URL=mysql://root:password@host.docker.internal:3306/geomag_operations
+      - OPENID_METADATA_URL=https://code.usgs.gov/.well-known/openid-configuration
+      - SECRET_KEY=$SECRET_KEY
+      - SECRET_SALT=$SECRET_SALT
+      - WEBSERVICE=false
+    volumes:
+      - ../.:/data
+    restart: always

localdev/geomag-edge/Dockerfile

+# dockerfile to build basic edge/cwb image
+
+FROM usgs/java:8
+
+LABEL maintainer="Jeremy Fee <jmfee@usgs.gov>"
+
+# set up vdl account
+# NOTE: USGS' public ftp server was decommisioned at the end of FY22.
+#       (i.e., ftp://hazards.cr.usgs.gov/CWBQuery/EdgeCWBRelease.tar.gz)
+#       As of January 2023, there is an experimental method to retrieve
+#       EdgeCWBRelease.tar.gz via a Gitlab package registry, which is
+#       what is used below (i.e., curl -O https://code.usgs.gov/...).
+#       -EJR
+RUN yum install -y net-tools sudo \
+    && mkdir /TEMP \
+    && cd /TEMP \
+    && curl -O https://code.usgs.gov/ghsc/neic/edgecwb/edgecwbgroup/edgecwbfiles/-/package_files/7388/download \
+    && mv -f download EdgeCWBRelease.tar.gz \
+    # && curl -O ftp://hazards.cr.usgs.gov/CWBQuery/EdgeCWBRelease.tar.gz \
+    && zcat EdgeCWBRelease.tar.gz | tar xf - \
+    && tar -xvzf EdgeCWBRelease.tar.gz \
+    && tar xf scripts_release.tar \
+    && cd scripts/INSTALL \
+    && ./addAccount vdl \
+    && sudo -u vdl cp /TEMP/EdgeCWBRelease.tar.gz ~vdl \
+    && cd ~vdl \
+    && sudo -u vdl /TEMP/scripts/installCWBRelease.bash \
+    && cp ~vdl/NoDB/DB/*.txt ~vdl/DB/. \
+    && rm -rf /TEMP \
+    && echo 'vdl - memlock 1024' >> /etc/security/limits.conf \
+    && echo 'vdl - stack 81920' >> /etc/security/limits.conf \
+    && echo 'vdl - nproc 20480' >> /etc/security/limits.conf \
+    && echo 'vdl - nofile 8192' >> /etc/security/limits.conf \
+    && mkdir /data \
+    && chown vdl:vdl /data /home/vdl
+
+# add docker entrypoint and configuration scripts
+COPY docker-configure.sh docker-entrypoint.sh /home/vdl/
+RUN chmod a+x /home/vdl/docker-entrypoint.sh /home/vdl/docker-configure.sh
+
+
+USER vdl
+EXPOSE 2060 2061 7974 7981
+WORKDIR /home/vdl
+CMD [ "./docker-entrypoint.sh" ]

localdev/geomag-edge/LICENSE.md

+Unless otherwise noted, This software is in the public domain because it
+contains materials that originally came from the United States Geological
+Survey, an agency of the United States Department of Interior. For more
+information, see the official USGS copyright policy at
+http://www.usgs.gov/visual-id/credit_usgs.html#copyright
+
+
+Disclaimers
+-----------
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+Information provided by this software may be preliminary or provisional and is
+subject to revision. It is being provided to meet the need for timely best
+science. The information has not received final approval by the U.S. Geological
+Survey (USGS) and is provided on the condition that neither the USGS nor the
+U.S. Government shall be held liable for any damages resulting from the
+authorized or unauthorized use of the information.

localdev/geomag-edge/README.md

+geomag-edge
+=============
+
+A lightweight configuration of an EDGE/CWB node in a container.
+
+
+Running the Docker image
+------------------------
+A version of the image created by this project is available on docker hub.
+```
+docker run -d --name localedge -p 2060:2060 usgs/geomag-edge
+```
+
+
+Building the Docker image
+-------------------------
+From the root of the project, run:
+```
+docker build -t usgs/geomag-edge:latest .
+```
+
+
+Related Projects
+--------------------
+
+- [EDGE/CWB wiki](https://github.com/usgs/edgecwb/wiki)
+
+The Dockerfile downloads the latest release of EDGE/CWB.
+
+- https://github.com/usgs/geomag-algorithms
+
+  Python library to process timeseries data, that can read from and write to an EDGE process.
+
+- https://github.com/usgs/geomag-edge-ws
+
+  JSON and IAGA2002 web service for data stored in EDGE.

localdev/geomag-edge/docker-configure.sh

+#! /bin/bash
+
+
+# only run configuration the first time
+RUN_ONCE=ran-docker-configure
+if [ -f "${RUN_ONCE}" ]; then
+  echo "## Already configured";
+  exit;
+else
+  touch "${RUN_ONCE}"
+fi
+
+
+################################################################################
+# configuration/defaults
+
+# files to write
+EDGE_CONFIG=edge.config
+EDGE_PROP=edge.prop
+EDGEMOM_SETUP=edgemom.setup
+QUERYMOM_SETUP=querymom.setup
+
+# figure out container ip address
+IP=$(ifconfig | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p')
+
+
+################################################################################
+# write edge.config
+echo "## Writing ${EDGE_CONFIG}"
+cat > "${EDGE_CONFIG}" <<-DONE
+Host=$(hostname)
+HostIP=$IP
+PublicIP=$IP
+DONE
+
+
+################################################################################
+# write edge.prop
+echo "## Writing ${EDGE_PROP}"
+cat > "${EDGE_PROP}" <<-DONE
+Station=DOCKER
+Network=DOCKER
+Node=1
+Instance=1
+
+DBServer=NoDB
+StatusDBServer=NoDB
+MetaDBServer=NoDB
+MySQLServer=NoDB
+AlarmIP=localhost
+StatusServer=localhost
+
+ndatapath=1
+daysize=3001
+extendsize=2000
+ebqsize=20000
+nday=10000000
+datapath=/data/
+logfilepath=/home/vdl/log/
+
+emailTo=
+SMTPFrom=
+SNWServer=
+SNWPort=0
+DONE
+
+
+################################################################################
+# write edgemom.setup
+
+echo "## Writing ${EDGEMOM_SETUP}"
+cat > "${EDGEMOM_SETUP}" <<-DONE
+Mom:EdgeMom:-empty >>edgemom
+Echn:EdgeChannelServer:-nodb >>echn
+# Load:MiniSeedServer:-nohydra -noudpchan -p 7965 >>load
+Replace:MiniSeedServer:-nohydra -noudpchan -p 7974 >>replace
+RawInput:RawInputServer:-nohydra -p 7981 -rsend 100 >>rawinput
+DONE
+
+
+################################################################################
+# write querymom.setup
+
+echo "## Writing ${QUERYMOM_SETUP}"
+cat > "${QUERYMOM_SETUP}" <<-DONE
+Mom:EdgeMom:-empty >>querymom
+Echn:EdgeChannelServer:-nodb >>echnqm
+QS:gov.usgs.cwbquery.EdgeQueryServer:-allowrestricted -mdsport 0 >>queryserver
+CWBWS:gov.usgs.anss.waveserver.CWBWaveServer:-allowrestricted -daysback 10000 -maxthreads 500 -mdsport 0 -nodb -p 2060 -nofilter -queryall >>cwbws
+DLQS:gov.usgs.cwbquery.DataLinkToQueryServer:-empty >>dlqs
+QSC:gov.usgs.cwbquery.QuerySpanCollection:-d 86400 -bands * >>qsc
+DONE
+
+
+################################################################################
+# Have EDGE process config files
+# NOTE: this overwrites the files above and so is commented out
+#
+#echo "## Running NoDBConfig"
+#java -cp ~vdl/bin/EdgeCWB.jar gov.usgs.anss.edgemom.NoDBConfig \
+#    -config "${EDGE_CONFIG}" \
+#    -once \
+#    -prop "${EDGE_PROP}"

localdev/geomag-edge/docker-entrypoint.sh

+#! /bin/bash
+
+
+# bash normally ignores SIGTERM
+_term () {
+  echo '## Caught SIGTERM, stopping edge processes'
+  kill -TERM "$alarm" "$edgemom" "$querymom"
+}
+trap _term SIGTERM
+
+
+################################################################################
+# configure environment
+
+source ~vdl/.bashrc
+
+echo '## Running docker-configure'
+./docker-configure.sh
+
+
+################################################################################
+# start edge processes
+
+echo '## Starting alarm'
+java -jar bin/EdgeCWB.jar alarm '^' 128 -alarm -noaction -nocfg -nodb &
+alarm=$!
+
+echo '## Starting edgemom'
+java -jar bin/EdgeCWB.jar edgemom '1#1' 499 -max -f edgemom.setup &
+edgemom=$!
+
+echo '## Starting querymom'
+java -jar bin/EdgeCWB.jar querymom '1#1' 161 -f querymom.setup &
+querymom=$!
+
+
+################################################################################
+# wait for edge processes to exit
+
+wait "$alarm" "$edgemom" "$querymom"

localdev/init.sql

+CREATE DATABASE IF NOT EXISTS geomag_operations;

localdev/local_development.md

+# Local Development
+
+## Background
+This docker-compose uses the main Dockerfile to create a `localdev-web-1` container and creates a mysql container called `localdev-database-1` and an edge container called `localdev-edge-1`.
+
+## Prerequisites
+For mac development:
+1. Download colima using homebrew.
+2. Start colima and edit virtual machine sizes using `colima start --edit`. Increase the number of CPUs to 16 and the memory to 16.
+3. Add the following env vars to your .bashrc or .zshrc or an .env file. See the instructions [here](../docs/metadata_webservice.md) for instructions on creating the OPENID_CLIENT_ID and OPENID_CLIENT_SECRET. The rest of the values can be set to strings of your choice. If you set the values in your .bashrc or .zshrc, run the source command to pick up the new values (ex `source ~/.bashrc`).
+```
+export OPENID_CLIENT_ID=""
+export OPENID_CLIENT_SECRET=""
+export SECRET_KEY=""
+export SECRET_SALT=""
+export MYSQL_ROOT_PASSWORD=""
+```
+
+## Use
+To simply start build and run all the containers, make sure you are in the /localdev directory and run the following command. This includes a reload command for the web container. It will restart the web container when changes are made locally.
+```
+docker-compose up
+```
+
+If you're not developing locally, use the following command to run the containers in the background. Reload will not work this way.
+```
+docker-compmose up -d 
+```
+
+## Helpful Docker Commands
+`docker ps` shows the running containers. 
+
+`docker ps -a` shows all the containers, even ones that have exited. This is helpful to see logs for containers that have unexpectedly exited.
+
+`docker exec -it {CONTAINER ID} /bin/bash` execs into the container. This can be helpful for exploring the mysql db. After you exec into the mysql container, run `mysql -u root -p` in the container. Input the password you set to MYSQL_ROOT_PASSWORD. Now you can use SQL to explore the db.
+
+## Helpful Tips
+Sometimes the mysql container exits immediately with code 139 due to insufficient memory. You can clean up your docker resources using the `cleanup-docker.sh` script in this folder. Restarting colima can also resolve this issue.