Runner intermittent failures
Here is a list of intermittent runner failures:
1.Running tests on ghsc/contractors intermittently fail with this error. This test is passing locally.
_______________ ERROR at setup of test_get_calculated_timeseries _______________
@pytest.fixture()
def bou_u_metadata():
> metadata = get_instrument_calibrations(observatory="BOU")
test/edge_test/MiniSeedFactory_test.py:49:Example pipeline https://code.usgs.gov/ghsc/contractors/ahobbs/geomag-algorithms/-/jobs/1316078
- Running Docker Docker Image intermittently fails with a few different errors.
- One is
TOOMANYREQEUSTSerror on thetrivy image --exit-code 1 --ignore-unfixed --severity LOW,MEDIUM "${LOCAL_IMAGE}";command. Example pipeline https://code.usgs.gov/ghsc/contractors/ahobbs/geomag-algorithms/-/jobs/1316079 - Another is a docker login error.
$ echo "$CI_REGISTRY_PASSWORD" | docker login $CI_REGISTRY -u $CI_REGISTRY_USER --password-stdin
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Error saving credentials: rename /root/.docker/config.json2274692847 /root/.docker/config.json: device or resource busExample pipeline https://code.usgs.gov/ghsc/contractors/ahobbs/geomag-algorithms/-/jobs/1316599
- The project is using the
ghsc/neic/motuswhich it shouldn't be.
Activity
As mentioned in the meeting, here's an example of the 1st issue (test stage failing) NOT happening for a non-contractor user: https://code.usgs.gov/ghsc/users/bgeels/geomag-algorithms/-/pipelines/512327
This test error message typically occurs if the system running the tests does not have access to staging-geomag.cr.usgs.gov.
Edited by Geels, Brendan Ryan@erigler @swilbur @bgeels @awernle @nshavers @ehunter @jdbrown We just had a meeting where we discussed most these issues. The test issue needs further investigation by logging in the tests.
The trivy image issue is known and being worked on.
The motus runner issue needs further investigation, which could include using exclude tags.
We are going to continue communicating in the teams chat about the ongoing investigations.
In regards to the Docker Login error, this was solved for geomag-plots by adding the following variable to the gitlab.yml file.
DOCKER_AUTH_CONFIG: '{"auths":{"$CI_REGISTRY":{"username":"$CI_REGISTRY_USER","password":"$CI_REGISTRY_PASSWORD"}}}' #for Docker Executor.Example:
Failing: https://code.usgs.gov/ghsc/contractors/nshavers/geomag-plots/-/jobs/1194730
Passing: https://code.usgs.gov/ghsc/contractors/nshavers/geomag-plots/-/jobs/1199402
I think it is a permissions issue, the docker executor starts off by using crendentials from the payload, or from an existing config.json file on the server, When calling the 'docker login' via the script, it attempts to change the config.json file at runtime, which sudo permissions would be necessary to do so. But by using the docker_auth_config variable, we force the docker executor to use the given credentials from the variable at 'compile time', so the config.json doesn't change at runtime, and sudo privileges are not necessary.
mentioned in merge request !353 (merged)
Regarding the 'motus' issue (number 3 in the description), from searching it looks like excluding tags/runners is a requested feature that is not yet implemented in gitlab. @jdbrown @ehunter would it be possible to add a
geomagtag to the runners on igskahcgvmd[12]ct[123] and mage[12].cr.usgs.gov?Edited by Geels, Brendan Ryan
Mage have their own runners,
https://code.usgs.gov/groups/ghsc/geomag/-/runners?search=mageAs for the container servers, here is d1ct1
https://code.usgs.gov/groups/ghsc/geomag/-/runners?search=d1ct1Yes, we could add an additional
geomagtags for those runners.Edited by Eddie J Hunter
mentioned in merge request !357 (merged)
mentioned in merge request !358 (merged)
