Skip to content

subpixel_register_point results not being written to database because of sqlalchemy Session autoflush

Created by: ladoramkershner

When doing large cluster batch processing (~300k+ jobs), I get this error intermittently.

sqlalchemy.exc.ProgrammingError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely)
(psycopg2.errors.UndefinedTable) relation "measures" does not exist
LINE 1: UPDATE measures SET "templateMetric"=0.9283186197280884, "te...
               ^

[SQL: UPDATE measures SET "templateMetric"=%(templateMetric)s, "templateShift"=%(templateShift)s WHERE measures.id = %(measures_id)s]
[parameters: {'templateMetric': 0.9283186197280884, 'templateShift': 14.545188207788856, 'measures_id': 1390828}]
(Background on this error at: http://sqlalche.me/e/13/f405)

It suggests to turn off the auto flush feature to prevent this, and according to the documentation on the sqlalchemy Session.init

autoflush – When True, all query operations will issue a Session.flush() call to this Session before proceeding. This is a convenience feature so that Session.flush() need not be called repeatedly in order for database queries to retrieve results. It’s typical that autoflush is used in conjunction with autocommit=False. In this scenario, explicit calls to Session.flush() are rarely needed; you usually only need to call Session.commit() (which flushes) to finalize changes.

Finally since the session_scope context manager in the NetworkCandidateGraph ends with

finally:
    session.close()

I think we are totally justified to turn this autoflush feature off. This is not detrimental to the control process, but does cause unnecessary errors.

Edited by Laura, Jason R.