Database Standardization
@awernle , @swilbur , @bgeels , @erigler , @nshavers This is a large MR to standardize and update the connections and interactions with the database.
Previously, we were using the databases library to connect to the mysql database. I replaced this with the sqlmodel library for 4 reasons.
- The databases library is not as actively developed and maintained as the sqlmodel library.
- There is outstanding (but not perfect) documentation for sqlmodel and it is designed to be used with FastAPI and Pydantic. They are all written by the same person and maintained by the same group.
- The sqlmodel wrapper around sqlalchemy is much easier to read.
- sqlmodel optimizes database connections through the use of the "engine" and it's not clear to me how the databases library was handling them. https://sqlmodel.tiangolo.com/tutorial/create-db-and-table/#create-the-engine
Unfortunately, a limitation of using sqlmodel is that the metadata key is already is being used for sqlalchemy's MetaData. Which means we need another layer between the sqlmodel metadata and metadata_history tables. I've solved this problem by setting an alias on the Table classes and an alias on the BaseModel classes to allow easy translation between the two classes. Only the factory layer should be aware of this issue. It should accept the user-facing BaseModels, translate into Table classes, and translate back again to BaseModels. Here is a diagram to help visualize this.
Another large change is around code organization. I've moved the metadata classes into a models folder under api. This is based on the suggested structure by sqlmodel. And it removes the many confusing metadata.py file names. I've also separated out the metadata_history database logic into it's own factory and models. These tables are very similar from a user perspective, but very unique in actuality and we should separate these concerns. I also removed a lot of the extraneous function calls around the session middleware.
These changes allow for these potential next steps to easily happen.
- Replace MetadataFactory with MetadataDatabaseFactory. See the above diagram. This will allow current and new typer interfaces to easily integrate with the existing MetadataDatabaseFactory.
- Validate sub metadata and potentially easily move sub metadata into different tables.
I know there are a lot of changes here. I'm open to all comments and suggestions!