Product JSON Size Limits
Based on group discussion:
-
Check current payload size of JSON messages being sent to web socket? - What are min/max/median/average sizes?
- Any outliers by product type?
-
Add check before sending product that if the generated JSON message size would exceed the maximum payload size (128 KB), then an exception is thrown. - Make this check configurable and disable-able. For example:
public static string MAX_PAYLOAD_BYTES_PROPERTY = "maxPayloadBytes"; public static long DEFAULT_MAX_PAYLOAD_BYTES = 128 * 1024; // 128 KB private long maxPayloadBytes = DEFAULT_MAX_PAYLOAD_BYTES; // ... during configuration ... this.maxPayloadBytes = ... // Read configured property value or default value // ... later before sending a product ... if (notificationBytes > this.maxPayloadBytes && this.maxPayloadBytes != 0) { throw SomeCustomException("Maximum payload exceeded!"); }
Original Description Below
When the client sends a product using the AWSProductSender, the content are uploaded and a Product JSON payload is sent to the `send_product` endpoint. This Product JSON payload is subsequently pushed into an SNS Topic and then onto an SQS Queue ultimately triggering a Lambda Function that notifies connected clients about the new product availability. This notification includes the Product JSON payload. Notifications currently leverages the API Gateway `PostToConnection` endpoint which has a payload maximum of 128 KB with 32 KB maximum frame size (https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html#apigateway-execution-service-websocket-limits-table). If this payload exceeds the maximum size, a `PayloadTooLargeException` occurs and clients are not notified.Question
Should this case be handled prior to the stage where clients are attempting to be notified? It seems like the sender should be notified their product is too large, otherwise the sender will assume the product has been sent, but the clients will never be notified, so effectively the product was sent, but only into the void.
Option 1 - Java client application checks payload size before calling send_product
endpoint on the server and throws an exception to short-circuit the process. This is the earliest possible time this error may be caught and handled.
Option 2 - The server send_product
endpoint checks the payload size before attempting to relay, and notifies clients if product is too large to be relayed. If users write their own client, they may not check payload sizes and may still end up sending products "into the void".
Option 3 - Both option 1 and option 2. This allows earliest checks possible while still handling case of user-created clients to upload products.
@khaynie - Depending on the decision, this may affect the infrastructure project as well.