Whatever can go wrong, will go wrong.<\/p>\n<\/blockquote>\n
It\u2019s so true in software engineering.<\/p>\n
To build resilient software applications, when architecting the integration points with downstream services, we shall consider all error scenarios. Robust error handling is essential. Retrying remote API calls is an important part.<\/p>\n
A retry can be done either synchronously or asynchronously. If the clients require a response of the execution status, not just the acknowledgement of the receipt of the request, it\u2019s appropriate to implement synchronous retries with limits on total retry numbers and time. On the other hand, if the clients don\u2019t care about the actual execution status, or have ways to receive responses asynchronously, it is almost always a good idea to adopt asynchronous retry architecture. Of course, before putting a request into asynchronous retry process, we can always implement synchronous retry first whenever it makes sense.<\/p>\n
In this article, we will focus on the asynchronous retry architecture.<\/p>\n
2. Queuing for Asynchronous Retry Architecture<\/strong><\/h1>\n
Queuing mechanism is the center of the Asynchronous Retry Architecture.<\/p>\n
The originating service constructs a Retry Message<\/strong> that includes the original request info, the destination URL and other metadata, puts the Retry Message<\/strong> into an Async Retry Queue<\/strong> based on the chosen queuing system. A trigger could be configured in the queue to trigger a processor. Or, an Async Retry Processor<\/strong> can pull the queuing system for new messages. The Async Retry Processor<\/strong> can then utilize the message received from the Async Retry Queue<\/strong> and make another call to the destination downstream service.<\/p>\n
A Dead Letter Queue<\/strong> is used to hold Retry Message<\/strong>s for certain period time after a (configurable) maximum number of retries have been reached.<\/p>\n
The below figure is a very high level workflow and message flow:<\/p>\n
In the above diagram, Service A<\/strong> is the calling service and Service B<\/strong> is the destination downstream service. If the initial call in Step 1<\/strong> fails, Service A<\/strong> will put a Retry Message<\/strong> into the Async Retry Queue<\/strong>.<\/p>\n
Depends on what Queuing System is chosen, either a trigger can be configured in the Async Retry Queue<\/strong> to trigger the Processor (3.1), or a Processor can be configured to poll the Async Retry Queue<\/strong> (3.2). If AWS SQS is chosen as the queuing system, a Lambda function can be configured to trigger the processor when a new message arrives.<\/p>\n
Once the Async Retry Processor<\/strong> receives the Retry Message<\/strong>, it can use the request info in the message to reconstruct the request, and send the request to the destination URL that is also included in the retry message.<\/p>\n
A Retry Message<\/strong> will be moved to the Dead Letter Queue<\/strong> if the maximum retry attempts have been reached as detected by the Async Retry Processor<\/strong> or the Async Retry Queue<\/strong>.<\/p>\n
4.<\/strong> Generic Data Model for Retry Message<\/strong><\/h1>\n
A Retry Message can have a generic data model as below:<\/p>\n
<span id="60ab" class="hc wp tx so wq b gg wr ws x wt" data-selectable-paragraph="">{\r\n "request":{\r\n "url":"http[s]:\/\/$host:$port\/$destitnation_endpoint_including_query_parameters",\r\n "method":"GET|POST|PUT|DELETE|PATCH", \r\n "payload":"$json_string",\r\n "headers":[$headers_to_pass_to_the_target_service]\r\n },\r\n "receivedCount": "$number",\r\n "async-retry-queue":"$async-retry-queue[optional]",\r\n "dead-letter-queue":"$dead-letter-queue[optional]"\r\n}<\/span><\/pre>\n
With this generic data model design for Retry Message<\/strong>, an Async Retry Processor<\/strong> can be designed to process any Retry Message<\/strong> constructed by any originating services (Service A<\/strong>) to any destination services (Service B<\/strong>).<\/p>\n
5. Retryable Errors<\/strong><\/h1>\n
Only non-functional errors are retryable. Below are some examples:<\/p>\n
a. No response at all;<\/p>\n
b. Temporary Network issue, usually 5xx<\/strong> (http status) errors;<\/p>\n
c. Request timeout: http status 408 errors;<\/p>\n
d. Conflict: http status 409 errors;<\/p>\n
e. Too many request: http status 429<\/p>\n
f. If there is a Retry-After header in the http response of the downstream service;<\/p>\n
h. Unauthorized: http status 401 errors with expired token error code\/message. These kind of errors usually require a new token. In this case, the Async Retry Processor is responsible for getting the proper token.<\/p>\n
6. <\/strong>Conclusion<\/h1>\n
Asynchronous Retry Architecture<\/strong> can be used to handle all retryable errors when the client is not expecting the execution result in the response of the call. It is extremely useful if a function may need to be tried many times for a long period of time.<\/p>\n
The number of maximum retry attempts, the async retry queue name\/url, and the dead letter queue name\/url can all be configurable. The configurable values can make architecture flexible for many different applications.<\/p>\n","protected":false},"excerpt":{"rendered":"
![\"\"](\"https:\/\/i0.wp.com\/blog.softwareclues.com\/wp-content\/uploads\/2020\/10\/1.png?resize=700%2C366\")