Improve Error Catching, Propagation and Error Logging

[davidgxue]

Please describe the feature you'd like to see
The current error logging mechanism isn't efficient. Retry errors can be spammy and provide less insightful information on where the error occurred. Other times, client side errors also are propagated as server side errors (e.g. Status code 400 from Azure OpenAI API due to either content moderation filter being triggered where prompt is rejected, or prompt is too long exceeding certain # of tokens)

Example 1 for client side error catching:

BadRequestError('Error code: 400 - {\'error\': {\'message\': "This model\'s maximum 
context length is 4096 tokens. However, your messages resulted in 4512 tokens. 
Please reduce the length of the messages.", \'type\': \'invalid_request_error\', 
\'param\': \'messages\', \'code\': \'context_length_exceeded\'}}')

Example 2 for client side error catching:

BadRequestError('Error code: 400 - {\'error\': {\'message\': "The response was filtered due to 
the prompt triggering Azure OpenAI\'s content management policy. Please modify your prompt 
and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766", \'type\': None, \'param\': \'prompt\',
 \'code\': \'content_filter\', \'status\': 400}}')

Describe the solution you'd like
Improve error propagation and logging

Related Issue

• This issue is created from this issue Alert only on relevant GCP error in slack #243

Acceptance Criteria

• Make code changes to improve the overall error propagation mechanism