In these days where everything is moving to "the cloud", new challenges arises that most of us where spared from before; temporary condition such as network connectivity issues or simply a service unavailability. The technical term for this is called Transient Fault Handling and one you must be conversant with.
If it seems a big over the top for your solution, you could consider the static TransientFaultUtility class found in the Cuemon namespace; it is fully compatible with cloud provides such as Windows Azure. It has several overloads for invoking a fault sensitive method, and will continue until the operation is successful, the amount of retry attempts has been reached, or a failed operation is not considered related to a transient fault condition.
The minimum required parameters for invoking a transient fault protected method is an integer specifying retryAttempts, a function delegate that will determine if isTransientFault and last but not least; a function delegate/action delegate pointing to the faultSensitiveMethod.
To see this in action, have a look at Figure 1. What we do here is simply throwing an HttpException should we encounter a HTTP 502. This can easily be extended to the ones listed in the IsTransientFault callback method. Otherwise we just write some debug information. Figure 2 shows how to consume the TransientFaultExample class and is intentionally set to fail in the first run. Figure 3 shows the Debug Trace.
Figure 1: A simple TransientFaultExample class that can easily be rewritten to more real-life scenarios
Figure 2: Consumes the class defined in Figure 1
Figure 3: The output of the test in Figure 2
If you don't like the default recovery wait time this can easily be added using one of the overloads on the TransientFaultUtility class. For your convenience, I have included the default implementation in Figure 4.
Figure 4: The default implementation of the function delegate recoveryWaitTime
I hope this little introduction have inspired you to master Transient Fault Handling one way or another. Happy coding.