tag:blogger.com,1999:blog-34935317373439451492024-03-13T05:23:16.232-07:00Frank (Kancheng) Cao's notesFrank's thoughs on C#, web services, SOA, .Net, design etc.Anonymoushttp://www.blogger.com/profile/06611918080745376316noreply@blogger.comBlogger1125tag:blogger.com,1999:blog-3493531737343945149.post-60334436114274772032012-06-14T22:10:00.000-07:002012-06-17T15:41:54.075-07:00Adding Timeout and Congestion Prevention in a WCF/.Net Remoting Multi-tier Server<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="font-size: large;"><span class="Heading1Char">Background</span>: </span></b><o:p></o:p></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<b><span style="font-size: large;"><br /></span></b></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
Visually
similar is an optional service in Bing.com shopping site which start from a clothing
or shoes item and finds other items that looks similar. If our response time is
beyond 750 milliseconds, the requests will timeout on client side. The shopping
team is investigating a problem that all requests fail against our service in one
center. We suspect that occasional timeouts in our service is causing their
server to go haywire. They wrote a test program which repeatedly query our
service, and found 13% of request timeout under normal load of 7 QPS (query per
second). I don't want to tune our search algorithm because it was developed by
a teammate who left the team. So, I decided to enforce a timeout on server
side.<o:p></o:p></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
In
the following graph: The response time occasionally was very high. <o:p></o:p></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-j2KL8xPiTu8/T9rDD884BxI/AAAAAAAAACA/-lQYsQRHrqo/s1600/QPS5_1.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="192" src="http://3.bp.blogspot.com/-j2KL8xPiTu8/T9rDD884BxI/AAAAAAAAACA/-lQYsQRHrqo/s320/QPS5_1.gif" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-IKQcksY7rrI/T9rDEB6fPKI/AAAAAAAAACI/13xwHt05Rs4/s1600/QPS7_1.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="192" src="http://1.bp.blogspot.com/-IKQcksY7rrI/T9rDEB6fPKI/AAAAAAAAACI/13xwHt05Rs4/s320/QPS7_1.gif" width="320" /></a></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: 'Times New Roman', serif;"><br /></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: 'Times New Roman', serif;"><br /></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<h1>
Analysis<o:p></o:p></h1>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
Architecture:
Our service has the following architecture: <o:p></o:p></div>
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt 27pt;">
WCF Client at
Shopping server -> WCF service/.Net remoting client -> Fan out connection
with 3 .Net remoting server, which perform actual search.<o:p></o:p></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
WCF
client is beyond our control, so I can insert the timeout at the following
places, in order of increasing complexity: <o:p></o:p></div>
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt 27pt; text-indent: -0.25in; vertical-align: middle;">
1.<span style="font-size: 7pt;">
</span>add timeout configuration in WCF service
definition<o:p></o:p></div>
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt 27pt; text-indent: -0.25in; vertical-align: middle;">
2.<span style="font-size: 7pt;">
</span>add timeout configuration in .Net remoting client
definition<o:p></o:p></div>
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt 27pt; text-indent: -0.25in; vertical-align: middle;">
3.<span style="font-size: 7pt;">
</span>add custom timeout code in WCF service <o:p></o:p></div>
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt 27pt; text-indent: -0.25in; vertical-align: middle;">
4.<span style="font-size: 7pt;">
</span>add custom timeout code in .Net remoting server <o:p></o:p></div>
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt 27pt;">
<br /></div>
<h1>
Attemp 1<o:p></o:p></h1>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
I
first tried #1 since it is easy and most up-stream to the call chain. All I had
to do is to add the following to the web.config file:<o:p></o:p></div>
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt 27pt;">
<span style="font-family: Consolas; font-size: 10pt;"> </span><span style="color: blue; font-family: Consolas; font-size: 10pt;"><</span><span style="color: #a31515; font-family: Consolas; font-size: 10pt;">bindings</span><span style="color: blue; font-family: Consolas; font-size: 10pt;">></span><span style="font-family: Consolas; font-size: 10pt;"><br />
</span><span style="color: blue; font-family: Consolas; font-size: 10pt;"><</span><span style="color: #a31515; font-family: Consolas; font-size: 10pt;">basicHttpBinding</span><span style="color: blue; font-family: Consolas; font-size: 10pt;">></span><span style="font-family: Consolas; font-size: 10pt;"><br />
</span><span style="color: blue; font-family: Consolas; font-size: 10pt;"><</span><span style="color: #a31515; font-family: Consolas; font-size: 10pt;">binding</span><span style="font-family: Consolas; font-size: 10pt;"> </span><span style="color: red; font-family: Consolas; font-size: 10pt;">name</span><span style="color: blue; font-family: Consolas; font-size: 10pt;">=</span><span style="font-family: Consolas; font-size: 10pt;">"</span><span style="color: blue; font-family: Consolas; font-size: 10pt;">Binding1</span><span style="font-family: Consolas; font-size: 10pt;">" ...<br />
</span><span style="color: red; font-family: Consolas; font-size: 10pt;">receiveTimeout</span><span style="color: blue; font-family: Consolas; font-size: 10pt;">=</span><span style="font-family: Consolas; font-size: 10pt;">"</span><span style="color: blue; font-family: Consolas; font-size: 10pt;">00:00:01</span><span style="font-family: Consolas; font-size: 10pt;">"<br />
</span><span style="color: red; font-family: Consolas; font-size: 10pt;">sendTimeout</span><span style="color: blue; font-family: Consolas; font-size: 10pt;">=</span><span style="font-family: Consolas; font-size: 10pt;">"</span><span style="color: blue; font-family: Consolas; font-size: 10pt;">00:00:01</span><span style="font-family: Consolas; font-size: 10pt;">"</span><span style="font-family: Consolas; font-size: 10pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
Unfortunately,
this did not work. I found the following web forum discussion saying the WCF
service timeout will be 22 seconds even if the value is configured to be
smaller.<o:p></o:p></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<a href="http://social.msdn.microsoft.com/Forums/en-US/wcf/thread/5f347965-13bf-4a2d-ae82-74ad38a8d7d1/"><span style="color: blue;">http://social.msdn.microsoft.com/Forums/en-US/wcf/thread/5f347965-13bf-4a2d-ae82-74ad38a8d7d1/</span></a><o:p></o:p></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<a href="http://stackoverflow.com/questions/9184424/enforce-wcf-client-sendtimeout"><span style="color: blue;">http://stackoverflow.com/questions/9184424/enforce-wcf-client-sendtimeout</span></a><o:p></o:p></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<a href="http://social.msdn.microsoft.com/Forums/en/wcf/thread/df63589e-64ee-4327-aa21-cc65938dbce9"><span style="color: blue;">http://social.msdn.microsoft.com/Forums/en/wcf/thread/df63589e-64ee-4327-aa21-cc65938dbce9</span></a><o:p></o:p></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<h1>
Attemp 2<o:p></o:p></h1>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
I
then tried #2, because it is next easiest. I just need to add a global
configuration to the .Net remoting client. When timeout expires, the .Net
remoting call will throw exception. The
following is the code:<o:p></o:p></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: #2b91af;">IDictionary</span>
t = <span style="color: blue;">new</span> <span style="color: #2b91af;">Hashtable</span>();<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: #2b91af;">SoapClientFormatterSinkProvider</span> sc = <span style="color: blue;">new</span> <span style="color: #2b91af;">SoapClientFormatterSinkProvider</span>();<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> t.Add(<span style="color: #a31515;">"timeout"</span>, (<span style="color: blue;">uint</span>)600);<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: #2b91af;">TcpClientChannel</span> clientChannel = <span style="color: blue;">new</span> <span style="color: #2b91af;">TcpClientChannel</span>(t,
sc);<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: #2b91af;">ChannelServices</span>.RegisterChannel(clientChannel, <span style="color: blue;">false</span>);<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
This
is very elegant programing model, and simple to implement, However, the actual
timeout was 485 milliseconds longer than given in the settings. (I tested
timeout setting of 200 and 600 milliseconds.) This is the response from the
"remoting support" team: <o:p></o:p></div>
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt 27pt;">
<span style="color: #1f497d;">"I
don’t think it guarantees the exception will be raised the exact time specified
in the timeout value only that it will be hit if the timeout value specified is
exceeded."<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
We
need accurate timeout values. So, I cannot use this approach. <o:p></o:p></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<h2>
Attempt 3<o:p></o:p></h2>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
I
then tried #3 and #4 with the following sample timeout code: <o:p></o:p></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">delegate</span> <span style="color: #2b91af;">QueryResults</span>
<span style="color: #2b91af;">MethodDelegate2</span>(<span style="color: #2b91af;">Decimal</span>
prodID, <span style="color: blue;">...</span>);<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">public</span> <span style="color: #2b91af;">QueryResults</span>
SearchVisualSimilarRangeWithTimeout(<span style="color: #2b91af;">Decimal</span>
prodID, <span style="color: blue;">...</span>)<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: #2b91af;">MethodDelegate2</span> dlgt = <span style="color: blue;">new</span>
<span style="color: #2b91af;">MethodDelegate2</span>(client.SearchVisualSimilarRange);<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: green;">// Initiate the asynchronous call.</span><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: #2b91af;">IAsyncResult</span> ar = dlgt.BeginInvoke(prodID, ..., <span style="color: blue;">null</span>, <span style="color: blue;">null</span>);<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: green;">// Wait for the WaitHandle to become signaled.</span><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">if</span> (!ar.AsyncWaitHandle.WaitOne(timeMsec))<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">throw</span> <span style="color: blue;">new</span> <span style="color: #2b91af;">TimeoutException</span>();<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: green;">// Get the results of the asynchronous call.</span><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">return</span> dlgt.EndInvoke(ar);<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
I
made the call to be asynchronous. When a call timeout, I will allow it to run
to completion asynchronously, but the main call thread will return immediately
after the timeout. <o:p></o:p></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
#3
did not work, and #4 partially worked in that timeout decreased from 13% to 3%.
Also, the problem is worse when there is more requests per second. The
following are the response time of reported by the test program:<o:p></o:p></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: 'Times New Roman', serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-CUwHuTKpPRw/T9rDaA3_X-I/AAAAAAAAACQ/-RGwIxuVKrU/s1600/QPS5_2.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="192" src="http://2.bp.blogspot.com/-CUwHuTKpPRw/T9rDaA3_X-I/AAAAAAAAACQ/-RGwIxuVKrU/s320/QPS5_2.gif" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-5e91GZhu448/T9rDafqTz2I/AAAAAAAAACY/gvG2zP4X0z0/s1600/QPS7_2.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="192" src="http://1.bp.blogspot.com/-5e91GZhu448/T9rDafqTz2I/AAAAAAAAACY/gvG2zP4X0z0/s320/QPS7_2.gif" width="320" /></a></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: 'Times New Roman', serif;"><br /></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<h1>
Attempt 4<o:p></o:p></h1>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
With
#4, it is possible that mess-up in the WCF service code could cause timeout.
So, I have to make #3 work. To troubleshoot why #3 did not work, I added
tracing with time stamp to the WCF service, and discovered the dynamics of the
timeout scenario: <o:p></o:p></div>
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt 27pt; text-indent: -0.25in; vertical-align: middle;">
<span style="font-family: Symbol; font-size: 10pt;">·<span style="font-family: 'Times New Roman'; font-size: 7pt;">
</span></span>Large part of perceived call time on the client
includes the time when the request is in the WCF request queue. My timeout code
doesn’t help in this situation. <span style="font-family: 'Times New Roman', serif; font-size: 12pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt 27pt; text-indent: -0.25in; vertical-align: middle;">
<span style="font-family: Symbol; font-size: 10pt;">·<span style="font-family: 'Times New Roman'; font-size: 7pt;">
</span></span>When WCF service has several threads running, it
may decide to hold more quests in the queue. This queue behavior is not
consistent, and WCF may decide to send a large number of requests to the
service code at once. This creates a wave of congestion in the .Net remoting
server and slow response time. <span style="font-family: 'Times New Roman', serif; font-size: 12pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt 27pt; text-indent: -0.25in; vertical-align: middle;">
<span style="font-family: Symbol; font-size: 10pt;">·<span style="font-family: 'Times New Roman'; font-size: 7pt;">
</span></span>The above 2 factors add together to become a more
severe congestion problem. <span style="font-family: 'Times New Roman', serif; font-size: 12pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
To
solve this problem, I added code in WCF service to drop new requests if the
number of concurrent requests is more than a threshold. Our search algorithm is
CPU intensive and use multiple threads, so if I allow more request through
while our system is already overloaded, it will cause much larger number of
requests to timeout. Furthermore, the ripple effect of the queue discussed
above can congest our entire system for a long period of time. So, this should
actually increase the overall success rate of our request.<o:p></o:p></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">volatile</span> <span style="color: blue;">static</span> <span style="color: blue;">int</span> concurr = 0; <o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">try</span><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;">
Interlocked.Increment(<span style="color: blue;">ref</span> concurr);<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">if</span> (concurr > InitState.Instance.maxConcurr)<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> { <span style="color: green;">// No locking here since this is heuristic anyway...</span><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">return</span> <span style="color: blue;">new</span>
KEXResponse();<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> ...<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">finally</span><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;">
Interlocked.Decrement(<span style="color: blue;">ref</span> concurr);<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
Alternatively,
I could investigate ways to solve the problem with different WCF queuing
configurations, but the above simple fix did the trick, so there is no need to
dig more into WCF for a solution. <o:p></o:p></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<h1>
Result<o:p></o:p></h1>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
The
result: after 2 days of work, the
combined dropped request& timeout rate observed by client dropped from 13%
to below 1%, with timeout rate dropping to 1 in 2000. Also, the system is much
more stable at large request per second load. The feature is back online after
the fix is deployed. <o:p></o:p></div>
<h1>
Lessons learnt: <o:p></o:p></h1>
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt 27pt; text-indent: -0.25in; vertical-align: middle;">
<span style="font-family: Symbol; font-size: 10pt;">·<span style="font-family: 'Times New Roman'; font-size: 7pt;">
</span></span>Never take documented behaviors or WCF and .Net
remoting for granted. Test it with a small program before applying it to
product code. <span style="font-family: 'Times New Roman', serif; font-size: 12pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt 27pt; text-indent: -0.25in; vertical-align: middle;">
<span style="font-family: Symbol; font-size: 10pt;">·<span style="font-family: 'Times New Roman'; font-size: 7pt;">
</span></span>Ripple effect between WCF queuing and backend
service may increase congestion, and a good test program is needed to simulate
real traffic and uncover the problem. <span style="font-family: 'Times New Roman', serif; font-size: 12pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 0in 0in 0.0001pt 27pt; text-indent: -0.25in; vertical-align: middle;">
<span style="font-family: Symbol; font-size: 10pt;">·<span style="font-family: 'Times New Roman'; font-size: 7pt;">
</span></span>Combination of good testing program, tracing, and
problem analysis can sometimes solve problems faster than profiling tools and
debugging. <span style="font-family: 'Times New Roman', serif; font-size: 12pt;"><o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>Anonymoushttp://www.blogger.com/profile/06611918080745376316noreply@blogger.com0