[ptt-users] Re: Understanding TM behaviour/results

Wed Sep 12 06:44:26 PDT 2007

Frank Cohen wrote:
> Dear Dominique: See below. -Frank
> 
> On Sep 12, 2007, at 7:26 AM, Dominique de Waleffe wrote:
> 
>> Frank,
>>
>> As I wrote, we are setting up our environment and test tools, based on
>> TestMaker/Selenium, as you know.
>>
>> In this process we are also trying to discover the power of TM and we
>> need to be able to correctly interpret the results...
>>
>> Currently, I have 2 puzzles:
>>
>> a) intuition not coherent with reports.
>>
>> I run a use case on 1 testnode with crlevel in cra,crb,crc where
>> crb=4*cra [cra=5]. TM gives me a TPS result of ra=15 and rb=22.
>> So this tell me that my server is able to provide 22TPS for a load 20 
>> CVUS.
> 
> Let me make sure I understand your cra, crb, crc settings. If cra = 5, 
> then crb = 20 and crc = 100. Is that correct?
> 
> Your answer that at 20 CVUs your server is able to provide 22 TPS is 
> what I would expect.
> 
>> Now I run the same thing, but on 4 different testnodes. I see that the
>> number of threads on each of them corresponds to the crx. So this leads
>> me to think that for cra  with 4 nodes I put a load equivalent to that
>> of crb with only one node, so I expect a TPS value close to that of crb
>> with one node. However, I still get a TPS value close to ra=15.
>>
>> So is my intuition correct?
> 
> Your intuition is correct.
> 
> With crb (20 crs) running on one node you should check the CPU, network, 
> and memory bandwidth of the node to make certain that the test is not 
> resource constrained.

Indeed. I am using PTTMonitor/Task manager/top/vmstat to monitor the 
machine and just discovered that apache was finally failing on me at 
some point...got better results after fixing configuration...

> 
>> How can we understand the figures reported when running on several node?
>> [Though I did only read quickly through the chapter on TM in FastSOA, I
>> did not spot an explanation on this aspect (more than 1 node)]
> 
>> If this intuition is correct, should the graphs/reported figures take
>> into account the number of nodes on which the usecase runs?
> 
> The figures you see in the Controller Panel for a load test are an 
> average of all the running nodes. The Performance Comparison Utility 
> will let you analyze the performance of each node. See:
> http://docs.pushtotest.com/docs/analyzingresults.html

Ok. So from the summary graph I get, I can conclude that my server is 
able to serve #nodes*TPS figure reported for #nodes*CVUs

In my example, running usecase with 5 CVUs on 4 nodes and seeing a 
result of 15, I may conclude that my server is able to provide 60 TPS 
under a load of 20 CVUs.

Now if you look at the attached screenshot, it show both the graph and 
the summary results in text form. And there it looks like the graph 
shows the sum of the TPS for each node and not the average.

Is a division by #nodes missing?

> 
>>
>> b) timing of usecase starts on different nodes.
>>
>> What is the sequence of events when starting a test on several nodes? 
>> How is the duration of the test related to this and how/when are 
>> successful transactions counted?
> 
> TestMaker distributes the test files to all of the nodes, starts the 
> first test use case (for example, cra) by instantiating threads for each 
> cr, once all of the nodes have instantiated their threads then TestMaker 
> begins logging the results for the period of the test use case, then 
> TestMaker stops logging and takes-down all of the threads, then 
> TestMaker repeats at the new level of crs.
> 
> There is information on the sequence of events in the TestMaker 5 
> specification document at: 
> http://downloads.pushtotest.com/tm5/TM5_Specification.pdf

Will print this out. Thanks.

>>
>> Is there any detailed explnations of this anywhere?
>>
> 
> 
> By the way, other TestMaker users would benefit from this information. 
> It would be great to answer your questions on the 
> users at lists.pushtotest.com email list so the answers go into our archive.

Indeed. Will forward this discussion to the list.

D.

-- 
Dominique de Waleffe
Technical director

Denali S.A., "Bridging the gap between Business and IT"
Rue de Clairvaux 8, B-1348 Louvain-la-Neuve, Belgium
Office: +32 10 43 99 51 Fax: +32 10 43 99 52
http://www.denali.be

Legal notice: this message and its attachments may contain
confidential and/or privileged information. If you are not the
addressee or authorized to receive this for the addressee, you must
not use, copy, disclose or take any action based on this message or
any information herein. If you have received this message by mistake,
please advise the sender immediately by return e-mail and delete this
message from your system. Thank you for your cooperation.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tm5-perf-res.jpg
Type: image/jpeg
Size: 115260 bytes
Desc: not available
Url : http://cake.pushtotest.com/pipermail/users/attachments/20070912/54d09cfe/tm5-perf-res-0001.jpg