[ptt-users] Re: Understanding TM behaviour/results

[Frank Cohen]

It could very well be that you discovered a bug where the reports are  
not dividing by the number of nodes. I will check this and let you  
know. -Frank



On Sep 12, 2007, at 8:44 AM, Dominique de Waleffe wrote:

> Frank Cohen wrote:
>> Dear Dominique: See below. -Frank
>> On Sep 12, 2007, at 7:26 AM, Dominique de Waleffe wrote:
>>> Frank,
>>>
>>> As I wrote, we are setting up our environment and test tools,  
>>> based on
>>> TestMaker/Selenium, as you know.
>>>
>>> In this process we are also trying to discover the power of TM  
>>> and we
>>> need to be able to correctly interpret the results...
>>>
>>> Currently, I have 2 puzzles:
>>>
>>> a) intuition not coherent with reports.
>>>
>>> I run a use case on 1 testnode with crlevel in cra,crb,crc where
>>> crb=4*cra [cra=5]. TM gives me a TPS result of ra=15 and rb=22.
>>> So this tell me that my server is able to provide 22TPS for a  
>>> load 20 CVUS.
>> Let me make sure I understand your cra, crb, crc settings. If cra  
>> = 5, then crb = 20 and crc = 100. Is that correct?
>> Your answer that at 20 CVUs your server is able to provide 22 TPS  
>> is what I would expect.
>>> Now I run the same thing, but on 4 different testnodes. I see  
>>> that the
>>> number of threads on each of them corresponds to the crx. So this  
>>> leads
>>> me to think that for cra  with 4 nodes I put a load equivalent to  
>>> that
>>> of crb with only one node, so I expect a TPS value close to that  
>>> of crb
>>> with one node. However, I still get a TPS value close to ra=15.
>>>
>>> So is my intuition correct?
>> Your intuition is correct.
>> With crb (20 crs) running on one node you should check the CPU,  
>> network, and memory bandwidth of the node to make certain that the  
>> test is not resource constrained.
>
> Indeed. I am using PTTMonitor/Task manager/top/vmstat to monitor  
> the machine and just discovered that apache was finally failing on  
> me at some point...got better results after fixing configuration...
>
>>> How can we understand the figures reported when running on  
>>> several node?
>>> [Though I did only read quickly through the chapter on TM in  
>>> FastSOA, I
>>> did not spot an explanation on this aspect (more than 1 node)]
>>> If this intuition is correct, should the graphs/reported figures  
>>> take
>>> into account the number of nodes on which the usecase runs?
>> The figures you see in the Controller Panel for a load test are an  
>> average of all the running nodes. The Performance Comparison  
>> Utility will let you analyze the performance of each node. See:
>> http://docs.pushtotest.com/docs/analyzingresults.html
>
> Ok. So from the summary graph I get, I can conclude that my server  
> is able to serve #nodes*TPS figure reported for #nodes*CVUs
>
> In my example, running usecase with 5 CVUs on 4 nodes and seeing a  
> result of 15, I may conclude that my server is able to provide 60  
> TPS under a load of 20 CVUs.
>
> Now if you look at the attached screenshot, it show both the graph  
> and the summary results in text form. And there it looks like the  
> graph shows the sum of the TPS for each node and not the average.
>
> Is a division by #nodes missing?
>
>>>
>>> b) timing of usecase starts on different nodes.
>>>
>>> What is the sequence of events when starting a test on several  
>>> nodes? How is the duration of the test related to this and how/ 
>>> when are successful transactions counted?
>> TestMaker distributes the test files to all of the nodes, starts  
>> the first test use case (for example, cra) by instantiating  
>> threads for each cr, once all of the nodes have instantiated their  
>> threads then TestMaker begins logging the results for the period  
>> of the test use case, then TestMaker stops logging and takes-down  
>> all of the threads, then TestMaker repeats at the new level of crs.
>> There is information on the sequence of events in the TestMaker 5  
>> specification document at: http://downloads.pushtotest.com/tm5/ 
>> TM5_Specification.pdf
>
> Will print this out. Thanks.
>
>
>>>
>>> Is there any detailed explnations of this anywhere?
>>>
>> By the way, other TestMaker users would benefit from this  
>> information. It would be great to answer your questions on the  
>> users at lists.pushtotest.com email list so the answers go into our  
>> archive.
>
> Indeed. Will forward this discussion to the list.
>
> D.
>
>
> -- 
> Dominique de Waleffe
> Technical director
>
> Denali S.A., "Bridging the gap between Business and IT"
> Rue de Clairvaux 8, B-1348 Louvain-la-Neuve, Belgium
> Office: +32 10 43 99 51 Fax: +32 10 43 99 52
> http://www.denali.be
>
>
> Legal notice: this message and its attachments may contain
> confidential and/or privileged information. If you are not the
> addressee or authorized to receive this for the addressee, you must
> not use, copy, disclose or take any action based on this message or
> any information herein. If you have received this message by mistake,
> please advise the sender immediately by return e-mail and delete this
> message from your system. Thank you for your cooperation.
> <tm5-perf-res.jpg>



--
Frank Cohen, PushToTest, http://www.PushToTest.com, phone 408 374 7426
TestMaker: The open-source SOA test automation tool