[Dev] <tally> and logging

[Frank Cohen]

Hi William: I'd like to make certain that the logging in TM5 is well  
defined. Would you please confirm the following:

Users create functional, scalability/performance, and reliability  
(service monitor) tests using TestMaker. All types of tests begin as  
a JUnit TestCase with setup, runtest, and teardown methods. Users  
write a TestCase in Java or Jython. (In releases later than TM 5.0  
users will also create a TestCase in JSR 223 scripting compliant  
languages, including Ruby, Groovy, PHP, etc.) The TestCases are  
wrapped in an XSTest wrapper (defined in a TestScenario.xml file,) a  
ServiceMonitor wrapper (coming in TestMaker 5.5,) or are called from  
a Jython script.

1) Users run a TestScenario.xml defined scalability test from:
    a) The com.pushtotest.testmaker.gui.guiTestMaker GUI interface
    b) The command line using the -commandline switch in the  
TestMaker.sh and .bat start-up scripts.

When the TestScenario runs there are three things to log:
i) system.out messages that happen while the test runs, including  
stack traces and exceptions

ii) Transactional data as the test runs, including each transaction  
entry and the system monitor entries. The amount of information in  
the transaction entries depends on the log level setting from the  
TestScenario.xml file and is defined as:

Log level 0 = No logging
1 = Information about major operations, including time to perform  
transaction (testusecase)
2 = Details about each operation, for instance, GET /examples, POST / 
examples
3 = Debugging information, including the request/response message  
content
4 = Ludicrous level, meaning everything and the kitchen sink
They are additive, so logging at level 3 gives debugging, details,  
and information.

iii) Results analysis, including the comma-separated-value tallied  
TPS values, aka the Scalability Index

2) Users run a Jython script from the TestMaker script editor (J  
editor, org.armedbear.j.Editor) using  
com.pushtotest.testmaker.gui.ScriptEditor.TMFrame.

Jython scripts log their output to the Output panel in the TestMaker  
script editor window as they do in TM 4.x

Please send me your comments or confirm that this is your understanding.

Thanks.

-Frank

--
Frank Cohen, PushToTest, http://www.PushToTest.com, phone 408 374 7426
TestMaker: The open-source SOA test automation tool