[Dev] <tally> and logging

[Frank Cohen]

Thanks, see below. -Frank


> Right now, there are five kinds of logs:
> 1. The system.out log, that has all text that is output to console.
> 2. The result log, old transactional log format for XSTest from  
> versions
> 4 and lower.
> 3. The TXLog, New transactional log from 4.4 and on. This is an XML
> format that includes tags for each scenario and each individual
> transaction. The content is based on timestamp data, duration of the
> transaction and result (true/false and any text that comes with it).
> 4. The SDLog, New System Data log that contains information for CPU,
> Memory and Networkd performance during the test.
> 5. The Summary, New log that contains the tallied information from the
> TXLog, consolidating all nodes and scenarios, plus system data.

That's right. Thanks for the summary.


> TestScenario.xml is the script to run a test suite. There is a class
> that will take that XML and upon reading the scenarios definitions,  
> will
> start to run the actual unit tests. This class is the one in a very  
> good
> position to record the TXLog, since it is starting and ending the
> transactions (in fact,the is another class that should do that,
> atomically, but under the control of the runner class).
>
> The System Monitor is a service that runs in the background, taking
> system readings and saving them into XML. This is in charge of SDLog
>
> The Tally class is the one in charge of reading both logs and
> consolidate them into Summary.
>
> Now, for legacy XSTest systems where the control program is written in
> jython (master.py), or any other case where the TestScenario is not
> used, the jython code should contain lines to save those log records.
> The TOOL contains an xml package where JaxB classes were created to  
> help
> generate or consume those logs.
>
> Now, the new requirement:
> To expand TXLog so support optional children tags for transaction,  
> that
> include details about the operation, Debuggin info or any other info
> further registered.
>
> Note that data for level 2-4 is not known byt the runner classes, but
> for the actual unit agents. That means, there should be a way for the
> unit agents, in any scrip language, to pass that information to the
> runner so it can log it into the transaction tag. This to be able to
> modify TXLog format without requiring to reimplement the agents.
>

In TM 4.4 the master.py script instantiates threads to run the  
TestCase over and over again for the duration of the test case.  
Master.py passes in parameters to the TestCase constructor. So, for  
TM 5 pass the log level from the TestScenario.xml file into the  
TestCase. The TestCase uses a base class (formerly agentbase.py but  
now a Java class like agentbase.java) to do the GET/POST commands and  
it automatically logs. The TM5 runner passes in the log level  
information which gets passed to the base class and the runner calls  
the base class to do the logging for each runtest call. That keeps  
from having to reimplement the agents and lets the user change the  
logging level in the TestScenario.xml file.

What do you think?

-Frank





> Please let me know if this makes sense.
>
> William.
>
> -----Original Message-----
> From: dev-bounces at lists.pushtotest.com
> [mailto:dev-bounces at lists.pushtotest.com]On Behalf Of Frank Cohen
> Sent: Monday, February 12, 2007 12:32 PM
> To: Load and TestMaker Developers List
> Subject: [Dev] <tally> and logging
>
>
> Hi William: I'd like to make certain that the logging in TM5 is well
> defined. Would you please confirm the following:
>
> Users create functional, scalability/performance, and reliability
> (service monitor) tests using TestMaker. All types of tests begin as
> a JUnit TestCase with setup, runtest, and teardown methods. Users
> write a TestCase in Java or Jython. (In releases later than TM 5.0
> users will also create a TestCase in JSR 223 scripting compliant
> languages, including Ruby, Groovy, PHP, etc.) The TestCases are
> wrapped in an XSTest wrapper (defined in a TestScenario.xml file,) a
> ServiceMonitor wrapper (coming in TestMaker 5.5,) or are called from
> a Jython script.
>
> 1) Users run a TestScenario.xml defined scalability test from:
>     a) The com.pushtotest.testmaker.gui.guiTestMaker GUI interface
>     b) The command line using the -commandline switch in the
> TestMaker.sh and .bat start-up scripts.
>
> When the TestScenario runs there are three things to log:
> i) system.out messages that happen while the test runs, including
> stack traces and exceptions
>
> ii) Transactional data as the test runs, including each transaction
> entry and the system monitor entries. The amount of information in
> the transaction entries depends on the log level setting from the
> TestScenario.xml file and is defined as:
>
> Log level 0 = No logging
> 1 = Information about major operations, including time to perform
> transaction (testusecase)
> 2 = Details about each operation, for instance, GET /examples, POST /
> examples
> 3 = Debugging information, including the request/response message
> content
> 4 = Ludicrous level, meaning everything and the kitchen sink
> They are additive, so logging at level 3 gives debugging, details,
> and information.
>
> iii) Results analysis, including the comma-separated-value tallied
> TPS values, aka the Scalability Index
>
> 2) Users run a Jython script from the TestMaker script editor (J
> editor, org.armedbear.j.Editor) using
> com.pushtotest.testmaker.gui.ScriptEditor.TMFrame.
>
> Jython scripts log their output to the Output panel in the TestMaker
> script editor window as they do in TM 4.x
>
> Please send me your comments or confirm that this is your  
> understanding.
>
> Thanks.
>
> -Frank
>
> --
> Frank Cohen, PushToTest, http://www.PushToTest.com, phone 408 374 7426
> TestMaker: The open-source SOA test automation tool
>
>
>
> _______________________________________________
> Dev mailing list
> Dev at lists.pushtotest.com
> http://lists.pushtotest.com/mailman/listinfo/dev
> _______________________________________________
> Dev mailing list
> Dev at lists.pushtotest.com
> http://lists.pushtotest.com/mailman/listinfo/dev
>



--
Frank Cohen, PushToTest, http://www.PushToTest.com, phone 408 374 7426
TestMaker: The open-source SOA test automation tool