[ptt-users] Pass or Fail criteria

[Frank Cohen]

Awesome Kyle. By the way, TestMaker 5 (now in development) will offer  
a new logger that would make it possible for you to create this kind  
of logger as something that everyone would benefit from. Thanks for  
your contribution. -Frank


On Nov 27, 2006, at 7:02 AM, Kyle Bell wrote:

> To add to Geoff's post, the way we've written our test cases to work
> with our testing and reporting system, goes something like this:
>
> try:
>
>   <test case logic>
>
>   assert <logical test>
>
> except:
>   self.logresult(tcid,'F',failure_reason)
> else:
>   self.logresult(tcid,'P','Test Case Passed')
>
>
> In my particular case, I took Frank's advice and implemented a mysql
> database results logger in agentbase.py which is inherited by every  
> test
> case and hence able to be referenced by the test case as  
> self.logresult
> ().
>
> Here is some code which implements a results report to my results
> database:
>
> def logresults(self, tcname, status, reasontext):
>   try:
>       from java.sql import DriverManager
>       from java.lang import Class
>       Class.forName("com.mysql.jdbc.Driver").newInstance()
>       con = DriverManager.getConnection
> ('jdbc:mysql://tool-1:3306/tcresults','testlink','testlink');
>       stmt = con.createStatement()
>       sql = "INSERT INTO `"+properties.resultsTable+"` ( `TCID` ,
> `STATUS` , `REASON` , `STAMP` , `RUNHOST`,`ID` ) VALUES ('"+tcname+"',
> '"+status+"', '"+self.reasontext+"', NOW( ) ,'"+os.environ[hostname] 
> +"',
> NULL)"
>       rs = stmt.execute(sql)
>       stmt.close()
>       con.close()
>       self.log(1, "Test Case Result successfully logged to the  
> database.
> "+time.strftime("%x %X",time.localtime()))
>   except:
>       self.etype, self.value, self.tb = sys.exc_info()
>       self.errmsg = string.join(traceback.format_exception 
> ( self.etype,
> self.value, self.tb ))
>       self.log(1, "Error while writing to the results database! " +  
> str
> ( self.errmsg ))
>
>
> Or, rather than catch an assertion error every time, we've also
> implemented our own test case exception we can raise with a reason:
>
> class TCfailError (Exception):
>     def __init__(self, reason):
>         self.reason = reason
>
>     def __str__(self):
>         return repr(self.reason)
>
>
> And rather than assert <logical test> we do:
>
> if <failure logic>:
>    raise TCfailError, "reason text"
>
> and reason text is a meaningful description of the failure that can be
> read from your results database...
>
> Regards,
>
> Kyle Bell
> TangoNetworks
>
>
> On Thu, 2006-11-23 at 10:13 +0000, Geoff Meakin wrote:
>>
>>
>> You can create whatever output you like. We use testmaker with
>> pass/fail criteria on each step.
>>
>> Instead of self.get  its useful to do resp=self.get in your scripts,
>> and then you have a response object “resp” which you can dump as a
>> full string  str(resp), and use whatever check parameters you like to
>> verify that your links/text is as required.
>>
>>
>>
>> You could either baseline this on your first run, or have a file to
>> check against- the possibilities are endless.
>>
>>
>>
>> My advice is learn a little bit of python if you don’t already so you
>> know how to do this- after all testmaker is based on it
>>
>>
>>
>> Really crude example:
>>
>>
>>
>> textcheckpoint1=’some text’
>>
>>
>>
>> resp=str(self.get(‘’’http://blah/blah/blah’’’, params))
>>
>> if string.find(str(resp), textcheckpoint1 )>-1:
>>
>>     print “textcheckpoint1: true”
>>
>> else:
>>
>>     print “textcheckpoint1: false”
>>
>>
>>
>> Cheers
>>
>> -Geoff
>>
>>
>>
>>
>> _____________________________________________________________________ 
>> _
>>
>>
>> From: users-bounces at lists.pushtotest.com [mailto:users-
>> bounces at lists.pushtotest.com] On Behalf Of Jacques Thomas
>> Sent: 22 November 2006 19:39
>> To: users at lists.pushtotest.com
>> Subject: [ptt-users] Pass or Fail criteria
>>
>>
>>
>>
>>
>>
>> So far so good seems like it just takes a little tweaking with the
>> advanced
>>
>> settings to get things working. The million dollar question I have is
>> can this
>>
>> nicely written application help my organization with daily web
>> monitoring?
>>
>> Not Load testing per se (we already use Keynote for that). I was
>> looking for
>>
>> something more along the lines of link & text checks, page layout and
>>
>> search test where I can compare the results of each run on a daily
>> basis.
>>
>> right now I’m getting this output file with:
>>
>>
>>
>> <testmaker version="126">
>>
>> - <message date="1164223031375">
>>
>> - <![CDATA[
>>
>> shoes11: Initialize
>>
>>   ]]>
>>
>>   </message>
>>
>> - <message date="1164223031375">
>>
>> - <![CDATA[
>>
>> test: setUp
>>
>>   ]]>
>>
>>   </message>
>>
>> - <message date="1164223031375">
>>
>> - <![CDATA[
>>
>> test: runTest
>>
>>   ]]>
>>
>>   </message>
>>
>> - <message date="1164223031390">
>>
>> - <![CDATA[
>>
>> Request step: 1, http://d.shoebuy.com/eluminate
>>
>>   ]]>
>>
>>   </message>
>>
>> - <message date="1164223037656">
>>
>> - <![CDATA[
>>
>> Request step: 2, http://www.shoebuy.com/womens-shoes.htm
>>
>>   ]]>
>>
>>   </message>
>>
>> - <message date="1164223038781">
>>
>> - <![CDATA[
>>
>>     Parsing images
>>
>>   ]]>
>>
>>   </message>
>>
>>
>>
>>
>>
>> Which looks good but to sell this to upper management - I need to be
>> able to clearly define
>>
>>  pass or fail criteria. If there is anybody that has used this  
>> tool in
>> a similar fashion, I look forward
>>
>> to hearing from you.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _____________________________________________________________________ 
>> _
>>
>>
>> Jacques Thomas Jr., QA Engineer
>>
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.pushtotest.com
>> http://lists.pushtotest.com/mailman/listinfo/users
>
> _______________________________________________
> Users mailing list
> Users at lists.pushtotest.com
> http://lists.pushtotest.com/mailman/listinfo/users
>