[ptt-users] Pass or Fail criteria

[Kyle Bell]

To add to Geoff's post, the way we've written our test cases to work
with our testing and reporting system, goes something like this:

try:

  <test case logic>

  assert <logical test>

except:
  self.logresult(tcid,'F',failure_reason)
else:
  self.logresult(tcid,'P','Test Case Passed')


In my particular case, I took Frank's advice and implemented a mysql
database results logger in agentbase.py which is inherited by every test
case and hence able to be referenced by the test case as self.logresult
().

Here is some code which implements a results report to my results
database:

def logresults(self, tcname, status, reasontext):
  try:
      from java.sql import DriverManager
      from java.lang import Class
      Class.forName("com.mysql.jdbc.Driver").newInstance()
      con = DriverManager.getConnection
('jdbc:mysql://tool-1:3306/tcresults','testlink','testlink');
      stmt = con.createStatement()
      sql = "INSERT INTO `"+properties.resultsTable+"` ( `TCID` ,
`STATUS` , `REASON` , `STAMP` , `RUNHOST`,`ID` ) VALUES ('"+tcname+"',
'"+status+"', '"+self.reasontext+"', NOW( ) ,'"+os.environ[hostname]+"',
NULL)"
      rs = stmt.execute(sql)
      stmt.close()
      con.close()
      self.log(1, "Test Case Result successfully logged to the database.
"+time.strftime("%x %X",time.localtime()))
  except:
      self.etype, self.value, self.tb = sys.exc_info()
      self.errmsg = string.join(traceback.format_exception( self.etype,
self.value, self.tb ))
      self.log(1, "Error while writing to the results database! " + str
( self.errmsg ))


Or, rather than catch an assertion error every time, we've also
implemented our own test case exception we can raise with a reason:

class TCfailError (Exception):
    def __init__(self, reason):
        self.reason = reason
        
    def __str__(self):
        return repr(self.reason)


And rather than assert <logical test> we do:

if <failure logic>:
   raise TCfailError, "reason text"

and reason text is a meaningful description of the failure that can be
read from your results database...

Regards,

Kyle Bell
TangoNetworks


On Thu, 2006-11-23 at 10:13 +0000, Geoff Meakin wrote:
> 
> 
> You can create whatever output you like. We use testmaker with
> pass/fail criteria on each step.
> 
> Instead of self.get  its useful to do resp=self.get in your scripts,
> and then you have a response object “resp” which you can dump as a
> full string  str(resp), and use whatever check parameters you like to
> verify that your links/text is as required.
> 
>  
> 
> You could either baseline this on your first run, or have a file to
> check against- the possibilities are endless.
> 
>  
> 
> My advice is learn a little bit of python if you don’t already so you
> know how to do this- after all testmaker is based on it
> 
>  
> 
> Really crude example:
> 
>  
> 
> textcheckpoint1=’some text’
> 
>  
> 
> resp=str(self.get(‘’’http://blah/blah/blah’’’, params))
> 
> if string.find(str(resp), textcheckpoint1 )>-1:
> 
>     print “textcheckpoint1: true”
> 
> else:
> 
>     print “textcheckpoint1: false”
> 
>  
> 
> Cheers
> 
> -Geoff
> 
>  
> 
>                                    
> ______________________________________________________________________
> 
> 
> From: users-bounces at lists.pushtotest.com [mailto:users-
> bounces at lists.pushtotest.com] On Behalf Of Jacques Thomas
> Sent: 22 November 2006 19:39
> To: users at lists.pushtotest.com
> Subject: [ptt-users] Pass or Fail criteria 
> 
> 
>  
> 
>  
> 
> So far so good seems like it just takes a little tweaking with the
> advanced
> 
> settings to get things working. The million dollar question I have is
> can this
> 
> nicely written application help my organization with daily web
> monitoring?
> 
> Not Load testing per se (we already use Keynote for that). I was
> looking for
> 
> something more along the lines of link & text checks, page layout and
> 
> search test where I can compare the results of each run on a daily
> basis.
> 
> right now I’m getting this output file with:
> 
>  
> 
> <testmaker version="126">
> 
> - <message date="1164223031375">
> 
> - <![CDATA[ 
> 
> shoes11: Initialize
> 
>   ]]> 
> 
>   </message>
> 
> - <message date="1164223031375">
> 
> - <![CDATA[ 
> 
> test: setUp
> 
>   ]]> 
> 
>   </message>
> 
> - <message date="1164223031375">
> 
> - <![CDATA[ 
> 
> test: runTest
> 
>   ]]> 
> 
>   </message>
> 
> - <message date="1164223031390">
> 
> - <![CDATA[ 
> 
> Request step: 1, http://d.shoebuy.com/eluminate
> 
>   ]]> 
> 
>   </message>
> 
> - <message date="1164223037656">
> 
> - <![CDATA[ 
> 
> Request step: 2, http://www.shoebuy.com/womens-shoes.htm
> 
>   ]]> 
> 
>   </message>
> 
> - <message date="1164223038781">
> 
> - <![CDATA[ 
> 
>     Parsing images
> 
>   ]]> 
> 
>   </message>
> 
>  
> 
>  
> 
> Which looks good but to sell this to upper management - I need to be
> able to clearly define
> 
>  pass or fail criteria. If there is anybody that has used this tool in
> a similar fashion, I look forward
> 
> to hearing from you.
> 
>  
> 
>  
> 
>  
> 
>  
> 
>  
> 
>  
> 
>  
> 
>  
> 
>                                    
> ______________________________________________________________________
> 
> 
> Jacques Thomas Jr., QA Engineer
> 
>  
> 
> 
> _______________________________________________
> Users mailing list
> Users at lists.pushtotest.com
> http://lists.pushtotest.com/mailman/listinfo/users