Tests failing on baseline

Mar 17, 2010 at 7:02 AM

Hi,

I'm seeing around 50 test errors and 18 failures with (what I believe is) unmodified code.  Is anyone else seeing the failures below?  I don't see a WorkItem for them.  I'm happy to investigate if the tests are passing for others.  Many are issues with the various web services, but a few appear to be local issues.  Most of the failures are in Bio.TestAutomation.dll.  The only failures in Bio.Test..dll are in BlastWebTest (they all fail except the TestBlastXmlParser - which presumably don't use the network). 

On a related note, it's a little annoying that almost all of the time needed to run the tests is taken by a very small number of tests (most are very fast, a handful take >5 minutes or more each).  Seems like there would be value in having a "long running" test category so that quick test passes could exclude those.  I'm in the process of doing that (at least for my own use).  Would anyone be interested in reviewing the test metadata I add for consideration for check-in?  Are all the developers today running all tests before every check-in, or is there some existing subset being used in practice?

Thanks!
   Rick

---

Some of the test failure results (let me know if you'd like my full test results XML file):

Bio.Test.BlastWebTest.TestNcbiQBlast_Casing:
System.Exception : SubmitHttpRequest failed. Status: WebException: Page not available.
Bio.TestAutomation.Algorithms.Assembly.PaDeNA.PaDeNAP1TestCases.ValidatePaDeNAStep6AssembledPathForForwardAndPalContig:
  Expected: True
  But was:  False
Bio.TestAutomation.Algorithms.Assembly.PaDeNA.PaDeNAP1TestCases.ValidatePaDeNAStep6CalculateDistanceForForwardPairedReadsWithRevCompl:
  Expected: 483.0f
  But was:  486.0f
Bio.TestAutomation.Algorithms.Assembly.PaDeNA.PaDeNAP1TestCases.ValidatePaDeNAStep6FilterPairedsForContigRevComplement:
  String lengths are both 1. Strings differ at index 0.
  Expected: "6"
  But was:  "3"
  -----------^
Bio.TestAutomation.Algorithms.Assembly.PaDeNA.PaDeNAP1TestCases.ValidatePaDeNAStep6ScaffoldPathsForReverseOrientation:
System.InvalidOperationException : Sequence contains no elements
Bio.TestAutomation.PAMSAM.PamSamBvtTestCases.ValidatePamsamHierarchicalClusteringStage2WithComplete:
  Expected: True
  But was:  False
Bio.TestAutomation.Web.AzureBlast.AzureBlastBvtTestCases.FetchResultsSynchronous:
System.ArgumentNullException : Value cannot be null.
Parameter name: s
Bio.TestAutomation.Web.Blast.BlastBvtTestCases.FetchResultsAsynchronous:
System.Exception : SubmitHttpRequest failed. Status: WebException: Page not available.
Bio.TestAutomation.Web.ClustalW.ClustalWServiceHandlerBvtTestCases.ValidateCancelRequest:
System.Net.WebException : The request failed with the error message:
--
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="/error.aspx?aspxerrorpath=/CBSUAppsWebSrvc/BioHPCWebSvc.asmx">here</a>.</h2>
</body></html>

--.
Bio.TestAutomation.Web.EbiBlast.EbiBlastBvtTestCases.ValidateEBIWuBlastManadatoryParams:
  Expected: True
  But was:  False

Mar 18, 2010 at 4:11 AM

Hello Rick,

The web-service test failures are because of the respective web-service availability. Their up-time is not guaranteed. 

We have taken up an internal work item to isolate these test cases and possibly make them optional under a default run of the test suite. 

The other test case failure within the PADENA module is because of parallelization - race condition - which at times leads to different output. We have a bug #10321 tracking this issue and should get fixed with our next release. 

thanks,

Vivek

Mar 18, 2010 at 6:26 AM

Thanks Vivek,

So I'll ignore the web service test failures and PADENA failures then.  Here's another one that appears to be a race (running multiple times I see failures about 10% of the time):

Bio.TestAutomation.PAMSAM.PamSamBvtTestCases.ValidatePamsamHierarchicalClusteringStage2WithComplete:
  Expected: True
  But was:  False
at Bio.TestAutomation.PAMSAM.PamSamBvtTestCases.ValidateHierarchicalClustering(String nodeName, List`1 nodes, List`1 edges) in C:\code\mbf\MBF\Bio.TestAutomation\PAMSAM\PamSamBvtTestCases.cs:line 1901
at Bio.TestAutomation.PAMSAM.PamSamBvtTestCases.ValidateHierarchicalClusteringStage2(String nodeName, MoleculeType moleculeType, UpdateDistanceMethodsTypes hierarchicalMethodName) in C:\code\mbf\MBF\Bio.TestAutomation\PAMSAM\PamSamBvtTestCases.cs:line 1855
at Bio.TestAutomation.PAMSAM.PamSamBvtTestCases.ValidatePamsamHierarchicalClusteringStage2WithComplete() in C:\code\mbf\MBF\Bio.TestAutomation\PAMSAM\PamSamBvtTestCases.cs:line 685

 I did a quick search of the issues list and didn't find it.  Known issue?

I guess all I'm trying to figure out here is what I should be doing to validate shelvesets I create...

Thanks,

    Rick

 

 

Mar 18, 2010 at 6:49 AM

Rick,

You are correct, this issue with PAMSAM is also a consequence of race condition with parallelization. It was however, fixed and checked in on 5th March changelist #106831.

Making sure the tests under Bio.Test & Bio.Test Automation suite execute fine is a good enough validation to have the change list created / submitted for review.

Please let us know if you face any other issues.

thx, Vivek