Here the feedback on VCL-TestTool, on the test environment for VCL-TestTool, on the test processes with VCL-TestTool and tooling which will be used for analyze test results (e.g. QUASTe) should be used.
Feedback on VCL-TestTool or simply TestTool
This section should be used to collect feedback which is related to the TestTool itself. Questions about test scripts should asked in the next section.
Feedback on Test Environment
This section should be used to collect feedback for the test scripts and the test environment (running test scripts in different languages or on different platforms/operating systems etc.).
Test results differ to the references made by Sun
- Description : My tests show more errors and warnings in a test result as the results which were made by Sun engineers.
- Information : Sun used for automated testing on Master builds one dedicated system per platform. Most of the tests are optimized to run in the Sun environment and on such dedicated systems. This isn't possible in the community until now. Therefore system dependencies can be one reason for different results in specific tests. Other reasons can be that languages of OOo are tested which aren't fully supported in the test scripts. If a language isn't supported at all, the test should throw warning/errors with such a content or does not start at all. Another problem can be that builds from buildbots were used for testing. It isn't clear until now, which different build environments leads to which errors. Also other and until now unknown dependencies can be a reason for different results.
- Solution : Only in QUASTe you can get the information, if your test results differ from the references by Sun. Please write an issue in issue tracker for each of the differences. Address them to the script owner which is written in the header of the script or to Helge Delfs (HDE). If in a by Sun unsupported platform differences exists, please contact the script writer. It isn't clear, if the issue can be evaluated by the script writer or has to be debugged by the supporter of the platform. In every case the highest priority is to bring the scripts in the situation, that the results in external and Sun test environments are as equal as possible. Do not forget : In most cases the differences are bugs in OOo. Perhaps it is only triggered by TestTool and not reproducible by manual testing. But then it is also a bug in OOo. But in most cases the fix of such a bug has less priority and therefore the test scripts will be adapted instead of fixing the bug.
Feedback on Test Processes with TestTool
This section should be used to collect feedback for processes where automated testing is integrated.
Feedback on QUASTe
This section should be used to collect feedback on QUASTe. QUASTe is the tooling/front end which can/should be used for analyzing the error-reports of the automated test runs.
entries for platform are missing
- Description : In QUASTe platforms like Mac PPC, Linux 64bit and other platforms which aren't fully supported by Sun are missing.
- Information : Currently only platforms are in QUASTe which are fully supported by Sun. It isn't a problem to add other platform in this tooling. But the consequence is, that references (results of a test run on a master build in one language - e.g. English) have to be created for each milestone. Only then it makes sense to make comparison with other platforms, languages and CWSs based on this platform.
- Solution : If somebody takes the responsibility for new platform in QUASTe the new platform will/can be integrated. Write an issue to Helge Delfs (HDE) with the naming of the platform.