Feedback on VCLTesttool or simply TestTool
This section should be used to collect feedback which is related to the TestTool itself. Questions about test scripts should asked in the next section.
- Description : Who to use the TestTool?
- Information : The TestTool isn't a one-click solution. It is more complex and some background information are needed to run tests or to work with TestTool. So please read the manual in the Automation project in OOo QA project : http://qa.openoffice.org/ooQAReloaded/AutomationTeamsite/ooQA-TeamAutomation.html
- Solution : For a quick start guide for TestTool read this : http://wiki.services.openoffice.org/wiki/VCLTesttool
Feedback on Test Environment
This section should be used to collect feedback for the test scripts and the test environment (running test scripts in different languages or on different platforms/operating systems etc.).
Test results differ to the references made by Sun
- Description : My tests show more errors and warnings in a test result as the results which were made by Sun engineers.
- Information : Sun used for automated testing on Master builds one dedicated system per platform. Most of the tests are optimized to run in the Sun environment and on such dedicated systems. This isn't possible in the community until now. Therefore system dependencies can be one reason for different results in specific tests. Other reasons can be that languages of OOo are tested which aren't fully supported in the test scripts. If a language isn't supported at all, the test should throw warning/errors with such a content or does not start at all. Another problem can be that builds from buildbots were used for testing. It isn't clear until now, which different build environments leads to which errors. Also other and until now unknown dependencies can be a reason for different results.
- Solution : Only in QUASTe you can get the information, if your test results differ from the references by Sun. Please write an issue in issue tracker for each of the differences. Address them to the script owner which is written in the header of the script or to Helge Delfs (HDE). If in a by Sun unsupported platform differences exists, please contact the script writer. It isn't clear, if the issue can be evaluated by the script writer or has to be debugged by the supporter of the platform. In every case the highest priority is to bring the scripts in the situation, that the results in external and Sun test environments are as equal as possible. Do not forget : In most cases the differences are bugs in OOo. Perhaps it is only triggered by TestTool and not reproducible by manual testing. But then it is also a bug in OOo. But in most cases the fix of such a bug has less priority and therefore the test scripts will be adapted instead of fixing the bug.
One-Click-Solution for running a test script
- Description : The TestTool and the Test Scripts are different parts. It is needed to have one place where I can download one package to run a test script by one-click or to have one compact application to run tests.
- Information : The TestTool itself is part of OOo install sets and will be installed by default in the program directory. Therefore the tool is always up-to-date and match to the office version/build. The test scripts aren't part of the OOo install sets and are in change for each build of OOo. When new features are added or functionality in OOo changed the test scripts has to be adapted. If this isn't done, the tests will throw errors and warning where isn't a bug. So it is important to take always the current test scripts for testing the corresponding OOO build. Only then it is possible to get a correct quality status of OOo with TestTool.
- Solution : There isn't a one-click-solution available until now and nobody worked on it. Because there isn't a clear picture of a such a solution. Currently the test scripts are be checked out automatically for each milestone. The packages can be downloaded from : http://ooopackages.good-day.net/pub/OpenOffice.org/qa/testautomation/
Feedback on Test Processes with TestTool
This section should be used to collect feedback for processes where automated testing is integrated.
Feedback on QUASTe
This section should be used to collect feedback on QUASTe. QUASTe is the tooling/front end which can/should be used for analyzing the error-reports of the automated test runs.
entries for platform are missing
- Description : In QUASTe platforms like Mac PPC, Linux 64bit and other platforms which aren't fully supported by Sun are missing.
- Information : The system information taken automatically by VCLTesttool cannot recognize all different platforms. Therefore only the groups of platforms (Mac, Linux, Solaris Sparc, Solaris Intel) are used. Because of different version numbers it is possible to differ on Windows like Win98, WinXP and Vista. But it isn't possible for the VCLTesttool to identify an 64 Linux system or a Mac PPC automatically. So it isn't a problem of QUASTe it is a problem of VCLTesttool.
- Solution : Currently there isn't a solution for this problem.