© 2004-2007 Reinhold Füreder (reinholdfuereder_AT_gmx_DOT_at)

• Introduction & documentation
• Sources
• Building
• Remaining internal dependencies
• Automatic skeleton code generation of test cases and test drivers
• Example usages for swWxGuiTesting
• Further work
• Updates

While I developed swWxGuiTesting as part of my MPhil study at the University of Exeter, UK, in collaboration with Simpleware Ltd -- a spin-off company from this university --, the company director Dr Philippe G Young is embracing the idea of contributing it to the open source community. For the sake of simplicity and striving/hoping for an integration into the wxWidgets framework we have decided to put it under wxWindows licence (thanks to Julian Smart's explanations and advise).

As discussed in the wxWidgets developers mailing list under subject "Contributing an automated wxWidgets GUI testing framework to the open source community" this site hosts the offered automated wxWidgets GUI testing framework -- named swWxGuiTesting. However, ideally this site will be only the temporary home of the project until it finds a better place that is hopefully within wxWidgets itself.

A first overview about swWxGuiTesting is given in the mailing list postings:

Reinhold Füreder, 30 Jan 2006    <<< recommended introductory starting point

Julian Smart, 31 Jan 2006

Reinhold Füreder, 31 Jan 2006

Julian Smart, 1 Feb 2006

Ron, 1 Feb 2006

Reinhold Füreder, 1 Feb 2006

Vadim Zeitlin, 5 Feb 2006

Reinhold Füreder, 5 Feb 2006

etc...

Much more details on the motivation and especially concepts of swWxGuiTesting are found in the standalone MPhil thesis chapter at MPhilThesisChapterSwWxGuiTesting.pdf (1.3MB). While this chapter also lists the required steps to make any wxWidgets based GUI application testable by this pragmatic framework, it admittedly does not replace a more developer-friendly tutorial...

The sources (including building instructions and via Doxygen generatable API documentation) are zipped into:

• swWxGuiTesting-1.5.zip release supporting latest stable release wxWidgets 2.8.6 (including Unicode) under Mac OS X (Carbon), Linux (GTK2) and MSW.
• swWxGuiTesting-1.4.zip release supporting latest stable release wxWidgets 2.8.0 under Linux (GTK2) and MSW.
• swWxGuiTesting-1.3.3.zip minor release with some minor fixes/improvements.
• swWxGuiTesting-1.3.2.zip minor release demonstrating easier approach for handling user-defined modal dialogs.
• swWxGuiTesting-1.3.1.zip minor release supporting building with release of CMake: version 2.4.3.
• swWxGuiTesting-1.3.zip release supporting testing release wxWidgets 2.7.0 under Linux (GTK2) and MSW.
• swWxGuiTesting-1.2.2.zip minor release supporting VTK 5; plus fix building for C&R of VTK render window interactions under Linux.
• swWxGuiTesting-1.2.zip release supporting stable release wxWidgets 2.6.3 under Linux (GTK2) and MSW.
• swWxGuiTesting-1.1.zip intermediate release supporting latest stable release wxWidgets 2.6.3 with patch 2 under MSW (not for any other platform).
• swWxGuiTesting-1.0.zip initial release supporting old wxWidgets version, see below for platform and version details.

For version details please see Updates.

Instructions for building are found in Building.txt, that is also part of the source archive.

The building was tested under Microsoft Windows (2000, XP) for Microsoft Visual C++ 6, Mac OS X 10.4 with GCC 4 (for versions >= 1.5 of swWxGuiTesting) and Ubuntu Dapper Linux with GCC 4 (for versions >= 1.2 of swWxGuiTesting; version 1.0 was tested with an ancient Red Hat 8 Linux installation with GCC 3.2 -- choosing this version is reasoned in the aforementioned mailing list postings).

The build process is based on the open source tool CMake (a release version >= 2.4.5 is required for wxWidgets 2.8.x), after the third party dependencies are built independently according to their respective standard build rules (including e.g. "make install" for the sake of simplicity, but not mandatory if additional CMake option adaptations are not frightening; non-standard building settings are outlined below, and potentially specified in some detail in the doxygen documentation).

• Starting from version 1.5 the stable release version 2.8.6 of wxWidgets (now also including Unicode builds) is supported and tested under Mac OS X (Carbon), Linux (GTK2) and MSW: Many thanks to John Ralls!
• Starting from version 1.4 the stable release version 2.8.0 of wxWidgets is supported and tested under MSW and Linux (GTK2). The build procedure is now based on standard CMake wxWidgets utilisation modules, which in turn requires CMake >= 2.4.5 in order to support wxWidgets 2.8.x. This step makes building easier because (a) no non-standard CMake modules need to be copied initially, (b) there is no need to edit CMakeLists.txt to specify wxWidgets version, and (c) configuration within CMake is easier: e.g. when wxWidgets was installed with the installer under MSW, or by means of "make install" under Linux everything is found automatically; alternatively just unzipping under MSW requires only the manual setting of wxWidgets_LIB_DIR (e.g. D:\CVS\wxWidgets-2.8.0-rc3\lib\vc_lib) and wxWidgets_ROOT_DIR (e.g. D:\CVS\wxWidgets-2.8.0-rc3).
• Starting from version 1.3 the testing release version 2.7.0 of wxWidgets is supported and tested under MSW and Linux (GTK2). Also the dependency of Xerces-C++ was replaced in favour of the expat library which already comes with wxWidgets. Choosing the wxWidgets version required:
  1. In all CMakeLists.txt replace the following:
     
     SET(WXW_FILE_VERSION "27")
SET(WXW_VERSION "2.7")

     with for instance:
     
     SET(WXW_FILE_VERSION "26")
SET(WXW_VERSION "2.6")

  2. Only under Microsoft Windows: When running CMake select the right wxWidgets root directory.
• Starting from version 1.2 the stable release version 2.6.3 of wxWidgets is supported and tested under MSW and Linux (GTK2). This change required a few changes in the swWxGuiTesting sources and also in the utilisation of the CMake based build system. On the other hand, XRC is already included in wxWidgets; cf. initial release:
  In the initial release (version 1.0) the supported wxWidgets was: 2.4.2 < used version from August 2003 < 2.5 (at that time the additionally required XRC library was not included in wxWidgets).
  (Static build; of course, dynamic should work as well, but may or may not require little adaptations in the build instructions. The used configuration of a recent wxWidgets version itself: setup.h for MSW; configure settings are straightforward for Linux). Also the wxrc executable is required during the building.
• CppUnit: versions 1.8 - 1.12.0 were tested

There is an optional dependency on VTK (tested with versions 4.x - 5.0 built with CMake) to allow Capture & Replay (C&R) of VTK render window interactions. However, the generated test cases for such interactions do not fulfil the ultimate goal of being easily maintainable.

Building notes/hints for Linux (Ubuntu Dapper):

• Configuration for wxWidgets build: "configure --with-gtk --with-opengl --disable-compat26" (for unicode add: "--enable-unicode"; for debugging add: "--enable-debug --enable-debug_gdb")
• Install lib and dev packages for CppUnit (and previously also for Xerces-C++)
• Install X11 header packages: xlibs-dev
• Install mesa-glx (incl. -dev) and/or nvidia-glx (incl. -dev) packages
• Potentially adapt LD_CONFIG_PATH to include /usr/local/lib to allow using/running the standard build of wxrc tool

Dependencies from swWxGuiTesting to SimplewareFramework (whose subcomponent it originally was) were removed to some extent by:

1. replacing wxWidgets based extensions again with wxWidgets base classes
2. removing any non-core classes, and related methods (e.g. ViewFrame + FrameFactory's CreateViewFrame() method)

Nevertheless, there are still dependencies that demand that the user of the testing framework builds her system under test (SUT) with some constraints (cf. so-called FrameFactory). In case swWxGuiTesting will be integrated into wxWidgets, one could get rid of these dependencies if wxWidgets takes over their strictly required responsibilities.

As outlined in the MPhil thesis chapter test case skeletons and test drivers can be generated by means of the free tool wxCRP. It should be sufficient to point the installed wxCRP application to the decompressed swWxGuiTesting wxCRP repository. A movie (4.4MB) with crap quality (sorry) shows the process and result of utilising wxCRP.

• Examples (except complex ScanCAD ones) can be found in "external" UsingSwWxGuiTesting project. (Note: Only test suites registered with the name "WxGuiTest" are part of a swWxGuiTesting test execution.)
• "Movies" initially recorded with the wonderful vncrec tool (http://www.sodan.org/~penny/vncrec/) are available for easy demonstration purposes. (Sorry for the partly disappointing movie quality.)
• Note that the "Simple Test" and "Simple C&R Test" are in fact testing the GUI testing framework, and the GUI is only for the sake of simplicity initialised in the setup() method instead of in the application under test (AUT).

1. Simple Test (UsingSwWxGuiTesting/Cxx/SimpleTest.*):
   • Test case source file: SimpleTest.cpp
   • Movie: SimpleTest.avi (2.0MB), step-by-step protocol
   • Based on original test of Capture & Replay (C&R) in Testing/Cxx/swWxGuiTesting/CppGuiTest/swCRCaptureTest.cpp
   • Artificially sleep for 1 second after each event simulation: wxThread::Sleep(1000) at end of WxGuiTestHelper::FlushEventQueue() method
   • Using temporary interactivity (breakpoint afterwards is not reached) to demonstrate control flow splitting and easy visual inspection
   • Using breakpoint to demonstrate control flow splitting and easy debuggability
2. Simple Capture & Replay Test (UsingSwWxGuiTesting/Cxx/SimpleCRTest.*):
   • Test case source file: SimpleCRTest.cpp
   • Movie: SimpleCRTest.avi (7.9MB), step-by-step protocol
   • Bootstrap GUI test case built solely upon a GUI initialisation in setup() and the CAPTURE macro in the test case:
     • during bootstrapping the test case is written directly into backup file (*.cr*)
     • each key stroke in a text control results in an event => collect them
   • Fix pragmatic code generation of capture phase:
     • adding include directories
     • removing duplicate container lookups and variable declarations
     • adding assertions
   • Replay captured test case like a normal test case
3. Complex Test - ScanCAD
   • Test case source file: sccCADFileImportGuiTest.cpp
   • Movie: ComplexTest_ScanCAD.avi (1.4MB, step-by-step protocol) for the CAD file import GUI test which is based on a modal dialog, and the first three minutes of executing the ScanCAD GUI test suite is shown in ComplexTest_ScanCAD_TestSuite.avi (7.8MB)
   • Deal with modal dialogs (message boxes and pop-up menus)
   • Whole use case (= integration/application test)
   • Disabling temporary interactivity for real automation

Most importantly see the conclusion section of the MPhil thesis chapter and the mailing list discussions so far. This project is more than a proof of concept for easy and maintainable cross-platform automated wxWidgets GUI tests, but by no means a product.

Must:

• Tutorial/cookbook that is more developer-friendly than the thesis chapter.

Fix:

• wxWidgets warnings in one of the examples (test case) with respect to resources.

Improve:

• C&R: looked up containers => cache them somehow to prevent duplicate lookups and variable declarations.
• C&R: do not emit code directly + fill map of include directories during emitting => all include directories (without duplicates) can be actually emitted first, before the rest of the code follows.

Comments:

• Dialog creation under M$ Windows seems to require FlushEventQueue() -- see swCRWindowHierarchyHandlerTest.cpp:

     #ifndef __WXGTK__
         WxGuiTestHelper::FlushEventQueue ();
     #endif
     

21.11.2007 - Release 1.5 supporting wxWidgets 2.8.6 (including Unicode) under Mac OS X, Linux GTK and MSW: Special thanks to John Ralls for his great work and feedback!!!

• Supporting Mac OS X required very moderate adaptations and is a significant corroboration of the initial hypothesis that swWxGuiTesting's concept allows easy and maintainble cross-platform automated wxWidgets GUI testing based on a single test case source
• Unicode wxWidgets builds are now supported too

18.12.2006 - Release 1.4 supporting wxWidgets 2.8.0 under MSW and Linux GTK:

• Switch to official CMake wxWidgets utilisation modules -- wxWidgets 2.8.0 requires at least CMake 2.4.5
• wxGuiTestHelper::CreateDefaultDialog() -- modal flag at dialog creation time is no more needed in order to have testable dialogs

Known bug with wxWidgets 2.8.0 under Linux GTK2:

• If C&R capturing is active, spin control scrolling seemingly leads to no less than 6 events, but spin control values do not change...

26.11.2006 - Minor release 1.3.3 with some minor fixes/improvements:

• Minor refactoring of emitting of pending events, plus also apply it before adding new comments.
• Improve C&R control: enable/disalbe start and stop button accordingly.
• Modal dialog timer approach - don't forget to call TransferDataFromWindow etc (cf. OnOk, OnCancel methods): call appropriate standard dialog event hanlder directly depending on the desired return code.
• Don't allow duplicate calling of event handler (cf. CppGuiTestEventSimulation): seemingly this cannot be prevented for tree item selection, or is the tree item even simulation buggy (MSW, Linux + GTK2)?

18.10.2006 - Minor release 1.3.2 demonstrating another approach (as in the MPhil thesis chapter) to handle user-defined modal dialogs (cf. CppGuiTestModalDialog test case):

Based on Julian Smart's hint a one-shot timer fires and allows the subsequent execution of some dialog interaction code before actually dismissing the modal dialog with the specified return code. Although this does not work for standard predefined modal dialogs in a platform independent way (e.g. wxDirDialog under MSW), it is an improvement to the previous solution by not strictly requiring to use an event-driven modal dialog usage instead of the easier function-like usage (cf. Section 1.5.4.1 in the MPhil Thesis chapter).

13.10.2006 - Minor release 1.3.1 supporting building with latest release of CMake: Version 2.4.3 required a few tiny but critial modifications in the CMakeLists.txt.

05.10.2006 - Release version 1.3 supporting wxWidgets 2.7 plus removed Xerces-C++ dependency in favour of the expat library which already comes with wxWidgets. Plus minor improvements:

• Make "real" test cases in CRCppEmitterTest.
• C&R: fix line break in emitting of non-integer numbers in code (e.g. was: "0.5" => "0.\n5").

14.08.2006 - Minor release version 1.2.2 supporting VTK 5; plus fix building for C&R of VTK render window interactions under Linux.

07.07.2006 - Release version 1.2 supporting latest stable release wxWidgets 2.6.3 also under Linux (GTK2).

27.06.2006 - Move website to new host, because the previous free webspace provider has shut down the page without any notice some time ago. Thanks to my friend Werner Fischer for hosting it.

09.06.2006 - Recreate VNC movies in the shape of MPEG videos to allow playing them on all systems. Thanks to Carl Godkin for reporting the problem.

03.05.2006 - Release version 1.1 supporting latest stable release wxWidgets 2.6.3 with patch 2 under MSW. Initiated by Carl Godkin.

14.04.2006 - Uploaded recompiled MPhil thesis chapter: now recognizable full sequence diagrams in Appendix A, Section 1.5 (page 95).

17.02.2006 - Initial "release".