Lessons learnt

In what follows, general comments (based on the assignment) on the usability and usefulness of the reverse engineering tools are presented. We will not cover all the comments on the specific tools in detail but discuss general problems encountered during the tool analysis.

Installation

The only commercial tool in this study (Imagix4D) had a clear and easy-to-use installation program. Installing the other tools was quite problematic. To some extent, installation problems with the academic tools are understandable. The research focus is typically on 'more interesting research problems', i.e., in developing the actual reverse engineering ideas behind the tools.

Documentation

Imagix4D contains a thorough tutorial on the functionalities and features of the tool. According to the students' experiences, going through the tutorial took two hours. The documentation for the academic tools was poor. Rigi contains few demos that help the users to get an idea of the features and activities supported by the tool. The documentation was mostly needed for the rcl script library and using scripts, on which the user heavily needs to rely on in order to be able to manipulate the views (and understand the views). However, Rigi does not provide such documentation (only few of the scripts are commented). The students found the user manual for Rigi (available on the web) quite useful. The main problem was the undocumented rcl library. The documentations for both both PBS and TkSee were out of date.

As in the case of installation problems, it is understandable that the academic tools do not have covering and up-to-date documentation. Making the tool more robust/stable and to write a proper tutorial and documentation are less intrigueing from the research point of view. Hence, the amount of time and effort put to that is typically low.

Learning the tool and analyzing the views

The learning curve seemed to be quite high for all of the tools, especially for Rigi, Bookshelf, and TkSee.  The development of any software engineering tool relies on user feedback. The functionality and the usability of the features need to be tested for evaluation purposes. If the tool is difficult to learn and it lacks proper documentation, it is more difficult to get users outside the research group and thus impartial feedback.

The tools provide primitive methods for manipulating the views. However, they give less support to the user for understanding the views. Typically, each reverse engineering tool has its own, unique graphical representation for visualizing the extracted information. The (naive) user is thus supposed to comprehend an unknown software system by analyzing an unfamiliar graphical representation describing this system. In such cases, it would be important that the reverse engineering tools give support for understanding what does the view represent, what does it not represent, how the view should be understood, what are the questions to which the user can find answers by studying the view, how can she find these answers, etc.

The Unified Modeling Language (UML), for example, contains different models that represent different aspects of the software on different levels of abstractions. Seeing a model, the user knows what does it represent and how to use it. In traditional reverse engineering tools, all the information is merged into a single view. The user has to manipulate the view to find the different aspects and abstractions. Furthermore, she does not know what is the correct form for expressing the aspects and the abstractions. More support for such activities in these tools would be desirable. The students wished for some kind of wizard-based guidance for specific goal-driven reverse engineering tasks. The scripts for manipulating the views, provided by the tools, were much tool low-level for giving such a support.

In this assignment, the students noticed that the (naive) user might have major difficulties in interpreting the initial view (formed on the basis of information extracted) and figure out what to do with it. The tools seemed to have nice ideas behind them but unfortunately they were quite well hidden from the user.