Keywords
Experimentation, verification, validation, model building,
proof-of-concept
INTRODUCTION AND THEORETICAL BACKGROUND
The need for experimentation in software engineering
is well documented in the literature, and many methods for experimentation
are available. However, in many cases, mostly due to time constraints,
too little actual experimentation is being done to provide proof-of-concept
before, during and after development of software artifacts [1, 7, 11, 12].
Formal experiments are commonly used in other disciplines to verify, validate
and refine concepts, but are not as prevalent in the engineering of software.
This paper proposes to investigate the role of experimentation in demonstrating proof-of-concept for artifacts to be developed in such projects as the Intelligent Systems (IS) project of NASA [6]. For a description of this project, see http://ic.arc.nasa.gov/ic/nra/index.html. Since the IS and similar projects tend to delve into the unknown, there will likely be a scarcity of previous work to draw upon for experience. It will be important to establish a baseline and conduct experiments to verify, validate and refine the concepts that are developed to support these cutting edge projects.
PREVIOUS RESEARCH IN THE AREA
Much work has been done in software engineering model
building [1]. For example, tools, processes, technologies and specification
languages have been built. However, less work has been done in validating
the models [1, 12]. The models have been built, but too few experiments
have been done to verify that the models are effective, or that new models
are better than other existing models. Some models may be perfect
in theory, but little thought has been given to whether the models can
successfully be used in practice, under what conditions the models will
be used, and what kind of training is necessary for the models to be used
to advantage.
Marvin Zelkowitz and Delores Wallace identified 12 methods of validation for software and software models in the paper "Experimental Models for Validating Computer Technology" [12]. The 12 methods identified were project monitoring, case study, assertion, field study, literature search, legacy data, lessons learned, static analysis, replicated experiment, synthetic, dynamic analysis, and simulation. They then examined over 600 software engineering publications from the 3 years 1985, 1990, and 1995, and classified the publications as to which of the 12 methods were used to validate the research presented. The study found that about one third of the papers had no experimental validation at all, although there was some improvement in that area between 1985 and 1995. Another one third of the papers used assertion to validate their research. However, assertion can be very biased since the researcher is often also the developer. In those documents which included lessons learned, the same comments about what should have been done were repeated in successive documents. The conclusion presented in the paper was that authors often fail to clearly state the goals or value expected and how they will validate their hypotheses. In addition, too many papers have no experimental validation or use informal validation that may be biased.
The Software Engineering Laboratory (SEL) has been in existence for over 20 years [2], and has done many experiments and studies on improving software and software processes. Although there are various other centers for software research such as IBM’s Cleanroom Software Technology Center [9] and the Software Engineering Institute of Carnegie Melon University (SEI), the literature shows that much more needs to be done [1, 7, 11]. A recent survey shows that although industry professionals believe data collection and analysis are important to validate new technology, they seldom collect the data [11]. The professionals rated vendor opinion as least important in the survey, yet admitted that vendor opinion often influences the decisions being made.
One good example of a formal experiment to validate computer technology is discussed in the paper "A Software Engineering Experiment in Software Component Generation" [4]. This paper includes the hypotheses, design of the experiment, methods for data collection and analysis, and detailed formal results.
The book “Experimentation in Software Engineering: An Introduction” outlines a process for conducting experiments in software engineering [10]. However, the authors do not discuss the problem of determining the situations in which an experiment will be useful and cost effective. We intend to go a step further by identifying concepts to be proved, determining if an experiment is feasible, and if it is feasible carrying out the experiment and documenting the results.
As more and more complex computer systems continue to be introduced into life-critical situations [3], better methods of verification and validation are needed to make sure these systems are safe. The proposed research will focus on formal experimentation at various stages of development to prove or disprove concepts before they are used extensively in the project.
GOALS OF THE RESEARCH
The research will involve identifying problem areas and
new concepts related to a large software development project, and then
conducting experiments as proof-of-concept. Methods for identifying
candidate experiments, carrying them out, and evaluating the results will
be researched. This should help to reduce the risk involved in high-risk
development projects by supplying methods for more efficient experimentation
with new concepts. In addition, models will be built and verified
based on the experimentation results.
Large distributed applications with significant architectural complexity and interactive human-computer interfaces will be targeted. These applications have a strong potential for improvement and for proof of various concepts through experimentation. One possible application to be considered for use in this research is the YieldTracker project [5], which will use data collected by satellite to provide farmers with information about crop yields in different local areas. First a method for determining the need for experimentation would be developed and documented. Then this method would be used to find the high-risk areas of the target project, and determine where experiments would be cost effective to reduce the risks. The experiments would then be carried out and documented.
The research will be based on the use of the Experience Factory paradigm as described in “The Software Engineering Laboratory – an Operational Software Experience Factory” [2]. With this paradigm, the experimentation is done as a separate activity from the development project by a different group of workers. This reduces the negative effect that the time used for experimentation might possibly have on the time-to-market of a project. Both the role of experimentation and the methods to facilitate more efficient experiments will be researched. For example, data collection methods and communication between the experiment team and the development team are major topics for consideration. The nature of the projects may require methods different from those currently documented in the literature [11, 12]. Currently available methods will be researched and used when applicable, and will also be expanded upon to form new methods specific to the new challenges.
The need for experimentation, even if the cost is high, needs to be emphasized and promoted. This research will promote experimentation in software engineering by presenting examples of successful experiments which are cost effective, and presenting a model for conducting such experiments.
CURRENT STATUS
The need for more formal experimentation in software
engineering has been established. The next step is to find one or
more projects for case studies to be done. The purpose of the case
studies will be to determine better methods of verifying and validating
new concepts.
INTERIM CONCLUSIONS
The conclusion so far is the fact that more formal experimentation
needs to be done to validate and verify software and software development
methods, and that better methods of experimentation are needed to encourage
its use in industry. In addition, better methods are needed to promote
widespread application of experimentation results. Instead of merely
stating "lessons learned" we need to study the situations where lessons
have been learned and make appropriate changes to avoid repeating the same
mistakes in the future.
OPEN ISSUES
How do we experiment without disrupting the production
environment? How do we convince developers of the importance of collecting
data, experimenting and using the results of the experiments? How
do we best insure that our products give predictable results and are safe?
CURRENT STAGE IN THE PROGRAM OF STUDY
All course work for the Ph.D. has been done. The
qualifying exam has been successfully completed. Admission to candidacy
for the Doctorate of Computer Science in the College of Engineering at
Texas Tech University has been granted, and the dissertation has been started.
The current research is in the beginning stages. A literature search has been done to determine the need for experimentation in software engineering, and has shown a strong need. In addition, I have met with members of the YieldTracker project [5] about working on a case study to apply experimentation to one or more concepts that need verification and validation. I have written a one-page proposal for research, which has been submitted to Dr. Victor Basili, and have received a somewhat promising response from Dr. Basili.
WHAT CAN BE GAINED FROM PARTICIPATING IN THE DOCTORAL
CONSORTIUM
Although I am extremely lucky to have a very powerful
and knowledgeable committee working with me on my dissertation research,
it is still an advantage to have the type of professors attending the SIGCSE
conference and Doctoral consortium see my work and ideas. I am hoping
for constructive feedback that will keep me on the right track in my research.
There is an opportunity at the conference to be exposed to the most current
information available in a very short time. It would take much longer
searching the literature, and I would not get the same quality of direct
feedback that is possible in a conference setting. I will also be
looking for collaboration opportunities which will enrich the work my committee
and I are doing.
BIBLIOGRAPHIC REFERENCES