Individualized Question Generation in an XML Framework

Justin Campbell
Dept. of Computer Science and Engineering, University of Washington

Introduction
One of the primary tools used by educators for learning, evaluation and student self-assessment is the problem set. Education research has shown that personalized assignments and questions can increase student collaboration, focus learning on fundamentals, motivate student success, and increase student mastery of the material, while maintaining positive student feedback [Toothman]. In general a personalized or individualized assignment is one that is created from a set of problem templates. The instructor writes the problems much like a normal assignment but it is then given to a tool which generates different instances of the problems for each student. Different tools to help instructors develop personalized exercises and/or assignments have been developed periodically over the last decades. However, the vast increase in computing power and availability in recent years has outpaced the sophistication of such tools.

Individualized exercises encourage students to focus on the concepts of a particular problem rather than on the answers. By changing various aspects of a question a student can no longer ask "Did you get answer C for question 2?" but must ask instead "How does casting a double work?" In addition, systems that provide instantaneous feedback often motivate students to solve every problem, rather than turning in partial solutions or partially completed assignments. [Morrissey]

Current tools to create individualized assignments often require a great deal of effort from the instructor. In addition, they often restrict exercise development to certain restricted classes of problems which may conflict with the learning goals or teaching style of the instructor. These problems are often too simplistic in nature or allow students to memorize problem patterns rather than learn the underlying concepts. My research focuses on creating a system that allows instructors to create problem templates for individualized assignments. These individualized exercises will be usable in a variety of ways including in homework, tests or self-assessment quizzes. The system will be straightforward to learn and use and yet allow significantly more sophisticated problems to be developed and used for individualized exercises.

Previous Research
Various tools and systems have been developed to create individualized assignments over the last several decades. Early work in this area includes the PLATO IV system used by Whitlock [Whitlock]. Michigan State University has a history of such tools including the Computer Assisted Personalized Approach (CAPA) system. Their system allows for various multiple-choice, multiple-selection and exact answer questions personalized per student and delivered over the Web [CAPA]. SAIL is another recent system which uses various LaTeX macros to add personalization to exercises [SAIL].

The drawback to these systems is that they rely heavily on simple forms of personalization. In general, question templates are generated with the intention of having random numbers placed into various aspects of the problem. This approach works well for many questions, for example "How many centimeters are in 3.14 inches?" or "Lexicographically sort the following numbers: 10 1 -18 38 12." While there are many useful exercises of this sort, there is a vast array of question types that cannot be created in a system that only does simple random number generation and substitution. In addition, many of these systems are designed for use in many different subject areas and therefore lack the power to generally address any particular area.

Goals of the Research
The primary goal of this research is to develop a system that allows complex individualized exercise templates to be created. The specific guiding principles are:

Current Status
I have begun to develop a system that uses XML as the framework for writing questions and personalizing them. Since many instructors maintain a course web page they are already familiar with HTML. XML can be used to add new tags to an HTML document that can then be processed to form individualized assignments.

The current system allows for many of the already available types of personalization like random number and random permutation generation. However, the real strengths from such a system will come from allowing more sophisticated templates. For example, the system should be able to individualize the following code:

int a[9] = {0, 1, 2, 3, 4, 5, 6, 7, 8};

for (i = 0; i < 6; i = i + 2)
     a[i+2] = a[i];
Not only the array initialization, but the expressions used in the for-loop can be further personalized. The system should dynamically generate and compile the exercise and then use that to verify a student's response. Current systems do not offer this level of complexity in writing the questions. In addition, I believe that allowing for the creation of simple BNF grammars and random expressions from them can open up a new class of problems to individualization. Consider the following problem:
Given the following expression:

!(P&&Q)

what values of P and Q will make the expression true?
Here the system should allow for dynamically created boolean expressions which can then be compiled and again used to check a student's response.

There are many other open avenues that lend themselves to individualization but require more complex forms of personalization than simple random number generation or variations on this theme. For example, constraint-based templates seem like a viable approach to many sophisticated template relations.

Interim Conclusions
More complex forms of individualization are possible with such a system. However, the extent to which such a system can be easily developed and learned is still to be explored.


Current Stage
I am in the process of passing my department's qualifying exam on the way to a Ph.D. All coursework has been completed.

What I Can Gain
This will be my first time attending SIGCSE. I am new to the field of computer science education research and I'm hoping to learn more about the current topics and methodologies researches use. I would like criticism and positive feedback on my research and its potentials. In addition, I hope to learn from the experience and knowledge of other doctoral students and expand my horizons. Lastly, I'd like to begin making contacts with others in this area.

Bibliographic References

Badros et al. "A Constraint Extension to Scalable Vector Graphics," WWW10, May 2001.

Bridgeman et al. SAIL: A System for Generating, Archiving, and Retrieving Specialized Assignments Using LaTeX. SIGCSE 2000.

CAPA: Computer-Assisted Personalized Approach. http://capa4.lite.msu.edu/homepage/

Preston, J and Shackelford, R. A System for Improving Distance and Large-Scale Classes. ITiCSE 1998.

Morrissey et al. Using Computer-Assisted Personalized Assisgnments for Freshman Chemistry. Journal of Chemical Education 1995 Vol 72 (2).

Toothman, B. and Shackelford, R. The Effects of Partially-Individualized Assignments on Subsequent Student Performance. SIGCSE 1998.

Whitlock, L. R. The Design and Implementation of a Dynamically Tailored Examination. Technical Report UIUCDCS-R-76-821, Dept. of Computer Science, UIUC, August 1976.