A questionnaire is one of the simplest data gathering mechanisms, and one which is well-suited to classroom use — most academics, for example, use questionnaires during or at the end of modules to ascertain the effectiveness of the teaching/learning during the module.
Questionnaires are a good way of ascertaining simple information quickly, but writing an effective questionnaire is a skill which is much more difficult to master than might be expected. Each question included — whatever type is chosen — must be carefully worded so as to be clear and unambiguous, and to allow for a full range of possible responses, and each question should be “neutral”, so as not to influence the choice of response. The layout of the questionnaire must not confuse the respondents, and should assist them in its completion. Furthermore, a questionnaire should be attractive so as to enthuse the respondent to complete it fully.
A questionnaire should be piloted (that is, trialled on a small group of respondents — possibly even colleagues) before it is actually deployed. This is really important — however well a questionnaire has been written, it is still possible for ambiguous or misleading phrases to have been accidentally used.
The types of question may be used for different purposes. Basic factual questions can help to provide a framework within which further questions make sense (such as age, gender, course enrolment, etc.). Questions about behaviour yield factual “added value” which is of direct interest. Other questions, which rely on the respondents’ opinions or judgements, may be difficult to interpret.
Questionnaires can be delivered on paper or online. Paper has the advantage that copies can be distributed at a suitable moment when respondents are co-located (for example, the start of a lecture), however they would then need manual or semi-manual processing (although OCR technology can assist). Online questionnaires can be managed with minimal human intervention, and consequent avoidance of transcription errors, but response rates may be low unless it must be completed as part of another online activity. The online approach, however, allows for innovative presentation (possibly inclusion of multimedia), for rapid turnround rates, and if appropriate for a questionnaire to be customised for individual respondents. Further issues to be considered include preserving anonymity online, since responses are potentially traceable, and avoiding duplicate responses.
Questions can include the following basic types:
- yes/no are quick to code responses, and forces the respondent to make a decision;
- multiple choice questions assume a range of expected responses;
- rank orderings are similar to multiple choice, but with added information about “priorities”;
- rating scales (such as Likert scales) offer flexible responses; and
- free response (open-ended) questions are appropriate when there is no expected range of answers.
Analysis of questionnaire data is normally quantitative, since a (relatively) large number of responses may be aimed for, sufficient to perform basic statistical analyses. Questions which require a “text field” response may need a simple qualitative analysis, but a questionnaire is normally inappropriate to elicit good qualitative data.
Anonymity is an issue. If a questionnaire asks for a respondent’s name or ID, then the responses to questions may not be accurate — for example, a student may not believe assurances that the data will not be used to grade them. On the other hand, knowledge of a respondent’s identity may help to correlate the data with other information, such as identifying factors which might have affected a student’s grade. This is a sensitive issue.
An approach which can be employed is the “follow-up” — respondents may be asked permission to be approached subsequently in order to gather further data. Also, where a significant proportion of potential respondents have failed to respond, it may be appropriate to contact them again for a second (or even a third) attempt at getting them to complete the questionnaire.
- Fast to administer.
- A good source of quantitative data.
- Anonymity may elicit genuine responses.
- Online questionnaires can be unintrusive.
- Significant time and resources are required to design a questionnaire.
- Anonymity may prevent the answers being correlated with other data.
- Responses may be inaccurate because respondents may give answers they think are expected.
How does students’ expertise in programming before University affect the perceived difficulty of their introductory programming module?
Most books on social science research address the basics of questionnaire writing, including Cohen (2000, 245-266), and we have found Foddy (1993), Gillham (2000) and Oppenheim (1992) particularly helpful. Oppenheim’s text, in particular, is accessible and easy to source.