Brief Description

This unit is concerned with questionnaire design. A first attempt at devising a questionnaire on school rules and punishments is reconsidered after work on fact finding and opinion polls. Through using poor questionnaires and examining what has gone wrong, pupils see the importance of questionnaire design in obtaining information.

Design Time: 4-5 hours.

Aims and Objectives

On completing the unit pupils should be able to identify design errors in other people's questionnaires and be better able to draw up sound questionnaires for themselves. Specifically they should be able to identify questions which are vague, biased or over-complicated and will have practised writing questions which are precise, unbiased, simple and clear. They also practise designing tables to take questionnaire answers, tallying and making critical comments on questions. They should be more aware of how data is collected in different circumstances, some problems that arise in data collection and some possible abuses of statistics.

Prerequisites

No mathematical or statistical prerequisites.

Equipment and Planning

Section A raises general problems of question formulation in the context of finding opinions on school rules and punishments. This is possibly best introduced as a class discussion.

Section B introduces design problems associated with fact-finding questionnaires. In Section B3 it is suggested that the class be split into groups of six.

Section C introduces design problems associated with opinion polls. The class has to be divided into two teams. One team answers questionnaire X and the other questionnaire Y. These are found on page R1. Page R2 is a Summary Sheet for the class answers. It is important that no pupil sees the other team's questionnaire until he has completed his own.

Section D makes more explicit the basic principles underlying sections B and C.

Section E summarizes the points to be remembered in good questionnaire design and includes a second attempt at designing a questionnaire for school rules and punishments.

Detailed Notes

Section A

This is a general introduction to some of the problems that may arise. The survey of Scottish pupils was reported in Spring 1977. It may be useful to have a class discussion after the pupils have attempted a to c, and some of their questions could be written on the board. It is probably better not to get too deeply involved at this stage as most of the points will come up again later in the unit.

The main point to be made is that writing questions is not easy. It is worth saying that some questions are unsatisfactory because they cannot be answered; others because the answers cannot easily be analysed. The question on school rules is difficult to answer because most schools do not have precise rules laid down and because there is an ambiguity between 'How many rules' and 'How many times', which could confuse.

Section B

The pupils are told in B4 that the questionnaire of B1 is 'very badly prepared'. Initially, though, the traps are there to be fallen into. To avoid duplicating discussion of the problems, try to get B1 and B2 done individually without discussion. If pupils spot all the mistakes in B1 and B2, then B3 can be covered more quickly as revision.

B1
Sex?
This is unlikely to be misunderstood, but to avoid answers such as 'boy', it would be better to include:

Height?
Here the units are not specified (metric or imperial), nor is the degree of accuracy. Some children will leave a blank because they don't know their height.

Hair colour?
Girls in particular may offer an impressive range of subtle shades of brown. It would have been better to offer a choice of four or five colours.

How many sweets do you eat?
This poses all sorts of problems of definition. Does 'sweets' include chocolate? What time-scale per day, per month, per year? Does it mean weight of sweets, or price paid for them, or actual number of sweets?

Mother's age at birth? Ambiguous. Hopefully some of the replies will say 'nought', but what is missing is the wotd 'your' before the last word. This question has actually appeared on an official questionnaire.

In general, most of these questions yield answers that cannot be analysed easily. Since they are trying to find out facts, there should not be much argument about the correct answer once the question has been sufficiently well-defined. A range of sensible answers, including 'Don't know', should be given to each question. This simplifies the analysis and helps remove ambiguities.

B2
You may need to emphasize to lower-ability pupils that we want the tables to show the number of pupils with different answers. A set of results like Male, Male, Female, Female, Male, ..., is not good enough. Below are some possible blank tables that could be used if the questions were more precise. Discussion on the formation of tables may well be necessary.

Pupils may well only see the inadequacy of their tables when they come to fill in the answers in B3. This can be used to indicate the importance of pre-testing a questionnaire on a different sample before carrying out a full survey.

B3
It would take too long for each pupil to record information from all the members of the class. The compromise suggested here is to divide the class into more manageable groups of six pupils each. Each group should meet at least some of the problems, but it may be valuable to summarize the problems which faced each group for the benefit of the rest of the class.

Alternatively, you can save time by pooling the ideas for summary sheet design. The class can then agree on the standard summary sheet which they all use. However, the major disadvantage of this approach is that the pupil is simply told his design is deficient. Until he uses his own summary tables and sees the snags he may not fully appreciate the important lessons to be learned.

B4
There are two different types of vagueness as indicated by questions 3 and 5 of 'Class Facts'. All respondents can be answering the same question, but use different terms in answering (as in question 3). Or they can be answering one of two distinct questions, as in question 5, where some will give age of mother at their birth and others will give age of mother at her birth (zero). This distinction is not made in the pupil notes. By incorporating a range of sensible alternatives, the error and four problems can often be avoided. Some ideas for an improved version are indicated in the notes on B2 above.

Section C

C1
Ostensibly the two questionnaires are trying to find out the same information. To bring out the effect of the wording it is important for the two teams to be the same size and have a similar distribution of opinions.

Each team must be ignorant of the other team's questionnaire. You will need multiple copies of the two questionnaires (page R1), and the summary (page R2) will also be needed.

It would be reasonable to use any selection process which is independent of pupils' opinions to give a suitable mix. Do be sure that the method adopted does give equal numbers in each team, otherwise a later comparison of the results is that much more difficult. Thus selection according to whether the pupil is born in an even or odd month of the year is independent of opinions but may yield teams of different size. Perhaps the simplest way is to rank pupils alphabetically and assign them alternatively to teams X and Y. If there is an odd number of pupils in the class, you may have to join in yourself.

Give the two teams just long enough to complete their questionnaire. If they stop too long to think about some of the questions, the bias may not come through as strongly.

C2

Section D

This section aims to bring out more positively the lessons of the previous two sections. A problem which is not covered, and may be brought to the attention of a more mature class, is that of getting the truth when asking a sensitive question. If smoking is against the school rules and you ask pupils about cigarette smoking, then no matter how unbiased the question is you are likely to get some untruthful answers.

This problem occurs to a greater or lesser extent with many surveys. In the Family Expenditure Survey the amount spent on alcoholic drinks is consistently understated, as is show by comparing with national sales figures.

D1
The three positive points to help find the truth are lack of bias, clarity and precision in the questions. It is worth collecting examples from real questionnaires that fail to meet these simple conditions. Not all the questions need to be done, extra ones have been put in for practice where required.

It is not easy to write unbiased questions. What constitutes clarity will depend on the population to be questioned. It is usually better to avoid double negatives such as 'Shouldn't you cross the road when the green light isn't showing?', since they are often misunderstood. Questions should clearly ask only one thing. In the quoted 'Do you agree that it is the responsibility of all qualified adults to exercise their democratic right at the ballot box in general elections', an 'Agree' could mean that it is the responsibility or that they should exercise their democratic right or that it should be at the ballot box rather than, say, at public meetings. Precision is often tied up with memory. Although 'How many hours do you watch television each week on average?' looks precise, without keeping records over a long period of time nobody would answer this accurately.

D2
This emphasizes that a careful wording of the questionnaire can make the subsequent analysis much easier.

Section E

The first seven guidelines are general and arise in the planning of a questionnaire. Guideline 1 is important and has not been emphasized elsewhere in this unit. A careful consideration of the purpose of the questionnaire often rules out many unnecessary questions and helps in writing good questions. Guidelines 3 and 4 are to encourage people to answer the questionnaire. Remember that it is their own time that is being taken up. People can also be encouraged to cooperate by being told the purpose. Guidelines 8 to 12 summarize the work of Section D and one of the questions answered by teams X and Y.

1. Guidelines 1 to 7 do not apply to this question, since the purpose of X was to show some of the faults' of questionnaire design rather than to find out particular information.
2. As a grand finale, after making sure that it satisfies the 12 guidelines, use one of the questionnaires from the class to find out what class opinion is on school rules and punishments.
3. This type of survey is often done by school geography departments. You may like to cooperate with them in designing the questionnaire and following up with analysing the results.
4. Alternatively a more topical survey could be carried out instead of option c.

Answers

(In many cases the answer given below is only one of many possible alternatives.)

Test Questions

1. The four questions a to d are all meant to find how much someone likes music. Which is the best question?
   1. Do you listen to cheap rubbish like 'Top of the Pops?'
   2. Do you spoil your concentration with constant radio?
   3. How many minutes did you spend yesterday:
      1. listening to music programmes on radio,
      2. listening to music on records, tape, or cassettes,
      3. singing or playing a musical instrument?
   4. Do you prefer classics, jazz or pop?
2. Which of the following questions a to e is the best to ask about someone's weight?
   1. Are you heavy?
   2. What do you weigh?
   3. Weight, without clothes, to the nearest kilogram.
   4. Even on occasions when in an unclad state, do you tip the scales to an excessive degree?
   5. Are you a measly weakling?
3. Look again at questions 2a to 2e.
   1. Name one question which is vague.
   2. Name one question which is biased.
   3. Name one question which is not simple and clear.
4. Questions often have a list of possible answers with them. You tick one, or cross out the others. Invent this type of answer for the questions below.
   1. Do you support Government policy on farm prices?
   2. Which day of the week do you do most of your shopping?
   3. How many children under 16 are there in your family?
   4. How long do you take to get home from school?
5. Write a summary table for the answers you would expect from guestion 4c.

Answers

1		c
2		c (b is second best)
3	a	2c, 2d, 2e (possibly b)
	b	2e
	c	2d
4	a	Yes / No / Don't know
	b	Monday / Tuesday / Wednesday / Thursday / Friday / Saturday (inclusion or omission of Sunday accepted)
	c	None / one / two / three / four / five or over (Omission of none accepted. More choices before open-ended class accepted)
	d	Up to 10 minutes 10 and up to 20 minutes 20 and up to 30 minutes 30 minutes or over - or alternatives
5		(0) 1 2 3 4 5 or more No. of children               No. of families               Space for tally and totals a bonus

Connections with Other Published Units from the Project

Other Units at the Same Level (Level 2)

Authors Anonymous
Fair Play
On the Ball
Getting it Right
Seeing is Believing

Units at Other Levels In the Same or Allied Areas of the Curriculum

Level 1

Leisure for Pleasure

Level 3

Car Careers
Pupil Poll

Level 4

Sampling the Census
Retail Price Index
Smoking and Health
Equal Pay

This unit is particularly relevant to: English, Humanities, Social Sciences.

Interconnections between Concepts and Techniques Used In these Units

These are detailed in the following table. The code numbers in the left-hand column refer to the items spelled out in more detail in Chapter 5 of Teaching Statistics 11-16.

An item mentioned under Statistical Prerequisites needs to be covered before this unit is taught. Units which introduce this idea or technique are listed alongside.

An item mentioned under Idea or Technique Used is not specifically introduced or necessarily pointed out as such in the unit. There may be one or more specific examples of a more general concept. No previous experience is necessary with these items before teaching the unit, but more practice can be obtained before or afterwards by using the other units listed in the two columns alongside.

An item mentioned under Idea or Technique Introduced occurs specifically in the unit and, if a technique, there will be specific detailed instruction for carrying it out. Further practice and reinforcement can be carried out by using the other units listed alongside.

Page R1

TEAM X QUESTIONNAIRE

Question
1	I prefer warm colours like brown, navy blue, and maroon.	Yes No
2	I'd rather be a humble doctor than a pop singer.	Yes No
3	The cane is needed to help keep order in schools.	Agree strongly Agree Don't know Disagree Disagree strongly
4	In general, which type of school would you prefer to attend?	Comprehensive Grammar Secondary Modern Independent Other
5	Should all pupils be allowed to stay on at school till they are 16?	Yes No Don't know
6	Are you often late for school?	Yes No
7	The differential calculus should be taught logically rather than genetically.	Agree Disagree Don't know

TEAM Y QUESTIONNAIRE

Question
1	I prefer dull colours like brown, navy blue and maroon.	Yes No
2	I'd rather be a doctor than have to sing for a living.	Yes No
3	Do you agree with the brutal system of beating innocent children with a cane?	Agree strongly Agree Don't know Disagree Disagree strongly
4	What are your views on secondary education?	______________________________________________
5	Should pupils be forced to stay on at school till 16, even when they don't want to?	Yes No Don't know
6	How often did you arrive late (after the first bell) last week?	0 1 2 3 4 5 times
7	The differential calculus should be taught logically rather ihan genetically.	Agree Disagree Don't know

Page R2

QUESTIONNAIRE SUMMARY SHEET

Back