EXPERIMENTAL DESIGNS:

THREATS TO VALIDITY AND

ASSOCIATED CONTROL TECHNIQUES

I. Internal Validity

Internal validity: Accuracy of the research study in determining that changes in the independent variable CAUSED changes in the dependent variable, as opposed to the possibility that some "confounding" variable may have caused the observed results.

 

Threat

Description

Control

A. Maturation

Changes in the DV that occur due to the natural maturation of the subjects.

Include a control group in your design.

B. History

Changes in the DV that are due to events that occurred during the study that were not part of the study.

Include a control group in your design.

C. Testing (Practice Effects)

Changes in the DV due to the fact subjects have been tested on the same instrument more than once.

Include a control group in your design.

D. Instrumentation

Changes in the calibration of a measuring instrument that affects scores on the DV.

Recalibrate instruments often.

E. Regression to the mean

Tendency for subjects who obtain extreme scores on a variable to have less extreme scores on follow-up testing.

Beware of this threat when selecting subjects for inclusion in your study.

F. Selection

Any factor that creates control and experimental groups that are not equivalent on some important subject variable.

Random assignment of subjects to control and experimental groups.

G. Diffusion of Treatment

Changes seen in subjects’ responses due to information they received from other subjects in the study.

Controls may not be possible depending on the design of your study.

H. Attrition

The differential loss of subjects during a study. (Those who drop out are likely to be different from those who continue in the study).

No real control.

I. Sequencing (Order Effects)

Effects on a subject’s performance or behavior due to the experiences the subject has had in earlier parts of the study.

Employ counterbalancing techniques.

 

II. Subject and Experimenter Effects

 

A. Experimenter Bias: Any effect that the "expectations" of the researcher may have had on the measurement and recording of the dependent variable.

Experimenter Effect

Description

Control

Knowledge, expectations, and information that could affect the outcome of the experiment.

Knowledge of which subjects are in the control and experimental groups.

Utilize a single (or double) blind design so that the researcher does not know which subjects are in each group.

Biased measurement (i.e., seeing what the experimenter wants to see).

Use objective (rather than subjective) measurements.

Utilize multiple observers or "research assistants" who are unaware of the hypothesis driving the study. (Check for inter-rater reliability).

B. Subject Effects: Any response by subjects in a study that does not represent the way they would normally behave if not in the study.

 

Subject Effects

Description

Control

Demand Characteristics

Elements in a study that may cue a subject as to the purpose of the study.

Use deception (i.e., keep the subjects in the dark as to the true purpose of the study).

Placebo Effect or Self-fulfilling Prophecy

An observed improvement in subjects because subjects believe a change will occur.

Provide a placebo for the control group and ensure that all subjects are "blind" as to which group they are in (a single-blind placebo design).

III. External Validity

External Validity: The generalizability of the results of the study. The extent to which the results of a particular study can extend to other subjects, times, and settings.

The more NATURAL* the setting, the LARGER the sample, and the more REPRESENTATIVE the sample is of the larger population from which it was drawn, the greater the external validity of the study.

REPLICATING a study using different subjects in different settings with slightly different procedures will help you demonstrate external validity of a study.

*Keep in mind that while a "natural" setting increases external validity, it greatly decreases internal validity

B. Subject Effects: Any response by subjects in a study that does not represent the way they would normally behave if not in the study.

 

Subject Effects

Description

Control

Demand Characteristics

Elements in a study that may cue a subject as to the purpose of the study.

Use deception (i.e., keep the subjects in the dark as to the true purpose of the study).

Placebo Effect or Self-fulfilling Prophecy

An observed improvement in subjects because subjects believe a change will occur.

Provide a placebo for the control group and ensure that all subjects are "blind" as to which group they are in (a single-blind placebo design).

III. External Validity

External Validity: The generalizability of the results of the study. The extent to which the results of a particular study can extend to other subjects, times, and settings.

The more NATURAL* the setting, the LARGER the sample, and the more REPRESENTATIVE the sample is of the larger population from which it was drawn, the greater the external validity of the study.

REPLICATING a study using different subjects in different settings with slightly different procedures will help you demonstrate external validity of a study.

*Keep in mind that while a "natural" setting increases external validity, it greatly decreases internal validity.

 


[Home]        [SYLLABUS]        [CALENDAR]       [STUDY GUIDES]       [REVIEWS]     [LINKS]  [HOMEWORK]           [HANDOUTS]           [UNIT CONTENTS]            [RESEARCH]