Unit Two: Evidence: Data Collection 

On-line Lesson

2.1 Qualitative and Quantitative Data: Choices to Make

Qualitative and Quantitative Research Methodologies

Quantitative : are based on collecting numbers, numerical calculations, or statistics.

bullet

Experiments: random treatment assignments and quasi experiments using non-randomized treatments.

bullet

Surveys: which are cross-sectional or longitudinal

book icon Trust for Public Lands Telephone Survey Results
 

Qualitative : refers to the use of words for data collection and result in patterns ascertained through the content analysis of peoples words. This type of data invite individuals to describe experiences in their own words, such as open ended questions.

bullet

ethnographic which are observations of groups 

bullet

grounded theory which uses multi-staged data collection

bullet

phenomenological studies which studying subjects over a period of time through developing relationships with them and reporting findings based on research "experiences."

bullet

case studies which use various data to investigate the subject over time and by activity.

book icon Job Competency Analyses of Entry-Level Resort and Commercial Recreation Professionals
 

Each research method has it's strengths and weaknesses. When designing a research study it is important to decide what the outcome (data) the study will produce then select the best methodology to produce that desired information. 

Worldviews (Paradigms) and Data: represent broad assumptions about data and how projects should be undertaken. Data are evidence. Methods are the tools for data collection.

bullet

Positivist worldview suggests facts and truth can be found and articulated. This view is associated with the scientific method and results in quantitative data. Goal-Attainment is closely related to this paradigm.

bullet

Interpretive worldview suggests that there are many perspectives and truths may exist within any evaluation or research study. the Goal-Free Model is often used to collect qualitative data.

bullet

Post-positivist perspective is more likely to be used instead of a pure interpretive paradigm. This approach allows for creativity in data collection and stays closer to the traditional scientific method by keeping a relatively narrow focus/topic.

Data Types

Both qualitative and quantitative types of data are used in research and evaluation projects. Most data collection methods can be sued to collect qualitative or quantitative data depending on the questions asked and how data are collected.

A simple distinction between qualitative and quantitative data suggests that:

* The nature of the data provides the framework to determine if a project uses qualitative or quantitative approaches.

Describing Differences in Data

Quantitative data usually follow standard procedures of rigor related to the instruments used and to statistical data analysis. Quantitative data are deductive, which means that particular evaluation questions or established theories serve as th start, and data are used to test those questions.

Qualitative data are more specific to a given context or situation. Qualitative data are generally part of a Goal-Free Model and are concerned with discovering new phenomenon or for developing theories.

See Making Choices about Quantitative and Qualitative Data table (p. 117).

2.2 Choosing Designs and Methods: The Big Picture

Major Categories of Evaluation or Research Design

bullet

experimental

bullet

descriptive/evocative (previously referred to an non-experimental)

The table below describes the basic characteristics of different methodologies.

Data Collection Methods Examples
Documents
bulletHistorical
bulletLiterature review
bulletMeta-analysis
bulletDiaries
bulletContent Analysis
bulletSecondary Data (data mining)
These methods identify trends in leisure research and practice. Participants keep diaries and journals researcher conducts content analysis of studies, reports and diaries.
Observations
bulletInterpretive
bulletEthnographic
bulletParticipant observer
bulletCase study
How people behave and interact in public open spaces. Observe systematically, become a participant observer.
Survey
bulletQuestionnaire
bulletInterview
bulletStandardized Scales/Instruments
To learn what people think about leisure motivation. To identify
relationships between motivation and satisfaction. Use interviews, surveys and standardized scales.
Experimental
bulletTrue experiments
bulletQuasi experiments
Obtain information under controlled conditions about leisure attitudes and experience with virtual reality. Subjects may be randomly assigned to various tests and experiences then assessed via observation or standardized scales.
Other Field Methods 
bulletNominal Group Technique
bulletDelphi
To identify trends and issues about leisure services, management and delivery systems. Focus Group systems. Various group, question and pencil paper exercises are used by facilitators.
Multimethods Approach
bullet Combination of methods shown
Interviews, journals and quantitative measures are combined to provide a more accurate definition and operationalization of the concept.
Source: Issac & Michael, 1985; Leedy, 1985; Dandekar, 1988; Thomas & Nelson, 1990.

Data Collection Techniques

There are two sources of data. Primary data collection uses surveys, experiments or direct observations. Secondary data collection may be conducted by collecting information from a diverse source of documents or electronically stored information. U.S. census and market studies are examples of a common sources of secondary data. This is also referred to as "data mining." 

Key Data Collection Techniques

bullet

Surveys

bullet

Questionnaires

bullet

Panel Questionnaire Designs

bullet

Interviews

bullet

Experimental Treatments

Experimental Treatments

Experimental designs are the basis of statistical significance. An example of the fundamentals of an experimental design is shown below.

A researcher is interested in the effect of an outdoor recreation program (the independent variable, experimental treatment, or intervention variable) on behaviors (dependent or outcome variables) of youth-at-risk. 

In this example, the independent variable (outdoor recreation program) is expected to effect a change in the dependent variable. Even with a well designed study, an question remains, how can the researcher be confident that the changes in behavior, if any, were caused by the outdoor recreation program, and not some other, intervening or extraneous variable ? An experimental design does not eliminate intervening or extraneous variables; but, it attempts to account for their effects.

Experimental Control

Experimental control is associated with four primary factors (Huck, Cormier, & Bounds, 1974).

  1. The random assignment of individual subjects to comparison groups;

  2. The extent to which the independent variable can be manipulated by the researcher;

  3. The time when the observations or measurements of the dependent variable occur; and

  4. Which groups are measured and how. 

Treatment Group: The portion of a sample or population that is exposed to a manipulation of the independent variable is known as the treatment group. For example, youth who enroll and participate in recreation programs are the treatment group, and the group to which no recreation services are provided constitutes the control group.

EXPERIMENTAL DESIGNS

  1. True Designs: require that an independent variable be controlled and manipulated.

  2. Quasi Designs: do not have all the qualities of a "true" experimental design. Quasi designs do not use random assignment into treatment or control groups.

  3. Ex Post Facto Designs: will determine which variables discriminate between subject groups.

True Designs - Five Basic Steps to Experimental Research Design

1. Survey the literature for current research related to your study.
2. Define the problem, formulate a hypothesis, define basic terms and variables, and operationalize variables.
3. Develop a research plan:
     a. Identify confounding/mediating variables that may contaminate the experiment, and develop methods to control or minimize them.
     b. Select a research design.
     c. Randomly select subjects and randomly assign them to groups.
     d. Validate all instruments used.
     e. Develop data collection procedures, conduct a pilot study, and refine the instrument.
     f. State the null and alternative hypotheses and set the statistical significance level of the study.
4. Conduct the research experiment(s).
5. Analyze all data, conduct appropriate statistical tests and report results.

Quasi Designs

The primary difference between true designs and quasi designs is that quasi designs do not use random assignment into treatment or control groups since this design is used in existing naturally occurring settings.

Groups are given pretests, then one group is given a treatment and then both groups are given a post-test. This creates a continuous question of internal and external validity, since the subjects are self-selected. The steps used in a quasi design are the same as true designs.

Ex Post Facto Designs

An ex post facto design will determine which variables discriminate between subject groups.

Steps in an Ex Post Facto Design

  1. Formulate the research problem including identification of factors that may influence dependent variable(s).

  2. Identify alternate hypotheses that may explain the relationships.

  3. Identify and select subject groups.

  4. Collect and analyze data

Ex post facto studies cannot prove causation, but may provide insight into understanding of phenomenon.

READ THIS SECTION BEFORE ATTEMPTING ASSIGNMENT 3

EXAMPLES OF EXPERIMENTAL RESEARCH DESIGNS

Three Types of Research Designs

bullet

Pre-Experimental

bullet

Quasi-Experimental

bullet

True Experimental

Experimental Notation

bullet

T1 = Pretest

bullet

T2 = Post-test

bullet

R = Randomly selected/assigned

bullet

X = Treatment of subjects (independent variable)

Pre-Experimental Designs

The difference between Quasi-experimental and Pre-experimental designs is pre-experimental designs, lack control of internal validity. For example, using the T1 X T2 design. Say you have a program for an existing youth at risk group and wish to determine if their self esteem would improve if they participated in a 14 day outdoor adventure program. So this is an existing group (no randomization or control group or treatment group) the entire group takes a pre-test (T1) then participates in a 14 day outdoor adventure program (treatment X) then takes the same test after the program. If there are any changes in the individuals test scores between the pre-test and post-test, it may be attributable to the 14 day outdoor adventure program. So a Pre-experimental design uses an existing population with no randomization which is called a convenience sample. It also has no control or treatment groups.

Another example would be if you wanted to conduct a community recreation services opinion survey. You use ALL registered voters in your City as the population for the survey. This is a very common design used in parks and recreation research (XT2).

bullet

Research Design with No Control: One-Group Post-test Only Design (X T2)

bullet

Research Design with Minimal Control: One-Group Pretest/Post-test Design (T1 X T2)

Quasi-Experimental Designs

Real life situations in leisure research create many instances when true experimental research is not possible, but some type of causal inference is needed. The purpose of the quasi-experimental design is to approximate the conditions of the true experiment. Quasi-experimental designs will cause compromises in terms of the internal and external validity of the design. The quasi-experimental research design is characterized by methods of partial control of the study. Otherwise, the steps used in quasi-experimental methods are essentially the same as in true experimental methods.

A quasi-experimental design will use an existing group/population but not randomize the participants. For example, you have two adult soccer teams. You wish to study if drinking caffeinated energy drinks will affect a players performance. So you may compare one existing soccer team A (no caffeinated energy drinks) or a treatment team B (caffeinated energy drinks). Then give each team a pre-test of soccer performance then a post-test of soccer performance: to another soccer team control group A (no caffeinated energy drinks) or the second soccer team B (caffeinated energy drinks). Then give each team a pre-test of soccer performance then a post-test of soccer performance.

A T1 X T2
B T1    T2

So any change in the treatment groups performance may be attributed to the caffeinated energy drinks. Because there was no randomization, this is NOT a true-experimental design but quasi-experimental. 

bullet

Nonequivalent Pretest-Post Test Control Group Design

A T1 X T2
B T1    T2
bullet

Single Group Interrupted Time Series Design

T1T2T3 X T4T5T6
bullet

Control Group Time Series Design

Group           Pretest     Treatment     Post Test
Experimental     T1                 X                  T
Control                T1                                               T2
bullet

Single Subject Designs

Baseline Data A - Intervention  - Treatment Data B
                                                 X X X X
T1T2T3T4T5T6T7T8T9T10

True Experimental Designs

True experimental designs provide the strongest indications for causal effects of the independent variable because of the control and ability to explain or account for extraneous factors. A true experimental design will have a random assignment to two or more groups.

A true experimental design will use a population and randomize the participants into two or more groups. For example, you have a group of adult soccer players. You wish to study if drinking caffeinated energy drinks will affect a players performance. So you randomly assign each player to a control group A (no caffeinated energy drinks) or a treatment group B (caffeinated energy drinks). Then give each team a pre-test of soccer performance then a post-test of soccer performance.

So any change in the treatment groups performance would be attributed to the caffeinated energy drinks.

bullet

Randomized Pretest-Post Test Control Group Design

Randomly Assigned Pretest Treatment Post test
Experimental Group T1 X T2
Control Group T1   T2
  OR
Randomly Assigned Pretest Treatment Post test
Experimental Group #1 T1 Xa T2
Experimental Group #2 T1 Xb T2
Control Group T1   T2
bullet

Randomized Control Group Post Test Only Design

Randomly Assigned Pretest Treatment Post test
Experimental Group   X T2
Control Group     T2

OTHER FIELD METHODS/GROUP TECHNIQUES

Nominal Group Technique (NGT)

The NGT is a group discussion structuring technique. It is useful for providing a focused effort on topics. The NGT provides a method to identify issues of concern to special interest groups or the public at large. Ewert (1990) noted that the NGT is a collective decision-making technique for use in park and recreation planning and management. The NGT is used to obtain insight into group issues, behaviors and future research needs.

Five Steps of the NGT 

  1. Members of the group identify their individual ideas in writing, without any group discussion;

  2. Each member lists his/her own ideas and then rank-orders them, again without any group discussion;

  3. A facilitator gives each participant an opportunity to state his/her ideas ( one item per person at a time, in round-robin fashion) until all ideas are exhausted; 

  4. As a group, participants discuss and consolidate ideas into a list; and 

  5. Finally, members vote to select priority ideas. The final list of ideas becomes the focus of further research and discussion. These ideas can also be used to generate a work plan for a formal strategic planning process, a basis for a survey or interview, or the development of a scale.

Source: (Mitra & Lankford, 1999)

Delphi Method

The delphi method was developed to structure discussions and summarize options from a selected group to:

bullet

avoid meetings, 

bullet

collect information/expertise from individuals spread out over a large geographic area, and

bullet

save time through the elimination of direct contact. 

Although the data may prove to be valuable, the collection process is very time consuming. When time is available and respondents are willing to be queried over a period of time, the technique can be very powerful in identifying trends and predicting future events.

The technique requires a series of questionnaires and feedback reports to a group of individuals. Each series is analyzed and the instrument/statements are revised to reflect the responses of the group. A new questionnaire is prepared that includes the new material, and the process is repeated until a consensus is reached. 

The reading below is a research study that used the delphi technique and content analysis to develop a national professional certification program.

book icon Sample Research Article: Job Competency Analyses of Entry-Level Resort and Commercial Recreation Professionals
 

Focus Groups

Richard Krueger (1988), describe the focus group as a special type of group in terms of purpose, size, composition, and procedures. A focus group is typically composed of seven to twelve participants who are unfamiliar with each other and conducted by a trained interviewer. These participants are selected because they have certain characteristics in common that relate to the topic of the focus group.

The researcher creates a permissive environment in the focus group that nurtures different perceptions and points of view, without pressuring participants to vote, plan, or reach consensus. The group discussion is conducted several times with similar types of participants to identify trends and patterns in perceptions. Careful and systematic analysis of the discussions provide clues and insights as to how a product, service, or opportunity is perceived.

A focus group can be defined as a carefully planned discussion designed to obtain perceptions on a defined area of interest in a permissive, nonthreatening environment. It is conducted with approximately seven to twelve people by a skilled interviewer. The discussion is relaxed, comfortable, and often enjoyable for participants as they share their ideas and perceptions. Group members influence each other by responding to ideas and comments in the discussion.

CHARACTERISTICS OF FOCUS GROUPS

Focus group interviews typically have four characteristics: 

  1. Identify the target market (people who possess certain characteristics);

  2. Provide a short introduction and background on the issue to be discussed;

  3. Have focus group members write their responses to the issue(s);

  4. Facilitate group discussion;

  5. Provide a summary of the focus group issues at the end of the meeting.

Other types of group processes used in human services (delphic, nominal, planning, therapeutic, sensitivity, or advisory) may have one or more of these features, but not in the same combination as those of focus group interviews.

Behavior/Cognitive Mapping

Cognitive and spatial mapping information provides a spatial map of:

bullet

current recreation use, 

bullet

the most significant recreation resources, and 

bullet

the approximate number of visitors to the recreation areas.

All types of recreation activities and travel involve some level of environmental cognition because people must identify and locate recreation destinations and attractions.

Cognitive mapping allows recreation resource managers the opportunity to identify where users and visitors perceive the best recreation areas are located. It is important to understand user perceptions in order to manage intensive use areas in terms of maintenance, supervision, budgeting, policy development and planning. 

Cognitive maps grid the research site into zones. The zones identify existing geographic, climatic, landscape, marine resources, and recreation sites. The grids allow respondents to indicate primary recreation sites, and then a  composite is developed to identify high impact areas. Researchers collect data at recreation areas (beach, campground, marina, trailhead, etc.) by interviewing visitors and recreationists. During the data collection process, random sites, days, times, and respondents (every nth) should be chosen to increase the reliability and generalizability of the data.

Observations

Observational research is used for studying nonverbal behaviors (gestures, activities, social groupings, etc.).

Sommer & Sommer (1986) developed the list shown below to assist in observation research.

  1. Specify the question(s) of interest (reason for doing the study).

  2. Are the observational categories clearly described? What is being observed and why?

  3. Design the measurement instruments ( checklists, categories, coding systems, etc.).

  4. Is the study designed so that it will be 'Valid (i.e., does it measure what it is supposed to measure, and does it have some generalizability)?

  5. Train observers in the use of the instruments and how to conduct observational research.

  6. Do a pilot test to (a) test the actual observation procedure and (b) check the reliability of the categories of observation using at least two independent observers.

  7. Revise the procedure and instruments in light of the pilot test results. If substantial changes are made to the instrument, run another pilot test to make sure changes will work under the field conditions.

  8. Collect, compile, and analyze the data and interpret results.

Casual observation is normally done like unstructured interviews. During the early stages of a research project, casual observation allows the researcher(s) to observe subjects prior to designing questionnaires and/or interview formats. 

Types of Observation Studies

bullet

Participant observer

bullet

Windshield surveys

bullet

Case study

Documents (also called Secondary Data or Data Mining)

Data mining is commonly used in both qualitative and quantitative research. Secondary data provides data which provides a framework for the research project, development of research question(s), and validation of study findings. 

Frequently used sources of secondary data are:

bullet

U.S. Census -  Extensive demographic data including age, sex, distribution, education, ethnicity, migration patterns, service industry, etc.

bullet

Bureau of Labor Statistics - Extensive information on such things as employment, unemployment, types of employment, income, etc.

bullet

National Center for Health & Information Vital rates such as births, State Department of Health deaths, health, etc. marriage and divorce rates, etc.

bullet

State Employment Departments - Number employed by industry, projected levels of employment growth, available jobs skills and skill shortages

bullet

Federal Land Management - National Parks, historic sites, scenic areas, forests by acres, budget and visitation rates.

bullet

State Highway Departments - Miles and condition of highways, bike lanes, and streets, capital and maintenance costs of highways

bullet

Law Enforcement Agency - Number and types of motor vehicles, types of crimes and violations, number of police officers by county and city, law enforcement

bullet

Outdoor Recreation - Number and type of parks, number and type of Agency/Dept. campgrounds, location, and rates for parks, lakes, rivers, etc.

bullet

Welfare/Human Services - Number of families on various types of Department assistance such as Aid to families with Dependent Children, Social Security, and SSI. Number of alcohol and drug abuse counselors, number of family counselors. Number and cases of child abuse, spouse abuse, desertions, child adoptions rate.

bullet

Newspapers - Scanning local newspapers is an excellent means to become better acquainted with a community and its principal actors as well as the issues that have been of greatest local concern

Content Analysis

Content analysis systematically describes the form or content of written and/or spoken material. It is used to quantitatively studying mass media. The technique uses secondary data and is considered unobtrusive research.

The first step is to select the media to be studied and the research topic. Then develop a classification system to record the information. The techniques can use trained judges or a computer program can be used to sort the data to increase the reliability of the process.

Content analysis is a tedious process due to the requirement that each data source be analyzed along a number of dimensions. It may also be inductive (identifies themes and patterns) or deductive (quantifies frequencies of data). The results are descriptive, but will also indicate trends or issues of interest.

The reading below is a research study that used the delphi technique and content analysis to develop a national professional certification program.

book icon Sample Research Article: Job Competency Analyses of Entry-Level Resort and Commercial Recreation Professionals
 

Meta-Analysis

Meta-analysis combines the results of studies being reviewed. It utilizes statistical techniques to estimate the strength of a given set of findings across many different studies. This allows the creation of a context from which future research can emerge and determine the reliability of a finding by examining results from many different studies. Researchers analyze the methods used in previous studies, and collectively quantify the findings of the studies. Meta-analysis findings form a basis for establishing new theories, models and concepts.

Thomas and Nelson (1990) detail the steps to meta-analysis:

  1. Identification of the research problem.

  2. Conduct of a literature review of identified studies to determine inclusion or exclusion.

  3. A careful reading and evaluation to identify and code important study characteristics.

  4. Calculation of effect size. Effect size is the mean of the experimental group minus the mean of the control group, divided by the standard deviation of the control group. The notion is to calculate the effect size across a number of studies to determine the relevance of the test, treatment, or method.

  5. Reporting of the findings and conclusions.

Historical Research

Historical research in leisure studies may focus on:
bullet

biographies of park and recreation professionals (Joseph Lee, Jane Adams, etc.), 

bullet

public, non-profit and private institutions (public parks and recreation, federal land management agencies), 

bullet

professional movements (playgrounds, leisure education), and 

bullet

related concepts (professionalism, certification and licensure, play). 

Historical research is also referred to as analytical research. Common methodological characteristics include a research topic that addresses past events, review of primary and secondary data, techniques of criticism for historical searches and evaluation of the information, and synthesis and explanation of findings. Historical studies attempt to provide information and understanding of past historical, legal, and policy events.

Five basic procedures common to the conduct of historical research were identified by McMillan & Schumacher (1984). They provide a systematic approach to the process of historical research.

Step 1: Define the problem, asking pertinent questions such as: Is the historical method appropriate? Are pertinent data available ? Will the findings be significant in the leisure services field?

Step 2: Develop the research hypothesis (if necessary) and research objectives to provide a framework for the conduct of the research. Research questions focus on events (who, what, when, where), how an event occurred (descriptive), and why the event happened (interpretive ). This contrasts with quantitative studies, in which the researcher is testing hypotheses and trying to determine the significance between scores for experimental and control groups or the relationships between variable x and variable y.

Step 3: Collect the data, which consists of taking copious notes and organizing the data. The researcher should code topics and subtopics in order to arrange and file the data. The kinds of data analysis employed in historical research include (based on McMillan & Schumacher, 1984):

  1. Analysis of concepts. Concepts are clarified by describing the essential and core concepts beginning from the early developmental stages. Clarification allows other researchers to explore the topic in other fashions.

  2. Editing or compilation of documents, to preserve documents in chronological order to explain events. For ex- ample, an edition of Butler's park standards, the National Recreation and Park Association's first minutes, or letters from early pioneers in the field preserves the documents for future researchers.

  3. Descriptive narration tells the story from beginning to end in chronological order, utilizing limited generalizations and synthesized facts.

  4. Interpretive analysis relates one event to another event. The event is studied and described within a broader con- text to add meaning and credibility to the data. For example, an examination of the development of a local jurisdiction's ability to dedicate land for parks may be related to the urbanization and loss of open space in our communities.

  5. Comparative analysis examines similarities and differences in events during different time periods-for example, the budget-cutting priorities and procedures of the Proposition 13 era of the early 1980s in parks and recreation as compared to the budget-cutting priorities and procedures of today.

  6. Theoretical and philosophical analysis utilizes historical parallels, past trends, and sequences of events to suggest the past, present, and future of the topic being researched. Findings would be used to develop a theory or philosophy of leisure. For example, an analysis of public recreation agency goals and objectives of previous eras can be used to describe the future in the context of social, political, economic, technological, and cultural changes in society.

Step 4: Utilizing external and internal criticism, the re- search should evaluate the data. Sources of data include documents (letters, diaries, bills, receipts, newspapers, journals/magazines, films, pictures, recordings, personal and institutional records, and budgets), oral testimonies of participants in the events, and relics ( textbooks, buildings, maps, equipment, furniture, and other objects).

Step 5: Reporting of the findings, which includes a statement of the problem, review of source material, assumptions, research questions and methods used to obtain findings, the interpretations and conclusions, and a thorough bibliographic referencing system.

2.3 Trustworthiness: The Sine Qua Non of Data Collection

Reliability - relates to whether a measure consistently conveys the same meaning.

Research should be valid and reliable.

Reliability: is based on the repeatability or replicability of results. The results of a research study should be the same when repeated with a similar setting and subjects. The three types of reliability checks are:

bullet

Test-Retest Reliability

bullet

Alternative or Equivalent Form Reliability

bullet

Internal Comparison or Consistency Reliability

Dependability: is when the evaluator keeps a record of how data were collected (called an audit trail) and how conclusions were drawn.

Validity: addresses whether the instrument measures the intended criteria (what it is supposed to measure). Data can be reliable without being valid.

There are two primary criteria for evaluating the validity of an experimental design. 

bullet

Internal validity. Determines whether the independent variable made a difference in the study? Can a cause-and-effect relationship be observed? To achieve internal validity, the researcher must design and conduct the study so that only the independent variable can be the cause of the results (Cozby, 1993). The most important type of internal validity is "content validity."

bullet

External validity, refers to the extent to which findings can be generalized or be considered representative of the population.

Usability: if the instrument cannot be effectively administered and if the data collected are not analyzed and applied the project is useless. Usability factors include:

bullet

instrument should be easy to administer

bullet

time required should be reasonable for the situation and population

bullet

instrument should be easy to score and interpret

bullet

directions should be consistent

bullet

providing a good atmosphere for testing is necessary

Trustworthiness: data are only as good and the means used to collect them. Evaluators should eliminate error as much as possible. Errors may come from the respondents, investigator, or sampling procedure. See table on page 120.

2.4 What are the Chances? Choosing a Sample

Sampling procedures vary depending on the methods used and the resources available.

Sampling Categories:

  1. probability sampling - means that everyone within a population has the same potential of being selected as part of the sample. Random sampling is the most common type of probability sampling and is superior to non-probability sampling.

  2. non-probability sampling - samples where not everyone has an equal chance and the likelihood of getting selected from a population is unknown.

  3. theoretical sampling - used primarily with qualitative data and refers to sampling until the evaluator reaches a point of data saturation.  This means no new data that contribute to the emerging themes are evident.

Key to Appropriate Sampling 

  1. the sample size required for a survey will depend on the reliability needed, which depends on how the results will be used.

  2. a sample size can be selected in various ways. Data is considered "fragile" if you get less than 60-70% of the sample to respond;

  3. methods of sampling are grounded in statistical theory and theories of probability. The sample used depends on the objective and scope of the method used and data desired.

  4. the relevant population to be sampled must be clearly identified.

  5. in sampling we seek to sample as few people as necessary to be an accurate probability representation.

Population:  refers to all the people who might comprise a particular group (zip code, age, gender, activity preference etc.) of people whose opinions are sought. In most cases, this group is so large and diverse that it is impossible to ask each member of the group every question so it becomes necessary to select a sample. If an entire population (called a convenience sample) is used then there is no sampling.

Randomization: a subset of a population chosen in such a way that all individuals of the specified population have equal probability of being selected.

Probability: the probability of an occurrence of any event ranges from .00 (no chance that the event will occur) to 1.00 (event is certain to happen). Researchers are interested in determining the probability of an event.

Before a sample can be obtained the universe (all members of the group ie., all city residents, all members of the hiking club, all 6th graders, etc.) must be identified.

Types of Probability Samples

bullet

simple random sample. This sample is the most commonly used in park, recreation and leisure services evaluations. It is based on the assumption that every member of a population has the same probability of being selected. The probability is the ratio of  the sample size and the population size (sample size / population = random sample ratio). For example, start with a participant list (N=559) and randomly pick a starting point, say 134 then pick every  6th name (nth) in the entire list for a total sample of 93. This will make the margin of error 10%.

bullet

stratification. This method divides the population into groups and then uses the categorization as the starting point for the sample selection.

bullet

systematic sampling determines a rationalization for some kind of routine sample.

bullet

cluster sampling. This method assumes there are groups with identifiable characteristics (geographic area, ethnicity, etc.) which can be used to select the sample.

Types of Non-probability Samples

bullet

Purposive sampling is arbitrarily selecting a sample because you believe evidence supports that the sample represents the total population.

bullet

Convenience sampling refers to a sample that happens to be available for a project.

bullet

Quota sampling is based on dividing the population into subgroups and drawing a sample to fulfill a specific quota.

bullet

Expert sampling is a process where people are chosen on the basis of an informed opinion that they are representative of a population that has needed information.

bullet

Network sampling or Snowball approach is when the evaluator gets people he/she knows to recommend others to be the sample for a project.

Population Lists and Concerns

Potential lists include: phone books, utility customers, registered voters, hospital or school records, drivers license records, etc.. Remember, not everyone has a phone, is a registered voter, or has a drivers license and therefore would be excluded from those lists.

List Concerns

bullet

Does the list contain any duplicate names?

bullet

Is it a complete list, any names missing?

bullet

Does the list contain any ineligible members?

bullet

Is the list accurate? Does it have correct phone and/or address information?

Representative Sampling

After the population has been identified it is imperative that any sample represents the entire population. The table below shows the proportion of ethnic groups in the Town of Parker.

Ethnic Groups

1990

Percent

African American

57

1.9

American Indian

664

22.9

Asian

46

1.5

Caucasian

1,956

67.5

Other

174

6.0

Total

2,897

100

Any sampling done, would need to include the same proportions of each ethnic group. The same concept applies to other socio-demographic characteristics such as age, gender, income level, education, etc.. Representation also is strongly related to probability.

Probability and Sampling Error

Sampling error: is the difference between the characteristics of a sample and the characteristics of the population from which the sample was selected (Babbie, 2006). Standard error is a function of sample size. As a sample size increases, the standard error decreases.

Non-sampling error: these have no simple and direct method of estimating. Non-sampling errors concern the biases that may exist due to who answers a survey compared to those people who did not. Errors may also come from sampling operations, non-interviews, participants not understanding the ideas being measured, lack of knowledge by the respondents, concealment of the truth, loaded questions, processing errors and interviewer errors.

A sample is a randomly selected subgroup of the population and therefore is not an exact representation. It is an estimate that reflects the larger population. This "estimate" is known as the "sampling error," and can be measured. The sampling error calculates the range of doubt, or degree of confidence, reflected in the sample measurement. 

Two Steps of the Sampling Process

  1. determine the kind of sample needed (population, data collection, etc.)

  2. determine the appropriate sample selection process.

Types of Samples and Sample Selection Process

Sample Selection Factors

bullet

goals of survey

bullet

population being studied

bullet

project budget

bullet

project timeline

bullet

sample selection difficulty

Determining Sample Size

The size of a sample will have a significant impact on the validity of the study. Primarily, sample size is determined by the degree of error that is acceptable to the researcher. Error can be calculated based on the sample size. It is usually expected to be within 3% to 5% for social sciences.

book icon Additional Reading: Calculating Sample Size and Margin of Error or page 123 of the text.

2.5 Choosing the Right Stuff: Measurement Instruments

After the criteria to examine and a research design has been selected, the next step is to create or identify a measuring instrument. For qualitative data the researcher is usually the instrument used (ask questions or make observations). For quantitative data a written instrument, observation checklist, questionnaire or test is necessary. If a valid and reliable instrument already exists that is a preferable choice to creating a new instrument. Researchers have two choices:

  1. Evaluate existing instruments to determine if they are reliable, valid and usable.

  2. Develop a new instrument and conduct the necessary process to confirm the new instrument is reliable, valid, and usable.

Measuring Instruments

bullet

Tests are standardized methods of asking for information. Types of tests include ability; achievement; attitude; and personality.

2.6 On Your Own Again: Developing Measurement Instruments

If a satisfactory instrument does not exist you may have to develop your own instrument.

book icon Additional Reading: ASA Brochures on Survey Research

book icon Additional Reading: Downtown Dew Questionnaire
 

Steps in Development of a Questionnaire

  1. Define the problem (see criteria)

  2. Determine the contents and ideas to be examined

  3. Identify and categorize the respondents

  4. Develop items, structure and format

  5. Write directions

  6. Pilot test, revise, field test, and revise

  7. Ensure response

A survey is a method to collect data, a questionnaire is the tool/instrument used to conduct a survey.

Five Areas Addressed in Developing a Question

  1. experience/behavior

  2. opinion/values (needs or interests)

  3. feelings (emotional responses)

  4. knowledge/facts

  5. background/demographics

Question Structures

Three major structures and the scales that can be sued wit each, include the following:

bullet

Open-ended, no predefined answers provided.

bullet

Close-ended With Ordered (Forced) Choices and Unordered Response Choices, specific responses are provided for the respondents to . Two types of close-ended questions: Ordered (Forced) Choices use a Likert scale, Semantic Differential scale, and rankings or Unordered Response Choices use multiple choice checklists.

bullet

Partially Close-Ended Questions, provide some structured responses, but provide space for respondents to write in their own answers or information.

Sample Open-ended Questions

What is your opinion of the condition of the public park and recreation facilities?

Sample Close-Ended Likert Type Scale

Funding Source

1

Do Not Favor

2

Slightly Favor

3

No Opinion

4

Favor

5

Strongly Favor

Bond issues be repaid by property taxes?

 

 

 

Wording Issues

  1. Use only one idea per question.

  2. Clear, brief, and simple are the rules for good questions.

  3. Avoid leading questions that suggest that you want a particular kind of response.

  4. Avoid asking people to make estimates.

  5. Think about trying to formulate a question from the respondent's viewpoint.

  6. Avoid advanced language and technical terms that is familiar to you but the respondent's may not understand.

  7. Be clear about what is being asked.

  8. Avoid negative questions that may con fuse the reader.

  9. Avoid the use of "iffy" words like "often" or "seldom."

  10. Do a pilot study.

  11. State the alternatives precisely.

  12. Some questions may require several stages to get adequate information.

Formats and Instrument Design

The general appearance may be critical to the success of a survey. Simple graphics and "white space" are important visual effects.

Questionnaire Format

bullet

questions should be ordered in a logical sequence

bullet

clear directions throughout the instrument are important

bullet

start with general questions and move to more specific ones

bullet

group controversial items with less controversial ones

bullet

group questions together according to content

bullet

use the same types of question structures together

bullet

build ties between questions

bullet

questionnaire of reasonable length (a two page questionnaire may get a higher response rate than 6 pages)

bullet

save the demographic questions for the end of the questionnaire

bullet

instrument should look professional and have no typographical errors.

book icon Additional Reading: Dead River Whitewater Boating Survey

2.7 Surveys: The Winner of the Popularity Contest

The survey is the most common method used in park and recreation evaluation.

Response Rate: is the percentage of questionnaires received based on the number that were distributed. The formula for calculating the response rate is:

Number of Completed Questionnaires Returned (divided by) Number of questionnaires distributed = Response Rate

Example: 20 questionnaires returned and usable divided by 25 questionnaires distributed = 80% response rate

Acceptable and Achievable Response Rates

bullet

70% from special interest groups (Henderson & Bialeschki, 2010)

bullet

60% professional groups (Henderson & Bialeschki, 2010)

bullet

55% from general interest groups (Henderson & Bialeschki, 2010)

bullet

25% - 35% leisure and recreation studies industry standard (Mitra & Lankford, 1999)

Example of Town of Parker Master Plan Survey Methodology and Response Rate

bullet

City utility customers (N=840) were used to develop the survey sample. 

bullet

Business customers and duplicate entries were deleted from the survey list and only residential customers (N=525) were used. 

bullet

A random sample of 230 addresses were drawn. 

bullet

A stamped, pre-addressed envelope was included within the survey to facilitate survey responses. 

bullet

Survey participants were asked to respond within five days. 

bullet

A total of 56 usable surveys were returned.

bullet

The response rate was 24.3% (56/230). 

bullet

This return rate is comparable to other Arizona city recreation surveys. 

book icon Additional Reading: Town of Parker Recreation Survey
 

Example of Whitewater Motivation Study Response Rate

bullet

Whitewater enthusiasts (N = 724) were identified from the initial list of 788, and were mailed a Letter of Introduction, as well as a Leisure Motivation Questionnaire. 

bullet

Five days after the initial mailing a follow-up postcard was sent. 

bullet

Of the 724 participants identified,

  1. 36 questionnaires were returned as undeliverable by the postal service.

  2. 454 were returned for analyses. 

  3. 16 were unusable (incomplete) leaving a total of 438 responses to be included in the analysis.

bullet

The response rate was 62% (438/724).

Factors that Reduce Return Rate

bullet

incomplete questionnaires

bullet

incorrect addresses (mail surveys)

bullet

questionnaires returned after deadline

bullet

duplicate listings

bullet

ineligible individuals in sample

Problems of Non-response

Low response rates introduce nonresponse error which is more difficult to estimate the the sampling error. Nonresponse creates a question of why these individuals did not reply. This also causes an image problem for the agency that is sponsoring the research.

Some researchers attempt to conduct a follow-up with a sample of nonrespondents to attempt to determine is any differences exist between the answers from respondents and nonrespondents. This would improve the validity of the study and improve it's generalizibility to the larger population.

Incentives to Increase Response Rates

Many researchers include some form of incentive to try to increase their response rate. It may be a one dollar bill included with the survey, a discount at a local business, or a drawing for a free prize.

Sample of Drawing in the Town of Parker Recreation Survey

SPECIAL DRAWING ENTRY FORM 

If you would like to have your name submitted for the Special Drawing for a free season swim pass, please print your name and phone number below. 

Name ______________________________________ 

Phone: _______________________ 

Thank you for your time and interest in the Parks and Recreation Department's efforts to improve our community’s recreation facilities and services. 

 Survey Techniques/Approaches

Self-Administered Questionnaires

bullet

mail

bullet

drop-off/pickup

bullet

group administrations

bullet

contact ahead/mail

bullet

Internet

Interviews

bullet

telephone

bullet

individual personal

bullet

focus group

Respondent Honesty in Survey Responses

bullet

face-to-face interviews = less honest answers

bullet

phone interviews = moderately honest answers

bullet

mail surveys = most honest answers

Multi-Methods

Many survey methodologies will combine several different data collection techniques to get the best response rate. Combinations may include:

bullet

mailing with a phone follow-up

bullet

mailing with a face-to-face interview

bullet

phone with an Internet site reply

2.8 Surveys: Administering Questionnaires and Conducting Telephone Interviews

Pilot test: All instruments should be pilot tested to identify any problems that may exist with the instrument or sample. The pilot test will include administering the instrument to a small sample (5-30 people) that are similar to the participants in the proposed study. The pilot study is also used to test reliability, validity, and usability of the instrument.

Field test: testing the process used to obtain data and not focus on the actual instrument.

Implementing Mailed Questionnaires

Mail - Self Response: A common form of survey research. It requires a carefully developed data collection technique to have a adequate response rate.
These techniques include:
 
  1. have a well written cover letter with a 2 week return date
  2. use first class personalized mailing with a stamped self-addressed envelope
  3. have a simple and concise questionnaire
  4. incentives may increase the response rate
  5. mailed by a date to reach all recipients at least a week before the return deadline.
  6. send a follow-up/reminder letter or postcard, ten days after initial mailing
  7. a second follow-up letter with the questionnaire should be sent out a few days after the initial due date
  8. a final contact/postcard should be mailed about 10 days after the second letter is sent to anyone who has not returned the questionnaire

The goal for the response rate is always 100% but a realistic response rate for a mail survey is 50% - 70%. Response rates below 50% may have a nonsampling or nonrespondent bias.

Administration to Groups

Giving surveys to groups requires creating the groups and giving them the questionnaire. This has a high response rate.
 
  1. provide writing implements
  2. provide a location conducive to taking the questionnaire
  3. provide clear directions an the purpose of the study and what do do when they complete the questionnaire

Telephone Interviews

Telephone interviews costs more than a mail survey but can be completely in a shorter time frame. Has a comparable response rate to face-to-face interviews but a better response rate than mail surveys. Problems associated with phone samples:

  1. it is estimated that 30% of the population no longer use a land line (house phone) and therefore do not appear in any phone book, thus eliminating them from any phone survey. This has serious implications on data collected from any phone survey, using a phone book as the participant sample.

  2. additionally, not everyone has a telephone, and some people prefer not to be listed in the phone directory.

  3. a phone book is not a current source of information, it is only printed once a year.

Random Digit Dialing (RDD)      

Two-digit RDD is the most popular form of RDD. In this case, the phone book is first used to select a phone number; this is followed by randomization of the last two digits by replacing them with a two-digit string of random numbers. For example, if the selected phone number is 123-4567, then the randomized number could be 123-4578, where "67" has been replaced by "78." This process increases the probability of finding a working number at the randomized location, since the number uses not only the prefix, but also the first two digits of an existing phone number, which makes it more likely that there is another working line in that bank of numbers. In other words, if 123-4567 is a working number, then 123-4578 is much more likely also to be a working number than 123-3489. Two-digit RDD thus reduces the likelihood of reaching nonworking numbers and increases the possibility of reaching numbers that are not included in the phone book.

The biggest drawback with the RDD process is the fact that it leads to the dialing of nonworking and nonresidential numbers instead of households. book icon Trust for Public Lands Telephone Survey

2.9 Surveys: Talking About Personal and Group Interviewing

Interviewing attempts to find out an individual's opinions, feelings and intentions through oral communication.

Interviewing Methods

bullet

telephone

bullet

face-to-face (individually or groups)

bullet

Internet

Approaches to Interviewing

  1. Structured close-ended interviews

  2. Standardized open-ended interviews - used to collect qualitative data. structured - semi-structured - unstructured

Structured approach, the interviewer follows the questions exactly as written.

Semi-structured approach, allows the interviewer to probe and ask questions in any order that is appropriate.

Unstructured approach, have no pre-established questions but are more of an informal conversation.

Content of Interviews

  1. contact interviewee via mail, email or phone

  2. interview takes 1/2 to 2 hours

  3. provide private location for the interview

Recording data: response sheet; tape recorder (digital or tape); notes

Focus Interview Groups

A focus group includes people who possess certain characteristics and who provide qualitat6ive data through a focused discussion (Casey, 2009).  A focus group is an organized open public meeting to collect data from individuals or community groups (users, advisory, special interests or stake holders).

Richard Krueger (1988), describe the focus group as a special type of group in terms of purpose, size, composition, and procedures. A focus group is typically composed of seven to twelve participants who are unfamiliar with each other and conducted by a trained interviewer. These participants are selected because they have certain characteristics in common that relate to the topic of the focus group.

The researcher creates a permissive environment in the focus group that nurtures different perceptions and points of view, without pressuring participants to vote, plan, or reach consensus. The group discussion is conducted several times with similar types of participants to identify trends and patterns in perceptions. Careful and systematic analysis of the discussions provide clues and insights as to how a product, service, or opportunity is perceived.

A focus group can be defined as a carefully planned discussion designed to obtain perceptions on a defined area of interest in a permissive, nonthreatening environment. It is conducted with approximately seven to twelve people by a skilled interviewer. The discussion is relaxed, comfortable, and often enjoyable for participants as they share their ideas and perceptions. Group members influence each other by responding to ideas and comments in the discussion.

CHARACTERISTICS OF FOCUS GROUPS

Focus group interviews typically have four characteristics: 

  1. Identify the target market (people who possess certain characteristics);

  2. Provide a short introduction and background on the issue to be discussed;

  3. Have focus group members write their responses to the issue(s);

  4. Facilitate group discussion;

  5. Provide a summary of the focus group issues at the end of the meeting.

Other types of group processes used in human services (Delphic, nominal, planning, therapeutic, sensitivity, or advisory) may have one or more of these features, but not in the same combination as those of focus group interviews.

2.10 Electronic Surveys and Mobile Devices: The Wave of the Present and the Future

Over the past decade, electronic surveys have been the fastest growing method for collecting data.

Types of Electronic Surveys

bullet

Email surveys: low cost, quick and reasonably easy and least intrusive. Sent in the body of the email, as attachment, or link to Internet hyperlink

bullet

Internet survey programs: online commercial programs such as Survey Monkey (www.surveymonkey.com)  and Zoomerang (www.zoomerang.com) have become very popular survey methods.

bullet

Tablets (Android and iPad) are growing in popularity because they are highly portable and require little training to use (QuickTapSurvey, iSurvey, iFormBuidler).

bullet

Cell phones are also being used for short and simple  surveys (iSurvey).

Issues in Electronic Evaluation

  1. lack of human interaction

  2. some loss of control including subject fraud

  3. reluctance to respond if they are not aware of the credibility of the sender

  4. may be viewed as junk mail or spam

  5. issues of privacy and confidentiality

  6. not everyone has access to the Internet

Questionnaires should include four types of elements

  1. words: communicate the question

  2. numbers: assist in moving through the questions

  3. symbols: are useful as arrows or bullets to communicate emphasis

  4. graphics: text boxes, squares, pictures, images and logos help create an interesting  and compelling questionnaire.

Happiness Survey (Extra Credit option) click on the button.

2.11 Observation: On a Clear Day You Can See Forever

Observation methods are not used as often as surveys in park and recreation evaluation but they may offer a useful data source.

Evaluation by judgment through accreditation and standards programs and checklists (maintenance) are common examples of observations.

Types of observations

bullet

checklists

bullet

professional judgment

bullet

fieldwork

bullet

participant observation

Roles of Observers

bullet

nonreactive (outside observer) will remove his/her self completely from any involvement with the individual or group observed.

bullet

pull participant (participant observer) is completely involved in an activity and observing.

bullet

unknown (covert observer) is not identified by the participants.

bullet

known (overt observer) is identified his/her role to the participants.

Observation Procedure

bullet

Choose a behavior or situation to observe

bullet

Decide on mode for recording observations and collecting data (quantitative or qualitative)

bullet

Determine sampling strategies

bullet

Grain observers or practice in the situation

bullet

Analyze data

bullet

Evaluate observation instrument

Quantitative Observations

Checklists or similar recoding scales are the most common quantitative observations.
Checklist strategies include:

bullet

interval sampling, where a series of brief intervals are observed and the evaluator notes what was observed.

bullet

frequency counts, event recordings, or tally methods counts the number of discrete events of a certain type of behavior as they occur at at given time (minute, hour, day, week).

bullet

duration refers to the length of time spent in a particular behavior or how long that behavior occurs.

bullet

latency recording is the time elapsed between a cue and the response.

bullet

rating system record the behavior from a pre-determined criteria (checklist).

bullet

time sampling or spot checking records behavior at a particular point in time.

Professional Judgments are the direct observation and evaluation by an expert or an individual familiar with the specific activity or program.

Qualitative Observations

  1. Anecdotal records and critical incidents are factual descriptions of meaningful incidents/events.

  2. field observations use the evaluator in a "natural" or "normal" situation to serve as a participant-observer.

Note Taking are data collected by recording information by an observer. The less time between the observation and recording the notes, the more accurate the data will be.

2.12 Unobtrusive Methods: Oddball Approaches

Unobtrusive (nonreactive) methods are observing, recording, and analyzing behavior in a situation where interaction with people usually does not occur and people are unaware that they are being observed. This method is most often used to provide supplemental data for evaluation projects or case studies. The anonymity of people being observed is common.

Unobtrusive Methods

bullet

body language and any non-verbal behavior

bullet

physical observation/evidence (social trails)

bullet

graffiti

bullet

amount of garbage

bullet

attendance records at events

bullet

archives (any record or document)

2.13 Experimental Designs: Focusing on Control and Interventions

Two Major Classifications of Evaluation and Research Designs

bullet

Experimental (true experimental designs, quasi-experimental designs)

bullet

Descriptive/evocative (non-experimental)

Types of Evaluation and Research Designs

Pre-Experimental - are not experimental but could be with additions to the design.

Quasi-Experimental - do not have a randomized sample or a control group or do both pretests and posttests.

True Experimental  - have a randomized sample with a control group.

Experimental Notation

bullet

T1 or O1 = Pretest

bullet

T2 or O2 = Post-test

bullet

R = Randomized sample (random means to assign people to groups by chance)

bullet

X = Treatment (independent variable)

bullet

E = group receiving the experimental treatment

bullet

C = control group receiving no treatment

bullet

O = observation

True Experimental Designs

True experimental designs provide the strongest indications for causal effects of the independent variable because of the control and ability to explain or account for extraneous factors.

bullet

Randomized Pretest-Post Test Control Group Design

Randomly Assigned Pretest Treatment Post test
Experimental Group T1 X T2
Control Group T1   T2
  OR
Randomly Assigned Pretest Treatment Post test
Experimental Group #1 T1 Xa T2
Experimental Group #2 T1 Xb T2
Control Group T1   T2
bullet

Randomized Control Group Post Test Only Design

Randomly Assigned Pretest Treatment Post test
Experimental Group   X T2
Control Group    

Quasi-Experimental Designs

Quasi-experimental methods do not satisfy the strict requirements of the true experiment. Real life situations in leisure research create many instances when experimental research is not possible, but some type of causal inference is needed. The purpose of the quasi-experimental design is to approximate the conditions of the true experiment. Quasi-experimental designs will cause compromises in terms of the internal and external validity of the design. The quasi-experimental research design is characterized by methods of partial control of the study. Otherwise, the steps used in quasi-experimental methods are essentially the same as in true experimental methods.

bullet

Nonequivalent Pretest-Post Test Control Group Design

A T1 X T2
B T1    T2
bullet

Single Group Interrupted Time Series Design

T1T2T3 X T4T5T6
bullet

Control Group Time Series Design

Group           Pretest     Treatment     Post Test
Experimental     T1                 X                  T
Control                T1                                               T2
bullet

Single Subject Designs

Baseline Data A - Intervention  - Treatment Data B
                                                 X X X X
T1T2T3T4T5T6T7T8T9T10

Pre-Experimental Designs

The difference between Experimental and Pre-experimental designs is pre-experimental designs, lack control of internal validity.

bullet

Research Design with No Control: One-Group Post-test Only Design (X T2)

bullet

Research Design with Minimal Control: One-Group Pretest/Post-test Design (T1 X T2)

2.14 Specific Applications to recreation: The More the Merrier

Importance -performance (I-P) - survey questionnaires to measure program effectiveness. I-P's are based on the concept that evaluation must be obtained fro the participant.

I-P Process

bullet

Attributes (criteria) what to measure are determined.

bullet

Two sets of questions are developed based on importance and satisfaction of the participant

bullet

Data are collected

bullet

Responses to the two sets of questions are matched and the means plotted on a two-way dimensional grid (importance & satisfaction axis) See text page 230.

bullet

Respondent's perceptions are translated into management action

Case Studies - are used to gather information about individuals or groups (communities, organizations or institutions) and may use many types of data.

Case Study Procedure

bullet

identify the focus of the investigation

bullet

outline what needs to be studied

bullet

select appropriate measurement tools

bullet

develop a plan for collect the data

bullet

collate all the data

bullet

interpret the data

bullet

make recommendations from the study

Single-Subject Techniques - allows the researcher to evaluate the effect of interventions on an individual participant. This is similar to the case study except only one person or a very small group are used as the sample. A series of measurements/observations occur over a period of time to determine how an individual may be changing as a result of a particular program or treatment plan. See text page 235.

System for Observing Play and and Leisure Activity in Youth (SOPLAY) and System for Observing Play and Recreation in Communities (SOPARC) - these observation "checklists" use a time sampling technique where periodic and systematic scans are conducted of areas (ex. playgrounds).

Devices to Measure Physical Activity - a variety of devices are used in research and evaluation to collect objective data and measure physical activity.

bullet

pedometer - counts the number of steps a person has taken.

bullet

accelerometers - measure intensity of body movement and time intervals.

bullet

heart rate monitors - measure physical activity and heartbeats over a specific time.

Economic-Analysis Techniques - economic impact and cost-analysis are commonly used in park and recreation agencies.

Consensus Techniques - nominal group technique and the Delphi technique are examples of consensus strategies.

Nominal Group Technique (NGT) - the NGT is a group discussion structuring technique. It is useful for providing a focused effort on topics. The NGT provides a method to identify issues of concern to special interest groups or the public at large. Ewert (1990) noted that the NGT is a collective decision-making technique for use in park and recreation planning and management. The NGT is used to obtain insight into group issues, behaviors and future research needs.

Five Steps of the NGT 

  1. Members of the group identify their individual ideas in writing, without any group discussion;

  2. Each member lists his/her own ideas and then rank-orders them, again without any group discussion;

  3. A facilitator gives each participant an opportunity to state his/her ideas ( one item per person at a time, in round-robin fashion) until all ideas are exhausted; 

  4. As a group, participants discuss and consolidate ideas into a list; and 

  5. Finally, members vote to select priority ideas. The final list of ideas becomes the focus of further research and discussion. These ideas can also be used to generate a work plan for a formal strategic planning process, a basis for a survey or interview, or the development of a scale.

Source: (Mitra & Lankford, 1999)

Delphi Method - the Delphi method was developed to structure discussions and summarize options from a selected group to:

bullet

avoid meetings, 

bullet

collect information/expertise from individuals spread out over a large geographic area, and

bullet

save time through the elimination of direct contact. 

Although the data may prove to be valuable, the collection process is very time consuming. When time is available and respondents are willing to be queried over a period of time, the technique can be very powerful in identifying trends and predicting future events.

The technique requires a series of questionnaires and feedback reports to a group of individuals. Each series is analyzed and the instrument/statements are revised to reflect the responses of the group. A new questionnaire is prepared that includes the new material, and the process is repeated until a consensus is reached. 

The reading below is a research study that used the delphi technique and content analysis to develop a national professional certification program.

book icon Sample Research Article: Job Competency Analyses of Entry-Level Resort and Commercial Recreation Professionals
 

Sociometry - are used to survey and analyze how groups operate by asking how people "get along?"

Miscellaneous Techniques

bullet

time diaries and time sampling

bullet

visual analyses

bullet

report cards

bullet

service quality - a score is obtained by subtracting the mean expectation rating from the mean performance rating.

2.15 Triangulation or Mixed Modes: Drawing on All the Resources

Using more than one evaluation method may provide additional information to a study.

bullet

within-method triangulation - includes using both qualitative and quantitative data using the same method. An example would be using open-ended and close-ended questions in a questionnaire.

bullet

between-method  (mixed-mode) triangulation - is the use to two or more methods to measure the same phenomena.

2.16 People Aren't All the Same: Considerations for Data Collection

In planning and conducting evaluation, youth, older adults, people with disabilities, people with low education levels, people from particular ethnic and cultural groups, and people from other "vulnerable populations" may require special considerations.

People with Disabilities - may require specific types of accommodations/adaptations based on their disability (physical, cognitive, visual).

Children and Youth - vary depending on their emotional, physical, social, and cognitive development. The child's developmental stage must be taken into account.

Older Adults - have a huge range in their physical and cognitive functions.

Cultural Differences - or cultural competence is required to interact with many cultures when undertaking community evaluation studies. A "culture" is a set of social and learned behavior patterns and beliefs. Evaluators must understand how a culture can influence data collection and interpretation. Culture is not the same as race and ethnicity. Culture includes visible traditions and aspects such as music, dance, food, clothing, language, skin color, and art. Also factors less obvious including religion, history, social class, concepts of life, body language, and use of leisure.

People with Low Socioeconomic Status - are difficult to reach and including them in evaluations are a challenge.

Participatory Inquiry - is focused on how to integrate knowledge with action and how to become co=collaborators in the field. Researchers are not separate neutral data collectors who study others but work together with people to bring about social justice.

Participatory Action Research (PAR) is a popular form of participatory inquiry. I addresses how knowledge can be produced that leads directly to action useful to a group or community. PAR techniques are similar to thee methods such as case studies, but the difference is in how individuals become part of the study team so that the research results in some type of direct social or personal change. The methods are diverse and experimental.

 

 

[Unit 2]

Copyright 2012. Northern Arizona University, ALL RIGHTS RESERVED