Dependability: is when the evaluator keeps a record of how data were collected (called an audit trail) and how conclusions were drawn.

Validity: addresses whether the instrument measures the intended criteria (what it is supposed to measure). Data can be reliable without being valid.

There are two primary criteria for evaluating the validity of an experimental design. 

Usability: if the instrument cannot be effectively administered and if the data collected are not analyzed and applied the project is useless. Usability factors include:

Trustworthiness: data are only as good and the means used to collect them. Evaluators should eliminate error as much as possible. Errors may come from the respondents, investigator, or sampling procedure. See table on page 120.

2.4 What are the Chances? Choosing a Sample

Sampling procedures vary depending on the methods used and the resources available.

Sampling Categories:

1. probability sampling - means that everyone within a population has the same potential of being selected as part of the sample. Random sampling is the most common type of probability sampling and is superior to non-probability sampling.

2. non-probability sampling - samples where not everyone has an equal chance and the likelihood of getting selected from a population is unknown.

3. theoretical sampling - used primarily with qualitative data and refers to sampling until the evaluator reaches a point of data saturation.  This means no new data that contribute to the emerging themes are evident.

Key to Appropriate Sampling 

Population:  refers to all the people who might comprise a particular group (zip code, age, gender, activity preference etc.) of people whose opinions are sought. In most cases, this group is so large and diverse that it is impossible to ask each member of the group every question so it becomes necessary to select a sample. If an entire population (called a convenience sample) is used then there is no sampling.

Randomization: a subset of a population chosen in such a way that all individuals of the specified population have equal probability of being selected.

Probability: the probability of an occurrence of any event ranges from .00 (no chance that the event will occur) to 1.00 (event is certain to happen). Researchers are interested in determining the probability of an event.

Before a sample can be obtained the universe (all members of the group ie., all city residents, all members of the hiking club, all 6th graders, etc.) must be identified.

Types of Probability Samples

Types of Non-probability Samples

Population Lists and Concerns

Potential lists include: phone books, utility customers, registered voters, hospital or school records, drivers license records, etc.. Remember, not everyone has a phone, is a registered voter, or has a drivers license and therefore would be excluded from those lists.

List Concerns

Representative Sampling

After the population has been identified it is imperative that any sample represents the entire population. The table below shows the proportion of ethnic groups in the Town of Parker.

Any sampling done, would need to include the same proportions of each ethnic group. The same concept applies to other socio-demographic characteristics such as age, gender, income level, education, etc.. Representation also is strongly related to probability.

Probability and Sampling Error

Sampling error: is the difference between the characteristics of a sample and the characteristics of the population from which the sample was selected (Babbie, 2006). Standard error is a function of sample size. As a sample size increases, the standard error decreases.

Non-sampling error: these have no simple and direct method of estimating. Non-sampling errors concern the biases that may exist due to who answers a survey compared to those people who did not. Errors may also come from sampling operations, non-interviews, participants not understanding the ideas being measured, lack of knowledge by the respondents, concealment of the truth, loaded questions, processing errors and interviewer errors.

A sample is a randomly selected subgroup of the population and therefore is not an exact representation. It is an estimate that reflects the larger population. This "estimate" is known as the "sampling error," and can be measured. The sampling error calculates the range of doubt, or degree of confidence, reflected in the sample measurement. 

Two Steps of the Sampling Process

1. determine the kind of sample needed (population, data collection, etc.)

2. determine the appropriate sample selection process.

Types of Samples and Sample Selection Process

Sample Selection Factors

Determining Sample Size

The size of a sample will have a significant impact on the validity of the study. Primarily, sample size is determined by the degree of error that is acceptable to the researcher. Error can be calculated based on the sample size. It is usually expected to be within 3% to 5% for social sciences.

Additional Reading: Calculating Sample Size and Margin of Error or page 123 of the text.

2.5 Choosing the Right Stuff: Measurement Instruments

After the criteria to examine and a research design has been selected, the next step is to create or identify a measuring instrument. For qualitative data the researcher is usually the instrument used (ask questions or make observations). For quantitative data a written instrument, observation checklist, questionnaire or test is necessary. If a valid and reliable instrument already exists that is a preferable choice to creating a new instrument. Researchers have two choices:

1. Evaluate existing instruments to determine if they are reliable, valid and usable.

2. Develop a new instrument and conduct the necessary process to confirm the new instrument is reliable, valid, and usable.

Measuring Instruments

2.6 On Your Own Again: Developing Measurement Instruments

If a satisfactory instrument does not exist you may have to develop your own instrument.

Steps in Development of a Questionnaire

1. Define the problem (see criteria)

2. Determine the contents and ideas to be examined

3. Identify and categorize the respondents

4. Develop items, structure and format

5. Write directions

6. Pilot test, revise, field test, and revise

7. Ensure response

A survey is a method to collect data, a questionnaire is the tool/instrument used to conduct a survey.

Five Areas Addressed in Developing a Question

1. experience/behavior

2. opinion/values (needs or interests)

3. feelings (emotional responses)

4. knowledge/facts

5. background/demographics

Question Structures

Three major structures and the scales that can be sued wit each, include the following:

Sample Close-Ended Likert Type Scale

Wording Issues

1. Use only one idea per question.

2. Clear, brief, and simple are the rules for good questions.

3. Avoid leading questions that suggest that you want a particular kind of response.

4. Avoid asking people to make estimates.

5. Think about trying to formulate a question from the respondent's viewpoint.

6. Avoid advanced language and technical terms that is familiar to you but the respondent's may not understand.

7. Be clear about what is being asked.

8. Avoid negative questions that may con fuse the reader.

9. Avoid the use of "iffy" words like "often" or "seldom."

10. Do a pilot study.

11. State the alternatives precisely.

12. Some questions may require several stages to get adequate information.

Formats and Instrument Design

The general appearance may be critical to the success of a survey. Simple graphics and "white space" are important visual effects.

Questionnaire Format

2.7 Surveys: The Winner of the Popularity Contest

The survey is the most common method used in park and recreation evaluation.

Response Rate: is the percentage of questionnaires received based on the number that were distributed. The formula for calculating the response rate is:

Number of Completed Questionnaires Returned (divided by) Number of questionnaires distributed = Response Rate

Example: 20 questionnaires returned and usable divided by 25 questionnaires distributed = 80% response rate

Acceptable and Achievable Response Rates

Example of Town of Parker Master Plan Survey Methodology and Response Rate

Example of Whitewater Motivation Study Response Rate

1. 36 questionnaires were returned as undeliverable by the postal service.

2. 454 were returned for analyses. 

3. 16 were unusable (incomplete) leaving a total of 438 responses to be included in the analysis.

Factors that Reduce Return Rate

Problems of Non-response

Low response rates introduce nonresponse error which is more difficult to estimate the the sampling error. Nonresponse creates a question of why these individuals did not reply. This also causes an image problem for the agency that is sponsoring the research.

Some researchers attempt to conduct a follow-up with a sample of nonrespondents to attempt to determine is any differences exist between the answers from respondents and nonrespondents. This would improve the validity of the study and improve it's generalizibility to the larger population.

Incentives to Increase Response Rates

Many researchers include some form of incentive to try to increase their response rate. It may be a one dollar bill included with the survey, a discount at a local business, or a drawing for a free prize.

Sample of Drawing in the Town of Parker Recreation Survey

SPECIAL DRAWING ENTRY FORM  If you would like to have your name submitted for the Special Drawing for a free season swim pass, please print your name and phone number below.  Name ______________________________________  Phone: _______________________  Thank you for your time and interest in the Parks and Recreation Department's efforts to improve our community’s recreation facilities and services.

 Survey Techniques/Approaches

Self-Administered Questionnaires

	mail
	drop-off/pickup
	group administrations
	contact ahead/mail
	Internet

Interviews

	telephone
	individual personal
	focus group

Respondent Honesty in Survey Responses

	face-to-face interviews = less honest answers
	phone interviews = moderately honest answers
	mail surveys = most honest answers

Multi-Methods

Many survey methodologies will combine several different data collection techniques to get the best response rate. Combinations may include:

	mailing with a phone follow-up
	mailing with a face-to-face interview
	phone with an Internet site reply

2.8 Surveys: Administering Questionnaires and Conducting Telephone Interviews

Pilot test: All instruments should be pilot tested to identify any problems that may exist with the instrument or sample. The pilot test will include administering the instrument to a small sample (5-30 people) that are similar to the participants in the proposed study. The pilot study is also used to test reliability, validity, and usability of the instrument.

Field test: testing the process used to obtain data and not focus on the actual instrument.

Implementing Mailed Questionnaires

Mail - Self Response: A common form of survey research. It requires a carefully developed data collection technique to have a adequate response rate. These techniques include:   have a well written cover letter with a 2 week return date use first class personalized mailing with a stamped self-addressed envelope have a simple and concise questionnaire incentives may increase the response rate mailed by a date to reach all recipients at least a week before the return deadline. send a follow-up/reminder letter or postcard, ten days after initial mailing a second follow-up letter with the questionnaire should be sent out a few days after the initial due date a final contact/postcard should be mailed about 10 days after the second letter is sent to anyone who has not returned the questionnaire

The goal for the response rate is always 100% but a realistic response rate for a mail survey is 50% - 70%. Response rates below 50% may have a nonsampling or nonrespondent bias.

Administration to Groups

Giving surveys to groups requires creating the groups and giving them the questionnaire. This has a high response rate.   provide writing implements provide a location conducive to taking the questionnaire provide clear directions an the purpose of the study and what do do when they complete the questionnaire

Telephone Interviews

Telephone interviews costs more than a mail survey but can be completely in a shorter time frame. Has a comparable response rate to face-to-face interviews but a better response rate than mail surveys. Problems associated with phone samples: it is estimated that 30% of the population no longer use a land line (house phone) and therefore do not appear in any phone book, thus eliminating them from any phone survey. This has serious implications on data collected from any phone survey, using a phone book as the participant sample. additionally, not everyone has a telephone, and some people prefer not to be listed in the phone directory. a phone book is not a current source of information, it is only printed once a year. Random Digit Dialing (RDD)       Two-digit RDD is the most popular form of RDD. In this case, the phone book is first used to select a phone number; this is followed by randomization of the last two digits by replacing them with a two-digit string of random numbers. For example, if the selected phone number is 123-4567, then the randomized number could be 123-4578, where "67" has been replaced by "78." This process increases the probability of finding a working number at the randomized location, since the number uses not only the prefix, but also the first two digits of an existing phone number, which makes it more likely that there is another working line in that bank of numbers. In other words, if 123-4567 is a working number, then 123-4578 is much more likely also to be a working number than 123-3489. Two-digit RDD thus reduces the likelihood of reaching nonworking numbers and increases the possibility of reaching numbers that are not included in the phone book. The biggest drawback with the RDD process is the fact that it leads to the dialing of nonworking and nonresidential numbers instead of households. Trust for Public Lands Telephone Survey

2.9 Surveys: Talking About Personal and Group Interviewing

Interviewing attempts to find out an individual's opinions, feelings and intentions through oral communication.

Interviewing Methods

	telephone
	face-to-face (individually or groups)
	Internet

Approaches to Interviewing

1. Structured close-ended interviews

2. Standardized open-ended interviews - used to collect qualitative data. structured - semi-structured - unstructured

Structured approach, the interviewer follows the questions exactly as written.

Semi-structured approach, allows the interviewer to probe and ask questions in any order that is appropriate.

Unstructured approach, have no pre-established questions but are more of an informal conversation.

Content of Interviews

1. contact interviewee via mail, email or phone

2. interview takes 1/2 to 2 hours

3. provide private location for the interview

Recording data: response sheet; tape recorder (digital or tape); notes

Focus Interview Groups

A focus group includes people who possess certain characteristics and who provide qualitat6ive data through a focused discussion (Casey, 2009).  A focus group is an organized open public meeting to collect data from individuals or community groups (users, advisory, special interests or stake holders).

Richard Krueger (1988), describe the focus group as a special type of group in terms of purpose, size, composition, and procedures. A focus group is typically composed of seven to twelve participants who are unfamiliar with each other and conducted by a trained interviewer. These participants are selected because they have certain characteristics in common that relate to the topic of the focus group.

The researcher creates a permissive environment in the focus group that nurtures different perceptions and points of view, without pressuring participants to vote, plan, or reach consensus. The group discussion is conducted several times with similar types of participants to identify trends and patterns in perceptions. Careful and systematic analysis of the discussions provide clues and insights as to how a product, service, or opportunity is perceived.

A focus group can be defined as a carefully planned discussion designed to obtain perceptions on a defined area of interest in a permissive, nonthreatening environment. It is conducted with approximately seven to twelve people by a skilled interviewer. The discussion is relaxed, comfortable, and often enjoyable for participants as they share their ideas and perceptions. Group members influence each other by responding to ideas and comments in the discussion.

CHARACTERISTICS OF FOCUS GROUPS

Focus group interviews typically have four characteristics: 

1. Identify the target market (people who possess certain characteristics);

2. Provide a short introduction and background on the issue to be discussed;

3. Have focus group members write their responses to the issue(s);

4. Facilitate group discussion;

5. Provide a summary of the focus group issues at the end of the meeting.

Other types of group processes used in human services (Delphic, nominal, planning, therapeutic, sensitivity, or advisory) may have one or more of these features, but not in the same combination as those of focus group interviews.

2.10 Electronic Surveys and Mobile Devices: The Wave of the Present and the Future

Over the past decade, electronic surveys have been the fastest growing method for collecting data.

Types of Electronic Surveys Email surveys: low cost, quick and reasonably easy and least intrusive. Sent in the body of the email, as attachment, or link to Internet hyperlink Internet survey programs: online commercial programs such as Survey Monkey (www.surveymonkey.com)  and Zoomerang (www.zoomerang.com) have become very popular survey methods. Tablets (Android and iPad) are growing in popularity because they are highly portable and require little training to use (QuickTapSurvey, iSurvey, iFormBuidler). Cell phones are also being used for short and simple  surveys (iSurvey).

Issues in Electronic Evaluation

1. lack of human interaction

2. some loss of control including subject fraud

3. reluctance to respond if they are not aware of the credibility of the sender

4. may be viewed as junk mail or spam

5. issues of privacy and confidentiality

6. not everyone has access to the Internet

Questionnaires should include four types of elements

1. words: communicate the question

2. numbers: assist in moving through the questions

3. symbols: are useful as arrows or bullets to communicate emphasis

4. graphics: text boxes, squares, pictures, images and logos help create an interesting  and compelling questionnaire.

Happiness Survey (Extra Credit option) click on the button.

2.11 Observation: On a Clear Day You Can See Forever

Observation methods are not used as often as surveys in park and recreation evaluation but they may offer a useful data source.

Evaluation by judgment through accreditation and standards programs and checklists (maintenance) are common examples of observations.

Types of observations

	checklists
	professional judgment
	fieldwork
	participant observation

Roles of Observers

	nonreactive (outside observer) will remove his/her self completely from any involvement with the individual or group observed.
	pull participant (participant observer) is completely involved in an activity and observing.
	unknown (covert observer) is not identified by the participants.
	known (overt observer) is identified his/her role to the participants.

Observation Procedure

	Choose a behavior or situation to observe
	Decide on mode for recording observations and collecting data (quantitative or qualitative)
	Determine sampling strategies
	Grain observers or practice in the situation
	Analyze data
	Evaluate observation instrument

Quantitative Observations

Checklists or similar recoding scales are the most common quantitative observations.
Checklist strategies include:

	interval sampling, where a series of brief intervals are observed and the evaluator notes what was observed.
	frequency counts, event recordings, or tally methods counts the number of discrete events of a certain type of behavior as they occur at at given time (minute, hour, day, week).
	duration refers to the length of time spent in a particular behavior or how long that behavior occurs.
	latency recording is the time elapsed between a cue and the response.
	rating system record the behavior from a pre-determined criteria (checklist).
	time sampling or spot checking records behavior at a particular point in time.

Professional Judgments are the direct observation and evaluation by an expert or an individual familiar with the specific activity or program.

Qualitative Observations

1. Anecdotal records and critical incidents are factual descriptions of meaningful incidents/events.

2. field observations use the evaluator in a "natural" or "normal" situation to serve as a participant-observer.

Note Taking are data collected by recording information by an observer. The less time between the observation and recording the notes, the more accurate the data will be.

2.12 Unobtrusive Methods: Oddball Approaches

Unobtrusive (nonreactive) methods are observing, recording, and analyzing behavior in a situation where interaction with people usually does not occur and people are unaware that they are being observed. This method is most often used to provide supplemental data for evaluation projects or case studies. The anonymity of people being observed is common.

Unobtrusive Methods

	body language and any non-verbal behavior
	physical observation/evidence (social trails)
	graffiti
	amount of garbage
	attendance records at events
	archives (any record or document)

2.13 Experimental Designs: Focusing on Control and Interventions

Two Major Classifications of Evaluation and Research Designs

	Experimental (true experimental designs, quasi-experimental designs)
	Descriptive/evocative (non-experimental)

Types of Evaluation and Research Designs

Pre-Experimental - are not experimental but could be with additions to the design.

Quasi-Experimental - do not have a randomized sample or a control group or do both pretests and posttests.

True Experimental  - have a randomized sample with a control group.

Experimental Notation

	T1 or O1 = Pretest
	T2 or O2 = Post-test
	R = Randomized sample (random means to assign people to groups by chance)
	X = Treatment (independent variable)
	E = group receiving the experimental treatment
	C = control group receiving no treatment
	O = observation

True Experimental Designs

True experimental designs provide the strongest indications for causal effects of the independent variable because of the control and ability to explain or account for extraneous factors.

Randomized Pretest-Post Test Control Group Design	Randomly Assigned Pretest Treatment Post test Experimental Group T1 X T2 Control Group T1   T2
	OR Randomly Assigned Pretest Treatment Post test Experimental Group #1 T1 Xa T2 Experimental Group #2 T1 Xb T2 Control Group T1   T2
Randomized Control Group Post Test Only Design	Randomly Assigned Pretest Treatment Post test Experimental Group   X T2 Control Group

Quasi-Experimental Designs

Quasi-experimental methods do not satisfy the strict requirements of the true experiment. Real life situations in leisure research create many instances when experimental research is not possible, but some type of causal inference is needed. The purpose of the quasi-experimental design is to approximate the conditions of the true experiment. Quasi-experimental designs will cause compromises in terms of the internal and external validity of the design. The quasi-experimental research design is characterized by methods of partial control of the study. Otherwise, the steps used in quasi-experimental methods are essentially the same as in true experimental methods.

Nonequivalent Pretest-Post Test Control Group Design	A T1 X T2 B T1    T2
Single Group Interrupted Time Series Design	T1T2T3 X T4T5T6
Control Group Time Series Design	Group           Pretest     Treatment     Post Test Experimental     T1                 X                  T Control                T1                                               T2
Single Subject Designs	Baseline Data A - Intervention  - Treatment Data B                                                  X X X X T1T2T3T4T5T6T7T8T9T10

Pre-Experimental Designs

The difference between Experimental and Pre-experimental designs is pre-experimental designs, lack control of internal validity.

	Research Design with No Control: One-Group Post-test Only Design (X T2)
	Research Design with Minimal Control: One-Group Pretest/Post-test Design (T1 X T2)

2.14 Specific Applications to recreation: The More the Merrier

Importance -performance (I-P) - survey questionnaires to measure program effectiveness. I-P's are based on the concept that evaluation must be obtained fro the participant.

I-P Process

	Attributes (criteria) what to measure are determined.
	Two sets of questions are developed based on importance and satisfaction of the participant
	Data are collected
	Responses to the two sets of questions are matched and the means plotted on a two-way dimensional grid (importance & satisfaction axis) See text page 230.
	Respondent's perceptions are translated into management action

Case Studies - are used to gather information about individuals or groups (communities, organizations or institutions) and may use many types of data.

Case Study Procedure

	identify the focus of the investigation
	outline what needs to be studied
	select appropriate measurement tools
	develop a plan for collect the data
	collate all the data
	interpret the data
	make recommendations from the study

Single-Subject Techniques - allows the researcher to evaluate the effect of interventions on an individual participant. This is similar to the case study except only one person or a very small group are used as the sample. A series of measurements/observations occur over a period of time to determine how an individual may be changing as a result of a particular program or treatment plan. See text page 235.

System for Observing Play and and Leisure Activity in Youth (SOPLAY) and System for Observing Play and Recreation in Communities (SOPARC) - these observation "checklists" use a time sampling technique where periodic and systematic scans are conducted of areas (ex. playgrounds).

Devices to Measure Physical Activity - a variety of devices are used in research and evaluation to collect objective data and measure physical activity.

	pedometer - counts the number of steps a person has taken.
	accelerometers - measure intensity of body movement and time intervals.
	heart rate monitors - measure physical activity and heartbeats over a specific time.

Economic-Analysis Techniques - economic impact and cost-analysis are commonly used in park and recreation agencies.

Consensus Techniques - nominal group technique and the Delphi technique are examples of consensus strategies.

Nominal Group Technique (NGT) - the NGT is a group discussion structuring technique. It is useful for providing a focused effort on topics. The NGT provides a method to identify issues of concern to special interest groups or the public at large. Ewert (1990) noted that the NGT is a collective decision-making technique for use in park and recreation planning and management. The NGT is used to obtain insight into group issues, behaviors and future research needs.

Five Steps of the NGT 

1. Members of the group identify their individual ideas in writing, without any group discussion;

2. Each member lists his/her own ideas and then rank-orders them, again without any group discussion;

3. A facilitator gives each participant an opportunity to state his/her ideas ( one item per person at a time, in round-robin fashion) until all ideas are exhausted; 

4. As a group, participants discuss and consolidate ideas into a list; and 

5. Finally, members vote to select priority ideas. The final list of ideas becomes the focus of further research and discussion. These ideas can also be used to generate a work plan for a formal strategic planning process, a basis for a survey or interview, or the development of a scale.

Source: (Mitra & Lankford, 1999)

Delphi Method - the Delphi method was developed to structure discussions and summarize options from a selected group to:

	avoid meetings,
	collect information/expertise from individuals spread out over a large geographic area, and
	save time through the elimination of direct contact.

Although the data may prove to be valuable, the collection process is very time consuming. When time is available and respondents are willing to be queried over a period of time, the technique can be very powerful in identifying trends and predicting future events.

The technique requires a series of questionnaires and feedback reports to a group of individuals. Each series is analyzed and the instrument/statements are revised to reflect the responses of the group. A new questionnaire is prepared that includes the new material, and the process is repeated until a consensus is reached. 

The reading below is a research study that used the delphi technique and content analysis to develop a national professional certification program.

Sample Research Article: Job Competency Analyses of Entry-Level Resort and Commercial Recreation Professionals
 

Sociometry - are used to survey and analyze how groups operate by asking how people "get along?"

Miscellaneous Techniques

	time diaries and time sampling
	visual analyses
	report cards
	service quality - a score is obtained by subtracting the mean expectation rating from the mean performance rating.

2.15 Triangulation or Mixed Modes: Drawing on All the Resources

Using more than one evaluation method may provide additional information to a study.

	within-method triangulation - includes using both qualitative and quantitative data using the same method. An example would be using open-ended and close-ended questions in a questionnaire.
	between-method  (mixed-mode) triangulation - is the use to two or more methods to measure the same phenomena.

2.16 People Aren't All the Same: Considerations for Data Collection

In planning and conducting evaluation, youth, older adults, people with disabilities, people with low education levels, people from particular ethnic and cultural groups, and people from other "vulnerable populations" may require special considerations.

People with Disabilities - may require specific types of accommodations/adaptations based on their disability (physical, cognitive, visual).

Children and Youth - vary depending on their emotional, physical, social, and cognitive development. The child's developmental stage must be taken into account.

Older Adults - have a huge range in their physical and cognitive functions.

Cultural Differences - or cultural competence is required to interact with many cultures when undertaking community evaluation studies. A "culture" is a set of social and learned behavior patterns and beliefs. Evaluators must understand how a culture can influence data collection and interpretation. Culture is not the same as race and ethnicity. Culture includes visible traditions and aspects such as music, dance, food, clothing, language, skin color, and art. Also factors less obvious including religion, history, social class, concepts of life, body language, and use of leisure.

People with Low Socioeconomic Status - are difficult to reach and including them in evaluations are a challenge.

Participatory Inquiry - is focused on how to integrate knowledge with action and how to become co=collaborators in the field. Researchers are not separate neutral data collectors who study others but work together with people to bring about social justice.

Participatory Action Research (PAR) is a popular form of participatory inquiry. I addresses how knowledge can be produced that leads directly to action useful to a group or community. PAR techniques are similar to thee methods such as case studies, but the difference is in how individuals become part of the study team so that the research results in some type of direct social or personal change. The methods are diverse and experimental.

 

 

[Unit 2]

Copyright 2012. Northern Arizona University, ALL RIGHTS RESERVED