- Thank you all for turning out on this
very rainy Monday night.
We are very pleased to have, at Smith,
Professor John Creswell.
He is professor of Educational Psychology
at the University of Nebraska-Lincoln.
In addition to teaching at the university,
is the author of numerous articles
on mixed-methods research,
qualitative methodology,
general research design.
He's also the author of 22 books,
including many new editions,
which focus on research designs,
comparisons of qualitative research methodologies,
and the nature and use of mixed-methods research.
His books are wildly popular,
and have been translated into many languages,
and hat are used around the world.
Now I get to have a little fun.
You all know who Dan Brown is?
"Angels and Demons?"
He's number one on Amazon.
John Creswell, writing a book on
mixed-methods methodology,
is number 1,005.
(wooing)
I have a book that's, like, 4,000,067.
(laughter)
You rarely get to compare your speaker
to the Amazon fame list of Dan Brown
and Stephen King, who are number one
and number two at the minute.
Not only does he have number one,
not only is he 1,005 in books,
but he's number one in
qualitative, quantitative and mixed methods
in the social sciences.
A prior book of Dr. Creswell's,
"Qualitative Inquiry and Research Design:
"Choosing Among Five Approaches,"
which, for a long time, was the most widely read
qualitative research book in the world,
is now number three.
So he's populating the top 10 very fully.
John Creswell held the Clifton Institute
Endowed Professor Chair for five years
at the University of Nebraska-Lincoln.
For the past five years, he's served
as co-director of the
Office of Qualitative and Mixed Methods Research
at Nebraska-Lincoln, and has provided support
for scholars, incorporating qualitative
and mixed methods into research projects
for external funding.
Another big achievement and real area
of leadership is that he has served
as the founding co-editor of the Sage journal
"The Journal of Mixed Methods Research."
He's also been an adjunct professor of family medicine
at the University of Michigan, where he's worked
on investigations in the health sciences and education,
and research methodology for the
National Institutes of Health
and National Science Foundation projects.
He's served extensively as a consultant
in the health sciences research area
for the Veterans Administration.
Dr. Creswell has also been a Senior Fulbright Scholar
to South Africa, and in 2008, lectured
to faculty at five universities on education
and health sciences.
In 2012, he was again a Senior Fulbright Scholar
to Thailand.
In 2011, he served as co-leader
of a national working group for the
National Institutes of Health, developing
best practices for mixed-methods research
in the health sciences.
In the spring of 2013, Dr. Creswell
has served as a visiting professor
at Harvard School of Public Health.
In summer 2013, he'll conduct
mixed-method training workshops
at Cambridge University in the U.K.
In 2014, he'll be awarded an honorary doctorate
from the University of Pretoria in South Africa.
As you can see, he mightily deserves that.
We're very proud to have Dr. John Creswell
with us tonight at Smith.
Dr. Creswell. (applause)
- Thank you. Very good.
Well, thank you for coming out tonight.
Jim, thanks for the great introduction here,
and the dean for your support
to come here,
and my daughter, Joanna Baez, who is
a third-year doctoral student,
who loves the doctoral program here at Smith,
and has just thrived so much.
I appreciate all that Smith has done for her.
She'll be a great asset when she goes out
to the social work community.
I'm going to introduce mixed-methods research tonight
in a way that, hopefully, will be very user friendly
and applied.
I'm going to start
with a very broad question.
What evidence do we use
to really makes sense of our world today?
Now I'm going to bring in an illustration.
This will, perhaps, tap into
your emotions.
It will tap into the affective side.
Hopefully it won't re-traumatize people
in the audience.
It's not my intent.
I'd have to say, when I put together these
next slides, though,
I just felt
a churning of my stomach.
It was very hard to put these together,
but I think it illustrates mixed methods.
So we have the Boston Marathon
on April 15th.
We can look at this event in a number of ways.
The weather was in the 50s.
There were 23,000 runners.
The prize was $108,000.
71 countries.
75%, three-quarters, of the individuals finished.
There were 18 hand cycle contestants
and 52 wheelchair contestants in that.
After the bombs,
which occurred at 2:50 in the afternoon,
there were three people killed, 170 wounded.
Taken to the Brigham Hospital, which was
close to where I had my office,
were 31 individuals.
Mass General had 31.
Boston Medical Center had 23.
Beth Israel had 21.
Boston Children had 10.
The first responders are a group
that I work with, actually, at Harvard,
the Harvard Humanitarian Initiative.
They were some of the first doctors
at these hospitals.
Because they were prepared for dealing
with crisis situations, did a great job
in responding to the people that were injured.
Now, I was at the mile 24 marker.
That morning, I decided not to go down
to the finish line.
24 mile marker was a place,
just as you would expect, two miles
from the finish line.
It was a point where the runners were getting tired
and wanting to drop out.
There must have been a thousand people
at this corner.
You can't see all of the people here
cause this is my picture.
There were about a thousand people.
Someone had set up a couple of big speakers.
They were blasting out this music on the speakers,
"Sweet Caroline,"
Chubby Checker's "Shout."
Imagine standing there with about
a thousand people, half of 'em singing
at the top of their lungs those songs.
So this was a moment of jubilation.
Actually, some of the runners
were doing that YMCA when the
YMCA song came on, you know,
as they were running by.
They were starting to smile.
A few minutes later, I was standing there,
and I watched the people that went
to the 24 mile marker and were asked
to be turned around.
Did you know that?
The police stopped 'em
at mile 24 because they couldn't go on
to miles 25.
I stood there, after this moment of jubilation,
and watched people, you know how
they have those aluminum capes around,
walking back
right through this area.
I've just presented two different kinds
of evidence for you.
One was the numbers,
which was informative, wasn't it?
I'll bet you didn't know all those
wheelchair people were in this race,
or even that there was a prize for over
$806,000.
Then, I told you
my personal story.
So which one did you like the best,
or which one was the best evidence for you?
People might be divided on this question.
In fact, if I just presented the quantitative statistics,
it probably wouldn't tell the whole story.
Or if I just told you my personal story,
without those statistics,
it wouldn't be as complete.
What I've given you here
is a mixed-methods rendering
of the Boston bombing situation,
where both statistics and person experiences
come together.
Are you beginning to understand mixed-methods research?
(audience agrees)
It's a very simple idea.
We don't need to complicate this.
Mixed methods is using qualitative
and quantitative data together
to have a larger understanding
of the problem we're working with.
That each one might give us a partial picture,
but when we put these two pictures together,
we have a more complete understanding.
Here's a formal definition.
It's an approach to investigating behavior,
social and health science problems by collecting
and analyzing
quantitative and qualitative data in response
to research questions, and integrating
or mixing the two to generate
a new and more complete insight
than what would have been obtained
by either than quantitative or the qualitative.
That's the pitch.
I've come to this
position, starting out as a quantitative researcher.
I spent the first 12 years of my life
doing correlational analysis, experimental studies,
survey projects, et cetera.
Then I started teaching qualitative research.
Then I added to that mixed methods
because I really felt that putting
the two approaches together would give us
greater insight.
This book, which first came out in 1994,
had a chapter called
"Combined Quantitative and Qualitative Research."
I got more emails
about that one chapter
than any other part of this book.
I didn't really have a lot to work from
at that point in the early '90s.
I've expanded this to different qualitative approaches,
five approaches here,
11 different approaches here.
This is the mixed methods intro book that I use
in all my workshops.
This is a reader of good studies.
This is an applied educational research book.
I've got six books that are very active now,
that really are beginning to focus
more and more on how you combine
multiple approaches,
how you use different approaches to research,
how you can bring quantitative and qualitative together.
Then I started looking into the social work field.
I've worked with Deborah Padgett down at NYU.
I've noted that she's done some writing here
on qualitative and mixed methods and social work.
"Knowledge Development," which is a guest editorial piece
where she begins to introduce people,
back in 2009,
to mixed-methods research.
I've looked at Jim's article,
Professor Driscoll's article, on
utilization-focused evaluation
of two intensive family preservation programs.
It's an article where he brought in
both qualitative and quantitative data.
He was ahead of his time because
this was in 1998
that he's been thinking about
how you might utilize both forms of data,
both forms of evidence, together.
There are more recent pieces.
This came out in 2012, looking at
group work for social work students
and the types of competencies.
You can see they're calling this a
mixed-methods study.
So now the words mixed methods
are beginning to flow into
the social work articles.
This one looks at school-based social workers,
and really compares those that
are school-hired versus agency-hired,
in terms of their competencies.
You start reading this.
First of all, it's described as a mixed-methods study.
They gathered survey data and they gathered
focus group data, and then they compared
those two databases.
This is a fairly recent article, in 2011.
I began finding some
mixed-methods studies in social work.
This research is being done right in your field.
Just a little bit about
the development of this.
It's often called the third methodological movement.
The first was quantitative, the second was qualitative,
and now mixed methods.
Depending on when
you view quantitative coming in, it definitely
came in around the turn of the century.
Qualitative, you could argue it stretched
way back to
some of the early anthropologists.
In the social sciences,
we often associate qualitative with
the Chicago school case study work
of the 1950s and '60s.
But mixed methods is a much more recent
development.
In 2009, one of my students
did an analysis of doctoral dissertations
using ProQuest.
As you can see here, dissertations
that had mixed methods in the title,
it increased from three, from 1984,
to over 2,500
in 2009.
One of my colleagues, Plano Clark,
says that the number now is over
6,000 dissertations.
Plano Clark also mapped out
the NIH funding,
National Institute of Health funding awards,
that self-identify as mixed methods.
You can see it's an upward climb here
to the number of 226
by 2008.
It's still tending to climb.
So there's interest, tremendous interest
across the country in different disciplines
for mixed-methods research.
It's only 25 years old.
I roughly tag this back to the
1985, 1990 period.
We had different people working.
Alan Bryman was working over in England.
He wrote a book on quantity and quality.
Jennifer Greene, out of the evaluation field
at Illinois, wrote an article
in 1989.
There were about eight people,
including myself, during this 1985, '90 time period,
that were beginning to think about how
we could put qualitative and quantitative together.
I felt it was just a matter of time before
it would develop into a full-blown methodology.
This book came out in 1998, called "Mixed Methodology."
The handbook on mixed-methods research
from Sage first came out in 2003.
That's a little over a decade ago.
Then in 2006,
the large, international Mixed Methods Conference started.
It was sponsored by a couple of universities
in England.
The "Journal of Mixed Methods" came in
in 2007.
In 2011, NIH developed some
best-practice standards for mixed-methods research
across all 17 institutes
and 11 centers,
talking about how to do good
mixed-methods research.
Then the latest event to unfold
is happening right now.
There's the
International Association of Mixed-Methods Research
that's forming that'll be interdisciplinary
and it'll be worldwide.
Those are some of the major events.
The characteristics of this methodology.
I think there are only just a few
that are key ones to look at.
You ask quantitative, qualitative
and mixed-methods
research questions
using open- and close-ended or mixed information.
Collecting and analyzing both quantitative
and qualitative data.
Using rigorous methods of data collection.
Integrating or bringing the two databases together.
We're gonna talk about some of these ways.
Using a specific mixed-methods design,
and often framing the study within
some larger theory or some philosophical assumptions.
You have quantitative research and qualitative.
We're gonna talk about integrating them,
bringing the two together.
Through that process of integration,
we're going to have stronger, a better understanding
of our research problem that what either one
would yield in itself.
Some people will come up say to me,
"I've been doing mixed methods since 1970, John."
What they've been doing is they've been gathering
qualitative and quantitative data,
usually very informally,
not using some of the features I'm gonna
show you here that have evolved in this
methodology of mixed methods,
often not seriously thinking about how
they're gonna integrate the two forms of data.
It's more informal approaches
rather than mixed methods, as we see it today.
Mixed methods is not just
taking your qualitative data and reporting
the results, and your quantitative and reporting 'em,
and keeping 'em separate.
We're gonna talk about integrating the two,
that step.
There are a lot of people that are just using
the word mixed methods
because it's a popular concept.
Somebody mentioned a software program today
where they're billing it as mixed methods,
but they're really not doing much in terms
of the methodology that I'm gonna be talking about.
There's a big controversy around the world
right now, especially in England,
about whether mixed methods is collecting
multiple quantitative forms of data
or multiple qualitative forms of data.
It's certainly multiple research.
We call this multi-method research,
not mixed methods.
If you go to the ICQI Conference,
they are collecting multiple qualitative forms,
such as interviews and observations,
but they're not collecting quantitative data,
so it's really not mixed methods.
It's more multi-method research.
There's this whole field out there
called content analysis.
It's where you do some interviews,
and then you analyze the data quantitatively.
You count how many times people
mention a code or a theme.
In this scenario, you can see
you're collecting qualitative data,
but you're not collecting quantitative data.
When those articles came in to our
"Journal of Mixed Methods Research,"
they were rejected.
We recommended that they be published
in a content analysis type of journal.
There are a lotta people around the country
that thinks mixed methods is just summative,
informative evaluation.
Certainly it could be, but it's much more than that.
It's how you would actually
integrate your qualitative and quantitative data
at both that summative and at the formative stage.
So what's emerged in the last 25 years
are a series of strategies.
I call these strategies that are developed
in the science of a rigorous mixed-methods study.
If you were to do a mixed-methods study
here at Smith, I would be going through
this checklist of 10 features
to see if you've incorporated them.
I think if you have, it would probably
be published in one of the leading journals.
(hum)
Asking, am I getting ahead here?
Here I go.
Asking questions.
We'll talk about framing within theory,
writing a rationale, collecting qualitative
and quantitative data using rigorous procedures,
selecting a mixed-methods design,
drawing a visual of it, adding in procedures
and integration, specifying validity threats
and creating joint displays of the data.
Are you tired yet? (laughter)
I'll go through these very quickly
and give you some illustrations.
In fact, the most important thing for me
is to adapt this to the field of social work.
What I always do in my presentations
is I go into the field that I'm presenting to,
and I look for good mixed-methods studies.
I looked for quite a while.
I found several that I thought
were good mixed-methods projects.
This is one that comes out of Connecticut.
Anne Farrell and Preston.
I understand Preston Britner was here
on the faculty
a few years ago.
He or she
said that he had been at Connecticut
for about 16 years, but prior to that,
they were here in Smith teaching.
This is a good mixed-methods study.
It's a very recent one, 2012.
I found it in the "Child and Family Social Work" journal.
Basically, what they're doing here
is they're talking about a program for families
with a dual vulnerability in housing
and child welfare.
They're very interested in how parents react.
They used a mixed-methods approach.
They gathered quantitative data using
an engagement measure,
which they labeled right here quantitative.
I hope I'm not in the way of anyone here.
Then they used open-ended interviews, qualitative.
They gathered data from 41 parents.
They then converged these two databases.
Then compared the two in terms
of high and low engagement scores.
A mixed-methods project, and it would fit
those characteristics that I talked about earlier,
gathering quantitative data, gathering
qualitative, integrating.
I'm gonna use this as an illustration.
I'm gonna present first their research questions.
The point that I'm trying to make in mixed methods,
what we typically do in mixed methods,
is we state both a quantitative,
a qualitative and a mixed-methods question.
The first question was, what is the level
and quality of client engagement
in this particular program
for individuals that are,
welfare families that are looking for housing?
Would you call this quantitative, qualitative
or mixed methods?
- [Voiceover] - Qualitative.
- What is it? - [Voiceover] Qualitative.
- Qualitative.
Okay, let's see.
We have two more response possibilities here.
(laughter)
- [Voiceover] You're measuring levels.
- Yeah, yeah, it's a measurement type
of question, isn't it?
Looking at the level and quality
of client engagement.
Here's the variable being measured,
and they're gonna measure that
in terms of level and quality.
The second question they posed in this article,
are there trends in engagement across
different program enrollments?
So people enrolled in this program at different times,
and after the program is all over,
then they're doing their study now.
Quantitative, qualitative or mixed?
- [Voiceover] Quantitative.
- Quantitative.
Why did you say that?
- [Voiceover] Because the trends are being measured.
- Right.
Usually in quantitative, we assess trends, don't we?
So knowing some of the terms.
How do clients perceive their relationship
with their case managers, especially
with respect to engagement?
- [Audience] Qualitative.
- That's an easy one, isn't it? Qualitative.
Yes, good, qualitative.
Now look at this one.
Do quantitative measures of engagement
match client narrative reports?
- [Audience] Mixed.
- Mixed.
They're looking at both the measures
and all of these narrative reports.
In fact, that's a very good
mixed-methods question, and it was
probably clear, what you've learned
about mixed methods up to this point.
Here's their last one.
Does the parent engagement measure
effectively capture client engagement?
Quantitative, qualitative or mixed?
- [Audience] Mixed.
- I'm gonna give ya a little hint on this.
Construct validity.
They've studied that in the courses.
This is quantitative.
Is this a valid instrument?
Does it have a good construct validity?
- [Voiceover] Just curious.
I'm really very new at this, but I was wondering
if number one could also,
I got thrown off by the quality of client engagement.
- I know.
- [Voiceover] I wondered if that could be qualitative
because who's deciding what the quality is?
- They probably had some measure of quality,
would be my guess.
That word, but that's important.
You're looking at the wording
in these questions to try to decipher
whether it's quantitative or qualitative.
Just because they say the word quality
doesn't mean it's gonna be qualitative.
They're trying to measure quality.
In fact,
here's a little primer on research questions.
What we hope to learn from qualitative questions.
Individual experiences, person meanings,
individual perceptions.
Let's look for key words in the questions
such as meaning, experiences, exploration,
individual views.
There's probably some other ones,
but you see that's how I was learning
which ones were quantitative and qualitative.
I was looking at their wording very carefully.
There are words that we often associate
with qualitative research.
Quantitative questions.
What we hope to learn
is cause and effect, probable cause and effect,
relationships among variables,
comparing groups, right?
The words that are gonna tip us
in that direction, factors, causes,
measures, determinants, correlations,
trends, level,
magnitude.
We could probably go on, but I was looking
for those types of words to see
whether they were really trying to measure something.
Your mixed-methods questions,
as you've saw, as you perceptively
identified, really combines both
experiences and some type
of quantitative measure,
relationship among variables or between groups.
It's some combination
of this and this.
Number two.
What we want in every mixed-method study
would be qualitative, quantitative
and mixed-methods questions.
The second thing is we need to frame
the study within some theory or philosophy, often.
This is from Crotty's book in 1998.
He has a nice flow chart.
He said, "You know, we have these
"paradigms and worldviews,
"epistemologies, ontologies, methodologies,
"that inform theoretical lenses that we use,
"that then inform our methodology,
"such as mixed methods,
"that then informs
"our method of data collection."
There are different worldviews out there.
In fact, if you look at the handbook
on qualitative research from Denzin and Lincoln,
and track, over the years, through their
different volumes the worldviews,
we know that they're changing.
I think in the latest one they have
critical theory as one of the worldviews.
I especially wanted to bring in,
we bring to research certain
predispositions, certain perspectives
that inform how we're going to do our research.
We may feel that there's a theory that's important,
and so we're most postpositivist.
We're gonna work more deductively.
We're gonna test out that theory.
Or constructivist, working with multiple realities,
or more of a participatory
where we wanna collaborate with our participants.
Or pragmatism,
singular and multiple realities, very practical,
looking at what works.
Starting with a question and asking,
"What data can we collect to answer it?"
So mixed-methods researchers have been
very concerned about philosophy
from the very beginning.
We even went through a historical period
they call the paradigm debate period
of the '90s.
What has surfaced now?
There are a number of people that
are talking about different worldviews
that seem to influence mixed-methods research,
pragmatism.
What works, the importance of the question,
looking at multiple ways of gathering data.
Donna Mertens down at Gallaudet
brings in a transformative
paradigm, where the objective of research
is to begin to address social justice issues
and bring about change.
Joe Maxwell brings in critical realism,
which is both, that reality is both
subjective and objective.
The latest is Burke Johnson's work
on dialectic pluralism, that we bring in
multiple worldviews, and out of this
comes some dialectic.
That's a good social work term, isn't it?
A dialectic development.
Some people have said we can use
several worldviews.
My writing tends to flow along the lines
of matching a worldview to the type of design.
For example, I'm gonna start with a survey.
I'm going to gather survey data.
I'm using a postpositivist perspective.
I followed up focus groups.
I'm using more of a constructivist.
I really have multiple paradigms operating
at different phases in my design.
If you go back to Thomas Kuhn's book,
and I saw that it was on the bookshelf
here at the Smith Bookstore, so you're reading
Thomas Kuhn.
I went back and re-read that last year,
but his idea is that worldview
is really shaped by the scholarly community
that you're in.
Is there a way that social workers
think about problems?
Are there accepted methods
for how you go about gathering data?
Are you socialized as graduate students
in social work to
buy into the community?
That's exactly what Thomas Kuhn was talking about.
You can see there's multiple perspectives
on philosophy
within mixed-methods research.
In terms of theory,
we do use, in mixed-methods research,
social science and behavioral theory
to often inform our work.
Sometimes we use transformative theories.
Often we use community-based participatory theories
that are brought throughout the studies.
For example, one of my doctoral students
in 2009
looked at the theoretical lenses being used
in the mixed-methods projects across
a number of journals.
Some of these are social work,
"Family Relations," "Critical Public Health,"
"Women and Health."
You can see that feminist studies
are very popular in the mixed-methods field.
There's a number that address racial issues.
Then it goes on down from there.
Yes, these different
theoretical lenses, both social and advocacy,
are being used.
Returning to that Farrell article
on the impact of this program.
they do have a passage at the beginning
where they talk about an ecological model of housing.
They refer to this author, and they refer
to family-centered practices.
So they have kind of a conceptual orientation
that flows into the beginning of this project.
Write a rationale for mixed-methods research,
why you're using both quantitative and qualitative.
It's a statement
in a mixed-methods project in which the author
advances the reasons for why, whoops, excuse me,
the reasons for why it's important
to collect it.
Why are you doing mixed-methods research?
Why is this important?
Back when qualitative was new
in the social sciences, you'd always find
a reason documented as to why
this was gonna be an ethnography.
Still today, you have
that pervading the qualitative.
Mixed methods is newer yet, and so
we need a rationale for why we're doing it.
You could say this is the types of problems
that are best suited for a mixed-methods project.
Think about your own study that you're doing
for a moment.
One rationale for using mixed methods
is you want different, multiple perspectives,
or to build a complete understanding.
There are perspectives to be gained
by administering a survey instrument.
There are perspectives from focus groups
or individual interviews.
They'll give you different perspectives.
One would be that in-depth perspective.
The other would be a more general impression.
Sometimes we need to confirm our
quantitative measures with qualitative experiences.
I was with the Massachusetts Department of Health
a few weeks ago.
They were talking about one of the surveys
that they administered across the state
of Massachusetts.
They were saying, "You know, there were really
"some interesting findings that came out
"of that survey."
Of course, I said, "Did you stop there?
"Why didn't you follow up a little bit
"with some qualitative interviews
"or talk to some people?
"Why didn't you confirm your quantitative measures
"with qualitative experiences to find out
"what was really going on?"
I had everyone in that room nodding.
This would be a pretty good idea.
I was framing it from a mixed methods.
Needing to explain quantitative results.
Sometimes these instruments that we have,
we're trying to fit 'em to the people
that we're studying.
They're not very contextually sensitive.
They may not fit the diversity
of the people we're working with.
Let's gather some qualitative data,
and then develop some instruments
or measures or interventions that might
actually work with the people we're studying.
My friends in global health down at Harvard
are all using this idea.
It's because my colleague
that's going to Sierra Leone
can't just drop a depression instrument
into Sierra Leone.
She's gotta start talking to some people
and gathering qualitative data.
Surprise, every one of her mixed-methods projects
begins with a strong qualitative part,
and then builds into that quantitative.
That quantitative is much better because
of that qualitative.
We need to enhance our experiments.
There's been a lot of discussion about
randomized controlled trials
and the problems with that.
We really don't understand the mechanisms.
Why can't we put some qualitative data
in, after the experiment is over,
to understand why that treatment
had such a positive impact,
would explain why it had that impact.
Or do we just stop with p-values,
confidence intervals, effect sizes?
Putting qualitative into an experiment.
I'm gonna show you some models of that.
When we're doing our community studies,
tomorrow I'll be talking to community researchers
in the Boston area.
They are gathering trend data
and individual perspectives,
keeping 'em separate.
They're not thinking about how they might
be linked to get a better understanding
of what those community members' needs might be.
Sometimes we need to evaluate success
of a program by using a needs assessment.
Then actually do a quantitative test
to see the success of the program.
Again, we're putting qualitative
together with quantitative.
Can you see these rationales for why
mixed methods is popular?
I looked at the Farrell article
on engagement in the supportive housing
for family program.
They had a few comments.
"Research has begun to include parent voices."
Then they said, "We then compared and contrast
"quantitative and qualitative findings."
Had they studied my list, maybe they
could think about what would be a good rationale
for this, such as "a complete understanding
"of the parents' experiences with the program
"can be attained through the combination
"of parent voices and assessment of measures
"of their engagement."
You see how I gave a rationale for both
quantitative and qualitative here?
Can be assessed through the voices,
qualitative, and the assessment of measures,
quantitative.
So whenever an article would come into
our "Journal of Mixed Methods Research,"
we'd be looking for a rationale statement.
It's very important.
A lot of people will just drop
quantitative instruments into a study,
and a lot of different qualitative stuff into a study.
It just leads to extra effort
and extra resources.
If you think through your rationale,
you're gonna be able to limit your data collection.
The fourth point, in qualitative data, we wanna collect
both quantitative and qualitative data,
so I came upon this book,
"Clinical Assessment for Social Workers,"
which some of you are probably reading this summer.
I found it to be a very good book,
very interesting.
They talked about quantitative
clinical assessment techniques,
which I'm sure you're familiar with.
Then qualitative clinical assessment techniques,
interviews, life stories, individual case analysis.
You know what I was looking for in this book?
Where they brought the two together.
Towards the front, in all fairness to these authors,
they had what they called an
integrated assessment checklist
where they did have both of 'em there,
but they never really gave us that thought
about how you might combine
these types of data to really feed
into each other to gain greater insight.
Maybe I'll write 'em and say, "In your next edition."
I thought it was an excellent, very nice book.
When we go to the Farrell study,
they collected data
from 41 parents.
They administered that instrument
to 41 parents.
They administered a rating scale
and they did on-on-one interviews.
Yes, they are collecting both forms of data
in this project.
The next thing is, in mixed methods,
it needs to be rigorous.
It needs to have some rigor in your methods.
I put together
John's foolproof list of rigorous methods.
I want something about sampling,
recruitment, sample size,
forms of data collection,
methods of recording data,
methods for transcribing, storing the data,
cleaning up the database, some initial analysis,
some advanced analysis.
I actually put together a checklist like this
for my colleagues in counseling psychology,
which they're using as they write articles,
especially qualitative journal articles.
It's not exhaustive list,
but it's a fair list of rigor.
If you were to put this in on both
the quantitative and qualitative side of your project,
I think you'd have a great doctoral dissertation.
Then I looked at the Farrell article
in terms of its rigor.
They did have information about sampling,
but it was non-probabilistic sampling.
They'd had some purposeful sampling
over on the qualitative side.
They had some recruitment techniques
on both sides.
They had the N of 41.
They had a questionnaire with 22 items
and open-ended questions.
They'd (mumbles) methods.
They didn't talk about an interview protocol.
I would have put that in that article.
They inputted this into SPSS.
They recorded the interviews and transcribed 'em.
They talked about cleaning up the database.
They then reported mean engagement scores.
They talked about Pearson correlations,
site comparisons.
They had a nice table on themes, qualitative themes.
They mentioned that they used grounded theory,
but I don't think they really used much grounded theory.
But kudos to them
for acknowledging the fact
that there are some larger strategies
of qualitative research out there.
When I looked down through this list,
they've got some really good, rigorous procedures.
If you were to read that article,
which was available out in the lobby,
some of you may have gotten a copy of it,
you will see, I think,
very good, sophisticated, rigorous methods
in a social work article.
Now, select a mixed-methods design.
There are a number of designs out there.
It can be very confusing to read some of the books,
but I talk about them in terms of basic
and advanced designs.
We're gonna look for the overall intent
of your design.
We're gonna start with a basic, and add in
more advanced if we need that,
and we're gonna consider some procedures
for the design.
What has happened in the mixed-methods field
is that we have these different ways
of doing mixed-methods research.
Just like in quantitative, we have
different ways of doing quantitative research.
We can do a true experiment.
We can do a quasi-experiment.
We can do single-subject research.
In qualitative, we have different ways
of doing qualitative.
My book on the five approaches,
ethnography, narrative, case study,
phenomenology, et cetera,
different ways of doing it.
So it's no surprise,
as this methodology has evolved,
that we now have different designs
for doing mixed-methods research.
We've given these some names.
The first is convergent.
It's where you gather quantitative data
and analyze it, and qualitative data,
and you bring the two together
and just compare the results.
I'll have to say,
the social work articles I've found
that are mixed methods,
this is your most popular design,
but think about other possibilities.
We can start quantitatively, let's say
with a survey,
come up with some interesting results,
and then follow up qualitatively to explain
those results in more detail.
That design right there
is a graduate student design, ready made.
In my mixed-methods class,
I'll have half my class choose this.
It's very clean.
You've got two phases.
Often people are coming, at least in my program,
from more of a quantitative background,
so starting quantitatively.
Very easy, elegant design.
The exploratory sequential design.
We're gonna first explore qualitatively.
From that qualitative data, we're gonna
build something quantitatively.
My colleague is building
in Sierra Leone a new depression instrument,
or putting in a new intervention,
or even coming up with some new variables
that may not be in the literature.
Then administering that quantitatively.
Mixed methods is not more complicated
than these three designs, for the most part.
If you were to take many of the projects
that are being published across the
social, behavioral, health sciences,
they're gonna fit one of these three designs.
We need to think about the intent here.
What's the intent of bringing these two databases together?
We can see if that interview data
reports similar results to what we found out
quantitatively.
We can confirm one with the other,
or even validate.
That may seem like you've never
heard of that before or thought about that,
but I'm at a project, an NSF project,
over in Boston right now
where they wanted to measure school use
of this mathematic curriculum by teachers.
They searched for almost a year to try
to find the variables that would work.
They finally settled on some that they would measure.
They decided to also gather some qualitative data.
What they were able to do with gathering
qualitative data is to actually refine
those variables and come up with new variables
that are much better.
They were putting these two databases together.
This one, the intent of this one, really,
we know that survey quantitative data
is very important,
but why people checked certain scores
on certain items,
we don't know much more than it was significant
or non-significant, so let's collect some
qualitative data.
Let's use the qualitative to explain
the quantitative results.
This one.
The intent is to first build something
to explore.
Almost all the people that come to me
that have a strong qualitative background,
such as what you've been given here
at Smith, they like this design.
You want to explore first, and then
build something quantitatively.
The intent is to first explore
because we may not know the variables,
we may not understand the population
we're studying, we may not even know
what questions to ask.
So what you intend to accomplish.
There's a certain timing,
your background, what's popular in your field,
what would be easiest for stakeholders
to understand.
What I wanna do is I wanna take these
basic designs and move to three more designs.
What I wanna do is I wanna frame 'em.
I'm gonna frame one of these designs
within an experiment,
within a social justice perspective, and within
an evaluation perspective.
Here's an experiment running.
What these authors did, it's looking
at the management of anti-psychotic medication.
They put qualitative data in before to help
design the intervention,
and they put qualitative in after the experiment
to explain the results.
We call these intervention mixed-methods designs,
really popular.
Here's another one.
This is a study of homeless individuals
and how they move from a hospital
to a shelter,
and what that transition looks like.
These authors, Grayson, they had
a survey instrument that had both closed
and open-ended questions.
That's not very strong on the qualitative side,
I call it mixed-methods light,
but it does have both qualitative
and quantitative data that they were trying
to bring together.
Then what they did was they had different
community input points throughout the project.
In other words, they put a framework around
a convergent design based upon
community participation.
Another framework would be a program evaluation
type of mixed-methods design.
This is a study of mental health needs
of youth in Sri Lanka.
They had several aspects of this project
going forward, from proposing the model
to developing a program to testing theory
and evaluating it.
They had quantitative and qualitative going
throughout these different phases.
We call this a multi-stage design,
large-scale research-funded project.
We can draw a picture of these designs,
and we can add procedures in there.
This is the Farrell study.
They did not have this picture.
It would have been helpful.
The qualitative interviews, qualitative theme analysis,
quantitative questionnaires analysis,
their comparison that went on.
They're integrating, they're bringing these
two databases together right here
at the end of the article, where they're
comparing the qualitative themes
for people that have lower and higher
mean engagement scores.
Can you see how that's integrating the
two forms of data?
One of the things,
there are different forms of integration that
detail to these different designs.
I'm gonna move forward here.
One of the complexities is understanding
the challenges in doing these projects.
By being able to picture the design
and look at how people are conducting 'em
and thinking about what the issues are,
we can actually come up with challenge points in this.
So for example,
you need to start with the similar questions
on both the quantitative and qualitative side
in this convergent design.
Farrell did that,
if you look at their study.
I don't know if they did it consciously,
but it was a great strategy.
If you also notice, they had equal sample size.
There's some controversy whether you need that.
Some people would say just a few people,
qualitatively, would work.
We don't need to generalize.
They began integrating this by bringing together,
in their different integration techniques
that go with this design,
a side-by-side
data transformation and a joint display.
They basically used a joint display discussion,
which I'm going to show you
in a moment.
We look at these designs in terms
of where integration occurs,
then if you find differences to explain
those differences towards the end.
There's one, two, three, four
challenge points in this design.
They didn't talk about any of those,
but if they had known that this
was the design they were working with,
they could have talked about 'em.
One thing that they didn't do, too,
is create a joint display of the results.
A joint display is a table or a graph
in which a researcher arrays quantitative
and the qualitative results together
so that a comparison of results can be made.
One thing that perplexes people
about this convergent design is you've got
numbers and you've got words.
How are you gonna bring the two together?
What they did was they took
the highest and lowest scores, and then
looked at how people talked about them qualitatively.
They did this in a discussion.
I would have put it into a table that's called
a joint-display table that we're getting
a lot of good examples in the
"Journal of Mixed Methods Research."
I created a table for them.
A low mean engagement,
high mean engagement,
and they were looking at the case manager's job.
They had some significant quotes.
You can look down through these quotes
and see if people are talking about
engagement differently
because of their
numeric score.
You could fill out a table like this.
There have been some very creative things
occurring on this front.
For example,
GIS, a geographic information system,
geocoding.
You could have a geocode map with a pullout
of some quotes and themes to illustrate
how people are talking about it.
That's what DataCore is doing right now
in the state of Delaware.
They're a major
research group looking at substance use
across different states.
My evaluation of their study.
I told Farrell, I checked with her.
I said, "I wanted to use your project,"
and I said, "It's really a very, very good project."
It was solid on many accounts.
They had all three types of questions.
They talked about models informing the study.
They stated a general rationale.
You saw my revision, I would have improved it.
They collected and analyzed qualitative
and quantitative data, yes, of course.
Did they use rigorous procedures?
Yes, I went down that checklist.
I thought it was pretty rigorous on both sides.
Did they select a mixed-method design?
Not really explicitly mentioned.
Did they draw a visual diagram of the design?
Really not in the article.
Did they add in procedures and integration
and threats?
Not really.
Did they create a joint display?
Yes, to a certain extent
in their discussion section.
So to me, this article scores actually very high.
I don't know whether they've read
the books mixed-methods research
or read much in the field.
My recommendations for social work,
my last slide.
Consider the full array of designs,
not just convergent designs.
How many of you before today knew
that you could sequence
these phases, qualitative and quantitative,
and thought about that?
A few.
I would suggest, if you wanted to engage
in a mixed-methods project for your first
mixed-methods study as a master's,
doctoral student, I think
this explanatory sequential design,
starting quantitatively and building qualitative
might work quite nicely for you.
I think you ought to think about
this checklist of 10 strategies,
and make sure they're all in your project.
You would have a very rigorous project.
I would recommend that you train
your graduate students in mixed methods.
Down at Johns Hopkins in the School of Public Health,
no, excuse me, the College of Nursing,
they have a qualitative course, and they've got
two units on mixed methods.
Next spring at Harvard in the School of Public Health,
they will have, for their master's-level students,
a completed mixed-methods course
that they're going to offer.
It could be added as an elective.
You might think about coming as a student
to one of these two- or three-day workshops.
Basically, the way it works out,
you bring a project in, and within two
or three days, we go through these steps.
You get feedback, and you have sketched out
a pretty good mixed-methods project.
I did it down at Chapel Hill
last year.
I'll be doing it at Johns Hopkins
this Thursday, Wednesday and Thursday.
I'll be doing it back at Harvard next week.
So do you know a little bit more about
mixed methods than when you came in here tonight?
- [Voiceover] Absolutely.
- How many are going to do a mixed-methods
project?
Almost all the hands went up, Jim. (laughter)
Thank you very much.
Let me take any questions you have.
I know a couple of you were
raising your hands throughout
and looking perplexed.
Barbara.
- [Voiceover] This is not the question I had earlier,
but that's okay.
You didn't address in your presentation
whether one researcher would do both,
or whether, what the difference is when you
have the team
reading the two elements
or one person.
Also, other eyes need to be brought in
and the point when you begin to merge the data?
That's a whole 'nother question, and I don't know if you
can say anything about that.
- I didn't talk about academic teamwork
in doing mixed methods, but
it's a popular subject in the mixed-methods field.
There are some good writings
on how to form a mixed-methods team.
For example,
in this NSF project down at Boston,
over at Boston,
the quantitative researchers are on one side
of the table, the qualitative are on the other side
(laughter) of the table.
You know where I sit?
In the middle. (laughter)
I'm often a translator going back and forth.
It takes good leadership to put together
a team like that.
It takes a certain openness.
Some people have learned one way
of doing their research.
They will stay by it the rest of their life.
Others
like to explore alleys
and take different routes to things.
If you go to the Mixed-Methods Conference,
it's this group of very creative artists.
That's the way I would say it.
These are people who are very open
to exploring possibilities.
Really, we have these problems
that we need to use what I call
our entire toolkit to understand 'em,
I think.
I didn't talk about academic teams.
In terms of one individual,
yes, you need some basic skills
in qualitative data collection analysis
and quantitative data collection analysis,
which I understand you get those
basic skills in your (mumbles) research courses here.
It's just now a matter of putting it together.
Some people may feel that that's calling
on a skill level that's too much.
I would say, in terms of our research across
the country, it's becoming more and more
international, interdisciplinary,
sophisticated.
Mixed methods, right now, is being seen
as the latest methodology,
throughout the health sciences, especially.
It's just going from field to field to field.
I don't know if I answered your question.
What other questions do you have?
Comments?
Jim.
- [Jim] I noticed in one of your tables
that there was the capital Quan
and small quan thing.
Has that pretty much been superseded
by the three design types
as a mode of analysis?
- What came forward in the mixed-methods field
is that if you're going to give emphasis
to either qual or quan,
you capitalize those words.
If you're gonna give them less emphasis,
one of 'em less emphasis, you put it
in small letters.
You'll notice
that qual and quan both had
four characters,
speaking to the issue of equality.
(laughter)
This whole area of emphasis,
to me, is not as strong
as what you intend to do.
It's a soft area.
For example, do I count how many pages
in your mixed-methods study are devoted
to qualitative versus how many pages quantitative?
Do I start, look at how it begins from a
strong postpositivist, quantitative,
theoretical orientation?
How do I really go about assessing
that weight?
When I look at mixed-methods projects,
I do see a different weight,
but I'm not sure that it's something that
is as important as intent.
I have demoted it
in my thinking some,
the weight, the priority.
Another interesting one that's surfaced
is the timing.
If I do my
survey,
and then as soon as you're done
completing that, I have you do an interview,
is that a sequential data collection,
or is it basically a data collection
at the same time?
In the medical field,
these doctors I work with, they're busy people.
They do a lotta convergent design work
because they can get out to the field
once and gather data.
The question of timing,
the question of priority,
are soft dimensions to me now.
I'll ask more, what is the intent?
People are always writing me with these
complicated designs.
Will they have qual or quan,
qual, quan,
all these drawings.
All I do is say, "Okay, what are you
"trying to accomplish?
"Are you trying to get a complete understanding?
"That's more of a convergent.
"Are you trying to explain results?
"What are you trying to accomplish?"
They might be trying to accomplish several things.
Then I say something like,
"What's at the heart of this?"
That Grayson article with the community participatory,
they had a lot going on in that project,
but at that heart of it was trying to merge
the quantitative and qualitative data
from that survey instrument.
The intent was to really get a more
complete understanding.
Some of these issues, for me, yes,
have kind of fallen not by the wayside,
but into
a lesser importance.
That's a good technical question.
Yes, sir?
- [Voiceover] Yes. (clears throat)
You had shown one type
of research that might be social justice oriented.
Could you give an example of that
or flesh out
what that might look like?
- A feminist mixed-methods study.
What would that look like?
First of all,
the feminist perspective.
We've studied this, and I've got some good articles
that would illustrate this.
The feminist perspective would flow into
the study in different places.
It would help shape the problem
at the beginning.
It would shape the theoretical orientation.
It would shape the data collection
where you don't want to further marginalize
the women that you're studying.
It would shape the outcomes.
Call for action.
What we have done in the mixed-methods field,
in analyzing the feminist mixed-methods projects,
is to be curious about how the feminist perspective
flows into different phases.
You could do an explanatory sequential
feminist study.
The feminist perspective would shape the problem.
It would probably dictate
the theoretical orientation
that you bring in in your quantitative first phase.
It would then shape, perhaps,
what qualitative follow-ups you would do
to explain
some of the injustices.
It would shape the outcomes.
I actually have a couple slides on this
where I show where
the feminist perspective has flown into
an explanatory sequential type of design.
But remember, I used social justice.
I said it's a framework that surrounds
one of these basic designs.
That's the way I look at it.
It helps to shape,
but still at the heart of that project
is an explanatory
sequential project.
The basic design is always in the center,
and then the advanced,
the social justice would be a framework around it.
Whether it's
a racial study,
a cultural study,
a critical theory.
Critical theory could be a framework
surrounding one of these basic designs as well.
What do you think about training in?
I'll take your question.
Then I wanna ask you what do you think
about training in mixed methods.
- [Voiceover] I was wondering if you
wouldn't mind telling us some about
what brought you to be so convinced
and passionate about these methods?
- That's a good question.
I started as a quantitative researcher,
then started teaching qualitative,
and then started going back and forth
between the two.
I had a doctoral proposal development class
where
some of the students were coming
with quantitative projects, and some were coming
with qualitative.
I would say, "Okay, tonight we're gonna
"talk about designing a hypotheses.
"Then following that, we're gonna talk about
"a good qualitative, open-ended question."
That got me thinking about
there's some crossover going there.
Back in 1991,
I saw a paper being presented
at the American Educational Research Conference,
AERA.
This was a doctoral student.
She had labeled it mixed-methods research.
I got so excited.
I said, "Oh, I'm gonna go to that."
I went to the session.
You know how many people were there?
- [Voiceover] The two of you?
- Two of us.
The student and myself. (laughter)
I said to her, I said, "You know,
"this is the methodology of the future."
She thought I was a crazy professor. (laughter)
I was just so excited about this.
I like the idea of working on
the frontiers of a new methodology,
and one that would really
bridge
our community or researchers, rather than
the division of quantitative and qualitative
we often have.
I see this as a bridging.
I see this as building a larger community
than either quantitative or qualitative
separately.
Then I think that 1994 book on research design
that had that one chapter on combining
qualitative and quantitative
really got me thinking about the potential
of that.
That book was used
in the undergraduate program
in a liberal arts college that my son was attending.
People were writing to me from Harvard,
they were using it in their doctoral program at Harvard.
I thought that there's a methodology here
that would transcend levels.
I've learned that.
I often get the question, is mixed methods
so rigorous,
so taxing,
that it's only for the best and brightest
of doctoral students?
But you know, you don't have to do
a full-blown quantitative and a
full-blow qualitative.
It could be a smaller part.
Do a survey with a few interviews.
I was in South Africa, and I was called upon
to be a judge
of the best research project among
the undergraduate students.
Out of the five students that presented that day,
four undergraduates were doing
mixed-methods research.
They were gathering both forms of data.
I think it's an approach
that transcends levels,
it transcends fields.
It's very practical.
I use the example of the Boston bombing,
but you could turn to CNN on any night,
and they are doing mixed-methods broadcasts
where they report the figures about Sandy,
and then they're interviewing people.
They're doing mixed-methods broadcasts.
Al Gore's documentary
on the
crisis in our environment,
if you look at his documentary, it's a
mixed-methods documentary.
That practical part reaches out to me.
It's something that, I think it's very intuitive.
It's not learning some statistical procedure.
It's not learning
the details of
how to do a good one-on-one interview.
It's something that we see in our daily life.
That's a long answer.
Other questions or comments?
Is this what you expected to hear tonight?
Just kind of an introduction
to mixed methods,
and also to make it user friendly
so you could see where you could do
a type of project.
Some of you were thinking about
your own studies, how they could be shaped
in that direction right now, weren't you?
Good.
- [Jim] Jack Creswell, thank you very much for
filling us in (applause).
- Thank you.
Thank you.
(applause)
Thank you.