Tim D'Annecy

Graduate student in North Carolina

layout: page

title: Class notes - INLS 581 (Research methods)


Table of contents

  • TOC GEN {:toc}

November 9, 2016

The professor started the class by discussing qualitative data and how to code this data.

Coding

Coding is a way to index and categorize data so that major data is organized together. There are a lot of specific methods to do this, but in general, most qualitative coding involves…

  1. An initial pass through your data. Make notes as you go and at the end, group cases into categories, look for themes and issues, and see what is represented. The professor noted this stage is often very open-ended.

  2. Go through the data again. Check the time, mark the text, underline and highlight, label text with descriptive or meta narrative comments, etc.

  3. Go through the data a third time. In this stage, you would systematically code the text using the labels developed in stage 2. After reviewing the codes and eliminating duplicates, think of which codes go together nicely (axial codes) and join them. This stage could have hundreds of code words.

  4. Relate theoretical ideas into the text and codes. While coding is only one part, now you have to take the time to interpret what you have found, connecting things to each other and also linking them to larger research questions.

Examples of codes include…

  • Setting/context

  • Perspectives held by subjects

  • Subjects’ ways of thinking

  • Process and activity codes

  • Strategy codes

Codebooks

The professor noted that codebooks are critical for understanding codes and for allowing others to understand your coding techniques. To demonstrate this, the class broke off to work on coding a block of example text.

Homework

The professor noted that moving forward, this course will move towards a more math-heavy focus and statistical analyses will be required.

October 24, 2016

The professor started the class by discussing research interviews and data collection for interviews. There are three distinct types of interviews.

  1. Unstructured interviews - Often used in conjunction with participant observation. Questions are generated on the fly in response to the interviewee’s narration or observed behavior.

  2. Structured interviews - Considered its own distinct mode of surveying. There are pre-coded response categories, with no deviation from the interview script. These are also typically analyzed quantitatively. Some strengths from this include a degree of control, a high level of reliability, these interviews are often faster, and, if done correctly, there is a minimal impact from the researcher. Some downsides include that there is a lack of depth in the answers, there is the possibility that you may miss concepts and answers may be partial. Additionally, there is often a lack of context and clarification that can make conclusions inaccurate later.

  3. Semi-structured interviews - More standardization than unstructured interviews and there is less rigidity than structured interviews. These are used when research questions do not have simple or brief answers, when the researcher expects that respondents may need to explain their answers or give examples, or when context is critical to the understanding of the gathered information.

In semi-structured interviews, the researcher has an interview schedule, which will guide the interview but not dictate it. Initially, there is an attempt to establish rapport with the respondent. The ordering of the questions in SS interviews is often less important and the interviewer is more free to probe interesting areas that arise, following up with previous statements and is often more free to jump around.

Interview questions

There are three types of questions:

  1. Open - What is it like at the library?

  2. Directive - Tell me about your work at the library.

  3. Reflective - It sounds like you think that the library is a good place to work.

  4. Closed - Do you like the library where you work? In most cases, the professor noted, these are the worst kind of questions and should be avoided.

Other types of questions also can be used for interviews. They don’t always have to lead to verbal responses, including talk-based interviews and drawing interviews.

Role of the interviewer

The interviewer has to listen carefully, adjusting the interview direction in response to interview context. Further, the interviewer must question, probe, and adjust the flow of the conversation by using appropriate/timely questions, monitoring and controlling directness of their questions, comments, gestures, and actions, guard against interpretation, filling in the blanks, and giving advice or passing judgment, and in all cases, must keep the conversation focused.

October 5, 2016

The professor started the class by discussing differences between mixed methods, qualitative methods, and quantitative methods.

September 26, 2016

The professor started the class by discussing qualitative methods.

Myths about qualitative research

  • If words are used or produced as data in a project, it’s qualitative research.

    • Truth: Qualitative DATA is not qualitative RESEARCH
  • Qualitative studies all use similar data collection and analysis techniques

    • Truth: Qualitative research” is an umbrella term that encompasses MANY specific methods/methodologies.
  • Qualitative research is less rigorous than quantitative research.

    • Qualitative research aims for “disciplined subjectivity”, there are MANY quality checks involved

Qualitative research is designed to discern how humans understand, experience, interpret, and produce the social world. Emphasis is on rich description, actor’s POV, context naturalism, and cases (vs. variables)

Naturalistic research

There are several concepts that make up the naturalistic approach:

  • Takes place in real-world settings

  • The research does not attempt to manipulate the phenomenon of interest

  • Within this broad approach, many different data collection methods can be used.

    • Observation (direct or participatory)

    • Interviews (open ended or other method)

    • Analysis of artifacts and other existing contexts.

Observation

In this type of qualitative research, the researcher’s role may be complete participant (researcher status unknown to participants), observer-as-participant, participant-as-observer, or complete observer.

  • Pros: Researcher gains first-hand experience and can record information as it occurs; unusual aspect or aspects that are “invisible” to participants can be observed; allows for exploration of sensitive topics.

  • Cons: The researcher may be seen as intrusive; some information not appropriate to report; quality of observations depend on researcher’s experience level.

Interviews

Content analysis of Artifacts and existing content

This type can include public and private documents, photographs, videos, art objects, software, film, etc.

  • Pros: It gives the researcher access to language/words/creative expression of respondents; can be accessed any time; no transcription necessary in some cases.

  • Cons: May be difficult to access and interpret; materials may be incomplete, inauthentic, or inaccurate

Ethnography

Ethnographic research is a subcategory of naturalistic research. It was originally developed in anthropology as an approach to interpreting cultures and explaining how everyday events and details of experience in a particular setting and time create “webs of meaning for members of the culture”. It can be applied to a wide variety of qualitative studies, so this term can cover a wide variety of studies.

The professor noted that ethnography gives you a clear, narrow focus or you can do an open-ended exploration of the issues. There are trade-offs between time and resources and the depth of insight. There are also differences between participant and non-participant approaches. There are also many ethical issues that come into the discussion, including confidentiality and anonymity, informed consent, conditions placed on access, and an observation of questionable behavior.

“Going native”

A term that was created during colonialism and is derogatory towards primitive cultures that represent earlier stages of human evolution. Now, the term is “over-rapport” and refers to when a researcher loses objectivity and ability to critically analyze the culture being studies. This concept is actually a problem that is under debate in the field.

September 14, 2016

The professor started the class by breaking students up into interest groups and encouraging them to share several things about their project:

  • Research question

  • Population of interest

  • Chosen sampling method

Share these items with the class, and share any challenges you encountered or realizations you had during the process of developing a research question and sampling plan.

My group agreed to do convenience sampling and

Our research question is:

How do special collections reconcile intellectual and physical arrangement?

Additionally, our challenges when choosing our research question, we had issues with the huge variety of fields, materials, organizations, and other things that fall under special collections. We founded we needed to be highly specific and narrow down our group and take a non-random sample to make sure we had an appropriate scope.

Research quality

Sometimes, the question comes up, “Why not use the same criteria for all studies?” The professor noted that while it seems simple (all research is different), but it is more a philosophical discussion that was explained as a dichotomy between the positivist (quantitative) paradigm and the naturalist (qualitative) paradigm.

Trustworthiness

COPY THESE NOTES FROM THE SLIDES

Truth-value: Internal validity/credibility

How can one establish confidence in the “truth” of the findings of a particular inquiry for the subjects (respondents) with whom, and the context in which, the inquiry was carried out?

For quantitative studies: Are we actually measuring what we think we’re measuring? For experiments, can we confidently attribute observed changes to our intervention?

For qualitative studies: Do our data adequately reflect the concept we’re studying?

Improving truth-values

To improve the truth values in your studies, it’s important to pay attention to several factors.

COPY THESE NOTES FROM THE SLIDES

September 12, 2016

The professor started the class by discussing Lab Assignment #1, which was due today in class or by midnight.

Sampling part 2

The professor then started by talking about sampling techniques with an eye towards example situations. In this situation, the professor gave a short paragraph of text and asked the groups to evaluate whether a simple random sample, a systematic sample, a stratified sample, or a cluster sample would be the most appropriate for the situation. The class agreed that either the stratified or cluster sample are the best options, but it depends on what the research question would entail. If she’s interested in looking at which age range is doing best or what the overall opinion is among a specific sex, it would make sense to take a stratified sample and use that to look into her question. If she wants to look at how each branch is doing, she could take a cluster sample of all of the branches or cluster those by region.

The professor noted that stratified samples have smaller numbers of groups while cluster samples tend to be larger numbers of groups. In her opinion, she thinks that a simple random sample would be best to address this population and to analyze the information.

Non-probability sampling

Any sampling technique that does not meet the criteria for probability (random) sampling. Sometimes the goal is still representativeness, but not always.

Purposive sampling

Defined as elements that are purposefully chosen because of some characteristic.

  • Typical cases - Looks at the full group and tries to select an approximate average or what the usual person is.

  • Maximize variability - Focuses on getting the most varied group of people to sample.

  • Minimize variability - Focuses on a minority or small group to find out information about that group. (e.g. “Why are people sad? What do they think?”)

  • Extreme cases or opposite views - Focuses on the outliers of the groups.

Quota sampling

Defined as determining which characteristics are of interest and set a quota for each level of that characteristic.

Similar to stratified sampling, but the participants are not randomly selected.

  • Snowball sampling - The initial participants identify more participants. The professor noted that in many cases, the social networks of these people are what motivates them to hand out the materials.

  • Convenience sampling - Choosing elements because they are easy to access. In this situation, the professor noted, this is why the most information is known about college-aged people because it’s easier to choose from the large pool in colleges.

Advantages and Drawbacks of non-probability sampling

Advantages

  • Flexible

  • Quicker and cheaper - you don’t need to purchase a sampling frame.

  • Can be precisely tailored to research goals.

  • Generally the preferred method of sampling when looking at small populations.

  • In many cases, there are higher participation rates.

Drawbacks

  • There is a greater risk for bias

  • There is no way to assess certain statistical measures

  • May not be possible to appropriate or generalize (if this is your goal)

Interest group activity

The professor then broke the class up into their interest groups and projected a slide:

Earlier this semester, you started to develop a research question in small groups based on some problems of practice you had identified in your fields. In those groups, discuss:

  • What types of “elements” would you be observing and analyzing - people, books, schools, etc.?

  • Do you want to be able to generalize to a larger population that you can reasonably study, and if so, what is that population?

  • Does a sampling frame exist for that population? If not, what could you use instead?

  • What sampling techniques might you use to conduct this study? What are the benefits and drawbacks of each?

My group agreed that we would only focus on special collections with the population focus on archivists. From this, we could construct a sampling frame of special collections, starting with a convenience sample of all of the institutions of collections within the area. We reasoned that we could find some of these institutions from listservs.

Additionally, we reasoned that we could only focus on primary source materials that are physical objects.

September 7, 2016

The professor started the class by noting that the discussion of sampling is one of the most important topics to discuss when writing research papers or theses. In today’s class, we will talk about probability sampling.

Why sample?

To begin to answer the question, the professor discussed the question, “Why even sample at all?” The class noted that to make a confident and accurate summary of a population or to prove or propose a useful theory, you must need the best sample group that you can do.

The class then answered the question, “What is the use of a study that doesn’t aim to generalize beyond its sample?” The class noted several reasons, mainly relating to the point that the studies that do this are normally for exploratory studies. In many cases where they don’t generalize, doesn’t mean they can’t.

To make this point, the professor projected a picture of a group of various “coconut” people. It was noted that sample frames are lists of statistics or information about the population in which you are studying. In this case, the image is not a sample frame. The frame itself is an abstraction of this image. Population parameters are the actual values or proportions in the population that are calculated from the sampling frame. Often (usually) this is not known for your variable of interest, but it can be derived through various sampling methods.

Two sampling methods include probability (random) sampling or non-probability sampling.

  • Probability sampling means that every individual has a known, non-zero probability of being selected. Probability sampling depends on a sampling frame and must have one. In this type of use, all selection of random.

  • Non-probability sampling is any sampling method that isn’t random sampling.

In all cases, the professor noted, that unless you already know the exact population parameters, you can never be entirely sure that you have a truly representative sample.

Simple random sampling

To do this, all you have to do is generate random numbers and to select the people who correspond with this number. When the sampling frame is compared with the actual parameters, there is a relatively good ratio when looked from top-down, but there are a lot of people missing. It was noted that this isn’t the best when you’re looking at representation.

Pros:

  • Simple!

  • Meets statistical test requirements

Cons:

  • Small subgroups are not always represented

  • Possible inadvertent clustering

Systematic sampling

To do this, generate one random number to start on, then take every nth of the population. For instance, you role a 1d20 and select every 18th person in your list. The professor noted that in the end, you won’t have a representative sample if the sampling frame has some sort of periodic pattern, but you’ll have results, nonetheless. The professor noted that, with all of these probability samples, “Because you’re taking a random sample, you have a good chance of finding a random bad sample.”

Pros:

  • Simple

  • Ensures even coverage of a sampling frame

Cons:

  • Hidden periodicity can create large errors

  • Inadvertent clustering is still possible.

Comparing stratified and cluster sampling

Stratified sampling is good for few subgroups and its implied that there is homogeneity within these subgroups. Heterogeneity between subgroups. Also, there must be random selection within each subgroup.

Cluster sampling is good for many subgroups, heterogeneity within subgroups, homogeneity among subgroups, and random sampling between groups.

Stratified sampling

Divide thee population into strata, or groups, and then poll those groups. It was noted that it’s important that the strata you choose should have some relationship to the study concepts or goals. If this concept is not taken into account, the group divisions are meaningless and can only distort your final outcomes.

The professor noted that stratified sampling frames are more expensive because they usually demand more work and more determinations to separate into the groups. Because of this, they aren’t used as often and when they are, tend to be for larger studies or from larger institutions that can afford it.

Pros:

  • Less errors

  • Ensures all subgroups are represented

Cons:

  • More expensive

  • Sub grouping information can be impossible to obtain

Cluster sampling

To use this technique, you take an initial sampling unit of groups (in some cases, they can be arbitrary). Then, taking these groups, you can then survey individuals in them. The professor noted that this sampling technique can skew your results and give you an inaccurate and non-representative survey of your population.

In the example on the board, if one of the groups is all men and another group is all women, you’ve already mis-represented your population because it appears that there are way more men than women. The sample frame differs completely from the population frame.

Pros:

  • Reduces cost

Cons:

  • High error if subgroups are different from each other

  • Statistical analysis is more difficult because the math changes due to weighted clusters

Non-responses and “dead to me” subjects

The professor noted that some individuals do not respond to your survey from the onset or drop off halfway through. The importance of this is that if the non-responses are representative of your remaining sample and you still have an adequate sample size, the study will not be negatively impacted. The professor noted it is often impossible to know how these people feel about your survey, but looking for commonalities that link them together can give you actionable information.

August 31, 2016

Grounding the project: Theories and models/literature reviews

The professor started the class by discussing a framework for analyzing academic literature and to define what “theory” means. Following the intro, the class broke off into groups to discuss Kulthau’s article from 2012, Information search process.

Questions posed to the class include:

  • How would you describe this theory? Is it descriptive, predictive, explanatory, or some combination of these?

  • What are some of the key concepts of this theory? How do they relate to each other?

  • Does this theory have any testable propositions? If so, how might you test them?

  • What could this theory help LIS researchers do or understand? What could it help practitioners understand?

What is a theory?

Before getting into what a theory is, the professor discussed several interconnected and related terms that are important to know beforehand.

  • A paradigm is defined as a broad, foundational assumption shared by essentially all researchers in a field.

  • A metatheory ia a theory concerned with the investigation, analysis, or description of theory itself; ideas about how concepts in a field should be thought about and researched.

  • A theory is a system of assumptions, accepted principles, and rules of procedure devised to analyze, predict, or otherwise explain the nature or behavior of a specific set or phenomena. A theory appears in abstract general terms and generates more specific hypotheses.

  • A model is a tentative ideational structure used as a testing device.

Group work

The class then broke off into groups again to discuss “possible study topics” from our discussion on Monday.

Intellectual vs. physical presentation of archival materials

Related disciplines:

  • Physical conservation

  • User expectations

  • Ethical considerations

Where to search:

  • Existing collections

  • Conversations with archivists

Search term suggestions:

  • Presentation

  • Cooperation between collections

  • Contextual relationship

  • Artifact integrity

  • Best practices

Ideas for narrative

  • The group did not fill out this information