Graduate student in North Carolina
title: Class notes - INLS 581 (Research methods)
The professor started the class by discussing qualitative data and how to code this data.
Coding is a way to index and categorize data so that major data is organized together. There are a lot of specific methods to do this, but in general, most qualitative coding involves…
An initial pass through your data. Make notes as you go and at the end, group cases into categories, look for themes and issues, and see what is represented. The professor noted this stage is often very open-ended.
Go through the data again. Check the time, mark the text, underline and highlight, label text with descriptive or meta narrative comments, etc.
Go through the data a third time. In this stage, you would systematically code the text using the labels developed in stage 2. After reviewing the codes and eliminating duplicates, think of which codes go together nicely (axial codes) and join them. This stage could have hundreds of code words.
Relate theoretical ideas into the text and codes. While coding is only one part, now you have to take the time to interpret what you have found, connecting things to each other and also linking them to larger research questions.
Examples of codes include…
Perspectives held by subjects
Subjects’ ways of thinking
Process and activity codes
The professor noted that codebooks are critical for understanding codes and for allowing others to understand your coding techniques. To demonstrate this, the class broke off to work on coding a block of example text.
The professor noted that moving forward, this course will move towards a more math-heavy focus and statistical analyses will be required.
The professor started the class by discussing research interviews and data collection for interviews. There are three distinct types of interviews.
Unstructured interviews - Often used in conjunction with participant observation. Questions are generated on the fly in response to the interviewee’s narration or observed behavior.
Structured interviews - Considered its own distinct mode of surveying. There are pre-coded response categories, with no deviation from the interview script. These are also typically analyzed quantitatively. Some strengths from this include a degree of control, a high level of reliability, these interviews are often faster, and, if done correctly, there is a minimal impact from the researcher. Some downsides include that there is a lack of depth in the answers, there is the possibility that you may miss concepts and answers may be partial. Additionally, there is often a lack of context and clarification that can make conclusions inaccurate later.
Semi-structured interviews - More standardization than unstructured interviews and there is less rigidity than structured interviews. These are used when research questions do not have simple or brief answers, when the researcher expects that respondents may need to explain their answers or give examples, or when context is critical to the understanding of the gathered information.
In semi-structured interviews, the researcher has an interview schedule, which will guide the interview but not dictate it. Initially, there is an attempt to establish rapport with the respondent. The ordering of the questions in SS interviews is often less important and the interviewer is more free to probe interesting areas that arise, following up with previous statements and is often more free to jump around.
There are three types of questions:
Open - What is it like at the library?
Directive - Tell me about your work at the library.
Reflective - It sounds like you think that the library is a good place to work.
Closed - Do you like the library where you work? In most cases, the professor noted, these are the worst kind of questions and should be avoided.
Other types of questions also can be used for interviews. They don’t always have to lead to verbal responses, including talk-based interviews and drawing interviews.
The interviewer has to listen carefully, adjusting the interview direction in response to interview context. Further, the interviewer must question, probe, and adjust the flow of the conversation by using appropriate/timely questions, monitoring and controlling directness of their questions, comments, gestures, and actions, guard against interpretation, filling in the blanks, and giving advice or passing judgment, and in all cases, must keep the conversation focused.
The professor started the class by discussing differences between mixed methods, qualitative methods, and quantitative methods.
The professor started the class by discussing qualitative methods.
If words are used or produced as data in a project, it’s qualitative research.
Qualitative studies all use similar data collection and analysis techniques
Qualitative research is less rigorous than quantitative research.
Qualitative research is designed to discern how humans understand, experience, interpret, and produce the social world. Emphasis is on rich description, actor’s POV, context naturalism, and cases (vs. variables)
There are several concepts that make up the naturalistic approach:
Takes place in real-world settings
The research does not attempt to manipulate the phenomenon of interest
Within this broad approach, many different data collection methods can be used.
Observation (direct or participatory)
Interviews (open ended or other method)
Analysis of artifacts and other existing contexts.
In this type of qualitative research, the researcher’s role may be complete participant (researcher status unknown to participants), observer-as-participant, participant-as-observer, or complete observer.
Pros: Researcher gains first-hand experience and can record information as it occurs; unusual aspect or aspects that are “invisible” to participants can be observed; allows for exploration of sensitive topics.
Cons: The researcher may be seen as intrusive; some information not appropriate to report; quality of observations depend on researcher’s experience level.
This type can include public and private documents, photographs, videos, art objects, software, film, etc.
Pros: It gives the researcher access to language/words/creative expression of respondents; can be accessed any time; no transcription necessary in some cases.
Cons: May be difficult to access and interpret; materials may be incomplete, inauthentic, or inaccurate
Ethnographic research is a subcategory of naturalistic research. It was originally developed in anthropology as an approach to interpreting cultures and explaining how everyday events and details of experience in a particular setting and time create “webs of meaning for members of the culture”. It can be applied to a wide variety of qualitative studies, so this term can cover a wide variety of studies.
The professor noted that ethnography gives you a clear, narrow focus or you can do an open-ended exploration of the issues. There are trade-offs between time and resources and the depth of insight. There are also differences between participant and non-participant approaches. There are also many ethical issues that come into the discussion, including confidentiality and anonymity, informed consent, conditions placed on access, and an observation of questionable behavior.
A term that was created during colonialism and is derogatory towards primitive cultures that represent earlier stages of human evolution. Now, the term is “over-rapport” and refers to when a researcher loses objectivity and ability to critically analyze the culture being studies. This concept is actually a problem that is under debate in the field.
The professor started the class by breaking students up into interest groups and encouraging them to share several things about their project:
Population of interest
Chosen sampling method
Share these items with the class, and share any challenges you encountered or realizations you had during the process of developing a research question and sampling plan.
My group agreed to do convenience sampling and
Our research question is:
How do special collections reconcile intellectual and physical arrangement?
Additionally, our challenges when choosing our research question, we had issues with the huge variety of fields, materials, organizations, and other things that fall under special collections. We founded we needed to be highly specific and narrow down our group and take a non-random sample to make sure we had an appropriate scope.
Sometimes, the question comes up, “Why not use the same criteria for all studies?” The professor noted that while it seems simple (all research is different), but it is more a philosophical discussion that was explained as a dichotomy between the positivist (quantitative) paradigm and the naturalist (qualitative) paradigm.
COPY THESE NOTES FROM THE SLIDES
How can one establish confidence in the “truth” of the findings of a particular inquiry for the subjects (respondents) with whom, and the context in which, the inquiry was carried out?
For quantitative studies: Are we actually measuring what we think we’re measuring? For experiments, can we confidently attribute observed changes to our intervention?
For qualitative studies: Do our data adequately reflect the concept we’re studying?
To improve the truth values in your studies, it’s important to pay attention to several factors.
COPY THESE NOTES FROM THE SLIDES
The professor started the class by discussing Lab Assignment #1, which was due today in class or by midnight.
The professor then started by talking about sampling techniques with an eye towards example situations. In this situation, the professor gave a short paragraph of text and asked the groups to evaluate whether a simple random sample, a systematic sample, a stratified sample, or a cluster sample would be the most appropriate for the situation. The class agreed that either the stratified or cluster sample are the best options, but it depends on what the research question would entail. If she’s interested in looking at which age range is doing best or what the overall opinion is among a specific sex, it would make sense to take a stratified sample and use that to look into her question. If she wants to look at how each branch is doing, she could take a cluster sample of all of the branches or cluster those by region.
The professor noted that stratified samples have smaller numbers of groups while cluster samples tend to be larger numbers of groups. In her opinion, she thinks that a simple random sample would be best to address this population and to analyze the information.
Any sampling technique that does not meet the criteria for probability (random) sampling. Sometimes the goal is still representativeness, but not always.
Defined as elements that are purposefully chosen because of some characteristic.
Typical cases - Looks at the full group and tries to select an approximate average or what the usual person is.
Maximize variability - Focuses on getting the most varied group of people to sample.
Minimize variability - Focuses on a minority or small group to find out information about that group. (e.g. “Why are people sad? What do they think?”)
Extreme cases or opposite views - Focuses on the outliers of the groups.
Defined as determining which characteristics are of interest and set a quota for each level of that characteristic.
Similar to stratified sampling, but the participants are not randomly selected.
Snowball sampling - The initial participants identify more participants. The professor noted that in many cases, the social networks of these people are what motivates them to hand out the materials.
Convenience sampling - Choosing elements because they are easy to access. In this situation, the professor noted, this is why the most information is known about college-aged people because it’s easier to choose from the large pool in colleges.
Quicker and cheaper - you don’t need to purchase a sampling frame.
Can be precisely tailored to research goals.
Generally the preferred method of sampling when looking at small populations.
In many cases, there are higher participation rates.
There is a greater risk for bias
There is no way to assess certain statistical measures
May not be possible to appropriate or generalize (if this is your goal)
The professor then broke the class up into their interest groups and projected a slide:
Earlier this semester, you started to develop a research question in small groups based on some problems of practice you had identified in your fields. In those groups, discuss:
What types of “elements” would you be observing and analyzing - people, books, schools, etc.?
Do you want to be able to generalize to a larger population that you can reasonably study, and if so, what is that population?
Does a sampling frame exist for that population? If not, what could you use instead?
What sampling techniques might you use to conduct this study? What are the benefits and drawbacks of each?
My group agreed that we would only focus on special collections with the population focus on archivists. From this, we could construct a sampling frame of special collections, starting with a convenience sample of all of the institutions of collections within the area. We reasoned that we could find some of these institutions from listservs.
Additionally, we reasoned that we could only focus on primary source materials that are physical objects.
The professor started the class by noting that the discussion of sampling is one of the most important topics to discuss when writing research papers or theses. In today’s class, we will talk about probability sampling.
To begin to answer the question, the professor discussed the question, “Why even sample at all?” The class noted that to make a confident and accurate summary of a population or to prove or propose a useful theory, you must need the best sample group that you can do.
The class then answered the question, “What is the use of a study that doesn’t aim to generalize beyond its sample?” The class noted several reasons, mainly relating to the point that the studies that do this are normally for exploratory studies. In many cases where they don’t generalize, doesn’t mean they can’t.
To make this point, the professor projected a picture of a group of various “coconut” people. It was noted that sample frames are lists of statistics or information about the population in which you are studying. In this case, the image is not a sample frame. The frame itself is an abstraction of this image. Population parameters are the actual values or proportions in the population that are calculated from the sampling frame. Often (usually) this is not known for your variable of interest, but it can be derived through various sampling methods.
Two sampling methods include probability (random) sampling or non-probability sampling.
Probability sampling means that every individual has a known, non-zero probability of being selected. Probability sampling depends on a sampling frame and must have one. In this type of use, all selection of random.
Non-probability sampling is any sampling method that isn’t random sampling.
In all cases, the professor noted, that unless you already know the exact population parameters, you can never be entirely sure that you have a truly representative sample.
To do this, all you have to do is generate random numbers and to select the people who correspond with this number. When the sampling frame is compared with the actual parameters, there is a relatively good ratio when looked from top-down, but there are a lot of people missing. It was noted that this isn’t the best when you’re looking at representation.
Meets statistical test requirements
Small subgroups are not always represented
Possible inadvertent clustering
To do this, generate one random number to start on, then take every nth of the population. For instance, you role a 1d20 and select every 18th person in your list. The professor noted that in the end, you won’t have a representative sample if the sampling frame has some sort of periodic pattern, but you’ll have results, nonetheless. The professor noted that, with all of these probability samples, “Because you’re taking a random sample, you have a good chance of finding a random bad sample.”
Ensures even coverage of a sampling frame
Hidden periodicity can create large errors
Inadvertent clustering is still possible.
Stratified sampling is good for few subgroups and its implied that there is homogeneity within these subgroups. Heterogeneity between subgroups. Also, there must be random selection within each subgroup.
Cluster sampling is good for many subgroups, heterogeneity within subgroups, homogeneity among subgroups, and random sampling between groups.
Divide thee population into strata, or groups, and then poll those groups. It was noted that it’s important that the strata you choose should have some relationship to the study concepts or goals. If this concept is not taken into account, the group divisions are meaningless and can only distort your final outcomes.
The professor noted that stratified sampling frames are more expensive because they usually demand more work and more determinations to separate into the groups. Because of this, they aren’t used as often and when they are, tend to be for larger studies or from larger institutions that can afford it.
Ensures all subgroups are represented
Sub grouping information can be impossible to obtain
To use this technique, you take an initial sampling unit of groups (in some cases, they can be arbitrary). Then, taking these groups, you can then survey individuals in them. The professor noted that this sampling technique can skew your results and give you an inaccurate and non-representative survey of your population.
In the example on the board, if one of the groups is all men and another group is all women, you’ve already mis-represented your population because it appears that there are way more men than women. The sample frame differs completely from the population frame.
High error if subgroups are different from each other
Statistical analysis is more difficult because the math changes due to weighted clusters
The professor noted that some individuals do not respond to your survey from the onset or drop off halfway through. The importance of this is that if the non-responses are representative of your remaining sample and you still have an adequate sample size, the study will not be negatively impacted. The professor noted it is often impossible to know how these people feel about your survey, but looking for commonalities that link them together can give you actionable information.
The professor started the class by discussing a framework for analyzing academic literature and to define what “theory” means. Following the intro, the class broke off into groups to discuss Kulthau’s article from 2012, Information search process.
Questions posed to the class include:
How would you describe this theory? Is it descriptive, predictive, explanatory, or some combination of these?
What are some of the key concepts of this theory? How do they relate to each other?
Does this theory have any testable propositions? If so, how might you test them?
What could this theory help LIS researchers do or understand? What could it help practitioners understand?
Before getting into what a theory is, the professor discussed several interconnected and related terms that are important to know beforehand.
A paradigm is defined as a broad, foundational assumption shared by essentially all researchers in a field.
A metatheory ia a theory concerned with the investigation, analysis, or description of theory itself; ideas about how concepts in a field should be thought about and researched.
A theory is a system of assumptions, accepted principles, and rules of procedure devised to analyze, predict, or otherwise explain the nature or behavior of a specific set or phenomena. A theory appears in abstract general terms and generates more specific hypotheses.
A model is a tentative ideational structure used as a testing device.
The class then broke off into groups again to discuss “possible study topics” from our discussion on Monday.
Intellectual vs. physical presentation of archival materials
Where to search:
Conversations with archivists
Search term suggestions:
Cooperation between collections
Ideas for narrative