The Theory and Method of Cultural Consensus

1. Introduction

 

1.1 First talk about theory, then if time, talk about the method

 

1.2 Theory concerns two observations

 

1.2.1 There is intracultural variability

 

- this comes as shock to anyone reading ethnographies, 99% of which are absolutely brimming with statements like "the samoans believe X" or "the mehinaku eat Y" like they were describing a nest of homogeneous ants.

 

- in past, only handful of people concerned about this: afc wallace in 60's, jack roberts also 60's, pelto & pelto 75, and roy d'andrade in 87 who called the situation "a scandal at the heart of the study of culture".

 

- today, recognizing intracultural variability is very hip. we celebrate diversity.

 

- your postmodernist, interpretive anthropologist fundamentally believes that every situation and every human being is unique, and

 

- that generalizations regarding human beings, are impossible, naive, and the desire to make them is indicative of western, white, male, hierarchical, militaristic, scientistic, positivistic, hegemonic discourse. which is not good. (to be fashionable your discourse should not privilege the hegemonic voice of science over the voices of ... well you get the picture).

 

- yet intracultural variation can still be a problem.

 

- although many anthropologists have dismissed empirical work as illusory at best (since all reality is constructed), some poor souls are still at it, and one of the hallmarks of anthropological method is the indepth interview of a limited number of key informants. but small samples are very dangerous in a diverse world because your handful of informants may have beliefs that are shared by very few others in the society. in fact, as you train your informant in anthropological ways, you can pretty much depend on him or her to look at the world very differently.

 

- there is also a formal problem, which has to do with the meaning and definition of culture. most of the dozens (hundreds?) of definitions contain something about it being shared. but if culture is not shared, what is it?

 

1.2.2 Agreement implies knowledge

 

- a folk method of ascertaining truth is to get agreement.

 

- e.g. the british and american jury system.

 

- system of film reviews and looking at boxoffice receipts to judge whether film is good

 

- a lot of social science works that way as well: today, the postmodernist researcher does not check her conclusions against the world out there, but rather against what others have said before her. If he can make copious citations to people who seem to agree with him, then he is right.

 

- early work by psychologists in the 20's showed that people as a group can estimate physical attributes such as length, weight, brightness with deadly accuracy, as long as you average the estimates of the entire group

 

- and anthropologists have always regarded that information which was confirmed by multiple informants, to be the most reliable.

 

- jim boster did interesting thing while studying the aguaruna. he collected multiple varieties of manioc and planted them in a garden. then he had aguaruna women walk through his garden and identify each plant. he found that the cluster of women who frequently agreed with each other seemed to also be the most knowledgeable. they were also more likely to be related by kin ties....

 

1.2.3 Of course, this doesn't always work. we may all agree that the more quickly and violently we press the elevator button, the faster the elevator will come, but our beliefs to do not make it so.

 

1.3 What the theory of cultural consensus does, like all formal theories, is specify the conditions under which agreement does imply knowledge.

- This then becomes the basis of the method of cultural consensus, because when the conditions are satisfied, the derivations of the theory may be used to estimate knowledge (and TRUTH) from observed patterns of agreement

 

- it also accounts for intracultural variability, while simultaneously asserting that fundamentally culture is shared.

 

1.4 The main parts of the theory are developed by romney, weller and batchelder in an american anthropologist article in '86 (?). several others have worked on it, including jim boster, who had the original insight. I was a research asst. for Romney.

 

2. Multiple Choice Exams

 

- give 5-choice multiple choice exam to 101 class. scantron graded.

 

- computer creates a student-by-question matrix that records each student's responses to each question. each row is a vector of numbers that represents a student's responses to each question. all values are between 1 and 5.

 

- presumably you as the instructor also have a vector recording your answers, which we call the answer key.

 

- your mission, as professor, is to construct a score for each student that represents how well the student did on the exam. i.e., what proportion of the material they have mastered.

 

- one way to think of this is that you must measure the similarity (agreement) between the instructor's vector and each student's vector. i.e. the percent correct is just a correlation between each student's row and the answer key.

 

- actually, if this were an SAT exam, you would be even more ambitious. you would distinguish between the student's level of knowledge or competence from how well they did on the exam, recognizing that sometimes when they don't know the answer, they guess it.

 

- a kind of latent/manifest or emic/etic distinction

 

- here is a simple model of how a student might go about answering a test question:

 

d_i knows right

Qj

1/L guess right

1-d_i doesnt know

1 - 1/L guesses wrong

d_i knows right

Qj

1/L guess right

1-d_i doesnt know

1 - 1/L guesses wrong

L = # of choices in multiple choice test

 

m_i = d_i + (1 - d_i)/L = prob of getting Qj right

 

d_i = (Lm_i - 1)/(L-1) = prob of knowing answer

 

- given this model, lets look at agreement among students. i.e. the proportion of questions on which the two students give the same answer, right or wrong.

 

which is the same thing as computing the probability that on any given question, the two students will answer the same thing.

 

- 4 ways that person i and j could give the same answer:

 

1. both know the right answer and they write it down. assuming there is only one right answer to each question, this means that they will agree on that question. this happens with probability

 

p1 = d_id_j

 

2. i knows the answer, and j guesses right:

 

p2 = d_i(1-d_j)/L

 

3. j knows the answer, and i guesses right:

 

p3 = d_j(1-d_i)/L

 

4. Neither knows, both guess the same:

 

p4 = (1-d_i)(1-d_j)/L

 

- adding these up and simplifying gets:

 

m_ij = d_id_j + (1-d_id_j)/L

 

- so left side of equation is amount of agreement between students i and j (i.e. proportion of questions they gave the same answer on), and right side is all about the amount of knowledge each person has, and L which is just a constant (the number of choices in the multiple choice test)

 

- in other words, agreement is a simple function of knowledge.

 

- makes sense. if two students know most of the material, they will get most of the questions right so will answer the same on most questions...

 

- so given how much two student know, we can estimate how much they will agree. what about the other way around. given how much they actually agree on a test, can we figure out how much they know?

 

re-arrange the equation as follows:

 

d_id_j = (Lm_ij-1)/(L-1) = m^*_ij

 

d_id_j = m^*_ij

 

m^* can be described the matrix of student-by-student agreements, corrected for chance agreement based on guessing

 

- this equation can be solved! in matrix terms, you are given what is essentially a correlation matrix (m^*) and asked to find a vector D which when multiplied by its transpose gives back the matrix m^*. this is what factor analysis does!

 

- now think about that. you give a test. you get students answers on each question. you correlate their answers (right or wrong) with each other to form a big student by student matrix called M which is just the proportion of questions they answered in common. then you adjust m by multiplying by L subtracting 1 etc. this gets M^*. then you factor analyze m^* and out comes the maximum likelihood estimate of the D vector, which gives the amount of knowledge of each student.

all that without looking at the answer key. in other words you can figure out how many questions each student got right, without knowing what the right answers are.

 

- now consider the anthrological problem of asking questions in the field. you don't know the culturally correct answers --- that's why you ask.

 

but even though you are asking questions that informants believe to have a right answer (i.e. what is the name of this plant?, is a boy permitted to look at the face of his mother's brother?), you find that different informants give you different answers. which is right?

 

now, there are two fundamental sources of variability.

 

- one is subcultures. all your informants may look the same to you, but they may come from different subcultures with different culturally right answers to the same questions.

 

- the other is amount of knowledge. most human cultures are too involved, too complex for every member to know completely. i, for example, am weak on plant and flower names, but i'm good with european history. people vary in the amount of competence they have in different cultural domains.

 

- so even if your informants are all drawn from the same culture, there will be variability due to differential amounts of knowledge. the D vector.

 

- is it too good to be true? can it really work? does AGREEMENT REALLY IMPLY KNOWLEDGE? can't we all agree that the world is flat and be dead wrong?

 

- yesbut. certain assumptions must hold. if they do, agreement implies knowledge. if they don't, agreement may or may not imply knowledge.

 

- the main value of this work, in my opinion, is that it spells out in rigorous form not only how to calculate competence and the underlying answer key from agreement data, but also under what conditions can we make the inference

 

 

 

ASSUMPTIONS OF

THE GENERAL CONDORCET MODEL

1. COMMON TRUTH

 

- Fixed answer key applicable to all students

 

- All informants are drawn from the same culture; share same cultural reality

 

 

2. LOCAL INDEPENDENCE

 

- Student-question random variables are independent, conditional on the answer key

 

- No bias

 

 

3. HOMOGENEITY OF ITEMS

 

- Each student has fixed competence across all items

 

- All questions drawn from same cultural domain

 

 

- when these conditions hold, then we can mathematically derive the theorem that relates agreement to knowledge.

 

 

 

A FEW NOTES ABOUT CULTURE

- culture is pattern

 

- intra-cultural variability is function of two factors:

 

- differential competence

 

- subcultural variability

 

 

- differential competence results in random, non-systematic departures from a single cultural truth

 

- in contrast, sub-cultural variability results in multiple truths: an agreement matrix with clusters in it rather than the center-periphery structure of variable competence.

 

 

- this methodology can be used to diagnose the existence of consensus or alternatively sub-cultures

 

TRANSLATING THE ASSUMPTIONS OF

THE GENERAL CONDORCET MODEL

1. COMMON TRUTH

 

- Fixed answer key applicable to all students

 

- All informants are drawn from the same culture; share same cultural reality

 

 

2. LOCAL INDEPENDENCE

 

- Student-question random variables are independent, conditional on the answer key

 

- No bias

 

 

3. HOMOGENEITY OF ITEMS

 

- Each student has fixed competence across all items

 

- All questions drawn from same cultural domain

 

 

 

 

SOURCES OF INTRA-CULTURAL VARIABILITY

1. COMPETENCE

 

- Random, unsystematic variation

 

- Single factor accounts much of the variance

 

- First eigenvalue is several times larger than the next largest

 

 

 

2. SUB-CULTURES

 

- Patterned, systematic variation

 

- More than one factor is needed to explain variance

 

 

OUTPUTS

- Averaging

 

 

- Measuring Consensus

 

 

- Measuring Competence