Is policy-driven research based on unsound and uncritiqued methodology? Could the subsequent policy conclusions which are generated therefore be flawed?
The problem of inaccurate and unsubstantiated assertions reflects a key problem in social science research in the humanitarian field, namely the methods used to conduct empirical research. Unlike more traditional social science studies, most humanitarian ‘field’ research has not been subjected to critique of its methods. In all sound research it is important that the methods used to obtain information and data be clearly explained in order to make it possible to replicate them and thereby validate findings. But in much humanitarian field research, key details about the approach are never revealed. Among the missing details not generally provided are: how many people were interviewed, by whom and under what circumstances; how the subjects were identified and selected and how translation issues were handled.
Methodological weaknesses and ethical problems
- Too many interviews – not enough data sets
Much of the published research on refugees and IDPs is based on data that has been collected using fairly unstructured interviews in a small-scale setting. Although in-depth interviews have given us a rich and useful store of descriptive and anecdotal data, they do not yield a sufficiently representative sample of the population to allow us to test competing hypotheses and causal relationships using statistical analysis. Aside from camp surveys, there are currently very few data sets based on large-scale surveys conducted by social scientists of self-settled refugee or IDP populations (i.e. those living outside camps), either in rural or urban areas. Existing surveys tend to focus on public health or nutritional issues. One particularly understudied population is urban refugees and IDPs.
- Missing control groups
Much social science research on refugees lacks any kind of control group, making it difficult to assess the extent to which refugee status is the problem or whether other factors are causing the variance. For example, a common research topic is the study of security problems in refugee camps, yet few studies are designed in such a way as to compare the security problems of refugees in camps with those living outside camps or with those of the host community.
- Representativeness and bias
Many social scientists in the field use a ‘snowball’ sample approach. Unlike a random sample, where everyone in the target population has an equal chance of being in the sample, in a snowball sample the subjects are more likely to be drawn from a particular segment of the community, and are likely to be similar in certain ways – church goers and their friends, for example, or those who are beneficiaries of an NGO. The sample will therefore be biased.
- The problem of bias and construct validity
Construct validity refers to the strength and soundness of the variable we are investigating. The conversation-like tone of in-depth interviews can potentially prompt particular responses, or inadvertently direct the answers, an unconscious process often difficult to avoid even by trained researchers. This potential problem is even more salient in participatory rural appraisal (PRA) techniques that are popular among field researchers. Many argue that PRA should be used as only one among a variety of measures – but in much reported field research it is the only one.
- Determining the size of the target population
Knowledge of the size of the target population is important in generating a proper sample. In the case of urban refugees and IDPs, no studies have yielded an accurate count in Africa. At best there are rough estimates.
Addressing these problems – the difficulties of studying self-settled refugees
- The problem of access
The problem of access to refugees means that most researchers work in camps, which present a more logistically manageable area, omitting from their study those refugees/IDPs who don’t live in camps yet who may be the majority. The difficulties of exploring the range of views held by the women in a community, for example, is particularly well known; this is also true for other more ‘hidden’ social groups, including the poor and those living in remote areas.
- Using local researchers
Western researchers work with local researchers because it is widely believed that this yields better results. While it is likely that the use of local researchers can increase the reliability and validity of data, there may be ethical and methodological problems issues involved, such as translation inaccuracies or – when refugee assistants are used – biased responses. Political or security problems may arise for refugee assistants/local researchers. When refugees are interviewed the information they reveal may be used against them either in the camp or in their areas of origin. In group discussions, a method commonly used by researchers, there can be no confidentiality; what may be inadvertently revealed cannot be fully controlled even by diligent researchers.
Making our work more relevant
As social scientists working in the humanitarian field, we have a duty to be honest researchers, conducting methodologically sound research while doing our best to confront the ethical problems that all fieldwork encounters. We need to:
- be more careful about making claims about representativeness and about representation within the community
- be more careful about construct validity
- write more clearly, use less jargon and avoid over-researched topics and places
- use common definitions of concepts
- be aware of political realities/incentive structures
Karen Jacobsen is the Director of the Refugees and Forced Migration Programme, Feinstein International Famines Centre, Tufts University, Massachusetts. Email: karen.jacobsen@tufts.edu