Building Block 7: Research Methods & Philosophy of Science

The Building Blocks:

Quantitative data analysis methods are important. In economics however, they do not require much defending: they are already the established status quo. Quantitative data collection methods, however, are rarely taught to economics students, despite the fact that students do learn how to analyse this data once it is collected. Philosophy of science, too, is largely accepted as a necessary component of academic programmes, even if it is still too often banished to the fringes of the programme. We believe it could be taught in a more integral and applied manner, as we discuss in the first section of this chapter.

The most unusual of our suggestions must be the inclusion of qualitative methods in the economist’s toolkit. Hence, we will start by briefly making the case for methods like interviews and case studies. A good example is found in Karen Ho’s work on Wall Street. Financial institutions, which are at the heart of the economy, are so quantitatively oriented that they generally prefer hiring physicists and mathematicians over economists and other social scientists. But when the curtain came crashing down in 2007, numbers were not enough to understand what had happened. In Liquidated, Karen Ho investigates financial instability by conducting over a hundred interviews and engaging in participant observation during her work as a consultant in various investment banks on Wall Street during multiple years (2009).

Her findings indicate how investment banks export their own insecure workplace labour arrangements to other sections of the economy. She identified a Wall Street culture that has distinct fads and fashions in its approach to business management. In the years before the crisis, this culture had come to focus relentlessly on downsizing and the flexibility of labour arrangements. Thereby, it had contributed both to the practices that led to the crisis throughout the broader economy, and to a banking landscape unable to withstand the shock, once it came.

Most students will not go on to extensively study topics such as the driving forces of Wall Street, nor will they have time for ethnographic research. Yet interviewing, absorbing and understanding bits of culture, and analysing case studies are skills that go beyond analysing pre-existing statistical data sets. They complement these data sets by providing context and new insights into mechanisms which can help to explain observable phenomena, and are crucial for any economist, regardless of their area of expertise.

Qualitative methods are particularly important for understanding institutions and culture, crucial in economic dynamics. They also allow us to gain insight into the nature of different kinds of economic relationships, such as employment, transactions, buyer-supplier relationships and competition. Qualitative methods can also inform us about unexpected developments within the economy. For example, interviews and participant observations have provided new insights into how the financial sector works and why financial instability arises. Qualitative research methods can be very helpful in understanding the context of a specific case and acquiring an overview of how those involved perceive the situation. These various skills are particularly useful when working on concrete problems as professional economists. In short, qualitative methods can contribute both to the development of theory and to practical, concrete understanding.

It can also be very useful and productive to combine quantitative and qualitative research methods, often called mixed methods research. We discuss this on our website.

We start this building block with the philosophy of science and a broad overview of available methods, from quantitative to qualitative and from data collection to data analysis. We then explore quantitative methods in more depth, first discussing data collection methods, then discussing various techniques for data analysis. We subsequently do the same for qualitative methods. The chapter ends with suggestions on how to teach and effectively combine these various aspects of research methods, and a list of useful reading and teaching materials.

The aim of teaching philosophy of science is not to teach students ‘the best way to do research’. Rather, it is to teach students how to make informed methodological choices and to be reflective on those choices. This requires explicit attention for the limitations of methods and the trade-offs involved in the process of making methodological decisions.

The two aspects of philosophy of science that are of particular importance are those that relate to the ontology and epistemology of economics. Here we only mention a few of important questions and approaches students should be exposed to; our website discusses further details.

Ontology is the study of the nature of the world. It asks questions such as: is there a world ‘out there’ that we can study objectively, or do we actively construct reality? Does the world consist of individual parts that relate to one another, or is it a systemic whole? It also questions whether the economic world fundamentally differs from the natural world.

Epistemology is about how we can or cannot know things. It asks whether we can objectively observe reality, or whether “knowledge” is always the result of our own interpretation and experience. It also considers the different ways in which we can or should acquire knowledge. For example, should we start from empirical observations or from logic? These questions are answered differently by different methodological traditions, such as positivism, interpretivism, (critical) realism, and pragmatism.

The goal here is not to convince students to choose one particular approach. Instead students should understand the different approaches, and in particular which arguments they make, so that they can make informed methodological decisions when studying a topic.

The ethics related to doing research are also an important aspect of the philosophy of science. Most universities have ethics committees to review the moral acceptability of studies, whether experiments, surveys or interviews. It is important to introduce students to ideas and debates surrounding these issues.

Philosophy of science does not have to be overly complicated or abstract. The easiest way to ensure this is to integrate it with other aspects of research methodology, which is why we combine them into this single building block. Philosophical issues can be discussed with the help of specific studies and concrete applications of methods, rather than only discussing the concepts in the abstract. When, for example, teaching students about the technicalities of regression analysis, it is important to discuss ideas about what statistical significance really tells us about the world (Ziliak & McCloskey, 2008).

Now we turn to the research methods. In figure 7 below, we present an overview of qualitative and quantitative data collection and analysis methods. This is meant to illustrate the wide variety of options there are when teaching research methods to economics students. But as teaching time is always limited, we allocate the methods in two categories, essential and additional. We suggest the methods in the ‘essential’ box are most relevant for all economists to become familiar with.

The additional methods can, however, be crucial for students specialising in certain directions. If a student, for example, decides to specialise in qualitative research, it is key that he or she also learns about doing observations and how to apply content analysis and grounded theory to qualitatively analysing data. On the other hand, a student focused on quantitative methods would, for example, be helped by learning about automated data collection, and mathematical and agent-based modelling. As such, there are many relevant methods that are often too much to teach to all economics students, but are of great importance for those specialising in a certain direction.

We are very aware that this categorisation is likely to be contested and we advise teachers to change it according to their own views. At the same time, we recommend keeping the list of essential methods short, in order to keep it practically feasible to teach in a programme.

The methods categorised as essential are elaborated upon in the headings 3-6 of this chapter, below. In the online resources, we discuss the additional methods mentioned in the table.

Statistics can provide very helpful insights into economic systems and dynamics. For this reason, we think it is important that economics students acquire a good basis in quantitative data analysis in their programmes. When teaching quantitative data analysis methods more diversity can also enrich economics education. Current programmes focus predominantly on regression analyses. Students could benefit from learning a broader set of statistical techniques, and in particular network analysis. It is becoming increasingly clear how such new methods can help us understand economic dynamics, such as the Global Financial Crisis of 2007-2008 and its build-up with massive international financial flows, or the COVID-19 crisis with the global spread of the virus.

More generally, we encourage putting less emphasis on mathematical modelling in compulsory economics courses. Surveys among employers of economists indicate that professional economists rarely need sophisticated mathematical or econometric skills (Yurko, 2018). What is generally needed in practice, is being able to work, make sense of and communicate relatively basic statistical analyses. As such, economics programmes can better prepare students for their future roles by putting less emphasis on mathematical skills. As Robert Frank, professor at Cornell University (2011, p. 408) writes:

This is not to say that mathematical modelling has no use. For those students with interest or talent for mathematics, like ourselves, it should be possible to specialise in this area through elective courses. In this way, the students who would later like to publish papers in mainstream economics journals have the opportunity to learn the research methods generally required there. At the same time, those less inclined towards mathematics are not prevented from becoming economists. In this way, economists, as a group, will be better able to fulfil their societal role as they will be able to apply a broader range of methods.

When mathematics is taught, we advise to mainly teach it mainly through tutorials and assigned homework as these allow for more differentiation and personal attention. Even more so than for other parts of economics education, students follow mathematics at different speeds and learn it in different ways. The traditional lecture teaching style, with a single professor standing at the front of the class writing out equations on the blackboard, therefore, does not seem to be the best way to enable students to develop their mathematical skills.

Beyond these techniques of analysis, we suggest that students gain hands-on experience with data collection methods, and in particular experiments and survey research. This will give them a feel for how datasets should be interpreted, but also help them gauge the reliability and limitations that come with any dataset. Let students design their own questionnaire and go out and gather a small dataset by themselves, and then work with that dataset on the aforementioned analytical techniques.

More and more programmes are incorporating behavioural economics and experimental methods. We applaud this development and encourage it, but we do think that learning how to collect quantitative data through surveys is even more critical. The main reason for this is that most data used by economists, both in academic and policy circles, still comes from surveys or processed government and tax files.

As Chang put it (2012, p. 1): “Some economists say [numbers] are like sausages: you don’t know what they really are until you cut into them.” If students do not learn how datasets are constructed during their education, they are likely to never really understand the numbers they will be intensively working with for the rest of their careers. Just as with statistics, it is important that students not only learn how to do the techniques, but are also taught how to reflect upon the advantages and disadvantages of methodological choices.

There are many different forms of qualitative data analysis, but case study research seems most essential for any economist. This is a useful approach to finding and describing essential information, and has the additional advantage that it almost automatically provides students with knowledge about the actual economy. Like interviewing, case study research is often applied by professionals working outside of academia. For this reason, it is useful to focus on teaching students to apply the method in practice and what aspects to be conscious of. Academic research is generally done more carefully in a longer time span and puts more emphasis on methodology. Practical applications of methods are often more concerned with the substance and conclusions. Students should therefore learn how to be able to acquire practically useful insights but in a methodologically proper manner.

For the systematic analysis of qualitative data, there are many other methods as outlined in the table above, such as content analysis, grounded theory, qualitative comparative analysis, and analytic induction. And like with quantitative analysis, it is useful for students to learn to work with software for analysis. However, while these methods are of great importance for students specialising in qualitative analyses, they seem to be less important for the average professional economist. Therefore, we recommend offering these qualitative data analysis methods in elective, rather than compulsory, courses. More details on these other techniques can be found in the online resources.

In addition to the large variety of qualitative data collection methods mentioned in the table above, we believe the most foundational method is interviewing. A key example of the usefulness of interviewing to economics is the study of Bewley into wage rigidity (1999). Rather than theorising about human behaviour in the abstract, he realised more insight into the matter could be gained by empirically investigating why both employers and employees are reluctant to let wages fall during recessions, creating a widely cited breakthrough study by using interviews.. Interviews are not only used by academic researchers. Many professionals in different types of organisations often use interviews to collect data, as it is a uniquely useful method for systematically acquiring knowledge about processes and people’s experiences and thoughts.

Students can easily start working with this method, just as running a regression analysis on a computer can be done with the push of a button. The challenge is, however, to do it well and to make students conscious of methodological issues and trade-offs, and helping them acquire skills and experience by practising the method. It is, for example, important to pay attention to issues such as the structure of an interview, the phrasing of the questions, the non-verbal communication during the interview, and the context in which the interview takes place.

We have three suggestions for how to teach these methods: be as hands-on as possible, use up-to-date software and teach specific approaches as part of a broader overview of the methods field.

First, ask students to apply the methods you teach them. Students find it much more interesting that way, and much more memorable. Perhaps most importantly, it prepares them for how they will use the methods in the future in their careers, as most students will become practical – rather than academic – economists. Another possibility is to teach methods in the context of a larger research project, which includes reading and evaluating existing research as well as letting students conduct empirical research themselves.

We understand that, for didactical reasons, it can sometimes be helpful to use fictional data to introduce students to the basics. However, we think it is important that this practice is kept to a minimum and students learn to work with real data as much as possible. The importance of this was expressed by a UK public sector manager who said (Yurko, 2018, p. 11):

To collect quantitative data students could design their own survey to investigate a particular research or policy question. Going door to door to conduct those surveys and finding respondents online will teach them much about the messy nature of statistical data, and it is a valuable personal experience that they will not forget easily. In addition, it breaks the school-like monotony of lectures and working groups. Once collected, the data can be analysed using the various descriptive and inferential statistical techniques mentioned above.

Teaching the qualitative data collection method of interviewing is an excellent opportunity to send students out to do fieldwork. For instance, students could be tasked to find a company, government institution, a bank or any other economic environment to do observations and conduct (brief) interviews with a number of employees. This data could then be analysed using qualitative data analysis techniques, such as content analysis and case study.

Second, we advise to use state-of-the-art software, both for quantitative and qualitative data analysis. Being able to work with up-to-date software is often a highly rewarded skill in employment and thus important to teach in economics programmes. We do recognise that this might sound easier than it is, as it requires continuous time investment from the teacher to be acquainted with recent developments in software. To find the right software, ask employers and academics at the research frontier, or search online.

Third and finally, we suggest it is useful to teach students to understand and be able to evaluate a broader set of methods, which students will not learn to actively apply themselves. Such overviews of the methods field will enable students to grasp more advanced work and other types of research, and to know what things to pay attention to. Approach the methods in an integrated way and connect the different aspects to each other. Instead of teaching and reflecting on the techniques (only) separately, discuss them together. An assignment could be to read a certain set of papers or research reports, and reflect upon the main methodological choices and steps, and explain how the results should be interpreted. Properly reading and summarising literature is a skill in itself, not only a preparation phase for conducting new research projects.

Unfortunately, there are limitations on how many methods students can be taught in up to three-year programmes. It is practically impossible to try to teach them all relevant methods. It is, however, possible to give them an overview and basic understanding of them, and to give them the skills required to learn new methods quicker and more thoroughly. We would therefore advise teaching students a wide range of methods relatively briefly, and select a few research methods for hands-on training, in more detail. This gives students both a rough idea of different methods as well as the experience of working more in-depth with some of them.