The Building Blocks:
A broad methodological toolkit with quantitative and qualitative data collection and analysis methods, and reflection upon them.
The philosophy of science provides an important foundation for evaluating research methods. We encourage discussing ontology, epistemology, and ethics with students. This helps them choose suitable methodologies for their own research projects and develop their critical thinking when encountering research results. Research methods are a vital element in the modern economist’s toolkit. This includes quantitative analysis methods, such as descriptive statistics, regression analysis and network analysis. It also includes qualitative analysis methods, such as case studies. Besides such data analysis tools, students should also gain experience with quantitative and qualitative data collection: designing and conducting experiments, survey research and interviews. This will give them a feel for data quality and put them in touch with the actual context that they are studying, beyond only the numbers.
Economists are knowledge workers. Much of our work, both inside and outside academia, consists of working with – more or less formalised – research methods. Hence, we need a broad range of methodological skills and knowledge, as well as the ability to reflect upon our methodological choices and explain the implications for the interpretation of our findings. These are unique skills that make economists valuable members of teams and organisations.
Contrast with current programmes
Methods courses in current undergraduate programmes are generally limited to mathematics and various forms of regression analysis. These are useful tools for proving and testing economic theories, and crucial for publishing in today’s academic journals. For a lot of work, however, it is essential to have a broader range of methodological tools available. The main purpose of most economists’ work is generally guiding action, rather than developing and improving theories. Hence, students need to learn how to apply various quantitative and qualitative data collection and analysis methods to real data.
“Research is formalized curiosity. It is poking and prying with a purpose.”Zora Neale Hurston (1942, p. 91)
Quantitative data analysis methods are important. In economics however, they do not require much defending: they are already the established status quo. Quantitative data collection methods, however, are rarely taught to economics students, despite the fact that students do learn how to analyse this data once it is collected. Philosophy of science, too, is largely accepted as a necessary component of academic programmes, even if it is still too often banished to the fringes of the programme. We believe it could be taught in a more integral and applied manner, as we discuss in the first section of this chapter.
The most unusual of our suggestions must be the inclusion of qualitative methods in the economist’s toolkit. Hence, we will start by briefly making the case for methods like interviews and case studies. A good example is found in Karen Ho’s work on Wall Street. Financial institutions, which are at the heart of the economy, are so quantitatively oriented that they generally prefer hiring physicists and mathematicians over economists and other social scientists. But when the curtain came crashing down in 2007, numbers were not enough to understand what had happened. In Liquidated, Karen Ho investigates financial instability by conducting over a hundred interviews and engaging in participant observation during her work as a consultant in various investment banks on Wall Street during multiple years (2009).
Her findings indicate how investment banks export their own insecure workplace labour arrangements to other sections of the economy. She identified a Wall Street culture that has distinct fads and fashions in its approach to business management. In the years before the crisis, this culture had come to focus relentlessly on downsizing and the flexibility of labour arrangements. Thereby, it had contributed both to the practices that led to the crisis throughout the broader economy, and to a banking landscape unable to withstand the shock, once it came.
Most students will not go on to extensively study topics such as the driving forces of Wall Street, nor will they have time for ethnographic research. Yet interviewing, absorbing and understanding bits of culture, and analysing case studies are skills that go beyond analysing pre-existing statistical data sets. They complement these data sets by providing context and new insights into mechanisms which can help to explain observable phenomena, and are crucial for any economist, regardless of their area of expertise.
Qualitative methods are particularly important for understanding institutions and culture, crucial in economic dynamics. They also allow us to gain insight into the nature of different kinds of economic relationships, such as employment, transactions, buyer-supplier relationships and competition. Qualitative methods can also inform us about unexpected developments within the economy. For example, interviews and participant observations have provided new insights into how the financial sector works and why financial instability arises. Qualitative research methods can be very helpful in understanding the context of a specific case and acquiring an overview of how those involved perceive the situation. These various skills are particularly useful when working on concrete problems as professional economists. In short, qualitative methods can contribute both to the development of theory and to practical, concrete understanding.
It can also be very useful and productive to combine quantitative and qualitative research methods, often called mixed methods research. We discuss this on our website.
We start this building block with the philosophy of science and a broad overview of available methods, from quantitative to qualitative and from data collection to data analysis. We then explore quantitative methods in more depth, first discussing data collection methods, then discussing various techniques for data analysis. We subsequently do the same for qualitative methods. The chapter ends with suggestions on how to teach and effectively combine these various aspects of research methods, and a list of useful reading and teaching materials.
1. Philosophy of Science
The aim of teaching philosophy of science is not to teach students ‘the best way to do research’. Rather, it is to teach students how to make informed methodological choices and to be reflective on those choices. This requires explicit attention for the limitations of methods and the trade-offs involved in the process of making methodological decisions.
The two aspects of philosophy of science that are of particular importance are those that relate to the ontology and epistemology of economics. Here we only mention a few of important questions and approaches students should be exposed to; our website discusses further details.
Ontology is the study of the nature of the world. It asks questions such as: is there a world ‘out there’ that we can study objectively, or do we actively construct reality? Does the world consist of individual parts that relate to one another, or is it a systemic whole? It also questions whether the economic world fundamentally differs from the natural world.
Epistemology is about how we can or cannot know things. It asks whether we can objectively observe reality, or whether “knowledge” is always the result of our own interpretation and experience. It also considers the different ways in which we can or should acquire knowledge. For example, should we start from empirical observations or from logic? These questions are answered differently by different methodological traditions, such as positivism, interpretivism, (critical) realism, and pragmatism.
The goal here is not to convince students to choose one particular approach. Instead students should understand the different approaches, and in particular which arguments they make, so that they can make informed methodological decisions when studying a topic.
The ethics related to doing research are also an important aspect of the philosophy of science. Most universities have ethics committees to review the moral acceptability of studies, whether experiments, surveys or interviews. It is important to introduce students to ideas and debates surrounding these issues.
Philosophy of science does not have to be overly complicated or abstract. The easiest way to ensure this is to integrate it with other aspects of research methodology, which is why we combine them into this single building block. Philosophical issues can be discussed with the help of specific studies and concrete applications of methods, rather than only discussing the concepts in the abstract. When, for example, teaching students about the technicalities of regression analysis, it is important to discuss ideas about what statistical significance really tells us about the world (Ziliak & McCloskey, 2008).
2. Research Methods: A Broad Overview
Now we turn to the research methods. In figure 7 below, we present an overview of qualitative and quantitative data collection and analysis methods. This is meant to illustrate the wide variety of options there are when teaching research methods to economics students. But as teaching time is always limited, we allocate the methods in two categories, essential and additional. We suggest the methods in the ‘essential’ box are most relevant for all economists to become familiar with.
The additional methods can, however, be crucial for students specialising in certain directions. If a student, for example, decides to specialise in qualitative research, it is key that he or she also learns about doing observations and how to apply content analysis and grounded theory to qualitatively analysing data. On the other hand, a student focused on quantitative methods would, for example, be helped by learning about automated data collection, and mathematical and agent-based modelling. As such, there are many relevant methods that are often too much to teach to all economics students, but are of great importance for those specialising in a certain direction.
We are very aware that this categorisation is likely to be contested and we advise teachers to change it according to their own views. At the same time, we recommend keeping the list of essential methods short, in order to keep it practically feasible to teach in a programme.
The methods categorised as essential are elaborated upon in the headings 3-6 of this chapter, below. In the online resources, we discuss the additional methods mentioned in the table.
3. Quantitative Data Analysis: Descriptive Statistics, Regression Analysis & Network Analysis
Statistics can provide very helpful insights into economic systems and dynamics. For this reason, we think it is important that economics students acquire a good basis in quantitative data analysis in their programmes. When teaching quantitative data analysis methods more diversity can also enrich economics education. Current programmes focus predominantly on regression analyses. Students could benefit from learning a broader set of statistical techniques, and in particular network analysis. It is becoming increasingly clear how such new methods can help us understand economic dynamics, such as the Global Financial Crisis of 2007-2008 and its build-up with massive international financial flows, or the COVID-19 crisis with the global spread of the virus.
More generally, we encourage putting less emphasis on mathematical modelling in compulsory economics courses. Surveys among employers of economists indicate that professional economists rarely need sophisticated mathematical or econometric skills (Yurko, 2018). What is generally needed in practice, is being able to work, make sense of and communicate relatively basic statistical analyses. As such, economics programmes can better prepare students for their future roles by putting less emphasis on mathematical skills. As Robert Frank, professor at Cornell University (2011, p. 408) writes:
“Most introductory courses (and my own was no exception in the early days) make little use of narrative. Instead, they inundated students with equations and graphs. Mathematical formalism has been an enormously important source of intellectual progress in economics, but it has not proved an effective vehicle for introducing newcomers to our subject. Except for engineering students and a handful of others with extensive prior training in math, most students who attempt to learn economics primarily through equations and graphs never really grasp [it].”
This is not to say that mathematical modelling has no use. For those students with interest or talent for mathematics, like ourselves, it should be possible to specialise in this area through elective courses. In this way, the students who would later like to publish papers in mainstream economics journals have the opportunity to learn the research methods generally required there. At the same time, those less inclined towards mathematics are not prevented from becoming economists. In this way, economists, as a group, will be better able to fulfil their societal role as they will be able to apply a broader range of methods.
When mathematics is taught, we advise to mainly teach it mainly through tutorials and assigned homework as these allow for more differentiation and personal attention. Even more so than for other parts of economics education, students follow mathematics at different speeds and learn it in different ways. The traditional lecture teaching style, with a single professor standing at the front of the class writing out equations on the blackboard, therefore, does not seem to be the best way to enable students to develop their mathematical skills.
4. Quantitative Data Collection: Experiments & Survey Research
Beyond these techniques of analysis, we suggest that students gain hands-on experience with data collection methods, and in particular experiments and survey research. This will give them a feel for how datasets should be interpreted, but also help them gauge the reliability and limitations that come with any dataset. Let students design their own questionnaire and go out and gather a small dataset by themselves, and then work with that dataset on the aforementioned analytical techniques.
More and more programmes are incorporating behavioural economics and experimental methods. We applaud this development and encourage it, but we do think that learning how to collect quantitative data through surveys is even more critical. The main reason for this is that most data used by economists, both in academic and policy circles, still comes from surveys or processed government and tax files.
As Chang put it (2012, p. 1): “Some economists say [numbers] are like sausages: you don’t know what they really are until you cut into them.” If students do not learn how datasets are constructed during their education, they are likely to never really understand the numbers they will be intensively working with for the rest of their careers. Just as with statistics, it is important that students not only learn how to do the techniques, but are also taught how to reflect upon the advantages and disadvantages of methodological choices.
5. Qualitative Data Analysis: Case Studies
There are many different forms of qualitative data analysis, but case study research seems most essential for any economist. This is a useful approach to finding and describing essential information, and has the additional advantage that it almost automatically provides students with knowledge about the actual economy. Like interviewing, case study research is often applied by professionals working outside of academia. For this reason, it is useful to focus on teaching students to apply the method in practice and what aspects to be conscious of. Academic research is generally done more carefully in a longer time span and puts more emphasis on methodology. Practical applications of methods are often more concerned with the substance and conclusions. Students should therefore learn how to be able to acquire practically useful insights but in a methodologically proper manner.
For the systematic analysis of qualitative data, there are many other methods as outlined in the table above, such as content analysis, grounded theory, qualitative comparative analysis, and analytic induction. And like with quantitative analysis, it is useful for students to learn to work with software for analysis. However, while these methods are of great importance for students specialising in qualitative analyses, they seem to be less important for the average professional economist. Therefore, we recommend offering these qualitative data analysis methods in elective, rather than compulsory, courses. More details on these other techniques can be found in the online resources.
6. Qualitative Data Collection: Interviews
In addition to the large variety of qualitative data collection methods mentioned in the table above, we believe the most foundational method is interviewing. A key example of the usefulness of interviewing to economics is the study of Bewley into wage rigidity (1999). Rather than theorising about human behaviour in the abstract, he realised more insight into the matter could be gained by empirically investigating why both employers and employees are reluctant to let wages fall during recessions, creating a widely cited breakthrough study by using interviews.. Interviews are not only used by academic researchers. Many professionals in different types of organisations often use interviews to collect data, as it is a uniquely useful method for systematically acquiring knowledge about processes and people’s experiences and thoughts.
Students can easily start working with this method, just as running a regression analysis on a computer can be done with the push of a button. The challenge is, however, to do it well and to make students conscious of methodological issues and trade-offs, and helping them acquire skills and experience by practising the method. It is, for example, important to pay attention to issues such as the structure of an interview, the phrasing of the questions, the non-verbal communication during the interview, and the context in which the interview takes place.
7. Practical Suggestions
We have three suggestions for how to teach these methods: be as hands-on as possible, use up-to-date software and teach specific approaches as part of a broader overview of the methods field.
First, ask students to apply the methods you teach them. Students find it much more interesting that way, and much more memorable. Perhaps most importantly, it prepares them for how they will use the methods in the future in their careers, as most students will become practical – rather than academic – economists. Another possibility is to teach methods in the context of a larger research project, which includes reading and evaluating existing research as well as letting students conduct empirical research themselves.
We understand that, for didactical reasons, it can sometimes be helpful to use fictional data to introduce students to the basics. However, we think it is important that this practice is kept to a minimum and students learn to work with real data as much as possible. The importance of this was expressed by a UK public sector manager who said (Yurko, 2018, p. 11):
“I’d basically make a lot of it more applied. I’m always slightly astonished you can go through three years of an undergrad learning macroeconomics without really knowing what GDP is or even knowing where to look on the internet to get GDP data. … It would be useful if they had a bit more of how to actually use econometrics rather than the technical, basic how to do econometric proofs.”
To collect quantitative data students could design their own survey to investigate a particular research or policy question. Going door to door to conduct those surveys and finding respondents online will teach them much about the messy nature of statistical data, and it is a valuable personal experience that they will not forget easily. In addition, it breaks the school-like monotony of lectures and working groups. Once collected, the data can be analysed using the various descriptive and inferential statistical techniques mentioned above.
Teaching the qualitative data collection method of interviewing is an excellent opportunity to send students out to do fieldwork. For instance, students could be tasked to find a company, government institution, a bank or any other economic environment to do observations and conduct (brief) interviews with a number of employees. This data could then be analysed using qualitative data analysis techniques, such as content analysis and case study.
Second, we advise to use state-of-the-art software, both for quantitative and qualitative data analysis. Being able to work with up-to-date software is often a highly rewarded skill in employment and thus important to teach in economics programmes. We do recognise that this might sound easier than it is, as it requires continuous time investment from the teacher to be acquainted with recent developments in software. To find the right software, ask employers and academics at the research frontier, or search online.
Third and finally, we suggest it is useful to teach students to understand and be able to evaluate a broader set of methods, which students will not learn to actively apply themselves. Such overviews of the methods field will enable students to grasp more advanced work and other types of research, and to know what things to pay attention to. Approach the methods in an integrated way and connect the different aspects to each other. Instead of teaching and reflecting on the techniques (only) separately, discuss them together. An assignment could be to read a certain set of papers or research reports, and reflect upon the main methodological choices and steps, and explain how the results should be interpreted. Properly reading and summarising literature is a skill in itself, not only a preparation phase for conducting new research projects.
Unfortunately, there are limitations on how many methods students can be taught in up to three-year programmes. It is practically impossible to try to teach them all relevant methods. It is, however, possible to give them an overview and basic understanding of them, and to give them the skills required to learn new methods quicker and more thoroughly. We would therefore advise teaching students a wide range of methods relatively briefly, and select a few research methods for hands-on training, in more detail. This gives students both a rough idea of different methods as well as the experience of working more in-depth with some of them.
- Economic Methodology: Understanding economics as a science by Marcel Boumans and John B. Davis, most recent edition from 2015. A sharp and accessible introduction into economic methodology and philosophy of science with explanations of different views on science and key debates on how economics should be practiced.
- Social Research Methods by Alan Bryman, most recent edition from 2015. A prominent textbook that introduces a wide variety of quantitative and qualitative research methods, such as interviews, structured and participant observation, content analysis, and survey research.
- The SAGE Handbook of Applied Social Research Methods by Leonard Bickman and Debra J. Rog, most recent edition from 2009. A leading textbook on applied research with attention to choosing the right method for the question at hand, practical considerations, and how to make informed methodological decisions for a variety of quantitative and qualitative methods.
- Handbook of Research Methods and Applications in Heterodox Economics by Frederic Lee and Bruce Cronin, from 2016. An instructive collection of essays with explanations, reflections on and applications of innovative research methods that deviate from the standard econometric approach usually taught in economics programmes, such as survey research, network analysis, experiments, ethnography, and agent-based computational modelling.
- Qualitative Research Practice A Guide for Social Science Students and Researchers by Jane Ritchie, Jane Lewis, Carol McNaughton Nicholls, and Rachel Ormston, most recent edition from 2013. A useful introduction into how to do rigorous and reflective quantitative research with chapters on interviews, focus groups, observation, research design, ethical considerations, and data analysis.