Quantitative and qualitative research are different. They vary in terms of data collection, methodologies, and analysis.
The internet is abuzz with experts and individuals championing the superiority of one approach over the other. But the truth is, there are advantages and disadvantages to both qualitative and quantitative research.
Read the following article to understand how to get the most from each research method. You’ll also learn how to combine them for the best results.
The following article focuses on 3 key areas:
- Quantitative research and analysis
- Qualitative research and analysis
- Qualitative vs quantitative research: a mixed-method approach for the best results
Quantitative research and analysis
What is quantitative research?
Quantitative research focuses on the collection and analysis of numerical data. If a researcher wants to use non-numerical data they have to quantify them before using them for statistical inference.
Quantitative research uses statistical measurements. These are mean, averages, median, standard deviation, t-tests, and regression. The goal of this approach is to collect quantifiable data to explain a specific phenomenon.
Quantitative research explains causality but also key differences between:
- Groups
- Demographics
- User behavior patterns
Data is not qualitative or quantitative. You can collect ‘categorical’ data and analyze it quantitatively, and you can ask for ‘numeric’ data and analyze it qualitatively.
In a recent research project, I asked employees how big their companies were. They told me a specific number. I used this number to better understand the context in which this person works, but I wasn’t doing any statistical analysis with them.
I did qualitative research even though the data is ‘numerical.’
In the app onboarding flow, we ask people about their role — they can answer “UX designer” or “Developer” and so on. But I might be trying to do some statistics with this data and conclude something like, ‘50% of our users have product-focused roles.
In this situation, I would be doing quantitative research even though the data is ‘categorical’.”
When performing quantitative research, be sure to:
- Use structured research instruments (e.g., survey or digital tools) to collect data
- Get a sample size that’s large enough for your analysis
- Be able to reproduce the same results under the same conditions (research reliability)
In general, you’ll use quantitative research methods in the advanced stages of research.
Qualitative research gives you a general sense of a phenomenon and lets you create hypotheses. Quantitative research lets you validate them.
Take a look at the following common approach to quantitative research:
Imagine you reviewed session recordings (qualitative method). Thanks to the recording, you noticed that something is causing friction for your visitors in the purchase funnel.
Based on session recordings, let’s say you come up with 3 hypotheses:
- You’re seeing a high drop-off rate because Safari users can’t access your shop’s website (bugs are occurring only in this browser)
- You recently changed the copy in the purchase funnel. It’s not as persuasive as you thought
- Something is wrong with your third-party payment system. As a result, visitors receive an error notification when trying to make purchases
You’ll now want to use quantitative research and analysis methods that will help you accept or reject your hypotheses. So, how do you go about collecting even more data for quantitative research? Let’s take a closer look.
How to collect quantitative data for your research
There are many quantitative data collection methods, including:
- Quantitative usability testing
- Web and mobile analytics
- A/B testing
- Surveys (with surveys, you can also collect qualitative data. If you want to do quantitative research, you’ll have to quantify qualitative data)
- Questionnaires
There are many ways of quantitative data collection.
But, according to Mikko Piippo, Senior Consultant & Founder at Hopkins, analytics implementation is rarely as reliable as it should be. Moreover, he says that quantitative data is often used solely for reporting purposes. It is seldom analyzed closely or utilized in decision-making.
Piippo suggests moving from what he calls the CRAP framework of analysis to the CARE framework.
In the following section, you’ll get to know the benefits and drawbacks of quantitative research. Let’s dive into it.
Strengths and weaknesses of quantitative research
1. Strengths of quantitative explorationIt’s relatively easy to collect a large enough sample size
It’s less time-consuming to ask closed-ended quantitative research questions. You can gather plenty of data by organizing surveys, pools, and experiments. Another way to collect valuable data is through web and mobile analytics. Choose analytics software that has data sampling turned off by default.
2. Quantitative research is less biased when you prioritize proper randomization
A random dataset refers to data where each individual has a known probability of appearing in a sample. Together with large sample size, randomization will help you reduce bias in your dataset. Although you can’t eliminate bias 100%, you can minimize its impact on the results. This is important as you should have a fair dataset, especially when it comes to making business decisions based on it.
“Quantitative data in web analytics is extremely valuable. Especially if you want to make data-driven decisions. Some analytics vendors offer reports based on sampled data sets — this allows for faster report building and is often good enough for spotting basic trends.
But when relying on sampled data, you have to acknowledge that only a part of the data is selected for analysis. In some cases (e.g., with heavily segmenting data), you can’t be sure whether that part is a good representation of your audience.
If you want to make precise decisions, you need access to precise data sets. That is especially important for businesses with a relatively short customer journey, where even a small change can make a huge impact.
That’s why the data analysts can run reports on a complete data set and gain access to raw analytics data.”
Maciej Zawadziński, CEO at Piwik PRO
3. Quantitative research is factual and aims at validating a hypothesis
While qualitative research aims to create hypotheses, quantitative research aims to test them. As quantitative research is factual, you can apply statistical analysis to reject or accept a hypothesis. Because quantitative research is factual, others can replicate it and get the same findings.
Weaknesses of quantitative exploration
Although quantitative research is valuable, it’s not a perfect science. Let’s take a look at the big picture.
1. Quantitative research results are hard to interpret when there is no reference point
When you don’t have another dataset with which to compare yours, it’s hard to interpret your findings. For example, say you’re in the hospitality business. Let’s pretend your website traffic exceeded 500,000 sessions. Is this a lot of traffic or not so much? The numbers won’t tell you anything if you don’t have a reference point.
“One of the drawbacks of using quantitative data is that it can be a hypothesis and a model for data collection. And analysis must be carefully developed while conducting quantitative research.
All of your findings can be thrown out if there are any problems relating to the research design, the bias of the researcher, or any other errors in implementation. Posing a question that you wish to prove or deny can make it challenging to develop a hypothesis.”
Robert Zeglinksi, Managing editor & researcher at Breaking Muscle
2. You only get a fraction of the bigger picture
When pursuing statistical relationships, p-values or correlations, it’s easy to overlook something. This is because quantitative research only gives you a fraction of the big picture.
Imagine you notice that your landing page has a bounce rate of 85%. Unfortunately, you won’t see why it’s at 85% or what’s causing visitors to bounce. Without this information, you won’t be able to optimize your landing page.
Sure, you could always test various hypotheses to determine why the bounce rate is so high, but this is time-consuming.
“Digital tools collect quantitative data to provide a picture regarding the ‘what’ and ‘how.’ You can try to develop the ‘why’ that will inform your strategy, but without qualitative data, it’s hard to know the ‘why.’ That’s one limitation.
But the most significant limitation, at least for marketing, is that collecting quantitative data at the scale we’re doing now will only become more complex. Already, opt-in systematization for online data collection impacts how much data marketers rely on it.
The upcoming end of third-party cookies will further limit our occasions to collect quantitative data. So, in our industry, the future of quantitative data is somewhat uncertain. It is already necessary to diversify how we analyze and evaluate activities.”
Elise Devaux, Marketing director at Statice
3. Potentially biased and misleading
Unfortunately, it’s impossible to state that the numbers are always 100% accurate and there is no bias. Although quantitative research is based on statistics, sometimes it isn’t more credible than observing 5 participants during qualitative usability testing.
The problem with numbers is that bias can be deeply rooted into the system or the mind of the professionals who analyze the data.
“The majority of quantitative research is performed using a subset of a target population rather than an entire population. The findings of this study are then generalized to represent the views of the entire population. This implies that the opinions of a small number of respondents in that study are taken as representative of the general public.
Although their viewpoints may be skewed or insincere, they are regarded as representative of the entire population.
As a result, the mistake of premature generalization is more likely to occur in quantitative research. It may not be the best idea to generalize the opinions of a group of people as their opinions may be prejudiced.”
Adam Fard, Founder & Head of design at Adam Fard
Qualitative research and analysis
What is qualitative research?
Qualitative research gathers descriptive data through observation, interviews, and questionnaires. Qualitative research aims to deeply understand an individual’s experiences, thoughts, and opinions.
The data is generally non-numerical and describes the context of a specific phenomenon. Sometimes, your qualitative research will contain numerical data, but you will use it for additional context, not mathematical computation.
Qualitative research methods include grounded theory, ethnography, action research, narrative research, and more. In the next section, we’ve chosen to focus on grounded theory.
Grounded theory collects qualitative data and derives theories based on descriptive information. As you can see from the image above, it’s an iterative and cyclical process.
How to collect qualitative data for analysis
There are many qualitative data collection methods, including:
- Interviews
- Discussions
- Focus groups
- Usability tests or usability research
- Behavior observations
- Session recordings
- Heatmaps
As for manual data collection through interviews, usability testing or focus groups, you’ll need an experienced researcher who knows how to collect and analyze the data.
As for automatic qualitative data collection, digital tools can collect qualitative data such as session recordings or heatmaps for you.
Now, let’s talk about the benefits and limitations of the research and qualitative data analysis.
Strengths and weaknesses of qualitative research
The strengths of a qualitative approach
- Answers the “why” of a specific phenomenon
One of the most significant benefits of qualitative research is that it deepens your understanding of the matter you’re exploring. While numbers give you a fragmented understanding, qualitative research’s stories and additional context give the complete picture. The qualitative data gives you the “why.”
Let’s see what Aditi Paul, Ph.D. a mixed-methods researcher wants to say about qualitative research.
“The biggest value qualitative data provides to a research project is that it adds context. It makes your story richer. It gives that texture to data-driven storytelling, which if you only rely on numbers, doesn’t make it sexy, for the lack of a better term doesn’t move people.
Numbers will convince you, but stories will move you to decisions. And that’s the beauty of qualitative research. That’s one thing. The second thing, qualitative research is super important in understudied or areas. In other words, in areas where there is a lack of sufficient quantitative research.”
- Provides flexibility
Qualitative data is semi-structured or unstructured. Because of this you get a lot of flexibility in the qualitative research process.
For example, in an interview, you can ask follow-up questions whenever you touch upon an interesting topic. By analyzing session recordings and heatmaps, you’re open to learning something new, often unexpected.
As there is no rigid format in this type of data research, you can be much more detailed in your studies. And details often provide surprising insights and nuances.
- Provides you with deep insight despite the lack of a specific measurement scale
With qualitative research, you don’t have to use specific measurement scales or statistical tools to derive value from qualitative data. The open-ended questions you’re free to ask bring context and depth to your study without thinking about statistics.
The weaknesses of a qualitative approach
- Operating on a small sample size that may not be big enough
As qualitative research is time-consuming, the sample size is often negligible. When looking into a specific case, it’s not uncommon to gather less data than you anticipate.
“To my mind, qualitative research entails amassing vast amounts of personal information, perspectives, and experiences from a large number of people.
As a result, gathering sufficient data from participants takes time, and compiling it is not as straightforward as plotting it on a graph.
Due to the nature of qualitative data, researchers must sift through each individual response to understand why participants felt or reacted in a certain way to a product.
Because responses are frequently descriptive rather than numerical, poring over the results will invariably take longer than parsing quantitative data.”
Bram Jansen, Chief editor at vpnAlert
Because of the small sample size, it’s often hard to obtain a sample of data that’s representative of the views of the wider population. It’s not uncommon for multiple studies to confirm a specific viewpoint, but it’s better to be careful.
BUT, sometimes a small sample size isn’t a bad thing.
Say you have a banking app where users can exchange Euros for US Dollars. If 10 users reported an error (both Android and iOS users), you don’t have to interview another 20 people to confirm it. In this case a small sample size is enough to spot a problem.
- It’s easy to spoil the data
It’s not that easy to conduct proper qualitative research. As there are multiple ways to ask research questions in usability testing sessions, interviews, case studies, and focus groups, influencing a response is not uncommon.
Even the order in which you ask a set of questions can lead to biased answers. Furthermore, the way you ask a question can imply specific inaccurate and biased answers. Let’s see what our expert has to say.
“A very common and highly recommended qualitative research method is simply talking to people. Still, there are many ways you can spoil your data. There is bias lurking in what you ask, how you ask, and how you interpret the data.
Let’s touch on the first theme. We as people are generally horrible at describing our behavior, and we tend to weave pleasant stories rather than give cold hard facts.
The good news is that you can get to a better level of precision by focusing on what you ask.
Predicting our future behavior — we suck at that. Don’t ask people if they would like to use your product or use a feature.
Generalizing past behavior — we suck less at this, but we’re not perfect. Don’t ask people how they typically select shoes or plan vacations.
Describing recent past behavior — there is some noise, but generally speaking, it’s good enough to use it as data. Ask people to tell you how they selected their last pair of shoes and planned their last vacation.
Describing current behavior — this is as good as it gets. Let people show you how they use your product or join them on their trip to a shoe store.”
Adam Fendych, Product manager at OUTFINDO
- Anonymity is not 100% guaranteed to participants
The topic of Individual privacy is growing in popularity. There is a big concern that qualitative data isn’t fully anonymous. During qualitative research, participants share unique insights that can be traced back to them. As a result, not everyone is comfortable sharing their thoughts. As a consequence, countless valuable answers never come to fruition.
According to mixed-methods researcher Aditi Paul, Ph.D., there is something else worth mentioning (it’s not a limitation, but it’s still good to know).
“Quantitative researchers have a straightforward way to establish reliability, versus qualitative researchers have to jump through more hoops to get to that reliability, because they don’t have statistical significance at their disposal.
So you have to refine your thinking even more. And one way of doing that (this goes for both qualitative and quantitative research) is to ground your assumptions as to ground your hypotheses in theory.
Statistical significance doesn’t mean that there is a story behind it and the story is valid. There are so many reasons that say “the higher the temperature, the more robberies” well, it’s statistically significant. But it’s not reliable. It doesn’t mean anything.
Now, where do you get that story from? You get the story from theories, you get that story from underlying principles that have been proven right over and over and over again.”
Qualitative vs quantitative research: use a mixed-method approach for best results
If you’re curious about whether you can combine qualitative and quantitative research, the short answer is yes.
What is a mixed-methods approach?
While both methods have their benefits and limitations, you don’t have to compromise and choose one method over another.
Instead of only looking at numerical or nonnumerical data, look at both to answer the “what” and “why.”For example, say your purchase process contains a bottleneck — quantitative research will show you that something is wrong. However, you won’t be able to see your users actually struggling with the issue. Qualitative research will tell you why users are experiencing this issue.
“It’s essential to understand the ‘what’ your visitors are doing on your website and, more importantly, ‘why’ they are doing it.
You need quantitative data from Google Analytics to understand opportunities for improvement, like entry pages with low conversion rates (the ‘what’). But you also need qualitative customer surveys, user testing, and session recordings to understand the issues causing the low conversion rate (the ‘why’).
A/B test results are only quantitative and don’t tell you ‘why’ a version was successful or not. That is why it’s so essential to gather user feedback and A/B test anything you’re launching — to discover the ‘why’ and what needs to be further improved.
I like using low-cost tools like Usabilityhub.com to ask the target audience what they like most and least about proposed improvements or A/B test variations. This helps ensure website improvements that have a more significant impact on conversion rates and sales.”
Rich Page, Founder at Rich Page: Website optimizer
Here’s another one:
“Combining quant (numbers) with qual (reasons) gives you a far better picture of your users. The more confidence you can achieve, the better the solution will be. It is crucial to base your hypotheses on as many views as possible.
The prioritization framework (PXL) is a good example. It assigns points for analytics, user testing, surveys, heatmaps, and session recordings.
Understanding user motivation is essential for coming up with a persuasive copy. Understanding the data informs you of the severity of the issue found. By combining these two, you improve your chances of coming up with the best solution.”
Ondřej Ilinčev, CRO expert at ILINČEV
Let’s explore another hands-on example. Say you saw a drop in your landing page’s conversions from 5% to 2% last week — the number of visitors remains roughly the same.
Here, the quantitative analysis indicates there is a problem. You’ll need to use qualitative data to understand why the conversion rate dropped.
You can interview your customers and ask what happened when they visited your landing page, but this will be time-consuming. A better way to analyze qualitative data is to watch session recordings and analyze the heatmaps of your landing page. This will give you a good understanding of why the drop in conversions occurred.
“For businesses, understanding the ‘why’ behind their successes and failures helps guide future decision-making.
Qualitative research enables consumers to explain the rationale behind their behavior, informing businesses on how to tailor their products to their target audiences better.
For example, a bank recently launched a new user interface for its mobile customers, but the number of mobile logins decreased significantly. Through a qualitative session, the bank can determine which aspects of its new UI are proving difficult for customers to navigate, whether it is usability, aesthetics, or simply acclimating to change.
While the quantitative data demonstrates that users are logging in less frequently, the qualitative data identifies specific issues that the bank can address.”
Shane Liuw, Chief executive officer at FIRST PAGE DIGITAL
How to approach qualitative vs quantitative research (mixed-methods analysis)
- Leave the misconceptions about qualitative vs. quantitative research behind
It’s a common misconception that quantitative and qualitative data provide opposite viewpoints. In reality, these approaches complement rather than compete with one another.
- Technology gives you ways to move past the limitations of qualitative and quantitative research
It’s no secret that qualitative and quantitative approaches have limitations. Thankfully, technology allows you to move past some of these obstacles. For example:
- A/B testing calculators, like the one on CXL, enable you to conduct statistically significant tests — all you need to do is input the numbers
- Marketing automation software like Hubspot makes it easier for you to reach thousands of prospective customers (so you can collect both qualitative and quantitative data)
- Mopinion gives you access to user feedback software for your online business in a matter of minutes
- Review sites can give you access to heaps of qualitative data (opinions, reviews, etc.)
- Popular e-commerce giants will also give you thousands of product reviews that you can use
- Smartlook collects qualitative and quantitative data automatically regarding your website or native mobile app (every user is recorded, and you can filter data to find exactly what you’re looking for)
- Hire data professionals to ensure your analyses are reliable
Skilled data professionals aren’t cheap. If you collect data but don’t know how to use it properly, you lose your competitive advantage. Besides investing in a digital stack that allows you to combine quality data, a skilled crew of analysts is also a good investment when putting your qualitative and quantitative data to work.
Grow your business even faster with a balanced combination of qualitative and quantitative approaches
If you’re ready to start tracking user interactions on your website or in a native mobile app, sign up for a Smartlook account (no credit card required). Our tool combines the power of quantitative and qualitative analytics, so you can track what users do on your website or mobile app and uncover why they do it.
If you want to see a Smartlook presentation that’s tailored to your business, schedule a demo with our team.
