How to know if a research is valid

In recent times the term “research” is all over the place. When listening to the radio, watching the television or reading a daily newspaper it is difficult to avoid it. Politicians often justify their policy decisions on the basis of “research”. Newspapers report the findings of research companies´ survey. Even advertisers sometimes point out the “results of research” that proof the quality of their product. This fact does not necessarily mean that the number of research has recently increased but just that the term has a wide range of meanings in everyday speech.

Therefore, the question to be answered in this post is what signs may tell us whether a research is NOT valid? See bellow a number of situations when the term is wrongly used:

1. When it is just collecting facts or information with no clear purpose (Wallinman, 2005). The lack of a clear purpose is one of the most common weaknesses among students. It is the result of mislead research with the simple fact of collecting information. In the so called information society, this venture is attainable for anyone able to type a single keyword on the Internet (sometimes committing plagiarism, by the way). But research is more than that. It is about a process that goes from the formulation of a research question to the final stages when the findings and conclusions are discussed. Gathering information is an important stage of the whole process, but not the only one. Gathering hundreds of charts and tables with statistic from Eurostat about the inflation rate in the European Union lacks of any purpose, except getting the reader bored.

Purpose means explaining, describing, understanding, comparing, criticizing, and analyzing (Gauri and Gronhaug, 2005). For instance, in the New York Times article “Poland is not yet lost” by the Nobel Prize economist Paul Krugman (2013), there is a clear purpose. He explains why Poland has avoided the severe slump that afflicted much of European periphery in a context of financial crisis. Based on facts, he suggests that it was thanks to have its own currency and be able to correct the real exchange rate quickly when crisis struck.

Another purpose might perfectly be to criticize such assertion and suggesting other conclusions based on facts (and not beliefs). Or perhaps compare these results with southern European periphery in order to come to a more consistent conclusion.

2. Reassembling and reordering facts or information without interpretation (Wallinman, 2005). “95 per cent of children in Britain had been victims of crime”. From a legal perspective, pushing a classmate or taking a pencil without the intention of returning it is a crime, isn´t it? So the results might be true, but without a clear interpretation of it, it may bring the reader to a sometimes ridiculous conclusion. But we live in the “age of bogus survey” (Kay, 2007) and the lack of interpretation of many researches does not seem to be the priority of many media today. On the contrary, producing eye-catching news does. In other words, research may aids publicist but not the public.

3. Other researches deserve the label of bogus for not providing information about the method used (Saunders et al., 2009). The interpretations must always be based on a carefully description of the method. For example, the results of a survey should always provide the sample size, who conducted it and when, among others. This is something essential to argue why the results obtained are meaningful. Additionally, any limitation associated to the method must be provided. Should you collect responses to your questionnaire from population of Trojmiasto urban area except the municipality of Gdynia, you must inform about this limitation.

4. When the data is not collected systematically. Could be the case that the method used is provided but its application was wrong. Selecting a too small sample size, omit certain groups of respondents that reduce the representativeness of the sample, or formulate the question of a questionnaire in different ways depending on the person interviewed are good reasons to think that a survey may be bogus. However, note that, sometimes, even very well-established researchers are exposed to this kind of vicissitude. Enough to mention Carmen Reinhart and Kenneth Rogoff´s fiasco; at the beginning of 2010 they circulated a paper that proof a significant relation between debt and economic performance. Apart from having received many critics since the beginning for the spurious nature of such relation, a few years later, other researchers, using seemingly comparable data on debt and growth, couldn’t replicate the results. They realized (and denounce) that Reinhart-Rogoff had omitted some data; used unusual and highly questionable statistical procedures. Worst of all, the excel spreadsheet contained a coding error. Unfortunately (this is out of topic), the bogus findings of their research did play a crucial role in the austerity policies implemented since then in many western economies, and especially in the so called PIGS countries.

5. When the word “research” may be used as a term to get some product or idea noticed and respected. “The quality of the product is constantly tested in our laboratories with the most advance technology”. More and more often companies advertisement use these and other bombastic-like phrases in order to give a more serious and rigorous impression. Lab coat experts revealing the benefits of the new yogurt product line are more than well known in television spots. The truth is that the private industry has learned to use such rhetoric with the only intention of improve their credibility, while being very far from what research really mean. In other words, profit interest might be replace what is consider to be the essential interest of a true research, i.e. find out things based on facts and not beliefs.

6. Finally, although it seems obvious, whenever there is no sense on what you are reading. Not just with regard to mass media, but also with academic works. There is a tendency to overvalue certain researches due to the high use of very sophisticated words. This classy style of writing (Becker, 2008) is very often used by certain academics to sound more erudite and as a form of distinction, while not having any sense at all. Worst of all, certain nonsense papers sometimes get round to be published in journals and magazine. A recent experiment led by Dragan Djuric and Boris Delibasic draw attention on this fact. Under the ostentatious title of “Evaluation of transformative hermeneutic heuristics for processing of random data” they got round to publish a fictitious paper. The publisher had accepted a paper with reference from 2012 from the long-gone Bernoulli and Laplace who haven’t published a paper in hundreds of years, as well as Michael Jackson and porn actor Ron Jeremy, who has been moonlighting as an author in the journal Transactions of the Chinese Mathematical Society, (a journal that, according to a simple Google search, doesn’t exist).

In conclusion, the massive production of pedantic, profit-seeker, non-reliable or quasi-scientific might be, in the so called (dis) information era, moving us away from the truth of our society, nature and our human lives or, such as the Polish sociologist Zygmunt Bauman (2003) suggests, the excess of information may be worse than the ignorance itself. Hopefully, the above signs will be helpful, together with an always necessary doses of critical-thinking, to recognize the worthy knowledge.

SEE ALSO

Is the only aim to do the best work

The nature of a good research

What is a good research

Technology, social change and the need of research

Reference list

Bauman, Z. (2003). Educational challenges of the liquid-modern eraDiogenes,50(1), 15-26.

Becker, H. S. (2008). Writing for social scientists: How to start and finish your thesis, book, or article. University of Chicago Press.

Ghauri, P. N. (2005). Research methods in business studies: A practical guide. Pearson Education. Seen in Saunders, M. N., Saunders, M., Lewis, P., & Thornhill, A. (2011). Research Methods For Business Students, 5/e. Pearson Education India.

Kay, J. (2007). Research that aids publicists but not the public,. Financial Times, 30 October. Seen in Saunders, M. N., Saunders, M., Lewis, P., & Thornhill, A. (2011). Research Methods For Business Students, 5/e. Pearson Education India.

Krugman, P. (2013). Poland is not yet lostNew York Times. 27 March.

Krugman, P. (2013). The Excel DepressionNew York Times18. Published: 18 April.

Reinhart, C. M., & Rogoff, K. S. (2010). Growth in a Time of Debt (No. w15639). National Bureau of Economic Research.

Saunders, M. N., Saunders, M., Lewis, P., & Thornhill, A. (2011). Research Methods For Business Students, 5/e. Pearson Education India.

Walliman, N. (2005). Your research project: a step-by-step guide for the first-time researcher. Sage. Seen in Saunders, M. N., Saunders, M., Lewis, P., & Thornhill, A. (2011). Research Methods For Business Students, 5/e. Pearson Education India.

Advertisements

The nature of research

In our second week´s lecture we had the pleasure of welcome five new students. After a brief introduction of themself we went over a few reminders of the past class, such as the main course objective. Bear in mind: “Learning how to collect relevant information for your own research project”. And I emphasized the word “relevant“, because in the so called “Information society” with such search tool as Google, collecting information is frankly easy, but other thing is selecting the most suitable for your research (click here to know more about this semester´s objectives. Bellow you can see the second week lecture structure´s main points:

  1. How to distinguish between a true research & bogus research?
  2. Types of research
  3. What things make management and business research different?
  4. Stages of Research process

Taking into account the great presence that the word research has reached in today´s society in the radio, television or newspaper, as well as the fact that in many occasions it doesn´t refer to research in the true meaning of the word (remind: “finding out information in a systematic way”) distinguishing between a true and bogus research becomes crucial to collect data for our own research. And for doing that I purpose you to ask three main questions. A true research should fetch “Yes” in the bellow questions:

  • Is the only interest finding out information?
  • Has the information been collected and interpreted systematically?
  • Has the research a clear purpose?

For a better understanding we have applied these questions on a example of a research under the title of” “What people really think about public smoking?” As long as a tobacco association is behind this report we should at least be prudent when sourcing the data provided. Secondly, assuming that “systematically” means using a method and providing appropriate details on it, as well as not having biased the answer, we finally considered it as a non-systematic research, since just a few lines were devoted to explain the method used and on top of that the questionnaire is not provided.

Despite it could be a bogus research, we should admit that it has a clear purpose, which is clearly visible in the title, it is, explains what people think about some particular issue, in this case, public smoking. Hopefully, your research will have a so clear purpose too. Although instead of explaining you could prefer comparing, criticizing or analysing, among other purposes.

Moving on the next point, it is also crucial to know the different types of research because it is probably going to be one of your first choices. Apart from the different types we have according to the discipline you are involved in, for example, economic, sociology, management (this is yours), or political sciences; the most elemental difference is between Pure or basic and applied research.

A research under the title “How to increase effectiveness of small business“, basic or applied? What about “Apple´s Ipod sales have decreased 20% last year. What is the reason for this decrease?” There are many differences between both, but probably the most relevant is regarding its application. The basic one aims to produce new knowledge, regardless whether it is useful in a short term, while applied one usually aims solving problems of particular organizations. Coming back to our example, basic or applied? Great, you got it. But, wait a second, what is a organization. Ok, Apple is one, Toyota another one. Yes sure, Google too. But those are just one type of organization. What about a university, a school, cultural association, government, family? Yes, why not. Our class? Yes, although we have just met. In management world there is a tendency to consider organization as a company, but organization may be many things, it is just about a group of people sharing rules and roles. Here we have another important question for you research, which organization is your research about?

Coming back to the basic/applied point. It seems that most of you prefer applied. So you are probably one of those that wonder “what is theory (basic) research for”, “is it really useful” Before Edison  invented the bulb, he had likely been asked these questions too. Perhaps some friend of him could have suggested that “why do you want to invent a new way to illuminate our houses when we already have candles” He was actually right, but could you imaging lead our lesson without electric light? In other words, although basic research findings might not have an immediate application, it could become, in a long term, essential. Therefore, a research under the title “How to increase effectiveness of small business” might not be useful for your neighbourhood bakery, but it could definitively change this industry in the future.

However, such as many authors suggest, although there are important difference between basic and applied research, it doesn´t mean that they are in conflict. In other words, it is about a basic-applied continuum rather that a categorical. It means that in a scale from 1 to 10, where “1” means completely basic and “10” completely applied, your research project might lay on 5.

In 2001, Hodgkinson et al, wanted to go further and suggest taxonomy for considering the relevance of a research, resulting in four quadrants. When a research is relevant in terms of theory and methodology and at the same time it is relevant in a practical meaning (it is applied) it is called pragmatic sciences. In the contrary situation would be puerile science (see “for beginners”). When the research has high theoretical and methodological relevance but low practical relevance, it is considered as pedantic science (in Spanish, “pedante”, in Italian “pedante” and in Chinese 迂) while populist sciences present the opposite characteristics, low practical relevance and high theoretical relevance.

As bachelor students, which quadrant you belong to? Probably to puerile one, right? And struggling to be in populist. This is your challenge, isn´t it?

Look out! Many of the papers you will find will likely belong to pedantic quadrant, or at least those in established scientific journals, hence make a good selection of your reading and not overestimate its complexity, you could save long time (Watch out, the fact that many researches are pedantic doesn´t mean they are boring or useless, it just means that are made for very specific and specialized intellectual circles and that you lack of many concrete concepts to understand them)

Moving on to the third point of our second week lesson, there are three things that make management and business research (M&B research) different and that you should take into account for you project. First it is its tansdisciplinary character. What does it mean? Despite many authors put into consideration whether Management is a science or not, the true is that it usually takes knowledge from many other discipline which is, at the same time, an advantage, provided that this provide many insights. Thinking over your own research, don´t discard draw up other disciplines like economic or sociology when looking for information.

On the other hand, management research is usually much related to decision making. It is very often crucial for successful manager´s decisions. Although many managers still base their decisions on personal experience (Rousseau, 2006), the truth is that in a more and more complex society and organizations, research-based decision making turn out to be essential. And there are many successful cases which are proving it and encouraging other organizations to adopt this strategy. As the one that came out in the movie Moneyball that changed the rules of baseball management. See bellow the official movie trailer.

Last but not least, B&B research must have relatively high practical relevance, i.e., in a basic-applied continuum, it should lay closer to applied, which doesn´t mean that you can rid of the theory, which is actually an essential factor when researching.
Finally, we went also over the research process and, concretely, over research stages, which actually matches with the chapters of our textbook and hence with our next lessons. Here is the result of activity completed by you ranking the different stages. Taking for granted that you wish to do research (at least to pass this course), in the next lecture we will go over the second stage which deals with how to formulate and clarify your research topic. Until then you can start reading my personal blog post on how to find a dissertation topic.

References

Becker, H. S. (2007). Writing for social scientists: How to start and finish your thesis, book, or article University of Chicago Press.

Camino, J. R. (2011). Cómo escribir y publicar una tesis doctoral ESIC Editorial.