3.6 Step 6: Appraise the quality of studies to include.
For this section, we define quality as the extent to which various parts of studies coherently fit the whole. Thus, the match among various parts of research design – research question and method, selection of research subjects, sample size data collection process, measurements, analysis, and reporting. In this section, particular attention needs to be paid to the quality of each section of the entire research design. For example, to prioritize papers according to their quality and to exclude certain papers deemed not useful due to inferior methodological quality (Okoli, 2015a). Okoli (2015a) posits that “perhaps the most significant distinction between classes of quality appraisal methods is whether the primary studies are quantitative (i.e., they obtain knowledge by measuring numbers) or qualitative (i.e., use text or other non-numeric data with discussion and argumentation to understand the phenomenon); hence” (p.896).
When appraising the quality of qualitative studies, Okoli highlights that study reviewers should endeavour to distinguish the lines of arguments in the context of inference, assertion, or supposition drawing on logics of how arguments were developed. For theoretical papers, particularly those that aimed at deriving or discovering theories such as grounded theory, the need to identify whether authors premised their arguments along deductive or inductive reasoning is of great importance.
Research quality in IS when focused on digital health has become a matter of life or death given the dearth of knowledge on the unintended consequences and side effects of the application of digital health technologies. As the primary objective of this study, to we posit that, given the increased application of IS in the healthcare environment, there is a growing evidence of identical interventions in the application of IS in the healthcare domain. This warrants stricter methodological quality of the highest standard. The implication of this new evidence unlike Fink’s, is that the advocacy of Petticrew and Roberts (2008) for “leniency of hierarchy of evidence” when conducting SLR in social sciences no longer really hold much on which to reckon, particularly for IS.
On incorporating non-peer reviewed/unpublished papers, we suggested above a scoping review on grey literature for insight or broader idea on the topic being researched as a starting point for systematic literature review. Additionally, high-quality reports with quality/well-cited references could be of great use. Again, as opposed to Fink (2005), Okoli broke ranks with these differences to assert that both qualitative and quantitative lend themselves to rigorous empirical structured methodology of SR to achieve explicit, comprehensive exhaustive, and reproducibility (Okoli, 2015a). To achieve the same level of quality among multiple authors, a standard form developed by all authors would be relevant. See an example developed by Fink.