Number of Citations: 1

Published online: 2 January, TAPS 2019, 4(1), 24-33
DOI: https://doi.org/10.29060/TAPS.2019-4-1/OA1063

Wee Shiong Lim^{1, 2}, Kar Mun Tham³, Fadzli Baharom Adzahar, Han Yee Neo⁴, Wei Chin Wong¹, Issac Lim², Charlotte Ringsted⁵ & Dujeepa D. Samarasekera⁶

¹Department of Geriatric Medicine, Institute of Geriatrics & Active Aging, Tan Tock Seng Hospital, Singapore; ²Health Outcomes and Medical Education Research, National Healthcare Group, Singapore; ³Department of Anaesthesiology, SingHealth, Singapore; ⁴Department of Palliative Medicine, Tan Tock Seng Hospital, Singapore; ⁵Centre for Health Science Education, Faculty of Health, Aarhus University, Denmark; ⁶Centre for Medical Education, National University of Singapore

Abstract

Background: In recent years, the Asia-Pacific region has become a hotbed of activity in medical education. Little is known about the progress in advancing the field through clarification research studies situated within a strong conceptual framework vis-a-vis descriptive (“What was done?”) and justification (“Did it work?”) research purposes. We aimed to determine the trend in clarification as opposed to descriptive and justification research purposes in the Asia-Pacific region over a 5-year period.

Methods: We conducted a scoping review of original research abstracts presented at the 2008, 2010 and 2012 Asia Pacific Medical Education Conferences. All eligible abstracts were classified as descriptive, justification or clarification using the modified Cook (2008) research purpose framework. All abstracts were reviewed by two researchers, with disagreement resolved by consensus. We performed trended Chi-square tests followed by logistic regression adjusted for covariates, to determine the longitudinal trend in clarification studies.

Results: Our sample comprised 517 abstracts (2008:136; 2010:195; 2012:186). There was a significant trend towards an increase in clarification studies from 2008 to 2012 (4.4% vs 8.7% vs 12.9%, p=.001), even after excluding submissions from non-Asian countries. When adjusted for covariates, abstracts in 2012 significantly predicted a clarification research purpose compared to 2008 (OR 2.74, 95% CI 1.04-7.23). A non-descriptive study design also significantly predicted clarification studies (OR 7.07, 95% CI 3.17-15.75).

Conclusions: Our results of a trend towards increased clarification research purpose affirm the longitudinal progress in quality of medical education research in the Asia-Pacific region. Efforts should be made to promote non-descriptive study designs that are undergirded in clarification research purpose.

Keywords:            Medical Education Research, Clarification Research Purpose, Asia-Pacific, Scoping Review

Practice Highlights

• We should aspire towards clarification studies situated within a strong conceptual framework vis-a-vis descriptive (“What was done?”) and justification (“Did it work?”) research purposes.
• There was a significant trend towards an increase in clarification studies from 2008 to 2012.
• A non-descriptive study design significantly predicted clarification studies.
• Efforts should be made to promote non-descriptive study designs that are undergirded in clarification research purpose.

I. INTRODUCTION

In recent years, the Asia-Pacific region has become a hotbed of activity in medical education (Chou, Chiu, Lai, Tsai & Tzeng, 2012; Fan et al., 2012; Bin Abdulrahman, Harden & Patrício, 2012; Samarasekera, Ooi, Yeo & Hooi, 2015). While much emphasis has been placed on improving standards in the scholarship of teaching (Zhang, Lee, Gruppen & Denian, 2013; E. Ahn & D. Ahn, 2014), comparatively less attention is paid to enhancing the quality of the research-related areas of scholarship, namely discovery, integration and application (Boyer, 1990; Obeidat, Alhaqwi & Abdulghani, 2015). With research being a key driver in informing the theory, practice and policy of medical education (Monrouxe & Rees, 2009; Frenk et al., 2010), we need systematic knowledge of the trends in quality of medical education research in the Asia-Pacific region.

In the ascertainment of quality of medical education research, it is imperative to recognize its uniqueness relative to its biomedical counterpart. Monrouxe & Rees (2009) eloquently pointed out that “medical education research is not a poor relation of medical research; it belongs to a different family altogether.” This necessitates an analogous broadening of the debate about quality in medical education research such that progress is judged from the “macroscopic” lenses of its contribution to the development of deeper insights and understanding of phenomenon to inform theory, practice and policy (Dornan, Peile & Spencer, 2009; Eva, 2009), as opposed to the prevalent “microscopic” focus on evidence hierarchies and detailed quality checklists. Such an approach emphasizes the importance of the conceptual framework to guide thinking about an idea, problem, or phenomenon by engendering generalizable knowledge from theories, models, evidence-based best practices or hypotheses (Rees & Monrouxe, 2010; Gibbs, Durning & Van Der Vleuten, 2011), such that research findings can be transferable to future practice and research (Bordage, 2009; Bunniss & Kelly, 2010).

Harnessing this macroscopic perspective, Cook, Bordage & Schmidt (2008) proposed a typology for classifying the purposes of medical education research, namely description, justification and clarification (Table 1). Description studies address the question: “What was done?” whereas justification studies seek to answer the question: “Did it work?” In contrast, clarification studies seek to answer the question: “Why or how did it work?” through the presence of a conceptual framework that can be affirmed or refuted by the study results (Cook et al., 2008; Ringsted et al., 2011). For instance, applying the Cook framework to an educational innovation that utilizes real-life elderly patients (as opposed to simulated patients) for teaching geriatrics assessment skills to junior medical students, a description study merely describes the educational intervention; a justification study compares the intervention group with a control group on pre-determined outcomes to determine if the intervention works; whereas a clarification study may invoke the situated learning theory to explain how the authenticity of exposure to real-life elderly patients complements aging simulation workshops by improving confidence and creating awareness of practical difficulties (Lim, Ng, Natesan, Wong & Tham, 2012). Applying this framework in his seminal study of 105 articles describing education experiments in 6 major journals, Cook et al. (2008) reported that clarification studies were uncommon (12%) relative to justification (72%) and description (16%) studies. A systematic survey of 850 experimental and non-experimental studies on problem-based learning (Schmidt, 2005) and a more recent study of research presentations at a medical education meeting in Mexico (García-Durán, Morales-López, Durante-Montiel, Jiménez & Sánchez-Mendiola, 2011) similarly reported a low rate of clarification studies.

The original iteration of the operationalized Cook framework was limited only to experimental studies (Cook, Beckman & Bordage, 2007; Cook et al., 2008). Expansion of the scope of the Cook framework to explicate trends in research purpose across non-experimental studies is sorely needed, as studies with a purely descriptive design without any underlying conceptual framework have historically constituted a significant proportion of the literature in medical education (Reed et al., 2008; García-Durán et al., 2011). It is therefore important to determine whether calls to enhance the quality of medical education research through more robust study designs beyond cross-sectional descriptive approaches (Gruppen, 2007) have resulted in concomitant shifts in research purpose beyond description. An opportunity to examine the complex relationship between research approach and research purpose, was afforded by the research compass framework first described by Ringsted et al. (2011). In this classification, research approach can be classified into 2 broad categories, namely experimental studies, with the main aim of justification to define appropriate interventions and outcomes, and non-experimental studies. The latter can be further sub-classified into explorative studies such as descriptive, qualitative and psychometric designs; observational studies which include cross-sectional, case-control and prospective cohort designs; and translational studies such as reviews, implementation studies and effect studies.

By successfully integrating the Cook and Ringsted frameworks, a recent systematic review reported that only one-eighth of medical education studies in the Asia-Pacific region have a clarification research purpose (Lim et al., 2017). This study also identified a clear study aim and non-descriptive research approach as important and potentially remediable areas to promote clarification studies. Little is known, however, of the trends in the quality of medical education research in the Asia-Pacific region and the factors which influence these trends. This represents a critical gap in the literature, since the “video” perspective afforded by longitudinal trends, as opposed to the “snapshot” of cross-sectional studies, can offer far richer insights into the impact of existing regional initiatives that promote scholarship in medical education (Lim, Chong & Sahadevan, 2007; Obeidat et al., 2015), such as centres/networks for research, national and regional conferences, and medical education journals.

In order to explore these issues, we conducted a scoping review of experimental and non-experimental original research abstracts presented at Asia Pacific Medical Education Conference (APMEC) from 2008 to 2012. The APMEC is a major regional conference that serves as an accessible “clearinghouse” that provides a timely and comprehensive snapshot of medical education research in the Asia-Pacific region (Samarasekera et al., 2015). Through this review, we aimed to determine the trend in clarification as opposed to descriptive and justification research purposes of APMEC original research abstracts during the study period. This can then serve as the basis from which we can shape a future research agenda to advance the regional progress of medical education research.

II. METHODS

A. Context

The first APMEC was held in 2004. Over the years, the APMEC has established itself as a major regional conference, attracting participants from South-East Asia, other parts of Asia (including Australia and New Zealand), as well as North America and Europe. Because we wanted to ascertain the trend in research purpose before the introduction of the Research in Essential Skills in Medical Education (RESME) course at APMEC in 2013, we studied the 5-year trend using abstracts from the 5^th, 7^th and 9^th APMECs in 2008, 2010 and 2012 respectively. We chose sampling of alternate years in order to make the workload manageable. The themes for the conferences were respectively “Medical education in a flat world”; “Excellence in medical education – quality in healthcare”; and “Towards transformative education for healthcare professionals in the 21^st century – nurturing lifelong habits of mind, behaviour and action.” Whilst the APMEC has traditionally employed the AMRaC (Aims, Methods, Results and Conclusion) abstract format, this was enforced only from 2009 onwards. The National Healthcare Group Institutional Review Board deemed this study exempt from review.

B. Study Eligibility

We studied original research abstarcts from the 5^th, 7^th and 9^th APMECs in 2008,2010 and 2012 respectively. Original research was defined as an educational intervention or trial; implementation of evidence based practice or guidelines; curriculum evaluation with subjective or objective outcomes; evaluation of an educational instrument or tool; surveys; qualitative research; and systematic reviews (Lim et al., 2017). Besides non-orignal research abstarcts, we also excluded abstracts from plenary lectures, workshops, special interest group meetings and discussions. Two authors (TKM and LWS) reviewed all abstracts to determine eligibility; any disagreement was resolved by discussion. Among 728 eligible abstracts, we excluded 211 that were not original research to yield the final sample of 517 abstracts comprising 136 (26.3%), 195 (37.7%), and 186 (36.0%) abstracts from 2008, 2010 and 2012 respectively (Figure 1). Each abstract was then rated independently and in duplicate. Disagreements were resolved by discussion, and if no consensus was reached, via adjudication by a third independent reviewer. Six reviewers were involved in data collection. We first performed a pilot study using randomly selected abstracts from APMEC 2011. After training and harmonization in the pilot phase, the six reviewers achieved good to excellent agreement in the coding (overall percentage agreement: 80-87%; ACI-statistic: 0.73 – 0.82) (Gwet, 1991).

C. Data Collection

1) Research Purpose

We classified research purpose as description, justification or clarification based upon modified definitions of the Cook framework (Table 1). The hallmark of clarification studies is the presence of a conceptual or theoretical framework that sets the stage for the presentation of the specific research question that drives the investigation being reported (Dine, Caelleigh & Shea, 2015). In our study, we classified the presence of a conceptual framework based on 3 crucial elements: 1) A theory, model, or hypothesis that asks “Why or how does it work?” 2) Transferability to new settings and future research; and 3) Confirmed or refuted by the results and/or conclusions of the study (Lim et al., 2017).

By integrating the Ringsted et al. (2011) framework, we expanded the original definitions of Cook et al. (2008) to accommodate both experimental and non-experimental studies. In the process, we were mindful to adhere to the original spirit of the definitions as far as possible (Lim et al., 2017). Firstly, even though the original definition of justification studies merited a comparison group, we waived this requirement for good quality psychometric studies for which we deemed that there was sufficient rigor in the measures of validity and reliability to answer the question “Does this assessment tool work?” Secondly, to address concerns that certain study designs, such as qualitative and observational studies, would be incongruent with a justification purpose, we delinked where appropriate the hierarchy of purpose from description to justification. Thus, a well-conducted observational study underpinned by a conceptual framework which explains the relationship between independent and dependent variables, would still qualify as a clarification study.

Figure 1. Flowchart of study inclusion and exclusion

*Modified from Cook et al., 2008
** Modified from Ringsted et al., 2011
Table 1. Typology of research purpose and research approach

2) Classification of Study Design

Our classification of study design was modified from the “research compass” framework (Table 1). Abstracts were classified into 2 categories: (1) Experimental, defined as any study in which researchers manipulated a variable (also known as the treatment, intervention or independent variable) to evaluate its impact on other (dependent) variables, including evaluation studies with experimental designs; and (2) Non-experimental, defined as all other studies that do not meet criteria for (1). Studies using mixed methods (for instance, an experimental design with a qualitative component) were classified according to the methodology that was deemed to be predominant.

Experimental studies were further sub-classified as experimental, quasi-experimental or pre-experimental according to established hierarchies of research designs (Creswell, 2013). We defined experimental studies by the presence of randomization; examples included factorial design, crossover design and randomized controlled trials. In contrast, for quasi-experimental studies, experimental and control groups were selected without random assignment of participants. Pre-experimental studies, namely single group pre-post and post-only designs, did not have a control group for comparison.

We also sub-classified non-experimental studies as descriptive, qualitative, psychometric, observational, or translational. Descriptive studies typically provide descriptions of phenomena, new initiatives or activities, such as curriculum design, instructional methods, assessment formats, and evaluation strategies (Ringsted et al., 2011). Because pure descriptive study designs may not strictly qualify as research by some authorities, they are ranked by default as lowest in the hierarchy of study designs (Crites et al., 2014). Hence, when two study designs were identified within the same study with one being descriptive, we coded based upon the “higher” non-descriptive study design.

3) Other Variables

We extracted data on other variables which may affect the quality of medical education research, such as presentation category, topic of medical education addressed, professional group being studied, country of the study population, number of institutions involved, Kirkpatrick’s learner outcomes (if applicable), and statement of study intent. We measured learner outcomes on 4 levels based upon Kirkpatrick’s expanded outcomes typology, namely learner reactions (level 1), modification of attitudes/perceptions (level 2a), modification of knowledge/skills (level 2b), behavioural change (level 3), change in organizational practice (level 4a) and benefits to patients or healthcare outcome (level 4b) (Kirkpatrick, 1967; Reeves, Boet, Zierler & Kitto, 2015). A study reporting more than one outcome was rated based upon the highest-level outcome, regardless of whether this was a primary or secondary outcome. The research question is arguably the most important part of any scholarly activity and is framed as a statement of study intent often in the form of a purpose, objective, goal, aim or hypothesis (Dine et al., 2015). We therefore collected data on whether there is an explicit statement of study intent, and if present, its quality as judged by correct location in the aims section; representation of study goals as opposed to mere stating of educational objectives; and completeness of information (i.e. whether any important objective was omitted).

D. Data Analysis

Results were summarized using descriptive statistics. Chi-square tests were used to compare differences in abstract characteristics between the years 2008, 2010 and 2012. To compare trends in research purpose from 2008 to 2012, we conducted trended Chi-square test with post-hoc pairwise comparisons. We also conducted sensitivity analysis by excluding non-Asian studies. Significant variables from bivariate analysis (P<.10) were included in binary logistic regression analysis to estimate the odds ratios (ORs) and 95% confidence intervals of a clarification study purpose associated with these factors. All analyses were performed using SPSS for Windows version 20.0 (SPSS Inc., Chicago, Illinois, USA). Statistical tests were two-tailed and conducted at 5% level of significance.

III. RESULTS

A. Abstract Characteristics

Among 517 eligible APMEC abstracts from 2008, 2010 and 2012, there was no significant difference in terms of presentation category, number of institutions studied, and Kirkpatrick’s learner outcomes (Table 2). In 2012, there were comparatively more abstracts from Singapore and fewer from the rest of South-East Asia, although the trend in number of abstracts from non-Asian countries remained stable. There were also more abstracts from postgraduate medical and non-medical participants in 2012 compared with 2010 (p=0.003). An explicit statement of study intent was absent or unclear in 45.6% of 2008 abstracts before the introduction of the AMRaC format, compared with 26.7% and 36.0% in 2010 and 2012 respectively (p=0.002). Compared with 2008, there was a trend away from descriptive study designs in 2010 and 2012 (2008–12: 66.9% vs 55.4% vs 54.8%, p=0.057).

*P < .01
Table 2. Comparison of abstract characteristics between 2008 and 2012

B. Research Purpose

There was a significant trend towards increase in clarification studies from 2008 to 2012 (4.4% vs 8.7% vs 12.9%; Trended c² = 11.12, p=.001) (table 3), with post-hoc analyses significant for clarification-descriptive (p=0.004) and justification-descriptive (p=0.04) but not clarification-justification (p=0.190) comparisons. The trend remained significant even after excluding 51 submissions from non-Asian countries (5.0% vs 7.4% vs 12.9%; Trended c² = 8.56, p=.003). In contrast, earlier studies that included a systematic survey of 850 experimental and non-experimental studies on problem-based learning (Schmidt, 2005), research presentations at a medical education meeting in Mexico (Garcia-Duran et al., 2011), and 105 articles describing education experiments in 6 major journals (Cook et al., 2008) reported prevalence figures of clarification studies that ranged between 0.4% to 12.0%.

*Trended Chi-square, P<0.01
^Abstracts from the Asia-Pacific Medical Education Conferences
Table 3. Comparison of trends in research purpose among various studies

C. Logistic Regression

Besides year of study, we also included in the regression model four independent variables with P<.10 in bivariate analysis, namely professional group, country of study, presence of clear study aims, and non-descriptive research approach (table 4). When adjusted for these covariates, abstracts in 2012 were still significantly more likely to have a clarification research purpose compared to 2008 (OR 2.74, 95% CI 1.04-7.23). The 2010 vs 2008 comparison was not signifcant (OR 1.63, 95% CI 0.60-4.41). Among the covariates, the only predictor of a clarification research purpose in multivariate analysis was a non-descriptive study design (OR 7.07, 95% CI 3.17-15.75).

*P < .05; **P < .01
Nagelkerke R square: 0.178
Adjusted for: Professional group, country of study, presence of clear study aims
Table 4. Logistic regression predicting likelihood of clarification research purpose

IV. DISCUSSION

Similar to developments elsewhere, there are calls for the medical education community in the Asia-Pacific Region to recognise and build on its existing research strengths in efforts to justify the value and legitimacy of medical education research (Dornan et al., 2009; Bin Abdulrahman, 2012). To our knowledge, this is the first study to report longitudinal trends in quality of medical education research using the Cook et al. (2008) research purpose framework. Our results demonstrated a trend towards increase in clarification studies from 2008 to 2012 even after excluding submissions from non-Asian countries, thus affirming the longitudinal progress in quality of medical education research in the Asia-Pacific region. In addition, we further explicated the observed trends by demonstrating that a non-descriptive research approach independently predicted a 7-fold increased odds of a clarification research purpose in multivariate analysis. The strengths of our study include duplicate review at all stages; clear and detailed description of the methods/procedures involved; adjustment for relevant covariates in multivariate analysis; and high inter-rater agreement among the coders.

We believe that this gradual regional shift in emphasis of research purpose within the Asian-Pacific field towards theory-driven and theory-building clarification research is a healthy and excellent development that is consonant with the wider evidence-based movement for research to inform, influence and impact the practice and policy of medical education (Eva & Lingard, 2008; Bordage, 2009). Theory can help structure and facilitate all aspects of the research process; it can help us generalise our study results beyond their immediate settings, promote collaboration within and between fields, open possibilities for new programs of research, attract funding and ultimately, sustain the virtuous cycle of quality medical education research (Gill & Griffin, 2009; Rees and Monrouxe, 2010; Kuper & Whitehead, 2013). As previously noted, trends in prevalence of clarification studies in our study compared very favourably with the 0.4% – 12.0% range which was reported in earlier cross-sectional studies in non-Asian settings (Schmidt, 2005; Cook et al., 2008; Garcia-Duran et al., 2011). It is reassuring that in our study, the longitudinal increase in clarification studies was accompanied by a corresponding decline in description research purpose. Taken together, this strongly supports the commendable progress made in promoting theory-grounded clarification studies even before the introduction of the RESME course at APMEC 2013. Possible explanations include the catalytic role of ongoing regional initiatives such as centres of excellence, research networks, national and regional conferences, and dedicated medical education journals to promote scholarship in medical education (Obeidat et al., 2015; Lim et al., 2017).

It is important to keep contextual developments in mind when interpreting the observed longitudinal trends in research purpose. For instance, adoption of the Accreditation Council for Graduate Medical Education International framework for postgraduate training in Singapore in 2009 could explain the surge in abstract submissions in the postgraduate medical category from Singapore at APMEC 2012 (Huggan et al., 2012; Samarasekera et al., 2015). Likewise, Interprofessional Core Curricula and Interprofessional Enrichment Activities were incorporated into Medical, Nursing, Dentistry, Pharmacy and Medical Social Work undergraduate programmes at the National University of Singapore (Jacobs et al., 2013), along with concomitant interest in interprofessionalism at the workplace (Tan, Adzahar, I. Lim, Chan, & W.S. Lim, 2014), can explain the encouraging rise in submissions from non-medical professional groups. Similarly, we believe that enforcing the structured format for abstract reporting in 2009 may have been the driving force behind the observed trends of improvement in aims statement. Our results corroborate evidence that structured formats improve the quality of reporting of research abstracts (Wong et al., 2005; Varpio, Amie & Richards, 2016), which is in turn positively associated with superior methodological quality (Cook, Levinson & Garside, 2011) and improved funding for medical education research (Reed et al., 2007). There is thus a case to be made for the consistent use of structured abstracts with relevant and thoughtful headings, for instance, a separate heading for conceptual framework or study hypothesis to spur the development of higher-order clarification studies.

The seven-fold increased odds of a clarification research purpose with non-descriptive research approach, supports the greater use of more rigorous non-descriptive study designs in medical education research (Gruppen, 2007). As opposed to strict adherence to a rigid hierarchy, the best study design would be the design that best answers the research question within a given context (Lim, 2013). Thus, awareness should be raised about the plurality of non-experimental non-descriptive approaches such as qualitative, psychometric, observational and translational research designs, which will encourage the broadening of the repertoire of research questions that can be asked to deepen understanding and advance the field. To borrow an analogy from the biomedical framework of translational science, this is akin to building up the T1 “bench to bedside” basic science evidence which can then inform educational practice and policy, and ultimately, patient care and outcomes (McGaghie, 2010).

Our findings should be interpreted within the limits of our study design. Our decision to sample APMEC abstracts may explain the predominance of abstracts from Singapore and the neighbouring South-East Asian countries. Nonetheless, the distribution of research purpose reported in our study is broadly consistent with existing literature. Future work could explore the generalizability of our results by comparing with other regional and international conferences (e.g. the Association for Medical Education in Europe [AMEE] conference) during the same timeframe. Our use of conference abstracts, which has a significant word constraint as compared to full-length papers, means that the validity of our findings is highly dependent on the reporting quality of the abstracts, such that our results may be more a reflection of the reporting quality rather than the actual quality of research. Notwithstanding, evidence affirming the positive relationship between reporting and ultimate methodological quality lends credence to the validity of assessing conference abstracts as an indirect quality indicator of research (Cook et al., 2011). Moreover, our research involved essential and fairly objective elements of reporting such as study aims and outcomes. Lastly, whilst our choice of timeframe before 2013 was effective in removing the direct influence of the RESME course on quality of APMEC abstracts, it will be prudent and salient to consider prospective studies to further ascertain if the trends in research purpose herein reported will prove sustainable.

V. CONCLUSION

Our results of a trend towards increased rigor of research purpose, even after excluding submissions from non-Asian countries, affirm the steady progress made in the quality of medical education research in the Asia-Pacific region in the last 5 years. We also highlighted the importance of promoting non-descriptive study designs that are undergirded in a clarification research purpose. We applaud recent regional initiatives to enhance the research capabilities of the Asian-Pacific community, such as the Asia Pacific Medical Education Network (APME-Net), the Asian Medical Education Association, regional journals such as The Asia-Pacific Scholar, formal research programs such as Masters and PhD degrees, and consensus statements to prioritize the research agenda. We encourage collaborative efforts with international partners to promote scholarship opportunities; examples include offering research courses such as RESME at regional courses, tie-ups with international journals to publish abstracts of regional conferences, and cross-cultural research to explicate mutual areas of interest involving pedagogy, issues and approaches (O’Sullivan, Stoddard & Kalishman, 2010; Suhoyo, Van Hell, Prihatiningsih, Kuks & Cohen-Schotanus, 2014). These initiatives augur well for fostering a culture of impactful educational research and scholarship in the Asia-Pacific region that aims to inform the theory, practice and policy of medical education through clarification studies.

Notes on Contributors

W. S. Lim conceived the study, and led the design, data collection, analysis, and drafting of the manuscript. K. M. Tham was involved in study selection, data collection and analysis. F. B. Adzahar, H. Y. Neo, W. C. Wong, and I. Lim were involved in the implementation of the study and data collection. C. Ringsted and D. D. Samarasekera were involved in the conception and design of the study, and provided revisions to the manuscript. All authors read and approved the final manuscript.

Ethical Approval

The study was approved by the institutional review board of the National Health Group (NHG DSRB Ref: 2012/00814).

Funding

The study was supported by an educational research grant (NHG-HOMER FY11/A08) from the National Healthcare Group Health Outcomes and Medical Education Research office.

Declaration of Interest

The authors report no declarations of interest.

References

Ahn, E., & Ahn, D. (2014). Beyond accreditation: Excellence in medical education. Medical Teacher, 36, 84-85.

Bin Abdulrahman, K., Harden, R., & Patrício, M. (2012). Medical education in Saudi Arabia: An exciting journey. Medical Teacher, 34, S4-S5.

Bin Abdulrahman, K. A. (2012). The value of medical education research in Saudi Arabia. Medical Teacher, 34, S1-S3.

Bordage, G. (2009). Conceptual frameworks to illuminate and magnify. Medical Education, 43(4), 312-319.

Boyer, E. L. (1990). Scholarship reconsidered: Priorities of the professoriate. New York, NY: Jossey-Bass Books.

Bunniss, S., & Kelly, D. R. (2010). Research paradigms in medical education research. Medical Education, 44(4), 358-366.

Chou, J. Y., Chiu, C. H., Lai, E., Tsai, D., & Tzeng, C. R. (2012). Medical education in Taiwan. Medical Teacher, 34, 187-191.

Cook, D. A., Beckman, T. J., & Bordage, G. (2007). Quality of reporting of experimental studies in medical education: A systematic review. Medical Education, 41(8), 737-745.

Cook, D. A., Bordage, G., & Schmidt, H. G. (2008). Description, justification and clarification: A framework for classifying the purposes of research in medical education. Medical Education, 42(2), 128-133.

Cook, D. A., Levinson, A. J., & Garside, S. (2011). Method and reporting quality in health professions education research: A systematic review. Medical Education, 45(3), 227-238.

Creswell, J. W. (2013). Research design: Qualitative, quantitative, and mixed methods approaches. Sage publications.

Crites, G. E., Gaines, J. K., Cottrell, S., Kalishman, S., Gusic, M., Mavis, B., & Durning, S. J. (2014). Medical education scholarship: An introductory guide: AMEE Guide No. 89. Medical Teacher, 36(8), 657-674.

Dine, C. J., Caelleigh, A. S., & Shea, J. A. (2015). Chapter 6: Problem statement, conceptual framework, and research question. In Durning, S. J., & Carline, J. D. (Eds.), Review criteria for research manuscripts 2nd ed. (pp. 3-6). Washington, DC: Association of American Medical Colleges.

Dornan, T., Peile, E., & Spencer, J. (2009). In defence of the existing strengths of medical education research. Medical Education, 43, 388–391.

Eva, K. W., & Lingard, L. (2008). What’s next? A guiding question for educators engaged in educational research. Medical Education, 42(8), 752-754.

Eva, K. W. (2009). Broadening the debate about quality in medical education research. Medical Education, 43(4), 294-296.

Fan, A. P., Tran, D. T., Kosik, R. O., Mandell, G. A., Hsu, H. S., & Chen, Y. S. (2012). Medical education in Vietnam. Medical Teacher, 34, 103-107.

Frenk, J., Chen. L., Bhutta, Z. A., Cohen, J., Crisp, N., Evans, T., & Zurayk, H. (2010). Health professionals for a new century: Transforming education to strengthen health systems in an interdependent world. Lancet, 376, 1923–1958.

García-Durán, R., Morales-López, S., Durante-Montiel, I., Jiménez, M., & Sánchez-Mendiola, M. (2011). Type of research papers in medical education meetings in Mexico: An observational study. Paper presented at the Annual Meeting of the Association for Medical Education in Europe, Vienna, Austria.

Gibbs, T., Durning, S., & Van Der Vleuten, C. (2011). Theories in medical education: Towards creating a union between educational practice and research traditions. Medical teacher, 33(3), 183-187.

Gill, D., & Griffin, A. E. (2009). Reframing medical education research: Let’s make the publishable meaningful and the meaningful publishable. Medical Education, 43(10), 933-935.

Gruppen, L. D. (2007). Improving medical education research. Teaching and Learning In Medicine, 19(4), 331-335.

Gwet, K. (1991). Handbook of Inter-Rater Reliability. STATAXIS Publishing Company.

Huggan, P. J., Samarasekara, D. D., Archuleta, S., Khoo, S. M., Sim, J. H. J., Sin, C. S. P., & Ooi, S. B. S. (2012). The successful, rapid transition to a new model of graduate medical education in Singapore. Academic Medicine, 87, 1268–1273.

Jacobs, J. L., Samarasekera, D. D., Chui, W. K., Chan, S. Y., Wong, L. L., Liaw, S. Y., … & Chan, S. (2013). Building a successful platform for interprofessional education for health professions in an Asian university. Medical Teacher, 35, 343-347.

Kirkpatrick, D. L. (1967). Evaluation of training. In Craig, R. L. & Bittel, L. R. (Eds.), Training and development handbook (pp. 87-112). New York, USA: McGraw-Hill.

Kuper, A., & Whitehead, C. (2013). The practicality of theory. Academic Medicine, 88, 1594–1595.

Lim, W. S., Chong, M. S., & Sahadevan, S. (2007). Utility of the Clinical Dementia Rating in Asian populations. Clinical Medicine & Research, 5, 61-70.

Lim, W. S., Ng, A., Natesan, S., Wong, W. C., & Tham, K. Y. (2012). Situated learning enhances authenticity of learning experience of geriatric assessment skills among junior medical students. Annals Academy Medicine Singapore, 41, S37.

Lim, W. S. (2013). More about the focus on outcomes research in medical education. Academic Medicine, 88(8), 1052.

Lim, W. S., Tham, K. M., Adzahar, F. B., Neo, H.Y., Wong, W. C., & Ringsted, C. (2017). Determinants of clarification research purpose in medical education research: A systematic review. The Asian Pacific Scholar, 2, 8-17.

McGaghie, W. C. (2010). Medical education research as translational science. Science Translational Medicine, 2, 19cm8. https://doi.org/10.1126/scitranslmed.3000679.

Monrouxe, L. V., & Rees, C. E. (2009). Picking up the gauntlet: Constructing medical education as a social science. Medical Education, 43, 196-198.

Obeidat, A. S., Alhaqwi, A. I., & Abdulghani, H. M. (2015). Reprioritizing current research trends in medical education: A reflection on research activities in Saudi Arabia. Medical Teacher, 37, S5-S8.

O’Sullivan, P. S., Stoddard, H. A., & Kalishman, S. (2010). Collaborative research in medical education: A discussion of theory and practice. Medical Education, 44, 1175–1184.

Reed, D. A., Cook, D.A., Beckman, T. J., Levine, R. B., Kern, D. E., & Wright, S. M. (2007). Association between funding and quality of published medical education research. JAMA, 298(9), 1002-1009.

Reed, D. A., Beckman, T. J., Wright, S. M., Levine, R. B., Kern, D. E., & Cook, D. A. (2008). Predictive validity evidence for medical education research study quality instrument scores: Quality of submissions to JGIM’s Medical Education Special Issue. Journal of General Internal Medicine, 23(7), 903-907.

Rees, C. E., & Monrouxe, L. V. (2010). Theory in medical education research: How do we get there?. Medical Education, 44(4), 334-339.

Ringsted, C., Hodges, B., & Scherpbier, A. (2011). ‘The research compass’: An introduction to research in medical education: AMEE Guide No. 56. Medical Teacher, 33(9), 695-709.

Reeves, S., Boet, S., Zierler, B., & Kitto, S. (2015). Interprofessional Education and Practice Guide No. 3: Evaluating interprofessional education. Journal of Interprofessional Care, 29, 305-12.

Samarasekera, D. D., Ooi, S., Yeo, S. P., & Hooi, S. C. (2015). Medical education in Singapore. Medical Teacher, 37, 707-13.

Schmidt, H. G. (2005). Influence of research in practices in medical education: The case of problem-based learning. Paper presented at the Annual Meeting of the Association for Medical Education in Europe, Amsterdam, The Netherlands.

Suhoyo, Y., Van Hell, E. A., Prihatiningsih, T. S., Kuks, J. B. M., & Cohen-Schotanus, J. (2014). Exploring cultural differences in feedback processes and perceived instructiveness during clerkships: Replicating a Dutch study in Indonesia. Medical Teacher, 36, 223-229.

Tan, K. T., Adzahar, F. B., Lim, I., Chan, M., & Lim, W. S. (2014). Transactive memory system as a measure of collaborative practice in a geriatrics team: Implications for continuing interprofessional education. Journal of Interprofessional Care, 28, 239-45.

Varpio, L., Amie, J., & Richards, B. F. (2016). Writing competitive research conference abstracts: AMEE Guide no. 108. Medical Teacher, 38, 1-9.

Wong, H. L., Truong, D., Mahamed, A., Davidian, C., Rana, Z., & Einarson, T. R. (2005). Quality of structured abstracts of original research articles in the British Medical Journal, the Canadian Medical Association Journal and the Journal of the American Medical Association: A 10-year follow-up study. Current Medical Research Opinion, 21, 467-473.

Zhang, Q., Lee, L. M., Gruppen, L. D., & Denian, B. A. (2013). Medical education: Changes and perspectives. Medical Teacher, 35, 621–627.

*Wee Shiong Lim
11 Jalan Tan Tock Seng,
Department of Geriatric Medicine,
Tan Tock Seng Hospital, Singapore 308433
Telephone: +65 6359 6474
Fax: +65 63596294
Email: wee_shiong_lim@ttsh.com.sg