Teaching Project Management on-line : lessons learned from MOOCs

Creating a course for teaching project management online in a full online distance-learning environment was a challenge. Working with adult learners from different continents that want to complete a Master degree was an additional challenge. This paper describes how different MOOCs were used to learn about teaching -(meta) e-learning. MOOCs provide diverse opportunities for teachers to learn and innovate in e-learning. From the analysis of 5 MOOCs in the broad field of project management we took important lessons on how to structure contents, how to prepare complex assignments and, the most important lesson of all, how to help students to learn. This paper describes our journey of learning from MOOCs how to be better online teachers.

Introduction to the challenge
Universidade NOVA de Lisboa has a fully online Master Degree on managing e-Learning Systems. Most students are adult learners, coming from different parts of the world, mostly from Portuguese speaking countries. In the 2015 edition, the programme was re-organized and a new teaching team was assigned to the course "Education Projects Management". The new teaching team, the authors of this paper, was given complete autonomy to restructure the course around the following Learning Outcomes (LOs): · The student will be able to identify and understand basic concepts and structure of project management in the educational context; · The student will be able to apply tools and techniques to develop an educational project, from idea generation to finalization; · The student will be able to understand the specificities of project management in the context of e-learning; · The student will be able to identify, group and organize information structures hierarchically, as well as to develop communication objects developed to disseminate information; · The student will get to know alternative approaches to project management, in particular design thinking.
Project management is a useful transferable skill that learners can apply to their learning activities, their profession and their personal life. Managing projects can be a very exciting and stimulating job. However, learning about project management can be exactly the opposite: too technical and bland. Students in this programme have different backgrounds and goals for the future. What they have in common is that they are adult learners, and even though they consider project management as a useful skill and an important tool, it is not their main interest.
In this context, the challenge we faced was mostly related with motivation: how do we teach project management online, in an international context, in a way that is engaging and rewarding to the learners and to the teachers? Some of the more specific questions we were facing were: Open Praxis, vol. 8 issue 4, October-December 2016, pp. 351-358 · What is the best approach to teach a fully online course in project management? · What is the right balance between theory, knowledge and practical work? · How to make learning interesting and engaging but not overwhelming to the students? · How to support students so that they don't fall behind or drop the course?
From the interpretation of the online learning continuum (Guàrdia, Maina, & Sangrà, 2013), in the far right end we can find MOOCs (Massive Open Online Courses), classified as fully online distributed learning, with complex and integrated use of ICT. Even though our goal was not to build a MOOC, we wanted to build a fully online course that uses technology in an integrated way, to deliver learning opportunities and to support students in their learning journey. For these reasons, we went to look for answers to our questions in existing courses, mainly in MOOCs.
This paper focuses on the lessons learned from analyzing different MOOCs in the area of project management. We consider this as being an innovative practice and perspective. MOOCs can be a tool for "Massive Learning", not only for students that want to learn about a topic but also for teachers and trainers that want to learn about how to develop and deliver online learning. We can say it is a type of meta-(e)Learning, as we are learning about e-learning through e-learning in practice.

Initial research and adopted pedagogical approach
With the challenge described above, we aimed at finding a student-centered approach to teaching project management that was useful and meaningful to each student and took advantage of their experiences. Most of our students are adult learners and we wanted to take advantage of their different backgrounds to build an enriching experience for all.
Given the nature of the course, we decided to adopt a problem based-learning approach (PBL) that allowed students to learn about project management, while doing a project-learning by doing. As described by Barrows and Tamblyn (1980), PBL is the ideal approach when you want studentcentered individualized learning that promotes both the acquisition of knowledge related with the problem and the development and application of specific skills. This facilitates the integration of knowledge and skills that students use to solving that problem. Another important aspect was that the projects should be realistic and relevant for the students, instead of being chosen by the teachers. This was, in our opinion a crucial factor to get students engaged with the work and be able to reuse the process when trying to solve new problems. The four goals of PBL as described by Barrows (1986) and adapted by Biggs (2007), are: · G1: Structuring knowledge for use in working contexts · G2: Developing effective reasoning processes · G3: Developing self-directed learning skills · G4: Increased motivation for learning · G5: Developing group skills, working with colleagues.
These goals are aligned with our idea for the course. We consider project management a set of transferable skills that students should be able to take to professional lives and apply them in different situations (G1). Project management is based on a structured process of making decisions and acting accordingly, i.e. effective reasoning (G2). Our view for the course was that students should learn-by-doing, taking hold of their learning, deciding every aspect of the project (G3). Motivation was the most challenging aspect of learning about project management. Using a PBL approach, with a strong a relevant practical component was the solution for this problem (G4). Finally, being a full online course with an international cohort of students, developing collaboration skills and networking is a desirable goal (G5). PBL provided us with the theoretical framework and pedagogical approach. But we needed to define the practical approach and tools for developing the course. We adopted the 7Cs of Learning Design (Conole, 2014) as an instrument to guide us in the design and development of the course. The framework consists of seven stages: The first stage, conceptualize, was in part addressed above. We described the background, the target audience, the LOs and the pedagogical approach. Still, we needed to go deeper in the conceptualize stage.
Our strategy was to research what was being done in e-learning in terms of project management. As stated above, in the online learning continuum (Guàrdia et al., 2013) MOOCs are at the far end, representing distance learning solutions fully online, highly dependent on ICT. These three features were the same as our course. Given the proliferation of MOOCs in the last few years, their openness, availability and diversity it was a logical decision to use them as a source for inspiration about teaching project management online but, most of all, MOOCs represent an opportunity for learning empirically about e-learning strategies, the "(meta)e-learning". This option was reinforced by literature. Guàrdia et al. (2013) have identified ten design principles for MOOCs from the learner perspective: 1. Competence based approach 2. Learner empowerment 3. Learning plan and clear orientations 4. Collaborative learning 5. Social networking 6. Peer assistance 7. Quality criteria for knowledge creation and generation 8. Interest groups 9. Assessment and peer feedback 10. Media-technology-enhanced learning These ten principles are aligned with our vision for the course, focusing on student-centred learning, networking, collaboration, competence based learning and students as producers of knowledge. So, for the conceptualize stage of the 7Cs, authors researched courses in the field of project management in the main MOOC providers: Coursera and EdX.
The first step was to analyse courses addressing project management. We selected three courses: "Project Management: The Basics for Success" by University of California in Coursera (University of California, n.d.); "Introducción a la Gestión de Proyectos" by UPValencia in EdX (UPValenciax, 2015); "Introduction to Project Management" by AdelaideX in EdX (AdelaideX, 2016). These courses followed a similar structure both in terms of contents and teaching/learning activities: · Structured units for each component of project management · Videos with lectures · Support documents · Quizzes for knowledge check Open Praxis, vol. 8 issue 4, October-December 2016, pp. 351-358 These courses are clearly xMOOCs, MOOCs with a teacher and content-centered approach (Conole, 2015;Guàrdia et al., 2013;Lugton, 2012). They focused on providing knowledge about each stage of project management. There was no alignment with the PBL model that we were looking for, nor with the 10 design principles for MOOCs listed above. Still, these MOOCs were useful in terms of providing an overview on contents in project management: idea, planning, risk analysis and evaluation.
Still, we wanted to have an approach that was more engaging for the learners and teachers. During our search, we came in contact with Design Thinking. Design thinking, as defined by Tim Brown from IDEO (2016) is "a human-centered approach to innovation that draws from the designer's toolkit to integrate the needs of people, the possibilities of technology, and the requirements for business success". It provides a creative and human-centered way to address challenges, problems or projects. IDEO has published a toolkit with a specific approach for using design thinking fin education (IDEO, 2012). There was a clear alignment between our vision for the course and the design thinking approach: the focus on the challenge; use of technology; people-centered; collaboration. We went back to the MOOC providers and analyzed two MOOCs in the field of design thinking. The two courses analyzed were cMOOCs, MOOCs with a focus on the student and building networks of collaboration (Conole, 2015;Lugton, 2012): · Design Thinking for Innovation, by University of Virginia (Coursera) · Product Design: The Delft Design Approach (EdX) The first MOOC is a very interesting course in terms of the way the lessons are structured: · Modules are built around questions · Video(s) introducing the module with general contents and concepts · Video(s) with case-studies related with the module content, including interviews with professionals in the field · Video(s) focusing on one specific approach or tool of the design-thinking framework Introducing the content in general terms gives the student the global perspective. Then, providing the "real-world" perspective of the same module, with concrete experiences narrated with people involved in the case. Finally, going back to the theory but focusing on practical tools or instruments that the students can use in different contexts.
In terms of assessment, this MOOC only has one assignment: an individual reflection by the student, assessed by peers. Even though the assessment is open, the MOOC provides several support tools to help students in completing the assignment and peer reviewing: · Clear instructions about what is expected from each role · Assessment rubric with instructions for applying and scoring · Example of the completed assignment · Peer-review practice quiz In terms of collaboration, the course includes different fora and study groups where students interact with each other and with mentors. Summarizing, the main lessons learned from this MOOC were as follows: · Structuring the course around questions · Using short videos with different perspectives of the same content/issue · Providing useful tools that students can take away with them to apply in other contexts · Robust guidance for completing the course · Networking and collaboration opportunities "Product Design: The Delft Design Approach" (TU Delft Online Learning, 2015), provides important insights in terms of structuring the course, interacting with students and problem-based assessment. The general course framework, the way the lessons are structured, how the assignments are introduced, all of these contribute to helping students engage with the course materials and activities. The course has 7 modules with a similar structure.
· Introduction video that reviews the work of the previous module and introduces the new work · Two or more short lectures in video · Quizzes to control the learning of the module · Exercises with clear and complete description of what the student is expected to do · Templates for the exercises to guide the student · Benchmark videos that document "model" students completing the exercises, so the student can compare their work · Example of completed exercises · Videos with experts related with the module · Videos with reflections about the assignments This MOOC has a strong social and collaborative component. Students use the discussion fora to publish and review their work and their colleagues' work. The course uses social networks for helping creating a community.
The main lesson retrieved from this MOOC is the importance of supporting and guiding students in online distance learning. The MOOC includes several strategies with this sole purpose. Lectures were very short videos, with no more than 7 minutes. These small chunks of information make it easy for the student to keep attentive. Following every lecture, there was a quiz, so that the student could check their learning. Another interesting strategy was having two "model" students that had go through the course and complete every task. These students are always present through videos and they interact with the camera in a very informal way to create empathy with the learner. The delivery of the assignments is also student-centered. Every assignment includes very clear and complete instructions of the tasks the student has to accomplish. Well-structured templates are provided to facilitate and guide the students. They can watch the "model" students going through the whole process of completing the assignment. Experts provide insights about the topic of the week and give real world context about the work that is being done. Every week, the teaching team prepares a video with reflections about some of the issues raised in the weekly forum. Another interesting aspect is that weekly assignments build on the assignment of the previous week. Using this strategy, at the end of the course, students have a complete project implemented step-by-step and did not feel overwhelmed by a big task or demotivated by completing small erratic tasks.
All these strategies represent a great amount of work and a huge effort on planning and producing resources. But it provides a structured support to the student that is a crucial aspect in MOOCs and in other distance learning contexts. Table 1 summarizes the analysis of the five MOOCs in the framework the 10 design principles listed above (Guàrdia et al., 2013), using "y" for compliance and "n" for non-compliance.

Implementing the lessons learned
After the conceptualization stage of the 7Cs of Learning Design, we proceeded to the next: capture, create, communicate and collaborate. These consist in defining the practical approach to the course. The first step was to define the content of the course (capture and create). We decided to have weekly modules, each built around one question related with project management. We were trying to make students understand, right from the start, the relevance of what they were learning: · Introduction video, reviewing the previous week and introducing the new week. · Short lecture videos · Quiz about the module · Additional readings · Assignment:

Completed example
Another important issue was the balance between the theoretical and practical components. We wanted students to feel that what they were learning was relevant to them. Working with adult learners, relevance is an important issue (Newman & Peile, 2002). Also, we wanted students to contribute with their experiences and making the course more enriching to all. Assignments are a great opportunity to achieve this. The strategy adopted was to have a small weekly assignment, linked to the theme of the module. In the first week they worked on the idea using a mind-mapping tool. On the second week, they identified stakeholders, defined priorities and analyzed their needs. The same strategy was used for the following weeks. At the end of the course students completed a full project, decided by them in every aspect, according to their ideas and interests. At the end, students had two weeks to review the project and present it in a single integrated document. As the assignments were aligned with course content, this final step was important to integrate the new knowledge (Consider and Consolidate).
A final issue of the course was the final exam that was compulsory for every course of the Master. As our course has a strong practical component, we decided to follow the same line and do a casestudy analysis focusing in critical aspects of project management. There was an intentional alignment between the exam and the weekly modules and assignments to increase the consistency of the course (Anderson, Krathwohl, & Bloom, 2001).
Collaborate and Communicate were the "Cs" where there was less investment. It is intended to improve these two missing Cs in the next edition.

Conclusion
Designing a e-learning course from scratch is an opportunity to innovate and learn. MOOCs are learning tools for learners to acquire knowledge and competences in different areas. But MOOCs create opportunities to "learn by example" how to teach in innovative ways, to (meta)e-learn. We have available courses from universities around the world, with different views, different strategies with different teams. It is a wonderful opportunity to learn about learning, teaching, assessing, motivating and engaging students.
In the course "Education Projects Management" we invested in analyzing these five MOOCs and it was an important step that helped us to define the strategy for the course. As a result, students were engaged and motivated and only two students of a total of twenty-two did not complete the course (one of them never logged on). All the others completed the 14 assignments on time, as requested. Feedback from students was very positive and the results of the final exam were coherent with their performance during the semester.
The main lesson learned from MOOCs was the importance of having a well-structured guidance system that facilitates and supports learning. Making the learning process easy for the students does not mean lowering the standard. It means putting the student at the center, creating a supportive and transparent learning environment where students know, at all times, what is expected from Open Praxis, vol. 8 issue 4, October-December 2016, pp. 351-358 them and what to expect from us. It is not an easy task and it requires thorough planning, thinking ahead and a lot of work. We were not able to implement every detail of our strategy but next year we will do better and the year after that one, we will do even better. Learning about teaching is a never-ending process.


Evaluation of high myopia complications prevention program in university freshmen

Abstract High myopia is a global eye health problem because of its high incidence of sight-threatening complications. Due to the role of awareness, self-examination, and preventive behavior in prevention of morbidity of high myopia complications, promoting knowledge, capabilities, and attitude of high myopic personnel are required in this regard. In this quasi-experiment study, 31 freshmen with high myopia in a national university were enrolled in 2014. The data were collected by validated and reliable questionnaire based on health belief model (HBM) and self-efficacy theory. The intervention program consisted of 1 educational session lasting 150 minutes by lecturing of high myopia complications, virtual reality experiencing, similarity modeling, and quibbling a film made on high myopia complications preventive concepts. Implementing the educational program showed immediate effect in knowledge, perceived susceptibility, perceived severity, self-efficacy, and preventive behavior intention. While 6 weeks after the educational program, significant increases were observed in cues to action, self-efficacy, and preventive behavior intention. This article provided that, after a single session, there was positive improvement in high myopia complication prevention behavior intention among participants. These positive effects confirmed the efficacy of the education program and will probably induce behavior change.

Introduction
The apparent worldwide rise in the prevalence of myopia has been a public health impact because of the associated concomitant increase in potentially sight-threatening ocular complications. Persons with high myopia (spherical equivalent -6.0 D) are more susceptible to some ocular abnormalities. In several studies, higher risks of cataract and open-angle glaucoma were reported in myopic adults. [1][2][3] And the risks of choroidal and retinal abnormalities such as choroidoretinal degeneration, lacquer cracks, subfoveal choroidal neovascularization, peripheral retinal degeneration or breaks, and rhegmatogenous retinal detachment, [4] also increased with the degree of myopia and axial length elongation. [5] The early detection and treatment of possible pathological ocular complications are essential in preventing or reducing the morbidity of outcomes in high myopic patients. The impacts of high myopic complications cannot be underestimated especially in young adults. But people with high myopia are often unaware that their high myopia puts them at considerable increased risk of pathological outcomes. There are well-established preventive actions, such as regular check-ups of intraocular pressure, central and peripheral retina, optic nerve, and any pathological signs by a certified ophthalmologist; checking one's own vision and visual field for each eye with same target in same distance and wearing protection glasses during work or sports. However, people with high myopia usually will not perform these actions because of ignorance or negligence. Therefore, the prevention or reduction of ocular complications associated with high myopia, a public health issue in the regions with high prevalent rate of high myopia, needs to be treated in educational dimension.
Health belief model (HBM) was one of the efficient models in studying preventive behaviors. [6] The model was first developed in the 1950s and cooperated with self-efficacy theory in 2008. [7] The components of this model are perceived as severity, susceptibility, benefits, barriers, and self-efficacy. Behavior was explained by HBM as the combination and interaction results of perceived susceptibility (belief that high myopia persons are susceptible to develop some kind of ocular complications), perceived severity (belief that high myopic complications are serious sight-threatening diseases), perceived benefits and perceived barriers to perform actions (high myopic complications prevention behavior intention), and self-efficacy (judgments of own capabilities to organize and execute courses of action required to prevent high myopic complications). The HBM has been applied frequently in studies for breast cancer self-screening, quit smoking, drug abuse, and sexual behavior. However, information and instruction courses about high myopic complications prevention were few in colleges and communities. The purpose of this study, based on the HBM and self-efficacy theory, was to evaluate the effectiveness of single session high myopic complications prevention program in promoting prevention behavior among high myopic university freshmen.

Methods
This was an 1-group pre-and posttest design of quasi-experiment study. The first-grade students of National Taiwan Normal University were recruited during freshman physical examination. Students with myopia spherical equivalent equaled or more than 6 diopters, detected by auto-refractor machine (Topcon KR800, Tokyo, Japan), without any previous ocular trauma, diseases, or surgery were invited to join the study. Those students, who were willing to participate in the study, were arranged to receive cycloplegic refractory examination to make sure their myopic spherical equivalence fit with the criteria described above. According to Cohen study in 1988, [8] the total number of 28 participants was needed to obtain a significant result at the given 2-tailed alpha value 0.05, effect size (r) 0.50, and power level equals 0.80. We estimated 20% drop-out rate, thus total 42 students were enrolled. Of these students, 5 students dropped out due to their absence for intervention program or did not complete secondary questionnaire (posttest). Furthermore, 6 students dropped out due to their inability to complete the third questionnaire (post posttest) 6 weeks after intervention program. Therefore complete data of 31 students were analyzed.
For gathering data, we designed a questionnaire based on the HBM and self-efficacy theory. The questionnaire was reviewed by 10 experienced colleagues (2 professors, 1 vice professor, and 4 assistant professors in health promotion and health education department, and 3 senior ophthalmologists) for revising its content and construct validity. Reliability analysis was conducted for knowledge, self-efficacy, preventive behaviors, and HBM components (perceived susceptibility, severity, benefits, barriers, and cues to action). Internal consistency of these sections of the questionnaire was tested in 30 sophomores with high myopia and calculated using Chronbach alpha technique (Chronbach a value was 0.55 for knowledge, 0.80 for self-efficacy, 0.89 for preventive behaviors, 0.92 for susceptibility, 0.88 for severity, 0.91 for benefits, and 0.86 for barriers). The whole document was approved by the Research Ethics Committee of National Taiwan University on September 11, 2014 (ID code: NTU-REC No: 201407ES021).
The questionnaire included questions regarding the sociodemographic characteristics of students (such as sex, department of college). To assess knowledge, we used 16 items on knowledge about eye anatomy, high myopia definition, incidence and correlated symptoms/signs of complications, self-examination and prevention activity with multiple choice questions. Then, each correct response was scored 1 point and each wrong response was scored 0. To assess cues to action, we used 11 items on whether the students have ever got related messages from any resource. Then, each "yes" response was scored one point and each "no" response was scored 0. Other questions consisted of perceived susceptibility (7 items on estimation of the probability of high myopia complications that will happen to oneself), perceived severity (10 items on estimation of the possible impacts of high myopia complications to oneself), perceived benefits (7 items on the awareness of the benefits to take preventive actions), perceived barriers (12 items on the possible difficulties for oneself to perform preventive behaviors), self-efficacy (10 items on assessing one's capability to perform required behaviors), and intention of preventive behaviors (13 items on self-examination, self-protection, and medical consulting behaviors about high myopia complications preventive activities). They were scored on a 5-point Likert scale as totally agree, agree, not decided, disagree, and definite disagree, and responses gained 5, 4, 3, 2, and 1 scores, respectively.
A pretest was performed while students receiving cycloplegic refractory examination. Group intervention program was held 1 week after pretest completed. The intervention program consisted of an educational session lasting 150 minutes and separated into 4 parts. The first part was a lecture with Power Point presentation about eye anatomy, high myopia definition, incidence and prevalence of correlated complications, symptoms/ signs of complications, simple self-examination methods, and daily prevention activity. In the second part, aimed at clarifying perceived threats of high myopia complications, students experienced the visual impacts of high myopia complications with virtual reality glasses ( Fig. 1) simulating retinal detachment, macular degeneration, advanced glaucoma, and cataract. In the  Tseng and Chen Medicine (2016) 95:40 Medicine third part, aimed at enactive attainments, perceived similarity to models, and verbal persuasion, students had an informal discussion with someone who had overcome retinal detachment caused by high myopia. In the last part, aimed at internalizing of knowledge and promoting effort feedback, students jointed a fault-finding game in a 5-section video which was recorded based on high myopia complications preventive concepts. After the educational session, posttest was done immediately with the same questionnaire. Then, 6 weeks later, post posttest was done also with the same questionnaire.
The obtained data were analyzed by SPSS (version 20.0, Chicago, IL) using descriptive statistics and 1-way analysis of variance (ANOVA) at the significant level of a < 0.05.

Results
Data from 31 freshmen of a national university who completed the whole program were studied. All of them had myopia (spherical equivalent) equaled or more than 6 diopters. There were 10 male and 21 female students in our study. Descriptive statistics on variations between pretest, posttest, and post posttest showed scores of posttest were the highest one in variables as knowledge, perceived susceptibility, perceived severity, selfefficacy, and preventive behavior. The scores of perceived benefits and perceived barriers in all 3 tests were almost the same, while the post posttest score of cues to action was the highest one compared to the other 2 (Table 1). ANOVA test on variations between pretest, posttest, and post posttest showed Table 1 Mean score and stander deviation of all variables in 3 tests.

Variables
Pretest (mean ± SD) Posttest (mean ± SD) Post posttest (mean ± SD)   Tseng and Chen Medicine (2016) 95:40 www.md-journal.com significant difference (P 0.05) in knowledge, perceived susceptibility, perceived severity, cues to action, self-efficacy, and preventive behaviors as described in Table 2. Post hoc comparison showed that posttest was significantly higher than the other 2 tests in knowledge, perceived susceptibility, perceived severity, self-efficacy, and preventive behaviors; while post posttest was significantly higher than pretest in self-efficacy and preventive behavior. In brief, immediate effect of education program was noted in knowledge, perceived susceptibility, severity, self-efficacy, and preventive behaviors, while delay effect only occurred in self-efficacy and preventive behavior. The effect in cues to action did not happen until post posttest. There were no effect at all concerning perceived benefits and perceived barriers.

Discussion and conclusions
Early diagnosis and early treatment are the most important methods in treating high myopia complications. Personal awareness about high myopia complications is the key point of early diagnosis. Nevertheless, our public health system is short of resources for people to be aware of the symptoms or signs of high myopic complications and how to protect themselves from the complications. In the year of 2006, about 16.85% of third grade high school students in Taiwan were high myopia, and the prevalence was more than 35% in university freshmen at the same time. [9] Thus, university or college became the most important community to promote health education of high myopia complications prevention.
In this study, the mean scores were increased significantly in knowledge about high myopia complications prevention immediately after educational program, but 6 weeks later there was no significant difference comparing with pretest. According to Ebbinghaus' forgetting curve there was a loss of about two-thirds of memory in a day and only 21.1% remained after 1 month. [10] Thus, it was not surprising that, after 6 weeks, education information received from our program could not be saved too much to make a difference. It was said that both cognitive and neural mechanisms could support the permanent storage of memory traces. [11] From a cognitive perspective, Averell and Heathcote [11] concluded that some memories remain free from the unfavorable effects of interference and recovery cues provided by the environment may provide a mechanism that reduces the interference. Thus, it seemed that multiple or repeated education programs may not be more effective in reviving memory than a warm-care phone call from important others or an environmental cue.
Turning to the results of our study, we found that perceived susceptibility and perceived severity had only immediate effect, while perceived benefits and barriers had no effect at all. An important review of HBM in health behavior studies conducted between 1974 and 1984 revealed perceived barriers was the only one and most powerful predictor of all health behaviors. [6,7] However it was not quite the story with the respect of perceived barrier under different medical care system. A study in Taiwan, 2.6% of patients had visited specialist of the same department at different hospitals on the same day and such a doctor-shopping behavior ratio increased up to 23.5% in a 7-day time setting. [12] The main reason was that patients in Taiwan can go to any hospitals or clinics for their own convenience without any transfer or appointment. And the reimbursement based on a fee-for-service is not high enough (ranging from 1.5 USD to 15 USD) to be a barrier for people who want to seek more consultations. [12,13] We would speculate that those results for perceived susceptibility, perceived severity, and perceived benefits in our study may be due to difficulties that study respondents have in conceptualizing these dimensions. These difficulties may be caused by the respondents being asymptomatic, not perceiving the health threats, or having little or no personal experience in such medical conditions. [7] Temporary rise in perceived threats may just due to images introduced in our education program, especially in part 1 and 2. It seemed that, according to the discussion above, HBM model may not suitable for predicting preventive behavior if the respondents are asymptomatic and the causal relationships are more complicated or specialized. [13] In our study, cues to action showed significant difference in post posttest. It was probably due to sleeper effect. The works of Pratkanis et al, [14] which were examined and verified by Kumkale and Albarracín, [15] produced data that documented and explained the sleeper effect. They found that memory of the "discounting cue" (information that the messenger was unreliable) disappeared more quickly than memory of the message. Naturally, when the negative information about the source was forgotten, but the message itself was remembered, the message gained more credibility. Thus, after a period of time, persuasive message of intervention program will make respondents pay more attention to information of high myopic complications and prevention issues.
Self-efficacy refers to people's judgments regarding their ability to perform a given activity [16,17] and is proposed to influence individual choices, goals, emotional reactions, effort, ability to cope, and persistence. [18] Many researches have shown that selfefficacy is the main determinant of preventive health behaviors. [19,20] According to studies of Bandura [17] and Schunk, [21] there are 4 types of interventions could affect self-efficacy: enactive attainments, perceived similarity to models, verbal persuasion, and feedback. Observing a coping model improved more self-efficacy for learning, motivation, and long lasting self-efficacy and skill, than observing a mastery model did. [21] One type of the coping models, coping-emotive model, directed to the highest self-efficacy for learning because subjects viewed themselves more competent than or not inferior to the model. [21] The belief that one is more talented than a model can raise self-efficacy and motivation to engage task. [21] In the third part of our program, students had informal talks with 2 patients who had overcome retinal detachment caused by high myopia. It was speculated that through the interventions of perceived similarity to models and feedback, the respondents raised their self-efficacy and motivation.
According to Bandura's theory, [16] one's sense of efficacy is the single and the most necessary motivational element, moving individuals to action. However, self-efficacy is not the only influence on behavior. High self-efficacy will not produce a competent performance when requisite knowledge and skill are lacking. [21] In our study, from picking wrong moves in a 5-section video which was recorded based on high myopia complications preventive concepts, self-efficacy and preventive skill of respondents were promoted.
Also, outcome expectations, or beliefs concerning the probable outcomes of actions, are important because people always go for positive outcomes. There were studies provided evidence that both outcome expectancies and perceived self-efficacy were the best couple predictors of the motivation in regular breast cancer detection behaviors, [22,23] and prostate cancer screening. [24] In our study, through lecturing and informal discussion with victims of high myopia complications, respondents realized the benefits of regular check-ups and good outcome expectations were speculated to enforce self-efficacy. As in high myopia complications, early detection and early treatment will reveal good prognosis. In our education program, we not only clarified knowledge about high myopic complications and prevention methods, but presented real models of high myopic complications patients who went through the task of examinations and treatments with good outcomes. Post hoc comparison showed only self-efficacy and preventive behavior intention had delay effect 6 weeks after intervention program. It was probably, according to the discussion above, due to improvement of knowledge, outcome expectancy, and selfefficacy after intervention program.
In conclusion, 1 session of education program could transmit prevention knowledge of high myopia complications, rise temporary perceived threats, induce a sleeper effect of cues to action, and promote a 6 weeks' self-efficacy coupled with prevention behavior intentions. Long-term changes in behavior could be expected with repeated education programs, goal progress feedback, and appropriate environmental cues. Major variables of HBM (perceived susceptibility, perceived severity, perceived benefits, and perceived barriers) were not quite distinguishable in our study. Unlike the well-established causal relationships studies (such as breast self-examination and early detection of breast cancer, [25] condom use and reduced sexual transmission diseases [26] ), theories other than HBM may be more suitable in predicting some highly specialized preventive behavior of asymptomatic respondents. Further educational programs should be designed with more emphasis on positive aspects of high myopic complications prevention behavior and with the aims of raising knowledge, attitude, and self-efficacy level of individuals. Besides, repeat sessions and environmental cues after educational programs would be also valuable.
There are several limitations of our study, as the sample size was not large enough, self-estimate and repeated questionnaire may be interfered by memory, interpretation error, or questionnaire itself. This was a study of a group of college freshmen with high myopia, the results could only represent the behavior changes of the population.


Non university sources of science in Chile

The following ideas are widely accepted in Chile with respect to scientific activity: is carried out mainly in universities and science is considered a naturally university activity, that was developed in a period of more than 150 years by isolated individual efforts, c) it was transformed into an institutionalized activity at the universities after the university reform movement at the end of the 1960 decade, d) the activity is finally institutionalized in the country with the creation of the “Comisión Nacional de Investigatión Científica y Tecnológica, CONICYT (National Commission for Scientific and Technological Research) in 1967. This work presents preliminary findings showing that there are other institutional initiatives, different to the efforts from universities and directly dependent of the Chilean Government, in order to produce science and technology in Chile. This governmental initiatives start at the beginning of the Republic of Chile circa of 1810.

Introduction
The scientific activity produces more results than can be handled by universities. These results impact all areas of society: economy, health, education, natural resources, environment, freedom and welfare of citizens, international relations, defense, popular sovereignty and national sovereignty. This has been understood many years ago by the developed countries, where scientific activity is carried out in universities and state institutes, private centers, public and private productive sector and armed forces. Contrary to what happens in developed countries, in Chile more than 95% of the scientific research is conducted at universities, virtually without connection with state institutions, the productive sector and the armed forces, which makes practically impossible that the cycle Research + Development + Innovation can be completed.
The stories about the history of science in Chile consider mainly the contribution of individuals to scientific activity. Starting back with the contributions of Claudio Gay, a French scientist, who in 1830 was hired by the Government of Chile for a study the natural history of the area and everything that contributes to know the industry of the country, its commercial activities and management. In 1838 the Chilean government hires the Polish scientist Ignacio Domeyko to work as a teacher at the "Liceo de Coquimbo" (High School of Coquimbo). After that, between 1840 and 1846 Domeyko makes trips through the territory of Chile and describes its geology.
Henceforth, the various stories of the history of science in Chile refer to the contributions of individuals without relation with a policy or institutional interest in developing scientific activity in Chile (see for example reference [1]). In fact, scientific research is not considered an important activity in Chilean universities until after the university reform from the sixties of the twentieth century.
Thus, nowadays one may find the following ideas in the Chilean intellectual imaginary with respect to the scientific activity in Chile: a) is carried out mainly in universities and science in Chile is considered a naturally university activity, b) that was developed in a period of more than 150 years by isolated individual efforts, c) it was transformed into an institutionalized activity at the universities after the university reform movement at the end of the 1960 decade, d) the activity is finally institutionalized in the country with the creation of the "Comisión Nacional de Investigación Científica y Tecnológica, CONICYT (National Commission for Scientific and Technological Research) in 1967.
However, Chile has about 17 State Research Institutes. State Research Institutes are institutions for scientific and technological research created by the State outside the universities. Their goal was to increase the productivity of various economic and strategic sectors through the introduction of technological innovations and the creation knowledge. To do so, among his tasks it to included the development of basic and applied research, coordinating and dictating policies in this area, and to train specialists [2].

Research institutes created by the Chilean State.
Although the Bacteriological Institute was created in 1929, and the Institute of Experimental Medicine in 1937, the golden age of the State Research Institutes is the second half of the twentieth century, between 1957 and 1970. To get an idea, apart from the above, during that period the following institutes were created: Geological Survey (1957) (1970).
From the mid-1970´s and 1980´s most of these institutions suffer a strong reduction in budget and self-financing policies are imposed to them [2]. During the subsequent governments, since 1989 to date, there are no policies oriented to the State Research Institutes. Government policies and instruments include incentives to universities and to the productive sector. None of these policies and instruments consider the particular reality of State Research Institutes. Thus, at present most of these are far from fulfilling one of their original objectives, mainly, to develop basic and applied research. In most of them their main activities are the provision of services and routine production activities. Even though, they have a small number of professional researchers, they have a lot of highly skilled and competent of other type of professionals and technicians. In addition, many of them have infrastructure and equipment that the universities do not have .

Others non-university initiatives previous to the State Research Institutes.
In a recent study entitled "Institutos de Investigación del Estado: Evolución y Perspectivas" (State Research Institutes: Evolution and Perspectives) presented at the "Encuentro Ciencia, Tecnología y Sociedad 2014" in Santiago, Chile [3], it is shown that there were other non-university initiatives, inclusive previous to the State Research Institutes, linked to government policies or public companies that allowed the creation of many organisms and institutions whose objectives were to conduct scientific and technological research.
A chronology regarding the creation of Chilean non-university scientific and technological institutions, it can be described as follows:


A Study on Moral Reasoning among Indian Police officers

Objectives: The aim of this research is to find out the moral reasoning levels of a sample of police officers in the North Zone in Tamil Nadu. Methods/Analysis: A descriptive research study has been used and the researcher used convenience sampling. The research instrument used is DIT2, the defining issues test 2. DIT2 instrument has been scored by the center for ethical development, University of Alabama. Findings: The respondents were found to be in the Maintaining Norms stage of moral reasoning. The police officers didn’t have high score of moral reasoning as indicated by the poor N2 scores. Police officers are not using higher moral reasoning in their profession. There is an urgent need for training the police officers in facing moral dilemmas with higher levels of moral reasoning. This would provide inner satisfaction to the officer and help mould the society into having higher standards of ethics. Novelty/Improvement: The research can be conducted by measuring the moral reasoning levels prior to a training program on ethics and post training program. *Author for correspondence

Introduction
Moral reasoning is a part of an internal component process involved in judging which action would be most justifiable, in a moral sense, that leads to moral behavior 1 . The development of moral reasoning is believed to be a "gradual and continuous process" as the individual advances through a sequence of increasingly complicated moral stages. It forms the cornerstone of ethical behavior [2][3][4][5][6][7] .
Ever since Piaget 2 and Kohlberg 3 conducted their pivotal studies on moral development, scholars have studied how children develop moral reasoning and how they learn the social and moral norms that regulate human interaction. In particular, Kohlberg identified three levels of moral reasoning: a preconventional level (i.e., a morality focused on the self in an egocentric manner), a conventional level (i.e., a morality totally affected by Indian Journal of Science and Technology, Vol 9(29) society's conventions and expectations), and a postconventional level (i.e., a morality based on ethical principles that protect basic human rights such as the right to life, liberty and justice). Individual reasoning should manifest itself as a single dominant stage across a wide spectrum of situations 8 . However, some studies showed that the level of moral reasoning a person uses is sensitive to context. For instance, people do not use their highest level of moral judgement when they reason about an everyday moral dilemma 9 or those involving drunk driving 10 . Thus, moral dilemmas possess properties that may change the participant's decision 11 .
Rest's (1986) Defining Issue Test is the most often used instrument for measuring moral development. Many studies have found support for its psychometric quality 12 . In his paper in "Police Stress: An Examination of the Effects of Stress and Coping Strategies" showed as to how police officers deal with stress that greatly affect as to how Keywords: DIT2, Dilemma, Moral Reasoning, N2, Police Officer, Postconventional Level they carry out their daily lives and how they treat family and friends 13 .
In general, the conduct and moral values of police officers, both on and off duty have been guided by a professional set of standards or "code of ethics". A review of these standards indicate that overall, police officers are entrusted with special authority and responsibility to enforce laws and serve the welfare of the public through believing in the dignity and worth of every individual and the constitutional rights of each individual. Police officers place a high value on objectivity, integrity, honesty and achieving the highest standards of professional service. It has been suggested that the use of psychometric instruments to measure job appropriate qualities, including values in the selection of police officer, is important 14 . Hence, police officers should possess characteristics associated with moderate to high levels of moral reasoning.
The purpose of this research is to find the moral reasoning of police officers in Tamil Nadu.

Procedure
This research study involved the voluntary participation of police officers. The North Zone police officers were circulated the questionnaire from the North Zone Inspector General of Police office. The permission to conduct the survey was obtained from the Director General of Police in police headquarters.
During the month of June, 2014 the researcher contacted the ADGP (Additional Deputy General of Police) Law & Order and submitted a petition explaining the purpose of the research study, permission to distribute research materials to police officers in North Zone, the usefulness of the study and a set of questionnaire to be circulated. The ADGP forwarded the request with a note to DGP. The questionnaire was scrutinized and approved by the Chief of Police to distribute the questionnaire to North Zone Police officers. The researcher was asked to contact the IGP (Inspector General of Police) North Zone for the purpose of collecting data. The researcher met the IGP North Zone and explained the purpose of research and asked him about the number of officers in North Zone. He told to submit 500 questionnaires as it required voluntary participation. So on July 10 th 2014 a total of 500 sets of questionnaires were submitted to the North Zone IGP office. Each police officer was to receive a large envelope containing a cover letter, the research materials (DIT2) and a pencil with eraser and sharpener. The dispatch clerk was responsible to distribute the questionnaires to the 6 districts of North Zone. The subjects were asked to complete and submit the questionnaire to the district police quarters.
Subjects were informed through a cover letter of the purpose of the study and that participation was voluntary and that the questionnaires were strictly for research purpose. They were also told that the information provided would be kept confidential. The subjects were advised that it would take approximately 75 minutes to complete all the questions. Subjects were advised to complete the research materials all at once or within several sittings as per their convenience when they were free from duty. Each subject was asked to read the instructions carefully and to be as honest as possible in responding to the data items. Upon completion of their research materials, participants were asked to submit it to their officer incharge.
The first set of responses was received from the district of Cuddalore in the month of September, 2014. Out of 80 questionnaires sent, 53 responses were received. Next from Villupuram district out of 80 questionnaires sent, 55 responses were received. A reminder was sent to the remaining 4 districts from the North Zone office. Since there was a delay in returning the questionnaires, the researcher was asked to telephone the SP (Superintendent of Police) in the 6 districts. The researcher contacted the Thiruvanamalai SP and requested for the return of research materials. She took action immediately and returned back 50 responses out of 80 questionnaires dispatched to them. The researcher then contacted Vellore DSP and he arranged for return of 60 responses out of 80 questionnaires dispatched to them. Researcher contacted Kancheepuram SP and he gave his assistant's details to coordinate the return of questionnaires. The inspector from kancheepuram district returned 40 responses out of 92 questionnaires dispatched to them. Last of all the researcher contacted Thiruvalur SP and requested him for return of questionnaires. Unfortunately, there were no responses obtained from 88 questionnaires sent to them. So out of 500 questionnaires sent the researcher could get back 258 responses. Out of 258 responses received 102 Vol 9 (29) | August 2016 | www.indjst.org responses were fully filled up. So roughly there has been a response rate of 20%.
The researcher was informed by the female inspector from Villupuram district were the researcher attended the crime meeting of a coming up 2-month training program in TNPA in december 2014 of 600 special sub inspectors. The researcher immediately consulted the North Zone IGP and got his consent to approach the TNPA. In November 2014, the researcher met the SP in TNPA and she advised me to get permission from ADGP Training in Ashok Nagar Police training college. The researcher wrote a petition to ADGP Training and met him in November in person and explained to him the purpose of research and the questionnaire. He immediately approved the petition and researcher was asked to contact TNPA Vandalore. By contacting ADSP in TNPA, the researcher was asked to circulate the questionnaire on 5 th of January 2015 to the group of 600 trainees who had assembled in the parade ground. The researcher was able to get 425 responses by January 25 th . Out of 425 responses the researcher could consider only 197 filled in responses for the study.
Once the data were collected and checked for completeness, the data was coded and entered in excel sheet for statistical analysis.

Instrument
This study used Rest and Narvaez's (1998) DIT-2. Rest indicated that the DIT measures moral reasoning at Stages 2 through 6.
Constructing a way to assess moral judgement is fundamental in moral judgement research; assessment precedes and is presupposed in every research operation. The four major sets of decisions in constructing a test of moral judgement are as follows: a) what features of thinking are to be used in characterizing someone's moral judgement? b) What information -collecting procedure will be used? Rest, 1969, used subjects' discussions of prototypic stimulus statements. c) Once data have been collected from a particular subject, then how does one use this information to index the subject's development? And d) how does one validate a test of moral judgement? What kinds of studies count in establishing reliability and validity?
Kohlberg's 3 theory of moral development asserts a sixstage sequence in which person's progress from lower to higher stages by a series of transformations. Each transformation produces qualitative changes that move the person toward more comprehensive, complex modes of reasoning about moral issues. This development progression is marked by the replacement of lower stage, less comprehensive modes of reasoning by higher stage, more comprehensive modes. The various stages of moral reasoning indicated in Table 1 shows that the police officers are mainly in the Maintaining Norms stage. The maintaining norms schema score represents the proportion of items selected that appeal to Stage 4 considerations. Stage 4 considerations focus on maintaining the existing legal system, maintaining existing roles and formal organization structure.

Results and Discussion
secondary schema. Type has been used by Yeap (1999) and Bebeau (2002) to provide a finer grained evaluation of pretest/ posttest change scores. The research findings indicate that majority of the police officers fall in the Maintaining Norms stage of moral reasoning. Also most of the police officers are in Type 3 developmental indice, which means they are in maintaining norms schema and transitional profiles. As officers strictly follow the rules and regulations it's logical to accept their moral reasoning levels. They have poor N2 score which shows that they are not having higher levels of moral reasoning like philosophers. The humanitarian/ liberalism exhibited by the officers is also very low. Which shows they aren't having a universal attitude while dealing with moral dilemmas.
This research study proves that police officers do not have high levels of moral reasoning. It's advised that ethical training be given to officers not only during initial entry training, but throughout their career progression.
The study needs to be taken after giving the respondents training on ethics. Larger sample can be used to find out the moral reasoning levels of police officers.

Conclusion
The research aimed at finding the moral reasoning level of a sample of police officers and found that they are not having higher levels of moral reasoning. As per the profession of police officers they can be expected to be in the Number of "Can't decide choices" is a variable to represent the decisiveness with which an individual selects action choices on the DIT. Because indecision is thought to be a product of the ease with which moral information is processed, there is the expectation that indecision will co vary with developmental phase 1 . Specifically, transitional phases should be associated with increased indecision given the multiple and potentially conflicting interpretations associated with transition.
Humanitarian/ Liberalism is a variable which indicates a humanitarian liberal perspective on moral issues. This variable measures to what extent individuals are similar to professionals in political science and philosophy who obtain highest P scores.
The Utilize score represents the degree of match between items endorsed as most important and the action choice on that story.
Type Indicator: Depending on schema preference and whether the profile is consolidated or transitional, it is possible to envision seven different types.
Type 2 refers to profiles that are predominant in personal interests schema, but are transitional. Type 3 refers to profiles that are predominant in maintaining norms schema, but are transitional; and in personal interests Table 3. Developmental Profile maintaining norms stage. But in order to improve the efficiency and satisfaction of the police officer we need to take steps to increase their moral reasoning level.
It would in turn fasten the judicial procedures and also lead to increase in public faith in the police service.


extfieldsofstudy: Sociology Medicine

text:
How Multidisciplinary Are the Multidisciplinary Journals Science and Nature?

Interest in cross-disciplinary research knowledge interchange runs high. Review processes at funding agencies, such as the U.S. National Science Foundation, consider plans to disseminate research across disciplinary bounds. Publication in the leading multidisciplinary journals, Nature and Science, may signify the epitome of successful interdisciplinary integration of research knowledge and cross-disciplinary dissemination of findings. But how interdisciplinary are they? The journals are multidisciplinary, but do the individual articles themselves draw upon multiple fields of knowledge and does their influence span disciplines? This research compares articles in three fields (Cell Biology, Physical Chemistry, and Cognitive Science) published in a leading disciplinary journal in each field to those published in Nature and Science. We find comparable degrees of interdisciplinary integration and only modest differences in cross-disciplinary diffusion. That said, though the rate of out-of-field diffusion might be comparable, the sheer reach of Nature and Science, indicated by their potent Journal Impact Factors, means that the diffusion of knowledge therein can far exceed that of leading disciplinary journals in some fields (such as Physical Chemistry and Cognitive Science in our samples).

Introduction
Interest in cross-disciplinary research knowledge interchange runs high. US funding agencies, including the National Science Foundation and the National Institutes of Health, as well as the National Academies of Science [1][2][3][4], promote programs with the explicit goal of fostering interdisciplinary research. And, over time, research in general is becoming more interdisciplinary [5].
Elsewhere we have explored interdisciplinary research counterposed against other forms of research, expressly disciplinary and multidisciplinary, with note of the potential of developing future fields via transdisciplinarity to go beyond previous fields [6][7][8]. Collectively, researchers in the field will sometimes bundle efforts extending beyond one discipline as cross-disciplinary. In this paper, we build on an operationally oriented definition from the National Academies report on Facilitating Interdisciplinary Research: "Interdisciplinary research (IDR) is a mode of research by teams or individuals that integrates information, data, techniques, tools, perspectives, concepts, and/or theories from two or more disciplines or bodies of specialized knowledge to advance fundamental understanding or to solve problems whose solutions are beyond the scope of a single discipline or area of research practice" (p. 188) [1].
The essence here is the requirement of integration of knowledge (of various forms, potentially done by teams or individuals). A key distinction arises between interdisciplinary-as reflecting a greater degree of integration-and multidisciplinary-as more reflecting juxtaposition [6,9].
In recognition of the promise that interdisciplinary research holds for addressing complex scientific problems with societal implications, the National Science Foundation (NSF) directs grant reviewers to consider a proposal's plan to disseminate findings across disciplinary bounds in order to have a broader impact. One of us (GS), a program director at NSF, has frequently heard panelists reviewing grant proposals, program directors reviewing progress reports, and external evaluators reviewing the impact of funding programs comment as if publication in Nature or Science (in addition to affording a not inconsiderable "Stamp of Scientific Approval" for appearing in arguably the most important scientific journals in the world) signified in itself the successful interdisciplinary integration and diffusion of findings.
But does it? How interdisciplinary are Nature and Science? Articles in them are explicitly chosen not only because they move their specific literatures in important ways, but also because of their potential to interest-and to influence-multiple segments of the scientific community. To be sure, one has merely to pick up an issue at random to see authors from a wide range of scientific departments and institutions, questions pursued using a wide range of methods, and references drawn from a wide range of literatures. In our investigation, we recognize that Nature and Science are surely multidisciplinary, but here we explore the extent to which their content (their articles) are interdisciplinary. What does multidisciplinarity at the level of the journals indicate about the integration and diffusion of knowledge at the level of individual articles? Researchers can appear to be promiscuously interdisciplinary, publishing with multiple collaborators in multiple fields, when at the individual project level they are serially monodisciplinary, each publication hewing closely to a particular literature. Similarly, the goal of many funding programs is the creation of a multidisciplinary portfolio of research projects. When evaluated at the level of the portfolio, they appear successfully interdisciplinary, but when evaluated at the level of individual funded projects they can look decidedly monodisciplinary [10]. Level of analysis is critical even in an exploratory evaluation, an important lesson to keep in mind in this time of data-driven accountability [11][12].
Our overarching research questions concern patterns of research knowledge transfer across fields, motivated by the hypothesis that interdisciplinary research offers compelling promise in advancing science and solving complex societal problems. We translate these interests to examine empirical evidence on citation patterns across scientific fields, comparing Nature and Science articles to those appearing in prominent disciplinary journals in those selected fields. Our focal questions: Are Nature and Science papers more interdisciplinary? And do Nature and Science papers diffuse strikingly more widely?

Data and Methods
Defining the fields, selecting the journals, and acquiring the dataset We chose Nature and Science because they are the two leading multidisciplinary journalsbased on reputation, explicit editorial policy, Web of Science Subject Category (i.e., MULTI-DISCIPLINARY SCIENCE), and prestige-fortified by their very high Journal Impact Factors (JIF). Though JIF tells a limited story [13][14], Nature's and Science's high JIFs do indicate that their articles are read and cited. A lot. Using JIFs from the 2010 Journal Citation Reports, we find Nature at JIF = 36.1 and Science at JIF = 31.4. We utilize Web of Science (WoS) as a leading database indexing fundamental research published in over 12,000 journals. Data challenges led us to limit our exploration to select fields, mainly because collection and cleaning of sets of citing records is extremely expensive, both in terms of time and funding. Articles typically cite 30 or more references, resulting in a huge multiplier. Furthermore, articles in the top multidisciplinary journals-of special interest to us-accrue many citations. For example, one compilation of 545 publications we examined drew 60,425 cites.
In order to address questions of the interdisciplinarity of Nature and Science, we focused on a sample of articles that appeared in a single year, 2009, in each of three fields-a life science (Cell Biology), a physical science (Physical Chemistry), and a behavioral science (Cognitive Science). These choices resulted from discussions with NSF colleagues about candidate fields in terms of coherence and mainstream interest, and empirical forays regarding journal coverage. We observed that any given issue of Nature or Science might have multiple articles on Cell Biology, whereas it might have none on Physical Chemistry or Cognitive Science.
We then sought to identify a comparison set of top disciplinary journals. As a starting place, we used the Web of Science Subject Categories (WCs) to identify the set of journals in each field covered by WoS. We then considered journals with relatively high JIFs for their fields (one indicator of impact), looked to whether they were the flagship journals of the major professional societies in the fields, and again consulted with NSF colleagues on prestige and reputation. We selected the following: We recognize that the wide range in impact of the journals affects the relative diffusion of research knowledge across disciplines. As the above JIFs attest, citation practices vary greatly by field [15]. JIF influence seems to peak after about three years [16]. Other variables influencing citation include how basic or applied the science is (basic research attracts more citations [17]), average number of authors per article (more authors attract more citations [18]), and average number of affiliations per article (more affiliations attract more citations [18]). Fields' publishing norms vary in terms of co-authoring, author ordering, average number of cited references, self-citing, and so on. The message is to be cautious in comparing fields in analyses such as this one [19]. Hence, we key on within-field comparisons of Nature and Science articles to disciplinary articles and refrain from field-to-field comparisons.
Number of publications to assess was another consideration. Obtaining sufficient publications factored into our field, journal, and temporal (one year) selections. We decided that looking at publications appearing in 2009 would support conclusions that are reasonably current, yet allow sufficient time (5 years) for papers to accrue citations; we note that there are many issues in tracking citation over time [20]. We followed up by collecting WoS records that have cited the 2009 publications through 2014. We used the WoS "document type" information to restrict our search to "articles" (setting aside reviews and various editorial content publications). Then we considered how to sample articles, deciding against going after the most cited papers as this might distort our assessment of knowledge diffusion patterns and differences. We moved toward a set of criteria that balanced sizes for samples from Nature and Science and the disciplinary journals, while taking best advantage of the articles available for 2009. We did not include in our sample articles that had fewer than four references to WoS journals as that would have rendered the interdisciplinarity metrics unreliable. Using these criteria, we gathered a sample of 1643 Nature and Science articles published in 2009.
Determining what articles are in which fields. A key challenge was to determine what constitutes an article in a target field. For the three disciplinary journals, we took publication in the chosen journal as the selection criterion-that is, an article appearing in Cognitive Science is considered by definition to be a Cognitive Science article. But how to categorize Nature and Science articles? We considered various ways to classify articles, including assignment based on review of titles, keywords, or abstracts by disciplinary experts. The greater subjectivity of such judgments, combined with the sheer numbers of papers to be examined, rendered the approach less attractive (though not inappropriate). We also considered coding authors' disciplines based on departmental affiliation. That would have proven problematic. First, it is not clear who the lead author of an article is. In some fields, the first author can be assumed to be the lead, in others it is the last. Moreover, Nature and Science do not always show departmental affiliation. And even when they do, the specificity of our fields does not always satisfactorily match the available information-"Physical Chemistry" would not usually be the name of a department or center. Furthermore, affiliation information gleaned from WoS records is inconsistent and deciding on field boundaries is tricky. Further ambiguity arises with multiple authors who reside in different departments or even individual authors with multiple departmental affiliations.
We shifted to examination of a Nature or Science article's cited references as a viable, reproducible, and somewhat more objective indicator of field ties. We follow Schunn [21] whose commonsense reasoning held that if 25% or more of the citations by a paper were to a given field, then that field should be considered to be a major influence. We therefore search the references of the articles appearing in Nature and Science and include all of those for which 25% of the references are in a particular field. We recognize that this approach tiptoes toward tautology, but we reasoned that if, for example, at least 25% of an article's references were not to articles appearing in Cell Biology journals, it would be difficult to consider the article to be a Cell Biology article. As a further check, we sampled entire issues of Nature and Science from our dataset and found that for every article there was at least one field that accounted for at least 25% of its references. This is not to say this need be true for the entire dataset, but our inclusion criterion would not appear to have biased our dataset in a systematic fashion.
To perform the categorization, we apply a thesaurus in VantagePoint software [22] to standardize journal names, then another to associate cited journals to their WCs (thesaurus provided by Thomson Reuters). Operationalizing Nature and Science article selection requires consideration of multiple factors. Reflecting on our selection of "disciplines" to study regarding multidisciplinary knowledge transfer patterns, we are using WCs. The names suggest the tendency of fields of study to intersect and integrate over time. Again, in consultation with disciplinary experts and scope provided by professional societies, we include the following WCs for our fields: Resulting dataset. Of the 1643 articles appearing in Nature and Science in 2009, our search, using the WC-based operationalization described above, yielded 339 Cell Biology articles, 61 Physical Chemistry articles, and 34 Cognitive Science articles. All of the selected Nature and Science articles are in only one of the three field categories; that is, there is no overlap. Preliminary analyses on the metrics described below showed no significant differences between articles in a given field appearing in Nature as opposed to those appearing in Science. Therefore, because the numbers of articles in Physical Chemistry and Cognitive Science appearing in Nature and Science were relatively low, and those in Cell Biology so high, the articles appearing in the two journals were pooled. As the pooled number of Cell Biology articles was so great, we randomly selected 50 articles from Science and 50 from Nature. Finally, we randomly sampled 50 articles appearing in 2009 from each of the three disciplinary journals (Cell, Journal of Physical Chemistry-A, and Cognitive Science) for inclusion in our dataset as the foil to the Nature and Science articles.

Metrics of interdisciplinarity
Integration and Diffusion scores. Measuring interdisciplinarity remains a challenge [23]. Integration and Diffusion scores provide the foundational metrics for the present analyses. They were devised by program evaluators in support of the US National Academies Keck Futures Initiative [4] to help assess the interdisciplinarity of bodies of publication-for example, of a field, a department or center, or even compilations at a national level [6][7][8]. Integration scores are based on the diversity of a paper's cited references [6,24], with diversity operationalized using WCs. As accessed through Thomson Reuters Web of Knowledge site for 2014 publications, WoS distinguishes about 224 WCs in the natural sciences (Science Citation Index) and social sciences (Social Sciences Citation Index).
Integration scores reflect distance among WCs based on the relative degree to which one WC's journals cite those of the other 223 WCs in a given year of WoS [25][26]. For example, WCs such as Ornithology and Forestry, whose journals frequently cite each other, are more similar than are, say, Acoustics and Economics, whose journals do not. Integration scores range from 0 toward 1, increasing as a paper's references span more WCs, are distributed more evenly across those WCs, and are more distant from each other. WCs are assigned at the journal level [8], so they don't reflect article content directly. Integration scores equate to Rao-Stirling diversity [24,26] as a tri-dimensional concept operationalized in terms of: (i) variety (number of WCs cited), (ii) balance (evenness of distribution of citing those WCs), and (iii) disparity (dissimilarity of the cited WCs). For some purposes, separating those three aspects may be fruitful [27][28], but we combine them into a single indicator of degree of interdisciplinarity here (i.e., Integration scores).
Integration scores can be expressed as: where p i is the proportion of references citing the WC i in a given paper. The summation is taken over the cells of the WC x WC matrix. s ij is the cosine measure of similarity between WCs i and j-based on cross-citation among WoS journals in a given year (we use 2010 here). We report findings in terms of means, though we recognize that citation distributions are generally non-normal. For reporting purposes, use of means appears reasonable in that our focus is on scores that are constrained to a range from 0 to 1. That said, the analyses below were conducted using nonparametric statistics. As noted, WCs whose journals frequently cite each will be scored as more similar. If a given article cites a number of references from an array of academic disciplines distantly related it is said to be relatively integrative. Diffusion scores can be thought of as Integration in reverse. The formulation is the same, except using the set of papers referencing the given paper to obtain the citing WCs. While Integration scores are backward looking in that they measure diversity among cited references, Diffusion scores are forward looking in that they measure diversity among papers that refer to a given body of work. Scholarship cited by a broad array of distantly related academic disciplines is said to be relatively diffuse [15].
Given that citations for a given article can accrue ad infinitum into the future, Diffusion scores can change over time. Hence, we want to compare papers of approximately the same age. Carley and Porter [15] provide benchmark Integration and Diffusion scores for WoS publications published in 1995, allowing an extended post-publication period in which to be cited. We note that suitable periods vary by field. For example, one would expect History citations to average much longer lags than, say, Nanotechnology. Here we calculate Diffusion scores for 2009 publications over an approximately 5-year period, through 2014, a reasonable period for citations to accrue [20].
In-field and out-of-field citations. We also use the WCs in a somewhat rougher fashion to determine whether a citing paper is out-of-field or in-field. If the citing article's journal is categorized into any of the WCs for one of our three fields of interest, we considered it to be an in-field citation, even if the journal is also categorized in some other WC. (Relatively few journals appear in more than two WCs, but multiple categories for a given journal pose analytical challenges [29].) For example, if a target Cognitive Science article were cited by a paper appearing in a journal that is included in the WC "PSYCHOLOGY, DEVELOPMENTAL," then the citation would be considered in-field. We tabulate the mean and median numbers of total citations for the articles in each field as well as the mean and median numbers of citations appearing in out-of-field journals.
To be sure, cross-disciplinarity is but one dimension of knowledge creation and diffusion; one could certainly consider others. And measures of cross-disciplinarity could use units other than WCs, such as departmental affiliations or final academic degrees of co-authors [30]. But WCs are highly suitable for study of cross-disciplinarity in several regards. First, their granularity corresponds well to that advanced by the National Academies Committee on Facilitating Interdisciplinary Research [1]. Second, WCs are widely used by the bibliometrics community, allowing for a greater basis of comparison. Third, WCs are readily accessible from WoS abstract records. And finally, though exact assignment of journals to WCs can be problematic [31], miscategorizations should exert relatively modest effects on similarity measures as "correct" categorizations tend to be nearby [32].

Results
The mean Integration scores for the journals in each field can be seen in Fig 1. Differences among the three fields under study are substantial. Carley and Porter [15] reported benchmark Integration and Diffusion scores for a set of disciplines based on all articles in those fields indexed by Web of Science in 1995. Consistent with those findings, we find differences between fields in the current study in the diversity of articles cited, even within Nature and Science. The mean Integration score of 0.22 for the Cell Biology articles in our study is highly disciplinary, approaching the lowest score derived in the benchmark study, 0.21 for "Mathematics." The mean of 0.42 for Nature and Science articles in Physical Chemistry is similar to the relatively interdisciplinary score of 0.40 derived in the benchmark study for "Atomic, Molecular, and Chemical Physics," and the mean of 0.54 for the Cognitive Science articles exceeds the highest Integration score in the benchmark study, the 0.43 derived for the related field of "Neuroscience." Critically, the articles in Nature and Science within each field are not significantly more interdisciplinary than are those sampled in the disciplinary journals, as indicated by preplanned Mann-Whitney U tests comparing the Integration Scores of the references in our sample. As can be seen more qualitatively in Table 1, the top five journals cited by the Nature and  Even if papers appearing in Nature and Science are not generally more interdisciplinary than are those appearing in top disciplinary journals, as measured by their cited references (putative influences on the research conducted), they might yet be shown to be more multidisciplinary in the diversity of the papers citing them (putative impact). Given the great number of eyeballs scanning the pages of Nature and Science and the disciplinary diversity of their owners, the question at hand is whether gaze translates into influence. In order to answer this question, we next examine the articles that cite the sampled articles. Fig 2 shows the mean Diffusion scores for the original target journals, reflecting the diversity of journals in which those citing articles appeared. Preplanned Mann-Whitney U tests reveal no significant mean differences between Cell Biology articles appearing in Cell and those appearing in Nature and Science, nor for the Physical Chemistry articles appearing in the Journal of Physical Chemistry-A versus those appearing in Nature and Science. As was the case when looking at the journals cited by our target articles, the top five journals citing the target Nature and Science articles in these fields are not qualitatively different from those of the disciplinary journals (see Table 1). The story is slightly different for Cognitive Science. The mean Diffusion score of the target articles appearing in Nature and Science is higher than that for Cognitive Science, reaching significance in a one-tailed Mann-Whitney U test (p = .006). Significant, but the relatively small differences between means is hardly what one would have expected given the reputations of Nature and Science for multidisciplinary reach.
Before we dismiss the idea of there being a substantial cross-disciplinary knowledge transfer advantage for articles appearing in Nature and Science, let us consider raw citation counts. Do articles appearing in Nature and Science garner more citations out-of-field? As can be seen in Table 2, the mean total citations of Cell Biology articles appearing in Nature and Science is about the same as the total citations of articles appearing in Cell. This makes sense, given their similar JIFs. More to the point, the mean number of citations in out-of-field journals is also about the same.
By contrast, articles appearing in the disciplinary Physical Chemistry and Cognitive Science journals receive dramatically fewer mean citations per article than do articles in those fields appearing in Nature and Science (Table 2). Similarly, the mean numbers of out-of-field citations are also dramatically lower. Unlike Cell, in the field of Cell Biology, no journal in Physical Chemistry or Cognitive Science has anything like the JIF of Nature and Science. As it turns out, sometimes JIF matters.

Discussion and Conclusion
Our exploratory paper has produced findings that are at once both consistent and at odds with the reputation of Nature and Science. Integration scores-a measure of the diversity of the cited references-for the Cell Biology, Physical Chemistry, and Cognitive Science articles appearing in Nature and Science were not significantly greater than were those appearing in leading disciplinary journals in each field. Similarly, the Diffusion scores-a measure of the diversity of the citing references-for the Cell Biology and Physical Chemistry articles were not significantly different from those for articles appearing in leading disciplinary journals, and the Diffusion scores for the Cognitive Science articles, while statistically different, do not suggest a substantial practical difference.
The lack of substantial differences in the rate of multidisciplinary diffusion notwithstanding, the raw impact of Nature and Science remains. Articles appearing in Cell (whose JIF is comparable to that of Nature and Science) are cited as often as are those appearing in Nature and Science, and they are cited as often out of field. But articles appearing in Nature and Science are cited far more often than are articles in such top disciplinary journals as the Journal of Physical Chemistry-A and Cognitive Science, and so the articles in them are cited more times out of field.
It is worth considering several methodological and analytic decisions we made in this study. We contrasted Cell Biology, a life science with relatively high numbers of articles appearing in Nature and Science, with Physical Chemistry and Cognitive Science, represented by relatively low numbers of articles. Indeed, articles in the latter two fields appear so infrequently that one might question how representative of the field they are. Certainly Nature and Science could not be said to be journals of record in those fields. It is an empirical question whether these results might hold for other disciplinary journals within those fields. For example, we chose what we thought would be more focused contrast cases. We might instead have chosen to look at journals that reach across subfields within each discipline, such as the Journal of the American Chemistry Society, Psychological Science, or PLOS Biology. We might also have chosen other Another important methodological decision in work of this type is choice of measures. As we noted above, there is no consensus on indicators of interdisciplinarity. We chose to use Integration and Diffusion scores, based on the Rao-Stirling diversity index, because of the appeal of its underlying conceptual basis (with its emphasis on disparity, balance, and variety), its standing in the scientometric community, and the existence of published benchmarks against which we could compare our results. But a case could certainly be made for using other measures. Zhang, Rousseau, and Glanzel [33], for example, argue for using their D2S measure that emphasizes variety over disparity and balance as an indicator of interdisciplinary diversity, and Bergmann and his colleagues [34] argue for quantifying interdisciplinarity by means of Jenson-Shannon divergence. Nonetheless, as suggested by our admittedly rough calculation of citation out of field and our qualitative comparison of the top cited and citing journals, the differences in integration and diffusion between the articles in Nature and Science and those in the disciplinary journals were not great. It is beyond the scope of this paper to address concerns about which measure of interdiscipinarity is best at addressing more subtle questions, such as what are optimal levels of interdisciplinarity in citations or in team composition. Suffice it to say that we were not looking for subtle effects.
We further acknowledge that bibliometric analyses themselves provide an imperfect perspective, especially for evaluative purposes, as Lane and others note, [13-14, 17, 35]. We might have chosen "Altmetrics" in order to tap into wider communities' referencing (e.g., Google Scholar) or topical attention (e.g., Twitter activity) in order to add dimensions beyond WoS citation analyses (c.f., [36]). Such data would offer interesting resources to expand our inquiry into the ramifications of publication in Nature and Science through the attention drawn by various publics quite distinct from citation by formal research communities [37][38][39]. And, of course, there are other ways to operationalize interdisciplinarity, including text analyses of content similarity and research network span [40]. We also recognize other forms of diversity pertinent to gauging the merits of publication in multidisciplinary, as opposed to disciplinary journals. For example, do the former generate more awareness (e.g., reading) and utilization (e.g., as indicated by citation) across geographical entities (a candidate indicator of breakthrough potential along with interdisciplinarity [41])?
Clearly, our paper is meant to be less of a definitive work and more of an exploration of how one might go about addressing such issues of interdisciplinarity and multidisciplinarity. And to that end our findings certainly demonstrate that level of analysis matters. As always in science, one must take care to square the kinds of claims one would like to make with the kinds of evidence produced. In this case, the extreme diversity of the articles appearing in any given issue of Nature or Science was not matched by a similar diversity at the level of the individual article.
How interdisciplinary should the articles in Nature and Science be? We could not say. Upon reflection, our finding of the lack of substantially increased interdisciplinarity in the articles is actually quite reasonable, given the missions and editorial policies of the journals. Articles are chosen for publication in Nature and Science because, foremost, they are the products of excellent science. The history of science suggests that paradigm-busting work, as it were, is most productive when the science demands it [42]. Interdisciplinarity is not a "scientific good" in and of itself, nor is gratuitous multidisciplinary citation a necessary indicator of good science.
Does publication in Nature and Science signify the successful multidisciplinary diffusion of findings? Well, yes, to some degree. The articles appearing in them are not inherently of greater interest to "outsiders" as indicated by measures of the diversity of citing articles, but, in keeping with differences in JIF, they can attract a markedly greater number of out-of-field citations, likely deriving in part from their vetted quality and high profile. But even this claim must be qualified, for it varies by field. Thus, the moral of our story, at least for Physical Chemists and Cognitive Scientists seeking a broader impact: Do excellent work and publish it in Nature or Science.