Open and Reproducible Science Summaries


The symbolstands for non-peer-reviewed work.

The symbolstands for summaries on the topic of Diversity, Equity, and Inclusion.

Trust Your Science? Open Your Data and Code (Stodden, 2011)◈

Main Takeaways:

  • Computational results suffer from problems of errors in final published conclusions.
  • In order to allow independent replication and reproducible work, release the scripts and data files, and if the researcher uses MATLAB for graphs etc, please provide the graphical user interface.
  • The standards for code quality are more precise definitions of verification, validation, and error quantification in scientific computing.
  • Research workflow involves changes made to data, including analysis, that affects data interpretation.
  • To conclude, open data is a prerequisite for verifiable research.

Quote

“Science has never been about open data per se, but openness is something hard fought and won in the context of reproducibility” (p. 22).

Abstract

This is a view on the reproducibility of computational sciences by Victoria Stodden. It contains information on the Reproducibility, Replicability, and Repeatability of code created by the other sciences. Stodden also talks about the rising prominence of computational sciences as we are in the digital age and what that means for the future of science and collecting data.

APA Style Reference

Stodden, V. C. (2011). Trust your science? Open your data and code. https://doi.org/10.7916/D8CJ8Q0P

You may also be interested in




Registered reports: a method to increase the credibility of published results (Nosek & Lakens, 2014)

Main Takeaways:

  • This editorial discusses the value of pre-registration and replication, as not all articles are published.
  • Direct replication adds data that increases the precision of effect size estimates for meta-analytic research. No direct replication, means it is difficult to identify false positives.
  • Conceptual replications are more popular than direct replications, as the former conceptualises a phenomenon from its original operationalisation, thus contributing to our theoretical understanding of the effect.
  • Direct replication encourages generalisability of effects, providing evidence that the effect was not due to sampling, procedural or contextual error.
  • If direct replication produces negative results, this improves the identification of boundary conditions for real effects.
  • The benefit of a registered report is that the feedback provided from peer review allows researchers to improve their experimental design.
  • Following peer review, the manuscript can be resubmitted for review and acceptance or rejection based on feedback.
  • Successful proposals tend to be high-powered, high quality, and faithful replication designs.
  • One benefit of a registered report is that this can be all done before the research is conducted.
  • Peer reviewers will focus on the methodological quality of the research, allowing conflict of interests to be  reduced and peer reviewers can provide a fair assessment of the manuscript.
  • The original studies can provide an exaggerated effect size. When this study is replicated, the effect size usually decreases as a result of a larger sample size.
  • Registered reports enable exploratory and confirmatory analyses, but a distinction is required. However, more trust can be placed in confirmatory analyses, as it follows a plan and ensures the interpretability of reported p value.

Quote

“No single replication provides the definitive word for or against the reality of an effect, just as no original study provides definitive evidence for it. Original and replication research each provides a piece of accumulating evidence for understanding an effect and the conditions necessary to obtain it. Following this special issue, Social Psychology will publish some commentaries and responses by original and replication authors of their reflections on the inferences from the accumulated data, and questions that could be addressed in follow-up research.” (p. 139)

Abstract

Professor Daniel Laken and Professor Brian Nosek provide an editorial on how pre-registration and registered reports can be used for the journal of Social Psychology in order to increase the credibility of individual results and findings.

APA Style Reference

Nosek, B. A., & Lakens, D. (2014). Registered reports : a method to increase the credibility of published results. Social Psychology, 45(3), 137-141. https://doi.org/10.1027/1864-9335/a000192

You may also be interested in




Education and Socio-economic status (APA, 2017b) ◈⌺

Main Takeaways:

  • Children from low socio-economic status take longer to develop academic skills than children from higher socio-economic status groups (e.g. poor cognitive development), leading to poorer academic achievement.
  • Children from low socio-economic status are less likely to attain experiences for the development of reading acquisition and reading competence.
  • As a result of fewer learning materials and experiences at home, children from low socio-economic status enter high school with literacy skills 5 years behind their affluent age-matched peers.
  • Children from lower socio-economic status households are twice as likely as those from high SES households to show learning related behaviour problems.
  • High school dropout rate was evident in low-income families compared to high-income families.
  • Placing low-socio-economic status students in higher-quality classrooms will help them earn more disposable income, more likely to attend college, live in affluent neighbourhoods and save more income for retirement.
  • Students from low socio-economic status are less likely to have access to resources about colleges (e.g. career offices and familial experience with higher/further education) and are more at-risk of being in debt to student loans than their affluent peers.
  • Low income students are less likely to succeed in STEM disciplines, 8 times less likely to obtain a bachelor’s degree by the age of 24 and have less career-related self-efficacy when it came to vocational aspirations than high income students.
  • These problems are worsened for people of colour, women, people who are disabled and LGBTIQ-identified individuals.

Abstract

This fact sheet explains the impact socioeconomic status on educational outcomes.

APA Style Reference

APA (2017, July). Education and Socioeconomic Status [Blog post]. Retrieved from https://www.apa.org/pi/ses/resources/publications/education

You may also be interested in




Ethnic and Racial minorities and socio-economic status (APA, 2017) ◈ ⌺

Main Takeaways:

  • The relationship between SES, race and ethnicity is intimately intertwined.  Communities are segregated by socio-economic status, race and ethnicity. Low economic development, poor health conditions and low levels of educational attainment are often comorbidities shared in these communities.
  • Discrimination hinders social mobility of ethnic and racial minorities. In the US, 39% of African American children and adolescents, and 33% of Latino children and adolescents are living in poverty, which is more than double than the 14% poverty rate for non-Latino, White and Asian children and adolescents.
  • Minority racial groups are more likely to experience multidimensional poverty than their White counterparts. American Indian/Alaska Native, Hispanic, Pacific Islander, and Native Hawaiian families are more likely than Caucasian and Asian families to live in poverty.
  • “African Americans (53%) and Latinos (43%) are more likely to receive high-cost mortgages than Caucasians (18%; Logan, 2008).” (p.9).
  • African American unemployment rates are double of Caucasian Americans. African American men working full time earn only 72% of Caucasian men's average earnings, and 85% of earnings of Caucasian women.
  • African Americans and Latinos are more likely to attend high-poverty schools than Asian Americans and Caucasians. From 2000 to 2013, dropout rates between racial groups narrowed significantly.  High school dropouts were highest for Latinos, followed by African Americans and Whites.
  • High achieving African American students may be exposed to less rigorous curriculums, attend schools with fewer resources, and have teachers who expect less of them academically than similarly situated Caucasian students.
  • 12% of African American college graduates were unemployed, which is more than double the rate of unemployment among all college graduates in the same age range.
  • Racial and ethnic minorities have worse health than that of White Americans.
  • Health disparities stem from economic determinants, education, geography, neighbourhood, environment, lower-quality care, inadequate access to care, inability to navigate the system, provider ignorance or bias, and stress.
  • “At each level of income or education, African American have worse outcomes than Whites. This could be due to adverse health effects of more concentrated disadvantage or a range of experiences related to racial bias (Braveman, Cubbin, Egerter, Williams, & Pamuk, 2010).” (p.10).
  • In pre-retirement years, Hispanics and American Indians are much less likely than Whites, African Americans, and Asians to have any health insurance. Negative net worth, zero net worth, and not owning a home in young adulthood are linked to depressive symptoms independent of other socio-economic indicators.
  • Hispanics and African Americans report a lower risk of psychiatric disorder relative to White counterparts, but those who become ill tend to have more persistent disorders.
  • African Americans, Hispanics, Asians, American Indians, and Native Hawaiians have higher rates of post-traumatic stress disorders than Whites, which is not explained by Socio-economic status and a history of psychiatric  disorders. However discrimination is factor that contributes to increasing mental health disorders among the Asian and African American communities (i.e., compared to the White community, African American communities are more frequently diagnosed with schizophrenia, a low prevalence but serious condition).

Abstract

Learn how socioeconomic status affects the lives of many racial and ethnic minorities.

APA Style Reference

APA (2017, July). Ethnic and Racial Minorities & Socioeconomic Status [Blog post]. Retrieved from https://www.apa.org/pi/ses/resources/publications/minorities

You may also be interested in




Faculty promotion must assess reproducibility (Flier, 2017) ⌺

Main Takeaways:

  • Inadequate training, increased competition, problems in peer review and publishing, and occasionally scientific misconduct are some of the variables behind irreproducible research in the biomedical field.
  • Diverse causes make finding solutions for the problem of irreproducibility  difficult, especially, as they must be implemented by independent constituencies including funders and publishers.
  • Academic institutions can and must do better to make science more reliable. One of the most effective (but least discussed) measures is to change how we appoint and promote our faculty members.
  • Promotion criteria has changed over time. Committees now consider how well a candidate participates in team science, but we still depend on imperfect metrics for judging research publications and our ability to assess reliability and accuracy is underdeveloped.
  • Reproducibility and robustness are under-emphasised when job applicants are evaluated and when faculty members are promoted.
  • Currently, reviewers of committees are asked to assess how a field would be different without a candidate’s contributions, and to survey a candidate’s accomplishment, scholarship, and recognition.
  • The promotion process should also encourage evaluators to say whether they feel candidates’ work is problematic or over-stated and whether it has been reproduced and broadly accepted. If not, they should say whether they believe widespread reproducibility is likely or whether work will advance the field.
  • Applicants should also be asked to critically evaluate their research, including unanswered questions, controversies and uncertainties. This signals the importance of assessment and creates a mechanism to judge a candidate’s capacity for critical self-reflection.
  • Evaluators should be asked to consider how technical and statistical issues were handled by candidates. Research and discovery is not simple and unidirectional, and evaluators should be sceptical of candidates who oversimplify.
  • Institutions need to incentivise data sharing and transparency. Efforts are more urgent as increasingly interdisciplinary projects extend beyond individual investigators’ expertise.
  • Success will need creativity, pragmatism and diplomacy, because investigators bristle at any perceived imposition on their academic freedom.

Quote

“Over time, efforts to increase the ratio of self-reflection to self-promotion may be the best way to improve science. It will be a slog, but if we don’t take this on, formally and explicitly, nothing will change.” (p.133)

Abstract

Research institutions should explicitly seek job candidates who can be frankly self-critical of their work, says Jeffrey Flier.

APA Style Reference

Flier J. (2017) Faculty promotion must assess reproducibility. Nature, 549(7671),133. https://doi.org/10.1038/549133a

You may also be interested in




Women and Socio-economic status (APA, 2010)◈ ⌺

Main Takeaways:

  • Socioeconomic status encompasses quality of life attributes and opportunities and privileges afforded to people in society.
  • Socio-economic status is a consistent and reliable predictor of outcomes across lifespan.
  • Low socio-economic status and its correlates (e.g., lower educational achievement, poverty and poor health) affect society.
  • Inequities in health distribution, resource distribution and quality of life are increasing in the US and globally.
  • Socio-economic status is a key factor in determining the quality of life for women and, by extension, strongly affects the lives of children and families.
  • Inequities in wealth and quality of life for women are long-standing and exist both locally and globally.
  • Women are more likely to live in poverty than men.
  • Men are paid more than women despite similar levels of education and fields of occupation.
  • Reduced income for women coupled with longer life expectancy and increased responsibility to raise children, increase probabilities that women face economic disadvantages.
  • Pay gap has narrowed over time but recently the progress has plateaued.
  • Women with a high school diploma are paid 80% of what men with the same qualifications are paid.
  • Single mother families are more than 5 times as likely to live in poverty as married-couples families.
  • Pregnancy affects work and educational opportunities for women and costs associated with pregnancy are higher for women than men.
  • 46% of women believed they have experienced gender discrimination.
  • Pregnant women with low socio-economic status report more depressive symptoms, suggesting the third trimester may be more stressful for low-income women.
  • At 2 and 3 months postpartum, women with low income have been found to experience more depressive symptoms than women with high-income.
  • Women with insecure and low-status jobs with little to no decision-making authority experience higher-levels of negative life events, insecure housing tenure, more chronic stressors, and reduced social support.
  • Depression and anxiety have increased significantly  for poor women in developing countries undergoing restructuring.
  • Women with low income develop alcoholism and drug addiction influenced by social stressors linked to poverty.
  • Improved balance in gender roles, obligations, pay equity, poverty reduction and renewed attention to maintenance of social capital redress the gender disparities in mental health.
  • SES also affects physical health, with women living with breast cancer being11% more likely to die if they live in lower SES communities.
  • Low-income women who have no insurance have lowest rates of mammography screening among women aged 40-64, increasing risk of death from breast cancer.
  • Obesity and staying obese from adolescence to young adulthood is linked to poverty among women.
  • Relative to HIV-positive men, women with HIV ahave disproportionately low-income in the US.

Abstract

Learn how socioeconomic status affects the lives of women.

APA Style Reference

APA. (2017, July). Women & Socioeconomic Status [Blog post]. Retrieved from https://www.apa.org/pi/ses/resources/publications/women 

You may also be interested in




The Gender Gap: Who Is (and Is Not) Included on Graduate-Level Syllabi in Social/Personality Psychology (Skitka et al., 2020) ⌺

Main Takeaways:

  • This article investigates whether there is a gender gap in Social/Personality Psychology syllabi.
  • One factor contributing to gender gaps is whose work we choose to teach in graduate seminars.
  • It is hypothesised that one link in the broad chain of factors contributing to the eminent gender gap is taht female authors are likely to be under-represented on graduate course syllabi compared to their male peers (gender gap hypothesis).
  • Reasons why female authors might be under-represented on course syllabi could be varied.
  • Instructors may internalise cultural prejudices and biases favouring men over women. This might result in a greater preference for male over female-authored papers (i.e., bias hypothesis).
  • Another possibility is that instructors might prefer older over contemporary papers (i.e., classic hypothesis).
  • Yet another possibility is that there are more male-authored papers available to include in syllabi than female-authored papers (i.e., availability hypothesis).
  • The present study investigates whether there is a gender gap in representation on graduate level syllabi and whether it is explained by preference for classic over contemporary papers or relative availability of male- versus female-authored manuscripts.
  • Method: The authors identified every social and/or personality PhD program in the US using the Social Psychology Network’s PhD ranking list and Graduate Programs GeoSearch.
  • 120 programs were identified and a list of social/personality faculty names and email addresses for each program were put together by going to psychology department websites.
  • Main interest was in courses for first-year graduate students.
  • Inclusion criteria for syllabi were: (1) course name includes words: social or personality, (2) course was at the graduate level.
  • Papers cited in the syllabi were coded for the following characteristics: gender of all authors, each author’s h-index, total number of authors, journal where the article was published, number of citations the article received since publication and topic in social/personality psychology.
  • To understand whether the gender representation on graduate syllabi is (or is not) consistent with the number of high-quality papers from which instructors can select, the present study obtained all names of authors, authorship order and year of publication for all papers published in the Journal of Social and Personality Psychology from 1965 to 2017 and published in the Personality and Social Psychology Bulletin from 1974 until April 2018. These journals accounted for 33% of reading on sample course syllabi and formed benchmarks.
  • Results: Less than 30% of papers referenced on syllabi were written by female first authors.
  • The gender gap on syllabi, differed as a function of instructor gender and decade papers were published: female instructors assigned more recently published papers (post-1990) and female first-authored papers at levels significantly higher than their male counterparts.
  • Difference in inclusion rates of female first-authored paper could not be explained by preference for classic over contemporary papers in syllabi or relative availability of female first-authored papers in the published  literature.
  • The gender gap differed depending on the content of the course. Male and female authors were approximately equally represented on graduate-level syllabi of topics as prejudice, close relationships, culture and health. The gender gap was much larger in syllabi of topics as best practices, replicability, attitude change and persuasion.
  • Male and female-authored papers included on syllabi had similar citation rates, although they had different h-index scores.
  • Increasing representation of female scholars’ work on graduate course syllabi would have beneficial consequences, moving toward greater gender inclusiveness in social/personality psychology.

Abstract

We contacted a random sample of social/personality psychologists in the United States and asked for copies of their graduate syllabi. We coded more than 3,400 papers referenced on these syllabi for gender of authors as well as other characteristics. Less than 30% of the papers referenced on these syllabi were written by female first authors, with no evidence of a trend toward greater inclusion of papers published by female first authors since the 1980s. The difference in inclusion rates of female first-authored papers could not be explained by a preference for including classic over contemporary papers in syllabi (there was evidence of a recency bias instead) or the relative availability of female first-authored papers in the published literature. Implications are discussed.

APA Style Reference

Skitka, L. J., Melton, Z. J., Mueller, A. B., & Wei, K. Y. (2020). The Gender Gap: Who Is (and Is Not) Included on Graduate-Level Syllabi in Social/Personality Psychology. Personality and Social Psychology Bulletin, 0146167220947326. https://doi.org/10.1177/0146167220947326

You may also be interested in




Leveraging a collaborative consortium model of mentee/mentor training to foster career progression of under-represented post-doctoral researchers and promote institutional diversity and inclusion (Risner et al., 2020) ⌺

Main Takeaways:

  • The goal of the study is to empower post-doctoral students and make them active participants in the mentoring relationships by emphasising the mentees’ contributions in shaping more productive interactions to be built upon to develop their own skills as a future mentor.
  • The study used several metrics by which they assessed the success of this collaborative, multi-institutional approach, using National Research Mentoring Network (NRMN), the Committee on Institutional Cooperation Academic Network (CAN) approach to provide mentor facilitator training for faculty and senior administrators and mentoring-Up training for post-doctoral students.
  • Background: “Establishing a functioning consortium needs buy-in and high-level cooperation from all partners. Prior to initiating programming, all potential institutional representatives set initial goals to address campus needs for mentor-up skill development for post-docs and mentor facilitator training for staff and faculty, establish sustainable communities of practice for mentor training and develop mechanisms for central coordination, outreach to campus constituents, templates for recruitment of participants and strategies to sustain collaboration and develop mechanisms for central coordination, outreach to campus constituents, templates for recruitment of participants, and strategies to sustain collaboration.” (p.4)
  • Method:  “The seven Core Principles [of “Mentoring-UP”] are: 1. Two-way communication, 2. Aligning expectations, 3. Assessing understanding, 4. Fostering independence, 5. Ethics, 6. Addressing equity and inclusion, 7. Promoting professional development. This curriculum provided postdocs opportunities for: i.) self-evaluation and reflection to become aware of their personal biases, attitudes, and behaviors; ii.) exploring strengths, weaknesses, and challenges in their interpersonal and professional relationships; iii.) understanding and learning how to use the mentor principles; and iv.) focusing on cognitive processes that may lead to behavioral changes and strategies to facilitate those changes in a process-based approach over 1.5–2 day workshops.” (p.5)
  • Method: “The 1.5–2 day workshops included case studies and activities that: i.) engage mentors in peer discussion of a mentor framework; ii.) explore strategies to improve mentoring relationships; iii.) address mentoring problems; iv.) reflect on mentoring philosophies; v.) and create mentoring action plans to model the interactive, collaborative, and problem-solving ways to develop and implement this set of trainings in the future. The training goals provided tools and mechanisms to implement mentor training venues at the participating institutions, thereby establishing sustainable Mentor-training programs for undergrads, graduate students, postdocs and faculty” (p.5).
  • “A specific NRMN-CAN survey was developed for all four postdoc cohorts to ascertain whether mentor training: i.) influenced career progression; ii.) impacted the postdocs’ relationship with their PIs; and iii.) components of the mentor training that were implemented by the postdoc mentees... A dedicated NRMN-CAN survey for faculty and senior administrators was also developed to ascertain whether participation in Mentor Facilitator training led to: i.) implementation of training workshops on their campuses; ii.) the level and number of participants; iii.) and whether facilitated sessions were carried out in partnership with others.” (p.5)
  • Results: Post-doctoral students reported improvements in their mentoring proficiency and improved relationships with the Principal Investigators. 29% of post-doc respondents transitioned to faculty positions, and 85% of these respondents were under-represented and 75% were female. 59 out of 120 faculty and administrators provided mentor training to over 3000 undergraduate, graduate and postdoctoral students and faculty on their campus for the duration of this  project.
  • The findings showed that the majority of post-doctoral students indicate that mentor training positively influenced their relationship with their Mentors in several domains (e.g. confidence building). In addition, this curriculum has guided most post-doctoral students to better understand their mentoring needs, develop strategies to manage their mentoring relationships and empower them to make critical career decisions to pursue an academic career.  In addition, early-career scientists stated they had more confidence to pursue an academic career with increased self-efficiency and advocacy.
  • Impressively, 29% of the responding postdocs, predominantly females (75%) and underrepresented postdocs (85%) have successfully migrated to faculty. Some postdocs also indicated that their mentor training and experiences were valuable skills when applying for academic positions and definitely aided in adapting to responsibilities as a faculty mentor.

Abstract

Changing institutional culture to be more diverse and inclusive within the biomedical academic community is difficult for many reasons. Herein we present evidence that a collaborative model involving multiple institutions of higher education can initiate and execute individual institutional change directed at enhancing diversity and inclusion at the postdoctoral researcher (postdoc) and junior faculty level by implementing evidence-based mentoring practices. A higher education consortium, the Big Ten Academic Alliance, invited individual member institutions to send participants to one of two types of annual mentor training: 1) “Mentoring-Up” training for postdocs, a majority of whom were from underrepresented groups; 2) Mentor Facilitator training—a train-the-trainer model—for faculty and senior leadership. From 2016 to 2019, 102 postdocs and 160 senior faculty and administrative leaders participated. Postdocs reported improvements in their mentoring proficiency (87%) and improved relationships with their PIs (71%). 29% of postdoc respondents transitioned to faculty positions, and 85% of these were underrepresented and 75% were female. 59 out of the 120 faculty and administrators (49%) trained in the first three years provided mentor training on their campuses to over 3000 undergraduate and graduate students, postdocs and faculty within the project period. We conclude that early stage biomedical professionals as well as individual institutions of higher education benefited significantly from this collaborative mentee/mentor training model

APA Style Reference

Risner, L. E., Morin, X. K., Erenrich, E. S., Clifford, P. S., Franke, J., Hurley, I., & Schwartz, N. B. (2020). Leveraging a collaborative consortium model of mentee/mentor training to foster career progression of underrepresented postdoctoral researchers and promote institutional diversity and inclusion. PloS one, 15(9), e0238518. https://doi.org/10.1371/journal.pone.0238518

You may also be interested in




An index to quantify an individual’s scientific research output (Hirsch, 2005)

Main Takeaways:

  • The author introduces the h index as a tool to quantify (in an unbiased way) the scientific output of researchers.
  • Scientists who earn a Nobel prize have unquestionably relevant and impactful research.  How do we quantify the impact and relevance of the work produced by other researchers?
  • Current evaluation criteria are based on number of publications, number of citations for each paper, journal where papers were published and impact parameters. All these parameters are likely to be evaluated differently by different people.
  • H is a preferable index to evaluate scientific output to a researcher.
  • Total number of papers: measures productivity, yet does not measure importance of impact of papers.
  • Citations measure total impact, yet hard to find and may be accentuated by “big hits” that are not representative of individuals if they are co-authors. Another disadvantage is that this measure gives a higher weight to review articles than original research articles.
  • Citations per paper allow comparison of scientists of different ages, yet hard to find, rewards low productivity and penalises high productivity.
  • Number of significant papers (defined as number of papers with citations higher than a certain number - let’s say “y” ). While this measure eliminates the disadvantages of the other measures (mentioned above), y is arbitrarily defined and will favour or disfavour individuals randomly. It needs to be adjusted for levels of seniority.
  • Number of citations to each of the q most-cited papers (let’s say q=5). While it overcomes many of the disadvantages mentioned above, this measure does not yield a single number and is difficult to obtain and compare.
  • The h index overcomes all the disadvantages of the other measures (mentioned above).
  • The higher the h, the more accomplished the scientist is. H should increase with time.
  • H will smoothly level off as the number of papers increases instead of a discontinuous change in slope.
  • In reality, not all papers contribute to the h index. This is especially the case of papers with low citations when the h index of the researcher is already an appreciable number.
  • H cannot decrease with time.
  • Contrary to other parameters, the h parameter is useful for cumulative achievement continuing over time even after the end of the scientist’s publication.
  • The author suggests that h ≈ 12 might be a typical value for advancement to tenure (associate professor) and that h ≈ 18 might be a typical value for advancement to full professor.
  • However, a single number can never give more than an estimation to an individual’s multi-faceted profile and many other factors need to be combined to evaluate an individual.
  • Differences in typical h values in different fields are expected (determined by average number of papers produced by each scientist in an specific field and size of field). Moreover, scientists working in non-mainstream areas will not achieve the same high h values as those working in highly topical areas.
  • High h is a reliable indicator of high accomplishment; the opposite is not always not true.
  • Although self-citations can obviously increase a scientist's h, their effect on h is much smaller than on the total citation count.
  • Nobel prize winners have an h index of 30, meaning their success did not occur in one stroke of luck but in a body of scientific work.
  • H index could be important for rankings of groups or departments in the chosen area and administrators could be interested in this.

Abstract

I propose the index h, defined as the number of papers with citation number >h, as a useful index to characterize the scientific output of a researcher.

APA Style Reference

Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. Proceedings of the National academy of Sciences, 102(46), 16569-16572. https://doi.org/10.1073/pnas.0507655102

You may also be interested in




High Impact =High Statistical Standards? Not Necessarily So (Tressoldi et al., 2013)

Main Takeaways:

  • The present study investigates whether there are differences in statistical standards of papers published in journals with high and low impact.
  • Journals with the highest impact factor are often taken to be a measure of high scientific value and rigorous methodological quality.
  • The present study investigated how often null hypothesis significance testing and alternative methods are used in leading scientific journals compared to journals with lower impact factors.
  • How many studies published in journals with the highest impact factor adopt the recommendations of basing their conclusions on their observed effect sizes and confidence intervals on those effect sizes? Are there differences with journals with lower impact factors in which editorial policy requires the adoption of these recommendations?
  • Method:  6 Journals with high impact and 6 journals with low impact were chosen. “Our aim was to compare across journals, using all relevant articles, noting that many variables could contribute to any differences we found.” (p.3).
  • Results: In 89% of Nature articles and 42% of Science articles, p values was more commonly used without any mention of confidence intervals, effect sizes, prospective power and model estimation, while other journals, both high- and low-impact factor, report confidence intervals and/or effect size measures.
  • The best reporting practice was present in 80% of articles published in New England Journal of Medicine and Lancet, while this dropped to less than 30% for articles published in Science and less than 11% in the articles published in Nature journals.
  • Reporting confidence intervals and effect sizes does not guarantee researchers use them in the interpretation of their findings or refer to them in text.
  • The lack of interpretation of confidence intervals and effect sizes means that just observing high percentage of confidence intervals and effect sizes reporting may overestimate the impact of the statistical reform.

Quote

“It is not sufficient merely to report ESs and CIs—they need to be used as the basis of discussion and interpretation.” (p.6).

Abstract

What are the statistical practices of articles published in journals with a high impact factor? Are there differences compared with articles published in journals with a somewhat lower impact factor that have adopted editorial policies to reduce the impact of limitations of Null Hypothesis Significance Testing? To investigate these questions, the current study analyzed all articles related to psychological, neuropsychological and medical issues, published in 2011 in four journals with high impact factors: Science, Nature, The New England Journal of Medicine and The Lancet, and three journals with relatively lower impact factors: Neuropsychology, Journal of Experimental Psychology-Applied and the American Journal of Public Health. Results show that Null Hypothesis Significance Testing without any use of confidence intervals, effect size, prospective power and model estimation, is the prevalent statistical practice used in articles published in Nature, 89%, followed by articles published in Science, 42%. By contrast, in all other journals, both with high and lower impact factors, most articles report confidence intervals and/or effect size measures. We interpreted these differences as consequences of the editorial policies adopted by the journal editors, which are probably the most effective means to improve the statistical practices in journals with high or low impact factors.

APA Style Reference

Tressoldi, P. E., Giofré, D., Sella, F., & Cumming, G. (2013). High impact= high statistical standards? Not necessarily so. PloS one, 8(2), e56180. https://doi.org/10.1371/journal.pone.0056180

You may also be interested in




Disability and Socio-economic status (APA, 2010) ◈ ⌺

Main Takeaways:

  • The Disabilities Act assures equal opportunities in education and employment for people with disabilities and prohibits discrimination based on disability.
  • Despite the Disabilities Act, people with disabilities remain over-represented among America’s poor and under-educated.
  • The federal government has two major programs to assist individuals with disabilities: the Social Security Disability Insurance and the Supplemental Security Income.
  • The Social Security Disability Insurance is a program  for workers who have become disabled and unable to work after paying Social Security taxes for at least 40 quarters. In this program, a higher income yields higher SSDI earnings.
  • The Supplemental Security Income is a welfare program for individuals with low income, fewer overall resources and no or an abbreviated work history.
  • Current federal benefit for a single person using Supplemental Security Income is $735 a month.
  • Despite these programs, people with disabilities are more likely to be unemployed and live in poverty.
  • For individuals who are blind and visually impaired, unemployment rates exceed 70 percent while for people with intellectual and developmental disabilities, the unemployment rate exceeds 80 percent. Also, one in ten veterans with disabilities are unemployed.
  • The American Association of People with Disabilities estimates that two thirds of people with disabilities are of working age and want to work.
  • There are disparities in median incomes for people with and without disabilities, such that individuals with disabilities often earn lower incomes.
  • A study surveyed human resources and project managers about perceptions of hiring persons with disabilities. Results show professionals held negative perceptions related to productivity, social maturity, interpersonal skills and psychological adjustment of persons with disabilities.
  • Disparities in education have been ongoing for generations. 20.9% of individuals 65 years and older without a disability failed to complete high school, relative to 25.1% and 38.6% of elder individuals with a non-severe or severe disability.
  • Great disparities exist when comparing attainment of higher degrees. 15.1% of the population aged 25 and over with disability obtain a bachelor’s degree, whereas 33% of individuals in the same age category with no disability attain the same educational status.
  • Individuals with a disability experience increased barriers to obtaining health care as a result of accessibility concerns, such as transportation, problems with communication and insurance.
  • Family members who provide care to individuals with chronic or disabling conditions are themselves at risk of developing emotional, mental and physical health problems due to complex caregiving situations.

Abstract

Learn how socioeconomic status affects individuals with disabilities.

APA Style Reference

APA (2010). Disability & Socioeconomic Status [Blog post]. Retrieved from https://www.apa.org/pi/ses/resources/publications/disability

You may also be interested in




A cry for help (Nature, 2019)

Main Takeaways:

  • 29% of 5700 respondents to a survey in 2017 listed their mental health as an area of concern while less than half of those sought help for anxiety or depression caused by their PhD study.
  • A new survey with 6300 graduate students from around the world show that 71% are satisfied with their experience of research, while 36% had help for their anxiety or depression related to their PhD.
  • How can graduate students be both broadly satisfied, but increasingly unwell? One of the reasons might lie on the fact that 1/5 of respondents report being bullied and experience harassment or discrimination.
  • Although universities should take more effective action, only ¼ of respondents said their institution provides support while 1/3 said they seek help elsewhere.
  • Another reason for graduate students to be broadly satisfied, but increasingly unwell is that career success is measured by publications, citations, funding and impact. To progress, a researcher must hit high scores in all of these measures.
  • Most students embark on a PhD as a foundation of an academic career. They believe they will have the freedom to discover and invent. However, problems arise when autonomy is reduced or removed, which occurs when targets for funding, impact and publications become part of the universities’ formal monitoring and evaluation systems.
  • As student’s supervisors judge their success or failure, it is not surprising many feel unable to open up about vulnerabilities or mental-health concerns.
  • Solutions do not solely lie in institutions doing more to provide on-campus mental health support, but also in the recognition of ill mental health as a consequence of excessive focus on measuring performance.
  • Much has been written about how to overhaul the system and find a better way to define success in research, including promoting that many non-academic careers are open to researchers.
  • The academic system is making young people ill and the research community needs to protect and empower the next generation of researchers.

Abstract

Without systemic change to research cultures, graduate-student mental health could worsen.

APA Style Reference

Nature. (2019). The mental health of PhD researchers demands urgent attention. Nature, 575, 257-258. https://www.nature.com/articles/d41586-019-03489-1

You may also be interested in




Registered reports (Jamieson et al., 2019)

Main Takeaways:

  • Stage I article submitted with title page, abstract, introduction, methods, analysis plan, and conclusions of a study before carrying out research.
  • Stage I includes the article and a cover letter, confirming all support and approval is in place, timeline for completing this study, statement confirming authors share raw data, digital materials, analysis.
  • Authors confirm registration of Stage I article with the Open Science Framework or another repository.
  • Method section includes justification of sample sizes compared to question, description of participants, problems investigated, a priori justification, procedures to deduce inclusion and exclusion criteria and clear protocol of experimental procedures.
  • Data analysis: outline and justify how data is treated including all pre-processing steps.
  • Two step review of Stage I paper. Triaged by an editorial team before passing to peer review. Peer reviewers assess importance of research question, introduction, plausibility, quality of hypotheses and methodological quality and appropriateness of data analysis plan, validity of inferential conclusions based on data.
  • One of three outcomes: conditional approval, revise decision, or reject. Revise decision allows authors to respond to editors and reviewers criticisms. Reject ends the review process.
  • Following conditional approval, authors submit a Stage II article. Stage II article must be consistent with Stage I report. Hypotheses, rationale, and reasoning approved in Stage I must reappear in Stage II.
  • Stage II provides a complete and final report of the approved Stage I article, which also includes raw data, digital materials and analyses. Stage II focuses on quality of data reported, soundness of conclusions drawn from data and consistency with arguments and reasoning.
  • Are data sufficiently resolved to support conclusions? Does the data answer the authors proposed hypotheses? Does the introduction and analyses match the Stage I submission? Any unregistered and post-hoc analyses justified, methodologically sound, and informative? Are conclusions consistent with collected data.
  • Editor can ask for revisions or reject Stage II articles.

Abstract

Professor Randall K. Jamieson provides an editorial on registered reports for the journal Canadian Journal of Psychology and how it works in this specific journal.

APA Style Reference

Jamieson, R. K., Bodner, G. E., Saint-Aubin, J., & Titone, D. (2019) Editorial: Registered reports. Canadian Journal of Experimental Psychology, 73, 3-4.

You may also be interested in




Postdocs in crisis: science risks losing the next generation (Nature, 2020)

Main Takeaways:

  • Post-doctoral researchers often spend years in a succession of short-term contracts, which creates immense anxiety and uncertainty.
  • Nature conducted a survey with postdocs and asked how the pandemic is affecting their current and future career plans, their health and well-being; and whether they feel supported by their supervisors.
  • Respondents spam across 93 countries (and different fields), but most are from the US and Europe.
  • Results show that the pandemic adds to postdocs’ distress. The pandemic worsened career prospects and supervisors have not done enough to support them during pandemic.
  • 51% of respondents are considering leaving active research due to work-related mental health concerns.
  • All efforts to help workers are welcome but on their own, small measures will not be enough to save many academic science careers.
  • Universities cannot be expected to bear this extra cost. Universities  are already feeling the consequences of the pandemic for their finances. This is especially the case of institutions dependent on income from international students’ fees.
  • Global student mobility will be much lower than usual in the coming academic year, and some institutions will lose a good fraction of their fee income as a result.
  • In places where research is cross-subsidized from tuition-fee income, contract-research workers such as post-docs are most vulnerable to losing their jobs and women and people from minority groups who form a high share of post-doctoral workforce, will likely be disproportionately affected.
  • As many post-docs are looking to leave their posts now, anticipating worse is to come, research and university leaders must find innovative ways to support early-career researchers.
  • Principal investigators should  show flexibility, patience and support for everyone in their group.
  • Principal investigators and their institutions must push harder than ever for accessible mental health services.

Abstract

The pandemic has worsened the plight of postdoctoral researchers. Funders need to be offering more than moral support.

APA Style Reference

Nature. (2020). Postdocs in crisis: science cannot risk losing the next generation. Nature, 585. 10.1038/d41586-020-02541-9 https://www.nature.com/articles/d41586-020-02541-9

You may also be interested in




Boosting research without supporting universities is wrong-headed  (Nature, 2020b) ⌺

Main Takeaways:

  • Coronavirus lockdowns have precipitated a crisis in university funding and academic morale.
  • Universities all over the world closed their doors. Classes and some research activities were moved online.
  • Staff were given little or no time to prepare and few resources or training to help them with online teaching.
  • Fewer students are expected to enrol in the coming academic year, instead waiting until institutions open fully. This means young people will lose a year of their education and universities will lose out financially.
  • Governments have plans to boost post-lockdown research but these plans will be undermined if universities make job cuts and end up with staff shortages. Universities need support at this crucial time.
  • Low- and middle-income countries face extra challenges from sudden transition to online learning. The main concern is for students unable to access digital classrooms (those who live in areas without fast, reliable and affordable broadband or where students have no access to laptops, tablets, smartphones and other essential hardware).
  • Teachers report students struggle to keep up since lockdown began. Students from poorer households in remote regions travel to the nearest city to access the Internet and pay commercial internet cafes to download course materials. To solve this issue, governments and funding bodies need to accept that students and universities should be eligible for the same kinds of temporary emergency funding as other industries are asking for.
  • Governments have denied requests to negotiate with universities or delayed decisions. In high-income countries, this is partly because universities are functioning and might be seen as less deserving of government help than businesses and professions that had no choice but to close. In poorer countries, public funding for universities is under threat because economies have crashed during lockdowns.
  • Cuts in universities’ budgets will disproportionately affect poorest students and more vulnerable members of staff (those with fixed-term contracts).
  • Students and staff on short-term contracts would welcome more support from academic colleagues in senior positions and from others with permanent positions.
  • Colleagues should make the case for managers that failing to provide more help to low-income students or cutting the number of post-doctoral staff and teaching fellows presents a harm to the next generation of researchers and teachers. It will reduce departments’ capacity to teach and increase load on those who remain.
  • Cutting back on scholarly capacity while increasing spending on research and development is wrong-headed, slowing down economic recovery and jeopardising plans to make research more inclusive.

Abstract

Universities face a severe financial crisis, and some contract staff are hanging by a thread. Senior colleagues need to speak up now.

APA Style Reference

Nature. (2020). Boosting research without supporting universities is wrong-headed. Nature, 582, 313-314. https://www.nature.com/articles/d41586-020-01788-6

You may also be interested in




Seeking an exit plan (Woolston, 2020)

Main Takeaways:

  • Full impact of COVID-19 pandemic on scientific careers might not be known for years, but hiring freezes and other signs of turmoil at universities shake faith in academia as career options.
  • A growing number of PhD students and other early-career researchers start to look at careers in industry, government and other sectors.
  • It is unclear how many of these researchers will eventually leave academia out of choice or necessity, but a significant academic exodus is expected.
  • It is suggested that the shortage of tenured and tenure-track university positions will deepen in coming years. History shows that, in the United States, recession coincided with a strong shift towards gig or temporary work.
  • Academic escapees have to prepare themselves to navigate a new career landscape. As competition for industry jobs will be stiff, it is important  to learn how to emphasise skills developed in university careers.

Abstract

The pandemic is prompting some early-career researchers to rethink their hopes for a university post. By Chris Woolston.

APA Style Reference

Nature. (2020). Seeking an ‘exit plan’ for leaving academia amid coronavirus worries. Nature 583, 645-646. Doi: 10.1038/d41586-020-02029-6.

You may also be interested in




Lesbian, Gay, Bisexual and Transgender Persons & Socioeconomic Status (APA, 2010) ◈ ⌺

Main Takeaways:

  • Individuals who identify as Lesbian, gay, bisexual and/or transgender are specially susceptible to socio-economic disadvantages.
  • Socioeconomic status is inherently linked to rights, quality of life, and general well-being of Lesbian, Gay, Bisexual and/or transgender persons.
  • Low income LGBT individuals and same-sex/gender couples have been found to be more likely to receive cash assistance and food stamps benefits compared to heterosexual individuals or couples.
  • Transgender adults were nearly 4 times more likely to have household income of less than $10,000 per year relative to the general population.
  • Raising the federal minimum wage benefits LGBT individuals and couples in the United States.
  • An increase in minimum wage should reduce poverty rates by 25% for same-sex/gender female couples and 30% for same-sex/gender male couples.
  • Due to an increase in minimum wage, poverty rates would be projected to fall for the most vulnerable individuals in same-sex/gender couples, including African American, couples with children, people with disabilities, individuals under 24 years of age, people without high school diplomas or the equivalent, and those living in rural areas.
  • The socio-economic position may be linked to experiences of discrimination.
  • Gay and bisexual men who earned higher income were less likely to report discrimination relative to those in lower socio-economic positions.
  • Discrimination against and unfair treatment of LGBT persons remains legally permitted. 47% of transgender individuals report being discriminated against in hiring, firing and promotion, over 25% had lost a job due to discrimination based on gender identity.
  • A lack of acceptance and fear of persecution lead many LGBT youth to leave their homes and live in transitional housing or on the street.
  • Many LGBT youth may be rejected by their family of origin or caregivers and forced to leave home as minors.
  • LGBT youth experience homeless at a disproportionate rate.
  • LGBT homeless youth are more likely than their homeless heterosexual counterparts to have poorer mental and physical health outcomes.
  • Alhough since 2015 states must issue marriage licenses to same-sex couples and recognise same-sex unions, legal barriers continue to exist.
  • Workplace and housing discrimination contribute to increasing socio-economic status disparities for LGBT persons and families.
  • 20 states and District of Columbia prohibit discrimination in workplace based on sexual orientation and gender identity, while 18 states have no laws prohibiting workplace discrimination against LGBT people.
  • 19% of transgender individuals report in a previous study that they were  refused a home or apartment and 11% report being evicted because of their gender identity or expression.

Abstract

Evidence indicates individuals who identify as lesbian, gay, bisexual and/or transgender (LGBT) are especially susceptible to socioeconomic disadvantages. Thus, SES is inherently related to the rights, quality of life and general well-being of LGBT persons.

APA Style Reference

APA (2010). Lesbian, Gay, Bisexual and Transgender Persons & Socioeconomic Status. [Blog post]. Retrieved from https://www.apa.org/pi/ses/resources/publications/lgbt

You may also be interested in




The Focus on Fame distorts Science (Innes-Ker, 2017) ◈ ⌺

Main Takeaways:

  • The author argues that instead of focusing on individual merit it is important that science is focused on scientific ideas and collaborative groups.
  • Asking if you are famous is a wrong question. It focuses on the individual scientist, as if science is a lonely enterprise of hopeful geniuses.
  • We should focus on ideas and knowledge and refining those ideas.
  • H-index is not an objective measure. It presupposes that peer-review papers are solid and that citations are a proxy for quality.
  • Science is argued to advance in an evolutionary manner. A wealth of ideas is produced, but only some are selected and survive depending on scientific merit and social process (production of papers, citations and engagement of groups of scientists).
  • Ideas that engage groups of scientists will grow and change and bring knowledge closer to the truth. Ideas that are not interacted with, on the other hand, will likely die. This is far from focus on eminence and individual fame prevalent in science.
  • Competition is a factor but cooperation is vital.
  • For ideas to survive, multiple labs need to engage with them as champions or severe adversarial testers.
  • If we focus on who may become eminent, we lose some power of the scientific process.
  • Eminent scientists would be nowhere without collaborators and adversaries willing to engage with the ideas.
  • The tendency to overwhelmingly publish only positive results with no clear avenue for publishing failures to confirm, means scientists are not grappling with the real field.
  • Recent work to improve methods, statistics and publishing practices is an example of collaboration.
  • In science, scientific ideas are the ones that need to be stress-tested, not scientists.
  • We need to move away from the cultural market model of science focusing on individuals rather than on robustness of ideas. Science is a low yield, high risk business.
  • Assigning individual merit based on productivity and citation encourages poor scientific practices and discourages collaboration and argumentative engagement with ideas. It results in a waste of talent.
  • Objectivity in Science is not a characteristic of individual researchers, but a characteristic of scientific communities.

Abstract

The 2016 symposium on Scholarly Merit focused on individual eminence and fame. I argue, with some evidence, that the focus on individual merit distorts science. Instead we need to focus on the scientific ideas, and the creation of collaborative groups.

APA Style Reference

Innes-Ker, Å. (2017). The Focus on Fame Distorts Science. https://psyarxiv.com/vyr3e/

You may also be interested in




Fame: I’m Skeptical (Ferreira, 2017) ◈ ⌺

Main Takeaways:

  • The author argues that fame and quality sometimes diverge and that reliance on fame helps to perpetuate stereotypes that keep women and underrepresented groups from participating in science.
  • Most of us believe we have the respect of our peers and acknowledge we wish to be admired and viewed as successful and important.
  • No psychologist and no rational person would deny that evaluating people and the quality of their work is necessary and inevitable in any field.
  • We like to admit most promising candidates to graduate programs, hire the best faculty, tenure only those who have long productive careers and reward scientists with prizes if they contributed more than most to uncover the nature of psychological processes.
  • We must not conflate fame and scientific quality, integrity and impact.
  • All of us point to colleagues who completed excellent work but are barely known or who are not famous until long after their research careers have ended.
  • Some scientists are well known because they have been called out for unethical practices, including data fabrication and other forms of cheating.
  • We need to discriminate between two questions: (i) what one must do to become famous and (ii) what leads a person to end up famous. While the second question is merely an attempt to reconstruct someone’s path to fame, the motivations of the first question need to be challenged.
  • Fame should not be a goal in science and valuing people or ideas because they are famous comes at a risk.
  • Fame should be viewed with caution and scepticism to avoid temptation to assume that if someone is famous, their work is significant.
  • Fame perpetuates discrimination and overlook excellent people and work.
  • Science is based on critical thinking. As such, we should never hesitate to question the ideas of someone who is famous.
  • We should not refuse to view the work of famous people positively or refuse to give it its due, but we must be careful to think an idea is useful due to the person being famous.

Abstract

Fame is often deserved, emerging from a person’s significant and timely contributions to science. It is also true that fame and quality clearly sometimes diverge: many people who do excellent work are barely known, and some people are famous even though their work is mediocre. Reliance on fame and name recognition when identifying psychologists as candidates for honors and awards helps to perpetuate a range of stereotypes and prevents us from broadening participation in our field, particularly from women and underrepresented groups. The pursuit of fame may also be contributing to the current crisis in psychology concerning research integrity, because it incentivizes quantity and speed in publishing. The right attitude towards fame is to use it wisely if it happens to come, but to focus our efforts on conducting excellent research and nurturing talent in others.

APA Style Reference

Ferreira, F. (2017). Fame: I'm Skeptical (2017).  https://psyarxiv.com/6zb4f/

You may also be interested in




Let’s Look at the Big Picture: A System-Level Approach to Assessing Scholarly Merit (Pickett, 2017) ◈ ⌺

Main Takeaways:

  • Why do we care about judging scientific merit? There is a need to have a system to determine whether to award tenure and promotion to faculty members, leading to a development of criteria in order to judge and measure the scholarly merit of individuals.
  • Science is a collective enterprise whose goal is to explain and understand the natural world and to build knowledge. Science cares about advancements and discoveries, not about individuals.
  • Individual scientists are valued to the extent that they further the goals of the collective system. However, science comprises lab workers, scientists, institutions, agencies, and broader society.
  • At organisation level, features facilitate scientific discovery-organisational autonomy, organisational flexibility, moderate scientific diversity and frequent and intense interaction among scientists with different viewpoints.
  • An individual scientist contributes to scientific discovery directly through their own scientific products or indirectly by positively affecting other aspects of the system.
  • More senior graduate students train incoming graduate students- when good at this the output of an entire lab can skyrocket as a result.
  • Graduate students not only conduct their own personal research but their presence in the lab facilitates scientific progress of others.
  • Scientists promote productivity of other scientists by reviewing manuscripts, sharing data, creating and serving scientific organisations, and developing scientific tools and paradigms used by others.
  • Individual research scientists do not have resources to create large research centres, but can organise conferences and symposia, create and contribute to scientific discussion platforms, and make their research protocols and data easily shareable.
  • Scholarly merit should include an individual’s system-level contributions, not only their productivity.

Abstract

When judging scientific merit, the traditional method has been to use measures that assess the quality and/or quantity of an individual’s research program. In today’s academic world, a meritorious scholar is one who publishes high quality work that is frequently cited, who receives plentiful funding and scientific awards, and who is well regarded among his or her peers. In other words, merit is defined by how successful the scholar has been in terms of promoting his or her own career. In this commentary, I argue that there has been an overemphasis on measuring individual career outcomes and that we should be more concerned with the effect that scholars have on the scientific system in which they are embedded. Put simply, the question we should be asking is whether and to what extent a scholar has advanced the scientific discipline and moved the field forward collectively.

APA Style Reference

Pickett, C. (2017). Let's Look at the Big Picture: A System-Level Approach to Assessing Scholarly Merit. https://psyarxiv.com/tv6nb/

You may also be interested in




“Fame” is the Problem: Conflation of Visibility With Potential for Long-Term Impact in Psychological Science (Shiota, 2017)◈ ⌺

Main Takeaways:

  • Fame is about visibility – who is seen. Ample evidence documents the influence of heuristics in determining who is visible, and whose contribution is considered important.
  • Explicit and implicit beliefs about competence influences peer review when methodological quality or potential impact is ambiguous.
  • The author is sceptical about the extent that fame is shaped by the quality of one’s work instead of confidence, dominance, persistence and demographics.
  • The pace of academic life accelerates, the pressure to depend on shortcuts in gatekeeping and evaluation will continue to grow.
  • The scientific community cannot remove implicit biases, there are ways to deflect the impact of these implicit biases.
  • Reviews of submitted work should be blind to identity and demographics, letting the quality of the product stand on its own.

Quote

“We specify criteria for good science flexibly but explicitly and in detail, including thorough and accurate contextualisation in relevant previous work, methodological rigour; innovation and problem solving and implications for theory, future research and/or intervention.  We should insist on diversity in career stage, gender, ethnicity and perspective instead of inviting first people who come to mind for invited opportunities such as conference talks, contribution to edited volumes, awards, and participation in committees that determine the direction of our field. We can resist temptation to track women and minorities into high profile, high-demand services roles, thinking that this solves problems of diversity in science. When, in fact, it does not.” (p.7)

Abstract

To be famous is to be widely known, and honored for one’s achievements. The process by which researchers achieve fame or eminence is skewed by heuristics that influence visibility; implications of these heuristics are magnified by a snowball effect, in which current fame leads to bias in ostensibly objective metrics of merit, including the distribution of resources that support future excellence. This effect may disproportionately hurt women and minorities, who struggle with both external and internalized implicit biases regarding competence and worth. While some solutions to this problem are available, they will not address the deeper problems of defining what it means for research to “make a difference” in our field and in society, and consistently holding our work to that criterion.

APA Style Reference

Shiota, M. N. (2017) “Fame” is the Problem: Conflation of Visibility With Potential for Long-Term Impact in Psychological Science. https://psyarxiv.com/4kwuq

You may also be interested in




Why a Focus on Eminence is Misguided: A Call to Return to Basic Scientific Values (Corker, 2017) ◈ ⌺

Main Takeaways:

  • The author argues that our current methods of scientific rewards are based on identifying research eminence. This reward system is not in line with scientific values of transparency and universalism and undermines scientific quality.
  • Why do we accord knowledge derived from scientific method a privileged position relative to common sense, appeals to authority figures, or other forms of rhetoric?
  • If scientists depend on their own expertise as justification to prioritise their claims, we are not better to make truth-claims than religious, political and other leaders.
  • Instead, science’s claim on truth comes not from its practitioners’ training and expertise, but rather from its strong adherence to norms of transparency and universalism.
  • Universalism means scientists reject claims of special authority. It matters far less who did the research than how it was done.
  • How do we square scientific ideals of universalism with scientific culture that fetishizes lone scientific genius?
  • We need to recognise the methods used to produce a scientific claim are more important than eminence of a person who produced it.
  • Focusing primarily on the individual researcher excellence hurts psychological science, as eminence reflects values that are counterproductive to maximise scientific knowledge.
  • The current system privileges quantity over quality, outcome of research instead of the process itself.
  • Systematic biases (e.g., structural sexism, racism, and status bias) affect how we identify who qualifies as eminent under status quo.
  • Gender, nationality, race or institution should not matter to measure research quality.
  • Structural changes should be initiated to help researchers reward and evaluate quality research (i.e., work that is reproducible, transparent and open, and likely to be high in validity).
  • We can do a much better job to recognise and reward many activities  researchers do that support scientific discovery beyond publishing peer reviewed articles (e.g. develop scientific software, generate large datasets, write data analytic code and construct tutorials to teach others to use it).
  • We need to re-evaluate ways to measure researchers’ excellence in light of value and promise of team-driven research. After all, science is a communal endeavour.
  • To combat structural and systematic problems linked to recognising eminence, double blind peer reviews need to be considered as standard practice for journal publication, grant funding and awards committee.
  • Technological solutions could even be developed to allow departments to blind in early stages of faculty hiring, as blinding is associated with higher levels of diversity.

Abstract

The scientific method has been used to eradicate polio, send humans to the moon, and enrich understanding of human cognition and behavior. It produced these accomplishments not through magic or appeals to authority, but through open, detailed, and reproducible methods. To call something “science” means there are clear ways to independently and empirically evaluate research claims. There is no need to simply trust an information source. Scientific values thus prioritize transparency and universalism, emphasizing that it matters less who has made a discovery than how it was done. Yet, scientific reward systems are based on identifying individual eminence. The current paper contrasts this focus on individual eminence with reforms to scientific rewards systems that help these systems better align with scientific values.

APA Style Reference

Corker, K. S. (2017). Why a Focus on Eminence is Misguided: A Call to Return to Basic Scientific Values. https://psyarxiv.com/yqfrd

You may also be interested in




Unequal effects of the COVID-19 pandemic on scientists (Myers et al., 2019) ⌺

Main Takeaways:

  • COVID-19 pandemic disrupted scientific enterprise.
  • Policymakers and institutional leaders have started to respond to reduce influences of pandemic on researchers.
  • For this study, authors reached out to US- and Europe-based scientists across institutions, career stages and demographic backgrounds.
  • The present paper solicited information about working hours and how time allocations changed since the onset of pandemic and asked scientists to report the range of individual and family properties, as these feature moderate effects of pandemic.
  • The sample was self-selected and it is likely that those who feel strongly about sharing situations, whether they experienced large positive or negative changes due to the pandemic, were the ones who chose to participate.
  • They found a decline in total working hours with the average dropping from 61 hours per week pre-pandemic to 54 hours at time of survey.
  • Only 5% of scientists report they worked 42 hours or less before the pandemic. This  share increased to 30% of scientists during the pandemic.
  • Time devoted to research has changed most during pandemic. Total working hours decreased by 11% on average, but research declined by 24%.
  • Scientists working in fields that rely on physical laboratories and on time sensitive experiments report largest declines in research time (in the range of 30-40% below pre-pandemic levels).
  • Fields that are less equipment intensive (e.g., mathematics, statistics, computer science and economics) report lowest declines in research time. The difference to other fields can be as large as fourfold.
  • There are differences between male and female respondents in how the pandemic influenced their work.
  • Female scientists and scientists with young dependents report ability to devote time to their research has been influenced and effects are additive - most impact was for female scientists with young dependents.
  • Individual circumstances of researchers best explain changes in time devoted to research during pandemic.
  • Career stage and facility closures did not contribute to changes in time allocated to research when everything else is held constant. Gender and young dependents contributed major roles.
  • Female scientists reported a 5% larger decline in research time than male scientists, but scientists with at least one child 5 years old or younger experienced a 17% larger decline in research time.
  • Having multiple dependents was linked to a  further 3% reduction in time spent on research. Scientists with dependents aged 6-11 years were less affected.
  • This indicates gender discrepancy can be due to female scientists being more likely to have young children as dependents.
  • Results indicate that the pandemic influences members of the scientific community differently.
  • Shelter at home is not the same as work from home, when dependents are also at home and need care.
  • Unless adequate childcare services are available, researchers with young children continue to be affected irrespective of reopening plans of institutions.
  • Pandemic will likely have longer-term impacts that are important to monitor. Further efforts to track effects of pandemic on the scientific workforce need to consider household circumstances.
  • Uniform policies do not consider individual circumstances and may have unintended consequences and worsen pre-existing inequalities.
  • The disparities may worsen as institutions begin the process of reopening given that different priorities for bench sciences versus work with human subjects or field-work travel may lead to new disparities across scientists.
  • Funders seeking to support high-impact programs adopt a similar approach, favouring proposals that are more resilient to uncertain future scenarios.
  • Senior researchers have incentives to avoid in-person interactions facilitating mentoring and hands-on training of junior researchers.
  • Impact of changes on individuals and groups of scientists could be large in short- and long-term, worsening negative impacts among those at a disadvantage.
  • We need to consider consequences of policies adopted to respond to pandemic, as they may disadvantage under-represented minorities and worsen existing disparities.

Quote

“The disparities we observe and the likely surfacing of new impacts in the coming months and years argue for targeted and nuanced approaches as the world-wide research enterprise rebuilds.” (p.882)

Abstract

COVID-19 has not affected all scientists equally. A survey of principal investigators indicates that female scientists, those in the ‘bench sciences’ and, especially, scientists with young children experienced a substantial decline in time devoted to research. This could have important short- and longer-term effects on their careers, which institution leaders and funders need to address carefully.

APA Style Reference

Myers, K. R., Tham, W. Y., Yin, Y., Cohodes, N., Thursby, J. G., Thursby, M. C., ... & Wang, D. (2020). Unequal effects of the COVID-19 pandemic on scientists. Nature Human Behaviour, 4, 880-883. https://doi.org/10.1038/s41562-020-0921-y

You may also be interested in




Don’t let transparency damage science (Lewandowsky & Bishop, 2016)

Main Takeaways:

  • Scientific communities have launched initiatives to increase transparency, open critique, and data sharing.
  • Good researchers include all perspectives but their openness can be abused by opponents who aim to stall inconvenient research.
  • Science is prone to attacks but rigour and transparency helps researchers and their universities respond to valid criticism.
  • Open data practices should be adopted and scientists should not regard all requests for data as harassment.
  • Researchers should explain why they cannot share their data. Confidentiality issues need to be considered, also researchers need control over how data is going to be used if the participant agrees to the sharing of this data.
  • Engagement with critics is a fundamental part of scientific practice. Researchers may feel obliged to respond even to trolls but can ignore abusive or illogical critics that make the same points.
  • Minor corrections and clarifications after publications should not be seen as a stigma against fellow researchers. Thus,
  • Publications should be seen as living documents with corrigenda being accepted (even if unwelcome) as part of scientific progress.
  • Self-censorship affects academic freedom and discussion. Publication retractions should be reserved for fraud or grave errors, but often are demanded by people who do not like a paper’s conclusions.
  • Complaints may undervalue researchers for legal but contentious science. Harassed scientists feel alone. They should not tolerate harassment dependent on race or gender nor if it is based on controversial science.
  • Training and support should be used to aid researchers in the ability to cope with harassment.

Abstract

Professor Stephan Lewandowsky and Professor Dorothy Bishop explain how the research community should protect its members from harassment, while encouraging openness and transparency as it is essential for science.

APA Style Reference

Lewandowsky, S., & Bishop, D. (2016). Research integrity: Don't let transparency damage science. Nature, 529(7587), 459-461.http://dx.doi.org/10.1038/529459a

You may also be interested in




Am I Famous Yet? Judging Scholarly Merit in Psychological Science: An Introduction (Sternberg, 2016)

Main Takeaways:

  • Quality, productivity, visibility and impact are judged by the department in terms of scientific merit. These evaluations are subjective.
  • Letters from distinguished referees provide a detailed and qualitative assessment of the referee’s future potential and the nature of the individual’s scientific contribution.
  • Letter writers are prone to fads, biases and personal idiosyncrasies that can positively or negatively affect the chances of tenure or promotion.
  • Quantity of publication is a reasonable measure of the researcher’s productivity. If one does not worry about the prestige of the journal, most articles get published. However, it does not inform us about the quality of the researcher’s work.
  • Quantity of publications controlling for impact factors of journals tells us how much, on average, articles in that specific journal get cited. However, the problem is that it focuses on the average, some articles get highly cited and some do not get cited at all. Also, prestigious journals are conservative in what they do publish.
  • Number of citations is a good measure and provides us information about how often is the researcher cited over their career. If you are in a controversial topic, or in an area that appeals to the broad audience or work in hot areas, it can provide a unique advantage to those researchers.
  • H index is the number of publications cited at least h times and takes into account how quality and quantity affect impact. I10 is the number of publications cited at least 10 times.
  • Grants and contracts show scholars have systematically and valued proposed programs of research.
  • Editorship shows scholar’s work is recognised in their field. Invited service on a grant panel is another recognition of success in one’s professional endeavours.
  • Awards are a useful measure of recognition by peers and measure quality of work instead of citation to work.
  • Honorary doctorates are recognitions by broader academic audiences of merit of a scholar’s work.
  • There are not many big psychological theorists left. Some would say the shift represents a natural progression as the field becomes more and more of a natural science.
  • The big thinkers of yesterday might be taken aback by the amount of work done in modern times.
  • The use of neuroimaging, behavioural experiments importance shrinks towards small-scale psychology without theory but with a large theory, they contribute to larger theory. Big thinking pays off.

Quote

“Most of us in academia go through a series of increasingly more challenging evaluations—first to get the PhD, next at the time of hiring, then at the time of reappointment, subsequently at the time of tenure, and finally at the time of promotion to full professor. And when we go through these evaluations, we almost inevitably wonder whether the criteria by which we will be judged are fair and whether the criteria, whatever they are, will be applied fairly.” (p.877)

Abstract

The purpose of this symposium is to consider new ways of judging merit in academia, especially with respect to research in psychological science. First, I discuss the importance of merit-based evaluation and the purpose of this symposium. Next, I review some previous ideas about judging merit—especially creative merit—and I describe some of the main criteria used by institutions today for judging the quality of research in psychological science. Finally, I suggest a new criterion that institutions and individuals might use and draw some conclusions.

APA Style Reference

Sternberg, R. J. (2016). “Am I famous yet?” Judging scholarly merit in psychological science: An introduction. Perspectives on Psychological Science, 11(6), 877-881. https://doi.org/10.1177/1745691616661777

You may also be interested in




Against Eminence (Vazire, 2017) ◈ ⌺

Main Takeaways:

  • The author argues that the drive for eminence is inherently at odds with scientific values and that insufficient attention to this problem is partly responsible for the recent crisis of confidence in psychology and other sciences.
  • Transparency makes it possible for scientists to discriminate robust from shaky findings.
  • The Replicability crisis shows a system without transparency does not work.
  • Those in charge of setting scientific norms and standards should strive to increase transparency, bolster our confidence that we trust published research.
  • However many high level decisions in science are made with a different goal in mind: to increase impact.
  • Professional societies and journals prioritise publishing attention-grabbing findings to boost visibility and prestige.
  • Seeking eminence is at odds with scientific value and affects scientific gatekeepers’ decisions.
  • Editors influenced by the status of submitting authors or prestige of institutions violate the basic premise of science. Science work should be evaluated on its own merit, irrespective of the source.
  • Lack of transparency in science is a direct consequence of the corrupting influence of eminence seeking.
  • Gatekeepers control incentive structures that shape individual researchers’ behaviour. Therefore they have a bigger responsibility to uphold scientific values and most power to erode those values.
  • Individual researchers’ desire for eminence threatens the integrity of the research process.
  • All researchers are human and desire recognition for their work. However, there is no good reason to amplify this human drive and encourage scientists to seek fame.
  • The glorification of eminence also reinforces inequalities in science. If scientists are evaluated based on ability to attract attention, those with the most prestige will be heard the loudest. Certain groups are overrepresented at a high level of status.
  • Eminence propagates privilege and raises barriers to entry for others.
  • How should scientific merit be evaluated? What does this mean for committees to select one or few winners?
  • First, it is important to admit that a larger number of scientists meet the objective criteria for these recognitions (i.e., do sound science).
  • It is also important to admit that selection of one or few individuals is not based on merit but on preference or partiality.
  • It is fine to select or recognise members who exemplify their values, but this should not be confused with exceptional scientific merit.
  • Whenever possible (for tenure, promotion and when journal space or grant fund permits), we should attempt to reward scientists whose work reaches a more objective threshold of scientific rigour or soundness instead of selecting scientists based on fame.

Abstract

The drive for eminence is inherently at odds with scientific values, and insufficient attention to this problem is partly responsible for the recent crisis of confidence in psychology and other sciences. The replicability crisis has shown that a system without transparency doesn’t work. The lack of transparency in science is a direct consequence of the corrupting influence of eminence-seeking. If journals and societies are primarily motivated by boosting their impact, their most effective strategy will be to publish the sexiest findings by the most famous authors. Humans will always care about eminence. Scientific institutions and gatekeepers should be a bulwark against the corrupting influence of the drive for eminence, and help researchers maintain integrity and uphold scientific values in the face of internal and external pressures to compromise. One implication for evaluating scientific merit is that gatekeepers should attempt to reward all scientists whose work reaches a more objective threshold of scientific rigor or soundness, rather than attempting to select the cream of the crop (i.e., identify the most “eminent”).

APA Style Reference

Vazire, S. (2017). Against eminence. https://doi.org/10.31234/osf.io/djbcw

You may also be interested in




Giving Credit Where Credit’s Due: Why It’s So Hard to Do in Psychological Science (Simonton, 2016)

Main Takeaways:

  • The article begins with indicators of scientific achievement before discussing some important precautions about the implications of these measures.
  • One measure is the lifetime career award that is based on the scientist’s cumulative record that spans over their career lifetime. This indicator is perceived as more reliable than early career award, as an early career award is founded on a much smaller and likely less representative sample of that career.  This limitation also applies to an award for a single publication such as best article recognition, as it may not predict the citations that the article later receives in the literature. The article argues that it makes most sense to concentrate on assessing lifetime contributions to psychological science.
  • Another measure to assess scientific achievement is an invitation to write a definitive handbook chapter, as it indicates that the scientist is a widely recognised expert on this specific topic.
  • A final measure to indicate that a scientist is well known is the simple count of total citations. However, this may be assessed by several alternative citation measures (e.g. h-index and i10).
  • However, these measures suffer from poor predictive validity, as the relationship between various predictors and criterion variables tend to be small to moderate, not large enough to make fine discriminations among scientists. In turn, these predictive utilities are contaminated with other potentially biasing factors (e.g. gender, ethnicity, specialty, methodology, ideology, affiliation, and publication type).
  • In addition, these measures suffer from interjudge reliability, as a psychologist receives mixed reviews after submitting a manuscript to a high-impact journal. For instance, one referee recommends the author to publish the manuscript with minor revision, while another advises an outright rejection. This frustrates the author and the editor, but proves the discipline lacks a strong consensus on what contributes to science or their specific area.
  • This lack of agreement should be reduced if evaluators operate with a larger sample of contributions such as lifetime career awards. However, the same problem from interjudge reliability applies to lifetime awards, as one committee member would argue that the scientist deserves this award, while a minority of the committee may disagree with the final decision and argue another scientist deserves this award. In the end, “the committee chair can then only assure the dissenters that their preferred candidate will most definitely emerge the winner in the next award cycle.” (p.890).

Quote

“Eminence in any scientific discipline will therefore be directly proportional to actual contributions. This expectation would be especially strong given that scientists purport to make inferences based on empirical fact and logical reasoning. Not only would peer assessments prove highly objective, but scientists’ self-assessments of their own contributions should depart relatively little from colleagues in the best position to evaluate their work. In short, a strong consensus should permeate all evaluations. One specific manifestation of this consensus would appear in the awards and honors bestowed on those scientists who have devoted a whole career to producing high-impact work. That is what would happen ideally, but does that happen in fact? And even if the ideal is closely approximated in most sciences, is it also reasonably attained in psychological science?” (p.888)

Abstract

More than a century of scientific research has shed considerable light on how a scientist’s contributions to psychological science might be best assessed and duly recognized. This brief overview of that empirical evidence concentrates on recognition for lifetime career achievements in psychological science. After discussing both productivity and citation indicators, the treatment turns to critical precautions in the application of these indicators to psychologists. These issues concern both predictive validity and interjudge reliability. In the former case, not only are the predictive validities for standard indicators relatively small, but the indicators can exhibit important non-merit-based biases that undermine validity. In the latter case, peer consensus in the evaluation of scientific contributions is appreciably lower in psychology than in the natural sciences, a fact that has consequences for citation measures as well. Psychologists must therefore exercise considerable care in judging achievements in psychological science—both their own and those of others.

APA Style Reference

Simonton, D. K. (2016). Giving credit where credit’s due: Why it’s so hard to do in psychological science. Perspectives on Psychological Science, 11(6), 888-892. https://doi.org/10.1177/1745691616660155

You may also be interested in




Eminence and Omniscience: Statistical and Clinical Prediction of Merit (Foss, 2016)

Main Takeaways:

  • The author asks the reader to take the perspective of the individual who has the final say in making a tenure, promotion, or hiring decision. The author also asks that you imagine the difference between the fallible human state we are in on such an occasion and what it would be like to be omniscient when making such decisions.
  • The author argues that there are two types of eminence: Deep eminence and surface eminence. The former refers to you as omniscient, you know, and future generations will know, if the candidate is doing work moving some part of the discipline toward “capital-T Truth”, while the latter is the basis for our mere-mortal judgment of ‘tenurability’ in a candidate.
  • Citation data predicts early prominence at least at extremes of citation distribution. However, longitudinal studies with large sample sizes are required to investigate this question.

Quote

“Diener suggests that the discipline will progress more rapidly if the most highly productive individuals are allowed to be even more productive, which will occur by further unburdening these worthies from their teaching responsibilities. I have three quick reactions to this point. One is that, to a considerable extent, his proposal has already been adopted. The typical teaching assignments in research universities are very substantially lower than they were in the days when modern experimental psychology took off. And it is not unusual to see less productive scholars with teaching assignments that involve, say, larger sections of undergraduates, as well as carrying out other service activities. Second, in many places it is still possible for successful grantees to “buy out” some of their teaching time. By definition, these are members of the publishing crew or they would not have the grant money that allows this exchange. And third, and most importantly, let’s revisit what a university is for. One of its primary goals is to develop the human capital of society. In order to keep faith with the funders of (at least the public) universities, we should be leery of allowing that mission to slip too low in our goal hierarchy.” (p.914)

Abstract

In this article, I review, comment upon, and assess some of the suggestions for evaluating scientific merit as suggested by contributors to this symposium. I ask the reader to take the perspective of the individual who has the final say in making a tenure, promotion, or hiring decision. I also ask that one imagine the difference between the fallible human state we are in on such an occasion and what it would be like to be omniscient when making such decisions. After adopting the terminology of “deep” and “surface” eminence, I consider what an omniscient being would take into account to determine eminence and to guide decision-making. After discussing how some proposed improvements in assessing merit might move us closer to wise decisions, I conclude by noting that both data and judgment are, and will continue to be, necessary. A clerk cannot determine eminence.

APA Style Reference

Foss, D. J. (2016). Eminence and omniscience: Statistical and clinical prediction of merit. Perspectives on Psychological Science, 11(6), 913-916. https://doi.org/10.1177/1745691616662440

You may also be interested in




Improving Departments of Psychology (Diener, 2016)

Main Takeaways:

  • Although we have excellent universities, our selection-based approach to talent and productivity is incomplete for creating the very best departments. What can we do to improve our department?
  • Current approach to excellence in scholarship rests largely on hiring the right individuals who have the right talent and motivation.

Abstract

Our procedures for creating excellent departments of psychology are based largely on selection—hiring and promoting the best people. I argue that these procedures have been successful, but I suggest the implementation of policies that I believe will further improve departments in the behavioral and brain sciences. I recommend that we institute more faculty development programs attached to incentives to guarantee continuing education and scholarly activities after the Ph.D. degree. I also argue that we would do a much better job if we more strongly stream our faculty into research, education, or service and not expect all faculty members to carry equal responsibility for each of these. Finally, I argue that more hiring should occur at advanced levels, where scholars have a proven track record of independent scholarship. Although these practices will be a challenge to implement, institutions do ossify over time and thus searching for ways to improve our departments should be a key element of faculty governance.

APA Style Reference

Diener, E. (2016). Improving departments of psychology. Perspectives on Psychological Science, 11(6), 909-912. https://doi.org/10.1177/1745691616662865

You may also be interested in




Varieties of Fame in Psychology (Roediger III, 2016)

Main Takeaways:

  • How do we determine the quality and impact of an individual and their research in psychological science? We use progression of fame to indicate how the system works. The author goes on to discuss routes to fame, but emphasizes that more likely than not, it is ‘local’ fame we are achieving that is not long lasting across time.
  • Once a researcher succeeds in graduate school, obtains a job in academia, industry or research institute,  then it is time to move up in one’s career. Below are different actions one could take:

Quote

“Fame is local, both by area and by time. This point has been made by scholars in other contexts (usually politics or other historical figures), but it is as true of psychology as of any other field. As with other writers in this series, the best advice is to do the research, the writing, and the teaching that you are passionate about. Fame may or may not come for a time, but should not be an all-consuming concern. Even if it comes, it will soon fade away”  (p.887)

Abstract

Fame in psychology, as in all arenas, is a local phenomenon. Psychologists (and probably academics in all fields) often first become well known for studying a subfield of an area (say, the study of attention in cognitive psychology, or even certain tasks used to study attention). Later, the researcher may become famous within cognitive psychology. In a few cases, researchers break out of a discipline to become famous across psychology and (more rarely still) even outside the confines of academe. The progression is slow and uneven. Fame is also temporally constricted. The most famous psychologists today will be forgotten in less than a century, just as the greats from the era of World War I are rarely read or remembered today. Freud and a few others represent exceptions to the rule, but generally fame is fleeting and each generation seems to dispense with the lessons learned by previous ones to claim their place in the sun.

APA Style Reference

Roediger III, H. L. (2016). Varieties of fame in psychology. Perspectives on Psychological Science, 11(6), 882-887. https://doi.org/10.1177/1745691616662457

You may also be interested in




Scientific Eminence: Where Are the Women? (Eagly & Miller, 2016)  ⌺

Main Takeaways:

  • Women’s scientific contributions in psychology may not be as numerous or influential as those of men.
  • What is the magnitude of the current eminence gender gap?
  • Women’s modest inroads into this list of eminent psychologists deserve respect, given this lag between obtaining a doctorate and attaining eminence and formidable barriers that women once faced in pursuing scientific careers.
  • Psychologists judge eminence by observing signs such as memberships in selective societies, career scientific achievement awards and honorary degrees.
  • Do men exceed women on both quantity and impact of their publication underlies h index?
  • Are these metrics tainted by unfair bias against women?
  • Does the h-index identify potential socio-cultural and individual causes of the eminence gap?
  • Women’s publications are cited less than men. This gap was larger in psychology.
  • Women received 20% fewer in psychology varying across subfields.
  • Gender gap on h-index and similar metrics has two sources: women publish less than men and articles receive fewer citations.
  • Metrics assessing scientific eminence may be tainted by prejudicial bias against female scientists in obtaining grant support, publishing papers, or gaining citations of published papers.
  • If psychologists are disadvantaged in publishing their work, bias may be limited to culturally masculine topics or male-dominated research areas.
  • Such topics and are no doubt becoming rarer in psychology, given women receive most US doctorates.
  • Men’s greater overall citations reflect higher rates of self-citation, women self-cite less often.
  • This reflects men’s larger corpus of their own citable papers.
  • Prejudicial gender bias is limited and presents ambiguity given most studies are correlational instead of experimental.
  • Little is known about possible gender bias in awards for scientific eminence such as science prizes and honorary degrees, which are imperfect indicators of the importance of scientists’ contributions.
  • Female scientists’ lesser rates of publication and citation reflect causes other than biases.
  • Broader socio-cultural factors shape individual identities and motivations.
  • Nature and nurture affects role occupancies so men and women are differently distributed into social roles.
  • Women excel in communal qualities of warmth and concern for others and for men to excel in agentic qualities of assertiveness and mastery.
  • Women are over-represented in less research intensive but more in teaching-intensive ranks and part-time positions.
  • Gender norms discourage female agency may disadvantage to gain status in departmental and disciplinary networks and garner resources.
  • Stereotypes erode women’s confidence in ability to become highly successful scientists.
  • Eminence gender gaps in psychology and other sciences shrink further over time as new cohorts of scientists advance in their careers.
  • Women’s representation among PhD earners has increased dramatically over recent decades.

Abstract

Women are sparsely represented among psychologists honored for scientific eminence. However, most currently eminent psychologists started their careers when far fewer women pursued training in psychological science. Now that women earn the majority of psychology Ph.D.’s, will they predominate in the next generation’s cadre of eminent psychologists? Comparing currently active female and male psychology professors on publication metrics such as the h index provides clues for answering this question. Men outperform women on the h index and its two components: scientific productivity and citations of contributions. To interpret these gender gaps, we first evaluate whether publication metrics are affected by gender bias in obtaining grant support, publishing papers, or gaining citations of published papers. We also consider whether women’s chances of attaining eminence are compromised by two intertwined sets of influences: (a) gender bias stemming from social norms pertaining to gender and to science and (b) the choices that individual psychologists make in pursuing their careers.

APA Style Reference

Eagly, A. H., & Miller, D. I. (2016). Scientific eminence: Where are the women?. Perspectives on Psychological Science, 11(6), 899-904. https://doi.org/10.1177/1745691616663918

You may also be interested in




Intrinsic and Extrinsic Science: A Dialectic of Scientific Fame (Feist, 2016)

Main Takeaways:

  • Fame and desire for a legacy provides meaning in one’s existence.
  • Scientists are not the only group driven by a desire to be famous. What does it mean to be famous in science and how do we measure scientific fame?
  • There is intrinsic motivation to follow one’s interest, curiosity, gut and intuition for important and undiscovered topics, while there is extrinsic motivation to follow money, grants and/or what is being published in top-tier journals.
  • There is a continuum of fame:  On one end, there is ‘mundane or imitative’ science - someone conducts a replication or slight advance of already published research. Its impact is little more than personal, influencing the person conducting it but impacts few other people.
  • ‘Normal’ science is when one takes an idea or theory from within an existing theoretical paradigm and tests it. Most scientific research falls in the normal category. Its impact is regional and/or narrowly national.
  • We have ‘creative science’ - this is moderate to high impact science, heavily cited by other scholars in the field and sometimes garner regional, national, or even international awards.
  • Finally, there is ‘rare transformational/revolutionary’ science that changes the entire field and whose impact is both internal and historic.
  • If a peer-reviewed article is the currency of scientific career, funding is its bread and butter. Research is not possible without finding increasing amounts of money (for most scientists).
  • Generative publications are not only highly cited themselves but also generate other works that are highly cited. If they generate enough new works of high impact, the original publication can be transformative.
  • Once published, articles are either ignored or exert some kind of influence on the field.
  • Publication and citation counts are reliable and robust measures of creative output in science.
  • Scientists could cite any and all work that affects their current research,but this appears to not be the case. Papers with several authors are more likely to be cited due to greater exposure.
  • Other more integrated measures of productivity have been developed to correct some problems. H index: when an author of ‘N’ articles has a number of publications cited at least X  number of times and the rest of articles receive few citations.
  • Traditional and citation-based metrics are impacted by time lag between when an article is published and when citation indexes catch up.
  • Altmetrics measures impact derived from online and social media data. Altmetrics assesses article outcomes such as: the number of times an article is viewed, liked, downloaded, discussed, saved, cited, tweeted, blogged, or recommended.
  • Altmetric data is faster than traditional citation count and h-index because it is counted immediately upon publication with real-time updates at any given time.
  • Publications are necessary but not sufficient conditions for citations, those who publish the most are cited the most.
  • It is important to remember there are individuals who publish a lot but not get cited, and those who publish not much but are heavily cited.
  • One can do very good work but the field may or may not pay much attention to it.
  • Many heavily cited papers make a methodological or statistical advance and are of practical, not theoretical, importance.
  • Psychologists would better understand the difference between individual success and disciplinary success. What is good for one’s career is not always what is good for science.
  • Researchers have begun to make recommendations to authors, editors, and instructors of research methodology to increase replicability such as pre-registering predictions by increasing transparency and clearly justify sample size and publish raw data.
  • Most psychological scientists find a way to marry their intrinsic interests with its extrinsic reward and impact.

Quote

“Finding that sweet spot between the two extremes of joy and recognition may be the best definition of success in science that we can come up with. So if I were to recommend a strategy for up and coming scientists it might be this: develop a research program that combines intrinsic fascination and interest with extrinsic recognition and career advancement. Follow your heart and your head. Explore and develop the riskier, more potentially transformative and creative lines of research at the same time that you develop the safer, more fundable ideas. This might occur by developing two separate lines of research, or better yet, by finding one research program that is both intrinsically motivated and then other people also recognize, appreciate, and reward you for it. If you can do both of these, you stand the best chance of surviving, succeeding, and maybe even becoming famous in the competitive world of academic psychological science” (p.897)

Abstract

In this article, I argue that scientific fame and impact exists on a continuum from the mundane to the transformative/ revolutionary. Ideally, one achieves fame and impact in science by synthesizing two extreme career prototypes: intrinsic and extrinsic research. The former is guided by interest, curiosity, passion, gut, and intuition for important untapped topics. The latter is guided by money, grants, and/or what is being published in top-tier journals. Assessment of fame and impact in science ultimately rests on productivity (publication) and some variation of its impact (citations). In addition to those traditional measures of impact, there are some relatively new metrics (e.g., the h index and altmetrics). If psychology is to achieve consensual cumulative progress and better rates of replication, I propose that upcoming psychologists would do well to understand that success is not equal to fame and that individual career success is not necessarily the same as disciplinary success. Finally, if one is to have a successful and perhaps even famous career in psychological science, a good strategy would be to synthesize intrinsic and extrinsic motives for one’s research.

APA Style Reference

Feist, G. J. (2016). Intrinsic and extrinsic science: A dialectic of scientific fame. Perspectives on Psychological Science, 11(6), 893-898. https://doi.org/10.1177/1745691616660535

You may also be interested in




The Nine Circles of Scientific Hell (Neuroskeptic, 2012)

Main Takeaways:

  • Neuroskeptic provides a humorous take on Dante’s nine circles of hell,  where each circle corresponds to a different questionable research practice that becomes increasingly problematic for scientific integrity.
  • The first circle is Limbo, which is reserved for people who turn a blind eye on scientific sins or reward others who engage in them (e.g., by giving them grants).
  • The second circle is Overselling, which is reserved for people who overstate  the importance of their work to get grants or write better papers.
  • The third circle is Post-hoc Storytelling- the scientist fires arrows at random; if a finding is noticed, a demon will explain at length or ramble that it aimed for this precise finding all along.
  • The fourth circle is P-value Fishing, which is reserved for those who “try every statistical test in the book” until they get a p-value of less than .05.
  • The fifth circle is Creative Use of Outliers, which is reserved for those who exclude “inconvenient” data points.
  • The sixth circle is Plagiarism- or presenting another individual’s work as their own work.
  • The seventh circle is the Non-publication of Data- scientists can free themselves from this circle if they write an article about it; however, the drawers containing these articles are locked.
  • The eighth circle is the Partial Publication of Data, where sinners are chased at random and prodded by demons, in analogy of the selective reporting and massaging of data.
  • The ninth circle is Inventing Data, which is reserved for Satan himself (i.e., people who make up their data).

Abstract

In the spirit of Dante Alighieri’s Inferno, this paper takes a humorous look at the fate that awaits scientists who sin against best practice.

APA Style Reference

Neuroskeptic. (2012). The nine circles of scientific hell. Perspectives on Psychological Science, 7(6), 643-644. https://doi.org/10.1177/1745691612459519

You may also be interested in




Attitudes Toward Open Science and Public Data Sharing: A Survey Among Members of the German Psychological Society (Abele-Brehm et al., 2019)

Main Takeaways:

  • Public data sharing is the only topic not discussed in open science. We should make data accessible for re-analyses in a secure, reliable and competently managed repository.
  • Although there is a positive attitude towards open science, some researchers argue whether data sharing will benefit the careers of early career researchers.
  • The present study investigated the attitude towards open science and public data sharing in general, as attitudes not only contribute to the research practice of the individual, but also the undergraduate student, the postgraduate student, the post-doctoral student, colleagues and the wider scientific community.
  • Method: 337 people were given scales and open-ended questions with 14 items that measured attitudes toward open science and public data sharing (e.g. what are the long-term consequences if a researcher shares raw data as part of a publication?).
  • Method: Attitudes toward open science were separated into hopes and fears.
  • Results: More hopes were related to open science and data sharing attitudes than fears. Hopes and fears were highest for ECRs, whereas for professors, hopes and fears were the lowest.
  • Attitudes towards open science and public data sharing were positive but there were fears that sharing data may have negative consequences for an individual’s career (e.g. data scooping).
  • Professors exhibited the least hopes and fears concerning the consequences of open science and  data sharing.

Quote

“This is, of course, true, but the idea of OS is transparency, and the question whether transparency and a higher commitment to data sharing and OS practices will eventually decrease QRPs and, thus, increase the robustness and replicability of psychological effects remains to be determined empirically.” (p.259).

Abstract

Central values of science are, among others, transparency, verifiability, replicability, and openness. The currently very prominent Open Science (OS) movement supports these values. Among its most important principles are open methodology (comprehensive and useful documentation of methods and materials used), open access to published research output, and open data (making collected data available for re-analyses). We here present a survey conducted among members of the German Psychological Society (N = 337), in which we applied a mixed-methods approach (quantitative and qualitative data) to assess attitudes toward OS in general and toward data sharing more specifically. Attitudes toward OS were distinguished into positive expectations (“hopes”) and negative expectations (“fears”). These were uncorrelated. There were generally more hopes associated with OS and data sharing than fears. Both hopes and fears were highest among early career researchers and lowest among professors. The analysis of the open answers revealed that generally positive attitudes toward data sharing (especially sharing of data related to a published article) are somewhat diminished by cost/benefit considerations. The results are discussed with respect to individual researchers’ behavior and with respect to structural changes in the research system.

APA Style Reference

Abele-Brehm, A. E., Gollwitzer, M., Steinberg, U., & Schönbrodt, F. D. (2019). Attitudes toward open science and public data sharing. Social Psychology, 50, 252-260. https://doi.org/10.1027/1864-9335/a000384

You may also be interested in




Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results (Wicherts et al., 2011)

Main Takeaways:

  • The American Psychological Association asks authors to sign a contract that data is available for individuals who wish to re-analyse the data to verify claims put forth in the paper. There has been no published research to assess this scenario in reality.
  • The present study examined the willingness to share data for re-analysis linked to strength of evidence and quality of reporting of statistical results.
  • Method: Wicherts et al. contacted corresponding authors of 141 papers published in the second half of 2004 in one of four high-ranking journals published by the American Psychological Association and determined whether the effects of outliers contributed to statistical outcomes.
  • Method: They included studies from the Journal of Personality and Social Psychology, and the Journal of Experimental Psychology: learning, memory and cognition, as authors were more willing to share data than other journals.
  • Method: They included tests results that were complete (i.e. test statistic, degrees of freedom, and p-value reported) and reported as significant effects.
  • Results: Higher p-values were more likely in papers from which no data were shared.
  • Conclusions: Reluctance to share was linked with weaker evidence and higher prevalence of apparent errors to report results. An unwillingness to share data was linked to reporting errors that affected statistical significance.
  • The authors seem to suggest that a reluctance to share data was linked to more errors in reporting of results and with weaker evidence. The unwillingness to share data was more pronounced when errors concerned significance.
  • Statistically rigorous researchers archive data better and are more attentive to statistical power than less statistically rigorous researchers.

Quote

“Best practices in conducting analyses and reporting statistical results involve, for instance, that all co-authors hold copies of the data, and that at least two of the authors independently run all the analyses (as we did in this study). Such double-checks and the possibility for others to independently verify results later should go a long way in dealing with human factors in the conduct of statistical analyses and the reporting of results” (pp.6-7).

Abstract

The widespread reluctance to share published research data is often hypothesized to be due to the authors’ fear that reanalysis may expose errors in their work or may produce conclusions that contradict their own. However, these hypotheses have not previously been studied systematically. We related the reluctance to share research data for reanalysis to 1148 statistically significant results reported in 49 papers published in two major psychology journals. We found the reluctance to share data to be associated with weaker evidence (against the null hypothesis of no effect) and a higher prevalence of apparent errors in the reporting of statistical results. The unwillingness to share data was particularly clear when reporting errors had a bearing on statistical significance.Our findings on the basis of psychological papers suggest that statistical results are particularly hard to verify when reanalysis is more likely to lead to contrasting conclusions. This highlights the importance of establishing mandatory data archiving policies.

APA Style Reference

Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PloS one, 6(11), e26828. https://doi.org/10.1371/journal.pone.0026828

You may also be interested in




Constraints on Generality (COG): A Proposed Addition to All Empirical Papers (Simons et al., 2017)

Main Takeaways:

  • When a paper identifies a target population and specifies constraints on generality (COG) of findings, researchers conduct direct replications that sample from the target population, leading to more appropriate tests of reliability of the original claim.
  • A COG statement indicates why the sample and target population is representative, justifying why subjects, materials, and procedures are representative of broader populations.
  • A COG statement does not limit the claim but leads the reader to correctly infer these findings are limited to the groups of populations being tested such as undergraduate students.
  • A COG statement establishes boundaries on generality rather than “replication failure.”
  • A COG statement inspires follow-up studies building on results by testing generality populations not originally tested.
  • A COG statement encourages reviewers and editors more receptive to next-step studies to test constraints identified.
  • A COG statement should be included in all papers, so editors support manuscripts with well-justified constraint on generality statements explicitly ground claims of generality.
  • Editors can evaluate whether claims are sufficiently important to justify publication.
  • A COG statement incentivises cumulative follow-up research, leading to greater reliability, influence and increased citations.
  • This COG statement values rigor, honesty, accuracy and supports the conclusion justified by evidence and theory, allowing readers to understand the limits of generalisability.
  • If science was more cumulative and self-correcting, broad generalisation might be justifiable.
  • A COG statement describes known or anticipated limits on finding and not mediation by unknown factors. It asks how our sample is representative of a broader population.

Abstract

Psychological scientists draw inferences about populations based on samples—of people, situations, and stimuli—from those populations. Yet, few papers identify their target populations, and even fewer justify how or why the tested samples are representative of broader populations. A cumulative science depends on accurately characterizing the generality of findings, but current publishing standards do not require authors to constrain their inferences, leaving readers to assume the broadest possible generalizations. We propose that the discussion section of all primary research articles specify Constraints on Generality (i.e., a “COG” statement) that identify and justify target populations for the reported findings. Explicitly defining the target populations will help other researchers to sample from the same populations when conducting a direct replication, and it could encourage follow-up studies that test the boundary conditions of the original finding. Universal adoption of COG statements would change publishing incentives to favor a more cumulative science.

APA Style Reference

Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017). Constraints on generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science, 12(6), 1123-1128.  https://doi.org/10.1177/1745691617708630 [ungated]

You may also be interested in




Most people are not WEIRD (Henrich et al., 2010)

Main Takeaways:

  • Much research on human behaviour is based on Western, Educated, Industrialized, Rich, and Democratic people (WEIRD).
  • They are the most unusual and psychological distinct individuals in the world.
  • Most research ignores the importance of generalizability and researchers tend to assume cognition and behaviours will be the same across all cultures.
  • However, across cultures, there are differences in terms of perceptual illusions, cultural biases and stereotypes.
  • There is a need for cross-cultural evidence in order to have a better understanding of cognition and behaviour.

Quote

“Recognizing the full extent of human diversity does not mean giving up on the quest to understand human nature. To the contrary, this recognition illuminates a journey into human nature that is more exciting, more complex, and ultimately more consequential than has previously been suspected” (p.29)

Abstract

To understand human psychology, behavioural scientists must stop doing most of their experiments on Westerners, argue Joseph Henrich, Steven J. Heine and Ara Norenzayan.

APA Style Reference

Henrich, J., Heine, S. & Norenzayan, A. (2010) Most people are not WEIRD. Nature 466, 29. https://doi.org/10.1038/466029a

You may also be interested in




Open Data in Qualitative Research (Chauvette et al., 2019)

Main Takeaways:

  • This article argues that as a result of epistemological, methodological, legal and ethical issues, not all qualitative data is appropriate for open access.
  • Open data allows researchers to test or refute new theories by validating research findings.
  • Although data is becoming more available, we need to consider hotly debated issues concerning open data and that not all data is created equally, especially data from qualitative research.
  • Qualitative research is not equally useful when decontextualized and requires contextualisation. Secondary analyses occur in teams or between collaborators when insider knowledge is shared.
  • Qualitative research design is not beneficial to secondary analysis. Researchers become part of the research and may bias the data. Also, preconceptions should not be removed from the analyses.
  • Personal knowledge is important for phenomenological research.
  • Open data is not captured in transcripts and participants may conduct research to become active contributors to the research process. Field notes are written by researchers.
  • Blanket consent forms have been recommended by some researchers in order to keep the participants’ data indefinitely and to make it potentially reusable by anyone.
  • However, confidentiality and anonymity becomes an issue for participants with open data, especially in small sample sizes.
  • In addition, there are other issues pertaining to open data such as sensitive issues, nature of questions and disclosure of information that may be harmful to the individual and researcher.

Quote

“Requirements for data access must consider the uniqueness and context of the data in each qualitative study. Consideration should be given to policies that grant the original research team adequate opportunities for involvement in publication of secondary analyses, perhaps with the rights to authorship to future publications if circumstances warrant. Alternatively, opportunities to comment on the new analysis and interpretation, considering the investigators’ understanding of the unique context of the study, would provide some additional accountability” (p.4).

Abstract

There is a growing movement for research data to be accessed, used, and shared by multiple stakeholders for various purposes. The changing technological landscape makes it possible to digitally store data, creating opportunity to both share and reuse data anywhere in the world for later use. This movement is growing rapidly and becoming widely accepted as publicly funded agencies are mandating that researchers open their research data for sharing and reuse. While there are numerous advantages to use of open data, such as facilitating accountability and transparency, not all data are created equally. Accordingly, reusing data in qualitative research present some epistemological, methodological, legal, and ethical issues that must be addressed in the movement toward open data. We examine some of these challenges and make a case that some qualitative research data should not be reused in secondary analysis.

APA Style Reference

Chauvette, A., Schick-Makaroff, K., & Molzahn, A. E. (2019). Open data in qualitative research. International Journal of Qualitative Methods, 18, 1609406918823863.  https://doi.org/10.1177/1609406918823863

You may also be interested in




How scientists can stop fooling themselves over statistics (Bishop, 2020b)

Main Takeaways:

  • Lab scientists should not be allowed to handle dangerous substances without safety training, researchers should not be allowed to be near a p value or similar measures of probability until researchers can demonstrate they understand their meaning.
  • Preconceived notions allow us to see a structure that is not there, whereas contradictory views provided from new data tend to be ignored.
  • People under-estimate how noisy small samples can be and conduct studies that lack the necessary power to detect an effect.
  • The more variables are investigated, the more likely a p value is to be significant due to type I error.
  • Basic statistical training is insufficient or counterproductive, providing misplaced confidence.
  • Simulated data allows students to discover how easy it is to find false results that are significant. Students learn with simulation that small sample sizes are useless to show a moderate difference.
  • Researchers need to build lifelong habits to avoid being led astray by specific confirmation bias.
  • It is easy to forget papers that counter our own instacts, albeit the papers had no flaws. Keeping tracks of these papers enables us to understand the blind spots and how to avoid them.

Abstract

Sampling simulated data can reveal common ways in which our cognitive biases mislead us.

APA Style Reference

Bishop, D. (2020). How scientists can stop fooling themselves over statistics. Nature, 584(7819), 9. https://doi.org/10.1038/d41586-020-02275-8

You may also be interested in




Publishing Research With Undergraduate Students via Replication Work: The Collaborative Replications and Education Project (CREP; Wagge et al., 2019)

Main Takeaways:

  • The Collaborative Replications and Education Project (CREP) allows undergraduates to participate in high-quality direct replication, using existing resources and by providing structure for research projects.
  • CREP samples seminal papers in 9 sub-disciplines published 3 years prior to the current year. Alumni students can then rate papers based on time and level of interest.
  • CREP teaches good scientific practices utilizing direct replications and open science methods.
  • CREP informs the original authors of the study selections and asks for materials and guidance for replication.
  • The skills acquired from CREP can be applied to non-academic careers. For instance, teaching students the ability to evaluate scientific claims.
  • CREP provides a forum and a community: for replication results to be presented and the institutionalization of replications, thereby contributing to science.
  • Students are invited to contribute to authorship, even if they do not involve lead authorship roles.
  • CREP deems that unaided, most student projects are not adequately powered for publication, thus do not lead to publication.
  • Working with CREP allows students to replicate/not replicate a seminal finding but also to provide them a publication.

Quote

“CREP offers a supportive entry point for faculty…new to open science and large-scale collaboration…helps with fidelity and quality checks…eliminates need for instructors to vet every hypothesis and design for student research projects…not be experts in a topic…do not need to learn new programs…documentable experience blending teaching, scholarship, and close mentoring.” (p. 4).

Abstract

The Collaborative Replications and Education Project (CREP; http://osf.io/wfc6u) is a framework for undergraduate students to participate in the production of high-quality direct replications. Staffed by volunteers (including the seven authors of this paper) and incorporated into coursework, CREP helps produce high-quality data using existing resources and provides structure for research projects from conceptualization to dissemination. Most notably, student research generated through CREP make an impact: data from these projects are available for meta-analyses, some of which are published with student authors.

APA Style Reference

Wagge, J. R., Brandt, M. J., Lazarevic, L. B., Legate, N., Christopherson, C., Wiggins, B., & Grahe, J. E. (2019). Publishing research with undergraduate students via replication work: The collaborative replications and education project. Frontiers in psychology, 10, 247.  https://doi.org/10.3389/fpsyg.2019.00247

You may also be interested in




CJEP Will Offer Open Science Badges (Pexman, 2017)

Main Takeaways:

  • This article introduces three badges: open data, open material, and pre-registration badges to the Canadian Journal of Experimental Psychology.
  • Open data badge: the data is digitally shareable and made publicly available to reproduce results.
  • Open materials badge: all materials that are necessary to reproduce reported results are digitally shareable with descriptions of non-digital materials being provided in order to replicate the study.
  • Pre-registration badge: researchers provide an analysis plan that includes a planned sample size, motivated research questions or hypotheses, outcome and predictor variables, including controls, covariates and independent variables. Results must be fully disclosed and distinguished from other results that were not pre-registered.
  • Pre-register + analysis: design a pre-register study with an analysis plan for research and the results are recorded according to the pre-registered plan.

Quote

“Indeed, in most cases, authors who wish to apply for badges will do so only after the editorial decision has been made. I understand that there are many reasons why it may not be possible to share data or materials, or to preregister a study, and so I certainly do not expect all authors to apply for badges. Nonetheless, I hope that many authors will devote the time required to make their data, materials, or research plans publicly available; these efforts are an important step toward improving our science.”  (p.1).

Abstract

This is a view on open science badges in the Canadian Journal of Psychology by Professor Penny Pexman. It describes the badges and their importance to open science. The badges are used as a mechanism to state that the author is following good research practices.

APA Style Reference

Pexman, P. M. (2017). CJEP will offer open science badges. Canadian Journal of Experimental Psychology= Revue Canadienne de Psychologie Experimentale, 71(1), 1-1. https://doi.org/10.1037/cep0000128

You may also be interested in




Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency (Kidwell et al., 2016)

Main Takeaways:

  • Researchers are not likely to share data and materials unless there are incentives such as badges. The inclusion of badges states that the journal values transparency and that the author has met the transparency standards for research by signalling to the reader that they have provided accessible data, materials, or pre-registered their study.
  • The present study investigated the influence of adopting badges by comparing data and material sharing rates before badges were adopted (i.e. 2012-2013) and after badges were adopted (2014-May 2015) in Psychological Science.
  • Method: “We used the population of empirical articles with studies based on experiment or observation (N = 2,478) published in 2012, 2013, 2014, and January through May 2015 issues of one journal that started awarding badges” (p.3).  Variables included were open data or open material badge, availability statement of data and material, and whether data or materials are available at a publicly accessible location.
  • Results: There was an increase in the reporting of open data after badges were introduced. However, reporting openness does not guarantee openness. When badges are earned, available data is provided, correct, usable and complete than when it was not earned.
  • Results: Open materials increased but not to the same extent.
  • Psychological science adopts badges, report sharing rates increases 10-fold to 40%. Without badges – a small percentage of reported sharing is a gross exaggeration of sharing.
  • Sharing data was larger when a badge was earned than when it was not earned.
  • Effects on sharing research materials were similar to sharing data but weaker with badges producing only three times more sharing.

Quote

“However, actual evidence suggests that this very simple intervention is sufficient to overcome some barriers to sharing data and materials. Badges signal a valued behavior, and the specifications for earning the badges offer simple guides for enacting that behavior. Moreover, the mere fact that the journal engages authors with the possibility of promoting transparency by earning a badge may spur authors to act on their scientific values. Whatever the mechanism, the present results suggest that offering badges can increase sharing by up to an order of magnitude or more. With high return coupled with comparatively little cost, risk, or bureaucratic requirements, what’s not to like?”  (p.13).

Abstract

Beginning January 2014, Psychological Science gave authors the opportunity to signal open data and materials if they qualified for badges that accompanied published articles. Before badges, less than 3% of Psychological Science articles reported open data. After badges, 23% reported open data, with an accelerating trend; 39% reported open data in the first half of 2015, an increase of more than an order of magnitude from baseline. There was no change over time in the low rates of data sharing among comparison journals. Moreover, reporting openness does not guarantee openness. When badges were earned, reportedly available data were more likely to be actually available, correct, usable, and complete than when badges were not earned. Open materials also increased to a weaker degree, and there was more variability among comparison journals. Badges are simple, effective signals to promote open practices and improve preservation of data and materials by using independent repositories.

APA Style Reference

Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L. S., ... & Errington, T. M. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS biology, 14(5), e1002456. https://doi.org/10.1371/journal.pbio.1002456

You may also be interested in




Signalling the trustworthiness of science should not be a substitute for direct action against research misconduct (Kornfeld & Titus, 2020)

Main Takeaways:

  • Truth is undermined by misconduct, fraud, failure to replicate, rise in the number of retractions, and the public media.
  • Fraudulent behaviour does not decrease the trust in science.
  • Fraudulent behaviour is a result of the fraudulent scientist, not untrustworthy science.
  • Reports indicate failure to publish will prevent academic appointment, tenure and ensuring funding of laboratories as the main concerns.
  • Educating the public about the high standards of science and scientists will not reduce the outrage concerning fraudulent research.

Quote

“When then will these leaders of the scientific community finally direct their talents and energy to the culprit per se, research misconduct, and its perpetrators”  (p.41).

Abstract

This is a response to the paper by Jamieson et al. (2019) on signalling trustworthiness in science. It contains information that the trust in science from the public and scientific community contributes to misconduct and fraudulent behaviour.

APA Style Reference

Kornfeld, D. S., & Titus, S. L. (2020). Signaling the trustworthiness of science should not be a substitute for direct action against research misconduct. Proceedings of the National Academy of Sciences of the United States of America, 117(1), 41. https://doi.org/10.1073/pnas.1917490116

You may also be interested in




Reply to Kornfeld and Titus: No distraction from misconduct (Jamieson et al., 2020)

Main Takeaways:

  • Funders should make research ethics a condition of support. Institutions should provide education and investigate misconduct fairly, quickly and transparently, while protecting whistle-blowers. Journals should act quickly to correct the record.
  • Scientists and outlets that publish their work should not only honor science’s integrity-protecting norms but also clearly signal when, and how, they have done so (e.g. statistical checks, plagiarism checks, badges, checklists).
  • These aforementioned methods should uncover and increase awareness of biases that undermine the ability to fairly interpret the authors’ findings.
  • “These indicators of trustworthiness clearly signal that the scientific community is safeguarding science’s norms and institutionalizing practices that protect its integrity as a way of knowing.” (p.42).

Abstract

This is a response to the commentary by Kornfeld and Titus (2020). It contains information about the importance of research ethics for funders, how institutions should protect whistleblowers and provide education to prevent misconduct and how scientists and outlets can provide evidence they honour scientific integrity.

APA Style Reference

Jamieson, K. H., McNutt, M., Kiermer, V., & Sever, R. (2020). Reply to Kornfeld and Titus: No distraction from misconduct. Proceedings of the National Academy of Sciences of the United States of America, 117(1), 42. https://doi.org/10.1073/pnas.1918001116

You may also be interested in




Stop ignoring misconduct (Kornfeld & Titus, 2016)

Main Takeaways:

  • The history of science shows irreproducibility is not a product of our times. These problems result from inadequate research practices and fraud. Current initiatives to improve science ignores fraudulent behaviour.
  • Reducing irreproducibility is a wasted opportunity, if dishonesty is not given much attention.
  • These unethical practices occurred long before people entered science. We need to consider reasons for misconduct: some researchers are perfectionists and are unable to cope with failure.
  • Funders should craft policies to ensure mentors are advisers, teachers, and role models, while limiting the number of trainees per mentor by discipline.
  • Established scientists are less likely to commit misconduct if they were more concerned about being detected and punished.
  • Whistle-blowers need to come forward and be protected. One method is to provide research integrity officers in the university who will protect them from retaliation.
  • Research funds should be given only when current certification about research integrity and honesty is provided by the institution. If misconduct occurs, institutions that fail to establish and execute these policies to assure integrity, will be held accountable.

Quote

“Government officials should be prepared to pursue repayments. The threat of such penalties should have a chilling effect on investigators contemplating research misconduct, and motivate institutions to establish and implement policies that reflect their commitment to institutional integrity.” (p.30)

Abstract

This is an editorial by Kornfeld and Titus (2016) who discusses that misconduct needs to be taken seriously and discussed. It contains solutions to resolve matters concerning research integrity for both the scientist and research institute.

APA Style Reference

Kornfeld, D. S., & Titus, S. L. (2016). Stop ignoring misconduct. Nature, 537(7618), 29-30.https://doi.org/10.1038/537029a

You may also be interested in




The Statistical Crisis in Science (Gelman & Loken, 2014)

Main Takeaways:

  • Authors argue that there is a statistical crisis in science because results are data-dependent, meaning that analytical decisions — a "garden of forking paths" — are so impactful that it can potentially produce different results depending on researchers' decisions.
  • Authors explain that p-value is short for probability value. It is the probability of obtaining an effect that is at least as extreme as the one you found in your sample - assuming the null hypothesis is true. Gelman and Loken define it as: “a way of measuring the extent to which a data set provides evidence against a so-called null hypothesis.”
  • Some ‘common’ practices (e.g., creating rules for data exclusion, for example) can flip analyses from non-significant to significant (and vice-versa). Such practices are particularly problematic when effect sizes or sample sizes are small, or when there are substantial measurement errors and variability.
  • The ‘garden of forking paths’, or researcher degrees of freedom, or data-dependent results, highlights that multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. This is because different choices about combining variables, inclusion and exclusion of cases, transformations of variables, tests for interactions in absence of main effects, and other steps, could occur with different data (depending on these decisions, which are often implicit, and unreported).
  • The Way Forward? Researchers should be made aware of choices involved in data analysis (pre-registration is practical but cannot be a general solution). Make a sharper distinction between exploratory and confirmatory data analysis, recognizing the benefits and limitations of each. Hence, researchers can perform two experiments: exploratory and confirmatory with its own pre-registered protocol. Authors also argue that statistically significant p-values shouldn’t be taken at face value even if linked to comparison consistent with existing theory. Researchers need to be aware of data dredging (the misuse of data analysis to find patterns in data that can be presented as statistically significant which dramatically increase the risk of false positives) and using both confidence intervals and p-values to avoid getting fooled by noise.
  • At the aggregate level, as the vast majority of papers are not published in high-impact journals without a significant p < .05 result (i.e., most journals, and the academic system in general, value ‘novel’ positive results rather than replications or pointing out mistakes in published literature), data-dependency results may be widespread. This is also known as perverse incentives.

Abstract

Data-dependent analysis— a "garden of forking paths" — explains why many statistically significant comparisons don't hold up.

APA Style Reference

Gelman, A., & Loken, E. (2014). The statistical crisis in science: data-dependent analysis--a" garden of forking paths"--explains why many statistically significant comparisons don't hold up. American scientist, 102(6), 460-466. [gated, ungated]

You may also be interested in




Only Human: Scientists, Systems, and Suspect Statistics A review of: Improving Scientific Practice: Dealing With The Human Factors, University of Amsterdam, Amsterdam, September 11, 2014 (Hardwicke et al., 2014)

Main Takeaways:

  • The h-index is used to evaluate scientists for hiring, promoting and funding decisions. This metric is affected by the number of publications, citations, and productivity. But papers can be cited in critiques, due to faults in methodology or failures to replicate. Does this mean citations are a good measure? No! (cf. Goodhart’s Law).
  • Academia provides short-term contracts to exploit without wasting resources. There is fierce competition for limited funding.
  • Publications aim for newsworthy results, leading to false positives and less integration with the literature. The pressure for positive findings can lead to unethical behaviour.
  • Questionable research practices are seen as unethical as it distorts data to support the researchers’ hypotheses, either intentionally or unintentionally. There is too much faith that scientists will self-correct.
  • Scientists need to be open about their results. Many scientists subscribe to the norm of communality (common ownership of scientific results and methods).
  • There is some data sharing but, most scientists don’t. Scientists are assumed to self-regulate, but this assumption is erroneous.
  • Incentives need to change and focus on quality, reproducibility, data sharing, and impact on society.
  • Pre-registration can help with publication biases and questionable research practice. A study should be published irrespective of findings.
  • Criticism of pre-registration is that workload will increase; evaluation of methodology and data collection to evaluate adherence to pre-registration plan. It is argued that pre-registration would save time in preventing manuscripts being rejected based on methodological issues or null results.
  • Pre-registration could backfire, as editors may require revisions to protocols, study is complete and changes may be impossible.
  • Pre-registration may address integrity issues before and during data collection.
  • Need to change the culture so scientists don’t need to prioritise their own research over scientific inquiry or credibility.

Quote

“The success of science is often attributed to its objectivity: surely science is an impartial, transparent, and dispassionate method for obtaining the truth? In fact, there is growing concern that several aspects of typical scientific practice conflict with these principles and that the integrity of the scientific enterprise has been deeply compromised.” (p.1)

Abstract

It is becoming increasingly clear that science has sailed into troubled waters. Recent revelations about cases of serious research fraud and widespread ‘questionable research practices’ have initiated a period of critical self-reflection in the scientific community and there is growing concern that several common research practices fall far short of the principles of robust scientific inquiry. At a recent symposium, ‘Improving Scientific Practice: Dealing with the Human Factors’ held at The University of Amsterdam, the notion of the objective, infallible, and dispassionate scientist was firmly challenged. The symposium was guided by the acknowledgement that scientists are only human, and thus subject to the desires, needs, biases, and limitations inherent to the human condition. In this article, five post-graduate students from University College London describe the issues addressed at the symposium and evaluate proposed solutions to the scientific integrity crisis.

APA Style Reference

Hardwicke, T E et al 2014 Only Human: Scientists, Systems, and Suspect

You may also be interested in




False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant (Simmons et al., 2011)

Main Takeaways:

  • Researchers are faced with many different decisions when analysing their data, such as whether to test more participants, how to exclude outliers, whether to use covariates or not. These “researcher degrees of freedom” can be exploited to run different variations of the analysis until a statistically significant result is found.
  • The combination of different researcher degrees of freedom makes it increasingly more likely to find a false positive result (i.e., a statistical fluke).
  • The authors use two real experiments and computer simulations to show how undisclosed flexibility in data analysis and the selective reporting of results makes it “unacceptably easy” to find significant results, even for hypotheses that are unlikely, or necessarily incorrect.
  • It is recommended that authors should: 1) determine data collection rules in advance of running the experiment; 2) collect at least 20 observations per cell; 3) report all variables and experimental conditions (including failed manipulations); and 4) if outliers are removed or covariates are included, authors should show how these actions change the results.
  • Reviewers should: 1) ensure that the rules for authors above are followed; 2) be tolerant towards imperfections of the study; 3) ask authors to show that the results don’t depend on specific analytical decisions; and 4) ask for direct replications when the authors’ justification is not compelling enough.
  • While other solutions are possible, such as correcting the alpha level, using Bayesian statistics, doing conceptual replications, and posting data and materials online, the authors consider them to be less practical and effective.

Abstract

In this article, we accomplish two things. First, we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.

APA Style Reference

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science, 22(11), 1359-1366. https://doi.org/10.1177/0956797611417632 [ungated]

You may also be interested in




Publication Prejudices: An Experimental Study of Confirmatory Bias in the Peer Review System (Mahoney, 1977)

Main Takeaways:

  • Cognitive bias (e.g. confirmation bias) is more prevalent in scientific publication. A piece of research is threatened by human decision making (i.e. the journal editor and reviewer).
  • The present study investigated whether confirmation bias is a problem for current review practices and how we can reduce confirmatory bias
  • To what extent do editors and referees weigh various components in evaluation? The ideal publication review system should focus on methodological quality and relevance, over data outcome and interpretation. Writing styles and conclusions can impact the decision made by the editor and peer reviewer.
  • Method: five groups of referees read manuscripts that had data that was consistent or inconsistent with the reviewer’s theoretical perspective.
  • Method con’t: Reviewers had to evaluate manuscripts based on relevance and methodology.
  • Method con’t: two final groups of reviewers received mixed findings, supporting one perspective of the reviewer and one was contradictory to the reviewer’s perspective.
  • Results: There was poor inter-rater reliability.  Reviewers were more likely to show confirmation bias for manuscripts in favour of their theoretical perspective and were more severe for manuscripts that contradict their perspective.
  • Referees should be asked to evaluate relevance and methodology of an experiment without seeing its results or interpretations (cf. registered reports).
  • Referees show little agreement on topics, thus they should be trained to produce better and unprejudiced consensus.
  • Peer review is perceived as an objective measure but ironically is prone to human biases.

Abstract

Confirmatory bias is the tendency to emphasize and believe experiences which support one's views and to ignore or discredit those which do not. The effects of this tendency have been repeatedly documented in clinical research. However, its ramifications for the behavior of scientists have yet to be adequately explored. For example, although publication is a critical element in determining the contribution and impact of scientific findings~ little research attention has been devoted to the variables operative in journal review policies. In the present study, 75 journal reviewers were asked to referee manuscripts which described identical experimental procedures but which reported positive, negative, mixed, or no results. In addition to showing poor interrater agreement, reviewers were strongly biased against manuscripts which reported results contrary to their theoretical perspective. The implications of these findings for epistemology and the peer review system are briefly addressed.

APA Style Reference

Mahoney, M. J. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive therapy and research, 1(2), 161-175. https://doi.org/10.1007/BF01173636

You may also be interested in




Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial (van Rooyen et al., 1999)

Main Takeaways:

  • The British Medical Journal wants to improve peer review. There was no evidence that investigated whether anonymous peer review is better than other forms of peer review.
  • Open peer review (i.e. reviewer signing their review) was argued to lead to better reviews, thus increasing credibility and accountability.
  • The article investigated whether the quality of reviews in open review was the same as traditional review.
  • Method: When both reviews were received, the reviews and the manuscript were passed to a responsible editor who was asked to rate the quality of reviews, using a validated review quality instrument.
  • A second editor was randomly chosen from the other 12 editors to measure review quality independently.
  • Method con’t: The corresponding author of each manuscript was sent anonymous copies of the two reviews and was told a decision on the manuscript had not been reached.
  • Method con’t: The corresponding author was asked to assess the quality of the review, using a review quality instrument.
  • Results: Twelve percent of reviewers were more likely to decline to review, if they were to be identified, as opposed to being anonymous.
  • Results: There was no difference between anonymous and identified reviewers in terms of quality of reviews, in recommendation of reviewers, or time taken to review the papers.
  • Results: The editors’ quality scores for reviews was higher than that of the authors.
  • There was no difference observed in terms of the quality of and time taken to produce the review of the manuscript.
  • Authors rated reviews that recommended the publication of the manuscript higher than those reviews that recommended rejection of the manuscript.
  • “Editors...did not seem to be influenced by a reviewer's opinion of the merit of a paper when they assessed the quality of the review” (p.26)

Abstract

To examine the effect on peer review of asking reviewers to have their identity revealed to the authors of the paper. Randomised trial. Consecutive eligible papers were sent to two reviewers who were randomised to have their identity revealed to the authors or to remain anonymous. Editors and authors were blind to the intervention. The quality of the reviews was independently rated by two editors and the corresponding author using a validated instrument. Additional outcomes were the time taken to complete the review and the recommendation regarding publication. A questionnaire survey was undertaken of the authors of a cohort of manuscripts submitted for publication to find out their views on open peer review. Two editors' assessments were obtained for 113 out of 125 manuscripts, and the corresponding author's assessment was obtained for 105. Reviewers randomised to be asked to be identified were 12% (95% confidence interval 0.2% to 24%) more likely to decline to review than reviewers randomised to remain anonymous (35% v 23%). There was no significant difference in quality (scored on a scale of 1 to 5) between anonymous reviewers (3.06 (SD 0.72)) and identified reviewers (3.09 (0.68)) (P = 0.68, 95% confidence interval for difference - 0.19 to 0.12), and no significant difference in the recommendation regarding publication or time taken to review the paper. The editors' quality score for reviews (3.05 (SD 0.70)) was significantly higher than that of authors (2.90 (0.87)) (P < 0.005, 95%confidence interval for difference - 0.26 to - 0.03). Most authors were in favour of open peer review. Asking reviewers to consent to being identified to the author had no important effect on the quality of the review, the recommendation regarding publication, or the time taken to review, but it significantly increased the likelihood of reviewers declining to review.

APA Style Reference

Van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999). Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial. Bmj, 318(7175), 23-27. https://doi.org/10.1136/bmj.318.7175.23

You may also be interested in




Early co-authorship with top scientist predicts success in academic careers (Li et al., 2019)

Main Takeaways:

  • Academic impact is complex and linked to citation number. The output of junior researchers can gain a competitive advantage based on visibility.
  • The present study asks whether a single event of interaction with ‘top scientists’ may alter junior researchers futures in academia.
  • Hypothesis: the more co-authorship with ‘top scientists’, the more junior researchers have competitive advantage.
  • Method: publication and citation data for four disciplines was indexed, since 1970 of selected journals for specific authors and institutions.
  • A paper’s prestige score is: the average prestige score of its authors’ institution (+) the average prestige score of the researchers’ papers.
  • Results: co-author with top scientists provide competitive advantage compared to peers of comparable early career profiles without top co-authors. Authors seem to suggest that students from less prestigious institutions would benefit junior researchers most.
  • Discussion:

Abstract

We examined the long-term impact of coauthorship with established, highly-cited scientists on the careers of junior researchers in four scientific disciplines. Here, using matched pair analysis, we find that junior researchers who coauthor work with top scientists enjoy a persistent competitive advantage throughout the rest of their careers, compared to peers with similar early career profiles but without top coauthors. Such early coauthorship predicts a higher probability of repeatedly coauthoring work with top-cited scientists, and, ultimately, a higher probability of becoming one. Junior researchers affiliated with less prestigious institutions show the most benefits from coauthorship with a top scientist. As a consequence, we argue that such institutions may hold vast amounts of untapped potential, which may be realised by improving access to top scientists.

APA Style Reference

Li, W., Aste, T., Caccioli, F., & Livan, G. (2019). Early coauthorship with top scientists predicts success in academic careers. Nature communications, 10(1), 1-9. https://doi.org/10.1038/s41467-019-13130-4 [ungated]

You may also be interested in




The Peer Reviewers’ Openness Initiative: incentivising open research practices through peer review (Morey et al., 2016)

Main Takeaways:

  • Openness and transparency is crucial to science. “Scientific progress is accelerated as more data are available for verification, theory-building and meta-analysis, and experimental materials are available for easier replications and extension studies.” (p.2)
  • Openness is an ethical obligation that provides further advantages and is being introduced as a policy change. It is not difficult to learn to be open, but implementing them may delay publications by a few days.
  • The relationship between reviewers and authors is important for the scientific process, especially when there is a missing figure or statistical results which requires the author to clarify.
  • The author must justify to the reviewer if the following is not included: link to the data, materials, a document with details on how to interpret any files, code, or explanation of how to run the software in the manuscript. If no real reason is given (e.g. legal, ethical or impracticality), reviewers should provide a short review for the lack of openness and failure to justify.
  • This Peer Reviewers’ Openness Initiative will ease the review load, as the reviewer can reject the manuscript, if the openness requirement is not met.
  • Open research is not a matter of policy, but a matter of scientific value and quality of product.
  • Open practices are not standardised and are driven by practice. Authors that lack training in open practices and scientists need to learn new skills and knowledge.
  • Senior researchers can help students curate data and research materials. Open data allows the reviewer the option to check the analysis.
  • The initiative is targeted at reviewers, not action editors. Researchers who value open research practices should join the Initiative to help promote open research.

Quote

“As a group, reviewers share the power to ensure that articles meet minimum scientific quality standards.What is needed is an affirmation that those minimum scientific quality standards include open practices. By acknowledging that open practices should be considered by reviewers alongside other research norms, reviewers can collectively bring about a radical positive change in the culture of science.” (p.3)

Abstract

Openness is one of the central values of science. Open scientific practices such as sharing data, materials and analysis scripts alongside published articles have many benefits, including easier replication and extension studies, increased availability of data for theory-building and metaanalysis, and increased possibility of review and collaboration even after a paper has been published. Although modern information technology makes sharing easier than ever before, uptake of open practices had been slow. We suggest this might be in part due to a social dilemma arising from misaligned incentives and propose a specific, concrete mechanism—reviewers withholding comprehensive review—to achieve the goal of creating the expectation of open practices as a matter of scientific principle.

APA Style Reference

Morey, R. D., Chambers, C. D., Etchells, P. J., Harris, C. R., Hoekstra, R., Lakens, D., ... & Vanpaemel, W. (2016). The Peer Reviewers' Openness Initiative: incentivizing open research practices through peer review. Royal Society Open Science, 3(1), 150547. https://doi.org/10.1098/rsos.150547

You may also be interested in




A 21 Word Solution (Simmons et al., 2012)◈

Main Takeaways:

  • The authors suggest a “21-word solution” to encourage researchers to disclose their analytical choices. If researchers did not engage in questionable research practices (e.g., dropping conditions/ variables, p-hacking, optional data stopping), they should explicitly say so in their paper.
  • Researchers should say what their sample size was in advance, be transparent and disclose information that they did not drop any variables or conditions.
  • We cannot trust our colleagues to run and report studies properly if some people believe it is okay to drop conditions and variables and others do not believe this is good scientific practice.
  • Researchers shouldn’t wait for journals and other colleagues to catch up if they want to achieve transparency in science. Rather, they should take the initiative themselves.
  • Scientific journals should ask researchers to disclose data collection and analysis decisions truthfully, but this doesn’t mean that they are responsible for policing researchers.
  • The 21-word solution can be easily included in your manuscript, even if this is done in the Supplemental materials to reduce word count. The ‘red tape’ of this transparency statement is arguably negligible compared to the thousands of idiosyncratic rules in APA’s Publication Manual.
  • False positives need to be scrutinised, as many p-hacking choices are encouraged.
  • Reviewers and the readers should ask if this study is a 1 or 2 dependent variable study.
  • Disclosure does not reduce p-hacking and does not reduce probability of false positives.
  • Papers should include this proposed 21 words to improve its credibility.

Quote

“We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study.” (p.1)

Abstract

APA Style Reference

Simmons, Joseph P. and Nelson, Leif D. and Simonsohn, Uri, A 21 Word Solution (October 14, 2012). Available at SSRN: https://ssrn.com/abstract=2160588 or http://dx.doi.org/10.2139/ssrn.2160588

You may also be interested in




Quality Uncertainty Erodes Trust in Science (Vazire, 2017)

Main Takeaways:

  • Quality uncertainty threatens the confidence people have in the findings and to build on them. Vazire argues that the lack of transparency in science has led to quality uncertainty and threatened to erode trust in science. They argue that greater transparency in scientific reporting will reduce quality uncertainty and restore trust in science.
  • In the scientific market, the source of quality uncertainty is that the authors know much more about what went into the articles than the potential consumers (e.g. the raw data, the original design and analysis plan, the exploratory analyses  and final analysis). Put simply, the more information that is hidden, the larger the quality uncertainty.
  • As a result of low levels of transparency in scientific publication, journal editors, reviewers and readers cannot differentiate between lemons and high-quality findings. Some, but not all, the information is presented in the method and results section. However, vital information is kept private (e.g. the raw data, the original design and analysis plan, the exploratory analyses  and final analysis), preventing consumers of their manuscript being certain about its quality.
  • The cost of lack of transparency will end up in building a science based on low-quality finding and shaky foundations, thus driving out rigorous science. When the low-quality findings do not stand up, it is too late, as high-quality research has been driven out.
  • The motto of the Royal Society is to “take no one’s word”, as we cannot rely on a few experts to evaluate the findings of specific research and then ask everyone to completely trust the author. To reduce quality uncertainty, we must be transparent in order to make a judgment about the quality of the manuscript.
  • Increased transparency will provide the consumers the information needed to detect many errors in the article, while making authors more accountable for their mistakes, thus encouraging further care for how their study is designed, analysed and written. This will not help consumers and researchers catch researchers who are willing to use fraudulent behaviours, but it will solve unintentional misrepresentations, thus rebuilding trust in science.
  • When journals choose to maximise citation impacts, instead of producing reliable, robust and reproducible science, the consumer is being given shoddy, as opposed to reliable product, thus making journals neglect their duties. To make journals more accountable, we must tie their reputation not to impact factor, but the quality of their articles instead and their policies on open science and transparency.

Quote

“In any market, consumers must evaluate the quality of products and decide their willingness to pay based on their evaluation. In science, consumers of new scientific findings must likewise evaluate the strength of the findings and decide their willingness to put stock in them. In both kinds of markets, the inability to make informed and accurate evaluations of quality (i.e., quality uncertainty) leads to a lower and lower willingness to put stock in any product – a lack of trust in the market itself. When there are asymmetries in the information that the seller and the buyer have, the buyers cannot be certain about the quality of the products, leading to quality uncertainty.” (p.1).

Abstract

When consumers of science (readers and reviewers) lack relevant details about the study design, data, and analyses, they cannot adequately evaluate the strength of a scientific study. Lack of transparency is common in science, and is encouraged by journals that place more emphasis on the aesthetic appeal of a manuscript than the robustness of its scientific claims. In doing this, journals are implicitly encouraging authors to do whatever it takes to obtain eye-catching results. To achieve this, researchers can use common research practices that beautify results at the expense of the robustness of those results (e.g., p-hacking). The problem is not engaging in these practices, but failing to disclose them. A car whose carburetor is duct-taped to the rest of the car might work perfectly fine, but the buyer has a right to know about the duct-taping. Without high levels of transparency in scientific publications, consumers of scientific manuscripts are in a similar position as buyers of used cars – they cannot reliably tell the difference between lemons and high quality findings. This phenomenon – quality uncertainty – has been shown to erode trust in economic markets, such as the used car market. The same problem threatens to erode trust in science. The solution is to increase transparency and give consumers of scientific research the information they need to accurately evaluate research. Transparency would also encourage researchers to be more careful in how they conduct their studies and write up their results. To make this happen, we must tie journals’ reputations to their practices regarding transparency. Reviewers hold a great deal of power to make this happen, by demanding the transparency needed to rigorously evaluate scientific manuscripts. The public expects transparency from science, and appropriately so – we should be held to a higher standard than used car salespeople.

APA Style Reference

Vazire, S. (2017). Quality Uncertainty Erodes Trust in Science. Collabra: Psychology, 3(1), 1. DOI: http://doi.org/10.1525/collabra.74

You may also be interested in




Rein in the four horsemen of irreproducibility (Bishop, 2019)

Main Takeaways:

  • Publication bias, low statistical power, p-hacking, and HARKing (hypothesising after the results are known) are threats to research reproducibility that make it difficult to find meaningful results.
  • Publication bias harms patients. The tendency to not publish negative results misleads readers and biases meta-analyses.
  • Low statistical power also misleads readers- when the sample size is small, there is a low probability that one will detect an effect even if it exists.
  • Time and resources are wasted on such underpowered studies.
  • P-hacking occurs when researchers conduct many analyses, but report only those that are significant.
  • HARKing is so wide-spread, that researchers may come to accept it as a good practice. Authors should be free to do exploratory analyses, but not when p-values are used outside of the context that was used to calculate them.
  • These four problems are older than most of the junior researchers working on them. New developments may help combat these issues:

Abstract

Dorothy Bishop describes how threats to reproducibility, recognized but unaddressed for decades, might finally be brought under control.

APA Style Reference

Bishop, D. (2019). Rein in the four horsemen of irreproducibility. Nature, 568(7753), 435-436. http://doi.org/10.1038/d41586-019-01307-2

You may also be interested in




Seven Easy Steps to Open Science: An Annotated Reading List (Crüwell et al., 2019)

Main Takeaways:

  • Students and academics with little knowledge of open science may not easily find and make use of resources.
  • Transparency and robustness may not guarantee increased rigour.
  • Researchers should plan data collection and analysis, be aware of assumptions of statistical models and understanding of statistical tools.
  • Credibility of scientific claims depend on replicability.
  • Open access removes barriers to access and distributes research.
  • The gold route refers to publicly available articles, while the green route relates to self-archiving or the works are made publicly available by people who created the manuscripts (e.g. preprints).
  • Open access articles are cited between 36%-600% more than non-open access work. It is given more coverage and discussed more in non-scientific settings.
  • Researchers need to consider how they share their data. Is it findable, accessible, interoperable and reusable (FAIR)?
  • All steps of data analysis should be recorded in open source programs (e.g. R or Python) or placed in a reproducible syntax file.
  • Pre-registration is an open science practice that: protects people from biases; encourages transparency about analytic decision-making; supports rigorous scientific research; enables more replicable and reproducible work.
  • Open science increases confidence and replicability of scientific results.
  • Direct replication duplicates the necessary elements in order to assess whether the original findings are reproducible, whereas conceptual replication changes one component of the original procedure such as sample or measure to measure whether the original results are reproducible.

Quote

“We hope that this paper will provide researchers interested in open science an accessible entry point to the practices most applicable to their needs. For all of the steps presented in this annotated reading list, any time taken by researchers to understand the issues and develop better practices will be rewarded in orders of magnitude. On an individual level, time and effort are ultimately saved, errors are reduced, and one’s own research is improved through a greater adherence to openness and transparency. On a field-wide level, the more researchers invest in adopting these practices, the closer the field will come toward adhering to scientific norms and the values it claims to espouse.” (p.245)

Abstract

The open science movement is rapidly changing the scientific landscape. Because exact definitions are often lacking and reforms are constantly evolving, accessible guides to open science are needed. This paper provides an introduction to open science and related reforms in the form of an annotated reading list of seven peer-reviewed articles, following the format of Etz, Gronau, Dablander, Edelsbrunner, and Baribault (2018). Written for researchers and students – particularly in psychological science – it highlights and introduces seven topics: understanding open science; open access; open data, materials, and code; reproducible analyses; preregistration and registered reports; replication research; and teaching open science. For each topic, we provide a detailed summary of one particularly informative and actionable article and suggest several further resources. Supporting a broader understanding of open science issues, this overview should enable researchers to engage with, improve, and implement current open, transparent, reproducible, replicable, and cumulative scientific practices

APA Style Reference

Crüwell, S., van Doorn, J., Etz, A., Makel, M. C., Moshontz, H., Niebaum, J. C., ... & Schulte-Mecklenbeck, M. (2019). Seven Easy Steps to Open Science. Zeitschrift für Psychologie. https://doi.org/10.1027/2151-2604/a000387

You may also be interested in




Many hands make tight work (Silberzahn & Uhlmann, 2015)

Main Takeaways:

  • It is argued that re-running the analysis produces the same outcome. An analysis run by single team-researchers takes on several roles: inventor: creates ideas and hypotheses; analysts: scrutinise data to support hypotheses; devil’s advocate: use different approaches to show weaknesses in the findings.
  • A crowdsourcing approach can be a useful addition to research. Several teams work with the same dataset, where the hypotheses and results are held closed.
  • All researchers discuss results via email exchanges and researchers add notes to their individual reports in light of  others’ work; to express doubt or confidence about their approach. Teams present findings in a draft manuscript, in which the participants are invited to comment and modify.
  • Researchers should not take any single analysis too seriously, as different analyses can produce a broad range of effect sizes.
  • Crowdsourcing analyses will not be the optimal solution for several research problems e.g. resource intensive.
  • Decision making should be made with care regarding: which hypothesis to test; data collection; and which variables to collect. Researchers will disagree about findings, making it difficult to present a manuscript with a clear conclusion.
  • Crowdsourcing research can allow us to evaluate whether analytical approaches and decisions drive findings. This would allow us to discuss the analytical approaches before we commit to a specific strategy.
  • Crowdsourcing reduces the incentives for novel and groundbreaking results and can reveal several scientific possibilities.

Quote

“Under the current system, strong storylines win out over messy results. Worse, once a finding has been published in a journal, it becomes difficult to challenge. Ideas become entrenched too quickly, and uprooting them is more disruptive than it ought to be. The crowdsourcing approach gives space to dissenting opinions. Scientists around the world are hungry for more-reliable ways to discover knowledge and eager to forge new kinds of collaborations to do so.” (p.191).

Abstract

Crowdsourcing research can balance discussions, validate findings and better inform policy, say Raphael Silberzahn and Eric L. Uhlmann.

APA Style Reference

Silberzahn, R., & Uhlmann, E. L. (2015). Crowdsourced research: Many hands make tight work. Nature News, 526(7572), 189. https://doi.org/10.1038/526189a

You may also be interested in




A user’s guide to inflated and manipulated impact factor (Ioannidis & Thombs, 2019)

Main Takeaways:

  • A widely misused metric is impact factor: reflecting the importance of publications in a specific journal.
  • The promotion and funding of an individual depends on the impact factor (cf. Goodhart’s Law). Based on the Declaration on Research Assessment, over 14000 researchers agree: let’s remove impact factor as the measure of an individual article’s quality.
  • There is a belief that a higher impact factor leads to more and better articles being submitted and published. If this is the case, a journal’s ratings may improve, as its impact factor increases.
  • Volume of submissions may increase, as many scientists naively decide where to send their paper based on journal impact factor. Volume may dissociate from quality.
  • An inappropriate use of impact factor is unlikely to stop (e.g. self citations and including citations to other recent articles without justification), especially with a large number of papers being cited without being counted.
  • In addition, certain manuscripts (e.g. review articles and papers with questionable scientific value) will get more citations than others research articles.
  • Papers should be submitted to target journals based on the quality, scientific rigour, and the relevance of the journal, not impact factor.

Quote

“Authors should pick target journals based on relevance and scientific rigour and quality, not spurious impact factors. Inspecting inflation measures is more informative for choosing a journal than JIF, because prominent inflation may herald spurious editorial practices and thus poor quality. Authors who submit to journals with high‐impact inflation may become members of bubbles. They even run the risk of having their work published in journals that are eventually formally discredited if Clarivate decides to make a more serious effort to curtail spurious gaming.” (p.5).

Abstract

This is a view on impact factor by Professor John P.A. Ioannidis and Dr Brett D. Thombs. It contains a discussion of the impact factor being misused, how it is misused by journals and reviewers but provides solutions to overcome the use of this metric. In addition, we should base journals not on the impact factor but the relevance, scientific rigour and quality of the journal.

APA Style Reference

Ioannidis, J. P., & Thombs, B. D. (2019). A user’s guide to inflated and manipulated impact factors. European journal of clinical investigation, 49(9), e13151. https://doi.org/10.1111/eci.13151

You may also be interested in




Promoting an open research culture (Nosek et al., 2015)

Main Takeaways:

  • The incentive system focuses on innovation, as opposed to replication, openness, transparency, and reproducibility.
  • There is no means to align individual and communal incentives via universal scientific policies and procedures.
  • We should reward researchers for the time and effort spent in open practices.
  • Citations should also extend to data, code, and research materials. Regular and rigorous citation of these materials should be cited as an original intellectual contribution.
  • Reproducibility increases confidence in the results and allows scholars to learn more about data interpretation.
  • The transparency guidelines are used to improve explicitness about the research process, while reducing vague or incomplete reporting of methodology.
  • Pre-registration of studies facilitates the discovery of research, allowing the study to be recorded in a public registry.
  • Four levels are used to encourage open science policy:
  • Level 1 is designed to have no barrier or incentive to adopting open science practices (e.g. code sharing). This reduces the effort on the efficiency and workflow of the journal.
  • Level 2 has stronger authorial expectations than Level 1. It avoids adding resource cost to editors or publishers who adopt this standard. In Level 2, journals would require codes to be deposited in a trusted repository (e.g. osf), also reviewers would need to check the link appears in the manuscript and access the code in the repository.
  • Level 3 is the strongest standard but provides some barriers to implementations in the journal. For instance, authors must provide their code for the review process and editors must be able to reproduce the reported analyses publication.
  • These higher level guidelines should reduce the time spent on communication with the authors and reviewers, improve standards of reporting, increase detectability of errors prior to publication and ensure that publication-related data is accessible for a long time.

Quote

“The journal article is central to the research communication process. Guidelines for authors define what aspects of the research process should be made available to the community to evaluate, critique, reuse, and extend. Scientists recognize the value of transparency, openness, and reproducibility. Improvement of journal policies can help those values become more evident in daily practice and ultimately improve the public trust in science, and science itself.” (p.1425).

Abstract

Author guidelines for journals could help to promote transparency, openness, and reproducibility.

APA Style Reference

Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., ... & Contestabile, M. (2015). Promoting an open research culture. Science, 348(6242), 1422-1425. http://doi.org/10.1126/science.aab2374

You may also be interested in




Promoting Transparency in Social Science Research (Miguel et al., 2014)

Main Takeaways:

  • The incentives, norms, institutions, and a dysfunctional reward structure make it difficult to improve research design in the social sciences.
  • Since social science journals do not instruct adherence to reporting standards (e.g. data sharing), researchers feel motivated to analyse and present data in a more publishable way, e.g. by selecting a subset of positive results.
  • Such practices result in a distorted body of evidence with too few null results having direct consequences on policies and eventually citizens.
  • This article surveys recent progress towards research transparency in the social sciences and provides standards and rules to realign scholarly incentives with good scientific practices based on three pillars: Disclosure, Preregistration, and Open data and materials.
  • Disclosure is about the systematic reporting of all measures, manipulations, data exclusions, and sample sizes.
  • Preregistration helps to reduce bias and increase credibility by pre-specifying, e.g.,  statistical models, dependent variables, and covariates.
  • Open data and materials allows researchers to test alternative approaches on the data, reproduce results, identify misreported or fraudulent results; reuse or adapt materials for replication or their own research.
  • Limitation: One might argue that preregistration counteracts exploratory analysis. Counterargument: Preregistration should just free an analysis from being reported as formal hypothesis testing.
  • Further work is needed, e.g., it is unclear how to preregister studies based on existing data which is a common approach in the social sciences.

Quote

“Scientific inquiry requires imaginative exploration. Many important findings originated as unexpected discoveries. But findings from such inductive analysis are necessarily more tentative because of the greater flexibility of methods and tests and, hence, the greater opportunity for the outcome to obtain by chance. The purpose of prespecification is not to disparage exploratory analysis but to free it from the tradition of being portrayed as formal hypothesis testing. New practices need to be implemented in a way that does not stifle creativity or create excess burden. Yet we believe that such concerns are outweighed by the benefits that a shift in transparency norms will have for overall scientific progress, the credibility of the social science research enterprise, and the quality of evidence that we as a community provide to policy-makers” (p.31).

Abstract

Social scientists should adopt higher transparency standards to improve the quality and credibility of research.

APA Style Reference

Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., ... & Laitin, D. (2014). Promoting transparency in social science research. Science, 343(6166), 30-31. http://doi.org/10.1126/science.1245317

You may also be interested in




Rebuilding Ivory Tower: A bottom-up experiment in aligning research with societal needs (Hart & Silka, 2020) ◈

Main Takeaways:

  • Scientists are trained to conduct good science, develop interesting research questions, be impartial to data, sceptical about conclusions and open to criticisms from our peers.
  • We are taught good science is a reward in itself for improving our world.
  • We need strong collaborations with diverse stakeholders in the public and private sectors, non-governmental organisations and civil society in order to identify and solve (sustainability/wicked) problems.
  • “So it turned out that social scientists as well as natural scientists had keen interest in a project aimed at bringing together their expertise and forming bonds with individuals and groups outside academia to solve local problems...also to identify best practices for interdisciplinarity and stakeholder engagement”. (pp. 80-81).
  • A shared culture, comprising a common set of beliefs and values and supported by organizational strategy and structure, is needed to streamline the commitment to excellence innate to many academics.
  • We try to create an atmosphere of learning from successes and failures. There is no sure-fire formula to match research with societal needs.
  • Older faculty are retiring but are being replaced by younger students who are able to move the initiative forward as a result of their skills to be interdisciplinary researchers.
  • Universities should use bottom-up (inner interest of academics to improve the world) and top-down (university programs; ideas from senior leaders) strategies to become more useful partners to society.
  • Put simply, “Although no single recipe will work in all contexts, it is our hope that the ingredients we’ve identified may prove useful to other universities in their own quests to help solve society’s greatest problems". (p.85).

Quote

“Two fundamental commitments [have emerged]: 1) In addition to the traditional focus on the biophysical components underpinning a problem, a much greater emphasis is needed on the human dimensions, including the complex interactions between society and nature; and 2) productive collaborations must be built between the university and diverse stakeholders to develop a sufficient understanding of sustainability problems and viable strategies for solving them.”

Abstract

Academic scientists can transcend publish-or-perish incentives to help produce real-world solutions. Here’s how one group did it.

APA Style Reference

Hart, D. D., & Silka, L. (2020). Rebuilding the ivory tower: bottom-up experiment in aligning research with societal needs. Issues Sci Technol, 36(3), 64-70. https://issues.org/aligning-research-with-societal-needs/ [accessed 14/08/2020]

You may also be interested in




Is science really facing a reproducibility crisis, and do we need it to? (Fanelli, 2018)

Main Takeaways:

  • Science is said to be in a crisis due to unreliable findings, poor research quality and integrity, low statistical power, and questionable publication practices caused by the pressure to publish.
  • Fanelli questions this “science in crisis” narrative by critically examining the evidence for the existence of these problems.
  • Fraud and questionable research practices exist, but they are likely  not common enough to seriously distort the scientific literature.
  • The pressure to publish has not been conclusively linked to scientific bias or misconduct.
  • Low power and replicability may differ between subfields and methodologies, and may be influenced by the magnitude of the true effect size, and the prior probability of the hypothesis being true.
  • There is little evidence to suggest that misconduct or questionable research practices have increased in recent years.
  • The “science in crisis” narrative is not well supported by the evidence and may be counterproductive, as it encourages values that can be used to discredit science. A narrative of “new opportunities” or “revolution” may be more empowering to scientists.

Quote

“Science always was and always will be a struggle to produce knowledge for the benefit of all of humanity against the cognitive and moral limitations of individual human beings, including the limitations of scientists themselves.” (p.2630)

Abstract

Efforts to improve the reproducibility and integrity of science are typically justified by a narrative of crisis, according to which most published results are unreliable due to growing problems with research and publication practices. This article provides an overview of recent evidence suggesting that this narrative is mistaken, and argues that a narrative of epochal changes and empowerment of scientists would be more accurate, inspiring, and compelling.

APA Style Reference

Fanelli, D. (2018). Opinion: Is science really facing a reproducibility crisis, and do we need it to?. Proceedings of the National Academy of Sciences, 115(11), 2628-2631. https://doi.org/10.1073/pnas.1708272114

You may also be interested in




Fallibility in Science: Responding to Errors in the Work of Oneself and Others (Bishop, 2018)

Main Takeaways:

  • Researchers may be under pressure, once they find a error, to not reveal it due to pressure from senior scientist and the institution;
  • Retraction produces fear in a scientist, as it is associated with shame.
  • Errors can be reduced with open science practices.
  • Raw data can never be made completely open due to confidentiality but we can modify it to remove identifiable information, so that other researchers reproduce what was done.
  • Stigma needs to be removed concerning error detection.
  • Making an analysis program open does not mean they are error-free. A reproducible result simply indicates that when the same data is analysed, the same result is obtained, even if incorrect.
  • Researchers whose error is noticed and respond with denial, anger or silence tend to damage their reputation for integrity. Resolving such issues via the journal that published the original article may be a better approach, though this process seldom proceeds smoothly.
  • Findings may be due to methodological concern, as opposed to errors in calculation or scripts, such as conducting a study without a control group, underpowered, using unreliable measures or that has a major confound.
  • Methodological errors may be due to ignorance instead of bad faith, including honest errors in the data, analysis or method which compromise conclusions inferred.
  • Replication is important, as confidence in the robustness of a finding cannot depend on a single study. When there is a failure to replicate, we should uncover why this happened (e.g. contextual factors or research expertise).
  • We should not say original researchers are incompetent, frauds, etc., but we should also not say that critics had malevolent motives and lack expertise. We need to be impartial.
  • We should avoid bias and identify publications that are ignored, as positive findings produce more citations than null findings.
  • Investigating misconduct is important but challenging. It is a difficult endeavour and requires evidence that takes time to accumulate.
  • Academic institutions take an accusation of misconduct against a staff member seriously but it takes a long time. We should consider whether people could have vested interests against this academic.
  • We should not mock or abuse other scientists who make honest errors, as this would encourage poor research practices and people may be less likely to be open about these errors.

Quote

“Criticism is the bedrock of the scientific method. It should not be personal: If one has to point to problems with someone’s data, methods, or conclusions, this should be done without implying that the person is stupid or dishonest. This is important, because the alternative is that many people will avoid engaging in robust debate because of fears of interpersonal conflict—a recipe for scientific stasis. If wrong ideas or results are not challenged, we let down future generations who will try to build on a research base that is not a solid foundation. Worse still, when the research findings have practical applications in clinical or policy areas, we may allow wrongheaded interventions or policies to damage the well-being of individuals or society. As open science becomes increasingly the norm, we will find that everyone is fallible. The reputations of scientists will depend not on whether there are flaws in their research, but on how they respond when those flaws are noted.” (p.6)

Abstract

This is a view on the fallibility of science, response to self-errors and errors made by others by Professor Dorothy Bishop. It contains a discussion on how open science should be the norm but being open and honest about oneself is not. It informs us that we should not mock or be hurtful to others concerning honest mistakes and that misconduct is a serious issue but we need to be supportive of both the researcher who is being accused and the individual who is accusing them.

APA Style Reference

Bishop, D. V. M. (2018). Fallibility in science: responding to errors in the work of oneself and others. Advances in Methods and Practices in Psychological Science, 1(3), 432-438. https://doi.org/10.1177/2515245918776632

You may also be interested in




Seven Steps Toward Transparency and Replicability in Psychological Science (Lindsay, 2020)

Main Takeaways:

  • Publication bias, small sample sizes, and p-hacking exaggerate effect sizes in the literature, contributing to the replication crisis.
  • Lindsay proposes seven steps to improve transparency and replicability:
  • 1.   Tell the truth. Be honest and advocate research-if the idea was inspired by data, state so. Report effect size with 95% confidence intervals around them.
  • 2.   Assess your understanding of inferential statistical tools. We need  improved statistical sophistication for researchers to test hypotheses about populations based on their samples - reward quality and accuracy of methods, not quantity and flashiness of results.
  • 3. Consider standardizing aspects of your approach to conducting hypothesis testing research. Create a detailed research plan providing a priori hypotheses, sample size planning, data exclusion rules, analyses, transformations, covariates etc. Be transparent and register a research plan (cf. Pre-registration and Registered reports).
  • 4.   Consider developing a lab manual. Include standardised procedures in data exclusion, data transformations, data-cleaning, authorship, file naming conventions, etc.
  • 5.  Make your materials, data, and analysis scripts transparent. They should be Findable, Accessible, Interoperable and Reusable (FAIR).
  • 6.   Address constraints on the generality of your findings. Under what conditions should your results replicate, and not replicate? Failure to replicate could be due to differences in procedures, albeit original work did not indicate such differences modulate effect.
  • 7.   Consider collaborative approaches to conducting research.

Quote

“The aim of the methodological reform movement is not to restrict psychological research to procedures that meet some fixed criterion of replicability. Replicability is not in itself the goal of science. Rather, the central aim of methodological reform is to make research reports more transparent, so that readers can gain an accurate understanding of how the data were obtained and analyzed and can therefore better gauge how much confidence to place in the findings. A second aim is to discourage practices that contribute to effect-size exaggeration and false discoveries of non-existent phenomena. As per Vazire’s analogy, the call is not for car dealerships to sell nothing but new Ferraris, but rather for dealers to be forthcoming about the weaknesses of what they have on the lot. The grand aim of science is to develop better, more accurate, and more useful understandings of reality. Methodological reform cannot in and of itself deliver on that goal, but it can help.” (p.19).

Abstract

Psychological scientists strive to advance understanding of how and why we animals do and think and feel as we do. This is difficult, in part because flukes of chance and measurement error obscure researchers’ perceptions. Many psychologists use inferential statistical tests to peer through the murk of chance and discern relationships between variables. Those tests are powerful tools, but they must be wielded with skill. Moreover, research reports must convey to readers a detailed and accurate understanding of how the data were obtained and analyzed. Research psychologists often fall short in those regards. This paper attempts to motivate and explain ways to enhance the transparency and replicability of psychological science. Specifically, I speak to how publication bias and p hacking contribute to effect-size exaggeration in the published literature, and how effect-size exaggeration contributes, in turn, to replication failures. Then I present seven steps toward addressing these problems: Telling the truth; upgrading statistical knowledge; standardizing aspects of research practices; documenting lab procedures in a lab manual; making materials, data, and analysis scripts transparent; addressing constraints on generality; and collaborating.

APA Style Reference

Lindsay, D. S. (2020). Seven steps toward transparency and replicability in psychological science. Canadian Psychology/Psychologie canadienne. Advance online publication. https://doi.org/10.1037/cap0000222 [ungated]

You may also be interested in




Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them (Flake & Fried, 2019)◈

Main Takeaways:

  • Questionable measurement practices (QMPs) undermine internal and external validity, both of the statistical conclusions and construct of interest.
  • The authors focus on  lack of transparency in reporting how measures are developed, used, and adapted, as well as reporting key psychometric information.
  • Often, information about the decisions researchers make is lacking, particularly when measures are created/adapted on the fly (e.g. changing response type, changing response style or options, changing item wording or content).
  • Increasing transparency of measurement development and use facilitates thorough and accurate evaluation of validity of results. The authors offer the following guidance:

Quote

“The increased awareness and emphasis on QRPs, such as p-hacking, have been an important contribution to improving psychological science. We echo those concerns, but also see a grave need for broadening our scrutiny of current practices to include QMPs (Fried & Flake, 2018). Recalling our example of depression at the outset, even if we increase the sample size of our depression trials, adequately power our studies, pre-register our analytic strategies, and stop p-hacking — we can still be left wondering if we were ever measuring depression at all.” (p.22)

Abstract

In this paper, we define questionable measurement practices (QMPs) as decisions researchers make that raise doubts about the validity of the measures, and ultimately the validity of study conclusions. Doubts arise for a host of reasons including a lack of transparency, ignorance, negligence, or misrepresentation of the evidence. We describe the scope of the problem and focus on how transparency is a part of the solution. A lack of measurement transparency makes it impossible to evaluate potential threats to internal, external, statistical conclusion, and construct validity. We demonstrate that psychology is plagued by a measurement schmeasurement attitude: QMPs are common, hide a stunning source of researcher degrees of freedom, pose a serious threat to cumulative psychological science, but are largely ignored. We address these challenges by providing a set of questions that researchers and consumers of scientific research can consider to identify and avoid QMPs. Transparent answers to these measurement questions promote rigorous research, allow for thorough evaluations of a study’s inferences, and are necessary for meaningful replication studies.

APA Style Reference

Flake, J. K., & Fried, E. I. (2019). Measurement schmeasurement: Questionable measurement practices and how to avoid them. https://psyarxiv.com/hs7wm/

You may also be interested in




A consensus-based transparency checklist (Aczel et al., 2020)

Main Takeaways:

  • This manuscript provides a checklist of research transparency practices researchers can use to evaluate the transparency in their study, or to help improve transparency at each stage of the research process.
  • Transparency is required to evaluate and reproduce findings, and also for research synthesis and meta analysis from the raw data.
  • There is a lack of transparency in the literature, but we should not assume an intention to be deceptive or misleading. Rather, human reasoning is prone to biases (e.g. confirmation bias and motivated reasoning) and few journals ask about statistical and methodological practices and transparency.
  • Journals can support open practices by offering badges, using the transparency and openness promotion guidelines, promote the availability of all research items, including data, materials and codes.
  • The consensus-based transparency checklist can be submitted with the manuscript to provide critical information about the process to evaluate the robustness of a finding.
  • The checklist can be modified by deleting, adding and rewording items with a high level of acceptability and consensus with no strong counter argument for single items.
  • Researchers can explain the choices at the end of each 36 section. There is a shortened 12-item version to reduce demands on the researchers’ time and facilitate broader adoption that fosters transparency and asks authors to complete a 36-item list.

Abstract

We present a consensus-based checklist to improve and document the transparency of research reports in social and behavioural research. An accompanying online application allows users to complete the form and generate a report that they can submit with their manuscript or post to a public repository.

APA Style Reference

Aczel, B., Szaszi, B., Sarafoglou, A., Kekecs, Z., Kucharský, Š., Benjamin, D., ... & Ioannidis, J. P. (2020). A consensus-based transparency checklist. Nature human behaviour, 4(1), 4-6. https://doi.org/10.1038/s41562-019-0772-6

You may also be interested in




Tell it like it is (Anon, 2020)

Main Takeaways:

  • Current research culture is defined by a pressure to present research projects as conclusive narratives that leave no room for ambiguity.
  • A manuscript answers a question(s) based on findings and how they support or contradict hypotheses.
  • Clean narratives represent a threat to validity and counter reality of what science looks like.
  • Clean narratives often report only outcomes to confirm original predictions or exclude research findings that provide messy results.
  • These questionable research practices create a distorted picture of research that prevents cumulative knowledge.
  • Pre-registration has little value if not heeded or transparently reported.
  • It is evident during peer review that a pre-registered analysis is inappropriate or suboptimal. Authors should have to provide deviations and explain why they did these deviations.
  • If a pre-registered analysis plan is flawed, or needs to be amended, authors can report results of pre-registered alongside new analyses.
  • Authors report multi-study research papers and authors report all work they executed, irrespective of outcomes.

Quote

“No research project is perfect; there are always limitations that also need to be transparently reported. In 2019, we made it a requirement that all our research papers include a limitations section, in which authors explain methodological and other shortcomings and explicitly acknowledge alternative interpretations of their findings… Science is messy, and the results of research rarely conform fully to plan or expectation. ‘Clean’ narratives are an artefact of inappropriate pressures and the culture they have generated. We strongly support authors in their efforts to be transparent about what they did and what they found, and we commit to publishing work that is robust, transparent and appropriately presented, even if it does not yield ‘clean’ narratives” p.1

Abstract

Every research paper tells a story, but the pressure to provide ‘clean’ narratives is harmful for the scientific endeavour.

APA Style Reference

Anon (2020). Tell it like it is. Nat Hum Behav 4, 1. https://doi.org/10.1038/s41562-020-0818-9 

You may also be interested in




Is pre-registration worthwhile? (Szollosi et al., 2020)

Main Takeaways:

  • Pre-registration should be an option to improve research. Pre-registration intends to solve statistical problems and forces people to think more deeply about theories, methods, and analyses.But, needing, rewarding, or promoting it is not worthwhile. Requiring pre-registration could harm the progress in our field.
  • Scientific inference is the process to develop better theories. Statistical models are simplified mathematical abstractions of scientific problems, simplifications to aid scientific inference but to allow abstraction.
  • Diagnosticity of statistical tests depends on how well statistical models map onto theories and improved statistical techniques does little to improve theories when mapping is weak.
  • Models are useful depending on how accurately the theory is matched to the model. Many statistical models (e.g. general linear model) in psychology are poor estimates of the theory.
  • Bad theories can be pre-registered with predictions barely better than randomly picking an outcome. Pre-registration does not improve theories but should allow researchers to think more deeply on how to improve theories through better planning, more precise operationalisation of constructs, and clear motivation for statistical planning.
  • We should improve theories when encountering difficulties with pre-registration or when pre-registered predictions are wrong. There is no problem with post-hoc scientific inference when the theories are strong.
  • Any improvement depends on a good understanding of how to improve a theory, and pre-registration provides no understanding. Pre-registration encourages thinking, but it is unclear whether the thinking is better or worse.
  • Poor operationalisation, imprecise measurement, weak connection between theory and statistical method should take precedence over problems of statistical inference.

Abstract

Proponents of preregistration argue that, among other benefits, it improves the diagnosticity of statistical tests. In the strong version of this argument, preregistration does this by solving statistical problems, such as family-wise error rates. In the weak version, it nudges people to think more deeply about their theories, methods, and analyses. We argue against both: the diagnosticity of statistical tests depends entirely on how well statistical models map onto underlying theories, and so improving statistical techniques does little to improve theories when the mapping is weak. There is also little reason to expect that preregistration will spontaneously help researchers to develop better theories (and, hence, better methods and analyses).

APA Style Reference

Szollosi, A., Kellen, D., Navarro, D. J., Shiffrin, R., van Rooij, I., Van Zandt, T., & Donkin, C. (2020). Is Preregistration Worthwhile?. Trends in cognitive sciences, 24(2), 94.https://doi.org/10.1016/j.tics.2019.11.009

You may also be interested in




Arrested theory development: The misguided distinction between exploratory and confirmatory research (Szollosi & Donkin, 2019)◈

Main Takeaways:

  • The article describes the current philosophy of science and explain how theory development and theory assessment should work under this framework. This will be followed by why proposed methodological solutions (e.g. direct replications and pre-registration) to the “replication crisis” are unlikely to eliminate factors that interrupt theory development. They conclude in how we can move  towards the development of good theories and explanations.
  • The aim of science is to develop good explanations. To bring about good explanations is to detect and correct flaws in our existing theories.
  • A theory can be criticised by argument, meaning the rejection of explanations that are bad according to the criteria of a good theory: “(1) explain what they are supposed to explain, (2) consistent with other good theories, and (3) cannot easily be adapted to explain anything” (p.4). This takes the form of an argument that a theory cannot account for some existing observation. Also, the theory can be criticised based on how easily it can be adapted to explain several unobserved data patterns. The theory can also be criticised by experimental testing, making a theory problematic by increasing the set of observations that a theory is meant to explain but is unable to do so.
  • For science to progress, accountability in theory change is important, we should predict that each adaptation of a theory makes it more inflexible, thus increasing the potential to be made problematic. In turn, this makes the current theory problematic, thus requiring new theories.
  • The distinction between exploratory and confirmatory research is meaningless. It does not matter when theories are changed but how easy it was or would have been to make these changes. Focusing on the distinction between exploratory and confirmatory research focus on the inflexibility of the predictions of a theory as important, while ignoring the flexibility of the theory. Hypotheses only matter for experimental studies when a theory is invariant, “experimental testing of a flexible theory will not move scientific progress, even if the predictions of the theory were pre-registered or directly replicated.” (p.10).

Quote

“The key property of a good explanation is that it is hard to vary (Deutsch, 2011). More specifically, a theory can be regarded as good if it satisfies the following criteria, proposed by Deutsch (2016): good theories (1) explain what they are supposed to explain, (2) are consistent with other good theories, and (3) cannot easily be adapted to explain anything. These criteria aim to ensure that a theory is constrained by all of our existing knowledge (existing observations and other good theories), without the benefit of flexibility to tailor the explanation to any possible pattern of observation. In other words, the conjectures that comprise a theory must be inflexible while still allowing the theory to account for its explicanda. This property of good theories constrains the way in which that theory can be changed. A good theory will resist most changes, because the explanation for any change must be consistent with the retained inflexible conjectures of that theory without making the theory inconsistent with existing observations.” (p.4).

Abstract

Science progresses by finding and correcting problems in theories. Good theories are those that help facilitate this process by being hard-to-vary: they explain what they are supposed to explain, they are consistent with other good theories, and they are not easily adaptable to explain anything. Here we argue that, rather than a lack of distinction between exploratory and confirmatory research, an abundance of flexible theories is a better explanation for current replicability problems of psychology. We also explain why popular methods-oriented solutions fail to address the real problem of flexibility. Instead, we propose that a greater emphasis on theory criticism by argument would improve replicability.

APA Style Reference

Szollosi, A., & Donkin, C. (2019). Arrested theory development: The misguided distinction between exploratory and confirmatory research. https://doi.org/10.31234/osf.io/suzej

You may also be interested in




From pre-registration to publication: a non-technical primer for conducting meta-analysis to synthesize correlation data (Quintana, 2015)

Main Takeaways:

  • This review will discuss how to conduct a meta-analysis following PRISMA guidelines.
  • Pre-register the meta-analysis protocol, as it allows the researchers to formulate the study rationale for a specific research question. In addition, pre-registration avoids bias by providing evidence of a priori analysis intentions, thus in turn, reducing p-hacking.
  • Although few journals need to consider meta-analysis registration, pre-registration is important for submission. As meta-analyses are often used to guide treatment for practice and health policy, its pre-registration is possibly even more important than the pre-registration of clinical trials.
  • Most journals do not explicitly state that pre-registration is a requirement, the submission of a PRISMA checklist is required, which includes a protocol and a study registration.
  • Although there are many databases available, it is the researcher's responsibility to choose the most suitable sources for their research areas. Numerous scientists use duplicate search terms within two or more databases to cover numerous sources. Researchers can also search reference lists of eligible studies for other eligible studies (i.e. snowballing).
  • It is important to note the number of studies returned and after using the specified search term, how many of these studies were discarded and the motivation behind discarding these studies. The search teams and strategies should be specific enough for a reader to reproduce the search, which includes the date range of studies, together with the date that the search was conducted.
  • Traditionally, it has been difficult to access the gray literature (i.e. research that has not been formally published), now it is becoming more accessible as libraries are posting dissertations in their online repository.  Regardless of whether gray literature studies should be included, explicit and detailed search strategies need to be included in the study protocol and method section.
  • There are two effect models generally used in a meta-analysis, fixed and random. The way to select one of these models is centered around "how much of the variation of studies can be attributed to variation in the true effect sizes" (p. 5) – assumptions of study homogeneity.  A variation is from random error and true study heterogeneity.
  • Forest plots visualise effect sizes and confidence intervals from included studies, together with summary effect size.  A funnel plot is a visual tool to investigate potential publication bias (i.e. significant findings are published, while non-significant results are not published) in meta-analyses. Funnel plots offer a useful visualisation for potential publication bias, it is important to consider that asymmetry may represent other types of bias like study quality, location bias and study size.
  • However, funnel plots suffer from subjective measures of potential publication bias. Two tests used to calculate objective measures of potential bias: trim and fill method and moderating variables.
  • The final step of the meta analysis is data interpretation and write-up. The PRISMA guidelines provide a checklist that includes all the items that should be included when reporting a meta-analysis.

Quote

Up to 63% of psychological scientists anonymously admit to questionable research practices(John etal.,2012). These practices include removing data points and analysing data, failing to report all measures analyzed, and HARKing.Such behavior has likely contributed to the lowrates of successful replication observed in psychology (Open Science Collaboration, 2015). The pre-registration of clinical trial protocols has become standard. In contrast,lessthan10%of meta-analysis refers to a study protocol,let alone make the protocol publically available (Moheretal.,2007).Thus,meta-analyses pre-registration would markedly improve the transparency of meta-analyses and the confidence of reported findings.” (p.8)

Abstract

Starting from the view that progress in science consists of the improvement of our theories, in the current paper we ask two questions: what makes a theory good, and how much do the current method-oriented solutions to the replication crisis contribute to the development of good theories? Based on contemporary philosophy of science, we argue that good theories are hard-to-vary: they (1) explain what they are supposed to explain, (2) are consistent with other good theories, and (3) cannot easily be adapted to explain anything. Theories can be improved by identifying problems in them either by argument or by experimental test, and then correcting these problems by changing the theory. Importantly, such changes and the resultant theory should only be assessed based on whether they are hard-to-vary. An assessment of the current state of the behavioral sciences reveals that theory development is arrested by the lack of consideration for how easy it is to change theories to account for unexpected observations. Further, most of the current method-oriented solutions are unlikely to contribute much to the development of good theories, because they do not work towards eliminating this problem. Instead, they reward only temporary inflexibility in theories, and promote the assessment of theory change based on whether the theory was changed before (confirmatory) or after (exploratory) an experimental test, but not whether that change yields a hard-to-vary theory. Finally, we argue that these methodological solutions would become irrelevant if we turned our focus to the explicit aim of developing theories that are hard-to-vary.

APA Style Reference

Quintana, D. S. (2015). From pre-registration to publication: a non-technical primer for conducting a meta-analysis to synthesize correlational data. Frontiers in psychology, 6, 1549. https://doi.org/10.3389/fpsyg.2015.01549

You may also be interested in




Pre-registration is Hard, And Worthwhile (Nosek et al., 2019)

Main Takeaways:

  • Pre-registration allows us to make exploratory and confirmatory analyses.
  • Pre-registration allows us to make the transparent uncertainty more certain, how many statistical tests were conducted and familywise error rate to be corrected.
  • Pre-registration reduces influence of publication bias and pre-registration is a skill that needs experience to be improved.
  • Pre-registration promotes intellectual humility and better calibration of scientific claims.
  • It allows us to provide information on how methodology is implemented, how hypotheses are tested, the exclusion rules, how variables are combined and what to use concerning the statistical model, covariates and characteristics.
  • Pre-registration converts general sense into precise and explicit plans that predict what has not yet occurred and decide what will be done.
  • It allows us to stop data collection. What are the steps required to assess questions of interest? What are the outcomes?
  • Having a plan is better than no plan, sharing plans to advance is better than not sharing them.
  • Planning will improve and benefits will increase for oneself and consumers of research.
  • Deviations make it harder to interpret with confidence what occurred to what was planned.
  • Transparency is important and all deviations should be reported, this is difficult due to narrative coherence, reviewer expectations and word limits.
  • We need to maximise credibility of reporting findings when possible, update pre-registration, deviations before observing data, mention all planned analyses to explain why a planned analysis was not reported.
  • Use supplements to share in full not hide inconvenient information and during analysis.

Abstract

Preregistration clarifies the distinction between planned and unplanned research by reducing unnoticed flexibility. This improves credibility of findings and calibration of uncertainty. However, making decisions before conducting analyses requires practice. During report writing, respecting both what was planned and what actually happened requires good judgment and humility in making claims.

APA Style Reference

Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., ... & Vazire, S. (2019). Preregistration is hard, and worthwhile. Trends in cognitive sciences, 23(10), 815-818.https://doi.org/10.1016/j.tics.2019.07.009 [ungated]

You may also be interested in




Preregistration of Modeling Exercises May Not Be Useful (MacEachern & Van Zandt, 2019)

Main Takeaways:

  • The present study focuses on modelling and data analysis and how each round improves analysis that builds richer understanding of data and processes that give rise to data.
  • Powerful software and improved graphical capabilities allows us to explore many more features of data.
  • The ease with which data is transformed and cleaned, with which a model can be fit may lead to overfitting.
  • Model development is intrinsically exploratory and creative.
  • The present article disagrees with pre- and post-registration of models. In highly exploratory settings, there is greater difficulty to pre-register a model and analysis.
  • Modelling depends on the modeller’s perspective and data collected. Each author performs exploratory analysis and may settle on the same transformation for the response variable.
  • When the model is combined with the Bayesian model averaging, the overall model provides a better description of the entire dataset than any single model on its own.
  • Reality is too complicated and covariates are sparse enough that it would be a challenge to identify the right model. Models are tools. Different models are used differently for distinct ends.
  • Model construction and development depend on analysing and re-analysing a dataset to determine which of its properties are crucial to understand a phenomenon and/or make predictions.
  • Confirmatory model implies truth to be discovered among models in competition but there tends to be model favouritism,which tends to be determined by which models the researcher has invested time in developing; how the modeller views the world; ease of implementation and so on.
  • One model is not true in the strictest sense as some data will be captured, but other data will not.
  • Bayesian methods need to be used, as datasets grow. If preregistration of analyses is required, Bayesian analysts may need to pay particular attention to the impact of the prior distribution on features of the analysis such as the Bayes factor, and the analyst must adopt techniques that can automatically provide robustness to the analysis.
  • Underfitting of the data is as problematic as overfitting. Pre-registration of model development may lessen the engagement of analysts with the data, contributing to less creative and fewer exploratory analyses.
  • Psychology departments should devote more resources to training in quantitative areas and training which include explicit content on under- and over-modelling. Also, we should partner with the statistics department to improve our modelling skills.

Abstract

This is a commentary on Lee et al.’s (2019) article encouraging preregistration of model development, fitting, and evaluation. While we are in general agreement with Lee et al.’s characterization of the modeling process, we disagree on whether preregistration of this process will move the scientific enterprise forward. We emphasize the subjective and exploratory nature of model development, and point out that “under-modeling” of data (relying on black-box approaches applied to data without data exploration) is as big a problem as “over-modeling” (fitting noise, resulting in models that generalize poorly). We also note the potential long-run negative impact of preregistration on future generations of cognitive scientists. It is our opinion that preregistration of model development will lead to less, and to less creative, exploratory analysis (i.e., to more under-modeling), and that Lee at al.’s primary goals can be achieved by requiring publication of raw data and code. We conclude our commentary with suggestions on how to move forward.

APA Style Reference

MacEachern, S. N., & Van Zandt, T. (2019). Preregistration of modeling exercises may not be useful. Computational Brain & Behavior, 2(3-4), 179-182. https://doi.org/10.1007/s42113-019-00038-x

You may also be interested in




Six principles for assessing scientists for hiring, promotion, and tenure (Naudet et al, 2018)

Main Takeaways:

  • Academic work is usually quantified by the quantity of publications. However, this is not a reliable measure.
  • An alternative measure is impact factor: the average number of citations to research articles over the preceding two years. This is an imperfect measure that does not capture the ethos of an academic institution.
  • Impact factor provides information about citation influence for a few papers but is less informative about an individual publication and the authors involved in the publication (cf. Goodhart’s Law - a valid measurement becomes useless when it becomes an optimisation target).
  • Promotions are based on questionable research practices that promote the quantity of publications, but reproducible research does not receive such support.
  • The incentive structure in academia is problematic, as the high impact factor is taken to be similar to high societal impact; this is not the case!
  • High impact factor leads to more funding, more citations and further funding (cf. Matthew’s Effect), whereas the opposite is observed for papers with low impact factor. Papers with high societal impact seem to fit papers with low impact factor.
  • We need to provide a more inclusive evaluation scheme that allows researchers and research to focus more on open science practices.
  • We need to consider societal and broader impact for promotions.

Abstract

The negative consequences of relying too heavily on metrics to assess research quality are well known, potentially fostering practices harmful to scientific research such as p-hacking, salami science, or selective reporting. The "flourish or perish" culture defined by these metrics in turn drives the system of career advancement in academia, a system that empirical evidence has shown to be problematic and which fails to adequately take societal and broader impact into account. To address this systemic problem,

APA Style Reference

Naudet, F., Ioannidis, J., Miedema, F., Cristea, I. A., Goodman, S. N., & Moher, D. (2018). Six principles for assessing scientists for hiring, promotion, and tenure. Impact of Social Sciences Blog. http://eprints.lse.ac.uk/90753/

You may also be interested in




Sample size and the fallacies of classical inference (Friston, 2013)

Main Takeaways:

  • The purpose of this target article was to make authors and reviewers think about their response to questions on sample size and effect size for their data.
  • It is important to get as much data as possible (i.e. to have a large sample) to reduce false positives.
  • Trivial effect sizes can be resolved by reporting confidence intervals.
  • The best studies use a large number of subjects and report the results in terms of confidence intervals or protected inference.
  • Large sample sizes will increase the efficiency of model comparison. Increasing sample size will allow us to do more model comparisons, that are otherwise not possible in small sample sizes.
  • “a trivial (standardised) effect size does not mean a very small effect — it is only small in relation to the random fluctuations that attend its expression or measurement. In other words, a miniscule effect is not trivial if it is expressed reliably.” (p.504).

Quote

“the proportion of true positives, in relation to the total number of significant tests, increases with sensitivity (i.e., the positive predictive value increases with sensitivity). This simply reflects the fact that the number of false negatives is fixed and the number of true positives increases with sensitivity...if we now assume that increasing sample size will increase sensitivity, increases in sample size should therefore increase the portion of true positives. However...if trivial effect sizes predominate, the PPV measures the proportion of significant results that are trivial. This means that increasing sample size is a bad thing and will increase the probability of declaring trivial effects significant (on average).” (p. 504).

Abstract

I would like to thank Michael Ingre, Martin Lindquist and their co-authors for their thoughtful responses to my ironic Comments and Controversies piece. I was of two minds about whether to accept the invitation to reply — largely because I was convinced by most of their observations. I concluded that I should say this explicitly, taking the opportunity to consolidate points of consensus and highlight outstanding issues.

APA Style Reference

Friston, K. (2013). Sample size and the fallacies of classical inference. Neuroimage, 81, 503-504.https://doi.org/10.1016/j.neuroimage.2013.02.057

You may also be interested in




Why small low-powered studies are worse than large high-powered studies and how to protect against “trivial” findings in research: Comment on Friston (2012) (Ingre, 2013)

Main Takeaways:

  • It is often argued that small underpowered studies provide better evidence due to a lack of small and trivial effect sizes yet underpowered studies are less likely to find a true effect in the data and failure does not come without consequences.
  • Failure to detect true effects may indicate that when significant findings are reported they may be due to type I error.
  • Findings from small low-powered studies are weaker than high-powered studies due to the fact that poor statistical power increases false positive rates, even with large effect sizes.
  • Researchers should make use of at least one additional statistic value (e.g. t value, effect size or confidence intervals) alongside p-values as this would protect against analysing meaningless effects;

Quote

“From a strictly scientific point of view, you can never have too much precision, and consequently, never too many subjects or too much statistical power (unless a researcher is doing something wrong when reporting and interpreting data). The limiting factors are cost (time, resources and money) and potential harm for the subjects involved in the study. The real question you need to ask is how much cost and harm you can afford to get as good answer as possible.” (p.498)

Abstract

It is sometimes argued that small studies provide better evidence for reported effects because they are less likely to report findings with small and trivial effect sizes (Friston, 2012). But larger studies are actually better at protecting against inferences from trivial effect sizes, if researchers just make use of effect sizes and confidence intervals. Poor statistical power also comes at a cost of inflated proportion of false positive findings, less power to “confirm” true effects and bias in reported (inflated) effect sizes. Small studies (n = 16) lack the precision to reliably distinguish small and medium to large effect sizes (r < .50) from random noise (α = .05) that larger studies (n = 100) do with high level of confidence (r = .50, p = .00000012). The present paper introduces the arguments needed for researchers to refute the claim that small low-powered studies have a higher degree of scientific evidence than large high-powered studies.

APA Style Reference

Ingre, M. (2013). Why small low-powered studies are worse than large high-powered studies and how to protect against “trivial” findings in research: Comment on Friston (2012). Neuroimage, 81, 496-498.https://doi.org/10.1016/j.neuroimage.2013.03.030

You may also be interested in




Ironing out the statistical wrinkles in “ten ironic rules” (Lindquist et al., 2013)

Main Takeaways:

  • This commentary discusses the concerns and premise of the Ten ironic rules for non-statistical reviewers by Professor Karl Friston (2012).
  • “working in a highly collaborative environment has taught us that both experts and non-experts alike can have good and bad ideas about statistics (as well as every other field) and that the idea of sharp boundaries between domains is inaccurate and counterproductive.” (p.499).
  • It is more difficult to interpret significant results in small samples, as sample sizes prevent sensitivity analyses to be conducted and specific assumptions to be checked.
  • Under-sampled studies do not allow us to ask questions about confounding variables such as age and gender. Increasing sample size allows us to detect small effects.
  • One argument in favour of small sample sizes is that  when we need to consider important non-statistical/ethical issues such as the lives of animals or side effects.
  • Hypothesis testing cannot discriminate between important, but subtle, effects and  trivial effects.

Quote

“In summary, sample size discussions, both prior to conducting a study and post-hoc in peer review, should depend on a number of contextual factors and especially specifics of the hypotheses under question. A small sample size is perfectly capable of differentiating gross brain morphometry between, say, children and adults. However, thousands of participants may be necessary to detect subtle longitudinal trends associated with human brain activation patterns in disease. That is, it is information content that is important, of which number of study participants is only a proxy” (p.501).

Abstract

The article “Ten ironic rules for non-statistical reviewers” (Friston, 2012) shares some commonly heard frustrations about the peer-review process that all researchers can identify with. Though we found the article amusing, we have some concerns about its description of a number of statistical issues. In this commentary we address these issues, as well as the premise of the article.

APA Style Reference

Lindquist, M. A., Caffo, B., & Crainiceanu, C. (2013). Ironing out the statistical wrinkles in “ten ironic rules”. Neuroimage, 81, 499-502. https://doi.org/10.1016/j.neuroimage.2013.02.056

You may also be interested in




Ten ironic rules for non-statistical reviewers (Friston, 2012)

Main Takeaways:

  • Reviewers may not have adequate statistical expertise to provide a critique during peer review in order to reject a manuscript.
  • Handling editors are happy to decline a paper and are placed under pressure to maintain a high rejection rate.
  • All journals need to maximise rejection rates in order to increase the quality of submission and impact factor. There are ten rules to follow.

Quote

“We have reviewed some general and pragmatic approaches to critiquing the scientific work of others. The emphasis here has been on how to ensure a paper is rejected and enable editors to maintain an appropriately high standard, in terms of papers that are accepted for publication. Remember, as a reviewer, you are the only instrument of selective pressure that ensures scientific reports are as good as they can be. This is particularly true of prestige publications like Science and Nature, where special efforts to subvert a paper are sometimes called for.” (p.1303)

Abstract

As an expert reviewer, it is sometimes necessary to ensure a paper is rejected. This can sometimes be achieved by highlighting improper statistical practice. This technical note provides guidance on how to critique the statistical analysis of neuroimaging studies to maximise the chance that the paper will be declined. We will review a series of critiques that can be applied universally to any neuroimaging paper and consider responses to potential rebuttals that reviewers might encounter from authors or editors.

APA Style Reference

Friston, K. (2012). Ten ironic rules for non-statistical reviewers. Neuroimage, 61(4), 1300-1310. https://doi.org/10.1016/j.neuroimage.2012.04.018

You may also be interested in




Using OSF to Share Data: A Step-by-Step Guide (Soderberg, 2018)

Main Takeaways:

  • Materials should findable, accessible, interoperable and reusable (FAIR) forms. Researchers should look for repositories to decide where and how to share their data.
  • A repository should contain unique and persistent identifiers, so data can be cited.
  • The data is publicly searchable with licenses clarifying how data is reused.
  • Rich meta-data descriptions are provided to allow data to be understandable and reusable.
  • Open science framework is a free and open-source Web tool to help researchers collaboratively manage, store and share the research process and the files related to their research.
  • Step 1: create an account on https://osf.io
  • Step 2: Sign-in your account. Enter name and password or login through your institution.
  • Step 3: Create a project. Press the green button to create a new project.
  • Step 4: Add Collaborators to the project.
  • Click on Contributors and press +Add green button. Search for contributors by name and click on the green + button.
  • If a collaborator does not come up in search, add them to the project by clicking add as an unregistered contributor link.
  • Step 5: upload files that are below the maximum storage of 5GB.
  • Step 6: Add a description of the project. In order to allow you and other users to know what files relate to the project.
  • Step 7: Add a License. reuse is one of the main purposes of data sharing. Other researchers need to know how they are allowed to reuse your work.
  • Step 8: Add component. data, analysis script and study materials should be placed in the project.
  • Step 9: Share your project with reviewers. The project is set up that you may want or need to give reviewers access to the contents of your project before you make it public.
  • Step 10: Make a project public. To make a project public, press the  “make public” button in the top right corner of the project page. Anyone will be able to view and download all files.
  • Step 11: Reference open science files in your work. Include the links in the manuscript, lab website or the published article to make the data accessible and useful.

Abstract

Sharing data, materials, and analysis scripts with reviewers and readers is valued in psychological science. To facilitate this sharing, files should be stored in a stable location, referenced with unique identifiers, and cited in published work associated with them. This Tutorial provides a step-by-step guide to using OSF to meet the needs for sharing psychological data.

APA Style Reference

Soderberg, C. K. (2018). Using OSF to share data: A step-by-step guide. Advances in methods and practices in psychological science, 1(1), 115-120. https://doi.org/10.1177/2515245918757689

You may also be interested in




On supporting early-career black scholars (Roberson, 2020)  ⌺

Main Takeaways:

  • Non-Black researchers need to take immediate support for early-career Black scholars.
  • “Maybe you were in a seminar where a Black doctoral student pushed back against a racist disciplinary norm, and you silently agreed and followed up with them afterwards to let them know that you support them.”
  • This silence in public signals to Black scholars that they are not welcome in these spaces
  • We must challenge white supremacy in academia. Speaking up about this is much more costly for Black scholars,  who face an onslaught of racist micro- and macro-aggressions on a daily basis. The burden should not fall on their shoulders.
  • We should be proactive in our outreach. We should invite early-career Black scholars, if they have expertise to improve a research project. Our careers and science will benefit from this help.
  • “Do not just encourage [Black scholars] to apply, provide material support to promote our successful applications; share funded grants with [Black scholars], work with [Black scholars] on developing compelling aims pages, and write [Black scholars] a persuasive letter of support. Supporting [Black scholars] on manuscripts and funding opportunities can mitigate some of the barriers in science that often stunt Black success.”
  • Inviting Black scholars will increase their credibility as experts and expand the audience’s familiarity with their scholarship. Manels are now being prohibited but we need to eliminate all-white speaker panels.
  • Educate yourself on rising Black scholars in your field, learn from early-career Black researchers, investigate journals that publish their scholarships, be familiar with the Black community’s professional societies, affinity groups and diversify your following list on Twitter.
  • Incorporate Black scholar’s work into your syllabi. This is necessary to eliminate structural racism. However, it requires individuals with the most amount of power. These steps will promote Black people to thrive among trainees and early-career scholars.
  • This will remove barriers to promote a more inclusive environment!

Quote

“Do not just encourage [Black scholars] to apply, provide material support to promote our successful applications; share funded grants with [Black scholars], work with [Black scholars] on developing compelling aims pages, and write [Black scholars] a persuasive letter of support. Supporting [Black scholars] on manuscripts and funding opportunities can mitigate some of the barriers in science that often stunt Black success.”

Abstract

Professor Mya Roberson provides a detailed commentary about the struggles that Black people encounter in academia and starting steps to eliminate structural racism.

APA Style Reference

Roberson, M. L. (2020). On supporting early-career Black scholars. Nature Human Behaviour, 1-1. https://doi.org/10.1038/s41562-020-0926-6

You may also be interested in




On the persistence of low power in psychological science (Vankov et al., 2014)

Main Takeaways:

  • Surveys of the literature have consistently shown that psychological studies have low statistical power and this problem has seen little, if any, improvement over the last few decades. Vankov et al. examine two arguments of why this may be the case.
  • The first possible reason is that researchers may fail to appreciate the importance of statistical power, since null hypothesis significance testing is a hybrid of two statistical theories- Fisher’s and Neyman and Pearson’s. While researchers readily adhere to the 5% Type I error rate, they pay little attention to the Type II error rate. Both need to be considered when evaluating whether a result is true.
  • A second possible reason is that scientists are humans and respond to incentives, such as the prestige of publishing a transformative study in a highly-regarded journal. However, producing such works is a high-risk strategy; a safer option may be to “salami-slice” works into multiple publications to increase the chance of producing publishable outputs.
  • To examine the merit of the first reason, Vankov et al. contacted authors of published papers and asked them for their sample size rationale. One third of the contacted authors were found to hold beliefs that would typically act to reduce statistical power.
  • There is a need for structural change, where editors and journals enforce rigorous requirements for statistical power. Journals introducing registered reports may also place greater emphasis on statistical power and robust designs.

Abstract

A comment by Dr Ivan Vankov, Professors Jeffrey Bowers and Marcus Munafo on the persistence of low power in psychological sciences. They discuss issues concerning false negatives, the importance of highly-regarded journals and that power is an issue to be discussed. They state that we need structural changes in journals in order to avoid the replicability crisis.

APA Style Reference

Vankov, I., Bowers, J., & Munafò, M. R. (2014). Article commentary: On the persistence of low power in psychological science. Quarterly journal of experimental psychology, 67(5), 1037-1040. https://doi.org/10.1080/17470218.2014.885986

You may also be interested in




Publication Decisions and their possible effects on inferences drawn from tests of significance or vice versa (Sterling, 1959)

Main Takeaways:

  • There is a risk to reject the null hypothesis, as it can lead to type I error.
  • “The experimenter who uses...tests of significance to evaluate observed differences usually reports that he has tested Ho by finding the probability of the experimental results on the assumption that Ho is true, and he does (or does not) ascribe some effect to experimental treatments”. (p.30).
  • Depending on the confidence of methodology and data collection, readers can reject or accept the null hypothesis.
  • Acceptance and rejection of null hypothesis is taken at p < .05.
  • When a fixed level of significance is used as a criterion for publishing in professional journals,, it may result in embarrassing and surprising results.

Quote

“What credence can then be given to inferences drawn from statistical tests of Ho if the reader is not aware of all experimental outcomes of a kind? Perhaps even more pertinent is the question: Can the reader justify adopting the same level of significance as does the author of a published study?” (p.33)

Abstract

There is some evidence that in fields where statistical tests of significance are commonly used, research which yields non-significant results is not published. Such research being unknown to other investigators may be repeated independently until eventually by chance a significant result occurs - an "error of the first kind" - and is published. Significant results published in these fields are seldom verified by independent replication. The possibility thus arises that the literature of such a field consists in substantial part of false conclusions resulting from errors of the first kind in statistical tests of significance.

APA Style Reference

Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American statistical association, 54(285), 30-34. https://doi.org/10.1080/01621459.1959.10501497

You may also be interested in




Replicability as a Publication Criterion (Lubin, 1957)

Main Takeaways:

  • How can publication lag be reduced?  Researchers should perform replications to ensure the results are repeated.
  • Replicability and generalisability should be used as a criteria to judge the rigour for these articles.
  • If results are replicated, there is no need to discuss other trivial factors. However, if it is not replicated, you should discuss contextual factors such as the time of day.

Abstract

A commentary by Dr Ardie Lubin on replicability being perceived as a criterion of publication. Replications are perceived as fundamental but not enough to publish. However, replication studies are important to conduct in order to remove any trivial variables that may explain the findings.

APA Style Reference

Lubin, A. (1957). Replicability as a publication criterion. American Psychologist, 12(8), 519-520. https://doi.org/10.1037/h0039746

You may also be interested in




The life of p: “Just significant” results are on the rise (Leggett et al., 2013)

Main Takeaways:

  • Computers make it easy to analyse data. Modern software allows simple calculations, enabling them to monitor data, while collecting it.
  • The ease of data analysis has made issues such as optional stopping and exclusion of outliers (i.e. p-hacking) easier to engage in.
  • The current study measured whether there is an over-representation of p < .05 for 2005 than 1965.
  • Method:  P values were collected from all articles published between 1965 and 2005.
  • Method: P values that were categorised as p < .05 and .01 were recalculated to provide a more exact p-value.
  • Method: If there was a lack of information to determine the exact p value, the data for this specific p value was excluded from the analysis.
  • Method: The distributions of p values used for the manuscript were between .01 and .10, any values outside of this range were excluded.
  • Results: The frequency of p values at or below .05 was greater compared to p frequencies in other ranges.
  • Results:  Although there is an over-representation for p values below .05 between 1965 and 2005,  there was a greater spike for p < .05 in 2005 than 1965.
  • Results:  In addition, p values close to but over .05 were more likely to be rounded down (e.g. p = .053 becomes p < .05) or incorrectly reported as significant in 2005 than in 1965.
  • As a result of shifting research climates, there are changes in how statistical analyses are executed.
  • Any values above .05 should be interpreted as non-significant, including trends such as .051.
  • Suboptimal research practices are easier to engage in, as calculations have become easier to compute.

Quote

“The use of confidence intervals, along with effect sizes, as well as registered reporting and mandatory methods disclosure, might decrease the emphasis placed on p values. This would, in turn, also encourage the use of optimal research practices. In the absence of additional, complementary statistics or registered reports, the use of p values as an isolated method for determining statistical significance remains vulnerable to human fallibility.” (p.2309)

Abstract

Null hypothesis significance testing uses the seemingly arbitrary probability of .05 as a means of objectively determining whether a tested effect is reliable. Within recent psychological articles, research has found an overrepresentation of p values around this cut-off. The present study examined whether this overrepresentation is a product of recent pressure to publish or whether it has existed throughout psychological research. Articles published in 1965 and 2005 from two prominent psychology journals were examined. Like previous research, the frequency of p values at and just below .05 was greater than expected compared to p frequencies in other ranges. While this overrepresentation was found for values published in both 1965 and 2005, it was much greater in 2005. Additionally, p values close to but over .05 were more likely to be rounded down to, or incorrectly reported as, significant in 2005 than in 1965. Modern statistical software and an increased pressure to publish may explain this pattern. The problem may be alleviated by reduced reliance on p values and increased reporting of confidence intervals and effect sizes.

APA Style Reference

Leggett, N. C., Thomas, N. A., Loetscher, T., & Nicholls, M. E. (2013). The life of p:" just significant" results are on the rise. Quarterly journal of experimental psychology (2006), 66(12), 2303. https://doi.org/10.1080/17470218.2013.863371

You may also be interested in




Psychologists Are Open to Change, yet Wary of Rules (Fuchs et al., 2012)

Main Takeaways:

  • The present study investigated whether psychologists support concrete changes to data collection, reporting and publication processes. If not, what are their reasons?
  • Method: 1292 psychologists from 42 countries were surveyed to assess whether each of Simmons et al.’s (2011) requirements and guidelines should be followed as a measure of good practice and whether these guidelines should be placed as mandatory conditions for publication in psychological journals.
  • Results:  98% of psychologists are open to change and agreed at least one requirement should be placed as a condition for publication, especially that “researchers must report all experimental conditions run in a study, including failed manipulations”(p.641).
  • Results: The reasons for not including a condition were: it was too rigorous; the condition did not agree with the argument; or the condition was not appropriate for all studies.
  • Psychologists are open to change for reporting and conducting research, and agree with the guidelines. However, some requirements are too rigid and questionable.

Quote

“Researchers and editorial staff alike must also ensure that standards are enforceable so as to avoid punishing honest researchers. The psychological community should capitalize on the current openness to change in order to develop and implement appropriate changes and thus improve the quality of published psychological research.” (p. 641).

Abstract

Psychologists must change the way they conduct and report their research—this notion has been the topic of much debate in recent years. One article recently published in Psychological Science proposing six requirements for researchers concerning data collection and reporting practices as well as four guidelines for reviewers aimed at improving the publication process has recently received much attention (Simmons, Nelson, & Simonsohn, 2011). We surveyed 1,292 psychologists to address two questions: Do psychologists support these concrete changes to data collection, reporting, and publication practices, and if not, what are their reasons? Respondents also indicated the percentage of print and online journal space that should be dedicated to novel studies and direct replications as well as the percentage of published psychological research that they believed would be confirmed if direct replications were conducted. We found that psychologists are generally open to change. Five requirements for researchers and three guidelines for reviewers were supported as standards of good practice, whereas one requirement was even supported as a publication condition. Psychologists appear to be less in favor of mandatory conditions of publication than standards of good practice. We conclude that the proposal made by Simmons, Nelson & Simonsohn (2011) is a starting point for such standards.

APA Style Reference

Fuchs, H. M., Jenny, M., & Fiedler, S. (2012). Psychologists are open to change, yet wary of rules. Perspectives on Psychological Science, 7(6), 639-642.

You may also be interested in




Experimental power comes from powerful theories – the real problem in null hypothesis testing (Ashton, 2013)

Main Takeaways:

  • Although null hypothesis testing is a powerful tool for decision making,  null hypothesis is no longer performed in how it was originally conceived.
  • A power analysis of a specific desired effect size must be carried out prior to the experiment being conducted, wherein a null hypothesis is compared against an alternative hypothesis based on this effect size.  This may not be possible with an effect size that was estimated from the data with no standard to compare with.
  • The advice to increase sample size and statistical power is sound, this may make any hypothesis in neuroscience become virtually untestable. For instance, if we fail to find an effect at 10%, we need to increase power to detect a 1% or 0.1% effect. In turn, poorly testable and hard-to-refute hypotheses become difficult to restrain, thus making these hypotheses more prevalent in the literature.
  • “The only way to resolve this dilemma while retaining the advantages of traditional null hypothesis testing is to be specific about the theoretical predictions that our experiments are designed to test” (p.1).

Quote

“The solution to the problem is to increase discipline not only in analysis and experimental design but also in relating experiments to explanatory theory. Much current practice instead seems to be an open-ended search for associations, reminiscent of old-style inductionism while superficially following the conventions of hypothetico-deductivism.” (p.1).

Abstract

A commentary by John C. Ashton who discusses the paper written by Professor Kate Button on small sample sizes. Ashton argues that power analyses and effects sizes should be used to estimate the alternative hypothesis.

APA Style Reference

Ashton, J. C. (2013). Experimental power comes from powerful theories—the real problem in null hypothesis testing. Nature Reviews Neuroscience, 14(8), 585-585. https://doi.org/10.1038/nrn3475-c2

You may also be interested in




Negative results are disappearing from most disciplines and countries (Fanelli, 2011)

Main Takeaways:

  • There are concerns that distort science. One concern is that the system disfavours negative findings, which gives a poor impression of science.
  • The study investigated whether positive results have increased in the recent scientific literature.
  • Method: 4600 papers in all disciplines between 1990 and 2007 were used, including variables such as frequency of papers to test a hypothesis and a report to support it.
  • Method: Country of location was included, information on year of publication and country was coded. Whether the evidence was positive or negative.
  • Results: Frequency of positive findings increased between 1990 and 2007 by 22%. This increase was larger in social and some biomedical disciplines.
  • Results: There were fewer positive results published by American than Asian countries. More positive results in American than in European countries.
  • Negative results decreased in frequency across disciplines due to publication bias.
  • The authors seem to suggest that science is now closer to truth today than 20 years ago.
  • There is an editorial bias that favours the United States that enables them to publish as many or more negative results than any other country, not fewer. The United States has a stronger bias against negative findings than Europe.

Quote

“However, even if in the long run truth will prevail, in the short term resources go wasted in pursuing exaggerated or completely false findings (Ioannidis 2006). Moreover, this self-correcting principle will not work efficiently in fields where theoretical predictions are less accurate, methodologies less codified, and true replications rare. Such conditions increase the rate of both false positives and false negatives, and a research system that suppresses the latter will suffer the most severe distortions.” (p.900)

Abstract

Concerns that the growing competition for funding and citations might distort science are frequently discussed, but have not been verified directly. Of the hypothesized problems, perhaps the most worrying is a worsening of positive-outcome bias. A system that disfavours negative results not only distorts the scientific literature directly, but might also discourage high-risk projects and pressure scientists to fabricate and falsify their data. This study analysed over 4,600 papers published in all disciplines between 1990 and 2007, measuring the frequency of papers that, having declared to have ‘‘tested’’ a hypothesis, reported a positive support for it. The overall frequency of positive supports has grown by over 22% between 1990 and 2007, with significant differences between disciplines and countries. The increase was stronger in the social and some biomedical disciplines. The United States had published, over the years, significantly fewer positive results than Asian countries (and particularly Japan) but more than European countries (and in particular the United Kingdom). Methodological artefacts cannot explain away these patterns, which support the hypotheses that research is becoming less pioneering and/or that the objectivity with which results are produced and published is decreasing.

APA Style Reference

Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90(3), 891-904. https://doi.org/10.1007/s11192-011-0494-7

You may also be interested in




Negativity towards negative results: a discussion of the disconnect between scientific worth and scientific culture (Matosin et al., 2014)

Main Takeaways:

  • There is pressure on scientists to choose investigative avenues that result in high-impact knowledge (as opposed to hypothesis-driven).
  • Negative results are not valued as positive results (positive results are valued, negative results are undervalued).
  • Scientific principles are under reconsideration and there are events, in which new evidence refutes old hypotheses (cf. paradigm shift).
  • Negative findings are seen as an inconvenient truth.
  • Science is a collaborative endeavour and we should value and report negative findings
  • When time is money, the current heuristic of judging  research output on impact and citations can lead to waste of funds & time.
  • It is commonplace for researchers to face resistance when presenting their work at scientific conferences. Then why is a negative finding viewed as a bad thing? What’s more, a negative result is often seen as philosophical rather than practical (voir real).
  • If negative questions are rephrased as positive questions, does that mean a negative finding is a positive finding?
  • Negative findings are seen as taboo and unworthy of publication in social sciences, but, for example, for clinical research, negative results are of absolute relevance and importance.
  • Negative results are not worthy of attention, thus placed in a file drawer and seen as less important.

Quote

“It means that the direction of scientific research should not be determined by the pressure to win the ‘significance lottery’, but rather systematic, hypothesis-driven attempts to fill holes in our knowledge. At the core, it is our duty as scientists to both: (1) publish all data, no matter what the outcome, because a negative finding is still an important finding; and (2) have a hypothesis to explain the finding. If the experiment has been performed to plan, the data has not been manipulated or pulled out of context and there is compiled evidence of a negative result, then it is our duty to provide an explanation as to why we are seeing what we are seeing. Only by truly rethinking the current scientific culture, which clearly favours positive findings, will negative results be esteemed for their entire value. Only then can we work towards an improved scientific paradigm.” (p.173)

Abstract

“What gets us into trouble is not what we don’t know, it’s what we know for sure that just ain’t so.” – Mark Twain. Science is often romanticised as a flawless system of knowledge building, where scientists work together to systematically find answers. In reality, this is not always the case. Dissemination of results are straightforward when the findings are positive, but what happens when you obtain results that support the null hypothesis, or do not fit with the current scientific thinking? In this Editorial, we discuss the issues surrounding publication bias and the difficulty in communicating negative results. Negative findings are a valuable component of the scientific literature because they force us to critically evaluate and validate our current thinking, and fundamentally move us towards unabridged science.

APA Style Reference

Matosin, N., Frank, E., Engel, M., Lum, J. S., & Newell, K. A. (2014). Negativity towards negative results: a discussion of the disconnect between scientific worth and scientific culture. Disease Models & Mechanisms, 7(2), 171. https://doi.org/10.1242/dmm.015123

You may also be interested in




A farewell to Bonferroni: the problems of low statistical power and publication bias (Nakagawa, 2004)

Main Takeaways:

  • There are several effect size measures: Cohen’s d and Pearson’s r. The former assesses the mean difference and the latter evaluates the strength of the relationship.
  • Bonferroni correction tries to reduce false positives when multiple tests or comparisons are performed.
  • Reviewers may demand a Bonferroni correction to remove irrelevant variables and reduce the number of false positives but it can still lead to publication bias.
  • The scientific community should discourage Bonferroni or the idea that reviewers should demand a Bonferroni correction.
  • These problems stem from a focus on statistical significance (i.e. p values) in journals instead of practical or biological significance (i.e. effect sizes). Researchers should be reporting effect sizes and the confidence intervals around these effect sizes.

Abstract

Professor Shinichi Nakagawa provides a commentary on low statistical power and the need to discourage Bonferroni corrections. In addition, we should rely on effect sizes and their confidence intervals to determine the value of science findings.

APA Style Reference

Nakagawa, S. (2004). A farewell to Bonferroni: the problems of low statistical power and publication bias. Behavioral ecology, 15(6), 1044-1045. https://doi.org/10.1093/beheco/arh107

You may also be interested in




The File-drawer problem revisited: A general weighted method for calculating fail-safe numbers in meta analysis (Rosenberg, 2005)

Main Takeaways:

  • There is a file-drawer problem in which studies are not published if they observe no significant effects. One measure to assess the number of non-significant findings is a fail-safe number. A fail-safe number is the number of non-significant, unpublished or missing studies would be needed to reduce the overall significant results to non-significant effect. If the fail-safe number is large compared to the number of observed studies, one can be confident in the summary conclusions.
  • The method to calculate a fail-safe number is to calculate the significance of multiple studies by calculating the significance of the mean Z-score (Rosenthal, 1979), whereas Orwin (1983) calculate a fail-safe number based on an effect size that measures the standardised mean difference between intervention and control. It also calculates the number of additional studies to reduce an observed mean effect size to a desired minimal effect size.
  • However, these approaches have several problems: “The first is that they are both explicitly unweighted. One of the primary attributes of contemporary meta-analysis is weighting; studies with large sample size or small variance are given higher weight than those with small sample sizes or large variance. Neither method accounts for the weight of the observed or the hypothesized unpublished studies. A second problem with Rosenthal’s method is that the method of adding Z-scores is not normally the method by which one combines studies in a meta-analysis; most modern meta-analyses are based on the combination of effect sizes, not simply significance values (Rosenberg et al. 2000). Rosenthal’s calculation is therefore not precisely applicable to the actual significance obtained from a meta-analysis. Orwin’s method is not based on significance testing; the choice of a desired minimal effect size to test the observed mean against seems unstable without a corresponding measure of variance” (pp.464-465).
  • The article proposes a general, weighted fail-safe calculation framework applicable to both fixed- and random-effects models.
  • The original fail-safe calculations are based on the fixed-effect model, but the authors estimate a fail-safe number for random-effects model meta-analysis.
  • Although the number of studies of null effect are perceived as important to change a significant outcome in the fixed-effect calculations, these studies for the random-effects model involve a sum-of-squares calculation that we need to assume have effects that are precisely zero. This assumption could be partially avoided by simulating missing studies with a desired variance.

Quote

“One needs to remember that a fail-safe calculation is neither a method of identifying publication bias nor a method of accounting for publication bias that does exist. It is simply a procedure by which one can estimate whether publication biases (if they exist) may be safely ignored...While perhaps not as elegant as some of these methods, a fail-safe number is much simpler to calculate. Hopefully, the approach presented here will allow us to better estimate the potential for unpublished or missing studies to alter our conclusions; a low fail-safe number should certainly encourage researchers to pursue the more complicated publication bias methodologies.” (p.467).

Abstract

Quantitative literature reviews such as meta-analysis are becoming common in evolutionary biology but may be strongly affected by publication biases. Using fail-safe numbers is a quick way to estimate whether publication bias is likely to be a problem for a specific study. However, previously suggested fail-safe calculations are unweighted and are not based on the framework in which most meta-analyses are performed. A general, weighted fail-safe calculation, grounded in the meta-analysis framework, applicable to both fixed- and random-effects models, is proposed. Recent meta-analyses published in Evolution are used for illustration.

APA Style Reference

Rosenberg, M. S. (2005). The file‐drawer problem revisited: a general weighted method for calculating fail‐safe numbers in meta‐analysis. Evolution, 59(2), 464-468.https://doi.org/10.1111/j.0014-3820.2005.tb01004.x

You may also be interested in




The “File Drawer Problem” and Tolerance for Null Results (Rosenthal, 1979)

Main Takeaways:

  • The file drawer problem is that 5% of articles are false positives, while file drawers have 95% non-significant results.
  • Researchers need to calculate the number of studies with null findings before the overall false positives are made.
  • A conservative alternative is to set Z = .00 when the exact p levels are not present for any non-significant findings, while setting Z = 1.645 when p < .05.
  • “A small number of studies that are not very significant, even when their combined p is significant, may well be misleading in that only a few studies filed away could change the combined significant result to a nonsignificant one.” (p.640).
  • Currently, there are no firm guidelines that can be given as to what constitute an unlikely number of unretrieved or unpublished studies.

Quote

“[...] more and more reviewers of research literature are estimating average effect sizes and combined ps of the studies they summarize. It would be very helpful to readers if for each combined p they presented, reviewers also gave the tolerance for future null results associated with their overall significance level.” (p.640)

Abstract

For any given research area, one cannot tell how many studies have been conducted but never reported. The extreme view of the "file drawer problem" is that journals are filled with the 5% of the studies that show Type I errors, while the file drawers are filled with the 95% of the studies that show nonsignificant results. Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed.

APA Style Reference

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638. [ungated]

You may also be interested in




At random: Sense and Nonsense (McNemar, 1960)

Main Takeaways:

  • 1) To what extent have biologists entered into psychology? A random sample of 100 American Psychological Association (APA)  members indicated that only 1% can be defined as fifth columnists for biology and from a random sample of 100 titles in Psychological Abstracts indicated that only 4% have a biological slant. This indicates that psychology has not risen to the level of biology.
  • 2) To what extent have statisticians taken over of psychology? Using the aforementioned samples, 1% of the APA members call themselves statisticians and 5% of abstracts deal primarily with statistics.
  • The development of a science depends largely on the invention of measuring instruments (e.g. The Thurstone and the Likert scaling techniques). However, “it does not require much imagination to predict that other instruments will lead to more and more unimaginative research. It is so easy to say: For your dissertation, why don't you apply the ZANY to such and such groups?” (p.296).
  • It is also important to look at the arsenal of statistics. The application of chi-square did not spread into psychology until the 1930s. However, the chi-square test was being misused, and its frequent misuse has contributed to some astoundingly  leading to fallacious significance levels.
  • In the late 1930s, the analysis of variance (ANOVA) invaded psychology and supporters of the ANOVA argued that the ANOVA would “rescue involve analysis of variance. Also, during the late 1930s, researchers seem to think that the more complex the design the better, although this complexity will introduce additional complexity of data interpretation.
  • Too many users of the ANOVA argue that the “reaching of a mediocre level of significance as more important than any descriptive specification of the underlying averages” (p.297). In addition, it was argued that significance testing is necessary but not sufficient for the development of a science.
  • In order to critically evaluate the literature and to plan their own research, it is important psychologists have a sound understanding of all commonly used statistical techniques. The teaching task is partly that we should maintain enthusiasm to sell but not oversell statistics. The research problem should come first, then at the design, the “available tools should be scrutinised but with the ever present though that there is merit in simplicity.” (p.299).

Quote

“Our reviewer, after noting that 9 of the 23 authors of Foundations were not psychologists, said that the problem of learning seemed to be the only one that a psychologist could call his own. He went on to point out that learning was too much concerned with statistical interpretation of empirical data. Then he concluded that "psychology as a science is now bankrupt" and should be turned over to two groups of receivers: the biologists (broadly defined) and the statisticians.” (p.295)

Abstract

A commentary by Dr Quinn McNemar who discusses sense and nonsense data, the difficulties of statistical teaching and the advancements of research design and statistics.

APA Style Reference

McNemar, Q. (1960). At random: Sense and nonsense. American Psychologist, 15(5), 295–300. https://doi.org/10.1037/h0049193

You may also be interested in




Replication Report: Interpretation of levels of significance by psychological researchers (Beauchamp & May, 1964)

Main Takeaways:

  • Importance: one of the first papers dealing with replication in psychology.
  • Study: Psychology lecturers were more cautious than psychology graduate students when making confidence judgments about research findings for the p value.
  • Also, psychology lecturers and students were more confident of p values based on a sample of 100 than on a sample of 10.
  • Methods: Subjects had to state the degree of belief in research findings as a function of associated p levels based on a sample sizes of 100 and 10. Subjects rated each of the 12 p levels (.001 to .90) for each sample size on a six point scale from 0 (i.e. complete absence of confidence or belief) to 5 (i.e. extreme confidence or belief).
  • Results: “Effects due to sample size (S) and p levels (P) were significant in both studies (p < .005) and the groups effect was significant in the replication (p < ,025). In addition, the S X P interaction was significant in the replication (p < .005), indicating that differences in confidence related to sample sizes varied across p levels.” (p.272).
  • Discussion:  There was no significant cliff effect found in intervals following p <  .05, .01, or any other p value.

Abstract

A commentary by Drs Kenneth Beauchamp and Richard May who investigated confidence judgments about research findings for p value.

APA Style Reference

Beauchamp, K. L., & May, R. B. (1964). Replication Report: Interpretation of Levels of Significance by Psychological Researchers. Psychological Reports, 14(1), 272-272. https://doi.org/10.2466/pr0.1964.14.1.272 [ungated]

You may also be interested in




Further evidence for the Cliff Effect in the Interpretation of Levels of Significance (Rosenthal & Gaito, 1964)

Main Takeaways:

  • Importance: one of the first papers dealing with replication in psychology.
  • There was a non-monotonicity decrease of confidence as p values increased.
  • 11 graduate student subjects showed a greater degree of confidence in .05 than that at .03 level. This indicates that p < .05 level has rather special characteristics.

Abstract

A commentary by Drs Robert Rosenthal and John Gaito who investigated confidence judgments about research findings and confidence in the levels of significance.

APA Style Reference

Rosenthal, R., & Gaito, J. (1964). Further Evidence for the Cliff Effect in the Interpretation of Levels of Significance. Psychological Reports, 15(2), 570-570. https://doi.org/10.2466/pr0.1964.15.2.570

You may also be interested in




Ten Simple Rules for Effective Statistical Practice (Kass et al., 2016)

Main Takeaways:

  • The 10 simple rules are:

Abstract

A commentary by Dr Robert Kass providing 10 rules about effective statistical practices and how to improve statistical practices.

APA Style Reference

Kass, R. E., Caffo, B. S., Davidian, M., Meng, X. L., Yu, B., & Reid, N. (2016). Ten Simple Rules for Effective Statistical Practice. Plos Computational Biology, 12(6), e1004961-e1004961. https://doi.org/10.1371/journal.pcbi.1004961

You may also be interested in




Registered Reports: Realigning incentives in scientific publishing (Chambers et al., 2015)

Main Takeaways:

  • Registered Reports allows peer reviews to focus on the quality and rigour of the experimental design instead of ground-breaking results. This should reduce questionable research practices such as selective reporting, post-hoc hypothesising, and low statistical power.
  • Registered reports are reviewed and revised prior to data collection.
  • A cortex editorial sub-team triages submissions within one week: to reject manuscripts; to invite for revision to meet the necessary standards; or to send out for Stage 1 in-depth review.
  • It takes approximately 8-10 weeks for a Stage 1 Registered Report to move from “initial review” to “in-principle acceptance”. This also includes 1-3 rounds of peer reviews.
  • Once the study is completed, it takes 4 weeks for a paper to move from Stage 2 review to final editorial decision.
  • Registered reports are not a one-shot cure for reproducibility problems in science and pose no threat to exploratory analyses.

Abstract

This is a view on registered reports in Cortex by Professor Chris Chambers and colleagues. It contains information on Registered Reports and the length of duration for submission and review. They discuss the editorial process and that a registered report is not a threat to exploratory research and is not a panacea to cure reproducibility problems.

APA Style Reference

Chambers, C. D., Dienes, Z., McIntosh, R. D., Rotshtein, P., & Willmes, K. (2015). Registered reports: realigning incentives in scientific publishing. Cortex, 66, A1-A2.

You may also be interested in




Raise standards for preclinical cancer research (Begley  & Ellis, 2012)

Main Takeaways:

  • Clinical trials in oncology have the highest failure rates.
  • Low success rate is not sustainable or acceptable.
  • Drug development heavily depends on literature.
  • Clinical endpoints are defined in terms of patient survival focus instead of intermediate endpoints e.g. cholesterol levels for statins.
  • It takes years before clinical applicability of the preclinical observation is known. Preclinical observation needs to withstand the challenges and rigorous nature of a clinical trial (e.g. blinding).
  • Claims in a preclinical study needs to be taken at face value.
  • Issue of irreproducible data has been discussed and received greater attention at costs of drug development.
  • Researchers need to contact original authors for mixed findings, exchange reagents and repeat experiments under authors’ direction.
  • In studies for which findings could be reproduced, authors pay close attention to controls, reagents, investigator bias and describing complete dataset.
  • Researchers need commitment and change of prevalent cultures to increase the robustness of published preclinical cancer research.
  • Researchers need to consider negative preclinical data and report all findings, irrespective of the outcome.
  • Funding agencies, reviewers and journal editors should agree negative data is as informative as positive data.
  • There are transparent opportunities for trainees, technicians and colleagues to discuss and report troubling or unethical behaviours without fearing adverse consequences. These should be reinforced and made easier and more general.
  • There needs to be a greater dialogue between physicians, scientists, patient advocates and patients: scientists need to learn about clinical reality, whereas physicians need better knowledge of challenges and limitations of preclinical studies. And both groups would benefit from improved understanding of patients’ concerns.
  • Institutions and committees should give more credit for teaching and mentoring, relying solely on publications for promotion or grant funding can be misleading and does not recognise the valuable contribution of greater mentors, educators and administrators.
  • The Academic system and peer review process encourages erroneous, selective or irreproducible data.

Abstract

Glenn Begley and Lee M. Ellis propose how methods, publications and incentives must change if patients are to benefit.

APA Style Reference

Begley, C. G., & Ellis, L. M. (2012). Raise standards for preclinical cancer research. Nature, 483(7391), 531-533. https://doi.org/10.1038/483531a

You may also be interested in




The cumulative effect of reporting and citation biases on the apparent efficacy of treatment: the case of depression (deVries et al., 2018)

Main Takeaways:

  • The authors analysed the cumulative influence of biases on efficacy and discussed remedies, using the evidence base for two effective treatments for depression: antidepressants and psychotherapy.
  • “Trials that faithfully report non-significant results will yield accurate effect size estimates, but results interpretation can still be positively biased, which may affect apparent efficacy.” (p.2453)
  • A spin occurs when a treatment is concluded to be effective, in spite of the fact that the results on the primary outcome was non-significant (e.g. concluding that treatment X was more effective than placebo, when it should be treatment X was not more effective than the placebo).
  • Positive trials are more likely to be published (cf. publication bias) and significant outcomes are more likely to be included in a published trial, while negative outcomes are changed or removed. Put simply, negative outcomes are reported but in an overly positive manner that makes the negative outcome into a positive outcome (i.e. spin).
  • Negative trials with either positive or mixed abstracts (e.g. concluding that the treatment was effective for one outcome but not another) were cited more often than those with negative abstracts.  These findings indicate that the effects of different biases accumulate to hide non-significant results from view.
  • Peer reviewers have an important role to ensure that important negative studies are cited and that the abstract accurately reports trial results. The peer reviewer can assess the study’s actual results, as opposed to their conclusions, and can conduct independent literature searches, since the authors’ reference list may have studies that disproportionately produce a number of positive findings.

Quote

“Close examination of registries by independent researchers may be necessary for registration to be a truly effective deterrent to study publication and outcome reporting bias. An alternative (or addition) to registration could be publication of study protocols or ‘registered reports’, in which journals accept a study for publication based on the introduction and methods, before the results are known. Widespread adoption of this format might also help to prevent spin, by reducing the pressure that researchers might feel to ‘oversell’ their results to get published. Hence, adoption of registered reports might also reduce citation bias by reducing the tendency for positive studies to be published in higher impact journals.” (p.2455)

Abstract

Dr deVries and colleagues discuss the importance of a spin on clinical trials, citation biases for positive trials and the benefits of registered reports and pre-registration.

APA Style Reference

De Vries, Y. A., Roest, A. M., de Jonge, P., Cuijpers, P., Munafò, M. R., & Bastiaansen, J. A. (2018). The cumulative effect of reporting and citation biases on the apparent efficacy of treatments: the case of depression. Psychological medicine, 48(15), 2453-2455. https://doi.org/10.1017/S0033291718001873

You may also be interested in




Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time (Kaplan & Irvin, 2015)

Main Takeaways:

  • The article investigates whether null results have increased over time in the National Heart, Lung, and Blood Institute.
  • Method: All large randomised controlled trials between 1970-2012 were identified.
  • Method: Two independent searches to improve probability to accurately capture all related trials- one by study author and second by grant databases from 1970-2012.
  • Results: 57% of papers were published prior to 2000 that showed benefit of intervention on primary outcome in comparison to only 2 among 25 (8%) trials published after 2000.
  • Results: Industry co-sponsorship was not linked to benefit but pre-registration linked to null findings.
  • Results: Pre-registration in clinical trials.gov was strongly related with the trend toward null findings.
  • The probability of finding a treatment benefit decreased, as opposed to increased, as studies became more precise.
  • Following the year 2000, file drawer problems became more prominent leading to over-reported positive findings.
  • There is a need to have stricter reporting standards for biases and greater rigour to suppress positive outcomes.

Quote

“All post 2000 trials reported total mortality while total mortality was only reported in about 80% of the pre-2000 trials and many of the early trials were not powered to detect changes in mortality. The effects on total mortality were null for both pooled analyses of trials that were registered or not registered prior to publication (see data in online supplement) In addition, prior to 2000 and the implementation of Clinicaltrials.gov, investigators had the opportunity to change the p level or the directionality of their hypothesis post hoc. Further, they could create composite variables by adding variables together in a way that favored their hypothesis. Preregistration in ClinicalTrials.gov essentially eliminated this possibility.” (p.9).

Abstract

We explore whether the number of null results in large National Heart Lung, and Blood Institute (NHLBI) funded trials has increased over time. We identified all large NHLBI supported RCTs between 1970 and 2012 evaluating drugs or dietary supplements for the treatment or prevention of cardiovascular disease. Trials were included if direct costs >$500,000/year, participants were adult humans, and the primary outcome was cardiovascular risk, disease or death. The 55 trials meeting these criteria were coded for whether they were published prior to or after the year 2000, whether they registered in clinicaltrials.gov prior to publication, used active or placebo comparator, and whether or not the trial had industry co-sponsorship. We tabulated whether the study reported a positive, negative, or null result on the primary outcome variable and for total mortality.17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000 (χ2=12.2,df= 1, p=0.0005). There has been no change in the proportion of trials that compared treatment to placebo versus active comparator. Industry co-sponsorship was unrelated to the probability of reporting a significant benefit. Pre-registration in clinical trials.gov was strongly associated with the trend toward null findings.The number of NHLBI trials reporting positive results declined after the year 2000. Prospective declaration of outcomes in RCTs, and the adoption of transparent reporting standards, as required by clinicaltrials.gov, may have contributed to the trend toward null findings.

APA Style Reference

Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of null effects of large NHLBI clinical trials has increased over time. PloS one, 10(8), e0132382. https://doi.org/10.1371/journal.pone.0132382

You may also be interested in




Publication Pressure and Scientific Misconduct in Medical Scientists (Tijdink et al., 2014)

Main Takeaways:

  • There is increasing evidence that scientific misconduct compromises the credibility of science.
  • There are different definitions and classifications for scientific misconduct: fabrication, falsification, plagiarism. Each of them is seen as fraud.
  • Publication pressure is a risk factor for scientific misconduct but has not been studied.
  • The present study addresses the relationship among publication pressure,  self-reported fraud and questionable research practices.
  • Method: All researchers received a survey and a publication pressure questionnaire that assessed scientific misconduct.
  • Method: 315 Respondents provided demographic information on gender, age, type of specialty; years working as a scientist; appointment status; main professional activity and Hirsch index.
  • Results: 15% of respondents admitted that they had fabricated, falsified and plagiarised or manipulated data.
  • Results: Fraud was more common among younger scientists working in a university hospital.
  • Results: 72% rated publication pressure as too high. Publication pressure was related to scientific misconduct severity score.
  • Discussion: Publication pressure is a psychological stress.  The pressure generated by this stress affects the amount of errors made in scientific research.
  • The data is more suited for the identification of potential determinants for self-reported misconduct than as a measure of the prevalence of misconduct.

Abstract

There is increasing evidence that scientific misconduct is more common than previously thought. Strong emphasis on scientific productivity may increase the sense of publication pressure. We administered a nationwide survey to Flemish biomedical scientists on whether they had engaged in scientific misconduct and whether they had experienced publication pressure. A total of 315 scientists participated in the survey; 15% of the respondents admitted they had fabricated, falsified, plagiarized, or manipulated data in the past 3 years. Fraud was more common among younger scientists working in a university hospital. Furthermore, 72% rated publication pressure as “too high.” Publication pressure was strongly and significantly associated with a composite scientific misconduct severity score.

APA Style Reference

Tijdink, J. K., Verbeke, R., & Smulders, Y. M. (2014). Publication pressure and scientific misconduct in medical scientists. Journal of Empirical Research on Human Research Ethics, 9(5), 64-71. https://doi.org/10.1177/1556264614552421 [ungated]

You may also be interested in




Using science and psychology to improve the dissemination and evaluation of scientific work (Buttliere, 2014)

Main Takeaways:

  • Buttliere advocates that the best way to optimise open science tools would be increasing their utility and lowering its costs and risks by centralizing existing  individual and group efforts. This centralized platform should be easy to use, have a sophisticated public discussion space and impact metrics using the associated data.
  • In order to be competitive, we have to publish in high impact journals. Being competitive can drive science and human progress but when we have questionable research practices, lack of open data and the file drawer problem, it is ineffective.
  • Researchers invest hours to set up their profile, learn the interface and build up their network.
  • Individuals post a paper, dataset, general comment, new protocol and shows up in the newsfeed of the system.
  • Other researchers interact with the post and the system notifies the original poster and displays the content from the same source.
  • If a question a researcher proposes is not found in the discussion of a paper or the subfield, the system could provide a list of experts to answer the question.
  • To increase impact and reduce questionable research practices, we need individuals to engage with prosocial activities.
  • Reviews should be done pre-publication and should privately provide feedback or reviews are made public and serve as a discussion of a certain number of comments.
  • To help science, people should adopt the new system.

Abstract

Here I outline some of what science can tell us about the problems in psychological publishing and how to best address those problems. First, the motivation behind questionable research practices is examined (the desire to get ahead or, at least, not fall behind). Next, behavior modification strategies are discussed, pointing out that reward works better than punishment. Humans are utility seekers and the implementation of current change initiatives is hindered by high initial buy-in costs and insufficient expected utility. Open science tools interested in improving science should team up, to increase utility while lowering the cost and risk associated with engagement. The best way to realign individual and group motives will probably be to create one, centralized, easy to use, platform, with a profile, a feed of targeted science stories based upon previous system interaction, a sophisticated (public) discussion section, and impact metrics which use the associated data. These measures encourage high quality review and other prosocial activities while inhibiting self-serving behavior. Some advantages of centrally digitizing communications are outlined, including ways the data could be used to improve the peer review process. Most generally, it seems that decisions about change design and implementation should be theory and data driven.

APA Style Reference

Buttliere, B. T. (2014). Using science and psychology to improve the dissemination and evaluation of scientific work. Frontiers in computational neuroscience, 8, 82. https://doi.org/10.3389/fncom.2014.00082

You may also be interested in




Bias against research on gender bias (Cislak et al., 2018)  ⌺

Main Takeaways:

  • Scientific inquiries often disregard the moderating roles of sex or gender. Moreover, some finding applies only to male participants, producing biased knowledge.
  • Findings related to men may be irrelevant and harmful to women.
  • Studies on gender bias are often met with lower appreciation in the scientific community compared to studies on race bias.
  • The present study investigated whether research on gender bias is prone to biased evaluation resulting in fewer and less prestigious publications and fewer funding opportunities.
  • The present study compared articles for gender bias and race bias in impact factor and grant support.
  • Method: 1485 articles published in 520 journals were assigned a numerical value based on type of bias. Two peer review criteria were used: Impact factor and whether the article was supported by finding or not.
  • Results: Articles on gender bias are funded less often and published in journals with lower Impact factor than articles on similar instances of social discrimination.
  • Discussion: Results suggest that bias against gender bias research is not merit based but reflects the topic's lower prestige and appreciation due to a generalised gender bias.
  • Another potential explanation for the observed difference in grant funding is the relative difference in availability of participant samples. Recruiting racially diverse samples may be more difficult, time-consuming and costly, while recruiting gender-diverse samples does not have similar issues.
  • It is less plausible, however, that differences in participant samples affect researcher’s decisions of the outlet for their work. It may be that researchers are aware of bias against gender bias research and consider their own work less suitable for more prestigious journals.
  • Research on gender bias is more often reviewed by male researchers than research on race bias.
  • Rejection by more prestigious journals show subtle bias in perceived quality of studies evidencing gender discrimination.

Quote

“This discussion is primarily important in order for gender bias to be properly acknowledged within the scientific community and to pursue further examination of this powerful source of inequality that severely affects many women in the world.” (p. 200)

Abstract

The bias against women in academia is a documented phenomenon that has had detrimental consequences, not only for women, but also for the quality of science. First, gender bias in academia affects female scientists, resulting in their underrepresentation in academic institutions, particularly in higher ranks. The second type of gender bias in science relates to some findings applying only to male participants, which produces biased knowledge. Here, we identify a third potentially powerful source of gender bias in academia: the bias against research on gender bias. In a bibliometric investigation covering a broad range of social sciences, we analyzed published articles on gender bias and race bias and established that articles on gender bias are funded less often and published in journals with a lower Impact Factor than articles on comparable instances of social discrimination. This result suggests the possibility of an underappreciation of the phenomenon of gender bias and related research within the academic community. Addressing this meta-bias is crucial for the further examination of gender inequality, which severely affects many women across the world.

APA Style Reference

Cislak, A., Formanowicz, M., & Saguy, T. (2018). Bias against research on gender bias. Scientometrics, 115(1), 189-200. https://doi.org/10.1007/s11192-018-2667-0

You may also be interested in




When it is fine to fail/ Irreproducibility is not a sign of failure, but an inspiration for fresh ideas (Anon, 2020)

Main Takeaways:

  • The past decade has seen a growing recognition that results must be independently replicated before they can be accepted as true.
  • It is argued that a focus on reproducibility is necessary in the physical sciences as well, although it should be viewed through slightly different lenses.
  • Questions in biomedicine and in the social sciences do not reduce as cleanly to the determination of a fundamental constant of nature as questions in physical sciences. As a result, attempts to reproduce results may include many sources of variability, which are hard to control for.
  • Experimental results of replications may question long-held theories or point to the existence of another theory altogether.
  • It is important to be cautious about assuming something is inherently wrong when researchers cannot reproduce a result when adhering to the best agreed standards.
  • When attempting to reproduce previous results, it helps to build trust and confidence in the research process. Researchers from different domains must talk and share the experiences of reproducibility.

Quote

“Irreproducibility should not automatically be seen as a sign of failure. It can also be an indication that it’s time to rethink our assumptions.” (p.192)

Abstract

The history of metrology holds valuable lessons for initiatives to reproduce results.

APA Style Reference

Anon (2020). It is fine to fail/Irreproducibility is not a sign of failure, but an inspiration for fresh ideas. Nature, 578, 191-192. https://doi.org/10.1038/d41586-020-00380-2

You may also be interested in




Signalling the trustworthiness of science (Jamieson et al., 2020)

Main Takeaways:

  • Authors argue that trust in science increases when scientists abide by the scientific norms.
  • Scientists reinforce trust in science when they promote the use and value of evidence, transparent reporting, self-correction, replication, a culture of critique, and controls for bias”.
  • There are already a number of practical ways scientists and scientific outlets have at their disposal to signal the trustworthiness of science: article badging, checklists, a more extensive withdrawal ontology, identity verification, better forward linking, and greater transparency.
  • The research community has started to thwart human biases and increase trustworthiness of scholarly work.
  • Scientists, policy makers and public base their decisions on inappropriate grounds such as irrational biases, non-scientific beliefs and misdirections by conflicted stakeholders and malicious actors.
  • It is important to communicate the value of scientific practices more explicitly and transparent to clarify misconceptions of science.
  • Scientific advances are built on previous work with new technological revolutions, new areas of research. As a result of these new approaches, interpretations can be corrected and advanced.
  • Central to this progress of science is a culture of critique, replication and independent validation of results, and self correction.
  • Science discourages group-think, countermands, human biases and rewards a dispassionate stance to the subject and institutionalised organised scepticism but fosters competition for scientists to replicate and challenge each other’s work.
  • To validate and build on the results of others, it is important to archive data and analysis plans in publicly available repositories.
  • Retraction statements to allow the issues that led to the retraction to be known and who was responsible for the paper’s shortcomings. If an official investigation commences, it can help the blame be narrowed as opposed to generalised to all authors (cf. CRediT, as it allows us to look and identify the contributor who caused this issue, without blaming all the authors).
  • We should also use a neutral term that encourages vigilance without disincentivizing disclosure such as relevant interest or relevant relationships, as opposed to conflict of interest, to indicate that not all ties are necessarily corrupt.
  • To complement peer review, badges, checklist, plagiarism and image manipulations, independent statistics and verification that authors comply with community endorsed reporting and archiving standards checks are used to signal trustworthiness of findings.
  • Authors organize their thinking in their helpful Table 1, where they describe 3 dimensions (competence, integrity, and benevolence) that communicate the level of trust warranted by an individual study, as well as their associated norms and examples of violation, while fleshing out the role of stakeholders.

Quote

“Science enjoys a relatively high level of public trust. To sustain this valued commodity, in our increasingly polarized age, scientists and the custodians of science would do well to signal to other researchers and to the public and policy makers the ways in which they are safeguarding science’s norms and improving the practices that protect its integrity as a way of knowing...beyond this peer-to-peer communication, the research community and its institutions also can signal to the public and policy makers that the scientific community itself actively protects the trustworthiness of its work.” (p.19235)

Abstract

Trust in science increases when scientists and the outlets certifying their work honor science’s norms. Scientists often fail to signal to other scientists and, perhaps more importantly, the public that these norms are being upheld. They could do so as they generate, certify, and react to each other’s findings: for example, by promoting the use and value of evidence, transparent reporting, self-correction, replication, a culture of critique, and controls for bias. A number of approaches for authors and journals would lead to more effective signals of trustworthiness at the article level. These include article badging, checklists, a more extensive withdrawal ontology, identity verification, better forward linking, and greater transparency.

APA Style Reference

Jamieson, K. H., McNutt, M., Kiermer, V., & Sever, R. (2019). Signaling the trustworthiness of science. Proceedings of the National Academy of Sciences, 116(39), 19231-19236.https://doi.org/10.1073/pnas.1913039116

You may also be interested in




On the reproducibility of meta-analyses: six practical recommendations (Lakens et al., 2014)

Main Takeaways:

  • Researchers on different sides of a scientific argument reach different conclusions in their meta-analyses of the same literature. The article recommends six recommendations that will increase the openness and reproducibility of meta-analyses.

Quote

APA Style Reference

Abstract

Meta-analyses play an important role in cumulative science by combining information across multiple studies and attempting to provide effect size estimates corrected for publication bias. Research on the reproducibility of meta-analyses reveals that errors are common, and the percentage of effect size calculations that cannot be reproduced is much higher than is desirable. Furthermore, the flexibility in inclusion criteria when performing a meta-analysis, combined with the many conflicting conclusions drawn by meta-analyses of the same set of studies performed by different researchers, has led some people to doubt whether meta-analyses can provide objective conclusions. The present article highlights the need to improve the reproducibility of meta-analyses to facilitate the identification of errors, allow researchers to examine the impact of subjective choices such as inclusion criteria, and update the meta-analysis after several years. Reproducibility can be improved by applying standardized reporting guidelines and sharing all meta-analytic data underlying the meta-analysis, including Quote from articles to specify how effect sizes were calculated. Pre-registration of the research protocol (which can be peer-reviewed using novel ‘registered report’ formats) can be used to distinguish a-priori analysis plans from data-driven choices, and reduce the amount of criticism after the results are known. The recommendations put forward in this article aim to improve the reproducibility of meta-analyses. In addition, they have the benefit of “future-proofing” meta-analyses by allowing the shared data to be re-analyzed as new theoretical viewpoints emerge or as novel statistical techniques are developed. Adoption of these practices will lead to increased credibility of meta-analytic conclusions, and facilitate cumulative scientific knowledge.

APA Style Reference

Lakens, D., Hilgard, J., & Staaks, J. (2016). On the reproducibility of meta-analyses: Six practical recommendations. BMC psychology, 4(1), 24.  https://doi.org/10.1186/s40359-016-0126-3

You may also be interested in




Specification Curve: Descriptive and Inferential Statistics on All Reasonable Specifications (Simonsohn et al., 2015) ◈

Main Takeaways:

  • To convert a scientific hypothesis into a testable prediction, researchers make several decisions for data analysis. However, these decisions are affected by implicit decisions such as conflict of interest or trying to publish a result that tells a publishable story.  This article introduces the Specification-Curve Analysis to reduce these problems. The steps include reporting results for “that (1) are consistent with the underlying theory, (2) are expected to be statistically valid, and (3) are not redundant with other specifications in the set.” (p.2).
  • Without specification-curve analysis, researchers selectively report a few specifications in their papers. However, the decisions that are conducted are based on arbitrary analytical decisions, thus specification-curve analysis aims to reduce the influence of arbitrary analytical decisions, while preserving the influence of non-arbitrary analytical decisions.
  • Competent researchers will disagree whether a data analysis is an appropriate test of the hypothesis of interest and/or statistically valid for the data at hand, specification-curve analysis will end debates about what data analysis to be conducted, but facilitate them further (cf. crowdsourcing; Tierney et al. ,2020; in press).
  • There are three main steps for Specification-Curve Analysis. 1. Define the set of reasonable specifications to estimate. Estimate all specifications and report the results in a descriptive specification curve. Finally, conduct joint statistical tests using an inferential specification.  A set of specifications can be produced by enumerating all data analytic decisions that are important to map the scientific hypothesis or construct of interest onto a statistical hypothesis, enumerating all reasonable alternative ways a researcher may make those decisions and generate the combination of decisions, to remove invalid and redundant combinations.
  • Different conclusions from the same data can be interpreted by different researchers based on theoretically justified or statistically valid analyses or may reflect on arbitrary decisions on how the shared views of the researchers are operationalised. Specification allows us to help reach consensus on the latter. To solve this issue, we need to do more or different theory or training, not data analysis.

Quote

“The Specification-Curve Analysis, (i) provides a step-by-step guide to generate the set or reasonable specifications, (ii) aids in the identification of the source of variation in results across specifications via a descriptive specification curve... (iii) and provides a formal joint significance test for the family of alternative specifications, derived from expected distributions under the null...If different valid analyses lead to different conclusions, traditional pre-analysis plans lead researchers to blindly pre-commit to one vs the other conclusion by pre-committing to one vs another valid analysis, while Specification-Curve allows learning what the conclusion hinges on.” (pp.5-6).

Abstract

Empirical results often hinge on data analytic decisions that are simultaneously defensible, arbitrary, and motivated. To mitigate this problem we introduce Specification-Curve Analysis, which consists of three steps: (i) identifying the set of theoretically justified, statistically valid, and non-redundant analytic specifications, (ii) displaying alternative results graphically, allowing the identification of decisions producing different results, and (iii) conducting statistical tests to determine whether as a whole results are inconsistent with the null hypothesis. We illustrate its use by applying it to three published findings. One proves robust, one weak, one not robust at all.

APA Style Reference

Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020). Specification curve analysis. Nature Human Behaviour, 1-7.https://doi.org/10.1038/s41562-020-0912-z [ungated]

You may also be interested in




Registered Reports: A new publishing initiative at Cortex (Chambers, 2013)

Main Takeaways:

  • We value novel and eye-catching findings over genuine findings, thus increasing questionable research practices.
  • Editorial decisions are one cause of questionable research practices, as they make decisions based on results.
  • Science undergraduates are taught about data analysis and hypothesis generation before the data is collected, ensuring the observer is independent of observation.
  • Cortex provides registered reports to allow null results and encourage replication.
  • Registered reports are manuscripts submitted before the experiment begins. This includes the introduction, hypotheses, procedures, analysis pipeline, power analysis, and pilot data, if possible.
  • Following peer review, the article is rejected or accepted in principle for publication, irrespective of the obtained results.
  • Authors have to submit a finalised manuscript for re-review, share raw data, and laboratory logs.
  • Pending quality checks and a sensible interpretation of findings, the manuscript is, in essence, accepted.
  • Registered reports are immune to publication bias and need authors to adhere to pre-approved methodology and analysis pipeline to prevent questionable research practices from being used.
  • A priori power analysis is required and the criteria for a registered report is seen as providing the highest truth value.
  • Registered reports do not exclude exploratory analyses but must be distinguished from the planned analyses.
  • Not all modes of scientific investigation fit registered reports but most will.

Abstract

This is an editorial by Chris Chambers who encouraged Registered Reports in Cortex as a viable initiative to reduce questionable research practices, its benefits, limitations and what information to include in a registered report.

APA Style Reference

Chambers, C. D. (2013). Registered reports: a new publishing initiative at Cortex. Cortex, 49(3), 609-610. https://doi.org/10.1016/j.cortex.2012.12.016 [ungated]

You may also be interested in




Prestige drives epistemic inequality in the diffusion of scientific ideas (Morgan et al., 2018)  ⌺

Main Takeaways:

  • There is no clear evidence that epistemic inequality is driven by non-meritocratic social mechanisms.
  • It remains unknown how an idea spreads in the scientific community.
  • If the origin does shape its scientific discourse, what is the relationship between the intrinsic fitness of the idea and its structural advantage by the prestige of origin?
  • The present study takes a different approach to define how faculty hiring drives epistemic inequality and can determine which researchers are situated in which institutions and the origin of the idea.
  • Method:  5032 tenured or tenure-track faculty data were collected. Data was collected from faculty hiring networks, nodes reflect university and the connections if a PhD was acquired at that university and if they held a tenure-track position.
  • Networks with a self-loop contained individuals who received their PhD at the same institution and held a faculty position.
  • Small departments have high placement power, while large departments have power. Elite institutions have a structural advantage.
  • Faculty hiring may not contribute to the spread of every research idea. Hiring contributes to others. Faculty hiring is a possible mechanism for the diffusion of ideas in academia.
  • The spread of information from a varying level of prestige for universities was investigated.
  • Results: Research from prestigious institutions spreads more quickly and completely than work of similar quality originating from less prestigious institutions.
  • Higher quality research from less prestigious universities has similar success as lower-quality research in more prestigious universities.
  • Even when the assessment of an idea’s quality is objective, idea dissemination in academia is not meritocratic,
  • Researchers at prestigious institutions benefit from structural advantage allowing ideas to be more easily spread throughout the network of institutions and impact discourse of science.
  • Lower quality ideas are overshadowed by comparable ideas from more prestigious institutions, high-quality ideas circulate widely, irrespective of origin.

Abstract

The spread of ideas in the scientific community is often viewed as a competition, in which good ideas spread further because of greater intrinsic fitness, and publication venue and citation counts correlate with importance and impact. However, relatively little is known about how structural factors influence the spread of ideas, and specifically how where an idea originates might influence how it spreads. Here, we investigate the role of faculty hiring networks, which embody the set of researcher transitions from doctoral to faculty institutions, in shaping the spread of ideas in computer science, and the importance of where in the network an idea originates. We consider comprehensive data on the hiring events of 5032 faculty at all 205 Phd.-granting departments of computer science in the U.S. and Canada, and on the timing and titles of 200,476 associated publications. Analyzing five popular research topics, we show empirically that faculty hiring can and does facilitate the spread of ideas in science. Having established such a mechanism, we then analyze its potential consequences using epidemic models to simulate the generic spread of research ideas and quantify the impact of where an idea originates on its long-term diffusion across the network. We find that research from prestigious institutions spreads more quickly and completely than work of similar quality originating from less prestigious institutions. Our analyses establish the theoretical trade-offs between university prestige and the quality of ideas necessary for efficient circulation. Our results establish faculty hiring as an underlying mechanism that drives the persistent epistemic advantage observed for elite institutions, and provide a theoretical lower bound for the impact of structural inequality in shaping the spread of ideas in science.

APA Style Reference

Morgan, A. C., Economou, D. J., Way, S. F., & Clauset, A. (2018). Prestige drives epistemic inequality in the diffusion of scientific ideas. EPJ Data Science, 7(1), 40. https://doi.org/10.1140/epjds/s13688-018-0166-4

You may also be interested in




Open Science Isn’t Always Open to All Scientists (Bahlai et al., 2019) ⌺

Main Takeaways:

  • Open science focuses on accountability and transparency, invites anyone to observe, contribute and create.
  • Open science focuses on conviction that research performs in dialogue with society. Science is a mainstreamed but increasing sense of competition rewards scientists who discover ideas and publish findings. “... science traditionally has rewarded only scientists who are the first to discover ideas and publish findings, there is resistance to move from “closed” practices…”
  • The broad term of Open Science and resulting vague scope is stalling the progress of the open science movement. We are now often caught up in detailed checklists about whether a project is “open” or not, rather than “focusing on the core goal of accountability and transparency.” All or nothing checklists reduce “the accessibility of science and may reify existing inequalities within this profession.”
  • Open science makes science accessible to everyone but there are systemic barriers (e.g. financial and social) that make open science more accessible to some not others such as career stage, power imbalance, employment stability, financial circumstance, country of origin and cultural context.
  • These barriers prevent scientists from pursuing further and should not be used to deny further participation, including receiving grant funding or job applications.
  • “To truly achieve open science’s transformative vision, it must be universally accessible, so that all people have access to the dialogue of science. Accessible in this context means usable by all, with particular emphasis on communities often not served by scientific products.”
  • Open science practices are not equally accessible to all scientists.  aywalls make research inaccessible but Open Access processing fees may prevent scientists from sharing their work, as not all institutions/individuals have the resources to overcome these barriers.
  • If open access is paid out of our personal funds, instead of grant or institution funding sources, it is an unsustainable solution for many scholars that do not have access to these funds.
  • “Yet open tools, code, or data sets are often not valued the same as “normal” academic products, and therefore those who spend their limited time and resources on these products suffer a cost in how they are evaluated for current and future jobs.”
  • Preprints and signed peer reviews may exacerbate inherent biases. “.. forcing transparency in practices that have traditionally operated in a “black box” may exacerbate inherent biases against women and people of color, especially women of color.”
  • Making data available is seen as high risk as someone can publish analyses with your data before you can.  Even “a small risk particularly affects members of the scientific community with fewer resources…”

Quote

“Power imbalance can play a large role in an individual’s ability to convince their research group to use openscience practices and as a result may cause them to not engage in these practices until they have stable employment or are in a senior position.”

Abstract

Current efforts to make research more accessible and transparent can reinforce inequality within STEM professions.

APA Style Reference

Bahlai, C., Bartlett, L. J., Burgio, K. R., Fournier, A., Keiser, C. N., Poisot, T., & Whitney, K. S. (2019). Open science isn’t always open to all scientists. American Scientist, 107(2), 78-82. https://doi.org/10.1511/2019.107.2.78

You may also be interested in




Surviving (thriving) in academia: feminist support networks and women ECRs (Macoun & Miller, 2014) ⌺

Main Takeaways:

  • This paper argues about how peer support networks may affect the experience of early-career scholars
  • Women who participated in the Feminist Reading Group (FRG) are actively intellectually engaged in theorising their own experiences.
  • The group perform functions linked to reading groups, create an informal space concerned with furthering disciplinary knowledge and developing academic skills.
  • FRG members created a community of belonging among themselves, in which personal support, knowledge, and cultural and social capital were provided.
  • Participants share resources and information about institutional processes and gain the confidence to navigate complex and hostile spaces of the University.
  • School’s official spaces are seen as gendered and not reflective of our research interests or intellectual backgrounds.
  • Participants state that FRG allowed them to continue their studies in times of difficulty.
  • FRG provides opportunities to broaden exposure to other fields and improve critical thinking skills.
  • FRG promotes the learn essential academic skills, since women are able to learn from experience with writing and publishing, and also developing presentation and analytical skills without fear of seeming to be an inadequate researcher.
  • Academic work can be isolating and early career researchers frequently report feeling unsettled, anxious and experiencing self-doubt.
  • FRG re-dresses this opacity and operates as an information sharing network for participants to learn about how things work at the University and in the department.
  • Women graduates receive less mentoring, less involvement in professional and social networking than their male peers.
  • Participation in the FRG also stimulated other academic activities, with members encouraging each other to attend conferences and present paper.
  • Most participants were white, straight, cis-gendered and middle class. The group was whiter than our department as a whole.
  • FRG provides participants with an opportunity to understand individual experiences of exclusion, exploitation, self-doubt, discrimination as shared and fundamentally political in character.
  • Our backgrounds and experiences are not homogeneous, most participants in the reading group are racially and socio-economically privileged.

Abstract

In this paper, we reflect upon our experiences and those of our peers as doctoral students and early career researchers in an Australian Political Science department. We seek to explain and understand the diverse ways that participating in an unofficial Feminist Reading Group in our department affected our experiences. We contend that informal peer support networks like reading groups do more than is conventionally assumed, and may provide important avenues for sustaining feminist research in times of austerity, as well as supporting and enabling women and emerging feminist scholars in academia. Participating in the group created a community of belonging and resistance, providing women with personal validation, information and material support, as well as intellectual and political resources to understand and resist our position within the often hostile spaces of the University. While these experiences are specific to our context, time and location, they signal that peer networks may offer critical political resources for responding to the ways that women’s bodies and concerns are marginalised in increasingly competitive and corporatised university environments.

APA Style Reference

Macoun, A., & Miller, D. (2014). Surviving (thriving) in academia: Feminist support networks and women ECRs. Journal of Gender Studies, 23(3), 287-301. https://doi.org/10.1080/09589236.2014.909718

You may also be interested in




Global gender disparities in science (Lariviere et al., 2013) ⌺

Main Takeaways:

  • Gender inequality is still rife in science.
  • There are gender inequalities in hiring, earnings, funding, satisfactions and patenting.
  • Men publish more papers than women. There is no consensus whether gender differences are a result of bias, childbearing or other variables.
  • The present state of quantitative knowledge of gender disparities in science was shaped by anecdotal reports and studies which are localized, monodisciplinary and dated. These studies take little account of changes in scholarly practices.
  • The present study presents a cross-disciplinary bibliographic research to investigate (i) the relationship between gender and academic output, (ii) the extent of collaboration and (iii) the scientific impact of all articles published between 2008 and 2012 and indexed in the Thomson Reuters Web of Science databases
  • Citation disadvantage is highlighted by the fact that women’s publication portfolios are more domestic than male colleagues and profit less from extra citations that international collaborations accrue.
  • Men dominate scientific production in nearly every country (the extent of this domination varies by region).
  • Women account for  fewer than 30% fractionalised authorships, while men representation is such publications was more than 70%.
  • Women are underrepresented when it comes to first authorships.
  • For every article with a female first author, there are nearly two (1.93) articles first-authored by men.
  • Female authorship is more prevalent in countries with lower scientific output.
  • Female collaborations are more domestically oriented than collaborations of males from the same country.
  • The present study analysed prominent author positions (sole, first- and last-authorship). When a woman was in any of these roles, paper attracted fewer citations than in cases wherein a man was in one of these roles.
  • Academic pipeline from junior to senior faculty leaks female scientists. Thus it is likely that many of the trends we observed can be explained by the under-representation of women among the elders of science.
  • Barriers to women in science remain widespread worldwide, despite more than a decade of policies aimed at levelling the playing field. For a country to be scientifically competitive, it needs to maximise its human intellectual capital.
  • Collaboration is one of the main drivers of research output and scientific impact. Programmes fostering international collaboration for female researchers might help to level the playing field.
  • No country can afford to neglect the intellectual contributions of half of its population.

Abstract

Cassidy R. Sugimoto and colleagues present a bibliometric analysis confirming that gender imbalances persist in research output worldwide.

APA Style Reference

Larivière, V., Ni, C., Gingras, Y., Cronin, B., & Sugimoto, C. R. (2013). Bibliometrics: Global gender disparities in science. Nature News, 504(7479), 211. https://doi.org/10.1038/504211a

You may also be interested in




The Pandemic and Gender Inequality in Academia (Kim & Patterson, Jr, 2020)◈ ⌺

Main Takeaways:

  • The COVID-19 pandemic worsened existing gender inequalities across society.
  • The present study investigated the influence of the current pandemic requires addressing an academic publication pipeline best measured in months, if not years.
  • If the pandemic disproportionately influences the productivity of female faculty, the effects on research productivity may not fully materialise for years and evaluation and promotion of female scholars could adversely be affected by gender-related inequalities woven into the system years before.
  • The present study determined the proportion of work- and family-related tweets sent by male and female academics using subject-specific keywords.
  • The pandemic caused the gender-related differences in professional tweeting to increase by 239%.  The lockdown increased the gap between male and female faculty member’s propensity to tweet about family and care-giving.
  • Women bear all care-giving activities- both men and women experienced an increase in family-related tweets- patterns we uncover reveal that female careers are more severely taxed by these commitments.
  • Method: Our sample was narrowed to tenure-track or tenured faculty based in the United States, producing approximately 3000 handles.
  • Method: We first identified all tweets related to career-promoting and family-related activities, and began with terms (e.g. publication, new paper, child care and home school).
  • Each tweet was coded as work- and family-related or not. A more extensive set of keywords classified the entire corpus.
  • Most papers and articles are shared on Twitter via URL, tweet was classified as work-related, if shared, URL address indicates file type, publication venue or data repository services.
  • Results: Faculty members of both genders were affected by the pandemic, the gap in work-related tweets between male and female academics roughly tripled following the work-from-home.
  • Variation in effects between junior and senior faculty indicates this relationship is not driven by an intrinsic gender difference. This effect is produced by gendered differences in adapting a work/life balance to the pandemic.
  • Female academics who reach full professor have overcome existing barriers to gender equality in academia.
  • Parenting obligations overshadow all other factors in limiting research productivity, indicating the influence of parenting on productivity.
  • Increased efforts to address these deep-rooted inequalities, the cracks in the pipeline continue to loom large.
  • Gender imbalances are less pronounced among the ranks of junior faculty, efforts to explain biases in early career trajectories would have the greatest long-term influence on the pipeline of female academics.

Quote

“With gender imbalances less pronounced among the ranks of junior faculty, efforts to account for biases in early career trajectories would have the greatest long-term impact on the pipeline of female academics. Moreover, as female role-models can positively influence young women’s propensities to enter male-dominated fields (Bonneau and Kanthak, 2018; Breda et al., 2020), administrators’ success or failure here could have downstream impacts on female representation in the academy for the next generation.” (p.15)

Abstract

Does the pandemic exacerbate gender inequality in academia? The temporal lag in publication pipeline complicates the effort to determine the extent to which women’s productivity is disproportionately affected by the COVID-19 crisis. We provide real-time evidence by analyzing 1.8 million tweets from approximately 3,000 political scientists, leveraging their use of social media for career advancement. Using automated text analysis and difference-in-differences estimation, we find that while faculty members of both genders were affected by the pandemic, the gap in work-related tweets between male and female academics roughly tripled following work-from-home. We further argue that these effects are likely driven by the increased familial obligations placed on women, as demonstrated by the increase in family-related tweets and the more pronounced effects among junior academics. Our causal evidence on work-family trade-off provides an opportunity for proactive efforts to address gender disparities that may otherwise take years to manifest.

APA Style Reference

Kim, E., & Patterson, S. (2020). The Pandemic and Gender Inequality in Academia. Available at SSRN 3666587. http://dx.doi.org/10.2139/ssrn.3666587

You may also be interested in




Gender in the editorial boards of scientific journals: A study on the current state of the art (Ghasemi et al., 2020) ◈ ⌺

Main Takeaways:

  • There is a large number of studies on gender in academia, gender in membership of editorial boards of scientific journals garner attention of research and little literature.  They make the policies and determine what is accepted for publication and what is not.
  • Admission or rejection of articles influences the academic careers of authors: full professors or PhD students. Gender in editorial boards attracted attention from several researchers, albeit studies focus on journals of a specific field of knowledge.
  • Works dealing with women and academia are addressed, those works focusing on editorial boards are reviewed. Male professors, male authors in journals and male dominance is higher than female counterparts.
  • Women’s receipt of professional awards, prizes and funding increased in the past two decades. Men continue to win a higher proportion of awards and funding for scholarly research than expected based on the nomination pool.
  • Stereotypes about women’s abilities, harsh self-assessment of scientific ability by women than by men; academic and professional climates dissatisfying to women and unconscious bias contribute to achieving fewer awards and funds.
  • Female board representations have improved over time, is consistent across countries, and gendered subdisciplines attract higher female board representations. Inequities persist at the highest level: women are under-represented as editors and on boards of higher ranked journals.  Three factors for women under-representation in editorial board: discipline, journal's prestige and editor’s gender.
  • The last 15 years hinders women’s ability to attain scholarly recognition and advancement and carries risk to the narrow nature and scope of research in the field. They all show a worrying trend of under-representation of women and agree on negative consequences for advancement of science.

Abstract

Gender issues have been studied in a broad range of fields and in many areas of society, including social relations, politics, labour, and also academia. However, gender in the membership of editorial boards of scientific journals is a topic that only recently has started to attract the attention of researchers, and there is little literature on this subject as of today. The objective of this work is to present a study of the current state of editorial boards with regard to gender. The methodology is based on a literature review of gender issues in academia, and more specifically in the incipient field of gender in editorial boards. The main findings of this work, according to the reviewed bibliography, are that women are underrepresented in academic institutions, that this underrepresentation is increasingly marked in higher rank positions in academia and in editorial boards, and that this carries the risk of narrowing the nature and scope of the research in some fields of knowledge.

APA Style Reference

Ghasemi, N. M., Perramon Tornil, X., & Simó Guzmán, P. (2019, March). Gender in the editorial boards of scientific journals: a study on the current state of the art. In Congrés Dones Ciència i Tecnologia 2019: Terrassa, 6 i 7 de març de 2019. http://hdl.handle.net/2117/134267

You may also be interested in




Something’s Got to Give (Flaherty, 2020) ◈ ⌺

Main Takeaways:

  • Women's journal submission rates fell as their caring responsibilities increased due to COVID-19 (see also) based on data from ongoing study of article submissions to preprint databases, whose preliminary results were published in Nature’s Index.
  • Submissions were up since COVID-19, but the share of submissions made by women was down.
  • Submissions by women as first authors (often junior scholars) were especially down, with some indication that they were shifting to middle authors.
  • Female first-author submissions to medRxiv, for example, dropped from 36% in December to 20% in April 2020.
  • Senior and author submissions by women decreased 6% over the same period, while male senior author submissions rose 5%.
  • Other researchers have found COVID-19 related papers in medicine and economics have fewer female authors than expected.
  • At one journal, male authors outnumbered female authors by more than three to one.
  • It was recommended by Melina R. Kibbe, editor of JAMA Surgery, that we should pause the tenure clock during the pandemic. However, critics of this approach have argued this can actually hurt, not help, women and under-represented minorities, as it can delay career progression and decrease lifetime earnings.
  • The status quo is such that men win the COVID-19 game, whereas women, in general, lose. We need to allow part-time work. Different work shifts should be available to those who need them. And agencies should extend grant end dates and allow for increased funding carryover from year to year.

Quote

“In any case, Power said, the challenge “needs more thinking about and a bigger public conversation, because this situation is not going away fast.” That conversation is long overdue, she added, in that “women and carers are supposed to just fit into a system designed for people without caring responsibilities. There is a saying working mothers have: ‘You have to work like you don’t have children and parents like you don’t have a job.’ And that was before COVID-19.”” (p.10).

Abstract

Women's journal submission rates fell as their caring responsibilities jumped due to COVID-19. Without meaningful interventions, the trend is likely to continue.

APA Style Reference

Flaherty, C. (2020, August, 20). Something's Got to Give. Inside Higher Ed. Retrieved fromhttps://www.insidehighered.com/news/2020/08/20/womens-journal-submission-rates-continue-fall

You may also be interested in




Publication metrics and success on the academic job market (Van Dijk et al., 2014)

Main Takeaways:

  • The number of applicants seeking academic positions vastly outnumber the available faculty positions. To date, there has not been a quantitative analysis of which characteristics lead researchers towards becoming a principal investigator (PI). Authors, based on their empirical results, defend that ‘success in academia’ is predictable, depending on number of publications, the journal’s impact factors (IF) of these publications and ratio between citations and IF. In addition, scientist’s gender and the rank of their university are important predictors suggesting that non-publication features play a statistically significant role in the academic hiring process.
  • Method: The authors qualified more than 200 different metrics of publication output for authors who became Principal Investors and those who did not.
  • Method: Whether or not a scientist becomes a scientist depends on the publication record, considering the first few years of publication and effect of each publication feature independent of other confounding variables.
  • Results: Authors with more first-author publications and more papers in high impact journals are more likely to have higher h index and take less time to become principal investigators.
  • Results: The actual number of citations is less predictive than journal impact factor.of becoming a Principal Investigator.
  • Results: Authors with more first or second author publications are more likely to become Principal Investigators. However, if you have a lot of co-authors, less credit is given to this publication.
  • Results: More middle author publications add little value in becoming a Principal Investigator, unless they are published in high impact journals.
  • Results: Authors who take longer than seven years to become a Principal Investigator have more citations per paper than authors who become Principal Investigators more quickly.
  • Results: Men are over-represented as Principal Investigator after correcting for all other publication and non-publication derived features, being male is positively predictive of becoming a Principal Investigator.
  • Quality of publication is given more weight than its actual quality. The number of citations a publication receives is correlated with the impact factor of the journal.
  • The authors found that citations/impact factor is the fourth most predictive feature after impact factor, number of publications and gender.
  • These authors have a two-fold increase in their first-author publication rate relative to authors who do not become Principal Investigator, indicating that more first-author publications per year can compensate for lack of high impact factor publications.
  • The Set of Principal investigator is enriched for scientists who attend higher-ranked universities, linked to many other features. It predicts becoming Principal Investigator independent of other publication features.
  • Scientists from higher-ranked institutions become Principal Investigators before those from lower-ranked institutions.
  • The author suggests that better universities attract better people and produce more Principal investigators.

Quote

“Our results suggest that currently, journal impact factor and academic pedigree are rewarded over the quality of publications, which may dis-incentivize rapid communication of findings, collaboration and interdisciplinary science.” (p.517)

Abstract

The number of applicants vastly outnumbers the available academic faculty positions. What makes a successful academic job market candidate is the subject of much current discussion 1, 2, 3, 4. Yet, so far there has been no quantitative analysis of who becomes a principal investigator (PI). We here use a machine-learning approach to predict who becomes a PI, based on data from over 25,000 scientists in PubMed. We show that success in academia is predictable. It depends on the number of publications, the impact factor (IF) of the journals in which those papers are published, and the number of papers that receive more citations than average for the journal in which they were published (citations/IF). However, both the scientist’s gender and the rank of their university are also of importance, suggesting that non-publication features play a statistically significant role in the academic hiring process. Our model (www.pipredictor.com) allows anyone to calculate their likelihood of becoming a PI.

APA Style Reference

Van Dijk, D., Manor, O., & Carey, L. B. (2014). Publication metrics and success on the academic job market. Current Biology, 24(11), R516-R517. https://doi.org/10.1016/j.cub.2014.04.039

You may also be interested in




Scientists’ Reputations are Based on Getting it Right, not being Right (Ebersole et al., 2016)

Main Takeaways:

  • What happens if my finding does not replicate?
  • The success of replications depends on the methodology used.
  • Many researchers argue that scientists should be evaluated only for things that they control (e.g. hypotheses, design, implementation, analysis and reporting).
  • Scientists produce ideas and insights that drive the discovery of the results, but results are determined by reality.
  • Exciting, innovative results are perceived as better than boring, incremental results.
  • However, certain and reproducible results are better than uncertain and irreproducible results.
  • It is ideal to have innovative and certain results. However, people prefer reproducible and boring findings than exciting but not reproducible results.
  • The authors ask whether we should chase the next exciting findings or should we work to achieve greater certainty via replication and other strategies?
  • How the scientist responds to other individuals’ replications or whether they pursue their own replication are closely tied to the reputation of the scientist.
  • If self-replication failure was reported or if their research failed to be replicated but was pursued with follow-up research, the reputation of the scientist whose work was being replicated increases.
  • A second survey was conducted between researchers and the general population. The same pattern of findings was observed.

Abstract

Replication is vital for increasing precision and accuracy of scientific claims. However, when replications “succeed” or “fail,” they could have reputational consequences for the claim’s originators. Surveys of United States adults (N = 4,786), undergraduates (N = 428), and researchers (N = 313) showed that reputational assessments of scientists were based more on how they pursue knowledge and respond to replication evidence, not whether the initial results were true. When comparing one scientist that produced boring but certain results with another that produced exciting but uncertain results, opinion favored the former despite researchers’ belief in more rewards for the latter. Considering idealized views of scientific practices offers an opportunity to address incentives to reward both innovation and verification.

APA Style Reference

Ebersole, C. R., Axt, J. R., & Nosek, B. A. (2016). Scientists’ reputations are based on getting it right, not being right. PLoS biology, 14(5), e1002460.

You may also be interested in




Rewarding Research Transparency (Gernsbacher, 2018)

Main Takeaways:

  • Reproducing results is an active ingredient of any science.
  • Pre-register studies’ goals and analysis plans make materials and data open to everyone and make reports available to everyone.
  • Taking steps to research transparency takes time and steps are not rewarded.
  • Reward research transparency when hiring, evaluating researchers for academic promotion and tenure and select researchers for society and national awards.
  • We reward during one of the most incentivised phases of academic life: hiring.
  • Departments argue that hiring announcements value transparent research practices and value job candidates who ascribe to transparent research practices.
  • Reward research transparency for people who serve search committees to evaluate job candidates in commitment to research transparency.
  • They can illustrate a commitment to research transparency- describing commitment in their cover letters, creating a research transparency section in research statements, annotating vita to indicate which of their studies are based on pre-registration, open materials, open data and open-access research reports.
  • Job candidates illustrate a commitment to research transparency by asking letter writers to address research transparency activities in a letter of recommendation.
  • If a department and candidates articulate commitment, the department evaluates them according to this commitment, and academic hiring incentivises research transparency.
  • Material deserves not only attribution, proper citations, and acknowledgement during evaluation.
  • Rewarding steps taken for greater research transparency should be based on a value-based metric. cc
  • Departments state their commitment to research transparency in job advertisements and promotion criteria, and candidates illustrate commitment in job applications and promotion dossiers.
  • Changing scientific culture needs top-down leadership, together with bottom-up enthusiasm, institutional commitment, supporting departmental agreement, publication and funding gatekeepers in sync with publication and funding gate knockers and actions, together with words.

Abstract

Cognitive scientists are increasingly enthusiastic about research transparency. However, their enthusiasm could be tempered if the research reward system fails to acknowledge and compensate these efforts. This article suggests ways to reward greater research transparency during academic job searches, academic promotion and tenure evaluations, and society and national award selections.

APA Style Reference

Gernsbacher, M. A. (2018). Rewarding research transparency. Trends in cognitive sciences, 22(11), 953-956. https://doi.org/10.1016/j.tics.2018.07.002

You may also be interested in




Registered Reports: A step change in scientific publishing (Chambers, 2014)

Main Takeaways:

  • Registered reports foster clarity and replication before the experiment is conducted.
  • Study protocols are reviewed before the experiments are conducted.
  • Readers feel more confident that work is replicable with initial study predictions and analysis plans that were independently reviewed.
  • Registered reports are a departure from traditional peer review.
  • Low power, high rate of cherry picking, post-hoc hypothesising, lack of data sharing, journal culture marked by publication bias, and few replication studies, have contributed to the reproducibility crisis.
  • Allows us to publish positive, negative, or null findings, thus producing a true picture of the literature.
  • We will not suffer from publication bias, when a manuscript is worthy of publication, editors and reviewers are driven by the quality of the methods, as opposed to results.
  • Registered reports are not an innovation but closer to restoration-reinvention of publication and peer review mechanisms.
  • Registered reports allow creativity, flexibility and reporting of unexpected findings.

Quote

“Ultimately, it is up to all of us to determine the future of any reform, and if the community continues to support Registered Reports then that future looks promising. Each field that adopts this initiative will be helping to create a scientific literature that is free from publication bias, that celebrates transparency, that welcomes replication as well as novelty, and in which the reported science will be more reproducible.” (p. 3)

Abstract

Professor Chris Chambers, Registered Reports Editor of the Elsevier journal Cortex and one of the concept’s founders, on how the initiative combats publication bias.

APA Style Reference

Chambers, C. (2014). Registered reports: A step change in scientific publishing. Reviewers’ Update. November, 13, 2014. https://www.elsevier.com/reviewers-update/story/innovation-in-publishing/registered-reports-a-step-change-in-scientific-publishing

You may also be interested in




Fast Lane to Slow Science (Frith, 2020)

Main Takeaways:

  • Fast Science is bad for scientists and bad for science.
  • Slow science may help us to make faster progress, how can we slow down? People hardly have time to read the original studies. There is little chance to cultivate broader interests: impairing mental health and well-being of a researcher.
  • We lost talented people resulting in decreased diversity.
  • Fast science cuts corners and contributes to the reproducibility crisis.
  • We could set up a working group-a small conference where practical ideas could be discussed.
  • We must look differently at timescales and consider bigger aims of science. Researchers need to be reminded that we contribute to a human effort that transcends an individual’s lifetime.
  • We work for the sake of truth and for the benefit of society, as they have reason to believe science continuously improves our models of the world.
  • A farsighted vision is important to create and test big theories, irrespective of obstacles.
  • How funders view lengths of grant proposals and intervals for evaluations.
  • Early career researchers believe they need to amass publications and grants. Established researchers assume that grants need to be maintained for their teams and facilities.
  • Researchers need to be encouraged and rewarded for long-term projects that depend on collaborations and may not have a short-term pay-off.
  • We must teach students about the history of science, its noble goals, how it moves forward through failure and success through collaboration and competition.
  • Researchers should actively model thinking pauses.
  • We need to inform researchers about regret and make them aware that in time they may feel similarly. Quality, as opposed to quantity, should be grounds for giving grants, for hiring people and promotion and awards.
  • Quality feels too subjective and tainted by bias stems from being part or wishing to be part of high-status networks.
  • How then do we assess quality, authors can be good judges of their work?
  • Best papers have something new and fascinating to say in a well-argued theoretical framework, are concise and use simple languages.
  • Collaborations are visible and replace the lone genius stereotype.
  • New solutions to big problems can be found more readily when researchers of diverse skills and different viewpoints interact. This is not difficult, first, we need to achieve common ground and language.
  • Need for vigilance to measure reliability and discriminate fact from fake. Engaging with those who bring different perspectives and make us aware of flaws in our theories and experiments. Why not develop a system that allows a listing in the manner of film credits?
  • We need to restrict the number of grants anyone holds at any one time and limit the number of papers published per year.
  • Funders, institutions and publishers regulate an initially voluntary triage to a prearranged number.
  • New models of science communication overcome some problems of traditional journal articles and provide answers to tricky problems of credits.
  • Doing less is better but we need to develop tools to measure quality. It would be exciting to set a goal and have content between those who continue in the fast lane and those who decide to switch lanes.

Abstract

Fast Science is bad for scientists and bad for science. Slow Science may actually help us to make faster progress, but how can we slow down? Here, I offer preliminary suggestions for how we can transition to a healthier and more sustainable research culture.

APA Style Reference

Frith, U. (2020). Fast lane to slow science. Trends in cognitive sciences, 24(1), 1-2. https://doi.org/10.1016/j.tics.2019.10.007

You may also be interested in




Lessons for psychology laboratories from industrial laboratories (Gomez et al., 2017)

Main Takeaways:

  • This proposal does not discuss outright fraud and reaches well-intentioned researchers to produce the best possible scientific work.
  • How can we increase the quality of the data in psychology and cognitive neuroscience laboratories?
  • Behavioural and social scientists are not less ethical than scientists from other disciplines but noisiness of data obtained from human behaviour contributes to these fields’ problems.
  • It is a research ethics imperative to reduce sources of noise in our data by implementing data quality systems.
  • Academic laboratories do not have external controls and scientists rarely get trained in quality systems.
  • Industrial laboratories have a very different culture as quality systems are widely used.
  • High-quality standards are imperative for industrial activities, as there are external forces that cannot be ignored.
  • Junior graduate students mess up times before they adopt their own quality habits and development of formal, explicit and enforceable quality policies would be beneficial for everyone involved, benefits would quickly outweigh costs of developing and enforcing these systems.
  • Reduce waste of resources on failed studies, facilitate the adoption of open science practices and improve the signal to noise ratio in the data.
  • Quality system needs of a group do survey-based studies might be different than the needs of a group collecting neurophysiological data.
  • Quality Assessment should be the responsibility of senior members of the team, as this process is strategic, pre-planned and has a long-term time frame.
  • A more stringent quality system would be to have an external group perform a quality verification audit.
  • A laboratory could be audited by a buddy laboratory from either the same or different institution.
  • Research could have verification badges the same way that some of the open science initiatives provide forms of certification for different levels of openness.

Abstract

In the past decade there has been a lot of attention to the quality of the evidence in experimental psychology and in other social and medical sciences. Some have described the current climate as a ‘crisis of confidence’. We focus on a specific question: how can we increase the quality of the data in psychology and cognitive neuroscience laboratories. Again, the challenges of the field are related to many different issues, but we believe that increasing the quality of the data collection process and the quality of the data per se will be a significant step in the right direction. We suggest that the adoption of quality control systems which parallel the methods used in industrial laboratories might be a way to improve the quality of data. We recommend that administrators incentivize the use of quality systems in academic laboratories.

APA Style Reference

Gomez, P., Anderson, A. R., & Baciero, A. (2017). Lessons for psychology laboratories from industrial laboratories. Research Ethics, 13(3-4), 155-160. https://doi.org/10.1177/1747016117693827

You may also be interested in




Let’s Publish Fewer Papers (Nelson et al., 2012)

Main Takeaways:

  • Authors agree with Nosek and Bar-Anan’s (2012) “Scientific Utopia: I. Opening Scientific Communication” that there is no longer a need for page limits, long lags between acceptance and publication, and prohibitive journal subscription fees, but worry that when all findings are made available, (a) it is harder to discriminate the true findings from the false findings; and (b) there will be more false findings.
  • We know authors file away less successful papers (file drawer problem and effect), leading to publication bias but we also need to focus on the cluttered office effect.
  • In an office full of papers, it is hard to tell good manuscripts from bad papers. Not all researchers should receive equal consideration: “What is less often pictured is the paper that landed in the file drawer, not because of the vagaries of the publication process but because it reports a study that was ill-conceived, poorly run, or generally uninteresting.”
  • The consequence is that the less established researcher is unlikely to be noticed and praised. Researchers seeking top jobs would better to comment on a paper by a famous and eminent researcher. Authors write: “When every paper is available, it becomes increasingly burdensome to find the good papers, and even harder to find the diamond in the rough—the paper that is not by a famous person, not from a famous school, and not in a popular research area.”
  • Advancement, as described by the proposal by Nosek and Bar-Anan’s (2012)  depends on the value of papers being rescued from the file drawer.
  • For every good idea we have, we need to consider many bad ideas.
  • It is a good idea to drop bad ideas before they mature into bad papers. Bad papers are easy to write but difficult to publish.
  • However, it is easy to publish papers now, making it easier to introduce more bad manuscripts
  • Some published papers are false-positives. An occasional false-positive manuscript is bad but lots of false positives are catastrophic to science and the field.
  • False positives are hard to identify and correct.
  • False positives produce severe costs on the scientific community, which is felt more by the field than the individual researcher.
  • We reward researchers heavily for having new and exciting ideas, and less so for being accurate (cf. Ebersole et al., 2016).
  • Researchers are trained to defeat the review process and conquer the publisher.
  • Researchers are rewarded for the quantity of papers and less for the truth value of our shared knowledge.
  • In a system that focuses on one paper per year, researchers can publish a paper on an effect that can be reliably obtained.
  • The researcher would be able to pursue their own work with improved clarity and focus, as there is only one paper to write per year.
  • It would also be easier to evaluate two candidates who differ in quality but are matched in quantity.

Abstract

This commentary is written by Professor Leif Nelson, Professor Jon Simons and Professor Uri Simonsohn about the importance of publishing fewer papers.

APA Style Reference

Nelson, L. D., Simmons, J. P., & Simonsohn, U. (2012). Let's publish fewer papers. Psychological Inquiry, 23(3), 291-293. https://doi.org/10.1080/1047840X.2012.705245

You may also be interested in




Informal Teaching Advice (Bloom, 2020)◈

Main Takeaways:

  • To be an excellent teacher. You need 12 points:
  • 1. Enthusiasm. Act enthusiastic like there’s no place in the world you would rather be. You should enjoy the material more. This will make your audience interested and want to learn more.
  • 2. Be confident. Even if you have practiced this talk 100 times, always act as if the talk has gone smashingly.
  • 3. Mix it up. Throw in some movies, demos and so on to cure boredom and make the talk more interesting.
  • 4. Bring in other people (e.g. guest lectures, interviews and debates). This will introduce variety to your course.
  • 5. Be modest in goals for each class. Do not cram too much material in any single session.
  • 6. Be yourself. Everyone has strengths. Use your strength to your advance and align it with the way you teach.
  • 7. Teaching prep can leech away all the time. Don’t let it.
  • 8. If you say well-timed “Great question. I don’t know but I’ll find out for next class”, it is perceived as charming and makes everyone feel good.
  • 9. Use specific students as examples in arbitrary ways.
  • 10. If a student asks a stupid question, don’t say they are stupid. Always respond with how interesting at a minimum level, no matter how off-topic.
  • 11. Use concrete examples from your own life. They do not necessarily have to be true.
  • 12. If you suffer from anxiety, self-medicate before teaching but do not get addicted.

Abstract

This commentary is written by Professor Paul Bloom about how to make your teaching more engaging with your students.

APA Style Reference

Bloom, P. (2020). Informal Teaching advice. https://www.dropbox.com/s/glm1agnxtz5tbww/informal-teaching-advice.pdf?dl=0

You may also be interested in




The Matthew effect in science funding (Bol et al., 2018)

Main Takeaways:

  • Why is academic success so unequally distributed across scientists? One explanation is the Matthew effect (i.e. a scientist’s past success positively influences success in the future). For example,  if one of two equally bright scientists is given an award, the award winning scholar will have a more successful career than the other equally bright scientist who did not receive an award. Put simply, the Matthew effect undermines meritocracy, by allowing an initially fortunate scientist to self-perpetuate, whereas an equally talented but less fortunate counterpart remains underappreciated.
  • “First, we address the causal inference problem using a regression-discontinuity approach. Second, we systematically study the Matthew effect in science funding... third, we identify a participation mechanism driving the Matthew effect whereby early stage failure inhibits participation in further competition through discouragement and lack of resources.” (p.4887).
  • Method: A single granting program, the Innovation Research Incentives Scheme, is the primary funding source for young Dutch scientists.  This was used to assess the Matthew effect, as it provided a dataset containing all review scores and funding decisions of grant proposals.
  • Method: “We isolate the effects of recent PhDs winning an early career “Veni” grant by comparing the subsequent funding success of nonwinners with evaluation scores just below the threshold to winners with scores just above it.” (p.4888).
  • Results: A scientist with an early career award is 2.5 times more likely to win a mid-career award than those who did not obtain an early-career award. This effect was not due to superior proposal quality or scientific ability but early funding itself.
  • Results: Winning an early-career grant explains 40% of differences in earning between the best and worst applicant and raises long-term prospects of becoming a professor by 47%.
  • The funding of early-career researchers show a Matthew effect, as candidates who won prior awards are evaluated more positively than non-winners, whereas scientists who were successful in obtaining grants select themselves into applicant pools for the following grants at higher rates than unsuccessful researchers.

Quote

“Recent studies have documented rising inequality among scientists across the academic world (38, 39). Not only do our findings suggest that positive feedback in funding may be a key mechanism through which money is increasingly concentrated in the hands of a few extremely successful scholars, but also that the origins of emergent distinction in scientists’ careers may be of an arbitrary nature.” (p.4880)

Abstract

A classic thesis is that scientific achievement exhibits a “Matthew effect”: Scientists who have previously been successful are more likely to succeed again, producing increasing distinction. We investigate to what extent the Matthew effect drives the allocation of research funds. To this end, we assembled a dataset containing all review scores and funding decisions of grant proposals submitted by recent PhDs in a V2 billion granting program. Analyses of review scores reveal that early funding success introduces a growing rift, with winners just above the funding threshold accumulating more than twice as much research funding (€180,000) during the following eight years as nonwinners just below it. We find no evidence that winners’ improved funding chances in subsequent competitions are due to achievements enabled by the preceding grant, which suggests that early funding itself is an asset for acquiring later funding. Surprisingly, however, the emergent funding gap is partly created by applicants, who, after failing to win one grant, apply for another grant less often.

APA Style Reference

Bol, T., de Vaan, M., & van de Rijt, A. (2018). The Matthew effect in science funding. Proceedings of the National Academy of Sciences, 115(19), 4887-4890. https://doi.org/10.1073/pnas.1719557115

You may also be interested in




Minimising Mistakes in Psychological Science (Rouder et al., 2018)

Main Takeaways:

  • The article discusses a few practices to improve the reliability of scientific labs by focusing on what technologies and elements and reduce common and ordinary errors.
  • Common, everyday and ordinary mistakes (e.g. reporting a figure based on incorrect data) can be detrimental to science and everyone has made these mistakes.
  • We need to consider practices in high risk fields where mistakes can have devastating consequences (e.g. healthcare and military). We have organisations that research this type of management and how to reduce these risks through high reliability organisations and the high reliability organisation principles. Should our lab be a high reliability organisation? Yes. Although mistakes in the labs do not have life-or-death consequences, they can produce knowledge waste and can threaten our reputations.  The principles of a high reliable organisation can transfer well to the academic lab setting.
  • The principles of a high reliable organisation are:

Quote

“We have been practicing open science for about two years. It is our view that there are some not-so-obvious benefits that have improved our work as follows: There are many little decisions that people must make in performing research. To the extent that these little decisions tend to go in a preferred direction, they may be thought of as subtle biases. These decisions are often made quickly, sometimes without much thought, and sometimes without awareness that a decision has been made. Being open has changed our awareness of these little decisions. Lab members bring them to the forefront early in the research process where they may be critically examined. One example is that a student brought up outlier detection very early in the process knowing that not only would she have to report her approach, but that others could try different approaches with the same data. Addressing these decisions head on, transparently, and early in the process is an example of how practicing open science improves our own science.” (p.9).

Abstract

Developing and implementing best practices in organizing a lab is challenging, especially in the face of new cultural norms such as the open-science movement. Part of this challenge in today’s landscape is using new technologies such as cloud storage and computer automation. Here we discuss a few practices designed to increase the reliability of scientific labs by focusing on what technologies and elements minimize common, ordinary mistakes. We borrow principles from the Theory of High-Reliability Organizations which has been used to characterize operational practices in high-risk environments such as aviation and healthcare. From these principles, we focus on five elements: 1. implementing a lab culture focused on learning from mistakes; 2. using computer automation in data and meta-data collection wherever possible; 3. standardizing organization strategies; 4. using coded rather than menu-driven analyses; 5. developing expanded documents that record how analyses were performed.

APA Style Reference

Rouder, J. N., Haaf, J. M., & Snyder, H. K. (2019). Minimizing mistakes in psychological science. Advances in Methods and Practices in Psychological Science, 2(1), 3-11. https://doi.org/10.1177/2515245918801915

You may also be interested in




Open Science at Liberal Arts Colleges (Lane et al., 2020)◈

Main Takeaways:

  • Authors offer suggestions on how open science can be fertile when promoted among the faculty that works with undergraduates in classrooms and in labs – i.e., faculty at Small Liberal Arts Colleges (SLACS). Authors also discuss how to use open science to encourage a transformation of the institutional culture and the development of professionals around open science practices.
  • SLACs' primary focus is the exceptionality of its undergraduate education (i.e., the integration of teaching and research defines the SLAC experience by meaningfully incorporating students into research, which is central to the institutional mission).
  • Authors discuss that faculty engaging in open science practices may be hesitant to discuss the replicability crisis because of the worry students will lose trust in the field or are not knowledgeable or qualified enough about open science to teach it. However, authors argue, SLAC has small class sizes and an interactive and educational approach that facilitates productive and robust discussion about open science, thus encouraging critical thinking and well-rounded education.
  • For example, open science can be included in statistics and advanced methods, as pre-registration can be used for students to plan their research question, hypothesis, methods and data analytic procedures before data collection begins.
  • Open science should be studied as part of the liberal arts experience or as general education requirements, as transferable skills can be taught (e.g. framing questions, thinking critically, working collaboratively, grappling with data and communicating clearly).
  • Students can be taught an explicit and transparent account of how discoveries are made, focusing primarily on the process itself, as opposed to the outcome.
  • Pre-registration can be used as a checkpoint during the research process to ensure the students understand their project prior to data collection, especially when SLACS have limited participant pools.
  • Open science will help the faculty at SLACs, as it allows the faculty to not compete in terms of quantity but also focus primarily on the research process, thus allowing for well-designed and robust studies and lines of research.
  • Sharing data and material encourages more productive collaborations with other researchers, current and future generations of undergraduate students, as it allows us to systematically track, organise and share materials and data. This will encourage good practices for future students and will have ready access to materials that were used in a study that was conducted several years ago, saving the research mentor’s time and energy that otherwise would be used to track down analyses, datasets and questionnaires.
  • SIPS: Inclusion is a primary focus on the Society of Improving Psychological Science: to make sure non-PhD granting institutions have a strong voice in its governance. Most projects at SIPS are reviewed with regard to diversity, including type and size of the institution.

Quote

“Sharing materials and data increases the trustworthiness of a research project and makes it possible for others to replicate our work. Sharing data requires greater accountability by researchers, who must demonstrate that they have handled the data properly, used data analysis tools adeptly, and did not overlook potential alternative explanations for their findings. The scrutiny that accompanies open science can begin to feel a little like inviting other researchers to look at how you’ve organized your bedroom closet.” (p.10).

Abstract

Adopting and sustaining open science practices is accompanied by particular opportunities and challenges for faculty at small liberal arts colleges (SLACs). Their predominantly undergraduate student body, small size, limited resources, substantial teaching responsibilities, and focus on intensive faculty-student interactions make it difficult to normalize open science at SLACs. However, given the unique synergy between teaching and research at SLACs, many of these practices are well-suited for work with undergraduate psychology students. In addition, the opportunities for collaboration afforded by the open science community may be especially attractive for those doing research at SLACs. In this paper, we offer suggestions for how open science can further grow and flourish among faculty who work closely with undergraduates, both in classrooms and in labs. We also discuss how to encourage professional development and transform institutional culture around open science practices. Most importantly, this paper serves as an invitation to SLAC psychology faculty to participate in the open science community.

APA Style Reference

Lane, K. A., Le, B., Woodzicka, J. A., Detweiler-Bedell, J., & Detweiler-Bedell, B. (2020, August 23). Open Science at Liberal Arts Colleges. https://doi.org/10.31234/osf.io/437c8

You may also be interested in




How to prove that your therapy is effective, even when it is not: a guideline (Cuijpers & Cristea, 2016)

Main Takeaways:

  • Treatment guidelines use randomised trials to advise professionals to use specific interventions and not others, policymakers and health insurance companies use evidence to indicate whether or not a specific intervention should be adopted and implemented.
  • “If this were your starting position, how could you make sure that the randomised trial you do actually results in positive outcomes that your therapy is indeed effective?” (p.1).
  • In fact, if you attained one important method to optimise the chance, results of a trial are favourable. The placebo effect may lead to an expectation that a therapy works.
  • Advertise your trial in the media as innovative, unique and the best among the available interventions.
  • “Another thing that you have to learn when you want to optimise the effects found for your therapy is that randomised trials have ‘weak spots’, also called ‘risk of bias’.” (p.3)
  • Consider the randomisation of participants – randomisation contributes to the trial – if participants are not randomised in groups, effects could be due to baseline differences between groups, not the intervention.
  • There are two important factors of randomisation: random numbers should be generated – use coin toss for instance or allocation concealment – researchers conduct trial or assistant to assign participants to respond well to intervention to intervention group instead of to control group.
  • Use non-blinded raters of clinical assessment of outcomes to influence outcomes of the trial.
  • Also, there is the issue of attrition for individuals who do not respond to intervention or experience side effects. It does not help them, harm them, so why continue?
  • Ignore attrition in analyses of outcomes and look exclusively at completers, participants who continued should be analysed.
  • Therapy had better outcomes for patients who completed the therapy than individuals who dropped out of the therapy. The correct alternative is to include all participants who are randomised in the final analyses.
  • Include multiple outcomes and analyse results so you can look at which outcome produced the best result and only report them., ignoring the other measures.
  • Published articles can fail to mention trial registration number, not prompting readers to dig up available protocol and check for selective outcome reporting.
  • You can say during presentations – the therapy works better than the existing one and user reports are positive but this is not examined in your manuscript.
  • If you find null effects for this specific intervention, do not publish the findings. If you think this is unethical to the participant and funder, just remind yourself, many other researchers do it, so it is an acceptable strategy!

Quote

“Research on the effects on therapies is no exception to this predicament. Many published research findings were found not to be true when other researchers tried to replicate these finding. When you want to show that your therapy is effective, you can simply wait until a trial is conducted and published that does find positive outcomes. And then you can still claim that your therapy is effective and evidence-based.” (p.6)

Abstract

Suppose you are the developer of a new therapy for a mental health problem or you have several years of experience working with such a therapy, and you would like to prove that it is effective. Randomised trials have become the gold standard to prove that interventions are effective, and they are used by treatment guidelines and policy makers to decide whether or not to adopt, implement or fund a therapy. You would want to do such a randomised trial to get your therapy disseminated, but in reality your clinical experience already showed you that the therapy works. How could you do a trial in order to optimise the chance of finding a positive effect? Methods that can help include a strong allegiance towards the therapy, anything that increases expectations and hope in participants, making use of the weak spots of randomised trials (risk of bias), small sample sizes and waiting list control groups (but not comparisons with existing interventions). And if all that fails one can always not publish the outcomes and wait for positive trials. Several methods are available to help you show that your therapy is effective, even when it is not.

APA Style Reference

Cuijpers, P., & Cristea, I. A. (2016). How to prove that your therapy is effective, even when it is not: a guideline. Epidemiology and Psychiatric Sciences, 25(5), 428-435. https://doi.org/10.1017/S2045796015000864

You may also be interested in




Many Analysts, One Data Set: Making Transparent How Variations in Analytical Choices Affect Results (Silberzahn et al., 2019)

Main Takeaways:

  • The article investigates ‘what if scientific results are highly dependent on subjective decisions at the analysis stage’? It also addresses  the current lack of knowledge on how much diversity in analytic choice there can be when different researchers analyse the same data, as well as whether these variations lead to different conclusions.
  • The study reports the influence of analytical decisions on research findings obtained by 29 teams that analysed the same dataset to answer the same research question.
  • The project had several key stages:

Quote

“The observed results from analyzing a complex data set can be highly contingent on justifiable, but subjective, analytic decisions. Uncertainty in interpreting research results is therefore not just a function of statistical power or the use of questionable research practices; it is also a function of the many reasonable decisions that researchers must make in order to conduct the research. This does not mean that analyzing data and drawing research conclusions is a subjective enterprise with no connection to reality. It does mean that many subjective decisions are part of the research process and can affect the outcomes. The best defense against subjectivity in science is to expose it. Transparency in data, methods, and process gives the rest of the community the opportunity to see the decisions, question them, offer alternatives, and test these alternatives in further research.” (p.354)

Abstract

Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.

APA Style Reference

Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., ... & Carlsson, R. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337-356.https://doi.org/10.1177/2515245917747646

You may also be interested in




ls There a Positive Correlation between Socioeconomic Status and Academic Achievement? (Quagliata, 2008) ◈ ⌺

Main Takeaways:

  • Poverty rates have been increasing together with a debate on socio-economic status. Parental income is an indicator of socio-economic status reflecting a potential for social and economic resources.
  • Parental education is a component of socio-economic status.
  • Learning in a meaningful context so at-risk students can immediately apply when they have learned and connect it to their own lives and individual experiences.
  • Many dropouts are not only from low SES backgrounds but also from mismatched learning styles.
  • SES affects children’s academic achievement. It is beneficial to determine the type of home environment, how educators will best support them at school.
  • Learning environment must be structured to achieve the highest level of internal motivation from all students.
  • School success is greatly determined by a family's socio-economic status. American society may be failing to provide educational opportunities for every student and citizen irrespective of socio-economic background.
  • Many poor students come to school without social and economic benefits available to most middle and high SES students. Sufficient resources for optimal academic achievement irrespective of socio-economic status.
  • The educational system produces an intergenerational cycle of school failures and short change an entire future American society as a result of family socio-economic status.
  • Method: 31 surveys were handed out and 13 were returned.  Some of the answers include health/nutrition; level of IQ; motivation or lack of motivation of teacher; amount of parental support; class size; quality of instruction/teaching resources; support available in home; school; student disabilities; language; education in culture; style of learning exposure to style; gender; peer influence; natural ability; attendance; family loss of tragic event; pregnancy full term; expectations and teacher/student relationship were also considered.
  • Method: Every teacher felt that the environment contributed most when considering academic achievement.
  • Method: Additional variables for socio-economic status were included: attitude; self-confidence; need to please; desire to do better; love of learning; acceptance; economics in the home; stability of family; siblings; age of parent(s); age of student maturity; family involvement; importance placed on learning; cognitive level; family history; neighbourhood; modelling of good work; ethics; pride; choices made; resources available; parental achievement; attending pre-k; home literacy; received early intervention; good nutrition; health; high IQ; oral language development; self-care skills; family life; class dynamics; personality and mood on any given day tells a specific teacher what they can or cannot do on a given day.
  • Results: The higher the socio-economic status, the higher the academic achievement.
  • The current literature is not available as specific students in low socio-economic status homes have high academic achievement.
  • Income, education and occupation are responsible for low academic achievement in many low SES families.
  • Socio-economic status causes less time with children and a result of lower education level of a parent, students from families of higher economic status tend to have parents who read to and with them, parents more apt to talk to them about the world and offer them more cultural experiences, many of the students' struggle with reading comes from low SES and parents that struggle with reading.
  • If a family does not have a good educational background or materials to use to work with their child, the child may suffer as a result of their environment.
  • If education is not valued in the home, students will not value education, more expectation for higher education in higher classes.

Abstract

In this literature review, family environments of low socioeconomic status (SES) students were examined and a comparison made in learning styles between low and high achievers Socioeconomic factors such as family income, education, and occupation play major role in the academic achievement of all students. There is a positive correlation between SES and academic achievement. The conclusions of this review have implications for all educators as well as the entire future of American society.

APA Style Reference

Quagliata, T. (2008). Is there a positive correlation between socioeconomic status and academic achievement?. Paper: Education masters (p. 78). https://fisherpub.sjfc.edu/cgi/viewcontent.cgi?article=1077&context=education_ETD_masters

You may also be interested in




Next