1.

The Credibility Problems in Scientific Publishing

The nature of the scientific publishing credibility problem is in two distinct areas. The first is in the aspect of the highest impact journals, where career pressures and financial programs have created a situation where the intensity of desire to publish is driven by nonscientific goals.¹ There have been high-profile retractions of fraudulent papers, and less acute but equally problematic lack of successes in translating therapeutics when going from basic animal studies to human clinical trials.² The highest profile retractions have been documented by the journals themselves in recent years and publicized on websites,^1,3 with some independent meta-analyses undertaken to understand the scope of the problem. The second distinct problem is at the bottom end of the journal impact range, where there has been enormous growth of journals with questionable credibility, willing to publish low-quality materials for a price with minimal or no real peer review.⁴ These two issues are separate, but both are symptoms of a system that is losing touch with core principles and appropriate driving influences of an open scientific community.

The high impact factor (IF) journal credibility crisis has been a growing and openly debated issue, documented by some of the same leading publishers. A poll of Nature readers indicated that 90% of respondents thought there was a credibility crisis in their own published papers, with 52% indicating it was a significant crisis.¹ This self-reported survey has been backed by scientific meta-analysis studies verifying the magnitude of the problem, but others also argue that it remains in proportion to the number of articles published.⁵ There have been major discussion panels sponsored to try and solve these issues, leading to white papers and proposed methodological solutions.⁶ Ultimately it is still up to each publisher to self-police their credibility. But societal and governmental bodies can get involved when expensive drugs fail to successfully translate from the preclinical or early clinical phase into pivotal trials. These are the most visible failures, and the most costly for companies and funding agencies, where the cost is ultimately to all of society. However, while there are clearly high-profile investigations and retractions from fraudulent research activities, most argue that these are still rare events, and the more insidious concerns are those where authors are accused of selective reporting, low statistical power, lack of self-replication, and insufficient oversight or mentoring of junior scientists doing the work and writing the papers. These factors, combined with pressures to publish for career success, lead to some irreproducible papers that make it through the peer-review system. The problem becomes further amplified by some local institutional or governmental financial systems that provide financial incentives for individual papers being published in high impact factor journals,⁷ thereby prioritizing short-term successes over long-term productivity goals, where quality and longevity of success might be better assessed. The more measured reward system of grant funding and scientist promotions over years provides a balance over multiple or sustained successes, which rewards achievements with demonstrated longevity over quick publication successes, and can take into account a better measure of quality metrics such as citation rates and translation efficacy of discoveries.^8,9

The growth of low-impact journals is also well documented through grassroots efforts to identify so-called “predatory journals,” which have the phenotype of low or absent standards for review.^10–12 There has been a robust grassroots effort to highlight these journals,^13,14 which can be identified by a number of features, such as the lack of an editorial board, lack of transparency in their website and publisher identity, and lack of indexing in known services such as Medline. Still, the number of these journals has ballooned, and while there are many shades of grey in validity, estimates indicate there are nearly 8000 such journals. Most obviously, these journals may not appropriately peer review their published papers. Unfortunately, the scientists who are predisposed to submit to these journals are often the most marginalized, by being located at low-resource countries or institutions, or junior investigators whose guidance and motivations may not be well placed.^11,15 It is critical for the development of these individuals, and for the field itself, to provide guidance on optimal routes for peer review and to help provide mentorship on the appropriate ways to complete and sustain impactful science.

The easiest way for scientists to know if the journal that they are submitting to is appropriate for them is to be aware of the other peer scientists in their field and choose journals that their peers use,¹⁴ and to focus on publishing with their community. This process contributes to the overall development of a network scientists who are working toward competitive success with each other. But beyond the single publishing event, the key linkage from scientist to scientist is the broader idea that scientific research requires open scholarly communication as well as scholarly competition. This is the missing ingredient that must be continually refreshed and renewed in our community, helping the field focus on long-term gains and successes instead of individual publishing events. The transition of the Journal of Biomedical Optics to fully open access and the ability to include supplementary materials with the original manuscript is the right direction for our field, so that all research can be fully understood and reproduced. We need to encourage our community to be open with their work, to ensure reproducibility and transparency in the process of promoting discoveries and developments.

2.

Accurate Measures of Success in Translational Research

The changes that are occurring in the publishing world need attention, but these shifts indicate that attention must also be paid to broader measures of success in scientific discovery, providing a reward system that has the right balance. The discussion is paralleled by a related issue, which is about how to evaluate scientists as they come up for tenure and promotion. Recently, a report by Ioannidis and Khoury¹⁶ proposed a new set of metrics for assessing success over the long term, proposing their PQRST approach, where this acronym represents the five metrics of productivity, quality, reproducibility, sharing, and translation, as outlined in Table 1. This set of metrics works quite well for a translational field such as biomedical optics.

Table 1

The five metrics in the PQRST approach.

These goals for measuring impact are all features of an academic pipeline for a translational researcher, who is seeking to go from publishing through to eventual translation. They form an outstanding way to think about the successes of an individual researcher, a research institute, or a scientific community publication. The steps in this sequence where the publishing world can help is to ensure that reproducibility and sharing are possible. Reproducibility must be part of the assessment of a published paper, with appropriate repeats of data, based upon the known or expected variance in it. Adjudicating this variance and repeatability is a key part of editing and reviewing a paper. Sharing of data, computer codes, materials, and protocols can be enforced by a publisher as a requirement to publish or can be stressed by reviewers and editors. Biomedical optical devices are often produced with unique algorithms, with competitive research intensity focused on making better algorithms than others in the field. In the end, these algorithms should either be freely available or commercialized, such that others can use them to validate the results. Without this level of transparency, it can lead to publications that lack merit or believability. Similarly, if there are unique data sets that would be required to reproduce the work, it seems logical that they need to be made available to others, so that the work can be fairly reviewed or reproduced. The field of biomedical optics should seriously consider these goals, as is happening in other fields.^17,18

The focus on eventual translation in biomedical optics is a goal for most studies, where this can mean translation either to industry development or to human clinical trial testing.^19–21 Ultimately if both are successful, a new optical device can change clinical practice by improving outcomes of the patients. There are many examples of such optical devices throughout surgery, pathology, cardiology, and all aspects of medicine. By ensuring that quality, reproducibility, and sharing are possible, journals can improve the impact of the scientist’s discovery as well as the journal itself.

3.

Translational Science Networks

In the field of biomedical optics, there is extreme geographic and scientific diversity, together with aspects of the work in the disparate areas of academia including medicine and industry,²² and so it can be a challenge to promote collaboration and cooperation in this setting where successes are measured by individual events such as publishing, grant funding, and commercial triumphs. However, the features that benefit individual researchers can be the same ones that benefit the field as a whole, namely strong mentor networks of collaborative researchers, who have a healthy balance of competition and collaboration. Sharing data and algorithms is not necessarily in conflict with individual innovation and successes; rather, focusing on one without the other at all can lead to suspicion of publications and results. In translational science, the local environment is critical—where solid biomedical optics inventions require a three-way blend in academics-medicine-industry—incorporating not just communication but active participation. One of the most tangible features of more integrated collaboration is the ability to efficiently cut off projects that may not be beneficial or have a viable translation conduit. The biomedical optics field today has matured to the point where there is extremely high creativity and a wide diversity of topics; however, the number of discoveries that penetrate out to viable medical devices and procedures still can be a small fraction of this whole activity. Indeed, it is apparent that most clinical optical devices use 40-year-old technologies of lamps, lenses, and glass fibers, because they are reliable and inexpensive. As money and time is spent on translational devices, hard decisions need to be made about what works, incorporating all aspects of technical performance, medical need, and commercial viability.²² If these were all truly folded into every research program, we could likely realize improved money spent on those programs that match the needed goals and possibilities.^19,20 To achieve this, it would be good for developing researchers to have stronger mentorship and advice on where focused efforts were needed. This does not need to displace technical innovation, but rather should complement it in an ongoing dialogue, which is often done in translational centers or in networks program grants.²³ One of the indicators that this is happening to some extent is the documented growth in dual-PI grants at the U.S. National Institutes of Health, as well as industry-academic partnership program funding.²⁴

4.

Initiatives at the Journal of Biomedical Optics

Some argue the problem is that the peer-review system is broken, leading to publications without true peer review and feedback. While this can be true in some fields and for some journals, the strength of the SPIE BiOS community is the enormous size and growth of the field, leading to a strong desire to be part of it. This journal has a large and recently expanded editorial board, which represents the diversity of our field, in gender, geographic submissions, and research topics.²⁵ The long-term impact of papers published here are still the highest of all biomedical optics journals today, and the peer-review system metrics appear strong. Papers published routinely get lengthy feedback, and still most papers that have been accepted and pass quality-control checks to enter the external peer-review system eventually gain publication after productive revisions. This process of a community-focused biomedical optics journal ensures that the individuals in the field assist in the process of making our science stronger.

A recent addition to the journal is the Perspective review series of papers, which provides a conduit for guidance from expert medical professionals to describe their vision for what technologies are needed, to hopefully foster more of a two-way dialogue.²⁶ Those who wish to participate should volunteer, participate in the forum, and seek ways to ensure that we provide strong friendly competition while not losing sight of the need for collaboration and mentoring. Additionally, the BiOS conference itself has become the world’s largest event for biomedical optics researchers to meet and present to each other. While this is just one annual opportunity for exchange, it is a critical piece of our overall community blending together engineering academics, medical professionals, and industry. I urge all involved in the field to work positively toward discoveries and innovations that are scientifically verifiable, translatable, and ultimately impactful to industry and medicine. This journal is one important part of the overall efforts of SPIE and the biomedical optics community in this direction.