NIAMS Centers Evaluation Working Group Report

May 2013

TABLE OF CONTENTS

  1. OVERVIEW & EXECUTIVE SUMMARY
    1. Introduction
    2. Observations
    3. CEWG Conclusions
  2. CEWG: COMMON GOALS
    1. Support Interdisciplinary Science to Promote Innovation That Will Lead to New Approaches to Understanding Disease in Humans
    2. Facilitate Development and Sharing of Research Resources
    3. Accommodate the Research Community’s Need for Shared Clinical Methodologies
    4. Increase Community Awareness of Shared Resources
    5. Foster Mentorship and Training Within Centers

APPENDICES

APPENDIX A: PROCESS & METHODOLOGY

APPENDIX B: MEETING SCHEDULE

APPENDIX C: MEETING/LISTENING SESSION SUMMARIES

  1. Bone Research
  2. Rheumatology Research
  3. Skin Biology Research
  4. Muscle Research
  5. Orthopaedic Research
  6. NIAMS Centers & Career Development of Clinician Scientists

APPENDIX D: REQUEST FOR INFORMATION (RFI)

APPENDIX E: CEWG MEMBERS

APPENDIX F: CLINICAL AND TRANSLATIONAL SCIENCE AWARDS (CTSAs) PROGRAM RECOMMENDATIONS

  1. CTSA (Skin and Rheumatic Diseases)
  2. CTSA (Musculoskeletal Diseases)

APPENDIX G: DATA ANALYSIS TOOLS

1. OVERVIEW & EXECUTIVE SUMMARY

  1. Introduction

    The National Institute of Arthritis and Musculoskeletal and Skin Diseases (NIAMS) supports research in its various mission areas through a range of means. In addition to its support of individual investigators, contracts, and clinical trials, NIAMS funds Center grants. These assemblies of investigators conduct research within the context of shared resources, a research theme, or a combination of both. NIAMS Centers offer a vehicle for integrative investigation in NIAMS mission areas ripe for such an approach.

    NIAMS periodically reviews the programs it funds to ensure that taxpayer dollars are used efficiently to promote the best possible science. In so doing, the Institute seeks input provided by the scientific community toward the optimal development and use of publicly funded programs and resources. NIAMS recently undertook an evaluation of its Centers program to ascertain whether the current configuration of Centers meets the needs of the NIAMS scientific community, given the rapid evolution of biomedicine.

  2. Observations

    NIAMS formed a Centers Evaluation Working Group (CEWG) to advise the Institute as to how its Centers program could be more responsive to current research needs and opportunities (see Appendix A: Process & Methodology). Concurrently, in September 2012, NIAMS began conducting listening sessions with investigators representing the Institute’s main mission-related research communities: skin, muscle, bone, rheumatology, and orthopaedics. As the listening sessions took place, consistent messages emerged that cut across the different disciplines. The CEWG undertook the task of assembling the themes expressed at the listening sessions and other inputs from the community into a coherent set of goals that would transform the current NIAMS Centers framework. In this report, the CEWG attempts to convey to NIAMS the needs and opportunities identified by investigators and patient groups within the NIAMS mission areas. The five Common Goals are listed below; a more in-depth exploration of each follows in the Common Goals section.

    • Foster interdisciplinary science to promote innovation that will lead to new approaches to understanding disease in humans
    • Facilitate sharing of research resources
    • Accommodate the research community’s need for shared clinical methodologies
    • Increase community awareness of shared resources
    • Foster mentorship and training within Centers
  3. CEWG Conclusions

    The CEWG concluded that modern biomedicine requires that diverse disciplines interact within and across academic and other institutions to cross traditional boundaries and achieve a higher level of functioning to address human diseases in an integrative and synergistic fashion. To meet this need, NIAMS should support both resource-only Centers that drive technical innovation and Centers that conduct research (some specialized resources may be developed with funding mechanisms other than Center support). Centers should facilitate collaborations that interrogate Big Data and employ innovative technology, and they should also facilitate access to as many patient samples and biological specimens as possible. The CEWG concluded that NIAMS should allow flexibility and dynamism in the design, structure, and conduct of its Centers, to accommodate the variable needs of NIAMS research areas that differ with respect to investigator community, resource availability, and knowledge depth and breadth.

    As a result of its discussions and deliberations, the CEWG drew two overarching conclusions:

    • NIAMS Centers should support the conduct of interdisciplinary research, in which a team of investigators pools their expertise in different disciplines to create a new, integrated approach to understanding human disease (as distinct from a multi-disciplinary approach, in which investigators with different expertise each focus on different aspects of a shared problem).
    • NIAMS Centers should prioritize improving access to resources, using as review criteria i) the importance of the resource and ii) the potential impact of providing access to that resource.

2. CEWG: COMMON GOALS

  1. Support Interdisciplinary Science to Promote Innovation that Will Lead to New Approaches to Understanding Disease in Humans

    NIAMS Centers should be vehicles for encouraging interdisciplinary science. Many advances in basic and clinical science that result in breakthroughs in the prevention and treatment of human diseases come from the creation of connections between diverse disciplines. Collaboration across disciplines requires the assembly of investigators normally embedded in a specific scientific discipline and its associated culture. The encouragement and organization of cross-disciplinary work often leads to plans that do not fit into the NIH R01 format, primarily because of the size of the needed enterprise and the trans-institutional nature of such collaborations. Importantly, research supported by Centers goes beyond what might be achieved through the funding of R01 grants.

    1. Interdisciplinary mindset benefits basic, translational, and clinical research

      The key goal of a Center is to have interdisciplinary teams develop a new approach to the study of a disease. For investigators pursuing fundamental basic research questions, rapid advances often come from the collaboration of investigators from multiple disciplines as diverse as mathematics, bioengineering, bioinformatics, epidemiology, and the medical sciences. History has shown that bringing together even a small number of investigators with diverse perspectives to address a common problem can be an engine for innovation. Furthermore, bringing together the perspectives of both basic and clinical scientists can inform the formulation of questions and priorities in a different manner from what might be formulated by either group alone. Therefore, NIAMS Centers, with their opportunities for shared resources and discovery research, should emphasize and promote the participation of basic and clinical scientists of diverse backgrounds and perspectives. Both face-to-face meetings and virtual gatherings of investigators who are more geographically dispersed have value, and thus both approaches should be promoted by Centers.

      Synergistic interactions among investigators should be a central goal for Centers that provide support for cores alone, as well as Centers that directly fund research projects. Integration around a common theme, whether from the design of cores or from investigators’ interests, would be the most obvious way to accomplish this goal, but investigators should have flexibility as to how they structure interactions amongst themselves to produce synergies. A shared theme can be built in different ways. For example, a shared resource Center might include cores that are integrated (e.g., around a disease-related registry and biorepository) or it might include a group of investigators who are integrated in working on a common research theme, but require a diverse set of cores that may not otherwise be related. Indeed, innovative approaches to promoting synergies should be a goal of every NIAMS Center. Integration can be achieved in different ways in Centers that provide direct support for research, such as based on shared resources (e.g., informatics, genomics, repository, outcomes); clustered around a disease-related topic (e.g., rheumatic, musculoskeletal, or skin diseases); or a shared approach (e.g., interdisciplinary systems biology).

      Successful Centers typically develop high visibility within a host institution, serving as a focal point for investigators and ideas. The CEWG heard from participants at several listening sessions about the importance of NIAMS Centers in contributing to the development of new collaborative research projects that were successful in obtaining independent grant funding, as well as the development of new research technologies. Pilot projects, in the context of team science across disciplines, can be a very useful vehicle for bringing together scientists of diverse backgrounds related to a common theme. Broadly announced requests for proposals that encourage established scientists to pursue a new area of study, or that invite new investigators to broaden their horizons, can expand the talent pool of investigators willing to undertake interdisciplinary science. The availability of well-designed core facilities and vigorous pilot and mentoring programs within Centers can expand the vision of NIAMS-funded scientists. When a pilot project program takes place in the context of core facilities that can make this transfer of skills possible, the barriers to interdisciplinary science will fall. Mentoring programs can further ease the challenges of testing new approaches and can help establish and expand teams that might not have existed at the time of the Center’s establishment (see Goal E).

    2. Understanding disease in humans

      Applications for NIAMS Center grants should be judged on their focus on human pathobiology, in particular their ability to provide new approaches to understanding the pathogenesis and outcomes of disease in humans. Analysis of tissue samples or groups of patients with NIAMS-related diseases and conditions will be the most direct avenue for understanding human diseases. In some cases, an alternate or supplemental research approach might employ an animal or other model that closely replicates a human disease. The choice of model will vary by disease. For example, in immune-mediated diseases, mouse models generally do not mirror human disease as a whole, while in bone research, single-gene defects may accurately reproduce human pathology.

      NIAMS Center application review panels should be formed such that the majority of reviewers are able to judge the potential value of a proposed Center to advance understanding of human pathobiology, and they should be explicitly charged to do so. As needed, review panels could also include individuals with the expertise to evaluate proposed non-human models. This approach acknowledges that, in some instances, studies in models will be relevant and more controlled. However, although clinical research studies are inherently less controlled — and of necessity, higher risk — studies in humans provide a more direct and ultimately essential path to understanding human disease.

      In welcoming a diversity of scientific approaches, the consensus of the CEWG was that NIAMS should not segregate research Centers into clinical and non-clinical subtypes. Optimally, Centers should be segregated by field (e.g., skin biology) for the purpose of review, because reviewers from the field could best determine the potential impact of an application on understanding disease. In this case, further segregation by clinical and non-clinical research would create too many subsets. Given the difficulties of comparing clinical and non-clinical study applications (some members of the CEWG were not convinced that the two types could be compared fairly), review outcomes should be monitored to ensure that Center applications proposing studies in humans are not disadvantaged by the need to compete with research employing non-human models.

    3. Flexibility in Center design, structure, and conduct facilitates innovation and collaboration

      The research community stated that NIAMS Center mechanisms should not constrain investigators in how they put their teams together or by what means they approach a question. Rather, the CEWG recognizes the need for NIAMS to ensure that funding mechanisms provide applicants with as much flexibility as possible to create networks and approaches that maximize innovation while maintaining scientific rigor. Center funding strategies should encourage and allow varied collaborations, including community organizations, scientific groups, or other novel, productive partnerships. Importantly, Center funding mechanisms should have the flexibility to include both non-clinical and clinical investigators. Collaborative efforts should also cut across disciplines and combine topics that bridge different areas of research.

      Rather than emphasizing specific structures, the goals for Centers outlined in this report should be established as review criteria with sufficient rigor such that review panels can make meaningful comparisons between applications. In parallel, it is essential that reviewers of Center grants comply by lessening the emphasis on process and enhancing the emphasis on proposed hypotheses and scientific rationale in applications. Centers should compete in peer review on the basis of a clear plan for promoting synergy among investigators and the likely impact of developing new approaches to understanding diseases in humans.

      Non-NIH examples of alternate models include those developed by the National Science Foundation (NSF) and the Defense Advanced Research Projects Agency (DARPA). NSF funds Engineering Research Centers (ERCs) and Science and Technology Centers. Its program announcements state broad goals, including development and training, and thus provide a potential model for NIAMS Centers. "Third Generation" ERCs call for a symbiotic relationship between ERC researchers, other investigators, partner organizations, and private sector entities. ERCs must create or enhance capacities emerging from transformational fundamental research toward the development of a financially sustainable resource. In the case of DARPA, staff and scientists work in a collaborative manner to formulate very specific topic areas without specifying how the topics should be approached. The DARPA approach relies on well-defined, broad parameters that frame a potential project’s impact, limitations, costs, and benefits.

    4. Bioinformatics and systems biology

      Integrating multiscale analytic technologies requires bioinformatics capabilities and systems biology expertise that builds quantitative models to predict phenotypic behaviors of networks of cells as a function of the extracellular communication signals and resulting intracellular signal transduction networks. Chronic diseases within the NIAMS mission that affect large numbers of people, whether inflammatory, autoimmune or degenerative, are mainly multifactorial diseases. These conditions result from the interaction of multiple genes and environmental influences, leading to activation of numerous downstream pathways through as yet unidentified mechanisms. They may also have variable etiologies in different subsets of patients. Adding further complexity, these conditions often involve dynamic networks of both immune and tissue cells that change with disease progression.

      This situation argues for the need to acquire highly multiplexed data at a greater hierarchy of information flow: across cell networks, cytokine networks, and intracellular signaling networks, stratified by segmentation of subsets of disease phenotypes and patient characteristics. An obvious challenge for the next decade is to understand causative signaling pathways in these complex diseases, which will provide targets for prevention or intervention or prevention. Many study methods are available to gather data in a system-wide fashion. Genome-wide association studies (GWAS) provide clues about genes (or regulatory regions) that are involved in pathogenesis, but these approaches yield little insight about how specific genetic loci contribute to disease. Encyclopedia of DNA Elements (ENCODE) analyses allow investigators to link polymorphisms, particularly in regulatory regions, to the expression of different genes in different cell types. Epigenetics can provide an additional link to the role(s) of the environment.

      Interdisciplinary teams at NIAMS Centers are ideally suited to plan and execute systems biology investigations. Importantly, systems biologists should be involved in framing the questions from the outset of a study. Even simple preliminary models of networks can provide a guide to where measurements might be made most strategically and help define the amount of data that must be collected. Systems biology expertise should also be centrally involved in designing collection methods so that samples are suitable for answering questions at multiple scales. Promoting systems thinking within NIAMS Centers will foster the spread of knowledge across research areas and Centers, and broadening the benefit for the diverse scientific community that NIAMS serves. To that end, careful coordination and leveraging of existing bioinformatics and computational resources among Centers, institutions, and other programs will maximize the federal investment in biomedical research.

  2. Facilitate Development and Sharing of Research Resources

    In addition to promoting environments that support interdisciplinary investigation (Goal A), NIAMS Centers should prioritize access to shared resources. The past decade has seen dramatic improvements in the ability of institutions to collaborate both intellectually and physically to achieve common goals. It has become increasingly practical for Centers that are geographically dispersed to create and promote scientifically integrated resources on a national scale. Fuller integration of resources and researchers sets the stage for a stronger scientific environment, and exposure to such an environment is helpful to trainees.

    1. NIAMS resource-focused Centers

      Core-only Centers should emphasize improved access to resources. Resources and investigators may be distributed at different institutions, particularly for technical resources that do not need to be duplicated at every institution. Shared-resource cores could also provide access to unique expertise or services, such as epidemiology, outcomes, genetics, or informatics, which may be necessary to address a common research problem. Resource-focused Centers should be reviewed on i) the basis of the importance of the resource(s) to be shared, ii) the science that can be expected from the community of investigators who are provided access, and iii) the Center’s potential ability to improve integration of that community. Benefits provided by Centers should go far beyond what could be provided on a fee-for-service basis alone: Centers should foster collaborations and technical development, as well as attract both new investigators and established investigators from various fields to NIAMS research areas. One way to facilitate outreach to other investigators could be to establish a research theme, but other approaches could also be employed. Similarly, openness to participation of investigators at other institutions would be one way to demonstrate a broader impact, but such a strategy would not be required.

    2. Development of new technologies and resources

      NIAMS resource-focused centers should be engines for the development of new technologies and methodologies. A clear benefit of interdisciplinary assemblies of scientists is the novel technology that grows from joint efforts that cross traditional disciplinary boundaries. Development of new technologies often involves engineers, mathematicians, physicists, and other quantitative specialists working in combination with individuals who are knowledgeable about human disease pathogenesis. Development of new methodologies often requires experts in epidemiology, bioinformatics, mathematics, and other disciplines working with clinical or outcomes researchers.

      NIAMS Centers should provide funding for technical development using a model similar to the Pilot & Feasibility Research Program. A limited set of funds should be set aside to promote new technologies that arise within a Center’s grant funding period. Like Pilot & Feasibility studies, specific details would not be reviewed, but the process for selection of the resource would be reviewed, as would outcomes at the time of competitive renewal. If Center funds are insufficient and additional support is necessary, the availability of institutional or other resources would be a plus.

      On occasion, highly unique resources are developed by investigators at a particular institution, and these resources are useful to investigators working in NIAMS mission areas across the nation. Supporting the development and dissemination of such resources may be best achieved through stand-alone cores not associated with other Center-related resources. Such single-resource facilities could serve as a mechanism to develop preliminary data using a new technology, while focusing on promoting awareness of the resource’s availability to the larger investigator community. NIAMS may wish to consider supporting the development of these unique resource facilities via non-renewable grants (perhaps through mechanisms other than Centers), and expect applicants to develop a post-grant period funding plan.

    3. Leveraging other research resources

      While Centers are one example of how interdisciplinary research can flourish, many institutions have their own programs and structures in place to foster the mixing of expertise and resources among diverse disciplines. NIAMS Centers should leverage institutional resources (e.g., NIH Clinical and Translational Science Awards (CTSA or similar organizations) that would be useful to Center investigators (for example, see bioinformatics resources, Goal A). Center applicants should describe institutional resources, other than those that receive funding from the Center, that are available to members of the Center. Such resources might include those supported by institutional funds or where a fee-for-service model is appropriate. Funds from NIAMS Centers can be used to provide additional support for existing institutional resources. The applicant should make it clear how this additional support benefits NIAMS investigators beyond what could be provided by user fees. Use of a consolidated facility that blends NIAMS and other resources should be encouraged in cases where such a resource can provide economies of scale, a wider range of techniques, and greater expertise than a stand-alone facility used only by NIAMS Center members.

    4. Unique resource needs of translational and clinical research communities

      Progress toward deciphering disease mechanisms will depend in part on the ability of diverse investigators to access appropriately consented patient samples in order to test specific hypotheses. A recurring theme from the listening sessions was that NIAMS, through Center funding, should provide critical infrastructure to support the use of accessible cohorts of patients and at-risk individuals " developed at a single institution or through multicenter operations " in a manner that facilitates and advances patient-oriented scientific research in NIAMS mission areas. Availability of such cohorts, biological samples, and associated information enables investigators from a range of disciplines to address questions that are not otherwise possible. In particular, the listening sessions produced a clear call for improved access to well-annotated clinical materials that would enable the study of novel hypotheses among integrated groups of investigators.

      Resource-sharing also ensures that NIAMS fulfills its responsibility to maximize the use of resources generously contributed by patients. While NIAMS Centers that support collections of patient-derived material would be primarily designed to conduct studies of pathogenesis and outcomes, they could also provide critical resources to assemble potential subsets of patients for additional approaches, such as accelerated clinical studies or interventional clinical trials of interest to the NIAMS research community (funded through other means). Such clinical resources would be useful for clinical studies of rare diseases and conditions that are hard to diagnose or characterize due to a lack of molecular diagnostic markers. Other potential applications include increasing participation in trials by underserved communities often neglected in traditional clinical trials, as well as enhancing recruitment for diseases without (m)any FDA-approved drugs. Finally, wider access to cohort registries could help advance personalized medicine by linking detailed genotype/phenotype information to help select specific individuals who have certain biomarkers, immune-phenotypes, or genotypes for participation in trials or clinical studies.

      Flexibility and creativity will be important to maximize the availability and use of clinical resources. For example, a Center would not be the best mechanism for developing de novo cohorts, registries or biorepositories; other sources of funding would be more appropriate. Possibilities include leveraging scientific foundation or patient-advocacy groups, accessing resources funded by other NIH components, cyber-based recruiting, and/or allowing investigators to contact registry participants to provide samples for new studies. NIAMS Center resources could be used to expand (or potentially repurpose) and to market and distribute existing clinical resources for use by a broader base of investigators.

      Consortia and patient collections are best built with a specific research question or clinical need in mind. NIAMS Centers should optimize and standardize collection methodologies to address relevant, critical, specific hypotheses, but they should remain sufficiently flexible to adapt to new questions and areas of investigation. Potential strategies include allowing for patient re-contact and re-consent, as well as facilitating standardized, documented sample collection and storage for future use.

      A key role played by NIAMS Centers is to provide a consortium of investigators that develop common platforms and collection tools for broad use across diseases and research areas. Developing standard data dictionaries and ontologies allows for data harmonization across datasets and simplifies the process of accessing, merging, and analyzing datasets. These steps also facilitate resource sharing among investigators at different institutions nationwide.

    5. Field-specific and/or unique resource needs

      Currently, there is a lack of multi-institutional technology, infrastructure, and standardization for interrogating at-risk target tissues involved in diseases in which NIAMS has a research interest. NIAMS Centers " perhaps a network of virtual entities " may provide a vehicle for the creation, coordination, and sharing of such technologies that include advanced imaging techniques, deep-sequencing methods, genetic counseling and preventive interventions, as well as functional validation of identified genetic loci. Ideally, such arrangements would foster rich collaborations between clinicians and basic scientists and allow for efficient and thorough investigation of pathological specimens toward understanding mechanistic detail. As noted previously in this section, single-resource facilities could serve as a mechanism to develop preliminary data using new technologies, while focusing on promoting awareness of resource availability to the larger investigator community.

  3. Accommodate the Research Community’s Need for Shared Clinical Methodologies

    The NIAMS Multidisciplinary Clinical Research Centers (MCRC) were established in 2001 to promote clinical, epidemiological, and health services research. Each MCRC consists of a Methodology Core and separate research projects. The MCRC have been successful in fostering the use of advanced methodologies to design clinical studies and perform statistical analyses in NIAMS mission areas, resulting in ground-breaking clinical research. Indeed, the areas of study supported by MCRC are now thriving and in less need of a direct stimulus. Furthermore, despite the ongoing importance of these areas of research, more recently the connectivity between MCRC Methodology Cores and research projects has been variable. NIAMS has perceived a problem in applications for MCRC funding in the dissociation in quality between proposed Methodology Cores and research projects, at least as judged by peer review.

    The CEWG observed that Methodology Cores continue to play an important role in fostering clinical research and collaborations. A critical mass of methodologists, working alongside subject-matter experts, can uniquely create a culture of research that enhances rigor, promotes creativity and discovery, and fosters training. Methodology Cores create an environment of sophisticated study design and analysis for the investigation of NIAMS-related diseases, and they also play an important role as the glue that binds together collaborative groups of clinical and epidemiological researchers within an institution. Despite differences of opinion among the CEWG membership, a majority concluded that the Methodology Core should be preserved as a component of NIAMS Centers. The types of research for which the MCRC was originally intended remain vitally important, but the CEWG recognized that the need for specialized Centers to support only this type of research is less clear. The CEWG concluded that a Center that supported a Methodology Core but not research projects would be soulless. A compromise supported by the majority was to include a research Pilot & Feasibility program in an NIAMS Center along with the Methodology Core. With the support of the Methodology Core and Pilot & Feasibility projects, investigators would be able to develop successful applications for independent funding. However, the CEWG also acknowledges that NIAMS should preserve investigators’ flexibility in the design of Centers that include Methodology Cores to best accommodate the needs and opportunities of their respective institutions.

    As with all NIAMS-funded projects, Methodology Cores should avoid duplicating existing resources and structures already available to NIAMS-funded investigators, such as those offered through CTSA or similar institutional resources. Cores to provide methodological support could be included independently in other types of Centers that support resource cores or research projects.

  4. Increase Community Awareness of Shared Resources

    1. Incentives for resource creators and users

      Beyond facilitating collaboration, broad sharing of resources — not just locally but potentially nationwide — is an important component of the responsible stewardship of federal funds. Particularly for patient-derived samples, which are valuable yet scarce, wise management and leveraging of resources is an important goal of NIAMS Centers. Increasingly, many academic-promotion models employ team-science metrics; these models should create incentives for the sharing of core resources.

      During listening sessions, the CEWG heard repeatedly about the inherent challenges of promoting and ensuring outside investigators’ access to clinical cohorts, biospecimens, and technology resources, all of which are extremely valuable but difficult to replicate. Several realities challenge the ease of sharing resources on a scale that would interest most investigators. First, there are financial and academic risks incurred when creators share their resources. Second, many resources are difficult to transport, such as sophisticated technologies, animals, and biospecimens. As such, users sometimes find it is easier to create their own local resource, often through separate funding, instead of trying to leverage existing resources created elsewhere. Third, "marketing" the availability of resources is difficult, since it is not always obvious how to connect with potential end users, especially those not directly related to NIAMS Center activities. Fourth, specific groups, such as basic/translational scientists seeking clinical samples and early-stage investigators without well-developed collaborations in place encounter challenges in finding appropriate resources.

      One metric is the extent to which resources developed in an NIAMS Center are used by a community of scientists. The community served may be local or national; in some circumstances, intensive use by a smaller community may have a greater impact than superficial use by a larger group. Nevertheless, a resource that is made available to a larger community is likely to have a greater impact than a resource that is restricted in its dissemination. However, achieving meaningful use of a particular resource by a community of investigators, particularly if widely distributed, requires awareness of the resource. Whether the community of investigators is located within a Center or broadly distributed, Centers should develop strategies for improving awareness of the resource throughout that community. Prioritizing resource access to certain user groups, including young investigators, can help ensure fair distribution of limited materials.

      Effective marketing of resources can be facilitated with online "push" technologies, social media, and networking platforms that both cast a wide net and are user-friendly. NIAMS Centers could build a common communication platform with the ability to "learn" investigator interests, and then match these interests with potential cores and resources. Raising awareness of resources will benefit both the creators and the users of the resources.

    2. Harmonization of resource collection, management, and dissemination

      The CEWG recognizes that well-intended efforts to standardize processes and methods can have the unintended effect of hampering innovation and discovery. Currently, there is no incentive for different groups of investigators to agree to shared standards for collecting general clinical data (e.g., family history, environmental exposures) nor to develop shared methods of biospecimen processing and handling. Thus, there is significant diversity in how each NIAMS core/Center currently functions, meaning that it is not always feasible to share resources because of the variability in how samples are obtained and processed, the multiplicity of diseases assessed, and the different approaches to clinical phenotyping and defining data.

      Despite the inherent challenges in gathering biospecimens and clinical data, some steps can facilitate broader usability of NIAMS Center-created resources. At a minimum, Center investigators should harmonize clinical data and specifications for tissue or other clinical material during sample/data collection. Doing so would include creating clear data definitions to allow mapping across clinical datasets or cross-walking between different biospecimen-processing methodologies to facilitate comparison of results. Where appropriate and efficient, NIAMS should incentivize efforts to standardize collection of general clinical data or frequently studied biospecimens. For example, "mega-Cores" could establish bioinformatics platforms to manage Big Data and other highly specialized technologies (e.g., CyTOF, large scale epigenomics, metabolomics) that can be accessed by multiple users.

    3. Multi-Center platforms

      Modern clinical and translational science requires large sample sizes that yield massive datasets that are difficult for an individual institution to create, manage, and integrate. Multi-Center networks, consortia, and other types of alliances are capable of amassing sufficiently large sample sizes needed for population science and the study of rare diseases (including biorepositories for rarely obtained tissues such as muscle, bone, and cartilage). However, a major barrier to creating robust clinical cohorts and biorepositories is the current Center structure that does not encourage multi-Center platforms and collaborations (although there are recently formed NIAMS Centers that either share a local resource nationally or share resources at different institutions). New NIAMS Center structures should encourage resource development across Centers, toward achieving the twin goals of addressing specific research questions and being flexible and permissive to enable discovery. An important component of this process would be the development of fair and transparent policies and procedures that encourage access by outside investigators.

  5. Foster Mentorship and Training within Centers

    1. Team-science benefits for career development

      Each new generation of scientists must be able to build bridges across disciplines to form the foundation for future science. Toward this goal, NIAMS Centers should integrate approaches and opportunities to engage trainees at all levels in the development and use of Center resources. Junior investigators are likely to be familiar with tools for social communication and to embrace the team-science approach required for research using complex clinical datasets. In turn, seasoned investigators can learn from new tools that have shaped the education and career development of researchers still in training. NIAMS should monitor these training and career development efforts by assessing Center “marketing” plans that define specific strategies Finally, and as noted earlier in this document (see Goal D), NIAMS-supported early-stage investigators should be given high priority for the use of Center resources, and they should be provided with mentoring opportunities that coincide with the use of samples, clinical information, and other resources.

      Several educational approaches fulfill NIAMS’ goals for training and career development within its Centers. These include, but are not limited to:

      • Seminar series that emphasize the research fundamentals employed by a Center’s supported cores and the advances resulting from their use
      • Immersion experiences for undergraduate, graduate, or postdoctoral trainees that refine technical understanding of new research approaches
      • Outreach and engagement of trainees in non-traditional scientific disciplines beyond those represented by Center expertise
      • Pilot projects that expand the breadth of a Center’s scientific base across disciplines
    2. Interinstitutional collaboration

      Team science undergirds 21st century biomedical investigation, and thus all efforts to promote interdisciplinary interactions are important NIAMS Center activities. A key strategy going forward will be identifying research partners at other institutions, along with targets for collaborative use of core facilities and resources. This integrative approach offers opportunities for trainees to seek and collaborate with investigators at partner institutions to enhance skill development as well as to acquire data to address research questions addressed by Center cores. Center investigators should also sponsor inter-institutional pilot grant activities with limited time frames. In the spirit of leveraging the NIAMS investment, Centers should form partnerships with other institutional training and mentoring resources (e.g. funding connections with CTSAs or other entities/organizations). Building public support for science in the area of the Center could be achieved by outreach aimed at high school students or undergraduates.

  6. Conclusion

    Rather than making specific recommendations for the structure of NIAMS Centers, the CEWG has defined a set of goals, based on input from the community. While the existing NIAMS Center program has great strengths, formulating future Centers should address a set of common goals. These include promoting interdisciplinary basic, clinical, and health services research; facilitating the sharing of important resources and promoting their visibility; preserving methodological excellence; and fostering training of future generations of researchers. Adhering to these goals will transform NIAMS Centers and allow the scientific community to assemble integrated, synergistic groups of investigators who will drive innovation and technical advancement toward improving health in all NIAMS mission areas.

 

APPENDIX A
PROCESS & METHODOLOGY

After internal discussion among NIAMS leadership and extramural program staff, the concept of a review of the NIAMS Centers programs was presented in open session to the NIAMS Advisory Council on September 11, 2012. The collective vision was that there “are areas of research in the NIAMS mission areas where progress would benefit from integrated, synergistic groups of investigators. Synergy could be built through interdisciplinary research, shared resource infrastructure, and cooperative technological innovation.” Because Centers present a unique opportunity to foster such groups, the stated goal of the review was to “inform the design of efficient, dynamic, and innovative funding strategies to facilitate research that requires integrated, synergistic approaches, significant infrastructure, and technological innovations.” Specifically excluded from the Centers review were: a change in the relative budget of the overall Centers program; an evaluation of any specific Center; P01-type projects; and the use of networks of Centers for clinical trials.

Methods for conducting the review included several parallel efforts:

  1. Hold "listening sessions" with groups of investigators and lay representatives within five mission areas — bone, rheumatic diseases, muscle, skin and orthopaedics (meeting dates in Appendix B, meeting summaries in Appendix C).
  2. Announce a Request for Information (RFI) to solicit input from the broader NIAMS community, including professional and advocacy organizations (the RFI text in Appendix D).
  3. Formation of a Centers Evaluation Working Group (CEWG) (membership in Appendix E).
  4. Pilot testing by NIAMS extramural program staff of a series of software tools designed to evaluate scientific networks and productivity (in Appendix G).
  5. Formation of two groups of investigators, independent of the CEWG, one representing the Division of Musculoskeletal Diseases and one the Division of Skin and Rheumatic Diseases, who were involved in CTSA leadership at their respective institutions, to advise NIAMS how Centers could optimally interact with and leverage CTSA resources (reports of these two groups in Appendix F).
  6. An analysis by the NIAMS staff members of the evaluations of centers by other Institutes from a "lessons learned" perspective.

The CEWG was comprised of four members of the NIAMS Advisory Council (including a lay representative), investigators from NIAMS mission-related research communities, and NIAMS staff. The CEWG was charged at its initial teleconference (November 6, 2012) by the NIAMS Director, Dr. Stephen Katz, to look at existing Center programs and advise NIAMS as to how these could be enhanced, integrating the results of the different efforts listed above. The CEWG was not charged with recommending changes to any specific type of Center, and in fact there was no a priori mandate for any change.

Members of the CEWG, as well as NIAMS staff, participated in listening sessions relevant to their fields of interest. As described in the "Observations" section of this report, it became clear that the community was providing NIAMS with consistent messages ("themes") as to what Centers should be designed to accomplish. The CEWG shifted its focus, and undertook the task of integrating the different themes from the listening sessions into a coherent set of goals for the NIAMS Centers program. Through a series of conference calls and emails, CEWG members refined the Center goals to reflect both emerging opportunities in research, as well as the need to maintain relevant strengths in support of the central thematic areas of importance to the NIAMS mission. In addition to the themes from the listening sessions, the CEWG also integrated input from outside organizations and individuals obtained from the other efforts listed above. During this process, CEWG members with substantial content expertise in the scientific or technical area relevant to a particular goal were assigned to prepare short summaries that contained key background information and concepts. Two CEWG members served as editors to assemble and consolidate written summaries that reflected input from the listening sessions and other sources. The initial drafts were circulated electronically among members of the CEWG for suggested changes.

The CEWG convened in person on March 11, 2013, in Bethesda, Maryland. The initial portion of the meeting was spent clarifying and reaching majority consensus on the goals outlined in the draft document. For the remainder of the meeting, the CEWG discussed a series of specific questions that had been raised by NIAMS staff to provide specific guidance in terms of certain types and structures of Centers. This report represents an integration of the draft document with the responses to the specific questions, followed by editing for conciseness and consistency.

The interim results of the pilot use of computer analytic tools to evaluate Centers were also presented by NIAMS staff at the in-person meeting of the CEWG. The consensus was that these were interesting and perhaps could be used in the future to apply objective, quantitative measures to include in the evaluation of Center programs, but that significant further development was needed before the tools were both practical and useful (Appendix G).

Future plans. The recommendations contained in this draft report will be presented to the NIAMS Advisory Council on June 4, 2013, for discussion in open session (and thus made available to the public). Contingent on the outcome, the intention is to solicit additional comment from the public through the issuance of another RFI. After receiving this input, NIAMS would then develop models for future Centers programs, which would be presented to Council before implementation.

 

APPENDIX B
MEETING SCHEDULE

Date Meeting
September 11, 2012 NIAMS Advisory Council briefing
October 12, 2012 Bone Research Listening Session, held at American Society for Bone and Mineral Research (ASBMR) annual meeting
November 6, 2012 CEWG Meeting Teleconference #1
Rheumatology Research Listening Session, held at American College of Rheumatology (ACR) annual meeting
November 29, 2012 Skin Biology Research Listening Session, by Teleconference
December 4, 2012 CEWG Meeting Teleconference #2
December 5, 2012 Muscle Research Listening Session, by Teleconference
December 12, 2012 NIAMS-CTSA (DSRD) Teleconference #1
December 13, 2012 Discussion about the role of NIAMS Centers in the career development of clinician scientists, held at the Forum for Clinical Mentored K Awardees
December 18, 2012 NIAMS-CTSA (DMD) Teleconference #1
January 7, 2013 CEWG Meeting Teleconference #3
January 28, 2013 Orthopaedic Research Listening Session, held at Orthopaedic Research Society (ORS) annual meeting
February 5, 2013 NIAMS Advisory Council update
February 12, 2013 NIAMS-CTSA (DMD) Teleconference #2
March 11, 2013 CEWG Meeting (In Person) #4
May 20, 2013 CEWG Meeting Teleconference #5

 

APPENDIX C
MEETING/LISTENING SESSION SUMMARIES

  1. Bone Research Listening Session, held at American Society for Bone and Mineral Research (ASBMR) Annual Meeting
  2. Rheumatology Research Listening Session, held at American College of Rheumatology (ACR) Annual Meeting
  3. Skin Biology Research Listening Session, by Teleconference
  4. Muscle Research Listening Session, by Teleconference
  5. Orthopaedic Research Listening Session, held at Orthopaedic Research Society (ORS) Annual Meeting
  6. Discussion about the role of NIAMS Centers in the career development of clinician scientists, held at the Forum for Clinical Mentored K Awardees

1. Bone Research Listening Session, held at American Society for Bone and Mineral Research (ASBMR) annual meeting

NIAMS CENTERS PROGRAM EVALUATION
Listening Session
American Society for Bone and Mineral Research (ASMBR) Annual Meeting
October 12, 2012

Introduction

Dr. Carter set the stage for the meeting by showing a few slides about the Institute’s Centers Program and evaluation plans and asked two questions:

  • What are the research opportunities for which progress needs synergistic, integrated groups of investigators?
  • What are the optimal Centers program structures to facilitate and support such research?

Discussion—Research Opportunities and Interdisciplinary Research Teams

The discussion began with participants talking about activities at the Clinical and Translational Science Awards (CTSA) institutions. Some CTSAs have formed regional consortia to promote team science. ASBMR is offering a planning grant that will support groups of bone researchers as they develop translational research proposals through CTSAs.

NIAMS could encourage its Centers to collaborate. Rare bone diseases were mentioned as a topic that could benefit from inter-Center teams.

Researchers emphasized the importance of engaging different disciples in thematic research (e.g., research that addresses a specific disease). Center cores can propel careers of non-physician health scientists. They also can bring health services researchers together to work under one umbrella. Citing the variety of health professionals who care for people who have lupus (e.g., rheumatologists, cardiologists, nephrologists), one person suggested that partnerships among NIH components could be a starting point for identifying the best research teams to address important questions.

Many participants would like to see bone and mineral researchers think more broadly about bone interactions with other tissues. Working across disciplines can be difficult but worthwhile. Scientists develop creative strategies for answering research questions when they have access to a range of technologies. One person cited an example of phosphorous imaging in bone research that involves physicists, mathematicians and a chemist who can modify beta sheets.

NIAMS and other NIH entities could play a more active role in bringing these groups together (e.g., bone researchers at adipocyte meetings). Centers could provide a platform from which bone, bone/muscle, bone/adipocyte, and other teams of investigators can propose studies. For example, collaborations that include muscle researchers and scientists who have expertise in applied physics could lead to the development of a tool for measuring muscle health and diagnosing sarcopenia (similar to DXA for bone density and the diagnosis of osteoporosis).

One researcher asked for suggestions regarding what the most effective mechanism to bring ideas together might be. Centers provide more than cores and clearinghouses; they integrate the scientific community. The NIAMS Building Interdisciplinary Research Teams (BIRT) funding opportunity (http://grants.nih.gov/grants/guide/rfa-files/rfa-ar-13-003.html) was mentioned as an example of a small program that could be expanded. BIRT has the advantage of encouraging people to work together on a topic of shared interest that is relevant to the NIAMS mission; NIAMS does not pick the science.

NIAMS has a history of supporting grant mechanisms that bring people together (e.g., through multidisciplinary P01s or P50s). Participants noted that shared funding—in the form of support for core facilities, joint science, and meetings, for example—can prompt information exchange among researchers from different disciplines. One person suggested that institutions invest seed money to unite multiple disciplines.

When discussing research approaches that could fill information and treatment gaps, someone contrasted biomedical engineering with osteoporosis. Much is known about the mechanisms that could be leveraged in biomedical engineering strategies, but interventions are limited, whereas clinicians have many treatment options for their patients who have osteoporosis, but the mechanisms by which the treatments work is unclear.

Participants noted that Centers are engines for R01 development, although they are “more than large R01s.” The tendency of collaborations fostered through a Center to turn into R01 projects was thought to be a strong sign of a program’s success.

One participant mentioned peer review, calling for the NIH to "increase the quality of study sections from the top. We need the help of [the] administration to shape study sections."1 Another noted difficulty in predicting which proposals will produce the most influential work and challenges in getting study sections to understand a combination of topics, such as adipogenesis and bone.

1 Of note, NIAMS convenes a special study section to review Centers applications. These programs are not reviewed by the same groups that review R01 applications.

Discussion—Core Facilities

Resources that encourage all types of translational research were mentioned as important. Investigators would benefit from broader access to technologies that are unique to institutions. Examples include mass spectroscopy, biotechnology tools, and Fourier transform infrared spectroscopy. One participant asked about establishing an NIH-NSF collaboration for mathematics.

Access to bioinformatics support can be challenging. One person described the current NIAMS-funded Research Core Centers (P30s) as "too timid," noting that bioinformatics alone can cost $400k. Pooled resources for national or regional bioinformatics facilities may be more cost effective than having multiple institutions maintain individual bioinformatics programs. A national resource with cost-sharing was mentioned as another possible structure. Cores that investigators can "plug into" could be desirable, although such fee-for-service structures are expensive.

Improved access to data from large, population-based studies in the United States would be helpful, particularly for those investigators who want to use it to explore innovative hypotheses. Datasets from Kaiser Permanente and the Framingham studies were cited as examples. As whole genome sequencing technology becomes less expensive, the research community would benefit from being able to analyze extensive databases from large groups of people.

Next generation sequencing tools were mentioned as an example of a shared resource that could be useful when studying bone biology. More expertise in epigenetics also would be helpful; bone researchers could benefit from learning how to use epigenetics data. Studies of non-coding regions of the genome, which may be the source of the "next big breakthrough," require bioinformatics and access to data and analytic tools. A virtual Center, where meta-issue informaticists would mine gene expression data to identify targets for drug repurposing and opportunities for drug development, was proposed as a potential structure.

Discussion—Awareness and Access

A participant noted that Centers have been largely successful internally, but questioned how best to support big science nationally (e.g., CTSAs). Dr. Carter asked about barriers that prevent P30s from being national resources. In response, another person noted that institutions lack incentives to function as national cores; they want their researchers to have priority access to facilities.

Facilities that can provide access to certain specialized techniques exist at various institutions around the country. Sometimes these facilities receive compensation under a fee-for-service structure; other times, the research is more collaborative. While fee-for-service can be more efficient or simpler, investigators who wish to access the resources can have difficulty paying the fees required for a Center to recover its costs (particularly for pilot studies).

Participants asked for ideas about how to communicate that their institutions have resources that others can access. One person expressed the opinion that the bone community is missing opportunities to organize and translate basic discoveries.

Conclusion

NIAMS plans to issue a Request for Information (RFI). Dr. Carter encouraged attendees to respond.

Participants

NIAMS Staff
  • Bob Carter
  • Joan McGowan
  • Laura Moen
  • Faye Chen
  • Gayle Lester
  • Bill Sharrock
  • Jonelle Drugan
Extramural Investigators
  • John Adams
  • Lynda Bonewald
  • Adele Boskey
  • Roberto Civitelli
  • Thomas Clemens
  • Karl Insogna
  • Gerard Karsenty
  • Sundeep Khosla
  • Hank Kronenberg
  • Nancy Lane
  • Jane Lian
  • Joe Lorenzo
  • Laurie McCauley
  • Kristy Nicks
  • Eric Orwoll
  • Cliff Rosen
  • Ken Saag
  • Elizabeth Shane
  • Steve Teitelbaum
  • Jennifer Westendorf
  • Nicole Wright

2. Rheumatology Research Listening Session, held at American College of Rheumatology (ACR) annual meeting

NIAMS CENTERS PROGRAM EVALUATION
Listening Session
American College of Rheumatology (ACR) Annual Meeting
November 14, 2012

Introduction

Dr. Katz began the meeting by thanking the participants and providing background. He emphasized that the NIAMS centers programs have contributed greatly to the Institute’s portfolio over the years, but that all investments need to be evaluated periodically to ensure that they represent the best use of federal resources.

Dr. Carter showed a few slides about the Institute’s Centers Program and evaluation plans and asked the participants to focus their ideas on two questions:

  • What are the opportunities in dermatology research that would benefit from integrated groups of researchers?
  • How should Centers be configured to optimally support such groups?

Discussion

Dr. Carter asked the group what the research needs are for organized groups of investigators.

One participant said that, in general, the technology is available through institutions or collaborators. On the other hand, access to patient populations provides the greatest opportunity and there is a need to find a way to allow investigators to tap into those that exist around the country. It was suggested that mechanisms be created to allow investigators to find and utilize well-defined, clinical patient cohorts.

Another member of the group pointed to the number of genes and the challenge in understanding what those genes do. A large population of people carrying the different variants of interest is needed, especially one that includes healthy individuals. The Genotype and Phenotype (GAP) Registry is a valuable population resource available to investigators as controls. Epigenomics and epigenetics is a field that lacks sufficient expertise except in a few places. It was asked how the knowledge that does exist can be leveraged in other places. It was also noted that ENCODE is not focused on the cells types that rheumatologists are interested in. However, there is a huge amount to learn from subsets of cells. There is a huge variability across cells about where these marks are — what cells are important and which variant is driving the molecular phenotype in a cell.

One researcher emphasized that registries and collections are very important, as are normal control collections. However, there are a couple of more operator-dependent technologies like metabolomics, for which broad access is not available. Therefore, there is a need to develop this expertise and have this on a national level. Operator-dependent cores should be national resources available to a broader audience.

One —big believer in cohorts— called them crucial to research, but pointed out that there are too many centers doing similar work like flow cytometry. There is a need, however, for the next generation of this technology. This individual asked if resources could be leveraged from outside, such as the Human Immunology Center, so that scientists are using the same techniques and can compare data across diseases. In addition, can data be leveraged from the Human Genome Center so new analytic outputs can be used? How can technologies that leverage tens of millions of dollars invested outside of the NIAMS community be applied to NIAMS scientists?

The NCI’s TTRC (Tobacco Translational Research Center) was given as an example of a center that encourages interdisciplinary research. The idea was presented that this kind of investment is more productive than the equivalent amount of monies spent on R01s. In response to Dr. Carter’s question about what made this tobacco center successful, the participant noted that the center was encouraged to collaborate with other partners, community organizations, and other scientific groups. The effort generated creativity, synergy, and new ideas.

One PI involved in a Multidisciplinary Clinical Research Center (MCRC) commented that it was important to preserve and grow the research enterprise that NIAMS is so good at supporting. This person’s perspective was that the MCRC program was a major source of clinical research, yet has seen its funding decrease.

One member of the group mentioned the need to promote translational and clinical research, which seems more critical for young researchers with Ph.D.s than those with M.D.s. There are needs and efforts in the immunology community to do translational research because the Ph.D. immunology group feels as if they are being disadvantaged if they do not have clinical opportunities.

It was noted that a lot of time is spent setting up registries and a critical aspect is asking the appropriate questions up front. Greater success is usually achieved when interdisciplinary teams formulate those questions. Another researcher continued by saying that the rules for building of interdisciplinary research teams [through BIRT supplements] are very strict and can be a disincentive.

There is a strong desire among the community to advance rheumatic disease research and focus on rheumatic disease patients. Consortia, electronic medical records, ACR registries, and other sorts of global or national efforts provide a tremendous amount of power to characterize patients from different points of view and samples. There is a huge opportunity but also big challenges.

In response to a question about how rheumatologists can interact with CTSAs (Clinical and Translational Science Awards), one individual identified the fact that there are a lot of the opportunities, people, and technologies. The greatest challenge in this person’s opinion is how to make resources widely available to individuals at other institutions. A completely different type of structure may be needed. The new IRP/EP Clinical Center mechanism may be a model.

Dr. Carter mentioned two mechanisms: One way is to have a local group built around a resource; it’s only locally supported but resources are used nationally. The other mechanism is to have multiple people from different institutions built into the core from the beginning.

Another participant added that any center should be built around a burning question, for example, pre-clinical autoimmune diseases. It’s a great opportunity to use the registries that exist.

A member of a P30 pointed out that his center has various geographic locations and that they want these to be national cores. Especially in the case of rare disease, the cores collect data and specimens, and the P30 is open to users and collaborators from other non-participating centers.

In addition, the Oklahoma P30 has a clinical core which has other collections built into it that are widely used by investigators from outside the Oklahoma Medical Research Foundation. The speaker said that there is probably a more formalized way of making the process more transparent, and that word of mouth and early investigator workshops are effective.

Another investigator made it clear that some cores have websites with downloadable forms for prospective users, which fills the need to extend resources readily to people in real time.

One participant stated that core centers are more important than ever. The P60 methodology core should support information exchange, not just statistics, and promote synergy between technologies and patient resources (e.g., CARRA registry). Moreover, outcome measure development is critical. Finally, pediatrics may provide a unique opportunity because there are fewer co-morbidities. Interactions between pediatric and adult rheumatologists are something that should be supported.

Some centers offer internal small targeted awards for people to use new cores on a local basis.

It was stated that face to face interactions are critical to the success of teams and cores. Building teams is an "underserved" area in science.

One participant mentioned that there are existing trials that could address pressing questions, but they are not fully utilized. It is therefore important to help other people understand what the resources are, such as good trial data with samples. Can a database of existing trials be created?

There was a feeling that any center model ought to contain an opportunity to train the next generation so they not only have access to the samples and data, but mentors across a number of institutions.

One researcher emphasized the need for good cohorts and good repositories with good targeted questions. This could be ensured by building a co-funding mechanism into the centers that relies on use. Furthermore, there are a lot of resources out there that people don’t know about. It would be useful to find out what resources exist and what companies want, and facilitate a matching program.

Flexibility is often a limiting factor, as centers and investigators sometimes struggle to find the right mechanism to fit their ideas. Similarly, forcing interactions restricts imagination and is sometimes not cost effective. Another person added that the Study of Etiology of Rheumatoid Arthritis (SERA) has had 6-7 external collaborations. It is viewed internally as something you want to do because it is a win-win situation. The 2020 taskforce that the ACR has been working with will address the changing attitudes and approaches about collaboration help these kinds of virtual centers to work much more effectively and use new technologies to function well. The key is to find the right structure amongst all of these ideas that promote the end result which is improvement in what we understand about health outcomes.

The idea of letting investigators be imaginative was echoed by another member of the meeting. This individual agreed with the point made about letting other people in the door to these cohorts, but said that it should not be required as a part of any center. For large cohorts, it is a win-win to ask outsiders in. The centers ought to be designed with the best science drawing from different disciplines to maximize the scientific value. Whether they are from the same institution or not, the face-to-face interactions on a regular basis are crucial for interdisciplinary research. That is what really drives interdisciplinary discussions [e.g., Affinity Research Collaboratives (ARC)].

One junior investigator pointed out that early-career researchers do not have as many resources as the established PIs. Having an online database of resources would be useful for his colleagues.

Dr. Carter pointed out that, in a time when it is hard to get an R01, the centers can be a really good way to stay involved, and some of the junior investigators think much more broadly than senior investigators. This point was amplified by another junior investigator who utilized the Patient-Reported Outcomes Measurement Information System (PROMIS) to propel her work into clinical research.

Training is very important to put people in the right places. Some cores share better than others. The successful ones have a very collaborative approach that builds a culture of collaboration and training.

One person spoke about programs that tap into new technologies, giving the new center on computational science at Roosevelt Island in NY as an example of a group looking to partner with industry. The participant asked about ways to tap into these partnerships. Dr. Carter responded by saying that integration with organizations and industry is key and that they are interested. Two of the biggest successes have been the North American Rheumatoid Arthritis Consortium (NARAC) and the Osteoarthritis Initiative (OAI), which are basically registries or repositories. They are difficult to start up on their own but, if done correctly, they can become something very powerful and fundable. With the OAI we had help with industry. This is a great example of when the interaction between NIAMS and organizations and industry has to happen. There is no other solution in the current environment.

One investigator added that interactions with other countries can be valuable, especially when studying rare diseases.

A member of the ACR pointed out that the College cannot take industry money and that a mechanism would have to be created to support important things that need to get done, in collaboration with the ACR (i.e., Foundation for the NIH). There’s a huge interest and great resources. Industry understands what is needed; better targets, better outcomes.

A challenge experienced with the development of the Patient-Centered Outcomes Research Institute (PCORI) was that multi-stakeholder input is really critical. People learned more from having industry and patient groups involved, and various government groups.

Dr. Carter pointed out that the Request for Information (RFI) was published recently and available to everyone.

Participants

NIAMS
  • Steve Katz
  • Bob Carter
  • Susana Serrate-Sztein
  • Carl Baker
  • Su Yao Mao
  • Yan Wang
  • Katy Marron
  • Justine Buschman
  • Liz Elliott
Extramural
  • Joan Bathon
  • Lou Bridges
  • Hermine Brunner
  • Jill Buyon
  • Bruce Cronstein
  • Leslie Crofford
  • Peggy Crow
  • David Daikh
  • Betty Diamond
  • David Felson
  • Liana Fraenkel
  • Sherine Gabriel
  • Steve Goldring
  • Peter Gregersen
  • Mike Holers
  • Judith James
  • Dinesh Khanna
  • Bob Kimberly
  • Bob Lafyatis
  • Peter Merkel
  • Jim O’Dell
  • Kyung Hyun Park-Min
  • Mary Wheatley

3. Skin Biology Research Listening Session, by Teleconference

NIAMS CENTERS PROGRAM EVALUATION
Listening Session
Dermatology Research Community
November 29, 2012

Introduction

Dr. Katz began the call by thanking the participants and providing background. He emphasized that NIAMS had no preconceptions about this evaluation and looked forward to hearing feedback from outside investigators, both during this call and through the Request for Information (RFI).

Dr. Carter showed a few slides about the Institute’s Centers Program and evaluation plans and asked the participants to focus their ideas on two questions:

  • What are the opportunities in dermatology research that would benefit from integrated groups of researchers?
  • How should Centers be configured to optimally support such groups?

It was made clear that NIAMS was about those challenges that fit the centers model, and not, for example, those that could be solved with expanded clinical trials.

Discussion

Dr. Carter asked the group whether anyone has been involved in collaborative projects.

One participant volunteered the area of regenerative medicine as one that would benefit greatly from coordinated centers, citing the multidisciplinary nature of the field. The reasoning was that because of the great diversity of expertise required, a large grant might be the only way to bring together the necessary teams of investigators. Dr. Carter agreed and added that this was some of the rationale behind the NIAMS Building Interdisciplinary Research Teams (BIRT) program.

A member of a P30 research center pointed out that P30s establish core centers, but also provide opportunities for pilot projects that are particularly important to young investigators.

Another individual touted multisystem approaches and made the point that standing up new centers is more difficult than supporting ones that are already established.

The Childhood Arthritis and Rheumatology Research Alliance (CARRA) was referenced as a model for the Pediatric Dermatology Research Alliance (PeDRA), which is currently forming a broad collaboration across institutions, and expressed a desire to see a funding mechanism to encourage these types of interactions. The group echoed the support for the CARRA model and pointed to the work of investigators at the University of Rochester Medical Center in the field of atopic dermatitis. He also mentioned the Boston University and University of Chicago collaboration on scleroderma and the lupus and Sjögren’s syndrome tissue bank at the Oklahoma Medical Research Foundation as good examples of resources within Centers that are being shared.

One researcher felt that it was important that initiatives not be too prescriptive as it could inhibit what could be done, and that PIs should be able to choose between a local vs. national model for their center. Some institutions have great local resources and others do not. Therefore, center design should be dictated by the researchers based on their particular environment. Dr. Carter added that some PIs feel the need to "shoehorn" their ideas to fit into a particular funding opportunity and many feel that fewer binding conditions would lead to great opportunity. The researcher expanded his point by saying that too much detail in the announcement inhibits innovation. When guidance is required in shaping a center, it should be well thought out and well defined. The NIH’s desire to see more translational work was given as an example. While this is an important priority, it has caused confusion because of the varied interpretations of translational research by applicants and study section members.

One participant supported efforts to build national infrastructures but pointed out that if the partner institutions don’t have the proper resources, things will begin to break down. Certain minimum resources (e.g. dedicated clinical coordinators) are needed at each site in the network. There was support for the establishment of a clinical trials network that would save costs and resources by having a single coordinating center as opposed to one at each location.

It was added that workflow needed to be clinician-friendly and consistent across sites. This individual felt that not having a coherent and common database is a major obstacle when collaborating between institutions, and that a common database and repository could provide the necessary uniformity.

Another person pointed to the significant need for experts in bioinformatics to analyze increasingly large datasets. Collaboration would enable those institutions that are lacking in this area of expertise to share data with their partners who could assist in data analysis. Dr. Carter agreed but noted that such a resource works best when run by an expert dedicated to on his/her own project, but who also sees value in collaborations.

One scientist explained that the atopic dermatitis session at the Society for Investigative Dermatology (SID) meeting was an attempt to nucleate clinical trials for this disease, and suggested that the nucleation of a new center often dictates its later success. New centers could be funded through a two-step process that would require investigators to demonstrate that they can convene a core group around an innovative idea before further funding is awarded. Dr. Carter pointed to the SID as an organization that could initiate new teams and ideas. However, the argument was made that other patient and professional groups don’t know how all the pieces fit together.

Autoinflammatory diseases were suggested as an area of great scientific opportunity where there was a lot of recent work done to identify genetic targets, and that now is the time to get a multidisciplinary group together to look at how that information could be analyzed and used to benefit patients. However, it was pointed out that the good work to date, particularly in genetics, was only a "parts list" and is still a long way from treatment. This illustrates the need for multidisciplinary efforts to apply the breakthroughs in genetics to new interventions that help patients.

This individual also supported the P50 mechanism. In their experience, it helps get people interested in a common problem and brings together PIs who might not normally be involved in a particular disease. R01’s are not good for interactions between investigators, and centers allow other experts to get involved and change the direction of their own research. The feeling was that large center grants provide incentives for this kind of engagement that other mechanisms cannot.

Finally, the point made earlier about how to treat established P30s was echoed by another participant. It was said that once the infrastructure is set up, it doesn’t make sense to cut off funding and lose the resources that were built. Dr. Carter responded by saying that existing centers should be more competitive for new funds because they have resources and a proven track record. He asked if these advantages made starting a new center too difficult. Another investigator pointed out that new centers have recently been funded and that this question of new versus old centers was not a problem that needed to be addressed through new policies.

One person asked about ways to encourage innovation, which is of particular interest to young investigators. Specifically, there was interest in the possibility of a center equivalent of a BIRT-like component that would pull interdisciplinary researchers into a center that might go beyond the building of cores and pilot studies.

Conclusion

Dr. Carter encouraged attendees and other members of the community to respond to the RFI.

Participants

NIAMS
  • Steve Katz
  • Bob Carter
  • Susana Serrate-Sztein
  • Anita Linde
  • Ricardo Cibotti
  • Carl Baker
  • Kevin Cheezum
  • Liz Elliott
  • Branden Brough
Extramural
  • Anna Bruckner
  • Zhoufeng Chen
  • Keith Choate
  • Kevin Cooper
  • George Cotsarelis
  • Beth Drolet
  • Craig Elmets
  • Peter Koch
  • Alfred Lane
  • Ethan Lerner
  • Robert Modlin
  • Alice Pentland
  • Richard Spritz
  • John Stanley
  • Cristina Strong
  • Kim Yancey

4. Muscle Research Listening Session, by Teleconference

NIAMS CENTERS PROGRAM EVALUATION
Listening Session
Muscle Research Community
December 5, 2012

Introduction

Dr. Carter set the stage for the meeting by showing a few slides about the Institute’s Centers Program and evaluation plans. He noted that the Institute does not have any preconceived notions about the outcome of the evaluation. This listening session focused on two questions:

  • What are the opportunities in muscle research for which progress needs synergistic, integrated groups of investigators?
  • What are the optimal Centers program structures to facilitate and support such research?

Dr. Carter emphasized that the Centers review is limited to the Research Core Centers (P30), the Centers of Research Translation (CORT; P50), and the Multidisciplinary Clinical Research Centers (MCRC; P60). While the experiences of the Senator Paul D. Wellstone Muscular Dystrophy Cooperative Research Centers (Wellstone Centers; U54) could inform the NIAMS effort, NIAMS will not be modifying the Wellstone Centers as a result of this evaluation.

Discussion—Imaging, Structure-Function Analyses

One participant suggested that the muscle research community would benefit from a synergistic, integrated group of investigators who were developing imaging techniques that could capture meso-scale muscle structure and neuromuscular function. Better knowledge regarding fiber length and the extracellular matrix organization, for example, would lead to stronger models that, in turn, could be applied to treat people who have muscle pathologies. A subsequent comment defined "meso-scale" as structures that are larger than single cells or cellular components, but smaller than the whole tissue images that can provide information about gross tissue volume.

Another scientist emphasized the need for other advances in imaging. Magnetic resonance and ultrasound approaches that would allow investigators to look closely at healthy muscle and tissue from patients who have myositis were cited as examples. Interdisciplinary research teams could develop new tools that would enable more precise comparisons among healthy people and patients. The resulting information would help both clinicians and basic researchers, and would facilitate translational research.

The ability to follow clinical populations with non-invasive imaging approaches and analytic tools would also benefit the field. Stronger bioinformatics that could distinguish between contractile and non-contractile tissue would further automate image analysis. A database that could help investigators interpret the implications of fat and connective tissue around and within muscle would guide the development of pharmaceutical treatments and physical rehabilitation strategies.

After Dr. Carter drew an analogy to the Osteoarthritis Initiative’s robust database of magnetic resonance images of participants’ knees, someone mentioned the importance of a centralized strategy for handling magnetic resonance images. Better data acquisition and analytic strategies, and help in setting up these approaches, would be useful. Access to imaging and spectroscopy experts who have genuine interests in muscle is essential for these advances.

Discussion—Interdisciplinary Research and Team Science

Increased cardiac research in the muscular dystrophies was mentioned as an important area. A participant asked how best to integrate cardiac and skeletal muscle into one grant that would be well received by a study section, and wondered whether the Centers mechanism would be appropriate for this type of research. Dr. Carter replied that Centers can help circumvent problems and facilitate collaborations to move the field forward in a way that will benefit patients.

While discussing collaborations and leveraging, a researcher described how a core supported the collection of pilot and feasibility data for a colleague’s cardiac imaging study in Duchenne muscular dystrophy. The principal investigator used the pilot data to get an American Heart Association grant. The project is now a big part of this investigator’s research program.

One participant noted that $20-30K of support can benefit younger researchers, but larger amounts are needed to garner attention from established researchers who have expertise in other areas. A Center that is configured to support such interactions, or a supplement or grant mechanism toward this aim, would be welcome. Dr. Carter mentioned the NIAMS Building Interdisciplinary Research Teams (BIRT) program (http://grants.nih.gov/grants/guide/rfa-files/rfa-ar-13-003.html) as a potential mechanism.

Participants noted the strengths that different types of researchers, such as magnetic resonance, sequencing, and informatics experts, bring to a project. Diverse groups can make advances that are more significant than those of researchers who use a single approach. Funding is important when forming teams.

Discussion—Core Facilities

The potential utility of a national diagnostics program that could provide next generation sequencing for people who have neuromuscular diseases was mentioned. Such an effort would pull research groups studying individual rare diseases away from the "diagnostic silos." Information could be processed through a standardized program.

One participant called for an integrated sequencing facility that can serve the needs of the entire country. A core-like sequencing facility could strengthen the research base by uniting different groups (e.g., groups with experience in patient care, magnetic resonance imaging, sequencing, biopsy, high end informatics). It would allow researchers to study new genes and ask how specific changes affect phenotype. If successful, such a resource would improve the diagnosis of genetic diseases.

A multi-institutional Center, or a group of multi-institutional Centers, appealed to at least some participants. Cores that can be used by all will benefit the community. Magnetic resonance imaging was cited as an example where working across institutions is valuable.

In response to a question about whether NIAMS Centers could build on Wellstone Centers, Dr. Carter noted that people who are involved in Wellstones are eligible to apply for additional Center grants. Members of the NIAMS Centers Evaluation Working Group are looking at how NIAMS Centers interact with the CTSAs, most of which have methodology cores.

One participant expressed a need for a "basic biology P30." Unlike the current core grants, most of which are oriented toward a specific disease, the basic biology grants could include supersolution imaging or channel biophysics cores that would use the muscle cell as the model system.

In follow-up to a point about available NIAMS resources, Dr. McGowan noted that the Institute is revising the information about existing Centers on its website so the community can easily see what the Institute is funding.

Conclusion

There is a challenge of balancing the community’s interests in constrained, specific RFAs and more general requests that will allow people to address problems of which the Institute is not aware. Dr. Carter voiced a concern that investigators were trying to "shoehorn" their research into the Centers mechanisms. Instead, funding mechanisms should support what researchers want to do.

Noting that the group did not address the question of how Centers could be configured to meet the research needs, a participant encouraged everyone to think about the question and share their opinions. Dr. Carter closed the meeting with the request that participants continue to think about the Centers program and respond to the request for information (RFI; http://www.niams.nih.gov/Funding/Centers_Evaluation/submission_form.asp).

Participants

NIAMS
  • Bob Carter
  • Joan McGowan
  • Amanda Boyce
  • Glen Nuckolls
  • Mary Beth Kester
  • Kevin Cheezum
  • Liz Elliot
  • Jonelle Drugan

Extramural Investigators and Advocates

  • Paul Allen
  • Kevin Campbell
  • Jim Dowling
  • Jim Ervasti
  • Karyn Esser
  • Troy Hornberger
  • Rick Lieber
  • Dick Moxley
  • Melissa Spencer
  • Brad Stephenson
  • Stephen Tapscott
  • Dave Thomas
  • Charles Thornton
  • Julio Vergara
  • Krista Vandenborne
  • Noah Weisleder

5. Orthopaedic Research Listening Session, held at Orthopaedic Research Society (ORS) annual meeting

NIAMS CENTERS PROGRAM EVALUATION
NIAMS Listening Session
Orthopaedic Research Society (ORS) Annual Meeting
Monday, January 28, 2013

Welcome and Overview

Dr. Carter opened the session by explaining why NIAMS was evaluating its Centers program.

After giving a brief history of the Institute's evaluation activities, Dr. Katz emphasized that this meeting provides NIAMS with an opportunity to hear from the orthopaedics community about the types of science that Centers could support.

Dr. Carter showed a few slides about the ongoing evaluation and asked two questions:

  • What are the research opportunities for which progress needs synergistic, integrated groups of investigators?
  • What are the optimal Centers program structures to facilitate and support such research?

Discussion—Team Science

Several participants described the composition of their research teams, which include basic biologists, material/implant scientists, surgeons, stem cell biologists and tissue engineers. Unlike some other medical specialties that are beginning to collaborate with quantitative physicists and investigators from related disciplines, orthopaedics has a robust history of interacting with these groups.

Researchers noted that the process of writing a Center grant application is very different than that of putting together an R01. Whereas the final R01 application typically resembles the Principal Investigator’s initial vision, Center proposals evolve. Team members bring their own visions to a project in ways that individuals cannot duplicate by themselves.

Participants had difficulty identifying even one orthopaedic research area that would not benefit from Centers. The ORS membership and meeting attendance is evenly divided among clinicians, biologists, and engineers. Not only do the research interests cross disciplines, they transcend scales (i.e., from the molecular to the clinical). Scientific questions also can involve multiple agencies: the FDA could play an important role if research teams tackle issues that extend from preclinical evaluation to post marketing surveillance.

Sustaining a team is as important as establishing one. For example, successful groups need access to expensive, specialized equipment. Centers could coordinate access to such resources and could help people form teams that would use them.

One member emphasized the need for NIAMS to maintain a balanced portfolio. The orthopaedics research community needs NIAMS to fund as many research project grants as possible, but teams are essential for moving research from the bench to bedside and back again. The NIAMS Research Core Center (P30) awards are not large enough to support all promising areas. Someone noted "[It would be] nice if there was a mechanism to support a critical number of teams."

The financial stresses that are facing the scientific enterprise were another topic. Deans are trying to invest strategically in programs where people can work together. Because the National Science Foundation is limiting the number of applications each institution can submit, universities are making local, strategic decisions, about which applications they will put forth. This is forcing institutions to develop clusters and constellations of scientific teams that can become Centers of Excellence.

Discussion—Training

Several researchers noted that collaborations influence the graduate student experience. Students should be in an environment where research is thriving and where they can participate as a component of a multidisciplinary team. One person noted that trainees do not believe they will be able to develop a productive research career until they see a large, successful group. The ability to visit multiple labs contributes positively to their experience.

Of note, students in engineering and related fields receive support from the Department of Defense and DARPA. These projects operate through a model of several labs that contribute to team science. Students and faculty learn and share technologies. An additive manufacturing facility (aka "3D printing") was cited as an example of people coming together synergistically.

Discussion—Core Facilities

Some investigators enter the orthopaedics field with expertise in different specialties. Multi-institution facilities that "everyone" can access and are renewed regularly would save time, particularly for these investigators. Researchers base the questions they ask on the tools available; investigators need to know about resources.

Access to large, staple equipment such as microCT scanners and biomechanics facilities that would focus on orthopaedic studies was viewed as valuable. Although equipment cores could serve all disciplines, some noted that tools are used more efficiently if they are targeted toward specific types of questions. When a resource is dedicated to a certain area or field, its users are more cognizant of its potential and limitations.

The reliance on large animal models, with their enormous infrastructure and cost, was mentioned as unique to the orthopaedics community. A participant noted that the FDA does not accept preclinical work in orthopaedics research on rabbits and smaller animals. Studies on larger animals require expensive imaging techniques (which are not "off the shelf projects") and sophisticated veterinary expertise. The number of large animal studies presented at this ORS meeting was described as "impressive," relative to previous years.

In response to Dr. Carter’s question about whether orthopaedics research could benefit from a series of stand-alone resources, which together would provide advanced techniques to the broader investigator community, participants made a distinction between a facility that supports a fairly developed technique that people can access as fee-for-service, and the need for Centers to bring people together to develop and refine tools to answer emerging research questions. Some institutions and organizations are soliciting applications for small projects to create and foster groups. The objective is to create groups that will contribute intellectually to scientific questions, rather than just being "contract research groups." Other comments included putting people together and pushing the method-development envelope and disseminating methods.

Discussion—NIAMS Centers and the Clinical and Translational Science Awards (CTSAs)

Dr. Katz asked participants to describe interactions with CTSA facilities. Some members work closely with CTSA leadership at their home institutions. One person acknowledged a "fair amount of overlap" between the CTSA and a P60 grant, but noted important distinctions.

The CTSA’s education component is helpful for providing mentorship and protected time for trainees’ research. CTSAs provide important infrastructure, but cannot offer detailed instrumentation. The CTSA institutions have to choose whether they're going to specialize ("deep and narrow") or generalize ("broad and shallow"). Continuing the metaphor, a participant stated that the broad and shallow approach does not allow for a deep dive to push science forward.

Several participants discussed the strengths and weaknesses of CTSA-funded cores at their institutions. Leveraging is important, but some types of instrumentation require specialized cores. Researchers at CTSA institutions who are working with small cohorts can have difficulty accessing CTSA resources. CTSAs have resource limitations too, and sometimes place small cohorts as a lower priority than larger studies.

Conclusion

One attendee suggested that NIAMS have a listening session with deans and provosts, as they could offer a different perspective on institutions’ priorities and needs. Partnership is becoming the model by which academic institutions survive.

Dr. Carter thanked the participants for their input. He also reminded everyone that the Institute is accepting comments from individuals, organizations, and institutions through its Request for Information (RFI).

Participants

NIAMS
  • Steve Katz
  • Bob Carter
  • Joan McGowan
  • Gayle Lester
  • Jim Panagis
  • Fei Wang
  • Kan Ma
  • Jonelle Drugan
Extramural
  • Peter Amadio
  • Joan Bechtold
  • Connie Chu
  • Mary Goldring
  • Farsh Guilak
  • Josh Jacobs
  • Brian Johnstone
  • Helen Lu
  • Ted Miclau
  • Elise Morgan
  • Regis O’Keefe
  • Claire Rimnac
  • Eddie Schwarz
  • Richard Souza
  • Marjolein van der Meulen

6. Discussion about the role of NIAMS Centers in the career development of clinician scientists, held at the Forum for Clinical Mentored K Awardees

NIAMS CENTERS PROGRAM EVALUATION
Listening Session
Established researchers and representatives from voluntary/professional organizations who attended the NIAMS Forum for Clinical Mentored K Awardees
December 14, 2012

Introduction

NIAMS held this listening session as part of a Forum for Clinical Mentored K Awardees. While Forum participants who have active K awards were meeting with NIAMS extramural staff, other guests (listed below) discussed the NIAMS Centers programs with Drs. Katz and Carter.

Dr. Carter spoke briefly about the Institute’s Centers Program and ongoing evaluation. He explained that other NIAMS listening sessions are focusing on scientific opportunities that would benefit from organized groups of investigators. The goal of this session was to discuss the Centers’ role in clinical career development activities. At present, NIAMS does not require its Centers to have formal mentoring activities.

Discussion—Integration with Clinical and Translational Science Awards (CTSAs)

Participants emphasized that NIAMS Centers should not duplicate established programs, such as those under CTSAs. Instead, NIAMS Centers at CTSA institutions could integrate arthritis and musculoskeletal and skin diseases research into existing infrastructure.

Extramural researchers at universities with CTSAs described their institutions’ training and mentoring activities. Some have formal networks for K awardees. Others have robust mentoring programs that connect trainees with advisors and with peers. Participants contrasted the mentoring and career development programs that are available at small and large CTSA institutions. Smaller, single-university awards may be more successful at bringing people together than larger, more diffuse programs. However, virtual institutions or consortia of CTSAs may be able to provide more opportunities for trainees to find mentors and collaborators. Participation in large networks might also mean easier access to different institutional resources. The group revisited this point when discussing how NIAMS Centers could be designed to help junior investigators find mentors and collaborators.

In general, participants felt that the KL2 Mentored Career Development Program in Clinical and Translational Science (a component of the CTSAs) and the K12 institutional Mentored Clinical Scientist Development Program Awards (including the Building Interdisciplinary Research Careers in Women’s Health, or BIRCWH, program) have built national and international research teams and consortia that directly benefit the mentored investigators. However, access to career development resources under an institutional award can be uneven when people assume that they cannot use certain pieces of infrastructure because the institution did not select them for a slot in the formal KL2 or K12 mentoring program.

As currently structured, the utility of CTSA programs and other Centers for career development may vary with a trainee’s clinical specialty. For example, surgical residents who could contribute greatly to clinical studies, but are unable to devote 50 percent of their time to research, could benefit from a less intensive program that would give them experience working on a collaborative team. With additional flexibility in Centers’ career development programs, institutions could further facilitate clinical and translational studies by connecting interested clinical fellows with research groups who could benefit from their skills and interests.

Like other programs, CTSA recipients are concerned about expenses associated with training, career development, and mentoring activities. Institutions are looking for strategies to offset or recover their costs. One participant suggested that NIAMS consider providing supplements that would support mission-relevant work, or mentoring to investigators studying mission-relevant questions, at CTSA institutions. The American Society for Bone and Mineral Research is partnering with NIH to support the development of clinical and translational research proposals on bone diseases through CTSAs.

Discussion—Training, Mentoring, Collaborations, and Research Independence

Centers add value to the grantee institution when they foster interactions across departments. Interdepartmental research opportunities expose junior investigators to new skills, additional resources, and potential collaborators and mentors. Participants noted that the NIAMS Building Interdisciplinary Research Teams (BIRT) program (http://grants.nih.gov/grants/guide/rfa-files/rfa-ar-13-003.html) provided similar opportunities, albeit on a smaller scale.

Another scientist emphasized that a Center’s leaders must value mentoring if the Center is to help investigators launch independent research careers. Some Centers assign mentors to investigators who propose pilot projects. Others provide junior members with free access to certain core facilities. Researchers must know that resources are available before they can access them. NIH, Centers, and individual mentors each have a role in letting investigators know about existing tools and opportunities. Many in the community are confused about whether Center resources are available to scientists outside of the grantee institution.

Dr. Katz noted that NIAMS funded the first Research Core Centers (P30s) because dermatology researchers did not have access to other resources. When the program began in 1988, the cores were supposed to be available to researchers across the country. Since then, cores have diverged into local or national resources. As part of the current evaluation, NIAMS is asking about the extent to which cores and other Center resources should be accessible nationally.

In addition to providing access to core resources, Centers can provide opportunities for clinical mentored K awardees to develop cohorts on which they can base new projects. Centers also can assist junior investigators by providing didactic training and experience in study design, statistical analyses, and other essential clinical research skills. Exposure to the numerous steps that must be taken before launching a new therapy can be valuable. While data on the experiences of a few patients or from a mouse model may be exciting, more work is needed before a treatment can be approved. A mentor who has experience with the drug development process could be useful. Independent of the Centers programs, NIAMS offers Midcareer Investigator Awards in Patient-Oriented Research (K24; http://grants.nih.gov/grants/guide/pa-files/PA-11-195.html) for scientists who are conducting patient-oriented research and are interested in mentoring junior clinical investigators.

Although an applicant’s "independence" is not part of the review criteria for R01 grants, early-stage investigators who recently received their first R01 awards noted that reviewers sometimes criticized them for being too closely aligned with their mentors. However, the more senior participants commented that the quality of some of the research supported under K awards is equal to that of R01 projects. Whereas the K awards used to go to very junior scientists who recently completed a fellowship, the grants have become increasingly competitive. This is placing stress on grantee institutions, which may need to provide promising clinical investigators with protected research time for 3 or 4 years before the scientists get enough experience and data to compete successfully for NIH funding.

Conclusion

Dr. Carter concluded the meeting by thanking participants for their comments and their commitment to the next generation of clinical researchers. He reminded them that the Centers Evaluation Working Group members will consider this input as they develop recommendations regarding appropriate goals for the NIAMS Centers program and steps that NIAMS might take to accomplish the goals.

Participants

NIAMS
  • Steve Katz
  • Bob Carter
  • Anita Linde
  • Gayle Lester
  • Bill Sharrock
  • Hung Tseng
  • Yan Wang
  • Justine Buschman
  • Liz Elliott
Researchers and voluntary/professional organization representatives
  • Lynda Bonewald
  • Keith Choate
  • Leslie Crofford
  • David Daikh
  • Gary Gilkeson
  • Heidi Kong
  • Vasileios Kyttaris
  • Francis Lee
  • Paul Muhlrad
  • Theodore Miclau
  • Sampath Prahalad
  • Robert Sah
  • Elizabeth Shane

 

APPENDIX D
Request for Information (RFI): Evaluation of NIAMS Centers Programs


Notice Number: NOT-AR-13-007

Update: The following update relating to this announcement has been issued:

  • January 9, 2013 — See Notice NOT-AR-13-008. Notice of Extension of the Expiration Date.

Key Dates

Release Date: November 5, 2012

Response Date: (Extended to February 15, 2013 per NOT-AR-13-008) January 15, 2013

Issued by

National Institute of Arthritis and Musculoskeletal and Skin Diseases (NIAMS)

Purpose

The National Institute of Arthritis and Musculoskeletal and Skin Diseases (NIAMS) is undertaking an evaluation of its Centers programs to inform the design of innovative funding strategies to address research challenges and opportunities that require synergistic, integrated groups of investigators, significant infrastructure and technological innovations. As part of this evaluation, NIAMS is seeking input from the investigator community and organizations involved in its mission areas.

Background

This evaluation will provide a framework for how to structure NIAMS Centers programs to best organize and facilitate multi-investigator research. Most importantly, NIAMS will seek input from the research communities in its mission areas on the needs and opportunities for organized approaches, and how to structure support for these in the future. This began with a discussion of the proposed evaluation at the NIAMS Advisory Council Meeting on September 11, 2012. Meetings and teleconferences with investigators are planned. In addition, NIAMS is seeking written input from investigators and organizations through this Request for Information (RFI).

Information Requested

This RFI starts from the premise that progress in certain areas of research within NIAMS’ mission requires, or is optimally done by, synergistic, integrated groups of investigators, beyond what would be feasible to support through Research Project Grants (RPG) such as R01s. Examples might include collaborations, infrastructure, technical development or resources that are widely shared across a research community, perhaps nationally. Specifically, NIAMS seeks information to help identify:

  1. The research opportunities within NIAMS’ mission that require synergistic, integrated groups of investigators.
  2. Innovative approaches to structure Center program(s) to optimally support such research.

Please note that NIAMS plans to continue to support clinical trials through the programs recently announced for R21, U34, U01 and UM1 applications. Although small trials have been done as part of a Center, discussion for funding of larger clinical trials should not be included in responses. Further, NIAMS no longer accepts applications for Program Project Grants (P01), as these more limited groups of investigators are best funded through other RPGs, so discussion of this mechanism should likewise not be included in responses. Finally, responses should not address the success of any one Center at a particular institution, although examples of novel or successful structures (or, alternatively, what does not work) are welcomed.

Confidentiality

Responses to this RFI are voluntary. Any personal identifiers (e.g., names, addresses, email addresses, etc.) will be removed when responses are compiled. Only the de-identified comments will be used. Proprietary, classified, confidential, or sensitive information should not be included in your response. The United States government reserves the right to use any non-proprietary technical information in any resultant solicitation(s).

This RFI is for information and planning purposes only, and should not be construed as a solicitation or as an obligation on the part of the United States government to provide support for any ideas identified in response to it. Please note that the United States government will not pay for the preparation of any information submitted or for its use of that information. Responses will be compiled and shared internally with staff from NIAMS and its Advisory Council, with one or more subcommittees of the Council, and with scientific working groups convened by the NIAMS, as appropriate. In all cases where responses are shared, the names of the respondents will be withheld.

How to Submit a Response

Interested extramural investigators and organizations are invited to respond.

Responses to this RFI will be accepted until January 15, 2013. You will not receive individualized feedback on any suggestions. No basis for claims against the United States government shall arise as a result of a response to this request for information or from the United States government's use of such information.

All comments must be submitted using this web page: http://www.niams.nih.gov/Funding/Centers_Evaluation/default.asp.

Inquiries

Please direct all inquiries to:

Dr. Robert H. Carter, M.D., Deputy Director
National Institute of Arthritis and Musculoskeletal and Skin Diseases (NIAMS)
Email: NIAMSCentersEval-RFI@mail.nih.gov

 

APPENDIX E
NIAMS CENTERS EVALUATION WORKING GROUP (CEWG)

Members

CARTER, Robert, M.D. (Chair)
Deputy Director
National Institute of Arthritis and Musculoskeletal and Skin Diseases
National Institutes of Health

*BUCKWALTER, Joseph A., M.S., M.D.
Professor and Head of Orthopaedic Surgery
Department of Orthopaedics and Rehabilitation
University of Iowa Hospitals and Clinics

GRIFFITH, Linda G., Ph.D.
Department of Biological & Mechanical Engineering
Biological Process Engineering Center
Massachusetts Institute of Technology

*HOLERS, V. Michael, M.D.
Professor and Division Head of Rheumatology
Departments of Medicine and Immunology
University of Colorado, Anschutz Medical Campus

JAMES, Judith A., M.D., Ph.D.
Chair, Arthritis and Clinical Immunology Program
Oklahoma Medical Research Foundation
Professor of Medicine
Department of Medicine, Rheumatology, and Immunology
University of Oklahoma Health Sciences Center

KATZ, Jeffrey N., M.D., M.Sc.
Professor, Harvard Medical School
Director, Orthopaedic and Arthritis Center for Outcomes Research
Professor of Medicine and Orthopaedic Surgery
Brigham and Women’s Hospital

KRONENBERG, Henry M., M.D.
Chief, Endocrine Unit
Professor of Medicine
Harvard Medical School
Massachusetts General Hospital

PENTLAND, Alice P., M.D.
James H. Sterner Professor and Chair
Department of Dermatology
School of Medicine and Dentistry
University of Rochester

SANDBORG, Christy, M.D.
Professor of Pediatrics
Stanford University School of Medicine

SPENCER, Melissa J., Ph.D.
Co-Director, Center for Duchenne Muscular Dystrophy
Professor of Neurology
University of California, Los Angeles
David Geffen School of Medicine

STEPHENSON, Bradley R., JD
Attorney At Law, PLLC
Member, MDA Task Force on Public Awareness
Member, NIAMS Council

*Editors

NIAMS Staff

ELLIOTT, Elizabeth A.
Ethics Coordinator
Office of the Director

KESTER, Mary Beth, M.S.
Health Science Policy Analyst
Science Policy and Planning Branch
Office of Science Policy, Planning and Communications

LINDE, Anita M., M.P.P.
Director
Office of Science Policy, Planning and Communications

MARRON, Kathryn, Ph.D.
Research Program Analyst
Division of Skin and Rheumatic Diseases
Extramural Program

SHARROCK, William J., Ph.D.
Director, Integrated Physiology and Genetics of Bone Program
Division of Musculoskeletal Diseases
Extramural Program

TSENG, Hung, Ph.D.
Director, Extracellular Matrix Biology and Diseases Program
Division of Skin and Rheumatic Diseases
Extramural Program

WANG, Yan, M.D., Ph.D.
Director, Rheumatic Diseases Genetics and Translational Research Program
Division of Skin and Rheumatic Diseases
Extramural Program

Report & Document Coordination

DAVIS, Alison, Ph.D.
Office of the Director
Contractor

Logistical Support

CHEEZUM, Kevin
Office of the Director
Contractor

 

APPENDIX F
CLINICAL AND TRANSLATIONAL SCIENCE AWARDS (CTSAs) RECOMMENDATIONS

Concurrently with the deliberations of the Centers Evaluation Working Group, NIAMS asked two groups, one representing musculoskeletal diseases and one skin and rheumatic diseases, of investigators who are involved in the leadership of CTSA at their respective institutions to advise NIAMS as to how to better integrate NIAMS Centers with CTSA. While NIAMS recognizes that not all institutions have CTSA, and NIAMS would not disadvantage those, the reports are more generally applicable in the sense of integrating and leveraging NIAMS Centers with other institutional resources while avoiding overlap and redundancy.

CTSA (Skin and Rheumatic Diseases)

General principles for NIAMS Center and CTSA interactions

January 13, 2013

Gary S. Firestein, Leslie Crofford, Bruce Cronstein, Robert Kimberly, and James G. Krueger

  1. Minimize overlap with existing cores or services
    1. Overlapping or duplicate cores are inefficient, can lead to silos, and limit the collaboration/multi-disciplinary interactions
    2. Significant overlap between a core/service in a proposed Center with existing cores in either CTSAs or within institutions that have created "virtual" CTSAs should be avoided.
    3. Any new proposed cores as part of a Center should explain how it is distinct from existing cores at the institution and why a separate infrastructure is needed.
  2. Leverage CTSA (or "virtual" CTSA) resources
    1. Many institutions have robust cores that can serve as a nidus for specialized expertise. Examples include biostatistics, biomedical informatics, and biorepository cores or other services where substantial economies of scale exist.
    2. Expanding an existing CTSA core in a way that creates specialized expertise within it but leverages institutional infrastructure or creates opportunities for collaboration within the core would be viewed as an advantage. Examples include:
      1. Biostatistics with specialized skill in musculoskeletal and skin diseases
      2. Biomedical informatics, including genomics analysis and linking genotype to phenotype with specialized platforms created for musculoskeletal and skin diseases.
      3. Biorepositories to assist with collection, processing, storage, annotating, or distribution of samples.
    3. Integrate with CTSA training and mentoring programs, such as K30 didactic content or KL2 mentoring
  3. Review criteria for Center grants

    Include criteria that evaluate leveraging CTSA or "virtual" CTSA institutional resources. For example, it would be considered an advantage to use infrastructure for an institution-wide biorepository to create a specialized repository for a NIAMS Center.

  4. Organizational structure
    1. Allow flexibility for core organization by providing incentives for collaboration and avoiding silos.
    2. Permit pooling resources to create a more robust core. For example, institutional resources, CTSAs or other centers (Cancer Centers) could create a shared resource with multiple “entry” sites while maintaining individual areas of specialized content.
  5. Interactions with other institutions
    1. Encourage Regional Centers for cores with highly specialized technology that could be used by many sites where appropriate.
    2. Encourage collaboration with CTSAs at other institutions if it makes programmatic sense, especially if similar infrastructure is not available at the proposed Center’s institution.
    3. Document that obstacles to inter-institutional use of cores can be mitigated (e.g., intellectual property, material transfer agreements, indirect costs)
    4. Facilitate use of existing consortium activities, including harmonized IRBs and multi-institutional pilot project grant programs.
  6. Addendum (Crofford): NIAMS Centers, CTSA and training
    1. NIAMS Centers and CTSAs have an important opportunity to collaborate in the area of career development and pilot programs. Institutional CTSAs must support career development through the KL2 program and most also support pre-doctoral training through the TL1 program. All CTSAs also have pilot grants programs, many of which have funding for young investigators as a priority. For NIAMS Centers with a component of training or pilot grant awards, working with CTSAs provides an infrastructure that will include other young investigators to facilitate peer-to-peer learning and mentoring in clinical and translational science in diverse areas including biostatistics, research ethics, good clinical practice, etc. There will also be didactic training programs and degree-granting programs in clinical and translational science that can be leveraged by NIAMS supported trainees. Another rationale for collaboration is to educate NIAMS supported investigators about the resources within institutional CTSA and the CTSA consortium. In many institutions, Center pilot programs can work together with CTSA to co-fund pilot grants. This allows increased funding for specific projects and potentially increases to pool of investigators. Any of these strategies to stretch the resources of NIAMS centers should be encouraged.

CTSA (Musculoskeletal Diseases)

February 12, 2013

Eric Orwoll, Marc Drezner, Sundeep Khosla, John S. Adams, and Rebecca Jackson

What are the benefits to be gained from partnerships between NIAMS Centers and the CTSA Consortium?

  1. Leveraging/utilizing existing CTSA core resources can extend the science in NIAMS Centers. Examples of CTSA cores: fiscal/HR management, statistics, informatics, labs (e.g. sequencing, proteomics), GCRC, recruiting, clinical study networks, community outreach, education. CTSA’s all have infrastructure for management of pilot programs that can be leveraged for exploratory or pilot programs within NIAMS centers.
    1. The existence of CTSA cores can save NIAMS resources that otherwise might be required to start new Center-specific cores.
    2. Leveraging well-functioning CTSA core resources should enhance efficiency and cost-effectiveness of NIAMS funding.
    3. Expertise in CTSA cores can extend/compliment what might be available in Center-specific cores
    4. By leveraging existing CTSA core resources, NIAMS Centers could focus on development of new, truly unique core functions that directly affect musculoskeletal and skin diseases (e.g. musculoskeletal and skin disease phenotyping) and could avoid redundancies in "standard" infrastructure such as clinical research units, biostatistics and translational informatics.
  2. The CTSA Consortium could contribute inter-institutional collaborative structures and resources. Opportunities could include:
    1. Existing CTSA collaborative and communication mechanisms among institutions could be leveraged to support inter-institutional NIAMS efforts
    2. The CTSA consortium could provide enhanced patient recruitment opportunities at collaborating institutions, including identifying patient data from EDWs for deep phenotyping
    3. Biospecimens for rare disease from multiple institutions could be collected using CTSA resources
    4. Extended core lab, analytical options, or expertise not available at every institution (e.g. metabolomics, PK PD assay development or lead optimization in medicinal chemistry) could be used in multi-institutional collaborations
    5. Coordinated regulatory and fiscal environment between institutions could be managed by CTSAs
    6. Integrated clinical trial networks exist in the CTSA consortium
  3. Potential benefits of non-CTSA elements of NCATS
    1. Enhanced partnering with industry
    2. Coordinated involvement of NCATS centers (especially preclinical innovation and rare disease network)

What budget model would support a CTSA — NIAMS Center collaborations?

Several models are possible, and are not mutually exclusive

  1. The use of CTSA resources (e.g. cores) by NIAMS Centers could be supported with appropriate full costs for services included in the Center budget. The availability of the CTSA core represents a commitment/contribution by the CTSA/institution to NIAMS Center functions.
  2. CTSAs could contribute to some extent (cost-sharing).
    1. There could be contributions by individual CTSA’s to NIAMS Centers at their institutions, either in the form of funding or in the form of available institution resources (for instance free biostat consultation or GCRC services)
    2. Contributions by the CTSA Consortium can be imagined in special circumstances

What organizational structure could support a CTSA — NIAMS Center partnership?

  1. There could be separate admin and fiscal structures in the NIAMS Center vs. the CTSA, with contractual arrangements between the CTSA and a NIAMS Center for needed services (e.g. core resources)
  2. NIAMS Centers could exist in a matrixed state with the CTSA’s, fully overlapping in some areas (e.g. the CTSA stat core IS the Center stat core) while not in others (e.g. the Center may have an independent bone/muscle phenotyping core)
  3. A NIAMS Center could exist fully within a CTSA with shared administrative and fiscal operations. In that case the NIAMS Center could be, in essence, an institutional core/capacity to support musculoskeletal science in partnership with the CTSA. This would demand clear delineation of the goals of NIAMS funding and the ability to track and evaluate outcomes.

Are there educational opportunities that may be enhanced by partnering with CTSAs?

The development of careers in NIAMS-relevant research areas may be an important part of the mission of Centers. CTSAs routinely offer robust educational opportunities and Centers can take advantage of existing curricular and mentoring programs offered by CTSAs. Leveraging the educational curricula in CTSAs can provide a foundation for translational research training while the disease-specific depth of expertise/mentoring would be provided by NIAMS Centers

How would a CTSA — NIAMS Center partnership be described in a FOA?

  1. It would be easy to include non-specific FOA language allowing Centers to propose to leverage or collaborate with CTSAs. This is the traditional FOA language and doesn’t provide any new or creative approaches.
  2. More uniquely, a NIAMS Center FOA could request applications that specifically mentioned collaborative advantages as above, or were built on any of the above organizational models.
  3. In the review process, a more favorable score could result if science or efficiency was enhanced by leveraging CTSA resources
  4. It is assumed that a FOA could not be limited to CTSA institutions, and that institutional resources outside CTSAs could also be leveraged to accomplish similar ends.

 

APPENDIX G
DATA ANALYSIS TOOLS

Evaluating Potential Tools for Analysis of Impact of Future Center Programs

As part of their review of Centers, NIAMS staff undertook a study of some available software tools that might help provide objective evidence of the impact of future Centers in terms of productivity and fostering collaborations. The goals were to see if any of these were useful and, if so, consider what information might be requested from a new Center in the terms of award. Each of four NIAMS staff (Katy Marron, William Sharrock, Hung Tseng, and Yan Wang) selected one tool. Each then selected a portfolio of grants as examples to explore the use of the tool (for some, these were existing Centers; others included other types of grants). eSPA and Web of Science were used to test methods for analyzing the number and citation index of publications, IN-SPIRE was used to map changes in topics of scientific papers, and Sci2 was used to look at collaborations.

These exploratory studies revealed unanticipated problems. In all cases, the effort needed to use the tool to analyze a portfolio far exceeded expectations. Most of the datasets required careful curation and, for some, manual entry. Some of the tools required manipulation of criteria to develop meaningful outputs. Other tools required a high event rate (e.g., number of publications) to produce meaningful data, restricting applicability. A brief summary of the exploratory studies of each of these tools follows. Each describes the potential question to be addressed, the methodology, the possible interpretation of the observations, and a list of caveats.

An initial look at each of these tools was shared with the CEWG. However, it is critical to note that none of these examples are being used to guide the recommendations of the CEWG as to goals or structure of Centers. Similarly, although some of these analyses used data related to existing or previous NIAMS Centers, none was used to judge the performance of those Centers. Rather, the purpose is simply to determine whether these tools might be useful in the future.

The conclusions are:

  1. The tools that measure publications are unlikely to be useful, as time needed to develop the datasets was excessive and the impact of Centers on these measures is difficult to assess. Further exploration is needed to understand the effects of a Center on productivity and the quality of research, and to create a model to describe that.
  2. IN-SPIRE produced an indication that the subjects of publications from a set of investigators changed after awarding of a Center grant, based on the words used in the texts. However, the words used in the text analysis require manipulation in order to produce meaningful clusters, and thus is operator-dependent. This case study is not sufficient to determine whether the change in the terms used in publications over time is a result of a Center.
  3. Sci2 produced visualization of expanding collaborative networks in association with a Center, but again comparative analyses would be needed to determine the effect of a Center program.

1. eSPA
Do Core Centers (P30s) really "… accelerate the pace of research, … ensure greater productivity, … expand the research base?"

Approach

This analysis used standard tools of grants administration to correlate the presence or absence of a Core Center (P30) with levels of published research output from the NIAMS “research base” at a given institution. The NIH Query View Report (QVR) system was used to define the research base in grant-years as the total years of R01, P01, P50, and P60 grant support awarded to an institution by the NIAMS in fiscal years 2008 through 2011. P01, P50, and P60 awards were counted as three grant-years for each active year, to normalize against the typically smaller R01s. Grant awards were linked to publications and impact factors using the NIAID Electronic Scientific Portfolio Assistant (eSPA). Research output was defined as the total number of peer-reviewed publications, excluding reviews, citing support by grants in the research base in calendar years 2009 through 2011, multiplied by the average journal impact factor for that group of publications. The analysis was limited to research grants administered under the NIAMS programs in Keratinocyte Biology and Diseases, Immunobiology and Immune Diseases of Skin, and Skin Repair, Regeneration and Pigmentation; and to P30s awarded for at least 3 of the 4 years (2008-2011) under Requests for Applications for Skin Diseases Research Core Centers.

Results

The research base (2008-2011) across 38 institutions varied from 2 to 36 grant-years, with a median of 8 grant-years. Only institutions with bases greater than 10 grant-years (total 13 institutions) were studied with respect to research output. The product of publication number by average impact factor varied from 42 to 452 within this sample, with a median of 164. There was no clear dependence of research output on either the presence/absence of a Core Center grant, or on the size of the associated NIAMS research base. The top four institutions for research output do not have P30s. Of 6 institutions above the median for research output, 2 have P30s; of 6 institutions below the median, 3 have P30s. [see chart on page 53]

Caveats

  1. The programs in skin-related areas represent 13% of the total NIAMS grant awards over the studied period. They are not necessarily reflective of the relationship between Core Centers and research output in other mission areas.
  2. Practices of support acknowledgement in publications may not be consistent.
  3. The product of publication number by journal impact factor is a crude measure of research output.
  4. The total NIAMS investment represented by the research base at these institutions ranged from $8,000,000 to $32,000,000, with a median of $22,000,000. The maximum support available through a P30 over the same period would have been approximately $2,400,000. This increment may be too small to make a detectable difference in research output, given the likely variation in staff research interests, infrastructure, and institutional support within the sample.
Bar graph showing the number of publications by impact 0 to 500 by the research base (grant years) 0 to 40. The graph also compares, by color, if grants were P30 grants.

2. Web of Science
Exploratory use of Web of Science as a tool to assess the research output from a grantee institute

One of the expected outcomes of research funding is a greater scientific literature output in both quantity and quality from the grantee institutes. Web of Science (WoS), the literature database created and managed by Thomson-Reuters, has a number of features that can facilitate assessing the research publication output from a research institution in a given area of science. Some of these features are: i) keyword search on topics, including multiple layers of keywords and Boolean operators; ii) attribution/categorization of research papers according to the authoring institution; iii) the number of times a paper is cited and the citing paper number, or the average citation number of a set of publications; iv) h-index. To explore the utility of these features in portfolio analyses, a number of tests were run to validate these features:

  1. To verify if the WoS can identify institutions with research strength in a given area, publication output (in number) from institutions with known eminence in a research topic (i.e., Marfan, PXE, FOP) was examined. The results show that WoS correctly identified the top contributor institutes to literature in these topic areas (Figure 1). These institutes are also known to host the strongest research group worldwide on each topic.
  2. To examine the correlation between publication number and impact (as measured by citation), the literature of gene-expression-profiling of rheumatoid arthritis was gathered by a two-layered Boolean keyword search (transcriptome + gene expression + genetics)/rheumatoid arthritis. Within this dataset of approximately 5000 papers, WoS provides a comparison of the publication number and the average citation/paper from 50 institutions (Figure 2). This result shows WoS’s ability to identify institutions that publish fewer but more cited papers.
  3. The question if h-index should be used in keyword-defined datasets needs to be addressed. This is because h-index is only suitable for research communities with similar citation behavior whereas keyword-defined datasets may contain research communities with diverse citation convention and behavior. A comparison was made among a group of institutions using a dataset selected by keyword (Methotrexate + rheumatoid arthritis) (Figure 3). The result shows that in this dataset, h-index follows closer to the publication number and obscures salient variations in citation/paper.

In order to use WoS as a tool for evaluating center grants, a number of problems remain to be resolved, which include how to choose keywords to select publications most relevant to the target of evaluation (a center grant, a group of individual grants, etc.,), what parameters of publication should be the focus of analysis, e.g., impact, topic coverage, authorship composition (e.g., age and interdisciplinary network) and how to measure effects and contributions of service facilities that are supported by center grants to assist research activities in a range of topic areas.

Figure 1. Web of Science (WoS) correctly identified the top institutes in Marfan, PXE and FOP disease research. Keyword (Marfan, PXE and FOP) search in WoS showed three research institutes/universities (de-identified) are, respectively, the top institutes in producing research literature in quantity in Marfan, PXE and FOP, which are consistent with their reputations as being the leader in each field

Figure 2. The average citation/paper feature of Web of Science is useful in capturing institutions that publish fewer but more frequently cited papers; citation frequency being considered as a surrogate indicator of impact.

Figure 2. The average citation/paper feature of Web of Science is useful in capturing institutions that publish fewer but more frequently cited papers; citation frequency being considered as a surrogate indicator of impact.

Figure 3. The h-index correlates more with the number of publications than citation/paper. The institutions (de-identified) are ranked by their literature output in a given area, 1 being the most productive. Web of Science (WoS) provides the average citation number of papers published by a research institute and the h-index of that set of publication. This analysis shows that h-index follows closer to the number of publication and bias against some institutions with extraordinarily highly cited papers.

Figure 3. The h-index correlates more with the number of publications than citation/paper. The institutions (de-identified) are ranked by their literature output in a given area, 1 being the most productive. Web of Science (WoS) provides the average citation number of papers published by a research institute and the h-index of that set of publication. This analysis shows that h-index follows closer to the number of publication and bias against some institutions with extraordinarily highly cited papers.

3. IN-SPIRE
About IN-SPIRE

IN-SPIRE (http://in-spire.pnnl.gov/index.stm) is a tool developed at Pacific Northwest National Laboratory, which enables text mining and clustering of datasets with large numbers of documents. IN-SPIRE can accept datasets from a variety of formats, including Excel, CSV, and text files. This tool can be used to identify themes within groups of documents, conduct time analysis, and compare different subsets or groupings that can be created from imported data elements in the dataset.

Exploratory Application of IN-SPIRE in the Context of a Center Award: Does Center Funding Contribute to the Emergence of Publications in New Thematic Areas?

The example we explored involved the analysis of publication data from a Center of Research Translation (CORT; P50) award with a single disease focus. The dataset was generated by searching PubMed (http://www.ncbi.nlm.nih.gov/pubmed) for publications meeting the search criteria listed below.

  • Authors included all key personnel (PIs, Investigators, Co-Investigators) of cores and projects
  • Publication dates from one year prior to the center application through sixteen months after the expected end of first funding period.
  • Exclude reviews
  • Limit to the disease focus of the center

Publication data were exported from PubMed in the MEDLINE format, and used to create the IN-SPIRE dataset. The settings in IN-SPIRE for numbers of clusters were adjusted to obtain reasonable visualization, and certain text words were excluded from the analysis to produce more meaningful clusters. The publications were sub-grouped by two year time frames according to publication date, as shown in Figure 1. In this example we were able to show that there three clusters of publications (clusters 2, 6, and 8 in Figure 1) that were new or increased after the center award, while cluster 7 was an active area of publication prior to the time of the center award, where activity in this area appeared to decline after award.

Caveats Based on Our Experience with This Example

While we were able to identify changes in publication activity in distinct thematic areas over time using IN-SPIRE in this example, it is clear that there are a number of caveats that need to be considered.

  • Clustering performance requires document numbers within certain thresholds (100 to 10,000 documents), not all projects will have publications within the threshold.
  • Labor intensive methodology to create and analyze the dataset for each grant.
  • Publications analyzed before and after award cannot be directly linked to the funded grant. If we were to limit the analysis to publications citing the center award, it would not be possible to compare publications prior to and after center award with the same criteria to see how themes in the publication change. In other analyses, we have also seen that there is wide variability in how investigators attribute publications to a grant.
Figure 1: IN-SPIRE galaxy view of publications associated with the key personnel from a NIAMS-funded P50 Center of Research Translation.

Figure 1: IN-SPIRE galaxy view of publications associated with the key personnel from a NIAMS-funded P50 Center of Research Translation.

4. Sci2

Co-Author Networks – the Science of Science (Sci2) Tool (https://sci2.cns.iu.edu/user/index.php)

The Sci2 is a modular toolset specifically designed for Science of Science Research & Practice. It supports temporal, geospatial, topical, and network analyses with advanced algorithms; and uses effective visualizations to explore and understand specific datasets. It is open source software made available for the research community

.

In Sci2, networks are comprised of two basic elements: points/nodes and lines/edges. In the Co-Author Network, each node represents an author and their complex interrelations as edges. The Co-Author Network analysis is based on the premise that having the names of two authors (or their institutions, countries) listed on one paper, patent, or grant is an empirical manifestation of scholarly collaboration. The more often two authors collaborate, the higher the weight of their joint co-author paper. The Weighted and Undirected co-authorship network analysis calculates, for each node/author, the total node degree (number of edges the author is connected to) and the total node strength (the sum of the weights of all the edges the author is connected to).

For example, in Figure 1, the annotated attributes for Author A (yellow node) is listed on the left side of the graph; its degree and strength are 37 and 51, respectively. Similarly, the degree and strength of Author B (blue node) are 20 and 26, respectively, while that for author C (green node) are 11 and 15, respectively. This suggests that Author A is connected to the highest numbers of people and has the highest numbers of co-author publications among the three, with Author B being the second and Author C as the last. This ranking is reflected in the size of the nodes. In addition, the weight of the edge (thicker line) between Authors A and B is higher than that between Authors A and C. This represents that Authors A and B have co-authored more than Authors A and C. These analyses provide clues as to who are the key players in a network, with whom they have co-published, and with whom they have published more often than with others.

Graphic reflects output of the Sci2 tool displaying authors of publications, identified by color, connected to other people and to co-authored publications. Their rankings are briefly described in the paragraph above.

Brief description of the exploratory question/example in which we tried to apply the tool

A goal of center programs is to facilitate interdisciplinary research where a team of experts from different disciplines work synergistically to create innovative approaches to a problem. We explored the Sci2 Co-author Network analysis as a potential way to visualize interdisciplinary collaborations of a research center. The assumption is that successful collaborations will lead to a growth over time in both the number of network investigators as well as the number of co-authored publications. As an example, Figure 2 shows an every two-year view of the co-author networks of a P60 research center over a ten-year period since its inception. It was based on publications of the Center director and co-authors that cite the grant number. The table below shows the associated data for each of the time slices; there appears to be a trend of increase over time in terms of the number of nodes and edges for the network as well as the calculated total node degree and total node strength of the principal investigator (yellow node). The disconnected sets of nodes represent papers that cite the Center grant but whose authors are not linked by co-authorship networks to the principal investigator.

This analysis suggests an expansion of the collaborative network.

Graphic reflects output of the Sci2 tool displaying, in color, a principal investigator's increase in connections over a 10 year period.

  Input Parameters for the entire network Calculated Attributes for the Principal Investigator
Years Number of Nodes Number of Edges Total Number of Degree Total Number of Strength
2002-2003 38 170 37 51
2004-2005 77 387 60 91
2006-2007 142 919 103 168
2008-2009 140 802 85 132
2010-2011 274 3,766 184 228

Caveats relating to the use of the tool based on the exploratory examples we tried

While temporal analysis of co-author networks provides a dynamic view of scholarly collaborations, there are limitations to this tool. For example, a “blossom” of the network at one time could be accidental due to one publication with a long list of co-authors. In this case, it is important to look at the trend of growth over a period of few years. In addition, supplemental analyses are needed in order to reveal the different disciplines of the co-authors or the interdisciplinary nature of the work. Moreover, this type of analysis can be used only to obtain a longitudinal view within a center using baseline as a control, rather than a cross sectional comparison with another center or other non-center projects. Furthermore, besides co-author publications, there are other outcome measures of a successful collaboration such as mentoring/training of early stage investigators or generation of patents and new grant funding.