CERC - Canada Excellence Research Chairs
CFI - Canadian Foundation for Innovation
CFREF - Canada First Research Excellence Fund
CGS - Canada Graduate Scholarships
CIHR - Canadian Institutes of Health Research
CRC - Canada Research Chairs
EST - Environmental sciences and technologies
HLTH - Health and Related Life Sciences and Technologies
HQP - Highly qualified personnel
ICT - Information and Communications Technologies
MoU - Memorandum of Understanding
NCE - Networks of Centres of Excellence
NRE - Natural Resources and Energy
NSERC - Natural Sciences and Engineering Research Council
PMS - Performance Measurement Strategy
R&D - Research and development
SSHRC - Social Sciences and Humanities Research Council
S&T - Science & technology
STIC - Science, Technology and Innovation Council

Executive Summary

Program Overview

Launched in 2008, the Canada Excellence Research Chairs (CERC) program supports Canadian universities in building a critical mass of expertise targeted within the four priority research areas of the Government of Canada’s Science & Technology (S&T) Strategy, in support of Canada’s growing reputation as a global leader in research and innovation.

The program aims to attract world-class researchers in S&T priority areas to eligible Canadian universities. Through a competitive peer-reviewed process, awards worth up to $10 million over seven years support the chairholder’s salary, the research team and other costs of research (i.e., the CERC unit). The first group of Canada Excellence Research Chairs was awarded in May 2010, and 18 chairholders from this cohort are now active at 13 host institutions in the four priority research areas. A second competition was launched in 2012 to allocate 11 new CERC awards in fields relevant to the four priority research areas, as well as other key fields of research.

As a tri-agency initiative of the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC), the program is administered by the Chairs Secretariat, housed within SSHRC.

Evaluation Scope, Objectives and Methodology

This evaluation covers the initial five years of the CERC program, from inception in 2008-09 to 2013-14 (to the end of phase 1 of the second competition). In accordance with section 42.1 of the Financial Administration Act and the Treasury Board Policy on Evaluation (2009), the evaluation addressed key questions of relevance, performance and efficiency. It was also designed to be of value for future program improvements and planning. Since it is still early in the program’s lifecycle, the assessment of program performance focused on the program’s immediate outcomes, and the assessment of economy and efficiency was limited to the operational efficiency of the program (i.e., the relationship between resources and outputs).

Ten evaluation questions were addressed via eight data collection methods: document review, review of administrative and performance data, cost-efficiency analysis, interviews, international comparison study, web survey, bibliometric analysis, and case studies of CERC units. Methodological challenges were mitigated in a proactive manner, resulting in the collection of robust evaluation data that was triangulated across multiple lines of evidence for each evaluation question. The impact of any limitations (e.g., small population size, current stage in the program lifecycle) is noted in this executive summary when related to specific findings or recommendations.

Key Findings, Conclusions and Recommendations

The evaluation’s conclusions and recommendations are presented in the narrative below along with the key findings to support them. The majority of these recommendations are intended to be implemented by the Chairs Secretariat, but the Secretariat’s response will likely require the development and implementation of actions in collaboration with other key stakeholders involved in the delivery of the CERC program, including host institutions and corporate divisions within the federal granting agencies.

Relevance
The evaluation evidence supports continuing the program. Given the current national and international contexts the program is as relevant today, if not more so, than when it was created. The CERC program will undergo another evaluation within the next five year period ^{Footnote 1}, at this time additional information will be available regarding the program’s relevance and need.

Recommendation 1: The CERC program should be continued for an additional five years. The current context reinforces the need to continue supporting the program to help ensure Canada remains competitive at the global level.

The CERC program was initially launched in response to the Government of Canada S&T Strategy (2007) to achieve global excellence in research. Evaluation evidence available at this early stage in the program’s lifecycle suggests that the CERC program continues to be relevant not only in the context of this Strategy, but also by contributing to the attraction of international research talent to Canada and by helping Canadian universities build research capacity in the four identified S&T priority areas. These needs remain strong because the current international landscape for science, technology and innovation is characterized by increasing government investments in R&D in many countries, including the emergence of new and important players such as the BRICS countries (i.e., Brazil, Russia, India, China and South Africa). These investments are driven by the recognition that research is a key driver to support strong economies.

The evaluation found that Canada faces fierce global competition to attract top research talent as a means to enhance the creation of new knowledge, technology and innovation. In fact, the international comparison study found that a growing number of countries have launched programs very similar to CERC (9 programs), as well as other initiatives (24 programs) aimed at attracting and retaining world-class researchers. Most of these programs were launched around the same time as the CERC program (i.e., 2008) or after. The international comparison study also found that, compared to similar international programs, the CERC award is competitive on a global scale based on its value and duration, and is therefore apt to help Canadian institutions attract world-class researchers.

To remain competitive in this shifting global research context, evaluation evidence indicates that Canada should not only continue to support R&D in higher education institutions, but also address its chronic challenges with respect to knowledge transfer from higher education institutions to companies and other receptors that have the ability to translate this knowledge into products and generate benefits for society. Relevant to this need, the CERC program was also intended to facilitate the strengthening of relationships between academia and other sectors that can apply or otherwise benefit from the innovation, knowledge and talent developed in the CERC host institutions.

The legitimate role of the federal government in continuing to support the development of research capacity in Canadian higher education institutions is widely recognized, and is clearly supported in the legislation (e.g., Department of Industry Act) and in the S&T Strategy. Ongoing federal investments for innovation and research programs—including the 2011 funding commitments for new CERC awards and the $1.6 billion five-year commitment to support research and innovation announced in the 2014 Economic Action plan, notably through the creation of the Canada First Research Excellence Fund—also demonstrate that such programs continue to represent a priority for the government. Acknowledging the important complementary role of provincial governments and industry in supporting innovation, evaluation stakeholders shared the view that federal government funding is essential to the achievement of the CERC program objectives, given that there are no other programs or sources of funding in Canada that can currently provide equivalent means (i.e., duration and value of CERC awards) to achieve similar outcomes.

The CERC program is also generally perceived to be complementary and to have synergies with other national funding programs that similarly respond to federal S&T policy commitments, particularly tri-agency programs (e.g., Canada Research Chairs [CRC], Networks of Centres of Excellence [NCE]) and the Canada Foundation for Innovation [CFI]. Several instances were observed in which the CERC program and investments from other national funding programs worked in concert to support individual CERC units.

Stakeholders inquired about the interaction between the CRC and CERC programs, given both operate through research chairs and pursue shared outcomes (e.g., research excellence, building capacity within Canadian universities), although the CERC program has a more explicit international perspective, smaller number of awards, and a larger award amount. The evaluation found some evidence of involvement of CRC chairholders in CERC units but no systematic data was collected or available to more adequately examine this interaction and to demonstrate the added-value of the CERC program over time. As such, it could be beneficial to better capture and assess the relative impact and synergies between these two programs as the CERC program matures.

Performance – Effectiveness
Although it is still early in the program’s lifecycle, the CERC program has made good progress towards achieving all of its expected immediate outcomes. Some outcomes have yet to be achieved to their full extent, as will be discussed below.

Among its most notable achievements, the CERC program has directly contributed to the attraction and recruitment of world-class researchers in a highly competitive global environment.

In this report, the term “world-class researchers” refers to researchers internationally recognized as world leaders or rising stars with exceptional potential, while “high-calibre researchers” refers to other key researchers involved in the CERC unit. Evaluation evidence confirms the exceptionally high level of scientific accomplishments of the group of researchers recruited as chairholders in the first CERC competition, all of whom came from institutions outside of Canada. The bibliometric analysis shows that CERC chairholders are world-leading scientists: as a whole, their peer-reviewed publications out-performed both researchers from Canada and the world, based on indicators of scientific impact and quality prior to being recruited. They also scored significantly higher in terms of scientific impact/quality than the group of high-performing nominees who were not retained for a CERC award. Moreover, in the first competition, the Selection Board’s recommendations for the top nominees were those who either met or in many cases exceeded the expectations of excellence established for the CERC program.

The CERC program also resulted in the attraction and/or involvement of hundreds of high-calibre researchers and highly-qualified personnel (HQP) in the supported CERC units. Based on self-reported data, over 300 researchers were closely involved with the 18 CERC units, along with over 800 HQP (e.g., students, postdoctoral fellows, and other research/technical personnel). More than 90% of the surveyed institutions reported an increase in the number of high-calibre researchers and all reported an increase in the number of HQP in priority research areas as a result of the CERC program.

The observed recruitment of world-class researchers was directly linked to the CERC program. Without the CERC award, most chairholders reported that they would not have moved to the institution or even to Canada; this statement was echoed by several high-calibre researchers and HQP. The attraction of world-class researchers was aided by the prestige of the CERC award and the pre-existing critical mass of research expertise at the host institutions. In fact, 70% of high-calibre researchers involved in CERC units were already working at the host institution when the CERC unit was established. Both the survey and case studies also showed that support received from the institution prior to and after the arrival of the chairholder (e.g., incremental funding, salaries, infrastructure, offer of associated faculty positions), as well as access to talented HQP, were both also critical to recruitment of world-class and high-calibre researchers.

Although the CERC program was successful in attracting world-class researchers to Canadian host institutions, clear evidence was found of ongoing barriers in the attraction and retention of world-class researchers; many of these barriers also applied (but to a lesser extent) to high-calibre researchers and HQP. Most notably, half of surveyed institutions were not able to recruit the world-class researcher originally identified by the institution for the CERC award and proceeded to nominate another qualified candidate. The difficulties in recruiting foreign world-class researchers can be partly explained by the fierce global competition for top-tier researchers. The main remaining barriers identified by successful chairholders and host institutions include institution-related issues (e.g., level of commitment from the host institution), non-renewability of the award and other award terms (e.g., amount and duration), personal issues (e.g., immigration, relocation, family/spouse), and issues related to program delivery in the first competition (i.e., short timeframe for submitting nominations). Not all of these challenges can be fully or effectively addressed by the CERC program alone, but recognition of the range of factors to be overcome in the attraction of world-class researchers have already led to refinement of the program’s second competition and should continue to inform the program’s design and delivery.

Concerns of potential barriers to access and equity were also raised after the first competition, at which time no women were nominated for a CERC award. This issue prompted a review of gender issues relating to the program by the Ad Hoc Panel on CERC Gender Issues, which identified potential factors related to barriers to equity in the CERC program design, and also in the broader university context. The panel’s recommendations resulted in changes to the second CERC competition (e.g., application and nomination process and criteria, program literature), which have been well-received and led to the development of university nomination strategies to proactively attract a larger, more diverse group of candidates. It is somewhat premature to assess the impact of these changes at this time other than to note that the first chairholder to be appointed in the second round was a woman.

The CERC program clearly enhanced the research capacity in S&T priority areas in host institutions, to the extent that some awards were reported to be “game changers” by the institution.

The concept of enhanced research capacity includes both the expertise and research infrastructure that are developed in the S&T priority areas at host institutions. The evaluation evidence shows that the CERC program helped build and/or structure a critical mass of expertise in the targeted fields within these institutions, in the form of the CERC units, which are led by the chairholder but may also include new hires or faculty positions. The resulting CERC units are typically large, involving a large pool of highly talented group of researchers and HQP. The CERC program has also been a strong incentive for host institutions to build or expand state-of-the-art research infrastructure, in partnership with CFI, as means to contribute to enhanced research capacity in priority areas in host institutions. In fact, 93% of host institutions reported positive impacts of the CERC program on their research infrastructure. In short, the case studies clearly showed that the CERC awards led to the development of highly interdisciplinary research teams and state-of-the-art facilities.

The evaluation also found strong evidence that, with the support of the CERC award, the chairholders act as a ‘catalyst’ or ‘nucleus’ in their role as core members of CERC units: they bring in new knowledge and ways of thinking, new technology and new collaborations, leading to benefits for researchers, HQP and collaborators/partners, as discussed below. The chairholders also played a key role not only in driving the development of this research capacity, but also in helping integrate and anchor this capacity within host institutions.

Indeed, the research capacity associated with the CERC units is usually well integrated within host institutions, meaning that CERC units have established strong linkages with or within existing groups, research centres and/or research areas. Conversely, the few CERCs that were found to be less integrated generally functioned as independent and more decentralized research units. Note that, given the early stage of the CERC awards, this assessment should be considered as preliminary (i.e., the degree of integration of CERC units may still evolve over the period of the CERC award). Evidence from several of the case studies also found several examples of enhanced synergy and cohesion generated by the CERC award across or within departments. However, in some cases, the CERC award had the unintended (and unwanted) effect of reinforcing competitiveness between or within departments.

Enhancement and integration of the CERC unit’s research capacity within universities was usually also facilitated by a high level of responsiveness and support—financial and otherwise—from the host institution. For example, many host institutions have invested large amounts in terms of space and equipment for the CERC unit to complement existing infrastructure, including new and recently created institutes in the CERC units’ research fields. Thus, 71% of host institutions reported positive impacts of the program on their reallocation of internal funding. The important contribution of host institutions in the success of the CERC program is shown by the fact that host institutions are one of the main sources of funding for CERC units: they provided $40 million between 2010 and 2013 across the 18 CERC units, and have committed a further $24 million, for a total of $64 million over the seven-year period of the awards.

To further support research capacity within host institutions, CERC units as a group have also successfully leveraged over $128 million in funds from a variety of other sources (as reported for 2012-13 and 2013-14), primarily from the federal government (72%, including from granting agencies and CFI). However, the amounts and sources leveraged from host institutions and other sources by individual CERC units vary widely, with about half of the CERC units receiving much larger amounts than the rest. Overall, CERC units were found to have sufficient resources to achieve their research objectives by the end of the CERC award.

The CERC program has created benefits for researchers and HQP, leading to positive impacts such as career opportunities and higher-quality research outputs.

The benefits of the CERC program reported by researchers and HQP revolve around the participation in large-scale, cutting-edge research with flexible, long-term funding provided by the CERC award. For example, in the absence of the CERC award, chairholders reported that they would not have had the opportunity to carry out research programs of such large scale and scope. Many of them would have dropped key components of their research program or carried out fewer, more focused projects within a much smaller and less integrated/coordinated and multidisciplinary group. Moreover, many high-calibre researchers and HQP cited the unique opportunity to work with a world-class researcher—often related to his exceptional personal qualities, vision and leadership—as a reason why they chose to join or remain the CERC unit, further emphasizing the chairholder’s central role in the CERC unit.

Many HQP reported they were given the opportunity to conduct risky research managed by a world-class researcher with strong leadership skills and supported by CERC funding, which are yielding high quality/high impact outputs. Some also reported having been given greater responsibility as part of the CERC unit compared to other research teams. HQP also gained increased exposure to a variety of expertise through exchanges/collaborations with other students during conferences, workshops, internships—even multidisciplinary international summer schools.

Notably, high-calibre researchers and HQP identified current and anticipated positive impacts of their participation in CERC units on their careers. They feel that their participation in a CERC unit has increased their employability, particularly in academic or industry sectors (e.g., commercialization skills leading to employment opportunities in spin-off and technology companies). While some high-calibre researchers secured tenured faculty positions when they joined the CERC unit, several HQP expressed a concern that these benefits and impacts would not extend beyond the end of the CERC term, as they perceived few career opportunities in academia and industry for researchers in Canada.

The benefits cited above were also reported to have contributed to increased scientific production of chairholders, high-calibre researchers and HQP. With respect to chairholders, bibliometric analyses indicate an increased international co-publication rate—from 50% to 85%—when comparing publications before and after the CERC award. The analysis also suggests a potential positive effect of the scientific impact/quality of papers, based on the impact factor of the journals in which they were published; however the difference was not statistically significant. Importantly, these bibliometric analyses are based on a relatively small number of peer-reviewed papers (i.e., those published 2011-2013 and for which the chairholder is an author), so this analysis could be repeated later in the CERC program’s lifecycle and include papers from other members of the CERC unit to obtain more robust and reliable data, and to help confirm the extent to which the scientific performance of the CERC units as a whole has benefited from the CERC program.

Delays in setting up the CERC facilities and/or research groups have sometimes hindered the scientific production of CERC units to a greater degree than expected.

The case study evidence showed that some CERC units experienced challenges, such as issues securing the required or committed funding and/or other forms of support from some expected sources, including from host institutions (e.g., faculty hiring, lab space/construction) and provincial governments, and applying for and using CFI funding (e.g., timing of grant application, flexibility in the use of CFI funds). These challenges were perceived to be most severe in the early stages of the award, which is a critical stage when a new research facility or team is being assembled, especially when this requires new infrastructure, the purchase of state-of-the-art equipment and the hiring of key personnel before the research program can get underway.

These challenges resulted in greater than anticipated delays in producing research outputs. For chairholders, this finding is confirmed by bibliometric trends, which indicate an eight-month lag on average from the start of the chair to the publication of the first related paper and an average two-year lag before recovery of the pre-CERC production level, with some CERC units having experienced a more noticeable lag than others. As discussed further below, this situation has implications in terms of developing the CERC units competitiveness and sustaining the research capacity of host institutions in the longer term.

Challenges Assessing Expected Outcomes
The evaluation evidence also points to the need for clarification of some expected program outcomes, particularly in cases where it was not possible to assess whether outcomes had yet been fully achieved. A clearer definition and shared understanding of these outcomes and of how the CERC program is expected to achieve them would help ensure that the CERC awards are selected, implemented and monitored accordingly.

Recommendation 2: Review and clarify expectations regarding the CERC program outcomes. Clearer definitions and expectations regarding branding, sustainability, as well as collaborations, partnerships, and relationships with users of research (non-academic sectors) need to be developed.

Evaluation evidence shows a lack of clarity among stakeholders and mixed findings on progress regarding the extent to which the CERC program has contributed to Canada becoming a global destination of choice for research and higher learning—in other words, in helping brand Canada in this role.

On one hand, efforts made by host institutions, the CERC units and chairholders have contributed to the visibility of host institutions in Canada as a destination of choice for research. For example, approximately 75% of host institutions reported that the CERC awards contributed to increasing their visibility nationally and internationally. According to case study interviewees, this increased visibility results primarily from networking and outreach efforts and activities undertaken by the chairholders and CERC units, such as hosting large-scale international symposia and initiatives involving academia, industry and/or government representatives, as well as through promotional materials produced and disseminated by the CERC units. Several chairholders also indicated that the establishment of formal collaborations with prestigious foreign institutions both exemplify and contribute to the increased visibility of the institution generated by the CERC award. For example, Memoranda of Understanding (MoU) and other agreements have been formalized between the CERC host institutions and international research institutions in relevant topics (e.g., quantum nonlinear optics, Arctic science).

On the other hand, limited evidence was found of increased visibility of the CERC program itself outside of host institutions and CERC award announcements. For instance, the CERC website received fewer than 100 unique visitors monthly prior to September 2013, and anecdotal evaluation evidence suggests that the CERC program is not well known outside of Canada among target audiences in the S&T community (e.g., representatives from other international programs with similar objectives). As such, program stakeholders suggested that more efforts could be made to better promote the program, as well as the research results of the chairholders, both nationally and internationally.

Since applications to the program are made by host institutions, and not by individual researchers, the main target audience for CERC program communications have been Canadian higher education institutions. With regard to Recommendation 2, the responsibility of the Chairs Secretariat and the extent to which program resources could be devoted to broadening and enhancing communication and promotional efforts to increase the visibility of the program beyond this core audience (e.g., to Canadian and international research communities and to the general public) is therefore up for debate. Of note, the Government of Canada also supports a range of other research programs with similar ‘branding’ policy objectives (e.g., CRC, Vanier CGS, and Banting Postdoctoral Fellowships), such that it may be appropriate and efficient to clarify and assess their joint contribution to this shared outcome.

The definitions and expectations for partnerships and collaborations in the context of the CERC program are not sufficiently clear, leading to challenges in assessing the level of achievement for this outcome.

There was considerable confusion and variability around the use of the terms “collaborations” and “partnerships” among case study respondents and program documents consulted, including the CERC program description, the logic model, and the application and reporting forms. This confusion stems from a lack of shared understanding of the definition of these terms observed across the various stakeholders and of how these were associated with specific expectations and/or outcomes. For instance, it could be made more explicit on how collaborations, partnerships, and relationships with users of research are to be associated with knowledge mobilization, leveraging and sustainability. Expectations related to the number and type of collaborations and partnerships established by the CERC units may also need to be assessed differently based on the nature of research conducted (e.g., fundamental, applied). This is important because the outcomes in the current CERC performance measurement strategy strongly emphasize the expectation of partnerships with industry and other end users.

Overall, the CERC program has been highly successful in contributing to the enhancement of beneficial research collaborations, but partnerships and relationships with users of research outside of academia (i.e., industry, government, associations, and other institutions) have been more limited to date.

CERC units have been highly successful in developing new relationships and building on academic collaborations in Canada and internationally. In fact, the large majority (two thirds) of the CERC units’ collaborations and partnerships reported in 2014 are with researchers in the academic sector (Figure 1), and over half of these are with foreign organizations. The major benefits of these relationships for both the CERC unit and their collaborators include diversification of expertise via access to a large pool of highly talented researchers from a variety of disciplines that contribute to the research endeavour, as well as access to cutting-edge infrastructure, equipment and data. These collaborations and partnerships lead to strengthened research outputs, improved training of HQP and enhanced career opportunities.

Several CERC units (but not all) work extensively with the private sector, such as in the context of applied research projects. However, relationships with the public sector and not-for-profit organizations at the provincial, federal and international level were less frequent as compared to the number of academic collaborations. In addition, 57% of host institutions reported a positive impact of the CERC unit on the number of collaborations and partnerships with the non-academic sector—a relatively modest share compared to other impacts on which they were surveyed.

self-reported international and Canadian collaborations and partnerships, by sector

Note: *Other includes hospitals, clinics, international agencies, etc.
Source: Case study questionnaires (2014)

Description of figure

Description of Figure 1, Self-reported international and Canadian collaborations and partnerships, by sector

This double bar graph shows the number and type of collaborations and partnerships developed by Canada Excellence Research Chairs units, collected through the 2014 case study questionnaires.

Along the x-axis there are five groups of two bars each. The groups are labelled by organization type: university, government, industry, association, and other (e.g., hospitals, clinics, international agencies, etc.). Each group contains a red bar for Canadian organizations and a blue bar for foreign organizations.

The y-axis shows the number of collaborations and partnerships. The scale begins at zero and goes to 160, in increments of 20.

The number of collaborations and partnerships with each organization type was reported as follows:

University, Canadian: 123
University, foreign: 148
Government, Canadian: 32
Government, foreign: 21
Industry, Canadian: 27
Industry, foreign: 5
Association, Canadian: 5
Association, foreign: 3
Other, Canadian: 8
Other, foreign: 27

The amount of funding leveraged from non-academic partners was also perceived by program stakeholders to be modest. For example, funding from foreign sources, provincial governments, trusts/foundations and corporations each account for between 3% and 7% of the total leveraged from sources outside the CERC program and host institutions. CERC units have identified challenges in developing partnerships and securing funding from industry and other sources, although many expect these investments to grow over time. This suggests an opportunity to increase the relatively modest contribution of companies/firms and other non-academic partners to help support the CERC units.

Indeed, the evaluation evidence suggests that the program may represent an effective means to build beneficial relationships with non-academic sectors in the future, and some stakeholders indicated that such partnerships are expected to be developed as programs of research evolve, leading to further benefits. Some early notable examples were observed of the benefits of such relationships (e.g., commercialization of research products, patent applications, creation of new companies, expert advice for the government sector). Industry partners also identified benefits of working with a CERC unit, such as a higher return on investment and reduced risk for R&D, as well as greater access to HQP that have gained industry-relevant skills. Similarly, government partners noted that working with the CERC unit was an opportunity for them to improve the economic development in their region, and to enhance decision-making through access to cutting-edge expertise and high-quality research results.

It should be noted, however, that a range of challenges to developing fruitful collaborations and partnerships were also observed, especially in the context of academia-industry partnerships, which may need time to establish. These ranged from focused issues, such as the management of intellectual property, to broader concerns, such as difficulties in securing matching funds and other investments, especially if partners were facing budget constraints as many were following the global economic crisis.

Performance – Program Design and Efficiency

The CERC program has demonstrated good operational efficiency, compared to what is typically expected from federal programs and to the performance of other similar programs. In particular, the evaluation found that the efficiency ratio of the CERC program was within the same low range as those of the NCE and CRC programs (i.e., $4 of administrative expenditures or less for each $100 of grant expenditures).

Stakeholders also generally expressed a high level of satisfaction with the amount, duration and flexibility of the award and the two-phase process. However, both survey respondents and case study interviewees were generally less satisfied with the non-renewability of the award, and with the requirement for matching funds (e.g., from the host institution and other sources) that was introduced in Competition 2.

The Chairs Secretariat has already identified and implemented several improvements to its design and delivery model following Competition 1, leading to more positive experiences reported by host institutions for Competition 2. It is nonetheless expected that the program will continue to review and adjust application and nomination processes on an ongoing basis. The evaluation identified a number of design and delivery features that could be further adjusted to improve program effectiveness, including promotion and visibility (see Recommendation 2), funding and post-award transition (see Recommendation 4). In particular, the evaluation concludes that the CERC program needs to improve the monitoring of program results over time given that the impacts of S&T investments are long-term.

Recommendation 3: Improve reporting procedures, mechanisms and tools (e.g., annual reports, mid-term review) to ensure that the Chairs Secretariat has more comprehensive information in order to monitor the program and to better capture evidence of program outcomes over the long-term.

In particular, the program should consider closely monitoring how the sustainability issue evolves as the program matures, as well as the impact of the changes implemented (e.g., improvements made after Competition 1, equity issues). The evaluation also identified key limitations with respect to the availability and consistency of performance and financial data for Competition 1 that are reported in the progress reports, statement of accounts, and data available for the promotion of the CERC program.

Emerging Issue – Sustainability of Research Capacity

Overall, the evaluation evidence showed that the CERC program awards have enabled host institutions to strengthen their leading position or to become world leaders in targeted research areas through attraction of world-class researchers and enhanced research capacity in these areas. This in turn is expected to contribute to the increased competitiveness of institutions to attract additional funding and ensure a certain degree of self-sustainability of the CERC units after the seven-year award term, and yield further benefits for researchers, HQP and collaborators/partners and other users of research.

Concerns were frequently raised with respect to the sustainability of the research capacity after the end of the CERC award, namely whether the critical mass of expertise and world-class research environment developed in S&T priority areas at host institutions will be maintained.

It is important to note that the sustainability outcome was perceived in different ways by stakeholders (e.g., sustainability of research capacity developed via the CERC unit vs. continuation of CERC-funded research projects/program vs. maintenance of CERC funding or equivalent level), such that this outcome could benefit from further clarification in the program description and definitions (see Recommendation 2). The ability to maintain state-of-the-art infrastructure and research spaces, and to retain a critical mass of outstanding tenured and non-tenured key researchers, HQP, and highly skilled professionals were all considered essential elements of the sustainability of the CERC-enabled research capacity developed in S&T priority areas within host institutions. This was also alternatively defined as “momentum” or as a “legacy” of the CERC award, such that it does not necessarily include the continuation of individual research projects supported by the award or the retention of the chairholder at the host institution.

The sustainability question had been a focus of the CERC program since the inception of the program. In their applications to the program, host institutions expected that their support, combined with additional sources of support would be sufficient to maintain the research capacity developed by the CERC units beyond the end of the award. However, the evaluation found that institutional sustainability plans (which may be formal or informal) may not result in the expected level of support, especially if the CERC units experienced delays in setting up research programs and are not yet in a position to be competitive for future research funding.

Indeed, in addition to the unanticipated delays experienced by some in the early stages of the award period, other challenges were identified with regard to the likelihood that the key elements of the research capacity developed by CERC unit would be adequately supported after the end of the CERC award. In particular, while faculty positions/salaries and major infrastructure and space for CERC units are expected to be supported beyond the duration of the CERC award by the host institution and other sources, a variety of elements/costs were less likely to be supported, including maintenance of equipment and facilities, administrative support for the CERC units, as well as direct and indirect costs of research (Figure 2).

Description of perceived likelihood that elements/costs will continue to be supported beyond the duration of the CERC award

Description of figure

Description of Figure 2, Perceived likelihood that elements/costs will continue to be supported beyond the duration of the CERC award

This stacked bar graph shows participating institutions’ perceptions of how likely it is that specific elements/costs will continue to be supported after the Canada Excellence Research Chairs (CERC) award term ends. The source of the findings is a survey of participating institutions completed by 20 respondents, a 100% response rate.

The y-axis lists key elements/costs. The x-axis shows percentage of respondents, from 0% to 100% in 10% increments. The bars are horizontal and each has up to four colours, representing the percentage of respondents who selected the following responses: very unlikely or unlikely, neutral, likely or very likely, or not applicable/don’t know.

The perceptions of how likely it is that elements/costs will continue were reported as follows:

Faculty positions created as a result of the CERC: 100% believed it was likely or very likely to continue
Chairholder salary: 7% believed it was very unlikely or unlikely to continue, 93% believed it was likely or very likely
Major infrastructure and space for CERC program of research: 14% were neutral, 79% believed it was likely or very likely to continue, 7% didn’t know or didn’t respond
Attraction and retention of highly qualified personnel: 7% believed it was very unlikely or unlikely to continue, 14% were neutral, 79% believed it was likely or very likely to continue
Maintenance of equipment and laboratories: 14% believed it was very unlikely or unlikely to continue, 21% were neutral, 57% believed it was likely or very likely to continue, 7% didn’t know or didn’t respond
Administrative support for CERC unit(s): 7% believed it was very unlikely or unlikely to continue, 36% were neutral, 50% believed it was likely or very likely to continue, 7% didn’t know or didn’t respond
Indirect costs of research for chairholder: 29% believed it was very unlikely or unlikely to continue, 21% were neutral, 43% believed it was likely or very likely to continue, 7% didn’t know or didn’t respond
Direct costs of research for chairholder: 36% believed it was very unlikely or unlikely to continue, 29% were neutral, 29% believed it was likely or very likely to continue, 7% didn’t know or didn’t respond

Note that a level of funding sufficient to maintain an appropriate level of research capacity may be different (higher or lower) than the CERC award amount, and the level of critical mass of expertise and funding required in order to ensure the momentum of the research capacity likely varies by field, by type of problem to be solved, etc.

Looking to the future, many chairholders explained that the most challenging issue would be the retention of the CERC unit’s personnel (i.e., key researchers and HQP), who are essential to ensure the continuity of research, support services and the maintenance of equipment. And while most chairholders expressed a clear intent to remain at the host institution beyond the CERC award and have been planning accordingly, some indicated that they would consider taking a position elsewhere if they are unable to drive their research program forward due to resource and capacity constraints.

In this context, many chairholders, key researchers, and host institutions have advocated for some form of renewal of the CERC award, although the non-renewability of the award has been a feature of the CERC awards since the onset of the program and has been communicated as such by the Chairs Secretariat. Currently, a one-year phase-out period is allowed over which previously-allocated funding can be spread.

All that being said, it is important to note that since the first cohort of CERC awards have not yet completed their terms, it is too early to definitively assess this outcome (i.e., the extent to which research capacity will be sustainable in S&T priority areas). Early ramp-up of CERC units and ongoing institutional support in the CERC unit’s research area are both likely to play an important role in contributing to sustainable research capacity in the longer term. Institution-led sustainability plans that extend beyond the funding period were also noted as a best practice in the international review.

Recommendation 4: Identify, monitor and promote best practices for the sustainability of the research capacity developed as a result of the CERC awards (i.e., critical mass of researchers and HQP; infrastructure).

In light of these challenges, the evaluation concludes that the Chairs Secretariat could adjust some of its delivery features to facilitate the early implementation phase of the CERC awards. Some changes have already been made for this purpose in the second Competition (e.g., universities may include a request for CFI funding with their CERC nomination), but there remains room for additional flexibility in the use of CERC funds. New chairholders could also benefit from additional orientation, such as guidelines and best practices, to facilitate the implementation of the CERC unit, especially in the early, critical stages of the award.

Finally, some synergies with other funding mechanisms and the development of partnerships were both identified as success factors for the enhancement and sustainability of research capacity (i.e., via leveraged funding). However, opportunities were identified to ensure that funding mechanisms that can and should work in concert with the CERC award are used more systematically or in a more timely manner (e.g., CFI, CRC, support programs for HQP). Note that the need for clarifying expectations with respect to partnerships and leveraging is addressed in Recommendation 2.

1.0 Introduction

1.1 Background of the CERC Program

In November 2006, the Government of Canada released an economic plan to make Canada a world leader for today and future generations within a changing global economy, known as Advantage Canada^{Footnote 2}. This plan acknowledged that Canada already possessed strengths such as the ingenuity and drive of its people, and a strong research base. However, the plan contended that Canada must do more, by creating a well-educated, highly talented workforce, and by turning research discoveries into innovation to provide solutions to environmental, health and other important challenges, in order improve Canada’s economic competitiveness. In order to achieve these goals, the government released its science and technology (S&T) strategy—Mobilizing Science and Technology to Canada’s Advantage—in 2007.^{Footnote 3}

Launched in 2008 ^{Footnote 4}, the Canada Excellence Research Chairs (CERC) program supports Canadian universities in building a critical mass of expertise targeted within the government priority research areas of the Government of Canada’s S&T Strategy, in support of Canada’s growing reputation as a global leader in research and innovation. It aims to respond to policy commitments in the S&T Strategy, particularly to help achieve two of the three main objectives, namely the “Knowledge Advantage” and the “People Advantage”. More specifically, the CERC program provides up to $10M over seven years through a competitive peer-reviewed process to world-class researchers established at eligible Canadian universities to support not only the chairholder’s salary, but also the research team and other costs of research.

The following profile provides more details on the objectives and delivery of the CERC program, followed by an overview of governance, beneficiaries, resources and expected results of the program.

1.1.1 Profile

Objectives
The Government of Canada designed the CERC program to help strengthen Canada’s ability to attract and retain world-class researchers, in order to undertake cutting-edge research in priority areas that can positively contribute to Canada’s global economic competitiveness and the well-being of Canadians. The program objectives are as follows:

to strengthen Canada’s ability to attract the world’s top researchers in order to be at the leading edge of breakthroughs in priority research areas expected to generate benefits for Canadians;
to help Canada build a critical mass of expertise in the priority areas outlined in the Government of Canada’s S&T Strategy, including: environmental sciences and technologies, natural resources and energy, health and related life sciences and technologies, and information and communications technologies;
to create a competitive environment to help Canadian universities attract a cadre of world-leading researchers in their pursuit of excellence in their research; and
to brand Canada as a location of choice for world-leading research in science and technology development.

For each CERC, Canadian universities receive up to $10 million over seven years to:

provide competitive salaries and benefits for chairholders;
assemble outstanding research teams and cover their salary and benefits (e.g., researchers, research associates, technicians, graduate students, postdoctoral fellows);
fund the direct costs of the research program (consumables, small equipment, travel, knowledge sharing and dissemination, etc.); and
support the indirect costs of the research (up to 25 per cent of direct cost of research).

Delivery
The first group of Canada Excellence Research Chairs was awarded in May 2010 through a highly competitive, multilevel peer review process in the four priority research areas of the federal government’s S&T Strategy:

Environmental sciences and technologies (EST)
Natural Resources and Energy (NRE)
Health and Related Life Sciences and Technologies (HLTH)
Information and Communications Technologies (ICT)

In June 2011, the Government of Canada announced the creation of 10 new Canada Excellence Research Chairs to be awarded in the four priority areas, as well as in other key areas of benefit to Canadians. A minimum of three Chairs will be awarded in areas related to the digital economy under the information and communications technologies priority area. At least one Chair will be allocated to each of the remaining three priority areas, and four Chairs will be open to all areas of inquiry. The CERC program is an ongoing program: it is expected that a new competition will be held every seven years. As such, another competition is anticipated to be launched in 2015, when the first round of CERC awards nears completion.

Review Process
The Canada Excellence Research Chairs are awarded through a highly competitive, two-stage process: ^{Footnote 5}

In phase 1, Canadian universities compete for the opportunity to establish chairs at their institution. The program is exclusively excellence-based, i.e., no special consideration is given to applications based on region, size of applying university or factors other than those noted in the evaluation criteria.
In phase 2, universities that submitted successful Phase 1 applications are invited to nominate world-class researchers for the available chair positions. Nominations must demonstrate excellence, based on the evaluation criteria, including the world-class excellence of the nominee, the proposed program of research and the fit with the university’s proposal in Phase 1.

Phase 1 applications and phase 2 nominations to the CERC program undergo a multilevel peer review. The selection process is based on the highest standards of research excellence. It includes evaluation by external experts, assessment by the review panel, strategic review by the selection board ^{Footnote 6} and final approval by the steering committee.

First CERC Competition

In phase 1, 40 proposals submitted by 17 Canadian universities competed for the opportunity to establish Canada Excellence Research Chairs at their institution. In phase 2, 19 Canada Excellence Research Chairs ^{Footnote 7}were awarded at 13 institutions (Table 1). Chairs are not renewable after the initial seven-year period.

Table 1 CERC award recipients (Competition 1)

Recipient institution	CERC recipient	Originating institution/organization	CERC title	S&T priority area
Laval	Marcel Babin	Centre national de la recherche scientifique	Remote Sensing of Canada's New Arctic Frontier	NRE
	Younès Messadeq	Universidade Estadual Paulista – UNESP (Brazil)	Photonic Innovations	ICT
Western Ontario	Adrian Owen	University of Cambridge	Cognitive Neuroscience and Imaging	HLTH
Manitoba	Søren Rysgaard	Greenland Institute of Natural Resources	Arctic Geomicrobiology and Climate Change	NRE
Waterloo	David Cory	Massachusetts Institute of Technology	Quantum Information Processing	ICT
	Philippe Van Cappellen	Georgia Institute of Technology	Ecohydrology	EST
Ottawa	Robert Boyd	University of Rochester	Quantum Nonlinear Optics	ICT
UPEI	Ian A. Gardner	University of California, Davis	Aquatic Epidemiology	EST
Alberta	D. Graham Pearson	University of Durham	Arctic Resources	NRE
	Michael Houghton	Epiphany Biosciences Inc. (USA)	Virology	HLTH
	Thomas Thundat	Oak Ridge National Laboratory (USA)	Oil Sands Molecular Engineering	NRE
	Patrik Rorsman ^{Footnote 8}	University of Oxford	Diabetes	HLTH
Toronto	Frederick Roth	Harvard Medical School	Integrative Biology	HLTH
	Oliver Ernst	Charité - Universitätsmedizin Berlin	Structural Neurobiology	HLTH
McMaster	Ali Emadi	Illinois Institute of Technology	Hybrid Powertrain	EST
Sherbrooke	Bertrand Reulet	Université de Paris XI	Quantum Signal Processing	ICT
Dalhousie	Douglas Wallace	Leibniz-Institut fuer Meereswissenschaften, Kiel	Ocean Science and Technology	EST
Saskatchewan	Howard Wheater	Imperial College of Science, Technology and Medicine	Water Security	EST
UBC	Matthew Farrer	Mayo Clinic College of Medicine	Neurogenetics and Translational Neuroscience	HLTH

Note: EST: Environmental sciences and technologies; NRS: Natural Resources and Energy; HLTH: Health and Related Life Sciences and Technologies; ICT: Information and Communications Technologies
Note: Chairholders’ names are linked to their respective profiles posted on the CERC program’s web site (www.cerc.gc.ca)
Source: CERC Evaluation Design Report (2013)

Second CERC Competition

In 2011, the Economic Action Plan ^{Footnote 9} announced $53.5 million over five years for 10 new CERC awards, including chairs in fields relevant to Canada’s Digital Economy Strategy.

The results for phase 1 of the second CERC competition were announced in November 2012. As part of this competition, 46 proposals were submitted by 27 universities. Following the multilevel peer review process,^{Footnote 10} 11 applications ^{Footnote 11} from 8 universities were retained for phase 2. At the end of phase 2, 11 new CERC awards will be allocated in fields relevant to the four S&T priority areas, as well as other fields of research.

1.1.2 Governance

The CERC program is a tri-agency initiative of the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC). It is administered by the Canada Research Chairs (CRC) Secretariat, which is housed within SSHRC.

The CERC program is represented in each agency’s Program Alignment Architectures (PAAs). For SSHRC, the CERC program is located within program 1.1 Talent, sub-program 1.1.1 Canada Research Chairs; for NSERC, the CERC program is located within program 1.1 People, sub-program 1.1.7 Canada Excellence Research Chairs; and for CIHR, the CERC program is located within 1.2 Health Researchers, sub-program 1.2.1 Salary Support Programs, sub-sub-program 1.2.1.3 CERC program.

The CERC program is governed by a steering committee and a management committee. The steering committee is comprised of the Presidents of the three granting agencies (CIHR, NSERC, and SSHRC), the Deputy Ministers of Industry Canada and Health Canada, and the President of the Canada Foundation for Innovation (CFI; as an observer). The steering committee is responsible for providing strategic direction for the program and is the body that makes final decisions on funding recommendations made by the selection board.

The CERC management committee is composed of a representative at the Vice-President level from each of the three granting agencies and the CFI (as an observer), Industry Canada, Health Canada, as well as the Executive Director of the CERC program. It is chaired by SSHRC’s Vice-President, Research Programs. The management committee oversees the operation and coordination of the program administration, monitoring and communications.

The Chairs Secretariat, housed within SSHRC, is responsible for day-to-day administration of the CERC program including:

organization of the multilevel peer review process which is highly international;
provision of advice and guidance to institutions and chairholders;
ongoing financial and operational monitoring of compliance of recipients with the terms and conditions of the program;
performance measurement;
evaluation and management audits (through the evaluation division and the corporate internal audit directorate of SSHRC-NSERC); and
reporting on the program to the minister of Industry Canada, the Treasury Board of Canada Secretariat and, ultimately, the Parliament of Canada.

The Chairs Secretariat reports to the management committee, which in turn reports to the steering committee.

1.1.3 Key Beneficiaries

As mentioned, the CERC program is delivered by CIHR, NSERC and SSHRC through the Chairs Secretariat housed at SSHRC. Canadian universities along with research affiliates are key co-deliverers and direct beneficiaries of the program. They are responsible for nominating candidates, as well as supporting the CERC unit.^{Footnote 12} Direct and other beneficiaries are listed below.

In addition, direct beneficiaries of the CERC program include chairholders; members of the research team involved within CERC units (i.e., high-calibre researchers working closely with the chairholder within the CERC unit); and highly qualified personnel (e.g., graduate students, postdoctoral fellows, research professionals, technicians, coordinators).

Other beneficiaries include: other academic and research institutions within Canada and abroad; federal government departments (e.g., Industry Canada, Department of Foreign Affairs, Trade and Development, Health Canada, Department of Finance); other federal, provincial, territorial and municipal government; private sector and not-for-profit organizations; and the Canadian public.

1.1.4 Resources

The 2008 federal budget provided $207.4 million over five years through the granting agencies, beginning in 2008-09, to support the creation of 20 Canada Excellence Research Chairs. An additional $53.5 million over five years was announced in the 2011 budget for the creation of 10 additional chairs. Table 2 presents a summary of the funding history for the CERC program, from program inception in 2008-09 to 2012-13.

Table 2 CERC program funding, 2008-2013

Fiscal year/exercise	2008-2009	2009-2010	2010-2011	2011-2012	2012-2013
Grants	$0	$0	$15,366,666	$25,533,333	$25,200,000
Operating expenditures	$503,516	$440,159	$788,099	$926,796	$936,616
TOTAL	$503,516	$440,159	$16,154,765	$26,460,129	$26,136,616

Note: The funding profile for each seven-year grant depends on the start date of the chairholder at the recipient institution. Since the start dates are staggered between April 2010 and August 2011, the CERC program requests will ramp up and down over the course of nine years.

Source: CERC Evaluation Design Report (2013); CERC financial datasets from the NSERC-SSHRC Finance Division (2014)

1.1.5 Expected Results

Through the achievement of the CERC program’s key objectives outlined above, the program is expected, over the long-term, to contribute to sustaining a world-class research environment in Canada that will enhance the country’s competitiveness in the global, knowledge-based economy, thus improving the overall prosperity of Canadians. A Performance Measurement Strategy (PMS) was developed and implemented for the CERC program in 2009 ^{Footnote 13} which included a risk assessment and mitigation strategy to ensure that the program is managed in a manner sensitive to risks, complexity, accountability for results and efficient use of resources.^{Footnote 14} The PMS outlines the activities, outputs and the expected immediate, intermediate and long term outcomes. These are graphically depicted in the CERC program Logic Model in Appendix A, along with a description of program outputs and outcomes (Appendix B).

Given that the nature of impacts of R&D investments are necessarily long-term, it is important to recognize that evidence of the return on this investment will only begin to materialize at the end of the initial seven-year funding period. At the same time, the PMS for the CERC program was designed to capture indicators of immediate and intermediate outcomes, as well as indicators of progress towards achieving longer-term outcomes in order to best capture and report on program results on an ongoing basis. ^{Footnote 15}

1.2 Evaluation Scope and Objectives

This is the first evaluation of the CERC program and, in accordance with section 42.1 of the Financial Administration Act and the Treasury Board Policy on Evaluation (2009),^{Footnote 16} it covers the first five years of the program, from its inception in 2008-09 to 2013-14. It should be noted that although the program was created in 2008, funds for the first competition only began to flow in 2010 and only phase 1 of the second competition is complete. As a result, the scope of the evaluation will cover the inception of the program to the end of phase 1 of the second competition. Since it is still early in the program’s lifecycle, the assessment of program performance focused on the program’s immediate outcomes, and the assessment of economy and efficiency was limited to the operational efficiency of the program (i.e., the relationship between resources and outputs).

The evaluation addressed key questions of relevance, performance, design/delivery and efficiency in order to be maximally useful to CERC management and staff for future program improvements and planning. In particular, it was designed so as to provide an opportunity for course corrections regarding the program’s two initial rounds of chairs, and the subsequent launch of the program’s third competition.

1.3 Evaluation Issues and Questions

The evaluation was designed to address the five core evaluation issues stipulated in the Treasury Board Policy on Evaluation (2009), which fall within two broad categories: relevance and performance. Table 3 identifies the specific evaluation questions that were developed for each core evaluation issue, which were grouped under main themes to limit redundancies and to present a coherent and structured narrative in this report.

Table 3 Evaluation issues and questions, grouped by main themes

Evaluation issues	Main themes	Evaluation questions and sub-questions
Relevance	Continued need for the CERC program	1. To what extent is there a continued need for the program in light of the current national and international contexts?
	Alignment with federal priorities, roles and responsibilities	2. Do the objectives of the CERC program continue to be relevant with government priorities (as articulated through the S&T Strategy)?
		3. Is there a legitimate and necessary role for the federal government in providing funding for the CERC program?
Performance – Effectiveness	Attraction and retention of world-class researchers, high-calibre researchers and Highly Qualified Personnel (HQP)	4. To what extent has the CERC program contributed to the capacity of host universities to attract and retain (i.e. sustain) high-calibre researchers and highly qualified personnel from Canada and the world?
		4.1 To what extent are there barriers to the attraction of world-class researchers and how has the CERC program addressed these barriers?
	Visibility of CERC units, host institutions, the CERC program and Canada	5. To what extent has the CERC program contributed to raising awareness of Canada as a location of choice for leading research?
	Partnerships and collaborations in S&T priority areas	7. To what extent have the CERC units established the necessary partnerships with co-creators and/or receptors of innovation?
	Enhanced and sustainable research capacity in S&T priority areas	6. To what extent has the CERC program contributed to enhanced and sustainable research capacity at universities in the S&T priority areas?
		8.1 To what extent do the CERC units have the level of resources required (from the program, from universities and from other sources) to build a sustainable critical mass in S&T priority areas?
Performance – Program Design and Efficiency	Program design and efficiency	8. To what extent are the most effective and efficient means being used to achieve program outcomes?

Source: CERC Logic Model; CERC Evaluation Design Report

2.0 Methodology

The evaluation used multiple lines of evidence and data sources for an in-depth and comprehensive analysis, with implementation shared between external consultants, Science-Metrix and SSHRC-NSERC’s internal evaluators. As well, an Evaluation Advisory Committee provided advice and reviewed main deliverables throughout the evaluation process (e.g., development of the evaluation design and implementation of the evaluation study).

A data collection matrix was used to map evaluation issues, questions, and indicators to the data collection methods and data sources (Appendix C). Eight data collection methods were used to address the ten evaluation questions: document review, review of administrative and performance data, cost-efficiency analysis, interviews, international comparison study, survey, bibliometric analysis and case studies.

2.1 Methods

The eight data collection methods and lead role for each team are presented in Table 4.

Table 4 CERC evaluation methods and lead team

Evaluation method

Lead role

Document review
Internal documents (180 documents):

Project file review (e.g., application and nomination files (Competitions 1 and 2), monitoring documents)
Administrative documents (e.g., Steering and Management committee minutes; program foundational and operational documents)
Special program studies and reports (e.g., report on best practices)

External documents (32 documents):

National and international studies/reports (e.g., gender issues, S&T Strategy, budget)

Literature on comparable programs (e.g., CRC evaluation report, websites)

SSHRC/NSERC Evaluation Division

Review of administrative/performance data

Award-specific files and data:

	Phase 1 Application Form	Phase 2 Nomination Form	Progress Reports (2010-2013)	Statements of Account (2010-2013)	Case-Study Questionnaire
1st competition	✔	✔	✔	✔	✔
2nd competition	✔

Media and promotional material relating to the CERC program and individual CERC awards (e.g., media analytics for the CERC website, media monitoring [Media Miser], presence/use of social media [Twitter, YouTube], media hits, press releases, announcements and communications-related CERC events)

SSHRC/NSERC Evaluation Division

Cost-efficiency analysis

Financial data available for administrative and grant expenditures, fiscal years 2010-11 to 2012-13
Financial data on comparable programs (i.e., NCE and CRC), fiscal years 2010-11 to 2012-13

SSHRC/NSERC Evaluation Division

Key informant interviews (n=26)
Design phase (n=21)
Internal stakeholders

CERC Management and Steering Committees (7)
Chairs Secretariat (4)
SSHRC Communications Division (2)

External stakeholders

CERC Review Panel (1)
University administrators (4)
CERC chairholders (3)

Evaluation phase (n=5)

CERC chairholder (1)
CERC Review Panel (1)

International funding organizations (3)

SSHRC/NSERC Evaluation Division

International comparison study

Initial review of 38 programs; in-depth review of 9 short-listed programs (selected based on criteria in terms of their similarity to the CERC program)
Review and validation of the publicly available data for 7 of 9 short-listed programs by program representatives

Chairs Secretariat/ SSHRC-NSERC Evaluation Division

Case studies (n=18)

Case study questionnaires (i.e., collection of baseline data on CERC units in preparation for site visits/case studies)
Site visits
Interviews with 10-20 representatives from each CERC unit (e.g. chairholders, key researchers in the CERC unit, HQP, collaborators/partners)
Document and literature review including promotional material from CERC units collected during site visits

Science-Metrix/ SSHRC-NSERC Evaluation Division

Survey of university representatives (n=20; 100% response rate)

All unsuccessful and successful nominating institutions for the CERC program (Competitions 1 and 2)

Science-Metrix

Bibliometric Analysis

All unsuccessful and successful nominees for the CERC program (Competition 1); analysis of scientific output (i.e., peer-reviewed papers) pre-and post-nomination to the CERC program

Science-Metrix

2.2 Limitations

Each of the methods used in this study has its strengths and limitations. Several of these challenges were identified early in the evaluation process and associated mitigation strategies were proactively built into the design. For instance, data collection tools for the case studies were pre-tested collaboratively to ensure that data collected by multiple analysts would be comparable; control groups were used in the bibliometric analyses to support conclusions that attribute observed changes to the CERC program; and data validation steps were included to compare information collected across multiple sources (e.g., surveys, administrative data review, case studies).

Indeed, triangulation of findings across multiple lines of evidence is a key strategy to determine areas of convergence or divergence; unless otherwise noted, the findings in this report converge across multiple sources. The main limitations relate primarily to data availability and quality issues, as well as to the specific context of the CERC program (e.g., small population size, current stage in the program lifecycle). When these limitations affected specific findings or conclusions, these are noted explicitly in the evaluation report so that the findings can be interpreted accordingly. More generally:

Lack of consistent performance and financial data (for Competition 1) (i.e., several key variables have not been consistently tracked as part of application/nomination forms and progress/financial reports throughout the duration of the CERC awards), as well as the gaps in the data available often reduced the scope of analyses conducted using these data, and weakened conclusions that could be drawn from these findings. Data drawn from progress reports and case study questionnaires were self-reported by institutions and/or chairholders, and as a result, these sources of evidence are subject to some degree of bias.
Despite an excellent response rate to the survey (100%), the overall population was quite small (N=20), thus limiting the generalization of findings and the analysis by subpopulation. As well, the small population size of successful (i.e., 24 individuals) and unsuccessful (12) nominees for the CERC program required that all bibliometric analyses be performed at the aggregated level, (i.e., pooling of data for each group of researchers, independent of their field of research). The small sample size likely also contributed to a lack of statistical power for statistical tests applied to these data (i.e., tests are not statistically significant even when large differences are observed).
Because the CERC program is still early in its lifecycle, the evaluation study focused primarily on immediate outcomes and only examined phase 1 of the second competition. It also remains too early to draw conclusions on a number of questions of interest (e.g., impact of measures taken by institutions to address equity issues, how CERC units will fare after the end of the awards).
At this stage of the program’s lifecycle, there were a limited number of outputs (i.e., peer-reviewed papers) produced since the launch of the CERC units. This constrained the scope of the bibliometric analysis to a small number of indicators that could be reliably calculated. Thus, bibliometric evidence on the impact of the CERC program should be considered preliminary; more robust analyses will be possible in coming years, as more outputs are produced by the CERC units.

3.0 Findings

3.1 Relevance

3.1.1 Continued need for the CERC program

Summary of Findings:
The evaluation evidence supports continuing the program. The CERC program was initially launched in response to the S&T Strategy (2007) to achieve global excellence in research, and the current national and international contexts reinforces the need to continue supporting the CERC program to help ensure Canada remains competitive at the global level.

Supporting world-class research excellence remains just as important—if not more so—given changes in the international research context since the CERC program inception in 2008, in particular with global shifts in R&D investments that are bringing new players to the forefront (e.g., BRICS countries). While other countries are now catching up to Canada with significant investments in research and higher education, Canada lags behind on several indicators relating to R&D expenditures. In particular, Canada’s R&D funding from the business sector has been declining since 2002; the country also faces ongoing challenges moving knowledge from higher education institutions to companies that can translate this knowledge into products that could benefit society.

In this context, Canadian higher education institutions continue to play a critical role in R&D. Indeed, Canada now faces fierce global competition to attract top research talent, with several countries having launched programs designed specifically for this purpose, many of which resemble the CERC program. The evaluation found that, compared to other similar international programs, the CERC award is still competitive on a global scale based on its value and duration.

In addition, the evaluation found that the CERC program is generally perceived to be complementary and has synergies with other national funding programs that, similarly, respond to specific S&T policy commitments, in particular tri-agency and CFI programs. Given similarities between the CRC and the CERC program in the pursuit of research excellence and in building research capacity, surveyed institutions confirmed the need to capture data and further explore potential synergies between these two programs, in order to demonstrate the added-value of the CERC program as it matures. However, the evidence seems also to point to some gaps in the data as well as a lack of uptake and awareness of some existing tri-agency funding opportunities, such as the Vanier CGS scholarships program.

Evolution of the international and national contexts
In order to understand whether there is a continuing need for the program, it is important to appreciate how the environment in which research is conducted has been changing. Ample documentary evidence pointed to an increased recognition worldwide that research excellence is a key driver in supporting innovation, as well as the fact that Canada faces fierce global competition to attract top research talent. ^{Footnote 17}

Global and national trends in R&D funding

Worldwide, the effects of the 2008 economic crisis led to a decline of business R&D in 2009 by close to 1%, down from the 5% growth seen in 2008, with global R&D expenditures decreasing from an annual growth of 4.7% in 2008 to 1.8% in 2009.^{Footnote 18} After the 2008 crisis, many governments identified innovation as being central to building stronger and more inclusive economies, capable of offering better jobs and long-term competitiveness. The current international landscape for innovation is therefore reflected by increasing investments in research and development (R&D). This landscape has also seen the emergence of new and important players such as the BRICS countries (i.e., Brazil, Russia, India, China and South Africa) and Asian countries in particular. For instance, China alone accounted for almost a third of the global increase in R&D between 2001 and 2006, as much as Japan and the EU combined. These countries are investing in the education system, and capitalizing on the commitment of their governments to innovate and rapidly ascend the value chain, which is a challenge for Canada.^{Footnote 19} As noted in a recent assessment from the Council of Canadian Academies: “The exceptionally rapid rise of China as a research powerhouse is now likely influencing a perception of relative decline in countries already at or near the top, like the United States and Canada.”^{Footnote 20}

In the context of shifting global investments in R&D, a range of documentary evidence was found that shows that Canada lags behind several countries on several indicators relating to R&D expenditures. For example, Canada’s R&D funding from the business sector has been declining since 2002,^{Footnote 21} and Canada ranked 15th among 20 peer OECD countries in 2011 for this indicator.^{Footnote 22} Canada also faces chronic challenges, especially with respect to knowledge transfer from higher education institutions to companies that have the ability to translate this knowledge into products that could benefit society. In particular, there is a lack of collaboration in R&D between academics and industries, for which Canada placed 15th out of 144 economies in the world.^{Footnote 23}

Need for ongoing support to Canadian higher education institutions to address innovation challenges

The document review also showed that Canada’s universities and colleges play a critical role in developing and advancing knowledge and its application. While the link between research and innovation is complex and the task of commercializing new knowledge is difficult and uncertain, advances in research conducted in the higher education sector are necessary to most innovation processes, not to mention their role in fostering research talent ^{Footnote 24}. Thus, according to the Science, Technology and Innovation Council’s (STIC) State of the Nation 2012 report,^{Footnote 25} higher education institutions still play a leading role in R&D in Canada and benefit from strong public funding: they received over 50% of total direct federal R&D funding in 2012. Indeed, the federal government has been making ongoing investments in higher education R&D since May 2007, when the federal government launched its national S&T Strategy, Mobilizing Science and Technology to Canada Advantage, which is committed to improving Canada’s long-term competitiveness and quality of life, to address Canada’s pressing economic and societal challenges.^{Footnote 26} In particular, Budget 2008 announced two new programs, the CERC program and Vanier Canada Graduate Scholarships (CGS), which were designed to help address these challenges by responding to specific S&T policy commitments—attracting and retaining international talent to Canadian higher education institutions, supporting world-leading research in S&T research priorities, and encouraging collaboration and commercialization.^{Footnote 27} Given the current national and international contexts, the evidence provided here therefore suggests that the need for the CERC program remains as relevant today, if not more so, than when the program was created. The CERC program will undergo another evaluation within the next five year period, at this time additional information will be available regarding the program’s relevance and need.

The ongoing need for such programs is further supported by a recent assessment from the Council of Canadian Academies, which stressed that Canada should “sustain its hard-won status as a global research leader because research excellence is essential to […] underpin the production of highly qualified people, trained at the leading edge so as to enhance the innovative capacity of Canadian business” and to “ensure that Canadians have ‘insider access’ to the latest global knowledge pools since inclusion in the best international networks depends on the quality of one’s contributions.”^{Footnote 28} University associations are also calling for the R&D talent and capabilities within Canadian higher education institutions to be further developed through sustained public investments.^{Footnote 29}

Competing programs targeted at attraction and retention of international talent

As noted above, other countries are now catching up to Canada with significant investments in research and higher education. A portion of foreign R&D investments were found to be targeted specifically at attracting world-class researchers. Governments throughout the world are developing programs to attract highly educated and highly skilled foreign talent which are deemed critical for creating new knowledge, technologies and innovation, key to economic growth.^{Footnote 30} The international comparison study examined similar and competing programs in light of the current international context and found that a growing number of countries had launched programs very similar to the CERC program (9 programs), as well as other initiatives (24 programs) aimed at attracting and retaining world-class researchers. These included EU countries (e.g., Germany, Denmark, Finland, Belgium, Luxembourg, Sweden, Ireland), as well as Singapore, the United Arab Emirates, China, Russia and Australia. Case study respondents also noted the recent surge of interest from Asia to recruit international researchers, but several programs in these countries could not be assessed as part of the international comparison study, given limited availability of publicly available material in English or French.

Furthermore, the majority of international programs under study have been launched fairly recently ^{Footnote 31}, which may be interpreted as a global trend in light of an increasingly competitive environment to attract world-class researchers. According to external key informants, these programs are similar to the CERC program, in that they have been designed to act as catalysts in order to strengthen their country’s research capacity, and overall research excellence, as well as encourage academic mobility. Several of these programs offer awards that are almost equivalent in value to the CERC award, but few are greater in value and duration (i.e., the Swedish Research Council and the Odysseus program funded by the Flemish Research Foundation offer awards that could exceed CDN$2 million per year). Thus, despite growing competition for top research talent, the CERC award remains competitive on a global scale.

It is worth noting that none of the short-listed programs are from the countries with which Canada has traditionally competed for S&T personnel, such as the United States and the United Kingdom. In contrast to the robust upsurge in research funding investments in emerging economies, some chairholders and high-calibre researchers reported that there have recently been substantial cuts in public funding for research in the US and UK. In a few cases, this was even reported as a reason for accepting the CERC award and relocating to Canada. According to a few interviewees and evidence from the international comparative study, Canada is well positioned to take advantage of this funding decline.

Complementarity with other national funding programs

In addition to the CERC program, the federal government supports a number of other programs specifically designed to support Canadian research excellence (including the international recruitment of research talent) and the commercialization enterprise. The evaluation found that the CERC program is generally perceived to be complementary and/or to have synergies with other national funding programs that, like the CERC program, respond to a range of specific S&T policy commitments—attracting and retaining international talent, supporting world-leading research in S&T research priorities, and encouraging collaboration and commercialization. These programs include the CRC, the Vanier CGS, the Banting Postdoctoral Fellowships, the Networks of Centres of Excellence (NCE), and the Industrial Research Chairs (IRC). For instance, some CERC chairholders have established collaborations with existing NCE networks and one chairholder even acts as scientific director of an NCE.

In fact, the evaluation evidence pointed to several instances in which the CERC program and investments from other national funding programs worked in concert to achieve the outcomes of the CERC program, particularly tri-agency and CFI programs. Documentary evidence and case study questionnaire data found that the second-most significant source of additional research funding obtained by CERC chairholders and high-calibre researchers was from tri-agency programs, which amounted to $31 million between 2010 and 2014 across the 18 CERC units. The state-of-the-art equipment and facilities established as part of the CERC units have also benefited from a total of $22 million in additional funding for research infrastructure provided by the CFI Innovation Fund and Foundation Scheme since 2010. Some CERC chairholders have also used the CREATE program to support their training programs. Similarly, 65 students and postdoctoral fellows (10%) received direct funding from tri-agency scholarships/fellowships programs. However, this percentage is relatively low in comparison to the total number of students and postdoctoral fellows involved within CERC units. As well, during the case study interviews, many foreign students seemed unaware of potential sources of funding, such as the Vanier CGS scholarships program.

Given similarities between the CRC and CERC programs in the pursuit of research excellence, and in building research capacity within Canadian universities based on institutional strategic plans, it is of interest to note that some CRC chairholders are currently collaborating with CERC units at their host institution, although there is no systematic data collected on the number of CRC chairholders involved within CERC units. The CERC program distinguishes itself from the CRC program based on the small number of chairs and the much larger award amount. The CERC program also has a more explicit international perspective than the CRC program: whereas the CRC program also serves to recruit international researchers, only 16% of CRC chairholders were recruited from outside Canada (active chairs as of March 2014),^{Footnote 32} whereas 100% of CERC chairholders came to Canada from abroad.^{Footnote 33} That said, given that some surveyed institutions inquired about the value of the CERC program in relation to the CRC program, it would be beneficial to capture and analyze data on the relative impact and synergies between these two programs in order to demonstrate the added-value of the CERC program as it matures.

3.1.2 Alignment with federal priorities, roles and responsibilities

Summary of Findings:
The objectives of the CERC program are closely aligned with the federal government priorities outlined in the Government of Canada S&T Strategy. In particular, CERC awards are provided primarily in the four priority research areas and directly support the “Knowledge Advantage” and the “People Advantage” outlined in this strategy. In line with the S&T Strategy, effective relationships with receptors of innovations have been highlighted as a key driver for innovation (“Entrepreneurial Advantage”), to which the CERC program contributes by promoting the development and application of leading-edge knowledge. Past and ongoing federal investments for innovation and research programs (including recent funding commitments for new CERC awards) also provide evidence that such programs continue to represent a priority for the government.

The evidence further confirms the legitimacy of the federal roles and responsibilities with regard to the CERC program, as per the legislated roles of Industry Canada and the granting agencies that oversee the program. Internal and external stakeholders, as well as a recent review of federal support for R&D, underscored the necessity of federal funding to support basic and applied research in the higher education sector. Finally, the role of provincial governments was seen to be complementary to that of the federal government, especially since the CERC chairholders have leveraged $21 million to date (13% of total funding leveraged from other sources) from their respective provinces to support the CERC units.

Alignment of CERC program objectives with government priorities

Ample evidence was found in reviewed documents that the objectives of the CERC program are closely aligned with the federal government priorities. More specifically, the CERC program serves to extend federal government’s overall support for higher education R&D in fields of strength, opportunity and relevance to Canada. Such efforts are guided by the S&T Strategy,^{Footnote 34} to which the CERC program directly contributes by funding chairs in its four priority areas and supporting Canada’s “Knowledge Advantage,” “People Advantage”, as well as contributing to the “Entrepreneurial Advantage”.

The S&T Strategy committed the federal government to create a competitive environment, based on its existing strengths (e.g., university-based research) that would attract and support international researchers, promote world-leading research, and use S&T to develop practical applications to address pressing challenges.^{Footnote 35} The majority of key informants and 70% of surveyed institutions confirmed that the CERC program was most relevant to help Canadian universities build a critical mass of expertise in these research priority areas. However, a few interviewees raised the importance of addressing emerging trends in addition to the S&T priority areas, in light of the current international context.

A majority (60%) of surveyed institutions also reported that the CERC program was relevant to a large or very large extent to attract/retain Canadian and international leading researchers at Canadian institutions and to maintain Canada’s international reputation for research excellence (i.e., “Knowledge Advantage”). These objectives are also aligned with the S&T Strategy, which aims to attract a world-class workforce crucial to the innovation process (i.e., the “People Advantage”).^{Footnote 36}

The CERC program also contributes to Canada’s efforts to build an “Entrepreneurial Advantage” by promoting the development and application of leading-edge knowledge.^{Footnote 37} Such efforts require the development of effective relationships with receptors of innovations to ensure knowledge transfer and application of research results to improve Canada’s competitiveness; as such, strengthening these relationships has been identified by the STIC as a priority for Canada in the context of R&D.^{Footnote 38} As noted in the previous section, Canada faces ongoing challenges with regard to knowledge transfer—effectively moving knowledge developed in higher education institutions to industry and other research users. Indeed, “universities and colleges can anchor clusters of innovative activity [...] and act as bridges or catalysts between businesses, governments and other countries”.^{Footnote 39} The CERC program was designed to facilitate the establishment of such bridges between academia and other sectors involved in the innovation chain, as reflected by its intermediate outcome of “Strengthened relationships with receptors of innovation, insights and HQP”.

Past federal investments for innovation and research programs that provide a benefit to the Canadian economy confirm that the government continuously strives to foster the development of a strong Canadian research capacity. Since 2006, over $11 billion has been allocated to initiatives supporting basic and applied research, talent development, research infrastructure, and innovative activities in the private sector.^{Footnote 40} The government also reaffirmed its commitment to the CERC program itself with ten new CERCs announced in the 2011 federal budget.^{Footnote 41} More recently, the government committed $1.6 billion over five years to support research and innovation as part of the 2014 Economic Action Plan, notably through the creation of the Canada First Research Excellence Fund (CFREF).^{Footnote 42} The CFREF will provide support to postsecondary institutions through increasing funding over the years, starting with $50 million in 2015-16, reaching $200 million annually in 2018–19 and beyond.

Alignment with federal roles and responsibilities

All lines of evidence support the legitimacy of the role of the federal government in the delivery of the CERC program. Two of the three granting agencies (i.e., SSHRC, NSERC) fall under the umbrella of Industry Canada, whose mandate and responsibilities relating to Canadian S&T activities and policy goals stem from the Department of Industry Act. The CERC program objectives are consistent with Industry Canada’s objectives as set out in its founding act, i.e. to “encourage the fullest and most efficient and effective development and use of science and technology” and “foster and promote science and technology in Canada.”^{Footnote 43}

There was also a general consensus among internal and external stakeholders consulted that federal government funding is essential to the achievement of the CERC program objectives, given that there are no other programs or sources of funding in Canada that currently provide equivalent funding (i.e., duration and value of awards) to achieve similar outcomes. The importance of government support for basic and applied research in the higher education sector was also clearly highlighted in the Expert Panel Report on Federal Support for R&D.^{Footnote 44} Most notably, the report states that, while primarily funded by the provinces, the federal government has traditionally played a key role over the years through transfer payments, student financial assistance and direct support. Furthermore, the report notes that “Canada’s innovation gap is partly an education gap” which can be reduced through “a collaborative approach that brings together our post-secondary institutions, federal and provincial agencies as well as industry and other partners to ensure appropriate recruitment [of talent], training and deployment for industrial innovation needs”^{Footnote 45}.

The evidence also confirms that the role of provincial governments is complementary to that of the federal government in the context of the CERC program. Provinces have jurisdiction over education and health, and therefore respond to similar policy commitments as the federal government, but their actions relating to support for S&T in the higher education sector are more focused on achieving regional benefits.^{Footnote 46} There is also considerable variation in the scale and form of provincial investments in the higher education sector.^{Footnote 47} For instance, larger provinces such as British Columbia, Alberta, Ontario and Quebec typically invest in larger and more diverse types of programs, while smaller provinces are making efforts to catch up. In comparison, the federal government has a greater capacity to provide S&T funding and operates at a pan-Canadian scale. Some provincial programs seek explicitly to complement federal agencies (e.g., seed and matching grants), while others seek to increase the competitiveness of provincial actors in obtaining federal grants and in attracting faculty and students.^{Footnote 48}

In the case of the CERC program specifically, chairholders were encouraged to reach out to their respective provinces for complementary funding, which resulted in $8.2 million of additional funds leveraged from provincial governments, representing 6.4% of total funding leveraged from other sources. Interestingly, some key respondents expressed interest in having the evaluation examine provincial government involvement and support for the program given the possibility of potential benefits at this level.

3.2 Performance – Effectiveness

3.2.1 Attraction & retention of world-class researchers, high-calibre researchers and HQP

Summary of Findings:
Overall, the CERC program enabled the attraction and the recruitment of world-class researchers, as well as facilitated the attraction and/or involvement of hundreds of high-calibre researchers and HQP in CERC units. A large proportion of this talent was recruited from foreign institutions. Without the CERC award, all chairholders, some key researchers and most HQP reported that they would not have moved to the institution or even to Canada.

Attraction of world-class researchers was aided by the prestige of the CERC award and the pre-existing critical mass of research expertise at the host institutions. The award itself was essential to recruitment and chairholders identified specific aspects of their research program that would not have been possible without the award. However, clear evidence supporting the existence of barriers in the attraction and retention of world-class researchers was found; many of these barriers also applied (but to a lesser extent) to other types of researchers. These barriers included institution-related issues, non-renewability of the award (and other award terms), personal issues, and issues related to program delivery for the first competition (i.e. short timeframe for submitting nominations).

Recruitment of high-calibre researchers and HQP was often greatly facilitated by the flexibility and long-term horizon of the CERC award (which were seen to allow for participation of high-quality/high-impact research, as well as other beneficial opportunities for career advancement). For example, recruitment was facilitated by the fact that about 70% of high-calibre researchers were already working at the host institution when the CERC unit was established, by the support offered by the institution, and by the unique opportunity to work with a world-class researcher—often related to his exceptional personal qualities, vision and leadership. Some specific barriers were noted for the recruitment of Canadian HQP, particularly doctoral students and postdoctoral fellows.

Concerns of potential barriers to access and equity were also raised after the first competition, at which time no women were nominated for a CERC award. This issue prompted a review of gender issues facing the program by the Ad Hoc Panel on CERC Gender Issues, which identified potential key factors related to barriers to equity in the CERC program design and also into the broader university context. The panel’s recommendations resulted in changes to the second CERC competition, which have been well-received. It is somewhat premature to assess the impact of these changes at this time other than to note that the first chairholder to be appointed in the second round was a woman.

Finally, the CERC award was found to have a positive impact on the scope of research conducted, the scientific production and career of chairholders, high-calibre researchers and HQP. In the absence of the CERC award, chairholders reported that they would not have had the opportunity to carry out research programs of such a large scale and scope. Many of them would have dropped key components of their research program or carried out fewer, more focused projects within a much less integrated/coordinated and multidisciplinary group. Notwithstanding, CERC units faced a few challenges, such as delays in setting-up the research group/lab and career-related issues for high-calibre researchers and HQP.

Attraction of world-class researchers, high-calibre researchers and HQP

Overall, the CERC program enabled the recruitment of world-class researchers and contributed to the attraction and/or involvement of hundreds of high-calibre researchers and HQP. Note that, in this report, the term “world-class researchers” refers to internationally recognized as world leaders or rising stars with exceptional potential (i.e., the calibre of researchers recruited to be CERC chairholders). The term “high-calibre researchers” is used to refer to other key researchers involved in the CERC unit.

This finding is supported primarily by bibliometric, documentary, survey and case study evidence. Additionally, the administrative data review provides descriptive quantitative information related to the distribution of chairholders by S&T priority area and the numbers of high-calibre researchers and HQP involved in CERC units, which can be used as a baseline to track these indicators over time. For example, more than half of all high-calibre researchers associated with CERC units are conducting research in the environmental sciences and technologies priority areas (note that a CERC unit can be working in more than one priority area), even though there are slightly more chairholders associated with the natural resources and energy (7) and the health and related life sciences and technologies (8) priority areas than in the area of environmental sciences and technologies (6).

To demonstrate the success of the CERC program in selecting the best candidates, the scientific publications of successful nominees were compared to the publications of unsuccessful nominees prior to the date of their application. Both groups consisted of very high performers and were quite comparable in terms of their scientific performance, based on most of the examined measures (Table 5). Successful nominees produced more papers (median) than unsuccessful ones, but this difference was not statistically significant, due to the small sample size. However, a significant difference was observed in the scientific impact/quality of the papers produced: the median Relative Impact Factor (RIF) score was significantly higher for successful nominees than for unsuccessful nominees, based on a gap of 22 percentage points between their scores.

In addition, successful nominees were compared to averages for Canada^{Footnote 49} and the world^{Footnote 50} in order to assess whether chairholders were outstanding on a national and international scale. This analysis shows that successful nominees are clearly world-leading scientists as they out-perform both Canada and the world based on all of the indicators of scientific impact and quality examined in the bibliometric analysis. For instance, between 2005 and 2009, the median Relative Citation (RC) and RIF scores of successful nominees show that their publications achieved a much higher level scientific impact or quality compared to either Canadian or world researchers.^{Footnote 51}

Table 5 Scientific performance of successful CERC nominees compared to unsuccessful nominees, Canada and the world, 2005 to 2009

Group	No. of Nominees	Int'l. Co-publication Rate†	Papers‡ (Median)	Relative Citation‡ (Median)	Relative Impact Factor‡ (Median)
Succesful nominees Unsuccessful nominees	24 12	45% 49%	39 28	1.30 1.23	1.31* 1.09
Canada in NSE and Health World in NSE and Health	n.c n.c	46% 21%	n.c n.c	0.70 0.53	1.00 0.90

Note: The comparative analysis of the various groups examined the five years prior to the date of nomination, which was near the end of 2009 for all nominees. ‡ The hypothesis testing was performed on the median using a non-parametric test (i.e., One-Tail Mann-Whitney U Test; H0: Successful ≤ Unsuccessful).†For the international co-publication rate, a Z-test for two independent proportions was used (H0: Successful ≤ Unsuccessful). Where the null hypothesis is rejected (p-value ≤ 0.05), the score for successful nominees is noted with an asterisk.

Source: Computed by Science-Metrix using the Web of Science (Thomson Reuters)

These findings confirm that, when grouped together, researchers nominated by institutions who were awarded CERC chairs are among the best researchers in the world and their accomplishments have resulted in a high level of scientific impact in their respective fields.^{Footnote 52} Some reviewers ranked CERC candidates among the top 5% in their respective fields, and a few chairholders were even considered to be contenders for the Nobel Prize. It is worth noting here that the chairholders were selected based not only on their excellence as a researcher, but also on the quality of their proposed program of research, as well as the fit with the university’s proposal in phase 1 (e.g., fit with the vision and institutional commitment; benefits to Canada of the proposed research). The review of the selection board,^{Footnote 53} based on input from the review panel, resulted in the recommendations of the top nominees^{Footnote 54} who met or exceeded the expectations of excellence established for the CERC program (i.e., in many cases^{Footnote 55} chairholders were rated as significantly exceeding expectations). All chairholders were recruited from institutions outside of Canada.

The survey and case study evidence also showed that the CERC units have attracted a high number of high-calibre researchers and HQP. More than 90% of the surveyed institutions reported an increase in the number of high-calibre researchers and all reported an increase in the number of HQP. Only one institution reported that the number of high-calibre researchers did not change because researchers involved with the CERC units had already been affiliated with the institution.

Overall, the analysis of self-reported data on CERC units’ composition (administrative data review) indicates that in total 303 high-calibre researchers were closely involved with the CERC units and 826 HQP benefited from CERC funding, via direct funding and/or state-of-the-art training opportunities. The HQP group includes 494 students at all levels of postsecondary education, 126 postdoctoral fellows and 206 “other” HQP, such as research professionals and technicians (Table 6). Within these totals, 35% of HQP are foreign (i.e., 62% of doctoral students, 71% of postdoctoral fellows). Many chairholders interviewed during the case studies stated that they recruited a substantial number of foreign students, as they sought to have their research teams comprised of the best researchers, wherever they come from.

Table 6 Number of Canadian and foreign students and non-students

	Undergraduate	Masters	Doctoral	Postdocs	Other†	Total
	Students			Non-students		Total
Canadian	109	117	68	37	191	522
Foreign	23	44	112	89	15	283
Total	153*	161	180	126	206	826

Notes: †“Other” non-students include coordinators, project managers, technicians, etc.. *21 undergraduate students were not classified as Canadian or foreign
Source: Case Study Questionnaire (2014)

Facilitators and barriers to attraction and retention of world-class researchers, high-calibre researchers and HQP

The evaluation found that, while most CERC units faced few or no barriers to the attraction of high-calibre researchers and HQP, some chairholders and institutions reported significant issues related to the attraction of world-class researchers. In this section, facilitating factors associated with the CERC program will be discussed first, followed by a discussion of barriers to the attraction and retention of research talent.

In terms of high-calibre researchers and HQP, stakeholders consulted for the case studies stated that the recruitment process was often greatly facilitated by the flexibility and long-term horizon of the CERC award, which were seen to allow for the production of high-quality/high-impact research, as well as provide other beneficial opportunities for their career advancement (see below). Recruitment was also aided by the fact that many high-calibre researchers were already working at the host institution when the CERC unit was established. More precisely, the review of self-reported data on CERC units’ composition shows that a high proportion—70%—of high-calibre researchers (212 out of a total of 303 researchers) were already working at the institution (Table 7); in contrast, 21% high-calibre researchers came from foreign institutions. It is also worth noting that some of the chairholders indicated that the CERC program enabled them to attract higher quality researchers than would have usually been possible.

Table 7 Number and involvement of high-calibre researchers in CERC units as of 2014

Involvement in CERC unit	Host institution	Other Canadian institution	International institution	Unknown	Total
	Provenance of key researchers
Ongoing involvement	202	27	49	10	288
No longer involved	10	1	4	0	15
Total	212	28	53	10	303

Note: Some researchers’ previous organization was not identified as being Canadian or foreign; these were therefore placed in the “Unknown” category.
Source: Case study questionnaires (2014)

Factors that may explain the success in recruiting high-calibre researchers and HQP are largely similar to the reasons cited for joining and remaining in the CERC unit. Many high-calibre researchers and HQP explained that they were attracted by the unique opportunity to work with a world-class researcher and often cited his exceptional personal qualities, vision and leadership, which in some cases was deemed even more important than their scientific reputation. Other cited reasons that were closely associated with the impacts of the CERC program, will be discussed in a subsequent section.

Survey responses and case studies also demonstrated the critical role played by the CERC program in enabling institutions to recruit world-class foreign researchers. Surveyed institutions considered that the factor that most aided the attraction of world-class researchers was the prestige of the award (100%), followed by the pre-existing critical mass of research expertise within the institution (94%) (see Figure 3 for survey results, ranked based on the percentage of respondents who perceived the factor to be aiding attraction of world-class researchers). Both the survey and case study evidence also support the finding that support received from the institution prior to and after the arrival of the chairholder (e.g., incremental funding, salaries, infrastructure, offer of associated faculty positions), as well as access to talented HQP, were also critical to recruitment of world-class and high-calibre researchers.

Furthermore, without the CERC award, most chairholders, key researchers and HQP reported that they would not have moved to the institution or even to Canada. In addition, few of the nominated but unsuccessful world-class researchers were recruited in the absence of the CERC award, according to surveyed institutions and documentary evidence. In addition, chairholders reported that, in the absence of the CERC award, they would not have had the opportunity to carry out research programs of such a large scale and scope, especially given the perceived limited availability of research funding in Canada and other countries. Many of them would have dropped key components of their research program or carried out fewer, more focused projects within a less integrated/coordinated and multidisciplinary group. Other reasons highlighted by chairholders for accepting the award were closely related to the impacts of the CERC program, which will be discussed in a subsequent section.

Evidence of barriers was found for the attraction and retention of world-class researchers (i.e., chairholders) and these barriers also impeded the recruitment of high-calibre researchers and HQP, albeit to a lesser extent than in the case of world-class researchers. Most notably, half of surveyed institutions were not able to recruit the world-class researcher originally identified by the institution for the CERC award and proceeded to nominate another qualified candidate.^{Footnote 56} The difficulties in recruiting foreign world-class researchers that were first considered by institutions for a CERC award may be partly explained by the fierce global competition that currently takes place to attract top-tier researchers. However, one should recall that bibliometric, program file, and case study evidence showed that the chairholders selected in the end are recognized as world-class based on their scientific performance and reputation and, as will be further discussed in the next sub-section, that they have generally been successful in their CERC enterprise up to this point.

Description of figure

Description of Figure 3, Factors that limit or aid institutions' ability to attract world-class researchers

This stacked bar graph shows participating institutions’ perceptions of the facilitators and barriers to attraction (recruitment and retention) of world-class researchers, high-calibre researchers and highly qualified personnel (HQP). The source of the findings is a survey of participating institutions completed by 20 respondents, a 100% response rate.

The y-axis lists key factors that limit or aid the attraction of world-class researchers. The x-axis shows percentage of respondents, from 0% to 100% in 10% increments. The bars are horizontal and each has up to four colours, representing the percentage of respondents who selected the following responses: greatly or moderately limited, no effect, greatly or moderately aided, or don’t know/not applicable.

The perceptions of the factors that limit or aid attraction were reported as follows:

Prestige of the award as a result of the Canada Excellence Research Chairs (CERC): 100% believed it greatly or moderately aided attraction
Critical mass of research expertise within the institution: 6% believed it had no effect, 94% believed it greatly or moderately aided attraction
The availability of HQP: 11% believed it had no effect, 89% believed it greatly or moderately aided attraction
Level of incremental support provided by the institution: 11% believed it greatly or moderately limited attraction, 89% believed it greatly or moderately aided attraction
Proximity to partners and collaborators in other sectors: 6% believed it greatly or moderately limited attraction, 6% believed it had no effect, 89% believed it greatly or moderately aided attraction
State of the research infrastructure at the institution: 11% believed it greatly or moderately limited attraction, 89% believed it greatly or moderately aided attraction
Duration of the CERC award: 6% believed it greatly or moderately limited attraction, 11% believed it had no effect, 83% believed it greatly or moderately aided attraction
Value of the CERC award: 22% believed it greatly or moderately limited attraction, 78% believed it greatly or moderately aided attraction
Institution’s capacity to leverage additional funding: 22% believed it had no effect, 78% believed it greatly or moderately aided attraction
The location of the institution: 6% believed it greatly or moderately limited attraction, 28% believed it had no effect, 67% believed it greatly or moderately aided attraction
Personal reason(s) (e.g., relocation, family/spouse): 61% believed it greatly or moderately limited attraction, 11% believed it had no effect, 22% believed it greatly or moderately aided attraction, 6% didn’t know or didn’t respond
Immigration regulations: 33% believed it greatly or moderately limited attraction, 44% believed it had no effect, 17% believed it greatly or moderately aided attraction, 6% didn’t know or didn’t respond
Non-renewability of the award: 50% believed it greatly or moderately limited attraction, 44% believed it had no effect, 6% believed it greatly or moderately aided attraction
Other: 40% believed it greatly or moderately limited attraction, 60%

Other specific challenges to attraction, recruitment and retention of world-class researchers, high-calibre researchers and HQP that were most often cited by successful chairholders and institutions are listed below. Note that detailed survey results on facilitators and barriers to attraction are presented in Figure 3 and generally align with findings from other lines of evidence. It is likely that some of these challenges cannot be addressed by the CERC program, but may deserve further consideration and communication/coordination with relevant parties (e.g., institutions, CIC, embassies).

Institution-related issues, such as:
- The level of commitment prior to and after arrival of the chairholders, or lack thereof. This finding is corroborated by the fact that chairholders who benefited from a high level of support from the institution reported this was a key facilitator to their recruitment.
- The perceived “attractiveness” of the institution for CERC chair candidates, or lack thereof (e.g. existing capacity in the research field, the specific location, size and the international reputation).
- Overly complex and/or lengthy institutional hiring procedures, which delayed the recruitment of tenured and non-tenured high-calibre researchers.

Non-renewability of the award and the award terms (i.e., amount and duration, when judged to be non-competitive), which are considered all the more problematic given the time required (i.e. on average two years) to fully establish the research group upon relocating to the host institution, and the perceived lack of equivalent funding after the end of the CERC award.
- For example, one institution reported that they were not able to recruit their first choice candidates as they were not interested in a unique 7-year mandate. Another survey respondent stated that $10M over 7 years “although very generous, is insufficient to encourage relocation of truly outstanding internationally recognized researchers”.
- The ability of chairholders to leverage additional and/or future funding varies greatly for several reasons linked notably to the context, scope and scale of their research and their familiarity with the Canadian research funding system. The non-renewability of the award also poses a challenge to the retention of HQP, especially of postdoctoral researchers, whose salaries cannot always be covered by other available funding programs. Several chairholders indicated that they are likely to leave the institution after the 7 years of the CERC award if they are unable to secure the funds they need to continue pursuing their research goals.

Personal issues, such as:
- Immigration and taxation issues (e.g., difficulties in obtaining visas for the chairholders, their family and members of their research team; double-taxation in certain countries such as Germany);
- Relocation expenses, especially for chairholders moving from Europe;
- Language barriers to attraction of world-class researchers and international scholarship students for francophone universities;
- Lack of dual employment opportunities for spouses (e.g., at the host institution).

Particularly for the first CERC competition, the short timeframe (i.e., seven months) for submitting nominations created a barrier for host institutions to successfully identify and nominate an appropriate candidate within this timeframe; this was often compounded by the other barriers listed above.

Barriers to access and equity
During the first competition, no women were nominated for a CERC award. In order to address this issue, an Ad Hoc Panel on CERC Gender Issues was established, which was tasked to identify the potential reasons why women were underrepresented and possible solutions and led to changes in the CERC application process and university nomination strategies for the second competition.

Some of the potential explanatory factors identified by the panellists relate to the CERC program design, while others relate to the wider university context (given that universities did not submit any female candidates). However, the latter factors were out of the scope of the Panel and thus were not discussed in-depth in the report (e.g., fewer female faculty in general, the fact that the proportion of female faculty declines with seniority and varies widely by discipline).^{Footnote 57} Aspects of the CERC program design which were deemed likely to have impacted the ability of the universities to find and enlist female candidates are described below.^{Footnote 58}

The CERC program focuses on the most senior international researchers with “eligible nominees required to be, or soon to be, full professors”^{Footnote 59} in a context where approximately only 20% of Canadian and 19% of US full professors are female.
The CERC program focuses on the S&T Strategy’s four priority research areas and the STIC sub-priorities, in which women are underrepresented.
The first competition had compressed/short program timelines for recruitment and the uncertainty of the outcome of the program for candidates being nominated (since only 50% of nominees would be awarded a CERC chair) were both found to be an issue because of “the interaction of factors such as family patterns, institutional requirements, and career expectations, women may be particularly vulnerable to factors such as compressed timelines, risk in career change, and changes in location of residence.”^{Footnote 60}

Interestingly, international funding organizations examined in the context of this evaluation have noted that women are unrepresented but the majority have not implemented any corrective measure to date. A few programs (e.g. Swedish Research Council) have implemented concrete measures related to equity issues, including specific application requirements, a two-track recruitment approach for established and emerging researchers and/or the use of a gender ratio on selection boards. However, none of the programs use quotas for gender for the chairholders themselves.

At the national level, the document review indicates that the CRC program has been proactive in addressing the equity issue. They have worked with universities to ensure that they follow open, transparent, and equitable recruitment practices and establish equity targets for the representation of the four designated groups (i.e. women, Aboriginal peoples, persons with disabilities and members of visible minorities). Universities are required to use a target-setting and reporting tool to monitor and report on the representation of the four groups among their chairholders. These measures resulted in the increase of women as chairs, who now represent 25.6% of chair holders in 2012.

The Ad Hoc Panel on CERC Gender Issues proposed four recommendations for the CERC program, and a fifth one directed at the Council of Canadian Academics requesting them to undertake an assessment of the advancement of women in university careers in Canada (i.e. to address wider university community issues).^{Footnote 61}

In direct response to these recommendations, the CERC program agreed to implement the following measures:

Universities are asked to report on their recruitment process and outreach efforts as part of phase 1 and phase 2. The information will be shared with reviewers. In phase 2, the quality of the institutional recruitment process has been added as a formal evaluation criterion.
The program literature will be amended to further clarify that both rising stars and established leaders are eligible, and to help the reviewers interpret the selection criteria for both types of researchers. Instructions to reviewers will also be clarified to ensure that measures beyond bibliometrics are to be used in assessing the quality of candidates.
The program will only short-list as many proposals as there are Chairs available. For the second competition, there are 11 Chairs available and therefore only 11 phase 1 applications were invited forward.
For the second competition, four of the 11 chairs are open to all areas of research.

Following the commitments made by the CERC program, universities have started to explore various proactive strategies to attract a larger, diverse group of candidates, as reported in their phase 1 applications. The majority planned to conduct an open and broad search to attract a great variety of talent. Many universities will also be appointing search and hiring committees including members representing a wide variety of social groups and expertise that will help to ensure an unbiased selection process, regardless of gender, ethnicity or career stage. Several institutions also intend to advertise the position to a wide range of audiences, through the use of less conventional advertising methods.

The findings presented in the above section stem from documentary and international review evidence. Limited evidence was found in case studies and the survey regarding potential/ongoing program barriers to gender equity. For instance, surveyed institutions were asked to comment on the effectiveness of the CERC program to attenuate or redress general issues of access, equity or inequity in its program design and delivery. Among the nine institutions that commented on this particular issue, a few perceived that the improvements made to Competition 2 processes will address issues of access. A few other institutions stated that gender issues go beyond the CERC program and relate to the academic system as a whole, which resonates with the findings of the Ad Hoc Panel on CERC Gender Issues.

While it is too early to assess the impact of measures taken by institutions to address equity issues, as 10 chairs remain be awarded at the time of writing this report, it is worth noting that one woman was appointed as a CERC chair in 2013—the first chairholder appointed as part of the second CERC competition.

Impact of the CERC award on chairholders, high-calibre researchers and HQP

Overall, the CERC award was found to have a positive impact on the scope of the research conducted, the scientific production and career of chairholders, high-calibre researchers and HQP. However, CERC units faced a few challenges, such as delays in setting-up the research group/lab and career-related issues for high-calibre researchers and HQP.

Chairholders and their teams reported very similar benefits, with the latter praising the leadership and scientific qualities of the chairholder as a key success factor in achieving these impacts. In essence, these benefits revolve around the participation in large-scale, cutting-edge research with flexible, long-term funding, which provide the opportunity to:

access a large pool of highly talented researchers and HQP;
benefit from a variety of expertise and a highly interdisciplinary environment;
set-up, calibrate and run state-of-the-art facilities; and
develop more academic and non-academic collaborations at the national and international level.

The participation of HQP in CERC units created further benefits for education and skills development through involvement in projects managed by a world-class researcher with strong leadership skills and supported by CERC funding. Many HQP reported they were given the opportunity to conduct risky research yielding high quality/high impact outputs. HQP also gained increased exposure to a variety of expertise through exchanges/collaborations with other students during conferences, workshops, internships, summer schools, etc. (see Exhibit 1).

Exhibit 1 International summer school

The Transatlantic Ocean System Science and Technology (TOSST) coordinated by Dalhousie University’s CERC chairholder was awarded $1.6M through the NSERC CREATE program to work with its German counterpart (HOSST) in order to train the next generation of HQP through a collaborative multidisciplinary summer school that can “address the responsible use and management of the ocean of the future.” The first summer school was held in 2013 in Halifax. In total, 12 PhD students from both Canada (6) and Germany (6) attended the two-week event. According to participants, the TOSST-HOSST summer school provided a great opportunity to network, exchange ideas, and develop skills with students and researchers in diverse research fields. For example, students participated in a competition during the summer school, based on “Dragon’s Den” where teams presented their potential tangible research applications to a business-like jury. See www.tosst.org

Postdoctoral fellows and research professionals often reported that they have been given more responsibilities as part of the CERC unit compared to other research teams, contributing to their training as independent investigators. Such increased responsibilities are illustrated by opportunities to supervise and train other students. In some cases, the postdoctoral fellows and/or research professionals play a pivotal role in the research team, as they are called upon to start/set-up the research lab, build and manage the research program, often by supervising research sub-groups. Some postdoctoral fellows also reported that they appreciated being given the opportunity to publish research results as principal or co-principal investigators. Postdoctoral fellows often perceived that all these opportunities significantly contributed to increased recognition by institutions of their status and value (i.e., as investigators rather than as trainees).

All the above benefits were reported to have contributed to increased scientific production of chairholders, high-calibre researchers and HQP. With respect to chairholders, bibliometric analyses indicate an increased international co-publication rate^{Footnote 62}—from 50% to 85%—and a potential positive effect of the scientific impact/quality of papers, based on the impact factor of the journals in which they were published (i.e., median RIF increased from 1.30 to 1.40; however the difference is not statistically significant). These increases were larger than those observed among the CERC nominees who were not granted a CERC award over the same period. However, these bibliometric analyses should be considered preliminary and should be repeated later in the CERC program’s lifecycle in order to obtain more robust and reliable data, as the population size (i.e., number of chairholders and number of publications) increases, in order to confirm the extent to which the scientific performance of the CERC units has benefited from the CERC program.

Finally, high-calibre researchers and HQP highlighted some positive impacts on their career. They feel that their participation in a CERC unit has increased their employability in the academic, the private or industry sectors (e.g., industry/commercialization skills leading to employment opportunities in spin-off and technology companies). In some instances, the CERC award contributed directly to the career advancement of high-calibre researchers, who obtained a faculty position when they joined the CERC unit (see Exhibit 2). However, with the notable exception of tenured faculty, some high-calibre researchers fear that these benefits and impacts will not extend beyond the CERC term, and they worry that there are few career opportunities in academia and industry for researchers in Canada.

Exhibit 2 Impact of the CERC on the career of high-calibre researchers

Under the leadership of the CERC chairholder in Neurogenetics and Translational Neuroscience, two high-calibre researchers were able to secure independent faculty positions at the University of British Columbia. From the beginning, the chairholder had envisioned building a multidisciplinary neurocentre primarily focused on Parkinson’s disease, supported by independent albeit closely linked programs on Parkinson’s and other related neurological diseases, such as Alzheimer’s, dementia, schizophrenia and multiple sclerosis. To achieve his vision, the chairholder strongly encouraged two of his former collaborators to move to UBC with him and build their own program under the umbrella of the CERC chair. One of these researchers left his position as an assistant professor in a leading US institution to lead an independent project on multiple sclerosis as a CRC chairholder at UBC, while the other obtained an assistant professor position and currently leads one of the labs associated with the CERC unit. Both researchers reported that the CERC award and the chairholder allowed them to take an important step in their career, which led to major scientific achievements and fruitful collaborations. One stated that his involvement in the CERC unit has allowed him to become fully independent and start a new research program on multiple sclerosis, which would not have happened without the support of the CERC award.

Despite the benefits described above, both qualitative and quantitative evidence found that delays in setting up the CERC facilities and/or research group have slowed down the scientific production of CERC units. The case study evidence showed that some CERC units experienced challenges in the early stages of the award that resulted in greater than anticipated delays in producing research outputs by various members of the unit. For chairholders, this finding is confirmed by bibliometric trends, which indicate an eight-month lag on average from the start of the chair to the publication of the first related paper and an average two-year lag before recovery of the pre-CERC production level (Figure 4, graphs 4 & 6 ^{Footnote 63}). However, it is worth noting in Figure 4 that not all chairholders were as similarly affected by such delays: some maintained their research publication output at a steady level before and after the CERC award (graphs 3 & 9), some have not yet regained pre-CERC levels (graph 7), while others appear to have already achieved an increase in their output (graph 14).

trends in the number of CERC-unrelated and CERC-related publications: examples from six CERC chairholders from 2000 to 2013

Description of figure

Description of Figure 4, Trends in the number of CERC-unrelated and CERC-related publications: examples from six CERC chairholders from 2000 to 2013

This image is a collection of six double bar graphs, showing trends in the number of publications from six Canada Excellence Research Chairs (CERC) chairholders. Each of the graphs shows the number of publications for one chairholder, differentiating between CERC-unrelated and CERC-related publications. These six examples were selected for illustrative purposes to show the different types of trends observed. They were computed by Science-Metrix using the Web of Science, a Thomson Reuters application.

Graphs are labelled with the following anonymous chairholder identifiers: 3, 4, 6, 7, 9, 14. Along the x-axis of each graph there are 14 groups of one or two bars each, labelled by year starting in 2000 and going up to 2013. Each group contains a red bar for CERC-unrelated papers and/or a green bar for CERC-related papers. The y-axis shows the number of CERC-unrelated and CERC-related papers, from 0 to 35 in intervals of 5.

The trends in the numbers of unrelated and related papers for all six CERC chairlholders are as follows:

CERC Chairholder #3

2000: 6 CERC-unrelated papers, 0 CERC-related papers
2001: 6 CERC-unrelated papers, 0 CERC-related papers
2002: 17 CERC-unrelated papers, 0 CERC-related papers
2003: 7 CERC-unrelated papers, 0 CERC-related papers
2004: 18 CERC-unrelated papers, 0 CERC-related papers
2005: 13 CERC-unrelated papers, 0 CERC-related papers
2006: 14 CERC-unrelated papers, 0 CERC-related papers
2007: 24 CERC-unrelated papers, 0 CERC-related papers
2008: 9 CERC-unrelated papers, 0 CERC-related papers
2009: 17 CERC-unrelated papers, 0 CERC-related papers
2010: 16 CERC-unrelated papers, 0 CERC-related papers
2011: 3 CERC-unrelated papers, 11 CERC-related papers
2012: 1 CERC-unrelated papers, 18 CERC-related papers
2013: 2 CERC-unrelated papers, 14 CERC-related papers

CERC Chairholder #4

2000: 3 CERC-unrelated papers, 0 CERC-related papers
2001: 11 CERC-unrelated papers,0 CERC-related papers
2002: 15 CERC-unrelated papers, 0 CERC-related papers
2003: 16 CERC-unrelated papers, 0 CERC-related papers
2004: 23 CERC-unrelated papers, 0 CERC-related papers
2005: 20 CERC-unrelated papers, 0 CERC-related papers
2006: 18 CERC-unrelated papers, 0 CERC-related papers
2007: 9 CERC-unrelated papers, 0 CERC-related papers
2008: 11 CERC-unrelated papers, 0 CERC-related papers
2009: 7 CERC-unrelated papers, 0 CERC-related papers
2010: 11 CERC-unrelated papers, 0 CERC-related papers
2011: 2 CERC-unrelated papers, 4 CERC-related papers
2012: 0 CERC-unrelated papers, 17 CERC-related papers
2013: 0 CERC-unrelated papers, 17 CERC-related papers

CERC Chairholder #6

2000: 9 CERC-unrelated papers, 0 CERC-related papers
2001: 4 CERC-unrelated papers, 0 CERC-related papers
2002: 10 CERC-unrelated papers, 0 CERC-related papers
2003: 6 CERC-unrelated papers,0 CERC-related papers
2004: 9 CERC-unrelated papers, 0 CERC-related papers
2005: 10 CERC-unrelated papers, 0 CERC-related papers
2006: 12 CERC-unrelated papers, 0 CERC-related papers
2007: 9 CERC-unrelated papers, 0 CERC-related papers
2008: 13 CERC-unrelated papers, 0 CERC-related papers
2009: 11 CERC-unrelated papers, 0 CERC-related papers
2010: 11 CERC-unrelated papers, 0 CERC-related papers
2011: 6 CERC-unrelated papers, 3 CERC-related papers
2012: 0 CERC-unrelated papers, 7 CERC-related papers
2013: 0 CERC-unrelated papers, 12 CERC-related papers

CERC Chairholder #7

2000: 16 CERC-unrelated papers, 0 CERC-related papers
2001: 9 CERC-unrelated papers, 0 CERC-related papers
2002: 18 CERC-unrelated papers, 0 CERC-related papers
2003: 17 CERC-unrelated papers, 0 CERC-related papers
2004: 10 CERC-unrelated papers, 0 CERC-related papers
2005: 8 CERC-unrelated papers, 0 CERC-related papers
2006: 8 CERC-unrelated papers, 0 CERC-related papers
2007: 14 CERC-unrelated papers, 0 CERC-related papers
2008: 4 CERC-unrelated papers, 0 CERC-related papers
2009: 6 CERC-unrelated papers, 0 CERC-related papers
2010: 1 CERC-unrelated papers, 4 CERC-related papers
2011: 0 CERC-unrelated papers, 5 CERC-related papers
2012: 0 CERC-unrelated papers, 6 CERC-related papers
2013: 0 CERC-unrelated papers, 5 CERC-related papers

CERC Chairholder #9

2000: 3 CERC-unrelated papers, 0 CERC-related papers
2001: 2 CERC-unrelated papers, 0 CERC-related papers
2002: 4 CERC-unrelated papers, 0 CERC-related papers
2003: 8 CERC-unrelated papers, 0 CERC-related papers
2004: 5 CERC-unrelated papers, 0 CERC-related papers
2005: 7 CERC-unrelated papers, 0 CERC-related papers
2006: 9 CERC-unrelated papers, 0 CERC-related papers
2007: 11 CERC-unrelated papers, 0 CERC-related papers
2008: 9 CERC-unrelated papers, 0 CERC-related papers
2009: 5 CERC-unrelated papers, 0 CERC-related papers
2010: 8 CERC-unrelated papers, 0 CERC-related papers
2011: 7 CERC-unrelated papers, 1 CERC-related papers
2012: 2 CERC-unrelated papers, 6 CERC-related papers
2013: 0 CERC-unrelated papers, 8 CERC-related papers

CERC Chairholder #14

2000: 3 CERC-unrelated papers, 0 CERC-related papers
2001: 3 CERC-unrelated papers, 0 CERC-related papers
2002: 4 CERC-unrelated papers, 0 CERC-related papers
2003: 3 CERC-unrelated papers, 0 CERC-related papers
2004: 2 CERC-unrelated papers, 0 CERC-related papers
2005: 3 CERC-unrelated papers, 0 CERC-related papers
2006: 1 CERC-unrelated papers, 0 CERC-related papers
2007: 8 CERC-unrelated papers, 0 CERC-related papers
2008: 3 CERC-unrelated papers, 0 CERC-related papers
2009: 2 CERC-unrelated papers, 0 CERC-related papers
2010: 2 CERC-unrelated papers, 0 CERC-related papers
2011: 4 CERC-unrelated papers, 1 CERC-related papers
2012: 0 CERC-unrelated papers, 8 CERC-related papers
2013: 0 CERC-unrelated papers, 11 CERC-related papers

According to case study interviewees, it is not surprising that such delays occur when a new research facility or team is assembled “from scratch”, especially when it requires new infrastructure and the purchase of state-of-the-art equipment. However, evidence from case studies revealed that some institutions did not provide the space or the human resources (i.e., faculty) that had been committed in their initial CERC proposal in a timely manner; this is discussed in more detail in Section 3.2.4 (Enhanced and sustainable research capacity of universities in S&T Strategy research priority areas).

3.2.2 Awareness of CERC units, the CERC program and Canada

Summary of Findings:
To achieve the CERC program’s long-term outcome of contributing to making Canada a global destination of choice for research and higher learning, it is expected that CERC units and host institutions will make efforts to promote their research activities, their group and institution, as well as the CERC program itself. The evaluation found that the CERC program has contributed to an increased visibility of the host institutions and the chairholders, and, to a more limited extent, of Canada as a location of choice for researchers. However, limited evidence was found of increased visibility of the CERC program itself outside chair announcements. As such, program stakeholders suggested efforts could be made to better promote and advertise the program and the research results of the CERC units, including to the general public, both nationally and internationally.

Visibility of the CERC units, host institutions and Canada
One of the long-term outcomes of the CERC program is that Canada becomes a global destination of choice for research and higher learning.^{Footnote 64} In order to achieve this outcome, the program expects CERC units and host institutions, supported by the Chairs Secretariat, to make efforts to promote their research activities, their group and institution, as well as the CERC program itself (see Appendix B).

The survey and case studies provide evidence of increased visibility of the CERC units, their host institutions and Canada as a location of choice to conduct research. For instance, approximately 75% of surveyed institutions reported that the CERC award(s) contributed to increasing their visibility nationally and internationally (Figure 5), although the observed effect was more pronounced at the national rather than at the international level. Moreover, all institutions reported that the CERC program contributed at least moderately to raising awareness of Canada as a location of choice for conducting world-class research (Figure 5).

extent of impact of the CERC award on the visibility of institutions and on the awareness of Canada as a location of choice to conduct world-class research

Description of figure

Description of Figure 5, the Extent of impact of the CERC award on the visibility of institutions and on the awareness of Canada as a location of choice to conduct world-class research

This stacked bar graph shows participating institutions’ perceptions of the impact of the Canada Excellence Research Chairs (CERC) award on the visibility of their institutions and awareness of Canada as a location of choice to conduct wold-class research. The source of the findings is a survey of participating institutions completed by 20 respondents, a 100% response rate.

The y-axis list types awareness/visibility that the CERC award may have influenced. The x-axis shows percentage of respondents, from 0% to 100% in 10% increments. The bars are horizontal and each has up to three colours, representing the percentage of respondents who selected the following responses: to a little or no extent, to a moderate extent, or to a large or very large extent.

The perceptions of the impact of the CERC award were reported as follows:

Awareness of Canada as a location of choice: 21% believed that the CERC award impacted awareness of this to a moderate extent, 79% believed the CERC award impacted this awareness to a large or very large extent
Visibility of your institution at the international level: 29% believed the CERC award impacted awareness of this to a moderate extent, 71% believed the CERC award impacted this awareness to a large or very large extent
Visibility of your institution at the national level: 7% believed the CERC award impacted awareness of this to a little or no extent, 21% believed the CERC award impacted this awareness to a moderate extent, 72% believed the award impacted this awareness to a large or very large extent

According to case study interviewees, increased visibility results primarily from networking and outreach efforts and activities undertaken by the chairholders and CERC units. Indeed, many chairholders have made significant networking efforts since their move to Canada, in order to raise stakeholders’ awareness of their research. In particular, chairholders and other members of the CERC units are presenting their research at international events and initiatives involving academia, industry and/or government representatives, such as conferences, workshops, open houses and consortia. As a notable example, one chairholder reported that the CERC unit jointly hosted a symposium with a neuroscience program of the university which attracted 400-500 students and many other participants, including international, leading scientists (including one Nobel Prize winner) and key researchers from Canada.

Several chairholders also indicated that the establishment of formal collaborations with prestigious foreign institutions are evidence of the increased visibility of the institution generated by the CERC unit and the chairholder, as illustrated by a Memorandum of Understanding (MoU) formalized between the Max Planck Institute (Germany) and the CERC in Quantum Nonlinear Optics; see also Exhibit 3.

Increased interest and recognition are also associated with the reputation of the chairholder, the calibre of the CERC unit (including infrastructure/equipment), and the quality of research produced to date. While delays in the implementation of the CERC awards have at times slowed down their research production, several members of the CERC units indicated that a number of high-quality, high-impact papers have already been produced as a result of the CERC funding, which also contributes to the prestige of the institution.

Exhibit 3 International visibility through collaboration

The chairholder at the University of Manitoba is one of the key researchers involved with the Arctic Science Partnership (ASP). ASP is a leading consortium on climate, ice, ecosystems and human interactions. The ASP began in July 2012 with the goal of developing a highly integrated and coordinated climate-related research and education collaboration in the Arctic. A MoU was signed by the Centre for Earth Observation Science (CEOS) at the University of Manitoba with research institutes located in Greenland and Denmark. As part of CEOS, the CERC chairholder noted that the ASP has made it possible to shift from a national to an international focus. For example, institutions now have access to researchers, students and networks in Europe and, as a result, to new collaborations in Greenland, Denmark, Germany, and UK. The ASP collaborations are continuing to expand through the coordination of field trips with the ARCTICNet Network of Centres of Excellence. Furthermore, a research proposal for a project in Baffin Bay has been developed with the Laval University CERC chairholder.

Analyses of the promotional materials produced and disseminated by the CERC units, and of the CERC media coverage also provided useful insights into the level of visibility of the CERC units. However, these should be interpreted with some caution, as they are based primarily on self-reported data and no comparative data is available; it is expected that more robust findings will be confirmed over time. Currently, this analysis shows an upward trend in the number of promotional tools, events and activities and use of social media between 2010 and 2013 (Figure 6). Furthermore, slightly over a quarter of the CERC units (28%) have used social media to promote their research (and/or the program).

Figure 6 Number and type of promotional tools and events/activities developed/held by year

Source: Case study questionnaires (2014); statistical data from CERC communications/media officer (2014)

Description of figure

Description of Figure 6, Number and type of promotional tools and events/activities developed/held by year

This line graph shows the number and type of promotional materials developed or events/activities held by Canada Excellence Research Chairs (CERC) units. Findings are drawn from 2014 case study questionnaires as well as statistical data from the CERC communications/media officer.

There are three lines, including: events and activities, promotional tools, and social media. The x-axis shows the year, starting in 2010 and going to 2013. The y-axis shows the number of materials, events or activities, from 0 to 90 in intervals of 10.

The perceptions of tools and events/activities developed/held was reported as follows:

Promotional tools: 23 in 2010, 28 in 2011, 32 in 2012, 79 in 2013
Events and activities: 0 in 2010, 10 in 2011, 28 in 2012, 24 in 2013
Social media: 2 in 2010, 1 in 2011, 0 in 2012, 13 in 2013

Visibility of the CERC program

Case study evidence, interviews and the administrative data review indicated that there is limited visibility of the CERC program in Canada outside of chair announcements and the CERC website. Case study interviewees reported that there was extensive media coverage of the chairs when they were initially announced (e.g., press releases, university and national newspaper articles/features, TV exposure), but that this coverage has been much more limited in the past couple of years. Furthermore, Google Analytics of the CERC website shows that, prior to September 2013, the site was fairly quiet: it was visited by less than 100 unique visitors monthly. A recent increase in the fall of 2013 might be due to the announcement of the latest chair (appointed at McGill University).

There is also limited evidence in terms of the extent to which the CERC program is known outside of Canada among target audiences in the S&T community; this evidence was not collected systematically for this evaluation and would likely be challenging to obtain. Most international program representatives consulted were not aware of the CERC program; however, they did know about similar EU and Australian programs (including some established at the same time as the CERC or after). In addition, chairholders themselves generally had not been aware of the program before they were contacted by their host institution. Anecdotally, a few chairholders reported that they systematically cite the CERC program (and use the CERC logo) in their presentations. For instance, one chairholder indicated that he has cited the CERC program in presentations that were given to 30 countries involved in an international initiative.

Given this perceived lack of visibility of the CERC program outside of host institutions, case study and key informant interviewees suggested that efforts could be made to better promote and advertise the program and the research results of the CERC units, including to the general public, and both at the national and international levels (e.g. in the form of documentaries; more advertising of the CERC on the institution’s website).

3.2.3 Partnerships in S&T Strategy research priority areas

Summary of Findings:
Definitions and expectations for partnerships and collaborations in the context of the CERC program are not clear, leading to challenges in assessing the achievement of this outcome. The evaluation evidence confirms that the CERC program has been highly successful in contributing to increased collaborations and associated benefits, primarily with academia, both within Canada and internationally. To date, CERC units have developed or established fewer collaborations and partnerships with non-academic sectors. However, there are some early notable examples suggesting that the program may represent an effective means to build these types of relationships in the future.

Initial considerations on the use of the terms “collaborations” and “partnerships”
Before the evidence for this outcome is presented, it is important to note that there was considerable confusion and variability around the use of the terms “collaborations” and “partnerships” in CERC program documents consulted. This confusion was closely related to a lack of shared understanding across program stakeholders of these terms and of the associated expectations (i.e., outcomes).^{Footnote 65}

On one hand, initial CERC program documents and phase 1 Applications Form make frequent (albeit inconsistent) references to partners across a broad range of sectors, receptors of innovation and research insights, and/or research users. Although not specifically defined, these terms are used in the context of leveraging of additional resources, knowledge transfer and application of research results; thus, these are explicitly or implicitly linked to program outcomes. On the other hand, phase 2 Nomination Forms and annual progress reports require far less detailed or specific information relating to collaboration and partnerships, refer to dissemination and collaboration in a broad manner (e.g., “within and outside the academic community”) and provide no definitions of the terms used. Thus, there are no clear references to program outcomes that would help point to the expectations of the program in this regard. In short, there was poor alignment between stated program outcomes relating to collaborations, partnerships and relationships with receptors/users of research, and the current application and reporting requirements.

It is not surprising that several enquiries arose from the CERC units during the case study process about information requested by the evaluation team on collaborators, partners and key receptors. Efforts by the evaluation team to clarify these terms for the CERC units were hampered by the fact that there were no definitions established for the CERC program that could be referred to. Confusion around these terms also resulted in the fact that some case study interviews were conducted with research collaborators instead of with partners or pertinent users of research, as originally planned. As such, the absence of clear definitions for these terms resulted in issues with the consistency and reliability of the data collected and used for this evaluation to address this outcome.

The wide range in the nature of the research conducted by CERC units across the S&T priority areas (e.g., basic, applied, etc.) may also have been a confounding factor: the opportunities for partnerships and the extent of involvement with research users varies across the innovation spectrum, such that some CERC units collaborate almost exclusively with academic researchers, while others work frequently with a broader range of sectors. However, program documents provide little direction as to how expectations should vary based on the nature of the CERC unit’s research. It is worth noting here that only a few international programs explicitly state that the research is expected to have an impact beyond academia and none of the international programs request a plan or some form of commitment regarding mobilization of knowledge to users of research. Thus, if the CERC program outcomes continue to be stated and assessed based on outcomes relating to partnerships and collaborations, this should be clarified, communicated and reporting processes improved in order to capture the necessary data for ongoing management and assessment of the program.

For the purpose of this evaluation, “collaboration” refers to researcher-to-researcher interactions (i.e., “collaborations” with other researchers, primarily within academia but also in other sectors), while “partnership” refers to the involvement of an external organization (i.e., “partners” are government, industry, associations, non-profit or other institutions) that have contributed or committed in-kind or cash contributions to support the CERC unit’s activities.^{Footnote 66}

Finally, “research users” refer to key receptors of the CERC units’ innovations and knowledge/insights. These terms are not exclusive, e.g., a research user could potentially also be a collaborator and/or partner. Note that the extent and impact of funding leveraged through partnerships are discussed in more detail in the next section.

Number and distribution of partnerships and collaborations

All lines of evidence confirm that the CERC program has been highly successful in contributing to the enhancement of research collaborations within Canada and with international collaborators. In 2014, CERC chairholders identified over 400 collaborators, partners and research users with whom they have been involved since the start of their CERC award. Case study and survey evidence indicated that the chairholders have not only enhanced existing collaborations through the CERC award, but also developed new relationships with other researchers at their host institutions, with other Canadian institutions, as well as internationally. According to key informants, collaborations between individual CERCs have also been developing to an extent that was unanticipated at the time of program initiation.

The large majority (two thirds) of the CERC units’ collaborations and partnerships reported in 2014 are with researchers in the academic sector (Figure 7). Less than 15% are with government organizations, and only 8% are with industry. These proportions are consistent with case study findings, which show that CERC units primarily collaborate with researchers in the academic sector. Nonetheless, several units (but not all) also work extensively with the private sector, such as in the context of applied research projects. Several CERC units also work closely with the public sector and (more rarely) not-for-profit organizations at the provincial, federal and international level. The finding that non-academic linkages are occurring to a lesser extent than academic collaborations is also supported by the fact that only 57% of university representatives reported a positive impact of the CERC on the number of collaborations and partnerships with the non-academic sector—a relatively modest share compared to other impacts on which they were surveyed.

Almost all collaborations and partnerships reported by chairholders in the case study questionnaire (91%) were categorized as “collaborations” (i.e., researcher-to-researcher interactions, as defined above); again, these categories are not exclusive (i.e., can be counted in more than one category). “Partnerships” account for about 7% and primarily involved universities and industry, while “research users” represent about 12% and involved government, industry, university and a variety of other types of organizations. Moreover, few partners and research users were located outside of Canada.

Figure 7 Number of self-reported international and Canadian collaborations, partnerships (including relationships with research users), by sector

Description of figure

Description of Figure 7, Self-reported international and Canadian collaborations and partnerships by sector

This graph is the same as Figure 1 above, just a slightly different title. This double bar graph shows the number and type of collaborations and partnerships developed by Canada Excellence Research Chairs units, collected through the 2014 case study questionnaires.
Along the x-axis there are five groups of two bars each. The groups are labelled by organization type: university, government, industry, association, and other (e.g., hospitals, clinics, international agencies, etc.). Each group contains a red bar for Canadian organizations and a blue bar for foreign organizations.

The y-axis shows the number of collaborations and partnerships. The scale begins at zero and goes to 160, in increments of 20.

The number of collaborations and partnerships with each organization type was reported as follows:

University, Canadian: 123
University, foreign: 148
Government, Canadian: 32
Government, foreign: 21
Industry, Canadian: 27
Industry, foreign: 5
Association, Canadian: 5
Association, foreign: 3
Other, Canadian: 8
Other, foreign: 27

Approximately half of all reported collaborations and partnerships were with foreign organizations. This proportion varies widely across types of collaborators or partners: more than half of academic collaborations were with researchers at foreign universities, but only 15% of relationships with industry involve foreign firms (Figure 7). It is, therefore, not surprising that almost 80% of university representatives indicated that the CERC has had a positive impact on the number of international collaborations at their institution, while 64% also reported a positive impact on the number of national collaborations.

Similarly, the bibliometric analysis of the scientific output (i.e., peer-reviewed papers) of CERC chairholders shows that their international co-publication rate increased significantly after they moved to their new institution. This increase is likely due in large part to the fact that chairholders have continued to co-author papers with researchers from their former countries, which are counted as “international co-authors” after the chairholders move to Canada. This nevertheless represents a net gain for Canada’s international publication rate, and the impact of the CERC is likely even larger as new or broader collaborative relationships are established that involve other members of the CERC unit. For example, both national and international collaborators working with several CERCs indicated that they had gained access to new collaborators from the host institution through the chairholder.

Nature and benefits of partnerships and collaborations

Collaborations and partnerships

The case studies found that the nature of the collaborations and partnerships varied across CERC units. Several CERCs reported the exchange or sharing of infrastructure, services, data and expertise, through formal or less formal agreements with academic, government and industry collaborators and partners. Collaborations with other researchers often involved joint projects, joint grants and funding applications, and co-authorship of publications. The major benefits of these relationships for both the CERC and their collaborators include diversification of expertise via access to a large pool of highly talented researchers from a variety of disciplines that contribute to the research endeavour, as well as access to cutting-edge infrastructure, equipment and data. These ultimately lead to strengthened research outputs.

Collaborations with researchers frequently involved co-supervision of students and development of joint training programs (including via formal agreements). Some industry partners also contribute to the salaries of students, interns or other HQP. As noted previously, these types of collaborations or partnerships greatly contribute to improved training of HQP and enhanced career opportunities. From the perspective of the collaborators and partners, they benefit from the contribution of the HQP to research projects and from access to HQP that have specific training and expertise in their field or relevant to their industry. As mentioned above, greater access to HQP that have gained industry-relevant skills was seen as an important benefit given the challenges faced by industry in attracting and hiring talent.

Research users

Less frequent were collaborations, partnerships or other forms of relationships with research users. The case studies provided some notable examples of commercialization of research products and the creation of new companies, as well as relationships with research users in the government sector. Similarly, CERC Progress Reports provide some evidence that chairholders are being called upon to a moderate extent to provide expert advice to the private sector, non-elected government officials, and to a minimal extent to elected government officials. These reports also show a slight increase in demand for expert advice and opinion across the types of research users between 2010-2012 and 2013; this trend remains to be confirmed over a longer timeframe.

Indeed, several lines of evidence (e.g., case studies, key informant interviews, progress reports) highlight the fact that CERC units are still early in their life cycle and that relationships with partners and research users are expected to increase. The case studies in particular provide some early examples showing that the CERC units can provide effective opportunities to build and further develop fruitful collaborations and partnerships. Their benefits will also likely extend beyond the CERC award, given the time it takes to translate research into applications.

Evidence of benefits for industry partners include, aside from those related to HQP, a higher return on investment and reduced risk associated with the R&D when working with a CERC unit (see Exhibit 4). Additionally, some federal and provincial government partners and research users consulted as part of the case studies noted an increase in their research capacity in the field, which is expected to lead to economic and policy benefits for the province or the country. More specifically, working with the CERC unit was an opportunity for them to improve their understanding of issues of interest, significantly contribute to the economic development of the province, and enhance decision-making through access to cutting-edge expertise, facilities and high quality research results.

Exhibit 4 Benefits for users in the industry sector

The McMaster University CERC chairholder has developed many collaborations and partnerships with industry, leveraging approximately $2 million from these sources in 2012-13. The chairholder strongly believes that the CERC award has had a huge impact on the size and nature of the collaborations developed by the unit. Industry partners indicated that they have benefitted from taking part in the CERC unit’s research and having access to students. Both the CERC chairholder and the partners indicated that their partnerships have resulted in timely and cost-effective use of resources, as well as an increased number of applications for patents. In addition, formalized partnerships between McMaster University and industry led to the creation of a university Industrial Research Chair, which fosters additional access and communication.

Notwithstanding, a range of challenges faced when developing fruitful collaborations and partnerships were also observed, especially in the context of academia-industry partnerships. For example, issues with intellectual property (IP) were noted in a few case studies: on one hand, institutional IP management systems or processes can delay the commercialization of research outputs, while on the other hand, research teams may face access limitations for certain technologies due to IP issues. More broadly, it was recognized that it takes time to establish such partnerships, and that there may be difficulties in securing matching funds and other investments, especially if industry partners are facing budget constraints as many are following the global economic crisis.

3.2.4 Enhanced and sustainable research capacity of universities in S&T Strategy research priority areas

Summary of Findings:
The CERC program has contributed to enhanced research capacity in S&T priority areas, including establishing state-of-the art infrastructure ^{Footnote 67} and building a critical mass of expertise (e.g., high-calibre researchers, HQP) that is usually well integrated within host institutions. Two success factors for this outcome include both institutional support (financial and otherwise) and when the chairholder plays a central “catalytic” role within the unit. As such, CERC awards are enabling many institutions to strengthen their leading position or to become world leaders in targeted research areas.

Moreover, CERC units have been successful overall in obtaining support from their host institutions and in leveraging funds from other sources, and generally reported that they have sufficient resources to carry out their research programs until the end of the CERC award. However, there is considerable variability in the amount of leveraged support across individual CERC units and several have experienced challenges and delays in securing funding and setting up facilities and teams.

While most of the infrastructure and some critical mass of expertise are expected to be maintained after the end of the CERC award, concerns were raised regarding the ability of chairholders and their host institutions to sustain the momentum generated by the CERC awards in terms of broader research capacity. Based on available evidence, it is likely that there will be adequate funding for salaries, major infrastructure and space, but there may be a lack of equivalent public and private funding sources to support HQP and direct and indirect research costs to drive the research forward beyond the CERC award. Institutional sustainability plans may not result in the expected levels of support of chairholders, especially if the CERC units experienced delays in setting up research programs and are not yet in a position to be strongly competitive for future research funding. Various solutions have been proposed to address this issue, and some exemplars and best practices related to sustainability issues were observed and can be used to support future improvements to the program. However, it is too early to determine exactly how CERC units will fare after the end of the awards and what the repercussion on the outcomes of the program related to the longer-term sustainability of research capacity that was enhanced through the CERC awards will be.

Enhanced research capacity in S&T priority areas

The case studies, document review, and survey have shown that the CERC awards have been “game changers” for the host institutions by integrating and building on existing research expertise in one or more S&T priority area(s), or in some cases, by developing new strengths in priority research areas within the institution. In both instances, the CERC chairholder has brought new knowledge, ways of thinking and collaborations to the institution, and has been instrumental for leveraging additional research investments.

It is worth highlighting again the large size of most CERC units, which usually involves dozens of high-calibre researchers and HQP, as shown in Section 3.2.1. As reported in the case studies, a key success factor for enhancing research capacity was found to be the CERC chairholder’s leadership qualities, acting as an “integrator”, “catalyst” or “nucleus” around which a critical mass of expertise is built. This critical mass of expertise includes not only key researchers already at the institution, but also new hires/faculty positions created to attract leading international researchers, high-quality training opportunities to attract HPQ (e.g., students, postdoctoral fellows, research professionals/coordinators).

Concurrently, the establishment of state-of-the art research infrastructure also contributed to enhanced research capacity within the institution. Notably, 93% of surveyed institutions reported improved research infrastructure as a positive impact of the CERC program, whereas 71% reported positive impacts on the reallocation of internal funding and 64% on the research capacity of the institution.

This finding is also supported by a mapping of the level of integration and the type of structure of CERC units that was conducted based on case study evidence. This mapping shows that the integration of CERC units within their respective host institutions varies along a spectrum: 14 out of 18 units were categorized as “highly” or “fully” integrated, while the remainder appeared to be less so. The degree of integration was assessed based on the linkages of the CERC unit with or within existing groups, research centres and/or research areas, as well as the creation of new centres or institutes around the CERC unit. Higher integration usually involved an increased level of responsiveness and support—financial and otherwise—from the host institution (via various levels of the administration and/or individual departments), also considered as being indicative of the university’s priorities (e.g., strategic research plan). Conversely, the few CERCs that were found to be less integrated generally functioned as independent and more decentralized research units, and some also had to overcome certain challenges (e.g., new research focus for the institution with the priority S&T area, delays, lack of visibility). Note that, given the early stage of the CERC awards, this assessment should be considered as preliminary (i.e., it is possible that the degree of integration of CERC units may still evolve over the period of the CERC award).

From the perspective of institutions, enhancement of research capacity as supported through the CERC program provides additional valuable benefits, such as the production of high-impact and high-quality research and intellectual property, and increased opportunities for academic collaborations with leading researchers at the national and international levels in fields related to S&T priority areas. While it is too early to detect a measurable impact on scientific performance of host department using bibliometrics, a positive effect on scientific impact has been reported based on the net gain observed on host departments’ international co-publication rates given the remarkably high rate of international co-publication—85%—of chairholders (see also section 3.2.1). Combined with the evidence on enhanced research capacity, this indicates that CERC awards are enabling many institutions to strengthen their leading position or to become world leaders in targeted research areas. As discussed below, this relates directly to the increased competitiveness of institutions in S&T priority areas, which has important implications for the sustainability of the research capacity being developed through the CERC award.

Additionally, within the host institutions, other departments/groups benefit from the existing and new collaborations and partnerships (including joint funding) enabled by the CERC unit, as well as shared infrastructure (e.g., laboratories) and equipment, considering the level of integration of the CERC unit (as described above). Evidence from several of the case studies found examples of greater synergy and cohesion generated across or within departments at several host institutions (see Exhibit 5). However, in some cases, the CERC award had the unintended (and unwanted) effect of reinforcing competitiveness between or within departments.

Exhibit 5 Synergy across departments of host institutions

The three CERC units in Quebec (at Université Laval and Sherbrooke) all demonstrated enhanced synergies within and across departments. The CERC chairholder in Photonic Innovations contributed to new collaborations across the Physics, Chemistry and Electrical Engineering departments after the chairholder proactively engaged with them and other faculty to participate in his program and share expertise, services and resources. The CERC chairholder in Remote Sensing of Canada's New Arctic Frontier brought new expertise to an already well-established team, and was also cited by key researchers to have reinforced the linkages, leading to increased collaborative research and teaching. Finally, the CERC chairholder in Quantum Signal Processing led to the creation of a research centre that allowed four faculty members to jointly hire staff and share resources and infrastructure for the benefit of their projects and students.

Resources required to support research program objectives

The administrative data review and case study evidence show that main sources of funding received by over half of the CERC units are the CERC program and their host institution; some have received larger amounts from other sources, as discussed below.

The CERC program provides $1.4 million per year for each CERC unit. Additionally, the administrative data review indicates that host institutions have planned to invest $64 million across the 18 CERC units examined,^{Footnote 68} and have reportedly provided $40 million, or over 60% of the planned amount, between 2010 and 2013.^{Footnote 69} Meanwhile, surveyed institutions reported that they were generally able to provide the level of resources required for the CERC chairholders and units to achieve their research objectives (at least to a moderate extent), especially in the form of human resources to conduct research (i.e., researchers and HQP), research space and infrastructure, as well as equipment and materials. The case studies further confirm that host institutions have invested large amounts in terms of space and equipment to complement existing infrastructure, including new and recently created institutes in the CERC units’ research fields. In addition to contributing to the chairholder’s salary and benefits, some new faculty positions were also created since the inception of the CERC units—mainly attracted by the CERC chair—which are supported by the institution. Finally, many CERC stakeholders reported that their universities provide them with ample administrative support (e.g., relating to research and training, grant applications, financial management).

However, the level of support provided by host institutions for individual CERC units varies considerably. For example, in 2012-13 and 2013-2014, 11 CERC units received support from their institution equal to or less than half of the CERC award amount (i.e., ≤$560,000), whereas the other 7 CERCs received 80% or more ($1.1 million to $8 million) ^{Footnote 70}. Because CERC units receive funding from a variety of sources, this does not necessarily mean that CERC units who received less funding from their host institution developed less research capacity. That said, due to data limitations, it was not always possible to determine if the amounts received reflect the initial commitments made by host institutions in the CERC proposals. Going forward, this suggests institutional investments could be more accurately tracked or monitored via the CERC program’s reporting tools and processes. Generally, based on the survey, the types of expenditures that have reportedly been covered to a more limited or moderate extent by host institutions (whether or not they had committed to do so) include human resources for administrative support, indirect costs and direct costs of research.

As shown in Table 8, CERC units, through the chairholders and high-calibre researchers, have leveraged over $128 million in funding from sources other than the CERC program and the host institution. A large proportion of this total, 72% ($93 million) comes from the federal government, granting agencies (other than from the CERC program) and CFI. Funding from foreign sources, provincial governments, institutions/trusts/foundations and corporations each account for between 3% and 7% of the total from other sources. Some of this leveraging has been obtained through collaborations (e.g., joint applications for grants) and partnerships with other organizations, which underscores both the importance of these collaborations and partnerships. However, as discussed in Section 3.2.3, CERC units have identified challenges in developing partnerships and securing funding from industry and expect these investments to increase, which suggests an opportunity to increase the relatively modest contribution of companies/firms to infrastructure and research funding for the CERC units.

Table 8 Leveraged infrastructure and research funding received by the CERC units (via high calibre researchers and chairholders), as reported in 2012-13 and 2013-14 ^{Footnote 71}

Source	Sum	Percentage of “leveraged funding”
Federal government	$39,419,051	30.6%
Granting agencies (NSERC, SSHRC, CIHR)	$31,008,204	24.1%
CFI	$22,316,524	17.3%
Foreign sources	$9,325,993	7.2%
Provincial government	$8,247,719	6.4%
Institutions, trusts or foundations	$5,346,161	4.2%
Corporations/firms	$3,846,437	3.0%
Other government	$1,442,894	1.1%
Voluntary organizations	$347,259	0.3%
Other	$7,443,840	5.8%
TOTAL	$128,744,082	100%

Source: Case Study Questionnaire (2014) and Progress Reports (2012-2013)

Similar to the findings on funding provided by host institutions, there is considerable variability in the funding received from other sources by individual CERC units. Indeed, in 2012-13 and 2013-2014, 10 CERC units reported leveraging a total equal to or less than the annual CERC award amount from other sources, i.e., ≤$1.4M, whereas the other 8 CERC units reported actual leveraging totals between $4 million and $40 million—greatly surpassing the annual value of the CERC award. These data, although partial or limited,^{Footnote 72} suggest that conclusions relating to leveraging (and associated implications for sustainability) should consider the distribution across individual CERC units rather than totals and averages across the program; it also remains early in the CERC awards’ lifecycle, so these findings are likely to change over time. Nonetheless, further evidence that CERC units have to date demonstrated mixed success in leveraging funding from external sources is provided by the survey, in which 57% of host institutions reported that external leveraging was a positive impact of the CERC program, whereas 43% reported that external leveraging occurred to a more moderate extent.

These findings also suggest a need to clarify the amount of funding that CERC units are expected to leverage from external sources, which may also depend on the field of research of the CERC units, in order to determine if the observed level of lower or higher than expected and if further leveraging should be encouraged. Note that for the second competition, host institutions must ensure 100 per cent in matching funds over the same period (excluding tri-agency and CFI funds);^{Footnote 73} this was not required in the first competition. Finally, this discussion also points to the need to better track and monitor leveraged funding, including planned versus actual, via the CERC program’s reporting tools and processes.

Indeed, the case study findings revealed that some CERC units have experienced challenges in securing the required or committed funding from some of the expected sources. More specifically, several CERC units faced challenges or delays in obtaining committed resources from provincial governments (and, in some cases substantial amounts committed at the time of nomination have not yet been received), and many reported delays in the fulfillment of university commitments (e.g., faculty hiring, lab space/construction). In some instances, this was said to be linked to changes in the economic and funding context since the inception of the CERC award. Moreover, some chairholders were unsuccessful or faced difficulties in applying for and using CFI funding (e.g., timing of grant application, flexibility in the use of CFI funds).

These funding challenges were often perceived to be more severe when they occurred at the front-end of the award period. Indeed, while CERC units generally have sufficient funds to carry out their research programs, the timing at which certain types of funding are available can be critical (e.g., infrastructure funding required at the front-end to set up essential facilities, equipment purchases, salary costs increase when HQP are hired). For example, equipment expenditures across the 18 CERC units exceeded or met the total (i.e., seven-year) planned amount within the first three years of the CERC, whereas actual salary expenditures across all types of HQP have only reached 18% of the total planned amount over the same period.^{Footnote 74} That said, not all CERC units may require the same amounts or types of funding at the same time. This suggests that greater flexibility in the use of funding would be beneficial in terms of addressing issues experienced with the implementation/ramp-up of CERC units, as well as helping to ensure the longer-term sustainability of the research capacity.

The challenges experienced by some CERC units in the early stages of the award resulted in greater than anticipated delays in setting up research facilities and teams, and the subsequent decreases in research output (e.g., publication) noted in previous sections. These delays have implications in terms of the longer-term sustainability of the critical research mass established through the CERC, and are also related to suggestions for potential improvements (e.g., to accelerate the ramp-up of CERC units) discussed in later sections.

Sustainability of the enhanced research capacity in priority S&T areas

Given that the initial 18 CERC awards are still ongoing, it is too early to assess the extent to which the enhanced research capacity in S&T priority areas will be sustained in the longer term. However, this evaluation was designed to help assess the likelihood that the enhanced research capacity attributed to the CERC units will be sustained beyond the duration of the seven-year award.^{Footnote 75} As stated in the CERC program’s Performance Measurement Strategy, “Given that the nature of impacts of R&D investments are necessarily long-term, it is important to recognize that evidence of the return on this investment will only begin to materialize at the end of the initial 7 year funding period.”^{Footnote 76} It is worth noting that chairholders also reported that, for the most part, their research programs have been designed with a longer-term horizon and scope than the CERC award.

Indeed, the achievement of intermediate and long-term outcomes of the CERC program requires that the critical mass of expertise and world-class research environment developed within host institutions be maintained at least to some extent (i.e., to maintain momentum and/or act as a legacy of the CERC award for the host institution), while strengthened partnerships with receptors of innovation, insights and HQP are required to increase the application of research-based knowledge, among other outcomes.^{Footnote 77} It should be noted that some key informants consulted have indicated that the achievement of these outcomes do not necessarily require the chairholders to continue to lead the next phase of the research program. Whether or not the chairholder remains at the host institution, it is expected that the momentum created through the CERC award will continue after the end of the award, through the critical mass of expertise, state-of-the-art facilities and other factors contributing to the comparative advantage developed by host institutions in S&T priority areas.
Overall, the evaluation found that, while most of the infrastructure and some of the critical mass of expertise are expected to be maintained after the end of the award, concerns were raised regarding the capacity of host institutions to sustain the research capacity generated by the CERC awards. The qualitative evidence collected as part of the case studies, surveys and document review point to specific types of resources that may be more challenging to obtain and the potential consequences of these challenges.

In the survey, the vast majority of institutions reported they would likely be able to sustain a critical mass in the research area(s) of the CERC(s) beyond the duration of the award to a large or very large extent. When asked to rate the likelihood that individual elements would continue to be supported beyond the duration of the CERC award (by the institution or via other sources), the majority of institutions identified faculty positions and the chairholder’s salary as the elements most likely to be supported (100% and 93%, respectively; Figure 8). Several institutions further specified that some of the faculty positions created were permanent or tenured, so they would continue to support them. Attraction and retention of HQP, and major infrastructure and space for the CERC’s programs of research were also likely to be supported (each rated as such by 80% of institutions). In the case studies, some of the stakeholders indicated that the infrastructure established as part of the CERC unit will constitute a “legacy” for the institution. However, chairholders and other members of the CERC unit were generally less likely than surveyed institutions to report that HQP would be attracted and retained, as will be discussed below.

Three categories appear to be of more concern (Figure 8): only around half of institutions consider it likely that support for maintenance of equipment and laboratories (57%), administrative support for CERC units (50%) and indirect costs (43%) of the research would be likely or very likely to be supported beyond the duration of the CERC award. Case study interviewees also generally expect that the facilities, equipment and centres could be sustained (including some of the associated administrative/technical support), but raised some challenges regarding longer-term support, from both institutional and external sources.

Finally, only 28% of surveyed institutions expect the direct costs of research to be supported (Figure 8). Many institutions specified that they expect chairholders to obtain grants or other external funds to cover the direct research costs, and while some chairholders have reportedly already secured ongoing funding, it is not always clear if these funds will be sufficient to support the future research program. In fact, a large majority of case study interviewees across groups explained that the current level of funding available from other grant programs (including “mainstream” NSERC and CIHR programs) is insufficient to drive their research forward as intended beyond the seven years of the CERC award. Some also expect that research grant funding available to them may continue to decrease in the future, as they have observed a progressive decline in the past few years, especially for basic research. In addition, while some funding can and has been leveraged from a variety of other sources (e.g., partnerships with industry or other sectors), there are also frequent challenges in obtaining sizeable and long-term investments from these sources (as noted previously).

Figure 8 Perceived likelihood that elements/costs will continue to be supported beyond the duration of the CERC award

Description of figure

Description of Figure 8, Perceived likelihood that elements/costs will continue to be supported beyond the duration of the CERC award

This graph is the same as Figure 2 above (including the title). This stacked bar graph shows participating institutions’ perceptions of how likely it is that specific elements/costs will continue to be supported after the Canada Excellence Research Chairs (CERC) award term ends. The source of the findings is a survey of participating institutions completed by 20 respondents, a 100% response rate.

The perceptions of how likely it is that elements/costs will continue were reported as follows:

Faculty positions created as a result of the CERC: 100% believed it was likely or very likely to continue
Chairholder salary: 7% believed it was very unlikely or unlikely to continue, 93% believed it was likely or very likely
Major infrastructure and space for CERC program of research: 14% were neutral, 79% believed it was likely or very likely to continue, 7% didn’t know or didn’t respond
Attraction and retention of highly qualified personnel: 7% believed it was very unlikely or unlikely to continue, 14% were neutral, 79% believed it was likely or very likely to continue
Maintenance of equipment and laboratories: 14% believed it was very unlikely or unlikely to continue, 21% were neutral, 57% believed it was likely or very likely to continue, 7% didn’t know or didn’t respond
Administrative support for CERC unit(s): 7% believed it was very unlikely or unlikely to continue, 36% were neutral, 50% believed it was likely or very likely to continue, 7% didn’t know or didn’t respond
Indirect costs of research for chairholder: 29% believed it was very unlikely or unlikely to continue, 21% were neutral, 43% believed it was likely or very likely to continue, 7% didn’t know or didn’t respond
Direct costs of research for chairholder: 36% believed it was very unlikely or unlikely to continue, 29% were neutral, 29% believed it was likely or very likely to continue, 7% didn’t know or didn’t respond

Looking to the future, most chairholders expressed concern about the limited resources of the institutions and existing public and private sector funding opportunities, and several expect they may need to reduce the size and scope of their CERC unit or research programs after the seven-year term of the program. Many chairholders explained that the most challenging issue is the retention of the CERC unit’s personnel (including key researchers and HQP), who are essential to ensure the continuity of research, support services, and the maintenance of equipment. This is especially true in cases where the CERC award has been the main source of support for personnel. Several researchers and HQP, and even some of the chairholders, will likely leave the host institution—by choice or because their position is no longer supported—at or near the end of the CERC term. Postdoctoral fellows and support staff (technical and administrative) are the most likely to leave, as their salaries cannot be fully supported by the level of funding provided by most research grants. Given these challenges, some researchers and HQP are said to have already started to look for other opportunities for the post-CERC period. In short, the ability of the university and/or the CERC unit to retain leading non-tenured key researchers and HQP and to support the expertise of highly skilled research professionals/technicians, will be a key factor of the sustainability of the CERC research programs and, more broadly, the research capacity in S&T priority areas within host institutions.

Finally, most chairholders expressed a clear intent to remain at the host institution beyond the CERC award and have been planning accordingly (see below); however, some indicated that they would consider taking a position elsewhere if they are unable to drive their research program forward due to resource and capacity constraints. As discussed throughout this report, chairholders play a central role in integrating and sustaining the research capacity within the CERC unit (e.g., catalytic effect; leadership and vision for research program; key success factor in attracting research talent). As such, it is likely that the departure of chairholders would lead to a negative impact on the sustainability of the research capacity established in host institutions as a result of the CERC award.

In this context, it is important to highlight that the sustainability question had been identified by the program and has been addressed directly from the inception of the program. More specifically, “sustainability” was a key risk identified as part of the CERC Performance Measurement Strategy, although the likelihood was rated low and the impact minor that there would be issues after the end of the award; as such the mitigation strategy was to “assess institutional plans to sustain the CERC unit and capitalize on the research advantage provided by the award”.^{Footnote 78} This was done as part of the application process, in which institutions were required to demonstrate their ability “to sustain the research advantage created by the proposed Chair after the seven-year term of the Chair expires”.^{Footnote 79}

A review of the completed application forms suggests that host institutions did not initially expect that sustainability would be a major issue. In summary, all host institutions expected to play a large part in ensuring the implementation sustainability of the CERC units by providing them with resources during and beyond the end of the award (e.g., faculty positions/salaries for the chairholder and/or key researchers in the CERC unit, planned/completed investments for infrastructure and equipment). Several also stated that their existing strengths and investments, the reputation of the chairholder, the CERC funding were expected to attract additional funding and ensure a certain degree of self-sustainability of the CERC units by the end of the award. In other words, CERC units would be competitive enough to raise the necessary support after the seven-year term. Moreover, host institutions often stressed that the CERC units had been selected based on their strategic areas (i.e., areas in which they are committed to maintaining or growing national or international leadership), and their contribution to the S&T priority areas. Both these elements were expected to help attract ongoing internal and external support to maintain the research capacity initially developed by the CERC units.

Now that the CERC units have been established, evidence from the case studies and the survey of institutions confirmed that, while these expectations remain valid for the most part, some institutional expectations have not yet been met and/or institutional sustainability plans have not been developed or implemented as planned. As discussed above, there have been delays in setting up lab facilities and hiring faculty, as well as challenges in leveraging investments from public and private sector. With regard to the hiring of faculty, some CERC units encountered challenges or delays specific to the hiring of Canada Research Chairs (CRCs) as planned in the initial CERC proposal, as well as setting up the infrastructure for the CRC chairholders. As noted previously, due to limited data availability, it was not always possible to determine the extent to which commitments from host institutions and other sources were received in full or in a timely manner (including funding and hiring of faculty to fill CRC positions); this suggests institutional and external commitments and investments could be more accurately tracked or monitored via the CERC reporting tools and processes.

The delays and/or challenges experienced by at least half of the first cohort chairholders in launching their research programs have often resulted in a slower production of scientific papers, as discussed in Section 3.2.1, and are therefore affecting the current level of competitiveness of their CERC units. In other words, the expected level of self-sufficiency and/or momentum has not yet been reached for all CERC units. It remains to be seen whether these CERC units will be able to catch up the delays and reach this point by the end of the seven-year term of the award. As indicated by CERC unit members and by host institutions, there is uncertainty with regard to the sustainability of the research capacity enabled by several of the CERC awards.

Given these challenges, many chairholders, key researchers, and host institutions have advocated for some form of renewal of the CERC award. It should be noted here that the non-renewability of the award has been a feature of the CERC awards since the onset of the program and has been communicated as such by the Chairs Secretariat. As an alternative to making the award renewable, some chairholders and institutions proposed that existing CERC chairholders could be allowed to compete against new nominees. Another option suggested by some surveyed institutions would be to allow a gradual phase-out of the award (possibly with additional funding), in order to facilitate the post-award transition, and thereby help ensure the sustainability of the CERC unit. Currently, a one-year phase-out period is allowed , over which previously allocated funding can be spread. Again, given the timing of this evaluation, it is not possible to provide conclusive evidence on the impact of the end of the CERC award (if any) on the research capacity in S&T priority areas, but the evidence presented across this evaluation suggests there will likely be some negative effects, and these effects may be more severe in some instances than in others.

In this context, several chairholders are exploring potential avenues to address sustainability issues. Given the importance of integration as a means to support research capacity, some are establishing strong linkages with other faculty within the department, faculty and institution with a view to mitigate the retention issue. Some chairholders are pursuing or have already secured alternate longer-term funding avenues, mechanisms or programs outside the Canadian system. Some are also seeking (with mixed success to date) to establish sustained collaborations and partnerships with other sectors, such as industry, government and not-for-profit. These collaborations and partnerships are expected to benefit the CERC units both by broadening the impact of their research and by attracting additional sources of support (e.g., via research translation into commercial products and spin-off companies, paid contract work). In addition, these collaborations and partnerships also were observed to generate career opportunities for researchers and HQP involved in CERCs to address attraction and retention issues.

Finally, the CERC program may find useful insights based on practices from comparable international programs, many of which have also encountered or sought to address similar sustainability issues. Notably, of the nine programs examined in depth, five are non-renewable; the exceptions are three shorter-term programs (2-5 years), and one 7-year program. Several non-renewable programs require a commitment from the institution to support the researcher’s position beyond the term of the award. For some programs (at least two of them), institutions are now required to submit detailed plans beyond the award (e.g., five additional years) for the long-term sustainability of the investment, including sources of external funding and faculty positions created or earmarked for the priority area. Note that only one program (in Russia) required leveraging of additional funds, and these funds came from the university, not from external partners. In sum, these programs largely place the responsibility of ensuring the sustainability of the research capacity on the side of the institutions rather than on other funding sources (e.g., granting agencies) or the program itself (e.g. via renewability of the award).

3.3 Performance – Program Design and Efficiency

3.3.1 Program design and efficiency

Summary of Findings:
Overall, the CERC program is administered in a cost-efficient way and has already identified and implemented several improvements. However, evaluation evidence points to several design and delivery features that could be further adjusted to improve program effectiveness. This includes program delivery features such as application and nomination processes, reporting, promotion and visibility, and funding and post-award transition.

Effectiveness and efficiency of the CERC program design

The cost-efficiency analysis that was conducted as part of this evaluation revealed that the CERC program is administered efficiently, when compared to what is typically expected from federal programs and to the performance of other similar programs. There is also strong qualitative evidence from the case studies and the survey indicating that the CERC program design is effective and efficient.

The CERC program was found to have a low average percentage of administrative expenditures compared to total expenditures (3.9%) over the period from 2010-11 to 2012-13 (Table 9). Typically, this percentage is expected to represent less than 10-15% of total expenditures, while less than 5% is generally considered to be low.^{Footnote 80} The ratio of administrative expenditures to grant expenditures was also low: administering the CERC program costs approximately $4 per each $100 of CERC funds granted. This ratio ranges from a high of 5.1 (2010-11) which is the first year the awards were granted and drops to less than 4 (3.6-3.7) in 2011-12 and 2012-2013. This appears to be due to the considerable increase of grant expenditures, compared to a more modest increase in administrative expenditures over the same period.

Table 9 CERC percentages of administrative to grant expenditures from 2010-11 to 2012-13

Fiscal year	Grant expenditures	Administrative expenditures	Ratio of administrative expenditures to grant expenditures (for each $100)	% Administrative expenditures / total expenditures
2010-2011	$15,366,666	$788,099	5.13	4.88%
2011-2012	$25,533,333	$926,796	3.63	3.50%
2012-2013	$25,200,000	$936,616	3.72	3.58%
Total	$66,099,999	$2,651,511	4.01	3.86%

Source: CERC financial datasets from NSERC-SSHRC Finance Division and program areas

The efficiency of a program is best assessed in light of other comparable programs. In that respect, the evaluation found that there was no meaningful difference between the efficiency ratios (administrative to grant expenditures) of the CERC program and those of two comparators, namely the Networks of Centres of Excellence (NCE) and Canada Research Chairs (CRC) programs (Table 10). Interestingly, these programs invest different proportions of funds in the various categories of administrative expenditures (e.g., direct salary, direct non-salary, indirect and direct non-attributable).^{Footnote 81} These differences in the distribution of administrative funds are likely due to differences in program design and delivery between the CERC, the CRC and the NCE programs.

Table 10 Cost-efficiency ratio for CERC and its comparators (2010-2011 to 2012-2013)

Program	Grant expenditures	Administrative expenditures	Ratio of administrative expenditures to grant expenditures (for each $100)	% Administrative expenditures / total expenditures
CERC	$66,099,999	$2,651,511	4.01	3.86%
CRC	$512,542,713	$10,136,042	1.98	1.94%
NCE	$223,530,500	$7,478,727	3.35	3.24%

Source: CERC financial datasets from NSERC-SSHRC Finance Division and program areas

Stakeholders were asked to comment on their level of satisfaction with various features of the CERC program design. Overall, there was a high level of satisfaction among case study interviewees and surveyed institutions with the amount, duration and flexibility of the award and the two-phase process (see Figure 9 for detailed survey results).

Many stakeholders praised the changes that were introduced for the second competition. These changes were intended to strengthen the program and reflect the recommendations of the Ad Hoc Panel on CERC Gender Issues. Aside from the equity-related changes, which have already been discussed in Section 3.2.1, key changes are described as follows:

Three new chairs in areas relating to the digital economy, targeting three S&T priority areas (environmental sciences, natural resources and energy, and health and related life sciences), while the rest are open to all areas of scientific enquiry that can be demonstrated to be of benefit to Canada.
To build on the program’s ability to leverage funds, and to reinforce the importance of partnerships within the innovation system, host institutions will be required in the competition process to identify a minimum of 100% ($10 million) in leveraged funds from partners.
Institutions were asked to invite only as many proposals to phase 2 as there are chairs available, to better enable the recruitment of top talent by providing a higher level of certainty to candidates, particularly women.
Adjusting competition timelines in response to universities’ request that more time is needed, particularly in phase 2, to recruit top candidates.

The positive impact of these changes is visible in survey responses which indicate a much higher level of satisfaction with the two-phase process for Competition 2 institutions. More specifically:

60% of institutions participating in Competition 1 and 88% of institutions participating in Competition 2 were satisfied with the application process.
Close to half of institutions participating in Competition 1 and 88% of institutions participating in Competition 2 were satisfied with the nomination process.

In particular, institutions are much more satisfied with the time provided to complete the application (i.e., phase 1; increase from 45% to 88% of institutions being satisfied or very satisfied), as well as the nomination process (i.e., phase 2; increase from 33% to 75%).

Table 9 CERC percentages of administrative to grant expenditures from 2010-11 to 2012-13

Satisfaction of institutions with various design and implementation features of the CERC program

Source: Survey of participating institutions (n=20; 100% response rate)

Description of figure

Description of Figure 9, the Satisfaction of institutions with various design and implementation features of the CERC program

This stacked bar graph shows participating institutions’ levels of satisfaction with various design and implementation features of the Canada Excellence Research Chairs (CERC) program. The source of the findings is a survey of participating institutions completed by 20 respondents, a 100% response rate.

The y-axis lists key design and implementation features of the CERC program. The x-axis shows percentage of respondents, from 0% to 100% in 20% increments. The bars are horizontal and each has up to four colours, representing the percentage of respondents who selected the following responses: very dissatisfied or dissatisfied, neither satisfied or dissatisfied, very satisfied or satisfied, or don’t know/not applicable.

The perceptions of the satisfaction with the design and implementation features were reported as follows:

Amount of award: 5% were very dissatisfied or dissatisfied, 5% were neither satisfied or dissatisfied, 90% were very satisfied or satisfied
Duration of the award: 10% were very dissatisfied or dissatisfied, 5% were neither satisfied or dissatisfied, 85% were very satisfied or satisfied
Promotion of the CERC program: 5% were very dissatisfied or dissatisfied, 20% were neither satisfied or dissatisfied, 75% were very satisfied or satisfied
Maximum number of Phase 1 applications that could be submitted by institutions: 10% were very dissatisfied or dissatisfied, 20% were neither satisfied or dissatisfied, 70% were very satisfied or satisfied
Allocation of chairs by government research priority areas: 5% were very dissatisfied or dissatisfied, 26% were neither satisfied or dissatisfied, 68% were very satisfied or satisfied
Feedback provided on recruitment strategies: 30% were neither satisfied or dissatisfied, 55% were very satisfied or satisfied, 15% didn’t know or didn’t respond
Leveraging of matching funds from the institution and other sources: 40% very dissatisfied or dissatisfied, 15% were neither satisfied or dissatisfied, 45% were very satisfied or satisfied
Non-renewability of the award: 40% were very dissatisfied or dissatisfied, 20% were neither satisfied or dissatisfied, 40% were very satisfied or satisfied
Requirement to report on the institutional recruitment process: 25% were very dissatisfied or dissatisfied, 60% were neither satisfied or dissatisfied, 15% were very satisfied or satisfied
Other: 5% were very dissatisfied or dissatisfied, 25% were very satisfied or satisfied, 70% didn’t know or didn’t respond.

However, both survey respondents and case study interviewees were generally less satisfied with the non-renewability of the award. Less than half of surveyed institutions (40%) reported being very satisfied or satisfied with non-renewability and the same proportion reported being very dissatisfied or dissatisfied. Similarly, about 45% of surveyed institutions were very satisfied or satisfied with leveraging of matching funds from the institution and other sources, while 40% were very dissatisfied or dissatisfied with this feature.

Suggested improvements to CERC program design and delivery

Stakeholders consulted during the case studies, survey and key informant interviews, as well as CERC Steering Committee members^{Footnote 82} suggested a series of enhancements that may constitute potential options for the continuous improvement process of the CERC program. A detailed list of suggestions is provided in Appendix D. Those considered to be best supported suggestions, and most relevant to the conclusions and recommendations are provided below.

Orientation of new chairholders and of institutions

Chairholders and institutions would benefit from more guidelines and best practices to facilitate the implementation of the chair, especially in the early, often critical stages of the award. These guidelines could take the form of:
- An orientation manual for chairholders covering issues such as how to navigate the Canadian research and research funding system, how to establish collaborations with Canadian industry, etc.;^{Footnote 83}
- Best practices for ramping up quickly (e.g., hiring of students, researchers and other HQP); and
- Suggestions on how to leverage funds and institutional support.

Promotion and visibility

Many stakeholders noted that efforts could be made to advertise the CERC program and disseminate the CERC units’ research results (e.g. through the media) more extensively and to a broader audience (i.e. international and general public).
The CERC program could also potentially consider improving student recruitment strategies (e.g., through posting of opportunities on the CERC program website) and implementing CERC student awards.

Funding issues (including leveraging and synergies with other programs)

Some chairholders and institutions suggested more flexibility in the use of funds:
- Ability to carry funds forward from one year to the other, to move funds from one budget line to another; with the possibility of using more funding for infrastructure expenses.
- Provide more front-end funding to chairholders and allow institutions to use CERC funding before the beginning of the seven-year term.

The CERC program could explore ways to facilitate the CFI application process and the use of CFI funds by chairholders. While this has been addressed to some extent in the second competition, as universities may include a request for infrastructure support from the CFI with their CERC nominations, some suggestions pointed to additional adjustments (e.g., setting aside CFI funds specifically for the CERC, separate from the institution’s general CFI envelope, to avoid creating competition within the institution for these funds).
Given uncertainties around the involvement of CRC chairholders in CERC units, the program could make efforts to better capture (and potentially to enhance) involvement of CRCs within CERC units.
As only 10% of HQP were found to receive funding support from other tri-agency programs, the CERC programs could explore opportunities to coordinate with these programs to increase this percentage (e.g., Vanier CGS, Banting, etc.).
Some high-calibre researchers and postdocs reported that they would benefit from being allowed to apply for tri-agency funding. The CERC program could explore whether opportunities exist for researchers to secure independent funding.

Post-award transition (for sustainability)

Some stakeholders and surveyed institutions suggested extending the duration of the award (e.g. ten years instead of seven years) to allow for more time to achieve research outcomes given the time needed to set up a chair.
Others suggested that chairholders could have access to a transitional award at the end of the CERC award, or that the program provides gradual phase-out of the CERC award (with or without additional funding). As noted previously, the CERC program recently announced a one-year extension with no additional funding.^{Footnote 88}

Reporting requirements and mechanisms

The evaluation team identified gaps in performance and financial reporting. The tools/processes were found to have some weaknesses, leading to data quality and availability issues. Some stakeholders noted that the volume/nature of data collected is not commensurate with the level of reporting that would be expected for an award of the CERC’s scale and duration.
- Interestingly, many international programs are currently being restructured to increase their focus on impacts (e.g., return on investment) through mid-term reviews and annual reports on progress.
Chairholders are eager to report on their scientific results/performance, especially as a means to position themselves for future funding opportunities.
The CERC program should consider addressing discrepancies in terms of alignment of the program outcomes (immediate and intermediate) with the current application and reporting requirements, regarding partnerships, collaborations and users of research (i.e. receptors of innovation). See also Section 3.2.3.

4.0 Conclusions and Recommendations

Overall, the evaluation found that there is a continued need for the CERC program, which is managed in an efficient manner. The CERC program was also found to have made good progress towards achieving all of its expected immediate outcomes. However, contextual changes have occurred since the inception of the CERC program that put greater pressure on its capacity to attract and retain world-class researchers. Given these shifts, as well as some challenges noted with regard to the level of achievement of longer-term outcomes, the evidence points to the need for greater clarification of expected program outcomes and further refinement of some design and delivery features to improve the program’s effectiveness.

The conclusions and associated recommendations are discussed below. The majority of these recommendations are intended to be implemented by the Chairs Secretariat, but will likely require developing and implementing actions in collaboration with other key stakeholders involved in the delivery of the CERC program, including host institutions and granting agencies.

Recommendation 1: The CERC program should be continued for an additional five years. The current context reinforces the need to continue supporting the program to help ensure Canada remains competitive at the global level.

The evaluation evidence supports continuing the program. The CERC program was initially launched in response to the S&T Strategy (2007) to achieve global excellence in research, and the current national and international contexts reinforces the need to continue supporting the CERC program to help ensure Canada remains competitive at the global level. Indeed, Canada faces fierce global competition to attract top research talent, with several countries having launched programs designed specifically for this purpose, many of which resemble the CERC program. The evaluation found that compared to similar international programs, the CERC award is competitive on a global scale based on its value and duration.

In the absence of the CERC funding, chairholders reported that they would not have had the opportunity to carry out research programs of such large scale and scope, especially given the limited availability of research funding in Canada and other countries. Many of them would have dropped key components of their research program or carried out fewer, more focused projects within a much smaller and less integrated/coordinated and multidisciplinary group.

The evaluation also found that the program is consistent and closely aligned with federal priorities outlined in the Government of Canada S&T Strategy because CERC awards are provided primarily in the four priority research areas and directly support the “Knowledge Advantage” and the “People Advantage” outlined in this strategy. The program also seeks to contribute to the “Entrepreneurial Advantage” by strengthening effective relationships with receptors of innovations in order to promote the development and application of leading-edge knowledge. Past and ongoing federal investments for innovation and research programs (including recent funding commitments for new CERC awards) also confirm that such programs continue to represent a priority for the government.

Although it is still early in the program’s lifecycle, the CERC program has made good progress towards achieving all of its expected immediate outcomes; however, some have yet to be achieved to the full extent. Most notably, the prestige of the award combined with the existing critical mass of research expertise in S&T priority areas within host institutions has contributed to the attraction of world-class researchers in a highly competitive global environment. The evaluation also found strong evidence that the CERC chairholders act as catalysts in their role as core members of CERC units: they bring in new knowledge and ways of thinking, new technology and new collaborations, leading to benefits for researchers, HQP and collaborators/partners.

CERC units have been highly successful in building on research collaborations in Canada and internationally, but partnerships and relationships with users of research have been more limited to date. There was considerable confusion and variability around the use of the terms “collaborations” and “partnerships” among case study respondents and program documents consulted. This confusion stems from a lack of shared understanding of the definition of these terms and of the associated expectations (i.e., outcomes) observed across the various stakeholders. However, while such partnerships are expected to be developed as programs of research evolve, evidence was found that the definitions and expectations vis-à-vis the CERC program related to partnerships and collaborations were unclear at various phases of the application and reporting processes, leading to challenges in assessing the achievement of this outcome. Clarifications are needed given the importance of strengthened relationships with partners and users of research for the achievement of program outcomes, including the sustainability of the CERC research capacity.

There was evidence that efforts made by host institutions, the CERC units and chairholders have contributed to the visibility of the host institutions and, to a more limited extent, of Canada as a destination of choice for research. However, limited evidence was found regarding the visibility of the CERC program outside of host institutions. As such, program stakeholders suggested that more efforts could be made to better promote the program and the research results of the chairholders, both nationally and internationally, including to the general public. Notably, one of the main limitations regarding the media analysis was the limited availability of promotional material from CERC units and institutions. As such, the evaluation team had to collect material from chairholders and/or their communications staff as part of the case studies.

Since the applications to the program are made by host institutions, and not by individual researchers, the main target audience for CERC program communications (e.g., CERC newsletter; program literature, etc.) have been Canadian higher education institutions. The responsibility of the Chairs Secretariat and the extent to which program resources could be devoted to broaden and enhance communication and promotion efforts to increase the visibility of the program beyond this core audience (i.e., to Canadian and international research communities, general public) is therefore up for debate and should be clarified. Furthermore, the Government of Canada also supports a range of other research programs with similar ‘branding’ policy objectives: CERC is part of a policy suite of programs that includes the CRC, Vanier CGS and Banting awards. It may therefore be more appropriate and efficient to develop joint promotional materials and/or assess the joint contribution of these programs to this shared outcome. Efforts to assess the visibility of the CERC program could therefore be limited to the key target population (i.e., Canadian higher education institutions).

As is discussed in more detail below, there was also a lack of clarity in the definitions and expectations regarding the program outcomes for sustainability. A clearer definition and shared understanding of each of these outcomes and of how the CERC program is expected to achieve them would help ensure that the CERC awards are selected, implemented and monitored accordingly.

Key considerations:

The program description and associated definitions should be reviewed and adjusted as needed to ensure a clearer and shared understanding of the policy expectations with regard to the following types of outcomes:
1. Collaborations, partnerships, and relationships with users of research, including making it more explicit how these relate to knowledge mobilization, leveraging and sustainability. Expectations related to partnerships and collaborations may differ based on the type of research conducted (e.g., fundamental, applied), which may require adjusting program outcomes. Definitions developed for similar terms in the context of the NCE program could serve as a model.
2. Branding, including its contribution in light of the range of other Government of Canada programs designed with similar policy objectives.
  - Clarify the role of CERC units and the expected level of effort the Chairs Secretariat should devote to communication and promotional efforts in light of the outcome to promote the recognition of Canada as a global destination of choice for research and higher learning.
  - Suggestions made by key researchers and HQP during case studies included the possibility of linking CERC units websites to the official CERC program website to enhance visibility of the CERC units.
  - To contribute to the assessment of this outcome, it would be beneficial to capture communications and promotional materials from CERC units and host institutions data in a more systematic way, including use of social media and other interactive tools and technologies, use for the promotion and visibility of the CERC units (e.g., research discoveries, feature articles, etc.). An approach for capturing this material as part of the reporting (e.g., annual progress reports) would be developed in consultation with the SSHRC Communications and Evaluation Divisions, to optimize resources for the efficient capture of data as per the performance measurement strategy, while also collecting material that can be used for contribute to the Tri-Council communications efforts.
  - The Chairs Secretariat, via the Communications Division may need to allocate additional resources to carry out enhanced communications activities if these are deemed necessary to support program outcomes.
3. Sustainability, including clarifying what is intended to be sustained beyond the award term (i.e., critical mass of expertise and/or research capacity rather than the research program); see also Recommendation 4.

There will likely need to be a series of subsequent adjustments to the CERC performance measurement strategy (PMS) and logic model, to the program’s application/nomination forms and review processes (e.g., revising the criteria that are associated with these outcomes), and to reporting mechanisms and tools (see also Recommendation 3) to ensure consistency and alignment across the program’s implementation and monitoring.

Recommendation 3: Improve reporting procedures, mechanisms and tools (e.g., annual reports, mid-term review) to ensure that the Chairs Secretariat has more comprehensive information in order to monitor the program and to better capture evidence of program outcomes over the long-term.

Overall, the CERC program has demonstrated good operational efficiency, and has already identified and implemented several improvements to its design and delivery model following Competition 1, as demonstrated by more positive experiences reported by institutions for Competition 2. It is expected that the program will continue to review and adjust application and nomination processes on an ongoing basis.

The evaluation identified a number of design and delivery features that could be further adjusted to improve program effectiveness, including program delivery (e.g., application and nomination processes, reporting), promotion and visibility, funding and post-award transition. Exemplars from the case studies and alternative practices identified in similar international programs have also been provided in this report.

In particular, the evaluation concludes that the CERC program needs to improve the monitoring of program results over time given that the impacts of S&T investments are long-term. In particular, the program should consider closely monitoring how the sustainability issue evolves as the program matures, as well as the impact of the changes implemented (e.g., improvements made after Competition 1, equity issues). The evaluation also identified limitations with respect to the availability and consistency of performance and financial data for Competition 1 that are reported in the progress reports, statement of accounts, and data available for the promotion of the CERC program. To address some of the limitations in the data available on CERC units, the evaluation team developed a case study questionnaire which was completed by all chairholders and used as baseline information for the evaluation.

Key considerations:

This recommendation aims to ensure that the Chairs Secretariat benefits from the collection of more comprehensive information to monitor the program and to capture program outcomes over the long-term (including financial data and implementation of sustainability plans), as explained in Recommendation 4.

This review should be done in consultation with the Evaluation Division and SSHRC’s Corporate Strategy and Performance Division, so that needs for future program evaluations will be effectively supported by enhanced performance measurement of the CERC program. Indeed, although the PMS was found to be adequate, the performance data collected did not fully support this program evaluation, such that additional data collection was required to address information gaps (i.e., case study questionnaires).

Given the duration and amount of CERC awards and the responsiveness of CERC units to requests made for this evaluation (i.e., case study questionnaires and offers of additional information on scientific outcomes), there is room to moderately increase reporting requirements.
Specific areas identified for improvements to reporting (via annual reports and/or mid-term review) are as follows:

To better assess progress towards sustainability outcomes (see Recommendation 4), including beyond the end of the award:
- Monitor matching funds and/or commitments made by institutions over time (including harmonizing financial data being collected in SOAs and progress reports).
- Capture information on the development and implementation of the sustainability plan.
- Track changes in research capacity within CERC units over time, including data on key researchers and HQP within CERC units (including CRC chairholders and those supported by other funding mechanisms), building on evaluation and/or baseline data.
Capture key scientific outcomes of the chairholder and other key members of the CERC unit to help assess achievement of related intermediate and long-term outcomes. It was requested by CERC chairholders that assessment of scientific performance be broadened as to not only include themselves, but other individuals within the CERC unit (i.e., graduate students, postdoctoral fellows, key researchers).
Collect data on the type and impact of collaborations and partnerships, as well as visibility of the program, aligned with clarified definitions and expectations (see Recommendation 2).

Review of reporting mechanisms of similar international and national programs as well as the case study questionnaire developed as part of the evaluation study, to be used as potential examples.
The Secretariat may have to seek additional resources to develop/adjust reporting tools (e.g., reporting templates) and ongoing track/analyse performance data on an ongoing basis.

Recommendation 4: Identify, monitor and promote best practices for the sustainability of the research capacity developed as a result of the CERC awards (i.e., critical mass of researchers and HQP; infrastructure).

The CERC program has been a strong incentive for host institutions to build or expand their infrastructure, in partnerships with CFI, which has clearly contributed to enhanced research capacity in S&T priority areas. Indeed, the CERC awards overall have been successfully used to create state-of-the-art research infrastructure and to build and/or structure a critical mass of expertise in the targeted fields and prompted the development of state-of-the-art infrastructure through university matching funding and the CFI program. This research capacity is usually well integrated within host institutions, meaning that CERC units have established strong linkages with or within existing groups, research centres and/or research areas. A higher achievement of this outcome was associated with two success factors: first, when chairholders played a central “catalytic” role within the CERC unit around which the research capacity and integration was developed, and second, sustained institutional support (financial and otherwise).

CERC units generally felt that they had sufficient resources to achieve their research objectives by the end of the CERC award. The main sources of funding for the CERC units were most often the CERC program itself and their host institution; some units also leveraged considerable funding from other sources. CERC units as a group have successfully leveraged over $128 million in funds from a variety of other sources, but primarily from the federal government (72%), and leveraging of individual units varies widely in amounts and sources. Foreign sources, provincial governments, institutions/trusts/foundations and firms each account for a small percentage of the remainder of leveraged funding (between 3% and 7%). Notwithstanding this success, there were also challenges experienced in developing their research capacity as per the intended plan, including delays at the early stages of implementation of the CERC award beyond those originally anticipated. These challenges slowed their research program and generation of scientific results, and also have implications in terms of the achievement of enhancing and maintaining the research capacity of host institutions.

In light of the challenges encountered by some CERC units during the ramp-up phase of the award (e.g., unexpected delays in setting up CERC facilities and/or research group), and the subsequent delays experienced by these units to generate research results, the evaluation concludes that the Chairs Secretariat could adjust some of its delivery features to facilitate the early implementation phase of the CERC awards. Some changes have already been made for this purpose in the second Competition (e.g., universities may include a request for CFI funding with their CERC nomination), but there remains room for additional flexibility in the use of CERC funds (e.g., more front-end funding, carry funds forward between years or move funds from one budget line to another). Currently, the CERC award can be extended for one year beyond the seven-year term (with no additional funding; additional extensions may be granted with justification). New chairholders could also benefit from orientation, such as in the form of guidelines and best practices, to facilitate the implementation of the CERC unit, especially in the early, critical stages of the award.

In this context, concerns were also raised with respect to the sustainability of the research capacity after the end of the CERC award, namely whether the critical mass of expertise and world-class research environment developed in S&T priority areas at host institutions would be maintained. Note that the sustainability outcome was perceived in different ways (e.g., sustainability of research capacity developed via the CERC unit vs. continuation of CERC-funded research projects/program vs. maintenance of CERC funding or equivalent level), such that this outcome could benefit from further clarification in the program description and definitions (see Recommendation 2).

For this evaluation, the ability to maintain state-of-the-art infrastructure and research spaces, and to retain a critical mass of outstanding tenured and non-tenured key researchers, HQP, and highly skilled professionals were all considered essential elements of the sustainability of the CERC-enabled research capacity developed in S&T priority areas within host institutions. This was also alternatively defined as “momentum” or as a “legacy” of the CERC award, such that it does not necessarily include the continuation of individual research projects supported by the award or the retention of the chairholder at the host institution. In other words, if essential elements of the CERC unit are not adequately supported after the end of the CERC award and if some key researchers, HQP, or chairholders leave the host institution—which was a possibility identified in the evaluation evidence—this raises potential issues with regard to the sustainability of the research capacity at host institutions in S&T priority areas.

The evaluation found that institutional sustainability plans (which may be formal or informal) may not result in the expected level of support, especially if the CERC units experienced delays in setting up research programs and are not yet in a position to be competitive for future research funding. Generally, it was perceived that institutions would provide adequate support for salaries, major infrastructure and space after the end of the award. However, there may be a lack of equivalent public and private funding sources to support HQP and direct and indirect research costs required to maintain momentum in the S&T priority area beyond the CERC award, i.e., the level of funding sufficient to maintain an appropriate level of research capacity may be different (higher or lower) than the CERC award amount. Indeed, the level of critical mass of expertise and funding required to maintain the CERC units’ research capacity likely varies by field, by type of problem to be solved, etc. Based on the evaluation findings, the most challenging issue in terms of critical mass is likely to be the retention of the CERC unit’s research personnel (including key researchers and HQP) and technical personnel, whose expertise is essential to ensure the continuity of research, support services, and maintenance of equipment within the host institution.

However, it is important to note that since the first cohort of CERC awards have not yet completed their terms, it is too early to definitively assess this outcome (i.e., the extent to which research capacity will be sustainable in S&T priority areas), or to identify robust predictors to help the Secretariat and reviewers identify CERC proposals that are more likely to lead to sustainable research capacity. That said, early ramp-up of CERC units and ongoing institutional support in the CERC unit’s research area are both likely to play an important role in contributing to sustainable research capacity in the longer term. Institution-led sustainability plans that extend beyond the funding period were also noted as a best practice in the international comparison study.

Finally, some synergies with other funding mechanisms and the development of partnerships were noted as part of this evaluation, and were both identified as success factors for the enhancement and sustainability of research capacity (i.e., via leveraged funding). However, opportunities were identified to ensure that funding mechanisms that can and should work in concert with the CERC award are used more systematically or in a more timely manner (e.g., CFI, CRC, support programs for HQP). Note that the need for clarifying expectations with respect to partnerships and leveraging is addressed in Recommendation 2.

Key considerations:

The Chairs Secretariat should examine ways in which it could facilitate post-award transition for the first cohort of CERC units, in collaboration with universities. For example, the current extension of the award term could also help CERC units who encountered delays to better position themselves for the post-award period; other options could be explored by the Secretariat in collaboration with host institutions.
Best practices to be explored that would be the responsibility of the Chairs Secretariat include providing greater flexibility in the use of CERC funds (e.g., carry-over, more front-end loading), which would allow CERC units to adjust to some emerging needs and issues in their particular context/case, given the variability in the issues encountered by individual CERC units during the early implementation of the CERC award.
Best practices to be explored that would be the responsibility of the host institutions include developing and implementing a sustainability plan outlining the institution’s plan for early ramp-up of CERC units and their ongoing support for the S&T research area relating to their CERC unit(s), including beyond the end of the CERC award; the CERC unit itself could also be encouraged to develop and implement a continuity/sustainability plan specific to its research program. The Chairs Secretariat could play a role here in providing guidance, in promoting or requiring such plans be developed at a given time (e.g., annual updates or at year 5), and in monitoring the plans (see also Recommendation 3).
To further address early implementation issues, build momentum more quickly, and contribute to enhancing sustainability, better orientation of newly appointed chairholders would be important and could take various forms. Orientation materials could include some form of mentoring by a researcher from an established CERC unit (chairholder or key researcher), and/or a “new chairholder manual” or guidelines, including on advice on aspects such as HQP recruitment, best practices on partnerships/collaborations, infrastructure funding and maintenance, etc.
- These guidelines would be distinct from the “Recruitment Best Practices” prepared for nominating institutions to provide advice on how to recruit world-class researchers for the CERC award, but could take a similar approach and/or format.

While it is expected that the next evaluation will be able to assess the sustainability issue more fully, in the interim, the Secretariat would benefit from monitoring this issue as the program matures as outlined in Recommendation 3, and reviewing the applications to see if there exist early predictors for the achievement of this outcome to inform the future selection of CERC awards. Given the risks associated with the potential loss of critical mass in host institutions (e.g., researchers, HQP, maintenance of infrastructure) after the end of the CERC awards, it would also be beneficial to assess the implications of this situation for other outcomes of the program (e.g., attraction of the next cohort of world-class researchers, branding of Canada as a location of choice, partnerships).
As outlined in Recommendation 3, it would be beneficial to capture the contribution of other programs within CERC units as part of the current reporting mechanism. It should be noted that some baseline data has been captured as part of this evaluation and that other available data (e.g., current reporting mechanism) provide an incomplete portrait of the actual situation.
- Hiring of CRC chairholders to work with or within the CERC units as well as other chair programs (e.g., IRCs) that could be associated with CERC units in a beneficial way.
- Given that only 10% of HQP were supported by complementary federal funding opportunities for HQP (e.g., Vanier CGS, Banting, etc.), there is likely room to improve this percentage, and thus contribute to improved retention of HQP and increased sustainability of CERC units.

Appendix A – CERC Logic Model

Description of figure

Description of Appendix A – CERC Logic Model

Appendix A shows a logic model for the CERC program. The logic model consists of seven rows, each containing between three and five boxes with text. From the bottom to the top, the rows are labelled as follows: inputs, activities, outputs, immediate outcomes, intermediate outcomes, long-term outcomes and strategic outcomes. There are also two long boxes or columns (one on each side of the page) labelled ‘Reach’ and ‘Critical Assumptions’.

The first row (on the bottom) has four boxes, each listing the inputs provided by one stakeholder group as follows:

Government (e.g., Industry Canada, Finance, CFI, Funding agencies, etc.) inputs include program design, policy input (STIC, Auto WG), partnerships, funding ($28.6 million), and previous investments.
CRC Secretariat/Funding agencies’ inputs include program design, implementation, management, and administration.
Universities’ inputs include research capacity, existing strengths (e.g., previous investments), infrastructure, and resources.
Broader community inputs include partnerships and alliances with universities/CERCs (private sector, public sector, e.g., provincial, federal, etc.)

The second row has four boxes, each listing the activities of one stakeholder group as follows:

Government activities include determining future research priorities.
Review Panels/Selection Board/CERC Management and Steering Committees activities include reviewing/approving high quality CERC proposals for funding, program governance and advice, and branding and promotion.
CRC Secretariat/Funding agencies’ activities include communications and marking, application management, selection board support, adjudication process support, program administration, performance and evaluation activities, support knowing mobilization, awards administration and financial monitoring, and institution/researcher assistance.
Universities’ activities include communications and marketing (partnerships with universities and CRC Secretariat, developing proposals for CERCs, recruiting and nominating world-class researchers in targeted research areas, providing support, leveraging funds from other sources, knowledge mobilization (transfer, dissemination, etc.), outcome reporting, awards administration.

The third row has three boxes, each listing one expected output as follows:

OUT #1 – Branding and communications (e.g., articles, events, etc.).
OUT #2 – Funded CERCs and their teams established at universities (up to 20 CERCs - $10 million/7 years).
OUT #3 – World-class research capacity in S&T Strategy research priority areas is in place in universities.

The fourth row has four boxes, each listing one expected immediate outcome as follows:

IMM #1 – Increased awareness of CERCs (promotion).
IMM #2 – Attraction and retention of world-class researchers, research trainees and other research professionals.
IMM #3 – Ambitious research results and partnerships in S&T Strategy research priority areas.
IMM #4 – Enhanced research capacity in universities in areas that align with S&T Strategy research priority areas.

The fifth row has five boxes, each listing one expected intermediate outcome as follows:

INT #1 – CERCs recognized for research leadership influence in S&T Strategy research priority areas.
INT #2 – Increased availability of HQP in S&T Strategy research priority areas.
INT #3 – Improved capacity to generate high-quality research results in S&T Strategy research priority areas.
INT #4 – Strengthened relationships with receptors of innovations, insights and HQP.
INT #5 – Strategic alignment of research enterprise to foster critical mass in S&T Strategy research priority areas.

The sixth row has three boxes, each listing one expected long-term outcome as follows:

LT #1 Branding / Research environment – Canada as a global destination of choice for research and higher learning.
LT #2 Research capacity – Internationally recognized critical mass of research results and discoveries, and capacity for application in S&T Strategy research priority areas.
LT #3 Research excellence – Canada recognized internationally for its preeminent researchers and breakthroughs in and/or application of knowledge generated by CERC groups.

The seventh row (on the top) lists three strategic outcomes as follows:

World-class Canadian research environment
Innovative and competitive economy
Improved prosperity of Canadians

The long box/column on the left is titled ‘Reach’ and lists the following areas of reach:

Government,
Universities,
Researchers,
Receptors of innovations, insights and HQP, and
General public.

The long box/column on the right is titled ‘Critical Assumptions’ and lists the following assumptions:

No compromise of research excellence,
Availability of international candidates,
Sufficient time for nomination and recruitment of world-class researchers,
Receptor capacity,
Sustainability, and
Leveraging.

Appendix B – CERC Logic Model Narrative

Outputs

O1 – Branding and communications:
- The CERC Communications Plan outlines an integrated mix of promotional and marketing initiatives, including public and media relations, special events and international marketing

O2 – Funded CERC Chairholders and their teams will be established at universities (up to 20 CERCs – $10M over 7 years):
- Researchers to establish research programs at Canadian universities, thus fulfilling the mandate of global research excellence in Canada, as outlined in the S&T Strategy

O3 – World-class research capacity in S&T Strategy research priority areas is in place in universities:
- Institutional leveraged funds invested in up to 20 new CERC units established in research priority and sub-priority areas. The governmental and institutional investment in the program will lead to regional investment and interests.
- Funded CERCs will also attract additional investment from other funding partners in research priority and sub-priority areas.

Immediate Outcomes

IM 1 – Increased awareness of CERCs (promotion):
- University administrators and the Chairholders act as ambassadors for the program in the research community; supported by the Secretariat and CERC Communications
- Promotion of the short- and long-term benefits of the program to Canadians and to Canada’s universities (i.e. research collaboration, commercialization opportunities, learning opportunities, positioning Canada as a research and learning destination, etc.)
- Canadian university administrators and the research community promote the CERC program, by supporting the CERC and their research teams within their institutions, and through collaboration with allied organizations (especially three federal funding agencies, and participating universities)
- Leveraging the credibility of the Chairholders and their innovative research through partnering with universities to draw public and media attention to the impact of their research. Accomplished through regional rollouts and regional media.

IM 2 - Attraction and retention of world-class researchers, research trainees and other research professionals—the program will enable:
- Research programs to be established in the S&T Strategy research priority areas in which Canadian universities have comparative strength assessed against global standards of excellence
- Canadian universities to create research opportunities that will retain the best Canadian researchers and attract the world’s best minds from other countries working in the research priority areas of the S&T Strategy
- The creation and reinforcement of existing research teams in these areas
- Attraction of the best students and post-doctoral fellows to a world-class research environment
- HQP training from leading researchers in their fields

IM 3 – Ambitious research results and partnerships in S&T Strategy research priority areas—the program will enable:
- CERCs to develop ambitious research programs around the research priority areas of the S&T Strategy
- CERCs and universities to formalize partnerships with the private sector, public sector, and other organizations in order to leverage additional support and to ensure knowledge transfer and application

IM4 – Enhanced research capacity in universities in areas that align with S&T Strategy research priority areas:
- CERC units will attract top scholars and students in research priority areas

Intermediate Outcomes

INT 1 – CERCs recognized for research leadership and influence in S&T Strategy research priority areas:
- The research undertaken by the CERCs and their teams will gain media and public attention as linked to the four priority areas
INT 2 - Increased availability of HQP in S&T Strategy research priority areas:
- Training of HQP will lead to the production of more and better graduates and researchers, and will therefore contribute to the increase in the pool of highly qualified personnel in Canada in S&T Strategy research priority areas.
- New training programs will be created in priority research areas
INT 3 - Improved capacity to generate high-quality research results in S&T Strategy research priority areas:
- Universities provide required financial support—through institutional investments and leveraging funds—to enable world-class research programs for up to twenty new research teams
- Attraction and retention of best researchers in the world, improved infrastructure, along with establishment of new research teams and expansion of existing teams will help universities improve their capacity to generate and apply new knowledge
INT 4 - Strengthened relationships with receptors of innovations, insights and HQP:
- Researchers within CERC units will increase collaborations with key partners across sectors and institutions
- Increased application of research-based knowledge to the work of partner organizations
- Increased capacity to orient, develop and partner in S&T Strategy research priority areas
INT 5 - Strategic alignment of research enterprise to foster critical mass in S&T Strategy research priority areas:
- CERC units assemble state-of-the-art facilities to perform research in priority areas
- Universities develop a comparative advantage in the S&T Strategy research priority areas within Canada and internationally, through targeting resources, and establishment of dynamic research teams in strategic areas of research

Long-Term Outcomes

Ultimately, the CERC program will help brand Canada as a global destination of choice for research and higher learning, helping maintain Canada’s competitiveness in the global market for the best and brightest.

It will also help Canada build or enhance internationally-recognized expertise in the S&T Strategy research priority areas, and allow the country to benefit from the application of the knowledge generated.

The CERC program will also foster research excellence and support the recognition of Canada as the home of pre-eminent researchers who are recognized internationally for their research breakthroughs and for the application of knowledge generated through the CERC units.

The achievement of these long-term outcomes will be assessed through evaluation.

Link to Overarching Strategic Outcomes

The CERC program is a key component of the S&T Strategy, and, together with the Government’s overall investments in research and higher-learning, contribute to:

Building a world-class research environment in Canada;
Building an innovative and competitive Canadian economy; and, ultimately,
Improving the prosperity of Canadians.

Appendix C – Data Collection Matrix

Appendix D – Detailed List of Suggested Improvements

Application and nomination processes

Increase the time allocated for the preparation of the application, and for the nomination and relocation processes ^{Footnote 86}
- Provide institutions with the timeframe for the competition period in advance so that they can start the application and/or recruitment process several months before the beginning of the competition.
- Allow nominated chairholders a longer period of time in order to decide whether to accept or decline the award.
- The review process could also use a panel of experts with a multidisciplinary approach.

The CERC program might consider establishing an open call or more regular competitions, to allow institutions to recruit the best researchers instead of taking the risk to fund those that were available, but not necessarily the top-ranked candidates.
- At the international level, three of the nine programs examined allow proposals to be submitted at any time during the year and evaluate the proposals on a case-by-case basis. Most of the other programs have a yearly cycle.

To ensure that institutions have sufficient capacity to host CERC chairs, the CERC program could request the submission of more detailed administrative information at the time of application.

Access and equity

The CERC program could explore strategies to coordinate with Citizenship and Immigration Canada with regard to accelerating and simplifying the immigration process for the chairholder and/or his family.
Some institutions offered employment opportunities to the chairholder’s spouse, some did not. While this was not widely reported as a major challenge, some chairholders suggested that the CERC program should communicate with institutions on this issue to facilitate dual employment offers for spouses.

Orientation of new chairholders and of institutions

Chairholders and institutions would benefit from more guidelines and best practices to facilitate the implementation of the chair, especially in the early, often critical stages of the award. These guidelines could take the form of:
- An orientation manual for chairholders covering issues such as cultural specificities, how to navigate the Canadian research and research funding system, how to establish collaborations with Canadian industry, etc.;^{Footnote 87}
- Best practices for ramping up quickly (e.g., hiring of students, researchers and other HQP); and
- Suggestions on how to leverage funds and institutional support.

Promotion and visibility

Many stakeholders noted that efforts could be made to advertise the CERC program and disseminate the CERC units’ research results (e.g. through the media) more extensively and to a broader audience (i.e. international and general public).
The CERC program could also potentially consider improving student recruitment strategies (e.g., through posting of opportunities on the CERC program website) and implementing CERC student awards.

Funding issues (including leveraging and synergies with other programs)

Some chairholders and institutions suggested more flexibility in the use of funds:
- Ability to carry funds forward from one year to the other, to move funds from one budget line to the other; with the possibility of using more funding for infrastructure expenses.
- Provide more front-end funding to chairholders and allow institutions to use CERC funding before the beginning of the seven-year term.

The CERC program could explore ways to facilitate the CFI application process and the use of CFI funds by chairholders. While this has been addressed to some extent in the second competition, as universities may include a request for infrastructure support from the CFI with their CERC nominations, some suggestions pointed to additional adjustments (e.g., setting aside CFI funds specifically for the CERC, separate from the institution’s general CFI envelope, to avoid creating competition within the institution for these funds).
Given uncertainties around the involvement of CRC chairholders in CERC units, the program could make efforts to better capture (and potentially to enhance) involvement of CRCs within CERC units.
As only 10% of HQP were found to receive funding support from other tri-agency programs, the CERC programs could explore opportunities to coordinate with these programs to increase this percentage (e.g., Vanier CGS, Banting, etc.).
Some high-calibre researchers and postdocs reported that they would benefit from being allowed to apply for tri-agency funding. The CERC program could explore whether opportunities exist for researchers to secure independent funding.
Given the challenges faced by some small-to-medium sized institutions to leverage additional funding, a few survey respondents suggested reducing the matching funds requirement (introduced in the second competition) for these institutions.

Networking and collaborations

There is widespread interest across members of CERC units to have more opportunities to meet and exchange with members from other CERCs and to establish collaborations between similar or complementary fields of research. For example, several HQP suggested holding an HQP-specific CERC Annual Meeting or HQP-oriented activities at the CERC Annual Meeting.

Post-award transition (for sustainability)

Some stakeholders and surveyed institutions suggested extending the duration of the award (e.g. ten years instead of seven years) to allow for more time to achieve research outcomes given the time needed to set up a chair.
Others suggested that chairholders could have access to a transitional award at the end of the CERC award, or that the program provides gradual phase-out of the CERC award (with or without additional funding). As noted previously, the CERC program recently announced a one-year extension with no additional funding.^{Footnote 84}

Reporting requirements and mechanisms

The evaluation team identified gaps in performance and financial reporting. The tools/processes were found to have some weaknesses, leading to data quality and availability issues. Some stakeholders noted that the volume/nature of data collected is not commensurate with the level of reporting that would be expected for an award of the CERC’s scale and duration.
- Interestingly, many international programs are currently being restructured to increase their focus on impacts (e.g., return on investment). They use mid-term reviews and require annual reports on progress.

Chairholders are eager to report on their scientific results/performance, especially as a means to position themselves for future funding opportunities.
The CERC program should consider addressing discrepancies in terms of alignment of the program outcomes (immediate and intermediate) with the current application and reporting requirements, regarding partnerships, collaborations and users of research (i.e. receptors of innovation). See also Section 3.2.3.

Footnotes

Footnote 1

As stipulated in the Transfer Payment Policy, all on-going programs of grants and contributions must be subject to a review of relevance and effectiveness every five years and, where appropriate, recommend the continuation, amendment or termination of the terms and conditions for these programs.

Return to footnote 1 referrer

Footnote 2

Department of Finance Canada. (2006). Advantage Canada: Building a Strong Economy for Canada. Retrieved from: http://www.fin.gc.ca/ec2006/plan/pltoc-eng.asp

Return to footnote 2 referrer

Footnote 3

Industry Canada. (2007). Mobilizing Science and Technology to Canada’s Advantage – Summary. Retrieved from: http://www.ic.gc.ca

Return to footnote 3 referrer

Footnote 4

The CERC program was announced in Budget 2008. Department of Finance Canada. (2008). The Budget Plan 2008: Responsible Leadership. Retrieved from: http://www.budget.gc.ca/2008/home-accueil-eng.html

Return to footnote 4 referrer

Footnote 5

The review process for both the application and nomination stages of the CERC program is described on the program’s web site. Retrieved from: http://www.cerc.gc.ca/program-programme/cpan-pccs-eng.aspx

Return to footnote 5 referrer

Footnote 6

The selection board includes world-renowned international and Canadian experts, and senior management of the three funding agencies, who assess proposals to determine which ones represent the best strategic investment made through the CERC program, based on the evaluation criteria. These recommendations are provided to the tri-agency steering committee which approves the successful applications or nominations.

Return to footnote 6 referrer

Footnote 7

There were 19 inaugural recipients of the CERC in the first competition. There are currently 18 CERC chairs.

Return to footnote 7 referrer

Footnote 8

Patrik Rorsman was initially awarded a CERC chair, but terminated his award after one year. The evaluation focused primarily on the remaining 18 active CERC awards.

Return to footnote 8 referrer

Footnote 9

Department of Finance Canada. (2011). Budget 2011: the next phase of Canada's Economic Action Plan—A Low-Tax Plan for Jobs and Growth. Retrieved from: http://www.budget.gc.ca/2011/glance-apercu/brief-bref-eng.html

Return to footnote 9 referrer

Footnote 10

This multi-level peer review process includes an evaluation by external experts, assessment by a review panel, strategic review by the selection board, and final approval by a steering committee

Return to footnote 10 referrer

Footnote 11

An additional Chair became available due to a vacancy from the inaugural competition.

Return to footnote 11 referrer

Footnote 12

CERC unit (definition): The group of researchers and highly qualified personnel (HQP) at a given institution that collaborate closely with the chairholder on the proposed program of research, whether they are funded through the CERC award or through other sources.

Return to footnote 12 referrer

Footnote 13

The initial version of the CERC PMS was prepared in 2009, and was revised in 2010 (latest version dated 18 November 2010).

Return to footnote 13 referrer

Footnote 14

SSHRC. (2010). Performance Measurement Strategy (PMS) for the CERC Program.

Return to footnote 14 referrer

Footnote 15

SSHRC. (2010). Performance Measurement Strategy (PMS) for the CERC Program.

Return to footnote 15 referrer

Footnote 16

Treasury Board of Canada Secretariat. (2009). Policy on Evaluation. Retrieved from: http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024

Return to footnote 16 referrer

Footnote 17

Science, Technology and Innovation Council (STIC). (2012). State of the Nation 2012. Canada’s Science, Technology and Innovation System: Aspiring to Global Leadership. Ottawa (ON)

Return to footnote 17 referrer

Footnote 18

Cornell University, INSEAD and the World Intellectual Property Organization. (WIPO). 2013. The Global Innovation Index 2013: The local dynamics of innovation. Dutta, S. & Lanvin, B. (eds), Geneva. Retrieved from: http://www.globalinnovationindex.org/content.aspx

Return to footnote 18 referrer

Footnote 19

Industry Canada. (2011). Innovation Canada: A Call to Action. Ottawa (ON): Independent Panel on Federal Support to Research and Development.

Return to footnote 19 referrer

Footnote 20

Council of Canadian Academies. (2013). Paradox Lost. Explaining Canada’s Research Strength and Innovation Weakness.

Return to footnote 20 referrer

Footnote 21

OECD (2013). OECD Science, Technology and Industry Scoreboard 2013. Innovation for Growth. OECD Publishing. Retrieved from:

Return to footnote 21 referrer

Footnote 22

Council of Canadian Academies. (2013). Paradox Lost. Explaining Canada’s Research Strength and Innovation Weakness

Return to footnote 22 referrer

Footnote 23

Science, Technology and Innovation Council. (2012). State of the Nation 2012. Retrieved from: http://www.stic-csti.ca/eic/site/stic-csti.nsf/vwapj/StateOfTheNation2012-may16-eng.pdf/$file/StateOfTheNation2012-may16-eng.pdf

Return to footnote 23 referrer

Footnote 24

Return to footnote 24 referrer

Footnote 25

Return to footnote 25 referrer

Footnote 26

Industry Canada. (2007). Mobilizing Science and Technology to Canada’s Advantage – Summary. Retrieved from: http://www.ic.gc.ca

Return to footnote 26 referrer

Footnote 27

Department of Finance Canada. (2008). The Budget Plan 2008: Responsible Leadership. Retrieved from: http://www.budget.gc.ca/2008/home-accueil-eng.html

Return to footnote 27 referrer

Footnote 28

Council of Canadian Academies. (2013). Paradox Lost. Explaining Canada’s Research Strength and Innovation Weakness.

Return to footnote 28 referrer

Footnote 29

Association of Universities and Colleges of Canada. (2014). Submission to the Federal Science, Technology and Innovation Review. AUCC Response to the Federal Science, Technology and Innovation (ST & I) Consultation Paper. Ottawa (ON).

Return to footnote 29 referrer

Footnote 30

OECD (2013). OECD Science, Technology and Industry Scoreboard 2013. Innovation for Growth. OECD Publishing. Retrieved from:

Return to footnote 30 referrer

Footnote 31

5 of the 9 short-listed programs were launched in 2008 or after, and only 4 of the other 24 programs examined were launched before 2000.

Return to footnote 31 referrer

Footnote 32

Chairs Secretariat. (2014). General Statistics of Currently Active Chairs (As of 2014-03-31) [for Canada Research Chairs]

Return to footnote 32 referrer

Footnote 33

The eligibility criteria for the CERC (2012 competition) indicate that researchers already based in Canada can be nominated for a CERC, but they may not be nominated by the university at which they hold the tenured or tenure-track position and the nominating university must demonstrate the net benefit to the country in moving the researcher from one Canadian institution to another. CERC. (2014). Application and Nomination Process: 2012 Competition. Accessed from: http://www.cerc.gc.ca/program-programme/cpan-pccs-eng.aspx#a5

Return to footnote 33 referrer

Footnote 34

Industry Canada. (2007) Science and Technology Strategy: Mobilizing Science and Technology to Canada’s Advantage. Ottawa (ON).

Return to footnote 34 referrer

Footnote 35

Industry Canada. (2007). Mobilizing Science and Technology to Canada’s Advantage – Summary. Retrieved from: http://www.ic.gc.ca

Return to footnote 35 referrer

Footnote 36

Industry Canada. (2007). Mobilizing Science and Technology to Canada’s Advantage – Summary. Retrieved from: http://www.ic.gc.ca

Return to footnote 36 referrer

Footnote 37

Industry Canada (2007). Mobilizing Science and Technology to Canada’s Advantage – Summary. Retrieved from: http://www.ic.gc.ca

Return to footnote 37 referrer

Footnote 38

Science, Technology and Innovation Council (STIC). (2012). State of the Nation 2012. Canada’s Science, Technology and Innovation System: Aspiring to Global Leadership. Ottawa (ON)

Return to footnote 38 referrer

Footnote 39

Science, Technology and Innovation Council (STIC). (2012). State of the Nation 2012. Canada’s Science, Technology and Innovation System: Aspiring to Global Leadership. Ottawa (ON)

Return to footnote 39 referrer

Footnote 40

Government of Canada (2014). Budget 2014: The Road to Balance: Creating Jobs and Opportunities.

Return to footnote 40 referrer

Footnote 41

Government of Canada (2014). Budget 2011: A Low-Tax Plan for Jobs and Growth.

Return to footnote 41 referrer

Footnote 42

Government of Canada (2014). Budget 2014: The Road to Balance: Creating Jobs and Opportunities.

Return to footnote 42 referrer

Footnote 43

Justice Canada. (1995). Department of Industry Act. Retrieved online from: http://laws-lois.justice.gc.ca/PDF/I-9.2.pdf

Return to footnote 43 referrer

Footnote 44

Innovation Canada: A Call to Action. Review of Federal Support to Research and Development – Expert Panel Report Retrieved online from:

Return to footnote 44 referrer

Footnote 45

Ibid.

Return to footnote 45 referrer

Footnote 46

Sa, C. (2010). Canadian Provinces and Public Policies for University Research. Higher Education Policy, 23, 335-357.

Return to footnote 46 referrer

Footnote 47

Statistics Canada. (2013). Gross Domestic Expenditures on Research and Development in Canada (GERD), and the Provinces. Catalogue 88.221-X. Retrieved from: http://www.statcan.gc.ca/pub/88-221-x/88-221-x2013001-eng.pdf

Return to footnote 47 referrer

Footnote 48

Sa, C. (2010). Canadian Provinces and Public Policies for University Research. Higher Education Policy, 23, 335-357.

Return to footnote 48 referrer

Footnote 49

Canadian papers were identified based on author addresses as they appear in Thomson Reuter’s Web of Science (WoS)

Return to footnote 49 referrer

Footnote 50

The international comparisons were performed by comparing the scientific impact of chairholders to the world average which reflects, to a great extent, the impact of countries publishing mostly their peer-reviewed papers in international journals written in English. Indeed, these countries (e.g., most European countries, Australia, and the US) account for a large fraction of the world’s scientific production and they together constitute a sound comparable for this study.

Return to footnote 50 referrer

Footnote 51

Relative Citation (RC) score: This is an indicator of the scientific impact of individual papers relative to the world average (i.e., the expected number of citations) in a given field or sub-field. The citation count of a paper is divided by the average count of all papers published the same year in the field/sub-field, to obtain a relative citation count (RC). The citation impact of an entity (e.g., group of researchers) can be obtained by taking the median of the RCs of the papers belonging to it.

Relative Impact Factor (RIF): This is an indicator of the scientific impact of individual papers based on the impact factor of the journal in which it was published. It reflects the scientific “quality” measured by the average citation rate of the publication venue instead of the actual publication. The citation impact of an entity (e.g., group of researchers) can be obtained by taking the median of the RIFs of the papers belonging to it.

Return to footnote 51 referrer

Footnote 52

A comparison with researchers from programs targeting the top international research talent would provide a more robust indication of the calibre of the nominated researchers.

Return to footnote 52 referrer

Footnote 53

For more details on the Selection Board, refer to the section on the Selection Process on page 7 of this report.

Return to footnote 53 referrer

Footnote 54

Nominations were judged according to the same three criteria used by the Review Panel, which were: 1) the quality of the nominee (i.e., Recognition as a world leader or rising star with exceptional potential); 2) the quality of the proposed program of research; and 3) the fit with the university's proposal in phase 1.

Return to footnote 54 referrer

Footnote 55

There were 19 chairs awarded. Patrik Rorsman was initially awarded a CERC chair, but terminated his award after one year. The evaluation focused primarily on the remaining 18 active CERC awards.

Return to footnote 55 referrer

Footnote 56

According to surveyed institutions, this situation includes cases in which nominations were submitted to the Chairs Secretariat and subsequently declined the award or were refused, and cases in which candidates who were originally approached by the institution declined the offer, such that a different chairholder was subsequently nominated.

Return to footnote 56 referrer

Footnote 57

Ad Hoc Panel on CERC Gender Issues, 2010. Report to the Minister of Industry. Retrieved online from: http://www.ic.gc.ca/eic/site/icgc.nsf/eng/h_05589.html#discussion. The Ad Hoc Panel on CERC Gender Issues Report suggested that the Council of Canada Academies undertake a “thorough assessment (…) of the data and issues around the advancement of women in Canadian university research”.

Return to footnote 57 referrer

Footnote 58

Ibid.

Return to footnote 58 referrer

Footnote 59

Ibid.

Return to footnote 59 referrer

Footnote 60

Ibid.

Return to footnote 60 referrer

Footnote 61

Ibid.

Return to footnote 61 referrer

Footnote 62

A co-publication is defined as a publication that was co-authored by at least two authors. When a publication involves authors from at least two different countries, it is defined as an international co-publication. The international co-publication rate of an entity (e.g., a group of researchers) is calculated as its number of international co-publications divided by its total number of papers.

Return to footnote 62 referrer

Footnote 63

The six examples in Figure 2 were selected for illustrative purposes to show the different types of trends observed, and should not be interpreted as a benchmarking of the performance of individual CERC units.

Return to footnote 63 referrer

Footnote 64

SSHRC/NSERC. (2013). CERC Logic Model, in CERC Evaluation Design Report.

Return to footnote 64 referrer

Footnote 65

CERC program outcomes are listed in Appendix B. In particular, outcomes IM 3 and INT 4 relate specifically to this topic.

Return to footnote 65 referrer

Footnote 66

NSERC. (2014) Evaluation of the Network of Centres of Excellence Program, page 35. Retrieved from. http://www.nserc-crsng.gc.ca/NSERC-CRSNG/Reports-Rapports/evaluations-evaluations_eng.asp. This definition is inspired by the one used in the last evaluation of the Networks of Centres of Excellence program and follows the categories defined and used in the administrative data review to categorize case study questionnaire data.

Return to footnote 66 referrer

Footnote 67

Funding for infrastructure-related costs if providing through university matching funds and/or the CFI programs.

Return to footnote 67 referrer

Footnote 68

Data from Phase 2 nomination forms, 7-year total.

Return to footnote 68 referrer

Footnote 69

Data from Progress Report 1 and 2 (2010-2013) submitted by host institutions to the Chairs Secretariat.

Return to footnote 69 referrer

Footnote 70

Matching funds from the host institution was not a requirement in the first competition round. For the second competition, the required matching funds will combine both the university contribution and contributions from other sources.

Return to footnote 70 referrer

Footnote 71

Additional information is regarding leveraged funds would be required from chairholders in order demonstrate an increase over time and to compute a ratio of CERC grant funds to leveraged funds.

Return to footnote 71 referrer

Footnote 72

Data on actual funding leveraged from other sources are based on the most recent annual data (i.e., 2012-13) provided in the progress report, complemented from the data from the case study questionnaire designed for the evaluation and administered in early 2014. Given lack of precision regarding financial data, actuals reported in the progress report are assumed to be for 2012-13 only. In addition, the progress reports seem to only capture financial leveraging (not in-kind contributions), although this is not clear based on the instructions.

Return to footnote 72 referrer

Footnote 73

CERC. (2014). Application and Nomination Process: 2012 Competition. Retrieved from: http://www.cerc.gc.ca/program-programme/cpan-pccs-eng.aspx

Return to footnote 73 referrer

Footnote 74

Data from Phase 2 nomination forms (7-year total), from Progress Report 1 and 2 (2010-2013) and from Statement of Accounts submitted by host institutions to the Chairs Secretariat.

Return to footnote 74 referrer

Footnote 75

CERC awards are non-renewable as per the terms and conditions of the program.

Return to footnote 75 referrer

Footnote 76

SSHRC. (2010). Performance Measurement Strategy (PMS) for the CERC Program

Return to footnote 76 referrer

Footnote 77

See Appendix B for the logic model narrative, which provides details on program outcomes. In the long-term, it is expected that the program will “foster research excellence and support the recognition of Canada as the home of pre-eminent researchers who are recognized internationally for their research breakthroughs and for the application of knowledge generated through the CERC units”.

Return to footnote 77 referrer

Footnote 78

Specific risks rated low likelihood and minor impact were related to “Universities’ ability to financially sustain the CERCs’ activities after the 7 years”, “Sustainability of the branding/global perception of Canada after the 7 years” and “Sustainability of the critical mass in an area of excellence of the 7 years (if different areas are targeted in next round)”. Only “Degree to which the overall system can accommodate increased demand on limited resources (e.g., incremental pressure on funding agencies)” was rated high likelihood and moderate impact.

Return to footnote 78 referrer

Footnote 79

CERC. (2008). Phase 1 Application Form.

Return to footnote 79 referrer

Footnote 80

Based on a review of other evaluations and reviews of S&T programs, there is no commonly accepted benchmark for this percentage and it is recognized that it will vary based on the characteristics of the program (e.g., delivery model, types/number of beneficiaries, maturity of the program). Previous evaluations of programs administered by the granting agencies have referred to 3% as “very low” (e.g., BL-NCE, see: http://www.nce-rce.gc.ca/_docs/reports/NCEReport-2013-RaportRCE_eng.pdf).

Return to footnote 80 referrer

Footnote 81

For example, the CERC program spends proportionally less (21%) than the NCE (35%) programs on direct salary expenses, but more (27%) than the CRC program (3%) on direct non-salary expenses. Also, the CERC program spends less indirect and direct non-attributable funding (53%) than the CRC program (71%) but more than NCE program (38%).

Return to footnote 81 referrer

Footnote 82

Steering Committee meeting minutes were examined as part of the document review.

Return to footnote 82 referrer

Footnote 83

A document outlining best practices for institutions on recruitment of world-class and high-calibre researchers, including expectations and advice is available at http://www.cerc.gc.ca/publications/recruitment-recrutement_e.pdf

Return to footnote 83 referrer

Footnote 84

CERC. (2014). Administer a Chair. Accessed from: http://www.cerc.gc.ca/program-programme/cpac-pcac-eng.aspx

Return to footnote 84 referrer

Footnote 85

Will not be addressing retention in the current evaluation but can collect baseline data

Return to footnote 85 referrer

Footnote 86

This suggestion should be reassessed in light of the results of the second competition, which is currently still in the nomination stage (phase 2), given the changes already made to provide more time for these processes.

Return to footnote 86 referrer

Footnote 87

Return to footnote 87 referrer

Footnote 88

CERC. (2014). Administer a Chair. Accessed from: http://www.cerc.gc.ca/program-programme/cpac-pcac-eng.aspx

Return to footnote 88 referrer

Evaluation of the Canada Excellence Research Chairs (CERC) Program Final Evaluation Report

Table of Contents

October, 2014

Table of Contents

Tables

Figures

Exhibits

List of Acronyms

Executive Summary

Program Overview

Evaluation Scope, Objectives and Methodology

Key Findings, Conclusions and Recommendations

1.0 Introduction

1.1 Background of the CERC Program

1.1.1 Profile

1.1.2 Governance

1.1.3 Key Beneficiaries

1.1.4 Resources

1.1.5 Expected Results

1.2 Evaluation Scope and Objectives

1.3 Evaluation Issues and Questions

2.0 Methodology

2.1 Methods

2.2 Limitations

3.0 Findings

3.1 Relevance

3.1.1 Continued need for the CERC program

3.1.2 Alignment with federal priorities, roles and responsibilities

3.2 Performance – Effectiveness

3.2.1 Attraction & retention of world-class researchers, high-calibre researchers and HQP

3.2.2 Awareness of CERC units, the CERC program and Canada

3.2.3 Partnerships in S&T Strategy research priority areas

3.2.4 Enhanced and sustainable research capacity of universities in S&T Strategy research priority areas

3.3 Performance – Program Design and Efficiency

3.3.1 Program design and efficiency

4.0 Conclusions and Recommendations

Appendix A – CERC Logic Model

Appendix B – CERC Logic Model Narrative

Appendix C – Data Collection Matrix

Appendix D – Detailed List of Suggested Improvements

Footnotes

Evaluation of the Canada Excellence Research Chairs (CERC) Program
Final Evaluation Report