Interim Evaluation of the Southern Ontario Prosperity Program

Final Report

Prepared for FedDev Ontario December 15, 2017

 

Table of Contents

• Executive Summary
• 1. Introduction
  • 1.1 Background
  • 1.2 Purpose of the Evaluation
  • 1.3 Structure of the Report
• 2. Evaluation Methodology
  • 2.1 Approach and Lines of Evidence
  • 2.2 Challenges and Mitigation Strategies
• 3. Southern Ontario Prosperity Program
  • 3.1 Number and Value of Projects Approved
  • 3.2 Overview of the Programs
• 4. Relevance
  • 4.1 Need for the Program
  • 4.2 Relationship to Other Programming
• 5. Effectiveness
  • 5.1 Current Status of the Projects
  • 5.2 Achievement of Project Objectives
  • 5.3 Outputs and Impacts of the Completed Projects
  • 5.4 Impacts of the Ongoing Projects
  • 5.5 Impact of the Projects on Beneficiaries
  • 5.6 On-going Impacts of the Projects
• 6. Program Design and Delivery
  • 6.1 Program Design and Delivery
  • 6.2 Cost-Effectiveness
• 7. Conclusions and Recommendations
  • 7.1 Conclusions
  • 7.2 Recommendations

Management Response and Action Plan

Annexes

• Annex I: Evaluation Methodology
  • I.1 Approach and Lines of Evidence
  • I.2 Evaluation Matrix
  • I.3 Challenges and Mitigation Strategies
• Annex II: The SOPP Program and Logic Model
  • II.1 Program Activity Architecture
  • II.2 Overview of the SOPP Programs
  • II.3 Program Logic Model

List of Tables and Figures

• Table 1: Evaluation of Issues and Questions
• Table 2: Number, Type and Value of Projects Approved by Program As of May 24, 2017
• Table 3: Overview of the Programs Included in the SOPP
• Table 4: Areas of Comparative Strength for Southern Ontario
• Table 5: Focus of the SOPP Programs
• Table 6: Summary of Funded Projects by Primary Focus or Objective
• Table 7: Other Programs That Share Similar Objectives to the SOPP Programs
• Table 8: Impact of FedDev Ontario Funding on the Project
• Table 9: Achievement of the Project Objectives
• Table 10: Average Rating by Program Regarding Achievement of Objectives
• Table 11: Number of Completed Projects Reporting Types of Impacts
• Table 12: Number of Projects Reporting Selected Outputs and Impacts
• Table 13: Direct Impacts of the Activities on the Beneficiary Organizations
• Table 14: Level of Agreement With Statements Regarding Program Delivery and Design – Proponents and Unfunded Applicants
• Table 15: Average Value of Projects Supported Under the Current and Previous Mandates
• Table 16: Value of Funding Approved by Year and Duration (as of May 24, 2017)
• Table 17: Average Funding Contributed From Other Sources For Every Dollar Provided By FedDev Ontario
• Table 18: Average Funding Contributed From Other Sources By Program For Every Dollar Provided By FedDev Ontario
• Table 19: Return on Contributions from FedDev Ontario to Projects Completed to Date
• Table 20: Planned and Actual G&C Expenditures Per Year and Program, 2014-15 to 2016-17
• Table 21: Planned and Actual Operating Expenditures per Year and Program, 2014-15 to 2016-17
• Table 22: Operating Expenditures as a Percent of Contribution Expenditures by Program, 2014-15 to 2016-17
• Table 23: Overview of the Hybrid Approach
• Table 24: Number of Representatives Participating in the Surveys, Key Informant Interviews and Case Studies
• Table 25: Summary of Recommended Issues, Performance Indicators and Data Sources
• Table 26: FedDev Ontario Program Alignment Architecture, 2017-18
• Figure 1: Overview of the Study Methodology
• Figure 2: Logic Model for Southern Ontario Prosperity Initiatives, Advanced Manufacturing Fund and Eastern Ontario Development Program, 2014-19

Acronyms

AAFC

Agriculture and Agri-Food Canada

AIME

Achieving Innovation and Manufacturing Excellence Program

AMF

Advanced Manufacturing Fund

APMF

Agency Performance Measurement Framework

ARC

Applied Research and Commercialization

BCA

Building a Competitive Advantage

BDC

Business Development Bank of Canada

BERD

Business Expenditures in Research and Development

CEDP

Collaborative Economic Development Projects

CFDCs

Community Futures Development Corporation

CFI

Canadian Foundation for Innovation

CME

Canadian Manufacturers & Exporters

CIHR

Canadian Institute of Health Research

CSSCTP

College Strategic Sector/Cluster/Technology Platform

EDC

Export Development Canada

EODP

Eastern Ontario Development Program

FedDev Ontario

Federal Economic Development Agency for Southern Ontario

FBI

Food & Beverage Initiative

FDI

Foreign Direct Investment

FTEs

Full-time Equivalents

GAC

Global Affairs Canada

GDP

Gross Domestic Product

GEI

Graduate Enterprise Internship

G&C

Grant and Contribution

IBGP

Investing in Business Growth and Productivity

IBI

Investing in Business Innovation

ICP

Investing in Commercializing Partnerships

ICT

Information and Communications Technology

IRAP

Industrial Research Assistance Program

IRCC

Immigration, Refugees and Citizenship Canada

IRD

Investing in Regional Diversification

ISED

Innovation, Science and Economic Development Canada

IRID

Investing in Regional Innovation and Development

JPC

Jobs and Prosperity Council

MEDG

Ministry of Economic Development and Growth

MNEs

Multinational Enterprises

MRIS

Ministry of Research, Innovation and Science

NFP

Not-for-Profit

NRCan

Natural Resources Canada

NSERC

National Sciences and Engineering Research Council

OBIO

Ontario Bioscience Innovation Organization

OCC

Ontario Chamber of Commerce

OCE

Ontario Centres of Excellence

OMAFRA

Ontario Ministry of Agriculture, Food and Rural Affairs

ONE

Ontario Network of Entrepreneurs

PAA

Program Alignment Architecture

PAC

Project Advisory Committee

PE

Productivity Enhancement

PI

Prosperity Initiative

PLM

Program Logic Model

PSI

Post-secondary institution

R&D

Research and Development

RDAs

Regional Development Agencies

SDTC

Sustainable Development Technology Canada

SME

Small and Medium Enterprises

SR&ED

Scientific Research and Experimental Development Tax

SEB

Scientists and Engineers in Business

SODF

Southwestern Ontario Development Fund

SODP

Southern Ontario Development Program

SOPP Program

Southern Ontario Prosperity Program

SOPIs

Southern Ontario Prosperity Initiatives

SOSCIP

Southern Ontario Smart Computing Innovation Platform

SOWC

Southern Ontario Water Consortium

SSHRC

Social Sciences and Humanities Research Council

TDP

Technology Development

UOIT

University of Ontario Institute of Technology

YLF

Yves Landry Foundation

Y-STEM

Youth in Science, Technology, Engineers and Mathematics

Interim Evaluation of the Southern Ontario Prosperity Program (SOPP) – Infographic

PDF version

8,837 KB, 1 page

Findings

Relevence

Government of Canada priorities

• Innovation & Skills plan
• Compliments other Government of Canada programs

Impact

• $2.45 leveraged for every $1 invested
  • 10,000 FTEs created
• $2.54 increase in revenue for every $1 invested
  • 2,900 business supported
• $2.38 in angel investment for every $1 invested

Expected Future Impact (including on-going projects)

• 33,000 FTEs created or maintained
• $4 B in North American sale
• $600 M in exports
• $150 M in angel investment
• Commercialization of 1,500 new products and technologies

Scope

• Eastern Ontario Development Program (EODP)
• Advanced Manufacturing Fund (AMF)
• Investing in Business Innovation (IBI)
• Investing in Business Growth and Productivity (IBGP)
• Investing in Commercialization Partnerships (ICP)
• Investing in Regional Development (IRD)

• 190 projects approved
• Up to $688 M invested
• 66% of projects still in progress as of May 24, 2017

Methodology

Secondary Lines of Evidence

• Document & literature review;
• Review of program & operational data

Surveys

61% response rate.

• 117 approved projects
• 40 non-approved projects
• 28 potential applicants
• 356 beneficiarie

Case Studies

• 6 completed

Interviews

• 64 completed

Going Forward

• Move to longer term ongoing funding mode
• Maintain fundamental program structure; continue to align with Government of Canada priorities
• Streamline application process
• Continue development of program officers
• Review and revise project reporting system

Executive Summary

Purpose and Method of Study

In August 2009, the Government of Canada created the Federal Economic Development Agency for Southern Ontario (FedDev Ontario) with a mandate to strengthen southern Ontario's economic capacity for innovation, entrepreneurship and collaboration, and promote the development of a strong and diversified southern Ontario economy. The Southern Ontario Prosperity Program (SOPP) served as the Agency's core program for its second five-year mandate (2014–15 to 2018–19). The SOPP consists of the Eastern Ontario Development Program (EODP), the Advanced Manufacturing Fund (AMF) and four programs under the Southern Ontario Prosperity Initiatives (SOPIs): the Investing in Business Innovation (IBI) initiative; the Investing in Business Growth and Productivity (IBGP) initiative; the Investing in Commercialization Partnerships (ICP) initiative; and the Investing in Regional Diversification (IRD) initiative. As of May 24, 2017, 190 projects had been approved under these programs, with approved funding totaling $688 million.

The purpose of this study was to conduct an interim evaluation of the SOPP, with a focus on its relevance, effectiveness, and design and delivery. This evaluation used a hybrid team approach (involving internal evaluators from FedDev Ontario and external consultants from Goss Gilroy Inc.) in implementing a mixed-methods research design involving multiple lines of evidence. Key lines of evidence included document and literature reviews (focused primarily on program relevance); project and program financial data; surveys of 117 project proponents, 40 applicants not approved for funding, 28 representatives from organizations eligible to apply for funding but which did not, and 365 beneficiary organizations that received financial or other assistance funded by FedDev Ontario but delivered by a third-party organization; interviews with 64 key informants; and case studies of six projects.

This is an interim evaluation. Data will not be available on the full impacts of the projects for several years. Of the 190 projects, only 34 percent were completed at the time of the evaluation and the completed projects are expected to continue generating impacts that extend beyond the term of their funding. Recognizing that only limited data is available to date on project outcomes, the evaluation also incorporated an extensive review of projected results to illustrate the impacts expected to be generated, reviews of progress reports, case studies of a sample of on-going projects, and a review of three recent FedDev Ontario studies which examined the longer-term impacts of a sample of past projects funded by FedDev Ontario.

Relevance

There is a continued need for the programs included in the SOPP. Key informants attribute the strong need to the importance of the Ontario economy, the significant opportunities for further growth and diversification that exist across a range of existing and emerging clusters, and the key challenges and constraints that are slowing this growth and diversification and need to be addressed. Project proponents, unfunded applicants, non-applicants and beneficiaries rated the need as high, noting how the programs help address challenges to economic development such as the need for capital and assistance for issues such as technology development and adoption, testing and commercialization, market development, staff development and training, business start-up and early stage development.

The SOPP programs are well aligned with each other and other programming available in southern Ontario, the constraints to development and needs of the key target groups. Taken together, the suite of SOPP programs employs a variety of delivery mechanisms to promote growth and diversification across various stages of businesses development, economic clusters, and regions within southern Ontario. Factors such as the place-based nature of FedDev Ontario^{Footnote 1}, the strong demand for funding, and coordination between FedDev Ontario and other programming organizations help to ensure that the SOPP programs complement rather than duplicate other federal or provincial government programs that promote innovation, business development and community development. The SOPP also supports the framework for the Innovation and Skills Plan (ISP) as well as Investing in Regional Innovation and Development (IIRD)^{Footnote 2}.

Effectiveness of the Programs

FedDev Ontario has made significant investments that are incremental and leveraged funding from other sources. FedDev Ontario funding fills a need that would not have been met by other programs. Only 4 percent of the projects would have proceeded as planned in the absence of FedDev Ontario funding. Each project dollar contributed by FedDev Ontario was leveraged with $2.45 in funding from other sources. The funding has targeted a range of existing and emerging economic clusters including manufacturing (associated with 60 percent of approved FedDev Ontario contributions, of which 37 percent could be categorized as advanced manufacturing), health care and biotech (26 percent), Information and Communications Technology (ICT) (13 percent), agri-food (8 percent) and clean tech and clean resources (7 percent).

The projects have contributed towards achievement of the priorities of the Government of Canada, including the Innovation and Skills Plan, as well as the mandate of FedDev Ontario. The investments have helped to:

• Attract, develop and retain highly skilled workers, researchers and entrepreneurs;
• Strengthen the regional innovation ecosystem through the further development of research and development capacity investments in technology development, testing and commercialization;
• Facilitate the development of collaborations and partnerships;
• Accelerate the start-up, early development, expansion and modernization of companies by attracting and facilitating investment;
• Support technology adaptation, adoption and commercialization;
• Support advisory services and market development activities; and
• Attract anchor^{Footnote 3} firms; and support community economic development and diversification.

The activities supported under SOPP have directly contributed to the Agency's core mandate of strengthening southern Ontario's economic capacity for innovation, entrepreneurship and collaboration and promoting development of a strong and diversified southern Ontario economy. The 65 projects which have been completed to date report increased sales of $241 million ($2.54 in increased revenues for every dollar provided by FedDev Ontario) while the 125 projects that are still ongoing are projected to generate $3.7 billion in increased North American sales, almost $600 million in export sales outside North America and nearly $1.1 billion in sales from the commercialization of 1,500 new products and technologies.

The impacts of the projects will continue to grow over time. Most projects resulted in improvements to operations, facilities, equipment and business practices or further development of the innovation capacity, the benefits of which will continue on well beyond the end of the project.

Program Design and Delivery

Most project proponents are satisfied with the design and delivery of the programs. Proponents report satisfaction in their dealings with FedDev Ontario staff and believe the design and delivery of the programs is appropriate, they were given sufficient time to complete the project, they received clear direction regarding the development of their proposal, and the application requirements and criteria are appropriate. The most significant concerns are the length of the application and approval process, the unpredictability of the timing of these steps, the reporting requirements, and usefulness of the performance measurement data. Given they invested in the process but were not successful in accessing funding, non-funded applicants tended to be less supportive of program design and delivery, particularly the length of the approval process, their dealings with FedDev Ontario staff, and the guidance and direction provided to them with respect to the preparation of their proposal.

Operating costs as a percent of grants and contributions are low relative to historical figures for the programs. Operating expenditures averaged 4.9 percent of program expenditures to March 31, 2017. Two factors contributing to the low percentage are an increase in average approved contributions per project and increased use of third parties to administer programs. Some reservations, from within the organization and externally, were expressed that the programs' operating budgets may have become too lean, particularly given the large number of legacy files that still require monitoring. Staffing constraints can slow the processing of applications; contribute to data challenges; increase staff turnover; and impact project monitoring.

The most serious issue regarding program design involves the five-year funding profile of FedDev Ontario. Three quarters of the funding (to May 24, 2017) was approved for projects of more than three years in duration. The five-year timeline means that most projects needed to be approved early in the mandate and little funding remained available for approvals in the later years. The five-year mandate also creates significant workload issues for program staff, particularly in the first year of the new mandate. During the first year, program staff are engaging with prospective applicants, and reviewing and processing applications with a particular focus on the larger scale, multi-year projects which tend to be more complex in nature. At the same time, the staff must also deal with legacy^{Footnote 4} projects from the previous mandate, reviewing final reports, conducting site visits, validating reported results, and preparing close-out reports. These challenges are further complicated by having program budgets divided relatively evenly across the five-year period. Reflecting the time required to approve, contract and launch new projects, actual program expenditures amounted to only 38 percent of the planned expenditures in 2014-15.

Recommendations

The recommendations arising from the evaluation are as follows:

1. Develop a formal plan for addressing the issues related to the five-year funding profile. The preference is to move to a longer-term funding model for the Agency or, if that cannot be achieved, take steps to mitigate some of the impacts. Mitigating measures could include (1) having additional trained staff resources in place for the first year of the new mandate so that the Agency is better able to process and approve new projects while still being able to effectively monitor and close-out legacy projects; and (2) allocating the program budgets so that planned expenditures are lower in year one than in subsequent years. Another option for subsequent years may be to move to a rolling funding model where FedDev Ontario secures funding for additional years part way through its next mandate, such that the program always has three to five years of funding remaining.
2. Maintain the same fundamental program structure for the next mandate, while exploring opportunities to refine and consolidate programs to address the current challenges and needs of the region. There is strong support within and outside the organization for maintaining the existing program structure given that the existing suite of programs is effectively designed, coordinated and delivered and doing so will enable the Agency to build off program awareness created over the past four years. Individual programs may need to be adjusted or adapted somewhat to reflect changing priorities of the federal government, the role of FedDev Ontario within ISED, and issues identified in the evaluation.
3. Offer potential applicants a single point of entry and regularly update publicly available information related to funding availability and timelines. It can be difficult for potential applicants to determine under which, if any programs, they may be eligible. An online form could be used to assess eligibility and guide prospective applicants to the appropriate program. Applicants also requested that additional information be publicly available on service standards, the balance of funding available for project approvals, the success rate of applications, and the range in timelines to decision.
4. Support the continued development of project officers. Program results can be directly impacted by the experience, knowledge and expertise of the project officers. The level of turnover has been high. A strategy should be developed to both reduce the level of turnover in the positions and accelerate the professional development of project officers through mechanisms such as the Professional Practice pilot. Consideration could also be given to augmenting internal resources by contracting with one or more outside agencies to assist in the review of certain aspects of project applications.

5. The project reporting system should be reviewed and revised, both in terms of the reporting process and the indicators on which proponents report. Consideration should be given to fully digitizing project files from cradle to grave (from expressions of interest to submission and review of proposals, implementation of projects, and project monitoring). The existing system of scanned documents and multiple excel files is cumbersome for applicants, proponents, officers, evaluators, and decision-makers; requires regular reentering of data; complicates validation; and restricts the ability to report on projects, proponents, and progress at the program and Agency level.

   The performance indicators should be refined to reflect the new departmental results framework and facilitate useful reporting on a broader range of results relevant to specific projects. An online system could enable the proponent to more easily report on key departmental results as well as indicators specifically relevant to their project (using skips patterns to adapt the indicators by project), which could then be validated (e.g. using automated procedures and personal follow-up where needed). The data could be rolled up easily to report on results by program, region, or sector or whatever parameter is of interest to FedDev Ontario managers on an on-going basis or in response to specific requests.

1. Introduction

1.1. Background

The Government of Canada created the Federal Economic Development Agency for Southern Ontario (FedDev Ontario) in 2009 with a mandate to strengthen southern Ontario's economic capacity for innovation, entrepreneurship and collaboration; and promote the development of a strong and diversified southern Ontario economy. The Southern Ontario Prosperity Program (SOPP), which served as the Agency's core program for its second five-year mandate (2014–15 to 2018–19), consists of the Eastern Ontario Development Program (EODP), the Advanced Manufacturing Fund (AMF) and the four Southern Ontario Prosperity Initiatives (SOPIs): Investing in Business Innovation (IBI); Investing in Business Growth and Productivity (IBGP); Investing in Commercialization Partnerships (ICP); and Investing in Regional Diversification (IRD).

1.2. Purpose of the Evaluation

The objective is to conduct an interim evaluation of the SOPP. Under the 2016 Policy on Results, evaluations are to be planned with consideration of using relevance, effectiveness and efficiency (design and delivery) as primary evaluation issues, where relevant to the goals of the evaluation (Directive on Results, C.2.2.1.5). For this evaluation, a series of evaluation questions were developed and grouped under those three issues.

This evaluation will enable FedDev Ontario to meet with the requirements of the Policy on Results, Transfer Payment Policy, and Section 42.1of the Financial Administration Act as well as contribute to renewal of the Agency mandate.

1.3 Structure of the Report

Chapter 2 summarizes the evaluation methodology. Chapter 3 provides a description of the programs involved in the SOPP and the projects which were funded. Chapters 4, 5 and 6 summarize the findings of the evaluation regarding relevance, effectiveness, and program design and delivery. Chapter 7 presents the conclusions and recommendations arising from the interim evaluation.

2. Evaluation Methodology^{Footnote 5}

2.1 Approach and Lines of Evidence

The evaluation was undertaken in three phases: planning, data collection involving various lines of evidence, and analysis and reporting (draft and final report). The planning phase involved a documentation review (on FedDev Ontario, the programs, and funded projects to identify the data available) and development of the evaluation matrix, methodology, data collection instruments and communication protocols. The data collection phase employed a hybrid team approach (involving internal evaluators from FedDev Ontario and external consultants from Goss Gilroy Inc.) in implementing a mixed-methods research design involving multiple lines of evidence.

As indicated above, the lines of evidence included a document and literature review (focused primarily on issues related to relevance); a review of project and program financial data; surveys of 117 proponents, 40 applicants not approved for funding, 28 representatives from organizations eligible to apply for funding but did not, and 365 beneficiary organizations that received financial or other assistance from third-parties which received funding from FedDev Ontario; interviews with 64 key informants; and case studies of six projects. The results were then analyzed to prepare the draft and final reports.

2.2 Challenges and Mitigation Strategies

The major challenge associated with this interim evaluation is that not enough time has elapsed for the impacts of the projects to be fully realized. Of the 190 projects, only 34 percent were completed at the time of the evaluation, and most completed projects will require additional time post-completion to realize their full impacts. To augment the limited data available to date on project outcomes, the evaluation team conducted an extensive review of the projected results to illustrate the impacts expected to be generated, conducted a review of progress reports, and focused the case studies primarily on on-going projects. The evaluation team also reviewed the results of three recent FedDev Ontario studies, which involved examining the longer-term impacts of some past projects funded by FedDev Ontario.^{Footnote 6}

The potential non-response error was addressed by achieving significantly high response rates for the surveys, while the potential for respondent bias was addressed by including a survey of organizations that applied but did not receive FedDev Ontario funding, a survey of potential applicants, and interviews with key informants not directly involved in the programming, as well as triangulating the results with data obtained through other lines of evidence. Given these mitigation measures, the evaluation team is of the opinion that the limitations of the study were adequately addressed and the results of the evaluation are deemed to be reliable and valid.

3. The Southern Ontario Prosperity Program

3.1 Number and Value of Projects Approved

As of May 24, 2017, 190 projects (totaling $688 million in funding) were approved under the SOPP programs. These included 113 private sector projects and 77 not-for-profit (NFP) projects undertaken by other organizations (not-for-profit organizations, post-secondary institutions and other government organizations).

3.2 Overview of the Programs^{Footnote 7}

Characteristics

The characteristics of the each of the six programs included in the SOPP are summarized in the table on the following page and in the points below:

• Investing in Business Innovation (IBI) provides support for mentorship, entrepreneurial support and financing to help new and early staged businesses grow and succeed. The objectives are to foster a culture of entrepreneurship focused on innovation by supporting start-ups to transform ideas into globally competitive products and services; increasing, stimulating and leveraging private sector investment; strengthening angel networks; and supporting mentorship and skills development activities to help start-ups grow and succeed. Of the 88 IBI projects, 56 supported SMEs, 22 supported angel investor networks and 10 supported not-for profit organizations which, in turn, provided support for skills development and seed funding to entrepreneurs and SMEs.
• Investing in Commercialization Partnerships (ICP) supports business-led partnerships with a focus on developing globally-competitive products and services. ICP works to increase collaboration among businesses, post-secondary institutions (PSIs) and research organizations to narrow the gap between innovation and commercialization, increase the capacity of existing and emerging innovation ecosystems, and promote the development of competitive economic clusters in southern Ontario. All ICP 12 projects involved not-for-profit organizations (including 7 PSIs). The projects support five of the six priority areas announced in the Innovation and Skills Plan in Budget 2017^{Footnote 8}.
• Investing in Business Growth and Productivity (IBGP) focuses on established southern Ontario businesses that have the potential to be global players. The IBGP helps businesses diversify markets and expand facilities, adopt new technologies and processes to improve productivity, and increase business capacity to grow and diversify markets. The objective is to position southern Ontario businesses to be more competitive globally by assisting established businesses with high growth potential; increasing investment in technologies and processes to improve productivity; and increasing the capacity of businesses to participate in global markets through exports and integration in global value chains. Of the 51 projects approved, 49 directly supported SMEs while two supported not-for-profit organizations which, in turn, assisted SMEs with productivity improvements or increased participation in global markets.

• Investing in Regional Diversification (IRD) provides unique regional assets and local expertise to attract new investment and opportunities and support the long-term development of stronger, more diverse economies in southern Ontario communities.
• Established as part of the 2013 Federal Budget, the Advanced Manufacturing Fund (AMF) supports research and innovation organizations, the private sector, PSIs and not-for-profit organizations to work together to accelerate development of large-scale, advanced technologies that will result in new market opportunities for Ontario businesses in manufacturing sectors. The objective is to increase firm productivity and enhance the competitiveness of Ontario's advanced manufacturers by addressing, within the Ontario delivery context, gaps in federal supports for advanced manufacturers; attracting projects that advance the development and/or adoption of cutting-edge technologies leading to product, process, and technological innovation; creating spillovers for manufacturing clusters and/or supply chains; and fostering collaboration between research institutes, post-secondary institutions and the private sector.
• The Eastern Ontario Development Program (EODP) was established in 2004 and has been administered by FedDev Ontario since the Agency was established in 2009. It is an economic development initiative aimed at addressing economic challenges in eastern Ontario and taking advantage of innovative opportunities in the region. The program is delivered through eastern Ontario's 15 CFDCs and promotes business development, job creation and strengthened economies in rural eastern Ontario communities. In addition, the EODP provides funding for Collaborative Economic Development Projects (CEDP) which generate benefits for multiple communities and promote broad-based collaborative economic development.

Governance

With the exception of the AMF, the SOPP programs are administered solely by FedDev Ontario. The Agency is responsible for program design, development, and promotion, review of applications, funding decisions, development and approval of contribution agreements, management of the funding agreements, project monitoring, and assessment of program outcomes.

The AMF is delivered under an MOU with Industry Canada (now ISED), which governs the review of applications. Under the MOU, FedDev Ontario retains responsibility for project approval but obtains input from ISED regarding the technical aspects (innovation), market relevance and potential spillover benefits of the proposed project. At times, ISED contracts with a private sector contractor to assist in that review. The AMF is the only SOPP program which also serves northern Ontario (one of the eight AMF projects, involving the largest contribution made to any SOPP project, was located in northern Ontario) and ISED provides support for program delivery in that region.

The contributions provided under each program are governed by contribution agreements made directly with businesses, not-for-profit organizations (including post-secondary institutions) that work with a collaborator or group of collaborators to implement the project, and third-party organizations that in turn use that funding to deliver support to businesses. The contribution agreement outlines the recipient's contractual obligation to provide information required for performance measurement and evaluation requirements.

4.0 Relevance

4.1 Need for the Program

Project proponents, unfunded applicants, non-applicants and beneficiaries each identified a continued need for the programming included in the SOPP. When asked to rate the extent to which there continues to be a need for these programs (on a scale of 1 to 5 where 1 is not at all and 5 is to a great extent), the average rating varied from 4.3 amongst non-applicants and 4.4 amongst unfunded applicants to 4.8 amongst beneficiaries and 4.9 amongst project proponents. Only 6 of the 550 people surveyed gave a rating of less than somewhat of a need; over three-quarters indicated that there was a great need. The average ratings given by the proponent and beneficiary groups were consistently high across the programs, ranging from 4.7 to 5.0.

In their responses, project proponents, unfunded applicants, potential applicants, and beneficiaries focused primarily on the need for assistance to help businesses deal with specific development constraints, highlighting the need for capital, funding to support further development or expansion of their operations, and assistance for particular issues such as technology development and commercialization, market development, staff development and training, and business start-up and early stage development. Those few respondents who provided ratings of 3 or less noted, generally, that the programming did not meet their needs (e.g., they required more funding than was available, the program did not move at the speed of business in terms of the timeline from application to the receipt of funding, low cost capital is already readily available from commercial sources and what was needed was a grant, or repayable grants programs would be better delivered by the private sector, such as a banks, rather than through government).

The demand for project funding also far exceeded the available funding. In total FedDev Ontario and its third party delivery agents received 4,215 applications for all SOPP programs between 2014-15 and 2017-18. 190 projects received funding.

Although not asked to rate the need for the programs on the same scale of 1 to 5, the key informants who were interviewed also expressed a strong need for the programs. In discussing the rationale for the programming, the key informants highlight various factors: the importance of the Ontario economy; the significant opportunities for further growth and diversification; the challenges or constraints that are slowing this growth and diversification; and the alignment between the SOPP suite of programs and those constraints; and alignment with the priorities of the Government of Canada^{Footnote 10} and the mandate of FedDev Ontario. Each of these factors was confirmed in the results of the document and literature review, as further described below:

1. As the largest regional economy in Canada, the health of the southern Ontario economy has a major impact on the overall health of the Canadian economy particularly in terms of manufacturing. According to Statistics Canada data, the region served by FedDev Ontario accounts for 39 percent of the Canadian population, 37 percent of its GDP, 41 percent of exports and 44 percent of business expenditures in research and development (BERD). Manufacturing is a key economic sector for Ontario, contributing nearly 13 percent of provincial GDP, 11 percent of employment (745,000 jobs), and over $170 billion in domestic exports for Ontario in 2015 (representing 86 percent of Ontario's domestic goods exports).

2. There are significant opportunities for further economic growth and diversification in southern Ontario. The Ontario economy encompasses a wide range of existing economic clusters^{Footnote 11} (e.g., manufacturing, finance, and ICT) and emerging clusters in areas such as health (e.g., biotech and regenerative medicine), bio-processing, agricultural technology and biotechnology, fintech, cybersecurity, big data, Internet of Things (IoT)^{Footnote 12}, networking, and quantum computing) where the province holds comparative advantages and the potential for further growth is significant. Further development of these clusters will generate spillover economic benefits for other economic sectors and benefits for society overall in areas such as environment, security, health, evidence-based policy making and communications.^{Footnote 13}

   The rate of development within existing and emerging economic clusters is best viewed as a function of multiple factors that create the conditions for growth such as access to capital, highly qualified personnel, entrepreneurs, markets, infrastructure and other key inputs, as well as capabilities related to research, development and commercialization. Governments and others work to accelerate the rate of development by influencing the factors that drive development. Varying somewhat from cluster to cluster, some of the key areas where southern Ontario holds comparative strengths^{Footnote 14} and how FedDev Ontario programs support those strengths are summarized in table 4 below.

3. However, the region also faces various challenges or constraints which can serve to slow the rate of development with the existing and emerging clusters^{Footnote 15}. As highlighted in the document and literature review, as well as in interviews and surveys, the region tends to lag other advanced economies in terms of labour productivity which, in turn, has been attributed to factors such as lower rates of business investment in R&D, technology adoption, and investment in new machinery and equipment. Some of the other issues that were highlighted in the research included:

   • While access to risk capital can be increasing, the level of investment remains lower than other competing regions (particularly the United States).
   • There are challenges associated with technology transfer and spinoffs. The high level of public sector investment in R&D has not resulted in high levels of technology commercialization.
   • While education levels in southern Ontario are high, there is a mismatch between the skills and experience required by companies (particularly in emerging sectors) and the available workers. The Conference Board of Canada has estimated that skills issues (including mismatches between education and market requirements) cost the Ontario economy over $24 billion per year.^{Footnote 16}
   • The rate of technological change in the economy is accelerating, which holds significant implications for businesses in terms of evolving skills requirements, the need to adapt business models, and the need to make further investments. In particular, developments such as the emergence of new platforms, systems and technologies, automation, the Internet of Things, 3D printing, big data analytics, introduction of composites and lightweight materials, and Industry 4.0 are having a major impact on traditional manufacturing.
   • Most SMEs are not pursuing global markets. For example, only 7 percent of Ontario SMEs are engaged in export activities.^{Footnote 17}

   The region is also being impacted by uncertainty regarding the economic outlook. While the declining Canadian dollar benefited exports (particularly in the manufacturing sector), it also made some key imports (such as machinery and equipment) more expensive which has slowed business investment. Going forward, uncertainty associated with the dollar, the ongoing NAFTA renegotiations, the global economic outlook and the potential for interest rate hikes may negatively impact the investment climate.

4. SOPP programs are well-coordinated with each other, employing a variety of delivery mechanisms to address these challenges and constraints and promote development across various stages of businesses' development, economic clusters, and regions within southern Ontario.

   Table 5 summarizes the relative focus of each of the programs in terms of the delivery mechanisms, development issues or challenges on which they focus as well as stages of business development, economic clusters and regions.

A description of the relative focus of the programs is provided below.

Delivery Mechanisms

FedDev Ontario employs various delivery mechanisms, including providing repayable funding directly to businesses, non-repayable funding for cluster or capacity development projects undertaken by post-secondary institutions or other not-for-profit organizations (sometimes in partnership with private sector organizations or others), and non-repayable contributions to third-party organizations (e.g., associations, research organizations, post-secondary institutions, CFDCs, or other not for-profit organizations) that, in turn, provide services and other support to business clients and others.

Stages of Business Development

The programs provide support for start-up and early stage companies, expanding and modernizing existing businesses, and increasing the involvement of MNEs in the further development of clusters in Ontario. To facilitate start-up and early stage development, IBI provides funding to SMEs to be matched with venture capital or funding from angel investors; to strengthen angel investor networks; and to not-for-profit organizations to facilitate skill development and seed financing for new entrepreneurs. ICP facilitates development, testing and commercialization of new technologies by bringing together businesses, post-secondary institutions and research organizations. IRD funds some projects that, in turn, made investments in early-stage companies.

Enhancing Ontario's productivity and growth requires increased investment in productivity enhancing advanced technologies and innovation. The IBGP and AMF have supported investments in the development and modernization of production capabilities and facilities as well as the adaptation or adoption of new technologies, materials or processes. Other IBGP and IRD projects have supported business growth through activities in areas such as market development and provision of advisory services.

Both AMF and ICP facilitated large-scale investments, including foreign direct investment in projects involving MNEs. For example, GE Healthcare indicated that, in the absence of the Centre for Commercialization of Regenerative Medicine and the funding provided by FedDev Ontario (an AMF project), they would have made their investment in regenerative medicine in another jurisdiction. Similarly, a significant investment by IBM into southern Ontario may not have been made in the absence of the SOSCIP project^{Footnote 19} and, to a lesser extent, the Southern Ontario Water Consortium (SOWC) project, both funded initially under the Technology Development Program (TDP) under Mandate 1 and continuing IBM's involvement under ICP. Project funding for Hanwha L&C Inc. of South Korea is another example of foreign direct investment attraction. Hanwha received $15 million from AMF to establish London, Ontario as Hanwha's North American headquarters, paving the way for future foreign direct investment in the region.

Development Issues

The programs have targeted a wide range of economic drivers including access to capital (e.g., increasing the supply of angel investment), expansion or modernization of production capabilities, linkages and networking between groups (e.g., the ICP has brought together representatives from business, post-secondary institutions and research organizations to narrow the gap between innovation and commercialization), technology development, testing and commercialization (e.g., through ICP, IRD and AMP projects), entrepreneurial and staff development, provision of advisory services through intermediary organizations funded by FedDev Ontario, market development and investment attraction.

As part of the evaluation, descriptions of each of the 190 projects were reviewed by the evaluation team to code the projects by primary focus or objective. Of the 190, 61 projects supported the development, expansion or modernization of manufacturing facilities and accounted for almost one-half (49 percent) of the approved funding. Twelve projects provided advisory services or support to businesses (e.g., services or support related to incubation and acceleration, business or technology assessment, business training, technology or business development support, or capital access advisory services).

The survey of beneficiaries included 150 businesses, 48 community projects and 60 CEDP projects funded under the EODP; 57 businesses that received assistance through the Achieving Innovation and Manufacturing Excellence (AIME) Global Program offered by the Yves Landry Foundation (funded through IBGP); and 50 businesses that were assisted though the CME SMART Advanced Technologies for Global Growth (ATGG) program (also funded through the IBGP). An overview of the objectives of their involvement in the program is provided below:

• The EODP business projects focused on a range of objectives including expansion of business operations, training, sales and marketing, and development of new products, processes and technologies.
• The focus of the EODP community innovation projects also ranged widely from development initiatives, strategic plans and feasibility studies to local infrastructure development, and support for business development activities.
• The beneficiaries involved in the CEDP were most commonly involved in activities related to mentorship, provision of business advice, and the development of technical and soft skills.
• Participants in the CME SMART ATGG program received advanced technology assessments or support for advanced technology adaptation or adoption projects.
• Participants in the Yves Landry program received training related to the development or adaption of new technology, processes or procedures within the company.

Existing and Emerging Clusters

The evaluation team also reviewed the 190 projects to identify the clusters or sectors on which they focus. The SOPP projects involve a range of existing and emerging economic clusters, with the manufacturing sector being the most significant. Ontario is the primary centre for manufacturing in Canada, accounting for 46 percent of Canada's manufacturing GDP and 52 percent of its manufacturing exports. While conditions have been improving, manufacturing employment declined by about one-quarter between 2005 and 2015. Of the 190 projects, 77 related to the manufacturing sector (mostly funded through the AMF and IBGP with some funding also provided through the ICP and IBI). These 77 projects accounted for 41 percent of the projects approved and 60 percent of the total approved value of FedDev Ontario contributions (of which approximately 37 percent could be categorized as advanced manufacturing).

Other leading sectors or clusters include health care and biotech (27 projects accounting for about 26 percent of approved funding), ICT (44 projects accounting for about 13 percent of approved funding), agri-food (18 projects accounting for about 8 percent of funding) and clean tech and clean resources (13 projects accounting for about 7 percent of funding).

Region

With the exception of the EODP (which is targeted specifically at communities in eastern Ontario) and the AMF (which could fund projects from all parts of Ontario)^{Footnote 21}, the programs are open to applicants from across southern Ontario. Uptake of the programs varies by region, depending largely on where industry is based. Overall, the leading regions include Toronto ($158 million was approved for projects based in Toronto, representing 23 percent of the total) and neighboring regions (a further $148 million, 22 percent, was approved for projects in census divisions adjacent to Toronto, mostly Peel and York).

Eastern Ontario was identified as a priority for funding because of weak economic conditions which contribute to a loss of businesses, investment and youth from the region. For example, as of May 2017, the employment and participation rates in eastern Ontario are lower than in the rest of the southern Ontario (54 percent v. 61 percent and 58 percent v. 66 percent respectively as of 2017) while, in 2014, employment incomes averaged 13 percent below those in southern Ontario. Of the 190 projects, 31 targeted the eastern Ontario region (excluding Ottawa and Kingston) with contributions totaling $81 million (12 percent of the total).

5. The SOPP programs are consistent with the priorities of the Government of Canada, including the Innovation and Skills Plan, as well as the mandate of FedDev Ontario. Based on the results of consultations undertaken for the Innovation and Skills Plan, the Government of Canada identified three priority areas: people (ensuring that people are equipped with the right skills and experience to drive innovation); technologies (taking full advantage of transformative emerging technologies that can elevate the competitiveness of established and new firms, industries, and clusters); and companies (growing the next generation of global companies in Canada)^{Footnote 22}. The investments made by FedDev Ontario have been consistent with these priorities, particularly with respect to:
   • Attracting, developing and retaining highly skilled workers, researchers and entrepreneurs. FedDev Ontario contributions have supported the delivery of training while proponents report that investments have helped southern Ontario attract and retain key workers, researchers and entrepreneurs.
   • Strengthening the innovation ecosystem through the further development of research infrastructure (e.g., construction of new buildings, adaptation of existing buildings, and provision of equipment for new or existing centres), investments in technology development, testing and commercialization, and facilitating the development of collaborations and partnerships.
   • Accelerating the start-up, early development, expansion and modernization of companies by attracting and facilitating investment, supporting technology adaptation, adoption and commercialization, supporting advisory services and market development activities and attracting anchor firms.

   As noted earlier, FedDev Ontario has made significant investment in clusters identified by the Government of Canada as priorities including advanced manufacturing, health and biotech, ICT, agri-food and clean tech and clean resources. The activities supported under the SOPP have also directly contributed to the Agency's core mandate of strengthening southern Ontario's economic capacity for innovation, entrepreneurship and collaboration and promoting the development of a strong and diversified southern Ontario economy.

4.2 Relationship to Other Programming

The major findings of the evaluation regarding the relationship between the SOPP and other programming are as follows:

1. Factors such as the place-based nature of FedDev Ontario, the strong demand for funding, and coordination between FedDev Ontario and other funding organizations help to ensure that the SOPP programs complement rather than duplicate other federal or provincial government programs that promote innovation, business development and community development. There are a variety of other programs available to support development in southern Ontario. When asked about other programs that are available in southern Ontario that share at least somewhat similar objectives to those of the programs included in SOPP, the key informants and those who were surveyed identified a range of different programs which are listed in the table below.

While the breadth of the SOPP programming creates the potential for some overlap with other programming, only a few key informants suggested that overlap exists with other programs and only one identified it as an issue (the potential for overlap between the recently announced Strategic Innovation Fund and the Advanced Manufacturing Fund). Some of the key characteristics of FedDev Ontario and its programming which enable the Agency to position its programs to complement the other available sources of assistance include:

• As a regionally-based organization, the Agency is well positioned to understand the specific needs of key target groups that are not being met by other programs. FedDev Ontario works closely with industry development organizations and companies in identifying needs in both existing and emerging clusters. It is able to deliver services at the ground level through supporting key intermediaries (such as CFDCs, industry associations, and other institutions) which provide capital and support services to both for-profit and not-for-profit organizations. Furthermore, FedDev Ontario undertakes analysis, outreach and engagement activities and periodic research (such as the consortia study completed in 2016) which further helps the Agency keep abreast of potential opportunities, issues, and constraints to development.
• By offering a range of programs, the support provided by FedDev Ontario can be tailored to meet the specific needs of clients.
• The contributions provided by FedDev Ontario allow for stacking within specific guidelines, enabling the funding to be leveraged with funding from a variety of other sources including other federal government programming and provincial programming. The co-funding arrangements enable proponents to increase their access to further funding and facilitate the sharing of risk. Approximately one-half of the other government representatives who were interviewed reported that the presence of FedDev Ontario funding for specific projects influenced their decision to provide funding or other assistance requested. Some elaborated that the commitment of FedDev Ontario provides them with an assurance that due diligence has been done and the organization or project is worthy of the investment. The presence of federal funding also brings more attention to a project and may increase opportunities to access funding from private sector investors.
• FedDev Ontario provides pathfinding assistance, referring organizations to other sources of assistance when relevant. In addition to the referral service offered by project officers, FedDev Ontario also hosts Canada Business Ontario which acts as a one-stop shop providing information and advice about available government grants and requirements, as well as other sources of financing. FedDev Ontario also works with federal partners on the Accelerated Growth Service initiative, helping high-growth firms scale-up through a coordinated and streamlined approach to accessing federal business support services.

• Efforts have been made by FedDev Ontario to improve the level of coordination in programming across various government organizations through regular meetings and established communication channels. These efforts aim to minimize potential overlaps, clarify roles and share information. A few key informants suggested that there may be opportunities to further enhance the level of coordination between programs. For example, it was suggested that applicants and proponents would welcome efforts to ease the administrative burden by achieving greater standardization across government programs with respect to application requirements, stacking limits and reporting requirements.

  Another issue that was identified relates to the inconsistent treatment by CRA of repayable loans for tax purposes. In some situations, CRA treats the repayable loans provided by FedDev Ontario to businesses as a grant rather than as a loan, which can increase the taxable income of the proponent and reduce their eligibility for a SR&ED Tax Credit. This issue was also raised in the evaluation of the Southern Ontario Development Program^{Footnote 23}. FedDev Ontario has discussed this issue with CRA extensively who has recommended that our program officers should advise clients to seek expert advice on the potential tax implications of the funding they receive from FedDev Ontario.

2. SOPP fills needs that would not be met by other programming. Only 4 percent of projects would have gone ahead as planned in the absence of FedDev Ontario funding. As indicated in the table below, most proponents indicated that the project would have been reduced in scope (44 percent), implemented over a longer period of time (33 percent), or delayed in the absence of FedDev Ontario support. About 15 percent of proponents said they would have had to cancel the project. Only 5 of the 40 applicants not approved for funding indicated that their projects proceeded as planned in the absence of FedDev Ontario support.

5. Effectiveness

This chapter summarizes the major findings regarding the effectiveness of the programs. It begins with an overview of the current status of the projects for contextual purposes and then reviews achievement of project objectives, impacts to date of the completed projects, projected impacts of the on-going projects, and extent to which the projects are expected to continue on and generate further impacts beyond the term of the FedDev Ontario funding.

5.1 Current Status of the Projects

Final reports and close-out reports had been filed for 65 of the 190 projects (34 percent) at the time of the evaluation. The final reports are prepared by the proponents at the conclusion of the project, adding to the progress reports prepared during project implementation. Close-out reports are prepared by project officers, typically after a site visit. The 65 completed projects tend to be of a shorter duration than the on-going projects and consisted primarily of projects funded under IBI (46 projects), IBGP (12 projects), and IRD (5 projects). The ICP, AMF and EODP projects were all multi-year projects. No ICP or EODP projects were completed and only one AMF project was completed. The other closed project was a SOPP Strategic Project.

Of the 117 projects represented in the survey, 35 (30 percent) were completed according to the proponents. No projects were identified as suspended, cancelled or yet to start. Only 6 of the 117 projects were identified as not being implemented as planned. Of those six, four reported having been delayed, three had some project partners change, three indicated the project focus had shifted somewhat, and one indicated the scope was reduced somewhat. Four of the six proponents indicated that the change impacted the effectiveness of the project (two indicated that effectiveness was enhanced and two indicated effectiveness was reduced, both of whom still rated the project as at least somewhat successful in meeting its objectives). Of those noting that effectiveness increased, one indicated that the change in strategy resulted in higher margins than projected while the other indicated timelines to commercialization were shortened because of bringing in new partners.

5.2 Achievement of Project Objectives

The proponents and beneficiaries were asked to rate how successful the project has been to date in terms of achieving its objectives, using a scale of 1 to 5, where 1 is not at all successful, 3 is somewhat and 5 is very successful. As indicated below, the average rating provided by both the proponents and the beneficiaries was 4.4.

The average ratings across the programs ranged from a low of 4.3 amongst the IBI and IBGP proponents and CEDP beneficiaries to a high of 4.9 amongst the EODP proponents.

When the few proponents and beneficiaries associated with lower rated projects were asked to identify areas where the project had been less successful than expected, the issues focused mostly on delays in project implementation caused by outside factors (e.g., delays in obtaining equipment for suppliers, regulatory issues, or in one case a fire) or by delays in the project approval process which delayed the results or required the project to be implemented in a shorter time period. Other issues that were identified related to project partners (e.g., an investor not following through with their commitments), changes in the market which required changes to the commercialization strategy, slower sales than expected, and not having enough funding to implement the project as planned.

5.3 Outputs and Impacts of the Completed Projects

The close-out reports, which are prepared by the project officer and signed off by a manager, summarize information on the proponent, project, expenditures, project outcomes, and project monitoring (e.g. site visits and audits). As part of the evaluation, a detailed review of each close-out report was undertaken to develop a database, capturing the impacts reported by each completed project. This section provides a summary of the impacts, drawn from the close-out reports and final reports, surveys, and case studies.

As indicated in table 11 below, the types of outputs and impacts most commonly reported for the projects included the creation or maintenance of permanent full-time positions, training, collaborations and partnerships, increased sales, capacity development, and the creation of intellectual property.

A further description of these outputs and impacts is provided in the following paragraphs, including examples of some of the impacts, largely drawn from the case studies (provided in the text boxes).

Provision of Services to Businesses and Other Beneficiaries

Nineteen projects reported providing businesses, particularly SMEs, with access to needed services, resources and other support ranging from business development services, technology development and testing services to training, capital and other resources. Most of the SOPP projects that provide significant support services to businesses are multi-year projects which are still on-going (and therefore not included in this total).

The closed projects that provided services to businesses largely involve angel investments. For example, the Maple Leaf Angel Capital Organization used FedDev Ontario funding to build on its network of investors and entrepreneurs by expanding its outreach, education, and engagement activities. Businesses benefited by not only having greater access to capital, but also greater access to guidance from experienced business minds. The National Angel Capital Organization operates as an Angel Network that is able to leverage the knowledge of its established investor community and professionalize the Angel asset-classes. The project led to more co-investment, investor syndication and improved access to capital for Ontario companies.

Maintained or Increased Employment

The 65 projects reported the creation of 1252 permanent FTEs and 1043 temporary FTEs, as well as the maintenance of 534 permanent FTEs and 7 temporary FTEs. Most of the permanent employment created to date is associated with manufacturers that undertook projects funded under IBGP and IBI. In addition to Bioamber, which is profiled in the box to the left, some examples of businesses that created or maintained FTEs with support from FedDev Ontario included:

• a robotics company that created 93 permanent FTEs and maintained 35 permanent FTEs by expanding its manufacturing capabilities, upgrading facilities, and implementing new technology;
• an automated food manufacturing and storage facility that expanded operations to service the Canadian and US markets, resulting in 145 permanent and 120 temporary jobs; and
• a food manufacturer which invested in freezer storage and advanced production equipment to enhance its competitiveness in the export market (the initial estimate was that 74 FTEs would be created; in the end, 80 permanent and 104 temporary FTEs were created).

Training

Most completed projects (55 of the 65) reported training of some kind. Those that reported training generally consisted of either:

• Individual companies that undertook training of their own staff, usually as part of a manufacturing expansion or modernization project. An example from another sector is a software company that provided training to assist in marketing its newly developed software.
• Most angel investment-related projects that provided training to others. For example, the Peterborough Region Angel Network brought together accredited investors and mentors who provided direction and advice to companies in the early stage of operations to promote the business and help them succeed.

Partnerships or Linkages

Fifty of the 65 projects reported the development of partnerships and collaborations with various other groups including investors, research collaborators, project partners, businesses and others. Examples include a medical instrument development company that commercialized a medical device which provides diagnoses in real-time (the organization reported developing relationships with 34 investors as part of the project), a lighting manufacturer that received funding to set up a pilot lighting manufacturing line in Toronto reported relationships with 27 organizations including investors, collaborators and service providers; and a software developer that developed partnerships with 19 organizations including technology partners, sources of funding, and investors.

Capacity Development

Capital expenditures represent a significant percentage of the total budgets of the SOPP projects. Over 60 percent of the combined total budget for all 190 projects (including the costs funded by FedDev Ontario and those funded from other sources) went towards the cost of equipment, buildings and land (the costs were budgeted under various cost categories including purchase of land, facility construction/renovations, facility expansion, furnishings and fixtures, building modifications and upgrades, manufacturing and equipment, leasehold improvements and other capital costs). Capital costs as a percent of the total budget did not vary much between the shorter-term closed projects (61 percent) and longer-term on-going projects (62 percent of costs).

Nineteen of the 65 closed projects reported making capital investments in research, technology development and testing, business development, and production capacity which will continue to be in use well beyond the term of the FedDev Ontario funded project. Fifteen of the proponents reported investments of more than $1 million in capital assets. For these projects, FedDev Ontario contributed towards the costs of obtaining equipment, constructing equipment, buildings, making renovations or modernizing operations. Examples of major investments include the Bioamber which was profiled earlier, as well as reopening of the Pembroke MDF plant, and retrofitting of the ArcelorMittal automotive parts plant.

Intellectual Property

Twenty-eight of the 65 projects reported the development or advancement of some form of intellectual property (IP), for which the organization may or may not seek patent protection or a copyright on created content. The form of IP ranged from software and hardware to medical devices, with software being the most common. Some examples include the final development and global commercialization of a clean energy storage technology on which six different patents were filed; further development and commercialization of six different products and services based on a product information software platform that allows manufacturers to provide retailers full product details in multiple formats in real time; a new high-quality mobile imaging device that helps doctors accurately diagnose cancer in real time (three trademarks and three patent applications were submitted for approval in three different jurisdictions); further development of a product that enables users to get technical support directly from their mobile devices (12 patent applications are being converted to full patent status and six new patent applications have been filed); and final development and commercialization of a fitness system that provides real time information to users (the project is associated with 15 patent applications in 5 countries to date with more expected).

Increased Sales and Revenue

Of the 65 completed projects, 44 proponents reported an increase in sales. Increased revenues resulted from expansion into a new market, developing and/or launching of a new product, or the expansion/building of production capacity. To date, the closed projects have reported increased sales of $205 million in Canada and a further $36 million in export sales. These figures are expected to increase significantly going forward as markets are further developed and production increases.

Reflecting the characteristics of the projects which have been completed to date, most of the revenues (73 percent of the sales in Canada and 94 percent of the export sales) have been generated by manufacturers. In addition to Pembroke MDF, other examples include a food producer that increased its capacity by expanding its production facility and adding a fifth food processing line, resulting in an increase in sales of $41 million; the production capacity of a manufacturer was expanded by relocating its operations into a much larger facility and expanding storage, resulting in a $40 million increase of sales to date; a frozen food manufacturer which increased its production and storage capacity to meet demand for products and improve competitiveness in the export market, resulting in $27 million in increased sales; and a project that automated a production line to fulfill the demands of Ontario's auto manufacturing industry and fulfill a long term commercial contract, increasing export sales by $16.3 million.

Enhanced Competitiveness and Sustainability

Many of the projects funded by FedDev Ontario help to enhance the competitiveness and sustainability of a company that is receiving funding or assistance provided by a funded intermediary organization. Twenty-one projects were flagged in the close-out reports as specifically enhancing competitiveness and sustainability. Examples include an energy usage reporting software company that developed marketing materials, and hired and trained sales and marketing staff to grow the business' customers and its geographic sales footprint; a project that enabled a company to enter the global market to build clean energy generation plants; a digital advertising company that finalized development of a digital display advertising system and executed a global sales and marketing plan, enhancing the company's competitiveness and presence in the international marketplace; a software company which enhanced and commercialized their software communications platform that provides analytics and monitoring solutions, enabling them to enter the international market; and an on-line voice over casting service that enables producers and casting agents to search for actors for the various projects in which they are involved. The project involved the expansion of its project management and sales teams to meet growing demand, along with the development and execution of a global market oriented marketing strategy.

Business Development and Angel Investments

Forty-three of the 65 projects reported further business development, usually further development of the proponent organization. Some intermediaries, mostly angel organizations, also reported that their services had supported or facilitated the development of other businesses.

Eight completed projects involved angel investments. The angel investor groups reported 356 investments totaling $227 million in funding. The funds provided by FedDev Ontario were used in the administration and development of the various angel investments networks, helping to identify both potential investors and companies, as well conduct the due diligence necessary to vet potential investments. Some examples of completed angel investment projects are Maple Leaf Angels Corporation which used funding to help build on its current outreach, education, and engagement activities for both investors and entrepreneurs; GreenSky President's Club used funding to grow the club to 100 members and continue to make the process of investing in good companies more efficient; and the Georgian Angel Network was able to sustain and build its member base, investable capital and sector expertise.

5.4 Impacts of the Ongoing Projects

Of the 190 projects, 125 of the projects are still going. There are two sources of data available on the projects: (1) data reported in the progress reports received to date; and (2) the projections regarding the impact of the project which were included in the contribution agreements signed between FedDev Ontario and the project proponents.

Of the 125 projects, 81 submitted data on the progress of their projects as of December 31, 2016 (these 81 projects submitted a total of 190 progress reports, an average of 2.3 progress reports per project). As per the available progress reports, the projects had created 10,901 permanent FTEs and 1,068 temporary FTEs to that point.

A more comprehensive indication of the impacts of the projects was developed by compiling the estimates of impacts developed at the time of the project approvals. To develop the database of impacts, a detailed review was conducted of the Contribution Agreements and the Project Approval Forms for each active project. While it is premature to determine whether the projects will meet or exceed their projections, the results of the evaluation indicate that the vast majority of these ongoing projects are proceeding largely as planned, they are still expected by the proponents to achieve their objectives, and many have already reported significant impacts. The impact most commonly projected for the projects includes the creation or maintenance of FTEs, increase in sales, and the provision of services to businesses or other beneficiaries as indicated in the table below.

A further description of these impacts is provided on the following pages.

Provided Services to Businesses and Other Beneficiaries

Fifty-nine of the 125 projects report providing services to over 2,900 businesses and other beneficiaries to date (as per the progress reports). The EODP program, though its business development, community innovation and CED projects, accounts for a majority of these. Through 2016, the leading projects in terms of the numbers of businesses served were the Yves Landry AIME Global Initiative (260 businesses) and the CME SMART ATGG program (225 businesses). Other leading organizations funded by FedDev Ontario include:

• The Ontario Centres of Excellence (OCE) reported assisting 204 businesses. OCE provides seed financing and support for product development and market entry to Ontario start-ups, helping them scale the companies and prepare for later-stage investment, commercial partners, and customers.
• The Waterloo Accelerator Centre reported assisting 127 businesses. The JumpStart program provides potential start-ups with matching seed funds and mentorship in partnership with the University of Waterloo, Wilfred Laurier University and Conestoga College.
• The George Brown College of Applied Arts and Technology reported assisting 93 businesses. George Brown College of Applied Arts and Technologies Food and Beverage Labs will offer applied research services for industry partners seeking to commercialize products and create jobs in the food and beverage industries.
• The Bioenterprise Corporation reported assisting 66 businesses. Bioenterprise is a business accelerator that helps to promote the creation, growth expansion of businesses in the agri-food, life sciences and bio-products sectors.
• The York University has developed a pipeline of 32 partners working to develop health technologies/services that will catalyze the growth of an emergent health tech cluster in York Region.
• The Canadian Film Centre reported assisting 29 businesses. The Canadian Film Centre established and expanded a continuum of integrated programming, known as ideaBOOST, programs to support more timely creation and commercialization of digital media products.
• The Ontario Bioscience Industry Organization reported assisting 20 businesses. The Enhance Capital Access Advisory Program is designed to assist high-potential bioscience companies to attract investment from Canadian and global investors.

Provided Training or Mentorship

Twenty-three companies reported training including nine companies that are undertaking training of their own staff (usually related to manufacturing expansion or modernization) and 14 intermediary organizations such as the Yves Landry Foundation that provide training to others.

Funded through IBI, Communitech Fierce Founders is an accelerator^{Footnote 24} program targeted at businesses founded by women that incorporates a significant mentorship component. Participants receive $30,000 of funding, mentorship from Communitech's growth coaches to help the businesses set and reach milestones, build and execute a sales strategy, and go to market plan and preparation for investor presentations.

Two other examples include Next Canada^{Footnote 25} (which provides mentoring, training and matching seed funding of up to $30,000 to high-potential start-ups) and the York Entrepreneurship Development Institute^{Footnote 26} which received support to expand its incubator program In total, FedDev Ontario funding is expected to provide training and mentoring support to over 900 businesses.

Maintained or Increased Employment

Of the 125 ongoing projects, 121 are projected to have a significant impact on employment. They are projected to create or maintain over 31,000 permanent FTEs and more than 2,500 temporary FTEs. Over 10,000 FTEs had been created as of December 2016.

About 60 percent of the FTEs will be generated through contributions to intermediary organizations which then work with businesses. For example, the CME SMART Advanced Technologies for Global Growth program is expected to generate more than 3,500 FTEs, the OCE project is expected to generate nearly 2,500 FTEs, and the AIME Global initiative is projected to create over 2,000 FTEs. The remaining 40 percent will be generated through funding provided directly to businesses which then expand or modernize their operations (a leading example of the impact of direct funding is a major AMF project which involves enhancing the capacity, productivity and quality of a major manufacturer; this is expected to create more than 2,700 FTEs).

Increased Sales

Increased sales were projected for 68 of the 125 ongoing projects, including $3.7 billion in increased North American sales, almost $600 million in sales outside North America and nearly $1.1 billion in sales from the commercialization of new products and technologies. Almost 1,500 new products, services and processes are expected to be commercialized.

Most of the projected increase in revenues is associated with contributions provided directly to businesses, particularly manufacturers who received support to further develop or upgrade their manufacturing facilities. Some notable examples include a food manufacturer that received support to invest in new equipment and build a processing plant to expand production, a fabricated metals manufacturer that is increasing its production capability, and a food additive company that is constructing a new building to accommodate expanded production and commercialization of proprietary natural food additives.

Supported Angel Investments

The angel investment networks match potential investors with those looking for funding. The funds provided by FedDev Ontario were used in the administration and development of the various networks, helping them identify both potential investors and companies as well as assisting with conducting the due diligence necessary when vetting potential investments.

In addition to the 356 angel investments totaling $227 million in funding reported by the completed projects, the on-going projects are projected to make nearly 450 investments totaling over $150 million. The Southwestern Ontario Angel Group received support to expand and enhance angel investing in southwestern Ontario. The Community Growth Accelerator Network is implementing a strategic plan focused on member attraction, promotion, and expansion of investment into early stage companies primed for growth. Angel One Network Inc. is focused on growing an investor base of accredited investors interested in early stage innovative companies.

Capacity Development

Fifty-six of the 125 ongoing projects involve the development of capacity which will continue to be utilized beyond the end of the FedDev Ontario funded project. Thirty-five proponents budgeted for investments of more than $10 million in capital assets including 20 which budgeted more than $20 million. FedDev Ontario has provided funding to companies to upgrade manufacturing capabilities for a wide variety of products such as steel, fabricated metals, food and confectioneries, automotive and aerospace components, biologics, rail car and aluminum trailer manufacturer manufacturing, paperboard packaging, recycled rubber, and vinyl upholstery fabrics.

FedDev Ontario has also made significant investments in research, development, and commercialization capacity which will have significant impacts in terms of cluster level. Some projects focused on existing and emerging clusters, such as regenerative medicine, water, cyber security, digital media, health information technology, and border logistics, where the potential markets are large and projected to grow rapidly. Other projects have focused more specifically on segments within a cluster, on the provision of equipment, technology and support services targeted at a range of sectors, or the development of a platform technologies targeted at a range of sectors.

For example, following up on an earlier project, FedDev Ontario provided funding to increase access to SOSCIP's high performance computing platforms and encourage new collaborations leading to commercialization outcomes. SOSCIP provides a platform that brings together research universities, IBM and SMEs to establish a collaborative model of R&D and innovation utilizing the latest advanced computing technologies.

Funding was also provided for the Development and operation of CCRM's Centre for Advanced Therapeutic Cell Technologies (CATCT), a 10,000 square foot process development facility dedicated to the development of technologies and optimization of processes that will enable industrial manufacturing of human cells for therapeutic purposes. Another example is the Collaborative Greenhouse Technology Centre, which involved retrofitting Vineland's newly built greenhouse into a state-of-the-art research greenhouse of an acre in size, comparable to commercial greenhouse operators. It will be a platform for SMEs across North America to test, refine, and develop their greenhouse technologies. The Southern Ontario Network for Advanced Manufacturing Innovation (SONAMI) is a one-stop shop providing access to equipment, development and testing facilities, and product development and applied research services. MARC provides a research facility, including access to engineers, scientists, and social scientists, to assist industry (ranging from major OEMs to SMEs) in undertaking applied R&D and product development for the auto industry, focusing on sustainable solutions such as development of hybrid and electric powertrains, highly efficient and cost-effective powertrain components, and lightweight materials.

5.5 Impact of the Projects on Beneficiaries

Project beneficiaries were asked to identify what impact the projects have had on their organization. As indicated below the beneficiaries in the IBGP projects (the CME SMART ATGG program and the Yves Landry program), as well as the businesses involved in the EODP reported that the projects generated a range of impacts including increased revenues, cost savings, and the commercialization of new technologies, products, processes, or services. The EODP community innovation projects reported a range of impacts including further development of the local businesses.

More specifically, participants in the CME SMART ATGG program reported that the assistance helped them to improve their competitiveness by lowering costs, improving quality and productivity, and reducing production time and lead times through acquiring new equipment and technology, training employees, implementing more modern technologies and manufacturing processes, developing new products and markets, and improving safety and quality. The improvements led to increased sales with five beneficiaries reporting increases of up to over $1 million.

Participants in the Yves Landry program reported that the assistance helped them to provide training to their staff to support lean manufacturing processes, innovation and continuous improvements, implement new systems (e.g., ERP, food safety, software & technology upgrades, and processes), and hire and train new employees. Impacts on the business included the development of new customers and markets, the ability to take on more orders and work, process improvements, improved productivity and costs savings, and reduced overtime, product handling, and maintenance.

Business participants in the EODP reported a wide range of impacts in terms of business start-up and development, new product development and commercialization, training and skills development, job creation, and increased access to other sources of capital. Local recipients of assistance reported that the community innovation projects provided support for economic development activities and initiatives such as research, the development of sector and regional development strategic plans, public infrastructure development (ranging from trails and wireless services to dock expansion and park revitalization), training, services for businesses start-ups and social enterprises, business counselling, business incubators, investment attraction activities, business retention and expansion, regional marketing and branding activities, and website development.

5.6 On-going Impacts of the Projects

The impacts of the projects will continue to grow over time. Many of the projects resulted in improvements to manufacturing operations, facilities and equipment or the development of research capacity or other permanent assets, the benefits of which will continue on well beyond the end of the project. In some cases, businesses have learned the effectiveness of marketing or training activities and will continue to implement these types of activities without funding support. When asked to rate the extent to which the project activities will continue on after completion of the FedDev Ontario project, the proponents provided an average rating of 4.1 (using a scale of 1 to 5, where 1 is not at all, 3 is somewhat and 5 is to a great degree). The elements that will not continue or will be reduced in scope in the absence of the FedDev Ontario funding consist primarily of the services and support implemented by third party organizations (e.g., the CME SMART ATGG program, regional EODP programs, organizations such as OBIO or Bioindustrial Innovation Canada, and some of the angel investor networks).

An analysis conducted by Statistics Canada in 2017 illustrates the longer-term impacts that the FedDev Ontario business support programs can have on businesses.^{Footnote 27} FedDev Ontario provided the Centre for Special Business Projects with a list of enterprises that received support during the period from 2008 to 2016 (covering companies funded under the SOPP suite of programs, as well as under programs from previous mandates including SODP, SOA and EODP). Of these companies, 2,779 enterprises were successfully matched to Statistics Canada's Business Register (BR). Statistics Canada then compared the growth of these firms to the growth of a comparison group and found that, over time the FedDev Ontario clients reported significantly higher average annual revenue growth of 10.1 percent, 6.2 percent higher employment growth and 3.4 percent higher productivity growth than non-clients. The firms also, on average, spent 21.9 percent more on R&D activities than non-clients. Furthermore, the survival rate of FedDev Ontario clients tended to be higher than in the case of non-clients.

Input/output modelling conducted by The Conference Board of Canada found that every $1.00 from FedDev Ontario core program investments generates $3.50 of value in the broader economy (via direct, indirect, and induced impacts). Finally, the boost to economic activity by Agency core investments is expected to return about $609 million to the government (municipal: $76 million; provincial: $279 million; federal: $254 million).

6. Program Design and Delivery

This chapter provides a summary of the findings regarding program design, delivery and cost-effectiveness.

6.1 Program Design and Delivery

A series of statements were developed regarding program design and delivery. The 117 proponents and the 40 unfunded applicants were asked to indicate whether they strongly agreed, somewhat agreed, neither agreed nor disagree, somewhat disagreed or strongly disagreed with each statement. The results are summarized in the table on the following page. As indicated, a majority of the proponents strongly agreed that:

• They were satisfied in their dealings with FedDev Ontario staff (78 percent);
• The design and delivery of the programs was appropriate (65 percent);
• Sufficient time was provided for the project to be completed (56 percent);^{Footnote 28}
• They received clear direction regarding the proposal (54 percent); and
• The application requirements and criteria were appropriate (53 percent).

The most significant concerns amongst the proponents were the length of the application and approval process, the reporting requirements, and usefulness of the performance measurement data. Given they invested in the process but were eventually not successful in accessing funding, the non-funded applicants tended to be less supportive of various aspects of the program design and delivery, particularly the length of the application and approval process (i.e., how long it took them to be notified that they would not be funded), their dealings with FedDev Ontario staff, and the direction provided with respect to what was expected in a proposal.

The proponents, unfunded applicants and non-applicants were asked to identify what they see as the primary advantages and disadvantages of the FedDev Ontario program. The advantages most commonly identified were the level of funding available (identified by 83 percent of respondents), the types of costs covered (67 percent), and ease of access (46 percent). The disadvantage most commonly identified was the reporting requirements (48 percent). A few applicants talked about the application process being unclear and cumbersome (e.g., lack of transparency in process, lack of clarity in guidelines and eligibility, multiple points of contact, long leadtimes for funding announcement, missed opportunities to promote, red tape). Others noted difficulties in accessing funding as the five-year mandate comes to a close (the need to wait for more funding to become available).

While perceptions of program design and delivery were largely consistent across programs, there were some variations. Not-for-profit proponents of large multi-year ICP projects, which can involve partnerships between MNEs, SMEs, institutions, and not-for-profit organizations, were those most likely to express concerns about the length of the application and approval process as well as the reporting requirements. Challenges with intellectual property rights were much more common amongst proponents in the AMF and ICP programs while challenges with the SR&ED tax credits were most common for businesses participants in the AMF and, to a lesser extent, the IBGP. Amongst the unfunded applicants, ICP, IBGP and IBI clients were those most likely to identify issues with the length of the application process and dissatisfaction with the FedDev Ontario staff.

The following sections summarize the input provided by key informants, proponents, and unfunded applicants regarding program design and delivery issues, including the key issues that were identified and their suggestions for improvement.

Program Design

The prevailing sentiment amongst proponents and key informants is that the programs are well-structured, each filling an important role or niche which targets one or more key issues constraining development of the economy (e.g., access to capital and talent, ability to develop and commercialize new technology, need to modernize the manufacturing sector, need to improve the economy in eastern Ontario, etc.). The programs are considered effective in reaching the key target groups. Demand^{Footnote 29} for most programs has been strong as demonstrated by the total of 4215 project applications received by FedDev Ontario and its third party delivery agents.

Targeting not-for-profit organizations, through non-repayable contributions, and for profit businesses, through repayable contributions, FedDev Ontario is able to implement the parallel strategies of solidifying the ecosystem that drives development (through increasing access to capital, research capacity, technology, and support services) and strengthening businesses (that generate economic growth by establishing, expanding and modernizing operations, developing new products and markets, and commercializing new technologies). The key informants particularly highlighted the role of the program in facilitating strategic growth of existing companies, enabling early-stage companies to bridge the commercialization gap and “valley of death”^{Footnote 30}. Through the EODP, IRD and other programming, FedDev Ontario has also been able to support communities in addressing tough economic conditions by, for example, leveraging their strengths and better supporting the development and survival of smaller companies and social enterprises as a means to compensate for the loss of larger companies.

There was support, both within and external to the organization, for maintaining the existing program structure given that the suite of programs works well together to address key needs and are well-designed and delivered. Continuation will mean that FedDev Ontario avoids a situation similar to that of 2013, needing to create awareness of a new suite of programs.

The movement to fund larger projects was supported by key informants, who indicated that making significant investments in the innovation ecosystem and leveraging significant investments by major anchor companies can be highly effective strategies for supporting cluster development and regional development. The average size of FedDev Ontario contributions increased across all programming versus the previous mandate (including EODP and IBI). As indicated below, the average contribution per project increased from $1.45 million per project to $3.62 million, an average increase of 150 percent.

Of the 443 projects funded prior to the current mandate, 17 (4 percent) involved contributions of $10 million or more including five which included contributions of $20 million or more. Of the 190 SOPP projects funded during the current mandate, 21 (11 percent) involved contributions of $10 million or more including seven which included contributions of $20 million or more.

FedDev Ontario reviews its programming on an on-going basis and makes some changes to improve program design and delivery. Examples of changes include development of the Professional Practice Pilot^{Footnote 31}; increased use of third party delivery models; revisions to centralize administration of the collaborative projects component (CEDP) of the EODP; easing the required timelines for completion of EODP projects funded at the local level; and other improvements to processes, tools, project databases, and templates. The quality of applications has also tended to improve over time as organizations and staff became more familiar with the new programming.

The most serious concern regarding the program design relates to the five-year funding mandates of FedDev Ontario, which according to both key informants and proponents impacts on what projects can be funded, particularly in the later years of the mandate. The finite nature of the mandate:

• Means that multi-year projects can only be approved earlier in the mandate and little funding remains available for approvals late in the mandate. The following table was developed based on approved funding and project start dates and end dates. As indicated, almost three quarters of the funding (to May 24, 2017) was approved for projects more than three years in duration, which meant that the projects had to be approved early. Of the approved funding, two-thirds (67 percent) was associated with a start date in 2014-15 and 19 percent was associated with a start date in 2015-16 (8 percent was associated with projects prior to April 1, 2014).

• Projects that are approved later in the mandate typically need to be restructured such that they can be completed within the time period remaining. This makes programming much less responsive to industry and regional needs (the programs virtually disappear for several years in terms of accepting new proposals) and can significantly compress the timelines available for project implementation.
• Creates significant workload issues for program staff, particularly in the first year of the new mandate. During that first year, program staff needs to engage with prospective applicants, and reviewing and processing applications with a particular focus on the larger scale, multi-year projects which tend to be more complex in nature. At the same time, the program staff must deal with legacy projects from the previous mandate, reviewing final reports, conducting site visits, validating reported results, and preparing the close-out report.

Other issues or opportunities that were identified by key informants or project proponents included:

• The need for more flexibility to enable the programs to better adapt to key opportunities (e.g., broaden eligibility requirements with respect to eligible projects, maximum contributions, eligible expenditures, and/or repeat funding). It was suggested that proposed projects sometimes need to be substantially restructured, not to improve the intended outcomes but, rather, to make the project fit better with the existing guidelines and timelines. It was suggested that program guidelines should focus more on the intended outcomes (what the project will achieve) rather than how it will get there.
• More actively promoting the programs to generate a larger pool of projects from which to select.
• The need for funding or additional funding for certain target groups, particularly women, Indigenous businesses, and youth entrepreneurs. This could involve developing sub-programs for specific target groups (which could, for example, involve conditionally repayable contributions), prioritizing certain groups in the assessment of applications, or simply more actively promoting programs to these groups.
• The desire, mostly amongst some corporate proponents, to make the repayable provision conditionally repayable depending on the success of the project. It was argued that this sharing of risk would encourage more companies to make investments in earlier stage companies and technologies.
• The potential need to adjust some of the programming to better align it with evolving government priorities.
• The potential to offer non-repayable contributions for certain large-scale private sector investment projects where such a contribution is needed to compete against other regions to secure significant investments.
• Having FedDev Ontario take a more proactive role in the identification of needs/opportunities and working with proponents to develop projects.

Application Requirements, Criteria, and Reviews

Most proponents, as well as those key informants who expressed an opinion, felt that the application requirements, criteria and process were appropriate given the objectives of the programs. Most concerns related to the definition of organizations eligible to apply for funding. Suggestions focused on:

• Rewarding success, by making it possible or easier for existing proponents to access additional funding to build on successful activities;
• Opening up EODP style funding to CFDCs in other regions;
• Expanding the EODP to include communities currently excluded in the east (Kingston and Ottawa) given the impact that developments in those communities can have on surrounding communities; and
• Opening up funding to regional innovation centres (RICs).

With respect to EODP, it was also suggested that consideration be given to varying the level of funding provided to each CFDC depending on local opportunities and needs (either through a pooling of funding or directly providing differing levels of funding); providing the CFDCs the option of offering repayable contributions; and, where warranted, allowing CFDCs to provide larger individual contributions to fill a perceived gap existing between the EDOP and the larger SOPP programs to which regional businesses and not-for-profits would not normally be able to access funding.

A few proponents, unfunded applicants and key informants expressed reservations as to whether project officers have the requisite experience and industry specific knowledge needed to fully review the applications, particularly for more complex, highly technical or niche projects. It was noted that there has been considerable turnover in the officer positions. Difficulties in assessing applications can lead to delays in the approval (or non-approval) of applications. To help ease some of these issues, FedDev Ontario is developing a Professional Practice pilot program to help accelerate the development of the Agency's capacity in terms of economic and business practice intelligence. Some of the key elements of the pilot program include business information sessions, recommended reading lists, and the development of personal learning plans.

A few key informants suggested that FedDev Ontario could complement its internal resources by contracting with one or more outside agencies (e.g., another government department or the private sector) to assist in the review of certain aspects of project applications; this strategy has been used by other RDAs. For example, both WD and ACOA have engaged NRC IRAP to conduct technical reviews of certain types of applications.

FedDev Ontario did develop an MOU with Industry Canada (now ISED) regarding the review of applications under the Advanced Manufacturing Fund. Under the MOU, FedDev Ontario retained responsibility for project approval but obtained input from ISED regarding the technical aspects (innovation), market relevance and potential spillover benefits of the proposed project. At times, ISED contracted with a private sector contractor to assist in that review. The arrangement was initially complicated by the inexperience the two groups had in working with each other, some differing expectations, and differences in the reporting relationships (at the beginning, ISED and FedDev Ontario reported to different Ministers). In addition, the large number of applications received relative to the number eventually approved placed some pressure on the model (41 applications were received including 20 in the first intake from December 2013 to October 2014 and 21 in the second intake which ran from January to October 2015, of which only 8 were approved). In the interviews, some concerns were expressed regarding the amount of time that the reviews would take. Different opinions were expressed by key informants regarding the utility of the Project Advisory Committee (PAC), which brought together representatives from ISED and FedDev Ontario to support management of the AMF. Most of the initial issues were worked out over time, as personal relationships developed and the reporting relationship of the organizations changed. Overall, the arrangement was considered effective in bringing together knowledge of the region (FedDev Ontario) with knowledge of the sector and technology knowledge (ISED).

Direction Provided to Applicants Regarding Proposals

The most common concern from unfunded applicants is that they would have liked to have known, earlier in the process (preferably before preparing a full proposal) that they were unlikely to be successful in accessing funding. It was suggested, for example, that it would be useful to know how much funding FedDev Ontario had remaining. There were some anecdotal stories that applicants submitted an application, only to be told that funds were no longer available and that they should try again when funds are available.

There was also some uncertainty as to what proposals should address, what is key to a successful proposal and, in the case of unsuccessful proposals, how the proposal scored and the specific reasons why it was not successful so that improvements can be made to future proposals. While the broad eligibility guidelines for the programs are published, some applicants went further and recommended that FedDev Ontario should inform applicants which types of projects are preferred (preferably supported by a formal scoring system) so that a more informed decision can be made regarding whether to prepare a proposal.

A few representatives suggested that FedDev Ontario offer a single point of entry, with a common application form rather than promoting each program separately. Some companies expressed confusion about what support each program provided and under which they would be eligible. By adopting a single point of entry, it was suggested that the responsibility for matching applications to programs would be shifted to FedDev Ontario to decide under which program a potential project would best fit. Another alternative would be to have the potential applicant fill out a short questionnaire defining the key characteristics of their organization and proposed project, which would then be matched against the program eligibility criteria to suggest programs or to indicate that the project as outlined is not eligible for funding.

Length of the Approval Process

The most common concern regarding program design and delivery relates to the length of the approval process and, perhaps more importantly, the unpredictability of the timing. Some applicants (including those that were approved) indicated that they did not receive clear guidance regarding how long the approval process would take. Even once projects are approved, there can still be an extended period before the contribution agreement is in place and the project can actually proceed. Unpredictability regarding timing is a major concern because it makes it very difficult for the organizations to plan and ensure that the proposed partnerships and resources will still be in place when the go ahead is received. Delays can also contribute to a loss of momentum, shorten the timelines available for implementation, and prevent projects from going ahead at the speed of business. One proponent, for example, indicated that their project waited for approval for about 18 months which meant that the project, once approved, had to both ramp up very quickly while being restructured in order to meet the new timelines.

Factors that can contribute to delays include challenges in reviewing the applications (e.g., gaps in the information provided by applicants, inexperienced officers, and the need to obtain input from outside agencies), delays in obtaining sign-off, and difficulties in the negotiation of contribution agreements. It was also noted that FedDev Ontario has little additional capacity to handle surges in application volumes. It was suggested that capacity constraints could be eased somewhat by further clarifying the eligibility requirements and finding ways to screen applications that are not going to be approved earlier in the process. It was also recommended that FedDev Ontario publish its standards regarding the time required to approve and sign contribution agreements for projects and that performance against those standards be made available publicly. Regular updates regarding expected timing should be given to proponents of active applications.

The timing of the election in 2015 may have delayed some projects. There is also at least the perception, amongst proponents, that it is more difficult to gain approval and sign-off for a follow-on project than for the initial project (some proponents reported that negotiations for the follow-up funding took over a year to finalize). It can be difficult to keep a consortium together during such a long waiting period.

The Reporting Requirements and the Usefulness of the Performance Measurement Data

Some concerns were expressed by both proponents and key informants regarding the reporting requirements and the usefulness of the performance measurement data itself. As part of this interim evaluation, a detailed review was conducted of the program data including project approval and project summary forms, contribution agreements, the tombstone data, progress reports and closing reports. Some of the key observations from the surveys, interviews and data review include:

• The system needs to be digitized. The process of compiling the data is cumbersome, labour intensive and difficult to validate which increases the likelihood of data errors. Rather than allowing proponents to enter data directly into a database, project reports are submitted in paper or electronic form (e.g., scanned forms) which then must be manually entered into database. A digital online system would streamline the reporting function, allow information to be carried forward from one reporting period to the next, facilitate automated validation, and eliminate the need for re-entering data (once by the proponents and once by FedDev Ontario).
• There frequently is a disconnect between the key outcomes on which the proponents are asked to report and what they see as the major impacts of the project. While there is a need for FedDev Ontario to have the proponent report on key outputs and impacts (e.g., to meet with requirements of the Policy on Results and the Agency's new Departmental Results Framework), it is also important to capture data on a wide variety of key measures. An effective data entry form would enable proponents to relatively easily report on a range of indicators relevant to their project (using skips patterns to adapt the indicators by program and project), which could then be validated (e.g., using both automated procedures and personal follow-up where needed). The data could be rolled up easily to report on results by program, region, or sector or whatever parameter is of interest to FedDev Ontario managers on an on-going basis or in response to specific requests.
• The amount and frequency of reporting was viewed as overly burdensome by some of the proponents, particularly those required to report quarterly and those who must report data on each of the beneficiaries with whom they work. Some suggested moving to a bi-annual or annual reporting system, or requiring more detailed information only once per year while more limited reports (providing basic information and highlighting any major issues or problems) could be provided quarterly. It was noted that, at present, quarterly reports can often take two or three days to compile and enter the information needed.

The Time Available to Complete Project

Most proponents indicated that the time available to complete the project was appropriate. The multi-year projects were those most likely to indicate that the time was not appropriate, noting that a variety of factors (e.g., later than expected approval) delayed the start of the projects. Given the fixed term of the mandate, the end date could not be extended.

Satisfaction with FedDev Ontario Staff

FedDev Ontario staff were praised for being knowledgeable, helpful to applicants navigating the application process and tailoring their applications to better meet the eligibility requirements, helpful to proponents in implementing the project, and responsive to inquiries. As indicated earlier, some unfunded applicants expressed dissatisfaction with not receiving guidance during the application process that would either have improved their likelihood of success or informed them earlier that they were unlikely to be successful. Some also expressed concern about the level of program staff turnover, disrupting the personal relationship and level of understanding that can develop between the project officer and the proponent.

In addition to funding, FedDev Ontario staff was praised for providing hands-on support and engaging in discussions during project implementation. This helped ensure that targets were met and the project continued on the right track. For example, in some projects, FedDev Ontario representatives participated in the conferences and meetings and engaged in open discussion with stakeholders, which helped increase awareness and understanding of the requirements, interests and operations amongst various contributors to the project. Having a single point of contact within FedDev Ontario who understands the project, issues, partners involved and the environment in which the project operates was noted by project proponents as an important aspect of FedDev Ontario contribution to success of the project. Flexibility in being able to move funding across years was noted as helpful in cases where projects were delayed for whatever reasons.

The key informants noted having project officers work with applicants during the application process improves the quality of applications and can reduce the extent to which they will need to go back and forth with requests for information during the approval process. There is at least the perception that review processes may be inconsistent across reviewers and over time, leading to some confusion.

Intellectual Property Rights or Ownership

Fourteen proponents identified issues with respect to intellectual property rights. Ownership of the IP varies from project to project, depending in part on the policy of the participating organizations (e.g., some universities retain ownership of any IP while, at others, the IP is owned by the creator and/or can be negotiated). Ownership is an issue because it directly affects the willingness of private sector partners to participate in a project and the potential economic benefits that could be generated, which impacts the ability of the commercializing entity to raise venture capital or other sources of financing. In most instances, the ownership issue can be addressed through negotiation and most commonly through the use of waivers. The key implications are the importance of identifying potential IP issues early in the application process, taking steps to address that issue so it will not hold up the project, and considering the impact of the issues on the potential for commercialization in decisions as to whether the project should be supported.

SR&ED Tax Credits

Fifteen proponents identified SR&ED tax credits as an issue, although only three strongly agreed it was a major issue. As noted earlier, CRA treats some repayable loans provided by FedDev Ontario as a grant rather than as a loan, which can increase taxable income and reduce eligibility for SR&ED tax credits. The tax interpretation of the contributions seems to vary from region to region within southern Ontario and even from case to case. FedDev Ontario had approached CRA previously about this issue, without a clear resolution. Rather than asking for a change in CRA policy, another option may be for FedDev Ontario to review and revise the design of its repayable loans to improve the likelihood that they will be treated as loans for tax purposes (perhaps structuring them as loans).

6.2 Cost-Effectiveness

The major findings of the interim evaluation regarding the cost effectiveness of the SOPP programs are as follows:

1. The funding contributed by FedDev Ontario has leveraged significant investments from the proponent organizations, other private sector investments, and other funding from the federal government, the provincial government and other sources. The table below illustrates the level of funding contributed by other sources for every dollar contributed by FedDev Ontario. Overall, the projects received $2.45 in funding for every dollar contributed by FedDev Ontario, with the leading source being private sector proponents (particularly those participating in the AMF and the IBGP). Other sources include not-for-profit proponents, other private sector sources (project partners or lenders), and the provincial government.

The following table illustrates the funding leverage by program. As indicated, the average degree of leverage ranged from no other sources of funding for the EODP and SOPP Strategic Project to $2.55 for the AMF and $4.33 for the IBGP.

2. While it is premature to assess the ultimate impacts of the programs, the outcomes generated by the projects completed to date indicate that the programs are already generating significant returns on the contributions made by FedDev Ontario and are on track to achieve outcomes identified in their Contribution Agreements. The following table compares some impacts reported to date by the closed projects to the value of FedDev Ontario contributions approved for those projects. As indicated, to date, the projects have generated $2.54 in increased revenues for every dollar provided by FedDev Ontario (including $2.16 in domestic sales and $0.38 in export sales) and $2.38 in angel investment per dollar contributed. These impacts will increase over time as products and technologies are commercialized and markets are further developed.

The figures are marginally understated to the extent that actual expenditures tend to be slightly lower than the level of contributions approved. For the 65 closed projects, actual expenditures were equal to 98.4 percent of the approved contributions.

3. Tables 18 and 19 contain planned and actual expenditure data that illustrate the impact of the five-year mandate on the programs, with expenditures tending to be very low in the initial years^{Footnote 32} and increasing as projects, particularly the multi-year projects, enter the full implementation stage and make claims. Actual expenditures as a percent of planned increased from 38 percent in 2014-15 to 85 percent in 2015-16 and 94 percent in 2016-17, averaging 73 percent over the three years.

Similarly, operating costs as a percent of budgeted expenditures increased year-over-year, from 84% in 2014-15 to 97 percent in 2015-16 and 99% in 2016-17, averaging 93% over the three years.

4. Operating costs as a percent of grants and contributions are very low. The financial data indicates that FedDev Ontario operates with a very lean operating structure, with operating expenditures averaging 4.9% of total program expenditures.

Two factors identified as contributing to the decrease in the operating costs were the increased average size of approved contributions (while larger contributions can be more expensive to administer they tend to be proportionately less expensive) and increased use of third parties to administer programs funded by FedDev Ontario. For example, organizations such as the Canadian Film Centre, Bioindustrial Innovation Canada, Alliance of Canadian Manufacturers & Exporters, Yves Landry Foundation, Ontario Centres of Excellence, Waterloo Accelerator Centre, Ontario Bioscience Industry Organization, Communitech Corporation, and Next Canada have received funding for administering programs funded by FedDev Ontario and delivered by them.

Some reservations, both from within the organization and externally, were expressed that the programs may be delivered with too lean of an operating budget, particularly given the large number of legacy files (related to projects in the previous mandate) that still require monitoring. The demand for staff resources tends to be highest in the first year of a mandate, given the pressure to approve new projects while still closing off projects from the previous mandate. Staffing constraints can slow the processing of applications, particularly during peak periods, contribute to some of the data challenges that were faced in the evaluation (e.g., the need to go back to the contribution agreements, progress reports, final reports and closing reports for additional data), and impact on project monitoring.

7. Conclusions and Recommendations

7.1 Conclusions

The major conclusions arising from the interim evaluation are as follows:

Relevance

1. There is a continued need for the programs included in the SOPP. According to key informants, there is a strong need for the programming given the importance of the Ontario economy, the significant opportunities for further growth and diversification that exist across a range of existing and emerging clusters, and the key challenges and constraints that are slowing this growth and diversification and need to be addressed. Project proponents, unfunded applicants, non-applicants and beneficiaries rated the need for the programming as high, focusing specifically on challenges to growth and diversification such as the need for capital, funding to support further development, expansion or modernization of operations, and assistance for issues such as technology development, testing and commercialization, market development, staff development and training, and business start-up and early stage development.
2. The SOPP programs are well aligned with each other and other programming available in southern Ontario, the constraints to growth and diversification and needs of the key target groups. Taken together, the suite of SOPP programs employs a variety of delivery mechanisms to promote growth and diversification across various stages of business development, economic clusters, and regions within southern Ontario. Factors such as the place-based nature of FedDev Ontario, the strong demand for funding, and coordination between FedDev Ontario and other programming organizations help to ensure that the SOPP programs complement rather than duplicate other federal or provincial government programs that promote innovation, business development and community development.

Effectiveness of the Programs

3. FedDev Ontario has made significant investments that are incremental and leveraged with funding from other sources. FedDev Ontario funding fills a need that would not have been met by other programs. Only 4 percent of the projects would have proceeded as planned in the absence of FedDev Ontario funding. Each project dollar contributed by FedDev Ontario was leveraged with $2.45 in funding from other sources. According to the project budgets, the primary sources of funding include private sector proponents (49 percent of funding), FedDev Ontario (29 percent), not-for-profit proponents (7 percent), contribution by private sector partners (6 percent) and the provincial government (6 percent).
4. The SOPP programs have contributed towards achievement of the priorities of the Government of Canada, including the Innovation and Skills Plan, as well as the mandate of FedDev Ontario. Based on the results of consultations undertaken for the Innovation and Skills Plan, the Government of Canada identified three priority areas: people (ensuring that people are equipped with the right skills and experience to drive innovation); technologies (taking full advantage of transformative emerging technologies that can elevate the competitiveness of established and new firms, industries, and clusters); and companies (growing the next generation of global companies in Canada)^{Footnote 34}. The investments made by FedDev Ontario are consistent with these priorities, particularly with respect to:
   • Attracting, developing and retaining highly skilled workers, researchers and entrepreneurs. Most projects have involved training or mentoring and proponents report that the projects have helped southern Ontario attract and retain key workers, researchers and entrepreneurs. Training is provided by a mixture of individual companies, which undertake training of their own staff (often as part of a manufacturing expansion or modernization project) and intermediaries funded by FedDev Ontario which provide training and mentoring for highly skilled workers, researchers, entrepreneurs and investors.
   • Strengthening the regional innovation ecosystem through the further development of research and development capacity, investments in technology development, testing and commercialization, and facilitating the development of collaborations and partnerships involving investors, research collaborators, project partners, businesses and others. FedDev Ontario has made significant investments in clusters identified by the Government of Canada as priorities. The funding targets a range of existing and emerging economic clusters. Manufacturing was a focus of 60 percent of approved FedDev Ontario contributions (37 percent could be categorized as advanced manufacturing). Other leading clusters include health care and biotech (26 percent of approved funding), ICT (13 percent), agri-food (8 percent) and clean tech and clean resources (7 percent).
   • Accelerating the start-up, early development, expansion and modernization of companies by attracting and facilitating investment, supporting technology adaptation, adoption and commercialization, supporting advisory services and market development activities and attracting anchor firms. For example, FedDev Ontario projects are contributing to the development of partnerships and collaborations, angel investments totaling over $375 million, and delivery of advisory and support services (ranging from business development services, technology development and testing services to training and other resources) to over 2,900 businesses.

   In addition, the programs have also supported community economic development and diversification. With the exception of the EODP, the programs were open to applicants from all regions within southern Ontario. Eastern Ontario has been identified as a specific priority for funding because of weak economic conditions, which contribute to a loss of businesses, investment and youth from the region. Of the 190 projects, 31 targeted the eastern Ontario region (excluding Ottawa and Kingston), accounting for 12 percent of approved funding.

   The activities supported under SOPP have also directly contributed to the Agency's core mandate of strengthening southern Ontario's economic capacity for innovation, entrepreneurship and collaboration and promoting development of a strong and diversified southern Ontario economy. The 65 projects which have been completed to date report increased sales of $241 million ($2.54 in increased revenues for every dollar provided by FedDev Ontario) while the 125 projects that were still ongoing were projected to generated $3.7 billion in increased North American sales, almost $600 million in export sales outside North America and nearly $1.1 billion in sales from the commercialization of 1,500 new products and technologies.

   The impacts of the projects will continue to grow over time. Most projects resulted in improvements to operations, facilities, equipment and business practices or further development of the innovation capacity, the benefits of which will continue on well beyond the end of the project.

Program Design and Delivery

5. Operating costs as a percent of grants and contributions are low. Operating expenditures averaged 4.9 percent of program expenditures to March 31, 2017. Two factors contributing to the low percentage are increased average approved contributions per project and increased use of third parties to administer programs. The FedDev Ontario contribution per project averaged $3.62 million for the SOPP programs, an increase of 150 percent over the contributions made under similar programs in previous mandates.

   Some reservations, from within the organization and externally, were expressed that the programs' operating budgets may have become too lean, particularly given the large number of legacy files that still require monitoring. Staffing constraints can slow the processing of applications, contribute to data challenges, and impact on project monitoring.

6. Most proponents are satisfied with the design and delivery of the programs. Proponents report satisfaction in their dealings with FedDev Ontario staff and believe the design and delivery of the programs is appropriate, they were given sufficient time to complete the project, they received clear direction regarding the development of their proposal, and the application requirements and criteria are appropriate. The most significant concerns are the length of the application and approval process (and the unpredictability of the timing), the reporting requirements, and usefulness of the performance measurement data. Some proponents also indicated challenges with the SR&ED tax credits. Given they invested in the process but were not successful in accessing funding, non-funded applicants tended to be less supportive of program design and delivery, particularly the length of the approval process, their dealings with FedDev Ontario staff, and the guidance and direction provided to them with respect to the preparation of the proposal.
7. The most serious issue regarding program design involves the five-year funding profile of FedDev Ontario. The five year timeline means that multi-year projects can only be approved early in the mandate. Given that almost three quarters of the funding (to May 24, 2017) was approved for projects of more than three years in duration, little funding remained available for approvals in the latter years. The five-year mandate also created significant workload issues for program staff, particularly in the first year of the new mandate. During that first year, program staff need to engage with prospective applicants, and review and process applications with a particular focus on the larger scale, multi-year projects which tend to be more complex in nature. At the same time, the staff must deal with legacy projects from the previous mandate, reviewing final reports, conducting site visits, validating reported results, and preparing close-out reports. These challenges were further complicated by having the budget for the SOPP programs divided relatively evenly across the five-year period. Reflecting the time required to approve, contract and launch new projects, actual program expenditures amounted to only 38 percent of the planned expenditures in 2014-15.

7.2 Recommendations

The recommendations arising from the evaluation are as follows:

1. Develop a formal plan for addressing the issues related to the five-year funding profile.

   The preferred option is to move to a longer-term funding model for the Agency or, if that cannot be achieved, take steps to mitigate some of the impacts. Mitigating measures could include by (1) having additional trained staff resources in place for the first year of the new mandate so that the Agency is better able to process and approve new projects while still being able to effectively monitor and close-out legacy projects; and (2) allocating the program budgets so that planned expenditures are lower in year one than in subsequent years. Another option for subsequent years may be to move to a rolling funding model where FedDev Ontario secures funding for additional years part way through its next mandate, such that the program always has three to five years of funding remaining.

2. Maintain the same fundamental program structure for the next mandate, while exploring opportunities to refine and consolidate programs to address the current challenges and needs of the region.

   There is strong support within and outside the organization for maintaining the existing program structure given that the existing suite of programs is effectively designed, coordinated and delivered and doing so will enable the agency to build off the program awareness created over the past four years. Individual programs may need to be adjusted or adapted somewhat to reflect changing priorities of the federal government, the role of FedDev Ontario within ISED, and issues identified in the evaluation. In refining the individual programs, consideration could be given to:

   • Targeting under-represented groups including Indigenous people, women and young entrepreneurs.
   • Better allocation of EODP funding. Steps could be taken to better align EODP funding (through a pooling of funding or directly providing differing levels of funding) with the regional demand for funding, areas of particular need, and capacity of CFDCs to deliver programming. Consideration could also be given to opening EODP style funding to CFDCs in other regions and giving CFDCs the option of offering repayable contributions and, where warranted, larger individual contributions.
   • Targeting MNEs. Attracting significant investments from MNEs and building linkages between those anchor organizations, SMEs, and research, development and commercialization centres can be a very effective cluster development strategy. However, availability of low cost capital through a program like AMF may not be an effective incentive to encourage that investment, and alternatives should be examined.
   • Refining the structure of the unconditionally repayable contributions such that they are more likely to be treated as loans by CRA.
   • Providing greater flexibility with respect to the program guidelines. While the basic program structure should be continued, consideration could be given to increasing the flexibility of the programming. The program guidelines should focus more on the intended outcomes and be less prescriptive in how projects must be structured to achieve those outcomes.

3. Offer potential applicants a single point of entry and regularly update publicly available information related to funding availability and timelines.

   It can be difficult for potential applicants to determine under which, if any programs, they may be eligible. An online form could be used to assess eligibility and guide prospective applicants to the appropriate program. Applicants requested that additional information be publicly available on service standards, the balance of funding available for project approvals, the success rate of applications, and the range in timelines to decision.

4. Support the continued development of project officers.

   Program results can be directly impacted by the experience, knowledge and expertise of the project officers. Concerns were expressed about the rate of turnover. A strategy should be developed to both reduce the rate of turnover in the positions and accelerate the professional development of project officers through mechanisms such as the Professional Practice pilot. Consideration could also be given to augmenting internal resources by contracting with one or more outside agencies to assist in the review of certain aspects of project applications.

5. The project reporting system should be reviewed and revised, both in terms of reporting process and the indicators on which proponents report.

   Consideration should be given to fully digitizing project files from cradle to grave (from expressions of interest to submission and review of proposals, implementation of projects, and project monitoring). The existing system of scanned documents and multiple excel files is cumbersome for applicants, proponents, officers, evaluators, and decision-makers, requires regular reentering of data, complicates validation, and restricts the ability to report on projects, proponents, and progress at the program and Agency level.

   The performance indicators should be refined to reflect the new departmental results framework and facilitate useful reporting on a broader range of results relevant to specific projects. An online system could enable the proponent to more easily report on key departmental results as well as indicators specifically relevant to their project (using skip patterns to adapt the indicators by project), which could then be validated (e.g. using automated procedures and personal follow-up where needed). The data could be rolled up easily to report on results by program, region, or sector or whatever parameter is of interest to FedDev Ontario managers on an on-going basis or in response to specific requests.

Annex I: Evaluation Methodology

I.1 Approach and Lines of Evidence

The evaluation was undertaken in three phases: Planning, Data Collection, and Synthesis, Analysis and Reporting. This evaluation used a hybrid team approach (internal evaluators and external consultants) in implementing a mixed-methods research design involving multiple lines of evidence. The following table outlines the roles of the FedDev Ontario Evaluation Directorate and GGI in undertaking the evaluation.

The planning phase involved detailed documentation review (on FedDev Ontario, its programs, and the funded projects to identify the data available and potential sources of further information) and development of the evaluation matrix, methodology, data collection instruments and communication protocols.

The purpose of the Data Collection phase was to systematically gather data and assemble the evidence. The secondary data sources included:

• Document and literature review: A comprehensive review, focused primarily on issues related to relevance, was undertaken of internal and external documents related to the programs, innovation and commercialization, federal policies and strategies, and previous evaluations.
• Review of project and operational data: Project data was used to develop a statistical profile of funded projects, client organizations, partnerships, intended and reported impacts, and the inter-relationship between the various programs. To do so, a detailed database of funded and unfunded projects was developed, drawing from various other databases and augmented by the results of a very extensive document review conducted by both GGI and the Evaluation Directorate. In addition, operational data regarding resource allocations was reviewed and used in assessing efficiency and economy.

The primary data sources included surveys, key informants and case studies. The surveys included:

• Survey of proponents: A web-based and telephone survey was undertaken, targeting the proponents of projects funded through SOPP. The survey obtained input on implementation and project results as well as perceptions of the program. Excluding duplicates (those involved in multiple projects) and those who could not be contacted, 117 representatives were contacted by email or telephone. Attempts were made to contact each of those who had not yet responded at least five times. Of these proponents, 117 responded which represents 65 percent of the proponents who were reached (only three formally declined) and 62 percent of the total population. Respondents most commonly included directors and CEOs (56 percent), project managers and project leads (27 percent), and heads of finance or controllers (20 percent). At a confidence level of 95 percent, the 117 respondents achieve a margin of error of about ±5.6 percent. The survey results were then linked with data from the project database for the purpose of detailed analysis.
• Survey of applicants not approved for funding under the SOPP. The survey obtained input on whether the project was implemented without FedDev Ontario funding, perceived need for the programming, program design and delivery, and opportunities for improvement. A list was developed of 107 organizations that applied but were not recommended for funding, of which 92 were contacted. Attempts were made to contact each of those who had not yet responded at least five times. Forty non-funded applicants responded, representing 43 percent of those reached and 37 percent of the population list.
• Potential applicants who had not applied for funding. Using various sources, a list was developed of approximately 150 companies and non-profit organizations that appeared similar to proponents who had been funded under the SOPP but that had not applied. It was difficult to generate a response because, in many cases, these representatives felt that they were not familiar enough with the FedDev Ontario or its programming to respond or simply willing to take the time to respond. In total, 115 potential applicants were reached, of which 28 responded (24 percent).
• Project beneficiaries, consisting of organizations that received financial assistance or other services, delivered by a third-party organization and funded by FedDev Ontario. A sample of over 1,600 beneficiary organizations was developed using names and contact information provided by third-party organizations to FedDev Ontario. In total, 365 organizations were surveyed online or by telephone including 48 assisted by projects supported through the EDOP Community Innovation program, 150 supported through the EODP Business Development Program, 60 supported through the EODP CEDP program, and 107 supported though the IBGP program. At a confidence level of 95 percent, the sample of 365 respondents achieves a margin of error of about ±4.5 percent.

In total, 550 people were surveyed as summarized in the table below.

Interviews were conducted with 65 key informants including:

• Eight management representatives of FedDev Ontario (one Vice-Present, one Director General, one Director and five Managers);
• Twenty-five representatives of other government departments and partners (fourteen representatives from regional or municipal development organizations, five representatives from the Government of Ontario, and six representatives from other federal government departments);
• Seventeen other stakeholders and experts (six representatives from industry associations, six from industry, two from innovation focused organizations, two from post-secondary institutions, and one venture capitalist); and
• Fifteen project proponents who were followed-up on to discuss key design and delivery issues raised in the proponent survey.

The primary focus was to obtain input on the need for the programming, relationship to other programs, factors that contribute to and constrain achievement of the intended outcomes, and opportunities for improvement.

Six case studies were conducted. The projects were selected using a variety of criteria including program, region, size and availability of performance data. The case studies illustrate the nature of the impacts, the lessons learned, contributing and constraining factors. The case studies involved a review of documents and data (e.g., proposals, tracking forms, progress reports, final reports, and publicly available information) complemented with input from proponent organizations, project partners, FedDev Ontario officers and beneficiaries. In total, input from 138 representatives was used in completing the case studies including 8 proponents and partners, 6 FedDev Ontario project officers, and 124 organizations that were surveyed as part of the project beneficiaries' survey.

I.2 Evaluation Matrix

The table on the following page summarizes the recommended performance indicators and data sources for each of the research questions to be addressed in the review.

I.3 Challenges and Mitigation Strategies

The major challenges associated with the evaluation and the mitigation strategies are described below.

• This is an interim evaluation which means that not enough time has elapsed for the impacts of the projects to be fully realized. Of the 190 approved projects, only 34 percent were completed and most of these projects will require additional time post-completion to realize their full impacts. To mitigate the limited data available on project impacts, an extensive review of the projected results was conducted to illustrate expected impacts. In addition, a review of progress reports was completed, and case studies focused primarily on on-going projects Results of three recent Fed Dev Ontario studies were reviewed, which involved examining longer-term impacts of some past projects funded by FedDev Ontario^{Footnote 37}
• The potential for non-response error. Factors contributing to the potential for non-response error include difficulties in reaching some respondents, the completeness and accuracy of some contact information, and the length of time since project completion. Mitigation strategies included multiple times to follow-up with target respondents; offering various modes of response and identification of alternative respondents within the same organization. It was confirmed that the characteristics of the survey respondents were broadly representative of the target population.
• Potential for respondent bias. To mitigate this, a survey of organizations that applied but did not receive funding from FedDev Ontario was conducted. Potential applicants who did not apply as well as key informants who not directly involved in the program were also surveyed. In addition, the survey and interview results were triangulated with data obtained through other lines of evidence.
• Limited knowledge and familiarity with SOPP. Some respondents, particularly those that are not directly involved in the programming, were familiar with the programming and activities at a high level but were unable to comment on more specific impacts or outcomes of FedDev Ontario investments. In these cases, the respondents' opinions were obtained on the needs and gaps with respect to support for technology commercialisation and innovation capacity building programs, the ecosystem of programming available at federal and provincial level, existing levels of collaboration, and recommendations regarding program design and delivery was obtained.

Given these mitigation measures, the evaluation team believes that the limitations of the study were adequately addressed and the results of the evaluation are deemed to be reliable and valid.

Annex II: The SOPP Programs and Logic Model

II.1 Program Activity Architecture

According to the Program Alignment Architecture (PAA)^{Footnote 38} and Agency Performance Measurement Framework (PMF), FedDev Ontario focuses its efforts on four program areas: Technological Innovation, Business Development, Community Economic Development, and Internal Services, as shown in the table below. The SOPP includes the two Technological Innovation sub-programs (AMF and ICP), two of the three Business Development sub-programs (IBI and IBGP), and two Community Economic Development sub-programs (EODP and IRD). The Agency plans to dedicate 222 full-time equivalents (FTEs) and $269.3 million to these activities in 2017-18.

II.2 Overview of the SOPP Programs

Investing in Business Innovation (IBI)

The Investing in Business Innovation (IBI) initiative provides mentorship, entrepreneurial support and financing to help new businesses grow and succeed. The initiative is designed to foster a more competitive southern Ontario economy by focusing on providing business support to new entrepreneurs, helping them transform their ideas into globally-competitive products and services, and increasing their access to private sector investment and advice. The objectives are to foster a culture of entrepreneurship focused on innovation by:

• Supporting start-ups to transform ideas into globally competitive products and services;
• Increasing, stimulating and leveraging private sector investment;
• Strengthening angel networks through improved standards and better investments; and
• Supporting mentorship and skills development activities to help start-ups grow and succeed.

Through the IBI, support could be provided for early stage SMEs, angel investor networks, and the delivery of skills development and seed financing for new entrepreneurs through not-for-profit organizations.

• The early stage SMEs (less than 50 employees) could receive up to $1 million in repayable funding to be leveraged with $2 in angel or VC funds for every $1 in IBI funding for start-ups to transform ideas into globally competitive products and services. The specific terms of repayment, including date of commencement and length of repayment term were determined on a case-by-case basis during the development of the contribution agreement; with repayments begin no later than one year following the completion of the project.
• Angel investor networks located in southern Ontario can receive up to $500,000 in non-repayable contributions for projects to strengthen angel networks through improved standards and better investments (up to 100 percent of direct eligible project costs).
• Not-for-profit organizations could receive up to $20 million for skills development and seed financing for new entrepreneurs (up to 100 percent of direct eligible project costs). The maximum is equal to $10,000 per entrepreneur for business training and $30,000 per SMEs to cover start-up costs (SMEs must provide 50 percent contribution).

Investing in Business Growth and Productivity (IBGP)

The Investing in Business Growth and Productivity (IBGP) initiative focuses on established southern Ontario businesses that have the potential to be global players with innovative and unique opportunities to accelerate growth and support job creation. This initiative supports economic growth and job creation by helping businesses diversify markets and expand facilities, adopt new technologies and processes to improve productivity, and increase business capacity to grow and diversify markets. The objective is to position southern Ontario businesses to be more competitive in the global market by:

• Assisting established businesses with high growth potential;
• Increasing investment in technologies and processes to improve productivity;
• Increasing the capacity of businesses to participate in global markets through exports and integration in global value chains.

Funding could be provided directly to SMEs as well as for services delivered to SMEs by not-for-profit organizations.

• SMEs could receive up to $20 million in repayable funding (up to 25 percent of direct eligible project costs, 50 percent direct eligible capital costs, and 75 percent of direct eligible non-capital costs). The SMEs must have at least 15 employees (no more than 1000 employees), a sustainable business model and a profitable track record with the potential to become a strong global player.
• Not-for-profit organizations (industry associations and/or regional economic development organizations) could receive up to $20 million ($100,000 in non-repayable funding per SME including 100 percent of direct eligible costs) to assist with productivity improvements or increased participation in global markets (SME must provide 50 percent contribution).

Investing in Commercialization Partnerships (ICP)

The Investing in Commercialization Partnerships (ICP) initiative supports business-led partnerships with a focus on developing globally-competitive products and services. Increased collaboration among businesses, post-secondary institutions and research organizations narrows the gap between innovation and commercialization. This initiative helps to increase the capacity of existing and emerging innovation ecosystems and the development of competitive economic clusters in southern Ontario.

The ICP could provide up to $20 million (up to 50 percent of eligible costs; the remaining 50 percent must be provided by other partners) to support business-led partnerships with a focus on developing globally competitive products and services.

Investing in Regional Diversification (IRD)

The Investing in Regional Diversification (IRD) initiative supports the long-term development of stronger, more diverse economies in southern Ontario communities. IRD leverages unique regional assets and local expertise to attract new investment and opportunities for economic growth and development.

Under the IRD, not-for-profit organizations located in southern Ontario whose mandate includes economic development could receive up to $20 million (up to 50 percent of eligible costs; the remaining 50 percent must be provided by the recipient as a cash contribution) to leverage regional assets and local expertise and attract new investment and opportunities for economic growth and development.

The Advanced Manufacturing Fund (AMF)

Established as part of the 2013 Federal Budget, the Advanced Manufacturing Fund (AMF) supports research and innovation organizations, the private sector, post-secondary institutions (PSIs) and not-for-profit organizations to work together to accelerate the development of large-scale, advanced technologies that will result in new market opportunities for Ontario businesses in manufacturing sectors. The objective is to increase firm productivity and enhance the competitiveness of Ontario's advanced manufacturers by:

• Addressing, within the Ontario delivery context, gaps in federal supports for advanced manufacturers;
• Attracting projects that advance the development and/or adoption of cutting-edge technologies leading to product, process, and technological innovation; and
• Creating spillovers for manufacturing clusters and/or supply chains, and fostering collaboration between research institutes, post-secondary institutions and the private sector.

The AMF could provide from $10 million to a normal maximum of $20 million:

• In repayable funding (up to 50 percent of direct eligible costs; other 50 percent from industry) to established, profitable businesses to support the development and/or adoption of cutting-edge technologies leading to innovation and new market opportunities for businesses in the manufacturing sector. Eligible manufacturing firms must be R&D activities in Ontario.
• In non-repayable funding (up to 50 percent of direct eligible costs; other 50 percent from industry) to not-for-profit organizations collaborating with an anchor firm to support the development and/or adoption of cutting-edge technologies leading to innovation and new market opportunities for businesses in the manufacturing sector. Not-for-profit organizations (e.g., research institutions, centres of excellence, or post-secondary institutions) must collaborate with an anchor firm and must demonstrate a significant benefit to the manufacturing sector.

Eastern Ontario Development Program (EODP)

The Eastern Ontario Development Program (EODP) was initially established in 2004 and has been administered by FedDev Ontario since the Agency was established in 2009. It is an economic development initiative aimed at addressing economic challenges in eastern Ontario and taking advantage of innovative opportunities in the region.The program is delivered through eastern Ontario's 15 CFDCs and promotes business development, job creation and strengthened economies in rural eastern Ontario communities. In addition, the EODP provides funding for Collaborative Economic Development Projects (CEDP) which generate benefits for multiple communities and promote broad-based collaborative economic development.

II.3 Program Logic Model

The figure on the following page illustrates the logic model for SOPI, the AMF, and EODP according to the Performance Measurement Strategy, which was developed in February 2014.

Activities

The logic model makes the distinction between the inputs and activities of FedDev Ontario and those of the funding recipients. With inputs such as operating and maintenance (O&M) expenditures, wages, salaries and benefits, and grant and contribution (G&C) funding, the Agency undertakes the following activities:

• Program development, planning and management: The Agency undertakes research, consultation, planning and program management activities that contribute to program design, delivery and administration.
• Non-financial support to southern Ontario stakeholders: In addition to its financial investments in economic development opportunities, the Agency acts as a resource by:
  • Directing stakeholders to relevant FedDev Ontario initiatives and/or to those of other government departments and agencies and other levels of government;
  • Convening key stakeholders in communities and industry sectors to capitalize on economic development opportunities; and
  • Providing guidance and advice to recipients of FedDev Ontario funding to assist them in carrying out the activities and obligations specified in contribution agreements.
• Contribution funding: The Agency provides unconditionally repayable and non-repayable contributions to recipients to carry out activities that will achieve SOPI, AMF and EODP objectives.

With G&C funding inputs, the recipients undertake the following activities:

• Product development and commercialization: This activity is undertaken by recipients of contributions under Investing in Business Innovation (IBI), Investing in Business Growth & Productivity (IBGP), Investing in Commercialization Partnerships (ICP) and the Advanced Manufacturing Fund (AMF).
  • IBI: Recipients undertake pre-commercialization and late-stage product development activities that will enable new businesses to move innovative products, services or processes to market.
  • IBGP: Recipients may undertake product development and commercialization activities that will support business expansion, market diversification and integration into global value chains.
  • ICP: Recipients bring together collaborations of research and innovation organizations, private-sector enterprises, post-secondary institutions and not-for-profit organizations to accelerate the development of globally competitive products and services that will result in new market opportunities for southern Ontario businesses.
  • AMF: Recipients undertake product development and commercialization activities including prototyping, demonstration projects, advanced product testing, and applied research leading to practical applications.
• Support to businesses/entrepreneurs: This activity is undertaken by recipients of contributions under all SOPI, AMF and EODP initiatives:
  • IBI: IBI supports not-for-profit organizations that will in turn support the development of entrepreneurs, help them to launch new start-up enterprises and develop investment-ready businesses. IBI will also provide direct support to early-stage businesses to undertake a variety of activities that will accelerate growth, create jobs, and diversify markets.
  • IBGP: IBGP supports eligible SMEs to undertake activities related to adapting or adopting new technologies, processes, and related skills development; business opportunity development, growth and integration in global value chains; facilities improvement or expansion; market development and expansion; and business expansion to support greater economic diversification.
  • ICP: Recipients will support businesses to develop globally-competitive products and services through increased collaboration with post-secondary institutions and research organizations and will increase the capacity of existing and emerging innovation ecosystems in southern Ontario.
  • IRD: Recipients provide support to strengthen regional businesses and clusters with the goal of economic diversification and sustainability.
  • AMF: The AMF supports Ontario manufacturers to undertake manufacturing and R&D activities related to prototyping, demonstration projects, advanced product testing, and applied research; improvements to existing materials, devices, products or processes, as well as the adoption or adaptation of highly innovative products, technologies, and processes that support product or process innovation.
  • EODP: Recipients of funds under the Business Development component of EODP provide financial support to new businesses and to support the growth of existing businesses through activities such as productivity enhancements, market diversification, product development and succession planning.
• Productivity improvement/process innovation: This activity is undertaken by recipients of contributions under the AMF, ICP and IBGP.
  • IBGP: Under the third-party delivery stream of IBGP, industry or sector associations further distribute contributions to SMEs to adapt/adopt new technologies, processes and skills that will enhance business productivity in their sector or industry.
  • AMF: Recipients under AMF receive support for the adoption or adaptation of highly innovative products, technologies (e.g., machinery and equipment), and processes that support product or process innovation leading to enhanced productivity.
  • ICP: Recipients will support businesses in the development. adoption or adaptation of highly innovative products, technologies (e.g., machinery and equipment), and processes that support product or process innovation leading to enhanced productivity.
• Third-party delivery: This activity is undertaken by recipients of IBI, IBGP, ICP, IRD and EODP.
  • IBI: Not-for-profit recipients of contributions under IBI will provide skills development, education, and seed financing to new entrepreneurs and businesses to improve their investment readiness.
  • IBGP: Not-for-profit recipients of contributions under IBGP will provide support to SMEs to adapt/adopt new technologies, processes and skills that will enhance business productivity in their sector or industry.
  • ICP: ICP will support not-for-profit organizations and post-secondary institutions to work with SMEs to undertake prototyping, demonstration projects, advanced product development, and applied research leading to practical commercial applications.
  • EODP: Under the Business Development stream of EODP, not-for-profit organizations will deliver support to promote the growth of new and existing businesses in rural eastern Ontario communities.

Outputs

The Agency outputs include:

• Guidelines, resources, reports: Program development, planning and management activities result in the creation of new initiatives and associated policies and practices that are intended to foster economic development.
• Outreach, information sessions, and advice: The extent to which the Agency provides path-finding services and other resources that support stakeholders to undertake economic development activities is reflected in the number of outreach activities, information sessions and other forms of advisory services provided.
• Networks and collaborations: Non-financial support to economic stakeholders is also reflected in the number of networks and collaborations the Agency facilitates.
• Approved projects/Contribution agreements: The Agency enters into contribution agreements with eligible recipients to support projects that will stimulate local economies and enhance the growth and competitiveness of local businesses and communities.

Recipient outputs are as follows:

• Partnerships/collaborations: All of the initiatives under the umbrella of the SOPI, AMF and EODP terms and conditions include outreach activities, partnerships and collaborations with stakeholders in economic development.
• Training/mentorship for entrepreneurs: IBI projects delivered through not-for-profit organizations will support the development of entrepreneurs, helping them to launch new start-up enterprises and supporting them to become investment-ready businesses. Early-stage businesses that receive direct funding support through IBI will also receive mentorship and support through angel and venture capital investors.
• Investments leveraged against FedDev Ontario contributions: It is anticipated that recipients of funding under all of the SOPI, AMF, and EODP will be able to use contribution funding from FedDev Ontario to leverage funds from third parties, including other federal departments, other levels of government, angel/venture capital investors, and private-sector partners.
• Businesses/organizations supported: All of the initiatives provide support to businesses, not-for-profit organizations or post-secondary institutions in the form of funding or technical/advisory support that will assist the Agency to accomplish its longer-term goals of improving the economic status of southern Ontario communities and the competitiveness of businesses.

Immediate Outcomes

The outputs are expected to result in a number of immediate outcomes (expected to be manifested in the first 1 to 2 years of project activities), including:

• Increased investment in research and development: Recipients receiving contributions through IBI and ICP and some IBGP direct-to-business and AMF projects will receive support to undertake research and development (R&D) and commercialization activities, including product and process applied research, engineering design, technology acceleration, product testing, certification, marketing studies, proof of concept, and piloting and demonstration activities. These contributions will in turn leverage further investment in R&D and commercialization activities from participating organizations, their partners and other funding organizations.
• Increased investment in community economic capacity: Recipients of contributions under IRD and under the Community Innovation and Community Economic Development components of EODP will receive support to diversify local economies that will in turn leverage further community investments to support local economic capacity.
• Increased access to capital: New enterprises participating in IBI and businesses engaged in IBGP, AMF and EODP Business Development projects will have increased access to capital to support their business development activities. Angel investment networks and their associations receiving non-repayable contributions through IBI will support this outcome by attracting new investments to southern Ontario angel networks.
• Increased investment in productivity improvements: Projects funded through the IBGP, ICP and AMF initiatives will result in investments that will facilitate the adoption and adaptation of new productivity-enhancing technologies.

Intermediate Outcomes

The immediate outcomes are expected to lead to the following intermediate outcomes within two to five years of support to projects:

• Increased commercialization of research: It is anticipated that the new products, services and processes developed as a result of investments in research and development activities undertaken by IBI, IBGP, ICP, and AMF projects will be commercialized and enter the market.
• Increased employment opportunities: Increased investments in R&D and community economic capacity and improved access to capital to undertake activities that will lead to business growth are in turn expected to contribute to the creation and retention of jobs in projects supported through all SOPI, AMF and EODP initiatives.
• Increased value and diversity of markets^{Footnote 39}: Businesses receiving increased investment in R&D and community economic capacity and improved access to capital through IBI, IBGP, and EODP Business Development projects are expected to benefit through growth that will result in increased sales and market diversity.
• Enhanced business productivity: Technologies adapted or adopted by businesses participating in IBGP, ICP and the AMF are expected to result in improved productivity.
• Improved survival rate of new businesses: New enterprises receiving capital and business advisory support through IBI are expected to have better survival rates than comparable businesses that have not received similar support. The performance measurement strategy will ensure collection of information about the survival rate or successful exit of new businesses to the end of the project lifecycle. In addition, start-up businesses receiving direct support through IBI generally repay their contributions over a two- to three-year period following project end. This will allow the Agency to continue to monitor the survival and or successful exit of individual businesses through annual financial reports during the control period. Finally, the collection of business numbers will enable the Agency to undertake longer-term follow-up of businesses receiving both direct support and support through intermediary not-for-profit organizations as a whole (not individually) as part of the overall program evaluation (i.e., through Business Registry data).

Ultimate Outcomes

Ultimate outcomes are generally associated with changes in societal conditions, are often subject to influences beyond the initiative itself and, as a result, take a longer time to be realized. The above intermediate outcomes are expected to result in the following ultimate outcomes in the longer term:

• Improved economic status of southern Ontario Communities: Diverse regional economies, a greater share of knowledge-based industries, and new and stronger start-up enterprises and SMEs are anticipated to result in more and larger businesses and increased employment opportunities in southern Ontario communities.
• More competitive businesses: The commercialization of new products, services and processes; more diversified markets; enhanced productivity and a talented labour force are expected to result in the improved competitiveness of businesses.