Interim Evaluation of the Southern Ontario Prosperity Program

Final Report
Prepared for FedDev Ontario December 15, 2017

Table of Contents

Management Response and Action Plan

Annexes

List of Tables and Figures

Acronyms

AAFC
Agriculture and Agri-Food Canada
AIME
Achieving Innovation and Manufacturing Excellence Program
AMF
Advanced Manufacturing Fund
APMF
Agency Performance Measurement Framework
ARC
Applied Research and Commercialization
BCA
Building a Competitive Advantage
BDC
Business Development Bank of Canada
BERD
Business Expenditures in Research and Development
CEDP
Collaborative Economic Development Projects
CFDCs
Community Futures Development Corporation
CFI
Canadian Foundation for Innovation
CME
Canadian Manufacturers & Exporters
CIHR
Canadian Institute of Health Research
CSSCTP
College Strategic Sector/Cluster/Technology Platform
EDC
Export Development Canada
EODP
Eastern Ontario Development Program
FedDev Ontario
Federal Economic Development Agency for Southern Ontario
FBI
Food & Beverage Initiative
FDI
Foreign Direct Investment
FTEs
Full-time Equivalents
GAC
Global Affairs Canada
GDP
Gross Domestic Product
GEI
Graduate Enterprise Internship
G&C
Grant and Contribution
IBGP
Investing in Business Growth and Productivity
IBI
Investing in Business Innovation
ICP
Investing in Commercializing Partnerships
ICT
Information and Communications Technology
IRAP
Industrial Research Assistance Program
IRCC
Immigration, Refugees and Citizenship Canada
IRD
Investing in Regional Diversification
ISED
Innovation, Science and Economic Development Canada
IRID
Investing in Regional Innovation and Development
JPC
Jobs and Prosperity Council
MEDG
Ministry of Economic Development and Growth
MNEs
Multinational Enterprises
MRIS
Ministry of Research, Innovation and Science
NFP
Not-for-Profit
NRCan
Natural Resources Canada
NSERC
National Sciences and Engineering Research Council
OBIO
Ontario Bioscience Innovation Organization
OCC
Ontario Chamber of Commerce
OCE
Ontario Centres of Excellence
OMAFRA
Ontario Ministry of Agriculture, Food and Rural Affairs
ONE
Ontario Network of Entrepreneurs
PAA
Program Alignment Architecture
PAC
Project Advisory Committee
PE
Productivity Enhancement
PI
Prosperity Initiative
PLM
Program Logic Model
PSI
Post-secondary institution
R&D
Research and Development
RDAs
Regional Development Agencies
SDTC
Sustainable Development Technology Canada
SME
Small and Medium Enterprises
SR&ED
Scientific Research and Experimental Development Tax
SEB
Scientists and Engineers in Business
SODF
Southwestern Ontario Development Fund
SODP
Southern Ontario Development Program
SOPP Program
Southern Ontario Prosperity Program
SOPIs
Southern Ontario Prosperity Initiatives
SOSCIP
Southern Ontario Smart Computing Innovation Platform
SOWC
Southern Ontario Water Consortium
SSHRC
Social Sciences and Humanities Research Council
TDP
Technology Development
UOIT
University of Ontario Institute of Technology
YLF
Yves Landry Foundation
Y-STEM
Youth in Science, Technology, Engineers and Mathematics

Interim Evaluation of the Southern Ontario Prosperity Program (SOPP) – Infographic

PDF version

Findings

Relevence

Government of Canada priorities
  • Innovation & Skills plan
  • Compliments other Government of Canada programs

Impact

  • $2.45 leveraged for every $1 invested
    • 10,000 FTEs created
  • $2.54 increase in revenue for every $1 invested
    • 2,900 business supported
  • $2.38 in angel investment for every $1 invested

Expected Future Impact (including on-going projects)

  • 33,000 FTEs created or maintained
  • $4 B in North American sale
  • $600 M in exports
  • $150 M in angel investment
  • Commercialization of 1,500 new products and technologies

Scope

  • Eastern Ontario Development Program (EODP)
  • Advanced Manufacturing Fund (AMF)
  • Investing in Business Innovation (IBI)
  • Investing in Business Growth and Productivity (IBGP)
  • Investing in Commercialization Partnerships (ICP)
  • Investing in Regional Development (IRD)
  • 190 projects approved
  • Up to $688 M invested
  • 66% of projects still in progress as of May 24, 2017

Methodology

Secondary Lines of Evidence

  • Document & literature review;
  • Review of program & operational data

Surveys

61% response rate.

  • 117 approved projects
  • 40 non-approved projects
  • 28 potential applicants
  • 356 beneficiarie

Case Studies

  • 6 completed

Interviews

  • 64 completed

Going Forward

  • Move to longer term ongoing funding mode
  • Maintain fundamental program structure; continue to align with Government of Canada priorities
  • Streamline application process
  • Continue development of program officers
  • Review and revise project reporting system

Executive Summary

Purpose and Method of Study

In August 2009, the Government of Canada created the Federal Economic Development Agency for Southern Ontario (FedDev Ontario) with a mandate to strengthen southern Ontario's economic capacity for innovation, entrepreneurship and collaboration, and promote the development of a strong and diversified southern Ontario economy. The Southern Ontario Prosperity Program (SOPP) served as the Agency's core program for its second five-year mandate (2014–15 to 2018–19). The SOPP consists of the Eastern Ontario Development Program (EODP), the Advanced Manufacturing Fund (AMF) and four programs under the Southern Ontario Prosperity Initiatives (SOPIs): the Investing in Business Innovation (IBI) initiative; the Investing in Business Growth and Productivity (IBGP) initiative; the Investing in Commercialization Partnerships (ICP) initiative; and the Investing in Regional Diversification (IRD) initiative. As of May 24, 2017, 190 projects had been approved under these programs, with approved funding totaling $688 million.

The purpose of this study was to conduct an interim evaluation of the SOPP, with a focus on its relevance, effectiveness, and design and delivery. This evaluation used a hybrid team approach (involving internal evaluators from FedDev Ontario and external consultants from Goss Gilroy Inc.) in implementing a mixed-methods research design involving multiple lines of evidence. Key lines of evidence included document and literature reviews (focused primarily on program relevance); project and program financial data; surveys of 117 project proponents, 40 applicants not approved for funding, 28 representatives from organizations eligible to apply for funding but which did not, and 365 beneficiary organizations that received financial or other assistance funded by FedDev Ontario but delivered by a third-party organization; interviews with 64 key informants; and case studies of six projects.

This is an interim evaluation. Data will not be available on the full impacts of the projects for several years. Of the 190 projects, only 34 percent were completed at the time of the evaluation and the completed projects are expected to continue generating impacts that extend beyond the term of their funding. Recognizing that only limited data is available to date on project outcomes, the evaluation also incorporated an extensive review of projected results to illustrate the impacts expected to be generated, reviews of progress reports, case studies of a sample of on-going projects, and a review of three recent FedDev Ontario studies which examined the longer-term impacts of a sample of past projects funded by FedDev Ontario.

Relevance

There is a continued need for the programs included in the SOPP. Key informants attribute the strong need to the importance of the Ontario economy, the significant opportunities for further growth and diversification that exist across a range of existing and emerging clusters, and the key challenges and constraints that are slowing this growth and diversification and need to be addressed. Project proponents, unfunded applicants, non-applicants and beneficiaries rated the need as high, noting how the programs help address challenges to economic development such as the need for capital and assistance for issues such as technology development and adoption, testing and commercialization, market development, staff development and training, business start-up and early stage development.

The SOPP programs are well aligned with each other and other programming available in southern Ontario, the constraints to development and needs of the key target groups. Taken together, the suite of SOPP programs employs a variety of delivery mechanisms to promote growth and diversification across various stages of businesses development, economic clusters, and regions within southern Ontario. Factors such as the place-based nature of FedDev OntarioFootnote 1, the strong demand for funding, and coordination between FedDev Ontario and other programming organizations help to ensure that the SOPP programs complement rather than duplicate other federal or provincial government programs that promote innovation, business development and community development. The SOPP also supports the framework for the Innovation and Skills Plan (ISP) as well as Investing in Regional Innovation and Development (IIRD)Footnote 2.

Effectiveness of the Programs

FedDev Ontario has made significant investments that are incremental and leveraged funding from other sources. FedDev Ontario funding fills a need that would not have been met by other programs. Only 4 percent of the projects would have proceeded as planned in the absence of FedDev Ontario funding. Each project dollar contributed by FedDev Ontario was leveraged with $2.45 in funding from other sources. The funding has targeted a range of existing and emerging economic clusters including manufacturing (associated with 60 percent of approved FedDev Ontario contributions, of which 37 percent could be categorized as advanced manufacturing), health care and biotech (26 percent), Information and Communications Technology (ICT) (13 percent), agri-food (8 percent) and clean tech and clean resources (7 percent).

The projects have contributed towards achievement of the priorities of the Government of Canada, including the Innovation and Skills Plan, as well as the mandate of FedDev Ontario. The investments have helped to:

The activities supported under SOPP have directly contributed to the Agency's core mandate of strengthening southern Ontario's economic capacity for innovation, entrepreneurship and collaboration and promoting development of a strong and diversified southern Ontario economy. The 65 projects which have been completed to date report increased sales of $241 million ($2.54 in increased revenues for every dollar provided by FedDev Ontario) while the 125 projects that are still ongoing are projected to generate $3.7 billion in increased North American sales, almost $600 million in export sales outside North America and nearly $1.1 billion in sales from the commercialization of 1,500 new products and technologies.

The impacts of the projects will continue to grow over time. Most projects resulted in improvements to operations, facilities, equipment and business practices or further development of the innovation capacity, the benefits of which will continue on well beyond the end of the project.

Program Design and Delivery

Most project proponents are satisfied with the design and delivery of the programs. Proponents report satisfaction in their dealings with FedDev Ontario staff and believe the design and delivery of the programs is appropriate, they were given sufficient time to complete the project, they received clear direction regarding the development of their proposal, and the application requirements and criteria are appropriate. The most significant concerns are the length of the application and approval process, the unpredictability of the timing of these steps, the reporting requirements, and usefulness of the performance measurement data. Given they invested in the process but were not successful in accessing funding, non-funded applicants tended to be less supportive of program design and delivery, particularly the length of the approval process, their dealings with FedDev Ontario staff, and the guidance and direction provided to them with respect to the preparation of their proposal.

Operating costs as a percent of grants and contributions are low relative to historical figures for the programs. Operating expenditures averaged 4.9 percent of program expenditures to March 31, 2017. Two factors contributing to the low percentage are an increase in average approved contributions per project and increased use of third parties to administer programs. Some reservations, from within the organization and externally, were expressed that the programs' operating budgets may have become too lean, particularly given the large number of legacy files that still require monitoring. Staffing constraints can slow the processing of applications; contribute to data challenges; increase staff turnover; and impact project monitoring.

The most serious issue regarding program design involves the five-year funding profile of FedDev Ontario. Three quarters of the funding (to May 24, 2017) was approved for projects of more than three years in duration. The five-year timeline means that most projects needed to be approved early in the mandate and little funding remained available for approvals in the later years. The five-year mandate also creates significant workload issues for program staff, particularly in the first year of the new mandate. During the first year, program staff are engaging with prospective applicants, and reviewing and processing applications with a particular focus on the larger scale, multi-year projects which tend to be more complex in nature. At the same time, the staff must also deal with legacyFootnote 4 projects from the previous mandate, reviewing final reports, conducting site visits, validating reported results, and preparing close-out reports. These challenges are further complicated by having program budgets divided relatively evenly across the five-year period. Reflecting the time required to approve, contract and launch new projects, actual program expenditures amounted to only 38 percent of the planned expenditures in 2014-15.

Recommendations

The recommendations arising from the evaluation are as follows:

  1. Develop a formal plan for addressing the issues related to the five-year funding profile. The preference is to move to a longer-term funding model for the Agency or, if that cannot be achieved, take steps to mitigate some of the impacts. Mitigating measures could include (1) having additional trained staff resources in place for the first year of the new mandate so that the Agency is better able to process and approve new projects while still being able to effectively monitor and close-out legacy projects; and (2) allocating the program budgets so that planned expenditures are lower in year one than in subsequent years. Another option for subsequent years may be to move to a rolling funding model where FedDev Ontario secures funding for additional years part way through its next mandate, such that the program always has three to five years of funding remaining.
  2. Maintain the same fundamental program structure for the next mandate, while exploring opportunities to refine and consolidate programs to address the current challenges and needs of the region. There is strong support within and outside the organization for maintaining the existing program structure given that the existing suite of programs is effectively designed, coordinated and delivered and doing so will enable the Agency to build off program awareness created over the past four years. Individual programs may need to be adjusted or adapted somewhat to reflect changing priorities of the federal government, the role of FedDev Ontario within ISED, and issues identified in the evaluation.
  3. Offer potential applicants a single point of entry and regularly update publicly available information related to funding availability and timelines. It can be difficult for potential applicants to determine under which, if any programs, they may be eligible. An online form could be used to assess eligibility and guide prospective applicants to the appropriate program. Applicants also requested that additional information be publicly available on service standards, the balance of funding available for project approvals, the success rate of applications, and the range in timelines to decision.
  4. Support the continued development of project officers. Program results can be directly impacted by the experience, knowledge and expertise of the project officers. The level of turnover has been high. A strategy should be developed to both reduce the level of turnover in the positions and accelerate the professional development of project officers through mechanisms such as the Professional Practice pilot. Consideration could also be given to augmenting internal resources by contracting with one or more outside agencies to assist in the review of certain aspects of project applications.
  5. The project reporting system should be reviewed and revised, both in terms of the reporting process and the indicators on which proponents report. Consideration should be given to fully digitizing project files from cradle to grave (from expressions of interest to submission and review of proposals, implementation of projects, and project monitoring). The existing system of scanned documents and multiple excel files is cumbersome for applicants, proponents, officers, evaluators, and decision-makers; requires regular reentering of data; complicates validation; and restricts the ability to report on projects, proponents, and progress at the program and Agency level.

    The performance indicators should be refined to reflect the new departmental results framework and facilitate useful reporting on a broader range of results relevant to specific projects. An online system could enable the proponent to more easily report on key departmental results as well as indicators specifically relevant to their project (using skips patterns to adapt the indicators by project), which could then be validated (e.g. using automated procedures and personal follow-up where needed). The data could be rolled up easily to report on results by program, region, or sector or whatever parameter is of interest to FedDev Ontario managers on an on-going basis or in response to specific requests.

1. Introduction

1.1. Background

The Government of Canada created the Federal Economic Development Agency for Southern Ontario (FedDev Ontario) in 2009 with a mandate to strengthen southern Ontario's economic capacity for innovation, entrepreneurship and collaboration; and promote the development of a strong and diversified southern Ontario economy. The Southern Ontario Prosperity Program (SOPP), which served as the Agency's core program for its second five-year mandate (2014–15 to 2018–19), consists of the Eastern Ontario Development Program (EODP), the Advanced Manufacturing Fund (AMF) and the four Southern Ontario Prosperity Initiatives (SOPIs): Investing in Business Innovation (IBI); Investing in Business Growth and Productivity (IBGP); Investing in Commercialization Partnerships (ICP); and Investing in Regional Diversification (IRD).

1.2. Purpose of the Evaluation

The objective is to conduct an interim evaluation of the SOPP. Under the 2016 Policy on Results, evaluations are to be planned with consideration of using relevance, effectiveness and efficiency (design and delivery) as primary evaluation issues, where relevant to the goals of the evaluation (Directive on Results, C.2.2.1.5). For this evaluation, a series of evaluation questions were developed and grouped under those three issues.

Table 1: Evaluation Issues and Questions
Issue Evaluation Questions
Relevance
  1. To what extent do the SOPP programs continue to address a demonstrable need?
  2. To what extent do the SOPP programs complement, duplicate, or overlap other government programs?
  3. To what extent is the SOPP aligned with current government priorities, including the elements of the Innovation and Skills Plan, the Innovation Charter's Areas of Action (People, Technologies, Companies) and Investing in Regional Innovation and Development (IRID)? Is there a need to reposition for the future?
Effectiveness
  1. To what extent is the SOPP achieving the expected outputs and outcomes (immediate and intermediate)?
  2. To what extent can these impacts be attributed to the support provided by FedDev Ontario?
  3. What unintended outcomes have been achieved?
  4. What are the facilitators and barriers to achieving expected outcomes?
Design and Delivery
  1. In what manner and to what extent is the design and delivery of the SOPP efficient and cost-effective?
  2. Are there more efficient and cost-effective ways of achieving expected results, taking into consideration alternative delivery mechanisms, promising practices and lessons learned?

This evaluation will enable FedDev Ontario to meet with the requirements of the Policy on Results, Transfer Payment Policy, and Section 42.1of the Financial Administration Act as well as contribute to renewal of the Agency mandate.

1.3 Structure of the Report

Chapter 2 summarizes the evaluation methodology. Chapter 3 provides a description of the programs involved in the SOPP and the projects which were funded. Chapters 4, 5 and 6 summarize the findings of the evaluation regarding relevance, effectiveness, and program design and delivery. Chapter 7 presents the conclusions and recommendations arising from the interim evaluation.

2. Evaluation MethodologyFootnote 5

2.1 Approach and Lines of Evidence

The evaluation was undertaken in three phases: planning, data collection involving various lines of evidence, and analysis and reporting (draft and final report). The planning phase involved a documentation review (on FedDev Ontario, the programs, and funded projects to identify the data available) and development of the evaluation matrix, methodology, data collection instruments and communication protocols. The data collection phase employed a hybrid team approach (involving internal evaluators from FedDev Ontario and external consultants from Goss Gilroy Inc.) in implementing a mixed-methods research design involving multiple lines of evidence.

Figure 1: Overview of the Study Methodology

Overview of the Study Methodology (the long description is located below the image)
Description of Figure 1

Figure 1 depicts an overview of the methodology used in the evaluation from June 2017 to October 2017. There were three phases undertaken: the planning phase, the lines of evidence and the analysis and reporting (draft and final report). The planning phase started in June 2017 and followed by the multiple lines of evidence phase, ending in August 2017. The lines of evidence phase includes a document and literature review; review of project and program financial data; there were 550 surveys (117 proponents, 40 applicants not approved for funding, 28 representatives from organizations eligible to apply for funding but did not, and 365 beneficiary organizations that received financial or other assistance from third-parties which received funding); interviews with 64 key informants; and case studies of six projects. Results used during analysis and reporting (draft and final reporting) which started September 2017 and ended October 2017.

As indicated above, the lines of evidence included a document and literature review (focused primarily on issues related to relevance); a review of project and program financial data; surveys of 117 proponents, 40 applicants not approved for funding, 28 representatives from organizations eligible to apply for funding but did not, and 365 beneficiary organizations that received financial or other assistance from third-parties which received funding from FedDev Ontario; interviews with 64 key informants; and case studies of six projects. The results were then analyzed to prepare the draft and final reports.

2.2 Challenges and Mitigation Strategies

The major challenge associated with this interim evaluation is that not enough time has elapsed for the impacts of the projects to be fully realized. Of the 190 projects, only 34 percent were completed at the time of the evaluation, and most completed projects will require additional time post-completion to realize their full impacts. To augment the limited data available to date on project outcomes, the evaluation team conducted an extensive review of the projected results to illustrate the impacts expected to be generated, conducted a review of progress reports, and focused the case studies primarily on on-going projects. The evaluation team also reviewed the results of three recent FedDev Ontario studies, which involved examining the longer-term impacts of some past projects funded by FedDev Ontario.Footnote 6

The potential non-response error was addressed by achieving significantly high response rates for the surveys, while the potential for respondent bias was addressed by including a survey of organizations that applied but did not receive FedDev Ontario funding, a survey of potential applicants, and interviews with key informants not directly involved in the programming, as well as triangulating the results with data obtained through other lines of evidence. Given these mitigation measures, the evaluation team is of the opinion that the limitations of the study were adequately addressed and the results of the evaluation are deemed to be reliable and valid.

3. The Southern Ontario Prosperity Program

3.1 Number and Value of Projects Approved

As of May 24, 2017, 190 projects (totaling $688 million in funding) were approved under the SOPP programs. These included 113 private sector projects and 77 not-for-profit (NFP) projects undertaken by other organizations (not-for-profit organizations, post-secondary institutions and other government organizations).

Table 2: Number, Type and Value of Projects Approved By Program as of May 24, 2017
Initiative For Profit Organizations Non-profits and Post-Secondary Total
Number $ million Number $ million Number $ million
Southern Ontario Prosperity Initiatives
Investing in Business Innovation 56 $34.8 32 $44.6 88 $79.7
Invest. in Business Growth & Prod 49 $175.9 2 $29.0 51 $204.9
Investing in Commercialization Part.  - - 12 $123.8 12 $123.8
Investing in Regional Diversification - - 12 $63.4 12 $63.4
SOPP Strategic Project - - 1 8 1 8
Advanced Manufacturing Fund (2013-2018)
Advanced Manufacturing Fund 8 $140.0 1 $20.0 9 $160.0
Eastern Ontario Development Program (2014-19)
EODP 17 $48.0 17 $48.0
Total 113 $350.7 77 $336.8 190 $687.8

3.2 Overview of the ProgramsFootnote 7

Characteristics

The characteristics of the each of the six programs included in the SOPP are summarized in the table on the following page and in the points below:

Table 3: Overview of the Programs Included in the SOPP
Program Focus Stage Projects FDO Approved Proponents Support Focus of Supported Projects
IBI Matched investment funding Start-up/Early stage 56 $34.8 million (avg. $621,000) SMEs under 50 employees
  • Repayable up to $1 million
  • To be leveraged with $2 in angel or VC funds for every $1 in IBI funding
  • Develop and commercialize new technologies
  • Establish or expand production capabilities
  • Undertake product and market expansion activities
Angel investment networks Start-up/Early stage 22 $7.5 million (avg. $340,000) Angel investment networks
  • Non-repayable to $500,000
  • Focus on strengthening angel networks
  • Attraction of members/angel investors and qualified applicant companies,
  • Outreach, education, mentoring and engagement for investors and entrepreneurs
  • Investor accreditation
  • Facilitating co-investment/investor syndication
  • Improved reporting and monitoring tools
Skills Development & seed funding Start-up/Early stage 10 $37.1 million (avg. $3.7 million) Not-for-profits (1 PSI and 9 non-profits)
  • Non-repayable to $20 million
  • Up to $10,000/entrepreneur for business training and $30,000/SMEs to cover start-up costs
  • SMEs must provide 50%
  • Seed financing, investment attraction (e.g., a Capital Access Advisory Program), training, mentorship, and incubation.
  • Individual projects targeted specific groups in terms of sector (e.g., bioscience, ag-tech, and medical technologies), stage of development (e.g., start-ups and early-stage), priority group (women), or region
ICP Businessled development/ commercialization of products and services Product Development and Commercialization 12 $123.8 million (avg. $10.3 million) Not-for-profits including 7 post-secondary institutions
  • Non-repayable to $20 million
  • Up to 50% of eligible costs
  • Remainder provided by other partners
  • Increases collaboration among businesses, PSIs and research organizations, narrows gap between innovation and commercialization, and increase capacity of ecosystems
  • Focused on range of activities including development, testing and validation of new technologies, applied research, providing access to R&D expertise and computing/data platforms, and support for commercialization and development of SMEs
  • Focused on a range of existing and emerging clusters: digital media, health technologies, ICT, agri-tech (greenhouse, food and beverage), water, bioengineering and manufacturing
IBGP Direct assistance to SMEs for growth/modernization Growth 49 $175.9 million (avg. $3.6 million)Footnote 9 SMEs (15 to 1000 employees)
  • Repayable up to $20 million
  • Target SMEs with sustainable business model, profitable track record and potential to become strong global player
  • Used most commonly to expand, modernize or relocate production capabilities
  • Acquisition of equipment, building of plants/ facilities and, to a lesser extent, develop/ expand markets and finance expansion
Assistance for manufacturers delivered through NFPs Growth 2 $29.0 million (avg. $14.5 million) Not-for-profits (industry associations or regional development organiz-ations)
  • Repayable up to $20 million
  • Up to $100,000 per SME
  • 100% of eligible costs
  • SMEs must provide 50%
  • Both projects targeted manufacturers
  • Up to $15,000 for advanced technology assessments by qualified professionals who examine company's manufacturing performance and recommend how advanced technologies could be implemented
  • Up to $100,000 for projects that improve productivity through adaptation or adoption of advanced technologies, materials or processes
  • Up to $50,000 to offset costs of training expenses related to supporting innovation
IRD Regional development and diversification Early stage and Growth 12 $63.4 million (avg. $5.3 million) Not-for-profits (regional development organiz-ations)
  • Non-repayable to $20 million
  • Up to 50% of eligible costs
  • Remainder provided by recipient as a cash contribution
  • Variety of projects ranging from technology research, development, testing and commercialization, building new facilities, business retention & expansion, business incubation and acceleration, digital media productions, investment attraction and regional investment and loan funds
AMF Increase productivity and competitiveness of advanced manufacturers Growth 8 $140 million (avg. $17.5 million) Established profitable businesses with R&D in Ontario
  • Repayable to a normal maximum of $20 million
  • Up to 50% of eligible costs
  • provided by industry
  • Projects focused on expansion of R&D and manufacturing capabilities, development of new products, technologies and processes, and the adoption, adaptation and commercialization of new technologies
Growth 1 $20 million Not-for-profits collaborating with an anchor firm
  • Non-repayable to $20 million
  • Up to 50% of eligible costs
  • Remainder provided by other partners
  • Support the development and/or adoption of cutting-edge technologies leading to innovation and new market opportunities for businesses in the manufacturing sector
  • Must demonstrate significant benefit for sector
EODP Business Development and Community Innovation Mixed 15 $37.5 million ($2.5 million per CFDC) 15 CFDCs in eastern Ontario
  • 10% budgeted for delivery
  • 54% budgeted for business development projects
  • 36% budgeted for community innovation projects
  • Business development projects leading to growth of new and existing businesses within rural eastern Ontario communities
  • Community innovation includes labour market development (skills gap analysis, skills development, worker transition), planning and research, and business infrastructure
Collaborative Economic Development Projects (CEDP) Mixed 2 $10.5 million 2 CFDCs
  • 10% budgeted for program delivery
  • 90% budgeted for CEDP projects
  • Projects that generate benefits for multiple communities and promote broad-based collaborative economic development. Two organizations received funding totaling $10.5 million.

Governance

With the exception of the AMF, the SOPP programs are administered solely by FedDev Ontario. The Agency is responsible for program design, development, and promotion, review of applications, funding decisions, development and approval of contribution agreements, management of the funding agreements, project monitoring, and assessment of program outcomes.

The AMF is delivered under an MOU with Industry Canada (now ISED), which governs the review of applications. Under the MOU, FedDev Ontario retains responsibility for project approval but obtains input from ISED regarding the technical aspects (innovation), market relevance and potential spillover benefits of the proposed project. At times, ISED contracts with a private sector contractor to assist in that review. The AMF is the only SOPP program which also serves northern Ontario (one of the eight AMF projects, involving the largest contribution made to any SOPP project, was located in northern Ontario) and ISED provides support for program delivery in that region.

The contributions provided under each program are governed by contribution agreements made directly with businesses, not-for-profit organizations (including post-secondary institutions) that work with a collaborator or group of collaborators to implement the project, and third-party organizations that in turn use that funding to deliver support to businesses. The contribution agreement outlines the recipient's contractual obligation to provide information required for performance measurement and evaluation requirements.

4.0 Relevance

4.1 Need for the Program

Project proponents, unfunded applicants, non-applicants and beneficiaries each identified a continued need for the programming included in the SOPP. When asked to rate the extent to which there continues to be a need for these programs (on a scale of 1 to 5 where 1 is not at all and 5 is to a great extent), the average rating varied from 4.3 amongst non-applicants and 4.4 amongst unfunded applicants to 4.8 amongst beneficiaries and 4.9 amongst project proponents. Only 6 of the 550 people surveyed gave a rating of less than somewhat of a need; over three-quarters indicated that there was a great need. The average ratings given by the proponent and beneficiary groups were consistently high across the programs, ranging from 4.7 to 5.0.

In their responses, project proponents, unfunded applicants, potential applicants, and beneficiaries focused primarily on the need for assistance to help businesses deal with specific development constraints, highlighting the need for capital, funding to support further development or expansion of their operations, and assistance for particular issues such as technology development and commercialization, market development, staff development and training, and business start-up and early stage development. Those few respondents who provided ratings of 3 or less noted, generally, that the programming did not meet their needs (e.g., they required more funding than was available, the program did not move at the speed of business in terms of the timeline from application to the receipt of funding, low cost capital is already readily available from commercial sources and what was needed was a grant, or repayable grants programs would be better delivered by the private sector, such as a banks, rather than through government).

The demand for project funding also far exceeded the available funding. In total FedDev Ontario and its third party delivery agents received 4,215 applications for all SOPP programs between 2014-15 and 2017-18. 190 projects received funding.

Although not asked to rate the need for the programs on the same scale of 1 to 5, the key informants who were interviewed also expressed a strong need for the programs. In discussing the rationale for the programming, the key informants highlight various factors: the importance of the Ontario economy; the significant opportunities for further growth and diversification; the challenges or constraints that are slowing this growth and diversification; and the alignment between the SOPP suite of programs and those constraints; and alignment with the priorities of the Government of CanadaFootnote 10 and the mandate of FedDev Ontario. Each of these factors was confirmed in the results of the document and literature review, as further described below:

  1. As the largest regional economy in Canada, the health of the southern Ontario economy has a major impact on the overall health of the Canadian economy particularly in terms of manufacturing. According to Statistics Canada data, the region served by FedDev Ontario accounts for 39 percent of the Canadian population, 37 percent of its GDP, 41 percent of exports and 44 percent of business expenditures in research and development (BERD). Manufacturing is a key economic sector for Ontario, contributing nearly 13 percent of provincial GDP, 11 percent of employment (745,000 jobs), and over $170 billion in domestic exports for Ontario in 2015 (representing 86 percent of Ontario's domestic goods exports).
  2. There are significant opportunities for further economic growth and diversification in southern Ontario. The Ontario economy encompasses a wide range of existing economic clustersFootnote 11 (e.g., manufacturing, finance, and ICT) and emerging clusters in areas such as health (e.g., biotech and regenerative medicine), bio-processing, agricultural technology and biotechnology, fintech, cybersecurity, big data, Internet of Things (IoT)Footnote 12, networking, and quantum computing) where the province holds comparative advantages and the potential for further growth is significant. Further development of these clusters will generate spillover economic benefits for other economic sectors and benefits for society overall in areas such as environment, security, health, evidence-based policy making and communications.Footnote 13

    The rate of development within existing and emerging economic clusters is best viewed as a function of multiple factors that create the conditions for growth such as access to capital, highly qualified personnel, entrepreneurs, markets, infrastructure and other key inputs, as well as capabilities related to research, development and commercialization. Governments and others work to accelerate the rate of development by influencing the factors that drive development. Varying somewhat from cluster to cluster, some of the key areas where southern Ontario holds comparative strengthsFootnote 14 and how FedDev Ontario programs support those strengths are summarized in table 4 below.

Table 4: FedDev Ontario Programs Support Areas of Comparative Strength for Southern Ontario
Area Description AMF EODP IBGP IBI ICP IRD
Innovation ecosystem Southern Ontario benefits from an extensive network of post-secondary institutions, government research labs, major research centres operated by multinational enterprises (MNEs) such as Xerox, IBM, Open Text, Cisco, Google, BlackBerry, Novartis, and Monsanto (Bayer) and is focused on areas such as telecommunications, digital technologies, health care, and biotech, numerous incubators, accelerators, and other cluster development organizations, and expanding capabilities in areas such as clinical trials, product testing, and technology demonstrations.
Investment in R&D The Government of Canada is a major funder of early and later stage research through programs such as CIHR, NSERC, SSHRC, CFI, IRAP, and SDTC as well as SR&ED. Targeted investments are also made by the provincial government and the private sector (particularly, ICT which accounts for a majority of private sector investment in Ontario).
Market access Some clusters benefit from access to large local markets for their products and technologies (e.g., local markets for energy or financial services), while some benefit from ready access to the US market and other export markets.
Skilled workers Thirty-five post secondary institutions in Southern Ontario produce thousands of graduates and provide industry with access to needed skilled workers such as engineers, researchers and technicians. Southern Ontario is home to 45 percent of all Canadian R&D scientists and engineers.
Access to capital The value of venture capital and private equity managed in southern Ontario has increased in recent years and companies in many of the sectors have received significant investments in recent years. Toronto is Canada's largest centre for private-equity activity and southern Ontario continues as a major destination for foreign direct investment (FDI) in North America.
Cost Competitiveness SR&ED and provincial matching programs significantly reduce the cost of R&D for companies. Even before the recent decline in the Canadian dollar, southern Ontario benefited from significant cost advantages in areas such as labour costs. For example, Toronto was ranked as the most cost competitive North American location in terms of labor for ICT operations.
Business Development Support Government, industry and not-for-profit organizations have made significant investments in the development of industry resources, networks and infrastructure. Representatives noted that southern Ontario features a business-friendly environment for start-ups and a range of organizations and programs, including accelerators and incubators that provide support to early stage companies.
Profile Southern Ontario has a long history and features leading companies in areas such manufacturing, finance, telecommunications, ICT, biotech, chemistry and agri-food.
  1. However, the region also faces various challenges or constraints which can serve to slow the rate of development with the existing and emerging clustersFootnote 15. As highlighted in the document and literature review, as well as in interviews and surveys, the region tends to lag other advanced economies in terms of labour productivity which, in turn, has been attributed to factors such as lower rates of business investment in R&D, technology adoption, and investment in new machinery and equipment. Some of the other issues that were highlighted in the research included:

    • While access to risk capital can be increasing, the level of investment remains lower than other competing regions (particularly the United States).
    • There are challenges associated with technology transfer and spinoffs. The high level of public sector investment in R&D has not resulted in high levels of technology commercialization.
    • While education levels in southern Ontario are high, there is a mismatch between the skills and experience required by companies (particularly in emerging sectors) and the available workers. The Conference Board of Canada has estimated that skills issues (including mismatches between education and market requirements) cost the Ontario economy over $24 billion per year.Footnote 16
    • The rate of technological change in the economy is accelerating, which holds significant implications for businesses in terms of evolving skills requirements, the need to adapt business models, and the need to make further investments. In particular, developments such as the emergence of new platforms, systems and technologies, automation, the Internet of Things, 3D printing, big data analytics, introduction of composites and lightweight materials, and Industry 4.0 are having a major impact on traditional manufacturing.
    • Most SMEs are not pursuing global markets. For example, only 7 percent of Ontario SMEs are engaged in export activities.Footnote 17

    The region is also being impacted by uncertainty regarding the economic outlook. While the declining Canadian dollar benefited exports (particularly in the manufacturing sector), it also made some key imports (such as machinery and equipment) more expensive which has slowed business investment. Going forward, uncertainty associated with the dollar, the ongoing NAFTA renegotiations, the global economic outlook and the potential for interest rate hikes may negatively impact the investment climate.

  2. SOPP programs are well-coordinated with each other, employing a variety of delivery mechanisms to address these challenges and constraints and promote development across various stages of businesses' development, economic clusters, and regions within southern Ontario.

    Table 5 summarizes the relative focus of each of the programs in terms of the delivery mechanisms, development issues or challenges on which they focus as well as stages of business development, economic clusters and regions.

Table 5: Focus of the SOPP Programs
Focus IBI ICP IBGP IRD AMF EODP
Sources: Program documentation and statistical review of approved projects
Approved Contribution from FedDev Ontario
Number 88 12 51 12 9 17
Value ($millions) $81 $132 $179 $69 $160 $48
Delivery Strategy
Direct to business
Funding for not-for-profit intermediaries
Funding for not-for-profit organizations or PSIs
Stage of Business Development
Start-up/Early Stage SMEs
Growth and Modernization
MNEs
Challenges and Constraints - targets needs related to:
Development/expansion of manufacturing capabilities
Research and commercialization
Advisory and support services
Regional Development - Eastern Ontario
Angel investment/investment funding
Product, prototype or technology development
Market development
Public infrastructure developmentFootnote 18
Investment attraction/business retention
Existing and Emerging Clusters
Manufacturing
Advanced manufacturing
Cleantech and clean resources
Health/bio-sciences
Agri-tech/Agri-food
Consumer Sector
ICT
Digital technology
Professional Services
Primary
Other
RegionFootnote *
Toronto
Regions Neighboring Toronto
Eastern Ontario
Other (including southwestern Ontario, Kingston and Ottawa)

A description of the relative focus of the programs is provided below.

Delivery Mechanisms

FedDev Ontario employs various delivery mechanisms, including providing repayable funding directly to businesses, non-repayable funding for cluster or capacity development projects undertaken by post-secondary institutions or other not-for-profit organizations (sometimes in partnership with private sector organizations or others), and non-repayable contributions to third-party organizations (e.g., associations, research organizations, post-secondary institutions, CFDCs, or other not for-profit organizations) that, in turn, provide services and other support to business clients and others.

Stages of Business Development

The programs provide support for start-up and early stage companies, expanding and modernizing existing businesses, and increasing the involvement of MNEs in the further development of clusters in Ontario. To facilitate start-up and early stage development, IBI provides funding to SMEs to be matched with venture capital or funding from angel investors; to strengthen angel investor networks; and to not-for-profit organizations to facilitate skill development and seed financing for new entrepreneurs. ICP facilitates development, testing and commercialization of new technologies by bringing together businesses, post-secondary institutions and research organizations. IRD funds some projects that, in turn, made investments in early-stage companies.

Enhancing Ontario's productivity and growth requires increased investment in productivity enhancing advanced technologies and innovation. The IBGP and AMF have supported investments in the development and modernization of production capabilities and facilities as well as the adaptation or adoption of new technologies, materials or processes. Other IBGP and IRD projects have supported business growth through activities in areas such as market development and provision of advisory services.

Both AMF and ICP facilitated large-scale investments, including foreign direct investment in projects involving MNEs. For example, GE Healthcare indicated that, in the absence of the Centre for Commercialization of Regenerative Medicine and the funding provided by FedDev Ontario (an AMF project), they would have made their investment in regenerative medicine in another jurisdiction. Similarly, a significant investment by IBM into southern Ontario may not have been made in the absence of the SOSCIP projectFootnote 19 and, to a lesser extent, the Southern Ontario Water Consortium (SOWC) project, both funded initially under the Technology Development Program (TDP) under Mandate 1 and continuing IBM's involvement under ICP. Project funding for Hanwha L&C Inc. of South Korea is another example of foreign direct investment attraction. Hanwha received $15 million from AMF to establish London, Ontario as Hanwha's North American headquarters, paving the way for future foreign direct investment in the region.

Development Issues

The programs have targeted a wide range of economic drivers including access to capital (e.g., increasing the supply of angel investment), expansion or modernization of production capabilities, linkages and networking between groups (e.g., the ICP has brought together representatives from business, post-secondary institutions and research organizations to narrow the gap between innovation and commercialization), technology development, testing and commercialization (e.g., through ICP, IRD and AMP projects), entrepreneurial and staff development, provision of advisory services through intermediary organizations funded by FedDev Ontario, market development and investment attraction.

As part of the evaluation, descriptions of each of the 190 projects were reviewed by the evaluation team to code the projects by primary focus or objective. Of the 190, 61 projects supported the development, expansion or modernization of manufacturing facilities and accounted for almost one-half (49 percent) of the approved funding. Twelve projects provided advisory services or support to businesses (e.g., services or support related to incubation and acceleration, business or technology assessment, business training, technology or business development support, or capital access advisory services).

Table 6: Summary of Funded Projects by Primary Focus or ObjectiveFootnote 20
Number Approvals ($ millions) % of FDO Funding
Development/expansion of manufacturing capabilities 61 $338.10 49.20%
Research and commercialization 12 $119.00 17.30%
Advisory Services to Business 12 $89.60 13.00%
Regional Development - Eastern Ontario 17 $48.00 7.00%
Investment fund 8 $34.20 5.00%
Product, Prototype or Technology Development 32 $22.70 3.30%
Market development 22 $12.90 1.90%
Public Infrastructure Development 2 $9.30 1.40%
Angel Investment 22 $7.80 1.10%
Investment attraction/business retention 2 $6.20 0.90%
Total 190 $687.80 100.00%

The survey of beneficiaries included 150 businesses, 48 community projects and 60 CEDP projects funded under the EODP; 57 businesses that received assistance through the Achieving Innovation and Manufacturing Excellence (AIME) Global Program offered by the Yves Landry Foundation (funded through IBGP); and 50 businesses that were assisted though the CME SMART Advanced Technologies for Global Growth (ATGG) program (also funded through the IBGP). An overview of the objectives of their involvement in the program is provided below:

Existing and Emerging Clusters

The evaluation team also reviewed the 190 projects to identify the clusters or sectors on which they focus. The SOPP projects involve a range of existing and emerging economic clusters, with the manufacturing sector being the most significant. Ontario is the primary centre for manufacturing in Canada, accounting for 46 percent of Canada's manufacturing GDP and 52 percent of its manufacturing exports. While conditions have been improving, manufacturing employment declined by about one-quarter between 2005 and 2015. Of the 190 projects, 77 related to the manufacturing sector (mostly funded through the AMF and IBGP with some funding also provided through the ICP and IBI). These 77 projects accounted for 41 percent of the projects approved and 60 percent of the total approved value of FedDev Ontario contributions (of which approximately 37 percent could be categorized as advanced manufacturing).

Other leading sectors or clusters include health care and biotech (27 projects accounting for about 26 percent of approved funding), ICT (44 projects accounting for about 13 percent of approved funding), agri-food (18 projects accounting for about 8 percent of funding) and clean tech and clean resources (13 projects accounting for about 7 percent of funding).

Region

Northumberland Case Study: Food Cycle Sciences Corporation in Stormont, Dundas & Glengarry

CEDP funding allowed this company to attract and retain FTEs with technical expertise from urban centres, including new Canadians. In addition, this project allowed Food Cycle Sciences to expand into global markets.

With the exception of the EODP (which is targeted specifically at communities in eastern Ontario) and the AMF (which could fund projects from all parts of Ontario)Footnote 21, the programs are open to applicants from across southern Ontario. Uptake of the programs varies by region, depending largely on where industry is based. Overall, the leading regions include Toronto ($158 million was approved for projects based in Toronto, representing 23 percent of the total) and neighboring regions (a further $148 million, 22 percent, was approved for projects in census divisions adjacent to Toronto, mostly Peel and York).

Eastern Ontario was identified as a priority for funding because of weak economic conditions which contribute to a loss of businesses, investment and youth from the region. For example, as of May 2017, the employment and participation rates in eastern Ontario are lower than in the rest of the southern Ontario (54 percent v. 61 percent and 58 percent v. 66 percent respectively as of 2017) while, in 2014, employment incomes averaged 13 percent below those in southern Ontario. Of the 190 projects, 31 targeted the eastern Ontario region (excluding Ottawa and Kingston) with contributions totaling $81 million (12 percent of the total).

  1. The SOPP programs are consistent with the priorities of the Government of Canada, including the Innovation and Skills Plan, as well as the mandate of FedDev Ontario. Based on the results of consultations undertaken for the Innovation and Skills Plan, the Government of Canada identified three priority areas: people (ensuring that people are equipped with the right skills and experience to drive innovation); technologies (taking full advantage of transformative emerging technologies that can elevate the competitiveness of established and new firms, industries, and clusters); and companies (growing the next generation of global companies in Canada)Footnote 22. The investments made by FedDev Ontario have been consistent with these priorities, particularly with respect to:
    • Attracting, developing and retaining highly skilled workers, researchers and entrepreneurs. FedDev Ontario contributions have supported the delivery of training while proponents report that investments have helped southern Ontario attract and retain key workers, researchers and entrepreneurs.
    • Strengthening the innovation ecosystem through the further development of research infrastructure (e.g., construction of new buildings, adaptation of existing buildings, and provision of equipment for new or existing centres), investments in technology development, testing and commercialization, and facilitating the development of collaborations and partnerships.
    • Accelerating the start-up, early development, expansion and modernization of companies by attracting and facilitating investment, supporting technology adaptation, adoption and commercialization, supporting advisory services and market development activities and attracting anchor firms.

    As noted earlier, FedDev Ontario has made significant investment in clusters identified by the Government of Canada as priorities including advanced manufacturing, health and biotech, ICT, agri-food and clean tech and clean resources. The activities supported under the SOPP have also directly contributed to the Agency's core mandate of strengthening southern Ontario's economic capacity for innovation, entrepreneurship and collaboration and promoting the development of a strong and diversified southern Ontario economy.

4.2 Relationship to Other Programming

The major findings of the evaluation regarding the relationship between the SOPP and other programming are as follows:

  1. Factors such as the place-based nature of FedDev Ontario, the strong demand for funding, and coordination between FedDev Ontario and other funding organizations help to ensure that the SOPP programs complement rather than duplicate other federal or provincial government programs that promote innovation, business development and community development. There are a variety of other programs available to support development in southern Ontario. When asked about other programs that are available in southern Ontario that share at least somewhat similar objectives to those of the programs included in SOPP, the key informants and those who were surveyed identified a range of different programs which are listed in the table below.
Table 7: Other Programs Offering Support for Innovation
Agency Program/Type of Program Early-stage Growth Maturity/Modern
Capital or Loan Programs
BDC Venture Capital
BDC Growth & Transition Capital
BDC Business Loans
IRCC Immigrant Investor Venture Capital Pilot
IRCC Start-up Visa
FedDev
Ontario
Community Futures
ISED Canada Small Business Finance Program
Government of Ontario
MEDG Ontario Venture Capital Fund
Market/Export Development Programs
NRCan Expanding Market Opportunities
EDC Export Development Canada
GAC CanExport
Innovation
AAFC Agri-Innovation
Food Research and Development Centre's Industrial Program
GAC Going Global Innovation (GGI)
ISED Consortium for Aerospace Research and Innovation in Canada (CARIC)
Automotive Innovation Fund
Automotive Supplier Innovation Program
Strategic Aerospace & Defence Initiative
Technology Demonstration Program
Strategic Innovation Fund
NRC Industrial Research Assistance Program (IRAP)
NSERC Various programs
SDTC Sustainable Development Technology Canada
Government of Ontario
OMAFRA Agri-Technology Commercialization Centre (ATCC)
Ontario Agri Food Technologies (OAFT)
MRIS Green Investment Fund
Regional Funding
MEDG Eastern Ontario Development Fund (Business Stream)
MEDG Southwestern Ontario Development Fund (Business Stream)

While the breadth of the SOPP programming creates the potential for some overlap with other programming, only a few key informants suggested that overlap exists with other programs and only one identified it as an issue (the potential for overlap between the recently announced Strategic Innovation Fund and the Advanced Manufacturing Fund). Some of the key characteristics of FedDev Ontario and its programming which enable the Agency to position its programs to complement the other available sources of assistance include:

  1. SOPP fills needs that would not be met by other programming. Only 4 percent of projects would have gone ahead as planned in the absence of FedDev Ontario funding. As indicated in the table below, most proponents indicated that the project would have been reduced in scope (44 percent), implemented over a longer period of time (33 percent), or delayed in the absence of FedDev Ontario support. About 15 percent of proponents said they would have had to cancel the project. Only 5 of the 40 applicants not approved for funding indicated that their projects proceeded as planned in the absence of FedDev Ontario support.
Table 8: Impact of FedDev Ontario Funding on the Project
Question: Proponents: If FedDev Ontario had not been able to provide funding for the project(s), what would your organization most likely have done? OR Unfunded: What happened to the proposed project when you were not able to obtain funding from FedDev Ontario? (select all that apply)
Proponents Unfunded Total
# % # % # %
Total Respondents 117 100.00% 40 100.00% 157 100.00%
Proceeded with the project as planned 5 4.30% 5 12.80% 10 6.40%
Reduced the scope of the project 51 43.60% 25 64.10% 76 47.80%
Implemented the project as planned but over a longer time period 38 32.50% 10 25.60% 48 30.60%
Delayed the start of the project 31 26.50% 17 43.60% 48 30.60%
Cancelled the project 17 14.50% 3 7.70% 20 12.70%
Undertaken a different type of project
Other, please specify
9 7.70% 2 5.10% 11 7.00%
Approached another program for funding to replace the requested FedDev Ontario funding 22 18.80% 6 15.40% 28 17.80%
Looked for private capital/investment 10 25.60% 10 6.40%
Other 14 12.00% 3 7.7%% 17 10.80%
Don't Know 5 4.30% 1 0.00% 6 3.80%
No Reply 0 0.00% 1 0.00% 1 0.60%

5. Effectiveness

This chapter summarizes the major findings regarding the effectiveness of the programs. It begins with an overview of the current status of the projects for contextual purposes and then reviews achievement of project objectives, impacts to date of the completed projects, projected impacts of the on-going projects, and extent to which the projects are expected to continue on and generate further impacts beyond the term of the FedDev Ontario funding.

5.1 Current Status of the Projects

Final reports and close-out reports had been filed for 65 of the 190 projects (34 percent) at the time of the evaluation. The final reports are prepared by the proponents at the conclusion of the project, adding to the progress reports prepared during project implementation. Close-out reports are prepared by project officers, typically after a site visit. The 65 completed projects tend to be of a shorter duration than the on-going projects and consisted primarily of projects funded under IBI (46 projects), IBGP (12 projects), and IRD (5 projects). The ICP, AMF and EODP projects were all multi-year projects. No ICP or EODP projects were completed and only one AMF project was completed. The other closed project was a SOPP Strategic Project.

Of the 117 projects represented in the survey, 35 (30 percent) were completed according to the proponents. No projects were identified as suspended, cancelled or yet to start. Only 6 of the 117 projects were identified as not being implemented as planned. Of those six, four reported having been delayed, three had some project partners change, three indicated the project focus had shifted somewhat, and one indicated the scope was reduced somewhat. Four of the six proponents indicated that the change impacted the effectiveness of the project (two indicated that effectiveness was enhanced and two indicated effectiveness was reduced, both of whom still rated the project as at least somewhat successful in meeting its objectives). Of those noting that effectiveness increased, one indicated that the change in strategy resulted in higher margins than projected while the other indicated timelines to commercialization were shortened because of bringing in new partners.

5.2 Achievement of Project Objectives

The proponents and beneficiaries were asked to rate how successful the project has been to date in terms of achieving its objectives, using a scale of 1 to 5, where 1 is not at all successful, 3 is somewhat and 5 is very successful. As indicated below, the average rating provided by both the proponents and the beneficiaries was 4.4.

Table 9: Achievement of the Project Objectives
Question: On a scale of 1 to 5, where 1 is not at all successful, 3 is somewhat successful, and 5 is very successful, how successful has the project been to date in meeting its objectives?
Response Proponents Beneficiaries Total
# % # % # %
1 Not at all 1 0.90% 6 1.60% 7 %
2 0 0.00% 3 0.80% 3 %
3 To some extent 11 9.40% 42 11.50% 53 %
4 41 35.00% 89 24.40% 130 1.50%
5 To a great extent 62 53.00% 177 48.50% 239 0.60%
N/A 0 0.00% 0 0.00% 0 11.00%
No reply 2 1.70% 48 13.20% 50 27.00%
Total Respondents 117 100.00% 365 100.00% 482 100.00%
Average Rating 4.4 4.4 4.4

The average ratings across the programs ranged from a low of 4.3 amongst the IBI and IBGP proponents and CEDP beneficiaries to a high of 4.9 amongst the EODP proponents.

Table 10: Average Rating By Program Regarding Achievement of Objectives
Program Average Rating Program Average Rating
Southern Ontario Prosperity Initiatives Eastern Ontario Development Program
Proponents Proponents 4.9
IBI 4.3 Beneficiaries
IBGP 4.3 EODP CEDP 4.3
ICP 4.4 EODP 4.4
IRD 4.6 EODP Total 4.4
Beneficiaries Advanced Manufacturing Fund
IBGP 4.4 AMF Proponents 4.5

When the few proponents and beneficiaries associated with lower rated projects were asked to identify areas where the project had been less successful than expected, the issues focused mostly on delays in project implementation caused by outside factors (e.g., delays in obtaining equipment for suppliers, regulatory issues, or in one case a fire) or by delays in the project approval process which delayed the results or required the project to be implemented in a shorter time period. Other issues that were identified related to project partners (e.g., an investor not following through with their commitments), changes in the market which required changes to the commercialization strategy, slower sales than expected, and not having enough funding to implement the project as planned.

5.3 Outputs and Impacts of the Completed Projects

The close-out reports, which are prepared by the project officer and signed off by a manager, summarize information on the proponent, project, expenditures, project outcomes, and project monitoring (e.g. site visits and audits). As part of the evaluation, a detailed review of each close-out report was undertaken to develop a database, capturing the impacts reported by each completed project. This section provides a summary of the impacts, drawn from the close-out reports and final reports, surveys, and case studies.

As indicated in table 11 below, the types of outputs and impacts most commonly reported for the projects included the creation or maintenance of permanent full-time positions, training, collaborations and partnerships, increased sales, capacity development, and the creation of intellectual property.

Table 11: Number of Completed Projects Reporting Types of Impacts
Type of Impact Projects Reporting Impact Overview/Examples
Number of Projects % of Projects
Projects Completed 65 100%
Delivery of services to businesses and other beneficiaries 19 29%
  • Provision of business advice for market entry
  • Advice on business development
  • Provided training and education to businesses
  • Provided locations for start-ups/SMEs to operate
Creation/maintenance of employment 65 100%
  • The creation or maintaining permanent or temporary jobs during and after the project lifecycle
Training of entrepreneurs and staff members 55 84%
  • Upskilling workers on new technology and infrastructure
  • Providing training for sales and marketing activities
  • Educate and train potential Angel Investors on investing in organizations
Linkages and partnerships 50 76%
  • Development of a R&D relationship when creating and testing new devices
  • Engaging educational institutions to assist with product development
  • Creation of mutually beneficial linkages between technology and manufacturing firms
Capacity development 19 29%
  • Creation of new facilities or product lines to increase production capacity or storage
  • Installation of new technology or software to increase the efficiency of production
Creation of IP 28 43%
  • The creation of new technology/registration of patents and copyrights
  • New software to provide greater efficiency for the means of production
  • New medical devices to assist medical staff in the diagnosis and treatment of patients
Increased revenue 44 68%
  • Increases in domestic and export sales
Enhanced competitiveness and sustainability 21 32%
  • Diversification of products offered or markets in which the company operates
  • Increasing sustainability of communities with income staying in the area and diversifying the business sector with the communities
Further development of the business and markets 44 68%
  • Increase in market presence
  • Engaging in sales globally
  • Development of marketing and sales materials
Angel investments 7 11%
  • Development of educational programs for potential investors
  • Attraction of new investors and organizations looking for investment.

A further description of these outputs and impacts is provided in the following paragraphs, including examples of some of the impacts, largely drawn from the case studies (provided in the text boxes).

Provision of Services to Businesses and Other Beneficiaries

Nineteen projects reported providing businesses, particularly SMEs, with access to needed services, resources and other support ranging from business development services, technology development and testing services to training, capital and other resources. Most of the SOPP projects that provide significant support services to businesses are multi-year projects which are still on-going (and therefore not included in this total).

The closed projects that provided services to businesses largely involve angel investments. For example, the Maple Leaf Angel Capital Organization used FedDev Ontario funding to build on its network of investors and entrepreneurs by expanding its outreach, education, and engagement activities. Businesses benefited by not only having greater access to capital, but also greater access to guidance from experienced business minds. The National Angel Capital Organization operates as an Angel Network that is able to leverage the knowledge of its established investor community and professionalize the Angel asset-classes. The project led to more co-investment, investor syndication and improved access to capital for Ontario companies.

Maintained or Increased Employment

Bioamber

Bioamber received support towards construction of a succinic acid manufacturing plant with the capacity to produce 30,000 tons of bio-based succinic acid per year. The project resulted in 58 permanent FTEs created and 200 temporary FTEs created during the project.

The 65 projects reported the creation of 1252 permanent FTEs and 1043 temporary FTEs, as well as the maintenance of 534 permanent FTEs and 7 temporary FTEs. Most of the permanent employment created to date is associated with manufacturers that undertook projects funded under IBGP and IBI. In addition to Bioamber, which is profiled in the box to the left, some examples of businesses that created or maintained FTEs with support from FedDev Ontario included:

Training

Most completed projects (55 of the 65) reported training of some kind. Those that reported training generally consisted of either:

Partnerships or Linkages

Fifty of the 65 projects reported the development of partnerships and collaborations with various other groups including investors, research collaborators, project partners, businesses and others. Examples include a medical instrument development company that commercialized a medical device which provides diagnoses in real-time (the organization reported developing relationships with 34 investors as part of the project), a lighting manufacturer that received funding to set up a pilot lighting manufacturing line in Toronto reported relationships with 27 organizations including investors, collaborators and service providers; and a software developer that developed partnerships with 19 organizations including technology partners, sources of funding, and investors.

Capacity Development

Mariposa Dairy

Mariposa Dairy is located in Lindsay and received $1 million through the IBGP initiative to increase production capacity of specialty cheese products as well as help with automation of the dairy company, training of staff, building of new lines and infrastructure. In April 2017, the company officially opened its brand new 40,000-square foot facility. This project is expected to create 15 full-time permanent jobs by the end of 2017.

Case Study: ArcelorMittal

The AMF project supported retrofitting a 93,000 square foot facility in Woodstock, Ontario with specialized machinery, to create a first-in-world automobile manufacturing process innovation involving high-strength lightweight auto parts production on a commercial scale. The investment increased domestic and export sales and created 21 permanent FTEs.

Capital expenditures represent a significant percentage of the total budgets of the SOPP projects. Over 60 percent of the combined total budget for all 190 projects (including the costs funded by FedDev Ontario and those funded from other sources) went towards the cost of equipment, buildings and land (the costs were budgeted under various cost categories including purchase of land, facility construction/renovations, facility expansion, furnishings and fixtures, building modifications and upgrades, manufacturing and equipment, leasehold improvements and other capital costs). Capital costs as a percent of the total budget did not vary much between the shorter-term closed projects (61 percent) and longer-term on-going projects (62 percent of costs).

Nineteen of the 65 closed projects reported making capital investments in research, technology development and testing, business development, and production capacity which will continue to be in use well beyond the term of the FedDev Ontario funded project. Fifteen of the proponents reported investments of more than $1 million in capital assets. For these projects, FedDev Ontario contributed towards the costs of obtaining equipment, constructing equipment, buildings, making renovations or modernizing operations. Examples of major investments include the Bioamber which was profiled earlier, as well as reopening of the Pembroke MDF plant, and retrofitting of the ArcelorMittal automotive parts plant.

Intellectual Property

Dejero Labs

Dejero Labs is located in Waterloo and received $925,000 through the IBI initiative to further develop technology to transmit live video from mobile devices in high-definition. Currently the holder of 12 innovative technology-based patents, Dejero leveraged funding from FedDev Ontario to attract up to $2 million from members of Golden Triangle AngelNet and Angel One Investor Network. It is expected that 166 highly skilled full-time jobs will be created in Waterloo region as a result of the project.

Twenty-eight of the 65 projects reported the development or advancement of some form of intellectual property (IP), for which the organization may or may not seek patent protection or a copyright on created content. The form of IP ranged from software and hardware to medical devices, with software being the most common. Some examples include the final development and global commercialization of a clean energy storage technology on which six different patents were filed; further development and commercialization of six different products and services based on a product information software platform that allows manufacturers to provide retailers full product details in multiple formats in real time; a new high-quality mobile imaging device that helps doctors accurately diagnose cancer in real time (three trademarks and three patent applications were submitted for approval in three different jurisdictions); further development of a product that enables users to get technical support directly from their mobile devices (12 patent applications are being converted to full patent status and six new patent applications have been filed); and final development and commercialization of a fitness system that provides real time information to users (the project is associated with 15 patent applications in 5 countries to date with more expected).

Increased Sales and Revenue

Case Study: Pembroke MDF

The company made investments in equipment, building improvements, and system upgrades to restart a 389,000-square foot and medium density fibreboard (MDF) molding manufacturing plant, creating employment for 190 people and initial revenues of $25 million. The project enabled former workers to be hired back, which injected needed income into the local economy. It also increased local sales of raw goods necessary for the manufacturing process (souring raw material for 22 local wood suppliers, contracting 33 freight carriers and purchasing equipment/parts from 350 suppliers in Ontario).

Of the 65 completed projects, 44 proponents reported an increase in sales. Increased revenues resulted from expansion into a new market, developing and/or launching of a new product, or the expansion/building of production capacity. To date, the closed projects have reported increased sales of $205 million in Canada and a further $36 million in export sales. These figures are expected to increase significantly going forward as markets are further developed and production increases.

Reflecting the characteristics of the projects which have been completed to date, most of the revenues (73 percent of the sales in Canada and 94 percent of the export sales) have been generated by manufacturers. In addition to Pembroke MDF, other examples include a food producer that increased its capacity by expanding its production facility and adding a fifth food processing line, resulting in an increase in sales of $41 million; the production capacity of a manufacturer was expanded by relocating its operations into a much larger facility and expanding storage, resulting in a $40 million increase of sales to date; a frozen food manufacturer which increased its production and storage capacity to meet demand for products and improve competitiveness in the export market, resulting in $27 million in increased sales; and a project that automated a production line to fulfill the demands of Ontario's auto manufacturing industry and fulfill a long term commercial contract, increasing export sales by $16.3 million.

Enhanced Competitiveness and Sustainability

Southeastern Ontario Angel Network

The project supported the Southeastern Ontario Angel Network's efforts to attract and retain members, as well as grow the group's investment activity, by increasing its capacity to identify, prepare and present better investment opportunities to members. Southeastern Ontario Angel Network grew its membership to 107 (nearly double the original goal of 55), and make 23 new investments totaling about $14 million over a 3-year period.

Many of the projects funded by FedDev Ontario help to enhance the competitiveness and sustainability of a company that is receiving funding or assistance provided by a funded intermediary organization. Twenty-one projects were flagged in the close-out reports as specifically enhancing competitiveness and sustainability. Examples include an energy usage reporting software company that developed marketing materials, and hired and trained sales and marketing staff to grow the business' customers and its geographic sales footprint; a project that enabled a company to enter the global market to build clean energy generation plants; a digital advertising company that finalized development of a digital display advertising system and executed a global sales and marketing plan, enhancing the company's competitiveness and presence in the international marketplace; a software company which enhanced and commercialized their software communications platform that provides analytics and monitoring solutions, enabling them to enter the international market; and an on-line voice over casting service that enables producers and casting agents to search for actors for the various projects in which they are involved. The project involved the expansion of its project management and sales teams to meet growing demand, along with the development and execution of a global market oriented marketing strategy.

Business Development and Angel Investments

Forty-three of the 65 projects reported further business development, usually further development of the proponent organization. Some intermediaries, mostly angel organizations, also reported that their services had supported or facilitated the development of other businesses.

Eight completed projects involved angel investments. The angel investor groups reported 356 investments totaling $227 million in funding. The funds provided by FedDev Ontario were used in the administration and development of the various angel investments networks, helping to identify both potential investors and companies, as well conduct the due diligence necessary to vet potential investments. Some examples of completed angel investment projects are Maple Leaf Angels Corporation which used funding to help build on its current outreach, education, and engagement activities for both investors and entrepreneurs; GreenSky President's Club used funding to grow the club to 100 members and continue to make the process of investing in good companies more efficient; and the Georgian Angel Network was able to sustain and build its member base, investable capital and sector expertise.

5.4 Impacts of the Ongoing Projects

Of the 190 projects, 125 of the projects are still going. There are two sources of data available on the projects: (1) data reported in the progress reports received to date; and (2) the projections regarding the impact of the project which were included in the contribution agreements signed between FedDev Ontario and the project proponents.

Of the 125 projects, 81 submitted data on the progress of their projects as of December 31, 2016 (these 81 projects submitted a total of 190 progress reports, an average of 2.3 progress reports per project). As per the available progress reports, the projects had created 10,901 permanent FTEs and 1,068 temporary FTEs to that point.

A more comprehensive indication of the impacts of the projects was developed by compiling the estimates of impacts developed at the time of the project approvals. To develop the database of impacts, a detailed review was conducted of the Contribution Agreements and the Project Approval Forms for each active project. While it is premature to determine whether the projects will meet or exceed their projections, the results of the evaluation indicate that the vast majority of these ongoing projects are proceeding largely as planned, they are still expected by the proponents to achieve their objectives, and many have already reported significant impacts. The impact most commonly projected for the projects includes the creation or maintenance of FTEs, increase in sales, and the provision of services to businesses or other beneficiaries as indicated in the table below.

Table 12: Numbers of Projects Projecting Selected Outputs and Impacts
Projects Percent Description
Projects Completed 125 100% --
Provided services to businesses and other beneficiaries 59 47%
  • Consisting primarily of organizations that work with businesses, providing capital, support services, training, and guidance
Maintained or increased employment 121 97%
  • 33,926 FTE position either created or maintained (31,263 permanent positions)
Provided training or mentorship 23 18%
  • Directly provided training to employees (9 projects)
  • Provide training to companies (14)
Created linkages / partnerships 53 42%
  • Enter into Partnerships and Collaborations with private businesses, not-for-profit organizations, educational institutions or other organizations
Increased sales 68 55%
  • $3.7 billion increase in North American sales
  • $597 million increase in sales outside of North America
  • $1 billion in sales from commercialized innovations
Supported angel investments 14 11%
  • 445 new angel investments and $154 million in new investment

A further description of these impacts is provided on the following pages.

Provided Services to Businesses and Other Beneficiaries

Yves Landry AIME Global Initiative

The AIME Global initiative, delivered by the Yves Landry Foundation, delivers two types of eligible training activities: training that supports the adoption of new technology, new processes or procedures or a change within the company to support innovation; and training that supports and develops highly skilled personnel in any area that leads to innovation. Through December 2016, the program reported assisting 260 businesses. Over its term, AIME projects creating more than 1,100 FTEs and maintain more than 1,000 FTES.

Fifty-nine of the 125 projects report providing services to over 2,900 businesses and other beneficiaries to date (as per the progress reports). The EODP program, though its business development, community innovation and CED projects, accounts for a majority of these. Through 2016, the leading projects in terms of the numbers of businesses served were the Yves Landry AIME Global Initiative (260 businesses) and the CME SMART ATGG program (225 businesses). Other leading organizations funded by FedDev Ontario include:

Provided Training or Mentorship

Case Study: Communitech Fierce Founders

The Fierce Founders Accelerator located in Kitchener is a six-month program offered twice per year to five to eight technology or tech-enabled companies which have at least one female founder. This is the only program of its kind in Canada that is exclusively for women. Companies receive up to $30,000 in matching funding, one-on-one mentorships and coaching. The program is projected to create or maintain 90 FTEs.

Participant Feedback:

“Fierce Founders gave me self-confidence!”
“Fierce Founders helped to double our revenues!”
“I am so grateful!”

Twenty-three companies reported training including nine companies that are undertaking training of their own staff (usually related to manufacturing expansion or modernization) and 14 intermediary organizations such as the Yves Landry Foundation that provide training to others.

Funded through IBI, Communitech Fierce Founders is an acceleratorFootnote 24 program targeted at businesses founded by women that incorporates a significant mentorship component. Participants receive $30,000 of funding, mentorship from Communitech's growth coaches to help the businesses set and reach milestones, build and execute a sales strategy, and go to market plan and preparation for investor presentations.

Two other examples include Next CanadaFootnote 25 (which provides mentoring, training and matching seed funding of up to $30,000 to high-potential start-ups) and the York Entrepreneurship Development InstituteFootnote 26 which received support to expand its incubator program In total, FedDev Ontario funding is expected to provide training and mentoring support to over 900 businesses.

Maintained or Increased Employment

Case Study: Innovation Centre at Bayview Yards

The Innovation Centre at Bayview Yards (the Centre) will provide industry with access to the Advanced Digital Media Lab (which will provide entrepreneurs, start-ups and SMEs with the technical capabilities required to develop, prototype and validate advanced digital media concepts), the Global Cybersecurity Resource Program (which provides access to cybersecurity expertise, resources and support) and the Maker Space (which provides access to leading-edge maker space infrastructure and capabilities including tools, technology and equipment). It opened in 2016 and, through December, had reported assisting 78 businesses. FedDev Ontario's support went towards the purchase and installation of software, hardware and other equipment for the Maker Space, Digital Media Lab, and Global Cybersecurity Program within the Innovation Centre at Bayview Yards. The project is expected to create 284 FTEs and maintain 71 FTEs.

Of the 125 ongoing projects, 121 are projected to have a significant impact on employment. They are projected to create or maintain over 31,000 permanent FTEs and more than 2,500 temporary FTEs. Over 10,000 FTEs had been created as of December 2016.

About 60 percent of the FTEs will be generated through contributions to intermediary organizations which then work with businesses. For example, the CME SMART Advanced Technologies for Global Growth program is expected to generate more than 3,500 FTEs, the OCE project is expected to generate nearly 2,500 FTEs, and the AIME Global initiative is projected to create over 2,000 FTEs. The remaining 40 percent will be generated through funding provided directly to businesses which then expand or modernize their operations (a leading example of the impact of direct funding is a major AMF project which involves enhancing the capacity, productivity and quality of a major manufacturer; this is expected to create more than 2,700 FTEs).

Increased Sales

Noblegen

Noblegen is a Peterborough-based advanced ingredients company, offering food and beverage companies non-GMO, cost effective and customized ingredients to satisfy consumer needs. The company received $600,000 in funding to expand its marketing activities and sell its advanced bio products on a global scale. This contribution through the IBI initiative is expected to create 22 skilled jobs by the end of 2017.

Increased sales were projected for 68 of the 125 ongoing projects, including $3.7 billion in increased North American sales, almost $600 million in sales outside North America and nearly $1.1 billion in sales from the commercialization of new products and technologies. Almost 1,500 new products, services and processes are expected to be commercialized.

Most of the projected increase in revenues is associated with contributions provided directly to businesses, particularly manufacturers who received support to further develop or upgrade their manufacturing facilities. Some notable examples include a food manufacturer that received support to invest in new equipment and build a processing plant to expand production, a fabricated metals manufacturer that is increasing its production capability, and a food additive company that is constructing a new building to accommodate expanded production and commercialization of proprietary natural food additives.

Supported Angel Investments

Ecosystem Capacity Investments to Promote Cluster Development

Examples of major investments in existing and emerging clusters:

  • Centre for Commercialization of Regenerative Medicine (Regenerative Medicine, Health Bio-Sciences)
  • The Health Ecosphere Innovation Consortium Pipeline (York University with Southlake Regional Health Centre), (Health Informatics)
  • Innovation Centre at Bayview Yards (Digital Media and Cybersecurity)
  • McMaster Automotive Innovation Centre (Automotive)
  • SOSCIP (Advanced Computing Platform)
  • Southern Ontario Network for Advanced Manufacturing Innovation (Advanced Manufacturing)
  • Southern Ontario Water Consortium (Water)
  • Sunnybrook Research Institute (Biomedical IGT)
  • Vineland Research and Innovations Centre (Horticulture)
  • Bioenterprise Corporation (Bio-Innovation/Manufacturing)
  • Canadian Film Centre (Digital Media Technologies)

The angel investment networks match potential investors with those looking for funding. The funds provided by FedDev Ontario were used in the administration and development of the various networks, helping them identify both potential investors and companies as well as assisting with conducting the due diligence necessary when vetting potential investments.

In addition to the 356 angel investments totaling $227 million in funding reported by the completed projects, the on-going projects are projected to make nearly 450 investments totaling over $150 million. The Southwestern Ontario Angel Group received support to expand and enhance angel investing in southwestern Ontario. The Community Growth Accelerator Network is implementing a strategic plan focused on member attraction, promotion, and expansion of investment into early stage companies primed for growth. Angel One Network Inc. is focused on growing an investor base of accredited investors interested in early stage innovative companies.

Capacity Development

Fifty-six of the 125 ongoing projects involve the development of capacity which will continue to be utilized beyond the end of the FedDev Ontario funded project. Thirty-five proponents budgeted for investments of more than $10 million in capital assets including 20 which budgeted more than $20 million. FedDev Ontario has provided funding to companies to upgrade manufacturing capabilities for a wide variety of products such as steel, fabricated metals, food and confectioneries, automotive and aerospace components, biologics, rail car and aluminum trailer manufacturer manufacturing, paperboard packaging, recycled rubber, and vinyl upholstery fabrics.

FedDev Ontario has also made significant investments in research, development, and commercialization capacity which will have significant impacts in terms of cluster level. Some projects focused on existing and emerging clusters, such as regenerative medicine, water, cyber security, digital media, health information technology, and border logistics, where the potential markets are large and projected to grow rapidly. Other projects have focused more specifically on segments within a cluster, on the provision of equipment, technology and support services targeted at a range of sectors, or the development of a platform technologies targeted at a range of sectors.

For example, following up on an earlier project, FedDev Ontario provided funding to increase access to SOSCIP's high performance computing platforms and encourage new collaborations leading to commercialization outcomes. SOSCIP provides a platform that brings together research universities, IBM and SMEs to establish a collaborative model of R&D and innovation utilizing the latest advanced computing technologies.

Funding was also provided for the Development and operation of CCRM's Centre for Advanced Therapeutic Cell Technologies (CATCT), a 10,000 square foot process development facility dedicated to the development of technologies and optimization of processes that will enable industrial manufacturing of human cells for therapeutic purposes. Another example is the Collaborative Greenhouse Technology Centre, which involved retrofitting Vineland's newly built greenhouse into a state-of-the-art research greenhouse of an acre in size, comparable to commercial greenhouse operators. It will be a platform for SMEs across North America to test, refine, and develop their greenhouse technologies. The Southern Ontario Network for Advanced Manufacturing Innovation (SONAMI) is a one-stop shop providing access to equipment, development and testing facilities, and product development and applied research services. MARC provides a research facility, including access to engineers, scientists, and social scientists, to assist industry (ranging from major OEMs to SMEs) in undertaking applied R&D and product development for the auto industry, focusing on sustainable solutions such as development of hybrid and electric powertrains, highly efficient and cost-effective powertrain components, and lightweight materials.

5.5 Impact of the Projects on Beneficiaries

Project beneficiaries were asked to identify what impact the projects have had on their organization. As indicated below the beneficiaries in the IBGP projects (the CME SMART ATGG program and the Yves Landry program), as well as the businesses involved in the EODP reported that the projects generated a range of impacts including increased revenues, cost savings, and the commercialization of new technologies, products, processes, or services. The EODP community innovation projects reported a range of impacts including further development of the local businesses.

Table 13: Direct Impacts of the Activities on the Beneficiary Organizations
Question: What impacts has the project had on your organization (for business projects) / community (for community projects)?
Impacts IBGP EODP
Projects Percent Projects Percent
Business Projects
Number Reporting 101 100.00% 193 100.00%
Cost savings and/or improvements in productivity for your organization? 85 84.20% 113 58.50%
Increasing the revenues of your organization? 66 65.30% 117 60.60%
Commercialization of new technologies, products, processes, or services by your organization or others?   34 33.70% 64 33.20%
Further or on-going investment by your firm in R&D?   34 33.70% 35 18.10%
Increasing your access to trained workers or highly qualified people (HQP)? 21 20.80% 42 21.80%
Increasing your access to capital (beyond the investment in the project itself)? 13 12.90% 36 18.70%
Further investment in your organization by others? 7 6.90% 20 10.40%
Licensing arrangements or transfer agreements between your organization and others including intellectual properties? 3 3.00% 8 4.10%
Community Projects
Number Reporting 47 100.00%
Further development of the local businesses 33 70.20%
Further development of community economic assets 32 68.10%
Retention (including succession) of businesses in the community 24 51.10%
Establishment of new businesses in the community 21 44.70%
Attraction of business investment to the community 19 40.40%
Further development of/access to management expertise  16 34.00%
Increased access of local employers to needed skilled workers 15 31.90%
CME SMART ATGG Case Study:

Several key informants whose companies received funding through CME SMART ATGG to replace aging machinery noted that it meant they could hire sales staff leading to additional contracts, provide training, and improve working conditions.

More specifically, participants in the CME SMART ATGG program reported that the assistance helped them to improve their competitiveness by lowering costs, improving quality and productivity, and reducing production time and lead times through acquiring new equipment and technology, training employees, implementing more modern technologies and manufacturing processes, developing new products and markets, and improving safety and quality. The improvements led to increased sales with five beneficiaries reporting increases of up to over $1 million.

Participants in the Yves Landry program reported that the assistance helped them to provide training to their staff to support lean manufacturing processes, innovation and continuous improvements, implement new systems (e.g., ERP, food safety, software & technology upgrades, and processes), and hire and train new employees. Impacts on the business included the development of new customers and markets, the ability to take on more orders and work, process improvements, improved productivity and costs savings, and reduced overtime, product handling, and maintenance.

Northumberland Case Study: Team Eagle Ltd., Campbellford

Received EODP funding for the development of the Runway Aircraft Braking Availability Tester as an airfield conditions reporting solution. Successful completion of this project led to further funding through the Build in Canada Innovation Program.

Business participants in the EODP reported a wide range of impacts in terms of business start-up and development, new product development and commercialization, training and skills development, job creation, and increased access to other sources of capital. Local recipients of assistance reported that the community innovation projects provided support for economic development activities and initiatives such as research, the development of sector and regional development strategic plans, public infrastructure development (ranging from trails and wireless services to dock expansion and park revitalization), training, services for businesses start-ups and social enterprises, business counselling, business incubators, investment attraction activities, business retention and expansion, regional marketing and branding activities, and website development.

5.6 On-going Impacts of the Projects

The impacts of the projects will continue to grow over time. Many of the projects resulted in improvements to manufacturing operations, facilities and equipment or the development of research capacity or other permanent assets, the benefits of which will continue on well beyond the end of the project. In some cases, businesses have learned the effectiveness of marketing or training activities and will continue to implement these types of activities without funding support. When asked to rate the extent to which the project activities will continue on after completion of the FedDev Ontario project, the proponents provided an average rating of 4.1 (using a scale of 1 to 5, where 1 is not at all, 3 is somewhat and 5 is to a great degree). The elements that will not continue or will be reduced in scope in the absence of the FedDev Ontario funding consist primarily of the services and support implemented by third party organizations (e.g., the CME SMART ATGG program, regional EODP programs, organizations such as OBIO or Bioindustrial Innovation Canada, and some of the angel investor networks).

An analysis conducted by Statistics Canada in 2017 illustrates the longer-term impacts that the FedDev Ontario business support programs can have on businesses.Footnote 27 FedDev Ontario provided the Centre for Special Business Projects with a list of enterprises that received support during the period from 2008 to 2016 (covering companies funded under the SOPP suite of programs, as well as under programs from previous mandates including SODP, SOA and EODP). Of these companies, 2,779 enterprises were successfully matched to Statistics Canada's Business Register (BR). Statistics Canada then compared the growth of these firms to the growth of a comparison group and found that, over time the FedDev Ontario clients reported significantly higher average annual revenue growth of 10.1 percent, 6.2 percent higher employment growth and 3.4 percent higher productivity growth than non-clients. The firms also, on average, spent 21.9 percent more on R&D activities than non-clients. Furthermore, the survival rate of FedDev Ontario clients tended to be higher than in the case of non-clients.

Input/output modelling conducted by The Conference Board of Canada found that every $1.00 from FedDev Ontario core program investments generates $3.50 of value in the broader economy (via direct, indirect, and induced impacts). Finally, the boost to economic activity by Agency core investments is expected to return about $609 million to the government (municipal: $76 million; provincial: $279 million; federal: $254 million).

6. Program Design and Delivery

This chapter provides a summary of the findings regarding program design, delivery and cost-effectiveness.

6.1 Program Design and Delivery

A series of statements were developed regarding program design and delivery. The 117 proponents and the 40 unfunded applicants were asked to indicate whether they strongly agreed, somewhat agreed, neither agreed nor disagree, somewhat disagreed or strongly disagreed with each statement. The results are summarized in the table on the following page. As indicated, a majority of the proponents strongly agreed that:

The most significant concerns amongst the proponents were the length of the application and approval process, the reporting requirements, and usefulness of the performance measurement data. Given they invested in the process but were eventually not successful in accessing funding, the non-funded applicants tended to be less supportive of various aspects of the program design and delivery, particularly the length of the application and approval process (i.e., how long it took them to be notified that they would not be funded), their dealings with FedDev Ontario staff, and the direction provided with respect to what was expected in a proposal.

The proponents, unfunded applicants and non-applicants were asked to identify what they see as the primary advantages and disadvantages of the FedDev Ontario program. The advantages most commonly identified were the level of funding available (identified by 83 percent of respondents), the types of costs covered (67 percent), and ease of access (46 percent). The disadvantage most commonly identified was the reporting requirements (48 percent). A few applicants talked about the application process being unclear and cumbersome (e.g., lack of transparency in process, lack of clarity in guidelines and eligibility, multiple points of contact, long leadtimes for funding announcement, missed opportunities to promote, red tape). Others noted difficulties in accessing funding as the five-year mandate comes to a close (the need to wait for more funding to become available).

While perceptions of program design and delivery were largely consistent across programs, there were some variations. Not-for-profit proponents of large multi-year ICP projects, which can involve partnerships between MNEs, SMEs, institutions, and not-for-profit organizations, were those most likely to express concerns about the length of the application and approval process as well as the reporting requirements. Challenges with intellectual property rights were much more common amongst proponents in the AMF and ICP programs while challenges with the SR&ED tax credits were most common for businesses participants in the AMF and, to a lesser extent, the IBGP. Amongst the unfunded applicants, ICP, IBGP and IBI clients were those most likely to identify issues with the length of the application process and dissatisfaction with the FedDev Ontario staff.

The following sections summarize the input provided by key informants, proponents, and unfunded applicants regarding program design and delivery issues, including the key issues that were identified and their suggestions for improvement.

Program Design

The prevailing sentiment amongst proponents and key informants is that the programs are well-structured, each filling an important role or niche which targets one or more key issues constraining development of the economy (e.g., access to capital and talent, ability to develop and commercialize new technology, need to modernize the manufacturing sector, need to improve the economy in eastern Ontario, etc.). The programs are considered effective in reaching the key target groups. DemandFootnote 29 for most programs has been strong as demonstrated by the total of 4215 project applications received by FedDev Ontario and its third party delivery agents.

Targeting not-for-profit organizations, through non-repayable contributions, and for profit businesses, through repayable contributions, FedDev Ontario is able to implement the parallel strategies of solidifying the ecosystem that drives development (through increasing access to capital, research capacity, technology, and support services) and strengthening businesses (that generate economic growth by establishing, expanding and modernizing operations, developing new products and markets, and commercializing new technologies). The key informants particularly highlighted the role of the program in facilitating strategic growth of existing companies, enabling early-stage companies to bridge the commercialization gap and “valley of death”Footnote 30. Through the EODP, IRD and other programming, FedDev Ontario has also been able to support communities in addressing tough economic conditions by, for example, leveraging their strengths and better supporting the development and survival of smaller companies and social enterprises as a means to compensate for the loss of larger companies.

Table 14: Level of Agreement with Statements Regarding Program Delivery and Design – Proponents and Unfunded Applicants
Statements Strongly Agree Somewhat Agree Neither Agree nor Disagree Somewhat Disagree Strongly Disagree N/A Total Average
# % # % # % # % # % # % # %
Proponents
The design and delivery was appropriate given what the programming is trying to accomplish. 76 65% 32 27% 5 4% 2 2% 0 0% 2 2% 117 100% 4.6
The application requirements and criteria were appropriate. 62 53% 44 38% 4 3% 4 3% 0 0% 3 3% 117 100% 4.4
FedDev Ontario provided clear direction to you regarding what was is expected in a proposal. 63 54% 47 40% 3 13% 1 1% 0 0% 3 3% 117 100% 4.5
The length of the application and approval process was appropriate. 36 31% 40 34% 13 11% 17 15% 8 7% 3 3% 117 100% 3.7
The reporting requirements were effective in reporting on the achievement of outcomes. 47 40% 44 38% 8 7% 12 10% 2 2% 4 3% 117 100% 4.1
The performance measurement data was useful to your organization for decision making. 30 26% 39 33% 28 24% 10 9% 4 3% 6 5% 117 100% 3.7
Sufficient time was provided for the project to be completed. 66 56% 36 31% 3 3% 7 6% 2 2% 3 3% 117 100% 4.4
You are satisfied in dealings with FedDev Ontario staff. 91 78% 20 17% 2 2% 1 1% 1 1% 2 2% 117 100% 4.7
The project created some challenges for your organization or others involved with respect to intellectual rights or ownership. 6 5% 8 7% 12 10% 17 15% 40 34% 34 29% 117 100% 2.1
The project created some challenges with respect to SR&ED tax credits. 3 3% 12 10% 18 15% 11 9% 31 26% 42 36% 117 100% 2.3
Unfunded
The design and delivery was appropriate given what the programming is trying to accomplish. 2 5% 17 43% 7 18% 7 18% 5 13% 2 5% 40 100% 3.1
The application requirements and criteria were appropriate. 3 8% 11 28% 6 15% 11 28% 6 15% 3 8% 40 100% 2.8
FedDev Ontario provided clear direction to you regarding what was is expected in a proposal. 1 3% 11 28% 4 10% 9 23% 12 30% 3 8% 40 100% 2.5
The length of the application and approval process was appropriate. 1 3% 8 20% 3 8% 7 18% 17 43% 4 10% 40 100% 2.1
You are satisfied in dealings with FedDev Ontario staff. 2 5% 8 20% 3 8% 8 20% 15 38% 4 10% 40 100% 2.3

There was support, both within and external to the organization, for maintaining the existing program structure given that the suite of programs works well together to address key needs and are well-designed and delivered. Continuation will mean that FedDev Ontario avoids a situation similar to that of 2013, needing to create awareness of a new suite of programs.

The movement to fund larger projects was supported by key informants, who indicated that making significant investments in the innovation ecosystem and leveraging significant investments by major anchor companies can be highly effective strategies for supporting cluster development and regional development. The average size of FedDev Ontario contributions increased across all programming versus the previous mandate (including EODP and IBI). As indicated below, the average contribution per project increased from $1.45 million per project to $3.62 million, an average increase of 150 percent.

Table 15: Average Value of Projects Supported Under the Current and Previous Mandates
Programs Projects Value ($ millions)
Contributions Avg./Project
Programs in Previous Mandates
SODP – First Intake 108 $183.10 $1.70
Graduate Enterprise Internship 10 $17.90 $1.79
Scientists and Engineers in Business 12 $15.50 $1.29
Youth STEM 15 $13.70 $0.91
ARC Initiative 47 $30.30 $0.64
Technology Development Program 6 $63.90 $10.65
Investing in Business Innovation 106 $60.00 $0.57
Prosperity Initiative – NFPs/PSIs 48 $211.40 $4.40
EODP 91 48 $0.53
Total 443 $643.80 $1.45
Current Five Year Mandate – SOPP Programs (2013-14 to 2017-2018)
Investing in Business Innovation 88 $79.7 0.91
Investing in Business Growth & Prod 51 $204.9 4.02
Investing in Commercialization Partnerships 12 $123.8 10.32
Investing in Regional Diversification 12 $63.4 5.28
SOPP Strategic Project 1 8.0 8.00
Advanced Manufacturing Fund 9 $160.0 17.78
EODP 17 $48.0 2.82
Total 190 $687.8 $3.62

Of the 443 projects funded prior to the current mandate, 17 (4 percent) involved contributions of $10 million or more including five which included contributions of $20 million or more. Of the 190 SOPP projects funded during the current mandate, 21 (11 percent) involved contributions of $10 million or more including seven which included contributions of $20 million or more.

FedDev Ontario reviews its programming on an on-going basis and makes some changes to improve program design and delivery. Examples of changes include development of the Professional Practice PilotFootnote 31; increased use of third party delivery models; revisions to centralize administration of the collaborative projects component (CEDP) of the EODP; easing the required timelines for completion of EODP projects funded at the local level; and other improvements to processes, tools, project databases, and templates. The quality of applications has also tended to improve over time as organizations and staff became more familiar with the new programming.

The most serious concern regarding the program design relates to the five-year funding mandates of FedDev Ontario, which according to both key informants and proponents impacts on what projects can be funded, particularly in the later years of the mandate. The finite nature of the mandate:

Table 16: Value of Funding Approved by Year and Duration (as of May 24, 2017)
Duration of Projects Year of Project Approval
Prior 2014-15 2015-16 2016-17 2017-18 Total Percent
Up to 1 year $0.70 $0.50 $1.20 0.20%
Up to 2 years $43.00 $7.30 $5.10 $0.20 $55.60 8.10%
Up to 3 years $5.40 $30.60 $54.80 $34.60 $125.40 18.20%
Up to 4 years $34.70 $111.60 $69.70 $216.00 31.40%
Over 4 years $15.10 $274.50 $0.50 $289.70 42.10%
Total $55.20 $460.40 $132.90 $39.70 $0.20 $687.80 100%
Percent 8.00% 66.90% 19.30% 5.80% 0.00% 100%

Other issues or opportunities that were identified by key informants or project proponents included:

Application Requirements, Criteria, and Reviews

Most proponents, as well as those key informants who expressed an opinion, felt that the application requirements, criteria and process were appropriate given the objectives of the programs. Most concerns related to the definition of organizations eligible to apply for funding. Suggestions focused on:

With respect to EODP, it was also suggested that consideration be given to varying the level of funding provided to each CFDC depending on local opportunities and needs (either through a pooling of funding or directly providing differing levels of funding); providing the CFDCs the option of offering repayable contributions; and, where warranted, allowing CFDCs to provide larger individual contributions to fill a perceived gap existing between the EDOP and the larger SOPP programs to which regional businesses and not-for-profits would not normally be able to access funding.

A few proponents, unfunded applicants and key informants expressed reservations as to whether project officers have the requisite experience and industry specific knowledge needed to fully review the applications, particularly for more complex, highly technical or niche projects. It was noted that there has been considerable turnover in the officer positions. Difficulties in assessing applications can lead to delays in the approval (or non-approval) of applications. To help ease some of these issues, FedDev Ontario is developing a Professional Practice pilot program to help accelerate the development of the Agency's capacity in terms of economic and business practice intelligence. Some of the key elements of the pilot program include business information sessions, recommended reading lists, and the development of personal learning plans.

A few key informants suggested that FedDev Ontario could complement its internal resources by contracting with one or more outside agencies (e.g., another government department or the private sector) to assist in the review of certain aspects of project applications; this strategy has been used by other RDAs. For example, both WD and ACOA have engaged NRC IRAP to conduct technical reviews of certain types of applications.

FedDev Ontario did develop an MOU with Industry Canada (now ISED) regarding the review of applications under the Advanced Manufacturing Fund. Under the MOU, FedDev Ontario retained responsibility for project approval but obtained input from ISED regarding the technical aspects (innovation), market relevance and potential spillover benefits of the proposed project. At times, ISED contracted with a private sector contractor to assist in that review. The arrangement was initially complicated by the inexperience the two groups had in working with each other, some differing expectations, and differences in the reporting relationships (at the beginning, ISED and FedDev Ontario reported to different Ministers). In addition, the large number of applications received relative to the number eventually approved placed some pressure on the model (41 applications were received including 20 in the first intake from December 2013 to October 2014 and 21 in the second intake which ran from January to October 2015, of which only 8 were approved). In the interviews, some concerns were expressed regarding the amount of time that the reviews would take. Different opinions were expressed by key informants regarding the utility of the Project Advisory Committee (PAC), which brought together representatives from ISED and FedDev Ontario to support management of the AMF. Most of the initial issues were worked out over time, as personal relationships developed and the reporting relationship of the organizations changed. Overall, the arrangement was considered effective in bringing together knowledge of the region (FedDev Ontario) with knowledge of the sector and technology knowledge (ISED).

Direction Provided to Applicants Regarding Proposals

The most common concern from unfunded applicants is that they would have liked to have known, earlier in the process (preferably before preparing a full proposal) that they were unlikely to be successful in accessing funding. It was suggested, for example, that it would be useful to know how much funding FedDev Ontario had remaining. There were some anecdotal stories that applicants submitted an application, only to be told that funds were no longer available and that they should try again when funds are available.

There was also some uncertainty as to what proposals should address, what is key to a successful proposal and, in the case of unsuccessful proposals, how the proposal scored and the specific reasons why it was not successful so that improvements can be made to future proposals. While the broad eligibility guidelines for the programs are published, some applicants went further and recommended that FedDev Ontario should inform applicants which types of projects are preferred (preferably supported by a formal scoring system) so that a more informed decision can be made regarding whether to prepare a proposal.

A few representatives suggested that FedDev Ontario offer a single point of entry, with a common application form rather than promoting each program separately. Some companies expressed confusion about what support each program provided and under which they would be eligible. By adopting a single point of entry, it was suggested that the responsibility for matching applications to programs would be shifted to FedDev Ontario to decide under which program a potential project would best fit. Another alternative would be to have the potential applicant fill out a short questionnaire defining the key characteristics of their organization and proposed project, which would then be matched against the program eligibility criteria to suggest programs or to indicate that the project as outlined is not eligible for funding.

Length of the Approval Process

The most common concern regarding program design and delivery relates to the length of the approval process and, perhaps more importantly, the unpredictability of the timing. Some applicants (including those that were approved) indicated that they did not receive clear guidance regarding how long the approval process would take. Even once projects are approved, there can still be an extended period before the contribution agreement is in place and the project can actually proceed. Unpredictability regarding timing is a major concern because it makes it very difficult for the organizations to plan and ensure that the proposed partnerships and resources will still be in place when the go ahead is received. Delays can also contribute to a loss of momentum, shorten the timelines available for implementation, and prevent projects from going ahead at the speed of business. One proponent, for example, indicated that their project waited for approval for about 18 months which meant that the project, once approved, had to both ramp up very quickly while being restructured in order to meet the new timelines.

Factors that can contribute to delays include challenges in reviewing the applications (e.g., gaps in the information provided by applicants, inexperienced officers, and the need to obtain input from outside agencies), delays in obtaining sign-off, and difficulties in the negotiation of contribution agreements. It was also noted that FedDev Ontario has little additional capacity to handle surges in application volumes. It was suggested that capacity constraints could be eased somewhat by further clarifying the eligibility requirements and finding ways to screen applications that are not going to be approved earlier in the process. It was also recommended that FedDev Ontario publish its standards regarding the time required to approve and sign contribution agreements for projects and that performance against those standards be made available publicly. Regular updates regarding expected timing should be given to proponents of active applications.

The timing of the election in 2015 may have delayed some projects. There is also at least the perception, amongst proponents, that it is more difficult to gain approval and sign-off for a follow-on project than for the initial project (some proponents reported that negotiations for the follow-up funding took over a year to finalize). It can be difficult to keep a consortium together during such a long waiting period.

The Reporting Requirements and the Usefulness of the Performance Measurement Data

Some concerns were expressed by both proponents and key informants regarding the reporting requirements and the usefulness of the performance measurement data itself. As part of this interim evaluation, a detailed review was conducted of the program data including project approval and project summary forms, contribution agreements, the tombstone data, progress reports and closing reports. Some of the key observations from the surveys, interviews and data review include:

The Time Available to Complete Project

Most proponents indicated that the time available to complete the project was appropriate. The multi-year projects were those most likely to indicate that the time was not appropriate, noting that a variety of factors (e.g., later than expected approval) delayed the start of the projects. Given the fixed term of the mandate, the end date could not be extended.

Satisfaction with FedDev Ontario Staff

FedDev Ontario staff were praised for being knowledgeable, helpful to applicants navigating the application process and tailoring their applications to better meet the eligibility requirements, helpful to proponents in implementing the project, and responsive to inquiries. As indicated earlier, some unfunded applicants expressed dissatisfaction with not receiving guidance during the application process that would either have improved their likelihood of success or informed them earlier that they were unlikely to be successful. Some also expressed concern about the level of program staff turnover, disrupting the personal relationship and level of understanding that can develop between the project officer and the proponent.

In addition to funding, FedDev Ontario staff was praised for providing hands-on support and engaging in discussions during project implementation. This helped ensure that targets were met and the project continued on the right track. For example, in some projects, FedDev Ontario representatives participated in the conferences and meetings and engaged in open discussion with stakeholders, which helped increase awareness and understanding of the requirements, interests and operations amongst various contributors to the project. Having a single point of contact within FedDev Ontario who understands the project, issues, partners involved and the environment in which the project operates was noted by project proponents as an important aspect of FedDev Ontario contribution to success of the project. Flexibility in being able to move funding across years was noted as helpful in cases where projects were delayed for whatever reasons.

The key informants noted having project officers work with applicants during the application process improves the quality of applications and can reduce the extent to which they will need to go back and forth with requests for information during the approval process. There is at least the perception that review processes may be inconsistent across reviewers and over time, leading to some confusion.

Intellectual Property Rights or Ownership

Fourteen proponents identified issues with respect to intellectual property rights. Ownership of the IP varies from project to project, depending in part on the policy of the participating organizations (e.g., some universities retain ownership of any IP while, at others, the IP is owned by the creator and/or can be negotiated). Ownership is an issue because it directly affects the willingness of private sector partners to participate in a project and the potential economic benefits that could be generated, which impacts the ability of the commercializing entity to raise venture capital or other sources of financing. In most instances, the ownership issue can be addressed through negotiation and most commonly through the use of waivers. The key implications are the importance of identifying potential IP issues early in the application process, taking steps to address that issue so it will not hold up the project, and considering the impact of the issues on the potential for commercialization in decisions as to whether the project should be supported.

SR&ED Tax Credits

Fifteen proponents identified SR&ED tax credits as an issue, although only three strongly agreed it was a major issue. As noted earlier, CRA treats some repayable loans provided by FedDev Ontario as a grant rather than as a loan, which can increase taxable income and reduce eligibility for SR&ED tax credits. The tax interpretation of the contributions seems to vary from region to region within southern Ontario and even from case to case. FedDev Ontario had approached CRA previously about this issue, without a clear resolution. Rather than asking for a change in CRA policy, another option may be for FedDev Ontario to review and revise the design of its repayable loans to improve the likelihood that they will be treated as loans for tax purposes (perhaps structuring them as loans).

6.2 Cost-Effectiveness

The major findings of the interim evaluation regarding the cost effectiveness of the SOPP programs are as follows:

  1. The funding contributed by FedDev Ontario has leveraged significant investments from the proponent organizations, other private sector investments, and other funding from the federal government, the provincial government and other sources. The table below illustrates the level of funding contributed by other sources for every dollar contributed by FedDev Ontario. Overall, the projects received $2.45 in funding for every dollar contributed by FedDev Ontario, with the leading source being private sector proponents (particularly those participating in the AMF and the IBGP). Other sources include not-for-profit proponents, other private sector sources (project partners or lenders), and the provincial government.
Table 17: Average Funding Contributed From Other Sources For Every Dollar Provided By FedDev Ontario
Sources of Funding Projects Value ($million) Percent Per $1 of FDO Funding
FedDev Ontario 190 $687.8 29.0% --
Private sector proponents 112 $1,156.0 48.7% $1.68
Not-for-profit proponents 33 $170,0 7.2% $0.25
Other private sector 21 $160.2 6.8% $0.23
Other federal government 16 $22.9 1.0% $0.03
Provincial government 38 $146.7 6.2% $0.21
Local government 4 $9.2 0.4% $0.01
Other 8 $19.9 0.8% $0.03
Total Sources of Funding 190 $2,372.7 100.0% $2.45

The following table illustrates the funding leverage by program. As indicated, the average degree of leverage ranged from no other sources of funding for the EODP and SOPP Strategic Project to $2.55 for the AMF and $4.33 for the IBGP.

Table 18: Average Funding Contributed From Other Sources by Program For Every Dollar Provided By FedDev Ontario
Program Number Value ($millions) Per $1 of FDO Funding
FDO Total Cost
Investing in Business Innovation 88 $79.7 $196.7 $1.47
Invest. in Business Growth & Prod 51 $204.9 $1,091.4 $4.33
Investing in Commercialization Part. 12 $123.8 $325.9 $1.63
Investing in Regional Diversification 12 $63.4 $133.9 $1.11
SOPP Strategic Project 1 8.0 $8.0 $0.00
Advanced Manufacturing Fund 9 $160.0 $568.8 $2.55
EODP 17 $48.0 $48.0 $0.00
Total 190 $687.8 $2,372.7 $2.45
  1. While it is premature to assess the ultimate impacts of the programs, the outcomes generated by the projects completed to date indicate that the programs are already generating significant returns on the contributions made by FedDev Ontario and are on track to achieve outcomes identified in their Contribution Agreements. The following table compares some impacts reported to date by the closed projects to the value of FedDev Ontario contributions approved for those projects. As indicated, to date, the projects have generated $2.54 in increased revenues for every dollar provided by FedDev Ontario (including $2.16 in domestic sales and $0.38 in export sales) and $2.38 in angel investment per dollar contributed. These impacts will increase over time as products and technologies are commercialized and markets are further developed.
Table 19: Return on Contributions from FedDev Ontario to Projects Completed to Date
Impacts Projects Reporting Impact Reported Value
FDO Contributions 65 -- $95.2 million
Increase Sales
Canadian 44 $205.3 million $2.16 per FDO $1.00
Export 9 $35.9 million $0.38 per FDO $1.00
Total 44 $241.2 million $2.54 per FDO $1.00
Employment
Created 59 2,295 FTEs
Maintained 20 541 FTEs
Total 60 2,836 FTEs
Angel Investment
Number of Deals 8 356 --
Total Investments 8 $226.9 million $2.38 per FDO $

The figures are marginally understated to the extent that actual expenditures tend to be slightly lower than the level of contributions approved. For the 65 closed projects, actual expenditures were equal to 98.4 percent of the approved contributions.

  1. Tables 18 and 19 contain planned and actual expenditure data that illustrate the impact of the five-year mandate on the programs, with expenditures tending to be very low in the initial yearsFootnote 32 and increasing as projects, particularly the multi-year projects, enter the full implementation stage and make claims. Actual expenditures as a percent of planned increased from 38 percent in 2014-15 to 85 percent in 2015-16 and 94 percent in 2016-17, averaging 73 percent over the three years.
Table 20: Planned and Actual G&C Expenditures Per Year and Program, 2014-15 to 2016-17
Program 2014-15 2015-16 2016-17
G&C Planned Actual % Planned Actual % Planned Actual %
EODP $9,600,000 $9,600,000 100% $9,600,000 $9,600,000 100% $9,600,000 $9,600,000 100%
AMF $40,000,000 $2,972,454 7% $52,000,000 $32,592,288 63% $51,000,000 $42,835,000 84%
IBI $20,258,252 $13,045,134 64% $18,986,466 $23,000,000 121% $15,858,231 $15,456,231 97%
IBGP $40,516,503 $16,539,346 41% $41,049,739 $40,766,577 99% $37,209,612 $41,996,041 113%
ICP $36,870,018 $4,314,956 12% $37,241,330 $20,463,485 55% $38,870,018 $34,296,182 88%
IRD $10,129,126 $14,003,722 138% $10,496,363 $17,499,846 167% $7,129,126 $6,563,799 92%
Total $157,373,899 $60,475,612 38% 169,373,898 143,922,196 85% $159,666,987 $150,747,253 94%
Table 21: Planned and Actual FTE and Operating ExpendituresFootnote 33 Per Year and Program, 2014-15 to 2016-17
Program 2014-15 2015-16 2016-17
FTE Operating FTE Operating FTE Operating
Plan Actual Plan Actual % Plan Actual Plan Actual % Plan Actual Plan Actual %
EODP 7 5 $648,023 $557,925 86 8 3 $408,279 $351,766 86 7 4 $744,339 $423,661 57%
AMF 8 8 $837,082 $961,054 115 9 9 $1,201,255 $1,071,130 89 9 10 $1,101,537 $1,079,542 98%
IBI 8 11 $891,768 $1,024,122 115 11 18 $1.083,883 $1,335,368 123 13 16 $1,275,825 $1,607,471 126%
IBGP 17 15 $2,014,525 $1,517,170 75 18 23 $1,644,695 $1,678,033 102 17 20 $1,552,279 $1,994,338 128%
ICP 14 8 $ 1,464,893 $1,049,876 72 12 12 $1,344,120 $1,424,018 106 10 9 $1,109,525 $1,017,677 92%
IRD 8 6 $929,781 $559,160 60 8 3 $ 734,526 $369,665 50 7 3 $744,339 $344,655 46%
Total 61 54 $6,786,072 $5,669,307 84 66 68 $6,416,758 $6,229,980 97 63 63 $6,527,844 $6,467,345 99%

Similarly, operating costs as a percent of budgeted expenditures increased year-over-year, from 84% in 2014-15 to 97 percent in 2015-16 and 99% in 2016-17, averaging 93% over the three years.

  1. Operating costs as a percent of grants and contributions are very low. The financial data indicates that FedDev Ontario operates with a very lean operating structure, with operating expenditures averaging 4.9% of total program expenditures.
Table 22: Operating Expenditures as a Percent of Contribution Expenditures By Program, 2014-15 to 2016-17
Program Program Expenditures Operating Percentage
Operating G&C Total
EODP $1,333,352 $28,800,000 $30,133,352 4.40%
AMF $3,111,726 $78,399,742 $81,511,468 3.80%
IBI $3,966,961 $51,503,365 $55,470,326 7.20%
IBGP $5,189,541 $99,301,964 $104,491,505 5.00%
ICP $3,491,571 $59,074,623 $62,566,194 5.60%
IRD $1,273,480 $38,067,367 $39,340,847 3.20%
Total  $18,366,631 $355,147,061 $373,513,692 4.90%

Two factors identified as contributing to the decrease in the operating costs were the increased average size of approved contributions (while larger contributions can be more expensive to administer they tend to be proportionately less expensive) and increased use of third parties to administer programs funded by FedDev Ontario. For example, organizations such as the Canadian Film Centre, Bioindustrial Innovation Canada, Alliance of Canadian Manufacturers & Exporters, Yves Landry Foundation, Ontario Centres of Excellence, Waterloo Accelerator Centre, Ontario Bioscience Industry Organization, Communitech Corporation, and Next Canada have received funding for administering programs funded by FedDev Ontario and delivered by them.

Some reservations, both from within the organization and externally, were expressed that the programs may be delivered with too lean of an operating budget, particularly given the large number of legacy files (related to projects in the previous mandate) that still require monitoring. The demand for staff resources tends to be highest in the first year of a mandate, given the pressure to approve new projects while still closing off projects from the previous mandate. Staffing constraints can slow the processing of applications, particularly during peak periods, contribute to some of the data challenges that were faced in the evaluation (e.g., the need to go back to the contribution agreements, progress reports, final reports and closing reports for additional data), and impact on project monitoring.

7. Conclusions and Recommendations

7.1 Conclusions

The major conclusions arising from the interim evaluation are as follows:

Relevance

  1. There is a continued need for the programs included in the SOPP. According to key informants, there is a strong need for the programming given the importance of the Ontario economy, the significant opportunities for further growth and diversification that exist across a range of existing and emerging clusters, and the key challenges and constraints that are slowing this growth and diversification and need to be addressed. Project proponents, unfunded applicants, non-applicants and beneficiaries rated the need for the programming as high, focusing specifically on challenges to growth and diversification such as the need for capital, funding to support further development, expansion or modernization of operations, and assistance for issues such as technology development, testing and commercialization, market development, staff development and training, and business start-up and early stage development.
  2. The SOPP programs are well aligned with each other and other programming available in southern Ontario, the constraints to growth and diversification and needs of the key target groups. Taken together, the suite of SOPP programs employs a variety of delivery mechanisms to promote growth and diversification across various stages of business development, economic clusters, and regions within southern Ontario. Factors such as the place-based nature of FedDev Ontario, the strong demand for funding, and coordination between FedDev Ontario and other programming organizations help to ensure that the SOPP programs complement rather than duplicate other federal or provincial government programs that promote innovation, business development and community development.

Effectiveness of the Programs

  1. FedDev Ontario has made significant investments that are incremental and leveraged with funding from other sources. FedDev Ontario funding fills a need that would not have been met by other programs. Only 4 percent of the projects would have proceeded as planned in the absence of FedDev Ontario funding. Each project dollar contributed by FedDev Ontario was leveraged with $2.45 in funding from other sources. According to the project budgets, the primary sources of funding include private sector proponents (49 percent of funding), FedDev Ontario (29 percent), not-for-profit proponents (7 percent), contribution by private sector partners (6 percent) and the provincial government (6 percent).
  2. The SOPP programs have contributed towards achievement of the priorities of the Government of Canada, including the Innovation and Skills Plan, as well as the mandate of FedDev Ontario. Based on the results of consultations undertaken for the Innovation and Skills Plan, the Government of Canada identified three priority areas: people (ensuring that people are equipped with the right skills and experience to drive innovation); technologies (taking full advantage of transformative emerging technologies that can elevate the competitiveness of established and new firms, industries, and clusters); and companies (growing the next generation of global companies in Canada)Footnote 34. The investments made by FedDev Ontario are consistent with these priorities, particularly with respect to:
    • Attracting, developing and retaining highly skilled workers, researchers and entrepreneurs. Most projects have involved training or mentoring and proponents report that the projects have helped southern Ontario attract and retain key workers, researchers and entrepreneurs. Training is provided by a mixture of individual companies, which undertake training of their own staff (often as part of a manufacturing expansion or modernization project) and intermediaries funded by FedDev Ontario which provide training and mentoring for highly skilled workers, researchers, entrepreneurs and investors.
    • Strengthening the regional innovation ecosystem through the further development of research and development capacity, investments in technology development, testing and commercialization, and facilitating the development of collaborations and partnerships involving investors, research collaborators, project partners, businesses and others. FedDev Ontario has made significant investments in clusters identified by the Government of Canada as priorities. The funding targets a range of existing and emerging economic clusters. Manufacturing was a focus of 60 percent of approved FedDev Ontario contributions (37 percent could be categorized as advanced manufacturing). Other leading clusters include health care and biotech (26 percent of approved funding), ICT (13 percent), agri-food (8 percent) and clean tech and clean resources (7 percent).
    • Accelerating the start-up, early development, expansion and modernization of companies by attracting and facilitating investment, supporting technology adaptation, adoption and commercialization, supporting advisory services and market development activities and attracting anchor firms. For example, FedDev Ontario projects are contributing to the development of partnerships and collaborations, angel investments totaling over $375 million, and delivery of advisory and support services (ranging from business development services, technology development and testing services to training and other resources) to over 2,900 businesses.

    In addition, the programs have also supported community economic development and diversification. With the exception of the EODP, the programs were open to applicants from all regions within southern Ontario. Eastern Ontario has been identified as a specific priority for funding because of weak economic conditions, which contribute to a loss of businesses, investment and youth from the region. Of the 190 projects, 31 targeted the eastern Ontario region (excluding Ottawa and Kingston), accounting for 12 percent of approved funding.

    The activities supported under SOPP have also directly contributed to the Agency's core mandate of strengthening southern Ontario's economic capacity for innovation, entrepreneurship and collaboration and promoting development of a strong and diversified southern Ontario economy. The 65 projects which have been completed to date report increased sales of $241 million ($2.54 in increased revenues for every dollar provided by FedDev Ontario) while the 125 projects that were still ongoing were projected to generated $3.7 billion in increased North American sales, almost $600 million in export sales outside North America and nearly $1.1 billion in sales from the commercialization of 1,500 new products and technologies.

    The impacts of the projects will continue to grow over time. Most projects resulted in improvements to operations, facilities, equipment and business practices or further development of the innovation capacity, the benefits of which will continue on well beyond the end of the project.

Program Design and Delivery

  1. Operating costs as a percent of grants and contributions are low. Operating expenditures averaged 4.9 percent of program expenditures to March 31, 2017. Two factors contributing to the low percentage are increased average approved contributions per project and increased use of third parties to administer programs. The FedDev Ontario contribution per project averaged $3.62 million for the SOPP programs, an increase of 150 percent over the contributions made under similar programs in previous mandates.

    Some reservations, from within the organization and externally, were expressed that the programs' operating budgets may have become too lean, particularly given the large number of legacy files that still require monitoring. Staffing constraints can slow the processing of applications, contribute to data challenges, and impact on project monitoring.

  2. Most proponents are satisfied with the design and delivery of the programs. Proponents report satisfaction in their dealings with FedDev Ontario staff and believe the design and delivery of the programs is appropriate, they were given sufficient time to complete the project, they received clear direction regarding the development of their proposal, and the application requirements and criteria are appropriate. The most significant concerns are the length of the application and approval process (and the unpredictability of the timing), the reporting requirements, and usefulness of the performance measurement data. Some proponents also indicated challenges with the SR&ED tax credits. Given they invested in the process but were not successful in accessing funding, non-funded applicants tended to be less supportive of program design and delivery, particularly the length of the approval process, their dealings with FedDev Ontario staff, and the guidance and direction provided to them with respect to the preparation of the proposal.
  3. The most serious issue regarding program design involves the five-year funding profile of FedDev Ontario. The five year timeline means that multi-year projects can only be approved early in the mandate. Given that almost three quarters of the funding (to May 24, 2017) was approved for projects of more than three years in duration, little funding remained available for approvals in the latter years. The five-year mandate also created significant workload issues for program staff, particularly in the first year of the new mandate. During that first year, program staff need to engage with prospective applicants, and review and process applications with a particular focus on the larger scale, multi-year projects which tend to be more complex in nature. At the same time, the staff must deal with legacy projects from the previous mandate, reviewing final reports, conducting site visits, validating reported results, and preparing close-out reports. These challenges were further complicated by having the budget for the SOPP programs divided relatively evenly across the five-year period. Reflecting the time required to approve, contract and launch new projects, actual program expenditures amounted to only 38 percent of the planned expenditures in 2014-15.

7.2 Recommendations

The recommendations arising from the evaluation are as follows:

  1. Develop a formal plan for addressing the issues related to the five-year funding profile.

    The preferred option is to move to a longer-term funding model for the Agency or, if that cannot be achieved, take steps to mitigate some of the impacts. Mitigating measures could include by (1) having additional trained staff resources in place for the first year of the new mandate so that the Agency is better able to process and approve new projects while still being able to effectively monitor and close-out legacy projects; and (2) allocating the program budgets so that planned expenditures are lower in year one than in subsequent years. Another option for subsequent years may be to move to a rolling funding model where FedDev Ontario secures funding for additional years part way through its next mandate, such that the program always has three to five years of funding remaining.

  2. Maintain the same fundamental program structure for the next mandate, while exploring opportunities to refine and consolidate programs to address the current challenges and needs of the region.

    There is strong support within and outside the organization for maintaining the existing program structure given that the existing suite of programs is effectively designed, coordinated and delivered and doing so will enable the agency to build off the program awareness created over the past four years. Individual programs may need to be adjusted or adapted somewhat to reflect changing priorities of the federal government, the role of FedDev Ontario within ISED, and issues identified in the evaluation. In refining the individual programs, consideration could be given to:

    • Targeting under-represented groups including Indigenous people, women and young entrepreneurs.
    • Better allocation of EODP funding. Steps could be taken to better align EODP funding (through a pooling of funding or directly providing differing levels of funding) with the regional demand for funding, areas of particular need, and capacity of CFDCs to deliver programming. Consideration could also be given to opening EODP style funding to CFDCs in other regions and giving CFDCs the option of offering repayable contributions and, where warranted, larger individual contributions.
    • Targeting MNEs. Attracting significant investments from MNEs and building linkages between those anchor organizations, SMEs, and research, development and commercialization centres can be a very effective cluster development strategy. However, availability of low cost capital through a program like AMF may not be an effective incentive to encourage that investment, and alternatives should be examined.
    • Refining the structure of the unconditionally repayable contributions such that they are more likely to be treated as loans by CRA.
    • Providing greater flexibility with respect to the program guidelines. While the basic program structure should be continued, consideration could be given to increasing the flexibility of the programming. The program guidelines should focus more on the intended outcomes and be less prescriptive in how projects must be structured to achieve those outcomes.
  3. Offer potential applicants a single point of entry and regularly update publicly available information related to funding availability and timelines.

    It can be difficult for potential applicants to determine under which, if any programs, they may be eligible. An online form could be used to assess eligibility and guide prospective applicants to the appropriate program. Applicants requested that additional information be publicly available on service standards, the balance of funding available for project approvals, the success rate of applications, and the range in timelines to decision.

  4. Support the continued development of project officers.

    Program results can be directly impacted by the experience, knowledge and expertise of the project officers. Concerns were expressed about the rate of turnover. A strategy should be developed to both reduce the rate of turnover in the positions and accelerate the professional development of project officers through mechanisms such as the Professional Practice pilot. Consideration could also be given to augmenting internal resources by contracting with one or more outside agencies to assist in the review of certain aspects of project applications.

  5. The project reporting system should be reviewed and revised, both in terms of reporting process and the indicators on which proponents report.

    Consideration should be given to fully digitizing project files from cradle to grave (from expressions of interest to submission and review of proposals, implementation of projects, and project monitoring). The existing system of scanned documents and multiple excel files is cumbersome for applicants, proponents, officers, evaluators, and decision-makers, requires regular reentering of data, complicates validation, and restricts the ability to report on projects, proponents, and progress at the program and Agency level.

    The performance indicators should be refined to reflect the new departmental results framework and facilitate useful reporting on a broader range of results relevant to specific projects. An online system could enable the proponent to more easily report on key departmental results as well as indicators specifically relevant to their project (using skip patterns to adapt the indicators by project), which could then be validated (e.g. using automated procedures and personal follow-up where needed). The data could be rolled up easily to report on results by program, region, or sector or whatever parameter is of interest to FedDev Ontario managers on an on-going basis or in response to specific requests.

Annex I: Evaluation Methodology

I.1 Approach and Lines of Evidence

The evaluation was undertaken in three phases: Planning, Data Collection, and Synthesis, Analysis and Reporting. This evaluation used a hybrid team approach (internal evaluators and external consultants) in implementing a mixed-methods research design involving multiple lines of evidence. The following table outlines the roles of the FedDev Ontario Evaluation Directorate and GGI in undertaking the evaluation.

Table 23: Overview of the Hybrid Approach
Task or Function FDO Evaluation Directorate GGI
Leadership of the Evaluation
Method Design and Implementation/Analysis of Data Collected
Development of the evaluation matrix and work plan
Literature and document review
Review of project and operational data
Case studies
Key informant interviews
Survey of project proponents
Survey of unapproved applicants
Survey of project beneficiaries
Integration of All Lines of Evidence/Presentation of Findings
Preparation of Draft and Final Evaluation Report

The planning phase involved detailed documentation review (on FedDev Ontario, its programs, and the funded projects to identify the data available and potential sources of further information) and development of the evaluation matrix, methodology, data collection instruments and communication protocols.

The purpose of the Data Collection phase was to systematically gather data and assemble the evidence. The secondary data sources included:

The primary data sources included surveys, key informants and case studies. The surveys included:

In total, 550 people were surveyed as summarized in the table below.

Table 24: Number of Representatives Participating in the Surveys Key Informant Interviews and Case Studies
Target Group Sample Population Reached Completed Percent Familiarity RatingFootnote 35
Surveys
Proponents 190 178 117 61.6% 4.8
Unfunded Applicants 107 92 40 37.4% 4.5
Non-Applicants 150 115 28 18.7% 2.6
Beneficiaries 1,935 1,677 365 18.9% 4.2
Total Surveys 2,382 2,062 550 23.1%

Interviews were conducted with 65 key informants including:

The primary focus was to obtain input on the need for the programming, relationship to other programs, factors that contribute to and constrain achievement of the intended outcomes, and opportunities for improvement.

Six case studies were conducted. The projects were selected using a variety of criteria including program, region, size and availability of performance data. The case studies illustrate the nature of the impacts, the lessons learned, contributing and constraining factors. The case studies involved a review of documents and data (e.g., proposals, tracking forms, progress reports, final reports, and publicly available information) complemented with input from proponent organizations, project partners, FedDev Ontario officers and beneficiaries. In total, input from 138 representatives was used in completing the case studies including 8 proponents and partners, 6 FedDev Ontario project officers, and 124 organizations that were surveyed as part of the project beneficiaries' survey.

I.2 Evaluation Matrix

The table on the following page summarizes the recommended performance indicators and data sources for each of the research questions to be addressed in the review.

Table 25: Summary of Recommended Issues, Performance Indicators and Data Sources
Evaluation Issues, Questions and Indicators Data Sources for Each IndicatorFootnote 36
Document
/Lit.
Review
Data
Review
Key
Inform.
Surveys Case Studies
Pro-
ponents
Benefic-
iaries
Un-
funded
Relevance
1. To what extent do the SOPP programs continue to address a demonstrable need?
Characteristics of the projects supported: timing, approved funding and total project costs, actual project expenditures, funding by cluster, region, program, type of project, type of proponent, partnerships/collaborations, major outputs (review of administrative data on the project approvals)
Consistency of the strategic investments made into the key economic drivers by SOPP programs and reported outcomes with the needs highlighted in recent industry and policy research and development strategies (results of the document and literature review including results of the 2016 Cluster study as well as past evaluations of FedDev Ontario programs and similar programs)
Extent to which SOPP projects are meeting the needs of industry and key stakeholders (feedback from proponents on the extent to which the support met their needs)
Evidence of continued need and/or demand for the programming (e.g., trends in funding requests from industry and other stakeholders in southern Ontario, perception of needs amongst project proponents, beneficiaries and unfunded applicants)
2. To what extent do the SOPP programs complement, duplicate or overlap other government programs?
Characteristics of other federal and provincial programs and initiatives that address the same needs in southern Ontario (e.g., alternative sources of funding identified by proponents and unfunded applicants, other similar programming identified by key informants, literature review on the characteristics of other similar programming)
Informed opinion on degree to which the SOPP programming complements, overlaps or duplicates other federal or provincial “programs”/initiatives in southern Ontario (perceptions of key informants, judgement by the evaluators)
Comparative advantages and disadvantages of SOPP programming relative to other similar programs (perceptions of proponents, unfunded applicants, and key informants)
Coordination and/or inter-relationship between FedDev Ontario programs and other programs in terms of referrals and joint funding of projects (program data on leverage of FedDev Ontario contributions/other sources of funding utilized; key informant interviews)
3. To what extent is the SOPP aligned with current government priorities, including the elements of the Innovation and Skills Plan (Innovation and Skills Plan), the Innovation Charter's Areas of Action (People, Technologies, Companies) and RDA 2.0? Is there a need to reposition for the future?
Key elements within the Innovation and Skills Plan, the Innovation Charter’s Areas of Action (People, Technologies, Companies) and broader government priorities (review of documentation)
Roles of the RDAs within the new structure (review of documentation, perceptions of FedDev Ontario management)
Consistency of the major investments, outputs and reported outcomes with the Innovation and Skills Plan, the Innovation Charter, IRID and other government priorities/gaps and areas of weak alignment (review of investments, outputs and intended outcomes; opinions of FedDev Ontario management)
Recent planned and potential changes that would better align the programming with these priorities (opinions of FedDev Ontario management)
Effectiveness
4. To what extent is the SOPP achieving the expected outputs and outcomes (immediate, intermediate and ultimate outcomes)?
Compilation of project data on targets and results reported to date (from project databases and review of project applications, contribution agreements, project summary forms and project application forms, progress reports, final site visit reports, completion or final reports, website information, press releases and communications)
Updating of projected and reported data on key project outputs and outcomes based on the results of surveys of proponents and beneficiaries as well as case studies in areas relevant to the programming such as (drawn from proponent and beneficiary surveys and case studies as well as available Statistics Canada analyses)
Evidence that the immediate outcomes are being achieved according to expected timelines (comparison of results to targets and timelines; opinions of the proponents)
Extent to which the projects and the resulting impacts (will) continue on and grow beyond the end of the original project funded by FedDev Ontario (evidence to date, plans and sources of support related to sustainability of the resources, capabilities and activities supported by the projects; projected future impacts of the projects as per the project documentation and perceptions and plans of the proponents and beneficiaries)
Plausibility of the linkages between immediate and intermediate outcomes (role of projects in promoting further development; mapping of the projects, activities and outcomes against the key economic drivers and the development needs of industry; opinions of key informants; judgement by evaluator)
5. Attribution: To what extent can these impacts be attributed to the support provided by FedDev Ontario?
Role of FedDev Ontario in the development, implementation and funding of specific projects and activities (interviews with FedDev Ontario project officers and survey of proponents)
Perceived likelihood that the projects/activities would have been implemented even in the absence of the support provided by FedDev Ontario (survey of proponents)
Percentage of unfunded projects that proceeded (survey of unapproved project applicants), extent to which they proceeded as planned (scope and timing), other sources of funding used, and impact on the success of the projects
FedDev Ontario's influence on the involvement of funding partners (interviews with representatives from other programs)
6. What unintended outcomes have been achieved?
Evidence regarding types and magnitude of unintended or unanticipated impacts generated by the projects (comparison of outputs and outcomes to intended outputs and outcomes, perceptions of key informants, proponents and beneficiaries)
Relationship of unintended impacts to the achievement of intended impacts, effects and goals (perceptions of key informants, proponents and beneficiaries)
7. What are the facilitators and barriers to achieving expected outcomes?
Extent to which activities were implemented as designed (perceptions of proponents; information from progress reports; results of case studies)
Specific factors which may have contributed to any differences in how projects and activities were implemented and what outputs and outcomes were generated (perceptions of proponents; information from progress reports; results of case studies)
(Other) factors that contributed to or constrained the impact of the SOPP programs and projects (perceptions of key informants and proponents; information from progress reports; results of case studies)
Efficiency
8. In what manner and to what extent is the design and delivery of the SOPP efficient and cost-effective?
Comparison of efficiency indicators across time, across SOPP programs and with other programming (e.g. program operational costs/total program budget; leverage of funding; repayable and non-repayable funding; allocation of project funding)
Perceptions regarding the efficiency and cost-effectiveness of program design and delivery including allocation of funding across programs; target groups; funding mechanisms (repayable and non-repayable contributions); program promotion, role of FedDev Ontario in the development of proposed projects; processes involved in the review and approval of applications, negotiation of contribution agreements, and provision of funding, monitoring and reporting procedures, and governance structure (interviews with key informants)
Partial cost-benefit analysis comparing the benefits of projects and programs to the costs (e.g., calculating return on investment for specific types of impacts and programming)
9. Are there more efficient and cost-effective ways of achieving expected results, taking into consideration alternative delivery mechanisms, promising practices and lessons learned?
Lessons learned and promising practices identified regarding the design and implementation of the funded projects/use of program funding to facilitate strengthening of the innovation eco-system and target clusters (results of case studies)
Recommendations for improvements to the existing programming (interviews with key informants, surveys of proponents. beneficiaries, and unfunded applicants, and case studies)
Views on whether there are alternative, more efficient, ways of delivering the programming (interviews with key informants, surveys of proponents. beneficiaries, and unfunded applicants, and case studies; reviews of alternative design and delivery structures used by other programs)

I.3 Challenges and Mitigation Strategies

The major challenges associated with the evaluation and the mitigation strategies are described below.

Given these mitigation measures, the evaluation team believes that the limitations of the study were adequately addressed and the results of the evaluation are deemed to be reliable and valid.

Annex II: The SOPP Programs and Logic Model

II.1 Program Activity Architecture

According to the Program Alignment Architecture (PAA)Footnote 38 and Agency Performance Measurement Framework (PMF), FedDev Ontario focuses its efforts on four program areas: Technological Innovation, Business Development, Community Economic Development, and Internal Services, as shown in the table below. The SOPP includes the two Technological Innovation sub-programs (AMF and ICP), two of the three Business Development sub-programs (IBI and IBGP), and two Community Economic Development sub-programs (EODP and IRD). The Agency plans to dedicate 222 full-time equivalents (FTEs) and $269.3 million to these activities in 2017-18.

Table 26: FedDev Ontario Program Alignment Architecture, 2017-18
Program Budget ($ million) FTEs Sub-Programs/Relevant Initiatives
Source: Planned FTES and Budget from FedDev Ontario's Departmental Plan 2017–18
Technological Innovation $93.1 19
  • 1.1.1 Advanced Manufacturing
    • Advanced Manufacturing Fund (AMF)
  • 1.1.2 Commercialization Partnerships
    • Investing in Commercialization Partnerships (ICP)
Business Development $55.0 51
  • 1.2.1 Business Investment
    • Investing in Business Innovation (IBI)
  • 1.2.2 Business Growth and Productivity
    • Investing in Business Growth and Productivity (IBGP)
  • 1.2.3 Business Services
Community Economic Development $105.3 40
  • 1.3.1 Community Futures Program
  • 1.3.2 Eastern Ontario Development Program (EODP)
  • 1.3.3 Official Language Minority Communities
  • 1.3.4 Regional Diversification
    • Investing in Regional Diversification (IRD)
  • 1.3.5 Infrastructure Delivery
    • Brantford/Mohawk Lake
    • Canada 150 Community Infrastructure Program
Internal Services $15.9 112
  • Management and Oversight, Communications, Legal, Human Resources Management, and Financial Management
Total $269.3 222

II.2 Overview of the SOPP Programs

Investing in Business Innovation (IBI)

The Investing in Business Innovation (IBI) initiative provides mentorship, entrepreneurial support and financing to help new businesses grow and succeed. The initiative is designed to foster a more competitive southern Ontario economy by focusing on providing business support to new entrepreneurs, helping them transform their ideas into globally-competitive products and services, and increasing their access to private sector investment and advice. The objectives are to foster a culture of entrepreneurship focused on innovation by:

Through the IBI, support could be provided for early stage SMEs, angel investor networks, and the delivery of skills development and seed financing for new entrepreneurs through not-for-profit organizations.

Investing in Business Growth and Productivity (IBGP)

The Investing in Business Growth and Productivity (IBGP) initiative focuses on established southern Ontario businesses that have the potential to be global players with innovative and unique opportunities to accelerate growth and support job creation. This initiative supports economic growth and job creation by helping businesses diversify markets and expand facilities, adopt new technologies and processes to improve productivity, and increase business capacity to grow and diversify markets. The objective is to position southern Ontario businesses to be more competitive in the global market by:

Funding could be provided directly to SMEs as well as for services delivered to SMEs by not-for-profit organizations.

Investing in Commercialization Partnerships (ICP)

The Investing in Commercialization Partnerships (ICP) initiative supports business-led partnerships with a focus on developing globally-competitive products and services. Increased collaboration among businesses, post-secondary institutions and research organizations narrows the gap between innovation and commercialization. This initiative helps to increase the capacity of existing and emerging innovation ecosystems and the development of competitive economic clusters in southern Ontario.

The ICP could provide up to $20 million (up to 50 percent of eligible costs; the remaining 50 percent must be provided by other partners) to support business-led partnerships with a focus on developing globally competitive products and services.

Investing in Regional Diversification (IRD)

The Investing in Regional Diversification (IRD) initiative supports the long-term development of stronger, more diverse economies in southern Ontario communities. IRD leverages unique regional assets and local expertise to attract new investment and opportunities for economic growth and development.

Under the IRD, not-for-profit organizations located in southern Ontario whose mandate includes economic development could receive up to $20 million (up to 50 percent of eligible costs; the remaining 50 percent must be provided by the recipient as a cash contribution) to leverage regional assets and local expertise and attract new investment and opportunities for economic growth and development.

The Advanced Manufacturing Fund (AMF)

Established as part of the 2013 Federal Budget, the Advanced Manufacturing Fund (AMF) supports research and innovation organizations, the private sector, post-secondary institutions (PSIs) and not-for-profit organizations to work together to accelerate the development of large-scale, advanced technologies that will result in new market opportunities for Ontario businesses in manufacturing sectors. The objective is to increase firm productivity and enhance the competitiveness of Ontario's advanced manufacturers by:

The AMF could provide from $10 million to a normal maximum of $20 million:

Eastern Ontario Development Program (EODP)

The Eastern Ontario Development Program (EODP) was initially established in 2004 and has been administered by FedDev Ontario since the Agency was established in 2009. It is an economic development initiative aimed at addressing economic challenges in eastern Ontario and taking advantage of innovative opportunities in the region.The program is delivered through eastern Ontario's 15 CFDCs and promotes business development, job creation and strengthened economies in rural eastern Ontario communities. In addition, the EODP provides funding for Collaborative Economic Development Projects (CEDP) which generate benefits for multiple communities and promote broad-based collaborative economic development.

II.3 Program Logic Model

The figure on the following page illustrates the logic model for SOPI, the AMF, and EODP according to the Performance Measurement Strategy, which was developed in February 2014.

Figure 2: Logic Model for Southern Ontario Prosperity Initiatives, Advanced Manufacturing Fund and Eastern Ontario Development Program, 2014-19

Logic Model for Southern Ontario Prosperity Initiatives, Advanced Manufacturing Fund and Eastern Ontario Development Program, 2014-19 (the long description is located below the image)
Description of Figure 2

Figure 2 depicts a logic model for Southern Ontario Prosperity Initiatives, Advanced Manufacturing Fund and Eastern Ontario Development Program from 2014 to 2019. A logic model shows how activities are expected to produce outputs and in turn how these outputs are expected to ultimately lead to the ultimate outcomes.

Agency inputs include Operation and maintenance; wages; salaries and benefits; grant and contribution funding. There are three activities, each of which align with the output components:

  • The first activity, Program development, planning and management, leads to the output of Guidelines, resources, reports.
  • The second activity, Non-financial support to southern Ontario stakeholders, leads to two outputs: Outreach, information sessions, advice and Networks and collaborations.
  • The third activity, Contribution funding, leads to the output Approved projects/contribution agreements.

Recipients’ inputs include, Grant and contribution funding. There are four activities:

  1. Product development & commercialization (IBI, IBGP, ICP and AMF)
  2. Support to business / entrepreneurs (all initiatives)
  3. Productivity improvement / process innovation (AMF, ICP and IBGP)
  4. Third-party delivery (IBI, IBGP, ICP, IRD and EODP)

Activity boxes lead to four outputs:

  1. Partnerships/collaborations (all initiatives)
  2. Training/mentorship for entrepreneurs (IBI)
  3. $ leveraged against FedDev Ontario contribution (all initiatives)
  4. Businesses/organizations supported (all initiatives)

Output boxes lead to four immediate outcomes:

  1. Increased investment in research and development (AMF, ICP, IBGP and IBI)
  2. Increased investment in community economic capacity (IRD and EODP)
  3. Increased access to capital (IBI, IBGP, EODP and AMF)
  4. Increased investment in productivity improvements (AMF, IBGP and ICP)

Outcome Increased investment in R&D, leads to two intermediate outcomes:

  1. Increased commercialization of research (IBI, IBGP, ICP and AMF)
  2. Increased employment opportunities (all initiatives)

Outcome Increased investment in community economic capacity, leads to intermediate outcome Increased employment opportunities (all initiatives).

Outcome Increased access to capital, leads to all five intermediate outcomes:

  1. Increased commercialization of research (IBI, IBGP, ICP and AMF)
  2. Increased employment opportunities (all initiatives)
  3. Increased value and diversity of markets (IBGP, IBI and EODP)
  4. Enhanced business productivity (AMF, IBGP and ICP)
  5. Improved survival rate of new businesses (IBI)

Outcome Increased access to capital, leads to intermediate outcome enhanced business productivity.

The intermediate outcomes lead to two ultimate outcomes:

  1. Improved economic status of southern Ontario communities (all initiatives)
  2. More competitive businesses (all initiatives)

The ultimate outcomes lead to the FedDev Ontario Strategic Outcome: A Competitive Southern Ontario Economy.

Activities

The logic model makes the distinction between the inputs and activities of FedDev Ontario and those of the funding recipients. With inputs such as operating and maintenance (O&M) expenditures, wages, salaries and benefits, and grant and contribution (G&C) funding, the Agency undertakes the following activities:

With G&C funding inputs, the recipients undertake the following activities:

Outputs

The Agency outputs include:

Recipient outputs are as follows:

Immediate Outcomes

The outputs are expected to result in a number of immediate outcomes (expected to be manifested in the first 1 to 2 years of project activities), including:

Intermediate Outcomes

The immediate outcomes are expected to lead to the following intermediate outcomes within two to five years of support to projects:

Ultimate Outcomes

Ultimate outcomes are generally associated with changes in societal conditions, are often subject to influences beyond the initiative itself and, as a result, take a longer time to be realized. The above intermediate outcomes are expected to result in the following ultimate outcomes in the longer term:

Date modified: