Evaluation of the Southern Ontario Development Program (SODP)

Final Report

Prepared by
Evaluation Directorate
Federal Economic Development Agency for Southern Ontario

Table of Contents

List of Acronyms

AIME Achieving Innovation and Manufacturing Excellence
ARC Applied Research and Commercialization Initiative
BDC Business Development Bank of Canada
BERD business enterprise R&D
BICD Business, Innovation and Community Development
CA contribution agreement
CAF Community Adjustment Fund
CFDC Community Futures Development Corporation
CME Canadian Manufacturers & Exporters
CSBP Centre for Special Business Projects (Statistics Canada)
EODP Eastern Ontario Development Program
FBI Food & Beverage Initiative (pre-SOA)
FTE full-time equivalent
G&Cs grants and contributions
GDP gross domestic product
GEI Graduate Enterprise Internship
GI general intake (pre-SOA)
IBI Investing in Business Innovation
ICT information and communications technology
IRAP NRC Industrial Research Assistance Program
KI key informant
MARC McMaster Automotive Resource Centre
MEDEC Medical Devices Canada
MNE multinational enterprise
NFP not-for-profit
NPV net present value
NRC National Research Council
OBI Ontario Brain Institute
OCC Ontario Chamber of Commerce
OECD Organisation for Economic Co-operation and Development
PAA Program Alignment Architecture
PI Prosperity Initiative
PI–PE Prosperity Initiative – Productivity Enhancement
PI–RD Prosperity Initiative – Regional Diversification
PI–BCA Prosperity Initiative – Building a Competitive Advantage
PMS Performance Measurement Strategy
PSI postsecondary institution
R&D research and development
RDA regional development agency
SEB Scientists and Engineers in Business
SME small and medium-sized enterprises
SOA Southern Ontario Advantage
SODP Southern Ontario Development Program
SOPI Southern Ontario Prosperity Initiatives
SOSCIP Southern Ontario Smart Computing Innovation Platform
SR&ED Scientific Research and Experimental Development
STEM science, technology, engineering and mathematics
TBS Treasury Board Secretariat
TDP Technology Development Program
Y-STEM Youth STEM
YLF Yves Landry Foundation

top of page

Executive Summary

An evaluation of the Southern Ontario Development Program (SODP) was undertaken by the Federal Economic Development Agency for Southern Ontario (FedDev Ontario) to meet Treasury Board requirements, as well as to provide program management with feedback on the design, implementation and success of the program. The objective of this evaluation was to determine the extent to which SODP continues to be relevant, is on track to achieve its expected program outcomes, and has demonstrated efficiency and economy.

Profile of SODP

FedDev Ontario developed and established SODP in 2009–10 as the core program for its initial five-year mandate (2009–10 to 2013–14). The main objective of this program was to support long-term economic growth in southern Ontario with contributions focused on competitiveness and productivity, innovation and commercialization, and community development and economic diversification.

SODP was initially designed to introduce new funding into southern Ontario during a time of need. FedDev Ontario, with limited capacity at that time and the need to act quickly, developed two initiatives and formed partnerships with third parties to help deliver the program.

Once the priority of quickly delivering SODP stimulus funding was addressed, FedDev Ontario in 2010–11 researched, designed and launched a more focused program aligned with its mandate to promote the development of a strong and diversified southern Ontario economy. This new program was the Southern Ontario Advantage (SOA), made up of seven initiatives.

While the scope of this evaluation was SODP and its initiatives, funding transfers were made from SODP to the following other programs: the National Research Council – Industrial Research Assistance Program (NRC–IRAP), the Business Development Bank of Canada (BDC), the Community Adjustment Fund (CAF), and the Eastern Ontario Development Program (EODP). These transfers were not included in this evaluation, as they have been or are being evaluated separately.

SODP expenditures covered in this evaluation were $597.1 million over the 2009–10 to 2013–14 period. They consisted of $177.4 million for the original SODP (pre-SOA) initiatives and $419.7 million for the SOA initiatives. In addition, total transfers to the other programs were $130.8 million.

Evaluation Methodology

The evaluation of SODP was based on multiple lines of evidence, including primary and secondary data from qualitative and quantitative sources:

In parallel with this evaluation, a Statistics Canada comparison study of SMEs assisted by SODP and similar unassisted SMEs was undertaken to provide information on the achievement of ultimate outcomes. The results of that study have been incorporated into this evaluation report.

Overall, the evaluation methodology provided the necessary evidence for reaching conclusions for all issues and questions. However, the evaluation methods had limitations. These limitations, described in Section 3.4, were taken into account during the analyses and were recognized in the interpretation of the findings.

Key Findings

Key evaluation findings for relevance, achievement of program outcomes, and efficiency and economy are provided below.

Program Relevance

There is a continuing need for a program like SODP to foster innovation, enhance productivity and competitiveness, and encourage the commercialization of research in southern Ontario. SODP was found to complement other federal and provincial government funding programs where they were available to recipients.

The requirement for funding in southern Ontario is substantial, but the existence of other programs was not a concern. The evaluation determined that other programs had a different emphasis and that the demand for funding far exceeded the limited funding available from SODP.

SODP was consistent and fully aligned with FedDev Ontario's Program Alignment Architecture (PAA) and strategic outcome and with federal government priorities and strategies, such as the Speech from the Throne (2013), the Budgets for the 2009 to 2014 period, and Canada's Science and Technology Strategy (2007). Further, SODP was found to be fully aligned with federal roles and responsibilities.

Performance: Achievement of Program Outcomes

The evaluation revealed that the SODP achieved the expected outcomes by the end of the program on , with the initiatives meeting or exceeding almost all their performance targets. It is expected that projects, many of which were completed on , will take varying lengths of time after project completion to achieve their full impact (e.g., sales and employment).

FedDev Ontario successfully delivered the SODP initiatives to support individuals and organizations, from start-ups to multinationals, by addressing needs across the business continuum within its mandate. The SODP delivery models were efficient and economical in producing outputs and progressing towards expected outcomes. No significant challenges were experienced in delivery, although some adjustments were made along the way.

The SOA initiatives leveraged $2.65 of client funding for each dollar of program funding expended, resulting in total SOA project expenditures of over $1.5 billion based on FedDev Ontario expenditures of $419.7 million. The initiatives were successful in supporting the development and commercialization of innovations and in creating sales, jobs and full-time employment. Of the SMEs in the SME survey, 94 percent rated their project as "good" to "excellent" in meeting its objectives. The three SOA People Advantage initiatives reached over two million children and youth, funded internships leading to employment, and helped entrepreneurs with their start-ups. The pre-SOA initiatives involved expenditures of $177.4 million and were found to be "very successful" or "successful" in achieving impacts.

Although a high percentage of SMEs met their project objectives, almost half the firms that responded in the SME survey stated that they faced barriers to fully exploiting their project achievements. Barriers included the unavailability of internal or external funding, followed by a lack of key staff, changes in market, establishing a distribution network, and needing to complete related components.

Further, cluster projects take a long time after their completion to reach their full potential, and many will need additional funding assistance if they are to achieve their full potential.Footnote 1 These projects, which are large and complex, met their objectives by the end of SODP in . The projects primarily focused on establishing the clusters, including putting in place the necessary infrastructure. At this point, many clusters have just entered the operating phase and, while committed to sustaining the cluster, are still in the process of ramping up and establishing partnerships with industry to use their facilities and expertise. Many clusters require additional funding to fully exploit their infrastructure, including recapitalization of equipment in future years. Some clusters reported that SMEs, including start-ups, would like to use the facilities but need assistance to pay.

In terms of achieving ultimate outcomes, the Statistics Canada study indicated that the SODP-assisted SMEs were generally more successful in post-funding employment and survivability than firms that were not assisted. However, as most of the projects were only completed on , the study will need to be repeated in a few years to reassess how project outcomes have contributed to SMEs' long-term performance. Given the total estimated investment of $284 million in the SMEs included in the survey, it is estimated that the minimum ratio of return on SODP contributions to sales generated in the first four years after completion is 1:5Footnote 2. In other words, for every dollar FedDev Ontario contributed to these SMEs, it is estimated it will generate $5.00 in sales.

Performance: Efficiency and Economy

Based on the evidence, the evaluation determined that the SODP delivery model, which involved delivery mechanisms tailored to each initiative, was efficient and economical in producing outputs and progressing towards expected outcomes. No major changes were identified; however, some improvements were suggested. Partnerships and leverage, including third-party delivery, helped expand FedDev Ontario's impact and increase the reach of FedDev Ontario's programs. SODP was able to leverage its impact as a champion or convener by using its contribution funding as an incentive. The balance between in-house and third-party delivery was instrumental in achieving efficiency and effectiveness. Because of the spending lifecycle of projects, some flexibility in authorities is required if the funding profile, which was flat-lined in SODP, is to match project funding needs.Footnote 3

As indicated by the number of applications, the demand for SODP initiative funding was much higher than available funding, so program staff had a range of qualified applicants from which to select recipients. SODP used an appropriate combination of repayable and non-repayable contributions in program delivery, providing flexibility for targeting different funding audiences.

Since its inception in 2009, FedDev Ontario has matured in building capacity and tools, although there still remains a need to refine some processes and integrate additional tools. Program management was supported by a performance measurement system in delivering the SOA initiatives. In a high-level comparison, FedDev Ontario delivery costs for its grants and contributions (G&C) programs were lower than those of four other regional development agencies (RDAs) delivering similar G&C programs. However, this comparison was undertaken at the RDA level and not at the program level, and programming and contextual factors (e.g., scale and type of programs, geography, and use of delivery partners) may explain some of the cost differences.

Funding recipients expressed a generally high level of satisfaction with FedDev Ontario and its delivery partners. Some concerns were the length of time taken for the application, assessment and approval processes; the lack of clarity in the reporting requirements; and the amount of administration required. However, there was recognition that program staff were very helpful and that a certain amount of due diligence is required in dealing with public money.

Third-party delivery, which involves trusted partners, was found to be both efficient and effective. Delivery partners expressed concern that the 5 percent of the contribution amount provided for program administration was insufficient. This was also the general view of FedDev Ontario interview respondents, who felt that 10 percent would be more reasonable for future programs, particularly if it resulted in better performance, such as improved reporting; greater promotion to ultimate funding recipients; increased outreach; and targeted outreach in geographic areas that require additional effort. Ultimate recipients of the funding delivered by third parties were often not aware that the source of the funding was FedDev Ontario. Further, a comprehensive database or databases of the recipients that were funded through the delivery partners was not available to program management.

Recommendations

On the basis of the evaluation, the following recommendations are made for consideration by FedDev Ontario management:

  1. Administrative processes
    1. Recipient concerns about the administrative processes, including simplifying and speeding up the application, assessment and approval processes and streamlining the reporting process, should be addressed to the extent possible, while meeting the minimum needs of program administration.
    2. Implementing initiative service standards and information and communications technology (ICT) solutions to improve performance should be considered. Service standards will assist program management in advising applicants of the time required for project approval.
  2. Third-party delivery
    1. Consideration should be given to increasing the 5 percent administration fee provided for third-party delivery, particularly if it results in better performance, such as improved reporting; greater promotion to ultimate recipients; increased outreach; and targeted outreach in geographic areas requiring additional effort. The specific percentage of increase should take into account the parameters of the initiative being delivered and the expectations related to delivery.
    2. FedDev Ontario should request that delivery partners increase the prominence of FedDev Ontario as the source of funding in their program communications with ultimate recipients.
    3. FedDev Ontario should establish a database of third-party funding recipients to facilitate the management of the Agency's third-party delivery projects. The database would identify SMEs receiving contributions for multiple projects and provide contact information for research purposes.
  3. Ongoing support
    1. FedDev Ontario management should undertake research into and analysis of mechanisms that would provide longer-term support to recipients. Mechanisms may be needed to assist clusters to realize their full potential and become self-sustaining beyond FedDev Ontario's current fixed five-year funding cycle. Additional support could also be used to assist potential SME users wishing to use the facilities but lacking the financial resources to do so. Due to the timeframes involved, the five-year lifecycle of FedDev Ontario programs may be an impediment to implementing this recommendation.
    2. To aid in future cluster planning, a review of cluster projects should be undertaken after approximately three years of operation to assess their ongoing sustainability, evaluate their success, identify barriers to achieving anticipated results, and document lessons learned.
    3. Consideration should be given to addressing barriers identified by SMEs in the survey that limit their ability to fully exploit their project achievements. This could involve additional assistance, financial or other, to help them address key barriers. If the barriers are known prior to the initial funding support, they should be identified as part of the funding application and taken into account in the funding approval decision.
  4. Program funding
    1. As part of third mandate renewal, FedDev Ontario management should ensure the five-year funding envelope aligns with project funding needs on an annual basis. This would address the difficulty of implementing projects that collectively have funding requirements matching the flat-lined five-year program funding, particularly in the first two years when projects are being assessed, approved and getting underway.
    2. As part of third mandate renewal, FedDev Ontario management should consider requesting continuous program funding. Continuous funding would better meet stakeholder requirements, improve internal efficiency, and facilitate strategic investment planning.
  5. Ultimate outcomes
    1. The Statistics Canada study, Business Performance Measurement of FedDev Ontario Program Beneficiaries, should be repeated in the future to assess how project outcomes will have contributed to the long-term performance of supported SMEs. The timing of the study would be based on at least three years' experience after project completion and the availability of data for the analysis in Statistics Canada databases. To the extent possible, future research should look at each initiative separately or in a group of similar initiatives in comparing assisted and unassisted firms.
    2. In addition, other methods of assessing ultimate outcomes should be considered, to provide additional lines of evidence to complement the Statistics Canada modelling.

top of page

1.0 Introduction

This report documents the evaluation of the Southern Ontario Development Program (SODP) undertaken by the Federal Economic Development Agency of Southern Ontario (FedDev Ontario). The evaluation was carried out by Goss Gilroy Inc., working in conjunction with FedDev Ontario's Evaluation Directorate, during the period from to . An evaluation advisory committeeFootnote 4 was established to advise the evaluation project team at key points during the evaluation.

FedDev Ontario developed and established SODP in 2009–10 as the core program for its initial five-year mandate in response to the economic challenges experienced at that time in southern Ontario. With SODP ending , FedDev Ontario undertook this evaluation of SODP to meet Treasury Board Secretariat (TBS) requirements, as well as to provide program management with feedback on the design, implementation and success of the program.

1.1 FedDev Ontario

FedDev Ontario, the federal government's regional development agency (RDA) responsible for southern Ontario, is headquartered in Waterloo, with offices in Toronto, Ottawa and Peterborough. Southern Ontario is Canada's most populous region and a key contributor to the Canadian economy.

Southern Ontario has a population of approximately 12.4 million residents living in 288 communities in 37 Statistics Canada census (2006) divisions, as shown in Exhibit 1.1. In 2011, southern Ontario represented 93.5 percent of the total population of Ontario and 35.9 percent of the population of Canada.

Exhibit 1.1: Geographic Area Covered by FedDev Ontario
Exhibit 1.1: Geographic Area Covered by FedDev Ontario (the long description is located below the image)
Description of Exhibit 1.1: Geographic Area Covered by FedDev Ontario

A map of southern Ontario that shows FedDev Ontario's scope of operations, from Cornwall in the east to Owen Sound in the west, and from Pembroke in the north to Windsor in the south. According to Statistics Canada's annual population estimates for 2013, southern Ontario has a population of more than 12 million people, representing 94 percent of Ontario's total population and 36.3 percent of the total population of Canada.

Exhibit 1.1: Geographic Area Covered by FedDev Ontario

FedDev Ontario was launched on , to help address the economic challenges in southern Ontario. It was created through Budget 2009, which provided $1.0 billion over five years for a new southern Ontario development agency. Its mandate was to "support economic and community development, innovation, and economic diversification, with contributions to communities, businesses and non-profit organizations." In addition to developing and establishing SODP in 2009–10 as its core program, FedDev Ontario was given responsibility for the delivery of a number of other programs, including new economic stimulus and infrastructure programs, as well as pre-existing economic development programs.

In Budget 2013, FedDev Ontario's mandate was renewed for an additional five years, from , to , with a budget of $920 million.

1.2 Objectives and Scope of the SODP Evaluation

The objective of the SODP evaluation was to determine the extent to which SODP continues to be relevant, is on track to achieve its expected program outcomes, and has demonstrated efficiency and economy. The evaluation was based on the TBS Directive on the Evaluation Function,Footnote 5 with a focus on the five core evaluation issues relating to Relevance and Performance (effectiveness, efficiency and economy). For each of the five core evaluation issues, key evaluation questions were developed. The evaluation methodology is described in Section 3.0.

The scope of the evaluation was SODP and its initiatives. In addition, funding transfers were made from SODP to the following programs (see Section 2.1.3), which have not been included in the evaluation because they have been or are being evaluated separately:

1.3 Outline of This Report

This report contains a profile of SODP, including its initiatives, the logic model and performance measurement strategy (PMS) and financial summary, in Section 2.0; a description of the evaluation methodology in Section 3.0; findings on relevance in Section 4.0; findings on performance in Section 5.0; findings on efficiency and economy in Section 6.0; and conclusions and recommendations in Section 7.0.

top of page

2.0 SODP Profile

FedDev Ontario developed and established SODP as its core program in 2009–10 in response to the economic challenges being experienced in southern Ontario. The objective of SODP was to support economic growth over the long term with contributions focused on competitiveness and productivity, innovation and commercialization, and community development and economic diversification. Six areas of activity were identified in the SODP Terms and Conditions:Footnote 6 community economic development; innovation; information and communications technology (ICT); trade and tourism; human capital; and business financing.

2.1 SODP Description

SODP consisted of the initial SODP initiatives, referred to here as the pre–Southern Ontario Advantage (pre-SOA) initiatives and the subsequent SOA initiatives. The pre-SOA and SOA initiatives are described below.

2.1.1 Initial SODP (Pre-SOA Initiatives)

SODP was initially designed to introduce new funding into southern Ontario during a time of need. FedDev Ontario, with limited capacity at that time and the need to act quickly, developed two initiatives and formed partnerships with third parties to help deliver the program.

The first SODP initiative was the general intake (GI), with $130.1 million in funding,Footnote 7 launched in and closing in late . The GI had broad criteria, and its projects focused on stimulating local economies and enhancing the growth and competitiveness of local businesses and communities. Although GI met the goal of supporting projects quickly, according to the evidence, its effectiveness was affected by the following:

However, GI did identify the need for more strategic and longer term support focused on productivity and competitiveness, in order to have a greater impact on the large and dynamic southern Ontario economy. This point was reinforced by the successful results of the second intake, which focused on the food and beverage industry. With $18.0 million in funding, the Food & Beverage Initiative (FBI) intake was smaller than the GI. The FBI was launched in and also had a closing date in late . FBI projects focused on small and medium-sized enterprises (SMEs) in the food and beverage industry, in recognition of that industry's key role in the southern Ontario economy. FBI was designed to help firms expand, modernize, innovate, or improve their competitiveness.

In addition to the two intakes, partnerships were established with three not-for-profit (NFP) organizations, which helped deliver the program by undertaking contribution programs on FedDev Ontario's behalf. These three NFP delivery partners and their contribution programs were the following:

In the initial SODP (pre-SOA) program, FedDev Ontario received 2007 applications, of which 90 were approved. Three of the approved applications involved the NFP delivery partnerships noted above. In addition, organizations that applied to the NFP delivery-partner funding programs resulted in a further 810 approved projects. In total, 897 projects involving $177.4 million in contributions were undertaken, as shown in Exhibit 2.1.

Exhibit 2.1: Overview of the Initial SODP (Pre-SOA)
Initial SODP (pre-SOA) Projects Expenditures ($ millions)
General intake (GI) 50 130.1*
Food & Beverage Initiative (FBI) 37 18.0
Delivery partnerships 810 (3)** 29.3
• Canadian Manufacturers & Exporters (CME) 349 15.7
• Ontario Chamber of Commerce (OCC) 186 1.6
• Yves Landry Foundation (YLF) 275 12.0
Total 897*** 177.4*

Source: FedDev Ontario Business, Innovation and Community Development (BICD), October–November 2013.

* Includes $26.6 million for two continuing pre-SOA projects expensed during the SOA period.
** 810 projects were delivered through three delivery-partnership projects.
*** Total for projects does not include the three delivery-partnership projects.

2.1.2 Southern Ontario Advantage (SOA) Initiatives

Once the priority of quickly delivering SODP stimulus funding was addressed, FedDev Ontario in 2010–11 researched, designed and launched a more focused program aligned with its mandate to promote the development of a strong and diversified southern Ontario economy. This new program was the Southern Ontario Advantage (SOA), which had seven initiatives.

In designing the seven initiatives, FedDev Ontario officials researched key areas of need within the region's economy by meeting with stakeholders and industry leaders to identify their challenges. This work was complemented by roundtable discussions across the region, which identified a number of regional economic challenges:

With these challenges identified, initiatives designed to foster innovation, enhance productivity and competitiveness, and encourage the commercialization of research were developed and subsequently announced on . The SOA vision for southern Ontario was to capitalize on new opportunities for innovation, build strategic advantages to position the region as a global leader, achieve greater global success, and once again be a driving force in the Canadian economy. The SOA initiatives were designed to address needs across the business continuum from the start-up phase, through the traction and growth phases, to the mature phase.

The seven SOA initiatives and their objectives and launch dates are given in Exhibit 2.2, grouped by the four Advantage categories.

Exhibit 2.2: SOA Initiatives
SOA initiative Objective Launch date
People Advantage
Y-STEM Youth STEM (Science, Technology, Engineering and Mathematics)
  • Encourage youth to pursue education and careers in STEM
  • Improve youth's understanding of the business of science
GEI Graduate Enterprise Internship
  • Develop business and management skills in graduate students and recent graduates of STEM programs to complement their technical skills
  • Provide career networking opportunities
  • Build the next generation of potential managers
  • Enable small and medium-sized enterprises (SMEs) to benefit from the technical knowledge of STEM graduate students and recent graduates
SEB Scientists and Engineers in Business
  • Develop improved business and management skills of STEM entrepreneurs
  • Improve access to financing and/or business support services needed to successfully launch and manage start-up SMEs in southern Ontario
Knowledge Advantage
ARC Applied Research and Commercialization
  • Support innovation in SMEs by encouraging greater collaboration and partnerships with postsecondary institutions
  • Accelerate innovation and improve productivity and competitiveness of SMEs
Pilot:
Extension:
TDP Technology Development Program
  • Bridge the gap between R&D and commercialization of market-driven, "game-changing" technologies
  • Increase collaborations involving private sector and academic and innovation organizations
  • Leverage private sector investment in game-changing technologies
Entrepreneurial Advantage
IBI Investing in Business Innovation
  • Accelerate the commercialization of new products, processes and practices
  • Increase, stimulate and leverage private sector investment
  • Encourage growth of angel investment funds
Prosperity Advantage
PI Prosperity Initiative
  • Enhance productivity, diversify the regional economy, and build competitiveness in southern Ontario; consists of three distinct components:
    • PE–Productivity Enhancement—addressed underinvestment in machinery, equipment and software in Ontario relative to such investments by their counterparts in the United States
    • RD–Regional Diversification—focused on regions where there are high concentrations of manufacturing industries in structural decline and low concentrations of high-growth industries, which could expose local economies to significant job losses if some plants were to close
    • BCA–Building a Competitive Advantage—emphasized the development or expansion of geographic concentrations (clusters) of interconnected companies and institutions in a particular field, which can provide a competitive advantage

Under FedDev Ontario's Program Alignment Architecture (PAA), SODP was designed to support two of the three program activities, which in turn supported its strategic outcome: a competitive southern Ontario economy.Footnote 8 The two program activities, Technological Innovation and Business Development, are described below:

A more detailed description of these seven SOA initiatives, including eligible applicants, beneficiaries, funding type, and funding available to an applicant, is provided in Appendix A. Three of the initiatives had multiple funding streams: SEB had two, IBI had three, and PI had three. In total, there were 12 distinct initiatives or streams in the SOA program.

The GEI, SEB and ARC initiatives were all delivered by FedDev Ontario's NFP and PSI delivery partners to assist third-party organizations. In addition, part of the Prosperity Initiative, Productivity Enhancement (PI–PE), was delivered by four delivery partners: three that delivered pre-SOA initiatives (CME, OCC and YLF) and Medical Devices of Canada (MEDEC), through its New Horizons for MedTech program. Third-party organizations that wished to obtain contributions via these delivery partners went through a second application process.

Information on the number of applications, approved agreements, and expenditures for each of the seven SOA initiatives is given in Exhibit 2.3. PI accounted for $234.4 million (or almost 56 percent of the total SOA expenditures of $419.7 million). The next largest initiatives, in dollar terms, were TDP and IBI, accounting respectively for $56.9 million (13.6 percent) and $56.4 million (13.4 percent). The smallest were the People Advantage initiatives (Y-STEM, GEI and SEB), which together accounted for $44.4 million (10.6 percent).

2.1.3 Other Programs Funded by SODP

In addition to the pre-SOA and SOA projects supported by SODP, funds were used as follows:

The above SODP funding transferred to these programs is not included as part of this evaluation, as each of these programs is being separately evaluated. Funding transferred to NRC–IRAP and BDC form part of their programs, for which they have an evaluation responsibility. In addition, the CAF and EODP are being separately evaluated within FedDev Ontario; therefore, those transfers have been excluded in this evaluation.

Exhibit 2.3: Key Statistics for SOA Initiative Implementation
(as of )
SOA initiative Total number of applications received Approved agreements Expenditures*
($ millions) (%)
People Advantage
Y-STEM Youth STEM (Science, Technology, Engineering and Mathematics) 44 15 13.3 3.2
GEI Graduate Enterprise Internship 19 10 15.9 3.8
SEB Scientists and Engineers in Business 14 12 15.2 3.6
Subtotal People Advantage 77 37 44.4 10.6
Knowledge Advantage
ARC Applied Research and Commercialization Initiative 41 47** 27.6 6.6
TDP Technology Development Program 19 6 56.9 13.6
Subtotal Knowledge Advantage 60 30 84.5 20.1
Entrepreneurial Advantage
IBI Investing in Business Innovation 203 106 56.4 13.4
Prosperity Advantage
PI–PE Prosperity Initiative – Productivity Enhancement 80 6 32.4 7.7
PI–RD Prosperity Initiative – Regional Diversification 181 19 74.7 17.8
PI–BCA Prosperity Initiative – Building a Competitive Advantage 257 32 127.3 30.3
Subtotal Prosperity Advantage 418 57 234.4 55.8
Total all SOA initiatives 863 230 419.7 100.0

Source: FedDev Ontario: SODP Performance Results (as of ), except the number of applicants is from FedDev Ontario: Key Results from Agency Launch to .
* Expenditures include payables at year end (PAYE) against 2012–13 and 2013–14.
** Includes 24 agreements in the ARC Pilot and 23 agreements in the ARC Extension, involving 24 different recipients.

2.1.4 Total SODP Funding

The total amount of SODP funding, including the pre-SOA and SOA projects and the transfers to other programs, is $727.9 million, as shown in Exhibit 2.4.

Exhibit 2.4: Total Approved SODP Contributions and Transfers (as of )
SODP funding components SODP expenditures
($ millions)
Pre-SOA and SOA 597.1
• Pre-SOA 177.4
• SOA 419.7
Transfers to other programs* 130.8
Total SODP 727.9

* Details provided in Section 2.1.3.

2.2 Logic Model and Performance Measurement Strategy

A performance measurement strategy (PMS) was prepared for SODP in . The PMS included a logic model and a performance measurement matrix containing a set of performance indicators for SODP overall. SOA initiatives were developed and new logic models and performance indicators were prepared for SODP overall and separately for each of the seven SOA initiatives. The revised logic models and performance indicators were approved by FedDev Ontario’s Executive Committee in , together with an updated version of the SODP PMS.Footnote 9

2.2.1 Logic Model

The approved SODP logic model (2012) is provided in Exhibit B.1 in Appendix B. It outlines the key activities being funded by the program; the outputs that will result from those activities; and the immediate, intermediate and ultimate outcomes the program is intended to achieve. The SODP logic model also identifies the key activities, outputs and outcomes for each of the initiatives under SOA and pre-SOA (summarized in Exhibit 2.5). Individual logic models for each of the SOA initiatives were also prepared to show how each initiative fits within the larger program.

2.2.2 Performance Measurement Strategy

In total, there were 20 outputs and outcomes and 38 performance indicators for SODP. The number of performance indicators applicable to each SOA initiative or stream depended on its objectives. The performance indicators for outputs and outcomes for each of the SOA initiatives or streams are provided in Exhibit B.2 in Appendix B.

Program management implemented an in-depth data collection and progress reporting system for each of the SOA initiatives. This system was based on the SODP PMS (2012), which provided performance information derived from project progress reporting from funding recipients. Where the program was delivered through delivery partners (NFPs and PSIs), the partners collected this information from the funding recipients and provided it to FedDev Ontario. This performance information was consolidated by FedDev Ontario Program Services in a summary report and a series of spreadsheets, one set for each initiative or stream.

In addition to individual project targets identified for recipients in the contribution agreements (CAs), targets were established for each SOA initiative. However, the SOA targets were not set until 2012, after the new PMS was approved. At this point, SOA projects had been launched and were well underway. As a new organization, FedDev Ontario did not have experience with similar programs to help establish program targets, so internal expertise and staff experience with other RDA programs were used to create the targets.

Funding recipients were also required to submit a final report within three months of project completion. These reports were to be prepared and submitted by organizations receiving funding directly from FedDev Ontario or through delivery partners. The initial final reports followed an outline in the CAs and asked for summary and high-level performance information pertinent to the project or initiative. However, these initial final report outlines were prepared prior to the finalization of the SODP logic models and performance indicators in 2012. Consequently, the information provided by the initial final reports is limited in comparison to the information included in subsequent final reports collected by the performance reporting system as of .

Exhibit 2.5: Relevancy of SODP SOA and Pre-SOA Performance Indicators for Each Initiative and Stream
Program outputs and outcomes SOA initiative Pre-SOA
Y–STEM GEI SEB–NFP SEB–PSI ARC TDP IBI–Angel IBI–SME PI–PE PI–RD PI–BCA
Outputs
1. Education sessions X X X
2. Internship programs X
3. Partnerships and collaborations X X* X* X X X** X** X** X** X
4. Investments in cash and in kind leveraged against FedDev Ontario contribution X X*** X*** X X X X X X X X
5. Businesses supported X X X X X X X X
Immediate outcomes
6. Enhanced STEM learning opportunities for children and youth X
7. Increased support to commercialize innovations X X X X X X X X
8. Increased exposure of STEM graduates to business X X X
9. Increased access to capital X X X X X X X X
10. Strengthened businesses and clusters X X X
11. Increased adoption or adaptation of new technologies X X X
Intermediate outcomes
12. Increased participation of children and youth in STEM outreach programs X
13. Improved employment of STEM graduates in the private sector X X X
14. Increased commercialization of research X X X X X X X X
15. Improved survival rate of businesses and start-ups X X X X
16. Increased employment opportunities in southern Ontario communities X X X X
17. Enhanced business productivity X X X
Ultimate outcomes
18. Increased innovation capacity in southern Ontario X X X X X X X X X X X X
19. Stronger southern Ontario communities X X X X X X X X X X X X
20. More competitive southern Ontario businesses X X X X X X X X X X X X

* Partnerships or collaborations formed with start-up businesses (NFP stream); partnerships or collaborations formed with donors (PSI stream); fellows attracted to PSI (PSI stream).
** "New" partnerships and collaborations.
*** Total value of investments in start-up businesses leveraged by source (NFP stream); total value of endowments for commercialization fellowships (PSI stream).

2.3 Stakeholders

FedDev Ontario worked with a variety of stakeholders to deliver SODP. Key stakeholders included SMEs and start-up enterprises, Not-For-Profits (NFPs) such as industry and sector associations and Community Futures Development Corporations (CFDCs), Post-Secondary Institutions (PSIs) and angel investment networks. Key stakeholder groups involved in SODP initiatives are identified in Exhibit A.1 in Appendix A.

In addition to potential recipients of contributions through the SODP, FedDev Ontario worked with interested parties such as municipal organizations; departments and agencies of the Government of Ontario; and other federal departments and agencies including Industry Canada, Regional Development Agencies (RDAs), NRC and the Natural Sciences and Engineering Research Council.

2.4 Governance

SODP funding contributions were governed by CAs between FedDev Ontario and funding recipients. FedDev Ontario outlined four types of relationships governed by SODP CAs:

  1. Direct Agency–recipient relationships where FedDev Ontario provided funding to a recipient for a specific project (e.g., through IBI, where FedDev Ontario supports a start-up enterprise to undertake a specific project);
  2. Direct Agency–recipient relationships where FedDev Ontario provided funding to a recipient for a specific project(s) with one or more collaborators (e.g., through the ARC initiative, where FedDev Ontario provided funding to a PSI that in turn worked with SMEs to undertake pre-commercialization projects);
  3. Indirect relationships where FedDev Ontario provided funding to an initial recipient that in turn distributed payments to ultimate recipients (e.g., through the PI–PE stream, where FedDev Ontario provides funding to an NFP, industry or sector association that in turn distributed funding to SMEs to support productivity improvements); and
  4. Intergovernmental third-party agreements where FedDev Ontario requested another government department to provide services on its behalf (e.g., FedDev Ontario requested BDC to deliver investments to early-stage firms in southern Ontario and Ontario-based venture capital funds focused on Ontario-based opportunities).

In all these models, the CA outlines the recipient's contractual obligation to provide information required for performance measurement and evaluation. In cases where the recipient delivers the program on behalf of FedDev Ontario, the recipient must obtain information from beneficiaries or ultimate recipients, in order to meet this requirement. The fourth type of relationship above, which was used in the pre-SOA program with NRC–IRAP and BDC, is not part of this evaluation, as these contributions form part of program evaluations within the recipient organizations.

2.5 Expenditures

Exhibit 2.6 shows SODP expenditures over the five-year period from 2009–10 to 2013–14.

Exhibit 2.6: SODP Expenditures, 2009–10 to 2013–14
Expenditures ($)
2009–10 2010–11 2011–12 2012–13 2013–14 Total 5 years
Salaries 544,448 2,189,168 3,220,865 3,269,129 3,429,125 12,652,735
Operations and maintenance 1,163,116 1,927,255 914,476 441,442 165,226 4,611,515
Grants and contributions 43,525,352 72,405,516 154,267,058 175,542,810 137,296,695 583,037,431
Transfer to the National Research Council* 27,500,000 17,000,000 44,500,000
Transfer to the Business Development Bank of Canada* 50,000,000 50,000,000
Program evaluation 150,000 150,000
Total 122,732,916 93,521,939 158,402,399 179,253,381 140,891,046** 694,801,681**

* These transfers are not included in this SODP evaluation (see Section 2.1.3).
** Excludes the costs of the program evaluation, as they will be incurred by the Evaluation Directorate.

top of page

3.0 Evaluation Methodology

This section describes the methods used in the evaluation of SODP and includes the evaluation issues covered, the approach taken, the design and application of the data collection methods, and the evaluation challenges and limitations.

3.1 Evaluation Issues

The SODP evaluation was based on the TBS Directive on the Evaluation Function,Footnote 10 which requires assessments of all five of its core issues relating to Relevance and Performance (effectiveness, efficiency and economy) in order to address value for money. However, departments and agencies have the flexibility to determine the evaluation approach and level of evaluation effort in accordance with the program's risks and characteristics and with the quality of performance information available for each program.

The evaluation was designed to address the five issues in the TBS Directive, as well as provide program management with feedback on the design, implementation and success of the program. The five core issues, as well as the key questions addressed in the evaluation, are presented in Exhibit 3.1.

Exhibit 3.1: Key Questions for the SODP Evaluation, by Core Issue
Issue Evaluation Question
Relevance
Issue #1: Continued need for program 1.1  Is there a continued need for a program similar to SODP?
1.2 Did the SODP complement, duplicate or overlap other government programs? Other private sector services?
Issue #2:priorities Alignment with government 2.1 To what extent is the SODP consistent with government priorities:
  1. FedDev Ontario's PAA and strategic outcome?
  2. Federal priorities and strategies?
Issue #3: Alignment with federal roles and responsibilities 3. To what extent is the SODP aligned with the federal government's activities, roles and responsibilities?
Performance (effectiveness, efficiency and economy)
Issue #4: Achievement of expected outcomes 4.1 To what extent have the expected immediate, intermediate and ultimate outcomes been achieved as a result of the SODP?
4.2 Did the SODP produce unintended positive and/or negative outcomes?
Issue #5: Demonstration of efficiency and economy 5.1 To what extent was the SODP delivery model efficient in producing outputs and progressing towards expected outcomes?
5.2 Is there a more cost-effective way of achieving the expected results, taking into consideration alternative delivery mechanisms, best practices and lesson learned?

3.2 Evaluation Approach

The evaluation design that addressed the issues and questions specified above recognized and took into account the following:

In parallel with the evaluation, FedDev Ontario contracted the Centre for Special Business Projects (CSBP) at Statistics Canada to compare the performance of SMEs that received contributions from SODP (both SOA and pre-SOA) with similar SMEs that did not receive contributions. CSBP documented its methods and findings in a report, Business Performance Measurement of FedDev Ontario Program Beneficiaries (). The study matched assisted SMEs with unassisted SMEs over the period 2009–2013. The findings of that study have been included in this evaluation.

The selection of data collection methods took the above research into account and was based on the most efficient means of rigorously addressing the evaluation issues and questions within the timeframe for the evaluation, while minimizing the response burden on funding recipients as much as possible.

Exhibit 3.2: Previous Relevant Research
SODP Data sources Description Year
Overall SODP Interim evaluation of FedDev Ontario's programs Determined to what extent FedDev Ontario's programs, including SODP, continue to be relevant, are on track to achieve their expected program outcomes, and demonstrate efficiency and economy 2012
Implementation review of the SODP PMS Assessed the quality and adequacy of ongoing performance measurement data collection, as well as examining the results achieved to date 2012
Preliminary internal delivery cost comparison Compared the direct delivery cost of People Advantage, IBI and PI, and pre-SOA
Compared the cost of repayable and non-repayable contributions
2012
Preliminary external delivery cost comparison Benchmarked FedDev Ontario's costs with the costs of other RDAs and NRC–IRAP 2012
Activity-based costing model for FedDev Ontario Forecast costing information at an activity level, by initiative, for the SODP replacement program 2013
Business performance measurement of FedDev Ontario program beneficiaries Compared the performance of SMEs that received contributions from SODP (both SOA and pre-SOA) with that of similar SMEs that did not receive contributions 2014
SODP methodology report Described in detail the methodology used for this evaluation 2014
SOA Inventory and forecast of performance indicators for the seven SOA initiatives Addressed the following question: When will results be available for the SOA initiatives? Included an inventory and forecast of SOA performance indicators 2012
Survey of ARC Pilot recipients Involved face-to-face interviews with all 24 PSIs and with 62 SMEs (20 percent statistical sample of the 309 SMEs supported by the initiative) 2012
Performance indicator data Provided (by program staff) on some performance indicators at the time of the interim evaluation 2012
Case studies Completed four case studies: one for Y-STEM, one for IBI and two for PI 2012
Economic analysis Assessed economic impacts of project spending for FedDev Ontario's contributions for ARC, IBI and PI 2013
Pre-SOA Survey of recipients Involved 33 interviews with direct recipients and 12 interviews and 51 file reviews with indirect recipients (CME, YLF and OCC) 2012
Case studies Completed two case studies, one of which included funding under both pre-SOA and SOA 2012
Economic analysis Assessed economic impacts of pre-SOA project spending of FedDev Ontario's contributions 2012

3.3 Data Collection Methods: Design and Application

Given the overall approach described above in Section 3.2, and taking into account previous research, the following section describes the data collection methods used in this evaluation and their application to the evaluation issues and questions.

3.3.1 Application of Data Collection Methods to the Evaluation Issues and Questions

The following data collection methods were used in this evaluation:

The application of the above data collection methods to the evaluation issues and questions is shown in Exhibit 3.3.

Exhibit 3.3: Data Methods for Key Evaluation Issues and Questions
Evaluation issues and questions Document review Review of performance, admin. & financial data Key informant interviews SME survey Mini case studies Data analysis
Relevance
Issue #1: Continued need for program 1.1  Is there a continued need for a program similar to SODP?
1.2  Did the SODP complement, duplicate or overlap other government programs? Other private sector services?
Issue #2: Alignment with government priorities 2.1  To what extent was the SODP consistent with government priorities:
  1. FedDev Ontario's PAA and strategic outcome?
  2. Federal priorities and strategies?
Issue #3: Alignment with federal roles and responsibilities 3. To what extent was the SODP aligned with the federal government's activities, roles and responsibilities?
Performance (effectiveness, efficiency and economy)
Issue #4: Achievement of expected outcomes 4.1  To what extent have the expected immediate, intermediate and ultimate outcomes been achieved as a result of the SODP?
4.2  Did the SODP produce unintended positive or negative outcomes?
Issue #5: Demonstration of efficiency and economy 5.1  To what extent was the SODP delivery model efficient in producing outputs and progressing towards expected outcomes?
5.2  Is there a more cost-effective way of achieving the expected results, taking into consideration alternative delivery mechanisms, best practices and lesson learned?

3.3.2 Description of the Data Collection Methods

The data collection methods used in the evaluation are described below.

Document Review

A review of relevant documents was carried out to obtain a good understanding of SODP and its initiatives, rationale, context and history. Many of these documents had been identified in the preparation of the evaluation methodology report. The review included program descriptions and background information on the SODP initiatives; the SODP performance measurement strategy (PMS); presentations and internal studies; Interim Evaluation of FedDev Programs; previous evaluation related studies and surveys; case studies and analyses; strategic studies related to the need for SODP programming; the Reports on Plans and Priorities; the Departmental Performance Reports; and transition binders and briefing notes.

In addition, the documents included the most recent research undertaken by FedDev Ontario's Strategic Policy Group in relation to the continuing need for an SODP type of program.

Review of Performance, Administrative, and Financial Data

Documents containing data on the SODP initiatives, collected and reported by FedDev Ontario Programs and Finance, provided key information for the evaluation. Performance data in those documents included the management summary and supporting spreadsheets for each SODP initiative as of .Footnote 11 This information was complemented by administrative and financial data on the projects undertaken, including final project reports and other financial summaries.

Key Informant Interviews

Interviews with key informants (KIs)Footnote 12 were a primary information source for the evaluation. Personal interviews, either by telephone or in person, were undertaken using structured interview guides. In total, 44 interviews were held with the following groups:

The individuals interviewed were identified in consultation with FedDev Ontario's Evaluation Directorate. Semi-structured interview guides were developed for the target groups to conduct in-depth interviews to address the key evaluation questions, given above in Exhibit 3.3.

Most of the government interviews were conducted in person at FedDev Ontario offices in Kitchener. The Government of Ontario interviews, several FedDev Ontario interviews, and all the delivery-partner and recipient interviews were by phone. Interviews were conducted in respondents' official language of choice. All interview findings were entered into an electronic database according to the evaluation issue, indicator and respondent type.

On-line and Phone Survey of SME Recipients Firms (SME Survey)

An on-line SME survey with phone follow-ups was undertaken with SME recipients of SODP contributions to obtain information on project results, both actual and expected; economic benefits; the value of SODP support to the SME; and satisfaction and feedback in relation to the delivery of the initiative.

The SME survey was used for the SEB, ARC, IBI, PI and pre-SOA SME projects, with the exception of the PI–BCA cluster projects, as these had a degree of complexity that was best handled by a case study approach. The six TDP projects were also complex, with an average cost of $10.65 million and an average of 15 partners, and were therefore addressed through case studies. The Y-STEM and GEI initiatives were not included, as they focused on people and not specifically on economic benefits. The survey involved SMEs that received SODP support directly from FedDev Ontario, as well as through third parties. The total SODP investment provided to these SMEs was approximately $284 million.

Lists of SMEs for the SOA initiatives were obtained from programs. Not all of the SMEs had email addresses, and considerable effort was made to obtain email addresses for the primary contacts. Good coverage was obtained for all initiatives. The initial number of SME projects available for the survey was 2,703, but that was reduced to 1,992 for the reasons given below:

Number of projects: 2,703
Reductions:
• Multiple projects 374
• No email available 202
• Indirect projects where a direct project was also funded 110
• KI and case study interviews (to reduce burden on respondent) 11
• Request from programs (e.g., bankruptcy) 14 711
Survey contact list: 1,992

Some of the 1,992 contacts had email addresses associated with multiple projects. In those cases, the respondent was asked to select one project for completing the survey.

Note that the reduction for indirect projects where a direct project was also funded was related to situations where a respondent would have received contributions for both indirect and direct projects. This action was taken to ensure that responses would be obtained for a reasonable number of the 163 direct projects in the frame.

The SME survey response rate was 27 percent (539 of the 1,992 contacts). The survey then obtained information on 539 projects (20 percent of the original project total of 2,703).

Mini Case Studies

Mini case studies were undertaken with organizations that benefitted from SODP contributions. The purpose of each case study was to provide detailed information on the project, how SODP contributed to the project's success, and what impacts were achieved. The organizations were selected from a list provided by FedDev Ontario. Mini case studies were used to illustrate the program and were not intended to be applied to other projects.

Ten case studies were undertaken: two for TDP, two for IBI, two for PI–RD and four for PI–BCA. Each case study involved a review of information in the project file, internet searches, and interviews with the recipient organization using a semi-structured interview guide. Each case study was summarized in a three- to six-page report. Seven case studies were completed by the consultants, and three were prepared by FedDev Ontario's Evaluation Directorate.

The evaluation used updated case studies that had been prepared by the Evaluation Directorate for the interim evaluation in 2012 and three success stories prepared by FedDev Ontario to illustrate how SODP contributed to projects in other initiatives and streams. These updated case studies and success stories are identified in this report.

Data Analysis

A number of data and cost analyses were undertaken in support of the evaluation, in particular for issue #5: demonstration of efficiency and economy. Several of these analyses examined the approaches used in delivering the program; these have been provided in Section 6.0.

3.4 Evaluation Challenges and Limitations

In undertaking this evaluation, the following challenges and limitations were identified.

3.4.1 Challenges

Key challenges related to the evaluation were the following:

3.4.2 Limitations

Limitations of the evaluation include the following:

top of page

4.0 Findings on Relevance

This section provides the evaluation findings for Relevance, which are based on the three core relevance issues described in Section 3.1:

4.1 Issue #1: Continued Need for Program

4.1.1 Continued Need for a Program Similar to SODP

Key Finding: There is a continuing need for a program similar to SODP to foster innovation, enhance productivity and competitiveness, and encourage the commercialization of research in southern Ontario.

The findings in this section are based on the KI interviews and on the document review, including research undertaken by FedDev Ontario's Strategic Policy Branch.

Need for SODP

In 2008, Ontario, traditionally a key driver of Canadian prosperity, was hit hard by the global recession, putting pressure on governments to rebuild the province's economic capacity and position the region to compete globally. The recession resulted in plant closures, restructuring and significant job losses, particularly in well-paying manufacturing jobs. The region lost almost 210,000 jobs, more than half of all the jobs lost in Canada, and two of the region's major automobile manufacturers (General Motors and Chrysler) were in crisis in 2009, leading to $13.7 billion in financial assistance from the provincial and federal governments.Footnote 16

FedDev Ontario was created in 2009 to help address the economic challenges that southern Ontario was facing. FedDev Ontario initially designed SODP to introduce new funding into southern Ontario quickly in response to these economic challenges. In 2010–11, following the priority of quickly delivering SODP stimulus funding, FedDev Ontario officials researched key areas of need within the region's economy, meeting with stakeholders and industry leaders to identify their challenges and to learn where FedDev Ontario could make a difference. As a result of this research, FedDev Ontario identified several regional economic challenges that it could get involved in:

The design of the seven SOA initiatives took into account those challenges, as well as the federal and provincial economic development landscape and the federal Science and Technology Strategy (2007). With the objectives of fostering innovation, enhancing productivity and competitiveness, and encouraging the commercialization of research, the SOA initiatives were designed to address needs across the business continuum from the entrepreneurial start-up phase, through the traction and growth phases, to the mature phase.

Continuing Need for a Program Similar to SODP

All interview respondents, both government and funding recipients, agreed that there is an ongoing need for a program similar to SODP. While the situation in southern Ontario is not viewed as being as critical as it was in 2009, the same challenges remain. The region is still struggling, and there continues to be a need to diversify and stimulate the economy, improve productivity and encourage innovation. There is also a need for the federal government to show leadership. While recognizing that FedDev Ontario does not have a lot of funding in relation to the size of the southern Ontario economy, the Agency has been able to demonstrate federal leadership by targeting funding to address specific needs and achieve results, leveraging matching funding, and encouraging collaborations and fostering partnerships.

Through SODP, FedDev Ontario has established itself as a champion for southern Ontario. Program officers are experienced in assessing projects and know the economic environment of the region, so the Agency is able to play an effective role in targeting areas of need. There is also an obligation to continue to solidify collaborations and capitalize on the investments made by SODP. Federal government leadership is particularly important for attracting foreign direct investment and often requires additional incentives to be competitive with jurisdictions outside of Canada. The provincial government respondents, while not as familiar with SODP initiatives as federal government recipients, strongly support the continuation of a program like SODP.

With the SODP expiring , FedDev Ontario undertook research and studies in 2012 and 2013 in support of efforts to renew its mandate for a subsequent five-year period. The following issues were investigated:

FedDev Ontario determined that while economic conditions in the region had improved since the global economic slowdown, longer term structural challenges persist. The 2012 and 2013 studies found that the impacts of the recession were still being felt in southern Ontario; this was corroborated by the interviews mentioned above and other studies. The service sector has been growing, but the manufacturing sector is declining in importance in both absolute and relative terms. Manufacturing was disproportionately affected by the recession, and production and employment in these industries have still not returned to pre-recession levels. A recent report by the Office of the Auditor General indicates that in 2007, approximately 1.5 percent ($21.4 billion) of the Canadian gross domestic product (GDP) was attributable to the auto industry, compared to about 1.1 percent ($19.1 billion) in 2013. In 2007, car manufacturers and parts suppliers employed 152,000 people. In 2013, the sector employed about 117,000 people.Footnote 17 The recent decline in the Canadian dollar relative to the US dollar should benefit southern Ontario manufacturers exporting in the future.

In  , Ontario's unemployment rate was 7.4 percent, which was greater than the national rate of 6.9 percent. Some of the highest unemployment rates in Canadian metropolitan areas were in southern Ontario: Peterborough with 11.6 percent; Windsor, 8.4 percent; Toronto, 7.8 percent; and London, 8.0 percent.Footnote 18 Research showed that Ontario lagged other G7 countries in productivity growth from 1984 to 2011 and lost considerable ground in productivity growth compared to the United States.Footnote 19

In , the Conference Board of Canada reported that Canada is a weak performer in business enterprise R&D (BERD),Footnote 20 which is associated with productivity and GDP growth. Canada ranked 15th of 16 peer nations and has had a poor ranking since the 1980s. Although Canadian businesses projected R&D spending of $15.6 billion in 2011, this is less than that of Canada's international peers when BERD is measured as a percentage of GDP. BERD spending in the United States, for example, is twice as high as in Canada.

Regarding start-ups and high-growth firms, a 2013 study by Deloitte, The Future of Productivity: A Wake-up Call for Canadian Companies, stated that "regardless of size or sector, few Canadian firms are able to sustain high levels of growth. Ironically, Canada has one of the Organisation for Economic Co-operation and Development's (OECD's) highest rates of new business entry and produces a greater proportion of young (five years or less) high-growth firms than the United States, Sweden or even start-up hotbed, Israel. But as our high-growth firms age, their performance slows and they fail to thrive. Canada's entrepreneurs may have mastered the art of creating fast-growing businesses with great potential, but they fall short when it comes to sustaining them."Footnote 21

From its economic studies, consultations, and economic program landscape reviews, FedDev Ontario identified entrepreneurs, productivity and growth, commercialization and communities as continuing priorities in its next program. It was announced in Budget 2013 that the Agency's mandate had been renewed with a new suite of programs and $920 million of funding over five years, starting .

Budget 2013 also included several announcements that affected FedDev Ontario's environment. In particular, $19 million was provided to promote education in high-demand fields, including STEM, at the national level; and $20 million was provided for a national NRC–IRAP voucher for SMEs to work with PSIs on small assignments, similar to the ARC program that had been introduced by FedDev Ontario and had good success in southern Ontario. Further, other organizations were offering programs similar to GEI. The decision was made to no longer offer the Y-STEM and GEI initiatives and to not retain ARC as a separate initiative. ARC-like collaborations would still be included as an eligible activity and delivery model for some of the SOPI. All other SODP initiatives have been included in the new SOPI suite.

In summary, the evidence substantiates the continuing need for a program similar to SODP to foster innovation, enhance productivity and competitiveness, and encourage the commercialization of research in southern Ontario.

4.1.2 Did SODP complement, duplicate or overlap other government programs or other private sector services?

Key Findings: SODP complemented other federal and provincial government funding programs where they were available to recipients. The existence of other programs targeting some of the same areas as SOA was not considered a concern: the other programs had a different emphasis, and the demand for funding in southern Ontario far exceeded the limited funding available from SODP. For example, Ontario provincial government funding was coordinated with SODP to provide complementary funding, usually with a different emphasis (e.g., operating vs. capital).

The SOA initiatives were based on needs identified through meetings with stakeholders and industry leaders to identify their challenges, research on the federal and provincial economic development landscape, and analysis of where FedDev Ontario programming could have the greatest impact. The objective was to achieve the best results with the finite funding available, given the needs and size of the southern Ontario economy.

The SOA initiatives provided support to individuals and organizations from start-ups to multinationals. While other federal and provincial government programs targeted some of the same areas as the SOA initiatives, they had different emphases, such as recipient, reason or purpose, type of support, size of support, lifecycle and geographic area affected. Provincial government respondents stated that the province coordinated its programs with SODP and provided complementary funding, usually with a different emphasis (for example, operating vs. capital). According to studies, SMEs frequently face more difficulties accessing financing due to greater loan default risk, greater year-to-year fluctuations in sales and earnings, shorter credit histories and inadequate collateral.Footnote 22 Without sufficient protection (i.e., collateral and/or co-signing agreements), lenders are more likely to reject these businesses' requests for loans.Footnote 23

All respondents agreed that that SODP complemented other federal and provincial government funding programs where they were available to recipients. Other programs that were available to recipients were not perceived as duplication or competing with each other, as each program had different emphases and the demand for funding in southern Ontario far exceeded the limited funding available.

4.2 Issue #2: Alignment with Government Priorities

Key Finding: SODP was consistent and fully aligned with FedDev Ontario's PAA and strategic outcome and with federal government priorities and strategies.

In the Speech from the Throne (), the government stated that "creating jobs and securing economic growth is and will remain our government's top priority." This was followed by Budget 2014, in which the federal government further stated that its priorities included a focus on jobs, growth and prosperity. In 2014, the federal government stated that it was "committed to achieving these priorities and is focusing on the drivers of job creation and growth—innovation, investment, skills and communities."

FedDev Ontario's strategic outcome—to achieve a competitive southern Ontario economy—was aligned with federal government outcomes and above priorities through its program activities and sub-activities, and its programs and initiatives, as shown in Exhibit 4.1.

Exhibit 4.1 shows the alignment between SODP and FedDev Ontario's PAA, illustrating how SODP initiatives contributed to the program sub-activities and in turn, to the program activities. SODP was designed to support Technology Innovation and Business Development and was the sole supporter among FedDev Ontario programs of these two program activities. Other FedDev Ontario programs supported Community Economic Development activities. The exhibit also shows how the SODP was aligned through the PAA with Government of Canada outcomes.Footnote 24

Exhibit 4.1: Alignment of SODP Initiatives with FedDev Ontario's Program Alignment Architecture and Government of Canada Outcomes
SODP initiative FedDev Ontario PAA (program activity and program sub-activity) Government of Canada outcome
Technological Innovation
Y-STEM (SOA) • Science, technology, engineering and mathematics (STEM) awareness An innovative and knowledge-based economy
GEI and SEB (SOA) • Skills development
ARC and TDP (SOA) • Technology development and commercialization
Business Development
IBI (SOA) • Business investment Strong economic growth
PI (SOA) and initial SODP projects (pre-SOA) • Business productivity and innovation
Community Economic Development
Other FedDev Ontario programs Strong economic growth; a diverse society that promotes linguistic duality and social inclusion

All federal government respondents considered SODP to be consistent and fully aligned with government priorities. FedDev Ontario and SODP, as its core program, were created and are consistent with the Speech from the Throne (2013), Budgets 2009 to 2014 and Canada's Science and Technology Strategy (2007).

4.3 Issue #3: Alignment with Federal Roles and Responsibilities

Key Finding: SODP was fully aligned with federal roles and responsibilities.

As an RDA and a representative of the federal government in southern Ontario, FedDev Ontario was aligned with federal government priorities through its SOA initiatives. These initiatives were designed to create a southern Ontario advantage by focusing on economic growth and job creation; fostering innovation, productivity and competitiveness, regional diversification, infrastructure improvements, and commercialization of research; and positioning southern Ontario as a strong force on the global stage.

All federal government respondents stated that SODP was fully aligned with federal roles and responsibilities and that the SOA initiatives were a formal and explicit match with government policy. During the evaluation period, the economy was the major focus of the federal government.

top of page

5.0 Findings on Performance Issue #4: Achievement of Expected Outcomes

This section provides the evaluation findings for performance issue #4: achievement of expected outcomes. The findings provide the answers to two key questions in Exhibit 3.1:

5.1 Achievement of the Immediate, Intermediate and Ultimate Outcomes

Key Findings: Evidence showed that SODP achieved the outcomes expected by the end of the program on . The initiatives met or exceeded almost all their performance targets. It is expected that projects, many of which were completed by , will take different lengths of time to achieve their full impacts (e.g., sales and employment).

The SOA initiatives received program expenditures of $419.7 million, and a further $1.1 billion was leveraged from clients. This co-investment approach resulted in $2.65 of client funding being raised for each dollar of program funding expended and total SOA project expenditures of over $1.5 billion. The initiatives were successful in terms of innovations developed and commercialized and related sales. They were also successful in creating jobs and full-time employment. In the SME survey, 94 percent of the respondents rated their project as "good" to "excellent" in meeting its objectives. The three People Advantage initiatives were successful in reaching over two million children and youth, funding internships leading to employment, and helping entrepreneurs with their start-ups.

The pre-SOA initiatives involved expenditures of $177.4 million. A subsequent survey rated pre-SOA projects as "successful" to "very successful" in achieving impacts.

Partnerships were key to expanding FedDev Ontario's impact. The program was able to leverage its impact as a champion or convener, using its contribution funding as an incentive. The program was able to leverage private sector involvement through NFPs and PSIs by requiring private sector partnerships as a condition of contribution funding. This contributed to long-lasting involvement and benefits.

From responses to the SME survey, for the four-year period following project completion three key outcomes were forecast: (1) total SME sales resulting directly from projects supported by SODP would be an estimated $1.85 billion minimum; (2) SME export sales would be a minimum of $0.81 billion; and finally (3) 3,833 full-time equivalent (FTE) positions would be created.

In terms of ultimate outcomes, the findings of a Statistics Canada study indicated that SODP has generally impacted the assisted SMEs positively in post-funding employment and survivability compared to firms that were not assisted. It is recommended that the study be repeated in a few years to assess how project outcomes will have contributed to SMEs' long-term performance.

For the SMEs included in the survey, the total estimated SODP initiative investment was $284 million. The estimated return ratio of SODP contributions to sales generated in the first four years after completion was a minimum of 1:5.

This section addresses the following question: To what extent have the immediate, intermediate and ultimate outcomes been achieved as a result of SODP?

Program outputs and outcomes result from the completion of SODP (SOA and pre-SOA) projects, when the initiative and program objectives are met. The linkage between the program activities, outputs and outcomes for each initiative is shown in the SODP logic model in Appendix B.

A summary of the achievement of immediate and intermediate outcomes is given below. Appendix C provides detailed SODP performance results. The assessment of these outcomes is based on comprehensive performance information collected by FedDev Ontario on the implementation of each SOA initiative, KI interviews, file reviews, prior research and the SME survey. Case studies and success stories have been used to illustrate how the initiatives contributed to project successes. Activities and outputs are included in the discussion, as they are directly linked to achieving the outcomes.

The objectives of the various initiatives differed, as did their immediate and intermediate outcome measures, as described above in Section 2.2. Further, some of the projects were designed to have an immediate impact (e.g., IBI and PI-PE), while for others (e.g., TDP and PI-BCA); the full benefits will take a number of years to materialize. It is important to recognize that the outputs and outcomes of the investments reported below (e.g. sales, employment), are based on early results for most projects and they would be expected to increase for a number of years following the investments. Eventually, all initiative projects will contribute to the ultimate (macroeconomic) outcomes.

The summary of the achievement of immediate and intermediate outcomes is followed by a discussion of achieving the ultimate outcomes, which involves the combined contributions of all the pre-SOA and SOA initiatives.

5.1.1 All Initiatives

SODP was found to be successful in achieving the outcomes that were expected by the end of the program on . The initiatives met or exceeded almost all of their performance targets. It was expected that projects would take different amounts of time after project completion to achieve their full impacts.

A summary of the findings for the SOA and pre-SOA initiatives is given below:

The number of innovations developed was just over 2,300. Excluding SEB, for which information was not available, more than 800 of 1,900 innovations had been commercialized as of , with sales estimated to be $186 million. In addition, just over 60,000 person–months of employment had been created or maintained by the IBI, TDP and PI initiatives, creating an estimated 1,300 jobs of which over 80 percent were full time. The NFP deliverers of the Y-STEM initiative held close to 30,000 outreach sessions and reached an estimated 2.1 million children and youths. GEI funded over 1,200 internships, with 71 percent leading to employment. SEB supported 425 start-ups and as of , almost all were still in business.

All FedDev Ontario interview respondents agreed that overall SODP had been successful in achieving expected outcomes. The program met or exceeded targets for most indicators. Partnerships were key in expanding FedDev Ontario's impact. Through the SODP, FedDev Ontario was able to leverage its impact as a champion or convener, using its contribution funding as an incentive. For example, FedDev Ontario was able to leverage private sector involvement through NFPs and PSIs by requiring private sector partnerships as a condition of contribution funding. Further, FedDev Ontario forged long-lasting relationships and built platforms with the objective of achieving continued involvement and benefits. In determining whether immediate or longer term benefits are better, key informants noted that initiatives with cluster projects like TDP and PI–BCA may be better suited to achieving benefits in the longer term. Despite the pre-SOA program being launched quickly with a limited amount of funding available, more than 90 percent of the applicants that received support are still in business. The pre-SOA experience also led to the successful design of the SOA initiatives.

In interviews with recipients, all delivery partners stated that their projects would not have gone ahead or the project outcomes been realized without SODP contributions. Recipients through third-party delivery stated that their projects would not have proceeded or would have proceeded more slowly without the SODP contributions. This was corroborated by the SME survey, in which 86 percent of the SMEs stated that if they had not received funding, their project would not have been carried out or would have been carried out with changes to size or scope. Close to 9 percent stated that the project would have been carried out with no changes to size and scope, and the remaining 5 percent provided a "don't know" response. Of those that would have proceeded or would have proceeded with changes to size or scope, more than half stated that the project would have been delayed. In addition, almost half stated that the funding assistance was helpful in obtaining funding from other sources.

5.1.1.1 Achievement of Project Objectives

When SMEs were asked to what extent their project had met its objectives, 94 percent of the 529 respondents stated that it had been "good" to "excellent", with 34 percent selecting "excellent" and 39 percent "very good".

SMEs also identified each of their project's objectives and whether each had been or will be achieved. The responses are given in Exhibits 5.1a and 5.1b. The exhibits show that a high percentage of SMEs have been successful in either achieving their project objectives or expecting to achieve their project objectives in the future.

Exhibit 5.1a: Achievement of Project Objectives (Products, Services, Processes and Practices)
Exhibit 5.1a: Achievement of Project Objectives (Products, Services, Processes and Practices) (the long description is located below the image)
Exhibit 5.1b: Achievement of Other Project Objectives
Exhibit 5.1b: Achievement of Other Project Objectives (the long description is located below the image)
5.1.1.2 Estimates of Sales, Employment and Jobs

The SME survey also asked SMEs to provide estimates of sales, employment and jobs resulting directly from projects supported by an SODP initiative. Estimates were requested for sales, export sales, and employment created and maintained: full-time equivalents (FTEs), part-time jobs and contract jobs. The estimates were requested for three periods: total up to project completion; total for first two years after project completion; and total for first four years after project completion. While the totals up to project completion are based on actual data, estimates for two and four years after project completion contain actual and forecast data, depending on when the project was completed. The survey only presents a partial picture of SODP impacts, as it focuses on SMEs included in the survey and does not include sales and jobs generated by TDP and PI cluster projects.

To assess the incremental impact, the SME survey asked what would have happened if the SODP funding had not been provided. According to the 539 responses, 9 percent of the SMEs stated that their projects would have been carried out with no changes to size and scope. The estimates of sales, employment and jobs for the 9 percent were removed from the analysis, leaving 493 responses. The results for the 493 respondents are given in Exhibit 5.2.

Exhibit 5.2: SME Survey Results for 493 Respondents
SME survey results(resulting directly from supported projects) Up to project completion Expected in the first 2 years after project completion Expected in the first 4 years after project completion Total up to project completion and for first 4 years after project completion
Total estimated sales ($ millions) 515.0 738.3 1,331.1 1,846.1
Total estimated export sales ($ millions) 275.1 356.7 539.1 814.2
Total estimated FTEs created* 1,598 1,304 2,235 3,833
Total estimated FTEs maintained 1,841 2,003 2,217 4,058
Total estimated full-time jobs created** 1,029 1,110 1,786 2,815
Total estimated part-time jobs created 204 256 356 560
Total estimated contract jobs created 196 192 193 389

Source: SME survey.

* Full-time equivalents (FTEs) in Person years - e.g., 1 person working full-time for 6 months is equivalent to 0.5 FTEs

** Full-time: position without a fixed end date, involving 35-40 hours per week; Part-time: position involving less than full-time without a set end date; and, Contract: a full-time or part-time position with a fixed end date.

As shown in Exhibit 5.2, for the 493 SMEs that responded, total sales up to project completion and including the first four years after the project completion were estimated to be $1.85 billion, while export sales were estimated to be $0.81 billion over the same period. The number of FTEs created in that period is estimated to be 3,833; and FTEs maintained is 4,058. This indicates that the SMEs were able to retain employment as a result of their supported projects during the period up to project completion and for four years afterwards. Over the same period, the number of part-time jobs created is estimated to be 560 and the number of contract jobs created is estimated to be 389.

The results in Exhibit 5.2 are only for 493 respondents to the survey, reflecting 27 percent of the total SMEs that received the survey and only 20 percent of the SME projects funded. These then are minimum estimates; the actual results may be higher.

5.1.2 Individual SOA and Pre-SOA Initiative Findings

The individual initiative findings have been organized by FedDev Ontario's PAA during SODP, in particular for the two program activities and five sub-activities that are relevant for SODP.Footnote 25 The sections that discuss each of the initiatives are listed in the third column of Exhibit 5.3.

Exhibit 5.3: Program Activities and Sub-Activities, SODP Initiatives, and Applicable Sections
PAA (program activity and program sub-activity) SODP SOA and pre-SOA initiatives Applicable section
Technological Innovation
• Science, Technology, Engineering and Mathematics (STEM) Awareness Y-STEM 5.1.2.1
• Skills Development GEI and SEB 5.1.2.2
• Technology Development and Commercialization ARC and TDP 5.1.2.3
Business Development
• Business Investment IBI 5.1.2.4
• Business Productivity and Innovation PI and pre-SOA 5.1.2.5
5.1.2.1 STEM Awareness: Youth STEM

STEM Awareness, a PAA sub-activity, encouraged youth to pursue education and careers in science, technology, engineering and mathematics (STEM) and improved their understanding of the business of science. Research had shown that only 37 percent of the Canadian teens enrolled in high school science were interested in taking a science course at the postsecondary level.Footnote 26 At the same time, employers in STEM-related industries were reporting current or anticipated skill shortages. The development of a more innovative and productive economy depends on graduates in STEM disciplines. When young people are encouraged early in their education, they can be inspired to pursue STEM education and careers.

Y-STEM

The Y-STEM initiative was supported by non-repayable contributions to NFPs delivering the program. Following the launch of Y-STEM in FedDev Ontario received applications from 44 NFPs, of which 15 were approved to deliver the Y-STEM initiative. Total Y-STEM expenditures were $13.3 million.

The contributions assisted the 15 NFPs in developing and expanding the delivery of programming to engage youth, educators and volunteers in STEM outreach programs with the objective of encouraging youth to consider STEM careers. Y-STEM targeted youth in southern Ontario and included Aboriginal and Francophone youth and youth living in distressed areas. Some NFPs targeted large numbers of youth across southern Ontario (e.g., Let's Talk Science, Earth Rangers Foundation and Perimeter Institute), while other NFPs focused on small, more intensive deliveries (e.g., SHAD and the Canadian Association for Girls in Science).

The initiative was a success: it exceeded its targets. It is estimated that FedDev Ontario's Y-STEM contributions resulted in STEM programming reaching an additional 2.1 million youth, considerably more than the initiative target of 1.5 million youth. Kindergarten to Grade 12 enrolment in southern Ontario was estimated to be 1.9 million in 2012–13; the program reached a large number of youth in southern Ontario, some multiple times. While it was expected that Y-STEM would contribute to an increased number of youth making a decision to pursue a postsecondary STEM education or a STEM career in the future, Y-STEM's effectiveness in encouraging youth was difficult to measure.

To provide some insight on the initiative, a case study of one of the NFPs delivering the initiative was prepared.Footnote 27 A summary is provided below.

Case Study (Let's Talk Science)—FedDev Ontario funding was used by Let's Talk Science to increase participation of children and youth, educators and volunteers in its STEM outreach programs across southern Ontario. This was achieved by increasing the accessibility of the STEM program (i.e., more outreach sites) and providing new educators and volunteers with the training and tools to engage children and youth. Outreach sites were located at various universities and colleges and were managed locally by student coordinators, who provided volunteers with the support and tools to implement Let's Talk Science programs in the communities. In addition, Let's Talk Science developed a group question and answer and design challenge competition for grade 6, 7 and 8 students to acquire science knowledge. It also developed CurioCity, an interactive, web-based meeting place where teens can connect with postsecondary students and science professionals to explore and discover STEM behind everyday life.

FedDev Ontario's $2.0 million in funding assisted Let's Talk Science in developing an additional 6,500 postsecondary volunteers and 800 professional volunteers, offering close to 6,000 sessions, and reaching an additional 487,000 youth across southern Ontario. The project provided 148 person–months of employment, and created 5.5 new FTE jobs.

5.1.2.2 Skills Development: GEI and SEB

Skills Development, a PAA sub-activity, was designed to expose STEM students to research opportunities and careers in the private sector; provide skills training to graduates and graduate students in business skills or financing related to commercializing innovations or to successful business start-ups in STEM fields; and to address future skills shortages in management in STEM fields, including engineering, architecture, science and information systems. It included the two SOA initiatives: GEI and SEB, discussed below.

Graduate Enterprise Internship

The GEI initiative had the following objectives: developing business and management skills in STEM graduate students; providing career networking opportunities; building the next generation of potential managers; and enabling SMEs to benefit from the technical knowledge of STEM graduate students and recent graduates. Research had shown that Canadian STEM graduate students and recent graduates were not being exposed to skills they needed to succeed in business and to lead innovation.Footnote 28 In addition, Canadian businesses and other organizations were underutilizing the skills, talent and knowledge of STEM graduates.

The FedDev Ontario non-repayable contributions assisted PSIs and NFPs in arranging internships with structured mentoring opportunities for graduate students and recent graduates of STEM programs with SMEs in southern Ontario. After the launch of GEI in , FedDev Ontario received 19 applications from PSIs and NFPs for delivery, of which 9 PSIs and 1 NFP were approved. Total expenditures for GEI were $15.9 million, and an additional $12.3 million was leveraged.

Due to the lead time required to establish partner relationships and then place interns with SMEs, initial uptake was slower than for some of the other SOA initiatives, but uptake eventually increased. Despite a slower start and not meeting all initiative targets, the GEI was considered a success.

The total number of participants with internships was 1,230, somewhat lower than the 1,480 targeted. However, the internships led to higher than expected levels of employment, with 71 percent of the students or graduates obtaining employment, compared to the targeted 50 percent. The number of SMEs involved was 931, which was lower than the 1,200 targeted. Some SMEs took multiple interns.

To illustrate the GEI initiative, a success story prepared by FedDev Ontario is given below.

Success Story (Mitacs)—FedDev Ontario assisted Mitacs (Mathematics of Information Technology and Complex Systems) in placing interns in SMEs across southern Ontario to receive business mentoring while enhancing the innovation and R&D capacities of the SMEs. Mitacs, based in Toronto, is a national NFP that designs and delivers research and training programs. It works with 60 universities, thousands of companies, and both federal and provincial governments to build partnerships that support innovation in Canada. During the GEI project, Mitacs placed 256 interns with 187 SMEs, which resulted in 204 jobs for the interns. The contribution funding provided was $4.23 million.

Scientists and Engineers in Business

The SEB initiative had two main objectives. The first was to improve the business and management skills of STEM entrepreneurs. The second was to improve access to financing and business support services needed to successfully launch and manage SMEs in southern Ontario. STEM graduates are generally not equipped with the business skills and financing needed to commercialize innovations or start successful STEM businesses. For starting a business, management talent has a more important role than does STEM knowledge.Footnote 29

The FedDev Ontario non-repayable contributions assisted PSIs and NFP in the following ways:

After launching SEB in , FedDev Ontario received 14 applications from PSIs and NFPs for the delivery of the initiative, of which 5 and 7 NFPs were approved. Total SEB expenditures were $15.2 million, and an additional $16.1 million was leveraged.

SEB was considered a success. A total of 425 entrepreneurs were supported through the initiative, exceeding the target of 320. The number of PSI commercialization fellowships was 98, compared to the target of 80; and the number of start-ups accessing financing through the NFPs was 327, compared to the target of 240. The survival rate of the start-ups continuing in business or having made a successful exit as of , was 98 percent, far exceeding the target of 15 percent. Respondents in the SME survey, undertaken in late 2014, indicated that the survival rate at that time was 92.5 percent, although the actual percentage may be lower, as not all SEB participants were reached or responded to the survey.

The SEB initiative is illustrated by the following success story, prepared by FedDev Ontario.

Success Story (University of Waterloo)—FedDev Ontario funding assisted the University of Waterloo through a contribution of $630,000 from the SEB initiative. The university's Commercialization Office (WATCO) provided commercialization fellowships to recent STEM graduates to help them create viable businesses in the Waterloo region. The purpose of the project was to help develop highly promising new technologies by providing companies, founded by recent graduates, with commercialization fellowships.

A total of 17 fellowships were awarded, including one to the founder of the successful start-up, Thalmic Labs. The fellowship accelerated the development of the company's Myo gesture control armband, leading to $14.5 million in venture capital investment in . The company has since grown to 34 employees and has started delivering products. The company has also received a number of awards.

5.1.2.3 Technology Development and Commercialization: ARC and TDP

Technology Development and Commercialization, a PAA sub-activity, was designed to develop stronger linkages between SMEs and PSIs and to support their partnerships. SMEs would be able to enhance their competitiveness and productivity by taking advantage of the research capabilities of the 35 PSIs in southern Ontario. The sub-activity had two SOA Knowledge Advantage initiatives: Applied Research and Commercialization (ARC) and the Technology Development Program (TDP).

Applied Research and Commercialization

The objective of the ARC initiative was to support innovation in SMEs by encouraging greater collaboration and partnerships with PSIs, linking applied R&D expertise with pre-commercialization needs. According to the Science and Technology Strategy (2007), business provided 54 percent of R&D in Canada, well below the OECD average of 68 percent. SMEs often lack the R&D capacity, working capital, and critical mass that would allow production to be taken off-line for product and process development.Footnote 30

When ARC was launched in as a two-year pilot, roughly six months ahead of the other SOA initiatives, FedDev Ontario received 41 applications from PSIs for the delivery of the initiative. Of those applicants, 24 received approval, receiving contributions of $14.5 million. With the success of the initiative, ARC was extended in through 23 of the original PSIs. Total expenditures for the ARC Pilot and ARC Extension were $27.6 million, leveraging an additional $28.9 million.

The ARC initiative supported partnerships between PSIs and SMEs in southern Ontario to improve innovation and productivity and to address pre-commercialization needs to help move innovative products and practices to market. The ARC guidelines were as follows:

All in all, the 24 PSIs collaborated with 574 SMEs (305 in the ARC Pilot and 269 in the ARC Extension), which considerably exceeded the target of 440. Interest in the initiative was high, as indicated by the fact that PSIs received 530 applications for the 269 projects that were funded under the ARC Extension.

The number of products, services, processes or practices developed was 875, far exceeding the target of  293. The number of products, services, processes or practices that were intended to enhance productivity was 574, again far exceeding the target of 147. As of , 142 products, services, processes or practices were commercialized (target was 22). Of the 155 SMEs that received support to enhance productivity, 50 reported an improvement, (compared to the target of 22) resulting in sales of $2.6 million.

A survey of the ARC Pilot was carried out in the spring of 2012 to assess the collaborations formed between the SMEs and PSIs and to gain an understanding of the benefits and impacts achieved. The survey involved face-to-face interviews with all 24 PSIs and 62 SMEs, representing a 20 percent statistical sample of the 309 SMEs supported in the pilot. The overall conclusion from the survey was that the ARC initiative was "wildly successful", meeting a critical need for PSI collaboration with SMEs; enabling PSIs to reach out to, work with, and be relevant to the community; and resulting in a noticeable expansion in applied research capacity. The survey method and results obtained are documented in The Applied Research and Commercialization Initiative: Survey of PSI and SMEs Experiences () by FedDev Ontario's Evaluation Directorate.

In the SME survey, 91 percent of the 116 SMEs that responded rated their project's success in achieving its objectives as being "excellent", "very good" or "good". In addition, just over 60 percent stated that the project would not have been carried out without the ARC funding contribution, and a further 30 percent stated that the project would have been carried out but with changes to size and scope.

FedDev Ontario interviewees considered the ARC initiative a "shining example" of outcomes achieved beyond expectations. The initiative resulted in increased productivity and new products or services being created and commercialized. It was viewed as a much needed intervention, as it sensitized PSIs to working with business and encouraged companies to invest in R&D. ARC was considered "a gift that keeps giving": the PSIs continue to work with the ARC companies. Both the initial ARC Pilot and subsequent ARC Extension were considered "wildly successful" in achieving ARC's objectives. The new suite of programming under the Agency's second mandate did not retain ARC as a separate initiative, but it did include ARC-like collaborations as an eligible activity and delivery model for some of the SOPIs. A national NRC–IRAP voucher (Credit Note Initiative) for SMEs to work with PSIs on small assignments, similar to the ARC program, was introduced through EAP 2013.

The ARC initiative is illustrated by the following success story, prepared by FedDev Ontario.

Success Story (Algonquin College)—The Algonquin College of Applied Arts and Technology in Ottawa received $750,000 in the ARC Pilot and a further $750,000 in the ARC Extension. Algonquin was able to enter into collaborations with 32 SMEs. One example was a project between Algonquin and Palomino Inc., a Toronto-based software and web solutions provider and creator of the WebPal Application Suite. WebPal enables medium-sized organizations to take their business processes online and interact with clients on a whole new level. Palomino has worked with a broad range of businesses and organizations over the years, from large academic institutions to grassroots small businesses. Hundreds of businesses are subscribed to the WebPal platform, and thousands log into WebPal every day.

This project resulted in the development of software to connect mobile vital sign monitoring devices with cloud-based electronic medical record software and with detailed simulation data, which allowed the company to greatly accelerate the time to market for their new system. Palomino is contemplating turning the product into a separate health care division of the company and aggressively targeting the Canadian long-term care market. Palomino has also endorsed the college's bid for long-term Natural Sciences and Engineering Research Council funding for a wellness, health and social innovation project and expects to carry out additional projects like this with the college in the future. The amount of funding support provided for this project was $50,000.

Technology Development Program

The TDP initiative had the following objectives: bridging the gap between R&D and the commercialization of market-driven "game-changing" technologies; increasing collaborations involving private sector, academic and innovation organizations; and leveraging private sector investment in game-changing technologies. In terms of context, Canada ranked 16th in the OECD for creating high-quality patents per million populationFootnote 31 and 14th of 17 advanced economies for innovation.Footnote 32

TDP recipient organizations are lead organizations that must be working with at least one private-sector collaborator, and the project must have at least 50 percent financing from nongovernmental sources. Projects need to contribute to the development of a globally competitive market-ready technology that has the potential to develop opportunities for businesses or an industry sector.

TDP provided 50 percent funding of direct eligible costs up to $20 million to established NFPs or PSIs in southern Ontario, with the remaining 50 percent to be provided by the private sector. After the launch of TDP in , FedDev Ontario received 19 applications from PSIs and NFPs, of which 6 received approval. Total funding expenditures were $56.9 million, with an additional $83.4 million leveraged from other sources.

FedDev Ontario interviewees noted that the six TDP projects were focused on the long term and were funded as game changers to diversify and provide comparative advantage in the southern Ontario economy. These projects were very large and involved many players and agreements, taking some time for FedDev Ontario to assess and approve. The focus was on technology development, not commercialization. It is expected that the TDP projects will do well in the future, as time is required for these types of cluster projects to build innovation and achieve benefits. To date, these clusters have had varying degrees of success, with some projects already providing benefits. In addition, some of the TDP clusters are proposing new projects under SOPI, FedDev Ontario's suite of programming in its second mandate.

Projects are expected to result in the creation of innovative products, substantial numbers of permanent jobs, and sales three to five years after project completion. As of , the six TDP projects have formed 109 partnerships, exceeding the target of 98. They had developed 118 new products, services, processes or practices, more than the 49 targeted. Of these, 24 had been commercialized with estimated sales of $8.0 million.

The six TDP project lead organizations, together with a description of the projects, are given in Exhibit 5.4.

Exhibit 5.4: Descriptions of TDP Projects
Project lead Project description
York University Research partners will create the Connected Wellness Platform, a cloud-based software system that will enable patients, their families, friends and professional care teams to collaboratively manage their health and wellness through the use of innovative applications
GreenCentre Canada GreenCentre commercializes new solvent technology to process or sanitize previously unrecyclable materials, such as styrofoam, contaminated plastic, oil sands tailings, and offshore drilling cuttings
University of Waterloo The Southern Ontario Water Consortium will build a watershed-level R&D platform to accelerate the commercialization of new "game-changing" technologies
Sunnybrook Research Institute (SRI) SRI will collaborate with 19 organizations to accelerate the commercialization of four image-guided therapy systems to establish a world-class image-guided therapy sector in southern Ontario
Ontario Brain Institute (OBI) OBI will accelerate the development of Ontario's neuroscience sector and contribute to the development of southern Ontario's growing neurotechnology cluster, NeuroTech Ontario, to commercialize brain-related technologies
Communitech The project will enable the commercialization of several new in-demand global data services, forge connections with new industry partners, and raise the profile of Canada as a leader in global data services

To illustrate how the initiative assisted one of the TDP clusters, a case study was prepared as part of this evaluation. A summary is given below.

Case Study (Ontario Brain Institute)—In 2011, the Ontario Brain Institute (OBI) began the NeuroTech Ontario initiative with assistance from FedDev Ontario. Acting as the research manager of the project, OBI catalyzed partnerships between universities and private sector companies to accelerate the commercialization of neurotechnologies and contribute to the development of Ontario's growing neurotechnology cluster (called NeuroTech Ontario).

Founded in 2010, the OBI is a provincially funded, not-for-profit research centre. Its mission is to maximize the impact of neuroscience and establish Ontario as a world leader in brain discovery, commercialization and care. FedDev Ontario's non-repayable contribution of $5.1 million through the TDP initiative was instrumental in getting the $13 million initiative off the ground. The remaining $7.9 million was provided by the private sector. According to OBI, the SODP funding got industry to come to the table to take part in the NeuroTech Ontario initiative and also attracted private sector investment. OBI brought together several organizations to partner in the initiative: 12 research institutions and universities across southern Ontario and 12 industry organizations (9 Ontario SMEs and 3 international companies). OBI had originally planned 14 projects; however, it was unable to launch 2 of the larger projects for reasons beyond its control; as a result only about half of OBI's originally approved FedDev Ontario contribution of $11 million was disbursed.

The 2.5-year initial phase of the NeuroTech Ontario initiative ended in , with 12 projects successfully completed. Those projects created innovative neurotechnology devices, developed software to improve cognitive abilities, and enhanced imaging technologies to advance and improve neurological disease diagnosis, intervention and treatment. At least 9 of the 12 projects are continuing via follow-on funding contributed by the original industry partners, new secured venture capital funding, and/or commercialization plans. OBI will continue its role supported by a renewed contribution of $100 million over five years, announced by the Government of Ontario in 2013.

In the three years since its inception, despite being unable to launch 2 of its planned 14 projects, OBI (NeuroTech Ontario) has helped Ontario's neuroscience sector expand, enhanced the competitiveness of local brain technology companies, and brought innovative brain technologies to market in Canada and abroad. OBI continues to grow the NeuroTech Ontario cluster via partnerships and financing from the public and private sector.

5.1.2.4 Business Investment: IBI

Business Investment, a PAA sub-activity, provided non-repayable contributions or seed financing to start-up businesses to accelerate the commercialization of new products, systems, processes and practices and to leverage private sector investment. A weak R&D and commercialization record and a lack of investment capital for companies had been identified as major impediments to getting innovative products to markets. Footnote 33 Ontario's venture capital activity had dropped from a peak of $3.5 billion in 2000 to $424 million in 2010. Angel and venture capital investment funds have untapped potential to increase access to capital.

Investing in Business Innovation

The IBI initiative had the objectives of accelerating the commercialization of new products, systems, processes and practices; increasing, stimulating and leveraging private sector investment; and encouraging the growth of angel investment funds. The initiative had two main streams of support:

Following the launch of IBI in , FedDev Ontario received 203 applications, of which 106 were approved, for total contributions of $60 million. The approved applicants consisted of 86 SMEs, with approved repayable contributions of $56.8 million; and 18 angel networks and 2 NFPs representing angel networks, with non-repayable contributions of $1.4 million and $1.8 million, respectively. The initiative has expended $56.4 million and has leveraged an additional $190.1 million from other sources.

The initiative was very successful in achieving results and exceeded all its targets. Total investments in SMEs of $246.9 million considerably exceeded the target of $147 million. The 86 SMEs that were supported developed 633 innovations, far exceeding the target of 50. Of these innovations, 364 had been commercialized as of , with resulting sales of $79.3 million. The projects have resulted in over 1,200 person–years of employment created or maintained and over 550 jobs, consisting of 76 percent full-time, 8 percent part-time and 17 percent contract. As of , there had been a 95 percent survival rate, and repayments were on track.

The angel investor stream resulted in 772 new partnerships with angel investors and investment organizations in southern Ontario, exceeding the target of 600. These new partnerships have brought in additional funds, including some from the United States and the United Kingdom. The amount of new investment attracted was estimated to be $95.9 million, which exceeded the target of $67.9 million.

In the SME survey, 95 percent of the 39 recipients of IBI funding that responded rated their project's success in achieving its objectives as "excellent", "very good" or "good". In addition, 35 percent stated that the project would not have been carried out without the IBI funding contribution, and a further 63 percent stated that the project would have been carried out but with changes to size and scope.

Interviews indicated that IBI was very popular and was well regarded by both government and recipients. IBI provides tremendous leverage for the dollars spent on the program, and FedDev Ontario benefits from reduced risk because the angel investors qualify the investments. To participate in IBI projects, angel investors must belong to a registered angel group. After the SME financings and towards the end of SODP, when IBI opportunities were reduced, some investors chose not to renew their membership, which created a financial planning challenge for the angel organizations. Further, several SMEs noted that they had been advised by the Canada Revenue Agency that the IBI repayable contributions would reduce their Scientific, Research and Experimental Development (SR&ED) tax credits, resulting in a negative financial impact.

To illustrate how IBI assisted an SME, a case study is summarized below.

Case Study (Wave Accounting)—Wave Accounting Inc. (Wave) is a Toronto-based company that provides a suite of online small business software products under the brand name Wave. This includes Wave Accounting, an online accounting software application for small businesses. Wave Accounting is 100 percent free, which provides a significant competitive advantage over the other international market leaders. Wave obtains revenue from payment transaction processing; paid advertisements for products and services targeting small businesses that use Wave's products; payroll; and other sources, such as priority support.

The $755,000 repayable financial contribution or loan provided by FedDev Ontario in 2011 under its IBI initiative was instrumental in attracting an additional $1.51 million in angel and venture capital that together provided key early-stage financial assistance. This funding was used for a development project to accelerate the commercialization of the company's online accounting and financial management tools for the growing small business market. Wave's management commented that "by helping Wave Accounting at this early stage, FedDev Ontario put us in a position to build a truly global enterprise, with impactful innovation and significant future growth here in the Toronto area."

The new and improved products have resulted in increased market penetration, user and sales growth, and employment. The number of users has increased from close to zero in 2011 to 1.5 million in , and employment is now at 61 jobs in late 2014, an increase of 40 from when the project started in 2011. Further, sales revenue has increased rapidly since 2011 and is expected to continue to grow rapidly, given the large and expanding customer base. In addition, Wave has been successful in obtaining subsequent rounds of financing from investors, which have provided additional resources for the company to continue its growth.

5.1.2.5 Business Productivity and Innovation: PI and Pre-SOA

Business Productivity and Innovation, another PAA sub-activity, provided repayable and non-repayable contributions to for-profit and not-for-profit corporations to improve the productivity of individual businesses, industry sectors, sub-regional economies and economic clusters and thus to improve the competitiveness of the southern Ontario economy. A number of studies had indicated that Ontario's productivity lagged that of its US counterparts. As a result, southern Ontario needed to increase productivity to remain competitive in the global economy.

PI was the SOA initiative that addressed this need. It made strategic investments with an emphasis on three priority areas: productivity enhancements, regional diversification and economic clusters. PI had expenditures of $234.4 million.

The SODP (pre-SOA) initiatives (GI and FBI), including three delivery partnerships, funded 87 projects directly and a further 810 indirectly, starting in early 2010, with expenditures of $177.4 million.

Prosperity Initiative

The PI objectives were enhancing productivity, diversifying the regional economy, and building competitiveness in southern Ontario. PI had three funding streams, which attracted a variety of projects:

The PI–PE stream assisted over 800 smaller projects, which were delivered, with one exception, through the use of delivery partners and generally focused on specific requirements that would deliver results in the near term. The PI–RD and PI–BCA streams assisted a smaller number of larger projects delivered directly by FedDev Ontario. These projects were estimated to have larger impacts but would take longer to achieve results.

PI was the largest of the SOA initiatives. Following its launch in , the program received 418 applications, of which 57 were approved, with total expenditures of $234.4 million. PI leveraged an additional $783.1 million from other sources, for a total of $1.02 billion. This initiative supported 1,298 businesses. Of those, 904 SMEs were estimated to have received a total investment of $605.9 million from the three PI streams and other sources.

Partial results as of , showed that for PI–RD and PI–BCA projects the number of innovations developed was 258, generating an additional $96 million in sales.Footnote 34 The number of new technologies adopted or adapted was just over 1,200, and the number of products, services, processes and practices that were commercialized was 271. The number of jobs created was estimated to be 553, consisting of 504 full-time, 27 part-time and 22 contract positions. For the three PI streams, 38,800 person–months of employment were generated. Targets had not been established for PI.

According to the FedDev Ontario interviews, this initiative was successful in creating jobs, commercializing products and expanding companies. For PI–PE, delivery partners were cited as being an effective and efficient approach to delivering the initiative to a large number of SMEs. For PI–RD, Dr. Oetker was mentioned as being successful in creating a substantial number of jobs. For PI–BCA, 11 clusters were established, similar to those in the TDP projects but focused more on the commercialization of innovations. The clusters were also to have SME involvement. Clusters cited included the McMaster Automotive Resource Centre (MARC), in Hamilton; the Advanced Manufacturing Park, in London; and the Southern Ontario Smart Computing Innovation Platform (SOSCIP), in Toronto and London.

a) Productivity Enhancement

The PI–PE funding stream addressed a lack of investment in machinery, equipment and software in Ontario businesses, relative to investments by similar companies in the United States. In 2009, the productivity gap between Canada and the United States was 23 percent and growing.Footnote 35 This underinvestment in capital equipment was estimated to have lowered Ontario's productivity by $800 per capita.Footnote 36 PI–PE supported the creation of new opportunities for economic diversification, market development and expansion; the attraction of businesses to diversify regional or community economies; and business expansion to support diversification.

Following the launch of PI–PE in , FedDev Ontario received 80 applications for PI–PE funding, of which 6 were approved, with total expenditures of $32.4 million. The PI–PE stream leveraged an additional $107.6 million. The recipients included four NFPs and one PSI. The four NFPs were delivery partners, three of which were the same organizations (CME, OCC and YLF) that helped FedDev Ontario deliver the pre-SOA initiatives. The number of projects delivered to SMEs under SOA by CME, OCC and YLF were 408, 191 and 179, respectively. The new NFP delivery partner was MEDEC, which delivered 30 projects to SMEs.

In the SME survey that included both PI–PE and pre-SOA recipients, 96 percent of the 290 delivery-partner recipients that responded rated their project's success in achieving its objectives as "excellent", "very good" or "good". In addition, 39 percent stated that the project would not have been carried out without the PI–PE funding contribution, and a further 45 percent stated that the project would have been carried out but with changes to size and scope. The remaining 16 percent consisted of 10 percent that would have carried out the project with no change in size or scope and 6 percent that provided a "don't know" response.

FedDev Ontario interviewees stated that the use of delivery partners was a major part of the SOA delivery strategy. It was considered to be a trusted delivery system and an excellent approach that had been very successful in reaching and supporting SMEs. Moreover, the delivery partners had the knowledge and ability to reach SMEs across southern Ontario and the capacity, including governance and administration, to support delivery. Administration costs were 5 percent of the contribution amount. This amount was considered too low to cover the effort involved.

To illustrate how the PI–PE stream assisted SMEs through delivery partners, a case study was prepared of one of the delivery partners for the Interim Evaluation. The description was updated for this evaluation and is summarized below.

Case Study (CME–SMART)—FedDev Ontario partnered with the CME,Footnote 37 one of four NFP delivery partners of SODP. The CME's SMART and SMART Prosperity Now programs helped SME manufacturers address productivity challenges; improve global competitiveness; and increase export sales, jobs and growth. SMEs assisted were located in all regions across southern Ontario, providing excellent coverage and opportunities for businesses in all communities.

In total, FedDev Ontario provided $38.85 million to the CME for southern Ontario manufacturers. An additional $131 million was leveraged from project partners. The CME–SMART program started in 2008 with funding from the Ontario provincial government. In 2009, FedDev Ontario made a $15.75 million contribution (Phase I) from the SODP pre-SOA program, which provided funding for 349 SMEs to facilitate lean implementationFootnote 38, energy conservation and management, quality improvements and IT process improvements. Then, under SOA PI–PE, CME received a $4.25 million contribution in 2010 (Phase II) for the same purpose, which helped a further 87 SMEs. Finally, in 2012, CME received an additional $18.9 million (Phase III) to deliver a new productivity enhancement program called SMART Prosperity Now, which helped 321 SMEs. This most recent funding went to southern Ontario manufacturers that were exporting, planning to export, or selling to an export supply chain. The funding supported productivity assessments and a variety of projects to improve productivity, such as integrating innovative technologies in products and processes, including alternative energy and clean technology.

The SMART program productivity assessment projects could receive up to 50 percent of eligible costs to a maximum non-repayable contribution of $50,000, while SMART Prosperity Now productivity enhancement projects could receive funding of up to 33.3 percent of eligible costs to a maximum non-repayable contribution of $75,000. CME received 5 percent of the funding to offset CME's administrative costs, but it found the percentage too low for the work entailed.

CME's estimated that the projects supported by FedDev Ontario funding created 5,550 jobs and retained 11,100.

b) Regional Diversification

The PI–RD funding stream focused on southern Ontario regions with high concentrations of manufacturing industries in structural decline and low concentrations of high-growth industries that could expose local economies to significant job losses if plants were to close. PI–RD supported the creation of new opportunities for economic diversification, market development and expansion; the attraction of businesses to diversify regional or community economies; and business expansion to support diversification.

Following the launch of PI–RD in , FedDev Ontario received 181 applications for PI–RD funding, of which 19 were approved, with total expenditures of $74.7 million; $176.7 million was leveraged from other sources. The recipients were 15 SMEs and 4 NFPs.

The following case study was prepared for this evaluation to illustrate how the PI–RD stream assisted SMEs.

Case Study (Flying Colours)—Flying Colours Corp (FCC) is an aviation service provider specializing in all aspects of aircraft customization, including executive conversions, interior completions, refurbishment and modification, upgrades and installation. Established in 1989, the company is family owned and operated. With its base of operations at Peterborough Airport and a subsidiary in St. Louis, Missouri, FCC is already a strong competitor in the field of corporate aviation in markets all over the world. FCC is well positioned to grow by accommodating increased traffic in Peterborough and rising demand for customized business jets internationally.

In 2013, FCC received a repayable contribution of up to $900,000 from FedDev Ontario for a total project value of $2.77 million to expand its facilities and acquire new technologies to service a growing number of business opportunities arising from the expansion of Peterborough Airport. The FedDev Ontario funding came from the SODP PI–RD stream.Footnote 39 Other funding was provided by the provincial government, as well as by FCC itself. This project created 38 highly skilled FTE jobs and increased FCC's reach in Canadian and international markets. This business expansion also allowed FCC to obtain Authorized Service Facility status with Bombardier Aerospace.

c) Building a Competitive Advantage

The PI–BCA funding stream emphasized the development or expansion of geographic concentrations of interconnected companies and institutions in a particular field, known as clusters. Clusters could provide a competitive advantage that would allow southern Ontario to compete with new emerging economies that have innate advantages, such as low labour costs. In 2010, Canada ranked 13th out of 26 countries in the Global Manufacturing Competitiveness Index. Southern Ontario has a relative strength in technology-intensive clusters in Canada, but it is only an average player in comparison to the United States.PI–RD stream.Footnote 40 PI–BCA supported activities to enhance the competitiveness of industry and sector clusters. These activities included the adaptation and adoption of new technologies, processes and skills development in an industry or sector; technology demonstration or piloting; business opportunity development and expansion; and facility improvement or expansion.

Following the launch of PI–BCA in , FedDev Ontario received 257 applications, of which 32 were approved, with total expenditures of $127.3 million. The PI–BCA stream leveraged an additional $498.7 million. The recipients were 13 SMEs and 19 NFPs or PSIs.

The following are illustrations of two cluster projects that were carried out in conjunction with private sector partners.

The following case study illustrates how the PI–BCA stream assisted in establishing a cluster involving seven southern Ontario PSIs and a major industrial partner.

Case Study (Smart Computing)—The Southern Ontario Smart Computing Innovation Platform (SOSCIP) was established on , as a new kind of collaborative research partnership to bring together university researchers and private-sector companies to use the latest advanced computing technologies. The objective of SOSCIP was to undertake collaborative research to solve important problems facing society and to build a competitive advantage for southern Ontario.

FedDev Ontario's PI–BCA initiative was instrumental in establishing SOSCIP by providing a non-repayable contribution of $20 million to the University of Toronto to form a collaborative research and innovation platform with six other Ontario universities and a lead industry partner, IBM Canada Ltd. The consortium would also foster partnerships with SMEs that could benefit from commercially promising research. Financial contributions of over $190 million were also provided by other organizations, including the Government of Ontario and in kind by IBM.

SOSCIP researchers at the partner universities, IBM and SMEs collaborate on projects in the areas of water, energy, cities and health to create jobs, solve industrial problems, develop new business opportunities, contribute highly skilled personnel to the workforce, and bring "made in Canada" products and services to market quickly and more efficiently. The platform is expected to boost Canadian competitiveness in the global economy by developing specialized skills in data management and analysis, software engineering and production in southern Ontario. It is expected that communities in southern Ontario will benefit from diversification and the creation of new economic opportunities.

After the initial phase of building the platform, SOSCIP is well positioned to embark on its next phase: an aggressive plan for growing the consortium and optimizing its collaborative research program. Promising projects are underway, and more will come. Although SOSCIP research projects have been active for only two years or so, inventions and innovations are starting to spur technology transfer. Economic benefits include revenue, cost avoidance, business and societal value, jobs, intellectual property, and higher valuations of companies and assets. There are already indications of commercial impact, and the introduction of new technologies, services and products is expected soon. While it is too early to measure the full benefits of the collaborative research and innovation being undertaken, forecasts from the Principal Investigators and SMEs regarding SOSCIP-sponsored research projects indicate that potential revenue generation will exceed $66 million over the next five years.

Cluster projects were successful in realizing their objectives by the end of SODP in March 2014. Most efforts were devoted to establishing the cluster, including putting in place the necessary infrastructure. At this point, many clusters have just entered the operating phase and are still in the process of ramping up and establishing partnerships with industry to use their facilities or expertise. While some results have been achieved, it will be some time before the potential of the clusters will be fully realized, with businesses benefitting from using the facilities.

One of the requirements for receiving funding was providing a sustainability plan for after the completion of the project. While committed to their sustainability plan, many of the clusters stated that they needed additional funding to fully exploit their infrastructure, including recapitalization of equipment in future years. Partnerships with industry are one source of funding—contract revenue is obtained for the use of the facilities. Other sources include provincial government programs. Many of the clusters have applied for additional government funding. Some clusters have found that SMEs would like to use the facilities but do not have the resources to pay for them.

The following case study, prepared for the Interim Evaluation, describes how the PI–BCA stream helped an SME to expand its facilities to exploit business opportunities.

Case Study (Ivaco)PI–BCA funding of $10 million was used as part of an $80 million project to expand the Ivaco steel plant in L'Orignal, Ontario, a francophone community approximately 100 km from Montréal and Ottawa and 90 km from the US border. The project was intended to increase Ivaco's annual billet capacity to 625,000 tonnes, from 400,000. The ability to produce additional billets reduced operating costs and allowed the company to manufacture grades of steel not currently produced. Ivaco is the largest employer in L'Orignal and the surrounding area, and the project provided a more stable and sustainable economy in eastern Ontario by retaining the company's existing employees (over 400) and creating an additional 51 full-time skilled jobs, as well as indirect jobs during project implementation and after completion.

Pre-SOA Projects

The initial SODP was designed to introduce new funding into the region during a time of need, with the goal of supporting projects quickly. The initial program supported economic and community development, innovation and economic diversification by making contributions to communities, businesses and non-profit organizations in southern Ontario. Given the need to act quickly, coupled with limited capacity within FedDev Ontario, partnerships were formed with third parties to help deliver the program on FedDev Ontario's behalf.Footnote 41

There were two initiatives that were launched quickly in 2009 to achieve the desired impact. The General Intake (GI) was launched in October 2009, and had few restrictions on project eligibility. The Food and Beverage Initiative (FBI) had a narrow focus and followed in . Both intakes closed in late .

FedDev Ontario interviewees indicated that GI and FBI provided funding for about 90 projects out of just over 2,000 applications. The 90 projects included CME, OCC and YLF delivery-partner projects. These in turn funded a further 810 projects, as shown in Exhibit 2.1. Because of the positive experience with CME, OCC and YLF, FedDev Ontario partnered with them again in the delivery of PI–PE.

A sample survey was undertaken in the spring 2012 to evaluate the impacts of pre-SOA program funding, as only limited information was available on the performance of pre-SOA projects.Footnote 42 The survey method and results are documented in Southern Ontario Development Project: Survey of Funding Recipients' Experiences (), by FedDev Ontario's Evaluation Directorate. The survey found that the private sector and indirect recipients were "very successful" and the NFPs were "successful", although not to the same extent as the other two groups. The survey is described in detail in Interim Evaluation of FedDev Programs.

The CME case study given above for PI–PE also includes the CME pre-SOA project, and an illustration of how the pre-SOA initiatives assisted SMEs is provided below.

Case Study (LiquiForce Services Inc.)—Through PI, FedDev Ontario made a repayable contribution of $1.35 million to LiquiForce towards a project with a budget estimated at $1.8 million (actual costs were $2.5 million). The purpose of the project was to create an innovative process that would allow the company to rehabilitate lateral sewer pipes without digging.Footnote 43 At that time, companies could perform trenchless rehabilitation of the main sewer pipes (below the street) but had to dig a hole in the homeowners' lawn and possibly their basement to do the lateral pipe rehabilitation. This project would allow LiquiForce to bid for lateral pipe re-lining at a lower cost to municipalities, which is expected to result in expansion of its existing business. In addition, the innovation also presents an opportunity for expansion through franchising and licensing agreements. Sixteen FTE jobs were created as a result of the project, including some at the main office and others in the field. Moreover, the creation of this lower cost rehabilitation process also ensured that existing jobs were not at risk.

5.1.7 Ultimate Outcomes

The ultimate outcomes for SODP and their performance indicators, discussed in Section 2.2, are shown in Exhibit 5.5. These ultimate outcomes result from the combined outcomes of the initiatives as a whole contributing to FedDev Ontario's strategic outcome: a competitive southern Ontario economy.

Exhibit 5.5: SODP Ultimate Outcomes and Performance Indicators
SODP ultimate outcome Performance indicator
Increased innovation capacity in southern Ontario
  • Change in business expenditures on R&D from baseline
  • Change in number of employees in southern Ontario considered "highly qualified personnel" from baseline
  • Change in business investment in machinery and equipment from baseline
Stronger southern Ontario communities
  • Change in number of southern Ontario businesses by size (SME, MNE) from baseline
  • Change in employment levels by sector from baseline
More competitive southern Ontario businesses
  • Change in inflation-adjusted gross revenue of southern Ontario businesses by sector from baseline
  • Change in export revenue of southern Ontario businesses from baseline

To measure these impacts, FedDev Ontario's Evaluation Directorate, with the assistance of the Strategic Policy Branch, undertook a study in 2014 comparing firms receiving funding assistance with a sample of similar firms that did not receive assistance. Prior to that, economic modelling was used to examine the impact on the Ontario economy of the pre-SOA and some SOA projects. In addition, a partial cost–benefit analysis of the sales estimates from the SME survey provides some information related to the performance indicators listed in Exhibit 5.5.

Comparison Study

In , FedDev Ontario contracted the Centre for Special Business Projects (CSBP) at Statistics Canada to compare the performance of SMEs that received contributions from SODP (both SOA and pre-SOA) with that of similar SMEs not receiving contributions. CSBP documented its methods and findings in a report, Business Performance Measurement of FedDev Ontario Program Beneficiaries ().

The CSBP study matched assisted SMEs with unassisted SMEs over the period 2009–2013. The resulting database was used to create tabulations of 11 selected indicators, which took into account SODP's PMS. The first four indicators were financial, and the remaining seven were nonfinancial. The study demonstrated that the approach was feasible; it also provided findings for the period 2009–2013. The study found that the indicators with statistically significant differences between the assisted and the unassisted firms related to post-funding employment. This included employment growth, changes in the average number of employees, and R&D employment growth. Survival rates were found to be higher for assisted firms over three years, although only one year was found to be statistically significant.

As most of the projects were not completed until , the study did not capture the full impact of the project funding (SODP and leveraged funding) on the SMEs. The study will need to be repeated in a few years to assess how project outcomes will have contributed to SMEs' long-term performance.

Economic Modelling

Economic modelling was used on two occasions: in 2012 as a pilot for pre-SOA projects and in 2013 for some of the SOA initiatives (ARC, IBI and PI).

Comparing the direct job figures from the model with the SME survey results above is difficult, given the differences in projects included and methods used. However, the model, the pre-SOA surveys and the SME survey showed substantial increases in direct employment resulting from the SODP projects. While the impacts of $504.3 million for the pre-SOA projects and $1,117.7 million for the ARC, IBI and PI SOA projects are substantial, they are small relative to the size of the Ontario economy.

SME Survey: Partial Cost–Benefit Analysis

The SME survey asked respondents to estimate total sales that would result directly from the SODP-funded project in the first four years after project completion. As discussed in Section 5.1 above, 493 SMEs responded. These respondents estimated that expected sales would amount to $1.3 billion.

The total SODP investments in these 493 SMEs were approximately $284 million.Footnote 46 Therefore, the minimum estimated ratio of return on SODP contributions in the first four years after project completion is 1:5.Footnote 47 In other words, for every dollar that FedDev Ontario contributed to these SMEs, $5.00 will be generated in sales.

5.2 Unintended Positive and Negative Outcomes

Key Findings: Most interview respondents found it difficult to think of any unintended positive or negative outcomes. Their comments largely related to positive outcomes and emphasized that the program or project had been more successful than expected. The survey found that SMEs' projects have contributed to social outcomes in addition to the outcomes that were the focus of the initiatives. The survey also found that almost half of the SMEs face barriers when trying to exploit their project achievements. Of these, the availability of internal and external funding is a major concern, followed by lack of key staff, changes in the market, establishing a distribution network, and needing to complete related components.

5.2.1 Interview Respondents

While most interview respondents had difficulty thinking of any unintended positive or negative outcomes, the comments provided largely related to positive outcomes.

Interviews with FedDev Ontario Staff

The FedDev Ontario interview respondents mentioned several unintended positive outcomes:

FedDev Ontario interview respondents also mentioned a few unintended negative outcomes:

Interviews with Delivery Partners

SODP helped delivery partners establish a reputation and build awareness of their capabilities, leading to further applied research projects, more funding support, and additional domestic and international relationships. A common view among delivery partners was that the involvement exceeded what was expected. One recipient "expected to be happy, but didn't expect to be euphoric."

Interviews with Ultimate Recipients

Some ultimate recipients (those whose funding was delivered through third parties) commented on unintended positive outcomes of their projects:

Some interviewees mentioned unintended negative outcomes:

5.2.2 Social Impacts

Respondents in the SME survey were asked if their projects had social impacts in addition to the impacts discussed above in Section 5.1. The SMEs that indicated that their projects had a "high" or "some" social impact have been combined, and their percentages are shown below:

The above percentages indicate that SME projects are contributing to social outcomes, in addition to those that were the focus of the initiatives.

5.2.3 Barriers to Exploiting Project Achievements

When asked whether their firms faced barriers in exploiting their project achievements, 46 percent of SMEs responding to the survey answered "yes." The proportions affected by barriers with a "significant", "substantial" or "high" impact are shown below:

As indicated above, for more than half of the SMEs experiencing barriers in exploiting their project achievements, the availability of internal and external funding is a major concern, followed by a lack of key staff, needing to complete related components, changes in market, establishing a distribution network, and so on. For the "other" category, only a few explanations were given: regulatory and adoption barriers; the need for a demonstration customer; and the cash flow problems resulting from IBI repayable contributions reducing eligibility for SR&ED tax credits.

top of page

6.0 Findings on Performance Issue #5: Demonstration of Efficiency and Economy

This section provides the evaluation findings for performance issue #5: demonstration of efficiency and economy. The findings have been structured to answer the two key questions in Exhibit 3.1. They also take into account TBS guidance on issue #5Footnote 48 and the three performance components: effectiveness, economy and efficiency. The two key evaluation questions are as follows:

6.1 Efficiency and Economy of the SODP Delivery Model in Producing Outputs and Outcomes

Key Findings: The SODP delivery model was found to have been efficient and economical in producing outputs and progressing towards expected outcomes.

This section addresses the following evaluation question: To what extent was the SODP delivery model efficient in producing outputs and progressing towards expected outcomes?

6.1.1 Program Planning

The objectives, target groups and activities of SODP and its initiatives became more defined and targeted as FedDev Ontario matured after its creation in . As described in Section 2.0, FedDev Ontario had the initial objective of supporting projects quickly to stimulate the southern Ontario economy. To that end it launched the GI initiative in , followed by the more focused FBI initiative in .

Once the priority of quickly delivering the SODP stimulus funding was addressed, FedDev Ontario developed a more focused program aligned with its mandate to promote the development of a strong and diversified southern Ontario economy. With input from stakeholders and industry on the regional economic challenges they faced, FedDev Ontario developed the seven Southern Ontario Advantage (SOA) initiatives with 12 funding streams to address these challenges. After the SOA initiatives were introduced, program management adjusted some funding parameters in two of the initiatives, SEB (NFP stream) and ARC, to better address market needs. The funding parameters for the other initiatives were not changed. A few FedDev Ontario interviewees noted that in retrospect the large number of SOA initiatives and streams was challenging and that a smaller number would have been more manageable.

6.1.2 Demand for SODP Funding

As indicated by the number of applications, the demand for pre-SOA funding was substantial. FedDev Ontario received 2,007 applications, but only 90 (4.5 percent) received approval. These pre-SOA initiatives were launched in late 2009, just after FedDev Ontario was created. At that time, FedDev Ontario was also involved in building its organization and capacity.

With the majority of SOA initiatives starting in late 2010, the proportion of applicants approved was 27 percent across the seven initiatives. However, the proportion for PI, the largest initiative, was 14 percent. The number of applicants and approved agreements are provided in Exhibit 2.3. The agreements in some of the initiatives were with delivery partners, and they also had more applicants than funding opportunities for their programs. One of the larger delivery partners commented that they were unable to address the "pent-up demand", as their program was fully allocated in a few weeks and they had a waiting list. The PSIs delivering ARC received 530 applications for the 269 projects that were funded under the ARC Extension, and IBI received 203 applications, of which 106 were funded. Some initiatives received more interest and some less, resulting in the reallocation of funding to higher interest programs from the original notional budgets.

It can be concluded that demand for the SOA initiative funding was higher than available funding and that program staff had a range of qualified applicants from which to select.

6.1.3 Use of Leverage

SODP had leverage built in as a requirement in all its initiatives (see Exhibit A.1 in Appendix A): that is, recipients were expected to provide a certain percentage of the project costs either in cash or in kind. The leverage requirement varied to some extent across the initiatives. This co-investment approach allowed FedDev Ontario to increase its impact through attracting other sources of funding. For the SOA initiative, this co-investment approach (described in Section 5.1.1) resulted in $2.65 of client funding being raised for each dollar of program funding expended.

In some cases, leverage was instrumental in attracting a greater client contribution than the initiative required. In the ARC initiative, the amount of leverage increased from the ARC Pilot to the ARC Extension. In the ARC Pilot, SODP funded 100 percent of directly eligible project costs for PSIs and up to 66.6 percent ($50,000) for SME applicants, with the SME contributing the remaining 33.3 percent. For the ARC Extension, the parameters were changed: SODP funded up to 50 percent ($100,000) of eligible project costs, with the SME contributing the other 50 percent.

6.1.4 Use of Authorized Funding

At its inception, SODP had difficulty spending the amount authorized, as shown in Exhibit 6.1. The funding was initially allocated evenly over FedDev Ontario's five-year mandate, not accounting for the need to build capacity and attract and approve program participants. In the first two years this resulted in transferring part of the funding allocation to BDC and NRC–IRAP to support their programs.

In the last two years, with capacity in place, FedDev Ontario was successful in spending almost the entire amount authorized (Exhibit 6.1).

Exhibit 6.1: Total SODP Contributions by Fiscal Year—Total Authorities and Actual Spending
Contributions ($ million)
Fiscal year* Total authorities Actual spending Variance
2010–11 159.2 72.2 (87.0)
2011–12 183.6 156.7 (26.9)
2012–13 181.3 181.3 0
2013–14 144.2 137.3 (6.9)

Source: FedDev Ontario Departmental Performance Reports.
* 2009–10 spending captured by Industry Canada, as FedDev Ontario was not created until .

Exhibit 6.2 shows the typical cost profile of a project over its lifecycle. Expenditures are shown to ramp up from zero during planning and start-up, continue growing through procurement and execution to peak expenditure, and then decline dramatically during project close out.

Exhibit 6.2: Illustration of a Typical Project Lifecycle
Exhibit 6.2: Illustration of a Typical Project Lifecycle (the long description is located below the image)

The challenge for FedDev Ontario program management is to approve a group of projects, each with the above profile but varying in cost and timeframe. When these projects are aggregated the flat-funding profile is maximized without exceeding the annual amount provided.

From the evaluation interviews it was estimated that for larger, more complex projects managed directly by FedDev Ontario it takes about 60 working days to obtain an application and then an additional 200 working days for FedDev Ontario to process the application to approval, with IBI being shorter and TDP being longer. Projects funded through delivery partners are much smaller and take considerably less time to launch. However, delivery partners must first submit a proposal to FedDev Ontario for approval before receiving the go-ahead to deliver their programs. Once approved, project expenditures are made after work is undertaken and claims are received.

Project expenditures do not start until after the project is announced and specific work is completed, and then time is taken to ramp up expenditures. This natural process has implications for the total amount that can be spent in a given fiscal year. Thus, it can be expected that expenditures considerably lag at the start of a program.

From the above, the following observations are made:

  1. FedDev Ontario's five-year mandate and flat-line annual budget established equal year-over-year spending expectations that are not necessarily reflective of the lifecycle of the types of multi-year projects funded by the Agency. Second, project proponents take time to understand new programming and its benefits, adding extra time to develop project applications. Third, after applications are received, it takes time for assessment, due diligence and the signing of contribution agreements that clearly outline terms and conditions of federal funding. And, finally, many of FedDev Ontario's projects are multi-year in nature and historical trends have shown that a majority of costs for these types of projects are typically incurred after the first year. Therefore, it is difficult for FedDev Ontario to fully expend the funding in the first two years of a five-year mandate when funding is distributed evenly across the five years. In addition, this flat-line funding profile could also create cash flow issues if projects that were approved in the first two years submitted maximum expenditures at the same time in years three to five.
  2. With a fixed end date for the program, all projects must finish by that time. In some cases, this may have implications for approved projects that may need to be adjusted or rushed to match the cash flow available.

The matching of aggregate project expenditures to program funding will be an ongoing concern of FedDev Ontario program managers as they seek to maximize the use of the flat-lined and time-limited funding without exceeding the annual amount provided. Allowing funding re-profiling would lead to improved cost-effectiveness. An ongoing program (A-base funding) would allow projects to be continuously approved and would avoid the expenditure peaking that occurs within a five-year program.

6.1.5 External Comparison of Delivery Costs

A "high-level" comparison was made of FedDev Ontario's costs for delivering its programs with the costs of other RDAs delivering similar programs. The source of information for the comparison was the 2013–14 Public Accounts of Canada,Footnote 49 a report that is produced annually and contains actual expenditures of federal government departments and agencies. In particular, the report contains G&Cs and operating expenditures.Footnote 50 This information is provided in Exhibit 6.3, together with the cost of delivering a "G&C program dollar" for each RDA, as well as the delivery cost of other RDAs as a percentage of FedDev Ontario's cost. However, the comparison was undertaken at the RDA level, which would have included all programs being delivered by the agencies; it is not just a comparison of SODP and similar programs of the other RDAs. Further, there may be other factors that might help explain some of the cost differences. For example, both Atlantic Canada Opportunities Agency and Western Economic Diversification deliver in multiple provinces in their jurisdictions. Other factors could include the type or complexity of programs delivered, whether programs have been in existence for some time, availability of delivery partners for smaller projects, and other non-G&C workload. The extent to which the above factors affect delivery costs would need to be ascertained.

Exhibit 6.3: Comparison of Program Delivery Costs for FedDev Ontario and Other Regional Development Agencies (Based on 2013–14 Public Accounts Information)
Regional development agency (RDA) (1)
Grants and contributions (G&Cs)* ($ million)
(2)
Operating expenses ($ million)
(1) ÷ (2)
Cost of delivering a G&C program dollar
($)
Other RDAs' delivery costs as a percentage of FedDev Ontario's
(%)
FedDev Ontario – Federal Economic Development Agency for Southern Ontario 204.2 32.0 0.157 N.A.
ACOA  – Atlantic Canada Opportunities Agency 236.7 77.4 0.327 209
CED – Canada Economic Development for Quebec Regions 260.9 43.4 0.166 106
WED  – Western Economic Diversification 143.0 45.3 0.317 202
CanNor – Canadian Northern Economic Development Agency 36.5 14.3 0.391 250

* The actual total value of G&Cs for other RDAs is somewhat larger than shown, as some of the other RDAs access part of their repayments for reinvestment.

Exhibit 6.3 provides the cost of delivering a G&C program dollar for each of these organizations. The comparison shows that FedDev Ontario's delivery cost of $0.157 is lower than those of the other RDAs. The other RDAs' delivery costs as a percentage of FedDev Ontario's delivery costs range from a low of 106 percent for Canada Economic Development for Quebec Regions to a high of 250 percent for the Canadian Northern Economic Development Agency. The same analysis using data from 2012–13 found that FedDev Ontario had the lowest cost of delivering G&C programs, with the next closest being Canada Economic Development for Quebec Regions with a relative cost of 126 percent of FedDev Ontario's cost.

In summary, this high-level comparison indicates that FedDev Ontario's delivery costs for G&C programs were considerably lower than those of four other RDAs, without accounting for the differences in programming and contextual factors. 

6.1.6 Third-party Delivery

FedDev Ontario made effective use of delivery partners for smaller projects. With a trusted delivery system, those partners have the reach and knowledge of the target organizations and individuals as well as the capacity to select and manage the projects with minimal monitoring from FedDev Ontario. This delivery approach was used for Y-STEM, GEI, SEB, ARC, PI–PE and part of pre-SOA. Third-party delivery is described in Section 5.0.

For FedDev Ontario to manage larger numbers of smaller projects in-house would require direct interaction with beneficiaries to obtain project applications, to evaluate and approve the submissions, to negotiate and execute contribution agreements, and to manage the projects while they are underway.Footnote 51 For some initiatives this would demand a local presence. Considerable effort would be involved, and in some cases FedDev Ontario's reach and knowledge of the target organizations and individuals might not be sufficient to successfully deliver a program or initiative. For larger projects where increased due diligence is required in assessing the application and monitoring the project, in-house management is required.

During SODP, concerns were raised by delivery partners that 5 percent of the total contribution amount was insufficient for the administrative work entailed. In one case the rate had been increased for PSIs in the ARC Extension to 12.5 percent: 5 percent for administration, 5 percent for outreach, and 2.5 percent for targeted outreach in geographic areas that had low or no uptake. The general view of FedDev Ontario interview respondents was that 5 percent was too low and that 10 percent would be more reasonable for future programs, as it would allow for improved reporting and more promotion. When the percentage is being set, the parameters of the initiative being delivered and the delivery partners' expectations should be taken into account.

While third-party delivery has been considered very successful, it has the drawbacks of not providing visibility on projects being considered for funding and not having direct involvement in the selection and announcement of funded projects. The ultimate recipients of the funding delivered by third parties were often not aware that the source of the funding was FedDev Ontario. They associated the funding with the delivery partner and its program.

Some issues were also identified in preparing for the SME survey:

An improved database would provide increased visibility on projects being funded and would allow the identification of SMEs receiving funding for multiple projects, as well as provide contact information for research purposes.

6.1.7 Partnerships

Partnerships were key to expanding FedDev Ontario's impact. The program was able to leverage its impact as a champion or convener, using its contribution funding as an incentive. The program was able to leverage private sector involvement through NFPs and PSIs by requiring private sector partnerships as a condition of providing contribution funding. Further, the program was able to forge long-lasting relationships and build platforms or clusters with the objective of achieving long-lasting involvement and benefits.

6.1.8 Repayable and Non-Repayable Contributions

SODP used a combination of repayable and non-repayable contributions in delivering the program. Both methods were found to work effectively. A combination of the two approaches was best, as it reached a broader number of organizations and provided flexibility.

6.1.9 Performance Measurement

FedDev Ontario implemented a performance measurement system based on the SODP PMS (2012). In-depth data was collected on 20 outputs and outcomes and 38 performance indicators derived primarily from progress reports submitted by funding recipients. When the program was delivered through delivery partners, the partners collected this information from funding recipients and forwarded it to FedDev Ontario. The number of performance indicators applicable to each SOA initiative or stream depended on the initiative's or stream's objectives. Ongoing performance reports were provided for each SOA initiative, updating FedDev Ontario management on the progress of the program. However, performance information on pre-SOA initiatives was limited.

6.1.10 Level of Satisfaction

In both the interviews and the SME survey, most funding recipients indicated that they were "satisfied" or "very satisfied" with FedDev Ontario and its delivery partners. In the SME survey, 84 percent of the 479 recipients that answered the satisfaction question were either "satisfied" or "very satisfied" with their interactions with FedDev Ontario or its delivery partners, and 8 percent were either "dissatisfied" or "very dissatisfied". When the delivery-partner recipients were asked about their level of satisfaction with FedDev Ontario, 81 percent indicated that they were "satisfied" or "very satisfied", and 6 percent were either "dissatisfied" or "very dissatisfied".

Comments regarding FedDev Ontario staff and the initiatives were almost universally positive. Recipients appreciated the staff assistance provided, which was seen as being helpful, professional, supportive and responsive.

6.1.11 Administrative Processes

While the level of satisfaction was high and the delivery process was effective in selecting appropriate projects, funding recipients expressed some concerns about the administrative processes:

6.1.12 Future Potential of Supported Projects

The evaluation found possible impediments to achieving the full potential of supported projects after project completion.

These impediments raise the question of whether some additional longer-term assistance should be provided subsequent to or as part of the initial contribution. The need for future rounds of support for the projects and the likelihood of making that support available should be considered. However, this may not be possible within the scope of a five-year program.

6.1.13 Partial Cost–Benefit Analysis of SODP

The allocative efficiency of SODP was analyzed by conducting a partial cost–benefit analysis of the economic impacts of the program.Footnote 53 The approach consisted of determining the sales attributed to SME projects supported by SODP and then comparing those numbers against the SODP investments.

The estimated sales expected in the first four years after project completion (discussed in Section 5.1) were $1.3 billion, according to SME survey respondents. As described in Section 5.1.7, the total SODP investments in the initiatives in the SME survey were approximately $284 million.Footnote 54 The estimated minimum return ratio on SODP contributions to sales generated in the first four years after completion would be 1:5.

6.2 Cost-effectiveness in Achieving Expected Results

Key Findings: The evaluation did not identify any major changes that need to be made to the program initiatives and their method of delivery. Some respondents commented on improving various aspects of the program. The most-mentioned areas were the need to simplify the application, reporting and claims processes and to speed up approvals. Internal interviewees suggested increasing the percentage of contributions that delivery partners receive, improving the processes, and integrating new tools.

This section addresses the following evaluation question: Is there a more cost-effective way of achieving the expected results, taking into consideration alternative delivery mechanisms, best practices and lesson learned?

Responses from the external interviews and SME survey did not suggest that major changes should be made to the program initiatives or to their method of delivery. Comments were more focused on improving various aspects of the program. The most-mentioned areas were the need to simplify the application, reporting and claims processes and to speed up approvals. These comments pertained to both FedDev Ontario and its delivery partners. In addition, the need for more funding and follow-on funding for commercialization and export support was frequently mentioned. Multiple respondents commented on the need for increased visibility for the program, distribution of ARC funds and increased support for training.

FedDev Ontario interviewees suggested that more third-party delivery be done where it makes sense. Third-party delivery is much more cost-effective for smaller projects, and the delivery partners have the required delivery capability and reach. The proportion of the contribution retained by the delivery partners should be increased to 10 percent from 5 percent to reflect the delivery and administration costs involved. There is also a need to refine some processes and to integrate tools, such as the CANVAS model to help with project proposal assessment, setting service standards for different types of projects, speeding up the processing of applications, and improving project reporting.Footnote 55 Other tools mentioned were a client portal and a system to track interaction with clients.

top of page

7.0 Conclusions and Recommendations

This evaluation was undertaken to determine the extent to which SODP continues to be relevant, is on track to achieve its expected program outcomes, and has demonstrated efficiency and economy.

7.1 Conclusions

The following conclusions are based on the evidence collected in this evaluation.

7.1.1 Program Relevance

There is a continuing need for a program similar to SODP to foster innovation, enhance productivity and competitiveness, and encourage the commercialization of research in southern Ontario. SODP complemented other federal and provincial government funding programs where they were available to recipients.

The need for funding in southern Ontario is substantial. Other programs that were available to recipients were not perceived as redundant, as they had a different emphasis and the demand for funding far exceeded the limited funding available from SODP.

SODP was consistent and fully aligned with government priorities, including FedDev Ontario's PAA and strategic outcome and federal government priorities and strategies, such as the Speech from the Throne (2013), the Budgets for the 2009 to 2014 period, and Canada's Science and Technology Strategy (2007).

7.1.2 Performance: Achievement of Program Outcomes

SODP achieved its expected outcomes by the time the program ended on . The program initiatives met or exceeded almost all of their performance targets. It is expected that projects will take some time following completion to achieve their full impacts.

FedDev Ontario was successful in delivering the SODP initiatives to support individuals and organizations by addressing needs across the business continuum within its mandate. The evaluation determined that SODP delivery models used for the initiatives were efficient and economical in producing outputs and progressing towards expected outcomes. No significant challenges were experienced in delivery, although some adjustments were made along the way.

The SOA initiatives leveraged $2.65 of client funding for each dollar of program funding expended, resulting in total SOA project expenditures of over $1.5 billion based on FedDev Ontario expenditures of $419.7 million. The initiatives resulted in innovations being developed and commercialized and in sales, jobs and full-time employment. When asked In the SME survey to what extent their project had met its objectives, 94 percent of SMEs selected "good" to "excellent". The three People Advantage initiatives (Y-STEM, SEB and GEI) reached more than two million children and youth, funded internships leading to employment, and helped entrepreneurs with their start-ups. The pre-SOA initiatives involved expenditures of $177.4 million. Pre-SOA projects were "very successful" or "successful" in achieving impacts.

In their survey, SMEs estimated three key outcomes for the first four years following project completion: a minimum of $1.85 billion in total sales resulting directly from SODP-supported projects; a minimum of $0.81 billion in export sales; and a minimum of 3,833 FTE positions created.

While a high percentage of SMEs were successful in meeting their project objectives, almost half of the firms that responded in the SME survey faced barriers in exploiting their project achievements. The main barrier was the availability of internal and external funding, followed by a lack of key staff, changes in market, establishing a distribution network, and needing to complete related components.

Cluster projects initiated by TDP and PI–BCA take a long time after completion to reach their full potential, and many will require additional funding to achieve this potential. These large and complex projects were successful in realizing their initial objectives (establishing the clusters and creating the necessary infrastructure) by the end of SODP in . At this point, many clusters have just entered the operating phase and, while committed to sustaining the cluster, are still in the process of ramping up and establishing partnerships with industry to use their facilities and expertise. Additional funding will be needed by clusters to fully exploit their infrastructure, including recapitalization of equipment in future years. Further, some clusters have found that SMEs would like to use the facilities but lack the financial resources to pay for services rendered.

The findings of a Statistics Canada study indicated that SODP-assisted SMEs had better ultimate outcomes (post-funding employment and survivability) than firms that were not assisted. However, as most of the projects were only completed on , the study will need to be repeated in a few years to reassess how project outcomes have contributed to long-term performance. Given the estimated total investment of $284 million in the SMEs included in the survey, it is estimated that the minimum return ratio of SODP contributions to sales generated in the first four years after completion is 1:5 (see Section 5.1.7 for details).

7.1.3 Performance: Efficiency and Economy

The SODP delivery model, which involved delivery mechanisms tailored to each initiative, was efficient and economical in producing outputs and progressing towards expected outcomes. Partnerships and leverage, including third-party delivery, were instrumental in expanding FedDev Ontario's impact. The program was able to leverage its impact as a champion or convener, using its contribution funding as an incentive. The balance between in-house and third-party delivery was instrumental in achieving efficiency and effectiveness. Because of the spending lifecycle of projects, some flexibility in authorities is required if the current flat-lined funding profile is to match project funding needs.

SODP was considered to have the right amount of risk and reward in selecting projects. The combination of repayable and non-repayable contributions in delivering the program was found to be effective, providing flexibility in targeting different organizations.

In a high-level comparison, FedDev Ontario delivery costs for its G&C programs were found to be considerably lower than the costs of four other RDAs delivering similar G&C programs. However, this comparison was undertaken at the RDA level and not at the program level, so programming and contextual factors may explain some of the cost differences.

Overall, funding recipients had a generally high level of satisfaction with FedDev Ontario and with its delivery partners. There were concerns about the amount of time taken in the application, assessment and approval processes, the lack of clarity in the reporting requirements, and the amount of administration required. However, there was recognition that program staff were very helpful and that a certain amount of due diligence is required in dealing with public money.

Third-party delivery involves trusted partners and was found to be both efficient and effective. However, delivery partners expressed concern that the 5 percent of the contribution amount provided for program administration was insufficient. Interview respondents indicated that 10 percent would be more reasonable for future programs, particularly if it resulted in improved reporting, greater promotion to ultimate recipients, increased outreach, and targeted outreach in geographic areas requiring additional support. Ultimate recipients of the funding delivered by third parties were often not aware that the source of the funding was FedDev Ontario.

7.2 Recommendations

On the basis of the evaluation, the following recommendations are made for consideration by FedDev Ontario management.

  1. Administrative processes
    1. Recipient concerns about the administrative processes, including simplifying and speeding up the application, assessment and approval processes and streamlining the reporting process, should be addressed to the extent possible, while meeting the minimum needs of program administration.
    2. Implementing initiative service standards and ICT solutions to improve performance should be considered. Service standards will assist program management in advising applicants of the time required for project approval.
  2. Third-party delivery
    1. Consideration should be given to increasing the 5 percent administration fee provided for third-party delivery, particularly if it results in better performance, such as improved reporting; greater promotion to ultimate recipients; increased outreach; and targeted outreach in geographic areas requiring additional support. The specific percentage of increase should take into account the parameters of the initiative being delivered and the expectations related to delivery.
    2. FedDev Ontario should request that delivery partners increase the prominence of FedDev Ontario as the source of funding in their program communications with ultimate recipients.
    3. FedDev Ontario should establish a database of third-party funding recipients to facilitate the management of the Agency's third-party delivery projects. The database would identify SMEs receiving contributions for multiple projects and provide contact information for research purposes.
  3. Ongoing support
    1. FedDev Ontario management should undertake research and analysis into mechanisms that would provide longer-term support to recipients. Mechanisms may be needed to assist clusters to realize their full potential and become self-sustaining beyond FedDev Ontario's current fixed five-year funding cycle. Additional support would be used to assist potential SME users wishing to use the facilities but lacking the financial resources to do so. Due to the timeframes involved, the five-year life of FedDev Ontario programs may be an impediment to implementing this recommendation.
    2. To aid in future cluster planning, a review of cluster projects should be undertaken after approximately three years of operation to assess their ongoing sustainability, evaluate their success, identify barriers to achieving anticipated results, and document lessons learned.
    3. Consideration should be given to addressing barriers identified by SMEs in the survey that limit their ability to fully exploit their project achievements. This could involve additional assistance, financial or other, to help them address key barriers. If the barriers are known prior to the initial funding support, they should be identified as part of the funding application and taken into account in the funding approval decision.
  4. Program funding
    1. As part of third mandate renewal, FedDev Ontario management should ensure the five-year funding envelope aligns with project funding needs on an annual basis. This would address the difficulty of implementing projects that collectively have funding requirements matching the flat-lined five-year program funding, particularly in the first two years when projects are being assessed, approved and getting underway.
    2. As part of third mandate renewal, FedDev Ontario management should consider requesting continuous program funding. Continuous funding would better meet stakeholder requirements, improve internal efficiency, and facilitate strategic investment planning.
  5. Ultimate outcomes
    1. The Statistics Canada study, Business Performance Measurement of FedDev Ontario Program Beneficiaries, should be repeated in the future to assess how project outcomes will have contributed to the long-term performance of supported SMEs. The timing of the study would be based on at least three years' experience after project completion and the availability of data for the analysis in Statistics Canada databases. To the extent possible, future research should look at each initiative separately or in a group of similar initiatives in comparing assisted and unassisted firms.
    2. In addition, other methods of assessing ultimate outcomes should be considered, to provide additional lines of evidence to complement the Statistics Canada modelling.

top of page

Selected References

top of page

Appendices

Appendix A: Description of SODP Initiatives

This appendix provides a short description of the seven SOA initiatives and the pre-SOA initiatives. Exhibit A.1 provides a description of each initiative: eligible applicant(s), beneficiaries, funding type, and funding available to an applicant.

A.1 Y-STEM: Youth STEM (Science, Technology, Engineering and Mathematics)

The Youth STEM (Y-STEM) initiative had two objectives:

Y-STEM was designed to nurture a love for science, innovation and entrepreneurship among children as young as kindergarten age and to build the next generation of innovators, entrepreneurs and problem solvers. Y-STEM provided support to NFPs for developing and expanding the delivery of programming to youth.

A.2 GEI: Graduate Enterprise Internship

The GEI initiative had the following objectives:

GEI was designed to build a pool of highly skilled workers in southern Ontario by providing business and management experience to graduate students and recent graduates of STEM programs. GEI provided support to NFP and PSIs in southern Ontario to arrange internships with structured mentoring opportunities in the region's SMEs.

A.3 SEB: Scientists and Engineers in Business

The SEB initiative had two objectives:

SEB was designed to provide funding to NFPs and PSIs to help graduates and graduate students in STEM fields improve their business, management and entrepreneurship skills. SEB also provided seed financing to help them commercialize their ideas.

A.4 ARC: Applied Research and Commercialization

The objective of the ARC initiative was to support innovation in SMEs by encouraging greater collaboration and partnerships with PSIs. These partnerships would link PSIs with applied R&D expertise and SMEs with pre-commercialization needs. The SMEs would improve their innovation skills and productivity and learn how to move innovative products and practices to market.

ARC was launched in as a two-year pilot initiative, roughly six months ahead of the other SOA initiatives. Because of high demand from PSIs and the success of the pilot, ARC was extended in and its budget increased by $15.8 million to a total of $30.3 million.

A.5 TDP: Technology Development Program

The TDP initiative had three objectives:

TDP was intended to strengthen and diversify southern Ontario's economy by providing financial support to bring emerging technologies to market more quickly. The program addresses the funding gap between business-driven R&D and commercialization by encouraging the private sector, NFPs and PSIs to collaborate on innovative breakthrough technologies, typically 1–7 years from market. Southern Ontario is home to some of the country's best PSIs and a number of globally competitive SMEs. Having these organizations work together is extremely beneficial for promoting southern Ontario's economy in a global market. Through TDP, the Agency supports highly complex projects that involve various partners. These projects are anticipated to have a great impact in southern Ontario, as well as internationally.

TDP recipient organizations are lead organizations that must be working with at least one private-sector collaborator, and the project must have a minimum of 50 percent financing from nongovernmental sources. Projects need to contribute to the development of a globally competitive, market-ready technology that has the potential to develop opportunities for businesses or an industry sector.

A.6 IBI: Investing in Business Innovation

The IBI initiative has three objectives:

IBI provided funding to

A.7 PI: Prosperity Initiative

The PI initiative had the following objectives:

PI had three funding streams:

A.8 Pre-SOA Initiatives

The initial SODP was designed to introduce new funding into the region during a time of need, with the goal of supporting projects quickly. The initial program supported economic and community development, innovation and economic diversification, with contributions to communities, businesses and non-profit organizations within southern Ontario. Given the need to act quickly coupled with limited capacity within FedDev Ontario, partnerships were formed with third parties to help deliver the program on FedDev Ontario's behalf.

The first intake of SODP (general intake) was launched in , followed by the Food & Beverage Initiative (FBI) intake in , with both closing in late . In addition, to help deliver the program, FedDev Ontario established delivery partnerships with the following:

Exhibit A.1: Description of the Southern Ontario Development Program—Pre-SOA and SOA Initiatives
SODP initiative Objective Eligible applicant(s) Beneficiaries Funding type Funding per applicant
PRE-SOA
GI - General intake
  • Stimulate local economies and enhance the growth and competitiveness of local businesses and communities
    • Improved competitiveness and productivity
    • Increased innovation and commercialization
    • Enhanced economic development and diversification of communities
    • Increased community mobilization and attractiveness
SMEs, NFPs and PSIs Businesses and communities Repayable and non-repayable
  •  Up to $20 million for profit and NFP

For profit:

  • Up to 50% of direct eligible costs
  • Up to 75% of direct eligible non-capital costs

NFP:

  • Up to 90% of direct eligible costs
FBI - Food and Beverage Industry
CME, OCC, YLF - Delivery partners
SOA
People Advantage
Y-STEM - Youth STEM (Science, Technology, Engineering and Mathematics
  • Encourage youth to pursue education and careers in STEM
  • Improve youth's understanding of the business of science
NFPs Youth from kindergarten to grade 12 Non-repayable
  • Up to $2 million
  • Up to 100% of direct eligible costs
GEI - Graduate Enterprise Internship
  • Develop business and management skills in STEM graduate students and recent graduates to complement their technical skills
  • Provide career networking opportunities
  • Build the next generation of potential managers
  • Enable SMEs to benefit from the technical knowledge of STEM graduate students and recent graduates
NFPs
PSIs
SMEs, recent STEM graduates with at minimum advanced level diplomas and students at graduate level or above Non-repayable
  • Up to $5 million
  • Up to 100% of direct eligible costs
  • Up to $10,000 undergrad or college-level grads
  • Up to $15,000 grad-level students or grads
  • Up to 50% of student salaries and benefits
SEB - Scientists and Engineers in Business
  • Develop improved business and management skills of STEM entrepreneurs
  • Improve access to financing and business support services needed to successfully launch and manage start-up SMEs in southern Ontario
NFP Science and Engineers in Business
NFPs
CFDCs
Recent STEM graduates or graduate student entrepreneurs Non-repayable
  • Up to $5 million (max. $35,000 per entrepreneur)
  • Up to 50% of direct eligible costs
PSI Commercialization Fellowships
PSIs Recent STEM graduates, graduate student entrepreneurs and post-doc entrepreneurs Non-repayable
  • Up to $5 million (max. $30,000 per fellowship)
  • Up to 50% of the cost of fellowships
Knowledge Advantage
ARC - Applied Research and Commercialization
  • Support innovation in SMEs by encouraging greater collaboration and partnerships with PSIs
  • Accelerate innovation; improve productivity and competitiveness of SMEs
PSIs SMEs Non-repayable
  • Up to $750,000

Pilot: 100% of direct eligible project costs (PSIs) consisting of up to 66.6% of total project costs up to $50,000; SME must contribute 33.3%

Extension: Same as pilot except up to 50% of total project costs up to $100,000; SME must contribute 50%

TDP - Technology Development Program
  • Bridge the gap between R&D and commercialization of market-driven game-changing technologies
  • Increase collaborations involving private sector, academic and innovation organizations
  • Leverage private sector investment in game-changing technologies
NFPs
PSIs
Private sector Normally non-repayable
  • Up to $20 million
  • Up to 50% of direct eligible costs; must be matched by private sector
Entrepreneurial Advantage
IBI - Investing in Business Innovation
  • Accelerate the commercialization of new products, processes and practices
  • Increase, stimulate and leverage private sector investment
  • Encourage growth of angel investment funds
Start-up SMEs
Start-up businesses (fewer than 50 employees) Start-up businesses Repayable
  • Start-up businesses up to $1 million
  • 1/3 of eligible costs, leveraging 2/3 angel or venture capital funds
NFP angel investor networks
NFP angel investor networks located in southern Ontario NFP angel investor networks Non-repayable
  • NFP angel investor networks: up to $50,000 (one time)
  • Up to 100% of direct eligible project costs
NFPs representing angel investors
NFP organizations representing angel investors NFP organizations representing angel investor networks, angel investors Non-repayable
  • NFP organizations representing angel investor networks: up to $2 million
  • Up to 100% of direct eligible project costs
Prosperity Advantage
PI - Prosperity Initiative Enhance productivity, diversify the regional economy, and build competitiveness in southern Ontario. Has three distinct components:  
  • PE (Productivity Enhancement): address companies' underinvestment in machinery, equipment and software in Ontario relative to that of their counterparts in the United States
NFPs representing a sector or industry SMEs and industry sectors Non-repayable
  • Up to $20 million
  • Up to 90% of direct eligible costs
  • RD (Regional Diversification): focus on regions with high concentrations of manufacturing industries in structural decline and low concentrations of high-growth industries, which could expose local economies to significant job losses if plants were to close
SMEs, regional or community-based NFPs Communities, industries and SMEs Non-repayable and repayable
  • Up to $20 million for profit and NFP

For profit:

  • Up to 50% of direct eligible capital costs
  • Up to 75% of direct eligible non-capital costs

NFP:

  • Up to 90% of direct eligible costs
  • BCA (Building a Competitive Advantage) : emphasize the development or expansion of geographic concentrations of interconnected companies and institutions in a particular field, known as clusters, which can provide a competitive advantage
NFPs, PSIs, SMEs or an existing or newly created economic cluster SMEs, industries and regions Non-repayable and repayable

top of page

Appendix B: SOA Logic Model and Performance Indicators

The logic model developed for SODP SOA is given in Exhibit B.1. The logic model shows the activities that were undertaken in the program; the outputs resulting from those activities; and the immediate, intermediate, and ultimate outcomes each of the initiatives was intended to achieve. The logic model for the pre-SOA initiatives was not as well developed. However, the outcomes were similar to those of the Prosperity Initiative and have been shown in Exhibit 2.5.

Individual logic models for each of the SOA initiatives were also prepared to show how each initiative fits within the larger program. Additional details may be found by referring to FedDev Ontario's SODP Performance Management Strategy (PMS).Footnote 56

In total, there were 20 outputs and outcomes and 38 performance indicators for SODP. The number of performance indicators applicable to each SOA initiative or stream varied, depending on the purpose of the initiative or stream. The performance indicators for outputs and outcomes for each of the SOA initiatives or streams are provided in Exhibit B.2.

Exhibit B.1: SODP SOA Logic Model
Exhibit B.1: SODP SOA Logic Model (the long description is located below the image)

Source:   FedDev Ontario: SODP Performance Measurement Strategy, Performance Results, as of .

Exhibit B.2: Relevancy of SODP SOA Performance Indicators for Each SOA Initiative or Stream
Program outputs and outcomes Performance indicators Y–STEM GEI SEB–NFP SEB–PSI ARC TDP IBI–Angel IBI–SME PI–PE PI–RD PI–BCA
Outputs
1. Education sessions Number of educational outreach sessions for children and youth X
Number of business skills education sessions for STEM entrepreneurs X X
2. Internship programs Number and $ value of Graduate Enterprise Initiative (GEI) contribution agreements X
3. Partnerships or collaborations Number of partnerships or collaborations formed as a result of FedDev Ontario support X X* X* X X X** X** X** X**
4. Investments in cash and in kind leveraged against FedDev Ontario contribution Total value of investments in businesses, communities and organizations leveraged by source (including not-for-profit organizations and other levels of government) (ARC, TDP, GEI, SEB, IBI, PI) X X*** X*** X X X X X X X
5. Businesses supported Number of businesses supported by initiatives (direct recipients, third-party [ultimate] recipients, collaborating businesses in SOA projects, etc.) X X X X X X X
Immediate outcomes
6. Enhanced STEM learning opportunities for children and youth Number of STEM learning activities created, enhanced or expanded (e.g., after-school programs, camps, competitions) X
7. Increased support to commercialize innovations Number of SMEs and start-ups receiving pre-commercialization support (ARC, SEB, IBI, PI–RD, PI–BCA). Note: SEB and IBI indicator will include both funding and business advisory support and mentoring X X X X X X
Number of new products, services, processes or practices developed (ARC, TDP, SEB, IBI, PI–RD, PI–BCA) X X X X X X X
% of leveraged $ directed to R&D (ARC, TDP, IBI, PI–RD, PI–BCA) X X X X X
8. Increased exposure of STEM graduates to business Total number of participants in GEI X
% of GEI participants who complete internships X
Number of entrepreneurs participating in SEB who receive business skills development X X
9. Increased access to capital Number of entrepreneurs accessing commercialization fellowships and total value of fellowships received (SEB PSI stream) X
Number of entrepreneurs accessing start-up financing and total value of financing (SEB NFP stream) X
Total value of new investments attracted to angel networks (IBI) X
Number of SMEs accessing support through IBI and PI and total value of support by source (angel or venture capital, FedDev Ontario, other) (IBI, PI) X X X X
10. Strengthened businesses and clusters Number and geographic location of businesses created, expanded or maintained (IBI, PI–RD, PI–BCA) X X X
Number and geographic location of economic clusters created or expanded (PI–BCA) X
11. Increased adoption or adaptation of new technologies Number of new productivity-enhancing processes, skills, technologies resulting from projects (ARC, PI–PE) X X
Intermediate outcomes
12. Increased participation of children and youth in STEM outreach programs Number of children and youth participating in STEM outreach programs supported by the Agency (Y-STEM) X
Number of children and youth participating in STEM outreach programs supported by the Agency prior to Agency funding (baseline) (Y-STEM) X
13. Improved employment of STEM graduates in the private sector % of GEI interns that have full-time, part-time and contract positions in STEM fields in the private sector following their internship (GEI) X
Number of start-up businesses established following participation in SEB X X
14. Increased commercialization of research Total value of sales resulting from innovations developed (TDP, IBI, PI–RD, PI–BCA) X X X X
Number of processes, practices products, or services commercialized (ARC, TDP, SEB, IBI, PI–RD, PI–BCA) X X X X X X X
15. Improved survival rate of businesses/start-ups % of businesses that continue to be in operation or have had a successful exit (e.g., purchase by another business) (SEB, IBI) X X X
16. Increased employment opportunities in southern Ontario communities Number of person months of employment created or maintained (IBI, PI–RD, PI–BCA) X X X
% of jobs created that are full-time, part-time or contract (IBI, PI–RD, PI–BCA) X X X
Total sales resulting from FedDev Ontario support (IBI, PI- RD, PI–BCA) X X X
17. Enhanced business productivity Total sales per FTE (ARC, PI) X X
Ultimate outcomes
18. Increased innovation capacity in southern Ontario Change in business expenditures on R&D from baseline X X X X X X X X X X X
Change in number of employees in southern Ontario considered "highly qualified personnel" from baseline X X X X X X X X X X X
Change in business investment in machinery and equipment from baseline X X X X X X X X X X X
19. Stronger southern Ontario communities Change in number of southern Ontario businesses by size (SME, MNE) from baseline X X X X X X X X X X X
Change in employment levels by sector from baseline X X X X X X X X X X X
20. More competitive southern Ontario businesses Change in inflation-adjusted gross revenue of southern Ontario businesses by sector from baseline X X X X X X X X X X X
Change in export revenue of southern Ontario businesses from baseline X X X X X X X X X X X

* Number of partnerships and collaborations formed with start-up businesses (NFP stream); number of partnerships and collaborations formed with donors (PSI stream); number of fellows attracted to PSI (PSI stream).
** "New" partnerships or collaborations.
*** Total value of investments in start-up businesses leveraged by source (NFP stream); total value of endowments for commercialization fellowships (PSI stream

top of page

Appendix C: SODP Performance Results

The performance results as of (Exhibits C.1, C.2 and C.3) were reproduced from a , presentation prepared for SODP by FedDev Ontario Program Services.

Exhibit C.1: SODP SOA Initiative Outputs (as of )
SOA initiative Approved contributions ($ millions) Leveraged funding ($ millions) Number of partners or collaborators Number of businesses supported People or STEM learning outputs
Y-STEM 13.7 * * * 29,250 outreach sessions
GEI 17.7 12.3 931 * 10 internship programs
SEB 15.5 16.1 425 425 Various**
ARC 30.3 28.9 579 575 *
TDP 63.9 83.4 109 * *
IBI 60.0 190.1 772 86 *
PI 269.5 783.1 1,288 1,298 *
Total 470.6 1,113.9 4,104 2,384 *

Source: FedDev Ontario: SODP Performance Measurement Strategy – Performance Results (as of ).
* Not relevant or not a specific program output.
** Included workshops, lectures, online training modules, boot camps and one-on-one mentoring.

Exhibit C.2: Results for SODP SOA Immediate Outcomes (as of March 31, 2014)
Outcome
Increased support to commercialize innovations Increased access to capital Strengthened businesses & clusters (IBI, PI–RD, PI–BCA) Increased adoption or adaption of new technologies People or STEM learning outcomes (n)
Indicator
Businesses receiving pre-commercialization support (n) Innovations developed (n) Proportion of leveraged funds directed to R&D (%) Businesses supported (n) / total value of support ($ million) Businesses/clusters created, expanded or maintained (n) Productivity-enhancing innovations as a result of project (n)
Y-STEM * * * * * * 569 learning activities
GEI * * * * * * 1,230 internships
SEB 425 425 * 425 start-ups and commercialization fellowships / accessing $23.9 million * * 425 entrepreneurs receiving businesses skill education
ARC 575 875 96.5 * * 574 *
TDP * 118 81 * * * *
IBI 86 633 100 86 businesses / accessing $246.9 million 86 businesses * *
PI 149 258 * 904 businesses / accessing $605.9 million 493 businesses / 31 cluster (BCA) projects 1,204 *
Total 1.235 2,309 92.5 1,415 businesses / accessing $882.5 million 579 businesses / 31 cluster projects 1,778

Source: FedDev Ontario: SODP Performance Measurement Strategy – Performance Results (as of ).
* Not relevant or not a specific program outcome.

Exhibit C.3: Results for SODP SOA Intermediate Outcomes (as of March 31, 2014)
Outcome
Increased commercialization of research Improved survival rate of businesses and start-ups Increased employment opportunities in southern Ontario Enhanced business productivity People or STEM learning outcomes
Indicator
Innovations commercialized (n) Sales resulting from innovations commercialized ($ million) Businesses still in operation or successful exit after participation in initiative (%) Person–months of employment created and maintained (n); Jobs created** (n) (full-time, part-time, contract)** Businesses/clusters created, expanded or maintained (n)
Y-STEM * * * * * 2.1 million children/youth
GEI * * * * * 71% of internships have led to employment
SEB Not available Not available 98 (418 of 425 projects) * * Of 425 start-ups, 418 are successful and 7 have ceased operation
ARC 142 2.6* * * 50 of 155 supported businesses reported improvement *
TDP 24 8.0 * 6,620 person–months * *
IBI 364 79.3 95 (based on 56 projects) 14,567 person–months
765 jobs (76% FT, 8% PT, 17% contract)
* *
PI 271 96.0 * 38,805 person–months
553 jobs (91% FT, 5% PT, 4% contract)
Not Available *
Total 801 185.9 * 59,992 person–months
1,318 jobs (82% FT, 7% PT, 11% contract)
Not Available *

Source: FedDev Ontario: SODP Performance Measurement Strategy – Performance Results (as of ).
* Not relevant or not a specific program outcome.
** ARC Extension only.
*** Closed projects only.

top of page

Management Response and Action Plan

Management Response and Action Plan—Final Evaluation of the Southern Ontario Development Program
Recommendation Management Response and Action Plan
State "Agreed"/ "Disagreed" and describe course of action
Management accountability
Identify responsible Office/Executive
Action completion date

1. Administrative Processes

  1. Recipient concerns about the administrative processes, including simplifying and speeding up the application, assessment and approval processes and streamlining the reporting process, should be addressed to the extent possible, while meeting the minimum needs of program administration.

Agreed. In advance of the launch of its new suite of programming, FedDev Ontario examined its existing intake, assessment, decision and contracting phases to identify improvements and efficiencies. This resulted in simplified forms and expedited processes, which were received positively by applicants.

DG, Infrastructure & Business Development
DG, Innovation & Community Development
Director, Business Development Programs
Director, Commercialization

Completed in

The Business Innovation and Community Development (BICD) Branch established a Modernizing Our Business (MOB) Working Group, composed of a cross-section of Agency staff, to continuously review and improve administrative processes and practices.

Director, Business Development

Ongoing: MOB Working Group meets bi-weekly to seek ways to improve the efficiency of internal program-level processes

BICD's Program Services is monitoring progress and identifying efficiencies for application assessment and approval timelines.

Vice President, BICD

BICD Placemat Review to be completed by

The Agency is reviewing contribution agreement processes and templates and establishing engagement protocols with the Department of Justice, offering more streamlined services to clients.

Director, Commercialization

  1. Implementing initiative service standards and information and communications technology (ICT) solutions to improve performance should be considered. Service standards will assist program management in advising applicants of the time required for project approval.

Agreed. Prior to the launch of the SOPIs, FedDev Ontario established a Service Charter that was posted on the Agency's website.

Vice President, BICD

Completed

The Agency is developing a revised Service Charter.

Vice President, BICD

FedDev Ontario is partnering with Treasury Board Secretariat and other federal partners to develop an Enterprise Grants and Contributions Solution that will support tracking of service standards.

Chief Financial Officer and Vice President, BICD

Underway. This is a complex project given the number of federal partners involved. A completion date for the system is currently planned for .

2. Third-Party Delivery

  1. Consideration should be given to increasing the 5 percent administration fee provided for third-party delivery, particularly if it results in better performance, such as improved reporting; greater promotion to ultimate recipients; increased outreach; and targeted outreach in geographic areas requiring additional effort. The specific percentage of increase should take into account the parameters of the initiative being delivered and the expectations related to delivery.

Agreed. In recognition of concerns raised by previous recipients, FedDev Ontario adjusted the guidelines for its new suite of programming, allowing for up to 10 percent of third-party delivery project costs to support outreach and project administration.

Vice President, BICD

Completed .

  1. FedDev Ontario should request that delivery partners increase the prominence of FedDev Ontario as the source of funding in their program communications with ultimate recipients.

Agreed. Communications Branch will continue to advise funding recipients, including third-party delivery partners, on federal visibility requirements. The Communications Branch is working with BICD to provide all third-party delivery partners with clearer direction to ensure FedDev Ontario receives adequate visibility for all investments and that third-party recipients comply with the Government of Canada Communication Policy.

Director General, Communications

  1. FedDev Ontario should establish a database of third-party funding recipients to facilitate the management of the Agency's third-party delivery projects. The database would identify SMEs receiving contributions for multiple projects and provide contact information for research purposes.

Agreed. The Agency has developed a database of direct and ultimate recipients of FedDev Ontario support through the Southern Ontario Prosperity Initiatives, Advanced Manufacturing Fund, and Eastern Ontario Development Program. The database indicates whether recipients have received support under multiple programs and whether any financial adjustments have been made. The Communications Branch is also developing success stories of selected program recipients to profile publicly.

Vice President, BICD

FedDev Ontario is partnering with Treasury Board Secretariat and other federal partners to develop an Enterprise Grants and Contributions Solution that will support a systematic approach for maintaining a database of clients and ultimate beneficiaries.

Chief Financial Officer and Vice President, BICD Underway. This is a complex project given the number of federal partners involved. A completion date for the system is currently planned for .

3. Ongoing Support

  1. FedDev Ontario management should undertake research into and analysis of mechanisms providing longer term support to recipients. Additional mechanisms are needed to assist clusters requiring additional time beyond FedDev Ontario's current fixed five-year cycle to realize their full potential and become self-sustaining. This additional support would be used to assist potential SME users wishing to use the facilities but lacking the financial resources to do so. Due to the timeframes involved, the five-year lifecycle of the program may be an impediment to implementing this recommendation.

Agreed in principle. As part of renewal discussions, the Agency will undertake research and analysis considering the extent to which programming gaps exist and the feasibility of longer term support. Longer term funding will be at the discretion of the government.

The Agency will continue to identify programming options that may be appropriate for the client's situation. For example, Canada Business Ontario (CBO) and FedDev Ontario program officers support businesses and not-for-profit organizations by identifying other government programming consistent with the Agency's role as a collaborator and convener.

Vice President, PPPM (lead) and Vice President, BICD

Research and analysis will take place between 2015/16 and 2017/18.

  1. To aid in future cluster planning, a review of cluster projects should be undertaken after approximately three years of operation to assess their ongoing sustainability, evaluate their success, identify barriers to achieving anticipated results, and document lessons learned.

Agreed. A review of cluster projects will be included in the methodology for the SOPI interim evaluation. It will be considered a separate line of evidence. The Evaluation Directorate will consult with BICD to ensure that contact information and performance data are available.

Senior Manager, Evaluation (lead) Directorate &
Director, Business Development Programs &
Director of Commercialization

SOPI Interim Evaluation has been identified in the Evaluation Plan for 2015/16 to 2019/20.

  1. Consideration should be given to addressing barriers identified by SMEs in the survey that limit their ability to fully exploit their project achievements. This could involve additional assistance, financial or other, to help them address key barriers. If the barriers are known prior to the initial funding support, they should be identified as part of the funding application and taken into account in the funding approval decision.

Agreed. In preparing FedDev Ontario's new suite of SOPIs, the Agency took into consideration issues identified by SMEs in the SODP evaluation survey. For example, the renewed Investing in Business Innovation initiative provides support to new entrepreneurs in the form of seed financing, as well as training in business and market development skills. The initiative supports early-stage enterprises with contributions toward business development, including building high-performance teams; adopting best management practices, processes and systems; and improving governance and investment readiness. It also supports late-stage product development geared toward market diversification and growth, customer and market development, and marketing and distribution strategies.

FedDev Ontario continues to engage with SODP and SOPI clients to proactively identify barriers and offer assistance. Engagement activities include implementation of project monitoring plans, updating of project risk profiles, and identification of program pathways and approaches.

Results from FedDev Ontario's current suite of programs will in turn be used to refine future programming.

DG, Infrastructure & Business Development &
DG, Innovation & Community Development &
Director, Business Development Programs &
Director of Commercialization

Completed for current suite of initiatives in November 2013.

Ongoing cycle of improvement required for new programs and initiatives.

4. Program Funding

  1. As part of third mandate renewal, FedDev Ontario management should pursue authorities to profile program funding within the five-year funding envelope to match project funding needs. This re-profiling addresses the difficulty of implementing projects that collectively have funding requirements matching the flat-lined five-year program funding, particularly in the first two years when projects are being assessed, approved and getting underway.

Agreed. The Agency will continue the dialogue with the central agencies to profile program funding to match project funding needs. The program funding profile will also be examined during the discussions on Agency and program renewal.

Vice President, PPPM (lead)
Vice President, BICD
Chief Financial Officer

2015/2016 to 2017/18
In advance of Agency and program renewal, PPPM will proactively engage with stakeholders to ensure the development of an optimal program funding profile for the Agency's third mandate.

Initial engagement to begin in 2016.

  1. As part of third mandate renewal, FedDev Ontario management should consider requesting continuous program funding. Continuous funding would better meet stakeholder requirements, improve internal efficiency, and facilitate strategic investment planning.

Agreed. Continuous program funding will be considered an option during the discussions on Agency renewal.

Vice President, PPPM

2015/2016 to 2017/18
In advance of Agency and program renewal, PPPM will proactively engage with relevant federal departments and agencies to assist in developing a business case for continuous program funding.

5. Ultimate Outcomes

  1. The Statistics Canada study Business Performance Measurement of FedDev Ontario Program Beneficiaries should be repeated in the future to assess how project outcomes will have contributed to the long-term performance of supported SMEs. The timing of the study would be based on at least three years' experience after project completion and the availability of data for the analysis in Statistics Canada databases. To the extent possible, future research should look at each initiative separately or in a group of similar initiatives in comparing assisted and unassisted firms.

Agreed. The Evaluation Directorate is working with the Strategic Policy Directorate on the Cross-Agency Demonstrating Impact project, looking at various methodologies to tell the impact story of FedDev Ontario. A follow-up to the Statistics Canada project is under consideration as part of the renewal story to be developed over the coming years.

Senior Manager, Evaluation Directorate

Strategic Policy is leading a working group to identify and implement strategies to strengthen the Agency's ability to demonstrate its impact on the southern Ontario economy. The working group includes members from BICD and the Evaluation Directorate.

As part of the overall work plan, FedDev Ontario will work closely with Statistics Canada and target completed results of the matching analysis by early 2017.

  1. In addition, other methods of assessing ultimate outcomes should be considered, to provide additional lines of evidence to complement the Statistics Canada modelling.

Agreed. Strategic Policy, jointly with other FedDev Ontario branches, will be exploring and identifying data requirements, methods and techniques, such as input–output analysis, that can be used to provide evidence for Agency impact in the southern Ontario economy across key metrics.

Director General, Strategic Policy and Evaluation

Strategic Policy is leading the Demonstrating Impact project that will consist of these stages:

2015/16—initiate a review of data availability and additional coding opportunities and implement required processes and coding activities.

2016/17—explore and undertake, if required, further follow-on surveys of recipients (particularly consortia-related projects) and the impact of investments for those regions and businesses.

2016/17—contract Statistics Canada to undertake matching analysis of FedDev Ontario recipients vs. non-recipients.

2016/17—explore the potential for undertaking input–output and growth accounting modelling of FedDev Ontario investments.

2016/17 and 2017/18—develop an integrated impact story.

top of page

Date modified: