Evaluation of the Eastern Ontario Development Program (EODP)

Final report
Prepared by Evaluation Directorate
Federal Economic Development Agency for Southern Ontario
April 2016

Table of Contents

List of Acronyms

BD
Business Development (business line)
CA
Contribution Agreement
CFDC
Community Futures Development Corporation
CFP
Community Futures Program
CI
Community Innovation (business line)
CP
Collaborative Projects (business line)
EAC
Evaluation Advisory Committee
EOCFDC
Eastern Ontario CFDC Network
EODF
Eastern Ontario Development Fund
EODP
Eastern Ontario Development Program
FedNor
Federal Economic Development Initiative for Northern Ontario
Gs&Cs
Grants and Contributions
GoC
Government of Canada
ICT
Information and Communications Technology
KI
Key Informant
MoSO
Minister of State’s Office
NFP
Not-for-Profit
PAA
Program Alignment Architecture
PMM
Performance Measurement Matrix
PMS
Performance Measurement Strategy
PSI
Post-secondary Institution
R&D
Research and Development
SD&G
United Counties of Stormont, Dundas and Glengarry
SODP
Southern Ontario Development Program
SOPI
Southern Ontario Prosperity Initiatives
SOPP
Southern Ontario Prosperity Program
SPD
FedDev Ontario’s Strategic Policy Branch
TBS
Treasury Board Secretariat

EVALUATION SUMMARY OF THE EASTERN ONTARIO DEVELOPMENT PROGRAM (EODP) – Infographic

PDF Version

EODP Promotes:

Evaluation Period: April 1, 2011 to March 31, 2014

What the Evaluation Found

Success in Achieving Results and Exceeding Targets

  • Undertook 2,050 projects
  • Leveraged $73 million in additional funding ($2.71 for every dollar invested)
  • Assisted 8,452 business
  • Provided 3,113 workshops, training and counselling sessions for 12,528 employees

Success in:

  • Building partnerships
  • Supporting businesses and communities
  • Creating and maintaining jobs
  • Contributing to diversification and competitiveness

Performance

  • Community Futures Development Corporations (CFDCs) delivering EODP works.
  • Each CFDC has knowledge of the community’s needs and can respond to local priorities.

Recommendations

  1. Plan collaborative projects, co-managed by more than one CFDC;
  2. Ensure planning studies can be implemented;
  3. Assess relative value of small vs. large projects;
  4. Review CFDC funding approach;
  5. Encourage CFDCs to share and leverage best practices and tools;
  6. Use 2016 Census information to study socioeconomic situation in eastern Ontario;
  7. Assess longer-term benefits for ultimate funding recipients; and
  8. Assess EODP’s impact on FedDev Ontario’s ultimate outcomes.

For More Information

For more information, visit FedDev Ontario’s website for a complete list of evaluation studies, or contact FDO.Evaluation-Evaluation.FDO@feddevontario.gc.ca.

Executive Summary

An evaluation of the Eastern Ontario Development Program (EODP) was undertaken by the Federal Economic Development Agency of Southern Ontario (FedDev Ontario). EODP is a federal government grants and contributions program designed to promote socioeconomic development in rural eastern Ontario, leading to a competitive and diversified regional economy and contributing to the successful development of business and job opportunities, as well as sustainable, self-reliant communities.

The objective of the EODP evaluation was to determine the extent to which the EODP, in the three-year period from 2011–12 to 2013–14, was relevant, achieved its expected program outcomes, and demonstrated efficiency and economy. The evaluation was based on the Treasury Board Secretariat’s Policy on Evaluation (2009) and it’s accompanying Directive on the Evaluation Function, with a focus on the five core evaluation issues relating to relevance and performance.

In Budget 2013, FedDev Ontario's funding was renewed for a second five-year mandate (2014–15 to 2018–19) and included a five-year extension of EODP beginning April 1, 2014. The EODP extension was included in FedDev Ontario's Southern Ontario Prosperity Program (SOPP) Terms and Conditions as part of the Southern Ontario Prosperity Initiatives (SOPI), with SOPI evaluations planned as part of FedDev Ontario's Evaluation Plan for 2014–15 to 2019–20. Given that the updated EODP, which came into effect as part of the SOPP on April 1, 2014, will be evaluated in conjunction with other SOPP initiatives, the scope of this evaluation was primarily on the implementation of EODP during the three-year period from 2011–12 to 2013–14. As a result, this evaluation was calibrated to balance the need for evaluation coverage with effective stewardship of government resources.Footnote 01, Footnote 02

The methodology used to address the five core evaluation issues in this calibrated evaluation of EODP involved the following data collection methods:

Overall, the evaluation methodology provided evidence that allowed FedDev Ontario to reach conclusions for all five core issues. However, there were limitations to the evaluation methodology. These limitations, described in the evaluation report, were taken into account during analysis and were recognized in the interpretation of the findings.

The evaluation was carried out by FedDev Ontario’s Evaluation Directorate during the period from October 2015 to January 2016. An Evaluation Advisory Committee (EAC) was established to provide advice to the evaluation project team at key points during the evaluation.

Profile of EODP

EODP has been an ongoing program since 2004 and was administered by Federal Economic Development Agency for Northern Ontario (FedNor) until 2009. Originally known as the Eastern Ontario Development Fund, it was launched as a one-year pilot program in response to program funding gaps and economic concerns expressed by regional stakeholders. The program was renamed EODP in 2006 and has been operating consistently since the original pilot. The current iteration of the program is scheduled to operate from 2014 to 2019.

FedDev Ontario assumed the overall management of EODP upon the Agency’s inception in 2009, and the 15 Community Futures Development Corporations (CFDCs) located across rural eastern Ontario have been responsible for third-party program delivery. CFDCs are independent, community‐based, legally incorporated not‐for‐profit (NFP) community development organizations. Each CFDC is managed by a board of directors made up of approximately 12 volunteers from the respective community. During 2011/12-2013/14, there were 37Footnote 03 CFDCs covering southern Ontario, of which 15 cover rural eastern Ontario.

During the 2011–12 to 2013–14 period, the CFDCs provided EODP grants and contributions (Gs&Cs) funding to eligible recipients, to support projects in the following three business lines:

Each CFDC developed a business plan and application to obtain EODP funding. In preparing their business plans, each CFDC consulted groups in their community, including county councils, municipalities, economic development and business groups, local politicians, provincial and other funding partners, and NFP organizations, among others. These consultations helped the CFDC to identify priorities that best responded to the needs of the community, within the program’s scope and activities as defined by FedDev Ontario. The business plan included the CFDC’s plans and targets for each of the BD, CI and CP business lines.

After reviewing the CFDCs’ business plans and applications, FedDev Ontario approved funding and signed a contribution agreement (CA) with each CFDC. The CFDCs then solicited applications for project funding in their communities, evaluated the applications, and decided which projects to fund. CFDCs put in place a project-level CA for each approved project, monitored the project’s progress, and approved and processed financial claims.

Key Findings

Key evaluation findings are provided below for relevance, achievement of program outcomes, and efficiency and economy.

Program Relevance

The evaluation of program relevance was calibrated, given that the relevance of EODP had been well established in multiple program evaluations and that the program was renewed for the period starting April 1, 2014, and ending March 31, 2019.

Regarding EODP (2011–14),Footnote 04 interview respondents were generally very pleased with EODP and were very supportive of continuing the program, as it stimulates investment and plays a key role in the community. Further, respondents felt that while EODP resources were small in comparison to the size of the rural eastern Ontario economy, the program was extremely effective and was able to have an impact by helping to address major challenges and encouraging business and community development.

EODP was found to complement other federal and provincial government funding programs available to EODP project funding recipients. With their different emphases and funding, those programs served to fill the gap between the demand for funding in eastern Ontario and the limited funding available from EODP.

EODP (2011–14) was found to be consistent and fully aligned with government priorities, including FedDev Ontario’s Program Alignment Architecture and strategic outcome and federal government priorities and strategies, including the 2010 and 2011 federal budgets. EODP was also fully aligned with federal roles and responsibilities.

Program Performance—Achievement of Program Outcomes

EODP (2011–14) was generally successful in achieving results and exceeded almost all targets established for the program. With only a few exceptions, respondents reported that the BD and CI business lines were very successful. However, the CP business line, which involved investments from two or more CFDCs and benefited one or more communities, had mixed results, with some respondents indicating that it had been successful and others pointing to significant challenges.

Over the 2011–14 period, 2,050 EODP projects were undertaken: 1,387 (68 percent) were BD; 623 (30 percent) were CI; and 40 (2 percent) were CP. The EODP funding contributions of $27 million resulted in $73 million in additional funding leveraged from other sources, for total project expenditures of $100 million. Every dollar invested in EODP resulted in $2.71 in leveraged funds. This result was much greater than the targeted leverage of $1.68 per dollar invested, identified in CFDC business plans.

EODP helped create new businesses and assisted existing businesses that were struggling. In total, 8,452 businesses were assisted: 289 (3 percent) were created, 1,207 (14 percent) were expanded, and 6,956 (82 percent) were maintained. EODP funding resulted in 2,687 jobs being created and 14,646 jobs being maintained, for a total of 17,333 jobs through the supported projects. The majority of the job impacts were through the BD projects, where 11,023 (64 percent) of the total 17,333 were realized. The final result of 17,333 jobs created or maintained substantially exceeded the target of 4,500, which included an allowance for missing data. Details of EODP performance results are shown in the following table.

Performance Results for EODP (2011–2014)

Projects
Results Business Development (BD) Community Innovation (CI) Collaborative Projects (CP) Total Program
Number 1,387 623 40 2,050
Funding
Results Business Development (BD) Community Innovation (CI) Collaborative Projects (CP) Total Program
EODP funding contribution* $14,279,989 $10,541,318 $2,187,027 $27,008,333
Leverage funding $42,522,720 $25,954,056 $4,754,186 $73,230,962
Total project funding $56,802,709 $36,495,373 $6,941,213 $100,239,295
Leverage ratio (leverage / EODP contribution) 2.98 2.46 2.17 2.71
Businesses
Results Business Development (BD) Community Innovation (CI) Collaborative Projects (CP) Total Program
Created 150 139 0 289
Expanded 805 384 18 1,207
Maintained 2,718 3,390 848 6,956
Total businesses 3,673 3,913 866 8,452
New markets and opportunities 1,352 1,482 229 3,063
Improved decision-making skills 1,308 1,451 293 3,052
Jobs
Results Business Development (BD) Community Innovation (CI) Collaborative Projects (CP) Total Program
Created 1,638 1,007 43 2,687
Maintained 9,386 4,177 1,083 14,646
Total jobs 11,023 5,184 1,126 17,333
Workshops, training and counselling
Results Business Development (BD) Community Innovation (CI) Collaborative Projects (CP) Total Program
Number of sessions 2,162 856 95 3,113
Trainees supported (including youth) 5,225 6,253 1,050 12,528
Youth trainees 3,551 3,818 59 7,428
Community economic development
Results Business Development (BD) Community Innovation (CI) Collaborative Projects (CP) Total Program
New economic sectors active 309 196 4 509
New staff & volunteers in economic development 641 1,254 36 1,931
Other areas
Results Business Development (BD) Community Innovation (CI) Collaborative Projects (CP) Total Program
Number of non-CFDC partnerships 1,147 1,725 215 3,087
Intern placements 297 20 7 324
Trade shows attended 239 55 10 304
Plans developed** 285 194 14 493

* The total EODP funding contribution for projects is 2.3 percent higher than the final expenditures shown in Exhibit 2.3 because final adjustments were made to expenditure figures after the program was completed.

** Business plans (activities) developed by CFDCs and their partners. The Contribution Agreements stipulate the development of such plans.

As an additional result of the EODP funding, 3,113 workshops, training and counselling sessions were undertaken in support of 12,528 trainees, including 7,428 young trainees. EODP funding also resulted in 509 new economic sectors becoming active across the 15 CFDC communities, and 1,931 new staff and volunteers becoming involved in community organizations dedicated to community economic development. EODP assistance resulted in an additional 3,087 non-CFDC partnerships, the placement of 324 interns, attendance at 304 trade shows, and the development of 493 plans (e.g., strategic plans, feasibility studies, business plans, community plans).

The performance information above indicates that EODP (2011–14) was successful in building partnerships, supporting businesses and communities, creating and maintaining jobs, and contributing to diversification and competitiveness.

Interview respondents viewed the CFDCs as being effective in contributing to rural economies and communities by supporting small business development; leveraging funding from other programs; and working with partners to promote economic stability, growth and diversification. Moreover, several respondents noted that given EODP’s limited budget, the program has been extremely effective in achieving results. While these results can be seen at the community level, it is difficult to assess the program’s impact on eastern Ontario given the size of the eastern Ontario economy.

Further, when interview respondents were asked if EODP had produced any unintended outcomes, the following comments were noted:

In terms of challenges, interview respondents noted that CP initiatives were not as successful as they could have been; and the late start of EODP (2011–14) created some difficulties for the CFDCs, such as a number of NFPs not being able to implement some of their planned activities.

Program Performance—Efficiency and Economy

The EODP (2011–14) delivery model was found to be efficient and economical in producing outputs and progressing toward expected outcomes. The delivery model was considered appropriate, as it provides each CFDC, which has knowledge of the community’s needs, with the ability to respond to local priorities.

The demand for EODP funding was much higher than available funding, and program staff had a range of qualified applicants from which to select. Partnerships and leverage, including collaborating with the CFDCs as third-party delivery agents, were found to be instrumental in expanding FedDev Ontario's impact. The requirements that supported projects had to be completed by the end of the fiscal year (without the flexibility of carry-over from one year to the next) created problems and inefficiencies for the CFDCs. Further, it was noted that administrative processes and templates had been developed by each CFDC independently, and there was considerable variability among the CFDCs.

The third-party delivery model was found to be both efficient and effective. The cost of delivering an EODP dollar was found to compare favourably with the overall delivery cost of Gs&Cs programs by FedDev Ontario and by other regional development agencies. CFDC respondents reported a good working relationship with the FedDev Ontario project officers and were very satisfied with the services received. The delays in starting the program caused difficulties for the CFDCs (as CFDCs did require some time to develop and integrate new materials into their EODP roll-out plans and packages); however, they were successful in delivering the program.

There was a consensus among interview respondents that FedDev Ontario’s use of CFDCs to deliver EODP was the most cost-effective approach to achieving program results. It was further observed that there is no better, more cost-effective group of organizations that are lean, well established and fully connected in the community.

Recommendations

On the basis of the evaluation findings, the following recommendations are made for consideration by FedDev Ontario management:

Program Design

1. It is recommended that FedDev Ontario program management undertake the following:

  1. Investigate whether there is a better approach to planning and managing Collaborative Projects (i.e., regional projects), which involve funding from two or more CFDCs and benefit one or more communities, to address the concerns identified in this evaluation.
  2. Examine approaches to reduce the number of NFP planning studies for projects that do not have available funds to implement them.
  3. Assess the relative value and balance of small projects versus large projects, given the decision to encourage larger, more strategic projects in 2013–14 and the fact that most businesses in eastern Ontario are small (micro) businesses.
  4. Review the funding approach where each CFDC receiving equal contribution funding to determine whether funding should be allocated on a different basis that reflects the needs and priorities of the geographic areas covered.

Third-Party Delivery

2. It is recommended that FedDev Ontario program management undertake the following:

  1. Revise the funding approach in which supported projects must be completed by the end of the fiscal year that they start in and allow multiyear projects.
  2. Encourage the CFDCs to benefit from the various approaches (e.g., processes, templates, software) developed and used by the CFDCs across eastern Ontario for EODP management and administration (promotion, application processing, project evaluation and selection, etc.) by adopting the best practices and lessons learned, with the objective of improving efficiency. The Eastern Ontario CFDC Network could potentially lead this initiative.

Economic Research

3. It is recommended that FedDev Ontario management undertake a study of the socioeconomic situation in eastern Ontario to help determine the continuing need for a program similar to EODP as input to future program renewal. This study should be carried out after the 2016 Census information becomes available. This study could also include determining if there is merit in extending the program to areas with similar economic needs in southwestern Ontario.

Intermediate and Ultimate Outcomes

4. It is recommended that FedDev Ontario management include the following methodologies in future evaluations of EODP:

  1. Assess the longer-term benefits to the project funding recipients. Also, investigate the role that EODP funding has in launching the supported projects; and determine the project funding recipients’ level of satisfaction and feedback in relation to the delivery of the program. This would likely include a survey of project funding recipients, interviews with CFDCs and project funding recipients, and case studies of supported projects.
  2. Assess the program’s impact on the ultimate outcomes, which would involve undertaking special analyses using Statistics Canada data, subject to data availability.

1.0 Introduction

This report documents the evaluation of the Eastern Ontario Development Program (EODP) undertaken for the Federal Economic Development Agency of Southern Ontario (FedDev Ontario). This evaluation was undertaken to examine EODP’s program relevance and performance during the three-year period from 2011–12 to 2013–14.

EODP is a federal government grants and contributions program delivered by FedDev Ontario. EODP was designed to promote socioeconomic development in rural eastern Ontario, leading to a competitive and diversified regional economy and contributing to the successful development of business and job opportunities and sustainable, self-reliant communities.

FedDev Ontario is the federal government’s regional development agency responsible for southern Ontario. FedDev Ontario is headquartered in Waterloo and has offices in Toronto, Ottawa and Peterborough. EODP is managed out of the Peterborough office.

The evaluation was undertaken to meet the requirement of the Treasury Board Secretariat (TBS) Policy on Evaluation (2009) and to satisfy Financial Administration Act Section 42.1, which states that "every department shall conduct a review every five years of the relevance and effectiveness of each ongoing program for which it is responsible."Footnote 05 Previous evaluations of the EODP were undertaken in 2005, 2007 and 2011. The last evaluation was approved in March 2011, which requires this evaluation to be completed by March 2016.

The evaluation was carried out by FedDev Ontario’s Evaluation Directorate during the period from October 2015 to January 2016. An Evaluation Advisory Committee (EAC) was established to provide advice to the evaluation project team at key points during the evaluation.

1.1 Objectives and Scope of the EODP Evaluation

The objective of the EODP evaluation was to determine the extent to which the EODP in the 2011–12 to 2013–14 period was relevant, achieved or was on track to achieve its expected program outcomes, and demonstrated efficiency and economy. The evaluation was based on the TBS Policy on Evaluation and its accompanying Directive on the Evaluation Function (2009),Footnote 06 with a focus on five core evaluation issues relating to relevance and performance (effectiveness, efficiency and economy). For each of the five core evaluation issues, key evaluation questions were developed. The evaluation methodology is described in Section 3.0.

In Budget 2014, FedDev Ontario’s mandate was renewed for a further five years, and the renewal included a five-year extension of EODP beginning April 1, 2014. The EODP extension was included in the terms and conditions of FedDev Ontario’s Southern Ontario Prosperity Program as part of the Southern Ontario Prosperity Initiatives (SOPI) and is therefore included as part of the SOPI evaluation schedule. Significant changes were made to EODP at the time of its renewal, effective April 2014. Recognizing the above, the evaluation is focused on the three-year period from April 1, 2011, to March 31, 2014, and has been calibrated to balance the need for evaluation coverage and the responsibility to ensure effective stewardship of government resources.

1.2 Outline of this Report

This report contains a profile of EODP, the logic model and performance measurement strategy, and financial summary (Section 2.0); a description of the evaluation methodology (Section 3.0); findings on relevance (Section 4.0); findings on performance (Section 5.0); findings on efficiency and economy (Section 6.0); and conclusions and recommendations (Section 7.0).

2.0 EODP Profile

2.1 EODP Objective

The EODP’s objective is to promote socioeconomic development in rural eastern Ontario as a means to foster a competitive and diversified regional economy and in turn, the successful development of new business and employment opportunities in communities that are both sustainable and self-reliant.Footnote 07

2.2 Background

In 2004, the Eastern Ontario Development Fund (EODF) was started as a one-year pilot program in response to program funding gaps and economic concerns expressed by regional stakeholders. The program has been continued since that time, with the name changing to EODP in 2006. EODP, as noted earlier, was most recently renewed for the 2014–19 period.

The following provides some highlights of EODP since its inception in 2004, EODF/EODP (2004–09)—The EODF program was announced with $10 million in funding for 2004–05, which was followed by successive annual renewals of $10 million. The name of the program was changed to EODP in July 2006. The program was delivered by the Federal Economic Development Initiative for Northern Ontario (FedNor) within Industry Canada until August 2009.

2.3 Previous Evaluations

In the period since its inception in 2004, EODP has been evaluated three times:

In each evaluation, the program was deemed successful, and the use of the CFDCs as a primary delivery partner was seen as a major contributor to that success.

The March 2011 EODP evaluation provided recommendations on the design and delivery of the program, as well as the outcome performance data, collection tools and the use of performance information for the management of EODP. A summary of the recommendations in the 2011 evaluation and the Management Response and Action Plan are provided in Appendix A. This appendix also provides a description of the actions that have been taken by management and indicates that all recommendations have been addressed.

2.4 The EODP, 2011–12 to 2013–14

This subsection describes the EODP (2011–14) program, including how it was delivered through the Community Futures Development Corporations (CFDCs) to the project funding recipients; program management; and a breakdown of the projects funded.

2.4.1 Program Description

EODP’s three business lines, eligible project funding recipients, and funding eligibility are described below.

Business Lines

EODP (2011-14) provided grants and contributions support for projects in the following three business lines:

Detailed descriptions of eligible activities for projects contained in the signed contribution agreements (CAs) between FedDev Ontario and each CFDC are provided in Appendix B.

Eligible Project Funding Recipients

Recipients eligible for EODP project funding through the CFDCs were the following:

Funding Eligibility

Eligibility for EODP project funding was as follows:

2.4.2 Program Delivery

FedDev Ontario was responsible for the overall management of EODP during 2011–12 to 2013–14 period. The program was delivered through 15 CFDCs located across rural eastern Ontario.

CFDCs are independent, community‐based, legally incorporated NFP community development organizations that were established to deliver the Community Futures Program (CFP) in a designated service area.Footnote 08 All CFDCs have corporate bylaws and have developed policies and procedures for administration of their general operations, personnel, and investment fund, as well as conflict of interest guidelines. Each CFDC is run by a board of directors made up of approximately 12 volunteers from the community. During 2011/12-2013/14, there were 37Footnote 09 CFDCs covering southern Ontario, of which 15 cover rural eastern Ontario.

The 15 eastern Ontario CFDCs deliver EODP as part of their program delivery. These CFDCs are the listed below:Footnote 10

Exhibit 2.1 shows rural eastern Ontario in yellow, with the geographic areas covered by each of the 15 CFDCs. It should be noted that rural eastern Ontario excludes the cities of Ottawa and Kingston.Footnote 11

Exhibit 2.1: Eastern Ontario and CFDC Delivery Areas

Exhibit 2.1: Eastern Ontario and CFDC Delivery Areas

Exhibit 2.1: Eastern Ontario and CFDC Delivery Areas (the long description is located below the image)
Description of Exhibit 2.1: Eastern Ontario and CFDC Delivery Areas

A map of Eastern Ontario showing the CFDC areas of the region highlighted in yellow including: 1000 Islands CFDC; CFDC of North & Central Hastings and South Algonquin; Community Futures Peterborough; Cornwall & The Counties CFDC; Frontenac CFDC; Greater Peterborough BDC; Grenville CFDC; Haliburton County Development Corporation; Kawartha Lakes CFDC; Northumberland CFDC; Prescott-Russel CDC; Prince Edward/Lennox & Addington CFDC; Renfrew County CFDC; South Lake CFDC; Stormont, Dundas & Glengarry CFDC; Trenval Business Development Corporation; Quinte Business Development Centre; and Valley Heartland CFDC.

2.4.3 Program Management

The 15 CFDCs in eastern Ontario applied to FedDev Ontario for EODP (2011–14) program funding. FedDev Ontario approved the applications and signed CAs with the CFDCs setting out terms and conditions for third-party delivery. Through the CA, FedDev Ontario was responsible for the overall management of EODP, while the CFDCs were responsible for delivery of the program. FedDev Ontario’s EODP program officers worked with CFDCs to ensure locally based projects were effectively administered.

Each CFDC developed a business plan and application to obtain EODP funding. In preparing their business plans, each CFDC consulted with groups in their community to identify the priorities that best responded to the needs of the community.Footnote 12 The priorities were required to align with the program’s scope and activities as defined by FedDev Ontario.Footnote 13 The business plan included the CFDC’s plans and targets for each of the BD, CI and CP business lines.

FedDev Ontario reviewed the business plans and applications and approved funding, and then signed a CA with each CFDC. The CFDCs solicited applications for project funding in their communities, evaluated the applications and decided which projects to fund. CFDCs required a project CA for each approved project, monitored the project’s progress, and approved and processed financial claims. CFDCs were required to submit quarterly reports to FedDev Ontario describing activities undertaken, outcomes and impacts of the projects funded, as well as one success story. In addition, an annual performance report was required.

A more detailed description of FedDev Ontario’s and the CFDCs’ respective roles and responsibilities for delivering EODP (2011–14) is provided in Appendix C.

2.4.4 EODP-Funded Projects

During EODP (2011–14), the 15 CFDCs approved 2,050 projects for assistance from the 6,714 enquiries and 2,683 applications received. These figures indicate that, on average, 3.3 enquiries and 1.3 applications were received for each approved project. The total funding contribution for the 2,050 projects was $27.0 million, as shown in Exhibit 2.2.

Exhibit 2.2 also shows a breakdown of the EODP projects for the three business lines, with BD representing 1,387 (67.7 percent) of the total projects, CI representing 623 (30.4 percent) projects, and CP representing 40 (2.0 percent) of the total projects. In terms of EODP approved project funding, BD accounted for $14.3 million (52.9 percent), CI for $10.5 million (39.0 percent), and CP for $2.2 million (8.1 percent). In addition, the table provides a breakdown of the BD and CP business lines by their initiatives.

As the CP business line was made up of BD and CI projects, a separate table has been provided at the bottom of Exhibit 2.2 to show the summary figures for these two projects.

The exhibit also shows the average amount of funding for each of the business lines and their initiatives. These figures show that, on average, BD projects received $10,296 in EODP funding, while CI and CP received $16,920 and $54,676, respectively.

Exhibit 2.2: EODP Projects, 2011–12 to 2013–14

Project Breakdown (2011–14) Projects Supported by EODP Total EODP Project
Funding Support
Average Funding per Project
Number Percent Dollars Percent Dollars
Business Development 1,387 67.7% $14,279,989 52.9% $10,296
Skills Development 344 16.8% $1,651,983 6.1% $4,802
Youth Retention and Attraction 310 15.1% $4,518,219 16.7% $14,575
Innovation 90 4.4% $1,717,161 6.4% $19,080
ICT 224 10.9% $1,877,298 7.0% $8,381
Other Business Development Projects 419 20.4% $4,515,329 16.7% $10,776
Community Innovation 623 30.4% $10,541,318 39.0% $16,920
Collaborative Projects 40 2.0% $2,187,027 8.1% $54,676
Business Development 25 1.2% $1,123,858 4.2% $44,954
Community Innovation 15 0.7% $1,063,169 3.9% $70,878
Total 2,050 100.0% $27,008,333 100.0% $13,175
Business Development 1,412 68.9% $15,403,846 57.0% $10,909
Community Innovation 638 31.1% $11,604,487 43.0% $18,189
Total 2,050 100.0% $27,008,333 100.0% $13,175

2.5 Logic Model and Performance Measurement Strategy

A Performance Measurement Strategy (PMS) was prepared for EODP (2011–14) and finalized in January 2012. The PMS included an EODP logic model and a Performance Measurement Matrix (PMM) containing performance indicators for the overall EODPFootnote 14.

2.5.1 Logic Model

The approved EODP logic model (2012) is provided in Exhibit D1 of Appendix D. It outlines the key activities of the program; the outputs that will result from those activities; and the immediate, intermediate and ultimate outcomes the program is intended to achieve.

2.5.2 Performance Measurement Strategy

In total, there were 11 outputs and outcomes and 45 performance indicators for the EODP (2011–14) program. EODP management implemented an in-depth data collection and progress reporting system to collect information on the performance indicators. The Performance Measurement Matrix (PMM) is provided in Exhibit D2 of Appendix D. The PMM also provides the source and frequency of performance information collection.

Performance information was collected by the 15 CFDCs from the project funding recipients and then summarized and provided to FedDev Ontario. Program management at FedDev Ontario checked and entered this information into a series of spreadsheets, which facilitated consolidation and stratification by CFDC, business line, initiative and key performance indicators.

The CFDCs provided target information for some of the performance indicators in the business plans that were submitted with their funding application. All CFDCs provided targets for the estimated number of projects to be undertaken and amount of leverage for the three business lines. In addition, most CFDCs provided targets for jobs created, or maintained, for the three business lines.

The quarterly reports provided performance information for almost all of the immediate outcomes, but for only some of the intermediate outcomes. The other indicator information would involve surveys or interviews with project funding recipients, which were not included in this evaluation.

Further, the measurement of the ultimate outcomes requires standardized methodology using Statistics Canada information such as the Census; the Labour Force Survey; and the Survey of Employment, Payrolls and Hours. These ultimate outcomes contribute to FedDev Ontario’s strategic outcome: a competitive southern Ontario economy. Since the program was completed on March 31, 2014, this evaluation was unable to provide an assessment of the program’s ultimate outcomes. This is a result of the two to five year period required for many projects to achieve their long term impacts as well as the time required for Statistics Canada to capture information regarding ultimate outcomes.

2.6 Stakeholders

FedDev Ontario worked with CFDCs to deliver EODP. Key stakeholders included the CFDCs, as well as the potential recipients of EODP funding. As noted above, the key stakeholder recipient groups that were eligible for EODP project funding were the following:

In addition to the recipient groups, the CFDCs worked with interested parties, such as municipal organizations, industry associations, volunteer groups, other CFDCs, departments and agencies of the Government of Ontario, and other federal departments and agencies (e.g., Business Development Bank of Canada).

2.7 EODP Expenditures

EODP (2011–14) was funded at $10 million per year for two years through Budget 2011 (March) from FedDev Ontario’s existing reference levels and was subsequently extended in 2013–14 for an additional $10 million, also from FedDev Ontario’s existing reference levels. In the first of the three years (i.e., 2011–12), the program was slow to launch. The program extension was announced in the late fall of 2011, and the 15 CFDCs submitted their applications and business plans near the end of 2011. The CAs between FedDev Ontario and the CFDCs were not signed until late February 2012.

The late signing of the CAs left only a short period until the end of March 2012 to undertake projects. Recognizing that it would not be possible to spend all the contribution funding identified for 2011–12 by March 31, FedDev Ontario adjusted the annual funding amounts over the three years so that fiscal 2011–12 received a smaller amount and 2012–13 and 2013–14 received larger amounts. Thus, the $30 million was expended over the three years.

The actual expenditures for the EODP during the three-year period from 2011 to 2014 are given in Exhibit 2.3.

Exhibit 2.3: EODP Final Expenditures, 2011–12 to 2013–14

EODP Actual Expenditures 2011–12 2012–13 2013–14 Total Percent
FedDev Ontario—EODP Administration $400,000 $400,000 $400,000 $1,200,000 4.0%
CFDCs—EODP Delivery $2,370,638 $13,178,759 $13,145,183 $28,694,580 96.0%
Administration $516,470 $889,608 $887,639 $2,293,717 7.7%
Contributions $1,854,168 $12,289,151 $12,257,544 $26,400,863 88.3%
Total EODP $2,770,638 $13,578,759 $13,545,183 $29,894,580 100.0%

Exhibit 2.3 shows that the contributions provided by the CFDCs for projects accounted for 88.3 percent of the total expenditures of $29.9 million, while the remaining balance included 7.7 percent for CFDCs’ administration costs for program delivery and 4.0 percent for FedDev Ontario EODP administration. Also, of the total of $28.7 million received by the CFDCs for EODP delivery, 7.7 percent ($2.3 million) was used by the CFDCs for administrative activities.

3.0 Evaluation Methodology

This section describes the methodology that was used in this calibrated evaluation of the Eastern Ontario Development Program (EODP), including the evaluation issues covered, data collection methods used, and the evaluation challenges and limitations.

3.1 Evaluation Approach

The EODP evaluation was based on the Treasury Board Secretariat (TBS) Directive on the Evaluation Function (2009),Footnote 15 which requires that evaluations assess all five of its core issues relating to relevance and performance (effectiveness, efficiency and economy) in order to address value for money. However, departments and agencies have the flexibility to determine the evaluation approach and level of evaluation effort in accordance with the program’s risks and characteristics and the quality of performance information available for each individual program.

The evaluation methodology was designed to align with the TBS Policy on Evaluation (2009), and as noted in Section 1.1, has been calibrated to adjust for the following factors:

Recognizing the above, this evaluation of EODP has focused on the years 2011–12 to 2013–14, capturing the period after the last evaluation and up to the start of the extension in 2014.

The evaluation was designed to address the five core issues relating to program relevance and performance required by the TBS Directive on the Evaluation Function. In assessing these five core issues, the evaluation focused on the key questions presented in Exhibit 3.1.

The evaluation design addressed the above evaluation issues and questions, recognizing and accounting for the following factors:

3.2 Data Collection Methods: Design and Application

Considering the overall approach described above in Section 3.1, and recognizing that a calibrated approach was required, the data collection methods used in this evaluation and their application to the evaluation issues and questions are described below.

3.2.1 Application of Data Collection Methods to the Evaluation Issues and Questions

The selection of data collection methods for the evaluation was based on the most efficient means of rigorously addressing the evaluation issues and questions within the calibrated scope, budget and timeframe for the evaluation.

The following data collection methods were used in this evaluation:

The application of the above data collection methods to the evaluation issues and questions is shown in Exhibit 3.3.

Exhibit 3.3: Data Methods for Key Evaluation Issues and Questions

Relevance
Key Evaluation Issues and Questions Document and Literature Review Review of Performance, Admin. & Financial Data Key Informant Interviews Data and cost analyses
Issue #1: Continued need for program 1.1 Is there a continued need for the EODP?  
1.2 Does the EODP complement, duplicate or overlap other government programs? Other private sector services?  
Issue #2: Alignment with government priorities 2.1 To what extent is the EODP consistent with government priorities:
  1. FedDev Ontario’s PAA and strategic outcome?
  2. Federal priorities and strategies?
 
Issue #3: Alignment with federal roles and responsibilities 3. To what extent is the EODP aligned with the federal government's activities, roles and responsibilities?  
Performance (effectiveness, efficiency and economy)
Key Evaluation Issues and Questions Document and Literature Review Review of Performance, Admin. & Financial Data Key Informant Interviews Data and cost analyses
Issue #4: Achievement of expected outcomes 4.1 To what extent have the expected immediate, intermediate and ultimate outcomes been achieved as a result of the EODP?  
4.2 Has the EODP produced unintended positive or negative outcomes?      
Issue #5: Demonstration of efficiency and economy 5.1 To what extent is the EODP delivery model efficient in producing outputs and progressing toward expected outcomes?    
5.2 Is there a more cost-effective way of achieving the expected results, taking into consideration alternative delivery mechanisms, best practices and lesson learned?  

3.2.2 Description of the Data Collection Methods

The data collection methods used in the evaluation are described below.

Document and Literature Review

A review of relevant documents was carried out to obtain a good understanding of EODP, including its business lines, rationale and historical context. The review included program descriptions and background information as follows:

File Review of Performance, Administrative and Financial Data

Documents containing performance, administrative and financial data on EODP that are collected and reported by FedDev Ontario programs and Finance were reviewed. Performance data for the program, collected from the CFDCs, was captured internally by FedDev Ontario in a series of spreadsheets. These spreadsheets facilitated consolidation and stratification by CFDC, business line, initiative and key performance indicators.

Key Informant Interviews

Interviews with key informants (KIs) were a primary information source for the evaluation. In total, 21 interviews were undertaken with the following groups:

The KIs were identified in consultation with FedDev Ontario’s EODP management and the Evaluation Directorate. For each of the targeted groups, semi-structured interview guides were developed to address the key evaluation questions in Exhibit 3.1. All the interviews were conducted by telephone. Interview findings were captured and entered into an electronic database and were summarized according to the evaluation issue, indicator and respondent type.

Data and cost analyses

A number of data and cost analyses were undertaken in support of the evaluation, in particular for issue #5 on demonstration of efficiency and economy. Several of these analyses examined the delivery of the program, and have been provided in Section 6.0.

3.3 Evaluation Challenges and Limitations

In undertaking this calibrated evaluation, the evaluation project team identified the following challenges and limitations that are discussed below.

3.3.1 Challenges

EODP was extended in 2014, and changes to the program were implemented—In selecting KIs, the team made an effort to identify individuals with knowledge of the program during the evaluation period from 2011–12 to 2013–14. FedDev Ontario and CFDC personnel were generally familiar with EODP (2011–14) and were able to respond to questions about EODP during the 2011–12 to 2013–14 period. However, many of the respondents also referred to the changes made in the subsequent EODP (2014–19) when commenting on EODP (2011–14). Other KIs had a general familiarity with the program but were not always able to distinguish between the specifics of each EODP.

3.3.2 Limitations

Coverage of government, CFDCs and program funding recipient stakeholders—The 21 KI interviews provided good coverage of government and the CFDCs that directly received funding from FedDev Ontario. However, the KI interview methodology is limited by the fact that all findings reflect the views and opinions of only the individuals who were interviewed, who did provide insightful information on the program delivery and benefits. However, project funding recipients were not included in this calibrated evaluation.

Timely and accurate information—To measure the performance of the program, FedDev Ontario depends on project funding recipients providing the information on their projects. This information is collected by the CFDCs and then passed on to FedDev Ontario through the quarterly reports. While the project funding recipients have provided the performance information as required by the CAs, FedDev Ontario relies on the CFDCs to review the collected information before it is forwarded to FedDev Ontario. FedDev Ontario has only limited capacity to verify the performance information provided by the project funding recipients, beyond a check for reasonableness by program management. However, this approach to collecting information on performance and results is the only cost-effective way of obtaining this information and is extensively used in similar evaluations.

Measurement of impacts—The performance information collected by FedDev Ontario was complete up to March 31, 2014; however, many of the project impacts would have occurred after that date. While some of the projects would achieve benefits in the short term (one to two years) after project completion, many would be expected to achieve benefits in the medium term (three to five years) or the long term (beyond 5 years). Further, the measurement of some of the intermediate outcome indicators requires follow-up with the project funding recipients to obtain quantitative and qualitative information on the impacts of the funding; however, a survey of project recipients was not included in this evaluation. Therefore, the information from the performance reports as of March 31, 2014, provides a limited picture of the benefits, and the indicators measured through the quarterly reports likely underestimate the overall impacts of the funded projects.

It takes two to five years after the program is completed, as noted above, for the projects to achieve their full benefits and for the Statistics Canada databases to provide information on how the program contributed to the ultimate outcomes. An assessment of the ultimate outcomes was not feasible within the scope and time frame of this evaluation.

Impact on the southern Ontario economy—The impact of the EODP contributions is relatively small when compared to the economy in the region. Moreover, other investment programs are also active in the region. It is therefore difficult to determine the extent to which EODP has contributed to FedDev Ontario’s strategic outcome: a competitive southern Ontario economy.

4.0 Findings on Relevance

This section provides the evaluation findings for relevance. These findings have been structured on the basis of the three core relevance issues described in Section 3.1.

4.1 Issue #1: Continued Need for Program

4.1.1 Was There a Continued Need for a Program Similar to EODP?

Key Findings: Past evaluations and renewal of the program for the 2014-2019 period confirm EODP’s relevance.

Interview respondents were generally satisfied with EODP and were very supportive of continuing the program, as it stimulates investment and plays a key role in the community.

The relevance of the Eastern Ontario Development Program (EODP) has been well established since its inception in 2004. Past evaluations of the program and the government’s renewal of the EODP for the 2014–19 period confirm its relevance.

The KIs noted that there have been economic challenges in rural eastern Ontario since the start of EODP (formerly known as the Eastern Ontario Development Fund [EODF]) in 2004 and that the recession resulting from the 2008–09 financial crisis hit rural eastern Ontario hard, resulting in more systemic challenges for the area.

The challenges that the KIs mentioned included layoffs and plant closings, increased unemployment, outmigration of youth (those who stayed lacked skills, and those who left didn’t come back), lack of opportunities for young people, underemployment, an aging workforce, a shortage of skilled labour, reduced access to capital (inability to fund small businesses), low productivity, no synergy from having business clusters or infrastructure, lack of diversification (traditional sectors of agriculture and tourism are low paying and seasonal), inability to attract businesses, limited access to high-speed Internet (broadband), and low web presence (website and e-commerce).

In addition, the high value of the Canadian dollar versus that of the US dollarFootnote 16 leading up to 2011 affected competitiveness and contributed to the loss of manufacturing and reduced tourism. US trade protection was also noted as a factor. Another challenge was the effect of lower commercial and industrial activity on the tax base of municipalities, which made it difficult for them to maintain infrastructure and provide services. Transportation infrastructure, both road and rail, was also noted as a challenge.

EODP (2011–14) was designed to help address the challenges noted above (see Section 2.0). Within the parameters of EODP, the 15 Community Futures Development Corporations (CFDCs) had the flexibility to target the assistance to meet the key challenges in their communities. Respondents felt that EODP had done a good job addressing the challenges, considering its size and resources; however, the resources did not come close to meeting demand. It was recognized that EODP could not address these challenges on its own, but had made a difference by contributing to the solutions. For example, while EODP (2011–14) did not have the resources to improve broadband coverage, it assisted businesses by supporting websites, e-commerce, and IT solutions. For example, it was observed that EODP "plants the seed" and helps mitigate risks of new initiatives. EODP may support a feasibility study that results in a major investment several years later.

Respondents were generally very pleased with EODP and were very supportive of continuing the program, as it plays a key role in the community. EODP stimulates investment in the community. Hundreds of youth would not have found work in their home towns. Worker skill shortages would not have been addressed. Economic development takes time, as it involves multiple years and multiple efforts, and EODP has been instrumental in keeping the momentum going. Several CFDCs commented that EODP strengthens the role of the federal government in rural eastern Ontario. An "Other" respondent stated that EODP is critical to rural eastern Ontario, particularly for smaller businesses.

In summary, the area benefits from EODP, as it encourages growth of new and existing businesses and enables community-led development projects that enhance and diversify local economies within rural eastern Ontario communities.

4.1.2 Did EODP Complement, Duplicate or Overlap Other Government Programs or Other Private Sector Services?

Key Findings: It was determined that EODP complemented other federal and provincial government funding programs such as, Small Business Centre (provincial program), where they were available to the recipients of EODP project funding assistance. The existence of other programs was not viewed as a concern, as these federal and provincial government funding programs had a different emphasis and the demand for funding in eastern Ontario was much greater than the funding available from EODP.

EODP is flexible, has broad application within its scope, and is available to a wide group of applicants. Other programs are more targeted or restricted (e.g., hi-tech or agriculture; larger firms) and have specialized requirements. A respondent commented that there is never money to develop strategy for economic development, which is supported by EODP, and this helps drive the other programs. Further, EODP focuses on small (micro) businesses with fewer than 10 employees, which accounts for almost all the businesses in rural eastern Ontario. However, EODP also works with larger businesses of up to 300 employees, of which there are very few.

All respondents commented that EODP was complementary to other federal and provincial government programs that may be available to its beneficiaries. In terms of complementarity, one approach is to nurture the small businesses forward with EODP, and then when the business is larger, support them with other programs. For example, Mariposa Dairy Ltd., received $63,827 under EODP for the period of this evaluation and has been awarded another $1M through the Southern Ontario Prosperity Program (SOPI) from June 26, 2014 to December 31, 2016. In addition, there are examples where EODP funds certain components of given projects, with complementary programs funding other components.

Respondents commented that EODP doesn’t duplicate other funds and that there is an absence of federal programs in eastern Ontario targeting the same beneficiaries. Further, EODP complements provincial programs, which work well together. For example, EODP does capacity support, such as business planning and skills development, which are critical precursors to the Province of Ontario funding capital (machinery and equipment) through its own Eastern Ontario Development Fund, which is designed for much larger projects. The province views EODP as being very important and complementary.

EODP is also complementary to the Community Futures Program (CFP)Footnote 17, a national loan program in southern Ontario, which is administered by FedDev Ontario and delivered by the CFDCs to the same recipients. Where appropriate, in delivering CFP the CFDCs may reduce the loan and include a grant component from EODP.

CFDCs coordinate efforts closely within their respective geographic areas. They are aware of their key community stakeholders, and those stakeholders are aware of the CFDC’s role. They work in collaboration with other funders, including the province and other FedDev Ontario program funds, to try to determine how to best leverage government programs to support companies. The organizations collaborate on larger projects. CFDCs are often co-located with other programs in municipal buildings, resulting in fewer stops for proponents, increased synergies, reduced costs, etc. Triaging applicants was mentioned by several respondents as the approach used to determine how best to support companies with programs and referrals. This involves identifying applicants’ core needs and then matching them with the appropriate programs.

All respondents agreed that that EODP complemented other federal and provincial government funding programs where they were available to the project funding recipients. The existence of other programs was not viewed as a concern, as other programs had a different emphasis and the demand for funding in eastern Ontario was much greater than the limited funding available.

4.2 Issue #2: Alignment with Government Priorities

Key Finding: EODP was consistent and fully aligned with government priorities, including FedDev Ontario’s Program Alignment Architecture and strategic outcome and federal government priorities and strategies.

Budget 2010 announced that funding would be provided for EODP for 2011–12 and 2012–13 as part of "Support for Communities." The Budget’s focus was on "helping Canadian firms create jobs, modernize their operations and better compete globally." The $20 million over two years was again mentioned in Budget 2011 and in Budget 2012. Funding for EODP was subsequently amended to include funding for 2013–14. Budget 2013 then extended EODP for five years to "continue to promote business development, job creation and self-reliant communities in rural eastern Ontario" as part of the $920 million, five-year renewal of FedDev Ontario’s mandate as of April 1, 2014.

EODP was designed to support the community economic development program activity in FedDev Ontario’s Program Alignment Architecture. In turn, the community economic development program activity supported FedDev Ontario’s strategic outcome ("a competitive southern Ontario economy") and contributed to the Government of Canada’s outcome ("strong economic growth").Footnote 18

EODP was therefore consistent and fully aligned with government priorities and strategies.

4.3 Issue #3: Alignment with Federal Roles and Responsibilities

Key Finding: EODP was fully aligned with federal roles and responsibilities.

FedDev Ontario, as a regional development agency and one of the federal government's representatives, was aligned with federal government priorities through EODP to encourage growth of new and existing businesses; and to enable community-led development projects that enhance and diversify local economies within rural eastern Ontario communities. EODP’s program activities were designed to contribute to the diversification and competitiveness of the rural eastern Ontario economy; economic stability and growth and job creation; and sustainable, self-reliant communities. Respondents considered EODP to be fully aligned with federal roles and responsibilities and to be a formal and explicit match with government policy.

Past Speeches from the Throne have placed emphasis on economic growth and stability, specifically, to support traditional industries, create jobs, and minimize loss of employment. The Speech (2011) noted that "jobs and growth will remain our government’s top priority."Footnote 19

5.0 Findings on Performance—Achievement of Expected Outcomes

This section provides EODP evaluation findings for achievement of expected outcomes. The findings have been structured around the two key questions provided in Exhibit 3.1.

5.1 Achievement of the Immediate and Intermediate Outcomes

Key Findings: Overall, EODP was generally successful in achieving results and exceeding almost all targets established for the program. Business Development (BD) and Community Innovation (CI) were found by respondents to have been very successful, with some reservations (section 3.3). Collaborative Projects (CP), which involved investments from two or more Community Futures Development Corporations (CFDCs) and benefited one or more communities, had mixed results, with some respondents feeling it had been successful and others feeling that it had significant challenges.

Interview respondents viewed the CFDCs as being effective in contributing to the rural economy and community by supporting small business development, leveraging funding from other programs, and working with partners to promote economic stability, growth, and diversification. Moreover, several respondents noted that given EODP’s limited budget, the program has been extremely effective in achieving results, and while these results can be seen at the community level, it is difficult to assess the program’s impact on eastern Ontario given the size of the eastern Ontario economy.

This subsection addresses the question "To what extent have the immediate and intermediate outcomes been achieved as a result of EODP?" An assessment of the ultimate outcomes was not feasible within the scope and timeframe of the evaluation.

Program outputs and outcomes result from the completion of EODP projects, which contribute to the achievement of the program objectives. The linkage between the program activities, outputs and outcomes is shown in the EODP logic model in Appendix D.

The achievement of immediate and intermediate outcomes is summarized in Section 5.1.1. The assessment of these outcomes is based on FedDev Ontario’s comprehensive performance information, key informant (KI) interviews, file review and document review. Success stories, prepared by FedDev Ontario, have been used to illustrate how EODP contributed to project successes. Activities and outputs are included in the discussion, as they are directly linked to achieving the outcomes.

5.1.1 Overall Results

Overall, EODP was successful in achieving results and exceeding almost all targets established for the program. The business lines Business Development (BD) and Community Innovation (CI) were found by respondents to have been very successful with some reservations (section 3.3). Collaborative Projects (CP), which involved multiple Community Futures Development Corporations (CFDCs) and benefited one or more communities, had a mixed result, with some respondents feeling that CP had been successful and others feeling that it had significant challenges.

5.1.2 Performance Results

A summary of the performance results for EODP (2011–14) for the three business lines (BD, CI, CP) is provided in Exhibit 5.1. This information was compiled from the quarterly performance reports submitted by the 15 CFDCs to FedDev Ontario throughout the program.

Performance Results for EODP (2011–14)

Projects
Results Business Development (BD) Community Innovation (CI) Collaborative Projects (CP) TotalProgram
Number 1,387 623 40 2,050
Funding
Results Business Development (BD) Community Innovation (CI) Collaborative Projects (CP) TotalProgram
EODP funding contribution* $14,279,989 $10,541,318 $2,187,027 $27,008,333
Leverage funding $42,522,720 $25,954,056 $4,754,186 $73,230,962
Total project funding $56,802,709 $36,495,373 $6,941,213 $100,239,295
Leverage ratio (leverage / EODP contribution) 2.98 2.46 2.17 2.71
Businesses
Results Business Development (BD) Community Innovation (CI) Collaborative Projects (CP) TotalProgram
Created 150 139 0 289
Expanded 805 384 18 1,207
Maintained 2,718 3,390 848 6,956
Total businesses 3,673 3,913 866 8,452
New markets and opportunities 1,352 1,482 229 3,063
Improved decision-making skills 1,308 1,451 293 3,052
Jobs
Results Business Development (BD) Community Innovation (CI) Collaborative Projects (CP) TotalProgram
Created 1,638 1,007 43 2,687
Maintained 9,386 4,177 1,083 14,646
Total jobs 11,023 5,184 1,126 17,333
Workshops, training and counselling
Results Business Development (BD) Community Innovation (CI) Collaborative Projects (CP) TotalProgram
Number of sessions 2,162 856 95 3,113
Trainees supported (including youth) 5,225 6,253 1,050 12,528
Youth trainees 3,551 3,818 59 7,428
Community economic development
Results Business Development (BD) Community Innovation (CI) Collaborative Projects (CP) TotalProgram
New economic sectors active 309 196 4 509
New staff & volunteers in economic development 641 1,254 36 1,931
Other areas
Results Business Development (BD) Community Innovation (CI) Collaborative Projects (CP) TotalProgram
Number of non-CFDC partnerships 1,147 1,725 215 3,087
Intern placements 297 20 7 324
Trade shows attended 239 55 10 304
Plans developed 285 194 14 493

The figures show that for the 2,050 projects undertaken, 1,387 (68 percent) were BD; 623 (30 percent) were CI; and 40 (2 percent) were CP. The performance results in Exhibit 5.1 are summarized below:

The quarterly reports provided information on the performance indicators for almost all of the immediate outcomes, but only some of the intermediate outcomes (e.g. leveraged amounts and job impacts).

5.1.3 Respondent Observations

The following comments were provided by the respondents interviewed as part of the evaluation. The comments provide some insight into the program, but these comments may not be inferable, given that not all CFDCs were included in the interview program.

Business Development

BD was generally viewed as being successful or very successful. The following observations were made by the respondents on the BD initiatives:

Community Innovation

CI was considered successful. Several respondents noted that plans (e.g., feasibility studies, business cases, engineering and architectural studies) that were supported by EODP are being used. It was also observed that these plans and studies would not have happened if they hadn’t been 100 percent funded. Respondents noted that CI supported an NFP for foresters, which led to third-party certification, which opened up markets and increased the value of the products produced; business training for Indigenous groups; trail building for local municipalities; studies for the building of tourist attractions (e.g., Thousand Islands Boat Museum in Gananoque, Brockville Aquatarium). It was suggested that the program should be extended to things that contribute to quality of life in a community, such as a community hall.

Collaborative Projects

CP, which involved multiple CFDCs and benefited one or several communities, received a mixed reaction from respondents. Some CFDCs had been successful in implementing projects with neighbouring CFDCs, while others commented that it was difficult to put together regional projects, as there is a need to establish a consortium. This involves having to go to each CFDC, municipality, etc., to get their support, with each assessing the cost and the impact that the project will have on their local interests and how the outputs and outcomes (e.g., jobs created) will be attributed.

During EODP (2011–14), one CFDC in the consortium of CFDCs would take the lead. There was no formal approval process or dedicated funding for CP. One respondent commented that for larger projects, a longer lead time is required, as it is difficult to plan a large or strategic project and complete the implementation within a fiscal year. During the program, there were 40 projects involving the participation of 128 CFDCs (some CFDCs participated in more than one project). This was an average of 3.2 CFDCs per project, which may be considered low, given that there are 15 CFDCs in eastern Ontario that could be involved in regional projects.

5.1.4 Success Stories

The following success stories, prepared by FedDev Ontario, illustrate how EODP has assisted projects and contributed to their success.

Success Story (Winchelsea Events)—Winchelsea, located in Winchester in the United Counties of Stormont, Dundas and Glengarry (SD&G), is a multipurpose event management and catering facility. With EODP assistance through the SD&G CFDC, the company expanded its facility to accommodate an addition of 298 square metres (3,210 square feet) of serviceable space and upgrade its IT to attract a diverse range of business and special event promotional functions. Six full-time and one part-time job were created. The project was undertaken in 2013 for a total cost of $216,000, with  $50,000 repayable contribution from EODP.

Success Story (Sprucewood Brands)—Sprucewood Brands, located in Warkworth in Northumberland County, is a bakery with humble beginnings that specializes in handmade shortbread cookies. Starting out in a small kitchen in the owner's home, Sprucewood had a limited production capacity and a limited ability to keep up with demand and access new markets. With EODP assistance through the Northumberland CFDC, Sprucewood has expanded into its own 316 square metre (3,400 square foot) facility—a revitalized 1920s' building, where eight full-time and four part-time staff are now employed. This project has played a vital role in helping the business take on larger accounts, which has led to an increase in staff and plans to add more. Sprucewood is now a $500,000 per year business. The project was undertaken between 2012 and 2014 for a total cost of $79,000, with a $35,000 EODP repayable contribution.

Success Story (Fenelon Falls)—Located on the Trent–Severn Waterway in the Kawarthas, Fenelon Falls was faced with the challenge of providing improved access to its downtown commercial core for the significant boat traffic that passes through the area. With EODP assistance through the Kawartha Lakes CFDC, Fenelon Falls reconstructed a wall with mooring capability for vessels passing through the area. More than 137 metres (450 linear feet) of docking capacity was added, allowing more boats to tie up in Fenelon Falls. In addition, patrons are now within walking distance of the main street, where they can visit various restaurants, craft shops and clothing stores. In the first year alone, the economic impact is estimated to have exceeded $300,000, greatly improving the economic viability of the area. The project was undertaken in 2012 for a total cost of $187,000, with a $72,000 EODP repayable contribution.

5.2 Unintended Positive or Negative Outcomes

Key Finding: Interview respondents noted a number of unintended positive outcomes, as well as a few challenges resulting from the program.

This subsection addresses the question "Did the EODP produce unintended positive or negative outcomes?" Respondents were asked this question during the KI interviews, and while some had no comments, others provided the following comments.

Positive

Negative

6.0 Demonstration of Efficiency and Economy

This section provides the Eastern Ontario Development Program (EODP) evaluation findings for demonstration of efficiency and economy. The findings have been structured around the two key questions in Exhibit 3.1 and take into account guidance provided by the Treasury Board Secretariat on issue #5Footnote 23 and its three components: economy, operational efficiency and allocative efficiency. These are the two key evaluation questions for this section:

6.1 Efficiency and Economy of the EODP Delivery Model in Producing Outputs and Outcomes

Key Finding: The EODP (2011–14) delivery model was found to be efficient and economical in producing outputs and progressing toward expected outcomes.

This subsection addresses the question "To what extent was the EODP delivery model efficient in producing outputs and progressing toward expected outcomes?"

6.1.1 Program Design

For program design, FedDev Ontario defines the overall scope and eligible activities and provides some guidance on areas of focus that relate to the government’s priorities. EODP (2011–14) was focused more on economic development than the prior EODP and less on community development.

The 15 Community Futures Development Corporations (CFDCs) in eastern Ontario apply to FedDev Ontario for EODP funding. Each CFDC consults with groups in their community to identify the priorities that best respond to the needs of the community and then prepares a business plan and application to obtain EODP funding. The business plan includes the CFDC’s plans and targets for each of the Business Development (BD), Community Innovation (CI) and Collaborative Projects (CP) business lines. When the application or business plan is approved by FedDev Ontario, a contribution agreement (CA) is signed with the CFDC.

EODP provides the CFDCs with the flexibility to deliver the program within its scope and eligible activities (defined by FedDev Ontario) to address their priorities, which vary by CFDC, in their geographic area. This approach is considered appropriate, as it provides each CFDC, which has knowledge of its community’s needs, with the ability to respond to local priorities.

The evaluation identified several areas of program design for consideration by program management:

6.1.2 Demand for EODP Funding

As indicated by the number of enquiries (6,714) and the number of applications (2,683)—resulting in 2,050 projects being approved for assistance—the demand for EODP project funding was substantial. These figures indicate that, on average, 3.3 enquiries and 1.3 applications were received for each approved project.

Respondents commented that there was far more demand for assistance than funds available. While this is accurate overall, demand was stronger in some areas of the program than in others. For example, one respondent noted that skills development seems to work better with larger businesses, as small (micro) businesses didn’t seem interested. It is noted that CFDCs have the flexibility within the program guidelines to allocate their funding to areas of greatest need or interest.

From the data gathered, it is evident that the demand for EODP funding was higher than available funding, and CFDC staff has a range of qualified applicants from which to select and have the flexibility to adjust their funding allocations, as appropriate within the EODP guidelines, to areas of greatest need or priority.

6.1.3 Use of Leverage

Leverage was a requirement for all projects involving commercial enterprises, but not for NFPs:

However, as shown in Exhibit 5.1, all three business lines were able to leverage the EODP contributions. The actual leveraging considerably exceeded the target for the program, which was based on aggregating the targets provided by the CFDCs in their business plans. The EODP funding contributions of $27 million to projects resulted in leveraging additional funding of $73 million from other sources, resulting in total project expenditures of $100 million. The leveraged funding was therefore 2.71 times the EODP contributions. The leveraging for BD was higher than for CI, which in turn was higher than for CP.

Further, the 2.71 leverage achieved by EODP is comparable to the 2.65 leverage achieved by the Southern Ontario Advantage initiatives on $419.7 million of contribution expenditures through FedDev Ontario’s Southern Ontario Development Program (SODP).Footnote 27

This co-investment approach, using leverage, allowed FedDev Ontario to considerably increase the impact of EODP in rural eastern Ontario communities by attracting other sources of funding.

6.1.4 Use of Authorized Funding

EODP (2011–14) was launched late, with the result that the CAs with the CFDCs were signed in late February 2012, delaying the CFDCs’ ability to fund projects. This meant that there was little time for projects to be carried out in 2011–12. FedDev Ontario recognized this problem and adjusted the cash flow for the CFDCs to $2.4 million in 2011–12 and $13.2 million in both 2012–13 and 2013–14. The delays in starting the program also led to further delays for some CFDCs, as they had to rehire staff (who had been let go earlier) to help deliver the program.

Despite the slow start, FedDev Ontario, with the assistance of the CFDCs, was successful in spending $29.9 million over the three years (see Exhibit 2.3), which was almost the entire $30 million budgeted.

In addition, projects that were supported in EODP (2011–14) had to be completed by the end of the fiscal year, and there was no carryover from one year to the next. This made it difficult to undertake larger projects or to start projects late in the fiscal year. One CFDC respondent commented that the March 31 deadline for completing projects did not give some of the funding recipients enough time to do the necessary preparatory work before embarking on their projects. Further, a respondent commented that a five-year horizon provides sufficient time to do proper planning, for partners to contribute, and to implement the solution. Another respondent stated that sometimes equipment deliveries can be delayed and miss the March 31 deadline.

6.1.5 Third-Party Delivery

FedDev Ontario made effective use of the 15 CFDCs in eastern Ontario as delivery partners for EODP. The CFDCs are a trusted delivery system and a critical component in the success of EODP. Once the CFDCs have signed the CAs, they work largely independently of FedDev Ontario program management to promote the program, evaluate requests for funding, determine the successful projects within the guidelines provided by FedDev Ontario, oversee the resulting contracts, collect performance information and pay claims.

Interview respondents, in commenting on the advantages of the CFDCs, stated that the CFDCs provide an existing structure located in the community, have staff expertise related to economic development, are aware of local issues, and have a relationship with the community. In addition, they are connected to the local businesses and NFPs and to the other players in economic development in the community. Their boards of directors are made up of approximately 12 volunteers from the community; and their staff works in the community and have for the most part been with the CFDCs for some time. The local decision making and flexible structure of the program allow it to address local challenges, within its defined scope and eligible activities.

In EODP (2011–14), the CFDCs used 7.7 percent of the contribution amount of $28.7 million for program administration activities, which is lower than the 10 percent that FedDev Ontario normally provides for third-party delivery. In the templates for the business plan, the CFDCs were limited to a maximum of $60,000 per year ($66,000 per year for the three bilingual CFDCs), for a total of $918,000 per year for the 15 CFDCs. For the three-year program, CFDC program administration activities could have cost a maximum of $2.75 million, or 9.6 percent of the total contribution that was available to the CFDCs. This indicates that the CFDCs did not budget for or charge up to the maximum amount and chose to spend the contribution funds on projects instead.

For FedDev Ontario to manage larger numbers of smaller projects in-house would require a direct interface with beneficiaries to obtain project applications, evaluate and approve the submissions, negotiate and execute project CAs, and manage the projects while they are underway. Further, considerable effort would be involved, and in some cases, FedDev Ontario’s reach and knowledge of the target organizations and individuals might be insufficient to successfully deliver the program. Moreover, effectively managing certain aspects of the program would require a local presence. For larger projects, however, where increased due diligence is required in assessing the application and monitoring the project, in-house management is required.

FedDev Ontario’s administrative cost of delivering the EODP, as shown in Exhibit 2.3, was $1.2 million for the delivery of $28.7 million in contribution funding to the CFDCs. This is $0.04 for each program dollar delivered to the CFDCs. This compares favourably with FedDev Ontario’s overall cost of $0.157Footnote 28 for the delivery of all its grants and contributions support programs when delivery partner costs are excluded. It also compares favourably with the delivery costs of the other regional development agencies.Footnote 29

If the CFDC delivery partner cost is added to the FedDev administrative cost above, the total delivery cost would be $3.49 millionFootnote 30 for $26.4 million in funding support provided to the project funding recipients (see Exhibit 3.1). This is $0.132 for each program dollar delivered by the CFDCs, which still compares favourably with the FedDev Ontario delivery cost of $0.157, which does not include delivery partner costs.

6.1.6 Partnerships

Partnerships expanded EODP’s impact by leveraging the EODP contributions. As noted above, each $1.00 of EODP contribution funding for projects leveraged, on average, $2.71 in funding from other sources.

The CFDC efforts are well coordinated within each of their geographic areas. They know all the other players in the community, who also know the CFDC. They work in collaboration with other funders, including other FedDev Ontario program funds, to try to determine how best to support companies. For larger projects, the organizations get together. CFDCs are often co-located with other programs in municipal buildings, and this arrangement makes for fewer stops for project proponents, facilitates synergy, reduces costs, etc.

EODP complements other funding programs, such as programs managed by the Province of Ontario. The province views EODP as being very important and complementary, as EODP works well with their funding. For example, EODP does capacity support, such as business planning and skills development, which are critical precursors to the Province of Ontario funding capital (machinery and equipment) through its own Eastern Ontario Development Fund, which is designed for much larger projects. CFDCs also worked with other funding programs, such as the National Research Council’s Industrial Research Assistance Program and the Business Development Bank of Canada. Further, EODP is complementary to the Community Futures Program (CFP). Where appropriate, CFDCs delivering CFP may reduce the loan and include a grant component from EODP. Several interview respondents noted that EODP is synergistic with CFP and resulted in the CFDCs having an increased profile in the community and better performance than CFDCs in southwestern Ontario without an EODP-type program.

6.1.7 Performance Measurement

FedDev Ontario was supported by a comprehensive performance measurement system based on the EODP Performance Measurement Strategy (2012), which had 11 outputs and outcomes and 45 performance indicators. Performance information on outputs, most immediate outcomes, and some intermediate outcomes was collected by the 15 CFDCs from the project funding recipients and then summarized and provided to FedDev Ontario. Program management at FedDev Ontario checked and entered this information into a series of spreadsheets, which facilitated consolidation and stratification by CFDC, business line and initiative and by key performance indicators.

The remaining performance indicator information would involve undertaking surveys or interviews with project funding recipients or using macroeconomic data and standardized methodology, both of which were not included in this evaluation.

6.1.8 Level of Satisfaction

CFDC respondents reported a good working relationship with the FedDev Ontario project officers and were very happy with the services received. CFDC staff consults with their project officers from time to time on grey areas, such as whether a project fits within the activity guidelines or how to classify it for performance tracking. FedDev Ontario project officers respond very quickly.

CFDCs reported that they provide an appropriate level of support to project applicants; help fill out application forms, etc.; and work with the funding recipients. As project funding recipients were not consulted in this calibrated evaluation, no information was available on their experiences or level of satisfaction. However, FedDev Ontario program management stated that they have received very few complaints, and no major problems had arisen.

6.1.9 Administrative Processes

Administrative processes and templates have been developed by each CFDC independently. While a few CFDCs mentioned talking to other CFDCs about processes, there may be potential benefits in making available best practices and lessons learned in key administrative areas, with the objective of improving efficiency. This might be a potential role for the EOCFDC Network.

6.2 Cost-Effectiveness in Achieving Expected Results

Key Finding: There was a consensus among interview respondents that use of CFDCs to deliver EODP was the most cost-effective approach to achieving program results.

This subsection addresses the question "Is there a more cost-effective way of achieving the expected results, taking into consideration alternative delivery mechanisms, best practices and lessons learned?"

There was a consensus among interview respondents that FedDev Ontario’s use of CFDCs to deliver EODP was the most cost-effective approach of achieving program results. One respondent noted that there is no better, more cost-effective group of organizations that are lean, well established and fully connected in the community. Another commented that there is no better delivery method at the local level that could ensure the quality of applications and with a board of directors from the local area with knowledge of the local business and NFP community.

Further, the costs incurred by FedDev Ontario in using the CFDCs to deliver EODP were largely incremental to CFDC operations, and as shown above, the delivery costs for EODP were found to be very reasonable (see Section 6.1.5 above). Moreover, EODP was synergistic with other CFDC operations, as it improved a CFDC’s overall performance and increased CFDC profile in the community.

7.0 Conclusions and Recommendations

This evaluation was undertaken to determine the extent to which the Eastern Ontario Development Program (EODP) continues to be relevant, is on track to achieve its expected program outcomes, and has demonstrated efficiency and economy. The evaluation involved a calibrated methodology that took into account timing of previous evaluations and changes to the EODP. Conclusions and recommendations for the evaluation are provided below.

7.1 Conclusions

The following conclusions are based on the evidence collected in this evaluation.

Program Relevance

Past evaluations and the government’s renewal of the program for the 2014–19 period confirm EODP’s relevance.

Interview respondents were generally satisfied with EODP (2011–14). They supported continuing the program, as it stimulated investment and played a key role in the community. Further, respondents felt that although EODP resources were small in comparison to the size of the rural eastern Ontario economy, the program was extremely effective and had an impact by helping address major challenges and encouraging business development and community development.

EODP was found to complement other federal and provincial government funding programs where they were available to the project funding recipients. The existence of other programs was not viewed as a concern, as other programs had a different emphasis and the demand for funding in eastern Ontario was much greater than the limited funding available from EODP.

EODP (2011–14) was found to be consistent and fully aligned with government priorities, including FedDev Ontario’s Program Alignment Architecture and strategic outcome; and federal government priorities and strategies, including Budget 2010 (March) and Budget 2011 (June). Further, EODP was fully aligned with federal roles and responsibilities.

Program Performance—Achievement of Program Outcomes

Overall, EODP (2011–14) was generally successful in achieving results and exceeding almost all targets established for the program. The business lines Business Development (BD) and Community Innovation (CI) were found by respondents to have been successful, with some caveats (section 3.3). Collaborative Projects (CP), which involved investments from two or more Community Futures Development Corporations (CFDCs) and benefited one or more communities, had a mixed result, with some respondents feeling it had been successful and others feeling it had significant challenges.

Performance information indicates that the program was successful in building partnerships, supporting businesses and communities, creating and maintaining jobs, and contributing to diversification and competitiveness.

Interview respondents viewed the CFDCs as being effective in contributing to the rural economy and community by supporting small business development; leveraging funding from other programs; and working with partners to promote economic stability, growth and diversification. Moreover, several respondents noted that given EODP’s limited budget, the program had been extremely effective in achieving results. However, while these results can be seen at the community level, it is difficult to assess the program’s impact on eastern Ontario given the size of the eastern Ontario economy. In addition, interview respondents noted a number of unintended positive outcomes, as well as a few challenges resulting from the program.

Program Performance—Efficiency and Economy

The EODP (2011–14) delivery model was found to have been efficient and economical in producing outputs and progressing toward expected outcomes. The delivery model was considered appropriate, as it provided each CFDC, which has knowledge of the community’s needs, with the ability to respond to local priorities.

The demand for EODP funding was much higher than available funding, and program staff had a range of qualified applicants from which to select. Partnerships and leverage, including third-party delivery (i.e., the CFDCs), were found to be instrumental in expanding FedDev Ontario’s impact. The requirement that supported projects had to be completed by the end of the fiscal year and could not be carried over from one year to the next created problems and inefficiencies for the CFDCs. Further, administrative processes and templates have been developed by each CFDC independently, and there was considerable variability among the CFDCs.

Third-party delivery involved trusted partners (e.g., CFDCs) and was found to be both efficient and effective. The cost of delivering an EODP dollar was found to compare favourably with the overall delivery cost of grants and contributions programs by FedDev Ontario and by other regional development agencies. CFDC respondents reported a good working relationship with the FedDev Ontario project officers and were very happy with the services received. The delays in starting the program caused difficulties for the CFDCs; however, they successfully delivered the program in the time remaining in the three-year period.

There was a consensus among interview respondents that FedDev Ontario’s use of CFDCs to deliver EODP was the most cost-effective approach to achieving program results. It was further observed that there is no better, more cost-effective group of organizations that are lean, well established and fully connected in the community.

7.2 Recommendations

The following recommendations, based on the evaluation findings, are made for consideration by FedDev Ontario management.

Program Design

1. It is recommended that FedDev Ontario program management undertake the following:

  1. Investigate whether there is a better approach to planning and managing Collaborative Projects (i.e., regional projects), which involve funding from two or more CFDCs and benefit one or more communities, to address the concerns identified in this evaluation.
  2. Examine approaches to reduce the number of not-for-profit (NFP) planning studies for projects that do not have available funds to implement them.
  3. Assess the relative value and balance of small projects versus large projects, given the decision to encourage larger strategic projects in 2013–14 and the fact that most businesses in eastern Ontario are small (micro) businesses with one or very few employees and do not require larger funding amounts.
  4. Review the funding approach where each CFDC receives equal contribution funding to determine whether funding should be allocated on a different basis that reflects the needs and priorities of the geographic areas covered.

Third-Party Delivery

2. It is recommended that FedDev Ontario program management undertake the following:

  1. Revise the funding approach in which supported projects must be completed by the end of the fiscal year that they start in and allow carryover to accommodate late-starting and multi-year projects.
  2. Encourage the CFDCs to benefit from the various approaches (e.g., processes, templates, software) developed and used by the CFDCs across eastern Ontario for EODP management and administration (promotion, application processing, project evaluation and selection, etc.) by adopting the best practices and lessons learned, with the objective of improving efficiency. The Eastern Ontario CFDC Network could potentially lead this initiative.

Economic Research

3. It is recommended that FedDev Ontario management undertake a study of the socioeconomic situation in eastern Ontario to help determine the continuing need for a program similar to EODP as input to future program renewal. This study should be carried out after the 2016 Census information becomes available. This study could also include determining if there is merit in extending the program to areas with similar economic needs in southwestern Ontario.

Intermediate and Ultimate Outcomes

4. It is recommended that FedDev Ontario management include the following methodologies in future evaluations of EODP:

  1. Assess the longer term benefits to the project funding recipients. Also, investigate the role that EODP funding has in launching the supported projects; and determine the project funding recipients’ level of satisfaction and feedback in relation to the delivery of the program. This would likely include a survey of project funding recipients, interviews with CFDCs and project funding recipients, and case studies of supported projects.
  2. Assess the program’s impact on the ultimate outcomes, which would involve undertaking special analyses using Statistics Canada data, subject to data availability.

Selected References

  1. EODP Draft Evaluation Methodology Report, FedDev Ontario, 2015.
  2. Evaluation of the Community Futures Program, FedDev Ontario, 2014.
  3. EODP Performance Measurement Strategy (PMS), FedDev Ontario, 2012.
  4. Evaluation of the Eastern Ontario Development Program, FedDev Ontario, 2011.
  5. Interim Evaluation of FedDev Programs, Westbay Research Inc., October 2012.
  6. Labour Market Bulletin—Ontario: 2012 (Annual Edition), Statistics Canada.
  7. Labour Market Bulletin—Ontario: 2014 (Annual Edition), Statistics Canada.
  8. Eastern Ontario’s Economic Development Strategy, Miller Dickinson Blais, February 2014.
  9. 2014 Economic Scan of Eastern Ontario, McSweeney & Associates, October 2014.
  10. Federal Economic Development Agency for Southern Ontario, 2013–14 Report on Plans and Priorities, FedDev Ontario.
  11. Selected Documents on the Evaluation Function, Treasury Board Secretariat, 2009.
  12. Federal Government Budgets, 2010 to 2013.
  13. Analysis of the Federal–Provincial Economic Development Program Landscape, Intergovernmental and Stakeholder Relations, FedDev Ontario, June and September, 2012.
  14. Ontario Economic Update 2016, Ontario Chamber of Commerce.

Appendix A: 2011 EODP Evaluation: Recommendations, Management Responses and Actions Taken

Recommendation Management Response Reported to Audit and Evaluation Branch Responsible Official Revised Completion Date Action Taken (as of December 2015)
Recommendation #1: FedDev Ontario should examine whether it is necessary to have four agreements for the administration of the EODP funds and determine if all of the funds could be administered through one agreement. Agreed. FedDev Ontario will streamline the contribution agreement for the delivery of the next generation of EODP so that a single agreement per Community Futures Development Corporation (CFDC) is used to fund all program components. This approach is consistent with The Government of Canada Action Plan to Reform the Administration of Grant and Contribution Programs, with a focus on simplification of funding programs and agreements. Director, Community Economic Development Fall 2011 Addressed. The four agreements per CFDC were reduced to one agreement per CFDC for EODP (2011–14).
Recommendation #2: FedDev Ontario should review its current Performance Measurement Strategy (PMS) and refine its data collection to ensure it is appropriate for measuring the outcomes of the program. Including CFDCs and academics in the review and identification of indicators would be an effective way of ensuring the appropriateness of the indicators for measuring success. Agreed. FedDev Ontario will redesign the EODP PMS. The number of performance indicators will be reduced to simplify reporting and to ensure data collection is relevant and appropriate for measuring the outcomes of the program. These efforts will be informed by the work toward consolidation and integration of performance measurement strategies for programs across the Agency.
  • Director General, Strategic Policy
  • Director General, Innovation and Economic Development
  • Director General, Audit and Evaluation
Fall 2011 Addressed. The EODP (2011–14) PMS was redesigned. In addition to the program no longer including access to capital, the number of performance indicators was reduced from 99 to 45 to simplify reporting while ensuring that data collection was relevant and appropriate for measuring the program outcomes.
Recommendation #3: FedDev Ontario and CFDCs should ensure that a standardized, up-to-date list of recipients is maintained. This list should include complete information on the recipients (e.g., name, email address) as well as specific project identifiers (e.g., project name, project type). This information will be required should FedDev Ontario wish to survey its recipient population for future evaluations. Alternatively, FedDev Ontario could implement a post-project feedback form, which could be administered to recipients following the completion of the project. This form could collect information on the impacts of the project. The results could be entered into a database and be used for both ongoing management and evaluation purposes. Agreed. FedDev Ontario will explore project and client management technology that will assist with ongoing monitoring and tracking data. A software/database program option will be investigated that could allow real-time access to project data and recipient information. Director, Community Economic Development Fall 2011 Addressed. As part of the EODP (2011–14) PMS redesign, the performance measurements tools were updated to incorporate recipients’ contact information along with the project impacts. The data was collected from the CFDCs’ quarterly reports and incorporated into a master database.

FedDev is currently working with other regional development agencies (RDAs) on a common Enterprise Grants and Contributions Solution, with an anticipated release date in 2018.
The evaluation raises questions about whether the current model for the delivery of regional projects is appropriate.

Recommendation #4: Since this issue has been identified in previous evaluations, FedDev Ontario should examine alternative delivery options for regional projects to determine whether they could be delivered more efficiently.
Agreed. FedDev Ontario will review the experience related to the delivery of the EODP regional projects, with the objective of determining a more efficient approach for the delivery of this component of the program. This will include the elimination of a separate and unique process for regional projects. Director, Community Economic Development Fall 2011 Addressed. Regional projects were not included in EODP (2011–14); however, a new category of CFDC projects, Collaborative Projects, was introduced. The intention was that CFDCs could use their EODP funding to work together on joint projects.

Appendix B: Eligible EODP (2011–14) Recipient Activities

The following descriptions of eligible EODP (2011–14) project funding recipient activities were extracted from the signed contribution agreements (CAs) between FedDev Ontario and each Community Futures Development Corporation (CFDC). The three business lines, which are described below, are as follows:

Eligible project funding recipients included the following:

  1. Non-for-profit organizations, including municipalities and municipal organizations, corporations, and community development organizations;
  2. Commercial enterprises, including individuals, corporations, partnerships, cooperatives and trusts; and
  3. Indigenous organizations.

B.1.0 Business Development

The BD business line supports projects that will lead to the growth of new and existing businesses within rural eastern Ontario communities. This business line includes initiatives in key areas, such as skills development, youth attraction, innovation, and information and communications technology (ICT) adoption.

Eligible Activities

Internship and Job Placement Initiatives

Ineligible Activities

B.2.0 Community Innovation

The CI business line supports community-led economic development that enhances and diversifies local economies. Working in partnership with community stakeholders, CI builds the economic and innovative capacity of rural eastern Ontario communities, and develops opportunities for sustainable economic growth and employment.

Eligible Activities

Eligible activities are CI project activities that

Examples of Eligible Activities

B.3.0 Collaborative Projects

EODP projects that involve and benefit one or more communities are encouraged. The CP business line is expected to result in a greater number of strategic economic development initiatives and improved partnerships that benefit eastern Ontario communities, including the strengthening of linkages between urban and rural areas. The objective of these projects is to support initiatives that will stimulate BD and CI opportunities by promoting socioeconomic development leading to a competitive and diversified regional economy within eastern Ontario. These projects will involve investments from two or more CFDCs.

Eligible Activities

Appendix C: Program Management

The following describes the roles and responsibilities of FedDev Ontario and the Community Futures Development Corporations (CFDCs) in the delivery of EOFP (2011–14).

C.1.0 FedDev Ontario

FedDev Ontario was responsible for the overall management of EODP. FedDev Ontario EODP program officers worked with the CFDCs to support them in effectively administering the locally based initiatives. Specific responsibilities included the following:

C.2.0 Community Futures Development Corporations

CFDCs in eastern Ontario applied for and subsequently received EODP funding through their own contribution agreement (CA) with FedDev Ontario. In addition to the development of a business plan and application, specific roles and responsibilities were set out in the terms and conditions of the CA. Specific responsibilities included the following:

Appendix D: EODP Logic Model and Performance Indicators

The logic model developed for EODP is given in Exhibit D.1. The logic model shows the activities that were undertaken in the program, the outputs resulting from those activities, and the immediate, intermediate and ultimate outcomes each of the initiatives was intended to achieve.

In total, there were 11 outputs and outcomes and 45 performance indicators for the EODP.

Exhibit D.1: EODP Logic Model (2012)

Exhibit D.1: EODP Logic Model (2012)

Exhibit D.1: EODP Logic Model (2012) (the long description is located below the image)
Description of Exhibit D.1: EODP Logic Model (2012)

A logic model showing how various stages of the program work together. The top section is the labelled "Inputs" which includes three subsections. Subsection 1 contains a field labelled "O&M, Salaries and Wages, and Gs&Cs". Subsection 2 contains a field labelled "Development/Maintenance". Subsection 3 contains a field labelled "Program Delivery".

The "Inputs" section has an arrow pointing downward to the "Activities" section below it. This section also contains three subsections. Subsection 1 contains a field labelled "Program Development, planning, and program management". Subsection 2 contains a field labelled "Monitoring and providing financial support to recipients". Subsection 3 contains a field labelled "Providing recipient funding".

The "Activities" section has an arrow pointing downward to the "Outputs" section below it. This section also contains three subsections each of which are connected with a downward pointing arrow from the respective subsection from the "Activities" subsection above. Subsection 2 of this section is also connected with an arrow pointing from itself to subsection 3. Subsection 1 contains a field labelled "Policies, plans, reports and program tools, and communication resources". Subsection 2 contains a field labelled "Advice, information and support". Subsection 3 contains a field labelled "Contribution Agreements and funding key partners".

The "Outputs" section has an arrow pointing downward to the "Immediate Outcomes" section below it. This section contains two subsections both of which are connected with a downward pointing arrow from subsection 3 of the "Outputs" subsection above. Subsection 1 contains a field labelled "Improved business development". Subsection 2 contains a field labelled "Improved community innovation".

The "Immediate Outcomes" section has an arrow pointing downward to the "Intermediate Outcomes" section below it. This section contains three subsections which are all connected with downward pointing arrows from both the subsections in the "Immediate Outcomes" section above. Subsection 1 contains a field labelled "Employment is created or retained". Subsection 2 contains a field labelled "Business are created, retained, expanded or attracted". Subsection 3 contains a field labelled "Economic sectors are established or strengthened".

The "Intermediate Outcomes" section has an arrow pointing downward to the "Ultimate Outcomes" section below it. This section contains three subsections which are all connected with downward pointing arrows from all three subsections in the "Immediate Outcomes" section above. Subsection 1 contains a field labelled "Diversified and competitive rural eastern Ontario economy". Subsection 2 contains a field labelled "Economic stability, growth and job creation". Subsection 3 contains a field labelled "Sustainable, self-reliant communities".

Source: FedDev Ontario, Performance Measurement Strategy for the Eastern Ontario Development Program (January 2012), p. 8.

Exhibit D.2: Performance Measurement Matrix

Outputs
Logic Model Component Indicators* Source/Frequency
Policies, plans, reports and program tools, and communication resources
  • # of program policies developed
  • # of program tools and resources developed
FedDev files (Delivery unit) / ongoing
Advice, information and support
  • # of info sessions, workshops or training sessions held with primary recipients, communities and other stakeholders
  • # of participants attending info sessions, workshops or training sessions
  • # of monitoring interactions between FedDev and primary recipients
FedDev files (Delivery Unit) / survey ongoing
Contribution agreements and funding key partners
  • # and $ value of agreements with Community Futures Development Corporation (CFDCs)
  • # and $ value of agreements with other eligible recipients
FedDev files (Payment and Monitoring / GCRS) / ongoing
Immediate outcomes
Logic Model Component Indicators* Source/Frequency
Improved business development
  • # and $ value of business development projects
  • # and $ value of marketing or export plans developed
  • # and $ value of strategic plans developed
  • # and $ value of feasibility studies completed
  • # and $ value of business plans developed
  • Identification of new markets and new opportunities
  • $ value leveraged from other partners
FedDev files (Payment and Monitoring / GCRS/CEDD) / Recipient project reports / surveys ongoing
Improved community innovation
  • # of new staff and volunteers in community organizations dedicated to community economic development
  • # of community or organizational plans developed
  • # and $ value of local community economic development projects
  • Extent to which EODP youth internship projects contribute to community economic development
  • # and $ value leveraged from other partners
FedDev files (Payment and Monitoring / GCRS/CEDD) / Recipient project reports ongoing
Intermediate outcomes
Logic Model Component Indicators* Source/Frequency
Employment is created or retained
  • # of jobs created and retained as a result of EODP funding
  • # and % of interns with ongoing employment in rural eastern Ontario
  • # and % interns (youth or non-youth) interested in starting own business or working in small business environment
  • # of interns subsequently hired by local business
  • Extent to which EODP projects have led communities to identify or secure future labour needs
Recipient project reports / survey of clients / key informant interviews ongoing / annual review
Businesses are created, retained, expanded, or attracted
  • # of businesses created (incl those attracted), maintained or expanded as a result of EODP funding
  • # of projects contributing to the development of new or external markets
  • $ and % increased revenues due to EODP funding
  • $ and % new or start-up revenue due to EODP funding
  • $ and % of increased profits due to EODP funding
  • $ and % of increased export revenues due to EODP funding
  • Survival rate of business
Recipient project reports/ survey of CFDC loan recipients/ key informant interviews ongoing/annual review
Economic sectors are created or strengthened
  • # of new economic sectors active in community as a result of EODP funding
  • # and $ value of projects developing and deploying information and communications technology ICT (including knowledge-building projects)
  • # and $ value of projects contributing to innovation (e.g., new products or processes)
  • $ and % profit growth in existing economic sectors due to EODP funding
  • Extent to which EODP projects have led to improved learning opportunities with respect to economic diversification
FedDev files (CEDD)/ Recipient project reports/ surveys ongoing/annual review
Ultimate outcomes
Logic Model Component Indicators* Source/Frequency
Diversification and competitiveness of rural eastern Ontario economy
  • # of regional, sub-regional and local partnership initiatives in place to respond to barriers to rural development
  • Increased # of employed persons working in various industry sectors
  • GDP growth in various sectors of the economy
Key informant interviews / Census, Statistics Canada Labour Force Survey / 2–5 years
Economic stability/growth and job creation
  • Employment levels relative to other rural or urban communities
  • Increasing trends in earnings data
  • # of businesses and organizations increasing employment, payrolls and hours
Census/ Statistics Canada Labour Force Survey / Survey of Employment, Payrolls and Hours / 2–5 years
Sustainable, self-reliant communities
  • Steady or growing municipal tax base
  • Increasing per capita income
  • Increasing educational attainment of residents
Census/ Statistics Canada Labour Force Survey/ 2–5 years

* Indicators measuring the impact of EODP programming on the socioeconomic development of official language minority communities will be incorporated into CFDC project reports to be consistent with relevant policies and regulations of the Official Languages Act.

Note: CEDD, Community Economic Directorate; GCRS, Grants and Contributions Reporting System.

Source: FedDev Ontario, Performance Measurement Strategy for the Eastern Ontario Development Program (January 2012), p. 13.

Date modified: