Archived — Applied Research and Commercialization Initiative – Intake and Award

Archived information

Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.

Table of Contents


1.0 Executive Summary

1.1 Background and Context

The Applied Research and Commercialization Initiative (ARC) falls within the Southern Ontario Development Program (SODP), one of FedDev Ontario's key funding programs designed to build a strong economy for Southern Ontario.

The ARC Initiative is a pilot program that was developed and delivered in a relatively short time frame. Contribution agreements were signed with 24 post secondary institutions with potential funding of $14.5 million.

The audit of ARC Intake and Award was conducted in accordance with the Audit and Evaluation Branch's Risk-Based Audit Plan for 2011-12. It was identified as being of high priority due to the complexity of third party delivery systems and the timing sensitivity to provide assurance.

The objective of this audit is to provide assurance that governance, risk management and control over the intake and award processes of the ARC Initiative are being executed in compliance with policies and guidelines established for the Initiative, as well as the Treasury Board of Canada Secretariat (TBS) Policy on Transfer Payments (2008).

1.2 Overall Assessment / Audit Opinion

The audit opinion is based on an assessment of the conditions, as they existed at the time of the audit, against pre-established audit criteria that were agreed to with management.

Overall, we observed that the ARC project team was successful in developing the pilot ARC Initiative and conducting assessments of application eligibility under considerably tight timelines. ARC guidelines were established defining the eligibility, funding and application processes. An application review team was created with application review criteria, and contribution agreements were signed in a timely manner.

In our opinion, the ARC intake and award process is well controlled (See Annex C for detail of audit opinions). Recommendations identified in our report represent opportunities to improve processes and improve the control environment.

1.3 Main Observations

The main observations of the audit are:

1.4 Key Recommendations

  1. ARC Objectives
    ARC objectives should be further articulated in terms of specific outcomes and performance goals/targets should be established.
  2. Risk Assessment Process
    Management should prepare an initiative-level risk assessment that is specific to ARC and its unique objectives, and ties directly to the ARC logic model.
  3. Application Evaluation Process
    1. Application guidelines should further define the acceptable form and format of applications;
    2. Criteria statements should be defined in more specific and measurable terms;
    3. Metrics should be developed that would enable a standard rating of PSI applications.
  4. Application File Documentation
    The application screening checklist should represent a summary of all application analysis performed, and be updated to reflect all information collected.

1.5 Statement of Assurance

The audit was carried out in accordance with the Treasury Board of Canada Secretariat (TBS) Policy on Internal Audit (2006) and the Internal Auditing Standards for the Government of Canada (2006). These standards require that the audit be planned and performed so as to obtain reasonable assurance that the existing management control framework ensures compliance with the Policy on Transfer Payments (2008) and other relevant policies and guidelines.

Based on our professional judgement as auditors, sufficient and appropriate audit procedures have been conducted in the audit of ARC Intake and Award. The evidence has been gathered to provide senior management with reasonable assurance of the accuracy of the conclusions drawn from this audit.

This report and audit was conducted for FedDev Ontario management purposes. Use of this report for other purposes may not be appropriate.

top of page

2.0 About the Audit

2.1 Background on the ARC Initiative

The Applied Research and Commercialization Initiative (ARC) was launched April 19, 2010 as a pilot initiative under the Southern Ontario Development Program (SODP) aimed at encouraging greater collaboration and partnerships between Post-Secondary Institutions (PSI's) with applied research development expertise and Small and Medium Enterprises (SME's) with pre-commercialization needs. The ARC Initiative's objectives were to (1) accelerate innovation and to improve productivity and competitiveness in SME's located in southern Ontario, and (2) expand capacity for PSI's to work with SME's.

Applications from PSI's were evaluated on their ability to meet the following criteria:

PSI's were given from the April launch until June 14, 2010 to submit applications for FedDev Ontario review. The ARC project team was successful at developing and delivering the intake and award process within a very short time frame.

Following a competitive application process, 24 southern Ontario post-secondary institutions received $14.5 million in contribution funding under the ARC Initiative. Funding for each institution ranged from $155,000 to $750,000, with up to five per cent permitted for outreach costs.

2.2 Audit Objective

The objective of this audit is to provide assurance that governance, risk management and control over the intake and award processes of the ARC Initiative, are being executed in compliance with policies and guidelines established for the Initiative, as well as the Treasury Board of Canada Secretariat (TBS) Policy on Transfer Payments (2008).

The overall audit objective is addressed through the following sub-objectives:

Governance

To ensure adequate mechanisms exist for oversight, timely and informed decisions, clear accountabilities, effective communication, appropriate skills and capacity.

Risk Management

To ensure that risk management processes are in place to identify and manage program risks.

Compliance

To ensure that intake and selection for the ARC Initiative is administered in accordance with applicable laws, regulations, guidelines and policies.

Detailed audit criteria related to these objectives are summarized in Annex A. Audit criteria represent the expected state of processes and controls.

2.3 Audit Scope

The audit scope includes all ARC activities leading up to and including the recommendation of PSI applications. This includes program development, promotion, applicant assessments, and recommendation. The scope includes all PSI applications received and assessed.

2.4 Audit Approach

The following key audit steps were performed:

Documentation Review

The audit involved a review of the ARC Initiative's management control framework including program documentation, policies, procedures and governing documents.

Interviews

Interviews were conducted with the following staff:

File Review

A random sample of 15 application and assessment files were selected for review including 11 funded and 4 non-funded applications.

The audit observations which follow result from a comparison of the desired criteria (see Annex A) to the actual conditions observed.

top of page

3.0 Observations and Recommendations

Internal Audit has conducted a review of the ARC intake and award process examining the management control framework and comparing conditions against pre-established audit criteria.

Overall, the ARC intake and award process was successful considering that it was a pilot initiative and under tight timelines. The following observations represent opportunities to improve processes and improve the control environment.

3.1 ARC Objectives

Criteria: Guidelines and procedures are adequately defined, including objectives for the ARC program.

Observation: ARC objectives remain at a high level and could be further articulated to more easily enable measurement of success and evaluation of the initiative. In addition, performance targets for these objectives have not been established that would define success in practical terms.

ARC’s objectives include:

From discussions with program management, the objective of expanding capacity for PSI’s to work with companies, although implied, has not been clearly articulated and documented as an ARC objective.

Impact: The initiative’s success becomes difficult to measure and evaluate when objectives have not been fully articulated and goals/targets have not been identified.

Recommendation: ARC objectives should be further articulated in terms of specific outcomes and performance goals/targets should be established.

3.2 Risk Assessment Process

Criteria: Management identifies the risks that may preclude the achievement of its objectives, assesses the risks, and formally responds to the risks.

Observation: Although a risk assessment had been prepared at the program-level for SODP, one had not been documented specifically for ARC. Management has indicated that risk management was a key consideration in the development and delivery of ARC intake. All the initiatives under SODP have unique characteristics, and a risk assessment for the ARC Initiative could be a useful tool to manage risks more effectively.

Impact: Unmitigated risks could result in ARC objectives not being met.

Recommendation: Management should prepare an initiative-level risk assessment that is specific to ARC and its unique objectives, and ties directly to the ARC logic model.

3.3 Application Evaluation Process

Criteria: The application evaluation process uses criteria that are definable, measureable and enables the achievement of ARC objectives.

Observation: The application evaluation process could be strengthened to improve the consistency and depth of analysis performed on applications, as follows:

Impact: A stronger application evaluation process will enhance the department’s ability to deliver funding in a fair, transparent and objective manner consistent with TBS policy.

Recommendation: Opportunities for improvement to application evaluation process include:

  1. Application guidelines should further define the acceptable form and format of applications;
  2. Criteria statements should be defined in more specific and measurable terms;
  3. Metrics should be developed that would enable standardized rating of PSI applications.

3.4 Application File Documentation

Criteria: Project decisions on application eligibility are fully documented and provide an audit trail.

Observation: Some improvements could be made to provide a more consistent audit trail linking application recommendations to supporting documentation.

The Project Summary Form (PSF) is the application evaluation document that is submitted to the Review Committee to recommend funding for an application. It is normally supported by an application screening checklist demonstrating analysis performed.

We found instances (4 of the 15 samples) where assessments made on the PSF were not aligned with those on the screening checklist. While supporting documentation was available, the screening checklist had not been updated to reflect all information gathered.

Impact: An incomplete audit trail reduces the ability to display fairness and transparency in the selection process.

Recommendation: The application screening checklist should be updated to reflect all information collected and represent a summary of all application analysis performed.

top of page

Supporting Annexes


Annex A: Audit Objectives and Criteria

Three audit objectives and criteria which include management control framework, risk management, and internal control
Objective 1: To ensure adequate mechanisms are in place for oversight, timely and informed decisions, clear accountabilities, effective communication, appropriate skills and capacity.
1 – Guidelines and procedures are adequately defined, including objectives for the ARC program.
2 – Roles, responsibilities, authority and accountability are defined and documented.
3 – The organization provides employees with the necessary training, tools, resources and information to support the discharge of their responsibilities.
4 – An oversight body has been created with a clearly communicated mandate
5 – Formal communication processes/mechanisms exist and support sharing of timely, relevant and reliable information to users and other external stakeholders.
Objective 2: Ensure that risk management processes are in place to identify and manage program risks.
1 – Management has a documented approach to risk management.
2 – Management identifies the risks that may preclude the achievement of its objectives, assesses the risks, and formally responds to the risks.
3 – Effective quality assurance process is in place to review ARC decisions and analysis.
Objective 3: To ensure that intake and selection for the ARC program is administered in accordance with applicable laws, regulations and policies.
1 – The application evaluation process uses criteria that are definable, measureable and enables the achievement of ARC objectives.
2 – All information critical to project decisions is accurately documented.
3 – Application and approval timelines have been established and adhered to.
4 – Project decisions on application eligibility are fully documented and provide an audit trail.
5 – A Contribution Agreement is on file and is consistent with the objectives and criteria of the ARC program.
6 – Contribution agreements were approved by the proper level of delegated authority and signed in a timely manner.

Annex B: Management Action Plan

management action plan regarding travel claims
Recommendation Planned Action or Justification for No Action Responsible Official Target Completion Date
1. ARC objectives should be further articulated in terms of specific outcomes and performance goals/targets should be established.

Agreed. A Performance Measurement Matrix (PMM) was developed in 2010 for ARC to correspond with the Southern Ontario Development Program (SODP) Performance Management Strategy (PMS). Specific outcomes, indicators and targets were described in the PMM, and reporting requirements corresponding to these measures were included in the Contribution Agreements (CA) with Recipients signed in 2010.

The PMS was updated in 2011 and is currently undergoing Agency approval. Any changes as a result of the PMS updates will be reflected in the amended CAs with Recipients under the ARC extension.

Taras Hollyer, Director, Program Policy and Innovation March 31, 2012
2. Management should prepare an initiative-level risk assessment that is specific to ARC and its unique objectives, and ties directly to the ARC logic model.

No Action. Although Treasury Board does not require initiative-level risk assessments, the Agency is developing a corporate risk assessment that includes a focus on programs. This risk assessment, once approved, will be incorporated into future ARC intake assessment processes.

Individual applications to the original ARC uptake in 2010 were assessed for risk even though a formal assessment document was not finalized by the Agency at that time. This risk assessment focused on the Applicant’s ability to administer and manage their proposed project, including project oversight, expenditure of funds, and reporting. Applications under the ARC extension will be assessed in the same way.

n/a n/a
3. Opportunities for improvement to application evaluation process include:
Application guidelines should further define the acceptable form and format of applications;
Criteria statements should be defined in more specific and measurable terms;
Metrics should be developed that would enable a standard rating of PSI applications.
i. Agreed. The initial ARC intake provided valuable insight into the application process. The ARC extension has already incorporated these insights into a standardized, streamlined application with specific fields that correspond with assessment criteria.
ii. Agreed. The initial ARC intake provided valuable insight into the application process. The ARC extension has already incorporated these insights and specific criteria statements have been included in the documentation provided to potential applicants. These criteria statements will be used as a foundation to assess each application going forward.
iii. Agreed. The criteria statements referred to in ii) above have already been incorporated into the ARC extension application assessment process. These criteria will have metrics associated with them, with weighting of risk as described in Recommendation #2 above. These metrics, though more standardized, will not be so rigid as to cause the Agency to conduct an erroneous assessment, as every southern Ontario Post-Secondary Institution is structured differently.
Taras Hollyer, Director, Program Policy and Innovation March 31, 2012
4. The application screening checklist should be updated to reflect all information collected and represent a summary of all application analysis performed.

Agreed. The screening checklist will be re-structured to focus on eligibility screening. The remaining parts of this document will be separated and used as a working tool for analysts to aid them in their assessment and to record "notes to file" related to the application.

The Project Summary Form (PSF) is the official write-up of the project, and contains the standardized assessment and funding recommendation for of the project. Future assessments will incorporate a process to ensure that the restructured checklist reflects the information found in the PSF.

Taras Hollyer, Director, Program Policy and Innovation To coincide with a second open intake for ARC (pending successful evaluation 12 2012.)

Annex C: Audit Opinion and Statement of Assurance

The auditor’s opinion is based on logical and specific deductions about the audit objectives based on observations against criteria and noteworthy accomplishments in the report. Standardized language is needed in order to ensure consistency in the formulation of an opinion across different audit areas and different time periods. In addition, it allows the reader to better understand the opinion being expressed.

The options for audit opinion that can be formulated:

Option 1: Well controlled. Denotes strong controls in all key areas (at least one of the first two criteria needs to be met and the third criterion needs to be met):

Option 2: Satisfactory. Denotes basic controls are effective in critical areas (At least one of the following two criteria need to be met):

Option 3: Marginal. Some key areas lack controls (Requires improvements in at least one of the following areas):

Option 4: Requires Significant Improvement. Situations where pervasive material weaknesses exist in overall internal control structure, which fails to provide management with reasonable assurance that the following management objectives are being met:

Statement of Assurance

The opinion is supported by a clear indication to users of the level of assurance being provided. Users need to be informed of the auditor’s judgement about the confidence that may be placed in the opinion above. The Statement of Assurance, as noted in the Executive Summary, informs the reader that the auditor has conducted the engagement in accordance with the Treasury Board Policy on Internal Audit.

This section also explains that the auditor has examined sufficient, relevant evidence and obtained sufficient information and explanations to provide a reasonable level of assurance on the reported opinion or conclusions. Please note that not all assurance engagements are designed to provide, or result in providing, a reasonable level of assurance. When this is the case, the auditor’s statement of assurance shall identify the circumstances (e.g. limited procedures, inconclusive evidence, etc.) that the level of assurance and shall indicate the auditor’s caution on reliance on these opinions or conclusions.

Date modified: