Evaluation is a key part of developing evidence-based policy, yet it is not currently embedded in the entrepreneurship policy process. In our latest Debate in Public Policy, it was fascinating to hear from Dr Jonathan Potter, Head of the Entrepreneurship Policy Unit in the Centre for Entrepreneurship, SMEs, Regions and Cities (CFE) at the OECD and Visiting Professor at the CIMR on how the evaluation landscape in entrepreneurship policy can be improved.
Dr Potter began by examining the context in which this research is needed. Entrepreneurship policy represents a significant portion of expenditure among OECD countries, accounting for 0.6 percent of GDP in the UK in 2004 and 1.3 percent of GDP in Sweden in 2011 – that’s equivalent to expenditure on the police force.
However, many entrepreneurship programmes have not been evaluated, or have been subject to a very light-touch evaluation process, which makes it difficult to understand their impact and make evidence-based policy decisions.
Dr Potter indicated that OECD governments have called for an update of the OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes which he co-authored in 2007, so as to take account of the availability of improved evaluation methods and data.
He talked through the OECD work now underway to review the evidence from the small body of truly reliable entrepreneurship policy evaluations available internationally and identify their lessons for policy and evaluation.
There are two key areas for debate:
- Which types of entrepreneurship policy are effective?
- What needs to be improved in entrepreneurship policy research?
Criticisms of entrepreneurship policy
Acs et al. (2016) offer some helpful starting criticism on entrepreneurship policy:
- Policies waste taxpayer money by targeting the wrong groups, for example by providing additional encouragement to those already intent on becoming entrepreneurs, rather than motivating other groups to grow their business
- Entrepreneurship could be stimulated in better ways than by programmes targeting entrepreneurs, for example investment in STEM education to produce more engineers, some of whom will start high-tech businesses.
The bottom line is, in its current form, does entrepreneurship policy make an impact, and is it value for money?
Unfortunately, the quality of the bulk of available entrepreneurship policy evaluations makes it difficult to draw conclusions. In Gibbons et al.’s 2015 Review of Government Evaluations for the National Audit Office, for example, none of the entrepreneurship policy evaluations reviewed gave convincing evidence, compared to 70 percent of the labour market evaluations. Among the weaknesses cited were a lack of control group to compare the outcomes on participants and non-participants, or drawing the conclusions on the basis of possibly biased or ill-informed questionnaire responses from programme beneficiaries.
Meta-evaluation and a mixed picture from OECD countries
Meta-evaluations conducted on entrepreneurship policy impact present a mixed picture: some meta-evaluations have broadly positive findings but others are inconclusive about whether the policy has an impact, or find mixed results, with impacts on some metrics, such as networking or financing, but not others such as employment or productivity. There is a strong bias in these meta-evaluations towards assessing financial interventions, and it is difficult to obtain good information on the impact of advice and training.
Examining 51 evaluations of high technical standard from across OECD countries, only half of policy interventions had a clearly positive impact on the substantial weight of metrics. Again it is difficult to find high quality control-group based evaluations in certain areas, such as business advice, entrepreneurship skills and cluster policies.
There are a few possible explanations for this mixed result:
- As Acs et al. (2016) argue, intervention is only justified where there is a clear market failure
- Programme outcomes are affected by the national and regional context, such as boom and bust economies
- There are disparities in the details of programme design and delivery on the ground
- The entrepreneurs and enterprises targeted vary in terms of sector and growth ambition
The future of entrepreneurship policy
Further research is needed in order to develop future entrepreneurship policy. The analysis undertaken so far has been useful in identifying key points for further consideration:
- Hard vs soft policies – what should be the balance between financial support and training/advice?
- Selective vs non-selective programmes – which is more likely to lead to positive entrepreneurship outcomes?
- Macro vs micro policies – should the balance shift between developing a favourable business environment and supporting specific entrepreneurs and start-ups?
Dr Potter concluded by emphasising the need for greater scale, frequency, coverage and technical quality of evaluation to compare the impacts of different types of entrepreneurship policy and support a rebalancing towards the most effective approaches.
The presentation was followed by a lively audience debate led by discussant Dina Mansour, who challenged Dr Potter on how policy can be developed that applies to different national contexts, and how business leaders can be given agency in the policy-development process.
The recording of the workshop is available here:
