Evaluation Report 2013

Early in 2013, we commissioned an independent evaluation of the NSO from the Centre of Science Education at Sheffield Hallam University.

The aims of the evaluation where threefold:

  1. to evaluate the effect of the NSO on attitudes to STEM education and careers,
  2. to provide evidence to the strengths and weaknesses of the NSO to inform future developments,
  3. to provide a framework against which longer-term impact can be assessed in the future.

The full report is available here, but the executive summary was:

  • This evaluation was carried out by the Centre for Science Education and the Centre for Education and Inclusion Research at Sheffield Hallam University.
  • The main aim of the evaluation focussed on how the NSO influences attitudes to STEM subjects
  • To guide the evaluation a programme logic model was developed, along with a set of programme hypotheses that described the rationale behind the intervention design
  • A ‘pre- and post-use, with comparator/control groups’ model was not possible due to the post-involvement timing of the evaluation
  • A mixture of questionnaires and school visits were used to gather data. Questionnaires were developed for primary pupils, pupils aged 11-14, pupils aged 14-16 and post 16 students. A separate teacher questionnaire was also used. Response rates for some of the questionnaires were too low to do a meaningful analysis, however, sufficient were obtained from 14-16 pupils, and teachers, to process into this report. The online service Survey Monkey was used.
  • The pupil survey was completed by a group of young people who were more disposed to science than a more randomly selected group (based on comparisons with other surveys)
  • Pupil responses were highly positive about the impact of the NSO on their attitudes to science and their awareness of how scientists and astronomers worked
  • Teacher responses were also highly positive, and echoed the pupil responses regarding impact on enthusiasm and awareness of the work of scientists
  • Most teachers were physics graduates, and counting subsidiary subjects studies at university, around one third had studied astronomy
  • Use of the NSO was made in a wide variety of contexts, including in class, in clubs, as an after school activity, with individual pupils, with groups and classes, linked to the curriculum, as a lesson support tool, to support teacher professional development
  • The Case Study visits to six schools were also highly positive. Teachers made use of the NSO independently of any school STEM policy or programme
  • A key use of the NSO is to support GCSE Astronomy
  • Pupil awareness of STEM careers was increased through use of the NSO, in particular the breadth of jobs related to astronomy and space science
  • We found evidence of achievement of all the shorter term aims of the project
  • We found evidence that led us to suggest that the longer-term aims of the project (such as progression on to STEM subjects and careers) which were not possible to measure during the evaluation, were likely to be met
  • The programme hypotheses were supported by the evidence from the evaluation
  • We found the reach and significance of the NSO to be considerable. In our view, this initiative has unprecedented reach, and is one of the most significant educational initiatives in the STEM field linked to an HEI
  • We provide a series of recommendations and criteria for future development, covering user engagement, curriculum coverage, curriculum resources and activities, marketing and communications, training and web site functionality
  • A key recommendation is to establish a lead user group to guide future developments and provide a test-bed for new activities and resources.