Process analyses try to find out how the intervention worked, by collecting data on how the intervention is used (usage analysis) and assessing if usage is related to outcomes or user characteristics (process analysis). We have developed two key concepts to support usage analyses:

1. Effective engagement is the level of user engagement needed to bring about intended outcomes. Identifying the minimum usage necessary for an intervention to be effective permits the design of ‘lean’ interventions that quickly achieve users’ goals and cost less to maintain and update. 
   
2. Meaningful measures of usage. Most usage data analyses are based on quantity of engagement (number of logins, time spent, pages viewed) - but often what matters is what people engage with. 
   

We use the intervention’s programme theory to identify the intervention elements that we predict will be important to engage with (of course, this may be different for different groups of users). Then we test the theory by evaluating whether engagement with those elements does relate to better outcomes.

Additional data and analyses

It is useful to collect additional quantitative data that may explain how the intervention worked. By collecting data about user characteristics (for example, age, gender, ethnicity, education, preferences) we can identify whether sub-groups of people used the intervention differently or had different outcomes.  

Often, we ask users about their beliefs about carrying out the target behaviour, for example: whether they feel they are able to do the behaviour; and if they believe doing it will help them. By comparing their answers at the beginning and end of the study, we can understand what changed and whether this relates to better outcomes. 

It can be difficult to decide in advance what data will need to be collected when implementing and evaluating an intervention. Our framework for Analysing and Measuring Usage and Engagement Data (AMUSED) can help you to plan and carry out quantitative process evaluations more efficiently and systematically. 

The AMUsED Framework

The AMUSED framework helps you plan your analysis in advance, to ensure all the data you need will be captured and in a format that will make it easier to analyse during the optimisation and evaluation phases. 

The framework is made up of 3 stages, each containing a checklist with generic questions to help guide the intervention development team in selecting questions that are specific to the intervention.