Barnaby Peacocke responds to Jayantha: using SoC/MSC to build baselines and the problems of analysing SoC with a historical perspective

I'm discussing this with Jayantha, but thought I'd share you some initial thoughts straight away.

Essentially the question Jayantha is raising is about the validity of collecting stories at the onset of a project

Myself I think this is a good approach - equivalent to a kind of narrative baseline. Instead of asking "over the last month, what have been the most significant changes..." the question is something like, "as things stand now, what are the most significant issues you are facing in relation to...[dimension of change]".

[Example:]
So for instance, for coir producers, "what are the most significant influences affecting your abilties to produce and market coir". From a range of stories shared by coir producers they could then select the one or two they feel best reresent the situation as it is today. It then becomes interesting to compare their selected stories over the next couple of years against their current perceptions

This in effect supports Jayantha's first bullet point - asking whether it is ok to collect stories where we expect future changes to happen as a resul of certain activities

Concerning documenting stories that look back to what was happening some time ago - I have done this kind of thing in field assessments asking people about how they perceive changes in things like production and rainfall patterns compared with 10 years ago or so; this timeline approach in participatory rural appraisal is quite common. But this is risky. We have to be careful not to overinterpret the data/stories we get. There are many biases that will confuse people's perceptions of change when looking back historically.

[Example:]
People who are facing current difficulties as a result, say, of economic downturns may share stories of the past that are in fact too rosey. "Oh the rains were much better then" - looking back to the good old days which actually never existed - so when you look at historical rainfall patterns there's actually been no change, it's the prices that slumped. Or vice versa.

So while the approach may be useful to see what people were doing in the past I'd suggest if anyone does do this they use the same principles of triangulation as recommended in PRA; for instance by getting historical stories from other stakeholders (market middle men, ministry and other actors) with which to compare (and triangulate) historical stories from producers.

Overall therefore you can see I am much less keen on this approach. I feel much more comfortable if we are asking for stories about the current situation and/or recent changes. Otherwise we do risk opening up a bit of a methodological minefield for ourselves

Best wishes
Barney

Lucho responds to Jayantha: Some suggestions (HELP! ANY IDEAS?)

Dear Jayantha:
Your question is very important.
I do not really know The Answer to it but I can reflect with you using the elements I have learned from Davies and Dart (a link to the doc is in the blog) and some common sense.

According to these authors (page 11), The kernel of the MSC process is a question along the lines of ‘Looking back over the last month, what do you think was the most significant change in [particular domain of change]?’

My first reflection is that even short times of interaction with the community can have impacts that we could document using SoCs. I would say that if you want to stick to the coir subsector it seems feasible to attempt an exploration of impacts related, for example, to the PMCA. If your team has already held PMCA workshops with coir producers and other market actors you could try to identify changes produced by these workshops. It does not matter if at the end you find no changes... but what if you do? It would be fascinating to understand how the initial PMCA workshops produce changes in the domains of change defined by your team (a discussion on domains of change can be found in the blog http://socmsc.blogspot.com/).

My second reflection is that if you feel that the coir process is still very raw, you can focus on a project that is a bit more advanced. No problem with that. I know that for the AMAP each team selected a subsector to work with but at this point I am more interested in allowing ourselves to learn how to use SoC/MSC than on the subsectoral focus.

Finally, I notice that you use the term 'story' in singular. This could be a language issue but I want to make sure that the concept is fully understood. The plurality of stories is fundamental to this technique. The MSCs are identified through a process of deliberative selection of many SoC at different levels of the organisation.

That is all for now.
Please come back to us with your reflections and thanks a lot for this important contribution.
Hugs,
Lucho

PS:
Barnaby... can you discuss this issues with Jaya when you meet with him please?
Could you please report your reflections to all of us (via email) or directly to the blog (even better)?

Jayantha: CAN WE USE SoCs FOR PROJECTS THAT ARE JUST BEGINNING?

Hi Lucho,

A small clarification

We have not made any interventions (apart from the PMCA and coir sector development strategic planning which is not implemented in full scale yet). Even if we made any intervention, we cannot expect a significant change during this short period. In this situation I am afraid whether we can document a story of change.

My question;
As we discussed during the int’l meeting are we going to start recording a story that we expect some changes in the long run (after two or three years) or
Can we document a story that initiated 2-3 years back? In this case not coir but paddy or other non farm enterprises

Regs

Jayantha

Jayantha Gunasekera
Team Leader - Markets and Livelihoods
Practical Action
Sri Lanka

The latest development: DEFINING THE DOMAINS

The following thoughts are based on the latest comments from Barnaby and Marshall (I copy them below):

Barnaby Peacocke said:
A. On the second domain of change the question about 'perceptions' leaves it very open to interpretation by those telling their stories as to what changes are relevant to the domain. The first domain is much clearer here. This isn't necessarily a bad thing but needs an eye kept on exactly what we want to learn in this domainB. Similarly the third domain about technologies is very much dependant on different interpretations as to what technologies are - are they just bits of kit, or do they include skills, knowledge, methods etc?

And Marshall Shumba from Zimbabwe suggested:
I think it is usefull to use domains but these domains should be built with markets chain actors. This will enable us to capture or cater for the various stories of change from different actors in the chain.

So... to keep the conversation moving, I suggest the following:

The domain of

• 
  Changes in the capacity of marginalised producers to engage with other market actors
  

seems to be clear.

I would say that, in order to make it clearer, examples of types of capacity to engage can be:

- Access to market information.
- Possibility to talk (face-to-face or on the phone or via email, etc) with other market actors (including politicians and other policy-makers).
- Possibility to organise meetings where they are the ones who invite other market actors (It is important to see how the level of attendance and interest from other market actors to participate in the meetings organised by farmers change through time).
- Access to spaces where policies are discussed.

The domain of

• 
  Changes in the ways in which marginalised producers perceive themselves as market actors.

seems to be less clear (according to Barney).

I do not agree totally with Barnaby, in the sense that self-perceptions are broad but once you specify that the self-perception we are interested in is as a market actor, it becomes more specific. For example:
- The person can feel that at the beginning of the project, he/she was feeling as a subsistence farmer and later on s/he feels that s/he can begin a small-enterprise; or s/he feels that s/he wants to move away from farming because s/he sees now as a trainer of farmers and the s/he can make a living out of it. etc.

On the domain of

• 
  Changes in the access of marginalised producers to technologies that contribute to their participation in a particular market system.

I do agree with Barnaby, I think that we can think of technology in a broad sense; i.e. hardware (machines and ICTs), software (techniques, knowledge, technocal info, procedures, standards, etc) and other technology-related inputs such as improved seeds and fertilizers.

This domain of change responds to the need we have to keep an eye on technology dynamics.

Concerning marshall's comment, Davies and Dart suggest that it is OK to allow participants to define domains of change. However, I think it is important to maintain the number of domains as small as possible. Otherwise, the excercise can become practically impossible to undertake in a reasonable time. It is also important to keep in mind that collecting the stories is time-consuming and that it might be very difficult to collect them form market actors who do not belong to the community of farmers.

To Marshall (and the other project managers), I have three suggestions:

1. You can allow the participants to define all the domains of change before hand and allow them to tell their stories within those domains.
2. You can simply collect the stories without suggesting any domain and then, when we analyse the stories, we can see if there are domains that emerge from the stories collected
3. You can use the domains we define here amongst us (three so far) with one domain that can be defined by the farmers (I suggest maximum one because experience seems to show that more than 4 domains make the excercise very time-consuming).

Each project manager can have a different approach. It will be interesting to contrast the results at the end.

I need your ideas. What do you all think? Marshall?

Domains recommended by Lucho

If you think that domains are a good idea, then I would suggest you use the following:

- Domains of change defined by the Markets and Livelihoods Programme strategy:

• Changes in the capacity of marginalised producers to engage with other market actors.
• Changes in the ways in which marginalised producers perceive themselves as market actors.

- Domains of change defined by Practical Action's strategy:

• Changes in the access of marginalised producers to technologies that contribute to their participation in a particular market system.

When you are asking people about these domains, I recommend that you always bear in mind the three dimensions of the market map and dig deep around them:

• The market chain: set of steps that bring a product to the final consumer.
• The service providers: actors who provide services and inputs (e.g. seeds, fertilisers and tools) to market chain actors.
• The business environment: the issues and factors that affect how market chain actors and service providers interact and make business (e.g. legislation, roads, telecommunications, physical environment, and business culture).

What do you think?

If you decided to use 'domains', which are the most appropriate?

This is a difficult question. Let's see what Davies and Dart (2005, pages 20 and 21) suggest...

What should domains focus on?

One question that often arises is whether the domains of change should only be about changes caused by the organisation that is using MSC, or include changes caused by other people, organisations or influences in general. For example, increased participation by individuals could result from changes in government legislation relating to the right to free association rather than anything to do with the organisation’s own activities. Nevertheless, in most societies, such changes would be considered significant.

In practice, most users of MSC have focused on changes that are a direct or indirect result of what their organisations are doing. There is, however, an argument for saying that many organisations already see the world too narrowly, and that it would be healthy to identify SCs arising from any cause.

These options do not need to be mutually exclusive. It should be possible to track both types of changes through the one application of MSC. One or more domains could be about changes caused by the organisation’s work, while another could ask specifically about changes not caused or influenced by the organisation. Doing so would help provide what Chris Roche (1999) has described as a ‘project out’ and a ‘context in’ perspective.

Who should be involved in determining domains?

In some organisations, existing commitments to the pursuit of specific objectives are likely to lead to the use of domains based on program objectives. Hopefully they will already be well-known and owned by the staff and clients. Where there is more freedom to select the domains, using a participatory process to identify appropriate domains is likely to encourage all participants to take a more active interest in the MSC process and its products.


PROPOSED QUESTIONS FOR REFLECTION:

• Do you think we should also look at changes generated by factors outside of our projects?
• Do you think that allowing key market actors to participate in the definition of the domains will encourage them to 'take a more active interest in the MSC process and its products'?

Step 2: Defining domains of change. (KEY POINTS)

Davis and Dart (2005, pages 17-22) say that domains are broad and often fuzzy categories of possible SC stories. In CCDB, participants in MSC were asked to look for significant changes in four domains:

• changes in the quality of people’s lives
• changes in the nature of people’s participation in development activities
• changes in the sustainability of people’s organisations and activities
• any other changes.

A domain of change is not an indicator. Good indicators are supposed to be SMART (specific, measurable, achievable, relevant and time-bound). Indicators must be defined so that everyone interprets them in the same way. Domains of change, on the other hand, are deliberately fuzzy to allow people to have different interpretations of what constitutes a change in that area.

How many domains should be used?

In our experience, between three and five domains is a manageable number. The limiting factor is how much time participants are willing to spend in discussing each domain. Participants may find the process too time-consuming if it takes more than two hours to review changes in all the domains in any one meeting.

Are domains essential?

Domains are not essential. MSC stories can be collected and analysed as a group without first being categorised into domains. Participants can be asked to go out and look for significant changes without being given guidance in the form of specific domains of concern. With smaller organisations where there are likely to be fewer SC stories to examine, the MSC approach will probably be easier without domains.

In organisations such as VSO, field staff are asked to identify and document MSC stories of any kind. It is only when the stories reach the country office level that they are categorised into domains that are of concern to the country office and to VSO headquarters in UK.

Letting middle and upper level staff within an organisation categorise MSC stories into domains produces some incidental benefits. If the domains are focused on organisational objectives, then the sorting decisions tell the rest of the organisation how those staff interpret the meaning of those objectives.

PROPOSED QUESTIONS FOR DISCUSSION:

• Should we use 'domains' or not?
• If yes, which 'domains' are appropriate for the Markets and Livelihoods Programme?

Step 1: How to start and raise interest. (KEY POINTS)

We have been using SoC in Practical Action for many years now, and despite the fact that we are not using them properly (see summary of 2003 Evaluation), we can assume that project managers are already managing to raise the interest of several people in telling their stories. However, it is always good to take some advice into account (from Davies and Dart 2005, pages 15-17)

• 'If you want to raise interest in MSC, you need to be clear about the purpose of MSC and the role it will play in your organisation'.

So, what is the purpose of SoC and MSC for the Markets and Livelihoods Programme?

I think it is a good way to open the team to the voices of other market actors (especially marginalised producers) AND to become more responsive during the implementation of our projects.

And what is the role that SoC/MSC will play in the organisation?

I think it will become one of our most powerful tools to learn and communcate with market actors (especially marginalised producers).

• 'Another really important lesson we have learned from experience is to start small'.

We will start small and step by step. One market system at a time in each of the countries where IA2 works.

PROPOSED QUESTIONS FOR REFLECTION:

• Do you think you will have problems to raise the interest of marginalised producers to tell you their stories? Why?

An evaluation of PA's experiences: How were things in 2003 in terms of SoC?

Summary evaluation of 2003 experiences

In introducing Stories of Change in 2003 we failed to distinguish their difference from human-interest stories. As a result, staff viewed Stories as just a more systematic way of reporting qualitatively on the (positive) aspects of the project. They often neglected to involve stakeholders and to explain why the story was significant, or what would be done differently in future to achieve the same successes. This learning is crucial to organizational learning.

While project staff identified with the need for a method like Stories of Change, it is not yet prioritised within the project. It is an initiative being pushed from QA with little reinforcement from the project managers or Aim Leaders. Having the QA and project managers reintroduce the method together might help staff to invest more effort into working with beneficiaries or partners to identify the most significant Stories of Change. Stories need a champion at the highest levels of a project and programme.

Practical training at the field level should make a real impact on the uptake of Stories as a core approach to evaluating impacts. Introducing the method through the Annual Review and expecting it to disseminate down meaningfully to project staff was unrealistic and counter to the bottom-up nature of the approach. No amount of written guidelines can substitute for going through the process with field staff and with partners. Practical training should help staff see the value of sharing stories - good and bad - as opposed to just passing them up the line, as the Annual Project Review has done in the past.

In the 2003 Review, it was found that a majority of projects responded to this section with ‘stories of success’ often affecting ‘communities of people‘ with no clear identification with the partner or pointer to the lessons learned. It is felt that ‘Domains of Change’ could help focus Stories of Change on a particular set of ‘relevant’ issues that were largely missed by he 2003 Annual Project Reviews. 2004 includes a domain for stories of disappointment. These are encouraged since they help project staff to make critical Action Recommendations for the year ahead.This could compensate for a tendency of staff to see M&E, the Annual Project Review and Stories of Change as important only for meeting accountability requirements. What we need to achieve is a constant feedback into lesson learning and better implementation approaches.

The steps we will follow

Dart (page 3, undated) suggests the following three steps:

A • establish domains of change
B • set in place a process to collect and review stories of change
C • conduct a secondary analysis of the stories.

I believe that these are easy to remember and we can stick to them. However, Davies and Dart (2005, page 15) propose a more detailed list of steps, which include the three steps mentioned above (numbers between brackets indicate the pages where you can read about the step)

1. Getting started: establishing champions and getting familiar with the approach (15-17)
2. Establishing ‘domains of change’ (17-22)
3. Defining the reporting period (22-23)
4. Collecting stories of change (23-28)
5. Reviewing the stories within the organisational hierarchy (28-34)
6. Providing stakeholders with regular feedback about the review process (34-35)
7. Setting in place a process to verify the stories if necessary (36-38)
8. Quantification (38)
9. Conducting secondary analysis of the stories en masse (38-44)
10. Revising the MSC process (44-45)

I think we can use both lists; the first one as a broad guideline and the second one as a more detailed 'protocol' to implement the process in the field.

Jessica J. Dart: the importance of transparency

'In the MSC approach, program stakeholders interpret their experiences with the program and select instances of significant change and record each as a story. They are also required to record why this change is significant to them. For example, when a farmer tells a story of significant change, she/he interact with the world and draw meaning from it, and it is in the telling of the story that meaning is constructed. Then when the reviewers read and evaluate the story, they engage with it and construct a further new meaning. When this is done in a group, this construction may be shared. In the MSC approach the criteria that are used to interpret the story are documented, made transparent and attached to the story itself. It is this transparency that makes the whole process even more open to new and more sophisticated constructions of meaning'. (quoted from Dart, undated, page 2)

How will we proceed?

OK. Very nice. It sounds good, doesn't it?

Now, how do we test the potential of SoC and MSC?

Here below there are some suggestions in order to move ahead:

1. Focus:

Between March and June 2007, the International Markets and Livelihoods Programme (IA2) will go through an intensive period of experimentation and reflection about the use of SoC and MSC to learn from marginalised producers.

2. Create a community of practice around this process:

This blog is part of this idea, but emails and telephone calls and conferences will also be used to maintain communication.

Additional to the IA2 team leaders in the country offices and the staff in the UK, it is important that project managers engage in this learning process as well. In fact, it would be great if they become the protagonists!

IA2 staff will be invited to become co-editors of the blog. Special co-editors from other organisations will also be invited.

3. Support:

Provide timely and high quality support to project managers before, during and after they have collected the SoC and identify the MSCs.

Barnaby Peacocke (our QA Manager and the internal champion of the idea of SoC) will be intensively involved with us providing support to the team leaders and project managers. In fact, he will be visiting the Sri Lanka and Sudan offices soon and will be providing hands-on support to the TLs and PMs.

4. Clear calendar of activities:

You can check the updated calendar here.

5. Production of a document to share with our experiences others:

By June, we hope to produce a short doc with some of the key lessons and insights from the process.

The AMAP-BELO programme and where does SoC/MSC fit in?

Under the AMAP programme, USAID, with the collaboration of Pact Inc., is supporting our efforts to become a more effective 'learning team'. The project begun in 2006 and will end in Nov 2007 but it is already helping us build learning processes and tools that will remain active for the foreseeable future.

The AMAP-BELO programme is helping CARE, Freedom from Hunger, WOCCU and Practical Action become effective learning organisations (there is where the accronym BELO comes from). BELO's conception is based on iterative (or repetivite) learning cycles... learning never stops; it revolves and expands like a spiral.

The design of Practical Action's project, called Practical Knowledge: Learning With and For The Poor, is based on four of these iterative learning cycles or loops:

1. The loop within the Markets and Livelihoods Programme (IA2).
2. The loop within Market Systems.
3. The loop between Mkt Sys (mainly marginalised producers) and the rest of the organisation.
4. The loop between IA2 and other organisations working in pro-poor market development.

So, where do SoC and MSC fit?

They fit in loop #3.

We want to test the potential of well-applied SoC and MSC to make the organisation more responsive to the voices of marginalised producers. We expect that these voices or stories will affect the project cycles, the annual business plans, the country/regional strategies and even (hopefully) our five-year strategies! We will see.

A final comment:

Remember the project of our project? Practical Knowledge: Learning With and For The Poor.

The differences in prepositions are important here; not only a nice name. Extracting knowledge from the marginalised has been a practice that, unfortunately, I have seen in many places. There is no sense of joint learning and no responsibility to feedback what we have learned from them.

'Learning WITH' makes us aware of the importance of a joint learning process: learning hand-in-hand with the others. 'Learning FOR' makes us aware of the responsibility we have towards marginalised communities to build knowledge also for them (downward accountability), not only for our line managers or the donors.

I hope we all, as a team, remember this along the way...

Introduction (where to begin?)

Hi everybody!

Hmmm... ok. Let's see... Here I am, trying to begin a blog to help us to use Stories of Change and the Most Significant Change technique to improve our initiatives to making market systems work better for marginalised producers.

SoC are simple narratives where people tell us that there have been changes in their lives related or due to the projects we are implementing. they can be positive or negative changes, but be prepared to hear that they feel no changes at all!

SoC can be a powerful and versatile tool to monitor and adjust the course of our initiatives. They are also a manifestation of the voices of the people who really matter to us (mainly marginalised producers in developing countries). The Markets and Livelihoods Programme is committed to listen carefully to those 'storytellers', learn from them and let them know how their stories help us to improve.

On the other hand, MSC is a technique that allows us to identify the changes that are affecting (for good or ill) the greatest number of people or that the majority of people perceive as the most important.

If we want to get the best out of these tools, we have to use them together. It always helps me to think of MSC as the (qualitative) average of a set of SoC. I cannot be serious if I say that 'my' project has been a success if one farmer, out of 3,000, says that the project has allowed him to buy a Mercedes Benz! I believe that, as a development practitioner, I can really be proud of (or worry about) MSCs generated by initiatives that I am involved in.

Do you also share this belief?