Plan, run and learn from a simple test

By the end of this lesson: Plan a small message test with a clear goal, a method and a sample audience, and note what you would change based on the results.

Download the slides (PDF)

Plan, run and learn from a simple test

A message test does not need a big budget or a research team. It needs a plan. This reading gives you a simple template and a worked example to adapt.

The four parts of a test plan

  1. Goal: what you want to learn, in one sentence.
  2. Method: how you will find out.
  3. Sample: who you will test with.
  4. Change note: what you found and what you would change.

Worked example

Imagine a community health group wants to send an SMS about a clinic outreach day. This example is for practice only. The details are made up.

  • Goal: Find out whether people understand the date, the place and what to bring.
  • Method: A quick conversation. Show the SMS and ask, "Can you tell me in your own words what this says?"
  • Sample: [add number] people from the village the SMS is meant for, with a mix of ages and both men and women.
  • Result: Several people were unsure where to go.
  • Change: Add the name of the nearby landmark. Move the date to the start of the message.

Choosing a method

Pick the lightest method that answers your goal.

| Method | Good for | Watch out for | |—|—|—| | Quick conversation | Checking if people understand | You may influence answers if you explain too much | | Side by side (A and B) | Choosing between two options | Change only one thing at a time | | Small send | Seeing real behaviour | Needs a channel you can already reach |

Think about your audience. If many people have limited data, a short phone call or a face to face chat may reach them better than an online form.

Choosing a sample

Your sample should look like your real audience. Think about:

  • language
  • age
  • location
  • the channel they use most

Avoid testing only with colleagues or friends. They know the context and may be too polite.

Running the test well

  • Use the same message and the same questions for everyone.
  • Record answers as they are given.
  • Do not defend or explain your message during the test.
  • Keep one change per version in an A and B test.

Reading the results

Compare what you saw with your goal. Look for the same problem appearing more than once. One odd answer may not matter. A repeated point of confusion usually does.

Write a change note

Keep it short so people will use it:

  • Goal
  • Method and sample
  • What we found
  • What we would change

Try it

Choose one real message you plan to publish. Fill in the four parts above before you send it.

Transcript

Imagine you run communications for a small organisation in Kumasi. You write a message reminding parents about a vaccination day. You send it to everyone. Then very few people turn up. What went wrong? You may never know, because you did not test.

Today you will learn to plan a small message test. You will set a clear goal, choose a method, pick a sample audience, and note what you would change. You can do all of this with a small budget and a few days.

Start with the goal. A goal says what you want the message to do. "Make the message better" is too vague. A good goal is specific.

For example: do people understand the date and the place? Do they know what to do next? Or, which of two subject lines makes more people open the email?

Your goal also tells you what to measure. Write it in one sentence. If you cannot, the test will wander.

Next, choose a method. Keep it simple. You have three easy options.

First, a quick conversation. Show the message to a few people. Ask them to explain it back in their own words. This is cheap and fast.

Second, a side by side comparison. Make two versions, A and B. Show each to a different group. See which one does better against your goal. Change only one thing between the versions. If you change the headline and the picture, you will not know which made the difference.

Third, a small send. Send the message through a channel you already use, such as a WhatsApp broadcast list or SMS, to a small group first. Then watch what happens. Look at replies, clicks or visits.

Choose the method that fits your audience and your budget. If your audience has little data or patchy internet, a face to face chat may work better than an online survey.

Now pick your sample. A sample is a small group that stands in for your wider audience. Choose people who look like the people you want to reach.

If your audience is market traders, do not test with colleagues in the office. Colleagues already know too much. Also avoid people who only want to please you. You need honest answers.

Think about language, age, location and the channel people use. You do not need big numbers. Even a small group can show you obvious problems. [add detail: your own sample size]

Then run the test. Keep conditions the same for everyone. Show the same message in the same way. Ask the same questions. Write down answers as you go, word for word where you can. Do not defend your message. Just listen.

Afterwards, compare the results with your goal. Did people understand the date? Where did they hesitate? Which version did better?

Look for patterns. If one person is confused, it may be about them. If several people are confused in the same place, change that part.

Last, write a short note. Record four things: your goal, your method and sample, what you found, and what you would change. A few lines is enough.

For example: "People were unsure where the clinic is. Change: add the landmark and the ward name." Next time, anyone on your team can read your note and not start from zero.

Let us recap. Set one clear goal. Choose a simple method that suits your audience and budget. Pick a small sample that looks like your real audience. Run the test the same way each time. Compare the results with your goal, and write down what you would change.

This is the last lesson in the course, so your next step is to try it for real. Pick one message you are about to publish. Plan a small test this week. Small tests, done often, make your messages clearer.