Encouraging your employees to be active users of your product is a great strategy to get early feedback, but it’s easy to get misled. That's where data comes to the rescue! Here are 5 steps to use data to dogfood better:
1. Flag All Employee Accounts
All user accounts that belong to employees must be flagged in the database. Employees inherently use the product differently from users, as shown in this case study from DataHero. You need to be able to remove this noise from your analysis.
Flag all employee accounts, not just administrators and developers
Have a simple process in place to easily flag new accounts, including any personal accounts employees may have.
2. Define an "Active User"
An "active user" could be someone who visits every day, every week, or at least once a month. "Active users" are a key metric for any startup, so you probably already have this. Make sure that it's well-defined and easy to leverage in your analysis.
Knowing who your "Active Users" are isn't just for pitch decks
3. Define an "Onboarded User"
An "onboarded user" is someone who has reached a certain usage threshold and has figured out the basics of your product. (They may have abandoned after they figured it out...but at least they figured it out.) Having this definition lets you excludes people who dropped off during the registration process, people who immediately realized that your product wasn't for them, etc.
Eliminating users who signed up and then immediately abandoned might be hard on the ego, but it's essential to making the right decisions
An "onboarded user" could be defined by a number of visits or the completion of certain actions. At DataHero, we defined an "onboarded user" as someone who visited on at least 4 separate occasions after completing the onboarding flow.
4. Define a "Customer"
Hopefully, this one is easy, but make sure you have it. It's important that you're able to separate paying customers from free users in your analysis.
Paying customers are your most important users. Period.
For most companies, power users by definition must be customers. If your pricing structure allows for non-paying power users, then you may need an additional definition here.
5. Look at the Data
Once you've got all your definitions, it's time to look at the data. Here is how to take the definitions you created and use data to dogfood better:
Start with All Users
First, look at the distribution across all users (including employees). Let's say you're trying to decide how many album covers to show on your music app's main screen. Start by looking at the distribution of the number of albums in each user's account:
Look at Only Employees
The next step is to look only at your employees:
If the distribution of your employees data matches the distribution for all users, then your dogfooding is an accurate representation and you can confidentially move forward.
If it doesn't match, then there are two possibilities:
- Your employees are power users
- Your employees are just plain different
Look at Everyone Except Employees
Remove your employees from the list of users and compare the results. Does the distribution of employees overlap with the distribution of non-employees?
If there is no overlap, then your employees are different from normal users and their data should be thrown out. For example, if the average employee has 10 albums, but no other user has more than 6.
Otherwise, it's possible that they are representative of power users.
Eliminate Non-Onboarded Users
If employees and non-employees overlap, the next step is to eliminate users who didn't pass the "onboarding" threshold. This gets rid of users who signed up but didn't really get anywhere. In most cases, this will increase the average for your distribution (for example, eliminating everyone who signed up but never had one album).
At this point, your employees may look more like power users if the distributions are closer together.
Look at Your Customers
Finally, it's time to look at your power users. If the distributions are similar, then you can conclude that your employees act like power users. If not, then your employees are different in some way. This doesn't mean that they don't have valid feedback thought.
Whether or not you investigate further is up to you. It may be important to your project to understand exactly why your employees are different, or it may be sufficient to know that their feedback is valid, but doesn't exactly match that of your users and customers. In either case, you've got enough information to move forward.
Here's real-world example of why it's important to not let dogfooding mislead you and how to use data to dogfood better.