Every email designer knows that resolving compatibility issues is a biyatch. But testing doesn’t have to be, if you have the right tool. Until all email clients act the same and fully support standards based HTML and CSS, the folks at the Email Standards Project will have a mission and we designers will need robust tools to figure out why something looks squished or spacey or whatever other flavor of bizarre the email client is spitting out that day.
Four tools you might consider are Campaign Monitor, MailChimp Inbox Inspector, Litmus, and Email on Acid. (Also, refer to these handy resources to help you make sense of the vagaries of email client CSS support.)
Which Is the Best Email Testing Tool?
It depends on you. All of these tools are robust, easy to use, and reasonably priced. I have had positive customer support experiences with all but Email on Acid (I have not had any contact with their support team). Campaign Monitor and MailChimp are campaign management/delivery services with a built in testing tool. Litmus and Email on Acid are dedicated testing tools.
How often do you test? Do you require spam and content testing, or are you only concerned with email design issues? Do you have team members who need to share the tool?
If you are looking to manage and send email campaigns, you might prefer Campaign Monitor or MailChimp. If you only test, then it might be a pain to have to set up a campaign first, so you might prefer to go with a dedicated testing tool like Litmus. But don’t take my word for it—try them all and then decide. (And if you have a favorite testing tool that I haven’t mentioned, let me know what it is and why it’s great!)
For instance, while writing this article, I almost overlooked the fact that MailChimp has a code analysis tool in my excitement over sharing the good news about Email on Acid. Reason being, I need a dedicated testing tool, and I run so many design tests per day that, for me, using MailChimp for testing is not feasible.
Campaign Monitor
Campaign Monitor provides simulation screen shots of your email in the following email clients:
Desktop Email Clients
- AOL 9
- Apple Mail 2.1
- Apple Mail 3
- Lotus Notes 6.5.4
- Lotus Notes 7
- Lotus Notes 8
- Outlook 2000
- Outlook 2003
- Outlook 2007
- Outlook Express 6
- Outlook XP
- Thunderbird
- Windows Mail
Web-based Email Clients
- AOL Web
- Gmail
- Mobile Me
- Windows Live Hotmail
- Yahoo! Classic
- Yahoo! Mail
Mobile Email Clients
- BlackBerry
- Windows Mobile 5
- Windows Mobile 6
Cost: $5 per test; pay as you go. Set up a free account to get started.
Campaign Monitor design test
This is a good, solid tool that I like to use. And I really like the nice folks at Campaign Monitor. Campaign Monitor’s design and spam testing tool was the first I ever tried, which was in 2007. It’s pretty straightforward: after you upload your campaign and before you send it, you click a button. My one complaint at the time was that sometimes too many screen shots would fail (I got a “screen shot could not be produced” error), which made my test a bust. This is really annoying when you’re on a tight deadline! However, on the few occasions that this happened, I wrote to Campaign Monitor’s excellent, friendly, and helpful support team, and they always cheerfully refunded my money.
That said, when I ran a Campaign Monitor test last month, it was fast and complete.
Campaign Monitor also runs your email through popular spam filters and lets you know which ones failed and why.
Pros
- Fast and easy
- Smoothly integrates with your campaign development process
- Tests in lots of clients and spam filters
- You don’t have to commit to a subscription or contract
- Super-awesome customer support team
Cons
- Not as convenient if you want to test code but not send campaigns (because you have to set up a campaign before you can test your code)
- Can get expensive if you routinely run lots of tests
- Retesting is not as convenient as with other tools, plus you have to buy another test
Overall, a great, reliable tool that is probably best for those who test less frequently and/or also need a service for managing/sending campaigns. I worked at a company that sent two to four emails a month, and this was perfect for them.
MailChimp Inbox Inspector
MailChimp‘s Inbox Inspector is also for design and spam testing. Like Campaign Monitor, MailChimp is primarily an email campaign sending/management service, so you have to set up a campaign in order to run a test. MailChimp says they check your campaign “by sending it to real ISPs and email services all over the world. These are not simulated screenshots.” I’ll admit my ignorance here: I’m not sure if that is any different from what the other services do. (If someone knows better than I about how these tests work, please write in and share!)
Inbox Inspector checks your email in over 25 desktop, web-based, and mobile clients. Frankly, I found the list overwhelming—nothing wrong with that! But I personally did not find some of the more obscure clients useful. I need to test in major clients used by folks in North America, so I did not derive value from viewing simulations in European and Asian clients.
Inbox Inspector also performs spam testing, content analysis, and code analysis. It identifies which lines of code are kicking up compatibility problems, which is vital for any email designer. It even makes suggestions for rephrasing your email content!
Cost: $14 per 3 tests; pay as you go. Set up a free account to get started.
Pros
- Easy to integrate into your campaign management workflow
- You don’t have to commit to a subscription or contract
- A good value if you don’t run frequent tests
- User interface makes navigating through results easy
- Code analysis tool
- Super-awesome customer support team
Cons
- Robust testing might be overkill for some people; for instance, I do not need all of the clients included in testing, nor do I need any of the spam or content analysis tools
- Can get expensive if you run frequent tests
- You have to set up a campaign in order to run a test
Litmus
Litmus provides simulation screen shots of test email in the following email clients:
- Live Hotmail
- AOL Mail
- Yahoo! Mail
- Outlook 2000
- Outlook 2002/XP
- Outlook 2003
- Outlook 2007
- Lotus Notes 6.5
- Lotus Notes 7
- Lotus Notes 8
- Apple Mail 2
- Apple Mail 3
- Gmail
- Yahoo! Mail Classic
- Thunderbird 2.0
- Thunderbird 3.0 Beta 3
Cost: Free limited testing (2 clients, 50 tests/month); $24 day pass, $49/month single user (unlimited browser and email testing, spam analysis), $199/month team (up to 10 users, unlimited browser and email testing, spam analysis)
Litmus email test
Litmus is not a sending service, so to get started you just copy and paste your code (or email it to them), and off you go. The user interface lists the email clients in the right-hand column so you can tab through your results quickly instead of always having to return to a main screen. You can retest an individual client or the entire batch with one click and tab through the versions you generate. You can generate compatibility reports for your coworkers. Litmus will even email, tweet, or IM you when your test is complete. This puppy is packed with features!
Pros
- Subscription pricing is a good value if you run frequent tests (no contract required)
- Easy to retest
- Great sharing tools
- Includes browser testing (if you need it anyway)
- User interface navigating through results easy
- Receive email, IM, or Twitter notice when test is complete
- Super-awesome customer support team
Cons
- No code analysis tool (though you can view your source code with one click)
- Maybe too expensive if you don’t run frequent tests
- Does not manage and send the email campaign (if you need that)
Overall, a very impressive tool with the most features I’ve encountered. I am using Litmus at a company I work for currently. I run many tests per day, which makes this a great value.
Email on Acid
I found this brand-spanking-new service while doing research on email testing tools last month. Email on Acid is still in beta, but I find myself gravitating there every day to run their Acid Tests. I am using Litmus and Email on Acid in tandem, in fact.
Email on Acid simulates your email in 15 popular email clients. The nifty tabbed interface lets you rotate quickly through the results. But what I love love LOVE about this service is the code analysis tool. For each client, you get not only a screen shot but a list of HTML/CSS errors. You can scroll through the source code to see the errors, highlighted in red, to identify which HTML or CSS declarations the client does not support. I did notice a difference in the rendering between Litmus and Email on Acid, and I can’t say which screen shot was inaccurate or why. But I am singing Email on Acid’s praises because it’s streamlined, code-oriented, and doesn’t have extra bells and whistles I don’t need.
Email on Acid design test
Email on Acid code analysis tool
Cost: Free testing while product is in beta. Help these folks out by trying an acid test and sending your feedback, will ya?
Pros
- Code analysis tool
- Free (for the time being)
- Testing tool only; you don’t have to set up a campaign to run a test)
Cons
- Product is in beta; not well established
- Doesn’t test in as many email clients as the other services
- Not a campaign sending tool (if you need that)
Overall…WOO HOO!! Email designers everywhere should be rejoicing over a tool that puts so much emphasis on code aalysis. I may be getting ahead of myself here, but this service looks extremely promising.
Resources
- Doctype (Q&A site for CSS issues): http://www.doctype.com
- Campaign Monitor Guide to CSS Support in Popular Email Clients (PDF and Excel spreadsheet available for download): http://www.campaignmonitor.com/css/
- MailChimp email marketing blog
- Email Standards Project blog (reports on email client compatibility issues): http://www.email-standards.org/blog
Amy,
Thanks so much for including us in your review of email testing tools! We are thrilled to hear that our code analysis feature is useful to you. We’ve also made some recent improvements to our Outlook07 analysis – instead of just highlighting unsupported HTML/CSS, we will display Outlook07’s translation of your original code. We hope to complete our testing phase and go live with that new feature this week.
If you do see a discrepancy between the Litmus test and our simulation, please don’t hesitate to report it to us. In our results interface, there is a button for each email client that reads “Report a Discrepancy.” We store your original HTML and the name of the email client in question making it really easy for users to help us improve our system.
We are hoping to come out of beta next month and when we do, you will be the first to hear about it!
Thanks again and happy testing.
Michelle Klann
Email on Acid
Insanely great post, Amy. Always love objective reviews based on real-world experience.
Like you, I use several of these tools often and in tandem, and am with you on virtually all your pros and cons. Looking forward to when Email on Acid works and plays well with Exploder, too.
Well done!
Email On Acid is a SIMULATION only — its not real screenshots and so it does not compare at all to the companies taking real screenshots.
Only 3 companies take real screenshots:
1. Pivotal Veracity.
2. Litmus App
3. Return Path
The other 2 companies you mentioned (Campaign Monitor and Mailchimp) don’t have their own technology – they are only reselling integrated versions of the above. Specifically:
- Campaign Monitor: is an integrated reseller of LitmusApp & Return Path
- MailChimp: is an integrated reseller of ReturnPath
You didn’t mention Pivotal Veracity in your review but our solution is primarily for large enterprises such as Apple, Amex, and Oracle and is probably not suitable directly for small businesses. However, our solution is exclusively sold and integrated and available at a lower price from all leading ESPs including: Acxiom Digital, Responsys, Bronto, BlueSky, Yesmail, Responsys, Silverpop, Exact Target, Clickmail, Click Squared and over 40 others as well as top agencies such as Ogilvy, Avenue A Razorfish, Digitas, MRM, etc.
Generally small businesses are best served by going to a reseller ESP. Medium and large enterprises should examine the options available directly from the 3 providers as they may have access to more features and/or other services when going direct.
Sorry and one more note – those w/ no budgets should look at emailonacid. even though its only a simulation (we do both but simulation is definitely never as accurate as a real screenshot), its better than qa’ing on nothing at all.
Great write up, Amy. I’m delighted you’ve found Litmus is a good fit for you.
Deirdre is correct regarding Email on Acid. The code analysis is a great feature, but the simulations don’t compare to screenshots in actual email clients.
We’re working on some code analysis features at Litmus. I think you’ll be very pleased with what we’re building, but it’s likely to be a month or so before we’re ready to release.
This is a great review. I have one question though. We were leaning toward integrating Litmus into our process, but were dismayed by how long it took to return results (as much as 20 minutes). Is there a faster or lighter test at there that can return results in, say, 30 seconds?
Thanks for great review.
For the fastest test results check-out previewmyemail.com.
They provide 22 web&email apps including iphone. Tests are completing just in minutes. I hope they will also provide ipad screen shots by near future.