Should QAs Care About Email Testing?
Yes, QAs should care about email testing. This article explains the reasons, elements to test, and how to make email testing effortless.
Foremost, you need to understand what kind of email testing we're talking about.
In general, email testing refers to a few methods of checking emails before sending them. For email marketers, it's more about content analysis and A/B testing campaigns. For devs and QAs who work with apps sending transactional emails, email testing refers to a bigger cycle of actions – from analyzing HTML to ensuring email deliverability.
I’ll cover:
- Importance Of Email Testing
- 4 Pain Points Of QA Email Testing + Workarounds
- Top Email Elements You Should Be Testing
First up, the importance of email testing for QAs.
You Can’t Let Testing Slide: Here’s Why
Statistically, 300+ billion emails are sent/received every day. It's hard to imagine the number of bugged emails sent by companies on a daily basis. However, the fact that such messages affect a brand reputation and provide a poor user experience is undeniable.
In this way, debugging emails is the responsibility of the dev/QA team to make sure a marketing team will be able to launch a proper campaign.
Skipping this stage leads to three major negative consequences:
Rendering Errors = Poor User Experience
Unfortunately, not all email clients support HTML and CSS to the same degree. Outlook, or Gmail app for non-Google accounts, for example, don’t render background images.
Similarly, email clients often have specific guidelines for designing emails - Yahoo Mail enforces margins while Gmail clips letters that are longer than 102kB.
Since designers are not necessarily considerate of rendering standards across a wide range of email clients, it’s a tester’s job to meet all the demands.
That’s why you should focus on testing campaigns before sharing them with users. Else, a person might see your letter clipped, with a shifted layout, unresponsive, or unsupported content. As a result, poor UX is guaranteed and there’s a higher chance customers won’t come back. If one considers that the entire campaign contains broken emails, it's frustrating.
Deliverability Takes A Toll
Ensuring a reliable pathway between in-app emails and end-users is critical to support large user bases. Since a lot of teams use email notifications to share passwords and notify the community about product updates, not being able to reach subscribers is a bummer.
In email marketing, email deliverability is an X factor that determines whether a user can access your important message. There are a lot of criteria determining a campaign’s deliverability rate: number of emails reported as spam, user interactions with letters, bounce rates, and others.
Building up smooth deliverability is hard work, and this is usually a responsibility of an engineering team. A QA needs to be mindful of when and how many transactional emails a website or app sends. On the contrary, sending all the effort down the drain is surprisingly easy – a few broken links or a failed spam check is enough to derail months of effort.
Deliverability testing is a way to prevent such frustrating setbacks as it lets a QA team:
- dodge spam traps (fake emails planted by ISPs all over the web that are often swept by bots and included in the subscriber base)
- find out what email infrastructure elements are misconfigured (IP, DNS records, email authentication records, etc.)
- make sure there are no spam triggers in the content
If a QA or dev ignores spam checks and deliverability testing, campaigns or important emails don't reach an end-user. How can a user reset a password or get a registration link if an untested email flies somewhere within a network? From undelivered emails, a company may suffer client loss and other business failures.
Reputation Is Affected
Now, highly personalized emails are a solid trend. However, when sending messages full of dynamic tags, things often get out of hand.
It’s not news for recipients to get letters with wrong name-tags or subject lines like “Hello, [username]”. For brands, small oversights kill the conversions of the entire campaign and bring down brands' media relations. The reason is simple: you don’t get a second chance at making the first impression. If you screw up once, subscribers will likely flag the letter as spam or leave negative feedback. And the brand will be associated with senders of broken emails just because someone skipped HTML/CSS testing.
Related Read: THE POSITIVE OUTCOMES OF NEGATIVE TESTING
4 Pain Points Of QA Email Testing (+ Ways To Work Around Them)
We outlined three major deteriorating effects of sending bugged emails. It's time to understand the pain points of those testers who struggle valiantly through email debugging.
We can’t and won't judge. For years, there were a lot of shortcomings in email testing workflows that made the process too manual, slow, and inefficient.
There are feasible workarounds that eliminate testing pain points. Let’s take a look at how to deal with the most annoying inconveniences.
1. Test Emails Go To Real Users
This annoying issue stems from the fact that QA teams use production domains to run test sessions. As a result, it’s easy to cross the line and accidentally send a test message to a list of subscribers.
On top of that, using the production server to run tests bloats out the sending volumes for a domain and hurts domain authority.
It’s easy to make sure you don’t email real users by mistake as long as you use a separate environment for testing. There are two ways to safely test emails:
- In-development environment testing using API integration
- Using tools that imitate the work of real SMTP servers with the ability to check common SMTP ports and other infrastructure elements.
2. Low Deliverability (Or Landing In Spam Folders)
If your preview emails go to ‘Spam’, it’s not necessarily a red flag. Before you alert the marketing team and double-check the infrastructure, cross out the following possibilities:
- Your preview email still has filler text. When sending test emails, make sure the body of the message is exactly the one a user will see. Artifacts like “Lorem Ipsum dolor” trigger spam filters and tank the deliverability of a test email.
- You don’t open your own test emails. If you use your own address to test emails, unless you interact with them, ISPs will flag the letters as irrelevant and start sending them to spam.
- Sender and recipient address match. To deliver emails successfully, email clients demand that sender and recipient’s addresses are not the same mailbox. So, when sharing a test message with yourself, choose a different email than the one you use to send the test.
- No “Unsubscribe” link. Batches that don’t have an “Unsubscribe” footer have a 99.9% chance of bouncing or being marked as spam.
3. Poor Rendering And Cross-Device Responsiveness
Another roadblock QAs hit is discovering that messages are displayed differently across email clients or device types. If this is the case for your test batch, here are a few client-specific rendering considerations you should validate the email body across:
Gmail:
- Images are supported by default.
- Emails over 102kB in size are automatically clipped.
- The <style> tag is placed at the head of the email.
- Auto-scaling emails on iPhones (images will seem off-center so it’s better to put “padding:0” in <body>.
- Minimal text size is 10.5pt for copy and 16.5pt for headings to ensure readability on smartphones.
Outlook:
- No support for background images.
- No support for interactive elements like forms or checkboxes.
- No support for HTML5 videos or GIFs.
- Limited padding support.
4. Low Testing Efficiency
Back in the 2000s, email testing was manual, static and tedious. Testing teams needed to generate emails from scratch and send them to test addresses. The good news is, nowadays, most of these steps can be automated easily.
Here are a few tools that help QA teams spend less time on testing of single email elements:
- Email preview: Litmus
- Email servers: GMass
- Email API: Mailosaur
- Spam check: SpamAssassin
- Email deliverability: Mail-Tester
- HTML check: HTML Email Check
- Browser automation system: Selenium
If you need a full-stack testing solution that will allow you to test all the tech aspects of email, including SMTP, API, HTML/CSS, prefer collaboration-friendly tools like Mailtrap.
Email testing is just one facet of quality assurance. For a holistic approach to QA, explore our guide to the best software testing tools.
Top Email Elements You Should Be Testing
Now that you know why there’s no way to pass on email testing, and understand how to deal with the main hurdles QA teams face when running sessions, it’s time to create a step-by-step testing strategy that’ll ensure high deliverability and impeccable rendering of your emails.
Here are the key types of email tests a QA team should run.
1. SMTP Monitoring
SMTP errors are a common culprit of email deliverability issues or entire email infrastructure fails. Here are the issues QAs need to look out for:
- The firewall is blocking communication.
- Server response takes too much time.
- SMTP server connects with the wrong hostname.
- SMTP doesn’t support given commands.
To streamline SMTP assessment, QA teams use dedicated tools: Web Biz or Wormly.
2. Email API Testing
API testing allows developers to test emails without exiting the IDE. By using APIs, you can:
- Maximally automate the process.
- Fetch emails in code.
- Extract and verify the content of a test email.
- Apply pattern matching.
- Send test emails with attachments.
Different programming languages run different scripts for API email testing. To streamline the process, consider implementing tools like Mandrill or MailSlurp.
3. Local Test Email Sending
Another way to test emails is by configuring a local server. This way, QA teams relieve the sending load from the production environment and separate testing from a real-life campaign.
Email testing on a local server is a helpful way to make sure you don’t send a test batch to subscribers by mistake. The tools to consider are Mailhog or Mailcatcher.
4. Email Deliverability And Spam Testing
As we mentioned, email deliverability and spam testing help control the reputation of your domain and IP and discover whether a sender address is not blacklisted by ISPs.
Mail-Tester or GlockApps might come in handy.
Conclusion
In the QA community, email testing often comes secondary to functional and performance testing. In truth, QA teams shouldn’t underestimate email debugging and infrastructure testing. Try out the approaches mentioned in this article. Tell us what email testing tools you prefer in the comments section.
For more on this and other advice from QA experts, don't forget to subscribe to the The QA Lead newsletter to keep up with the best in the world of quality engineering.
Related List of Tools: 10 BEST EMAIL TESTING TOOLS FOR OPTIMIZED DELIVERY
Don't stop learning now! Check this podcast out: AUTOMATED TESTING W/ TESTRIGOR CEO ARTEM GOLUBEV & PAUL GROSSMAN
