What’s the best day to send emails?

Generally speaking, weekdays are good, weekends aren’t, and Mondays seem to fall somewhere in between. There is some variation within weekdays - from the charts below, you can see that Friday open rates seem moderately better than Tuesday through Thursday, and that Mondays appear worse than other weekdays for both open and click rates.

But don’t read too much into that. There’s not enough certainty in these data to say that the distribution of the next 200 emails will be quite as favorable to Fridays and unfavorable to Mondays, and there’s too much noise in the click rate data to make any definitive statements at all. So the one solid conclusion that you can take to the bank - open rates are better on weekdays then on weekends.

What’s the best time to send emails?

We have more categories for send time, which means fewer observations per category and therefore a little less certainty. But one thing we can say with relative certainty is that in terms of open rates, emails sent before 9am don’t perform very well. So for morning emails, take advantage of MailChimp’s “Send with Timewarp” option - this will deliver your campaign according to the recipient’s time zone, and avoid hitting any West Coast inboxes during the 6-8am window.

Note that 9am emails don’t appear to have great open rates - but we haven’t sent enough emails at that hour to firmly say that their poor performance is due to the timing rather than some other factor. If you google “what time should I send this email,” a lot of sites will recommend the 9am hour, so it wouldn’t be a bad idea to send a few more out and see how they do.

One additional interesting tidbit - there was a statistically significant difference between open rates for emails sent at 5pm and 6pm, with the 5pm emails outperforming the 6pm. The average click rates at 6pm are moderately higher, but the difference isn’t significant. So if you’re sending something late in the afternoon, shoot for 5 instead of 6.

There were a handful of emails sent at 7pm or later that aren’t included in the below plots since there are only one or two observations in each hour.

The Sample

A few emails were removed from this dataset. First, those sent by users outside of the division, both because we have little control over when they are sent, and because their audience and purpose differ in scope from the emails that this discussion is intended to help inform. Second, all emails that included the word “COVID” in the email name. My assumption was that emails addressing the initial COVID-19 outbreak were sent in extremely abnormal circumstances, and therefore the results of those emails wouldn’t be generalizable to future emails. In an effort to introduce as little subjectivity as possible, all of these datapoints were removed before any analysis was performed.

One outlier was removed during the analysis portion of this project. For a full explanation, see the “Outlier” section below.

All email send times are rounded down to the nearest hour. I.e., an email sent at 9:47 would be categorized as a 9:00 email. This rounding was done based partly on the assumption that emails are typically targeted to be sent on the hour, partly on the assumption that most people operate on the hour. In other words, someone’s behavior at 11:45 is more likely to match what they’re doing at 11:00 (sitting at their desk) than at 12:00 (eating lunch). I contemplated breaking the data into half hour increments, but that would have created twice as many time categories resulting in a corresponding drop in certainty in our results. Additionally, I could have treated send time as a numerical rather than categorical variable. I opted for categorical/binned numerical primarily so that we could consider averages at given points throughout the day.

You might have noticed that I found basically no solid conclusions about the best day or time to send emails in regards to click rates. My best guess is that this is because the dataset includes both emails with a call to action and without. There was no easy way to parse these categories out, so I made no attempt to account for that variables. That means that an email with no call to action that gets a 0% click rate is seen as “bad,” even if that’s not really an accurate reflection of the email. The best solution to this will be better email naming convention practices going forward, which will help to separate out emails intended to receive clicks versus those that aren’t when considering click rates. But keep in mind that narrowing our scope will also narrow our sample size, which will make it more difficult to draw solid conclusions.

Variables

We’re looking at four variables: Day sent, time sent, open rate, and click rate. It’s useful to look at the distribution of each observation before diving into the analysis.

A couple of things jump out. First, a relatively low number of weekend emails was sent. Second, there aren’t a ton of observations in the 7pm and later categories, so we’ll remove those when analyzing send times since there’s not enough data there to draw solid conclusions. Third, the open rate distribution is relatively normal, which is good, because the ANOVA testing that we’ll do later assumes normality. But the click rates most definitely do not have a normal distribution, and are skewed pretty heavily to the right. Which is OK, because we can fix that with a simple logarithmic transformation:

Outlier Detection

A note on outliers - there are a lot of ways to define “outliers” and none of them are perfect. In general you shouldn’t remove an extreme data point from your data unless you have a really good reason to. A good reason might be that you think the result was recorded incorrectly or the datapoint is unique from the others in the data. A bad reason would be that the point changes your underlying assumptions about your data. If that’s the case, it says more about the rest of your data (largely that you don’t have enough of it) than it does about the one extreme point in question.

## 
##  Shapiro-Wilk normality test
## 
## data:  dat$Open_Rate
## W = 0.99039, p-value = 0.1869

Hypothesis Testing

##              Df Sum Sq  Mean Sq F value Pr(>F)  
## Day           6 0.0312 0.005193   2.728 0.0144 *
## Residuals   199 0.3789 0.001904                 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

##              Df Sum Sq  Mean Sq F value   Pr(>F)    
## Time         14 0.0897 0.006409   3.822 1.04e-05 ***
## Residuals   191 0.3203 0.001677                     
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

##              Df Sum Sq Mean Sq F value Pr(>F)
## Day           6   3.31  0.5521   0.488  0.817
## Residuals   199 224.96  1.1304

##              Df Sum Sq Mean Sq F value Pr(>F)
## Time         14  21.35   1.525   1.407  0.153
## Residuals   191 206.93   1.083

Document Information

All of the statistical analyses in this document were performed using R version 4.0.2 (2020-06-22). R packages used will be maintained using the packrat dependency management system. For questions, data, or to view full code, contact Jon Crenshaw at jon.crenshaw@colby.edu

## R version 4.0.2 (2020-06-22)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 10 x64 (build 19041)
## 
## Matrix products: default
## 
## locale:
## [1] LC_COLLATE=English_United States.1252 
## [2] LC_CTYPE=English_United States.1252   
## [3] LC_MONETARY=English_United States.1252
## [4] LC_NUMERIC=C                          
## [5] LC_TIME=English_United States.1252    
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
##  [1] kableExtra_1.3.4 tibble_3.1.0     agricolae_1.3-3  tidyr_1.1.2     
##  [5] fmsb_0.7.0       lubridate_1.7.10 mvnormtest_0.1-9 gridExtra_2.3   
##  [9] scales_1.1.1     forcats_0.5.1    ggplot2_3.3.3    dplyr_1.0.4     
## [13] readxl_1.3.1    
## 
## loaded via a namespace (and not attached):
##  [1] httr_1.4.2        sass_0.3.1        jsonlite_1.7.2    viridisLite_0.3.0
##  [5] bslib_0.2.4       shiny_1.6.0       highr_0.8         cellranger_1.1.0 
##  [9] yaml_2.2.1        pillar_1.5.1      lattice_0.20-41   glue_1.4.2       
## [13] digest_0.6.27     promises_1.2.0.1  rvest_0.3.6       colorspace_2.0-0 
## [17] htmltools_0.5.1.1 httpuv_1.5.5      klaR_0.6-15       pkgconfig_2.0.3  
## [21] labelled_2.7.0    haven_2.3.1       questionr_0.7.4   bookdown_0.21    
## [25] purrr_0.3.4       xtable_1.8-4      webshot_0.5.2     svglite_2.0.0    
## [29] later_1.1.0.1     combinat_0.0-8    farver_2.1.0      generics_0.1.0   
## [33] ellipsis_0.3.1    withr_2.4.1       magrittr_2.0.1    crayon_1.4.1     
## [37] mime_0.10         evaluate_0.14     fansi_0.4.2       nlme_3.1-148     
## [41] MASS_7.3-51.6     xml2_1.3.2        blogdown_1.2      tools_4.0.2      
## [45] hms_1.0.0         lifecycle_1.0.0   stringr_1.4.0     munsell_0.5.0    
## [49] cluster_2.1.0     compiler_4.0.2    jquerylib_0.1.3   systemfonts_1.0.1
## [53] rlang_0.4.10      grid_4.0.2        rstudioapi_0.13   miniUI_0.1.1.1   
## [57] labeling_0.4.2    rmarkdown_2.7     gtable_0.3.0      AlgDesign_1.2.0  
## [61] R6_2.5.0          knitr_1.31        fastmap_1.1.0     utf8_1.1.4       
## [65] stringi_1.5.3     Rcpp_1.0.6        vctrs_0.3.6       tidyselect_1.1.0 
## [69] xfun_0.21