upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Text Advertising Blindness: The New Banner Blindness?

Justin W. Owens, Barbara S. Chaparro, and Evan M. Palmer

Journal of Usability Studies, Volume 6, Issue 3, May 2011, pp. 172 - 197

Article Contents


Results

The following sections discuss task success, ad recall, task difficulty, task duration, and eye-tracking metrics.

Task Success

Task success was examined across target location (Content, Top Ad, or Side Ad) and search type (exact or semantic search). Figure 4 shows task success by search type and target location.

Participants were most successful at identifying targets in the Content AOI (82%), followed by the Top Ad AOI (52.9%), and then the Side Ad AOI (36.8%). Analysis of search type indicated that participants were more successful in exact searches (67%) than semantic searches (60%). Differences in success were examined with Chi-Square tests of association and further examination of residual values. The relationship between target location and task success was significant, Χ2(2, N = 396) = 64.531, p < .001, indicating that task success was significantly higher when the target was within the Content AOI (z = 3.3) and significantly lower in the Side Ad AOI (z = -3.3). Additionally, participants were slightly less successful, but not significantly, when the target was located in the Top Ad AOI (z = -1.3). The relationship between task success and search type was not significant, Χ2(2, N = 396) = 1.976, p > .05. The results of both Chi-Square tests indicate that target location, but not search type, has an effect on the success of the trial.

Figure 4

Figure 4. Percentage of task success for search type and target location, error bars are +/- one standard error.

Ad Recall

If one considers the implication of text advertising blindness to be that users ignore areas resembling text advertising, ad recall should be in sync with the task success results. With a higher success rate for the Content and Top Ad target locations, participants should have more recall of text advertisements in this region compared to Side Ad target locations. Ad recall was recorded in this study when users indicated the presence of an ad in the web page sketch or reported ads being present in the open-ended question at the end of the experiment. Overall, 12 of 25 participants recalled advertisements located in the Top Ad AOI. When ads were not recalled, participants typically reported that the Top Ad AOI contained information related to the content, such as related links. Twenty-four of 25 participants reported that the Side Ad AOI contained advertisements. No participants reported advertising in the Content AOI, which never contained ads. Figure 5 contains frequency of ad recall per advertising region.

Figure 5

Figure 5. Ad recall across advertising regions

Task Difficulty

Not surprisingly, users rated the unsuccessful trials as more difficult than successful trials. Results of an independent-samples t-test indicate that task difficulty between successful (M = 1.99, SD = .842) and unsuccessful trials (M = 4.014, SD = 1.103) was significantly different, t(239.013) = 19.051, p > .001, d = 1.49.

A two-way ANOVA was conducted to examine whether task difficulty differed across search type (exact, semantic) and target location (Content, Top Ad, Side Ad) conditions when participants were successful in finding the target. The ANOVA returned a significant main effect of target location, F(2, 246) = 5.179, p = .006, η2 = .04. The main effect for search type and the interaction between target location and search type were not significant. Figure 6 illustrates the differences in task difficulty between conditions. Post-hoc analyses showed that participants rated the Content target location tasks as significantly easier than the Side Ad target location (MD = -0.4553, p = .007).

Figure 6

Figure 6. Mean task difficulty ratings for target location for successful trials, error bars are +/- one standard error.

Task Duration

Similar to task difficulty, if participants were experiencing text advertising blindness, task duration for search tasks where the target information was located in advertising areas should have been longer than non-advertising areas. Additionally, participants should have been faster at completing exact searches than semantic searches (Burke et al., 2005). Task duration was calculated from the onset of the trial to either when the target was clicked, 105 seconds expired, or the participant gave up on completing the trial and announced that the target could not be located.

A two-way ANOVA was conducted to examine whether differences existed for task duration across search type (exact, semantic) and target locations (Content, Top Ad, Side Ad) when participants successfully found the target locations. A significant interaction was reported between search type and target location, F(2, 246) = 5.39, p = .005, η2 = .042. Figure 7 illustrates the differences in task duration between conditions.

Post-hoc analyses showed that exact searches in Content target trials were the quickest and exact searches with Side Ad targets were the slowest. There was little difference in task duration across target locations for the semantic searches. Participants completed exact searches in Content target location trials quicker than Top Ad target location trials (MD = 12.689, p < .01), Side Ad target location trials (MD = 27.19, p < .01), semantic searches in Top Ad location trials (MD = 11.876, p < .01), Content target location trials (MD = 16.443, p < .01), and Side Ad target trials (MD = 19.249, p < .01). Participants were slower during exact searches with Side Ad target locations than Top Ad location trials (MD = 14.5, p < .01), semantic searches with Content target location trials (MD = 10.746, p < .01), Top Ad target location trials (MD = 15.323, p < .01), and Side Ad location trials (MD = 7.94, p < .01). Finally, participants completed semantic searches during Side Ad target trials slower than Top Ad target trials (MD = 7.383, p < .01) and during exact searches with Top Ad targets (MD = 6.56, p < .05).

Figure 7

Figure 7. Task duration in seconds for search type and search target for successful trials, error bars are +/- one standard error.

Eye-Tracking Metrics

Several eye-tracking metrics were examined to explore how participants viewed the text advertisements. Eye-tracking data analyses were limited to visitations to the web page that contained the target location.

In previous studies (Zelinsky, 1996; Zelinsky & Sheinberg, 1997), fixation counts have been highly correlated with other measures of participant gaze behavior and can thus serve as a useful summary statistic and greatly simplify the analyses of eye-tracking data. Correlation coefficients were calculated between fixation count, gaze time, and scan path length. Primarily, fixation count was positively correlated with gaze time for both successful and unsuccessful trials (r = .962, p < .001, r2 = .925; r = .979, p < .001, r2 = .958) and scan path length for successful and unsuccessful trials (r = .98, p < .001, r2 = .96; r = .978, p < .001, r2 = .957). Given these results, fixation count served as the primary data in subsequent analyses and further examination of gaze time and scan path length were not explored.

Heatmaps

Heatmaps were generated for successful and unsuccessful tasks for each type of search task. Heatmaps were aggregated across all similar tasks. Polygons were superimposed over the AOI to indicate the location of the Content, Top Ad, and Side Ad AOIs. Cooler colors (blue, green) indicate fewer fixations while warmer colors (yellow, orange, red, purple, white) indicate increases in fixations. Content search tasks are represented in Figure 8. Top Ad search tasks are represented in Figure 9. Finally, Side Ad search tasks are represented in Figure 10.

It can be seen that the heatmaps show fewer fixations on the Content AOI when participants failed to find targets during Content search tasks, which suggests that they were not thoroughly searching the Content AOI of the page. During Top Ad search tasks, participants that failed to successfully find search targets primarily searched the Content AOI and largely ignored the Top Ad. Even when successful, participants still searched the Content and Side AOIs. While completing Side Ad search tasks, participants primarily searched the Content and Top Ad AOIs even when successful. When participants were unsuccessful, they ignored the Ad AOIs.

Figure 8

Figure 8. Heatmaps for successful and unsuccessful Content target location tasks

Figure 9

Figure 9. Heatmaps for successful and unsuccessful Top Ad target location tasks

Figure 10

Figure 10. Heatmaps for successful and unsuccessful Side Ad target location tasks

Fixation order by AOI

If an area on a web page was related to advertising and users actively ignored it, then it was expected that those areas would be fixated after the main content area of the page, if at all. The AOI rank order was calculated for both successful and unsuccessful trials in an attempt to understand successful and failed search strategies. AOI rank order was defined as the order the Content AOI, Top Ad AOI, and Side Ad AOI were fixated. The order was determined by calculating the time to first fixation for each AOI, and assigning the appropriate rank. Mean rank order was calculated by a Friedman test.

Successful Trials

A Friedman's Chi-Square was conducted to determine if fixation order of AOIs were significantly different. The results indicated reliable differences in rank order for successful vs. unsuccessful trials, Χ2(2, N = 60) = 30.833, p < .001. Additionally, Kendall's W Coefficient of Concordance indicated a small effect size, W = .257. Figure 11 contains the means for rank order of each AOI. The rank orders of AOIs in successful trials indicated that the Side Ad AOI was viewed after the Content and Top Ad AOIs. This supports the notion that participants fixated on regions unrelated to advertising before advertising related regions. Post-hoc analyses showed that the ranks between the Content AOI and Side Ad AOI were significantly different, z = -4.141, p < .001. Additionally, significant differences were noted between the Top Ad AOI and Side Ad AOI, z = -4.593, p < .001. The rank order of fixation for the Content and Top Ad AOIs were not significantly different from each other (p > .05).

Unsuccessful Trials

The rank orders of AOIs in unsuccessful trials indicate that the Side Ad AOI was viewed after the Content and Top Ad AOIs, as in the successful trials analysis. A Friedman's Chi-Square was conducted to determine if rank order of AOIs differed. The results indicated significant differences in rank order for unsuccessful trials, Χ2(2, N = 42) = 27, p < .001. Additionally, Kendall's W Coefficient of Concordance indicated a medium effect size, W = .321. Figure 11 contains the means for rank order of each AOI. Post-hoc analysis showed significant differences between the Content AOI and Side Ad AOI ranks, z = -4.357, p < .001, and between the Top Ad AOI and Side Ad AOI, z = -3.735, p < .001. The rank order of fixation for the Content and Top Ad AOIs were not significantly different from each other (p > .05).

Figure 11

Figure 11. Mean rank fixation order for successful and unsuccessful trials by AOI, shorter bars indicate the region was fixated earlier. For both successful and unsuccessful trials, the Side Ad AOI was viewed after the Content and Top Ad AOI. Error bars represent 95 percentile Bootstrap Confidence Intervals.

In the sample, there was approximately an equal split between participants that thought the Top Ad region were text advertisements or a collection of links related to the content. Previous analyses showed Content and Top was searched first, and that those two were not significantly different from each other. For successful searches, Top Ad AOI rank preceded the Content area, even though the difference was not significant. By examining fixation order, it can be determined whether the classification of the Top Ad region influenced the beginning of the perceived Content region. Results indicated significant differences in rank order for both advertisement and related links classifications, Χ2(2, N = 25) = 15.92, p < .001 and Χ2(2, N = 35) = 20.8, p < .001, respectively. Additionally, Kendall's W Coefficient of Concordance indicated small effect sizes, W = .318 and W = .297, respectively. Figure 12 contains the mean fixation order for AOIs based on ad recall. Overall, if the Top Ad region was perceived as being related content, participants began their searches in this region. In comparison, if they considered it advertisements, they started searching in the true Content region and fixated more on the Content AOI t(250) = 2.162, p = .032.

Figure 12

Figure 12. Mean rank fixation order for trials by AOI and Top Ad perception, shorter bars indicate the region was fixated earlier. Participants who associated the Top AOI with ads tended to fixate first on the Content, followed by the Top Ad AOI, while those that associated the Top AOI with Content fixated the Top Ad AOI first, followed by the Content. Both groups fixated on the Side Ad AOI last.

Whole page fixation count

If participants experienced text advertising blindness, it was expected that they would have more fixations on the entire target page in conditions where target items were located in advertising regions. Moreover, in conjunction with previous literature, it was expected that semantic searches would result in more whole page fixations than exact searches (Burke et al., 2005). Whole page fixation count was defined as the total number of fixations across all areas on the target web page.

A 2x3 ANOVA examined the fixation count across search type (exact, semantic) and target locations (Content, Top Ad, Side Ad) for successful trials. A significant interaction was observed between search type and target location, F(2, 246) = 7.379, p = .001, η2 = .057. Figure 13 illustrates the differences in fixation count between conditions.

Post-hoc analysis revealed that both the exact and semantic searches showed an increase in fixation count for Content and Top Ad target location trials with semantic searches having more fixations overall. However, the fixation count for the Side Ad target location condition was equal between search types.

Figure 13

Figure 13. Whole page fixation count for search type and target for successful trials, error bars are +/- one standard error.

Fixation count by AOI

Eye-tracking metrics were analyzed further as a function of AOI—Content, Top Ad, and Side Ad.

A within-subjects ANOVA was conducted to examine the overall differences in fixation counts across the three AOIs (Content, Top Ad, Side Ad). Mauchly’s test indicated that the assumption of sphericity had been violated (χ²= 27.232, p < .001), therefore degrees of freedom were corrected using the Huynh-Feldt estimates of sphericity (epsilon = .913). Results indicated a significant main effect for AOI, F(1.825,458.160) = 207.565, p < .001, η2 = .453. Participants fixated more in the Content AOI than the Top or Side Ad AOIs. Additionally, participants fixated more in Top Ad than Side Ad AOIs.

A three-factor mixed (2x3·3) ANOVA was conducted to examine differences in fixation counts for the three AOIs (Content, Top Ad, Side Ad) as a function of search type (exact, semantic) and target location (Content, Top Ad, Side Ad) on successful trials. Results indicated a significant interaction of AOI by target location, Wilks' Λ = .235, F(4, 490) = 130.343, p < .001, multivariate η2 = .516. Post-hoc tests revealed several significant differences. When the task target was located in the Content region, participants fixated on the Content AOI more than the Top Ad AOI or Side Ad AOI. When the task target was located in the Top Ad region, participants had more fixations in the Top Ad AOI followed by the Content AOI and then the Side Ad AOI. However, when the task target was located in the Side Ad region, participants fixated less in the Top Ad AOI than the Side Ad and Content AOIs. It should be noted that when the target information was in the Side AOI, the number of fixations in the Content and the Side AOI was equal. Figure 14 illustrates the relationships between AOIs and fixation count. Figure 15 illustrates the relationships between AOIs and target location.

Figure 14

Figure 14. Fixation count of AOIs for successful trials, error bars are +/- one standard error.

Figure 15

Figure 15. AOI fixation counts across target locations in successful trial completion, error bars are +/- one standard error.

Mean fixation duration by AOI

Mean fixation duration of AOIs should help differentiate between exact and semantic searches. In previous literature, it has been said that cognitive load varies with different search types (Rayner, 1998). AOI fixation counts were calculated by summing the number of fixations located within an AOI on the web page. Fixations were defined using the preceding definition. AOI mean fixation durations were calculated by averaging the duration of fixations found within an AOI on the web page.

A small number of samples were found in several conditions because mean fixation duration cannot be calculated in AOIs where no fixations exist. Therefore, the three factor mixed (2x3·3) ANOVAs could not be calculated for mean fixation duration. Target location was collapsed due to previous findings of a significant main effect for search type in the overall page analysis for mean fixation duration.

A two-factor mixed (2·3) ANOVA was conducted to examine differences of mean fixation duration for three AOIs (Content, Top Ad, Side Ad) for each participant across search type (exact, semantic) for successful trials. Results indicated a significant main effect for AOI, Wilks' Λ = .779, F(2, 57) = 10.528, p < .001, multivariate η2 = .27. There was no main effect of search type. Post-hoc testing showed that participants had longer fixation duration in the Side Ad search than Content and Top Ads searches. Figure 16 illustrates the relationships between AOIs.

Figure 16

Figure 16. AOI mean fixation duration in milliseconds in successful trial completion, error bars are +/- one standard error.

 

Previous | Next