Professional Development

Article Library » Guest Writers

Website Ranking Consistency

Share This Article: On Twitter On Facebook Print


Brief Report: Consistency of Search Engine Rankings for Autism Websites

Brian Reichow, Adam Naples, Yale Child Study Center, Timothy Steinhoff, Jason Halpern, Temple University School of Medicine, Fred R. Volkmar, Yale Child Study Center

The World Wide Web is one of the most common methods used by parents to find information on autism spectrum disorders and most consumers find information through search engines such as Google or Bing. However, little is known about how the search engines operate or the consistency of the results that are returned over time. This study presents the results of analyses of searches from 2009, 2010, and 2011 World Wide Web searches for information on autism. We found that over time, consumers are likely to have different search experiences yielding different results, and we urge consumers to use caution when using the World Wide Web to obtain information on autism.

Brief Report: Consistency of Search Engine Rankings for Autism Websites

The World Wide Web is one of the most common methods for obtaining health-related information (Fox, 2011; Fox & Jones, 2009; Khoo, et al., 2008; Wainstein, et al., 2006), and is the most common method parents use to find information on autism spectrum disorders (ASDs; Chowdhury et al., 2002; Mackintosh et al., 2005). There are now millions of results that are returned when searching for the term autism1 (64.5 million results are returned in Google on June 4, 2011). However, the information available on the World Wide Web is constantly changing and consumers are likely to find information about ASDs that have mixed quality (Chowdhury et al., 2002; Reichow et al., in press; Scullard, Peacock, & Davies, 2010). Popular search engines (e.g., Google, Bing, Yahoo) are the means by which most people locate health-related information (Eysenbach & Köhler, 2002; Khoo et al., 2008) and have great influence on the websites consumers locate and visit (Wacogne & Scott-Jupp, 2010). The exact ways in which search engines operate consist of proprietary information to which only a select few employees are granted access. Companies are regularly modifying their algorithm (McCracken, 2011, March), which controls the content that is returned to the user. User interfaces of search engines are also constantly changing with new aspects being added (e.g., location search, instant results, voice search; Google, 2010). Significant differences between search engines have been reported (Bar-Ilan, 2005), however it appears that individuals use the sites similarly (Hotchkiss, 2005). Most consumers locate health-related information by searching for one to three terms, deciding to visit one of the top three websites listed (ranked or sponsored) which they scan in less than 5 seconds (Eysenbach & Köhler; Hotchkiss, Alston, & Edwards, 2005). Although consumer behavior appears to be consistent, the changing algorithms along with different and changing user interfaces of search engines and updated information on web pages are likely to create a situation in which searching for the same terms will lead to different results. Because Internet search engines are the most used source for obtaining information, a deeper understanding about how they retrieve information for consumers is of great interest. In this study we sought to evaluate the consistency of website rankings across time and between search engines when searching for information about ASDs.



We used lists of websites for the samples of this study. Our first sample consisted of the top 100 websites returned when autism was entered into the Google (, Yahoo (, and Bing ( search engines on June 4, 2009. Our second sample were the results of Google searches of autism conducted on June 4, 2009, July 4, 2009, June 4, 2010, and June 4, 2011 (location set to United States; the results of these searches are shown in Table S1). Our third sample included the results when autism and one or two additional terms (i.e., autism spectrum disorders, autism causes, autism symptoms, autism treatment, autism vaccines) were entered into the Google search engine in June 2010 and June 2011. Finally, we examined seven lists of recommended websites containing information about ASDs (Bloomquist, 2005; Charman, 1999; Coates, 2009; D’Auria, 2010; Polirstok & Lesser, 2003; Sabo & Lorenzen, 2008; Seeman, 2005).

Data Analysis

We used analytic statistics (i.e., Spearman rank correlations with pairwise deletion) and descriptive statistics to examine the consistency of the results obtained when looking for information on ASDs using search engines across time and between search engines. We examined search engine consistency across time using results obtained from Google. Specifically, we examined the (a) 1-month consistency by comparing the June 4, 2009 and July 4, 2009 searches; (b) 1-year consistencies by comparing the June 4, 2009 and June 4, 2010 searches; (c) 2-year consistency by comparing the June 4, 2009 and June 4, 2011 searches. We examined the consistency of rankings between search engines by comparing the June 4, 2009 Google, Yahoo, and Bing searches. Finally, we used descriptive statistics to examine (a) the percentage of the same websites in the pairwise time comparisons, (b) the percentage of URLs from the original June 4, 2009 search that were active in June 2011, (c) the percentage of URLs in the recommended lists that were active in July 2009, and (d) the percentage of websites from the multi-word searches that were in both the June 2010 and June 2011 results.


The rank-order correlations for the consistency of Google results slightly decreased over time after one-month, rs = .92, and more markedly after one- and two-years, rs = .79 and rs = .82, respectively. Although these correlations are high, the percentage of websites in the top 100 at these time points decreased significantly; after one month, 82%, 54%, and 48% of the original results remained in the top 100 1-month, 1-year, and 2-years later. We also examined the consistency of results across search engines for the June 4, 2009 search. The consistency of website rankings between search engines were moderate. The rank order correlations for the 31 websites in the top 100 for Google and Yahoo was rs = .70 and the rank order correlation for the 31 websites in the top 100 for Google and Bing was rs = .61. Only 21 websites appeared in the top 100 across all three search engines (these 21 websites are shown in Table 1). With the exception of one website (, the top 10 results from Google were found in Yahoo and Bing. Because consumers typically only use the first page of results, we also examined the consistency of the top 10 sites when the term autism was searched alone and with additional term(s) using our Google June 4, 2009, June 4, 2010, and June 4, 2011 searches. The consistency of the top 10 websites returned when autism was used alone was high; of the top 10 results of the June 4, 2009 Google autism search, 100% were located in the top 31 results for all subsequent Google searches and the top 5 sites found on June 4, 2009 remained in the top 10 of all subsequent searches (the top ten results for the “autism” search using Google on June 4, 2009, June 4, 2010, and June 4, 2011 are shown in Table 2). The 1-year consistency when autism and additional terms were searched was lower. When autism was searched with additional terms in Google, on average, 56% of websites located in June 2010 were located in the same search in June 2011 (range 0% for autism vaccine to 90% for autism spectrum disorder). Finally, we examined the ability to locate previously recorded websites. Our first examination involved determining the percentage of URLs from the original June 4, 2009 that remained active in June 2011. Of the top 100 websites from 2009, 88% were active in 2011, including 100% of the top 10. We also examined the currency of URLs from lists of recommended websites that had been published in peer-reviewed journals (e.g., Bloomquist, 2005; Charman, 1999; Coates, 2009; D’Auria, 2010; Polirstok & Lesser, 2003; Sabo & Lorenzen, 2008; Seeman, 2005) that were active in July 2009. There was high variability in the percentage of recommendations that were still active (Bloomquist: 1 of 8, 13%; Charmam: 1 of 5, 20%; Coates: 9 of 12, 75%; D’Auria: 17 of 17, 100%; Polirstok & Lesser: 19 of 41, 46%; Sabo & Lorenzen: 12 of 14, 86%; Seeman: 14 of 24, 58%).


This report presents findings from our multi-year evaluation of autism websites. As the use of World Wide Web continues to grow, especially with the increasing access people have with portable devices, the influence websites have on consumers is likely to increase. Therefore, it is important to understand what consumers are likely to encounter when using the World Wide Web to locate health-related information. While our previous work has focused on the characteristics and quality of autism websites (Reichow et al., in press), our current analysis focused solely on the consistency of search engine returns between search engines and over time.

Our findings suggest that over time, consumers are likely to have different search experiences yielding different results. Although the top ten results were similar across time, the remainder of sites located had great variability. Given research suggesting consumers often limit their use to the top ten results of a search, many consumers may find similar results across time. However, changes in search engines (e.g., addition of location feature, news results, different search algorithms) and differences between search engines, create a situation in which consumers using common web-searching strategies are likely to encounter different information over time and when using different search engines even when searching for the exact same term. Therefore, consumers should remain attentive when using the World Wide Web to obtain information on ASDs. Based on the results of this examination of search engine results, we offer four observations and recommendations to assist consumers when searching for information on ASDs. Our first observation and recommendation regards the top websites listed in a search, which are the sites consumers are most likely to visit. Of the top 10 results of the Google autism searches, 100% were located in the top 31 1-year and 2-years later. This suggests there is likely less variability at the higher ranked positions, which are the websites consumers are most likely to visit. Furthermore, the top 5 sites found on June 4, 2009 remained in the top 10 of all subsequent Google searches. The consistency of the two- and three-word searches was much lower (average 56%). Our observational analyses of the returns suggest that many of the multiword searches returned sites with more specific information and current news stories, which might be the cause of this lowered level of consistency. Regardless of the underlying cause, consumers should be aware that while multiple-word searches might lead to more specific

results, they might also encounter greater variability across time. Our second observation and recommendation regards changes in the user interface of search engines. Between our 2010 and 2011 searches, Google enabled a feature that determined your geographic location and tailored search results for that location (Google, 18 October 2010). We found that differences in location created large differences in the top 10 returns for a search; 3 of the top 10 sites (30%) were different in Google when the search location was set to different cities (e.g., New York, Chicago, Los Angeles) compared to the more general United States location search. Smaller changes were seen in the top 100 returns; up to 5 of 100 sites were different between searches. Many consumers might find it helpful to be directed to local resources (e.g., state autism societies, university clinics), but highly relevant sites might be missed by these substitutions. It might also create confusion for individuals searching for someone in a different town (e.g., a grandmother in Orlando searching for information about autism for her grandson living in Seattle). Other user interface changes have included insertion of news results and images in website searches. Given that most individuals typically only search the top three or four sites listed after a term is searched (which may or may not include advertisements or sponsored sites) and rarely view a second page of results, changes like this can produce dramatic differences. We urge consumers to be aware of the influence these features have on returns and to not let the changes distract them from obtaining the most relevant returns. Our final observation and recommendation centers around making lists of recommended websites. We found that 12% of websites in the top 100 of a Google search in 2009 could no longer be accessed using the same URL in 2011. The retrieval rate is even lower when examining lists of recommended websites in peer-reviewed journals; our results showed that up to 87% of a website recommendation list might have broken links. Thus, experts wishing to pass along valuable information to practitioners or parents might instead be passing along the frustration of not being able to find the recommended information. We think there are two possible explanations for the loss of so many websites from recommended lists. First, it is possible that the websites remain active but the page has changed locations (i.e., the same information is available but at a different URL). To ensure website locations remain current creators of recommendation lists should update the list on a regular basis. The second possibility is that a website had ceased to exist. This can be problematic if someone located quality information they wish to access later but are unable. Based on analysis of the lists with thegreatest number of active links (e.g., D’Auria, 2010; Sabo, 2008), we feel that websites containing a shallower or less complex directory tree (URL), which is characterized by fewer back-slashes, will likely create a recommendation that was more likely to withstand the natural decay of unmaintained websites’ subpages (i.e., will be more likely to be found after the passage of time). We suggest that when creating a list of recommended websites to limit the recommendation to the protocol (e.g., http), the host (e.g., www), the subdomain (e.g., cdc) and the top-level domain (.gov). When it is not possible to access a site with these four elements, we suggest minimizing the umber of sub-domains, file names, or file extensions, (underlined in example; included in the recommendation list. Although we attempted to systematically evaluate the consistency of search engine results across time and between search engines, our results do have limitations. First, our study evaluated search engine results when approximating consumer searches for autism information. We do not know if our results would generalize to the World Wide Web beyond autism (e.g., the consistency of results when using search engines to locate information on childhood cancer) or to other search engines (e.g., less popular search engines, meta-search engines). Second, our results show website rankings can be highly variable and volatile (for reasons that may not be completely known), which likely influenced our results. For example, when two members of our research team searched autism in Google 30 minutes apart, the 3rd and 4th results had switched

places. Although we do not feel that these small differences exerted great influence on our findings and conclusions, we feel they should be acknowledged as a limitation.



Bar-Ilan, J. (2005). Comparing rankings of search results on the web. Information Processing

and Management, 41, 1511-1519.

Bloomquist, S. (2005). Autism resources for public libraries: Issues, challenges, and

recommended resources. Indiana Libraries, 24(3), 23-31.

Charman, T. (1999). Autism resources and information on the Internet or World Wide Web.

Autism, 3, 97-100.

Chowdhury, J., Drummond, J., Fleming, D., & Neitfeld, S. (2002). Content analysis of online

autism specific sites. Journal on Developmental Disabilities, 9(2), 157-165.

Coates, H. (2009). Autism spectrum disorders: Wading through the controversies on the web.

Medical Reference Services Quarterly, 28, 259-267.

D’Auria, J. P. (2010). Autism on the web: “Oh, the places you’ll go!”. Journal of Pediatric

Health Care, 24, e11-e15.

Eysenbach, G., & Köhler, C. (2002). How do consumers search for and appraise health

information on the World Wide Web? Qualitative study using focus groups, usability

tests, and in-depth interviews. British Medical Journal, 324, 573-577

Fox, S. (2011). The social life of health information, 2011. Washington, DC: Pew Internet &

American Life Project.

Fox, S., & Jones, S. (2009). The social life of health information: Americans’ pursuit of health

takes place within a widening network of both online and offline sources. Washington,

DC: Pew Internet & American Life Project.

Google (18 October 2010). Google public policy blog: More transparency and control over

location. Information obtained 17/6/2011 from


Hotchkiss, G., Alston, S., & Edwards, G. (2005). Google eye tracking report: How searchers see

and click on Google search results. Information obtained 17/6/2011 from

Hotchkiss, G. (2006). Eye tracking report: Google, MSN and Yahoo! compared: An in depth

look at interactions with Google, MSN & Yahoo! using eye tracking methodology.

Information obtained 17/6/2011 from


Khoo, K., Bolt, P., Babl, F. E., Jury, S., & Goldman, R. D. (2008). Health information seeking

by parents in the Internet age. Journal of Paediatrics and Child Health, 44, 419-423.

Mackintosh, V. H., Myers, B. J., & Goin-Kochel, R. P. (2005). Sources of information and

support used by parents of children with autism spectrum disorders, Journal on

Developmental Disabilities, 12, 41–51.

McCracken, H. (2011, March 14). Engine overhaul: Google takes steps to downgrade shoddy

content in search results. Time, 177(10), 58.

Polirstok, S. R., & Lesser, D. R. (2003). Useful online information, resources, and interventions

for speech language pathologists and teachers of students with autistic spectrum

disorders. Topics in Language Disorders, 23, 166-167.

Reichow, B., Halpern, J., Steinhoff, T., Letsinger, N., Naples, A., & Volkmar, F. R. (in press).

Characteristics and quality of autism websites. Journal of Autism and Developmental


Sabo, R. M., & Lorenzen, J. M. (2008). Consumer health web sites for parents of children with

autism. Journal of Consumer Health on the Internet, 12, 37-49.

Scullard, P., Peacock, C., & Davies, P. (2010). Googling children’s health: reliability of medical

advice on the interent. Achieves of Disease in Childhood, 95, 580-582.

Seeman, C. (2005). Autism spectrum disorders: A resource for ASD information on the web.

College & Resource Libraries, 66(9), 646-649.

Wacogne, I., & Scott-Jupp, R. (2010). The role of Google in children’s health. Achieves of

Disease in Childhood, 95, 576-577.

Wainstein, B., K., Sterling-Levis, K., Baker, S. A., Taitz, J., & Brydon, M. (2006). Use of the

Internet by parents of paediatric patients. Journal of Paediatrics and Child Health, 42,





1 – throughout this paper, the search terms entered into the search engine will be italicized since

searching with or without quotation marks can alter search results.



Table 1. Twenty-one websites appearing in all three autism searches conducted in Google,

Yahoo, and Bing on June 4, 2009 and their corresponding rank.








1 11 1

2 2 3

3 1 5

4 6 8

5 4 2

6 10 4

7 8 87

9 5 45

10 7 11

11 13 10

15 14 7

16 57 9

17 45 34

19 33 31

26 50 46

27 26 88

41 12 17

44 16 19

50 90 41

52 15 100

76 66 57



Table 2. URLs for the top ten websites for autism search in Google on June 4, 2009, June 4,

2010, and June 4, 2011 (location for 2011 set to “United States”).

Rank June 4, 2009 June 4, 2010 June 4, 2011












Ask Dr. Susan