You deleted how many pages? 130M and heres why - brightonSEO Autumn 2021
•
1 like•2,456 views
In this talk David will tell the story of Trainline’s multi-year crawlability project code named Black Widow. Find out why millions and millions of pages were deleted, the results that followed and what we learnt along the way.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to You deleted how many pages? 130M and heres why - brightonSEO Autumn 2021
Similar to You deleted how many pages? 130M and heres why - brightonSEO Autumn 2021 (20)
Recently uploaded
Recently uploaded (20)
You deleted how many pages? 130M and heres why - brightonSEO Autumn 2021
- 1. You deleted how many pages? 130M! And here’s why David Lewis | Trainline SLIDESHARE.NET/DAVIDLEWIS93
- 4. 10 September 2021 #brightonSEO 4 David Lewis Senior Technical SEO Manager @ Trainline 3.5 years at Trainline, 6 years in SEO Advocate of personal development Keen meditator Guitar player Big sports fan https://www.linkedin.com/in/david-lewis-48462751/
- 5. About Trainline Trainline is the leading independent rail and coach travel platform selling rail and coach tickets to millions of travellers worldwide. Via our highly rated website and mobile app, people can seamlessly search, book and manage their journeys all in one place. We bring together millions of routes, fares and journey times from more than 270 rail and coach carriers across 45 countries. We offer our customers the best price for their journey and smart, real time travel information on the go. Our aim is to make rail and coach travel easier and more accessible, encouraging people to make more environmentally sustainable travel choices. i. Based on transactions from weeks commencing 06/10/2019 to 23/11/2019 ii. iOS rating from UK AppStore as of 29 November 2019 iii. Trainline data for November 2019 (incl. UK consumer number Web/App and EU Web/App) iv. Average monthly visits for year ended 29 February 2020 v. UK consumer mobile app transactions for the year ended 29 February 2020 Trainline is the world’s leading independent rail and coach travel platform We sell tickets on behalf of more than 270 rail and coach companies and are adding more all the time We sell train and coach tickets worldwide, helping our customers travel in and across 45 countries in Europe and the rest of the world We sell tickets to people living in more than 175 countriesi in 14 languages Our customers can book in 10 currencies and payment methods include Apple Pay, PayPal, SOFORT & iDEAL We have built Europe’s leading train and coach app – with a 4.9/5* ratingii There are more than 32.4m cumulative downloads of our appiii Our platforms host more than 90m visits per monthiv 76% of our transactions are mobilev Our team numbers more than 600 people and more than 40 nationalities, including 300+ travel tech specialists and engineers >160 Train companie s >110 Coach companie s
- 6. Crawl Budget = Crawl Rate Limit + Crawl Demand 10 September 2021 #brightonSEO 6 https://www.botify.com/blog/crawl-budget-optimization
- 7. How did we get to a place of so much bloat? 10 September 2021 #brightonSEO 7
- 8. The customer came to us… 10 September 2021 #brightonSEO 8
- 9. …rather than us going to them 10 September 2021 #brightonSEO 9
- 10. How did we define value? 10 September 2021 #brightonSEO 10
- 12. What was the data showing? 10 September 2021 #brightonSEO 12
- 13. 1. Google was crawling valuable pages on average between 1-3 weeks 10 September 2021 #brightonSEO 13
- 14. 2. On average, 36% of all monthly crawls were to pages that were not valuable 10 September 2021 #brightonSEO 14
- 15. 3. We had inadequate internal linking architecture – only 36% coverage 10 September 2021 #brightonSEO
- 16. Our project goals 10 September 2021 #brightonSEO 16
- 17. 1. Delete 130 million pages that had no value 10 September 2021 #brightonSEO 17
- 18. 2. Achieve 100% internal link coverage 10 September 2021 #brightonSEO 18
- 19. 3. Reach 90% Googlebot crawl share to valuable pages 10 September 2021 #brightonSEO 19
- 20. 4. Improve average position of optimised URLs 10 September 2021 #brightonSEO 20
- 21. A new internal linking architecture 10 September 2021 #brightonSEO 21
- 22. 10 September 2021 #brightonSEO 22 Dave’s Zoom
- 23. We created new internal linking logic which enabled us to reach 100% coverage 10 September 2021 #brightonSEO 23
- 24. 10 September 2021 #brightonSEO 24 At the same time we orphaned roughly 1.5M pages that had no value
- 25. 6 months later… 10 September 2021 #brightonSEO 25
- 26. 10 September 2021 #brightonSEO 26 We observed an improvement in crawl performance, but it wasn’t enough
- 27. …so, we deleted 5M pages that had no value 10 September 2021 #brightonSEO 27
- 29. But 5M wasn’t anywhere near enough… another 125M pages needed to be deleted 10 September 2021 #brightonSEO 29
- 30. We had a duplicate version of every single UK train times page, across 6 subdomains 10 September 2021 #brightonSEO 30
- 31. 10 September 2021 #brightonSEO 31 And they were a total waste of crawl budget
- 32. Then… we monitored 10 September 2021 #brightonSEO 32
- 34. 10 September 2021 #brightonSEO 34 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 4 / 1 2 / 2 0 1 6 / 1 2 / 2 0 2 2 / 1 2 / 2 0 2 3 / 1 2 / 2 0 0 4 / 0 1 / 2 0 2 1 0 5 / 0 1 / 2 0 2 1 0 6 / 0 1 / 2 0 2 1 1 1 / 0 1 / 2 0 2 1 1 2 / 0 1 / 2 0 2 1 1 3 / 0 1 / 2 0 2 1 1 4 / 0 1 / 2 0 2 1 1 5 / 0 1 / 2 0 2 1 1 8 / 0 1 / 2 0 2 1 1 9 / 0 1 / 2 0 2 1 2 0 / 0 1 / 2 0 2 1 2 1 / 0 1 / 2 0 2 1 2 2 / 0 1 / 2 0 2 1 2 5 / 0 1 / 2 0 2 1 2 6 / 0 1 / 2 0 2 1 2 7 / 0 1 / 2 0 2 1 Percentage Indexed Date Indexed vs. De-indexed - Orphaned Pages % Indexed % de-indexed Sitemaps removed from GSC First sitemap data in GSC 86% of all deleted pages were de-indexed in 6 weeks
- 35. The results 10 September 2021 #brightonSEO 35
- 36. 10 September 2021 #brightonSEO 36 Googlebot crawl share for valuable pages improved from 64% to 87% Stage 1: Orphaning and Internal Linking Stage 2: 410’s 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Jan-19 Feb-19 Mar-19 Apr-19 May-19 Jun-19 Jul-19 Aug-19 Sep-19 Oct-19 Nov-19 Dec-19 Jan-20 Feb-20 Mar-20 Apr-20 May-20 Jun-20 Jul-20 Aug-20 Sep-20 Oct-20 Nov-20 Dec-20 Jan-21 Feb-21 Mar-21 Apr-21 May-21 Jun-21 Jul-21 Googlebot Crawl Share Percentage Date Googlebot Crawl Share Over Time
- 37. 10 September 2021 #brightonSEO 37 3X more URLs are now crawled by Googlebot and Botify on a monthly basis from where we started April 2020 November 2020 2 million URLs 1.7 million URLs 1.7 million URLs 800k URLs 2.6 million URLs 1.7 million URLs July 2021 1 million URLs 2.5 million URLs 1.7 million URLs
- 38. 10 September 2021 #brightonSEO 38 For previously orphaned URLs that were internally linked to for the first time - we improved their average position by 2, jumping from P8 to P6
- 39. 10 September 2021 #brightonSEO 39 We improved page depth from an average of 7 to 5
- 41. Sitemaps It’s not enough to just remove sitemaps from GSC to prevent crawling, you must delete the files too 10 September 2021 #brightonSEO 41 https://www.searchenginejournal.com/google-deleting-a-sitemap-wont-stop-us-crawling-your-site/409108/
- 42. Sitemaps Or so we thought… Deleting the sitemap files still isn’t enough, because of Google’s long- term memory 10 September 2021 #brightonSEO 42 https://www.searchenginejournal.com/googlebot-404/239325/
- 43. 10 September 2021 #brightonSEO 43 410’s – de- indexation We expected the 410’s to be de- indexed within 48 hours, but didn’t account for the time it would take Google to crawl through all the URLs
- 44. 10 September 2021 #brightonSEO 44 410’s - crawling Google will crawl 410’s again, and again, and again
- 45. Research If you think you’ve found the answer, keep looking 10 September 2021 #brightonSEO 45
- 46. 10 September 2021 #brightonSEO 46 Data That said, you can research all day long, but ultimately, be led by your data and make a judgement call
- 47. Thank you David Lewis Senior Technical SEO Manager Trainline