This video is from our Website Optimization Course, free in HubSpot Academy: https://bit.ly/3duJqSF
In addition to this free course, try out Hubspot’s free Website Grader and see how your website scores (and learn how to fix it):: https://bit.ly/2VcDpDL
The most important part of on-page SEO allows search engines to crawl and index your pages. On-page SEO is all about increasing your discoverability in search engines. You can’t do this without allowing these search engines to access your pages. This is where crawling, indexing, and ranking comes into play.
In this video, we will take a look at how this process works and learn how to enable it on your webpages. Search engines have 3 primary functions. Crawl, or search webpages and look over the code and content for each URL. Index, or store and organize the information it finds from crawling. And rank, which allows search engines to provide the pieces of content that best match a person’s search query. And how does Google find your content? Basically, Google uses a huge set of computers to crawl billions of pages on the web. This crawler essentially begins with a list of webpage URLs generated from previous crawls and then augments those pages with the sitemap data that are provided.
During the crawling process, the Googlebot looks for new sites, updates to existing pages, and any broken links. Once the crawling process is complete, all of the results are fed into Google’s index, and any new sites or updated content will be listed accordingly. While it processes the results, Google looks at information on your pages such as title tags, meta descriptions, alt tags, and more. Think of this process as a library. Books need to be identified, categorized, and placed in the right order on shelves. Crawlers are performing similar tasks as a librarian. They are organizing all of the contents on the internet into ways that make them easily accessible.
Can your pages already be found by Google? The best place to start is to understand which pages on your website are already being indexed. If your page is being indexed, that means that Google is able to crawl your page and store its contents. The best way to do this is by using an advanced search operator. An advanced search operator sounds complicated, but it’s actually pretty simple. In the search engine, type “site:yourdomain.com” replacing the domain with your own. Example of an advanced search operator. So if we’re doing this for HubSpot, we’d type “site:hubspot.com” into Google. We see that HubSpot has 217,000 pages currently being indexed by Google. This isn’t going to be an exact number, but it will at least give you an idea of how many of your own pages are currently being indexed. If no content is indexed yet for a site, Google will let you know that your search did not match any results. If no content is found, your next step should be to create a sitemap that you can submit to Google.
A sitemap is a file of code that lives on your web server and lists all of the relevant URLs your website is carrying. Your sitemap helps search engine web crawlers understand how your website is built so they can evaluate and rank it more easily. XML sitemaps are designed specifically for search engines like Google. Google needs to be able to find webpages anchored within a website no matter how old or deeply nested they might be in that website’s domain. For this reason, an XML sitemap is a crucial component of a blog, where article pages are constantly bumped further back into the website’s archive as new content is published.