Almost every business or niche blog relies heavily on organic traffic for its survival. To ensure consistent traffic from search engines, we use several SEO techniques apart from creating useful content on a regular basis. But all these techniques will fail if your blog is NOT properly crawled and indexed by a search engine bot. And that's where Google Webmaster Tools (GWT) comes handy. Let's see how this excellent tool can help you find issues (if any) that may prevent your blog from being indexed correctly.
To kick things off, start by adding your blog's home page URL to the GWT dashboard. You will be asked to verify the ownership of your blog through one of the given methods. The easiest one is adding a site verification
tag on the home page of your blog. You can remove this tag once verification process is completed.
Submit Sitemap
This is the most important task that must be completed after adding your blog to the GWT dashboard. Sitemap is simply a file containing a list of web pages that may be crawled by the indexing bot. The most common sitemap format is the XML file format that is well understood by all major search engine crawlers. Nowadays, all popular blog content management systems like WordPress and Blogger automatically generates a sitemap. Self-hosted WordPress blogs generally use Google XML Sitemaps plugin to generate an optimized sitemap for their blog.
Now the question arises - Why at all we need to submit a sitemap? The answer is very simple. GWT system checks for any discrepancies in your sitemap that may generate errors or warnings during the crawl process. The most common error is the presence of a broken link (non-existent page) in the sitemap. Warnings may include those pages which are inaccessible (https secure page behind a login) or are wrongly redirected pages that leads to nowhere. Sitemap submission ensures you're well aware of these errors and warnings and can take necessary steps to reduce these discrepancies to the minimum.
Submitting a WordPress sitemap - You can access sitemap submission page through 'Site Configuration -> Sitemaps'. Simply type 'sitemap.xml' in the submission box and press the 'Submit Sitemap' button.
Submitting a Blogger sitemap - You have to type a bit more while submitting a sitemap for a Blogger blog. Type-in the following text 'atom.xml?redirect=false' and you're done.
You can also use robots.txt file to tell the crawlers where to find the sitemap through this directive - sitemap: http://www.yourblogname.com/sitemap.xml.
Check Robots.txt
Head over to 'Site Configuration -> Crawler access' option to review your directory and file access permissions for indexing bots. It is done through robots.txt file located in the root directory of your blog. GWT dashboard shows the accessibility and content of this file along with a list of user agents (crawlers) for the specified URLs.
Robots.txt file generation - You can also create a custom robots file through this option. Remember, robots file generated through this option must be replaced with the old version present on your web server.
Content removal request - Although non-existent pages are automatically removed from the Google index, still you can use 'Remove URL' option to submit the removal requests. These submissions do speed up the removal process and more importantly, you can also initiate re-inclusion requests for previously submitted removal requests.
Manage Sitelinks
Sometimes Google displays sitelinks right below the search result for a given website as shown in the image below. These sitelinks are not available for every blog or website. Google's search algorithm decides when and which sitelinks (if any) should appear in SERPs. Generally, you can find these links in your Google analytics dashboard in 'Top Landing Pages' report.
Internally, Google creates a bunch of sitelinks and shows them randomly in subsequent search queries for a given domain. You cannot control the content (blog posts) which would appear as sitelinks, BUT you can surely ask Google to demote a post (URL), thus preventing it from appearing as a sitelink for at least 90 days. You can access this option from 'Site Configuration -> Sitelinks'. It's an important tool in your arsenal for preventing selective links from appearing as sitelinks.
Declare Change of Address
If you're moving your blog to a new domain, this option can speed up things. Once you've completed the migration process, make sure landings on all old links result in permanent 301 redirect towards new links. After implementing this redirection, add the new domain in your GWT account and head on to 'Site Configuration -> Change of address' option for the old domain. Specify the new domain where your new blog exists and press the submit button.
This ensures indexing of the new domain is speedy and accurate supplemented with fast removal of corresponding old links in the database. Note: While migrating to a new domain, ensure that all the internal links within the content are changed or redirected as well.
Apply Geo-targeting & Adjust Crawl Rate
If you're writing for local audience or for visitors' from specific geographical location, you must use the 'Geographic target' feature accessible through 'Site Configuration -> Settings' option. Here you're clearly indicating to Google about the geographical location of your audience. This helps Google in serving customized and tailored search results of your blog for that particular geographic location. End-result is highly targeted traffic leading to good conversions.
Another important feature that can be leveraged is tweaking the crawl rate of the indexing bot. This option is particularly useful for blogs having massive amount of content with multiple sub-domains. Generally, on each visit crawler grabs a limited number of web pages so that it does not overwhelm your web server. You can use the 'Crawl rate' option to change this default behavior instructing the crawler to go deeper into the content tree gobbling up more web pages in a single visit.
Leverage SERP Report
One of the best things about GWT is the inclusion of comprehensive SERP report that can be used to refine your on-page optimization strategy. This report gives you the last 30 days SERP data in aggregated as well as in segregated form. Confused? Let's try to understand both forms and see how this data can be extremely beneficial for your SEO efforts as a whole.
Aggregated data - The image shown below presents the last 30 days SERP data in aggregated form. Remember, this graphic simply tabulates the approximate figures and should not be treated as accurate display of results. This handy result sheet shows you three important metrics associated with your blog's SERP performance. Each of these figures gives you a fair idea about how organic traffic from Google search engine is flowing towards your blog.
Queries - This important metric show the number of queries visitors typed on Google search engine to find the relevant content. Needless to say, your blog's content was listed somewhere in the result pages for all these queries.
Impressions - It's the number of times your blog posts appeared in SERPs on different user agents on different platforms. For a given query, your blog post may rank on a certain position, but it may not be counted as an impression or a view unless that very page is fetched by the user agent (generally a browser) where your content appears in the result listing.
Clicks - And last but not the least is the number of clicks received by your content's entries in SERPs. For a popular blog, this figure can be much much higher resulting in massive organic traffic from this well-known search engine.
Note: You can filter the above chart on the basis of 'search type' - (web, image, mobile, video) or/and the 'geolocation' (country) of the visitor.
Segregated data - The next figure shown below shows the break up of the above chart in tabular format. You can export both these data sheets in csv Excel format. The following table gives you some extremely useful insights about how your content is performing in SERPs. Let's take a closer look and try to understand how this data is like a goldmine for optimizing your SEO efforts.
Query - This is the meatiest part of the report where you get the actual search queries that visitors use to reach your blog. It contains highly competitive prime keywords as well as long tail keywords for your analysis.
Impressions & deviation - The next important metric is the number of appearances in SERPs for each of these keywords (search queries). The change compared to previous time frame is also shown beside each of these metrics.
Clicks, CTR & deviation - The appearances in SERP alone doesn't guarantee traffic. It's the click-through rate that matters the most, and that's what highlighted in this column.
Average position & deviation - And last but not the least is the average relative position on SERP pages for each given keyword as compared to previous set of data of similar duration. For each of these columns, the deviations (positive/negative) are also shown to get a fair idea about how your content is performing with passage of time.
Now the big question is - How we can leverage this report to the fullest? Well, as I said earlier, this is a goldmine and you must know how to extract the required information for improving your on page SEO. Fortunately, you can easily export this data to an excel sheet with all columns intact. Here's what you can do to work on improving your on page SEO after exporting the data. Check out for top keywords (sending maximum traffic) and filter those with large negative deviations. Run the same query on Google search engine and see which site has outranked you recently for this keyword. Skim through the HTML and content of that page and make necessary improvements in your web page. This won't give you instant results, but can certainly prove vital in at least holding the current position (minimize damage) for a longer duration.
The second important trend that can be leveraged in this report is the list of new keywords that have recently gained prominence although not sending that much traffic compared to the prime search queries. These long tail keywords generally have large positive deviations contributing small amount of traffic. Pick out those that are relevant to your niche and work on optimizing your content for these long tail keywords. If you manage to rank well even for half a dozen of these keywords, you'll notice large inflation in your organic traffic.
Now Your Turn
There are several other sections for diagnosing crawl errors, for checking Google+ author stats and for analyzing search impact due to +1's received by your content. I've left these sections for you to explore and see what can be gained out of them. In addition to this, I'll love to get your inputs about how you are using Google webmaster tools to optimize your website or blog.