How to do keyword research

"It might sound crazy but I am extremely passionate about keyword research because I take pride in really digging deep into a market and delivering a comprehensive report that can make all the difference to an SEO campaign and to the business as a whole. Keyword research is about much more than just crunching data, it's about finding opportunities." That's me rambling on about my enjoyment of keyword research.

This guide is like a recipe book for SEO keyword success; it details all the ingredients (or tools) that you are going to need - don't worry they are free too! It also guides you through the whole process so you can literally sit at your computer with this guide open and follow it step-by-step until you've got yourself a hit list of keywords that your business should be focusing on.

This guide is about spotting opportunities in your marketplace those keyword gems that your competitors have missed that provide laser targeted traffic to your website and mean sky-high conversions all for as little money as possible.

I want this lens to be a priceless tool that empowers you to make a difference to your website's search engine ranking.

Why bother? That's always the question on peoples' lips and quite rightly so because why spend your time doing something that won't be beneficial or make an impact in your business?

Firstly, I think it is important to explain why search engine optimisation as a practice is so crucial to your business.

There was a time when individuals and businesses would buy products from a high street and search for services in a phonebook. Now, the majority of this happens online.

In a survey, it was found that over 80% of people looking for something offline searched for it online which means that your business needs to be visible.

A jaw-droppingly high percentage of these said that if they can't find a website or contact details online for a business then they'll use a competitor.

If you own a business you will probably be familiar with the 80/20 principle - where 80% of something comes from 20% of the work.

In the world of keywords, 20% of potential keywords (or sometimes even less) will provide 80% of your natural search traffic.

To put it bluntly, if you're not hitting the right 20% of keywords you are missing out on money.

Let me give you another fact; 40% of search traffic will click on the first result of the search result page, close to 90% of search traffic won't make it past the first page. That means that if your website isn't on the first page and preferably at the top of the pile, you are missing out on a massive opportunity.

Getting to the top of that pile in a natural and ethical way is what SEO is all about and that starts with intelligent keyword selection, this guide will show you how to determine your most relevant and commercially worthwhile keywords.

Keyword selection in reality involves a great deal of common sense, experience, number crunching and climbing inside your potential customer's mind to think how they'd think - what words would they use to search for your business?

To answer my original question, why bother?

* It's the foundation of a successful SEO campaign (less waste)
* It helps you to focus your efforts and resources for maximum return (more money)
* It is a step towards more potential customers finding you (more eyeballs)

* A sprinkle of patience
* A measure of buyer behaviour
* A dash of common sense
* Microsoft Excel or Google Docs - but preferably Excel
* Google Keyword Tool - you are going to need a Google Account (it's free!)
* Microsoft Ad Intelligence - this is an add-on to Excel so you see now why Excel was preferable.
* Bing Webmaster tools
* SEO Book Rank Checker
* SEO Book Toolbar
* SEOMoz Term Target
* Google Analytics

So you've got all your tools together, now is the time to pull together masses and masses of raw data - the more time you can invest here the better. Obviously there is that fine-line between valuable time investment and getting too bogged down in details but I would say you should prepare yourself for around 10-20 hours work here depending on your market, its competitiveness and the depth you wish to go into.

Bing and Yahoo's recent search results tie up has resulted in a large 'hypothetical' consolidated search player (hypothetical because they haven't merged but Yahoo is now utilising Bing's search results). This means that Bing now effectively holds 30% of the search market still way behind Google's approximate 60% share of the market but a sizeable chunk of the market and many millions of eyeballs to get your business in front of.

In order to see the best results you should really research and optimise for the two major search engines separately; this is because they have slightly different algorithms and potential differences in traffic, keywords used and so on.

Here's another insiders trick on how to spot keyword gems

In my opinion, SEO best practice would be to research keywords in both search engines and compare results, helping you to determine which search engine it will be more worthwhile concentrating on for that particular keyword. For example, 'accountants in Manchester' might be a highly-competitive keyword in Google but barely touched in Bing - no prizes for guessing which search engine I would focus my optimisation for that keyword on!

7 simple steps%u2026

1. Excel Spreadsheet
2. Mine data from Google Analytics
3. Mine data from Google AdWords Keyword Tool
4. Mine data from Google
5. Mine data from Microsoft AdCentre
6. Mine data from Bing
7. Setup conditional formatting

Step 1 - Spreadsheet Setup

Fire up Microsoft Excel and create a spreadsheet to input data to. It really is much easier if you use Microsoft Excel (as this guide has been created to guide you through the process using Excel) but of course any spreadsheet software is fine.

You need the following columns:

* Keyword
* Search Volume (Google)
* Search Volume (Bing)
* Competitors (Google)
* Competitors (Bing)
* KEI Google ("Search Volume" divided by "Competition")
* KEI Bing ("Search Volume" divided by "Competition")
* Relevance (as you perceive it - core=1, outer core=2, long tail=3)
* Current rank/current visibility Google (if applicable)
* Current rank/current visibility Bing (if applicable)
* Highest ranking page (if applicable)
* Expected percentage of search traffic (if applicable)
* Potential Traffic if ranked no1 Google (40% of total search volume)
* Potential Traffic if ranked no1 Bing (40% of total search volume)

Initially you will need to setup conditional formatting to highlight any duplicates in the keyword column because once you start bringing keywords in from multiple sources it can soon start to add up.

Step 2 - Head to Google Analytics to pull together all that useful data you already have about your website

You've got a head start on everyone else reading this guide because you are part of that exclusive club which has an insight into the market beyond what is publicly accessible which = competitive advantage.

If you are like 99% of Google Analytics users, you probably only use 0.01% of the power that this application has to offer; you installed it on your website because you'd heard it was a good idea and you probably look at it every once in a while to see how many people looked at your website.

Don't worry if this is you because there is no time like the present to start harnessing the awesomeness of Google Analytics and putting all that data to good use.

Once you've logged into your Google Analytics account, for the purposes of this exercise you are really looking to ascertain which keywords currently bring you traffic, what the bounce rate is for these keywords, which pages visitors land on for those keywords and what percentage these keywords convert at.

Please note that the conversion rate will only be available if you have set up goals in Google Analytics for your website.

I would advise having a separate section within the master spreadsheet to analyse keywords that you are already visible for. This allows you to make comparisons and judgements on keywords you already rank for, otherwise they might end up lost in a sea of other keywords.

Traffic sources > Keywords > Non-paid > Expand rows to 250 > Bounce rate + Conversion.

Step 3 - Head to the Google AdWords Keyword Tool

You need to create a Google account but it's free and it opens up the use of loads of fantastic applications and online services.

Using exact phrase match (add speech marks before and after the keyword), enter a few of the keywords that you think your customers are likely to use.

This should provide you with a lengthy list that is exportable into CSV format. Select all of the keyword suggestions and download to your computer.

For the purposes of this exercise, you really want to get rid of all the columns except for the keyword and local search volume as this populates the first two columns of your master spreadsheet.

Once you've done this, you can eradicate any keywords which are obviously no good as the list is auto-generated by Google and remember how I said that no machine is as good at keyword research as a human.

Step 4 - Mine data from Google

Using a regular search query (exact phrase match again) and the advanced 'allintitle' search you can build a competitive landscape for each keyword.

Simply search (using Google of course) for each keyword making a note of the number of results, then search again but this time add allintitle: to the front of your search query. This can help to give you an idea of the number of pages currently being optimised for that keyword.

I then find the average of the two figures to build a more accurate picture. Complete your spreadsheet accordingly.

This is quite a time-consuming process but an essential part if you are to find under-served niches in your market.

You can also use Google to search for your highest ranking page for a keyword and find the current rank/visibility of each keyword. To get an accurate picture of these things however you must log out of your Google account first.

Step 5 - Mine data from Microsoft AdCentre Intelligence

This is Microsoft/Bing's version of the Google keyword tool. It works in much the same way except it is an add-on to Microsoft Excel.

It has a keyword suggestion tool and the information it outputs can then be copied into your master spreadsheet.

Step 6 - Mine data from Bing

You need to do the same thing as you did for Google (in terms of finding the number of competitors) so head to Bing and enter your keyword with quotation marks around it to ensure exact phrase match and then just make a note of the number of pages in the search results (the number is at the top, below the search query box).

Step 7 - Setup conditional formatting

This is an extremely useful functionality of Microsoft Excel, it allows you to automatically highlight potential target keywords because you can set a box to turn blue if the KEI is above average or if the competing pages is below average.

It can help you create a shortlist (for further investigation) before the final hit list.

You should now have a full spreadsheet with an abundance of data and a plethora of keywords to choose from. Now you should move onto keyword selection below.

A note to anyone wondering why I haven't talked about Google Search trends or the fluctuations in search volumes; these figures in my experience rarely seem to portray an accurate representation of the way the market is moving. Search volumes rise and decline for all sorts of reasons, I understand that it can be useful in some circumstances but to date I have yet to find them add any real value to my keyword research or analysis. I will occasionally use it as a yardstick in order to make a judgement call on a keyword which I've flagged up as being seasonal/time dependant.

You should now have a list of between 250-500 potential keywords to filter, crunch and sort in order to develop a hitlist to target.

There are essentially two ways to select keywords:

1. Look at your existing site structure and work backwards looking to keywords to fit the pages
2. Look for keyword opportunities and then fit into your site structure adding new pages and content as necessary

I recommend the latter - it just helps to ensure not a single opportunity is missed.

Keyword Selection Criteria

When selecting keywords you are looking at the following:

* Relevance - how closely does it match what you offer?
* Search volume - how many people locally and globally searched using that keyword
* Competitiveness - how many other websites are competing to rank well for that keywords
* Strength of competitors - what sites will you be competing with? It doesn't matter if there are only handful, if they are big websites with many thousands of backlinks, huge budgets and SEO teams that work round the clock. Use SEOMoz's term target tool to help you work out how well optimised a page is and use SEOBook's SEO x-ray to dig deeper into competing pages.
* Searcher intent - are they browsing or buying? Using your common sense and market experience to make a judgement about what that person is looking to do.
* Current visibility/rank*
* Conversion*
* Bounce rate*

*only available to existing websites with a web analytics programme installed.

There are a few Microsoft Excel tactics that can help you initially weed out the 'dead-end' keywords. This includes keywords with very low search volumes, particularly high levels of competition or a particularly low KEI. Conditional formatting is a handy Excel feature which can help you spot these with relative ease.

KEI (keyword effectiveness index) is a useful metric calculated by taking the search volume and dividing it by the number of competing web pages. The resulting figure should be taken into account when making a decision rather than being the reason for selecting a keyword because the KEI doesn't truly reflect the competitiveness of a keyword as it doesn't take into account the quality of competitors but rather the quantity of competitors.

Once you've done this, it allows you to highlight potential keywords that you can then investigate and analyse a little further.

Group similar keywords and keyphrases together for example there might be a core keyword with only slight variations like 'golf clubs', 'buy golf clubs', 'where to buy golf club'. Selecting a core keyword but with multiple variations and qualifiers can allow you to optimise a web page for more than one keyword which gives traffic to your site a boost for little or no extra work.

This strategy will also help to enhance the natural appearance of your website in the eyes of the search engines as you can vary anchor text on inbound links and boost your visibility across the keywords you selected.

If you find a large number of variations of the keyword, try to select a maximum of 15 by refining your list using the selection criteria discussed earlier. Sometimes you may be able to consolidate qualifiers or variations that are too similar.

The idea is that you will be selecting keywords to focus individual web pages on 'champion pages'. You can then create a hub of content and relevance around these or inject into existing pages on your website. By having this narrow focus (but clever spread of variations) on each of your web pages you can really improve your search engine ranking.

Keyword selection is a science and does require some practice but the only way to really get to grips with it is to have a go.

Relevance

You might have found that dream keyword but it should be relevant to your business either through content that you already have or content that you are going to produce.

You should have graded your keywords already so this is an extra yardstick to determine which keywords make the final hitlist, remember that you want a good spread of core, outer-core and longer tail keywords.

Search volume

Some might tell you to only target keywords with massive search volumes, this isn't a particularly savvy move because in reality everyone else is chasing those 'big head' terms. Why not pool a few less trafficked keywords and really mop up?! Even if it is a few hundred visitors a month, if you can really dominate that keyphrase with minimal effort and repeat this process across a number of keywords you can build a really effective SEO campaign.

Competitiveness & Strength of competitors

You should invest some time really looking at your shortlist of keywords and making judgements about the strength of competing pages.

Look at the number of backlinks a website has, it's PageRank (a good indication of strength as it takes into account links, age of domain and perceived authority) and also look at the business behind the website, if they are a large company then they could well have a number of in-house SEOs or at least a hefty SEO budget administered externally.

Searcher intent

This is a hugely important aspect of a keyword and one so often overlooked by other SEOs. This is the area which really sets UP apart from the other keyword research services.

We analyse each keyword to ensure that the traffic that lands on our websites will do what we want them to. No use in driving loads of visitors to your website if they are just going to hang about and not really convert into sign-ups, sales and cash!

Look for indicative statements like 'buy', 'purchase', 'compare' and these can usually result in high-converting keywords.