I was fortunate to be able to give an introductory SEO session recently at FireBootcamp.
The presentation was originally going to be a 45min intro, but thanks to tons of interesting questions, we ended up going for close to 2.5 hours. The presentation is broken into two parts. Part 1 is below, and Part 2 is coming soon.
It’s primarily aimed at developers and is pitched at an introductory level. But that said, a number of the questions lead us into advanced territory, which was lots of fun.
In this first part we covered the following main points:
A general introduction to SEO and where it fits in overall web marketing
The importance of understanding the business goals of a web site (this is especially important for developers who consult to enterprise clients)
The key factors to optimize on web sites, and how optimizing 3 simple areas can make a big difference to Google rankings
Things that can get a site penalised by Google
An in-depth look at Google analytics and how to use the main reports
An overview of Google analytics goals and how they can be used for ecommerce tracking
Discussion about the range of different analytics packages, and their varying levels of privacy and personally identifiable information functionality
Understanding the robots.txt file (and the disaster that can happen when it is incorrectly implemented)
The value of preparing an XML sitemap
A big thank you to Adam Cogan for inviting me, and to Adam Stephensen for hosting the day. Also, thanks to Titus and Raj for recording and editing the presentation.
The video is embedded below, and you can also access it here.
Introduction to SEO – Part 1 (Craig Bailey at FireBootCamp)
The slides for the presentation are available here:
FireBootCamp Introduction to SEO by XEN Systems from Craig Bailey
The full transcript is below (I used SpeechPad as the transcription service):
Craig: As you know, we’re doing an intro to SEO. As I was just saying before, it’s pitched at an introductory level. What I’m going to do is just give you some quick tips about SEO. Today is really an introductory session. We do have the option later, if there’s time and schedule and interest allows, we might look at a more advanced SEO topic later. Might do it after hours or something like that. We’ll see.
We can chat about that separately, but today is really just an introduction of the basics. I also want to go through some tools with you. Things like Google Analytics, which is something that I then want you to set up today. I’m not sure how today is structured in terms of what you’ve got to do, but I’m around all day, and helping you set up some of these tools is what I became to do, because they’re quite important for web developers to know.
Just a little bit about me. My name’s Craig Bailey. Hi. I run a company called XEN Systems and we’re a digital agency. We’re small. We look after SEO and online strategy for clients, and I do a fair bit of analytics training as well. But my background is actually as a developer. I was a developer for 20 years. More than 20 years.
I was a database developer and then a web developer. I was a Microsoft MVP in 2007 and 2009 as well. After that, I went more from developing stuff to more promoting stuff. So the online marketing side of sites. So no longer developing sites, but promoting the sites. It’s a little bit about me. Kind of a technical background. That’s kind of my focus today.
Today’s goals. We’re going to go through three things. The first is an introduction to SEO. I’m just going to tell you what that means and what it covers and some things to be aware of. Second thing is I want to go through some tools that you can get implemented as soon as possible.
Then the third thing, depending on time and how we go, it’s just the importance of content. We’ll look at kind of how we develop content for the sites and also pick up on something that Andrew Coates covered weeks ago, which was around this idea of an online profile. So your blogs and things like that and the importance of that, how that all feeds into Google.
At the end, here’s kind of what I, at the end, if I’ve succeeded, here’s what I want you to take away. I kind of want you to understand the basics of SEO. I’d like you to set up a Google account if you don’t already have one. I’d like you to set up Google Analytics. I’d like you to set up Google Webmaster Tools. I’d like you to set up Bing Webmaster Tools. Okay?
These are all really quite easy to do, and I’ll go through each of them. I’d like you to understand the value of good content. I’d also like you to start to be thinking about this concept of keywords and ranking for specific terms, and also understanding your audience, what they’d be searching for. Thinking about your customer.
That’s kind of how we’re going to go. Actually, just before I get into SEO, I’ll just tell you the format. Today I’ve got a whole bunch of slides ready, but I don’t really want to spend a lot of time in the slides. I’m actually going to flip out to the tools and we’ll spend most of the time in the browser. But in terms of the format, I’m really keen to make it interactive.
It’s always hard to pitch presentation. Is it too simple, too basic? So I’ll just rely on your feedback. Okay? If there’s something I just gloss over, just pull me up. If it’s too basic, just give me the “Yeah, yeah” kind of signal. I’m more than happy for you to interrupt. So I’m happy for this to kind of transgress as we go along.
But I’ll give you a few introductory points. What is SEO? SEO stands for search engine optimization. Some of you know that. But what is search engine optimization? Well, it is kind of what it sounds like and it’s optimizing your site so that Google will rank it higher. But then what does optimizing mean? That’s really what we’re going to talk about today. It’s really how the site is presented so that Google can rank it higher.
Now, what is SEO? It’s search engine optimization. It’s also about maximizing the number of keywords that drive traffic to your site. In Google, when you search for something, we want to be kind of ranking for as many of those as possible. We want to increase traffic to the site.
However, it’s also about matching the right kind of traffic. It’s not just about traffic for traffic’s sake. It’s actually about matching what your site delivers to who your audience is, who your project audience is, who your customer is. It’s all well and good to rank for the term “Miley Cyrus,” but if you’re selling a SAAS product, that’s no value at all.
The other part of SEO really is about actively monitoring it and improving it. SEO is not just a set and forget. It’s just not like do these few things and we’re all good. As you know, the Google ranking’s always changing, so there’s this kind of ongoing process of what’s happening, what’s changing, are there issues with my site? Let’s improve it. That incremental ongoing improvement. That’s SEO as well.
Does that make sense so far? Let me just check. Is that all good? If you want me to cover anything else, just let me know. So I’ll push on. What I wanted to do, before we dive into the technical bits and the hands on bits, I actually wanted to take a step back and think about why we build a website in the first place.
Really what I’m getting at here is business goals. I’m going to spend a couple minutes on this because I think it’s really important as web developers that we kind of understand this approach, what the website we’re building is for, and also, more importantly, if you’re going to be .NET developers for say a company like SSW or other consulting companies, when you’re placed as a consultant into an enterprise or something to deliver a solution, it’s really important that you’re thinking about what are the business goals for that client.
Too often you kind of see developers going, even senior developers, they’ll go in and they think, “Well, just give me the requirements.” In their mind they’re thinking, “Right.” I’ve done this. Maybe you’ve done this.
It’s all about, “Well, what’s the cool framework that I can use to deliver this solution?” or “What’s the new technology I can play with? This is going to be fun.” We’ve got to have fun developing, right? But it’s about a balance. Sometimes the balance skews too far to just getting stuff done, but not actually being in tune with what the business goals for the client is. This is really important.
Why build a website? What I’m saying is to drive traffic to it, obviously. Traffic comes from a variety of sources. We’re going to touch on that first one, organic, today, which is the Google stuff. There’s a whole bunch of other sources. Paid, social, referral, direct, email campaigns, all that kind of stuff. We want to drive traffic, but we want it to be the right kind of traffic.
For that website, so for our client, we’re developing a website. They’ve probably got some kind of conversion process in mind, how they want to convert a client. Now, that could be getting them to sign up for a service. Could be buying something if it’s an e-commerce site. It could be just awareness. You’re not for profit. You’re trying to get awareness of an issue. It could just be for advertising. You just want eyeballs on the site.
There’s always different kinds of conversion ideas that a site might have and different customers that you might be appealing to, customer or audience. Ultimately though, most websites have a financial component. They want to make money. Not all cases, but most of the cases.
Even if you’re working, say you’re doing a big government website where you might think, “It’s not really about money,” actually it probably is, because someone’s probably got to justify the budget for that site, even if it’s just the development this year but even the maintenance of it next year. Getting a mindset around what are the business goals for a website is really important as developers.
Now, today we’re just going to touch on kind of the start of that driving traffic. But I really wanted to kind of harp on this point that we really need to be thinking of the whole process. As developers, as we become more experienced developers, understanding the business goals behind a website is really important. Any questions on that or any comments on that?
Is that interesting to people or it perhaps annoying? Don’t annoy me with what the company wants. I just want to build the site. I get that. I’ve done that plenty of times. Okay.
Just to kind of set the scene, if you’re looking at marketing for a website, you know there’s offline marketing or other, TV or that kind of stuff. But then in the online marketing space, this kind of umbrella term we talk about, online marketing, we’ve got the paid stuff, we’ve got direct, we’ve got social, and then we’ve got SEO, the organic stuff. Quite often we think of this as the fray traffic. Although, as we know, there’s a lot of hard work and time and cost that goes into ranking for stuff.
But today we’re really looking at that organic part. When it comes to SEO, there’s kind of on page SEO and off page SEO. These things I’m talking about, we will actually kind of review them, keep coming back to them, but this is kind of broadly setting it up. Today we’re really mostly going to talk about the on page SEO. By that, I mean the stuff that we can actually do on the site. Off page, just to let you know, is stuff like getting a link from people, recommendations from other sites, that kind of stuff.
Let’s think about the Google algorithm. I’m really just going to refer to Google, but this includes Bing and Yahoo! There’s hundreds of search engines, as I’m sure you’re aware. Let’s just talk about the algorithms and how they rank stuff. This is still kind of high level stuff, just setting the scene.
You may be aware that Google uses hundreds of input factors, and these are changing over time. It’s not just, like, three things you’ve got to do. There’s hundreds of them. They’re continually testing stuff.
In the past, Google has actually said they test any number of variations each week or daily. They’re always testing things. So it’s not like they just do one kind of…You kind of hear sometimes about these Google algorithm changes, and we think it’s like, oh, it happens three times a year. No, it’s happening every day, they’re testing things.
So it’s constantly changing. The other thing about it is that the results that you get are personalized. This is becoming more the case. The results you get even on just different devices or just different browsers on the same machine will often be different.
Search for a term on my browser. It’s different to the results you get on another. That’s because it’s being personalized based on your previous history, what you’ve looked at, all that kind of stuff. Also, your location and device. You search for “airport” on your mobile. You’re probably going to get a different result than on your desktop. We know that that’s reasonably complex as it is.
Then the Google algorithm takes into its calculations on page items, I’m going to talk about page titles, and off page items, which is getting links from other sites. High quality content is important. It’s becoming more important. Gone are the days when you could just rank any old rubbish based on some spammy links. The push to quality is definitely on. That’s not to say you won’t find rubbish when you’re searching Google or Bing. It’s not perfect, but it’s definitely getting better quality.
The other thing that I wanted to tell you about, the Google algorithm and Bing as well, is that if you’re a big brand, that helps. The reason I say this is because if you’ve got a small website and you get penalized…Have you ever heard about getting penalized on Google? Is that something you’ve…Okay, I’ll tell you a little bit about penalties.
Google has these guidelines. They think they control the world, and they kind of do in some ways. So they put out guidelines where they tell you what’s right or wrong to do. You can often think that they’re like a law onto themselves. Like, if I break the guidelines, am I doing something wrong? No, you get penalized. It’s not like it’s an illegal crime or anything, but the way some people think about it is you get penalized, it’s like you’ve broken the law. That’s how kind of strong Google is.
But getting back to brands. If you ever get penalized, which what happens is your site just completely falls out of Google. You’re searching for your site one day and then the next day it’s just gone. A bit of a heart attack. But when you get penalized, if you’re a small website, it’s very hard to get back in. If you’re a big brand, you’ll get back in in a couple of weeks. You just kind of cry, apologies and that kind of thing to get back in.
The reason I’m telling you this is because you might think, “Well, a big brand website is ranking for all these terms. I’ll just go and look at what they’re doing and I’ll just copy it.” Right? It makes sense. To a certain extent you would do that. You go, “Oh, well, they’re obviously working. I’ll do some of that.” Just saying it’s not always the case. Sometimes they get away with stuff that you and I can’t get away with.
Anyway. Just to quickly recap that. Google algorithm, quite complex. Yeah?
Man: I was just going to ask, so what kind of things will get you penalized?
Craig: Okay, so that’s a good question. The question was asked, what kind of things will get you penalized? You wake up one morning and you’re out. Now, the answer to that is there’s tons of things you can do. Plus, it’s changing. But I’ll give you just some examples.
Have you ever seen when you get those spammy links? You’ve got a blog and you get these comment notifications. It’s just a spammy comment and it’s got a link in it. Get all that? That’s because about five years ago if you spammed comments, you were building links into the comment part of people’s sites.
The thinking was, if I get all these links from all these comment blogs, all these links, Google will look at these links and say, “Oh, well everyone’s linking to me. This site might be better.” And it actually worked five years ago, maybe 10 years ago.
Hasn’t worked for years. I don’t know why we still get all this comment spam, but that’s an example. Google basically cut down on that. If they notice all your links to your site that previously propped you up in the rankings were spammy backlinks, they’re called, they basically started penalizing sites for that.
You might think, “Well…” This is getting a bit off track. You might think, “Well, hey. Let’s say I want to be mean and kind of get someone else penalized? Well, I’ll just build a whole bunch of spammy backlinks to them as well.” Then you’ve got this idea of negative SEO, I don’t know if you’ve heard this term, where you try and penalize other sites.
Then the algorithm came back with, “Well, we’ve got to be more aware of this so that people can’t just destroy someone else’s.” It all gets very complex. They have their whole bunch of signals where they try and work this out. As much as I hate Google, I kind of love them in other ways. They’ve got a very difficult job trying to work out what is quality. So that’s one example.
Then when that stopped working, I’ll just give you one more example. People started building their own websites that just had rubbish content. They’d basically say, “I’m going to build a whole bunch of low quality websites and just link back to my website.”
So you’ve got this idea of blog networks purely just for creating sites that you could then put links on back to your site. That worked for a while, and then Google comes in and says, “Hang on. We’ve wised up to this and we’re basically going to penalize that behavior as well.” They’re kind of some examples, that kind of stuff.
Back in the day, back in the day…10, 15 years ago, when this was all kind of the wild west of just manipulating Google, one of the things you could do on your site, so not offsite getting links, but on your site you could include, you could hide all this dodgy keywords within your content. Yeah, and it was called keyword stuffing. You’d hide it so only Google saw it, and that worked for, I don’t know, a year or two.
That’s 10 years ago. Google clamped down that. Some big brands actually got called out. I think BMW was famously called out doing that. Of course they apologized and fixed it, and now back in a couple months later. They might have gotten a three-month suspension. I don’t know. That answer your question?
Man: Yeah.
Craig: Yeah, it’s a good question. By the way, all you need to do is kind of jump onto some of these SEO forums and you’ll get the latest tip for how to rank. It’s always something scalable and automated and spammy. Basically, the rank well for a while, you’ve got to be delivering content and working hard to promote it to get good quality links. If you’re just building thousands of spammy backlinks, it might work briefly, but it’s not a long term plan.
So the Google algorithm. Let’s actually look at a Google result. I’ll actually just flip out to the browser now and I’ll try to do most of this from here if we can. I’m just going to search for the term, “FireBootCamp,” because what I thought I’d do is I’d actually use FireBootCamp as an example today for most of the things we look at.
The reason for that is because it’s actually a really new site. It’s quite new. It’s just starting to rank for things. We’re just starting to get good content on it and improve it. It’s probably a good example, if we’re just building our own websites, to be thinking about, as opposed to some aged site that’s ranking well for everything, it’s had thousands of visits, and could be a little demoralizing.
So we’re going to look at FireBootCamp. I want you to notice a few things here. This is a typical Google result. First thing, see this? I don’t know if you can see that color shading. It’s quite faint. What’s that? Is that a Google result?
Man: Ads.
Craig: Yup, they’re ads. Quite right. It’s good that you know that, but in fact most people don’t know that. They see this one at the top and they think, “Oh, that’s the best result. Google is telling me that’s the best result.” These are actually paid. We’re not really going to talk about that today unless you want to. We can talk about AdWords and advertising.
But really, this one here is the first organic result, the first natural result in Google. I want you to notice two things. There’s the what I’ll call the title or the heading. It says FireBootCamp. Then the second…
There’s actually three things, but two of them we’re going to focus on. The second is the URL, which is useful. See how it’s slightly bolded and matches our search term? Then this third one, which is actually the second thing I’m going to harp on today, is the meta description. That’s a result.
I just want to show you. If I actually go to the FireBootCamp’s site, see this thing here? I don’t know if you can. Actually, a better way to show it, I’ll just flip over to Safari. FireBootCamp. At the top of the browser or the tab, you see this, the FireBootCamp. That’s actually called the title, page title. You’re familiar with this and the code behind it? Yeah.
The page title is actually one of the key influences on what Google ranks that page for. Out of the hundreds of things that you could be doing for your site, this is actually the very first thing that I recommend you do, setting the page titles correctly, because they basically directly flow into the search results. Now, I’m going to harp on this, because you know the 80/20, where it’s kind of like do 20% and get 80% of results? Getting page titles right is the single best thing you can do for SEO.
I actually do work for government departments, and I’ll go in and find that page titles are not set. Just by fixing that on a government’s site can have a mass impact on their Google rankings. So don’t underestimate it. Every presentation you’ll see on SEO will mention page titles, every article. It is because it’s really important. That’s a key thing that I’ll just harp on.
Actually, let me flip back and I’ll just show you an example of when people get it wrong. We’re going to look at the source in a minute. Did I not put it here? Here’s an example. I don’t know if you can see this, but here is a government site and that’s their page title. That’s a massive opportunity they’ve missed out on there. They don’t even have their brand or department name. So Google kind of has to guess what the site’s about.
In fact, this little snippet I’ve overlaid here is actually how it appeared in the Google ranking. It’s just default. That’s no use to anyone. The reason I mention this is because you might think, “Oh, page titles. Yeah, everyone talks about that.” Government departments, people, big brands often get this wrong, or they just have “Home” on their homepage.
They don’t even have their brand name. That kind of stuff. Really important that you get this right. Okay. Does that make sense? Is this new to people or is this kind of you knew this before? How many people is this new to? Okay, good. That’s good. Thanks for that. Just want to make sure I’m telling you stuff that’s useful.
Now, the second thing…Actually, let me just…I know you probably know this, but I’ll just flip into the page source for FireBootCamp. This is this section here. Here’s the two things. When we talk about page title, that’s it, the title there. Can you actually see that at the back? Joanna, you might not be able to read that. Sorry. Let me just see if I can zoom in. Why is my [inaudible 00:23:20] not working? There we go. Is that a bit easier to read? There we go.
So these are the things that I’m talking about. The page title and the code behind. Now, how you set that, if you’re just writing pure HTML, you could obviously just set that yourself, but quite often the way you set that is different based on how you’re developing it or whether you’re using content management system and, yeah, the kind of framework that you’re building your site with. Those kind of specifics, we might look at after the presentation depending on how you’re building your sites. But that’s important that you set that for each page.
Now, the second one, which is the meta description. This normally flows…Google normally picks that up to put here. Now, what’s interesting is in FireBootCamp, and this is good. I kind of like to show you this, because this is an example of where we didn’t quite get FireBootCamp optimized early on. It’s only been for a couple months. We’ve only now set this description, because that’s actually what we wanted to say, “Become a world class beginner .NET developer.”
That’s actually what we wanted to say here, but we’re lucky because Google’s actually pulled…That description’s actually quite good, but where have I got that from? What Google has done is when they’ve crawled the page, they’ve actually pulled that here from some text in the content of the page. It’s kind of lucky that they pulled that and not something that was a bit off topic. But we actually have the chance to influence what Google uses in those two things.
Now, let me tell you. I had said the page titles are really important because that guides Google on how to rank the page and for what terms. This meta description, it actually has no influence on rankings. You might think, “Well, if it doesn’t actually influence the ranking, why would we do it?” The reason is you should write that so that it’s like a call to action. Okay? Because when you get actually…Let’s go for a term, “SharePoint Consulting” maybe. Here’s results for this.
See how there’s a whole bunch of things competing for our attention? Which one do I click? What we actually want to do is make this description the most compelling to click through, because if you’re lucky enough to be competing with someone that doesn’t set their meta descriptions properly, they might just have garbage or Google’s pulled their menu structure or something in, then you can often, actually maybe not often, but you can for some of the time, even if you’re not the top ranking result, say you’re two or three, if you have a more compelling description here, you’ll get more click-throughs.
This is actually really important. You’ll get more traffic even though you don’t rank as high. That’s why I’m harping on those two. There’s set from page title and the meta description. Those two things are the most important thing that you can do for your site to help with SEO. If nothing else today, take away that little tip. Okay, let me just flip back to PowerPoint. Any questions on that? Any thoughts? Okay.
Here, so what I’ll say is you might think, “Well, what do I actually put in that page title?” I would say if you’re starting a new site, especially the homepage, put your brand terms in there. Okay? You often see page titles where it says, “Welcome to the best site about blah, blah, blah.” Put your brand terms there, especially if you’re getting it started. Okay?
Now, I’ve skipped over a few of those slides. Now we got up to there. I’m actually going to give you a little bit of an SEO checklist. I’m going to explain a few of these and then I’m going to jump into tools. Okay? Is this all making sense so far? Google crawls the Web and they’re crawling pages and they’re looking for stuff on your sites.
We’ve talked about page titles. We’ve talked about page meta descriptions. The third one you can use is headings on the page to influence what Google ranks it for and what they think the page is about. That’s why even, probably FireBootCamp’s not the best example, but just the use of headings throughout the page can influence. If these are headings, this is a guide to Google. Yeah. “Development manages new .NET developers.” Maybe that’s what the site is about.
Okay, so that’s the first three. Page titles, page meta descriptions, which are done in the code behind or in your content management system, headings, which is the content that you have on the site. Of course, the content that you write about, the more you can match that to your audience, the better. It guides Google.
Now, one of the things about SEO that as web developers we should be doing is this concept of test and measure. Everything we should do has a test and measure aspect when it comes to ranking studies. We try something, we test it, and we measure whether it works. Then the obvious question is, “Well, how do we measure it?” That’s where Google Analytics comes in. I’m going to dive into Google Analytics in a second.
I did want to mention the robots.txt file. Are people familiar with this? Who’s come across the robots? No one’s? Yep. One or two has come across the robots.txt file. I’ll tell you what that is just to get it out of the way. Also, XML sitemaps. Have you heard of XML sitemaps as well? Yeah, okay. So they might be more familiar with that.
I’ll tell you what the robots file is. We’ll just jump to that, and then I’ll come back to Google Analytics. Here’s the robots.txt file. This is a file that sits on your server. Quite often you’ll just generate the file itself or your content management system can look after it sometimes. What it is, it’s actually a directive to the search engines as to what they can and can’t or what they should and shouldn’t crawl on your site.
Now, it’s called the robots.txt file because it’s applying to the Google crawlers and robots or the search bots that crawl through your site. It’s not really for people to look at. It’s for the search engine crawlers to look at. Now, this has a lot of power over your site. You can use it to block the entire site or just to block parts of it.
Let me tell you the use cases you might have. You’re building a staging site for a client and you’ve got it publicly available, which you might have to do because there’s external testers or for a variety of reasons. You can actually use your robots.txt file to completely block it from Google, because there’s nothing worse than in the Google results turning up some staging site, especially if you’re about to launch something that’s time sensitive or it’s got some new features.
You don’t want that turning up in Google results. So you can block it. On the left there I’ve given you the directive that you can use to block everything. It’s basically saying, “Disallow the entire site.” That’s great for staging. The problem is often when staging sites go live, and maybe they’re replacing an existing site, you can see what’s going to happen, right? People copy over their robots file and it blocks the entire production site. This happens more often than you’d think. It happens all the time.
Just over Christmas I had a client. Their entire site, just I’d been testing something, basically dropped out of Google for a couple of weeks because someone had incorrectly set the robots.txt file. This is something that you should all be aware of. It should actually be part of one of your checklist items when a staging goes to production. I don’t know. You can probably automate this somehow in your CI retains or whatever, but it should just basically check that robots isn’t blocking everything.
Now, the second on here one the right, the go live. Your user agent, you can just get it to block…Yes?
Man: In Windows Azure we have option to put out staging type site to production site. Then what happened to this robots.txt?
Craig: Okay, so the question is, in Azure websites… In your website you have that process. Now, I’m not sure what happens, what Microsoft’s process is, so you might actually have to have a manual check, but I would imagine if it’s already set under your staging folder, it’s going to get copied over.
I don’t know what scripts you use. Maybe it doesn’t copy text scripts. I’m not sure. But let’s assume it gets copied over. You’re actually going to have to come up with some checkpoint as part of that staging copy that checks the robots.txt file or maybe pulls over an existing one or maybe just doesn’t copy that over, because you know that the one on your production side. The other comment on that, Adam, how that would…
Adam: It’s interesting. I’ve actually made a note of it. The way Azure works with staging, it actually has two versions of the website. What it actually does is it does a virtual [inaudible 00:33:03].
Craig: Ah, right.
Adam: So how to handle this is actually going to be an interesting question.
Craig: Okay.
Adam: My thing would be we probably will have a look at browsing. Robots.txt is just the website address, /robots.txt.
Craig: Yes.
Adam: So we could dynamically generate robots.txt as long as it looks to Google, /robots.txt returns a text file.
Craig: Yes.
Adam: That would be okay. So if we had a route that dynamically generated it…
Craig: Yes.
Adam: …depending on whether or not it was our staging instance.
Craig: Yep. That would be a good solution, yes.
Adam: I think the answer for Azure selects is going to be not to actually have a robots.txt. Good to have a custom route that returns robots.txt based on whether or not it’s staging or not staging. We’ll actually have to have it in code. Every website will be able to return it depending on whether it’s a staging instance or not.
Craig: Yeah, so that’s a good solution. I’ll just repeat that for the mic. What Adam is saying is basically that file will be generated dynamically. Actually, this is the way WordPress works most of the time. WordPress, there’s not actually a physical .txt file on the server. It is generated dynamically. If you have plugins, they often hook into that.
In your SEO plugin on WordPress, I don’t know if many of you use WordPress, but if it says, or even actually the default WordPress, it will say, “Hide this from search engines.” It’s actually putting that robots.txt file, virtual file, in place. I think you’re quite right. On the Azure side you should probably generate that. So yeah, that’s a good solution.
Okay, now the advantage of…You might think, “Well, why do we even have it on the production side at all?” The reason is quite often you do want to hide folders from Google. You just don’t want them searching stuff. So you might have an FTP directory that you don’t want Google somehow finding and crawling or things like that. That’s where you can actually put these disallow directives in and block folders or specific files and all of that kind of stuff in robots.txt.
That’s something definitely to check. It’s becoming more important because in the past if you just didn’t have a robots.txt file, it was generally okay. It’s kind of like Google assumed if it wasn’t there, “Well, we’ll just crawl everything.” Sometimes you find though that, especially if you’re refreshing a preexisting site, you find if they did have a robots.txt file in the past, Google is expecting it.
Then when you put the new site there, you kind of say, “Oh, I’ll get rid of robots all together. We don’t need it. We don’t need to block anything,” Google is actually expecting to see it. What they tend to do now, if they’re expecting to see a robots file and it isn’t there, they tend to assume, “Well, we best be careful and block the entire site.” That’s quite often how you see it happen. So just something to be aware of. Very important.
Okay, robots. Sorry. Back to my checklist. That was robots, and the other one was the XML sitemap. Nowhere near as important as the robots file, but is a guide. It’s just an XML markup of the pages that you want to guide Google to look at.
It’s just a file that you put in that same directory. When we look at Google Webmaster Tools, we’ll look at how we actually tell Google to look at that sitemap. It’s just a guide. If an XML sitemap is a guide to Google, robots is the gate. Make sure the gate’s open and then guide it all you want.
But what I want to do now is I actually want to flip over to Google Analytics and kind of walk you through that for a little while because I think it’s really important and something that you’ll find really interesting about your site. Here’s the tools that we’re going to look at today, time permitting. I probably should check time. There we go.
Google Analytics. Google Webmaster Tools. We might have a look at Google AdWords if we have time and there’s interest. We’ll look at some keyword tools, and there’s a whole bunch of other tools now. There’s millions of tools. If you ever want to lose a weekend, just kind of look up some SEO tools and go crazy. It’s kind of one of those things. Or if you like to procrastinate and don’t want to actually do work, go and find a new SEO tool to play with. All right? I do it all the time.
Man: The next shiny object.
Craig: Yeah, right. It’s like that. “What’s the new framework? What’s the new SEO tool?” Anyway, so what is Google Analytics? You probably know this. It tracks visitor sources, behaviours, conversions, and things now. If you don’t already have a Google account, you’ll need to set one up. That’s really easy. Or just set up a Gmail account and then create your account. We’ll actually pop in to Analytics now. I’ll just walk you through what it looks like.
I’m going to show you the…Actually, let me just ask. Who’s actually played with Google Analytics? One, two. Okay, so most of you haven’t. So most of this will be new to you, I’m assuming. Let me just collapse all these. Here’s the interface. You come in. Later we’ll look at how to implement this on your site. Basically, it’s a bit of JavaScript you just put on your site and it starts tracking.
But what it tracks after a while is you get all these stats. Now, this is FireBootCamp’s site, which, as I said, is pretty new. So these numbers are not huge yet, but they’re at least indicative. We can get some interesting insights into what people are doing on the site.
Now, what you do is you come in and there’s a whole bunch of menus I won’t get into. But down the left hand side are the main sections. There’s three main sections that you’re thinking about in Analytics. You’re thinking about who’s coming to my site, how did they get here, and what are they doing? They’re kinds of the questions that you’re asking.
Who are they? What country? What device? How did they get here? Was it from Google? Was it from AdWords? Was it a referral site? And three, what are they doing, which is stuff like what pages are they visiting, how long are they staying on pages, that kind of thing. That’s when we have this audience section. That’s who are they. Acquisition is how did they get here, and behavior is what are they doing.
I just wanted to jump into a few of these reports to show you the kind of stuff you can get out about your site. Now, I told you at the start that my background was a database developer, right? So I love data and analyzing data. This kind of gets me excited, but I realize that might not be the case for everyone. If I see you kind of nodding off, I’ll just realize this is not that exciting to most people. But to me it’s like…
Also, the reason why this is important is if you’re optimising your site, working out what people are doing and liking is kind of important. If you’re working for a big client, it’s kind of like, “Well, what are all the people on their site doing? Maybe that’s going to influence the IA, the information architecture that we set up for the new site,” or whatever. Analytics is like a good source that fades into a whole bunch of stuff.
Now, before I get into each of these three sections, I’ll look at the real time section. This is really cool. On the FireBootCamp site, let’s say I go to a page. So this. That’s firing, is it? A little bit slow. Oh, I think I’ve lost my…Oh, no. It’s coming back. Yeah. I’ve gone to one of the pages. Actually, let me just bring this back to actual size. I zoomed in before.
So this is the site. If we go over and look over at real time, this is the section. This is quite nice. It tells you real time what people are doing on your site. There’s one visitor. That’s me, obviously. It would be nice if there were lots more, but you know. That’s the way the site is at the moment.
You might think, “Well, what’s a high traffic site?” Say you’re on a big government site that’s getting millions of visitors per year. You might think, “Oh, how many active visitors have they got?” Often it’s just between 50 and 100. You might think, that’s actually quite a lot. If you’re ever looking at Analytics and you’re getting 50 to 100 active visitors on your site, you’re actually, no, you’re doing actually okay. Yeah?
Man: How does Google Analytics compare to things like Red Sheriff that are similar?
Craig: Okay, so the question is, how does Google Analytics compare to Red Sheriff? I don’t know what Red Sheriff is.
Man: It was, around a few years back.
Craig: So it’s an analytics package?
Man: I think it was, yeah.
Craig: Okay. Well, let’s Google it.
Man: Around 2005.
Craig: Is it? Website traffic statistics. Well, I don’t know the answer to that particular question, but the question is good because it’s like, “Is Google Analytics the only analytics package?” No. There’s hundreds of them. Google Analytics is free, which makes it very popular. Most of the other analytics packages are paid and there’s quite a range.
You can get something like Clicky or GetClicky, which is actually one I use a fair bit for on sites. I think it’s, like, $9 a month. There’s other that focus on particular features, like KISSmetrics is another popular one. I use that on some sites where we actually want to get more personally identifiable information about the visitors.
It actually raises a good point, which is within analytics you’ll never see…Oh, five. Is that all you? Thank you. Yeah, they’re all from Sydney. Thank you for that. We have got such a popular site. It’s just gone off, trending… 
With analytics there’s nothing ever personally identifiable. That’s actually in Google Analytics’s standard terms and conditions. So much so that when you go and do AdWords stuff, they actually make it when you do remarketing…We won’t go into that today, but remarketing is basically where ads follow you around. Have you ever done that? You go to a site and then wherever you go it’s like, “I’m just seeing their banner ads everywhere.”
It can be good or it can be annoying, depending on your point of view and whether you’re the advertiser. But that kind of stuff can get close to personally identifiable, because what you can do is you can actually kind of drive people into silos on your site and work out who they are. You might send them a specific email just to a page and then you’ve kind of been able to tag them.
Google Analytics doesn’t let you do that. They say nothing personally identifiable. In fact, if you’re doing advertising campaigns that break that, they’ll often close your account. The reason I’m mentioning all of that is because other analytics packages do allow you to do that. Red Sheriff might have been one of those tied in directly.
While Analytics is free and really useful and by far the most popular, other analytics packages will give you more stuff, especially when you want to get into a lot of that data mining and really specific behavioral analysis. Any other questions straight away? That’s a good question. That was a good one.
Man: Can you just clarify what the bars on the second graph show?
Craig: Okay. Yeah. The question was, what do these bars actually mean as they scroll along? They’re basically indicating, you probably can’t see it, but the y-axis here in per second is, that’s a one where those are. That’s saying there’s one visitor or probably one page view probably, per second there. So if we all jump on now, let me actually flick around. You should see these boost up. I was hoping we might see that jump up to two for some of it.
Man: I think [inaudible 00:45:48].
Craig: Yeah, might be.
Man: Then why is the disc…So the bars don’t seem to be coming through every second though. It’s simply getting [inaudible 00:45:57] as opposed to…
Man: That’s because there’s several [inaudible 00:45:59].
Craig: There is a little bit of a delay. Let me just jump to something that you’re probably not on. I’ll jump to the blog page. You’ll see a little delay before it pops in to these active pages. I’ll be one of these visitors already, and then you’ll see it switch over to blog probably. There it goes. That one.
So there’s actually a delay of a couple of seconds. It’s not super real time. There is a delay, which is probably fine. But yeah, that’s prob-…Oh, here we go. There we’ve got a four. You might not be able to see that on the screen, but that’s actually four. We had four people there at that same point on the site within that few seconds where it was capturing. That’s what those mean.
Then this is per second. This one here with the tiny bars. Then this is per minute. So you get a 30 minute kind of window of activity. In this section, two minutes ago, we had about 16 page views on this site. Right now we’ve got seven. This is just…We’re nailing it.
Locations. Traffic sources. It will also tell you where they came from. Someone came organically. Did anyone type in Google? I wonder if that’s me. No, no. I came direct. Someone has actually come via the search engine, so that’s useful. Yeah, so that’s cool. It’s telling you at any one time. Now, why that’s good is sometimes if you’re a big brand and you’re running a competition and let’s say you just announced some sponsored tweet on Twitter or something and you just see suddenly your social sources ramp up. That can be quite cool.
Man: So if I was coming from a link from Twitter, I would see Twitter as a source.
Craig: Yeah, it would either appear as social media or if you’re ever in Analytics and see t.co as the referral source, that’s Twitter. Yeah. Question?
Man: Can you get Google Analytics off any website? You are opening this FireBootCamp now?
Craig: Sorry. I don’t quite understand the question.
Man: We can see the analytics off any website? You are opening with this FireBootCamp…
Craig: Oh, right.
Man: …a Reddit one or other site?
Craig: So the question is, can you view the analytics for someone else’s site? The answer to that is possibly, if they give you permission. But there’s a process where you create an Analytics account, you put the script on your site, and then you have permissions to view that Analytics account. I have permission to view the FireBootCamp site and a bunch of my client sites, but there’s no way I can tap in to see what’s trending on BuzzFeed or something, unless they gave me access.
I won’t go into it today, but there’s a whole bunch of stuff under this admin section. We have user management and things like that where you can give all kinds of permissions. You can give some people just access to reports and specific areas of the site. But I’ll just go back to reporting. Does that answer your question? Yeah?
Man: Analytics privacy issues that you need to be aware of?
Craig: Good question. The question is, does Analytics have a privacy legislation? It probably depends, country specific, but I know in Australia most people will put a little disclaimer. In our privacy section I’m pretty sure we’ve got it here that says we do collect information about you. What’s important, and that’s actually another really good question because…Yeah, here we are. Personal information collected from websites where we talk about what is collected and how we collect it.
Now, this is actually going into a bit of detail, but because we use remarketing for the FireBootCamp site, remember I said we fully ran the web with banner ads. They work really well. You might love them or hate them. But we actually have to give people, this is in Google’s terms and conditions, we actually have to give them an opt-out option. We do that here where you can actually opt out. Actually, it’s this second bit here. You can opt out of that remarketing cookie.
Whether legally in Australia we have to do that, I’m not exactly sure. I think so. There’s some…You should mention it. But this second bit about opting out, that’s a Google requirement.
Man: [Inaudible 00:50:36] legal.
Craig: Oh, right.
Man: [Inaudible 00:50:40] privacy checklist we have to go through.
Craig: Yeah, right.
Man: [Inaudible 00:50:44].
Craig: Right. It’s a good question. The comment there, just for people, was that at Telstra, whenever they publish a new piece of content or a new section of the site, you’re always getting the call from legal saying, “Have you done the privacy checklist?” It’s a good question. I don’t really know the specifics of what are required, but I think it’s a really good thing that you’d have to work out for whichever client you’re working for.
Okay, so that’s Google Analytics. Any other questions on that before we…? I’ll push on. Okay, so that’s real time. I’ll just dig into some of the other sections, the audience. Now, what’s cool about this, I’ll just pick on a few. We can actually have a look. Now, once we’ve gone out of real time, we’re now into a date range of people visiting the site.
That’s actually, you can see the date range I’ve got up here on the top right. We’re just looking at the last month. You can change that to the last years or whatever, last week if you want. It gives you some roll ups on stats. We can see that Australia gives us the most traffic. United States, India, Brazil. That’s probably Tiego just maintaining the site, I’d say, one of the SSW developers.
Yeah, so that can be quite interesting. That’s always interesting if you’re targeting an Australian audience and suddenly you’re getting a whole lot of traffic from the Philippines or something, which looks a little bit like that’s probably not the traffic you wanted or something like that. So that can be useful.
Man: [Inaudible 00:52:22], Europe.
Craig: As in Europe. Yeah. Russia. So a lot of linked networks in Russia that would build the spammy links. Yeah, and you getting a lot of Russian traffic. It’s probably not a good sign. Don’t know. There’s a bunch of things like technology.
The only other one I’ll look at here in the audience section is this overview of the mobile, because mobile’s becoming, as you would know, far more important. We don’t get a lot of mobile traffic here yet. That’s about 10%. No, maybe 12, 13%. But really on a lot of sites you’re seeing that jumping up around 25, 40% even. Mobile is increasing, especially on text sites.
You would imagine…If you get a lot of traffic from sites social, from Twitter or something, a lot of that’s going to be mobile traffic, because people are tweeting stuff. They get a link. They just flick through. That’s when you’ll see your mobile stuff go up. But let me ask you. We’re going to look at the devices. What do you think the most popular device for viewing the site is going to be?
Man: PC.
Craig: Sorry, what was that?
Man: Just a PC.
Craig: PC. Sorry. In devices we’re going to look at mobile devices. Sorry. You’d be quite right. PC. Into the mobile section, I’m just going to drill into. Android.
Man: Android.
Craig: How many for Android? How many for Apple? So most are Android. Okay. Here’s what we got. iPhone.
Man: Yes. All hipster.
Craig: All hipsters, yeah.
Man: They’re the most dominant one.
Craig: That’s right.
Man: I think they’re blaming on Twitter and…
Man: Maybe it’s [inaudible 00:54:10].
Craig: Sorry. Say that again. What is that? Yeah, probably is. Yeah. The other thing is these numbers are so low that they’re pretty unreliable anyway. You really wanted to be getting thousands of visits to draw some conclusions, but yeah, it’s always interesting. iPhone and iPad. On most client sites that I look at, they’re the big device usage.
Man: Even though Android’s been more of [inaudible 00:54:40], people are browsing more on Apple products.
Craig: Seems to be the case. And I’m talking mostly about Australia actually when I made that last comment. Yeah, Australia. Yeah, I think so. I have seen stats where they look at that and they say the Internet consumption by mobile device, there’s more people with Android, but there’s more pages viewed with Apple. Yeah. That will probably change.
Man: [Inaudible 00:55:10].
Craig: Right, yeah. I’ll just repeat the comment. The comment is, yeah, that’s an important consideration for design and testing, what tools you test your sites on. Take a look at analytics to see whether there’s particular devices that get a lot of traffic and make sure you’re testing for those. Yeah. No, that’s a good point.
Man: I thought Apple was most lately the device in Australia anyway.
Craig: Is it? Yeah. Maybe Apple’s still ahead.
Man: [Inaudible 00:55:39] where it actually starts to change.
Craig: Okay. You might be right. The comment there is maybe Australia was still more iOS centric. You might be right. I don’t actually know the stats. Comment.
Man: Is there any information [inaudible 00:55:52] people who come through from an app that has [inaudible 00:55:56]?
Craig: Good question. How they came. So the question is, can you tell the app that they came from? I don’t know the full answer to that question, but I can tell you that you can tell whether they’ve come from apps, especially when you’re looking at the advertising traffic.
But there is an issue with mobile as well that quite often when you look at the acquisitions, so it’s probably a good lead in to acquisition source. I’ll just show you the channels. One of the problems that we’ve had with trying to gauge mobile visitors, I’ll explain what these are…Actually, no. Let me explain that first and I’ll come back to your question.
In terms of channel sources, you know we talked about organic and we talked about paid. There’s this thing called direct traffic. Direct traffic, it basically means we don’t really know where they came from. They might have just typed it directly into the address bar and they came through as direct traffic. But what we often found in some of the early devices, and iOS 6 was a key one here with Safari, even if you typed into Google, you’re in Google in their browser, and you typed and came through, that actually came through as direct traffic in analytics. So we were kind of blocked from even seeing the source they came from.
But then your further question is, well, what app? Was it the Twitter app