Explaining The 3 Types of Crawlers Google Uses To Understand The Web

, ,

To understand and rank websites in search results, Google is constantly using tools called crawlers to find and analyze new or recently updated web pages. What may surprise you is that the search engine actually uses three different types of crawlers depending on the situation with web pages. In fact, some of these crawlers may ignore the rules used to control how these crawlers interact with your site.

In the past week, those in the SEO world were surprised by the reveal that the search engine had begun using a new crawler called the GoogleOther crawler to relieve the strain on its main crawlers. Amidst this, I noticed some asking “Google has three different crawlers? I thought it was just Googlebot (the most well-known crawler which has been used by the search engine for over a decade).”  

In reality, the company uses quite a few more than just one crawler and it would take a while to go into exactly what each one does as you can see from the list of them (from Search Engine Roundtable) below: 

However, Google recently updated a help document called “Verifying Googlebot and other Google crawlers” that breaks all these crawlers into three specific groups. 

The Three Types of Google Web Crawlers

Googlebot: The first type of crawler is easily the most well-known and recognized. Googlebots are the tools used to index pages for the company’s main search results. This always observes the rules set out in robots.txt files.

Special-case Crawlers: In some cases, Google will create crawlers for very specific functions, such as AdsBot which assesses web page quality for those running ads on the platform. Depending on the situation, this may include ignoring the rules dictated in a robots.txt file. 

User-triggered Fetchers: When a user does something that requires for the search engine to then verify information (when the Google Site Verifier is triggered by the site owner, for example), Google will use special robots dedicated to these tasks. Because this is initiated by the user to complete a specific process, these crawlers ignore robots.txt rules entirely. 

Why This Matters

Understanding how Google analyzes and processes the web can allow you to optimize your site for the best performance better. Additionally, it is important to identify the crawlers used by Google and ensure they are blocked in analytics tools or they can appear as false visits or impressions.

For more, read the full help article here.

/0 Comments

Twitter Makes Its Source Code Public, Here’s What You Need To Know

, ,

Following leaks, Twitter has made its content recommendation algorithm completely available to the public – laying bare how the social network works and what sort of posts are most likely to succeed. 

Along with a lot of interesting details involving which types of content are best received, how your interactions with others affect you, and how poor grammar may hurt you, the code also includes a number of concerning details that have made human rights groups concerned. 

Let’s talk about all the most notable parts below:

Likes Count Most

Likes seem like the easiest type of interaction you can get from other users but don’t underestimate them. The code shows that likes are easily the most important type of engagement compared to retweets or replies. 

The system assigns points to each type of interaction, with each point giving a boost to a post’s visibility. In the current system, a single like gives a post 30 points. Retweets are not far behind, giving 20 points. Shockingly, replies are practically meaningless in comparison, giving just a single point for each reply. 

This means that all the conversation in the world doesn’t matter if users aren’t also liking your posts. 

Pics and Videos Are Important

Less surprisingly, the source code confirms that posts containing visual media are largely preferred over plain text. 

Linking Out Is Frowned Upon

This is another one that has been suspected for a while but has been confirmed by the source code. 

For the most part, Twitter does not want you to link users off the platform. It makes a simple type of sense. Twitter’s goal is to keep people on the app as long as possible, and each link represents a chance for users to leave the app. 

To combat this, the site largely downplays posts containing links unless they are coming from accounts that already have a lot of interaction on their posts. 

Twitter Blue Helps

Elon Musk has not been shy about his plans to make Twitter more of a pay-for-play platform through his pet project, Twitter Blue. Since its reveal, one of the touted benefits of the premium subscription is increased visibility, which is backed by the source code. 

This is not a guarantee you’ll suddenly get a ton of exposure if you sign up for Twitter Blue, though. Accounts are just given points toward their overall algorithm ranking if they are subscribed. 

Poor Spelling Costs You

For a site with the most limited options for editing posts after they go live (only available to Twitter Blue subscribers for 30 minutes after a tweet is posted), Twitter is surprisingly uptight about spelling and grammar. The source code indicates that posts with poor spelling and grammar may be demoted as a form of spam prevention.

The Controversial Stuff

Lastly, we come to the most eyebrow-raising details contained within the source code because of how Twitter appears to be handling international conflicts and vulnerable groups. 

Based on the available code, Twitter seems to be limiting the visibility of posts talking about the ongoing war in Ukraine as hate-based content. This is particularly problematic as many human aid groups have relied on social networks like Twitter to drive donations, awareness, and support.

Another hot-button topic that seems to be directly targeted by Twitter’s code are transgender individuals. Users found that several terms relating to transgender people are suppressed on the platform, particularly when sharing links to other sites containing those terms. Meanwhile, activists say that the platform is not limiting pages containing hateful terms.

Musk says part of the decision to make this source code open to the public is the hope to identify problems that can be quickly fixed by the team to improve the recommendation algorithm. As such, the code should be seen as a work in progress. Still, it is worth taking time to familiarize yourself with everything in the recent code if you drive sales for your business through Twitter.

/0 Comments

Google Announces 5 New Ways To Learn More About Your Search Results

, ,

Just ahead of International Fact-Checking Day on April 2, Google announced a wave of new features intended to make it easier for users to learn more about where their information is coming from.

As the company revealed in a recent announcement, Google is introducing five new features to verify information online:

  • Expanding the “About this result” feature worldwide
  • Introducing an “About this author” section
  • Making it easier to learn about websites using the “About this page” feature
  • Providing context for top stories with “Perspectives”
  • Helping spot information gaps

Expanding the “About this result” feature worldwide

Launched in 2021, the “About this result” feature gives searchers access to additional information about the sources and websites that appear in search results. 

Though English-speaking countries have been able to find this information by clicking the three vertical dots next to most search results for a while, users in other countries or speaking other languages are just now getting access to the feature. 

Introducing an “About this author” section

Google is adding a new section to the “About this result” feature which gives information specifically about the author of the content you see. 

At the time, it is unclear exactly where Google will be gathering this information, but it is worth keeping an eye on as the feature rolls out – especially if your site publishes blog content.

Making it easier to learn about websites using the “About this page” feature

Google is adding a new way to access the “About this page” feature, which details information about a webpage similar to the “About this result” feature.

Now, you can learn more about a page by typing the URL of a site into Google’s search. The following search results will include information from the “About this page” feature at the top of the page. 

Here, you’ll see information about how the site describes itself and what others across the web have said about the site. 

Providing context for top stories with “Perspectives”

The Perspectives carousel aims to provide additional context around Top Stories by sharing helpful insights from journalists and other experts.

The feature has been in testing since 2022, but Google says it will be widely available in the coming days. 

Helping spot information gaps

When Google is unable to confidently provide information about a topic – either because there are few good sources available or because the information is changing quickly around that topic – the search engine will display a content warning with the search results it provides. 

To learn more about these new features, read the complete announcement from Google here.

/0 Comments

LinkedIn Launches 4 New Features For Businesses

, ,

LinkedIn revealed a slew of new features designed exclusively for businesses this week, including content creation, new ways to engage with other companies, and easier ways to have live discussions with others.

Let’s look at all the new features in a bit more detail:

Improved Post Scheduling

Though LinkedIn has offered the ability for regular users to schedule posts ahead of time since late 2022, the feature has not been available to business pages until now. 

Now, business pages can create and schedule posts up to three months ahead of time. Not only does this make it ensure you’ll be delivering steady content to your users, it gives you the chance to prepare a content calendar ahead of time to be more considerate of what type of content you are posting and when. 

While the feature is currently only available to desktop users, LinkedIn says it will be coming to mobile apps soon.

First-Party Audio Events

LinkedIn is rolling out a new, built-in way to hold live audio events on the platform. This means you’ll be able to host audio-based discussions with your followers and other experts in the field without needing to use third-party external apps. 

While hosts will have control over the discussion and who gets to speak, users will be able to engage with the conversation using emojis or requesting to speak in the chat.

Automated Job Postings

LinkedIn has created a way for small businesses with under 1,000 employees to automatically create and share job postings.

Once activated, the feature generates and shares an open role job posting as a pre-scheduled post which can be edited after it is posted.

However, the announcement says that the feature will not be available for what are vaguely described as “basic jobs.”

Follow Other Pages From Your Page

LinkedIn Pages can now start following other Pages, allowing you to have B2B conversations without necessarily going through a personal LinkedIn account. Additionally, this makes it easier to find conversations relevant to your field happening across the platform.

All of these new features aim to expand the functionality of business Pages and make it easier to engage with your audience. By using these, you’ll be able to find new connections, foster better relationships with leads, and be able to become an authority in your field faster.

/0 Comments

TikTok Adds 500,000 Licensed Songs Brands Can Use In Ads and Content

, ,

TikTok is making it easier for brands of any size to use licensed music on the platform with its new Sounds for Businesses service. 

With this, small businesses will be much more able to afford popular songs in their ads, videos, and other organic content through the TikTok Audio Library. 

Why Licensed Music Is Important For Marketing

For brands trying to rise above the noise on social media, using licensed songs is a crucial part of getting users’ attention.

Even more, TikTok says its research shows 68% of users find remembering a brand’s message easier when it is paired with a popular song they like. Of those, 62% said they would take time to learn more about a brand if they enjoy the song. This is because users feel like a brand may have similar tastes to theirs if they choose music they like as well. 

Unfortunately, licensing music is not easy – especially for smaller brands. It is not always clear who to contact to license a piece of music and costs can quickly stack up. 

Trying to slip a popular song into your marketing materials without paying isn’t likely to work either. TikTok (and all other social networks) use automated tools to identify music that may be infringing on a copyright claim. When this happens, the app removes your audio entirely – potentially ruining your message in the process. 

With this new library of licensed music, brands can now avoid this entire complicated and expensive process by choosing from over 500,000 licensed tunes and sounds. 

How To Use TikTok Sounds for Businesses 

TikTok’s Sounds for Businesses library is available to brands creating content and ads on both desktop and mobile.

On desktop, the Audio Library is within the Creative Center. Here, you can filter audio by theme, genre, mood, and length to find the perfect match. 

For mobile users, licensed songs can be added when creating a new post by tapping “Add Sounds” and filtering the results to licensed music. 

Sounds for Businesses is rolling out now. For more, read TikTok’s announcement here.

/0 Comments

Google’s Danny Sullivan Addresses Using AI-Generated Content To Rank

,

Google continues to be relatively tight-lipped about its stance on AI-generated content, but a new statement from Google’s Danny Sullivan suggests the search engine may not be a fan.

Artificial Intelligence has become a hot-button issue over the past year, as AI tools have become more complex and widely available. In particular, the use of AI to generate everything from highly-detailed paintings to articles posted online has raised questions about the viability of AI content.

In the world of SEO, the biggest question about AI-generated content has been how Google would react to content written by AI systems.

Now, we have a bit of insight into how the search engine’s stance on AI-created content – as well as any content created solely for the purpose of ranking in search results.

In a Twitter thread, Google Search Liaison, Danny Sullivan, addressed AI-generated content, saying:

“Content created primarily for search engines, however it is done, is against our guidance. If content is helpful & created for people first, that’s not an issue.”

“Our spam policies also address spammy automatically-generated content, where we will take action if content is “generated through automated processes without regard for quality or user experience.”

Lastly, Sullivan says:

“For anyone who uses *any method* to generate a lot of content primarily for search rankings, our core systems look at many signals to reward content clea/rly demonstrating E-E-A-T (experience, expertise, authoritativeness, and trustworthiness).”

In other words, while it is possible to use AI to create your content and get Google’s stamp of approval, you are walking a very thin line. In most cases, having content produced by experts with experience providing useful information to those who want it will continue to be the best option for content marketing – no matter how smart the AI tool is.

/0 Comments

Google Says It Can Take Quite a While to Recover From Algorithmic Search Penalties

, ,

If your site gets hit with an algorithmic penalty from Google, you’ll likely be eager to fix the issue and improve your rankings again. However, Google’s top experts say it can take quite some time to recover if they believe your site is spammy.

In a recent Google SEO Office Hours session, representatives were asked how long it can take to recover from an algorithm penalty related to content quality problems. 

While many details about the question remain unclear – such as how significant the penalty is – the search engine’s spokespeople encouraged site owners to be proactive. Otherwise, it may be months before they regain ground in the search results.

Specifically, the question posed in the video is:

“If a website gets algorithmically penalized for thin content, how much of the website’s content do you have to update before the penalty is lifted?”

There are a few ways the question could be read, so in this case, the experts kept it simple and straight to the point:

“Well, it’s generally a good idea to clean up low-quality content or spammy content that you may have created in the past.

For algorithmic actions, it can take us months to reevaluate your site again to determine that it’s no longer spammy.”

In other words, it is always better to share high-quality original content than to risk being labeled as spam. Once that happens, you’ll likely be in the doghouse for at least a few months.

To hear the answer, check out the video below beginning at 24:24.

/0 Comments

LinkedIn Previews 2023’s Upcoming Features and Updates

, ,

To kick off 2023, Linkedin is giving a preview of the upcoming features it is working on. Along with updates to the platform’s job search systems, the company revealed it is prioritizing making content more accessible, new ways to find B2B products and services, better analytics, and more. 

Let’s explore all 7 of the new features LinkedIn revealed:

1. Improving Accessibility for Video Content

To help content creators make their videos more accessible, LinkedIn will start automatically generating captions for all videos on the platform. Currently only available for English users, creators can edit these captions or upload manually generated captions when uploading videos. 

Additionally, LinkedIn is adding a high contract mode on the LinkedIn app to make videos and other content easier to see for those with difficulty seeing. 

2. Standardized Accessibility Job Titles

LinkedIn is making it easier for those working in accessibility to find job opportunities and connections by providing a set of standardized job titles for use across the platform. 

3. Alt Text In Campaign Manager

Another feature intended to make visual content more accessible, LinkedIn’s Campaign Manager has added the ability for advertisers to add alt-text descriptions to images in ads. 

4. Personalized Job Collections 

According to LinkedIn’s research, more people are casually looking for job opportunities than ever – particularly with the goal of finding a position that better aligns with their values and interests. 

With this in mind, LinkedIn is creating personalized job collections to help users find opportunities without conducting specific searches. 

5. Improvements to B2B Product Search

To make it easier to search and filter through the nearly 90,000 B2B available across LinkedIn, the platform is adding new category filters and ways to share information with prospects 

Along with updates to the B2B product search engine, the company is improving Product Pages to better showcase the benefits and unique features of B2B products.

6. Post Scheduling

After years of pleas from users, LinkedIn is finally letting creators schedule posts ahead of time. 

Users will begin seeing a “schedule” button directly next to the “Post” button when preparing a post. If selected, the icon will let you select exactly when you want your content to go live. 

7. Improvements to Analytics

Lastly, LinkedIn is updating its analytics dashboard for creators to include more relevant data like audience insights and top-performing content. 

The company says highlighting these specific types of data will help brands understand their overall growth and performance at a glance.

/0 Comments

Twitter Relaunches Twitter Blue and Other Profile Verification Labels

,

After a false start, Twitter is relaunching its verification system including the much-discussed Twitter Blue program. 

To distinguish those who are paying for verification from celebrities or well-known businesses, the new take on the system is also introducing unique labels for businesses and official accounts. 

During the initial rollout of the Twitter Blue system, there was a wave of confusion as parody accounts, trolls, and other bad actors were spending the $8 fee for Twitter Blue to impersonate public figures and companies. 

Following this confusion and complaints from several companies facing PR fiascos, Twitter CEO Elon Musk temporarily retracted the program. In theory, the new badges for official and business accounts, along with restrictions on who can apply for Twitter Blue, should help prevent bad actors from abusing the system. 

Below, we will get into what each badge stands for and who is eligible for each.

Twitter Verified (Twitter Blue) – Blue Checkmark

The classic blue checkmark will be available to users subscribing to the Twitter Blue verified program.

The program costs $8 per month if you subscribe through a web browser or $11 for those subscribing through the iOS app. Musk says the increased cost for Apple users is reflective of Apple’s commission on in-app purchases.

Along with the blue checkmark, Twitter Blue accounts will soon start receiving a number of benefits including:

  • Priority placement in replies, mentions, and search results
  • 50% fewer advertisements
  • The ability to edit tweets
  • The ability to publish longer videos

To be eligible, accounts must meet a number of criteria including:

  • Be Complete: Your account must include a display name and profile photo
  • Recent Activity: There must be activity on the account in the last 30 days
  • Established: Accounts must be older than 90 days and include a phone number for verification
  • Non-Deceptive: Twitter must find no signs of deceptive activity on your account, including signs of platform manipulation or spam.

Verified Businesses – Gold Checkmark

While verified businesses or public figures used to share the blue checkmark, they will now be distinguished with a gold checkmark. The intent is to prevent confusion between individuals paying for verification and brands established on the platform. 

For now, brands that already have a blue checkmark will see their badge automatically converted to the gold option. However, Twitter’s product lead, Esther Crawford, says the company will soon be opening up portals for businesses to apply for gold checkmarks.

Important Verified Figures – “Official” Labels

Lastly, the social network is granting important public figures an “Official” label next to their account names. This label will be applied to all of the following types of accounts:

  • Government accounts
  • Political organizations (such as parties) 
  • Commercial companies & business partners
  • Major brands
  • Media outlets

For more, explore the new Help Center page for profile labels.

/0 Comments