Bot traffic – what is it and how to block it?

Bot traffic is an issue that increasingly spends the sleep of website administrators. On the one hand, this phenomenon can be useful, but on the other, it can cause many problems and risks. What is it really and how to block bot traffic? In this article you will find out:

• What is bot traffic?
• What are good and bad bots?
• What are the ways of identifying bot traffic?
• How do I block bot traffic?

Bot traffic – definition

Bot traffic is a type of internet traffic that is generated by automatic computer programmes, commonly known as bots. Internet robots are used for various purposes, among which we can mention indexing of content by search engines, monitoring of prices in online shops, but also illegal activities – DDoS attacks or spreading spam.

Bot traffic is website traffic generated by bots, i.e. automatic computer programmes.

What is bot traffic?

The presence of bots on a website can also distort the results, extracted from analytical tools.

Good and bad bots

Bots can be divided into good and bad. Among the former, we can mention indexing bots (software that scans websites to collect information, used by web browsers), chatbots (serving e.g. online shop customers via chat), intelligent assistants (such as Siri or Google Assistant), marketing bots (tracking ads and monitoring customer feedback and trends) and aggregation bots (collecting news from multiple sources and delivering it to users of a given service, such as Google Discover or Google News).

‘Bad’ bots, on the other hand, include spam bots (create fake user accounts and spread spam across the web), web copying bots (copy web content and share it with other sources), credential snooping bots (retrieve leaked passwords and try to access other web resources using them), DDoS bots (spread malware) and advertising fraud bots (software that clicks on ads to generate revenue in a fraudulent manner).

Ways to identify bot traffic

How can we recognise the presence and activity of robots on our website? Your concerns should include:

• Extended or shortened session time – the presence of bots on the website may be evidenced by, among other things, non-standard (extended or shortened) session time. Robots browse the website very quickly or very slowly, which distinguishes this type of traffic from the traffic generated by users.
• Unusual number of page views – bot traffic may also be identified by an unusual number of page views – if you observe a significant increase in the number of page views for no apparent reason, this may mean that the traffic is generated by robots.
• A significant increase in traffic from a single location – the presence of robots will also be indicated by a marked increase in site traffic coming from a single location.
• A sharp increase in Bounce Rate – bot traffic can also be identified by a sharp increase in Bounce Rate – this could mean, for example, that robots have targeted one site.
• Falsified user data – Bot traffic can also be identified by falsified user data (e.g. in forms on the website) – this is generated by robots automatically filling in such documents.

Methods of blocking bot traffic

What should you do if you recognise bot traffic on your website and want to block it? Below are some methods for blocking or limiting bot traffic completely. However, it is important to bear in mind that overly restrictive safeguards may lead to a hindrance to the use of the site by genuine users. Any steps should therefore be well thought out and implemented with a balance between security and usability. Methods of blocking bot traffic include:

• CAPTCHA security – CAPTCHAs are tests that require users to perform a specific action, such as recognising characters or images, to confirm that they are humans and not bots. This is an effective way to filter out automated bots, as most bots are unable to solve such a task.
• Create a robots.txt file – this file is an instruction to index the site for bots visiting it. It allows you to designate areas that bots cannot access.
• Honeypots – otherwise known as ‘honey’ traps created specifically for bots. These are hidden elements on a page that are visible to bots, but not to real users. Bots attempt to interact with honeypots, which allows them to be identified and blocked.
• Setting up a WAF web application firewall – this is used to filter suspicious requests and block IP addresses based on various factors.
• Setting request frequency limits – this involves limiting the number of requests that can be sent from a given IP address at a certain time. This can help minimise the impact of bot traffic on site performance.
• Use of bot detection systems – implementing anti-bot systems that detect malware, including PerimeterX.

Keep in mind that the key to effective bot traffic management is to understand the differences between bot types and to use appropriate security solutions. Keep in mind that not all bots have a negative impact on your site – some perform important functions, such as helping search engines index your content. With the right approach, you can effectively protect your site from unwanted traffic, keeping your users safe and comfortable with your services.