On this page
Ralf Gundelach, Michael Mühlhauser, Dominik Herrmann · Full Paper, ARES 2026, August 24–27, 2026, Linköping (Sweden)
Summary
Many websites deploy bot detection that blocks automated browsers – the very tools researchers use to measure privacy and security practices on the web. Our literature survey shows that 83% of 81 measurement papers at top venues (2020–2025) do not report this problem. In a measurement study across the Tranco top 10,000 (four browser configurations, 40,000 page visits), we quantify the effect: headless Chromium is blocked on 15.2% of sites, the other configurations on 6.8–7.2%; 82% of blocks are attributable to bot detection. The resulting sample loss is not random but correlates with the infrastructure provider – and therefore cannot be corrected by stratified sampling.
Key numbers
- 83% of the 81 surveyed measurement papers (USENIX Security, IEEE S&P, CCS, NDSS, WWW, IMC, PETS; 2020–2025) do not report blocking rates; only 5% quantify them.
- 772 vs. 14: Of 6,936 pairwise comparable sites, 772 block only headless Chromium, while just 14 block only headed Chromium. For Firefox the difference is negligible (15 vs. 12).
- 82% of the 1,554 observed blocks are attributable to bot detection: 59% confirmed via vendor signatures (Cloudflare challenges, DataDome, PerimeterX/HUMAN, CAPTCHA, Imperva), 23% inferred from condition-dependent blocking. Only 18% are uniform infrastructure blocks.
- 75% of the sites that block only headless Chromium can be unblocked by adjusting HTTP headers alone (the user agent and
sec-ch-uaclient hints expose “HeadlessChrome”); 25% keep blocking – deeper detection layers at work. - Detection infrastructure exceeds active blocking: 34% of successfully scanned sites probe
navigator.webdriver, and 46% probe at least one unambiguous automation signal.
Taxonomy of detection techniques
The paper classifies client-side bot detection along two axes: by the specificity of the probed signals and by the instrumentation method used to observe these accesses.
| Signal tier | Meaning | Examples |
|---|---|---|
| Tier 1 | No legitimate purpose other than automation detection | navigator.webdriver, window._selenium |
| Tier 2 | Strong automation indicators that also serve fingerprinting | chrome.runtime, navigator.plugins |
| Tier 3 | General browser fingerprinting, only interpretable in context | navigator.userAgent, localStorage |
| Instrumentation method | How it works |
|---|---|
| Property interception | A recursive JavaScript proxy on window logs every property access including its full path |
| Prototype patching | Non-configurable getters on built-in prototypes are replaced with instrumented versions |
| Honeypot properties | Planted decoy properties (e.g., window._selenium) whose access proves detection intent |
The 28 identified tier-1 signals fall into eight framework families (WebDriver API, PhantomJS, NightmareJS, WebDriver Protocol, Selenium, Chrome DevTools, Awesomium, ChromeDriver). Commercial vendors combine them with fingerprinting across up to eight categories (canvas, audio, WebGL, speech, fonts, dimensions, permissions, plugins).
Self-check: Does your browser look like a bot?
This test probes a subset of the client-side signals from the taxonomy above, directly in your browser. It runs entirely locally – no data is transmitted or stored.
Recommendations for measurement studies
Report the browser setup
Document engine, version, and display mode (headless/headed) – the difference amounts to up to a factor of 2 in blocking rates.
Quantify blocking rates
Measure and report the observed block and failure rates during the crawl instead of treating them as noise.
Document the CDN composition
The infrastructure provider is a confounder: Cloudflare-hosted sites drop out of the sample systematically more often than others.
Disclose stealth measures
Countermeasures in use – or their deliberate omission – belong in the methods section.
Methods and limitations
Methods. Literature survey: 132 papers screened, 81 included, 12 criteria, double-coded subsample (Cohen’s κ = 0.90). Measurement study: Tranco top 10,000 (list of 2026-02-25), four configurations in a 2×2 design (Chromium/Firefox × headless/headed, Playwright without stealth measures), 40,000 page visits from datacenter IPs with IP rotation, 2026-02-27 to 2026-03-02. Block = HTTP 403/429/503; attribution to bot detection via manually verified vendor signatures and cross-condition analysis.
Limitations. Single European vantage point and a single measurement period; homepages of the top 10K only; the study captures explicit HTTP blocks, not degraded content served with status 200; the headed configurations only approximate human browsing. The client-side detection prevalence is a lower bound, as server-side techniques (TLS fingerprinting, IP reputation, behavioral analysis) remain invisible.
How to cite
The camera-ready version and DOI will appear at the conference; until then, please cite the preprint:
@misc{gundelach2026botdetection,
title = {Detecting Bot Detection: Prevalence, Techniques, and
Implications for Web Measurement Research},
author = {Gundelach, Ralf and M{\"u}hlhauser, Michael and Herrmann, Dominik},
year = {2026},
eprint = {2606.14525},
archivePrefix = {arXiv},
primaryClass = {cs.CR},
note = {Accepted at ARES 2026}
}Crawler, dataset, analysis notebooks, and the literature survey codings are openly available: doi.org/10.5281/zenodo.20334022 (screenshots excluded for copyright reasons).
Contact
Prof. Dr. Dominik Herrmann
Chair Privacy and Security in Information Systems
University of Bamberg, 96045 Bamberg
Chair Privacy and Security in Information Systems
University of Bamberg
96045 Bamberg
dh.psi@uni-bamberg.de | +49 951 863-2661
uni-mal-anders.de | LinkedIn
Prof. Dr. Dominik Herrmann
Chair Privacy and Security in Information Systems,
University of Bamberg, 96045 Bamberg
dh.psi@uni-bamberg.de
+49 951 863-2661
uni-mal-anders.de | LinkedIn