CVE-2021-41125: HTTP authentication credential leak to target websites in scrapy
Impact
If you use HttpAuthMiddleware (i.e. the httpuser and httppass spider attributes) for HTTP authentication, all requests will expose your credentials to the request target.
This includes requests generated by Scrapy components, such as robots.txt requests sent by Scrapy when the ROBOTSTXTOBEY setting is set to True, or as requests reached through redirects.
Patches
Upgrade to Scrapy 2.5.1 and use the new httpauthdomain spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials.
If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead.
Workarounds
If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basicauthheader function to convert your credentials into a value that you can assign to the Authorization header of your request, instead of defining your credentials globally using HttpAuthMiddleware.
For more information If you have any questions or comments about this advisory: Open an issue Email us
Other sources
Scrapy is a high-level web crawling and scraping framework for Python. If you use HttpAuthMiddleware (i.e. the httpuser and httppass spider attributes) for HTTP authentication, all requests will expose your credentials to the request target. This includes requests generated by Scrapy components, such as robots.txt requests sent by Scrapy when the ROBOTSTXTOBEY setting is set to True, or as requests reached through redirects. Upgrade to Scrapy 2.5.1 and use the new httpauthdomain spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead. If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basicauthheader function to convert your credentials into a value that you can assign to the Authorization header of your request, instead of defining your credentials globally using HttpAuthMiddleware.
— MITRE
Affected Software
Remediation
Recommended actions to resolve this vulnerability, in priority order.
- Upgrade
Upgrade
pip/Scrapyto a version that resolves this vulnerability.Fixed in 2.5.1 - Upgrade
Upgrade
pip/Scrapyto a version that resolves this vulnerability.Fixed in 1.8.1 - Upgrade
Upgrade
scrapyto a version that resolves this vulnerability.Fixed in 2.5.1 - Upgrade
Upgrade
scrapyto a version that resolves this vulnerability.Fixed in 1.8.1 - Configuration
When using Scrapy HTTP authentication with HttpAuthMiddleware, upgrade to Scrapy 2.5.1 and set the new `http_auth_domain` spider attribute so only intended domains receive the configured HTTP authentication credentials.
Scrapy spider http_auth_domain = (use new spider attribute to control allowed domains) - Configuration
If you cannot upgrade, stop defining HTTP authentication credentials globally via `HttpAuthMiddleware`; instead set HTTP authentication credentials per-request by generating an `Authorization` header value using `w3lib.http.basic_auth_header` and assigning it to each request’s `Authorization` header.
Scrapy HttpAuthMiddleware usage HttpAuthMiddleware (global credentials) = (avoid for global exposure)
Event History
Frequently Asked Questions
What is the severity of CVE-2021-41125?
CVE-2021-41125 is classified as a moderate severity vulnerability due to the exposure of HTTP authentication credentials.
How do I fix CVE-2021-41125?
To resolve CVE-2021-41125, upgrade Scrapy to version 2.5.1 or later, or to version 1.8.1.
What type of systems are affected by CVE-2021-41125?
CVE-2021-41125 affects specific versions of Scrapy and Debian Linux 9.0.
What is the nature of the vulnerability in CVE-2021-41125?
CVE-2021-41125 allows for the exposure of HTTP authentication credentials when using HttpAuthMiddleware.
Is there a workaround for CVE-2021-41125?
There are no known workarounds for CVE-2021-41125; upgrading to a fixed version is the recommended solution.