Re: Fingerprinting lessons from 'curl-impersonate'
Hi Rodrigo,
I experienced problems with the user-agent being banned, and having to impersonate Firefox to load some sites. I haven't found yet examples of this deep fingerprinting for TLS or similar, you?
Here are a few example sites which seem to be doing something like that: https://www.spacecoastdaily.com https://www.lowes.com/ https://elokata.santanderconsumer.pl And of course washingtonpost.com, which you are already aware of. I'm sure there are more. Regards, Alex PS. Mails from the list don't seem to be coming through to my email anymore... maybe my provider changed something recently, will have to investigate.
Hi, On Wed, Jan 01, 2025 at 04:16:58PM +0100, a1ex@dismail.de wrote:
Hi Rodrigo,
I experienced problems with the user-agent being banned, and having to impersonate Firefox to load some sites. I haven't found yet examples of this deep fingerprinting for TLS or similar, you?
Here are a few example sites which seem to be doing something like that:
This one seems to be loading for me.
Fails with: SSL_read() failed: SSL_ERROR_SYSCALL Tls_close_by_key: Avoiding SSL shutdown for: https://www.lowes.com/
Fails with: TLS ALERT on write: decode error SSL_read() failed: error:0A000126:SSL routines::unexpected eof while reading Tls_close_by_key: Avoiding SSL shutdown for: https://elokata.santanderconsumer.pl Thanks for the list, I will check later each one in case we are doing something wrong.
And of course washingtonpost.com, which you are already aware of. I'm sure there are more.
Yeah, the case of washingtonpost.com is truly appaling. They were just banning right away anything that is not HTTP/2. When I complained with a curl reproducer so they allow Dillo via HTTP/1.1, they also banned curl... Similar problem with ebay.com, due to Akamai (edgesuite). I also reported it, but so far no reply (this time, no curl reproducer). I also contacted CloudFlare to see if there could be an option for non-JS browsers to solve a similar challenge so we are not excluded.
PS. Mails from the list don't seem to be coming through to my email anymore... maybe my provider changed something recently, will have to investigate.
Hmm, I haven't touched anything in the settings... I'll CC you. Best, Rodrigo.
Hi Rodrigo, Rodrigo Arias <rodarima@gmail.com> wrote:
This one seems to be loading for me.
Interesting. I don't get a TLS error, but it returns a 403 (that's with a FF UA in Dillo). In FF proper, the page loads fine.
Yeah, the case of washingtonpost.com is truly appaling. They were just banning right away anything that is not HTTP/2. When I complained with a curl reproducer so they allow Dillo via HTTP/1.1, they also banned curl...
Doubling down on their BS. Luckily nobody needs their site, but it's troubling that these BigCo shits seem to have zero accountability or consequences for their antisocial behavior. Regards, Alex
Rodrigo Arias wrote:
On Wed, Jan 01, 2025 at 04:16:58PM +0100, a1ex-J7K0XVabL0iELgA04lAiVw@public.gmane.org wrote:
Here are a few example sites which seem to be doing something like that:
This one seems to be loading for me.
Also for me with Dillo 3.1.0 using mbed TLS 2.26.0.
Fails with:
SSL_read() failed: SSL_ERROR_SYSCALL Tls_close_by_key: Avoiding SSL shutdown for: https://www.lowes.com/
This URL loads for me. Log: Nav_open_url: new url='https://www.lowes.com/' Dns_server [0]: www.lowes.com is 23.54.30.24 23.54.30.14 Connecting to 23.54.30.24:443 www.lowes.com TLSv1.2, cipher TLS-ECDHE-ECDSA-WITH-CHACHA20-POLY1305-SHA256
Fails with:
TLS ALERT on write: decode error SSL_read() failed: error:0A000126:SSL routines::unexpected eof while reading Tls_close_by_key: Avoiding SSL shutdown for: https://elokata.santanderconsumer.pl
For me this redirects to: https://elokata.santanderconsumer.pl/Error/SecurityError?Reason=125733341256... Log: Nav_open_url: new url='https://elokata.santanderconsumer.pl' Dns_server [0]: elokata.santanderconsumer.pl is 195.138.208.195 Connecting to 195.138.208.195:443 elokata.santanderconsumer.pl TLSv1.2, cipher TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384 Nav_open_url: new url='https://elokata.santanderconsumer.pl/Error/SecurityError?Reason=125733341256...' Connecting to 195.138.208.195:443 But that second URL never loads (no error logged). https://elokata.santanderconsumer.pl redirects to and sucessfully loads an error page in Firefox 128 with NoScript.
Thanks for the list, I will check later each one in case we are doing something wrong.
And of course washingtonpost.com, which you are already aware of. I'm sure there are more.
Yeah, the case of washingtonpost.com is truly appaling. They were just banning right away anything that is not HTTP/2. When I complained with a curl reproducer so they allow Dillo via HTTP/1.1, they also banned curl...
https://washingtonpost.com/ redirects to https://www.washingtonpost.com/ and loads fine for me.
Similar problem with ebay.com, due to Akamai (edgesuite). I also reported it, but so far no reply (this time, no curl reproducer).
https://ebay.com/ redirects to https://www.ebay.com/ and loads for me (I browse ebay.com.au in Dillo often).
Hi, On Thu, Jan 02, 2025 at 09:26:13AM +1100, Kevin Koster wrote:
https://washingtonpost.com/ redirects to https://www.washingtonpost.com/ and loads fine for me.
I tested it and it continues to fail for me with several UA. But after changing the user agent to mimick an Apple phone it seems to work both with openssl and mbedtls. Here is a working UA I found online: http_user_agent="Mozilla/5.0 (Apple-iPhone7C2/1202.466; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543 Safari/419.3" Same result as you for the previous hosts, both with openssl and mbedtls. It looks like they only discriminate the UA.
https://ebay.com/ redirects to https://www.ebay.com/ and loads for me (I browse ebay.com.au in Dillo often).
Ebay bans User-Agent: Dillo/3.1.1, but others seem to work. The above iPhone UA does. Best, Rodrigo.
On Thu, 2 Jan 2025 00:28:20 +0100 Rodrigo Arias <rodarima@gmail.com> wrote:
On Thu, Jan 02, 2025 at 09:26:13AM +1100, Kevin Koster wrote:
https://washingtonpost.com/ redirects to https://www.washingtonpost.com/ and loads fine for me.
I tested it and it continues to fail for me with several UA. But after changing the user agent to mimick an Apple phone it seems to work both with openssl and mbedtls.
For all my tests User-Agent was: Dillo/3.1.0 I'm in Australia. Maybe CDNs are directing my requests to Web servers with different rules than others?
Kevin Koster wrote:
I'm in Australia. Maybe CDNs are directing my requests to Web servers with different rules than others?
Yes, that seems to be the case:
Nav_open_url: new url='https://www.lowes.com/' Dns_server [0]: www.lowes.com is 23.54.30.24 23.54.30.14
I get a completely different DNS result from Europe: Nav_open_url: new url='https://www.lowes.com' Dns_server [0]: www.lowes.com is 2.16.1.211 2.16.1.234 Which is an edgekey/akamaiedge cdn located in France, whereas yours is a is a different akamai cdn which appears to be located in Sydney.
Hi Kevin, On Thu, Jan 02, 2025 at 11:58:19AM +1100, Kevin Koster wrote:
For all my tests User-Agent was: Dillo/3.1.0
I'm in Australia. Maybe CDNs are directing my requests to Web servers with different rules than others?
Yeah, maybe there are some geo-fencing rules, although I was not expecting it to be that way. There is also the possibility that something is broken in Dillo or elsewhere. I also tested https://ebay.com.au/ and https://ebay.com/ with mbedtls 2.28.9 and 3.6.1, and they are both blocked for me (from Spain). I cannot reproduce what you are observing. In curl works for me, or in Dillo if I change the user agent to the one curl uses. It also works if I set the user agent to "Dillo/3.1.0" in curl as well as force it to use HTTP 1.1. Not sure what is going on. Could you send your http options? $ grep '^http' ~/.dillo/dillorc Best, Rodrigo.
Rodrigo Arias wrote:
On Thu, Jan 02, 2025 at 11:58:19AM +1100, Kevin Koster wrote:
For all my tests User-Agent was: Dillo/3.1.0
I'm in Australia. Maybe CDNs are directing my requests to Web servers with different rules than others?
Yeah, maybe there are some geo-fencing rules, although I was not expecting it to be that way. There is also the possibility that something is broken in Dillo or elsewhere.
I also tested https://ebay.com.au/ and https://ebay.com/ with mbedtls 2.28.9 and 3.6.1, and they are both blocked for me (from Spain). I cannot reproduce what you are observing.
In curl works for me, or in Dillo if I change the user agent to the one curl uses. It also works if I set the user agent to "Dillo/3.1.0" in curl as well as force it to use HTTP 1.1.
Not sure what is going on. Could you send your http options?
$ grep '^http' ~/.dillo/dillorc
http_language="en-AU,en,en-US" http_strict_transport_security=NO http_referer=path
participants (3)
-
a1ex＠dismail.de -
Kevin Koster -
Rodrigo Arias