- Dillo-dev - lists.mailman3.com

Re: Fingerprinting lessons from 'curl-impersonate'

by Kevin Koster

Rodrigo Arias wrote: > On Thu, Jan 02, 2025 at 11:58:19AM +1100, Kevin Koster wrote: >> For all my tests User-Agent was: Dillo/3.1.0 >> >> I'm in Australia. Maybe CDNs are directing my requests to Web servers >> with different rules than others? > > Yeah, maybe there are some geo-fencing rules, although I was not > expecting it to be that way. There is also the possibility that > something is broken in Dillo or elsewhere. > > I also tested https://ebay.com.au/ and https://ebay.com/ with mbedtls > 2.28.9 and 3.6.1, and they are both blocked for me (from Spain). I > cannot reproduce what you are observing. > > In curl works for me, or in Dillo if I change the user agent to the one > curl uses. It also works if I set the user agent to "Dillo/3.1.0" in > curl as well as force it to use HTTP 1.1. > > Not sure what is going on. Could you send your http options? > > $ grep '^http' ~/.dillo/dillorc http_language="en-AU,en,en-US" http_strict_transport_security=NO http_referer=path

Jan. 2, 2025

Re: Fingerprinting lessons from 'curl-impersonate'

by Rodrigo Arias

Hi Kevin, On Thu, Jan 02, 2025 at 11:58:19AM +1100, Kevin Koster wrote: > For all my tests User-Agent was: Dillo/3.1.0 > > I'm in Australia. Maybe CDNs are directing my requests to Web servers > with different rules than others? Yeah, maybe there are some geo-fencing rules, although I was not expecting it to be that way. There is also the possibility that something is broken in Dillo or elsewhere. I also tested https://ebay.com.au/ and https://ebay.com/ with mbedtls 2.28.9 and 3.6.1, and they are both blocked for me (from Spain). I cannot reproduce what you are observing. In curl works for me, or in Dillo if I change the user agent to the one curl uses. It also works if I set the user agent to "Dillo/3.1.0" in curl as well as force it to use HTTP 1.1. Not sure what is going on. Could you send your http options? $ grep '^http' ~/.dillo/dillorc Best, Rodrigo.

Jan. 2, 2025

Re: Fingerprinting lessons from 'curl-impersonate'

by a1ex＠dismail.de

Kevin Koster wrote: > I'm in Australia. Maybe CDNs are directing my requests to Web servers > with different rules than others? Yes, that seems to be the case: > Nav_open_url: new url='https://www.lowes.com/' > Dns_server [0]: www.lowes.com is 23.54.30.24 23.54.30.14 I get a completely different DNS result from Europe: Nav_open_url: new url='https://www.lowes.com' Dns_server [0]: www.lowes.com is 2.16.1.211 2.16.1.234 Which is an edgekey/akamaiedge cdn located in France, whereas yours is a is a different akamai cdn which appears to be located in Sydney.

Jan. 2, 2025

Re: Fingerprinting lessons from 'curl-impersonate'

by Kevin Koster

On Thu, 2 Jan 2025 00:28:20 +0100 Rodrigo Arias <rodarima(a)gmail.com> wrote: > On Thu, Jan 02, 2025 at 09:26:13AM +1100, Kevin Koster wrote: > > https://washingtonpost.com/ redirects to > > https://www.washingtonpost.com/ and loads fine for me. > > I tested it and it continues to fail for me with several UA. But after > changing the user agent to mimick an Apple phone it seems to work both > with openssl and mbedtls. For all my tests User-Agent was: Dillo/3.1.0 I'm in Australia. Maybe CDNs are directing my requests to Web servers with different rules than others?

Jan. 2, 2025

Re: Fingerprinting lessons from 'curl-impersonate'

by Rodrigo Arias

Hi, On Thu, Jan 02, 2025 at 09:26:13AM +1100, Kevin Koster wrote: > https://washingtonpost.com/ redirects to > https://www.washingtonpost.com/ and loads fine for me. I tested it and it continues to fail for me with several UA. But after changing the user agent to mimick an Apple phone it seems to work both with openssl and mbedtls. Here is a working UA I found online: http_user_agent="Mozilla/5.0 (Apple-iPhone7C2/1202.466; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543 Safari/419.3" Same result as you for the previous hosts, both with openssl and mbedtls. It looks like they only discriminate the UA. > https://ebay.com/ redirects to https://www.ebay.com/ and loads for > me (I browse ebay.com.au in Dillo often). Ebay bans User-Agent: Dillo/3.1.1, but others seem to work. The above iPhone UA does. Best, Rodrigo.

Jan. 1, 2025

Re: Fingerprinting lessons from 'curl-impersonate'

by Kevin Koster

Rodrigo Arias wrote: > On Wed, Jan 01, 2025 at 04:16:58PM +0100, a1ex-J7K0XVabL0iELgA04lAiVw(a)public.gmane.org wrote: >> Here are a few example sites which seem to be doing something like that: >> >> https://www.spacecoastdaily.com > > This one seems to be loading for me. Also for me with Dillo 3.1.0 using mbed TLS 2.26.0. >> https://www.lowes.com/ > > Fails with: > > SSL_read() failed: SSL_ERROR_SYSCALL > Tls_close_by_key: Avoiding SSL shutdown for: https://www.lowes.com/ This URL loads for me. Log: Nav_open_url: new url='https://www.lowes.com/' Dns_server [0]: www.lowes.com is 23.54.30.24 23.54.30.14 Connecting to 23.54.30.24:443 www.lowes.com TLSv1.2, cipher TLS-ECDHE-ECDSA-WITH-CHACHA20-POLY1305-SHA256 >> https://elokata.santanderconsumer.pl > > Fails with: > > TLS ALERT on write: decode error > SSL_read() failed: error:0A000126:SSL routines::unexpected eof while reading > Tls_close_by_key: Avoiding SSL shutdown for: https://elokata.santanderconsumer.pl For me this redirects to: https://elokata.santanderconsumer.pl/Error/SecurityError?Reason=12573334125… Log: Nav_open_url: new url='https://elokata.santanderconsumer.pl' Dns_server [0]: elokata.santanderconsumer.pl is 195.138.208.195 Connecting to 195.138.208.195:443 elokata.santanderconsumer.pl TLSv1.2, cipher TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384 Nav_open_url: new url='https://elokata.santanderconsumer.pl/Error/SecurityError?Reason=12573334125…' Connecting to 195.138.208.195:443 But that second URL never loads (no error logged). https://elokata.santanderconsumer.pl redirects to and sucessfully loads an error page in Firefox 128 with NoScript. > Thanks for the list, I will check later each one in case we are doing > something wrong. > >> And of course washingtonpost.com, which you are already aware of. I'm >> sure there are more. > > Yeah, the case of washingtonpost.com is truly appaling. They were just > banning right away anything that is not HTTP/2. When I complained with a > curl reproducer so they allow Dillo via HTTP/1.1, they also banned > curl... https://washingtonpost.com/ redirects to https://www.washingtonpost.com/ and loads fine for me. > Similar problem with ebay.com, due to Akamai (edgesuite). I also > reported it, but so far no reply (this time, no curl reproducer). https://ebay.com/ redirects to https://www.ebay.com/ and loads for me (I browse ebay.com.au in Dillo often).

Jan. 1, 2025

Re: Fingerprinting lessons from 'curl-impersonate'

by Schimon Jehudah

Greetings, and Happy New Year! Client side ----------- Some check/restriction pages are pseudo and can be bypassed with a refresh. See https://greasyfork.org/en/scripts/493323-xhreload It can also work by clicking on the buttons back and forward with some browsers. Server side ----------- However, some are server side, and therefore the so called "PassKey" be required, which is the only reason that I have FF installed, until it is available for Falkon. fixoutlook.org -------------- As was with Outlock which has foisted a modified HTML format, which could not be displayed with other email clients, I would suggest to organize a similar campaign, or better yet, a boycott. It was "fixoutlook.org" and later redirected to "email-standards.org". I sense that it was a form of a controlled opposition in order to make a benevolent display or distration to the public. Boycott ------- Therefore, I suggest to begin a boycott campaign, and consequently teach and inform people of recommended means to publish and serve contents on the internet, including utilization of P2P file-sharing software. It might be too radical to you, yet I am a Jew and I am of the orthodox Jewish sector, and that sector has mostly inner communication exchanges and less with other sectors and authorities (i.e. government). So the practical mean to them, is to boycott and inform others of recommended commercial or cultural practices, if necessary. As a Jew, I am a part of such community of well over a 100K for almost a decade, and I highly recommend to try that practice of boycott. Happy New Year, Schimon On Wed, 1 Jan 2025 14:24:05 +0100 Rodrigo Arias <rodarima(a)gmail.com> wrote: > Hi Kevin, > > On Wed, Jan 01, 2025 at 12:00:32PM +1100, Kevin Koster wrote: > > Yes that seems to work for me too, strangely. Anyway that > > particular URL was just a random example I found in Firefox's > > history. > > I see, here is a similar thread in Palemoon: > https://forum.palemoon.org/viewtopic.php?f=3&t=31339 > > > Seems unlikely that it will be tied to a JS-free option for the > > initial "proof-of-work" challenge. But yes it might be useful after > > completing the first challenge in Firefox, if switching browser > > can work in practice. > > Privacy pass is not tied to a particular challenge. Cloudflare now > supports a JS captcha but other non-JS challenges may be added in the > future. On the other hand, this also opens the door to Apple > enforcing only non-modified iOS systems to access a website, which is > just pure evil, > see: https://educatedguesswork.org/posts/private-access-tokens/ > > I think the idea of private tokens may eventually solve the spam > problem, but I think in its current form is harmful. > > Best, > Rodrigo. > _______________________________________________ > Dillo-dev mailing list -- dillo-dev(a)mailman3.com > To unsubscribe send an email to dillo-dev-leave(a)mailman3.com

Jan. 1, 2025

Re: Fingerprinting lessons from 'curl-impersonate'

by a1ex＠dismail.de

Hi Rodrigo, Rodrigo Arias <rodarima(a)gmail.com> wrote: > > https://www.spacecoastdaily.com > > This one seems to be loading for me. Interesting. I don't get a TLS error, but it returns a 403 (that's with a FF UA in Dillo). In FF proper, the page loads fine. > Yeah, the case of washingtonpost.com is truly appaling. They were > just banning right away anything that is not HTTP/2. When I > complained with a curl reproducer so they allow Dillo via HTTP/1.1, > they also banned curl... Doubling down on their BS. Luckily nobody needs their site, but it's troubling that these BigCo shits seem to have zero accountability or consequences for their antisocial behavior. Regards, Alex

Jan. 1, 2025

Re: Fingerprinting lessons from 'curl-impersonate'

by Rodrigo Arias

Hi, On Wed, Jan 01, 2025 at 04:16:58PM +0100, a1ex(a)dismail.de wrote: > Hi Rodrigo, > > > I experienced problems with the user-agent being banned, and having to > > impersonate Firefox to load some sites. I haven't found yet examples > > of this deep fingerprinting for TLS or similar, you? > > Here are a few example sites which seem to be doing something like that: > > https://www.spacecoastdaily.com This one seems to be loading for me. > https://www.lowes.com/ Fails with: SSL_read() failed: SSL_ERROR_SYSCALL Tls_close_by_key: Avoiding SSL shutdown for: https://www.lowes.com/ > https://elokata.santanderconsumer.pl Fails with: TLS ALERT on write: decode error SSL_read() failed: error:0A000126:SSL routines::unexpected eof while reading Tls_close_by_key: Avoiding SSL shutdown for: https://elokata.santanderconsumer.pl Thanks for the list, I will check later each one in case we are doing something wrong. > And of course washingtonpost.com, which you are already aware of. I'm > sure there are more. Yeah, the case of washingtonpost.com is truly appaling. They were just banning right away anything that is not HTTP/2. When I complained with a curl reproducer so they allow Dillo via HTTP/1.1, they also banned curl... Similar problem with ebay.com, due to Akamai (edgesuite). I also reported it, but so far no reply (this time, no curl reproducer). I also contacted CloudFlare to see if there could be an option for non-JS browsers to solve a similar challenge so we are not excluded. > PS. Mails from the list don't seem to be coming through to my email > anymore... maybe my provider changed something recently, will have to > investigate. Hmm, I haven't touched anything in the settings... I'll CC you. Best, Rodrigo.

Jan. 1, 2025

Re: Fingerprinting lessons from 'curl-impersonate'

by a1ex＠dismail.de

Hi Rodrigo, > I experienced problems with the user-agent being banned, and having to > impersonate Firefox to load some sites. I haven't found yet examples > of this deep fingerprinting for TLS or similar, you? Here are a few example sites which seem to be doing something like that: https://www.spacecoastdaily.com https://www.lowes.com/ https://elokata.santanderconsumer.pl And of course washingtonpost.com, which you are already aware of. I'm sure there are more. Regards, Alex PS. Mails from the list don't seem to be coming through to my email anymore... maybe my provider changed something recently, will have to investigate.

Jan. 1, 2025