- Dillo-dev - lists.mailman3.com

Re: HTTPS Proxy Support

by Kevin Koster

Rodrigo Arias <rodarima-Re5JQEeQqe8AvxtiuMwx3w(a)public.gmane.org> wrote: > On Sat, Jun 22, 2024 at 09:43:00AM +1000, Kevin Koster wrote: >>But the proxy approach allows old versions/binaries to keep working >>too. If Dillo 3.0.5 had supported it, then the HTTPS issues from >>lacking SNI support could have been worked around by running an >>'old-style' HTTPS proxy with SNI support on localhost. > > A related term seems to be "reverse proxy": > > https://en.wikipedia.org/wiki/Reverse_proxy Yes, though in practice that tends to imply a caching proxy run by website operators, hence "reverse" because a normal proxy is run by the person accessing the content on other people's servers, but these are run by the server operators. But the mechanism is the same as my usage. Real-world "reverse proxies" just tend to be geared away from internet-wide usage to serve requests to any server, not least because HTTPS requires the proxy to have the target website's certificate. Or at least based on looking at Squid and Nginx, internet-wide "reverse" proxy isage isn't documented clearly enough for me to spot the easy way of doing it (while also manipulating the content being retrieved). I did see a very hard way to do it with Squid and an abandoned 3rd-party library that wouldn't even compile for me. > From the "Fun with Crypto Ancienne" post I understand that you want > Dillo to get a HTTP or HTTPS URL and always perform a HTTP GET towards > your proxy, as the Mosaic configuration suggests: > >> https 127.0.0.1 8765 http >> http 127.0.0.1 8765 http Yes. Which, if it helps to decode my earlier posts, is equivalent to this Wget command before they secretly changed its https_proxy behaviour: http_proxy=127.0.0.1:8765 https_proxy=127.0.0.1:8765 wget https://example.com > Other that those old browsers, I don't think you can do this with any > (relatively) modern tool. Which is annoying because I didn't need it until (relatively) recently. It looks like current Lynx is designed to use HTTPS tunneling through proxies now, based on use of the do_connect variable in WWW/Library/Implementation/HTTP.c of its source code. Though with Lynx 2.8.9rel.1 and 2.9.0dev.6 I can't actually get it to talk to anything (inc. Netcat on localhost) with http_proxy or https_proxy, even though setting each variable causes page loads to fail over the respective protocol. Not a firewall issue. > This is what Dillo is currently doing for https and http URLs: > > hop% http_proxy=http://localhost:1234 dillo http://www.google.com > hop% nc -l 1234 > GET http://www.google.com/ HTTP/1.1 > Host: www.google.com > User-Agent: Dillo/3.1.1 > Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > Accept-Encoding: gzip, deflate > DNT: 1 > Referer: http://www.google.com/ > Connection: keep-alive > > hop% http_proxy=http://localhost:1234 dillo https://www.google.com > hop% nc -l 1234 > CONNECT www.google.com:443 HTTP/1.1 > Host: www.google.com:443 > > While for the latter you'll want: > > GET https://www.google.com/ HTTP/1.1 > Host: www.google.com > User-Agent: Dillo/3.1.1 > Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > Accept-Encoding: gzip, deflate > DNT: 1 > Referer: https://www.google.com/ > Connection: keep-alive > > You'll need to instruct Dillo to ignore the HTTPS handling and just > treat it as an HTTP GET request to the proxy (using the HTTPS url), so > no CONNECT is used. Yes, though ideally like in Wget this behaviour wouldn't be caused by setting 'http_proxy'. You'd have a variable/setting like Wget's 'https_proxy' which only causes HTTPS connections to try and use the proxy, while HTTP connections go through directly. Then you could set 'http_proxy' as well if you wanted to use a proxy for HTTP connections too. For example (with 'https_endpoint' instead of the confusing 'https_proxy' variable/setting name I proposed earlier): http_proxy=127.0.0.1:1234 dillo http://www.google.com [ 127.0.0.1:1234 ] GET http://www.google.com/ HTTP/1.1 http_proxy=127.0.0.1:1234 dillo https://www.google.com [ 127.0.0.1:1234 ] CONNECT www.google.com:443 HTTP/1.1 https_endpoint=127.0.0.1:5678 dillo http://www.google.com [ www.google.com:80 ] GET http://www.google.com/ HTTP/1.1 https_endpoint=127.0.0.1:5678 dillo https://www.google.com [ 127.0.0.1:5678 ] GET https://www.google.com/ HTTP/1.1 http_proxy=127.0.0.1:1234 https_endpoint=127.0.0.1:5678 dillo http://www.google.com [ 127.0.0.1:1234 ] GET http://www.google.com/ HTTP/1.1 http_proxy=127.0.0.1:1234 https_endpoint=127.0.0.1:5678 dillo https://www.google.com [ 127.0.0.1:5678 ] GET https://www.google.com/ HTTP/1.1 So the behaviour of 'http_proxy' alone isn't changed. Also I don't think that 'https_endpoint' actually needs to be set from an environment variable, just a dillorc setting would do.

June 23, 2024

Re: 'Dillo Downloads' box cutnpaste

by pastebin＠gmx.com

Hi any chance that the 'mouse exit from file name location' in the 'Dillo Downloads' dialog box (where 'target file tooltip' disappears) to 'mouse exist from 'Dillo Downloads' dialog box itself? i think this would allow cutnpaste of the information in the tooltip or could you tell me where the code might be for 'mouse exit from file name location' action? On Sat, 8 Jun 2024 09:47:59 -0600 pastebin(a)gmx.com wrote: > Hi > > if i have a page of links > > i cab do a mouse-right-click on one of the url links and i get the 'link menu' and then i can click on 'save link as..' and it saves the html fine in the current directory in most cases and then there is now a 'Dillo Download' box with 'Done' on it on the screen and running the mouse over the filename causes a 'Target File: xxx' box to be displayed with all the files information displayed > > The problem is when after i run mouse over the saved file name on the 'Dillo Download' box the 'Target File: xxxx) information disappears from the screen when i attempt to cutnpaste the information there > > can this behavior be changed so that moving the mouse completely from the 'Dillo Download' box and not just the file name location removes the 'Target File: xxx' information from the screen so a cutnpaste can be performed on the information in the 'Target File: xxx' box? > _______________________________________________ > Dillo-dev mailing list -- dillo-dev(a)mailman3.com > To unsubscribe send an email to dillo-dev-leave(a)mailman3.com

June 22, 2024

Re: extra /tmp on file name line when save html

by pastebin＠gmx.com

Hi i haven't had this problem since upgrading to fltk-1.3.9 On Sat, 8 Jun 2024 21:31:32 +0200 Rodrigo Arias <rodarima(a)gmail.com> wrote: > Hi, > > On Sat, Jun 08, 2024 at 10:04:40AM -0600, pastebin(a)gmx.com wrote: > >> Which version of Dillo and FLTK do you have? Cannot reproduce with > >> Dillo 3.1.1 and FLTK 1.3.9. > >> > >> I'm thinking it may be related to file access permissions, > > > >i'm thinking it has to do with 'accidentally hitting an arrow key > >before i want to type a file name to save' but nothing is reproducable > > I reviewed the code and it doesn't seem this is likely to be on Dillo > side, as we only call fl_file_chooser() of FLTK: > > https://github.com/dillo-browser/dillo/blob/e89a2293ebb3331dca46f5edf010823… > > Please report your FLTK version, as it may be already solved on 1.3.9 > and that could explain why I don't see it. > > Rodrigo. > _______________________________________________ > Dillo-dev mailing list -- dillo-dev(a)mailman3.com > To unsubscribe send an email to dillo-dev-leave(a)mailman3.com

June 22, 2024

Re: dialog box dimensions

by pastebin＠gmx.com

Hi it turns out this has nothing to do with dillo but a [transient] entry in the fluxbox 'apps' file takes care of it https://fluxboxwiki.github.io/category/howtos/en/Editing_the_apps_file.html http://fluxbox.org/help/man-fluxbox-apps.php right clicking on the dillo 'save page as' title bar and 'remember' options sets it up > > >see attached pngs > > > >1.png 'save page as file' dialog - dialog size i want > > > >2.png 'save page as file' dialog - file already exists - dialog size i want > > > >3.png 'file exists' dialog - is the same size as 'save page as file' > >dialog - the size is to large for what i want so if i reduce the > >dimensions - the next 'save page as file' dialog i get will be to small > >(the same size as what i made the 'file exists' dialog > > >dillo saves one dimension size for the two dialog boxes > >maybe 2 separate size dimensions for 'save page as file' and 'file > >exists' dialog > > If you have a working patch I could merge it into 3.1.1. Otherwise I can > queue it among the other issues, as I would need to investigate further > how to solve it. > > I have a patch somewhere that switches to a native file picker, which > may solve one of the problems (it also remembers where you saved the > last file). But this will probably be added in 3.2.0, as I'm finishing > bug fixes for 3.1.1 (no more features for now). >

June 22, 2024

Re: HTTPS Proxy Support

by Rodrigo Arias

Hi, On Sat, Jun 22, 2024 at 09:43:00AM +1000, Kevin Koster wrote: >But the proxy approach allows old versions/binaries to keep working >too. If Dillo 3.0.5 had supported it, then the HTTPS issues from >lacking SNI support could have been worked around by running an >'old-style' HTTPS proxy with SNI support on localhost. A related term seems to be "reverse proxy": https://en.wikipedia.org/wiki/Reverse_proxy From the "Fun with Crypto Ancienne" post I understand that you want Dillo to get a HTTP or HTTPS URL and always perform a HTTP GET towards your proxy, as the Mosaic configuration suggests: > https 127.0.0.1 8765 http > http 127.0.0.1 8765 http Other that those old browsers, I don't think you can do this with any (relatively) modern tool. This is what Dillo is currently doing for https and http URLs: hop% http_proxy=http://localhost:1234 dillo http://www.google.com hop% nc -l 1234 GET http://www.google.com/ HTTP/1.1 Host: www.google.com User-Agent: Dillo/3.1.1 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Encoding: gzip, deflate DNT: 1 Referer: http://www.google.com/ Connection: keep-alive hop% http_proxy=http://localhost:1234 dillo https://www.google.com hop% nc -l 1234 CONNECT www.google.com:443 HTTP/1.1 Host: www.google.com:443 While for the latter you'll want: GET https://www.google.com/ HTTP/1.1 Host: www.google.com User-Agent: Dillo/3.1.1 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Encoding: gzip, deflate DNT: 1 Referer: https://www.google.com/ Connection: keep-alive You'll need to instruct Dillo to ignore the HTTPS handling and just treat it as an HTTP GET request to the proxy (using the HTTPS url), so no CONNECT is used. >Sure, or actually I'd suggest using a separate variable/option name >to set that proxy server because using "https_proxy" for both proxy >systems is what's already confused me now with Wget. Something >like "https_endpoint=[hostname]" maybe? I think I would have to see it implemented to determine what would be a good name that prevents users from accidentally misconfiguring it. Best, Rodrigo.

June 22, 2024

Re: HTTPS Proxy Support

by Kevin Koster

Rodrigo Arias <rodarima-Re5JQEeQqe8AvxtiuMwx3w(a)public.gmane.org> wrote: > On Thu, Jun 20, 2024 at 04:07:27PM +1000, Kevin Koster wrote: >>My content-modifying proxy application might be very niche, but I still >>think it would be more widely useful for people running Dillo on very >>old computers. Let me know whether it's something that might be accepted >>for Dillo even if I decide to try implementing it myself. > > Provided that those machines cannot deal with HTTPS traffic, I think is > a reasonable use case. I haven't had time to dig a bit more into this > issue to see if there is a way to make those old machines use TLS > instead Oh there is, the Crypto Ancienne TLS library is designed for this: https://github.com/classilla/cryanc I started work on supporting it for HTTPS in Dillo, and got far enough that I'm sure the library would work for it, though I didn't finish debugging my attempt. That was while Dillo 3.1 looked abandoned so I did it as a HTTPS DPI plug-in, which makes my work obsolete now. But the proxy approach allows old versions/binaries to keep working too. If Dillo 3.0.5 had supported it, then the HTTPS issues from lacking SNI support could have been worked around by running an 'old-style' HTTPS proxy with SNI support on localhost. > I won't oppose to it, but we probably want to ensure that we don't send > plain traffic to a proxy when browsing HTTPS pages by accident. We > should add an option like "allow_insecure_proxy" (false by default) or > similar, so it is harder to leak HTTPS traffic accidentally. Sure, or actually I'd suggest using a separate variable/option name to set that proxy server because using "https_proxy" for both proxy systems is what's already confused me now with Wget. Something like "https_endpoint=[hostname]" maybe?

June 21, 2024

Re: problem with compressed or encrypted url

by Rodrigo Arias

Hi, On Fri, Jun 21, 2024 at 09:52:06AM +1000, Kevin Koster wrote: >pastebin-KK0ffGbhmjU(a)public.gmane.org wrote: >> Hi >> >> someone please tell me what is wrong when i do >> >> dillo https://ubuntu.com/tutorials/configure-ssh-2fa#2-installing-and-configuring… >> >> it seems to be encrypted or compressed? > >It failed the first time for me, then trying a little later it >loads. In between I tried using "wget -S --spider" and the first >time the header showed it was compressed using Brotli compression: >"content-encoding: br" > >On subsequent tries it returned uncompressed data to Wget and >Dillo, which is correct since neither supports Brotli compression. >It appears that the originating Web server is mis-configured to >use Brotli compression even if the client doesn't announce support >for it. >[...] >The difference appears to be whether the page is served from a >cache server. The cache server still latter notices the browser's >Accept-Encoding header and doesn't use Brotli compression. Very good analysis, thanks :-) I reported the issue upstream: https://github.com/canonical/ubuntu.com/issues/13980 Best, Rodrigo.

June 21, 2024

Re: HTTPS Proxy Support

by Rodrigo Arias

Hi Kevin, On Thu, Jun 20, 2024 at 04:07:27PM +1000, Kevin Koster wrote: >On Thu, 20 Jun 2024 10:03:59 +1000 >"Kevin Koster" <dillo(a)ombertech.com> wrote: > >> Rodrigo Arias <rodarima-Re5JQEeQqe8AvxtiuMwx3w(a)public.gmane.org> >> wrote: >> > However, if you start Dillo from a shell where the http_proxy >> > variable is set in the environment, I would imagine it would be >> > already picked by Dillo and the internal wget command. The dpid >> > daemon has to start from this shell, so you should do a "dpidc stop" >> > command before to ensure it. >> >> I'm not sure if we're on the same page, Wget supports http_proxy >> and https_proxy environment variables. The former may or may not >> tunnel HTTPS requests through a Web proxy, but that's not what I >> want since then the proxy can't manipulate the data and the client >> needs to handle the encryption. https_proxy with Wget uses the >> proxy to do all the encryption and therefore those issues don't >> apply. More browsers support the HTTPS tunneling proxy feature, >> but that doesn't achieve anything that I'm interested in >> personally. > >I did some tests and yes the https_proxy environment variable is picked >up by Wget for the downloads DPI. It doesn't affect Dillo itself. The >http_proxy environment variable is picked up by Dillo, but as I say that >doesn't do what I want with Dillo or Wget. > >It appears that GNU Wget's behaviour has changed. Newer versions _do_ >try to tunnel the HTTPS connection through the proxy set to https_proxy >(not what I want), whereas older versions ask that proxy to establish >the HTTPS connection and send data back unencrypted (which is what I >want). So I picked a bad example with Wget. See that blog post for other >examples like Lynx, but I haven't tried them myself. > >I can't find this change in the Wget changelog. One version that works >as I intend it to is GNU Wget 1.9, circa 2003. Looking through the Wget >mailing list archives it seems like many people either considered the >old behaviour a bug or a useless feature, so I may be swimming against >the tide. > >My content-modifying proxy application might be very niche, but I still >think it would be more widely useful for people running Dillo on very >old computers. Let me know whether it's something that might be accepted >for Dillo even if I decide to try implementing it myself. > >Sorry for the confusion. Provided that those machines cannot deal with HTTPS traffic, I think is a reasonable use case. I haven't had time to dig a bit more into this issue to see if there is a way to make those old machines use TLS instead, which I think it would be preferred. I won't oppose to it, but we probably want to ensure that we don't send plain traffic to a proxy when browsing HTTPS pages by accident. We should add an option like "allow_insecure_proxy" (false by default) or similar, so it is harder to leak HTTPS traffic accidentally. In any case, support for https_proxy should be added to Dillo first. Best, Rodrigo.

June 21, 2024

Re: problem with compressed or encrypted url

by pastebin＠gmx.com

Hi Kevin, Thanks - for spending the time figuring this out - it is interesting what you did - i never would have imagined it was 'brotli and a misconfigured server' it never came through uncompressed here - even after a few attempts - i wonder what someone looking at the logs thougth about - so many multiple loadings of a url about secureing sshd lol brotli appears to be a google invention - i wonder about how it got into a misconfigured server :) if it ever comes up for a vote - i vote no to include brotli into dillo ;) On Fri, 21 Jun 2024 09:52:06 +1000 "Kevin Koster" <dillo(a)ombertech.com> wrote: > pastebin-KK0ffGbhmjU(a)public.gmane.org wrote: > > Hi > > > > someone please tell me what is wrong when i do > > > > dillo https://ubuntu.com/tutorials/configure-ssh-2fa#2-installing-and-configuring… > > > > it seems to be encrypted or compressed? > > It failed the first time for me, then trying a little later it > loads. In between I tried using "wget -S --spider" and the first > time the header showed it was compressed using Brotli compression: > "content-encoding: br" > > On subsequent tries it returned uncompressed data to Wget and > Dillo, which is correct since neither supports Brotli compression. > It appears that the originating Web server is mis-configured to > use Brotli compression even if the client doesn't announce support > for it. > > I found another page on the Ubuntu site where the problem still > happens for me: > > ---- > $ wget --spider -S https://ubuntu.com/navigation > Spider mode enabled. Check if remote file exists. > --2024-06-21 09:41:41-- https://ubuntu.com/navigation > Resolving ubuntu.com... 185.125.190.20, 185.125.190.29, 185.125.190.21, ... > Connecting to ubuntu.com|185.125.190.20|:443... connected. > HTTP request sent, awaiting response... > HTTP/1.1 200 > server: nginx/1.14.0 (Ubuntu) > date: Thu, 20 Jun 2024 23:41:14 GMT > content-type: text/html; charset=utf-8 > x-view-name: webapp.views.navigation_nojs > x-clacks-overhead: GNU Terry Pratchett > permissions-policy: interest-cohort=() > cache-control: max-age=60, stale-while-revalidate=86400, stale-if-error=300 > x-frame-options: SAMEORIGIN > x-content-type-options: NOSNIFF > x-vcs-revision: 1718884808-1dd74d9 > x-request-id: c211f00fd2212b0b8771952d7fa04558 > strict-transport-security: max-age=15724800 > link: <https://assets.ubuntu.com>; rel=preconnect; crossorigin, <https://assets.ubuntu.com>; rel=preconnect, <https://res.cloudinary.com>; rel=preconnect > content-encoding: br > x-cache-status: STALE from content-cache-il3/0 > Length: unspecified [text/html] > Remote file exists and could contain further links, > but recursion is disabled -- not retrieving. > ---- > > After retrying a few times I got an unencrypted response: > > ---- > $ wget --spider -S https://ubuntu.com/navigation > Spider mode enabled. Check if remote file exists. > --2024-06-21 09:46:12-- https://ubuntu.com/navigation > Resolving ubuntu.com... 185.125.190.21, 185.125.190.29, 185.125.190.20, ... > Connecting to ubuntu.com|185.125.190.21|:443... connected. > HTTP request sent, awaiting response... > HTTP/1.1 200 > server: nginx/1.14.0 (Ubuntu) > date: Thu, 20 Jun 2024 23:45:45 GMT > content-type: text/html; charset=utf-8 > content-length: 202990 > x-view-name: webapp.views.navigation_nojs > x-clacks-overhead: GNU Terry Pratchett > permissions-policy: interest-cohort=() > cache-control: max-age=60, stale-while-revalidate=86400, stale-if-error=300 > x-frame-options: SAMEORIGIN > x-content-type-options: NOSNIFF > x-vcs-revision: 1718884808-1dd74d9 > x-request-id: 67baf729cf4344e46b689b28c5efb56f > strict-transport-security: max-age=15724800 > link: <https://assets.ubuntu.com>; rel=preconnect; crossorigin, <https://assets.ubuntu.com>; rel=preconnect, <https://res.cloudinary.com>; rel=preconnect > x-cache-status: HIT from content-cache-il3/1 > accept-ranges: bytes > Length: 202990 (198K) [text/html] > Remote file exists and could contain further links, > but recursion is disabled -- not retrieving. > ---- > > The difference appears to be whether the page is served from a > cache server. The cache server still latter notices the browser's > Accept-Encoding header and doesn't use Brotli compression. > > While checking docs for the headers, I found this page has the same > problem (again in both Dillo and Wget): > > ---- > $ wget -S --spider --compression=gzip https://http.dev/compression > Spider mode enabled. Check if remote file exists. > --2024-06-21 09:30:10-- https://http.dev/compression > Resolving http.dev... 34.120.39.70 > Connecting to http.dev|34.120.39.70|:443... connected. > HTTP request sent, awaiting response... > HTTP/1.1 200 OK > content-type: text/html; charset=utf-8 > strict-transport-security: max-age=63072000; includeSubDomains; preload > pragma: public > cache-control: public, max-age=86400 > x-content-type-options: nosniff > x-frame-options: SAMEORIGIN > x-ua-compatible: IE=Edge,chrome=1 > x-xss-protection: 1; mode=block > vary: Accept-Encoding > x-versionid: 8Bp0x4p9 > x-production: True > link: <https://http.dev/css/app.min.css?v=8Bp0x4p9>; rel="preload"; as="style" > x-request-id: fc33d19c-441a-4fab-ae1b-113e5e9028f5 > content-encoding: br > x-cloud-trace-context: a7d7fda8062ff6c0a9a86187afcacecb;o=1 > Content-Length: 8989 > date: Thu, 20 Jun 2024 23:29:42 GMT > server: Google Frontend > via: 1.1 google, 1.1 google > Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 > Length: 8989 (8.8K) [text/html] > Remote file exists and could contain further links, > but recursion is disabled -- not retrieving. > ---- > _______________________________________________ > Dillo-dev mailing list -- dillo-dev(a)mailman3.com > To unsubscribe send an email to dillo-dev-leave(a)mailman3.com

June 21, 2024

Re: problem with compressed or encrypted url

by Kevin Koster

pastebin-KK0ffGbhmjU(a)public.gmane.org wrote: > Hi > > someone please tell me what is wrong when i do > > dillo https://ubuntu.com/tutorials/configure-ssh-2fa#2-installing-and-configuring… > > it seems to be encrypted or compressed? It failed the first time for me, then trying a little later it loads. In between I tried using "wget -S --spider" and the first time the header showed it was compressed using Brotli compression: "content-encoding: br" On subsequent tries it returned uncompressed data to Wget and Dillo, which is correct since neither supports Brotli compression. It appears that the originating Web server is mis-configured to use Brotli compression even if the client doesn't announce support for it. I found another page on the Ubuntu site where the problem still happens for me: ---- $ wget --spider -S https://ubuntu.com/navigation Spider mode enabled. Check if remote file exists. --2024-06-21 09:41:41-- https://ubuntu.com/navigation Resolving ubuntu.com... 185.125.190.20, 185.125.190.29, 185.125.190.21, ... Connecting to ubuntu.com|185.125.190.20|:443... connected. HTTP request sent, awaiting response... HTTP/1.1 200 server: nginx/1.14.0 (Ubuntu) date: Thu, 20 Jun 2024 23:41:14 GMT content-type: text/html; charset=utf-8 x-view-name: webapp.views.navigation_nojs x-clacks-overhead: GNU Terry Pratchett permissions-policy: interest-cohort=() cache-control: max-age=60, stale-while-revalidate=86400, stale-if-error=300 x-frame-options: SAMEORIGIN x-content-type-options: NOSNIFF x-vcs-revision: 1718884808-1dd74d9 x-request-id: c211f00fd2212b0b8771952d7fa04558 strict-transport-security: max-age=15724800 link: <https://assets.ubuntu.com>; rel=preconnect; crossorigin, <https://assets.ubuntu.com>; rel=preconnect, <https://res.cloudinary.com>; rel=preconnect content-encoding: br x-cache-status: STALE from content-cache-il3/0 Length: unspecified [text/html] Remote file exists and could contain further links, but recursion is disabled -- not retrieving. ---- After retrying a few times I got an unencrypted response: ---- $ wget --spider -S https://ubuntu.com/navigation Spider mode enabled. Check if remote file exists. --2024-06-21 09:46:12-- https://ubuntu.com/navigation Resolving ubuntu.com... 185.125.190.21, 185.125.190.29, 185.125.190.20, ... Connecting to ubuntu.com|185.125.190.21|:443... connected. HTTP request sent, awaiting response... HTTP/1.1 200 server: nginx/1.14.0 (Ubuntu) date: Thu, 20 Jun 2024 23:45:45 GMT content-type: text/html; charset=utf-8 content-length: 202990 x-view-name: webapp.views.navigation_nojs x-clacks-overhead: GNU Terry Pratchett permissions-policy: interest-cohort=() cache-control: max-age=60, stale-while-revalidate=86400, stale-if-error=300 x-frame-options: SAMEORIGIN x-content-type-options: NOSNIFF x-vcs-revision: 1718884808-1dd74d9 x-request-id: 67baf729cf4344e46b689b28c5efb56f strict-transport-security: max-age=15724800 link: <https://assets.ubuntu.com>; rel=preconnect; crossorigin, <https://assets.ubuntu.com>; rel=preconnect, <https://res.cloudinary.com>; rel=preconnect x-cache-status: HIT from content-cache-il3/1 accept-ranges: bytes Length: 202990 (198K) [text/html] Remote file exists and could contain further links, but recursion is disabled -- not retrieving. ---- The difference appears to be whether the page is served from a cache server. The cache server still latter notices the browser's Accept-Encoding header and doesn't use Brotli compression. While checking docs for the headers, I found this page has the same problem (again in both Dillo and Wget): ---- $ wget -S --spider --compression=gzip https://http.dev/compression Spider mode enabled. Check if remote file exists. --2024-06-21 09:30:10-- https://http.dev/compression Resolving http.dev... 34.120.39.70 Connecting to http.dev|34.120.39.70|:443... connected. HTTP request sent, awaiting response... HTTP/1.1 200 OK content-type: text/html; charset=utf-8 strict-transport-security: max-age=63072000; includeSubDomains; preload pragma: public cache-control: public, max-age=86400 x-content-type-options: nosniff x-frame-options: SAMEORIGIN x-ua-compatible: IE=Edge,chrome=1 x-xss-protection: 1; mode=block vary: Accept-Encoding x-versionid: 8Bp0x4p9 x-production: True link: <https://http.dev/css/app.min.css?v=8Bp0x4p9>; rel="preload"; as="style" x-request-id: fc33d19c-441a-4fab-ae1b-113e5e9028f5 content-encoding: br x-cloud-trace-context: a7d7fda8062ff6c0a9a86187afcacecb;o=1 Content-Length: 8989 date: Thu, 20 Jun 2024 23:29:42 GMT server: Google Frontend via: 1.1 google, 1.1 google Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 Length: 8989 (8.8K) [text/html] Remote file exists and could contain further links, but recursion is disabled -- not retrieving. ----

June 20, 2024