- Dillo-dev - lists.mailman3.com

Re: HTTPS Proxy Support

by Kevin Koster

On Thu, 20 Jun 2024 10:03:59 +1000 "Kevin Koster" <dillo(a)ombertech.com> wrote: > Rodrigo Arias <rodarima-Re5JQEeQqe8AvxtiuMwx3w(a)public.gmane.org> > wrote: > > However, if you start Dillo from a shell where the http_proxy > > variable is set in the environment, I would imagine it would be > > already picked by Dillo and the internal wget command. The dpid > > daemon has to start from this shell, so you should do a "dpidc stop" > > command before to ensure it. > > I'm not sure if we're on the same page, Wget supports http_proxy > and https_proxy environment variables. The former may or may not > tunnel HTTPS requests through a Web proxy, but that's not what I > want since then the proxy can't manipulate the data and the client > needs to handle the encryption. https_proxy with Wget uses the > proxy to do all the encryption and therefore those issues don't > apply. More browsers support the HTTPS tunneling proxy feature, > but that doesn't achieve anything that I'm interested in > personally. I did some tests and yes the https_proxy environment variable is picked up by Wget for the downloads DPI. It doesn't affect Dillo itself. The http_proxy environment variable is picked up by Dillo, but as I say that doesn't do what I want with Dillo or Wget. It appears that GNU Wget's behaviour has changed. Newer versions _do_ try to tunnel the HTTPS connection through the proxy set to https_proxy (not what I want), whereas older versions ask that proxy to establish the HTTPS connection and send data back unencrypted (which is what I want). So I picked a bad example with Wget. See that blog post for other examples like Lynx, but I haven't tried them myself. I can't find this change in the Wget changelog. One version that works as I intend it to is GNU Wget 1.9, circa 2003. Looking through the Wget mailing list archives it seems like many people either considered the old behaviour a bug or a useless feature, so I may be swimming against the tide. My content-modifying proxy application might be very niche, but I still think it would be more widely useful for people running Dillo on very old computers. Let me know whether it's something that might be accepted for Dillo even if I decide to try implementing it myself. Sorry for the confusion.

June 20, 2024

Re: HTTPS Proxy Support

by Kevin Koster

Rodrigo Arias <rodarima-Re5JQEeQqe8AvxtiuMwx3w(a)public.gmane.org> wrote: > On Wed, Jun 19, 2024 at 01:15:52PM +1000, Kevin Koster wrote: >>As mentioned in the discussion about link handlers and webpage >>content rewriting, I once looked into using Web proxy software to >>do the latter. One complication was that Dillo doesn't support >>a feature equivalent to setting the https_proxy environment >>variable for GNU Wget, whereby the proxy acts as the HTTPS >>endpoint for encrypted Web server connections and sends the data >>back to the browser unencrypted. This allows the proxy to >>manipulate page content without needing to transform HTTP requests >>into HTTPS requests and rewrite https:// in page URLs as http://. > > This should be easy to do, as Dillo can forward the http_proxy variable > to the wget command. > > However, if you start Dillo from a shell where the http_proxy variable > is set in the environment, I would imagine it would be already picked by > Dillo and the internal wget command. The dpid daemon has to start from > this shell, so you should do a "dpidc stop" command before to ensure it. I'm not sure if we're on the same page, Wget supports http_proxy and https_proxy environment variables. The former may or may not tunnel HTTPS requests through a Web proxy, but that's not what I want since then the proxy can't manipulate the data and the client needs to handle the encryption. https_proxy with Wget uses the proxy to do all the encryption and therefore those issues don't apply. More browsers support the HTTPS tunneling proxy feature, but that doesn't achieve anything that I'm interested in personally. Anyway Wget https_proxy is only a common example of the feature (which is described in more detail at the blog link), I'm not so much interested in using it when Dillo launches Wget for downloads, but for data retrieved by Dillo itself to display within the browser. >>Besides proxies, for old computers where performance or available >>TLS library versions prevent using modern encryption, this also >>allows the encrypted communications to be offloaded to more modern >>computer. Or using a different encryption library that's optimised >>for old computers, as described here: >> >>http://oldvcr.blogspot.com/2020/11/fun-with-crypto-ancienne-tls-for.html >> >>For using this with old computers, it would be ideal if the HTTPS >>proxy feature was available even if Dillo is compiled without >>linking to a TLS library. Also the https_proxy environment variable >>should be set to the HTTPS proxy hostname when running Wget for >>downloads so that it applies to those as well. > > Sounds doable, but I would have to review how the HTTPS traffic is > handled in Dillo when a proxy is present. I did look into this with Dillo 3.0.5 and I'm confident this HTTPS proxy feature wasn't supported already there, but I never worked out exactly where the feature would fit into that code (which is likely quite different now anyway since moving out of the DPI environment).

June 20, 2024

Re: Option to define external link handler

by Kevin Koster

Rodrigo Arias <rodarima-Re5JQEeQqe8AvxtiuMwx3w(a)public.gmane.org> wrote: > On Wed, Jun 19, 2024 at 10:03:20AM +1000, Kevin Koster wrote: >>This mechanism might suit an idea I've had to do remote downscaling >>of extremely large images, which are increasingly being included in >>web pages. The script would send a list of URLs in all <img> tags >>within the HTML to a remote server (eg. on a VPS), or ideally just >>the ones for large image files, then rewrite the URLs in the HTML >>to point to the remote server where the converted images are >>available over HTTP/S. > > You can create a script that rewrites the <img> src attribute > > <img src="https://foo.com/img1.png"> > > To point to an endpoint of your server: > > <img src="https://yourserver.com/downscale?url=https://foo.com/img1.png"> Yes that would work too, however the idea of sending a URL list was that it allows my server to re-use the HTTPS connection to the image host server while downloading lots of images for a webpage, whereas the way you have it each image request by my server would be a new HTTPS connection, and therefore potentially slower. It's a minor difference though. > And then in the server you simply downscale it. Here is how you could do > it with rules: > > # Script that would rewrite images to a server for downscaling > action downscale filter 'rewrite-img.sh' > define mime header 'Content-Type' > match mime 'text/html' action downscale That should work. I would personally find it clearer to read if there was a character indicating assignment values, since this would make it obvious which words are commands and which are arguments. Such as: # Script that would rewrite images to a server for downscaling action downscale filter='rewrite-img.sh' define mime header='Content-Type' match mime='text/html' action=downscale Using "action" as an argument to "match" as well as the name of a command might still be a little confusing. >>Or a deeper approach would be to apply the same approach as this >>rewrite engine to binary content as well, and have Dillo do it >>transparently via 'rewrite'/convert rules for image MIME types. >>Then the HTML would stay the same and Dillo would trigger a command >>that requested a downscaled image from the converter server instead >>of the original image's server. That would be more elegant, but >>expands the scope of your proposed system a little. > > Rewriting the binary image directly would be possible, but then you > would have wasted the bandwidth bringing it to Dillo, and now you have > to send it to the server to downscale it. No the idea is to reduce the bandwidth usage on Dillo's connection, so for this approach Dillo would have to abort the connection to the image server if the image size was over the limit and fetch it from the script instead, which might do: wget -q -O - "https://yourserver.com/downscale?url=https://foo.com/img1.png" A way to get Dillo to take the replacement URL from the script would be better, but I suspect that would make this system more complicated to implement because then state matters between different connections. Granted I've lost the ability to re-use the HTTPS connection at my server with this approach. I like how it wouldn't affect the performance of fetching small images by waiting for them to be needlessly downloaded by my server first though, which would generally be a bigger advantage overall. Most images are still small enough that I wouldn't down-scale them, just the 1MB+ ones would get that treatment. The surprise 10MB+ ones are the real evil, and just the option to block these outright (abort the connection and don't run any script) would be better than nothing. To complicate things further, it would be good to have a right-click menu option to bypass this rule and allow fetching the full-size image. Alternatively an external handler could be assigned (via the earlier-discussed mechanism) to a script that downloads and opens the image URL in an external image viewer, which might actually be better to use for that. > In any case, imagine you want to downscale it locally anyway. Here is > how I can think about it: > > # Script that would downscale an image and write to stdout > action downscale filter 'downscale-img.sh' > > # Define headers from the HTTP content with shorter names > define mime header 'Content-Type' > define size header 'Content-Length' > > # Downscale big images > match mime =~ 'image/.*' and size > 10K action downscale > > Notice that this can be triggered for any image, not only ones provided > via HTTP/HTTPS, but also via other protocols like gemini that are > adapted to speak HTTP and also provide a Content-Length header. > > I added the =~ and > operators, as the former would match a regex and the > latter will use a numeric comparator. You can assume that the default if > the header is not present is to make any comparison fail. > > I have also added the "define" keyword to define properties like "mime" or > "size" which are parsed from the HTTP headers and are shorter and easier > to write. That looks good. Maybe I'd find it clearer without the whitespace between operators as with my '=' example above. >>Maybe since it still requires a remote Web server this problem >>would be better solved via a Web proxy (I did look into Squid >>before, but drowned in confusing documentation). But I just thought >>I'd mention it as an example of a more complex usage for this >>proposed rewrite system. > > But then you will need to pass all the traffic through the server so it > performs the substitution there. > > Another solution which may be better is to mark from Dillo which > requests are being done from img elements (filtering them before going > to the network). > > If Dillo marks those requests in the HTTP headers for example, then you > could do: > > # Script that transforms image HTTP requests to a server that > # downscales the image > action downscale filter 'downscale-req.sh' > > define source header 'Dillo-Request-Source' > > # Downscale images comming from <img> elements > match source 'img' action downscale Yes, that's neat. > This would have the benefit that Dillo already performs the parsing of > the HTML for you, and only the images that are loaded are passed to the > downscaling server. Additionally, cookies would be sent in the HTTP > request, so you can access login protected images this way too. Good point, although for my usage login protected images wouldn't be much of a concern.

June 19, 2024

Re: HTTPS Proxy Support

by Rodrigo Arias

Hi, On Wed, Jun 19, 2024 at 01:15:52PM +1000, Kevin Koster wrote: >As mentioned in the discussion about link handlers and webpage >content rewriting, I once looked into using Web proxy software to >do the latter. One complication was that Dillo doesn't support >a feature equivalent to setting the https_proxy environment >variable for GNU Wget, whereby the proxy acts as the HTTPS >endpoint for encrypted Web server connections and sends the data >back to the browser unencrypted. This allows the proxy to >manipulate page content without needing to transform HTTP requests >into HTTPS requests and rewrite https:// in page URLs as http://. This should be easy to do, as Dillo can forward the http_proxy variable to the wget command. However, if you start Dillo from a shell where the http_proxy variable is set in the environment, I would imagine it would be already picked by Dillo and the internal wget command. The dpid daemon has to start from this shell, so you should do a "dpidc stop" command before to ensure it. >Besides proxies, for old computers where performance or available >TLS library versions prevent using modern encryption, this also >allows the encrypted communications to be offloaded to more modern >computer. Or using a different encryption library that's optimised >for old computers, as described here: > >http://oldvcr.blogspot.com/2020/11/fun-with-crypto-ancienne-tls-for.html > >For using this with old computers, it would be ideal if the HTTPS >proxy feature was available even if Dillo is compiled without >linking to a TLS library. Also the https_proxy environment variable >should be set to the HTTPS proxy hostname when running Wget for >downloads so that it applies to those as well. Sounds doable, but I would have to review how the HTTPS traffic is handled in Dillo when a proxy is present. Best, Rodrigo.

June 19, 2024

Re: Option to define external link handler

by Rodrigo Arias

Hi, On Wed, Jun 19, 2024 at 10:03:20AM +1000, Kevin Koster wrote: >> Notice that the HTTP content can be compressed. So, for example, this >> simple rewrite script: >> >> #!/bin/sh >> sed 's_www.youtube.com_inv.vern.cc_g' >> >> Would only work well in the SED node *after* the HTTP content is >> uncompressed and the headers removed. The rewrite rules should indicate >> in which position of the chain they apply. > >This mechanism might suit an idea I've had to do remote downscaling >of extremely large images, which are increasingly being included in >web pages. The script would send a list of URLs in all <img> tags >within the HTML to a remote server (eg. on a VPS), or ideally just >the ones for large image files, then rewrite the URLs in the HTML >to point to the remote server where the converted images are >available over HTTP/S. You can create a script that rewrites the <img> src attribute <img src="https://foo.com/img1.png"> To point to an endpoint of your server: <img src="https://yourserver.com/downscale?url=https://foo.com/img1.png"> And then in the server you simply downscale it. Here is how you could do it with rules: # Script that would rewrite images to a server for downscaling action downscale filter 'rewrite-img.sh' define mime header 'Content-Type' match mime 'text/html' action downscale >Or a deeper approach would be to apply the same approach as this >rewrite engine to binary content as well, and have Dillo do it >transparently via 'rewrite'/convert rules for image MIME types. >Then the HTML would stay the same and Dillo would trigger a command >that requested a downscaled image from the converter server instead >of the original image's server. That would be more elegant, but >expands the scope of your proposed system a little. Rewriting the binary image directly would be possible, but then you would have wasted the bandwidth bringing it to Dillo, and now you have to send it to the server to downscale it. You probably want to use the previous approach for this case. In any case, imagine you want to downscale it locally anyway. Here is how I can think about it: # Script that would downscale an image and write to stdout action downscale filter 'downscale-img.sh' # Define headers from the HTTP content with shorter names define mime header 'Content-Type' define size header 'Content-Length' # Downscale big images match mime =~ 'image/.*' and size > 10K action downscale Notice that this can be triggered for any image, not only ones provided via HTTP/HTTPS, but also via other protocols like gemini that are adapted to speak HTTP and also provide a Content-Length header. I added the =~ and > operators, as the former would match a regex and the latter will use a numeric comparator. You can assume that the default if the header is not present is to make any comparison fail. I have also added the "define" keyword to define properties like "mime" or "size" which are parsed from the HTTP headers and are shorter and easier to write. >Maybe since it still requires a remote Web server this problem >would be better solved via a Web proxy (I did look into Squid >before, but drowned in confusing documentation). But I just thought >I'd mention it as an example of a more complex usage for this >proposed rewrite system. But then you will need to pass all the traffic through the server so it performs the substitution there. Another solution which may be better is to mark from Dillo which requests are being done from img elements (filtering them before going to the network). If Dillo marks those requests in the HTTP headers for example, then you could do: # Script that transforms image HTTP requests to a server that # downscales the image action downscale filter 'downscale-req.sh' define source header 'Dillo-Request-Source' # Downscale images comming from <img> elements match source 'img' action downscale This would have the benefit that Dillo already performs the parsing of the HTML for you, and only the images that are loaded are passed to the downscaling server. Additionally, cookies would be sent in the HTTP request, so you can access login protected images this way too. Working on these examples is very helpful to design the rule system, so feel free to mention more cases. Best, Rodrigo.

June 19, 2024

HTTPS Proxy Support

by Kevin Koster

As mentioned in the discussion about link handlers and webpage content rewriting, I once looked into using Web proxy software to do the latter. One complication was that Dillo doesn't support a feature equivalent to setting the https_proxy environment variable for GNU Wget, whereby the proxy acts as the HTTPS endpoint for encrypted Web server connections and sends the data back to the browser unencrypted. This allows the proxy to manipulate page content without needing to transform HTTP requests into HTTPS requests and rewrite https:// in page URLs as http://. Besides proxies, for old computers where performance or available TLS library versions prevent using modern encryption, this also allows the encrypted communications to be offloaded to more modern computer. Or using a different encryption library that's optimised for old computers, as described here: http://oldvcr.blogspot.com/2020/11/fun-with-crypto-ancienne-tls-for.html For using this with old computers, it would be ideal if the HTTPS proxy feature was available even if Dillo is compiled without linking to a TLS library. Also the https_proxy environment variable should be set to the HTTPS proxy hostname when running Wget for downloads so that it applies to those as well.

June 19, 2024

Re: Option to define external link handler

by Kevin Koster

Rodrigo Arias <rodarima-Re5JQEeQqe8AvxtiuMwx3w(a)public.gmane.org> wrote: > The solution with the new design would involve: > > 1) Open a "gemini:" link > 2) The request is routed to the gemini: dpi handler (like now) > 3) The gemini plugin returns the .gmi file as-is as an HTTP response, > instead of converting it to HTML > 4) The .gmi mime type matches a rewrite rule and is rewritten into HTML > in the SED node. > > Now, if we open a .gmi via HTTP: > > 1) Open a "https:" link > 2) The request is routed to the usual HTTP/IO/TLS chain > 3) The HTTP server returns the .gmi file as-is as an HTTP response. > 4) The .gmi mime type matches a rewrite rule, and is rewritten into HTML > in the SED node. > > Notice that the HTTP content can be compressed. So, for example, this > simple rewrite script: > > #!/bin/sh > sed 's_www.youtube.com_inv.vern.cc_g' > > Would only work well in the SED node *after* the HTTP content is > uncompressed and the headers removed. The rewrite rules should indicate > in which position of the chain they apply. This mechanism might suit an idea I've had to do remote downscaling of extremely large images, which are increasingly being included in web pages. The script would send a list of URLs in all <img> tags within the HTML to a remote server (eg. on a VPS), or ideally just the ones for large image files, then rewrite the URLs in the HTML to point to the remote server where the converted images are available over HTTP/S. Or a deeper approach would be to apply the same approach as this rewrite engine to binary content as well, and have Dillo do it transparently via 'rewrite'/convert rules for image MIME types. Then the HTML would stay the same and Dillo would trigger a command that requested a downscaled image from the converter server instead of the original image's server. That would be more elegant, but expands the scope of your proposed system a little. Maybe since it still requires a remote Web server this problem would be better solved via a Web proxy (I did look into Squid before, but drowned in confusing documentation). But I just thought I'd mention it as an example of a more complex usage for this proposed rewrite system.

June 19, 2024

Re: Option to define external link handler

by Rodrigo Arias

Hi Alex, On Tue, Jun 18, 2024 at 03:28:20PM +0200, a1ex(a)dismail.de wrote: >> This also allows patching the HTML of sites so you can fix them to >> work better (or at all) in Dillo. This is also done by Firefox from >> the webcompat[2] project in what they call "interventions", as >> sometimes page authors don't fix them or take a long time, so they >> patch it from the browser directly. You can open about:compat to see >> the long list of patches, here[3] is one for YouTube. > >This would be very impressive, a real step forward for Dillo in my >opinion. Do you think this is something that would be relatively >straight-forward to implement, or is it more of a long-term goal with >lots of work required to get there? Either way, sounds like there are >exciting times ahead for Dillo! Adding a mechanism to rewrite the HTML is surprisingly not super complicated, as the internal design of Dillo is centered around the CCC, the "Concomitant Control Chain", which is basically a chain of bi-directional pipes connected together to pass data around. Here is how Dillo currently receives data from the a TLS server (AFAIK). I'm only drawing the incoming direction, but the outgoing link is similar. Net +--------+ +-------+ +------+ +-------+ ---->| TLS IO |--->| IO |--->| HTTP |--->| CACHE |-... +--------+ +-------+ +------+ +-------+ src/tls.c src/IO.c src/http.c src/capi.c And adding a new rewrite module (named SED in the diagram) would require rerouting the chain to add a new element (not hard): Net +--------+ +-------+ +------+ +=====+ +-------+ ---->| TLS IO |--->| IO |--->| HTTP |---># SED #--->| CACHE |-... +--------+ +-------+ +------+ +=====+ +-------+ src/tls.c src/IO.c src/http.c | src/capi.c | +---------+ | rulesrc | | ... | +---------+ The module can then forward the content parsed from the HTTP module to the appropriate scripts defined in the rules, and then read the output and forward it to the next steps in the chain. When no rules apply, it can just forward the content to the cache as-is. Now, the interesting part is that we can place another SED module between the IO and the HTTP nodes, so we can rewrite the HTML content *and* the HTTP headers too. This would allow for example writing a plugin that matches a given mime type and on-the-fly rewrites it into an HTML file changing the Content-Type header. This is already done by the plugins, but they mix the two things together. For example we can display a .gmi file served via the "gemini:" protocol, but we cannot display a local .gmi file. Same for manual pages with the "man:" protocol, which cannot open manual pages served via "file:" or "http(s):". The solution with the new design would involve: 1) Open a "gemini:" link 2) The request is routed to the gemini: dpi handler (like now) 3) The gemini plugin returns the .gmi file as-is as an HTTP response, instead of converting it to HTML 4) The .gmi mime type matches a rewrite rule and is rewritten into HTML in the SED node. Now, if we open a .gmi via HTTP: 1) Open a "https:" link 2) The request is routed to the usual HTTP/IO/TLS chain 3) The HTTP server returns the .gmi file as-is as an HTTP response. 4) The .gmi mime type matches a rewrite rule, and is rewritten into HTML in the SED node. Notice that the HTTP content can be compressed. So, for example, this simple rewrite script: #!/bin/sh sed 's_www.youtube.com_inv.vern.cc_g' Would only work well in the SED node *after* the HTTP content is uncompressed and the headers removed. The rewrite rules should indicate in which position of the chain they apply. As a side note, keep in mind that all of these pieces work in stream mode. Each node reads a bit of data, process it and sends it to the next node of the chain, without the need to store the whole thing in memory. Same with that sed command I wrote as an example. Best, Rodrigo.

June 18, 2024

Re: Option to define external link handler

by a1ex＠dismail.de

On Mon, 17 Jun 2024 18:23:32 +0200 Rodrigo Arias <rodarima(a)gmail.com> wrote: > >> Executing a user script is fine, but I also want to be able to > >> rewrite the page and bring it back to Dillo for display, which > >> requires more cooperation. > > > >I'm curious what the use-case for this is. Sounds interesting. > > For example, you could write a filter program that parses HTML and > rewrites the links to JS hungry websites to alternative ones, in the > same way libredirect[1] works. > > [1]: https://libredirect.github.io/ > > This also allows patching the HTML of sites so you can fix them to > work better (or at all) in Dillo. This is also done by Firefox from > the webcompat[2] project in what they call "interventions", as > sometimes page authors don't fix them or take a long time, so they > patch it from the browser directly. You can open about:compat to see > the long list of patches, here[3] is one for YouTube. This would be very impressive, a real step forward for Dillo in my opinion. Do you think this is something that would be relatively straight-forward to implement, or is it more of a long-term goal with lots of work required to get there? Either way, sounds like there are exciting times ahead for Dillo! Thanks! Alex

June 18, 2024

Re: Option to define external link handler

by Rodrigo Arias

Hi, On Mon, Jun 17, 2024 at 02:57:44PM +0200, a1ex(a)dismail.de wrote: >> So there is no need to download the file completely before it is >> piped to the tools. > >I don't think mupdf will open a file piped from stdin. Neither does >nsxiv. Maybe I'm missing something. Apparently that's the case, although it should be possible to do: https://mupdf.readthedocs.io/en/1.22.0/progressive-loading.html However, Okular and Zathura seem to be able to open PDFs from the standard input as a pipe. Same with feh for images. >It is possible to determine the Content-Type without downloading the >file with something like: > >curl -XHEAD -s -w '%{content_type}' $url Yes, but it will require another GET request to download the actual file. I think we can just begin a GET request from Dillo, parse the HTTP headers, select the appropriate handler from the mime, and either pipe the content to it or write it to a file and then pass the file path. The first option would allow doing it without waiting for the whole file to download. In fact, it should be possible to not download more than what is being consumed by the handler program. One of the Dillo main objectives is to support slow download speeds (or metered) gracefully. >> Executing a user script is fine, but I also want to be able to rewrite >> the page and bring it back to Dillo for display, which requires more >> cooperation. > >I'm curious what the use-case for this is. Sounds interesting. For example, you could write a filter program that parses HTML and rewrites the links to JS hungry websites to alternative ones, in the same way libredirect[1] works. [1]: https://libredirect.github.io/ This also allows patching the HTML of sites so you can fix them to work better (or at all) in Dillo. This is also done by Firefox from the webcompat[2] project in what they call "interventions", as sometimes page authors don't fix them or take a long time, so they patch it from the browser directly. You can open about:compat to see the long list of patches, here[3] is one for YouTube. [2]: https://webcompat.com/ [3]: https://hg.mozilla.org/mozilla-central/rev/1fa7de8dec52 This already happened to Dillo with Hacker News, and there are still some minor issues not solved. The matching rules should apply those corrections only to the set of matching URLs. Best, Rodrigo.

June 17, 2024