June 2024 - Dillo-dev - lists.mailman3.com

Re: HTTPS Proxy Support

by Rodrigo Arias

Hi, On Wed, Jun 19, 2024 at 01:15:52PM +1000, Kevin Koster wrote: >As mentioned in the discussion about link handlers and webpage >content rewriting, I once looked into using Web proxy software to >do the latter. One complication was that Dillo doesn't support >a feature equivalent to setting the https_proxy environment >variable for GNU Wget, whereby the proxy acts as the HTTPS >endpoint for encrypted Web server connections and sends the data >back to the browser unencrypted. This allows the proxy to >manipulate page content without needing to transform HTTP requests >into HTTPS requests and rewrite https:// in page URLs as http://. This should be easy to do, as Dillo can forward the http_proxy variable to the wget command. However, if you start Dillo from a shell where the http_proxy variable is set in the environment, I would imagine it would be already picked by Dillo and the internal wget command. The dpid daemon has to start from this shell, so you should do a "dpidc stop" command before to ensure it. >Besides proxies, for old computers where performance or available >TLS library versions prevent using modern encryption, this also >allows the encrypted communications to be offloaded to more modern >computer. Or using a different encryption library that's optimised >for old computers, as described here: > >http://oldvcr.blogspot.com/2020/11/fun-with-crypto-ancienne-tls-for.html > >For using this with old computers, it would be ideal if the HTTPS >proxy feature was available even if Dillo is compiled without >linking to a TLS library. Also the https_proxy environment variable >should be set to the HTTPS proxy hostname when running Wget for >downloads so that it applies to those as well. Sounds doable, but I would have to review how the HTTPS traffic is handled in Dillo when a proxy is present. Best, Rodrigo.

June 19, 2024

Re: Option to define external link handler

by Rodrigo Arias

Hi, On Wed, Jun 19, 2024 at 10:03:20AM +1000, Kevin Koster wrote: >> Notice that the HTTP content can be compressed. So, for example, this >> simple rewrite script: >> >> #!/bin/sh >> sed 's_www.youtube.com_inv.vern.cc_g' >> >> Would only work well in the SED node *after* the HTTP content is >> uncompressed and the headers removed. The rewrite rules should indicate >> in which position of the chain they apply. > >This mechanism might suit an idea I've had to do remote downscaling >of extremely large images, which are increasingly being included in >web pages. The script would send a list of URLs in all <img> tags >within the HTML to a remote server (eg. on a VPS), or ideally just >the ones for large image files, then rewrite the URLs in the HTML >to point to the remote server where the converted images are >available over HTTP/S. You can create a script that rewrites the <img> src attribute <img src="https://foo.com/img1.png"> To point to an endpoint of your server: <img src="https://yourserver.com/downscale?url=https://foo.com/img1.png"> And then in the server you simply downscale it. Here is how you could do it with rules: # Script that would rewrite images to a server for downscaling action downscale filter 'rewrite-img.sh' define mime header 'Content-Type' match mime 'text/html' action downscale >Or a deeper approach would be to apply the same approach as this >rewrite engine to binary content as well, and have Dillo do it >transparently via 'rewrite'/convert rules for image MIME types. >Then the HTML would stay the same and Dillo would trigger a command >that requested a downscaled image from the converter server instead >of the original image's server. That would be more elegant, but >expands the scope of your proposed system a little. Rewriting the binary image directly would be possible, but then you would have wasted the bandwidth bringing it to Dillo, and now you have to send it to the server to downscale it. You probably want to use the previous approach for this case. In any case, imagine you want to downscale it locally anyway. Here is how I can think about it: # Script that would downscale an image and write to stdout action downscale filter 'downscale-img.sh' # Define headers from the HTTP content with shorter names define mime header 'Content-Type' define size header 'Content-Length' # Downscale big images match mime =~ 'image/.*' and size > 10K action downscale Notice that this can be triggered for any image, not only ones provided via HTTP/HTTPS, but also via other protocols like gemini that are adapted to speak HTTP and also provide a Content-Length header. I added the =~ and > operators, as the former would match a regex and the latter will use a numeric comparator. You can assume that the default if the header is not present is to make any comparison fail. I have also added the "define" keyword to define properties like "mime" or "size" which are parsed from the HTTP headers and are shorter and easier to write. >Maybe since it still requires a remote Web server this problem >would be better solved via a Web proxy (I did look into Squid >before, but drowned in confusing documentation). But I just thought >I'd mention it as an example of a more complex usage for this >proposed rewrite system. But then you will need to pass all the traffic through the server so it performs the substitution there. Another solution which may be better is to mark from Dillo which requests are being done from img elements (filtering them before going to the network). If Dillo marks those requests in the HTTP headers for example, then you could do: # Script that transforms image HTTP requests to a server that # downscales the image action downscale filter 'downscale-req.sh' define source header 'Dillo-Request-Source' # Downscale images comming from <img> elements match source 'img' action downscale This would have the benefit that Dillo already performs the parsing of the HTML for you, and only the images that are loaded are passed to the downscaling server. Additionally, cookies would be sent in the HTTP request, so you can access login protected images this way too. Working on these examples is very helpful to design the rule system, so feel free to mention more cases. Best, Rodrigo.

June 19, 2024

HTTPS Proxy Support

by Kevin Koster

As mentioned in the discussion about link handlers and webpage content rewriting, I once looked into using Web proxy software to do the latter. One complication was that Dillo doesn't support a feature equivalent to setting the https_proxy environment variable for GNU Wget, whereby the proxy acts as the HTTPS endpoint for encrypted Web server connections and sends the data back to the browser unencrypted. This allows the proxy to manipulate page content without needing to transform HTTP requests into HTTPS requests and rewrite https:// in page URLs as http://. Besides proxies, for old computers where performance or available TLS library versions prevent using modern encryption, this also allows the encrypted communications to be offloaded to more modern computer. Or using a different encryption library that's optimised for old computers, as described here: http://oldvcr.blogspot.com/2020/11/fun-with-crypto-ancienne-tls-for.html For using this with old computers, it would be ideal if the HTTPS proxy feature was available even if Dillo is compiled without linking to a TLS library. Also the https_proxy environment variable should be set to the HTTPS proxy hostname when running Wget for downloads so that it applies to those as well.

June 19, 2024

Re: Option to define external link handler

by Kevin Koster

Rodrigo Arias <rodarima-Re5JQEeQqe8AvxtiuMwx3w(a)public.gmane.org> wrote: > The solution with the new design would involve: > > 1) Open a "gemini:" link > 2) The request is routed to the gemini: dpi handler (like now) > 3) The gemini plugin returns the .gmi file as-is as an HTTP response, > instead of converting it to HTML > 4) The .gmi mime type matches a rewrite rule and is rewritten into HTML > in the SED node. > > Now, if we open a .gmi via HTTP: > > 1) Open a "https:" link > 2) The request is routed to the usual HTTP/IO/TLS chain > 3) The HTTP server returns the .gmi file as-is as an HTTP response. > 4) The .gmi mime type matches a rewrite rule, and is rewritten into HTML > in the SED node. > > Notice that the HTTP content can be compressed. So, for example, this > simple rewrite script: > > #!/bin/sh > sed 's_www.youtube.com_inv.vern.cc_g' > > Would only work well in the SED node *after* the HTTP content is > uncompressed and the headers removed. The rewrite rules should indicate > in which position of the chain they apply. This mechanism might suit an idea I've had to do remote downscaling of extremely large images, which are increasingly being included in web pages. The script would send a list of URLs in all <img> tags within the HTML to a remote server (eg. on a VPS), or ideally just the ones for large image files, then rewrite the URLs in the HTML to point to the remote server where the converted images are available over HTTP/S. Or a deeper approach would be to apply the same approach as this rewrite engine to binary content as well, and have Dillo do it transparently via 'rewrite'/convert rules for image MIME types. Then the HTML would stay the same and Dillo would trigger a command that requested a downscaled image from the converter server instead of the original image's server. That would be more elegant, but expands the scope of your proposed system a little. Maybe since it still requires a remote Web server this problem would be better solved via a Web proxy (I did look into Squid before, but drowned in confusing documentation). But I just thought I'd mention it as an example of a more complex usage for this proposed rewrite system.

June 19, 2024

Re: Option to define external link handler

by Rodrigo Arias

Hi Alex, On Tue, Jun 18, 2024 at 03:28:20PM +0200, a1ex(a)dismail.de wrote: >> This also allows patching the HTML of sites so you can fix them to >> work better (or at all) in Dillo. This is also done by Firefox from >> the webcompat[2] project in what they call "interventions", as >> sometimes page authors don't fix them or take a long time, so they >> patch it from the browser directly. You can open about:compat to see >> the long list of patches, here[3] is one for YouTube. > >This would be very impressive, a real step forward for Dillo in my >opinion. Do you think this is something that would be relatively >straight-forward to implement, or is it more of a long-term goal with >lots of work required to get there? Either way, sounds like there are >exciting times ahead for Dillo! Adding a mechanism to rewrite the HTML is surprisingly not super complicated, as the internal design of Dillo is centered around the CCC, the "Concomitant Control Chain", which is basically a chain of bi-directional pipes connected together to pass data around. Here is how Dillo currently receives data from the a TLS server (AFAIK). I'm only drawing the incoming direction, but the outgoing link is similar. Net +--------+ +-------+ +------+ +-------+ ---->| TLS IO |--->| IO |--->| HTTP |--->| CACHE |-... +--------+ +-------+ +------+ +-------+ src/tls.c src/IO.c src/http.c src/capi.c And adding a new rewrite module (named SED in the diagram) would require rerouting the chain to add a new element (not hard): Net +--------+ +-------+ +------+ +=====+ +-------+ ---->| TLS IO |--->| IO |--->| HTTP |---># SED #--->| CACHE |-... +--------+ +-------+ +------+ +=====+ +-------+ src/tls.c src/IO.c src/http.c | src/capi.c | +---------+ | rulesrc | | ... | +---------+ The module can then forward the content parsed from the HTTP module to the appropriate scripts defined in the rules, and then read the output and forward it to the next steps in the chain. When no rules apply, it can just forward the content to the cache as-is. Now, the interesting part is that we can place another SED module between the IO and the HTTP nodes, so we can rewrite the HTML content *and* the HTTP headers too. This would allow for example writing a plugin that matches a given mime type and on-the-fly rewrites it into an HTML file changing the Content-Type header. This is already done by the plugins, but they mix the two things together. For example we can display a .gmi file served via the "gemini:" protocol, but we cannot display a local .gmi file. Same for manual pages with the "man:" protocol, which cannot open manual pages served via "file:" or "http(s):". The solution with the new design would involve: 1) Open a "gemini:" link 2) The request is routed to the gemini: dpi handler (like now) 3) The gemini plugin returns the .gmi file as-is as an HTTP response, instead of converting it to HTML 4) The .gmi mime type matches a rewrite rule and is rewritten into HTML in the SED node. Now, if we open a .gmi via HTTP: 1) Open a "https:" link 2) The request is routed to the usual HTTP/IO/TLS chain 3) The HTTP server returns the .gmi file as-is as an HTTP response. 4) The .gmi mime type matches a rewrite rule, and is rewritten into HTML in the SED node. Notice that the HTTP content can be compressed. So, for example, this simple rewrite script: #!/bin/sh sed 's_www.youtube.com_inv.vern.cc_g' Would only work well in the SED node *after* the HTTP content is uncompressed and the headers removed. The rewrite rules should indicate in which position of the chain they apply. As a side note, keep in mind that all of these pieces work in stream mode. Each node reads a bit of data, process it and sends it to the next node of the chain, without the need to store the whole thing in memory. Same with that sed command I wrote as an example. Best, Rodrigo.

June 18, 2024

Re: Option to define external link handler

by a1ex＠dismail.de

On Mon, 17 Jun 2024 18:23:32 +0200 Rodrigo Arias <rodarima(a)gmail.com> wrote: > >> Executing a user script is fine, but I also want to be able to > >> rewrite the page and bring it back to Dillo for display, which > >> requires more cooperation. > > > >I'm curious what the use-case for this is. Sounds interesting. > > For example, you could write a filter program that parses HTML and > rewrites the links to JS hungry websites to alternative ones, in the > same way libredirect[1] works. > > [1]: https://libredirect.github.io/ > > This also allows patching the HTML of sites so you can fix them to > work better (or at all) in Dillo. This is also done by Firefox from > the webcompat[2] project in what they call "interventions", as > sometimes page authors don't fix them or take a long time, so they > patch it from the browser directly. You can open about:compat to see > the long list of patches, here[3] is one for YouTube. This would be very impressive, a real step forward for Dillo in my opinion. Do you think this is something that would be relatively straight-forward to implement, or is it more of a long-term goal with lots of work required to get there? Either way, sounds like there are exciting times ahead for Dillo! Thanks! Alex

June 18, 2024

Re: Option to define external link handler

by Rodrigo Arias

Hi, On Mon, Jun 17, 2024 at 02:57:44PM +0200, a1ex(a)dismail.de wrote: >> So there is no need to download the file completely before it is >> piped to the tools. > >I don't think mupdf will open a file piped from stdin. Neither does >nsxiv. Maybe I'm missing something. Apparently that's the case, although it should be possible to do: https://mupdf.readthedocs.io/en/1.22.0/progressive-loading.html However, Okular and Zathura seem to be able to open PDFs from the standard input as a pipe. Same with feh for images. >It is possible to determine the Content-Type without downloading the >file with something like: > >curl -XHEAD -s -w '%{content_type}' $url Yes, but it will require another GET request to download the actual file. I think we can just begin a GET request from Dillo, parse the HTTP headers, select the appropriate handler from the mime, and either pipe the content to it or write it to a file and then pass the file path. The first option would allow doing it without waiting for the whole file to download. In fact, it should be possible to not download more than what is being consumed by the handler program. One of the Dillo main objectives is to support slow download speeds (or metered) gracefully. >> Executing a user script is fine, but I also want to be able to rewrite >> the page and bring it back to Dillo for display, which requires more >> cooperation. > >I'm curious what the use-case for this is. Sounds interesting. For example, you could write a filter program that parses HTML and rewrites the links to JS hungry websites to alternative ones, in the same way libredirect[1] works. [1]: https://libredirect.github.io/ This also allows patching the HTML of sites so you can fix them to work better (or at all) in Dillo. This is also done by Firefox from the webcompat[2] project in what they call "interventions", as sometimes page authors don't fix them or take a long time, so they patch it from the browser directly. You can open about:compat to see the long list of patches, here[3] is one for YouTube. [2]: https://webcompat.com/ [3]: https://hg.mozilla.org/mozilla-central/rev/1fa7de8dec52 This already happened to Dillo with Hacker News, and there are still some minor issues not solved. The matching rules should apply those corrections only to the set of matching URLs. Best, Rodrigo.

June 17, 2024

Re: Option to define external link handler

by a1ex＠dismail.de

Hi Rodrigo, > >Just for fun, here is a version which uses curl instead of wget. This > >has the advantage that curl can output the Content-Type from the > >server, and we can use this to perform actions instead of relying on > >the file extension. Another advantage is that curl can tell us the > >output filename, so we don't have to mess around to get that > >anymore. > > These examples are very useful to test the ideas of rules and > actions. Here is how I could imagine it being written: > > action youtube label "Open YouTube in MPV" shell "mpv $url" > action audio label "Open audio in MPV" shell "mpv $url" > action video label "Open video in MPV" shell "mpv $url" > action image label "Open image in NSXIV" download shell "nsxiv > $out" action mupdf label "Open PDF in MuPDF" download shell "mupdf > $out" action default open-new-tab "$url" > > match url "outu" action youtube > match url "\.(mp3|ogg)$" action audio > match url "\.(mkv|mp4|webm)$" action video > # From here on Dillo needs to fetch the headers > match mime-type "image" action image > match mime-type "pdf" action mupdf > # Catchall for non-matched urls > match any action default This looks pretty reasonable. I guess the trick is to not make things too complicated, and yet still make it flexible enough. > Although, we may use Dillo to download the content on the fly, like > this: > > action image label "Open image in NSXIV" pipe "nsxiv -" > action mupdf label "Open PDF in MuPDF" pipe "mupdf -" > > So there is no need to download the file completely before it is piped > to the tools. I don't think mupdf will open a file piped from stdin. Neither does nsxiv. Maybe I'm missing something. It is possible to determine the Content-Type without downloading the file with something like: curl -XHEAD -s -w '%{content_type}' $url Also, it may be desirable to restrict the download filesize to a reasonable limit, which can also be determined with a similar method. > Anyway, I need to think more about it. I also need to consider how > this could be moved out of Dillo, so we don't bring more complexity > to it. > > Executing a user script is fine, but I also want to be able to rewrite > the page and bring it back to Dillo for display, which requires more > cooperation. I'm curious what the use-case for this is. Sounds interesting. > Just a small comment, you can use `exec` to avoid nesting the ifs as > it won't return: > > if echo $1 | grep outu; then > exec $youtube "$1" > fi > > Or exit: > > if echo $1 | grep outu; then > $youtube "$1" > exit $? > fi Oops! I should know better than that, but there is is.. Thanks for pointing it out. Regards, Alex

June 17, 2024

Re: Option to define external link handler

by Rodrigo Arias

Hi Alex, On Sun, Jun 16, 2024 at 06:54:41PM +0200, a1ex(a)dismail.de wrote: >> Sure, here is a very basic example which mainly just goes by the >> extension in the URL to handle some common filetypes. I suppose that >> for more complex links in which the filetype can't be parsed like >> this, you could curl the link into the 'file' command to determine >> the type: curl -s "https://example.com/filename" | file - >> But that's overkill for my needs. >> There are many ways to improve on this, but hopefully it at >> least illustrates the concept. > >Just for fun, here is a version which uses curl instead of wget. This >has the advantage that curl can output the Content-Type from the >server, and we can use this to perform actions instead of relying on >the file extension. Another advantage is that curl can tell us the >output filename, so we don't have to mess around to get that anymore. These examples are very useful to test the ideas of rules and actions. Here is how I could imagine it being written: action youtube label "Open YouTube in MPV" shell "mpv $url" action audio label "Open audio in MPV" shell "mpv $url" action video label "Open video in MPV" shell "mpv $url" action image label "Open image in NSXIV" download shell "nsxiv $out" action mupdf label "Open PDF in MuPDF" download shell "mupdf $out" action default open-new-tab "$url" match url "outu" action youtube match url "\.(mp3|ogg)$" action audio match url "\.(mkv|mp4|webm)$" action video # From here on Dillo needs to fetch the headers match mime-type "image" action image match mime-type "pdf" action mupdf # Catchall for non-matched urls match any action default Although, we may use Dillo to download the content on the fly, like this: action image label "Open image in NSXIV" pipe "nsxiv -" action mupdf label "Open PDF in MuPDF" pipe "mupdf -" So there is no need to download the file completely before it is piped to the tools. On the other hand, I can already spot a problem. When we right-click on a link to find out which actions are available, some of them require the mime type to be known, which would require a request to the server. So we cannot determine which actions will match beforehand. Maybe a better way is to define the set of menu entries first, which could also be altered by match rules: menu label "Open in default program" tag open menu label "Save in notes" tag save-notes Then we use the information of which menu item was selected *and* the clicked url to do the full handling: action youtube shell "mpv $url" action audio shell "mpv $url" action video shell "mpv $url" action image download shell "nsxiv $out" action mupdf download shell "mupdf $out" action save-notes shell "echo $url >> ~/.dillo/notes.txt && echo Saved" action default open-new-tab "$url" match tag open { match url "outu" action youtube match url "\.(mp3|ogg)$" action audio match url "\.(mkv|mp4|webm)$" action video # From here on Dillo needs to fetch the headers match mime-type "image" action image match mime-type "pdf" action mupdf } match tag save-notes action save-notes match any action default So when the user clicks "Save in notes" it doesn't go through the normal "open with" rules, but only matches the save-notes rule. Anyway, I need to think more about it. I also need to consider how this could be moved out of Dillo, so we don't bring more complexity to it. Executing a user script is fine, but I also want to be able to rewrite the page and bring it back to Dillo for display, which requires more cooperation. ># temporary file location >tmp_dir="/tmp/dillo" > ># youtube. checks for "outu" string in URL >if echo $1 | grep outu >then $youtube "$1" >else Just a small comment, you can use `exec` to avoid nesting the ifs as it won't return: if echo $1 | grep outu; then exec $youtube "$1" fi Or exit: if echo $1 | grep outu; then $youtube "$1" exit $? fi Best, Rodrigo.

June 16, 2024

Re: Option to define external link handler

by a1ex＠dismail.de

On Sun, 16 Jun 2024 13:07:38 +0200 <a1ex(a)dismail.de> wrote: > On Sat, 15 Jun 2024 22:16:12 +0200 > Rodrigo Arias <rodarima(a)gmail.com> wrote: > > > Thanks! I'm interested in how you are using a script to match > > several URLs and do some actions automatically, not sure if you > > would be interesting in sharing it (or some parts). > > > Sure, here is a very basic example which mainly just goes by the > extension in the URL to handle some common filetypes. I suppose that > for more complex links in which the filetype can't be parsed like > this, you could curl the link into the 'file' command to determine > the type: curl -s "https://example.com/filename" | file - > But that's overkill for my needs. > There are many ways to improve on this, but hopefully it at > least illustrates the concept. Just for fun, here is a version which uses curl instead of wget. This has the advantage that curl can output the Content-Type from the server, and we can use this to perform actions instead of relying on the file extension. Another advantage is that curl can tell us the output filename, so we don't have to mess around to get that anymore. --------------------------------------------------------------------- #!/bin/sh # Example Dillo URL action handler script using curl # handlers youtube="mpv" audio="mpv" video="mpv" image="nsxiv" pdf="mupdf" # temporary file location tmp_dir="/tmp/dillo" # youtube. checks for "outu" string in URL if echo $1 | grep outu then $youtube "$1" else # audio. uses file extension from URL if echo $1 | tail -c -5 | grep -e mp3 -e ogg then $audio "$1" else # video. uses file extension from URL if echo $1 | tail -c -5 | grep -e mkv -e mp4 -e webm then $video "$1" else # use curl to download the file, and save Content-Type and filename if [ ! -d $tmp_dir ] ; then mkdir $tmp_dir ; fi curl -s --write-out "%{content_type} \n%{filename_effective}" \ --remote-name --output-dir "$tmp_dir" "$1" > $tmp_dir/file_info set the filename based on curl output to file_info filename=$(tail -1 $tmp_dir/file_info) # images. checks content type from file_info if head -1 $tmp_dir/file_info | grep image ; then $image "$filename" fi # pdf. checks content type from file_info if head -1 $tmp_dir/file_info | grep pdf ; then $pdf "$filename" fi fi fi fi -------------------------------------------------------------------- Regards, Alex

June 16, 2024