- Dillo-dev - lists.mailman3.com

Re: Issues with HTTP multipart/form-data file upload

by Xavier Del Campo Romero

Hi Rodrigo, > Dillo has a mechanism to read chunks of data from different sources as they are arriving and pass them to the next stage for processing. However, AFAIK it always reads a chunk and appends it to a large buffer. It doesn't free the processed part until is done with the whole thing. > > This would require a change in the way Dillo processes data, but I think it would be required for large files. There are more details in the > devdoc/CCCwork.txt file and in src/chain.c if you want to take a closer look. > > As I'm planning to change the design of the CCC, I think I can take this into account too so it would be doable. I'll add it to the list of shortcomings of the current design. Thank you. I am still unfamiliar with that part of Dillo, so please let me know about any progress. > Okay, I'll focus on the boundary patch first, which is the easiest to merge and then I'll take a closer look at the others. > > Yeah, I would assume a lot of implementations are broken, so we want to try to minimize the chances we run into problems. Limiting ourselves to a-z, A-Z and 0-9 would still account for 62 out of the 75 possible characters, so roughly 82% of the set. I think that removing the quoting in favour of the limited set reduce the risk for broken implementations, yet still provide a good amount of randomness. > Check sizeof " ": https://godbolt.org/z/7Tso8ooYz Interestingly, the " " character on your last email is not really a <space> (<U0020>): $ printf "%s" " " | hd 00000000 e2 80 88 |...| 00000003 Compared to an ASCII whitespace: $ printf "%s" " " | hd 00000000 20 | | 00000001 Both Godbolt and my editor also flag that multi-byte character with a yellow rectangle around it because it would be highly confusing otherwise. For example: printf("len=%zu\n", strlen(" ")); Confusingly returns "len=3". I am not sure whether this was an intentional modification from your side. My patch is adding a <space> as defined by POSIX.1-2017 [1], so that sizeof " " would always return 2. Was it your intention to flag this potential confusion? Also, there was not strict reason to use sizeof " ". Any other character would do e.g.: sizeof "x", sizeof "A", etc. > You can also use dStr_append_c() to only append one character, so you only need a single character. That would be an unnecessary use of the heap, because the size is static. > If we only use alphanumeric characters, we can just use isalnum() right? According to POSIX.1-2017 [2], isalnum(3) depends on the current locale configured by the system. For example, characters such as Ä or ú could return non-zero. To avoid this, there are two possible solutions: 1. Use isalnum_l(3) to specify a locale_t object corresponding to the "POSIX" locale (equivalent to "C" [3]), which must be previously allocated by the newlocale(3) function [3] and released by the freelocal(3) function [4]. A minimalist example is shown below: locale_t l = newlocale(LC_CTYPE, "POSIX", NULL); for (unsigned char i = 0; i < 255; i++) printf("hhu=%hhu, c=%c, isalnum=%d\n", i, i, isalnum_l(i, l)); freelocale(l); 2. Define a known subset from the portable character set defined by POSIX.1-2017 [5] and use strspn(3), as already suggested by the patch. IMHO this approach is better because: - It does not deal with locales, so developers not familiar with them would understand the code better. - It is also portable outside a POSIX environment (not sure if this a requirement, though). - It does not require dynamic allication via newlocale(3). - It is the only possible option if non-alnum characters, such as ':' or '/', are appended to the boundary string. [1]: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap06.html [2]: https://pubs.opengroup.org/onlinepubs/9699919799/functions/isalnum.html [3]: https://pubs.opengroup.org/onlinepubs/9699919799/functions/newlocale.html [4]: https://pubs.opengroup.org/onlinepubs/9699919799/functions/freelocale.html [5]: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap06.html > I meant when is the next KoVoꓘ concert :-) No gigs ahead, but I will keep you informed. :) Best regards, Xavi On 28/8/24 22:47, Rodrigo Arias wrote: > Hi Xavier, > > On Wed, Aug 28, 2024 at 01:04:04AM +0200, Xavier Del Campo Romero wrote: >> Hi Rodrigo, >> >>> Glad to read that you also consider Dillo for slcl, and thanks for >>> preparing the patches :-) >> >> Thank you! I want slcl to be useful to anyone, including users who care >> about minimalist software like Dillo. The web is already too crowded >> with bloated "webapps" and other terrible things. :) > > Agreed! > >>> Sounds good, not sure how complicated it would be to do this. >> >> I still need to investigate this further, but I assume this would >> require Dillo to at least implement a sink callback. >> >> In other words, the component responsible for transmitting the data >> (probably src/IO/IO.c) should trigger a user-defined callback with an >> arbitrarily-sized buffer (typically, of BUFSIZ bytes, as defined by >> stdio.h) that must filled with file data. Then, the user-defined >> callback can fill from zero up to BUFSIZ bytes, which are eventually >> trasmitted to the server. > > Dillo has a mechanism to read chunks of data from different sources as > they are arriving and pass them to the next stage for processing. > However, AFAIK it always reads a chunk and appends it to a large buffer. > It doesn't free the processed part until is done with the whole thing. > > This would require a change in the way Dillo processes data, but I think > it would be required for large files. There are more details in the > devdoc/CCCwork.txt file and in src/chain.c if you want to take a closer > look. > > As I'm planning to change the design of the CCC, I think I can take this > into account too so it would be doable. I'll add it to the list of > shortcomings of the current design. > >> That said, I am still not sure how much actual effort this would take. >> But I am glad to receive positive feedback so far - I will then continue >> to find a solution. >> >>> However, being able to upload multiple files at the same time sounds >>> reasonable, so feel free to try on your own in the meanwhile. >> >> Uploading multiple files at once seems doable - the patches I sent on my >> previous email are probably already doing most of the required work. >> Again, the trickiest task is to send data on-the-fly for each selected >> file. > > Okay, I'll focus on the boundary patch first, which is the easiest to > merge and then I'll take a closer look at the others. > >> >>> Shouldn't it be 68 then? >> >> I understand the opposite: the boundary string with the two leading >> dashes ("--") included can be up to 72 bytes long, and 74 bytes long for >> the ending boundary (which includes two more dashes after the boundary >> string). This is confirmed by reading the BNF defined by RFC 2046 (some >> bits omitted for simplicity), section 5.1.1 [1]: >> >>> boundary := 0*69<bchars> bcharsnospace >>> bchars := bcharsnospace / " " >>> bcharsnospace := DIGIT / ALPHA / "'" / "(" / ")" / >>> "+" / "_" / "," / "-" / "." / >>> "/" / ":" / "=" / "?" >>> dash-boundary := "--" boundary >>> ; boundary taken from the value of >>> ; boundary parameter of the >>> ; Content-Type field> multipart-body := >>> [preamble CRLF] >>> dash-boundary transport-padding CRLF >>> body-part *encapsulation >>> close-delimiter transport-padding >>> [CRLF epilogue] >>> delimiter := CRLF dash-boundary >>> close-delimiter := delimiter "--" > > Oh right! I see that we are already using 70 characters anyway. > >> Note: even if the specification tells receivers to handle transport >> padding, for the time being I am assuming "transport-padding" as zero >> length since composers must not generate non-zero length transport >> padding. I am still not sure where transport padding would apply, >> anyway. Probably outside web browsers? >> >>> I would leave out all the symbols to avoid quoting and only use A-Z >>> a-z and 0-9. >> >> Interestingly, Dillo would always quote boundary strings [2], even if >> only using A-Z, a-z and 0-9. In fact, this is one of the wrong >> assumptions I spotted when testing slcl against Dillo. > > Yeah, I would assume a lot of implementations are broken, so we want to > try to minimize the chances we run into problems. > > Apart from slcl we should also test this with some sites and see if they > continue to work okay. > > This will also increase the fingerprinting information to distinguish > Dillo among other browsers, but I think it is not more information that > the already leaked by the user agent. > >>> Which, if I computed it correctly, is still too small to worry about. >> >> Not only it is too small of a chance: if we really wanted to do "the >> right thing" and make Dillo absolutely sure the boundary string is not >> contained within the selected files, this would imply a noticeable >> performance impact when dealing with large files, much likely for a >> near-zero benefit. >> >> I have not inspected their source code yet (and I do not want to), but I >> understand both Gecko and Chromium are also making that assumption, >> because otherwise it would take them a lot of CPU time to upload large >> files. > > But then they would be doing such assumption with a "much larger" > probability it hits the file. > > Skipping it with 70 characters is safe for one file, but also probably > safe for all files ever uploaded with Dillo. > > Maybe curl or other small codebases are easier to read, but not really > needed. > >> >>> Why sizeof " " instead of just 2? >> >> Because, to my eyes, sizeof " " has more meaningful semantics, compared >> to a magic integer constant such as 2. However, for this simple >> scenario, I would still consider both acceptable. > > Check sizeof " ": https://godbolt.org/z/7Tso8ooYz > > You can also use dStr_append_c() to only append one character, so you > only need a single character. > > If we only use alphanumeric characters, we can just use isalnum() right? > >> I can replace it with 2 if you find the other construct unacceptable. >> >>> PS: When are you playing? >> >> Sorry, I did not understand your last sentence. Could you please give a >> bit more context? :) > > I meant when is the next KoVoꓘ concert :-) > > Best, > Rodrigo. > _______________________________________________ > Dillo-dev mailing list -- dillo-dev(a)mailman3.com > To unsubscribe send an email to dillo-dev-leave(a)mailman3.com

Aug. 30, 2024

Re: Firefox bookmarks plugin

by Diego

El mar, 27 ago 2024 a las 23:17, Rodrigo Arias (<rodarima(a)gmail.com>) escribió: > > Hi, > > On Mon, Aug 26, 2024 at 01:31:13AM +0200, Diego wrote: > >Good work and very interesting idea. > > Thanks! > > >If I can suggest a change I would add this lines to dpidrc at Makefile > >the installation > > > ># Default dillo bookmarks dpi > >#bookmarks=bookmarks/bookmarks.dpi > ># ffbm: A Dillo plugin to sync bookmarks with Firefox > >bookmarks=$(DPI_DIR)/$(BIN) > > > > > >maybe it's time to patch dpidrc to make it more descriptive > > > >I sugest this > >dpi_dir=/usr/local/lib/dillo/dpi > > > ># standard dillo distributed DPIs > >#bookmarks=bookmarks/bookmarks.dpi > >#cookies=cookies/cookies.dpi > >#downloads=downloads/downloads.dpi > >#vsource=vsource/vsource.filter.dpi > > The file dpidrc only registers the protocols (ftp: -> ftp dpi), the dpis > are taken from the dpi_dir or ~/.dillo/dpi automatically. > I know. > It doesn't require describing the available dpis in the file. Not sure > what would be the advantage of doing so. For reference. To explain to the user how to switch between a built-in DPI and another DPI for the same service. Maybe a documentation patch better? I can try to write some documentation about it Diego.

Aug. 30, 2024

Re: TLS connect error: "an EOF was observed that violates the protocol"

by a1ex＠dismail.de

Hi Rodrigo, On Wed, 28 Aug 2024 23:02:38 +0200 Rodrigo Arias <rodarima(a)gmail.com> wrote: > I think your IP may be banned from narkive. Yeah, that does sound like it could be the case. Funny, because I've never even used that site before. I have a static IP address, so that might have something to do with it. On the plus side, without this issue we wouldn't have found the crash in Dillo, so, thank you narkive for inadvertently helping Dillo :) Regards, Alex

Aug. 29, 2024

Re: TLS connect error: "an EOF was observed that violates the protocol"

by Rodrigo Arias

Hi, >I have confirmed that this patch prevents Dillo from crashing. >Thanks! Nice! >> However, it doesn't explain the EOF you are getting in narkive. Are >> you using tor or some proxy? > >No vpn/tor/proxy. I wouldn't be surprised if my network provider was >doing some weird stuff though... Maybe thats an area for further study. Look at what I just found: hop% torsocks curl -v https://narkive.com * Host narkive.com:443 was resolved. * IPv6: (none) * IPv4: 149.248.211.108 * Trying 149.248.211.108:443... * Connected to narkive.com (149.248.211.108) port 443 * ALPN: curl offers h2,http/1.1 * TLSv1.3 (OUT), TLS handshake, Client hello (1): * CAfile: /etc/ssl/certs/ca-certificates.crt * CApath: none * OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to narkive.com:443 * Closing connection curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to narkive.com:443 Same if I pass Dillo traffic via tor. I think your IP may be banned from narkive. Try in some days or reset your router if you have a dynamic IP. Or mail them if persists. Best, Rodrigo.

Aug. 28, 2024

Re: Issues with HTTP multipart/form-data file upload

by Rodrigo Arias

Hi Xavier, On Wed, Aug 28, 2024 at 01:04:04AM +0200, Xavier Del Campo Romero wrote: >Hi Rodrigo, > >> Glad to read that you also consider Dillo for slcl, and thanks for preparing the patches :-) > >Thank you! I want slcl to be useful to anyone, including users who care >about minimalist software like Dillo. The web is already too crowded >with bloated "webapps" and other terrible things. :) Agreed! >> Sounds good, not sure how complicated it would be to do this. > >I still need to investigate this further, but I assume this would >require Dillo to at least implement a sink callback. > >In other words, the component responsible for transmitting the data >(probably src/IO/IO.c) should trigger a user-defined callback with an >arbitrarily-sized buffer (typically, of BUFSIZ bytes, as defined by >stdio.h) that must filled with file data. Then, the user-defined >callback can fill from zero up to BUFSIZ bytes, which are eventually >trasmitted to the server. Dillo has a mechanism to read chunks of data from different sources as they are arriving and pass them to the next stage for processing. However, AFAIK it always reads a chunk and appends it to a large buffer. It doesn't free the processed part until is done with the whole thing. This would require a change in the way Dillo processes data, but I think it would be required for large files. There are more details in the devdoc/CCCwork.txt file and in src/chain.c if you want to take a closer look. As I'm planning to change the design of the CCC, I think I can take this into account too so it would be doable. I'll add it to the list of shortcomings of the current design. >That said, I am still not sure how much actual effort this would take. >But I am glad to receive positive feedback so far - I will then continue >to find a solution. > >> However, being able to upload multiple files at the same time sounds reasonable, so feel free to try on your own in the meanwhile. > >Uploading multiple files at once seems doable - the patches I sent on my >previous email are probably already doing most of the required work. >Again, the trickiest task is to send data on-the-fly for each selected file. Okay, I'll focus on the boundary patch first, which is the easiest to merge and then I'll take a closer look at the others. > >> Shouldn't it be 68 then? > >I understand the opposite: the boundary string with the two leading >dashes ("--") included can be up to 72 bytes long, and 74 bytes long for >the ending boundary (which includes two more dashes after the boundary >string). This is confirmed by reading the BNF defined by RFC 2046 (some >bits omitted for simplicity), section 5.1.1 [1]: > >> boundary := 0*69<bchars> bcharsnospace >> bchars := bcharsnospace / " " >> bcharsnospace := DIGIT / ALPHA / "'" / "(" / ")" / >> "+" / "_" / "," / "-" / "." / >> "/" / ":" / "=" / "?" >> dash-boundary := "--" boundary >> ; boundary taken from the value of >> ; boundary parameter of the >> ; Content-Type field> multipart-body := [preamble CRLF] >> dash-boundary transport-padding CRLF >> body-part *encapsulation >> close-delimiter transport-padding >> [CRLF epilogue] >> delimiter := CRLF dash-boundary >> close-delimiter := delimiter "--" Oh right! I see that we are already using 70 characters anyway. >Note: even if the specification tells receivers to handle transport >padding, for the time being I am assuming "transport-padding" as zero >length since composers must not generate non-zero length transport >padding. I am still not sure where transport padding would apply, >anyway. Probably outside web browsers? > >> I would leave out all the symbols to avoid quoting and only use A-Z a-z and 0-9. > >Interestingly, Dillo would always quote boundary strings [2], even if >only using A-Z, a-z and 0-9. In fact, this is one of the wrong >assumptions I spotted when testing slcl against Dillo. Yeah, I would assume a lot of implementations are broken, so we want to try to minimize the chances we run into problems. Apart from slcl we should also test this with some sites and see if they continue to work okay. This will also increase the fingerprinting information to distinguish Dillo among other browsers, but I think it is not more information that the already leaked by the user agent. >> Which, if I computed it correctly, is still too small to worry about. > >Not only it is too small of a chance: if we really wanted to do "the >right thing" and make Dillo absolutely sure the boundary string is not >contained within the selected files, this would imply a noticeable >performance impact when dealing with large files, much likely for a >near-zero benefit. > >I have not inspected their source code yet (and I do not want to), but I >understand both Gecko and Chromium are also making that assumption, >because otherwise it would take them a lot of CPU time to upload large >files. But then they would be doing such assumption with a "much larger" probability it hits the file. Skipping it with 70 characters is safe for one file, but also probably safe for all files ever uploaded with Dillo. Maybe curl or other small codebases are easier to read, but not really needed. > >> Why sizeof " " instead of just 2? > >Because, to my eyes, sizeof " " has more meaningful semantics, compared >to a magic integer constant such as 2. However, for this simple >scenario, I would still consider both acceptable. Check sizeof " ": https://godbolt.org/z/7Tso8ooYz You can also use dStr_append_c() to only append one character, so you only need a single character. If we only use alphanumeric characters, we can just use isalnum() right? >I can replace it with 2 if you find the other construct unacceptable. > >> PS: When are you playing? > >Sorry, I did not understand your last sentence. Could you please give a >bit more context? :) I meant when is the next KoVoꓘ concert :-) Best, Rodrigo.

Aug. 28, 2024

Re: TLS connect error: "an EOF was observed that violates the protocol"

by Theo Buehler

Thanks to both of you for tracking this down. I pulled the fix into the OpenBSD port for now.

Aug. 28, 2024

Re: TLS connect error: "an EOF was observed that violates the protocol"

by a1ex＠dismail.de

On Wed, 28 Aug 2024 21:40:20 +0200 Rodrigo Arias <rodarima(a)gmail.com> wrote: > Hi, > > >This is enough for me to at least fix this case on Dillo side. > > I see, I was trying to set the conn->in_connect = FALSE after conn > was freed. Try this patch: > > diff --git a/src/IO/tls_openssl.c b/src/IO/tls_openssl.c > index 3e68e928..5ad12b8c 100644 > --- a/src/IO/tls_openssl.c > +++ b/src/IO/tls_openssl.c > @@ -1186,7 +1186,10 @@ static void Tls_connect(int fd, int connkey) > if (a_Klist_get_data(conn_list, connkey)) { > conn->connecting = FALSE; > if (failed) { > + conn->in_connect = FALSE; > Tls_close_by_key(connkey); > + /* conn is freed now */ > + conn = NULL; > } > a_IOwatch_remove_fd(fd, DIO_READ|DIO_WRITE); > a_Http_connect_done(fd, failed ? FALSE : TRUE); > @@ -1195,7 +1198,8 @@ static void Tls_connect(int fd, int connkey) > } > } > > - conn->in_connect = FALSE; > + if (conn) > + conn->in_connect = FALSE; > } > > static void Tls_connect_cb(int fd, void *vconnkey) I have confirmed that this patch prevents Dillo from crashing. Thanks! > However, it doesn't explain the EOF you are getting in narkive. Are > you using tor or some proxy? No vpn/tor/proxy. I wouldn't be surprised if my network provider was doing some weird stuff though... Maybe thats an area for further study. -Alex

Aug. 28, 2024

Re: TLS connect error: "an EOF was observed that violates the protocol"

by Rodrigo Arias

Hi, >This is enough for me to at least fix this case on Dillo side. I see, I was trying to set the conn->in_connect = FALSE after conn was freed. Try this patch: diff --git a/src/IO/tls_openssl.c b/src/IO/tls_openssl.c index 3e68e928..5ad12b8c 100644 --- a/src/IO/tls_openssl.c +++ b/src/IO/tls_openssl.c @@ -1186,7 +1186,10 @@ static void Tls_connect(int fd, int connkey) if (a_Klist_get_data(conn_list, connkey)) { conn->connecting = FALSE; if (failed) { + conn->in_connect = FALSE; Tls_close_by_key(connkey); + /* conn is freed now */ + conn = NULL; } a_IOwatch_remove_fd(fd, DIO_READ|DIO_WRITE); a_Http_connect_done(fd, failed ? FALSE : TRUE); @@ -1195,7 +1198,8 @@ static void Tls_connect(int fd, int connkey) } } - conn->in_connect = FALSE; + if (conn) + conn->in_connect = FALSE; } static void Tls_connect_cb(int fd, void *vconnkey)

Aug. 28, 2024

Re: TLS connect error: "an EOF was observed that violates the protocol"

by Rodrigo Arias

Hi, On Wed, Aug 28, 2024 at 08:14:20PM +0200, a1ex(a)dismail.de wrote: >Hi, > >On Wed, 28 Aug 2024 19:12:43 +0200 >Rodrigo Arias <rodarima(a)gmail.com> wrote: > >> You may be able to get some info on where it is failing by building >> Dillo with the address sanitizer enabled: >> >> $ git clone https://github.com/dillo-browser/dillo >> $ cd dillo >> $ ./autogen.sh >> $ mkdir build >> $ cd build >> $ ../configure 'CFLAGS=-Og -g -fsanitize=address' 'CXXFLAGS=-Og -g >> -fsanitize=address' $ make >> $ src/dillo https://narkive.com > >That doesn't seem to work: > >$ ../configure 'CFLAGS=-Og -g -fsanitize=address' 'CXXFLAGS=-Og -g >-fsanitize=address' >checking build system type... x86_64-unknown-openbsd7.6 >[...] >configure: error: C compiler cannot create executables >See `config.log' for more details Oh, I didn't knew it was not supported in OpenBSD. Well, I added a simple patch to reproduce your problem, and I got a heap use after free with asan: hop% LD_LIBRARY_PATH=/usr/lib/libressl src/dillo https://narkive.com dillo_dns_init: Here we go! (threaded) TLS library: LibreSSL 3.9.2 Enabling cookies as from cookiesrc... Nav_open_url: new url='https://narkive.com' Dns_server [0]: narkive.com is 149.248.211.108 Connecting to 149.248.211.108:443 TLS connect error: "an EOF was observed that violates the protocol" Tls_close_by_key: Avoiding SSL shutdown for: https://narkive.com fd 6 is done and failed ================================================================= ==1646866==ERROR: AddressSanitizer: heap-use-after-free on address 0x5030002c8649 at pc 0x64733a019b37 bp 0x7ffd6d8bfe00 sp 0x7ffd6d8bfdf0 WRITE of size 1 at 0x5030002c8649 thread T0 #0 0x64733a019b36 in Tls_connect IO/../../../src/IO/tls_openssl.c:1201 #1 0x64733a019d88 in a_Tls_openssl_connect IO/../../../src/IO/tls_openssl.c:1266 #2 0x64733a019e9a in a_Tls_connect IO/../../../src/IO/tls.c:133 #3 0x64733a01a10d in Http_connect_tls IO/../../../src/IO/http.c:521 #4 0x64733a01b7c4 in Http_connect_socket_cb IO/../../../src/IO/http.c:555 #5 0x769620b7c1f3 in fl_wait(double) (/usr/lib/libfltk.so.1.3+0xa51f3) (BuildId: 80ecc607ef6caa03d5591ba84ff58945ff1be889) #6 0x769620b1b09a in Fl::wait(double) (/usr/lib/libfltk.so.1.3+0x4409a) (BuildId: 80ecc607ef6caa03d5591ba84ff58945ff1be889) #7 0x769620b1b15d in Fl::run() (/usr/lib/libfltk.so.1.3+0x4415d) (BuildId: 80ecc607ef6caa03d5591ba84ff58945ff1be889) #8 0x647339fd912c in main ../../src/dillo.cc:583 #9 0x769620039c87 (/usr/lib/libc.so.6+0x25c87) (BuildId: 32a656aa5562eece8c59a585f5eacd6cf5e2307b) #10 0x769620039d4b in __libc_start_main (/usr/lib/libc.so.6+0x25d4b) (BuildId: 32a656aa5562eece8c59a585f5eacd6cf5e2307b) #11 0x647339fb8894 in _start (/home/ram/dev/dillo/git/build-libressl/src/dillo+0xe1894) (BuildId: 83bcb2b57d527264c53219ed2204e01b019e3e34) 0x5030002c8649 is located 25 bytes inside of 32-byte region [0x5030002c8630,0x5030002c8650) freed by thread T0 here: #0 0x769620cfb422 in free /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:52 #1 0x64733a083ca3 in dFree ../../dlib/dlib.c:70 #2 0x64733a019782 in Tls_connect IO/../../../src/IO/tls_openssl.c:1192 #3 0x64733a019d88 in a_Tls_openssl_connect IO/../../../src/IO/tls_openssl.c:1266 #4 0x64733a019e9a in a_Tls_connect IO/../../../src/IO/tls.c:133 #5 0x64733a01a10d in Http_connect_tls IO/../../../src/IO/http.c:521 #6 0x64733a01b7c4 in Http_connect_socket_cb IO/../../../src/IO/http.c:555 #7 0x769620b7c1f3 in fl_wait(double) (/usr/lib/libfltk.so.1.3+0xa51f3) (BuildId: 80ecc607ef6caa03d5591ba84ff58945ff1be889) #8 0x769620b1b09a in Fl::wait(double) (/usr/lib/libfltk.so.1.3+0x4409a) (BuildId: 80ecc607ef6caa03d5591ba84ff58945ff1be889) #9 0x769620b1b15d in Fl::run() (/usr/lib/libfltk.so.1.3+0x4415d) (BuildId: 80ecc607ef6caa03d5591ba84ff58945ff1be889) #10 0x647339fd912c in main ../../src/dillo.cc:583 #11 0x769620039c87 (/usr/lib/libc.so.6+0x25c87) (BuildId: 32a656aa5562eece8c59a585f5eacd6cf5e2307b) #12 0x769620039d4b in __libc_start_main (/usr/lib/libc.so.6+0x25d4b) (BuildId: 32a656aa5562eece8c59a585f5eacd6cf5e2307b) #13 0x647339fb8894 in _start (/home/ram/dev/dillo/git/build-libressl/src/dillo+0xe1894) (BuildId: 83bcb2b57d527264c53219ed2204e01b019e3e34) previously allocated by thread T0 here: #0 0x769620cfca31 in malloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:69 #1 0x64733a083c3a in dMalloc ../../dlib/dlib.c:47 SUMMARY: AddressSanitizer: heap-use-after-free IO/../../../src/IO/tls_openssl.c:1201 in Tls_connect Shadow bytes around the buggy address: 0x5030002c8380: fa fa fd fd fd fa fa fa fd fd fd fd fa fa fd fd 0x5030002c8400: fd fd fa fa 00 00 00 00 fa fa fd fd fd fd fa fa 0x5030002c8480: fd fd fd fd fa fa fd fd fd fd fa fa fd fd fd fd 0x5030002c8500: fa fa 00 00 00 00 fa fa 00 00 00 00 fa fa 00 00 0x5030002c8580: 00 fa fa fa 00 00 00 00 fa fa 00 00 00 00 fa fa =>0x5030002c8600: 00 00 00 fa fa fa fd fd fd[fd]fa fa fd fd fd fa 0x5030002c8680: fa fa fd fd fd fa fa fa 00 00 00 fa fa fa 00 00 0x5030002c8700: 00 00 fa fa 00 00 00 00 fa fa 00 00 00 fa fa fa 0x5030002c8780: fd fd fd fd fa fa fd fd fd fd fa fa fd fd fd fd 0x5030002c8800: fa fa fd fd fd fd fa fa fd fd fd fd fa fa fd fd 0x5030002c8880: fd fd fa fa fd fd fd fd fa fa fd fd fd fd fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==1646866==ABORTING Hopefully is the same you are seeing. Here is the patch I'm using: diff --git a/src/IO/tls_openssl.c b/src/IO/tls_openssl.c index 3e68e928..05d795a5 100644 --- a/src/IO/tls_openssl.c +++ b/src/IO/tls_openssl.c @@ -1108,8 +1108,11 @@ static void Tls_connect(int fd, int connkey) a_IOwatch_remove_fd(fd, -1); + ret = 0; + if (ret <= 0) { int err1_ret = SSL_get_error(conn->ssl, ret); + err1_ret = SSL_ERROR_SYSCALL; if (err1_ret == SSL_ERROR_WANT_READ || err1_ret == SSL_ERROR_WANT_WRITE) { int want = err1_ret == SSL_ERROR_WANT_READ ? DIO_READ : DIO_WRITE; This is enough for me to at least fix this case on Dillo side. However, it doesn't explain the EOF you are getting in narkive. Are you using tor or some proxy? Best, Rodrigo.

Aug. 28, 2024

Re: TLS connect error: "an EOF was observed that violates the protocol"

by a1ex＠dismail.de

On Wed, 28 Aug 2024 20:22:37 +0200 Theo Buehler <tb(a)theobuehler.org> wrote: > > dillo(76315) in malloc(): write to free chunk > > 0x1957a5e82c0[24..31]@32 allocated at /usr/lib/libc++.so.10.0 > > 0x8b918 (preceding chunk 0x1957a5e82a0 allocated at > > /usr/local/lib/libcrypto.so.53.0 0x1a7f51) > > what's the output of > > addr2line -e /usr/lib/libc++.so.10.0 0x8b918 /usr/src/gnu/lib/libcxx/../../../gnu/llvm/libcxx/src/new.cpp:67 > and > > addr2line -e /usr/local/lib/libcrypto.so.53.0 0x1a7f51 stack/stack.c:128 I am also including part of a ktrace of the crash, just in case that would be useful: 57506 dillo GIO fd 1 wrote 34 bytes "Connecting to 149.248.211.108:443 " 57506 dillo RET write 34/0x22 57506 dillo CALL connect(6,0x759b98292280,16) 57506 dillo STRU struct sockaddr { AF_INET, 149.248.211.108:443 } 57506 dillo RET connect -1 errno 36 Operation now in progress 57506 dillo CALL futex(0xc4d868bc00,0x82<FUTEX_WAKE|FUTEX_PRIVATE_FLAG>,2147483647,0,0) 57506 dillo RET futex 0 57506 dillo CALL sigprocmask(SIG_BLOCK,0<>) 57506 dillo RET sigprocmask 0<> 57506 dillo CALL poll(0x759b982923c8,1,INFTIM) 57506 dillo STRU struct pollfd { fd=5, events=0x5<POLLIN|POLLOUT>, revents=0x5<POLLIN|POLLOUT> } 57506 dillo RET poll 1 57506 dillo CALL recvmsg(5,0x759b98292280,0) 57506 dillo GIO fd 5 read 32 bytes "\^N\0l\b\^N\0\M^@\^B\0\0>\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0" 57506 dillo STRU struct msghdr { name=0x0, namelen=0, iov=0x759b98292270, iovlen=1, control=0x759b98292220, controllen=0, flags=0x80<MSG_DONTWAIT> } 57506 dillo STRU struct iovec { base=0xc554d1a04c, len=4064 } 57506 dillo RET recvmsg 32/0x20 57506 dillo CALL futex(0xc4d86b3ef0,0x82<FUTEX_WAKE|FUTEX_PRIVATE_FLAG>,1,0,0) 57506 dillo RET futex 0 57506 dillo CALL writev(5,0x759b98292470,3) 57506 dillo STRU struct iovec [3] { base=0xc53d099000, len=504 } { base=0x0, len=0 } { base=0xc504b50156, len=0 } 57506 dillo GIO fd 5 wrote 504 bytes "8\^B\^E\0\^E\0\M^@\^B\^D\0\b\0\M^Y\M^Y\M^Y\0\0\0\0\0;\^C\^E\0\^E\0\M^@\^B\0\0\0\0\ \0\0\M-1\^C\M-k\^D\^T\0;\^C\^E\0\^E\0\M^@\^B\0\0\0\0\^A\0\M-2\^C\M-i\^D\^R\0;\^C\ \^E\0\^E\0\M^@\^B\0\0\0\0\^A\0\M-2\^C\M-i\^D\^R\08\0\^D\0\^E\0\M^@\^B\^D\0\0\0qqq\ \0B\0\a\0\^N\0\M^@\^B\^E\0\M^@\^B\0\0\M-D\^C\M-j\^D\M-D\^C\M-j\^D\M-C\^C\M-j\^D\ \M-1\^C8\M^@\^D\0\^E\0\M^@\^B\^D\0\0\0\0\0\0\0B\0\a\0\^N\0\M^@\^B\^E\0\M^@\^B\0\0\ \M-1\^C\M-i\^D\M-1\^C\0\0\M-C\^C\0\0\M-2\^C8C\^D\0\^E\0\M^@\^B\^D\0\0\0\^A\^A\^A\0\ F\0\^E\0\^N\0\M^@\^B\^E\0\M^@\^B\^A\0\M-2\^C\M-i\^D\^R\0\M^K\a\^B\0\a\^A\M^@\^B\ \M^K\^D\^E\0\b\^A\M^@\^B\^N\0\M^@\^B)\0\0\0\0\0\0\0\M^K\^F\^E\0\b\^A\M^@\^B\0\0\0\ \0\^A\0\M-2\^C\M-i\^D\^R\0\M^K\^T\^_\0\^B\0\M^@\^B\^A\0\0\0&\0\0\0\b\0\f\0\0\0\v\0\ \0\0\0\0\0\0\0\^E\0\0\0\0\0i\M-]\M-~\M-fz\^B\0d\M-nY W\M-pn\0\M-\o\0\0\0z\ \M-L\b\M-~0\0\0\0\^XI\^_\M^?\^\\0\0\0\0\0\^_\M^?\^\\0\0\0\0\0\r\M^?+\0\0\0\0\0\0\ \M-g_\0\0\0`\M-T\0z\M-`0\0%\M-X\M^I\0\^F\M^R\M-}\M-v\M^?\M-# \0\0\0 ,\^O\0\0\ \M^K\^X\^R\0\^C\M-U\M-b\0(\0\M^@\^B\b\^A\M^@\^B$\0\0\0\^B\0\M^@\^B\0\0\0\0\^R\0\0\ \0\^B\0\M-@\^C&\0t\^Bj\^B\M-2\^B\M-d\^A\^F\^B\M-2\^BC\^Bj\^B1\^B\^C\09\^Bt\^B\M-'\ \^B\M-2\^BP\^DP\^DP\^D;\^C\^E\0\^E\0\M^@\^B\0\0\0\0\^A\0\M-2\^C\M-i\^D\^R\0;\^C\^E\ \0\^E\0\M^@\^B\0\0\0\0\0\0\M-1\^C\M-k\^D\^T\0\M^Q\^C\^D\0\^A\0\0\0\^D\0\M^@\^B\^C\ \0\0\0" 57506 dillo RET writev 504/0x1f8 57506 dillo CALL futex(0xc4d86ad1a0,0x82<FUTEX_WAKE|FUTEX_PRIVATE_FLAG>,2147483647,0,0) 57506 dillo RET futex 0 57506 dillo CALL futex(0xc4d86b3ef0,0x82<FUTEX_WAKE|FUTEX_PRIVATE_FLAG>,1,0,0) 57506 dillo RET futex 0 57506 dillo CALL recvmsg(5,0x759b98292370,0) 57506 dillo RET recvmsg -1 errno 35 Resource temporarily unavailable 57506 dillo CALL recvmsg(5,0x759b982920d0,0) 57506 dillo RET recvmsg -1 errno 35 Resource temporarily unavailable 57506 dillo CALL recvmsg(5,0x759b98292370,0) 57506 dillo RET recvmsg -1 errno 35 Resource temporarily unavailable 57506 dillo CALL select(7,0x759b98292390,0x759b98292410,0x759b98292490,0) 57506 dillo STRU struct fd_set { 3,5 } 57506 dillo STRU struct fd_set { 6 } 57506 dillo STRU struct fd_set { } 57506 dillo STRU struct fd_set { } 57506 dillo STRU struct fd_set { 6 } 57506 dillo STRU struct fd_set { } 57506 dillo RET select 1 57506 dillo CALL kbind(0x759b98292288,24,0xb28d566424727b4) 57506 dillo RET kbind 0 57506 dillo CALL getsockopt(6,SOL_SOCKET,SO_ERROR,0x759b98292358,0x759b9829235c) 57506 dillo RET getsockopt 0 57506 dillo CALL kbind(0x759b98292248,24,0xb28d566424727b4) 57506 dillo RET kbind 0 57506 dillo CALL kbind(0x759b98292248,24,0xb28d566424727b4) 57506 dillo RET kbind 0 57506 dillo CALL kbind(0x759b98292248,24,0xb28d566424727b4) 57506 dillo RET kbind 0 57506 dillo CALL kbind(0x759b98292218,24,0xb28d566424727b4) 57506 dillo RET kbind 0 57506 dillo CALL kbind(0x759b98292248,24,0xb28d566424727b4) 57506 dillo RET kbind 0 57506 dillo CALL kbind(0x759b982911d8,24,0xb28d566424727b4) 57506 dillo RET kbind 0 57506 dillo CALL mmap(0,0x5000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0) 57506 dillo RET mmap 846576910336/0xc51bea8000 57506 dillo CALL write(6,0xc48c3b0640,0x123) 57506 dillo GIO fd 6 wrote 291 bytes "\^V\^C\^C\^A\^^\^A\0\^A\^Z\^C\^C\M-=\^[j 2\M-r_\M-5\M-Z*|^Bm\M-+\M-I\M-LD\M^G]-e\^Xa\M^H\M-0\^]6\M-J'g\M-k \M^^\M-n\M-S\M-D\ \M-B\M-}\^QWyqV\M-d,\M^K\M^L\M-/\M-l\f< \M--~\M-ax\^V,^`\M^Y\M-FV\M-]\0V\^S\^B\^S\^C\^S\^A\M-@0\M-@,\M-@(\M-@$\M-@\^T\M-@ \0\M^_\0k\09\M-L\M-)\M-L\M-(\M-L\M-*\0\M-D\0\M^H\0\M^]\0=\0005\0\M-@\0\M^D\M-@/\ \M-@+\M-@'\M-@#\M-@\^S\M-@ \0\M^^\0g\0003\0\M->\0E\0\M^\\0<\0/\0\M-:\0A\M-@\ \^R\M-@\b\0\^V\0 \0\M^?\^A\0\0{\0\0\0\^P\0\^N\0\0\vnarkive.com\0\v\0\^B\^A\0\0\r\0\^X\0\^V\b\^F\^F\ \^A\^F\^C\b\^E\^E\^A\^E\^C\b\^D\^D\^A\^D\^C\^B\^A\^B\^C\0 \0 \0\b\0\^]\0\^W\0\^X\0\^Y\0003\0&\0$\0\^]\0 \^O\M-+\r\M-k$\M-f\^\\f]K\M-Fr\M->&\M-s\ a \M^Y\M-0\M-]F`NU\M^X(\^XT\M-MTQ5\0+\0\^E\^D\^C\^D\^C\^C\0#\0\0" 57506 dillo RET write 291/0x123 57506 dillo CALL read(6,0xc48c3a58b0,0x5) 57506 dillo RET read -1 errno 35 Resource temporarily unavailable 57506 dillo CALL kbind(0x759b982911d8,24,0xb28d566424727b4) 57506 dillo RET kbind 0 57506 dillo CALL futex(0xc4d868bc00,0x82<FUTEX_WAKE|FUTEX_PRIVATE_FLAG>,2147483647,0,0) 57506 dillo RET futex 0 57506 dillo CALL sigprocmask(SIG_BLOCK,0<>) 57506 dillo RET sigprocmask 0<> 57506 dillo CALL poll(0x759b982923c8,1,INFTIM) 57506 dillo STRU struct pollfd { fd=5, events=0x5<POLLIN|POLLOUT>, revents=0x4<POLLOUT> } 57506 dillo RET poll 1 57506 dillo CALL writev(5,0x759b98292470,3) 57506 dillo STRU struct iovec [3] { base=0xc53d099000, len=508 } { base=0x0, len=0 } { base=0xc504b50156, len=0 } 57506 dillo GIO fd 5 wrote 508 bytes "8\^B\^E\0\^E\0\M^@\^B\^D\0\b\0\M^Y\M^Y\M^Y\0\0\0\0\0;\^C\^E\0\^E\0\M^@\^B\0\0\0\0\ \0\0\M-1\^C\M-k\^D\^T\0;\^C\^E\0\^E\0\M^@\^B\0\0\0\0\^A\0\M-2\^C\M-i\^D\^R\0;\^C\ \^E\0\^E\0\M^@\^B\0\0\0\0\^A\0\M-2\^C\M-i\^D\^R\08\0\^D\0\^E\0\M^@\^B\^D\0\0\0qqq\ \0B\0\a\0\^N\0\M^@\^B\^E\0\M^@\^B\0\0\M-D\^C\M-j\^D\M-D\^C\M-j\^D\M-C\^C\M-j\^D\ \M-1\^C8\M^@\^D\0\^E\0\M^@\^B\^D\0\0\0\0\0\0\0B\0\a\0\^N\0\M^@\^B\^E\0\M^@\^B\0\0\ \M-1\^C\M-i\^D\M-1\^C\0\0\M-C\^C\0\0\M-2\^C8C\^D\0\^E\0\M^@\^B\^D\0\0\0\^A\^A\^A\0\ F\0\^E\0\^N\0\M^@\^B\^E\0\M^@\^B\^A\0\M-2\^C\M-i\^D\^R\0\M^K\a\^B\0\b\^A\M^@\^B\ \M^K\^D\^E\0 \^A\M^@\^B\^N\0\M^@\^B)\0\0\0\0\0\0\0\M^K\^F\^E\0 \^A\M^@\^B\ \0\0\0\0\^A\0\M-2\^C\M-i\^D\^R\0\M^K\^T\^Y\0\^B\0\M^@\^B\^A\0\0\0\M-<\^B\0\0\a\0\ \0\M^?\M^?\b\0 \0\0\0J/\0\0\0T%\0\M-7s\0\0\0\M-O[\0\M-7s\0\0\0\M-O[\0\M-7\ s\0\0\0\M-O[\0\M-7s\0\0\0\M-P[\0\M-2\^?\0\0\^A\M-i[\0\M^H\M-S*\^Dy\M-q[\0\^W\M-O\ \M^?\M-kp\M-C[\0\0\0\^U\^C\0\0\0\0\M^K\^X\^Y\0\^C%\M-X\M^I(\0\M^@\^B \^A\M^@\^B\ $\0\0\0\^B\0\M^@\^B\0\0\0\0 \0\0\0\^B\0\M-@\^C\M-F\0\^U\^B\^F\^B\M-<\^B\M^]\^B\^U\ \^B\^C\0\^F\^Bt\^Bj\^Bj\^B\^U\^B\^F\^B\M-2\^BC\^Bt\^Bj\^B\^C\0j\^B\^U\^B1\^Bt\^B\ \M-2\^BC\^B\M-d\^A\M-2\^BC\^Bt\^Bj\^BP\^DP\^DP\^D;\^C\^E\0\^E\0\M^@\^B\0\0\0\0\^A\ \0\M-2\^C\M-i\^D\^R\0;\^C\^E\0\^E\0\M^@\^B\0\0\0\0\0\0\M-1\^C\M-k\^D\^T\0\M^Q\^C\ \^D\0\^A\0\0\0\^D\0\M^@\^B\^C\0\0\0" 57506 dillo RET writev 508/0x1fc 57506 dillo CALL futex(0xc4d86ad1a0,0x82<FUTEX_WAKE|FUTEX_PRIVATE_FLAG>,2147483647,0,0) 57506 dillo RET futex 0 57506 dillo CALL futex(0xc4d86b3ef0,0x82<FUTEX_WAKE|FUTEX_PRIVATE_FLAG>,1,0,0) 57506 dillo RET futex 0 57506 dillo CALL recvmsg(5,0x759b98292370,0) 57506 dillo RET recvmsg -1 errno 35 Resource temporarily unavailable 57506 dillo CALL select(7,0x759b98292390,0x759b98292410,0x759b98292490,0) 57506 dillo STRU struct fd_set { 3,5,6 } 57506 dillo STRU struct fd_set { } 57506 dillo STRU struct fd_set { } 57506 dillo STRU struct fd_set { 5 } 57506 dillo STRU struct fd_set { } 57506 dillo STRU struct fd_set { } 57506 dillo RET select 1 57506 dillo CALL recvmsg(5,0x759b982920d0,0) 57506 dillo GIO fd 5 read 32 bytes "\^C$\M-/\bZ\^C\0\^A\M-G\^B\0\0\^D\0\M^@\^B\0\0\0\0j\^Fi\0 \^A\^R\0\0\0\^A\0" 57506 dillo STRU struct msghdr { name=0x0, namelen=0, iov=0x759b982920c0, iovlen=1, control=0x759b98292070, controllen=0, flags=0x80<MSG_DONTWAIT> } 57506 dillo STRU struct iovec { base=0xc554d1a04c, len=4064 } 57506 dillo RET recvmsg 32/0x20 57506 dillo CALL futex(0xc4d86b3ef0,0x82<FUTEX_WAKE|FUTEX_PRIVATE_FLAG>,1,0,0) 57506 dillo RET futex 0 57506 dillo CALL recvmsg(5,0x759b982920d0,0) 57506 dillo RET recvmsg -1 errno 35 Resource temporarily unavailable 57506 dillo CALL recvmsg(5,0x759b98291ed0,0) 57506 dillo RET recvmsg -1 errno 35 Resource temporarily unavailable 57506 dillo CALL recvmsg(5,0x759b982920d0,0) 57506 dillo RET recvmsg -1 errno 35 Resource temporarily unavailable 57506 dillo CALL recvmsg(5,0x759b98292370,0) 57506 dillo RET recvmsg -1 errno 35 Resource temporarily unavailable 57506 dillo CALL select(7,0x759b98292390,0x759b98292410,0x759b98292490,0) 57506 dillo STRU struct fd_set { 3,5,6 } 57506 dillo STRU struct fd_set { } 57506 dillo STRU struct fd_set { } 57506 dillo STRU struct fd_set { 6 } 57506 dillo STRU struct fd_set { } 57506 dillo STRU struct fd_set { } 57506 dillo RET select 1 57506 dillo CALL read(6,0xc48c3a58b0,0x5) 57506 dillo RET read 0 57506 dillo CALL write(1,0xc4e11d5000,0x44) 57506 dillo GIO fd 1 wrote 68 bytes "TLS connect error: "an EOF was observed that violates the protocol" " 57506 dillo RET write 68/0x44 57506 dillo CALL write(1,0xc4e11d5000,0x41) 57506 dillo GIO fd 1 wrote 65 bytes "Tls_close_by_key: Avoiding SSL shutdown for: https://narkive.com " 57506 dillo RET write 65/0x41 57506 dillo CALL kbind(0x759b982911d8,24,0xb28d566424727b4) 57506 dillo RET kbind 0 57506 dillo CALL write(1,0xc4e11d5000,0x18) 57506 dillo GIO fd 1 wrote 24 bytes "fd 6 is done and failed " 57506 dillo RET write 24/0x18 57506 dillo CALL close(6) 57506 dillo RET close 0 57506 dillo CALL getpid() 57506 dillo RET getpid 57506/0xe0a2 57506 dillo CALL write(2,0x759b98291bf0,0x18) 57506 dillo GIO fd 2 wrote 24 bytes "dillo(57506) in free(): " 57506 dillo RET write 24/0x18 57506 dillo CALL write(2,0x759b98291cd0,0x29) 57506 dillo GIO fd 2 wrote 41 bytes "write to free mem 0xc48c3dbd80[24..31]@32" 57506 dillo RET write 41/0x29 57506 dillo CALL write(2,0x759b98291bf0,0x1) 57506 dillo GIO fd 2 wrote 1 bytes " " 57506 dillo RET write 1 57506 dillo CALL sigprocmask(SIG_SETMASK,~0x20<SIGABRT>) 57506 dillo RET sigprocmask 0<> 57506 dillo CALL thrkill(0,SIGABRT,0) 57506 dillo PSIG SIGABRT SIG_DFL 57506 dillo NAMI "dillo.core"

Aug. 28, 2024