October 2003 - Dillo-dev - lists.mailman3.com

hairy bug in IO fixed!

by Jorge Arellano Cid

Hi there! The last five or six days were spent on trying to find (& fix) a bug that caused the CPU to hog under very hard to reproduce conditions. I know that heavy duty dillo users had noticed that in rare ocassions dillo starts taking all the CPU, and that's almost impossible to reproduce it again with another dillo... Well, the first interesting thing is that the CPU hog is "soft" (i.e. doesn't block dillo) because it is caused by a polling loop. The other surprising fact is that the CPU-hog occurs outside of dillo (!). That is, not in any part of its code. So if you were to attach GDB to find the bug (once on hog), you'd be always pointed outside dillo's source, but with a clue: polling. I was working with the ftp dpi when I finally came accross a way to reproduce the bug, so I took the chance and started digging. A whole revision/tweaking of the IO engine close/abort handling, a revision of glib's sources (gio channels) to finally find a way to stop the polling loop. Short story, there were several involved parts: threads, the IO, external processes, several open connections in parallel, glib (and the kernel). It turned out to be a race condition in the window between the actual close of the file descriptor, the glib source event removal for the watch, and the kernel reuse of FD number. The source code for gio channels (inside glib) hinted a way to remove the event source cleanly, but outside the gio channel's explicit API. The details are in IO.[ch] in CVS! Cheers Jorge.- PS: Indan wrote: > 3) There is no efficient way of (un)registering file descriptor events in > GTK. (FLTK has a perfect interface). Yes there is! (subtle but effective) > Did anyone found the bug yet, without reading the diff? If not, then it's > as harmless as I thought ;-). Still not nice of course, but it's unclear > where it goes wrong exactly, doesn't seem to be easily fixed without > changing Dillo a lot. I hope this patch and explanations helps you with point number 3. More comments later... Cheers 2 Jorge.-

Oct. 31, 2003

Re: [Dillo-dev] HTTPS Patch

by Indan Zupancic

Updated the diff, gcc 2.x should work now, link is still http://www.xs4all.nl/~dorinek/dillo/dillo-ssl.diff.gz Did anyone found the bug yet, without reading the diff? If not, then it's as harmless as I thought ;-). Still not nice of course, but it's unclear where it goes wrong exactly, doesn't seem to be easily fixed without changing Dillo a lot. It would be nice to get some feedback from the Dillo developers, I'd like to know what they think should change before the code is good enough to go into the official Dillo, and if they even consider that of course. I have plenty of ideas, but most of them aren't feasible without merging the two IOData structures (now there is one for the read part, and one for the write part, why, I don't know, doesn't make things simpler), doing that makes it much easier to do certain, more complex things that need to have all the fd related info in one place, like ssl, connection caching (currently Dillo opens and closes a connection for each action, although it seems to work well, it's still more efficient to keep connections open longer, especially with slow/bad network connections), "keep alive" (to avoid timeout problems in ssl, quite annoying when you're taking too much time typing an email), simple ftp support, etc. I know that the Dillo developers want to implement most "extra" features like https/ftp/bookmarks/etc as plugins, but I think it's beter to implement the features that are used a lot, or just very important, like https and ftp, natively in Dillo with a small implementation. The overall code is most times smaller too, can say that Dillo is just 300 K, but it would be a bit weird if the plugins together are bigger than Dillo after a while. https took around 2 Kb, I think I can implement basic FTP (and possibly ftps) in about 15 K, using no external ftp library or program (hopefully 10, worst case 20K). If you let me loose on Dillo's code then this is what I would do: # Cleanup the IO engine, one IOData per fd. # Moving the dns stuff to IO. # Add ftp support. # Add file upload support in html forms. # Add connection caching to IO. # Move all the socket stuff into IO.c, the module that wants to handle opening the connection should also close it (all non-internet sockets, like files. Unix sockets may be added to IO.c.). # Auto keep alive like behaviour for ssl connections. # Maybe always ACCEPT_SESSION for cookies when they come via a https connection. # Get rid of the ccc construction, or change it so that it behaves more logical. My opinion is that you should have either a clear unique API for each module, or one general API that is the same for each module. Something in between like the current ccc functions is only confusing (if you want a chain like behaviour, then use function pointer stuctures or something). # When changing the internal API anyway, making shared library plugins possible too, at least for the IO part (maybe ftp library plugin as proof of concept). # Get rid of the splash stuff, move it to an external html file that is opened by default when nothing else is opened. Currently the splash screen is even opened when you do "dillo http://site.org". Loading a file is almost as fast as the build in splash screen. Removing it makes Dillo smaller (about 10 K), simpler and lets people make their own startup pages (for instance a to hd saved google.com) As you could notice, I'm mainly interested in the internet/IO part, graphical stuff doesn't attract me much. If it did, I would help/start porting Dillo to FLTK. If Dillo decides to go with GTK 2 then I will probably stop using it :(. I don't have enough faith in GTK2, it's already too big (and I saw the test results). If you have GTK 2 running anyway it doesn't matter much if you run Dillo too of course, but that's the case too with all those KDE/Gnome programs. IMHO, you should look at all the code that a program uses, including the one in dependencies. I prefer a program that has it's own small and efficient implementation of something to one that is smaller but uses huge libraries. I'm also thinking about making a Dillo installer (sh script) that downloads and installs the cvs version of Dillo and applies the patches the user wants. This needs some coordination, I saw links to other Dillo patches, but I don't know if they are up to date, nor if the maker is willing to keep that so. Currently the only patches I know that are up to date are Frank's patches and my own https patch. Greetings, Indan ------------------------------------------------- This mail is written in SquirrelMail with Dillo -------------------------------------------------

Oct. 30, 2003

Fonts

by Carlos Daniel Ruvalcaba Valenzuela

Does <font size=5> exist in current CVS?, or any plans to support it and <basefont>? I think they are really useful, and should be in already. Cheers Carlos Daniel Ruvalcaba Valenzuela _________________________________________________________ Do You Yahoo!? La mejor conexión a internet y 25MB extra a tu correo por $100 al mes. http://net.yahoo.com.mx

Oct. 28, 2003

Re: [Dillo-dev] HTTPS Patch

by Indan Zupancic

> Declaring variables in mid-code (ie, anywhere in the block) is valid > only for compilers complying with the C99 standard (such as GCC3). The code is declared at the start of the block, and is also the last code declared, it's just also initialized, but if you look at the code all declarations are before the statements. Please send the emails only to the list, otherwise I receive two emails, thanks.

Oct. 27, 2003

Re: [Dillo-dev] HTTPS Patch

by David McKee

Indan, This is extremely cool. I am pleased and excited. It rates as high with me as frame support, and leaves me waiting only for printing to be content. On Mon, 27 Oct 2003, Indan Zupancic wrote: > > First, the build doesn't know where to find my openssl headers > > and libraries [snippage] > I don't know much about autoconfigure/make/build, so it's no wonder. In > the final version it should be autodetected. Fair enough. Neither do I. > Seems like you have an old openssl version, on my computer all headers > are in openssl/, you probably need to change the includes at the top > of IO.c. If they are just in a non-default directory then run > "CFLAGS=-I/path/to/ssl ./configure". Also don't forget to run autogen > after applying the patch. New openssl in a weird place because I compiled from source. I fixed this by installing some carefully chosen links in the file system, so it should work for other builds as well... But thanks for the advice: I hadn't realized that ./configure could take paths from environment variables. > I'll change that St thing, but I'm pretty sure it's allowed to do it that > way, my compiler (gcc 3.2.2) didn't give any warnings either. Will change > it anyway, if old gcc has problems with it (not home now, so will change > it later). It is (as has already been noted) a c99 thing, not allowed in the old ANSI standard. Thanks for your work. It looks great. -- -- David McKee -- dmckee(a)jlab.org -- (757) 269-7492 (Office)

Oct. 27, 2003

Re: [Dillo-dev] HTTPS Patch

by Livio Baldini Soares

Indan Zupancic writes: [...] > I'll change that St thing, but I'm pretty sure it's allowed to do it that > way, my compiler (gcc 3.2.2) didn't give any warnings either. Will change > it anyway, if old gcc has problems with it (not home now, so will change > it later). Declaring variables in mid-code (ie, anywhere in the block) is valid only for compilers complying with the C99 standard (such as GCC3). IMHO this type of construct should still be avoided, as it breaks compatibility with other ANSI-C-only compilers (including GCC 2.95). best regards, -- Livio B. Soares

Oct. 27, 2003

Re: [Dillo-dev] HTTPS Patch

by Indan Zupancic

> First, the build doesn't know where to find my openssl headers > and libraries, and there does not seem to be a ./configure option > for this. (BTW: My goodies are in /usr/local/ssl/...) I can fix > this locally, but a config option might be in order. I don't know much about autoconfigure/make/build, so it's no wonder. In the final version it should be autodetected. Seems like you have an old openssl version, on my computer all headers are in openssl/, you probably need to change the includes at the top of IO.c. If they are just in a non-default directory then run "CFLAGS=-I/path/to/ssl ./configure". Also don't forget to run autogen after applying the patch. I'll change that St thing, but I'm pretty sure it's allowed to do it that way, my compiler (gcc 3.2.2) didn't give any warnings either. Will change it anyway, if old gcc has problems with it (not home now, so will change it later). greetings, Indan

Oct. 27, 2003

Re: [Dillo-dev] HTTPS Patch

by Mike Eriksen

My Red Hat 7.3 didn't have problems finding the ssl-headers, but I couldn't compile before I made the two changes to St mentioned below. Yahoo! Now I have webmail with Dillo. Even with Yahoo! ;-) Mike On Mon, 27 Oct 2003, David McKee wrote: > Folks, > > When I apply and attempt to compile (on a Redhat 6.2 machine, gcc > 2.95), I have two problems. > > First, the build doesn't know where to find my openssl headers > and libraries, and there does not seem to be a ./configure option > for this. (BTW: My goodies are in /usr/local/ssl/...) I can fix > this locally, but a config option might be in order. > > > > Secondly, I get two compilation errors in IO.c. Both are related to the > code: > > errno = 0; > ssize_t St = SSL_read(io->ssl, io->Buf, io->BufSize); > > which I believe should (strictly) be > > ssize_t St; > errno = 0; > St = SSL_read(io->ssl, io->Buf, io->BufSize); > > because declaration should come before statements. In anycase, > making this change in both places (around lines 822 and 852) > fixes the problem for me. > > Now to try it out... > > Thanks, > > -- Thinstation FAQ maintainer http://thinstation.sourceforge.net - a light, full featured linux based thin client OS

Oct. 27, 2003

Re: [Dillo-dev] HTTPS Patch

by David McKee

Folks, When I apply and attempt to compile (on a Redhat 6.2 machine, gcc 2.95), I have two problems. First, the build doesn't know where to find my openssl headers and libraries, and there does not seem to be a ./configure option for this. (BTW: My goodies are in /usr/local/ssl/...) I can fix this locally, but a config option might be in order. Secondly, I get two compilation errors in IO.c. Both are related to the code: errno = 0; ssize_t St = SSL_read(io->ssl, io->Buf, io->BufSize); which I believe should (strictly) be ssize_t St; errno = 0; St = SSL_read(io->ssl, io->Buf, io->BufSize); because declaration should come before statements. In anycase, making this change in both places (around lines 822 and 852) fixes the problem for me. Now to try it out... Thanks, -- -- David McKee -- dmckee(a)jlab.org -- (757) 269-7492 (Office)

Oct. 27, 2003

Re: [Dillo-dev] HTTPS patch

by Mike Eriksen

Sorry - apparently I can't read :-( I'll go for the cvs version. Thanks Mike On Mon, 27 Oct 2003, Indan Zupancic wrote: > > Which version does the patch require? I tried against 0.7.3 and got some > > errors - one in each of IO.c, Url.c and capi.c. > > Quote: "It was made against the current cvs" > > I guess that's pre 0.8. > > it shouldn't be much work to get it working with 0.7.3, if you want that > then just say, I'll make another patch then. The current cvs version seems > very stable, and Frank's patches are also cvs only, so it seems more than > logical to me that if you want experimental new features, you use the cvs > version. > > Two things I forgot to mention: > > If you want a lean and mean Dillo, then comment out or, remove "#define > DEBUG_MSG 5" at the top of IO.c, the extra overhead of the spam isn't > negligible on my 233 mhz computer (it also makes the https code much > bigger). > > The openssl licence is a "GPL-Incompatible, Free Software Licenses", so > it's needed to add an exception that says that DIllo may be linked to the > openssl library, see for more info: > http://www.gnu.org/philosophy/license-list.html#TOCGPLIncompatibleLicenses > > > > _______________________________________________ > Dillo-dev mailing list > Dillo-dev(a)lists.auriga.wearlab.de > http://lists.auriga.wearlab.de/cgi-bin/mailman/listinfo/dillo-dev > -- Thinstation FAQ maintainer http://thinstation.sourceforge.net - a light, full featured linux based thin client OS

Oct. 27, 2003