November 2004 - Dillo-dev - lists.mailman3.com

[Dillo-dev]Re: dillo: umask(0) in dpid/main.c?

by Jorge Arellano Cid

On Sun, Nov 28, 2004 at 01:03:23PM +0100, Richard Zidlicky wrote: > Hi, Hi. > I stumbled upon this as I noticed that bookmarks and saved files > are world writable. Is there still a good reason to set umask > to 0 for all plugins? Good question! Basically the umask(0) call is there because the "Linux Programming Second Edition Unleashed" recommends so as a standard for daemons ;). Now, looking into it, it makes much more sense to use 077 as a mask for dpis (i.e. only allowing the owner). > > Another detail I noticed while checking file creation > is this: > --- dillo-0.8.3/src/cookies.c.rz 2004-11-28 12:39:33.000000000 +0100 > +++ dillo-0.8.3/src/cookies.c 2004-11-28 12:40:22.000000000 +0100 > @@ -100,7 +100,7 @@ > > if ((F_in = fopen(filename, "r+")) == NULL) { > /* Create the file */ > - fd = open(filename, O_WRONLY | O_CREAT | O_TRUNC, S_IRUSR | S_IWUSR); > + fd = open(filename, O_WRONLY | O_CREAT | O_TRUNC, S_IRUSR | S_IWUSR, 0777); > if (fd != -1) { > if (init_str) > write(fd, init_str, strlen(init_str)); Well, with the proper mask set to 0077 this would not be required. If the need to add explicit permissions arise, it'd be 0077 too. Comments are welcome. -- Cheers Jorge.-

Nov. 30, 2004

Re: [Dillo-dev]Memory leak bug (was dillo-0.8.3)

by Jorge Arellano Cid

Hi there, Do you remember this 4KB memory leak mail-thread? On Thu, Oct 28, 2004 at 09:14:15AM +0200, Roberto A. Foglietta wrote: > robert w hall wrote: > > >Hmm - could this be distro/kernel dependent? > > Doing something similar (toggling reload when monitoring the dillo cvs > >page IIRC) and monitoring memory via top on a remote terminal shows No > >systematic effect - but this is playstation2-linux - an old 2.2.21 > >kernel on a redhat-like implementation for a mips. > > > >In detail:- > >On first coming from the home page to the CVS changelog, top shows > >9460k; this goes up to 9564 on first and stays there for subsequent > >presses of reload. Returning to Home then back to cvs recovers the 9460, > >9564, 9564 sequence. > >Bob > > It should be a glibc bug much more than kernel one, IMHO. > > Sometimes (but very rarely) I supposed to have managed to fix it > because I didn't see it anymore, but recompiling THE SAME code or simply > restarting dillo I saw it again. Obviously I was reloading THE SAME > trivial html page (html, head, body, 4 links in 4 rows, closing-end). The pthreads bug was not deterministic, sometimes it got detached some others not, depending on the kernel scheduler and pthreads implementation. > > I'm working on MDK 10.1 but I could test it on a slackware too. Or > better I will test dillo running on A but compiled in B and all others > combinations foreach A, B in {mdk,slack}. At least after the pthread bug interim patch (CVS) it doesn't leak anymore in my system. And there's this gentle side effect of Dillo using less than 1/5 memory than before! -- Cheers Jorge.-

Nov. 30, 2004

Re: [Dillo-dev]dillo.c patch

by Jorge Arellano Cid

Hi Russell, Sorry for the late answer, the parser and pthreads kept me away from everything else. On Wed, Nov 03, 2004 at 08:05:17AM +0800, Russell J. Wood wrote: > Hello all, > > Attached is a patch for dillo.c that merely ensures all command line arguments > are parsed and complete before the GUI environment. > > Previous to this, the locale and GUI functions where run and then the command > line, of which, in my opinion, should be run after the CLI because the CLI has > the immediate commands. There is no point firing up the GUI when all one would > like to know is the version they are running, right? > > Again, it's nothing major and doubtfully increases performance in any way. The locale needs to be set before parsing the options because some of them can change its meanings (e.g. 1,2 != 1.2). gtk_init(&argc, &argv) needs to be called to get rid of gtk specific args (hopefully not much of an issue when working under FLTK though ;). -- Cheers Jorge.-

Nov. 30, 2004

[Dillo-dev]Dillo doesn't work!! :(

by Andrea Spadaccini

Hi everyone, i compiled and installed Dillo 0.83, but it doesn't open anything! Here there are the error messages I get: [andrea@iridiumII andrea]$ dillo dillorc:98: unexpected string constant "force_visited_color", expected symbol dillo_dns_init: Here we go! Disabling cookies. Nav_open_url: Url=>about:splash< Nav_open_url: Url=>dpi:/bm/< dpi_socket_dir.c:61: tst_dir: access : No such file or directory - /tmp/andrea-0c137u/ debug_msg - init_sockdir: The socket directory /tmp/andrea-0c137u does not exist or is not a directory dpid.c:500: init_all_dpi_sockets: bind : Address already in use/tmp/andrea-3vbac6/bookmarks.dpi dpid started Capi_dpi_conn_timeout:: try 1 main.c:153: get_command: dpid tag is NULL : main.c:326: main: get_command failed : ** ERROR **: file main.c: line 351 (main): assertion failed: (numsocks >= 0) aborting... waitpid: No child processes IO_write_chunk: Pipe rotta It seems like it cannot access the dir /tmp/andrea-0c137u, but it should be able to do it: [andrea@iridiumII andrea]$ ll /tmp/ totale 4 drwx------ 2 andrea users 4096 2004-11-30 00:59 andrea-3vbac6/ srwxr-xr-x 1 andrea users 0 2004-11-30 00:59 sylpheed-1000= srwxr-xr-x 1 andrea users 0 2004-11-29 22:20 xmms_andrea.0= srwxr-xr-x 1 giorgia users 0 2004-11-03 16:57 xmms_giorgia.0= srwxr-xr-x 1 luther users 0 2004-11-27 11:50 xmms_luther.0= [andrea@iridiumII andrea]$ cd /tmp/andrea-3vbac6/ [andrea@iridiumII andrea-3vbac6]$ ls bookmarks.dpi= dpid.srs= ftp.filter.dpi= https.filter.dpi= downloads.dpi= file.dpi= hello.filter.dpi= I get the same problems if I'm root too, so it shouldn't be a permissions problem... I've tried installing dpis in my home directory, but it doesn't work.. Can you help me, please? Thanks! :) -- [ Andrea Spadaccini a.k.a. Lupino / Lupin85 - from Catania ] [ MAIL: lupin85(at)email.it - ICQ: 91528290 - Linux 2.4.22 ] [ Linux Registered User #313388 - Powered by Slackware 9.1 ]

Nov. 30, 2004

Re: [Dillo-dev]pthreads

by Jorge Arellano Cid

On Fri, Nov 26, 2004 at 02:14:53PM -0200, Livio Baldini Soares wrote: > Hi Jorge! > > Jorge Arellano Cid writes: > > Errata: > > > > > > > > but it usually ends with: > > > > > > jcid 19262 0.0 0.4 1652 400 pts/10 S+ 08:18 0:00 ./pth_mem2 cd > > > [...] > > > jcid 19027 0.0 0.4 2676 408 pts/10 S+ 08:14 0:00 ./pth_mem2 cd > > > > Oh, the the PID and time in the above example are quoted wrong, > > it should say: > > > > jcid 19262 0.0 0.4 1652 400 pts/10 S+ 08:18 0:00 ./pth_mem2 cd > > [...] > > jcid 19262 0.0 0.4 2676 408 pts/10 S+ 08:18 0:00 ./pth_mem2 cd > > I can't really tell if that's a leak or not. In my system, there is > an increase use of memory between the first iteration and the second, > but from then on, things become _very_ stable (box with Linux 2.4 and > libc 2.3.2 from Debian/unstable): I understand. In my box the "leak" usually occurs near the fifth if at all... > livio 20097 0.0 0.0 1664 336 pts/21 S+ 10:24 0:00 ./pth_mem cd > livio 20097 0.0 0.0 32556 468 pts/21 S+ 10:24 0:00 ./pth_mem cd > livio 20097 0.0 0.0 1796 396 pts/21 S+ 10:24 0:00 ./pth_mem cd > livio 20097 0.0 0.0 32556 476 pts/21 S+ 10:24 0:00 ./pth_mem cd > livio 20097 0.0 0.0 1796 396 pts/21 S+ 10:25 0:00 ./pth_mem cd > > So there is certainly an increase in memory use (in my case a lot > smaller than yours). But look, there is no leak of memory, because > with each interation, the memory is equal to the previous phase. > > Or do you get an increase of 0.5-1 MiB for _each_ phase? No but I get a "random" one in one of them. For instance: I just run this one and there was no leak: <q> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND jcid 26194 0.0 0.3 1640 336 pts/29 S+ 17:50 0:00 ./pth_mem2 cd jcid 26194 0.0 0.5 32412 476 pts/29 S+ 17:50 0:00 ./pth_mem2 cd jcid 26194 0.0 0.4 1652 404 pts/29 S+ 17:50 0:00 ./pth_mem2 cd jcid 26194 0.0 0.5 32412 484 pts/29 S+ 17:50 0:00 ./pth_mem2 cd jcid 26194 0.0 0.4 1652 404 pts/29 S+ 17:50 0:00 ./pth_mem2 cd jcid 26194 0.0 0.5 32412 484 pts/29 S+ 17:50 0:00 ./pth_mem2 cd jcid 26194 0.0 0.4 1652 404 pts/29 S+ 17:50 0:00 ./pth_mem2 cd jcid 26194 0.0 0.5 32412 484 pts/29 S+ 17:50 0:00 ./pth_mem2 cd jcid 26194 0.0 0.4 1652 404 pts/29 S+ 17:50 0:00 ./pth_mem2 cd jcid 26194 0.0 0.5 32412 484 pts/29 S+ 17:50 0:00 ./pth_mem2 cd jcid 26194 0.0 0.4 1652 404 pts/29 S+ 17:50 0:00 ./pth_mem2 cd jcid 26194 0.0 0.5 32412 484 pts/29 S+ 17:50 0:00 ./pth_mem2 cd jcid 26194 0.0 0.4 1652 404 pts/29 S+ 17:50 0:00 ./pth_mem2 cd jcid 26194 0.0 0.5 32412 484 pts/29 S+ 17:50 0:00 ./pth_mem2 cd jcid 26194 0.0 0.4 1652 404 pts/29 S+ 17:50 0:00 ./pth_mem2 cd jcid 26194 0.0 0.5 32412 484 pts/29 S+ 17:50 0:00 ./pth_mem2 cd jcid 26194 0.0 0.4 1652 404 pts/29 S+ 17:50 0:00 ./pth_mem2 cd </q> Then I run this other one (w/leak): <q> jcid 26878 0.0 0.3 1640 336 pts/29 S+ 18:27 0:00 ./pth_mem2 d jcid 26878 0.0 0.5 32412 472 pts/29 S+ 18:27 0:00 ./pth_mem2 d jcid 26878 0.0 0.4 1652 400 pts/29 S+ 18:27 0:00 ./pth_mem2 d jcid 26878 0.0 0.5 32412 480 pts/29 S+ 18:27 0:00 ./pth_mem2 d jcid 26878 0.0 0.4 1652 400 pts/29 S+ 18:27 0:00 ./pth_mem2 d jcid 26878 0.0 0.5 32412 480 pts/29 S+ 18:27 0:00 ./pth_mem2 d jcid 26878 0.0 0.4 1652 400 pts/29 S+ 18:27 0:00 ./pth_mem2 d jcid 26878 0.0 0.5 33436 484 pts/29 S+ 18:27 0:00 ./pth_mem2 d * jcid 26878 0.0 0.4 2676 404 pts/29 S+ 18:27 0:00 ./pth_mem2 d jcid 26878 0.0 0.5 33436 484 pts/29 S+ 18:27 0:00 ./pth_mem2 d jcid 26878 0.0 0.4 2676 404 pts/29 S+ 18:27 0:00 ./pth_mem2 d jcid 26878 0.0 0.5 33436 484 pts/29 S+ 18:27 0:00 ./pth_mem2 d jcid 26878 0.0 0.4 2676 404 pts/29 S+ 18:27 0:00 ./pth_mem2 d jcid 26878 0.0 0.5 33436 484 pts/29 S+ 18:27 0:00 ./pth_mem2 d jcid 26878 0.0 0.4 2676 404 pts/29 S+ 18:27 0:00 ./pth_mem2 d jcid 26878 0.0 0.5 33436 484 pts/29 S+ 18:27 0:00 ./pth_mem2 d jcid 26878 0.0 0.4 2676 404 pts/29 S+ 18:27 0:00 ./pth_mem2 d </q> I can easily get results with an without leaks for the three ways though. As from my tests, it seems like the "leak" can appear on any iteration > > To make my point clearer, look at how my home box handles the fixed > test program (I'm using Linux kernel 2.6 with NTPL, and libc 2.3.2 > from Debian/unstable): > > livio 4300 0.0 0.0 1440 316 pts/7 S+ 10:40 0:00 ./pth_mem d > livio 4300 0.0 0.0 93812 432 pts/7 Sl+ 10:40 0:00 ./pth_mem d > livio 4300 0.0 0.0 34356 372 pts/7 S+ 10:40 0:00 ./pth_mem d > livio 4300 0.0 0.0 93812 436 pts/7 Sl+ 10:40 0:00 ./pth_mem d > > etc. Even though I use humengeous use of memory, I'm pretty sure > it's not leaking (since the numbers are the same after the first > iteration). A thing that's not happening on my box. > > Looking at straces, I see that libc is doing mmap2() on anonymous > memory for allocation, and in the first phase, actually allocates 8 > MiB worth of stack for each thread! (I know it's stack because it's > setting PROT_EXEC on, whoch it doesn't for regular > mallocs()). Subsequently, each thread mmap2() _2_ MiB of anonymous > memory (instead of only one). > > Anyways, not wanting to get too much into the details, it's hard to > predict what your libc is doing for you. I can see that libc correctly > munmaps() all the mapped segments (effectively freeing the > memory). _And_ the VSZ stays the same throughout interations. > > REMINDER: In this case a big VSZ does not mean big memory > usage. It means a _potential_ to use a lot of memory (I've seen rare > cases where a thread actually makes use of more than 1MiB of stack, > for example, let alone 8MiB). If the mmap2() is is never touched, a > page never gets assigned to the process' page table. BTW, Is the VSZ reserved somehow, on swap at least? or is just a potential size that we can happily ignore? I'm adding six more test results, this time with Dillo and a small page with a hundred tiny images. It has three samples for each technique: Note: I'm using the file dpi server, so dillo uses a pthread for each IO. Using 'd' technique: <q> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 13820 11168 ./dillo 15492 12624 ./dillo 15600 12736 ./dillo </q> Using 'cd' technique: <q> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 15496 12640 ./dillo 9272 6888 ./dillo 10408 7940 ./dillo </q> Here the RSS is smaller for 'cd' so I opted this one for the interim patch just commited to CVS. -- Cheers Jorge.-

Nov. 28, 2004

Re: [Dillo-dev]pthreads

by Livio Baldini Soares

Hi Jorge! Jorge Arellano Cid writes: > Errata: > > > > > but it usually ends with: > > > > jcid 19262 0.0 0.4 1652 400 pts/10 S+ 08:18 0:00 ./pth_mem2 cd > > [...] > > jcid 19027 0.0 0.4 2676 408 pts/10 S+ 08:14 0:00 ./pth_mem2 cd > > Oh, the the PID and time in the above example are quoted wrong, > it should say: > > jcid 19262 0.0 0.4 1652 400 pts/10 S+ 08:18 0:00 ./pth_mem2 cd > [...] > jcid 19262 0.0 0.4 2676 408 pts/10 S+ 08:18 0:00 ./pth_mem2 cd I can't really tell if that's a leak or not. In my system, there is an increase use of memory between the first iteration and the second, but from then on, things become _very_ stable (box with Linux 2.4 and libc 2.3.2 from Debian/unstable): livio 20097 0.0 0.0 1664 336 pts/21 S+ 10:24 0:00 ./pth_mem cd livio 20097 0.0 0.0 32556 468 pts/21 S+ 10:24 0:00 ./pth_mem cd livio 20097 0.0 0.0 1796 396 pts/21 S+ 10:24 0:00 ./pth_mem cd livio 20097 0.0 0.0 32556 476 pts/21 S+ 10:24 0:00 ./pth_mem cd livio 20097 0.0 0.0 1796 396 pts/21 S+ 10:25 0:00 ./pth_mem cd So there is certainly an increase in memory use (in my case a lot smaller than yours). But look, there is no leak of memory, because with each interation, the memory is equal to the previous phase. Or do you get an increase of 0.5-1 MiB for _each_ phase? To make my point clearer, look at how my home box handles the fixed test program (I'm using Linux kernel 2.6 with NTPL, and libc 2.3.2 from Debian/unstable): livio 4300 0.0 0.0 1440 316 pts/7 S+ 10:40 0:00 ./pth_mem d livio 4300 0.0 0.0 93812 432 pts/7 Sl+ 10:40 0:00 ./pth_mem d livio 4300 0.0 0.0 34356 372 pts/7 S+ 10:40 0:00 ./pth_mem d livio 4300 0.0 0.0 93812 436 pts/7 Sl+ 10:40 0:00 ./pth_mem d etc. Even though I use humengeous use of memory, I'm pretty sure it's not leaking (since the numbers are the same after the first iteration). Looking at straces, I see that libc is doing mmap2() on anonymous memory for allocation, and in the first phase, actually allocates 8 MiB worth of stack for each thread! (I know it's stack because it's setting PROT_EXEC on, whoch it doesn't for regular mallocs()). Subsequently, each thread mmap2() _2_ MiB of anonymous memory (instead of only one). Anyways, not wanting to get too much into the details, it's hard to predict what your libc is doing for you. I can see that libc correctly munmaps() all the mapped segments (effectively freeing the memory). _And_ the VSZ stays the same throughout interations. REMINDER: In this case a big VSZ does not mean big memory usage. It means a _potential_ to use a lot of memory (I've seen rare cases where a thread actually makes use of more than 1MiB of stack, for example, let alone 8MiB). If the mmap2() is is never touched, a page never gets assigned to the process' page table. hope this helps, -- Livio B. Soares

Nov. 26, 2004

Re: [Dillo-dev]pthreads

by Jorge Arellano Cid

Errata: > > but it usually ends with: > > jcid 19262 0.0 0.4 1652 400 pts/10 S+ 08:18 0:00 ./pth_mem2 cd > [...] > jcid 19027 0.0 0.4 2676 408 pts/10 S+ 08:14 0:00 ./pth_mem2 cd Oh, the the PID and time in the above example are quoted wrong, it should say: jcid 19262 0.0 0.4 1652 400 pts/10 S+ 08:18 0:00 ./pth_mem2 cd [...] jcid 19262 0.0 0.4 2676 408 pts/10 S+ 08:18 0:00 ./pth_mem2 cd -- Cheers Jorge.-

Nov. 26, 2004

Re: [Dillo-dev]pthreads

by Jorge Arellano Cid

On Thu, Nov 25, 2004 at 04:48:07PM +0100, Lars Segerlund wrote: > > I REALLY understand chasing the leaks and then finding > something like this stings ... Yes it does. Note that even knowing that it was my fault (assuming wrongly about the API), the irony remains. ...and the soothing side of it, is that now we have a Dillo running on less than 1/5 of the memory it used to. -- Cheers Jorge.-

Nov. 26, 2004

Re: [Dillo-dev]pthreads

by Jorge Arellano Cid

On Thu, Nov 25, 2004 at 02:39:01PM -0200, Livio Baldini Soares wrote: > Hi Jorge, > > I just got the chance to look at your little test program. It has a > few bugs: > > 1) In pthread_create() you are passing a pointer to the unintialized > thrdID[i] variable. Pthreads does not guarantee that the threadID > argument is initialized before the start_routine is called. If you > try printing the value of p_thrID in the 'thr_function', you'll see > you'll get some correct and some random values. Gotcha! This explains most of it. So the quid is that Pthreads does not guarantee that the threadID argument is initialized before the start_routine is called. My fault! Even though some other pthread implementations seem to do it, the document from the opengroup you cite is clear: <q> APPLICATION USAGE There is no requirement on the implementation that the ID of the created thread be available before the newly created thread starts executing. The calling thread can obtain the ID of the created thread through the return value of the pthread_create() function, and the newly created thread can obtain its ID by a call to pthread_self(). </q> Now, even after patching the test program to use: > - pthread_detach(*p_thrID); > + pthread_detach(pthread_self()); it still leaks some memory in my machine! (near half a MB). It is interesting to note that as Brian reports: > [Brian wrote:] > > [Jorge wrote:] > > > > Has anyone had the opportunity to run the test program under > > Solaris? This is an interesting test because Solaris has its own > > threads implementation. I'd expect the detached pthreads test not > > to leak memory on Solaris. > > did it just now. all three ways leak zero memory. Zero leak on Solaris. On my machine, the three ways can still leak, but sometimes they don't. For instance, running the test should give: jcid 19262 0.0 0.4 1652 400 pts/10 S+ 08:18 0:00 ./pth_mem2 cd [...] jcid 19262 0.0 0.4 1652 400 pts/10 S+ 08:18 0:00 ./pth_mem2 cd but it usually ends with: jcid 19262 0.0 0.4 1652 400 pts/10 S+ 08:18 0:00 ./pth_mem2 cd [...] jcid 19027 0.0 0.4 2676 408 pts/10 S+ 08:14 0:00 ./pth_mem2 cd (the same amount leaked either with "d", "cd" or "j" mode). At some point in time "cd" seemed to be the "better" one, but surely more testing is required. Or better, finding the bug. ;) -- Cheers Jorge.-

Nov. 26, 2004

Re: [Dillo-dev]pthreads

by Livio Baldini Soares

Hi Brian, Brian Hechinger writes: > i just did a test on my mac, which runs NetBSD 2.0 RC4, and it also does > not leak memory in either of the three cases. > > i know they use the pth library, but i wonder if they do something > differently than linux does? Did you get a chance to read my bugreport messages? http://lists.auriga.wearlab.de/pipermail/dillo-dev/2004-November/002448.html and http://lists.auriga.wearlab.de/pipermail/dillo-dev/2004-November/002450.html It is possible that different pthread implementations initialize (or set) the threadID (first argument passed to pthread_create()) in different orders. In Linux, I have seen different systems behave randomly (probably because it issues a clone(), and from then on let's the OS take over the scheduling, which is usually non-deterministic). Even if you tried sucessfully on Solaris and *BSD, it does not mean that it is a safe construct to rely on. The OS might happen to schedule the threads in a certain order, or the specific version of the libraries synchronizes the thread creation in a specific manner, etc. Bottom line, the POSIX threads specificiation does *not* require the threadID to be set before the new thread begin execution:

Nov. 26, 2004