- Dillo-dev - lists.mailman3.com

Memory usage
by jcid＠dillo.org June 26, 2008

June 26, 2008

Hi there, I've been reviewing memory usage these days, and found some weird numbers that I'd like to understand. Taking the huge mysql page (13MB), I made the following memory measures (with ps). Load a bare dillo, load the simple directory where the huge page is located, then the huge page, then back, then forward, back, forward and back. I did this for dillo2.19Jun, dillo2.21Jun, dillo-fltk (with zone allocators) and dillo1. All in GNU/Linux with image_off. Here are the numbers: ----------- Memory test ----------- %MEM VSZ RSS TTY STAT START TIME COMMAND 0.1 9672 4144 pts/10 S+ 09:37 0:00 ./dillo-fltk.19Jun bare 0.2 10376 4756 pts/10 S+ 09:37 0:00 ./dillo-fltk.19Jun dir list 6.9 176176 144776 pts/10 S+ 09:37 0:02 ./dillo-fltk.19Jun huge page 5.4 123556 113216 pts/10 S+ 09:37 0:02 ./dillo-fltk.19Jun back 7.0 176176 146240 pts/10 S+ 09:37 0:05 ./dillo-fltk.19Jun forward 5.6 127020 117720 pts/10 S+ 09:37 0:05 ./dillo-fltk.19Jun back 7.0 176172 146240 pts/10 S+ 09:37 0:07 ./dillo-fltk.19Jun forward 5.6 127016 117716 pts/10 S+ 09:37 0:08 ./dillo-fltk.19Jun back %MEM VSZ RSS TTY STAT START TIME COMMAND 0.1 9672 4140 pts/10 S+ 09:26 0:00 ./dillo-fltk.21Jun bare 0.2 10376 4760 pts/10 S+ 09:26 0:00 ./dillo-fltk.21Jun dir list 5.9 147904 123988 pts/10 S+ 09:31 0:02 ./dillo-fltk.21Jun huge page 4.9 111668 101944 pts/10 S+ 09:31 0:02 ./dillo-fltk.21Jun back 6.0 147780 124792 pts/10 S+ 09:31 0:05 ./dillo-fltk.21Jun forward 5.0 115008 105780 pts/10 S+ 09:31 0:05 ./dillo-fltk.21Jun back 6.0 147780 124856 pts/10 S+ 09:31 0:07 ./dillo-fltk.21Jun forward 5.1 115008 105844 pts/10 S+ 09:31 0:07 ./dillo-fltk.21Jun back %MEM VSZ RSS TTY STAT START TIME COMMAND 0.1 9676 4084 pts/10 S+ 11:16 0:00 ./dillo-fltk bare 0.2 10464 4768 pts/10 S+ 11:17 0:00 ./dillo-fltk dir list 5.4 136960 112920 pts/10 S+ 11:17 0:04 ./dillo-fltk huge page 4.3 100724 90864 pts/10 S+ 11:17 0:04 ./dillo-fltk back 5.4 136952 113756 pts/10 S+ 11:17 0:09 ./dillo-fltk forward 4.5 104180 94744 pts/10 S+ 11:17 0:09 ./dillo-fltk back 5.4 136888 113776 pts/10 S+ 11:17 0:13 ./dillo-fltk forward 4.5 104116 94764 pts/10 S+ 11:17 0:14 ./dillo-fltk back %MEM VSZ RSS TTY STAT START TIME COMMAND 0.1 8408 3524 pts/10 S+ 10:59 0:00 ./dillo1 bare 0.1 24788 3564 pts/10 S+ 11:01 0:00 ./dillo1 dir list 8.7 226996 182220 pts/10 S+ 11:01 0:07 ./dillo1 huge page 6.8 165880 141792 pts/10 S+ 11:01 0:09 ./dillo1 back 8.8 226952 184176 pts/10 S+ 11:01 0:17 ./dillo1 forward 6.9 166264 144024 pts/10 S+ 11:01 0:18 ./dillo1 back 8.8 226968 184216 pts/10 S+ 11:01 0:26 ./dillo1 forward 6.9 166264 144060 pts/10 S+ 11:01 0:27 ./dillo1 back Besides showing clearly a sane memory usage reduction pattern, there's a strange fact I don't understand. Once back from the huge page, I'd expect memory usage to drop to what it was before plus the 13MB of cached data, plus some KB of overhead, BUT the RSS keeps near 90MB. This happens with dillo1 too. My thoughts: * If this memory is not freed, it'd be great to find the huge leak. * If it's freed but not returned to the OS., dillo would consume as much memory as the biggest page it has loaded. Clearly not a desirable situation. * It may be that the O.S. knows it can claim back this memory when necessary but prefers to keep it assigned to the same process until it finds better use for it. ------------------------------------------- Huge page with different memory allocators: ------------------------------------------- %MEM VSZ RSS TTY STAT START TIME COMMAND 5.4 134912 113740 pts/17 S+ 18:56 0:02 ./dillo-fltk step: 2x 5.4 134916 113752 pts/17 S+ 18:57 0:02 ./dillo-fltk step: 2x 5.4 134828 113660 pts/17 S+ 18:58 0:02 ./dillo-fltk step: 2x 5.0 114940 105432 pts/17 S+ 18:53 0:02 ./dillo-fltk step: 1 5.0 111976 105488 pts/17 S+ 18:54 0:03 ./dillo-fltk step: 1 5.0 114872 105416 pts/17 S+ 18:54 0:02 ./dillo-fltk step: 1 5.1 117796 106436 pts/17 S+ 18:59 0:02 ./dillo-fltk step: cst 5.1 117568 106404 pts/17 S+ 19:00 0:02 ./dillo-fltk step: cst 5.1 117820 106460 pts/17 S+ 19:00 0:02 ./dillo-fltk step: cst This data comes from a memory usage reduction by tunning the simpleVector memory allocator. The first is the current one, which allocates in chunks of 2*numAlloc, the second allocates one by one and the third is a custom one I made. The 1-allocator is the best memory usage reducer but it's slower (roughly 20%). The cst-allocator reduces a bit less but is almost as fast as the 2x-allocator (2% slower or so). Memory reduction: VSZ RSS ----------------------------- 1-allocator 15% 7.3% cst-allocator 13% 6.5% ----------------------------- Well, compared with the 1000% "leak" above... :-) -- Cheers Jorge.-

2 5

[patch] Region is no Shape
by Johannes.Hofmann＠gmx.de June 23, 2008

June 23, 2008

Hi, attached patch cleans up Region. It no longer derives from Shape. Region has grown pretty specialize for handling redraws. Also it was never used as Shape object anyway. Cheers, Johannes

2 1

form.cc: more refactoring
by onepoint＠starurchin.org June 23, 2008

June 23, 2008

A set of small patches that clean up form.cc , mostly to do with DilloHtmlSelect . * refactor by creating new functions: - a_Html_parse_stash() - DilloHtmlSelect::ensureSelection() - DilloHtmlSelect::addOptionsTo() - DilloHtmlSelect::appendValuesTo() - DilloHtmlSelect::getCurrentOption() (cf. other getCurrent* methods) * make DilloHtmlSelect::options private * rename DilloHtmlInput::getInputValues() to appendValuesTo() (the new name is more mnemonic and consistent with the DilloHtmlSelect method above) Regards, Jeremy Henty

2 5

back to that SELECT segfault
by corvid＠lavabit.com June 23, 2008

June 23, 2008

In SELECT, if standard_widget_colors is set, color and backgroundColor are set to NULL. This tells FltkResource::setWidgetStyle() to leave the colors on the widget alone. If an inline element inside a SELECT calls style->color->getColor() or style->backgroundColor->getColor(), we get that crash. This currently means FONT and IMG. OBJECT and A would as well if that a_Color_vc code were uncommented. options: - four IN_SELECT tests. - letting Html_stack_cleanup_at_open() be called for inline elements in order to close the SELECT or something if IN_SELECT and the new tag isn't OPTION or OPTGROUP. - generating our own standard widget colors to put into color and backgroundColor. I don't like the first one because someone innocently adding a getColor somewhere in the future will never know about this danger. (a good example would be adding the a_Color_vc to open_frame())

3 5

Textblock::addText()
by jcid＠dillo.org June 22, 2008

June 22, 2008

Hi there, - Changed Textblock::addText() to internally allocate its text string, making the memory handling opaque to the caller. Does anybody remmeber the topic of addText() deleting a malloc'ed block of memory? I decided to make the change for API cleanliness/correctness and set the Textblock do the whole memory handling (i.e. now the Textblock is the one that allocates/deallocates the text string). The HTML parser was adjusted a bit to keep its old performance. -- Cheers Jorge.-

2 9

Splitting html.cc: what next?
by onepoint＠starurchin.org June 21, 2008

June 21, 2008

Off the top of my head, some suggestions for more files that could usefully be split out from html.cc: - table.cc , handles <table> , <tr> , <td> , <th> - list.cc , handles <ul> , <ol> , <li> , <dl> , <dt> , <dd> - text.cc , handles , , , , <strike> - paragraph.cc , handles , <blockquote> , <center> , <address> - page.cc , handles <html> , <head> , <title> , <meta> , <body> Thoughts? Regards, Jeremy Henty

3 7

occasional bus errors
by Johannes.Hofmann＠gmx.de June 20, 2008

June 20, 2008

Hi is anyone else seeing these rare bus errors? This is not new and happens every now and then. I'm trying to track it down, but it would be helpful to have a method to reliably reproduce it. Cheers, Johannes BTW: It's currently the only type of crash I noticed. Program terminated with signal 10, Bus error. #0 0x28595b1a in ?? () (gdb) bt #0 0x28595b1a in ?? () #1 0x0808ec51 in ~DeepIterator (this=0x28565720) at iterator.cc:515 #2 0x08091dc8 in dw::core::SelectionState::resetSelection (this=0x2850a47c) at selection.cc:85 #3 0x08091e20 in dw::core::SelectionState::reset (this=0x2850a47c) at selection.cc:75 #4 0x08091809 in dw::core::Layout::removeWidget (this=0x2850a400) at layout.cc:167 #5 0x080970ff in ~Widget (this=0x285aaf00) at widget.cc:269 #6 0x0807d554 in ~Textblock (this=0x285aaf00) at textblock.cc:117 #7 0x080916c7 in dw::core::Layout::setWidget (this=0x2850a400, widget=0x285b0300) at layout.cc:175 #8 0x08056ddc in a_Web_dispatch_by_type (Type=0x285a4560 "text/html; charset=us-ascii", Web=0x285659c0, Call=0x285958d0, Data=0x285958d4) at web.cc:94 #9 0x08058583 in Cache_process_queue (entry=0x28582400) at cache.c:1042 #10 0x08058d3c in a_Cache_process_dbuf (Op=0, buf=0x2873c000 "HTTP/1.1 200 OK\r\nDate: Fri, 20 Jun 2008 15:51:22 GMT\r\nServer: Apache/1.3.34 (Debian) PHP/4.4.4-8+etch6 mod_ssl/2.8.25 OpenSSL/0.9.8c\r\nCache-control: no-cache\r\nConnection: close\r\nTransfer-Encoding: chu"..., buf_size=1440, Url=0x285a1d00) at cache.c:871 #11 0x080564b8 in a_Chain_fcb (Op=134802368, Info=0x285a2e80, Data1=0x285c9ac0, Data2=0x80eb3e3) at chain.c:111 #12 0x080743dc in a_Dpi_ccc (Op=2, Branch=2, Dir=1, Info=0x28595b40, Data1=0x285c9200, Data2=0x0) at dpi.c:223 #13 0x080564b8 in a_Chain_fcb (Op=134802368, Info=0x285a2e80, Data1=0x285c9200, Data2=0x0) at chain.c:111 #14 0x08074cfd in a_IO_ccc (Op=2, Branch=2, Dir=1, Info=0x28595ba0, Data1=0x28595e00, Data2=0x0) at IO.c:418 #15 0x08074e8a in IO_callback (fd=<value optimized out>, io=0x28595e00) at IO.c:205 #16 0x08074f63 in IO_fd_read_cb (fd=4, data=0x20e) at IO.c:291 #17 0x080c28cd in fltk::wait () #18 0x080c2a7d in fltk::run () #19 0x0804e63d in main (argc=-1958539068, argv=0x58390446) at dillo.cc:125

4 4

[patch] move highlighting data into Textblock
by Johannes.Hofmann＠gmx.de June 20, 2008

June 20, 2008

Hi, as promised here comes a patch to move highlighting information from struct Word into Textblock to save memory. I had to change the semantics of the highlighting API slightly (see comments) to ensure that the highlighted part is always contiguous. Here are some measurements (with load_images=NO): Page dillo-fltk patched_dillo-fltk dillo1 firefox ============================================================== splash 7240K 7236K xslt [1] 15016K 14220K freebsd [2] 70264K 58084K mysql [3] 186M 148M 232M 200M Obviously the difference is mostly visible on large pages. And firefox is pretty close regarding memory usage. However it needed more than 3 minutes to load the local mysql page compared to 6 seconds for dillo-fltk! Please look out for possible highlight related regressions. Cheers, Johannes [1] http://www.w3.org/TR/xslt [2] http://www.freebsd.org/doc/en/books/handbook/book.html [3] http://downloads.mysql.com/docs/refman-6.0-en.html.tar.gz

3 8

easy way to get many root certificates?
by corvid＠lavabit.com June 20, 2008

June 20, 2008

I'm tired of clicking on the "remote certificate cannot be verified" message. I added a couple of lines to https.c so that it would save certificates. When I encountered the next https page, "openssl x509 -text -in [certificate]" showed that Equifax was the issuer. I got their root certificate, stuck it in .dillo/certs, found the hash value with "openssl x509 -hash -in [certificate]", and linked it to [hash].0. And, hey, no more message! But I don't want to do that over and over for all of the common root certificates. It seems that you can get the certificates out of firefox if you have some program called certutil, but, so far as I can tell, this requires getting a bunch of mozilla code known as NSS that is something like six megs of source. This is nowhere near worth the headaches that would surely result from trying to compile such a thing. So, does anyone happen to know an easy and convenient way to deal with this?

3 5

[patch] fix FltkPlatform destructor
by Johannes.Hofmann＠gmx.de June 19, 2008

June 19, 2008

Hi, this fixes a small memory leak in FltkPlatform. Johannes

2 1