- Dillo-dev - lists.mailman3.com

[patch] dw2 memory reduction

by Johannes.Hofmann＠gmx.de

On Sun, May 25, 2008 at 06:47:11PM +0200, Johannes Hofmann wrote: > On Sun, May 25, 2008 at 09:12:11AM -0400, Jorge Arellano Cid wrote: > > On Sun, May 25, 2008 at 10:08:12AM +0200, Johannes Hofmann wrote: > > > Hi, > > > > > > On Sat, May 24, 2008 at 04:39:42PM -0400, Jorge Arellano Cid wrote: > > > > On Sat, May 24, 2008 at 02:58:08PM +0200, Johannes Hofmann wrote: > > > > > Hi, > > > > > > > > > > attached is patch to reduce the memory usage of dw2: > > > > > > > > Excellent! > > > > (AFAIU this is near 30% memory usage reduction in dw2) > > > > > > > > Committed. > > > > > > > > > > > > > > * Styles are now reused if they are equal. This brings down the > > > > > number of Style objects on the mysql page from around 180000 to > > > > > 23000 and reduces memory usage by about 25M at the cost of an > > > > > additional HashTable. > > > > > Maybe the hash table should be part of fltkplatform > > > > > as the hash tables for Color and Font? > > > > > There would be even more sharing of styles possible, if > > > > > x_link would not be part of style. On the other hand about 3M for > > > > > styles on the huge mysql page is pretty ok. > > > > > > > > > > * Size of struct Word is reduced by 4 integers by using shorts. I > > > > > know that this is questionable, but I think in the case of struct > > > > > Word that can be allocated a lot it is worth the hassle. > > > > > Please check whether my assumption that space lengths and word > > > > > lengths can be stored in shorts are valid in general. > > > > > > > > It looks OK at first sight, but to be sure it'd be good to know > > > > what happens when the limit is surpassed. > > > > > > Just tried it on a page with 40000 char long word. With short for > > > hlStart/hlEnd, selection is - obviously - no longer possible. > > > That's clearly a regression. > > > Please revert with attached patch. > > > > A max of 32K for a single word, looks quite reasonable to me. > > I meant, if the browser doesn't crash, and fails gracefully, why > > not. > > > > BTW, it can be extended to 64K with unsigned, by using 65535 as > > sentinel value (instead of -1), but again, is it worth? > > > > It'd be good to know how much "struct Word" is reduced in size > > (after compiler alignment), and how much it impacts overall > > memory usage. I mean if struct word is reduced 20% and that > > yields a 4% overall memory reduction, it may be not worth > > complicating the data type. > > > > OTOH, ... > > On the very long mysql page, addWord is called 1192256 times. But > due to the allocation in 2^n steps in SimpleVector memory for > some more of them could be allocated > structs on 32bit systems are 8 byte aligned. struct Word with Argh, structs are 4byte aligned of course! Cheers, Johannes

May 26, 2008

Re: (working) prototype of charset decoding in cache.c

by corvid＠lavabit.com

Jorge wrote: > On Fri, May 02, 2008 at 08:02:14PM +0000, corvid wrote: > > I've been playing around with this now and then over the past few months > > when I haven't had anything else going on. I've been running on it for > > a day or so without incident, but I haven't gone through it with a > > fine-toothed comb because I know there will be changes. > > > > It's 35K, so I realize I'm in little danger of hearing back too quickly :) > > Besides, I know you said you have some nbsp thing that needs attention... > > I've made a second review over the code, and it doesn't look > that bad to me. This is, the decoding in cache, even the stuffed > cache-entry; if it can be isolated/tidied in the future better. hurray! > The part that looks inconvenient to me is the charset choosing > dialog. I'd prefer Dillo do this automatically. There's no need > to try to recognize every charset in the world, but just a few > "most used" ones like koi8-r. BTW, UTF-8 has become the preferred > solution. For one thing, I was thinking of a site I knew of where they had a Russian forum and the message board was too limited to deal with it, so they told people to change each page in that part to windows-1251 or something like that. But okay. That's obviously not any sort of fundamental requirement. (And because I liked changing to text/plain to have source in a proper bw.) > The idea is that we can leave space for a expandable function > just in case. BTW, there's an improved version of the data > guesser in the ftp dpi. Please send me a koi8-r encoded page to > test (or just let me know how to find it with google :). Here's one that's apparently about 0.7.0: http://gazette.linux.ru.net/lg79/arndt.html > IIRC you sent me an explanation message on what this patch was > for and remaining issues. I can't find it :(. Please resend it to > me. Oh, it was just...let's see... "> BTW, please explain what is/are the main point(s) of it. After > reading the comment it was a bit scary to apply. :-o It's not for applying. It's "here's what I was able to come up with to get encoding translation for page source and plain text, but there are obviously issues, so I need suggestions for... how to do what needs to be done in the most dillolike manner possible" or... or something like that..." > > There are plenty of issues. Going in order through the diff: > > > > - *_get_content_type(). I searched for some Russian text files where a > > charset is given in the HTTP header. None of them were regarded as > > text by the old test for ascii chars. > > Maybe the way to go is to have tests for more filetypes and treat > > unknown as text. Checking for all possible charsets would be no fun, > > anyway. (Tests for more filetypes might be needed for file uploading, > > as well.) > > The problem with "unknown as text" is that some HTTP servers > send downloadable files as text. It is quite practical in this > case to get a download dialog from dillo, intead of a binary > rendering (especially for newbies). That's why the current data > guesser is conservative. Quite true. > The data guesser can be improved. e.g. taking ASCCI control > characters as a strong indication of a binary, and their absence > as a text hint. I can give it a look after you send me the files. > > > > - CacheEntry_t now has a horrifying amount of junk in it. Maybe it > > would be good to pull out about half of it into some sort of Data thing. > > It's stuffed but still understandable. If it can be tidied in a > second pass, better. > > > o TypeSet to be set from Meta or a user dialog. Meta can set it once; > > the dialog can set it repeatedly. > > I'd like to avoid that dialog if possible. > > > o UTF8Data (LocalData?) that's been translated. cache.c has a lot of > > "entry->UTF8Data ? entry->UTF8Data : entry->Data" in it now that > > perhaps should be hidden behind something. > > Yes. > > > o DataReaders reference count. UTF8Data is destroyed when not needed. > > o CharsetDecoder > > > > - a_Cache_unref_buf, a_Capi_unref_buf, a_Nav_unref_buf all just to > > provide notification that the view source window isn't reading it anymore. > > Ugh. > > Not pretty, but tolerable. > > I'm thinking along the line of getting the core of the patch > committed and then polish the rough edges. > > > > - a_Cache_[gs]et_content_type and capi and nav. Ugh again. > > I don't know whether there's a timing issue with changing decoders > > due to a meta tag at the same time that a view source window is being > > opened. > > Yes there is. > > Simple to solve by allowing view source after parsing the HEAD > element only. Do you think badly-formed html with, e.g., <head>..</head><body>...</body><head><meta..charset..></head> <body>...</body> is worth worrying about? > > (Why content-type and not charset? I find switching to text/plain to > > use findtext on the source to be useful.) > > Sorry, I don't understand the question. I was saying that I was pushing around content-type instead of charset because I liked changing it to text/plain in the dialog. If there's no dialog, maybe I'll go back to charset. > > - Decode_charset() no longer free()s its input. This inconsistency would > > be an excellent source of bugs, so it would probably be good to make it so > > that none of the decoders free input. > > Yes, this is a must. > Consistency should be our goal/horizon/friend/helper. :-)

May 26, 2008

Re: (working) prototype of charset decoding in cache.c

by jcid＠dillo.org

Hi, On Fri, May 02, 2008 at 08:02:14PM +0000, corvid wrote: > I've been playing around with this now and then over the past few months > when I haven't had anything else going on. I've been running on it for > a day or so without incident, but I haven't gone through it with a > fine-toothed comb because I know there will be changes. > > It's 35K, so I realize I'm in little danger of hearing back too quickly :) > Besides, I know you said you have some nbsp thing that needs attention... I've made a second review over the code, and it doesn't look that bad to me. This is, the decoding in cache, even the stuffed cache-entry; if it can be isolated/tidied in the future better. The part that looks inconvenient to me is the charset choosing dialog. I'd prefer Dillo do this automatically. There's no need to try to recognize every charset in the world, but just a few "most used" ones like koi8-r. BTW, UTF-8 has become the preferred solution. The idea is that we can leave space for a expandable function just in case. BTW, there's an improved version of the data guesser in the ftp dpi. Please send me a koi8-r encoded page to test (or just let me know how to find it with google :). IIRC you sent me an explanation message on what this patch was for and remaining issues. I can't find it :(. Please resend it to me. > > There are plenty of issues. Going in order through the diff: > > - *_get_content_type(). I searched for some Russian text files where a > charset is given in the HTTP header. None of them were regarded as > text by the old test for ascii chars. > Maybe the way to go is to have tests for more filetypes and treat > unknown as text. Checking for all possible charsets would be no fun, > anyway. (Tests for more filetypes might be needed for file uploading, > as well.) The problem with "unknown as text" is that some HTTP servers send downloadable files as text. It is quite practical in this case to get a download dialog from dillo, intead of a binary rendering (especially for newbies). That's why the current data guesser is conservative. The data guesser can be improved. e.g. taking ASCCI control characters as a strong indication of a binary, and their absence as a text hint. I can give it a look after you send me the files. > - CacheEntry_t now has a horrifying amount of junk in it. Maybe it > would be good to pull out about half of it into some sort of Data thing. It's stuffed but still understandable. If it can be tidied in a second pass, better. > o TypeSet to be set from Meta or a user dialog. Meta can set it once; > the dialog can set it repeatedly. I'd like to avoid that dialog if possible. > o UTF8Data (LocalData?) that's been translated. cache.c has a lot of > "entry->UTF8Data ? entry->UTF8Data : entry->Data" in it now that > perhaps should be hidden behind something. Yes. > o DataReaders reference count. UTF8Data is destroyed when not needed. > o CharsetDecoder > > - a_Cache_unref_buf, a_Capi_unref_buf, a_Nav_unref_buf all just to > provide notification that the view source window isn't reading it anymore. > Ugh. Not pretty, but tolerable. I'm thinking along the line of getting the core of the patch committed and then polish the rough edges. > - a_Cache_[gs]et_content_type and capi and nav. Ugh again. > I don't know whether there's a timing issue with changing decoders > due to a meta tag at the same time that a view source window is being > opened. Yes there is. Simple to solve by allowing view source after parsing the HEAD element only. > (Why content-type and not charset? I find switching to text/plain to > use findtext on the source to be useful.) Sorry, I don't understand the question. > - Decode_charset() no longer free()s its input. This inconsistency would > be an excellent source of bugs, so it would probably be good to make it so > that none of the decoders free input. Yes, this is a must. Consistency should be our goal/horizon/friend/helper. :-) -- Cheers Jorge.-

May 26, 2008

Fuzzing dillo

by onepoint＠starurchin.org

On Mon, May 26, 2008 at 08:49:09AM -0400, Jorge Arellano Cid wrote: > Corvid has been working around forms lately so please try to > coordinate ideas with him OK. > > There's a lot of cut-and-paste code like "foo->get (foo->size () - > > 1)". Adding getLast() methods to some lout classes and using them > > wherever possible would remove a lot of clutter. > > OK. OK, that's next on the TODO. > Just beware of not converting the "horrible case code" into > "horrible virtual method subclassing". :-) To be honest I don't think I can promise that it will be worth doing until I have a go, I am just optimistic enough to want to have a go. > FLTK2 does this quite nicely with the virtual handle() method. Yes, that's a good model. I guess the analogy would be for the Input classes to have virtual update_form_from_html() and form_values() methods. There problem with creating Input objects as there is no such thing as a virtual constructor or virtual class method. So some case code might remain, but it would be hidden in the class API and the switch would simply select which derived class constructor to call. That's not so bad. Regards, Jeremy Henty

May 26, 2008

patch: http_proxyuser preference

by jcid＠dillo.org

On Sun, May 25, 2008 at 06:44:22PM +0000, corvid wrote: > After I sent the preferences mail yesterday, I found that I overlooked one. > > The headers look good, but I have no way to test it for real. Committed. (better to have it there than to leave it out waiting for testing). -- Cheers Jorge.-

May 26, 2008

Fuzzing dillo

by jcid＠dillo.org

On Sun, May 25, 2008 at 05:13:13PM +0100, Jeremy Henty wrote: > On Sun, May 25, 2008 at 10:31:23AM -0400, Jorge Arellano Cid wrote: > > On Sun, May 25, 2008 at 01:12:38PM +0200, Justus Winter wrote: > > > > Furthermore I'd suggest to break down html.cc into several files. > > > > The hard part is a good choice on what to separate. ;) > > I suggest splitting out everything that handles forms and form > elements. That's a fair amount of code that represents a reasonably > self-contained chunk. > > The forms code can also be be cleaned up in a few ways. It looks like you have a good set of ideas to start. Corvid has been working around forms lately so please try to coordinate ideas with him > There's a lot of cut-and-paste code like "foo->get (foo->size () - > 1)". Adding getLast() methods to some lout classes and using them > wherever possible would remove a lot of clutter. OK. > html.cc looks like a C file that was subsequently converted to C++ > because it makes little use of C++. html.cc is a C file that was subsequently converted to C++! :-) > If DilloHtmlForm and > DilloHtmlInput were given constructors > and destructors and managed > with new and delete then lots of allocation/deallocation code would be > neatly factored out instead of sitting inline with all the other > logic. OK 2. > The type element of DilloHtmlInput leads to much horrible case code. > If DilloHtmlInput were an abstract class and the different Input types > were subclasses then all the type-dependent code could go into virtual > methods. This would break up the huge case statements that make much > of this code hard to follow. > > I'll propose patches if people are positive about these suggestions. BTW: > [corvid wrote:] > It's at least definitely worth trying, I think. Just beware of not converting the "horrible case code" into "horrible virtual method subclassing". :-) FLTK2 does this quite nicely with the virtual handle() method. This is IMHO remarkable because the underlying event handling model is certainly complex, but the abstraction for it attains simplicity at the widget level. -- Cheers Jorge.-

May 26, 2008

Fuzzing dillo

by corvid＠lavabit.com

Jeremy wrote: > html.cc looks like a C file that was subsequently converted to C++ > because it makes little use of C++. If DilloHtmlForm and > DilloHtmlInput were given constructors and destructors and managed > with new and delete then lots of allocation/deallocation code would be > neatly factored out instead of sitting inline with all the other > logic. > > The type element of DilloHtmlInput leads to much horrible case code. > If DilloHtmlInput were an abstract class and the different Input types > were subclasses then all the type-dependent code could go into virtual > methods. This would break up the huge case statements that make much > of this code hard to follow. It's at least definitely worth trying, I think.

May 26, 2008

patch: goodbye to use_oblique and transient_dialogs

by jcid＠dillo.org

On Sun, May 25, 2008 at 08:19:05PM +0000, corvid wrote: > It's nice to get to delete things, and especially preferences, > since they have a natural tendency to grow and grow and grow. Committed. -- Cheers Jorge.-

May 26, 2008

remaining preferences

by jcid＠dillo.org

On Sun, May 25, 2008 at 08:32:02PM +0000, corvid wrote: > Jorge wrote: > > On Sat, May 24, 2008 at 07:28:47PM +0000, corvid wrote: > > > # Set this to YES, if you want to limit the word wrap width to the vieport > > > # width (may be useful for iPAQ) > > > limit_text_width > > > > > > It doesn't do anything. I don't think it's ever done anything. > > > It sure would be nice if it did something. But I don't have the expertise > > > to change that. > > > > For instance: Some times HTML pages for mailing lists render > > one looong line for a message. The MUA packs the message as a > > single line and the mail2html filter wraps it with PRE. AFAIR the > > SPEC said original formatting was to be honoured in PRE, but > > browsers, in practice, tend to wrap it. Sometimes you need one > > behaviour and then another. > > Are you saying that it does this, or that the intention is for it to do this? The intention is for it to do this. -- Cheers Jorge.-

May 26, 2008

remaining preferences

by corvid＠lavabit.com

Jorge wrote: > On Sat, May 24, 2008 at 07:28:47PM +0000, corvid wrote: > > # Set this to YES, if you want to limit the word wrap width to the vieport > > # width (may be useful for iPAQ) > > limit_text_width > > > > It doesn't do anything. I don't think it's ever done anything. > > It sure would be nice if it did something. But I don't have the expertise > > to change that. > > For instance: Some times HTML pages for mailing lists render > one looong line for a message. The MUA packs the message as a > single line and the mail2html filter wraps it with PRE. AFAIR the > SPEC said original formatting was to be honoured in PRE, but > browsers, in practice, tend to wrap it. Sometimes you need one > behaviour and then another. Are you saying that it does this, or that the intention is for it to do this?

May 25, 2008