- Dillo-dev - lists.mailman3.com

[patch]: html.cc cleanup: DilloHtml::currentForm()

by onepoint＠starurchin.org

Here's the next patch: it turns DilloHtmlForm into a class with a constructor and destructor to manage the data members. It depends on the previous patch. Note that this changes DilloHtml slightly to manage DilloHtmlForm pointers instead of DilloHtmlForm structs. This improves consistency as DilloHtml already manages DilloLinkImage and DilloUrl objects by pointers. Next I plan to refactor common code into DilloHtmlForm members just as I did with DilloHtml, then address DilloHtmlInput, DilloHtmlSelect and DilloHtmlOption in the same way. I expect this to be straightforward. Then we will be ready to consider moving the form handling code to a separate file and replacing the DilloHtmlInput case statements with virtual methods. This will probably be trickier and need some more planning. (I already tried once to move out the form handling code and gave up because it is so entangled. Hopefully the above cleanups will make it feasible.) Please comment on the patch and the plan, especially if you have an interest in how html.cc should be cleaned up. Regards, Jeremy Henty

May 26, 2008

Re: (working) prototype of charset decoding in cache.c

by jcid＠dillo.org

On Mon, May 26, 2008 at 03:01:53PM +0000, corvid wrote: > > > [...] > > > - a_Cache_[gs]et_content_type and capi and nav. Ugh again. > > > I don't know whether there's a timing issue with changing decoders > > > due to a meta tag at the same time that a view source window is being > > > opened. > > > > Yes there is. > > > > Simple to solve by allowing view source after parsing the HEAD > > element only. > > Do you think badly-formed html with, e.g., > <head>..</head><body>...</body><head><meta..charset..></head> <body>...</body> > is worth worrying about? This is already handled in the parser. But now that you mention this, we have to be careful with reload. It deletes the entry from the cache before making the request. A simple solution is just to close the view-source window in this corner case. > > > (Why content-type and not charset? I find switching to text/plain to > > > use findtext on the source to be useful.) > > > > Sorry, I don't understand the question. > > I was saying that I was pushing around content-type instead of charset > because I liked changing it to text/plain in the dialog. > If there's no dialog, maybe I'll go back to charset. "View source" as a bw is not a bad idea. It reuses code and adds findtext for free, but currently loses the line numbers. (which match nicely with the bug meter). This is a separate problem. One way to have both features in "View source" is with a dpi that gets the source from dillo and sends back an html page. The simple way is to prefix line numbers and wrap with PRE. It can be sent with a no-cache directive, or removed from cache upon window close. The problem of sending back and forth a huge page may be not as bad as it looks because huge pages are seldom inspected for source from a browser, and it can be handled anyway. -- Cheers Jorge.-

May 26, 2008

[patch] dw2 memory reduction

by joerg.sonnenberger＠web.de

On Mon, May 26, 2008 at 05:01:34PM +0200, Johannes Hofmann wrote: > > On the very long mysql page, addWord is called 1192256 times. But > > due to the allocation in 2^n steps in SimpleVector memory for > > some more of them could be allocated > > structs on 32bit systems are 8 byte aligned. struct Word with > > Argh, structs are 4byte aligned of course! On most sane architectures a structshave the same alignment as the strictest type in it. There's one important exception -- ARM, where the minimal alignment is 16 Byte (IIRC). Joerg

May 26, 2008

[patch]: html.cc cleanup: DilloHtml::currentForm()

by onepoint＠starurchin.org

This patch adds a DilloHtml::currentForm() method and uses it to refactor wherever possible. This reduces the number of instances of the "foo->size() - 1" idiom in html.cc from 23 to 13! Regards, Jeremy Henty

May 26, 2008

[patch] dw2 memory reduction

by Johannes.Hofmann＠gmx.de

On Sun, May 25, 2008 at 06:47:11PM +0200, Johannes Hofmann wrote: > On Sun, May 25, 2008 at 09:12:11AM -0400, Jorge Arellano Cid wrote: > > On Sun, May 25, 2008 at 10:08:12AM +0200, Johannes Hofmann wrote: > > > Hi, > > > > > > On Sat, May 24, 2008 at 04:39:42PM -0400, Jorge Arellano Cid wrote: > > > > On Sat, May 24, 2008 at 02:58:08PM +0200, Johannes Hofmann wrote: > > > > > Hi, > > > > > > > > > > attached is patch to reduce the memory usage of dw2: > > > > > > > > Excellent! > > > > (AFAIU this is near 30% memory usage reduction in dw2) > > > > > > > > Committed. > > > > > > > > > > > > > > * Styles are now reused if they are equal. This brings down the > > > > > number of Style objects on the mysql page from around 180000 to > > > > > 23000 and reduces memory usage by about 25M at the cost of an > > > > > additional HashTable. > > > > > Maybe the hash table should be part of fltkplatform > > > > > as the hash tables for Color and Font? > > > > > There would be even more sharing of styles possible, if > > > > > x_link would not be part of style. On the other hand about 3M for > > > > > styles on the huge mysql page is pretty ok. > > > > > > > > > > * Size of struct Word is reduced by 4 integers by using shorts. I > > > > > know that this is questionable, but I think in the case of struct > > > > > Word that can be allocated a lot it is worth the hassle. > > > > > Please check whether my assumption that space lengths and word > > > > > lengths can be stored in shorts are valid in general. > > > > > > > > It looks OK at first sight, but to be sure it'd be good to know > > > > what happens when the limit is surpassed. > > > > > > Just tried it on a page with 40000 char long word. With short for > > > hlStart/hlEnd, selection is - obviously - no longer possible. > > > That's clearly a regression. > > > Please revert with attached patch. > > > > A max of 32K for a single word, looks quite reasonable to me. > > I meant, if the browser doesn't crash, and fails gracefully, why > > not. > > > > BTW, it can be extended to 64K with unsigned, by using 65535 as > > sentinel value (instead of -1), but again, is it worth? > > > > It'd be good to know how much "struct Word" is reduced in size > > (after compiler alignment), and how much it impacts overall > > memory usage. I mean if struct word is reduced 20% and that > > yields a 4% overall memory reduction, it may be not worth > > complicating the data type. > > > > OTOH, ... > > On the very long mysql page, addWord is called 1192256 times. But > due to the allocation in 2^n steps in SimpleVector memory for > some more of them could be allocated > structs on 32bit systems are 8 byte aligned. struct Word with Argh, structs are 4byte aligned of course! Cheers, Johannes

May 26, 2008

Re: (working) prototype of charset decoding in cache.c

by corvid＠lavabit.com

Jorge wrote: > On Fri, May 02, 2008 at 08:02:14PM +0000, corvid wrote: > > I've been playing around with this now and then over the past few months > > when I haven't had anything else going on. I've been running on it for > > a day or so without incident, but I haven't gone through it with a > > fine-toothed comb because I know there will be changes. > > > > It's 35K, so I realize I'm in little danger of hearing back too quickly :) > > Besides, I know you said you have some nbsp thing that needs attention... > > I've made a second review over the code, and it doesn't look > that bad to me. This is, the decoding in cache, even the stuffed > cache-entry; if it can be isolated/tidied in the future better. hurray! > The part that looks inconvenient to me is the charset choosing > dialog. I'd prefer Dillo do this automatically. There's no need > to try to recognize every charset in the world, but just a few > "most used" ones like koi8-r. BTW, UTF-8 has become the preferred > solution. For one thing, I was thinking of a site I knew of where they had a Russian forum and the message board was too limited to deal with it, so they told people to change each page in that part to windows-1251 or something like that. But okay. That's obviously not any sort of fundamental requirement. (And because I liked changing to text/plain to have source in a proper bw.) > The idea is that we can leave space for a expandable function > just in case. BTW, there's an improved version of the data > guesser in the ftp dpi. Please send me a koi8-r encoded page to > test (or just let me know how to find it with google :). Here's one that's apparently about 0.7.0: http://gazette.linux.ru.net/lg79/arndt.html > IIRC you sent me an explanation message on what this patch was > for and remaining issues. I can't find it :(. Please resend it to > me. Oh, it was just...let's see... "> BTW, please explain what is/are the main point(s) of it. After > reading the comment it was a bit scary to apply. :-o It's not for applying. It's "here's what I was able to come up with to get encoding translation for page source and plain text, but there are obviously issues, so I need suggestions for... how to do what needs to be done in the most dillolike manner possible" or... or something like that..." > > There are plenty of issues. Going in order through the diff: > > > > - *_get_content_type(). I searched for some Russian text files where a > > charset is given in the HTTP header. None of them were regarded as > > text by the old test for ascii chars. > > Maybe the way to go is to have tests for more filetypes and treat > > unknown as text. Checking for all possible charsets would be no fun, > > anyway. (Tests for more filetypes might be needed for file uploading, > > as well.) > > The problem with "unknown as text" is that some HTTP servers > send downloadable files as text. It is quite practical in this > case to get a download dialog from dillo, intead of a binary > rendering (especially for newbies). That's why the current data > guesser is conservative. Quite true. > The data guesser can be improved. e.g. taking ASCCI control > characters as a strong indication of a binary, and their absence > as a text hint. I can give it a look after you send me the files. > > > > - CacheEntry_t now has a horrifying amount of junk in it. Maybe it > > would be good to pull out about half of it into some sort of Data thing. > > It's stuffed but still understandable. If it can be tidied in a > second pass, better. > > > o TypeSet to be set from Meta or a user dialog. Meta can set it once; > > the dialog can set it repeatedly. > > I'd like to avoid that dialog if possible. > > > o UTF8Data (LocalData?) that's been translated. cache.c has a lot of > > "entry->UTF8Data ? entry->UTF8Data : entry->Data" in it now that > > perhaps should be hidden behind something. > > Yes. > > > o DataReaders reference count. UTF8Data is destroyed when not needed. > > o CharsetDecoder > > > > - a_Cache_unref_buf, a_Capi_unref_buf, a_Nav_unref_buf all just to > > provide notification that the view source window isn't reading it anymore. > > Ugh. > > Not pretty, but tolerable. > > I'm thinking along the line of getting the core of the patch > committed and then polish the rough edges. > > > > - a_Cache_[gs]et_content_type and capi and nav. Ugh again. > > I don't know whether there's a timing issue with changing decoders > > due to a meta tag at the same time that a view source window is being > > opened. > > Yes there is. > > Simple to solve by allowing view source after parsing the HEAD > element only. Do you think badly-formed html with, e.g., <head>..</head><body>...</body><head><meta..charset..></head> <body>...</body> is worth worrying about? > > (Why content-type and not charset? I find switching to text/plain to > > use findtext on the source to be useful.) > > Sorry, I don't understand the question. I was saying that I was pushing around content-type instead of charset because I liked changing it to text/plain in the dialog. If there's no dialog, maybe I'll go back to charset. > > - Decode_charset() no longer free()s its input. This inconsistency would > > be an excellent source of bugs, so it would probably be good to make it so > > that none of the decoders free input. > > Yes, this is a must. > Consistency should be our goal/horizon/friend/helper. :-)

May 26, 2008

Re: (working) prototype of charset decoding in cache.c

by jcid＠dillo.org

Hi, On Fri, May 02, 2008 at 08:02:14PM +0000, corvid wrote: > I've been playing around with this now and then over the past few months > when I haven't had anything else going on. I've been running on it for > a day or so without incident, but I haven't gone through it with a > fine-toothed comb because I know there will be changes. > > It's 35K, so I realize I'm in little danger of hearing back too quickly :) > Besides, I know you said you have some nbsp thing that needs attention... I've made a second review over the code, and it doesn't look that bad to me. This is, the decoding in cache, even the stuffed cache-entry; if it can be isolated/tidied in the future better. The part that looks inconvenient to me is the charset choosing dialog. I'd prefer Dillo do this automatically. There's no need to try to recognize every charset in the world, but just a few "most used" ones like koi8-r. BTW, UTF-8 has become the preferred solution. The idea is that we can leave space for a expandable function just in case. BTW, there's an improved version of the data guesser in the ftp dpi. Please send me a koi8-r encoded page to test (or just let me know how to find it with google :). IIRC you sent me an explanation message on what this patch was for and remaining issues. I can't find it :(. Please resend it to me. > > There are plenty of issues. Going in order through the diff: > > - *_get_content_type(). I searched for some Russian text files where a > charset is given in the HTTP header. None of them were regarded as > text by the old test for ascii chars. > Maybe the way to go is to have tests for more filetypes and treat > unknown as text. Checking for all possible charsets would be no fun, > anyway. (Tests for more filetypes might be needed for file uploading, > as well.) The problem with "unknown as text" is that some HTTP servers send downloadable files as text. It is quite practical in this case to get a download dialog from dillo, intead of a binary rendering (especially for newbies). That's why the current data guesser is conservative. The data guesser can be improved. e.g. taking ASCCI control characters as a strong indication of a binary, and their absence as a text hint. I can give it a look after you send me the files. > - CacheEntry_t now has a horrifying amount of junk in it. Maybe it > would be good to pull out about half of it into some sort of Data thing. It's stuffed but still understandable. If it can be tidied in a second pass, better. > o TypeSet to be set from Meta or a user dialog. Meta can set it once; > the dialog can set it repeatedly. I'd like to avoid that dialog if possible. > o UTF8Data (LocalData?) that's been translated. cache.c has a lot of > "entry->UTF8Data ? entry->UTF8Data : entry->Data" in it now that > perhaps should be hidden behind something. Yes. > o DataReaders reference count. UTF8Data is destroyed when not needed. > o CharsetDecoder > > - a_Cache_unref_buf, a_Capi_unref_buf, a_Nav_unref_buf all just to > provide notification that the view source window isn't reading it anymore. > Ugh. Not pretty, but tolerable. I'm thinking along the line of getting the core of the patch committed and then polish the rough edges. > - a_Cache_[gs]et_content_type and capi and nav. Ugh again. > I don't know whether there's a timing issue with changing decoders > due to a meta tag at the same time that a view source window is being > opened. Yes there is. Simple to solve by allowing view source after parsing the HEAD element only. > (Why content-type and not charset? I find switching to text/plain to > use findtext on the source to be useful.) Sorry, I don't understand the question. > - Decode_charset() no longer free()s its input. This inconsistency would > be an excellent source of bugs, so it would probably be good to make it so > that none of the decoders free input. Yes, this is a must. Consistency should be our goal/horizon/friend/helper. :-) -- Cheers Jorge.-

May 26, 2008

Fuzzing dillo

by onepoint＠starurchin.org

On Mon, May 26, 2008 at 08:49:09AM -0400, Jorge Arellano Cid wrote: > Corvid has been working around forms lately so please try to > coordinate ideas with him OK. > > There's a lot of cut-and-paste code like "foo->get (foo->size () - > > 1)". Adding getLast() methods to some lout classes and using them > > wherever possible would remove a lot of clutter. > > OK. OK, that's next on the TODO. > Just beware of not converting the "horrible case code" into > "horrible virtual method subclassing". :-) To be honest I don't think I can promise that it will be worth doing until I have a go, I am just optimistic enough to want to have a go. > FLTK2 does this quite nicely with the virtual handle() method. Yes, that's a good model. I guess the analogy would be for the Input classes to have virtual update_form_from_html() and form_values() methods. There problem with creating Input objects as there is no such thing as a virtual constructor or virtual class method. So some case code might remain, but it would be hidden in the class API and the switch would simply select which derived class constructor to call. That's not so bad. Regards, Jeremy Henty

May 26, 2008

patch: http_proxyuser preference

by jcid＠dillo.org

On Sun, May 25, 2008 at 06:44:22PM +0000, corvid wrote: > After I sent the preferences mail yesterday, I found that I overlooked one. > > The headers look good, but I have no way to test it for real. Committed. (better to have it there than to leave it out waiting for testing). -- Cheers Jorge.-

May 26, 2008

Fuzzing dillo

by jcid＠dillo.org

On Sun, May 25, 2008 at 05:13:13PM +0100, Jeremy Henty wrote: > On Sun, May 25, 2008 at 10:31:23AM -0400, Jorge Arellano Cid wrote: > > On Sun, May 25, 2008 at 01:12:38PM +0200, Justus Winter wrote: > > > > Furthermore I'd suggest to break down html.cc into several files. > > > > The hard part is a good choice on what to separate. ;) > > I suggest splitting out everything that handles forms and form > elements. That's a fair amount of code that represents a reasonably > self-contained chunk. > > The forms code can also be be cleaned up in a few ways. It looks like you have a good set of ideas to start. Corvid has been working around forms lately so please try to coordinate ideas with him > There's a lot of cut-and-paste code like "foo->get (foo->size () - > 1)". Adding getLast() methods to some lout classes and using them > wherever possible would remove a lot of clutter. OK. > html.cc looks like a C file that was subsequently converted to C++ > because it makes little use of C++. html.cc is a C file that was subsequently converted to C++! :-) > If DilloHtmlForm and > DilloHtmlInput were given constructors > and destructors and managed > with new and delete then lots of allocation/deallocation code would be > neatly factored out instead of sitting inline with all the other > logic. OK 2. > The type element of DilloHtmlInput leads to much horrible case code. > If DilloHtmlInput were an abstract class and the different Input types > were subclasses then all the type-dependent code could go into virtual > methods. This would break up the huge case statements that make much > of this code hard to follow. > > I'll propose patches if people are positive about these suggestions. BTW: > [corvid wrote:] > It's at least definitely worth trying, I think. Just beware of not converting the "horrible case code" into "horrible virtual method subclassing". :-) FLTK2 does this quite nicely with the virtual handle() method. This is IMHO remarkable because the underlying event handling model is certainly complex, but the abstraction for it attains simplicity at the widget level. -- Cheers Jorge.-

May 26, 2008