- Dillo-dev - lists.mailman3.com

Cookies performance

by corvid＠lavabit.com

Johannes wrote: > On Mon, Jan 18, 2010 at 03:58:25PM +0900, furaisanjin wrote: > > 2010/1/18 Johannes Hofmann <Johannes.Hofmann(a)gmx.de>: > > > Please test attached patch. It limits the overall number of > > > concurrent connections to 4. > > > > I tested this with TCP_NODELAY option and it seems that further image > > downloading stops when the page repush happens due to remote CSS. So I > > don't see all images. Good thing is that I don't get the error > > message. > > Hm, it seems to work ok here... > > Does loading other pages still work, after image loading has > stalled, or is the complete dillo instance broken after that? > > Do you have a fast or slow internet connection? > > Do you know what kind of proxy you are connecting to? I've tried with image loading and remote stylesheets on to the degree that I can tolerate :) , and I like this connection limiting a whole lot. So far, all I've noticed was once getting one of those "IO_write, closing with pending data not sent" msgs when I left a page while things were loading. It didn't seem to cause any harm, though.

Jan. 18, 2010

Cookies performance

by Johannes.Hofmann＠gmx.de

On Mon, Jan 18, 2010 at 03:58:25PM +0900, furaisanjin wrote: > 2010/1/18 Johannes Hofmann <Johannes.Hofmann(a)gmx.de>: > > Please test attached patch. It limits the overall number of > > concurrent connections to 4. > > I tested this with TCP_NODELAY option and it seems that further image > downloading stops when the page repush happens due to remote CSS. So I > don't see all images. Good thing is that I don't get the error > message. Hm, it seems to work ok here... Does loading other pages still work, after image loading has stalled, or is the complete dillo instance broken after that? Do you have a fast or slow internet connection? Do you know what kind of proxy you are connecting to? Regards, Johannes

Jan. 18, 2010

Cookies performance

by furaisanjin＠gmail.com

2010/1/18 Johannes Hofmann <Johannes.Hofmann(a)gmx.de>: > Please test attached patch. It limits the overall number of > concurrent connections to 4. I tested this with TCP_NODELAY option and it seems that further image downloading stops when the page repush happens due to remote CSS. So I don't see all images. Good thing is that I don't get the error message. Regards, furaisanjin

Jan. 18, 2010

Cookies performance

by Johannes.Hofmann＠gmx.de

On Fri, Jan 15, 2010 at 01:09:39PM +0100, Johannes Hofmann wrote: > On Fri, Jan 15, 2010 at 06:20:41PM +0900, furaisanjin wrote: > > 2010/1/14 Johannes Hofmann <Johannes.Hofmann(a)gmx.de>: > > > Ok, could you please try this combination: > > > > > > - TCP_NODELAY not included (e.g. hg up 239f9fc6c313) > > > - cookies disabled > > > - proxy is used > > > > > > My guess is that this combination also shows the problem, but of > > > course we need to verify this. > > > > Yes, I see the same error messages with above combination, so > > definitely this is communication problem between dillo and my local > > proxy. I guess that dillo creates socket connection too often. > > Ok good. I'm working on a connection limit right now and hope to have > something to test soon. Please test attached patch. It limits the overall number of concurrent connections to 4. Cheers, Johannes

Jan. 17, 2010

some code for better cookie rejection

by corvid＠lavabit.com

Yesterday I made a rough draft of some code to allow the use of the http://publicsuffix.org/list/ to determine which domains dillo shouldn't allow cookies to be set for: http://www.dillo.org/test/public_suffix.diff I just put it there for safekeeping for now, as I have no plans to clean it up and get it into dillo until after the next release, but it occurred to me that I should at least mention it's there for anyone who uses cookies and lives somewhere where the domain situation is complicated (e.g., Japan). A user would have to add a line to cookiesrc like public_suffix_file /path/to/list/file

Jan. 16, 2010

Cookies performance

by Johannes.Hofmann＠gmx.de

On Fri, Jan 15, 2010 at 06:20:41PM +0900, furaisanjin wrote: > 2010/1/14 Johannes Hofmann <Johannes.Hofmann(a)gmx.de>: > > Ok, could you please try this combination: > > > > - TCP_NODELAY not included (e.g. hg up 239f9fc6c313) > > - cookies disabled > > - proxy is used > > > > My guess is that this combination also shows the problem, but of > > course we need to verify this. > > Yes, I see the same error messages with above combination, so > definitely this is communication problem between dillo and my local > proxy. I guess that dillo creates socket connection too often. Ok good. I'm working on a connection limit right now and hope to have something to test soon. Thanks for the continuous testing, Johannes

Jan. 15, 2010

Cookies performance

by furaisanjin＠gmail.com

2010/1/15 corvid <corvid(a)lavabit.com>: > Okay, committed a fix for --disable-cookies. I checked myself and now I can compile with disabling cookies. Regards, furaisanjin

Jan. 15, 2010

Cookies performance

by furaisanjin＠gmail.com

2010/1/14 Johannes Hofmann <Johannes.Hofmann(a)gmx.de>: > Ok, could you please try this combination: > > - TCP_NODELAY not included (e.g. hg up 239f9fc6c313) > - cookies disabled > - proxy is used > > My guess is that this combination also shows the problem, but of > course we need to verify this. Yes, I see the same error messages with above combination, so definitely this is communication problem between dillo and my local proxy. I guess that dillo creates socket connection too often. Regards, furaisanjin

Jan. 15, 2010

Cookies performance

by corvid＠lavabit.com

furaisanjin wrote: > I did gmake clean and then ./configure --disable-cookies. I compiled > codes and found that I had to modify like this. Okay, committed a fix for --disable-cookies.

Jan. 14, 2010

Cookies performance

by Johannes.Hofmann＠gmx.de

On Thu, Jan 14, 2010 at 11:04:03PM +0900, furaisanjin wrote: > 2010/1/14 Johannes Hofmann <Johannes.Hofmann(a)gmx.de>: > > Is this without TCP_NODELAY? > > The code contains TCP_NODELAY since I don't have any problem if I > remove TCP_NODELAY. > > Now I tried > - cookie is enabled > - TCP_NODELAY is included > - no proxy is used > > Then I don't any see error message. So the problem seems to be > communication between dillo and my local proxy. Ok, could you please try this combination: - TCP_NODELAY not included (e.g. hg up 239f9fc6c313) - cookies disabled - proxy is used My guess is that this combination also shows the problem, but of course we need to verify this. Regards, Johannes

Jan. 14, 2010

Cookies performance

by furaisanjin＠gmail.com

2010/1/14 Johannes Hofmann <Johannes.Hofmann(a)gmx.de>: > Is this without TCP_NODELAY? The code contains TCP_NODELAY since I don't have any problem if I remove TCP_NODELAY. Now I tried - cookie is enabled - TCP_NODELAY is included - no proxy is used Then I don't any see error message. So the problem seems to be communication between dillo and my local proxy. Regards, furaisanjin

Jan. 14, 2010

Cookies performance

by Johannes.Hofmann＠gmx.de

On Thu, Jan 14, 2010 at 09:40:58PM +0900, furaisanjin wrote: > 2010/1/14 Johannes Hofmann <Johannes.Hofmann(a)gmx.de>: > > Could you try with cookies disabled and a dillo version before > > the TCP_NODELAY change? > > I did gmake clean and then ./configure --disable-cookies. I compiled > codes and found that I had to modify like this. > > diff -r 9049a634eb0d src/cookies.c > --- a/src/cookies.c Wed Jan 13 15:27:56 2010 +0900 > +++ b/src/cookies.c Thu Jan 14 21:33:37 2010 +0900 > @@ -22,7 +22,11 @@ > */ > void a_Cookies_init(void) > { > - MSG("Cookies: absolutely disabled at compilation time.\n"); > + printf("Cookies: absolutely disabled at compilation time.\n"); > +} > + > +void a_Cookies_freeall() > +{ > } > > I stopped dpid and installed cookie dpi just in case, then tried dillo > again. I feel I see more error messages by this change. Is this without TCP_NODELAY? If so it would indicate that the real problem is the massive number of connections dillo is establishing if there are man images on a page and some servers / proxies don't like it. We probabely need to look at the experimental patch Jorge had posted here: http://lists.auriga.wearlab.de/pipermail/dillo-dev/2009-August/006778.html Cheers, Johannes

Jan. 14, 2010

Cookies performance

by furaisanjin＠gmail.com

2010/1/14 Johannes Hofmann <Johannes.Hofmann(a)gmx.de>: > Could you try with cookies disabled and a dillo version before > the TCP_NODELAY change? I did gmake clean and then ./configure --disable-cookies. I compiled codes and found that I had to modify like this. diff -r 9049a634eb0d src/cookies.c --- a/src/cookies.c Wed Jan 13 15:27:56 2010 +0900 +++ b/src/cookies.c Thu Jan 14 21:33:37 2010 +0900 @@ -22,7 +22,11 @@ */ void a_Cookies_init(void) { - MSG("Cookies: absolutely disabled at compilation time.\n"); + printf("Cookies: absolutely disabled at compilation time.\n"); +} + +void a_Cookies_freeall() +{ } I stopped dpid and installed cookie dpi just in case, then tried dillo again. I feel I see more error messages by this change. Regards, furaisanjin

Jan. 14, 2010

Cookies performance

by Johannes.Hofmann＠gmx.de

On Thu, Jan 14, 2010 at 08:24:54PM +0900, furaisanjin wrote: > 2010/1/14 Johannes Hofmann <Johannes.Hofmann(a)gmx.de>: > > What page is that, or does it happen on all pages with many images? > > Is it using https? > > There would be more pages but these are just examples. > > http://www.hatena.ne.jp/ > http://f.hatena.ne.jp/hotfoto > > The first one doesn't contain so many images but relatively many and > the second one contains lots of images. I don't think this is the site > specific issue. Both seem to work fine here. What happens if you disable cookies? Could you try with cookies disabled and a dillo version before the TCP_NODELAY change? I remember that we once wanted to limit concurrent connections to a reasonable number and maybe the slow DPI communication is just doing that. So I suspect that without cookies even the version without TCP_NODELAY has issues - but that's just a wild guess. Regards, Johannes

Jan. 14, 2010

Cookies performance

by furaisanjin＠gmail.com

2010/1/14 Johannes Hofmann <Johannes.Hofmann(a)gmx.de>: > What page is that, or does it happen on all pages with many images? > Is it using https? There would be more pages but these are just examples. http://www.hatena.ne.jp/ http://f.hatena.ne.jp/hotfoto The first one doesn't contain so many images but relatively many and the second one contains lots of images. I don't think this is the site specific issue. Regards, furaisanjin

Jan. 14, 2010

Cookies performance

by Johannes.Hofmann＠gmx.de

On Thu, Jan 14, 2010 at 11:25:31AM +0900, furaisanjin wrote: > Hi all. > > This may be FreeBSD (8.0R) specific or my local proxy specific but if > src/IO/dpi.c contains TCP_NODELAY, I got lots of this error messages > when the page contains lots of pictures. > > Http_connect_socket ERROR: Connection reset by peer What page is that, or does it happen on all pages with many images? Is it using https? > > so that several pictures aren't drawn. I agree that page rendering is > much faster if TCP_NODELAY is set but that isn't good on my > environment. We need find the issue for sure, but generally TCP_NODELAY should not cause any connection resets, so I think there is a bug somewhere that get's triggered by the different timing now. Cheers, Johannes

Jan. 14, 2010

Cookies performance

by furaisanjin＠gmail.com

Hi all. This may be FreeBSD (8.0R) specific or my local proxy specific but if src/IO/dpi.c contains TCP_NODELAY, I got lots of this error messages when the page contains lots of pictures. Http_connect_socket ERROR: Connection reset by peer so that several pictures aren't drawn. I agree that page rendering is much faster if TCP_NODELAY is set but that isn't good on my environment. Regards, furaisanjin

Jan. 14, 2010

release process (Re: [Dillo-dev] Re: even more cookies SECURITY)

by Johannes.Hofmann＠gmx.de

On Wed, Jan 13, 2010 at 07:54:52AM +0000, corvid wrote: > Jorge wrote: > > On Mon, Jan 04, 2010 at 11:38:03PM +0000, corvid wrote: > > > Do we currently have anything that needs to be > > > fixed/implemented before a release would be sensible? > > > > Not that I remember. The only problem I recall was the error > > message you reported with cookies and later seemed to be working > > somehow. > > > > Besides, AFAIS the repo is stable enough to start rolling the > > release process. > > The matter of the display code prompts the question of where we want > to draw a feature freeze line. Yes, that's the question. I think the display stuff will need about three weeks until we can release it. If we want to release earlier I would leave it out. Cheers, Johannes

Jan. 13, 2010

release process (Re: [Dillo-dev] Re: even more cookies SECURITY)

by corvid＠lavabit.com

Jorge wrote: > On Mon, Jan 04, 2010 at 11:38:03PM +0000, corvid wrote: > > Do we currently have anything that needs to be > > fixed/implemented before a release would be sensible? > > Not that I remember. The only problem I recall was the error > message you reported with cookies and later seemed to be working > somehow. > > Besides, AFAIS the repo is stable enough to start rolling the > release process. The matter of the display code prompts the question of where we want to draw a feature freeze line.

Jan. 13, 2010

Cookies performance

by corvid＠lavabit.com

Johannes wrote: > Can you please test attached patch, whether it compiles and makes a > difference for you? I speeds things up quite a bit here. We might > also want to cache SecretKey in a_Dpip_check_auth(). It sure seems faster to me!

Jan. 13, 2010

Cookies performance

by Johannes.Hofmann＠gmx.de

On Tue, Jan 12, 2010 at 09:46:01PM +0100, Johannes Hofmann wrote: > On Tue, Jan 12, 2010 at 07:06:12PM +0000, corvid wrote: > > Johannes wrote: > > > On Tue, Jan 12, 2010 at 05:42:03PM +0000, corvid wrote: > > > > Johannes wrote: > > > > > I noticed that since the recent DPI update pages which use > > > > > cookies load slower (e.g. fltk.org) > > > > > Do others also see that? It could also be OS specific. > > > > > > > > I thought it seemed weirdly slow _sometimes_. > > > > I'll have to throw some gprof at it and hope for something > > > > simple and embarrassing. > > > > > > > > (I wonder whether it would be easy to do valgrind with dpid and the > > > > plugins.) > > > > > > I did some truss on the cookies dpi and found that it is reading > > > ~/.dillo/dpid_comm_keys quite often. Maybe that is related. > > > > It looks like it checks it through a_Dpip_check_auth() when it > > gets or sets a msg. As for whether it needs to, I don't know. > > > > I tried a little gprof, but a modest amount of browsing and > > accepting cookies wasn't enough for the dpi to use up enough > > time for gprof to generate stats. > > Yah, I don't think it's CPU bound. There must be something else. I > will have a look at it. Can you please test attached patch, whether it compiles and makes a difference for you? I speeds things up quite a bit here. We might also want to cache SecretKey in a_Dpip_check_auth(). Cheers, Johannes

Jan. 12, 2010

title attrs (Re: [Dillo-dev] display: inline)

by corvid＠lavabit.com

Johannes wrote: > On Tue, Jan 12, 2010 at 08:43:41PM +0000, corvid wrote: > > At the mention of setNonCssHints, I was hoping that whatever the > > prototype did would magically make it so that I could implement tooltips > > on arbitrary elements easily, but no such luck. > > Is the title attribute allowed on all elements and always results in > a tooltip? If so, we could rearrange things to parse common > attributes in a special function. title is part of %coreattrs along with id, class, and style. (and then there's also a title attr on the STYLE element) %coreattrs is part of %attrs, And what has %coreattrs or %attrs? Most things. As for behaviour, the spec says 'visual browsers frequently display the title as a "tool tip"'.

Jan. 12, 2010

display: inline

by Johannes.Hofmann＠gmx.de

On Tue, Jan 12, 2010 at 08:43:41PM +0000, corvid wrote: > Johannes wrote: > > On Tue, Dec 08, 2009 at 06:56:55PM +0100, Johannes Hofmann wrote: > > > Hi, > > > > > > I started to experiment with the real solution, which means that the > > > display member of Style determines whether a (Text)block is created > > > or a ListItem, or whether just normal text is added. > > > The patch is still small, but as the change is pretty > > > fundamental, so I decided to put it in a separate repo for now. > > > You can find it at: > > > > > > http://www.flpsed.org/hgweb/dillo_display/ > > > > > > > I've updated this prototype a bit. It works quite well, the only > > drawback is the added complexity in html.cc. > > In addition to Html_tag_open_*() and Html_tag_clone_*(), there now is > > an optional Html_tag_content_*() function for tags that create > > widgets or other content. This function is not called in the > > display:none case. > > The reason for the splitting into Html_tag_open_*() and > > Html_tag_content_*() is that styleEngine->setNonCssHints() is > > needed before we can get the current style. > > Only after that is done, we know the value of the display property > > and can decide whether to call Html_tag_content_*() or not. > > > > I'd like to get this into main line soon if you agree. > > Please let me know what you think. > > Maybe if Tags[] gets complicated enough, it will become possible to > talk Jorge into using NULL for the other default fields as well. > > At the mention of setNonCssHints, I was hoping that whatever the > prototype did would magically make it so that I could implement tooltips > on arbitrary elements easily, but no such luck. Is the title attribute allowed on all elements and always results in a tooltip? If so, we could rearrange things to parse common attributes in a special function.

Jan. 12, 2010

Cookies performance

by Johannes.Hofmann＠gmx.de

On Tue, Jan 12, 2010 at 07:06:12PM +0000, corvid wrote: > Johannes wrote: > > On Tue, Jan 12, 2010 at 05:42:03PM +0000, corvid wrote: > > > Johannes wrote: > > > > I noticed that since the recent DPI update pages which use > > > > cookies load slower (e.g. fltk.org) > > > > Do others also see that? It could also be OS specific. > > > > > > I thought it seemed weirdly slow _sometimes_. > > > I'll have to throw some gprof at it and hope for something > > > simple and embarrassing. > > > > > > (I wonder whether it would be easy to do valgrind with dpid and the > > > plugins.) > > > > I did some truss on the cookies dpi and found that it is reading > > ~/.dillo/dpid_comm_keys quite often. Maybe that is related. > > It looks like it checks it through a_Dpip_check_auth() when it > gets or sets a msg. As for whether it needs to, I don't know. > > I tried a little gprof, but a modest amount of browsing and > accepting cookies wasn't enough for the dpi to use up enough > time for gprof to generate stats. Yah, I don't think it's CPU bound. There must be something else. I will have a look at it.

Jan. 12, 2010

display: inline

by corvid＠lavabit.com

Johannes wrote: > On Tue, Dec 08, 2009 at 06:56:55PM +0100, Johannes Hofmann wrote: > > Hi, > > > > I started to experiment with the real solution, which means that the > > display member of Style determines whether a (Text)block is created > > or a ListItem, or whether just normal text is added. > > The patch is still small, but as the change is pretty > > fundamental, so I decided to put it in a separate repo for now. > > You can find it at: > > > > http://www.flpsed.org/hgweb/dillo_display/ > > > > I've updated this prototype a bit. It works quite well, the only > drawback is the added complexity in html.cc. > In addition to Html_tag_open_*() and Html_tag_clone_*(), there now is > an optional Html_tag_content_*() function for tags that create > widgets or other content. This function is not called in the > display:none case. > The reason for the splitting into Html_tag_open_*() and > Html_tag_content_*() is that styleEngine->setNonCssHints() is > needed before we can get the current style. > Only after that is done, we know the value of the display property > and can decide whether to call Html_tag_content_*() or not. > > I'd like to get this into main line soon if you agree. > Please let me know what you think. Maybe if Tags[] gets complicated enough, it will become possible to talk Jorge into using NULL for the other default fields as well. At the mention of setNonCssHints, I was hoping that whatever the prototype did would magically make it so that I could implement tooltips on arbitrary elements easily, but no such luck.

Jan. 12, 2010

Cookies performance

by corvid＠lavabit.com

Johannes wrote: > On Tue, Jan 12, 2010 at 05:42:03PM +0000, corvid wrote: > > Johannes wrote: > > > I noticed that since the recent DPI update pages which use > > > cookies load slower (e.g. fltk.org) > > > Do others also see that? It could also be OS specific. > > > > I thought it seemed weirdly slow _sometimes_. > > I'll have to throw some gprof at it and hope for something > > simple and embarrassing. > > > > (I wonder whether it would be easy to do valgrind with dpid and the > > plugins.) > > I did some truss on the cookies dpi and found that it is reading > ~/.dillo/dpid_comm_keys quite often. Maybe that is related. It looks like it checks it through a_Dpip_check_auth() when it gets or sets a msg. As for whether it needs to, I don't know. I tried a little gprof, but a modest amount of browsing and accepting cookies wasn't enough for the dpi to use up enough time for gprof to generate stats.

Jan. 12, 2010

Cookies performance

by Johannes.Hofmann＠gmx.de

On Tue, Jan 12, 2010 at 05:42:03PM +0000, corvid wrote: > Johannes wrote: > > I noticed that since the recent DPI update pages which use > > cookies load slower (e.g. fltk.org) > > Do others also see that? It could also be OS specific. > > I thought it seemed weirdly slow _sometimes_. > I'll have to throw some gprof at it and hope for something > simple and embarrassing. > > (I wonder whether it would be easy to do valgrind with dpid and the > plugins.) I did some truss on the cookies dpi and found that it is reading ~/.dillo/dpid_comm_keys quite often. Maybe that is related.

Jan. 12, 2010

display: inline

by Johannes.Hofmann＠gmx.de

On Tue, Dec 08, 2009 at 06:56:55PM +0100, Johannes Hofmann wrote: > Hi, > > I started to experiment with the real solution, which means that the > display member of Style determines whether a (Text)block is created > or a ListItem, or whether just normal text is added. > The patch is still small, but as the change is pretty > fundamental, so I decided to put it in a separate repo for now. > You can find it at: > > http://www.flpsed.org/hgweb/dillo_display/ > I've updated this prototype a bit. It works quite well, the only drawback is the added complexity in html.cc. In addition to Html_tag_open_*() and Html_tag_clone_*(), there now is an optional Html_tag_content_*() function for tags that create widgets or other content. This function is not called in the display:none case. The reason for the splitting into Html_tag_open_*() and Html_tag_content_*() is that styleEngine->setNonCssHints() is needed before we can get the current style. Only after that is done, we know the value of the display property and can decide whether to call Html_tag_content_*() or not. I'd like to get this into main line soon if you agree. Please let me know what you think. Cheers, Johannes

Jan. 12, 2010

Cookies performance

by corvid＠lavabit.com

Johannes wrote: > I noticed that since the recent DPI update pages which use > cookies load slower (e.g. fltk.org) > Do others also see that? It could also be OS specific. I thought it seemed weirdly slow _sometimes_. I'll have to throw some gprof at it and hope for something simple and embarrassing. (I wonder whether it would be easy to do valgrind with dpid and the plugins.)

Jan. 12, 2010

web bug save?

by corvid＠lavabit.com

I wrote: > I wrote: > > I wrote: > > > Johannes wrote: > > > > On Thu, Dec 31, 2009 at 12:27:36AM -0300, Jorge Arellano Cid wrote: > > > > > Sometimes I wonder whether > > > > > connecting through to a TOR network could help us a lot with it. > > > > > > > > You can already use dillo with privoxy and tor. We could perhaps > > > > make it more convenient to configure such a setup. > > > > We also would need to make sure that really all requests go through > > > > the proxy, e.g. I'm not sure, whether wget called from dowload.dpi > > > > uses the proxy from dillorc. > > > > > > I have warnings in the FAQ and dillorc that the wget env variables would > > > need to be set appropriately, but it would be nice to have dillo send that > > > to the dpi in requests so that it can set the environment for sure before > > > execing wget. > > > > I just committed a patch to set the environment variable from the preference > > if it wasn't already set. > > > > issues: > > - maybe we should have an ftp_proxy pref, too. > > - ISTR that dillo doesn't understand user/pass in URLs. > > This means that dillo doesn't do the right thing if > > - the env var has the user/pass > > - dillo is setting the env for wget. > > It turns out that if I want to add the ability for ftp/downloads > to do the right thing with the no_proxy pref as well, > I would have to send a no_proxy arg to the dpi. > > It would be cleaner to send the proxy URL to the dpi > (and set/clear it there in the environment for wget) every time > that it's needed than to have the proxy URL in the environment > and send a no_proxy arg when it's needed. > > So I'm going to back out that change. This turns out to be a somewhat complicated situation. I naively imagined that whatever I found in wget's 1500-line manpage was what there was, but since wget is a gnu program, it's a subset selected who-knows-how. wget understands the http_proxy, https_proxy, and ftp_proxy environment vars. dillo understands the http_proxy environment var. If it is not set, dillo will use the http_proxy pref. dillo uses its http_proxy value for both http and the https plugin. Whatever we might do to get *_proxy set a certain way in the environment in the downloads/ftp dpis, wgetrc can override them. - should the https plugin use the https_proxy environment var if it's set? I'm leaning toward no. - should dillo try to set the https_proxy env var from the http_proxy pref for the benefit of the downloads dpi? Well, the downloads dpi doesn't like https anyway right now because we haven't given it any sort of #ifdef ENABLE_SSL add argument "--no-check-certificate" #endif, but we could and probably should add the argument. As for the question, ... not sure. (Maybe we should just let wget do all of our https, at least instead of the current plugin.) - should dillo have https_proxy prefs and ftp_proxy prefs? On one hand it might minimize confusion, but then I could take the same amount of space in dillorc to write a big comment explaining all of this. (The unpleasant part of putting things in dillorc or explaining things in dillorc is that I know that no one who upgrades is going to look through it meticulously to see what tiny things have changed.) wget also understands the no_proxy environment variable and wgetrc option. Dillo understands the no_proxy preference. Coincidentally(?), they have the same format. - Should dillo set the env var for wget from the pref if not set? I think so, for the benefit of http downloads. - Should dillo use the env var for itself if set? I kind of don't want to add the complexity. I don't know. (And to get user/pass into wget for proxy, it seems that the proper thing is to stuff them into the env var *_proxy urls.) Anyway, I'm finding this all somewhat tricky to think about.

Jan. 12, 2010

Cookies performance

by Johannes.Hofmann＠gmx.de

Hi, I noticed that since the recent DPI update pages which use cookies load slower (e.g. fltk.org) Do others also see that? It could also be OS specific. Cheers, Johannes

Jan. 12, 2010

web bug save?

by Johannes.Hofmann＠gmx.de

On Tue, Jan 12, 2010 at 06:50:01AM +0000, corvid wrote: > I wrote: > > I wrote: > > > Johannes wrote: > > > > On Thu, Dec 31, 2009 at 12:27:36AM -0300, Jorge Arellano Cid wrote: > > > > > Sometimes I wonder whether > > > > > connecting through to a TOR network could help us a lot with it. > > > > > > > > You can already use dillo with privoxy and tor. We could perhaps > > > > make it more convenient to configure such a setup. > > > > We also would need to make sure that really all requests go through > > > > the proxy, e.g. I'm not sure, whether wget called from dowload.dpi > > > > uses the proxy from dillorc. > > > > > > I have warnings in the FAQ and dillorc that the wget env variables would > > > need to be set appropriately, but it would be nice to have dillo send that > > > to the dpi in requests so that it can set the environment for sure before > > > execing wget. > > > > I just committed a patch to set the environment variable from the preference > > if it wasn't already set. > > > > issues: > > - maybe we should have an ftp_proxy pref, too. > > - ISTR that dillo doesn't understand user/pass in URLs. > > This means that dillo doesn't do the right thing if > > - the env var has the user/pass > > - dillo is setting the env for wget. > > It turns out that if I want to add the ability for ftp/downloads > to do the right thing with the no_proxy pref as well, > I would have to send a no_proxy arg to the dpi. > > It would be cleaner to send the proxy URL to the dpi > (and set/clear it there in the environment for wget) every time > that it's needed than to have the proxy URL in the environment > and send a no_proxy arg when it's needed. So the idea is to always use command line arguments instead of environment variables? Sounds good to me. Cheers, Johannes

Jan. 12, 2010

web bug save?

by corvid＠lavabit.com

I wrote: > I wrote: > > Johannes wrote: > > > On Thu, Dec 31, 2009 at 12:27:36AM -0300, Jorge Arellano Cid wrote: > > > > Sometimes I wonder whether > > > > connecting through to a TOR network could help us a lot with it. > > > > > > You can already use dillo with privoxy and tor. We could perhaps > > > make it more convenient to configure such a setup. > > > We also would need to make sure that really all requests go through > > > the proxy, e.g. I'm not sure, whether wget called from dowload.dpi > > > uses the proxy from dillorc. > > > > I have warnings in the FAQ and dillorc that the wget env variables would > > need to be set appropriately, but it would be nice to have dillo send that > > to the dpi in requests so that it can set the environment for sure before > > execing wget. > > I just committed a patch to set the environment variable from the preference > if it wasn't already set. > > issues: > - maybe we should have an ftp_proxy pref, too. > - ISTR that dillo doesn't understand user/pass in URLs. > This means that dillo doesn't do the right thing if > - the env var has the user/pass > - dillo is setting the env for wget. It turns out that if I want to add the ability for ftp/downloads to do the right thing with the no_proxy pref as well, I would have to send a no_proxy arg to the dpi. It would be cleaner to send the proxy URL to the dpi (and set/clear it there in the environment for wget) every time that it's needed than to have the proxy URL in the environment and send a no_proxy arg when it's needed. So I'm going to back out that change.

Jan. 12, 2010

web bug save?

by corvid＠lavabit.com

I wrote: > Johannes wrote: > > On Thu, Dec 31, 2009 at 12:27:36AM -0300, Jorge Arellano Cid wrote: > > > Sometimes I wonder whether > > > connecting through to a TOR network could help us a lot with it. > > > > You can already use dillo with privoxy and tor. We could perhaps > > make it more convenient to configure such a setup. > > We also would need to make sure that really all requests go through > > the proxy, e.g. I'm not sure, whether wget called from dowload.dpi > > uses the proxy from dillorc. > > I have warnings in the FAQ and dillorc that the wget env variables would > need to be set appropriately, but it would be nice to have dillo send that > to the dpi in requests so that it can set the environment for sure before > execing wget. I just committed a patch to set the environment variable from the preference if it wasn't already set. issues: - maybe we should have an ftp_proxy pref, too. - ISTR that dillo doesn't understand user/pass in URLs. This means that dillo doesn't do the right thing if - the env var has the user/pass - dillo is setting the env for wget.

Jan. 12, 2010

HTML4 testpage containing all elements

by onepoint＠starurchin.org

On Mon, Jan 11, 2010 at 01:46:19PM +0100, Johannes Hofmann wrote: > does anyone have a test page containing all possible HTML4 element? There's a test suite at http://www.w3.org/MarkUp/Test/HTML401/current/tests/ but it's split into multiple pages. I couldn't find an all-in-one version. Regards, Jeremy Henty

Jan. 11, 2010

HTML4 testpage containing all elements

by Johannes.Hofmann＠gmx.de

Hi, does anyone have a test page containing all possible HTML4 element? Cheers, Johannes

Jan. 11, 2010

want to test some experimental cookies code?

by Johannes.Hofmann＠gmx.de

On Sun, Jan 10, 2010 at 01:11:37AM +0000, corvid wrote: > Jorge wrote: > > AFAIS, you're our cookies expert now. > > Please feel free to replace the current dpi when you see net gain. > > Committed. Excellent!

Jan. 10, 2010

dpi send cmd error msg (Re: [Dillo-dev] Re: even more cookies SECURITY)

by corvid＠lavabit.com

Jorge wrote: > On Mon, Jan 04, 2010 at 11:38:03PM +0000, corvid wrote: > > Do we currently have anything that needs to be > > fixed/implemented before a release would be sensible? > > Not that I remember. The only problem I recall was the error > message you reported with cookies and later seemed to be working > somehow. a_Dpi_send_blocking_cmd() gets upset when Dpi_blocking_read() doesn't get anything. So in the cookie dpi's srv_parse_tok(), after the Cookies_set(), I tried putting a a_Dpip_dsh_write_str(sh, 1, "text"), and that makes the error msgs go away for me.

Jan. 10, 2010

want to test some experimental cookies code?

by corvid＠lavabit.com

Jorge wrote: > AFAIS, you're our cookies expert now. > Please feel free to replace the current dpi when you see net gain. Committed. Is there an easy way that we could get dpid to send the cookies dpi a DpiBye to end the session on demand? I tried this once, long ago, and the problem was something like dpid expected a reply, so it kept restarting the dpi to send it the bye.

Jan. 10, 2010

want to test some experimental cookies code?

by jcid＠dillo.org

On Tue, Jan 05, 2010 at 09:12:39PM +0000, corvid wrote: > In recent days, I've been making a version of cookies dpi that > follows the current cookies draft of the http state working group. > > I haven't tested it exhaustively yet, but if anyone who uses > cookies wants to try it and watch the msgs and so on, let me know. > > > As for how much validity or meaningfulness this draft has, let's see... > - They've made the decision to describe the way things currently behave, > having learned from the bitter experience of others who made RFC 2109 > and RFC 2965*. > In 2010, no one has any way to be King of Cookies. They write testcases and > try all of the major browsers and decide what they want to write down in > their spec when there are differences. > - On their mailing list, from the user agents, they have an opera guy, the > curl guy, and someone related to firefox, although I didn't take any notice > of whether he has a position of authority in firefox or not. > - Of course there will be changes. Smallish, I think. > > > A cookie set without an explicit domain goes only to that domain. > A cookie with a domain attribute is sent to that domain and any subdomain. > A .domain attr cookie can go to domain. > As for how to decide to reject cookies for, e.g., .co.uk, I don't think they're > ever going to say too much. > > A cookie can be set for any path, regardless of the url path of the page > setting it. > Paths are matched when selecting cookies to return, though. > Path-matching does work by slashes instead of simple prefixes. > > An http page can set a secure cookie (just like currently). > > An example of the little things that may change are that a cookie that is > just VALUE instead of NAME=VALUE is currently allowed, but it's not decided > for good yet. There are corner cases such as this that maybe they could > make illegal without rendering themselves irrelevant. > > > * To be precise, the main author of the RFCs has written that it was at > least initially reasonable to think that what they were putting together > in rfc 2109 would be adopted by user agents. AFAIS, you're our cookies expert now. Please feel free to replace the current dpi when you see net gain. -- Cheers Jorge.-

Jan. 9, 2010

Re: even more cookies SECURITY

by jcid＠dillo.org

Hi, I want to start this 2010 year thanking all the people that have contributed to make Dillo what it is. It's been 10 years and a long way lays behind. Lots of people have shared their time/work/advice/help/money/. After all, what we have now is the addition of all those efforts. Thanks to you all!!! On Mon, Jan 04, 2010 at 11:38:03PM +0000, corvid wrote: > Do we currently have anything that needs to be > fixed/implemented before a release would be sensible? Not that I remember. The only problem I recall was the error message you reported with cookies and later seemed to be working somehow. Besides, AFAIS the repo is stable enough to start rolling the release process. BTW, the cookie story is frustrating. Generally speaking: vested interests strive to kill commodity technologies to introduce their own and henceforward profit monopoly-like. This happens everywhere in the area we deal with e.g. HTTP, HTML, SGML, XML, scripting, flash, java, video streams, audio, cookies, AJAX, etc. Remember that not even the big players were able (nor willing) to implement the SPECS they co-designed. Dillo's game so far had been to support a subset with emphasis on efficiency and privacy. The game is never solved, the point is to decide well where to prune (see bottom line). Speaking of technology: Ten years ago it was images and tables, the FTP, then CGI, then cookies (the stateless HTTP, now with state), then frames (which faded 5 years later), then SGML (which was replaced by XML), then XML (which never succeeded over HTML tag soup). Then CSS (which is necessary these days. Despite IE never gave much for it). Somewhere in the middle there was java (and java beans), but it died. Then Flash, then AJAX, then silverlight, the cloud... IMHO, beyond the hyped technologies "du jour", Dillo is relevant as long as it's able to serve as an efficient information gathering tool, with emphasis on privacy. -- Cheers Jorge.-

Jan. 9, 2010

Doctree changes

by Johannes.Hofmann＠gmx.de

Hi, I just committed a modification of the Doctree class which is used for CSS matching. Before it was just an interface that was implemented by StyleEngine. Now it is a real class that is used by StyleEngine. I hope the new origanization makes things a bit clearer. It also avoids virtual method calls in the performance critical CSS matching path. To me it seems that complex pages render a bit faster now, but I did not make any proper measurements. Cheers, Johannes

Jan. 7, 2010

Query on interfacing IMX27 keyboard with Dillo browser.

by newman.x＠gmail.com

On Thu, Jan 7, 2010 at 4:48 PM, corvid <corvid(a)lavabit.com> wrote: > (Trying firefox now, I see that I can't type tabs into textareas; > it always takes me to the next focusable element. I'm a bit surprised.) It always been this way. I remember Konqueror allowed me to insert Tab the usual way and one was also able to disable this behavior via Preferences or context menu.

Jan. 7, 2010

Query on interfacing IMX27 keyboard with Dillo browser.

by corvid＠lavabit.com

Rajesh wrote: > I am using Dillo as a browser on IMX27 system. (working with > latest CVS LTIB code). I wanted to interface imx27 > Keyboard with the browser so that the UP/DOWN/etc. keys work on the > browser pages. > I have been using this with Touchscreen and I wanted to know if the > Keyboard can be interfaced too. > > Presently, I saw that on any Linux installation with Dillo on it, only mouse > works as mouse can be moved from one link to other link. But keyboard was > not working. I would want to make the Keyboard UP/DOWN arrows take me up and > down the links on a page in the Dillo browser. Is it possible to do this ? Not currently. I've sort of idly considered what it might take to be able to tab between the various links and form controls, but was stopped by "Hmm, how would you know to break out of a <textarea> again?". (Trying firefox now, I see that I can't type tabs into textareas; it always takes me to the next focusable element. I'm a bit surprised.)

Jan. 7, 2010

Query on interfacing IMX27 keyboard with Dillo browser.

by rmarathe11＠gmail.com

Hello, I am using Dillo as a browser on IMX27 system. (working with latest CVS LTIB code). I wanted to interface imx27 Keyboard with the browser so that the UP/DOWN/etc. keys work on the browser pages. I have been using this with Touchscreen and I wanted to know if the Keyboard can be interfaced too. Presently, I saw that on any Linux installation with Dillo on it, only mouse works as mouse can be moved from one link to other link. But keyboard was not working. I would want to make the Keyboard UP/DOWN arrows take me up and down the links on a page in the Dillo browser. Is it possible to do this ? regards, Rajesh Marathe.

Jan. 7, 2010

want to test some experimental cookies code?

by corvid＠lavabit.com

In recent days, I've been making a version of cookies dpi that follows the current cookies draft of the http state working group. I haven't tested it exhaustively yet, but if anyone who uses cookies wants to try it and watch the msgs and so on, let me know. As for how much validity or meaningfulness this draft has, let's see... - They've made the decision to describe the way things currently behave, having learned from the bitter experience of others who made RFC 2109 and RFC 2965*. In 2010, no one has any way to be King of Cookies. They write testcases and try all of the major browsers and decide what they want to write down in their spec when there are differences. - On their mailing list, from the user agents, they have an opera guy, the curl guy, and someone related to firefox, although I didn't take any notice of whether he has a position of authority in firefox or not. - Of course there will be changes. Smallish, I think. A cookie set without an explicit domain goes only to that domain. A cookie with a domain attribute is sent to that domain and any subdomain. A .domain attr cookie can go to domain. As for how to decide to reject cookies for, e.g., .co.uk, I don't think they're ever going to say too much. A cookie can be set for any path, regardless of the url path of the page setting it. Paths are matched when selecting cookies to return, though. Path-matching does work by slashes instead of simple prefixes. An http page can set a secure cookie (just like currently). An example of the little things that may change are that a cookie that is just VALUE instead of NAME=VALUE is currently allowed, but it's not decided for good yet. There are corner cases such as this that maybe they could make illegal without rendering themselves irrelevant. * To be precise, the main author of the RFCs has written that it was at least initially reasonable to think that what they were putting together in rfc 2109 would be adopted by user agents.

Jan. 5, 2010

table/cursor oddity

by Johannes.Hofmann＠gmx.de

On Mon, Jan 04, 2010 at 01:19:37AM +0000, corvid wrote: > A little bug that I just stumbled upon. > > <table border=1> > <tr><td style="cursor:help">one<td style="cursor:wait">two<td>three > <tr><td colspan=3>row > </table> > > The style of the first td seems to be the style of the whole table. > > (I wasn't surprised that the text has the default cursor, since I > know there's that weirdness with wordStyle and regular style. > Speaking of which, is it spelled out anywhere what is going on with > that exactly? I know there have been times when I've been in the area > of the table code and some reason why it's necessary has become clear, > but I don't think I've ever had the whole shape of the matter in my head.) > > Hmm. I just changed the first td to style="cursor:help; color:red", > and it colored the first td correctly, and now the rest of the table > shows the wait cursor. Interesting. Should be fixed now. Turned out to be a stupid bug in style.cc.

Jan. 5, 2010

css :visited and security

by Johannes.Hofmann＠gmx.de

On Tue, Jan 05, 2010 at 03:36:24PM +0000, corvid wrote: > The other day, I saw another of those pages about using :visited and > display in order to allow others to learn whether the user has visited > a page. (I remember Johannes pointing to a page about this last year, too.) > > I wonder whether it would be sensible to have a dillorc option defaulting > to NO for whether to use :visited. Yes, that sounds good. We should distinguish between remote / embedded CSS and user agent and user stylesheet though. For the latter two we don't need any restrictions. We might also do some other restrictions on remote CSS in the future, e.g. possible dillo specific extensions should also be only accessible via the user agent and user stylesheet.

Jan. 5, 2010

css :visited and security

by corvid＠lavabit.com

The other day, I saw another of those pages about using :visited and display in order to allow others to learn whether the user has visited a page. (I remember Johannes pointing to a page about this last year, too.) I wonder whether it would be sensible to have a dillorc option defaulting to NO for whether to use :visited.

Jan. 5, 2010

Query on Dillo browser supporting "frames"

by corvid＠lavabit.com

Rajesh wrote: > Will Dillo2 browser support "Frames" in future ? IF yes, what is the > time frame for that release ? I'm not aware of any particular plans to work on it, but of course I'm sure a good patch that integrates well with the existing code would be gladly accepted...

Jan. 5, 2010