Hi, I noticed that since the recent DPI update pages which use cookies load slower (e.g. fltk.org). Do others also see that? It could also be OS specific. Cheers, Johannes
Johannes wrote:
I noticed that since the recent DPI update pages which use cookies load slower (e.g. fltk.org). Do others also see that? It could also be OS specific.
I thought it seemed weirdly slow _sometimes_. I'll have to throw some gprof at it and hope for something simple and embarrassing. (I wonder whether it would be easy to do valgrind with dpid and the plugins.)
On Tue, Jan 12, 2010 at 05:42:03PM +0000, corvid wrote:
Johannes wrote:
I noticed that since the recent DPI update pages which use cookies load slower (e.g. fltk.org). Do others also see that? It could also be OS specific.
I thought it seemed weirdly slow _sometimes_. I'll have to throw some gprof at it and hope for something simple and embarrassing.
(I wonder whether it would be easy to do valgrind with dpid and the plugins.)
I did some truss on the cookies dpi and found that it is reading ~/.dillo/dpid_comm_keys quite often. Maybe that is related.
Johannes wrote:
On Tue, Jan 12, 2010 at 05:42:03PM +0000, corvid wrote:
Johannes wrote:
I noticed that since the recent DPI update pages which use cookies load slower (e.g. fltk.org). Do others also see that? It could also be OS specific.
I thought it seemed weirdly slow _sometimes_. I'll have to throw some gprof at it and hope for something simple and embarrassing.
(I wonder whether it would be easy to do valgrind with dpid and the plugins.)
I did some truss on the cookies dpi and found that it is reading ~/.dillo/dpid_comm_keys quite often. Maybe that is related.
It looks like it checks it through a_Dpip_check_auth() when it gets or sets a msg. As for whether it needs to, I don't know. I tried a little gprof, but a modest amount of browsing and accepting cookies wasn't enough for the dpi to use up enough time for gprof to generate stats.
On Tue, Jan 12, 2010 at 07:06:12PM +0000, corvid wrote:
Johannes wrote:
On Tue, Jan 12, 2010 at 05:42:03PM +0000, corvid wrote:
Johannes wrote:
I noticed that since the recent DPI update pages which use cookies load slower (e.g. fltk.org). Do others also see that? It could also be OS specific.
I thought it seemed weirdly slow _sometimes_. I'll have to throw some gprof at it and hope for something simple and embarrassing.
(I wonder whether it would be easy to do valgrind with dpid and the plugins.)
I did some truss on the cookies dpi and found that it is reading ~/.dillo/dpid_comm_keys quite often. Maybe that is related.
It looks like it checks it through a_Dpip_check_auth() when it gets or sets a msg. As for whether it needs to, I don't know.
I tried a little gprof, but a modest amount of browsing and accepting cookies wasn't enough for the dpi to use up enough time for gprof to generate stats.
Yah, I don't think it's CPU bound. There must be something else. I will have a look at it.
On Tue, Jan 12, 2010 at 09:46:01PM +0100, Johannes Hofmann wrote:
On Tue, Jan 12, 2010 at 07:06:12PM +0000, corvid wrote:
Johannes wrote:
On Tue, Jan 12, 2010 at 05:42:03PM +0000, corvid wrote:
Johannes wrote:
I noticed that since the recent DPI update pages which use cookies load slower (e.g. fltk.org). Do others also see that? It could also be OS specific.
I thought it seemed weirdly slow _sometimes_. I'll have to throw some gprof at it and hope for something simple and embarrassing.
(I wonder whether it would be easy to do valgrind with dpid and the plugins.)
I did some truss on the cookies dpi and found that it is reading ~/.dillo/dpid_comm_keys quite often. Maybe that is related.
It looks like it checks it through a_Dpip_check_auth() when it gets or sets a msg. As for whether it needs to, I don't know.
I tried a little gprof, but a modest amount of browsing and accepting cookies wasn't enough for the dpi to use up enough time for gprof to generate stats.
Yah, I don't think it's CPU bound. There must be something else. I will have a look at it.
Can you please test attached patch, whether it compiles and makes a difference for you? I speeds things up quite a bit here. We might also want to cache SecretKey in a_Dpip_check_auth(). Cheers, Johannes
Hi all. This may be FreeBSD (8.0R) specific or my local proxy specific but if src/IO/dpi.c contains TCP_NODELAY, I got lots of this error messages when the page contains lots of pictures. Http_connect_socket ERROR: Connection reset by peer so that several pictures aren't drawn. I agree that page rendering is much faster if TCP_NODELAY is set but that isn't good on my environment. Regards, furaisanjin
On Thu, Jan 14, 2010 at 11:25:31AM +0900, furaisanjin wrote:
Hi all.
This may be FreeBSD (8.0R) specific or my local proxy specific but if src/IO/dpi.c contains TCP_NODELAY, I got lots of this error messages when the page contains lots of pictures.
Http_connect_socket ERROR: Connection reset by peer
What page is that, or does it happen on all pages with many images? Is it using https?
so that several pictures aren't drawn. I agree that page rendering is much faster if TCP_NODELAY is set but that isn't good on my environment.
We need find the issue for sure, but generally TCP_NODELAY should not cause any connection resets, so I think there is a bug somewhere that get's triggered by the different timing now. Cheers, Johannes
2010/1/14 Johannes Hofmann <Johannes.Hofmann@gmx.de>:
What page is that, or does it happen on all pages with many images? Is it using https?
There would be more pages but these are just examples. http://www.hatena.ne.jp/ http://f.hatena.ne.jp/hotfoto The first one doesn't contain so many images but relatively many and the second one contains lots of images. I don't think this is the site specific issue. Regards, furaisanjin
On Thu, Jan 14, 2010 at 08:24:54PM +0900, furaisanjin wrote:
2010/1/14 Johannes Hofmann <Johannes.Hofmann@gmx.de>:
What page is that, or does it happen on all pages with many images? Is it using https?
There would be more pages but these are just examples.
http://www.hatena.ne.jp/ http://f.hatena.ne.jp/hotfoto
The first one doesn't contain so many images but relatively many and the second one contains lots of images. I don't think this is the site specific issue.
Both seem to work fine here. What happens if you disable cookies? Could you try with cookies disabled and a dillo version before the TCP_NODELAY change? I remember that we once wanted to limit concurrent connections to a reasonable number and maybe the slow DPI communication is just doing that. So I suspect that without cookies even the version without TCP_NODELAY has issues - but that's just a wild guess. Regards, Johannes
2010/1/14 Johannes Hofmann <Johannes.Hofmann@gmx.de>:
Could you try with cookies disabled and a dillo version before the TCP_NODELAY change?
I did gmake clean and then ./configure --disable-cookies. I compiled codes and found that I had to modify like this. diff -r 9049a634eb0d src/cookies.c --- a/src/cookies.c Wed Jan 13 15:27:56 2010 +0900 +++ b/src/cookies.c Thu Jan 14 21:33:37 2010 +0900 @@ -22,7 +22,11 @@ */ void a_Cookies_init(void) { - MSG("Cookies: absolutely disabled at compilation time.\n"); + printf("Cookies: absolutely disabled at compilation time.\n"); +} + +void a_Cookies_freeall() +{ } I stopped dpid and installed cookie dpi just in case, then tried dillo again. I feel I see more error messages by this change. Regards, furaisanjin
On Thu, Jan 14, 2010 at 09:40:58PM +0900, furaisanjin wrote:
2010/1/14 Johannes Hofmann <Johannes.Hofmann@gmx.de>:
Could you try with cookies disabled and a dillo version before the TCP_NODELAY change?
I did gmake clean and then ./configure --disable-cookies. I compiled codes and found that I had to modify like this.
diff -r 9049a634eb0d src/cookies.c --- a/src/cookies.c Wed Jan 13 15:27:56 2010 +0900 +++ b/src/cookies.c Thu Jan 14 21:33:37 2010 +0900 @@ -22,7 +22,11 @@ */ void a_Cookies_init(void) { - MSG("Cookies: absolutely disabled at compilation time.\n"); + printf("Cookies: absolutely disabled at compilation time.\n"); +} + +void a_Cookies_freeall() +{ }
I stopped dpid and installed cookie dpi just in case, then tried dillo again. I feel I see more error messages by this change.
Is this without TCP_NODELAY? If so it would indicate that the real problem is the massive number of connections dillo is establishing if there are man images on a page and some servers / proxies don't like it. We probabely need to look at the experimental patch Jorge had posted here: http://lists.auriga.wearlab.de/pipermail/dillo-dev/2009-August/006778.html Cheers, Johannes
2010/1/14 Johannes Hofmann <Johannes.Hofmann@gmx.de>:
Is this without TCP_NODELAY?
The code contains TCP_NODELAY since I don't have any problem if I remove TCP_NODELAY. Now I tried - cookie is enabled - TCP_NODELAY is included - no proxy is used Then I don't any see error message. So the problem seems to be communication between dillo and my local proxy. Regards, furaisanjin
On Thu, Jan 14, 2010 at 11:04:03PM +0900, furaisanjin wrote:
2010/1/14 Johannes Hofmann <Johannes.Hofmann@gmx.de>:
Is this without TCP_NODELAY?
The code contains TCP_NODELAY since I don't have any problem if I remove TCP_NODELAY.
Now I tried - cookie is enabled - TCP_NODELAY is included - no proxy is used
Then I don't any see error message. So the problem seems to be communication between dillo and my local proxy.
Ok, could you please try this combination: - TCP_NODELAY not included (e.g. hg up 239f9fc6c313) - cookies disabled - proxy is used My guess is that this combination also shows the problem, but of course we need to verify this. Regards, Johannes
2010/1/14 Johannes Hofmann <Johannes.Hofmann@gmx.de>:
Ok, could you please try this combination:
- TCP_NODELAY not included (e.g. hg up 239f9fc6c313) - cookies disabled - proxy is used
My guess is that this combination also shows the problem, but of course we need to verify this.
Yes, I see the same error messages with above combination, so definitely this is communication problem between dillo and my local proxy. I guess that dillo creates socket connection too often. Regards, furaisanjin
On Fri, Jan 15, 2010 at 06:20:41PM +0900, furaisanjin wrote:
2010/1/14 Johannes Hofmann <Johannes.Hofmann@gmx.de>:
Ok, could you please try this combination:
- TCP_NODELAY not included (e.g. hg up 239f9fc6c313) - cookies disabled - proxy is used
My guess is that this combination also shows the problem, but of course we need to verify this.
Yes, I see the same error messages with above combination, so definitely this is communication problem between dillo and my local proxy. I guess that dillo creates socket connection too often.
Ok good. I'm working on a connection limit right now and hope to have something to test soon. Thanks for the continuous testing, Johannes
On Fri, Jan 15, 2010 at 01:09:39PM +0100, Johannes Hofmann wrote:
On Fri, Jan 15, 2010 at 06:20:41PM +0900, furaisanjin wrote:
2010/1/14 Johannes Hofmann <Johannes.Hofmann@gmx.de>:
Ok, could you please try this combination:
- TCP_NODELAY not included (e.g. hg up 239f9fc6c313) - cookies disabled - proxy is used
My guess is that this combination also shows the problem, but of course we need to verify this.
Yes, I see the same error messages with above combination, so definitely this is communication problem between dillo and my local proxy. I guess that dillo creates socket connection too often.
Ok good. I'm working on a connection limit right now and hope to have something to test soon.
Please test attached patch. It limits the overall number of concurrent connections to 4. Cheers, Johannes
2010/1/18 Johannes Hofmann <Johannes.Hofmann@gmx.de>:
Please test attached patch. It limits the overall number of concurrent connections to 4.
I tested this with TCP_NODELAY option and it seems that further image downloading stops when the page repush happens due to remote CSS. So I don't see all images. Good thing is that I don't get the error message. Regards, furaisanjin
On Mon, Jan 18, 2010 at 03:58:25PM +0900, furaisanjin wrote:
2010/1/18 Johannes Hofmann <Johannes.Hofmann@gmx.de>:
Please test attached patch. It limits the overall number of concurrent connections to 4.
I tested this with TCP_NODELAY option and it seems that further image downloading stops when the page repush happens due to remote CSS. So I don't see all images. Good thing is that I don't get the error message.
Hm, it seems to work ok here... Does loading other pages still work, after image loading has stalled, or is the complete dillo instance broken after that? Do you have a fast or slow internet connection? Do you know what kind of proxy you are connecting to? Regards, Johannes
Johannes wrote:
On Mon, Jan 18, 2010 at 03:58:25PM +0900, furaisanjin wrote:
2010/1/18 Johannes Hofmann <Johannes.Hofmann@gmx.de>:
Please test attached patch. It limits the overall number of concurrent connections to 4.
I tested this with TCP_NODELAY option and it seems that further image downloading stops when the page repush happens due to remote CSS. So I don't see all images. Good thing is that I don't get the error message.
Hm, it seems to work ok here...
Does loading other pages still work, after image loading has stalled, or is the complete dillo instance broken after that?
Do you have a fast or slow internet connection?
Do you know what kind of proxy you are connecting to?
I've tried with image loading and remote stylesheets on to the degree that I can tolerate :) , and I like this connection limiting a whole lot. So far, all I've noticed was once getting one of those "IO_write, closing with pending data not sent" msgs when I left a page while things were loading. It didn't seem to cause any harm, though.
On Mon, Jan 18, 2010 at 06:19:28PM +0000, corvid wrote:
Johannes wrote:
On Mon, Jan 18, 2010 at 03:58:25PM +0900, furaisanjin wrote:
2010/1/18 Johannes Hofmann <Johannes.Hofmann@gmx.de>:
Please test attached patch. It limits the overall number of concurrent connections to 4.
I tested this with TCP_NODELAY option and it seems that further image downloading stops when the page repush happens due to remote CSS. So I don't see all images. Good thing is that I don't get the error message.
Hm, it seems to work ok here...
Does loading other pages still work, after image loading has stalled, or is the complete dillo instance broken after that?
Do you have a fast or slow internet connection?
Do you know what kind of proxy you are connecting to?
I've tried with image loading and remote stylesheets on to the degree that I can tolerate :) , and I like this connection limiting a whole lot.
We definately need a connection limit. Currently dillo can be spawning hundreds of connections to a single server which is not nice. HTTP 1.1 even suggests a limit of 2 connections per host though according to http://www.stevesouders.com/blog/2008/03/20/roundup-on-parallel-connections/ modern browsers us something between 2 and 6 now, so I think an overall limit of 4 is ok for dillo. There even is a benefit when loading long pages with images: With the connection limit in place the first (and probabely visible) images on the page get more bandwidth and load faster.
So far, all I've noticed was once getting one of those "IO_write, closing with pending data not sent" msgs when I left a page while things were loading. It didn't seem to cause any harm, though.
This might be connections that get aborted while still queued and therefore not connected. I will check that. Cheers, Johannes
On Mon, Jan 18, 2010 at 06:59:08PM +0100, Johannes Hofmann wrote:
On Mon, Jan 18, 2010 at 03:58:25PM +0900, furaisanjin wrote:
2010/1/18 Johannes Hofmann <Johannes.Hofmann@gmx.de>:
Please test attached patch. It limits the overall number of concurrent connections to 4.
I tested this with TCP_NODELAY option and it seems that further image downloading stops when the page repush happens due to remote CSS. So I don't see all images. Good thing is that I don't get the error message.
Hm, it seems to work ok here...
Ok, I'm also seeing an issue now. Not sure if it's the problem you are having. With an overall limit of open connections a slow server can eat up all 4 connections. Then dillo can't connect to other hosts anymore, e.g. in an other window or tab. This needs to be addressed.
2010/1/19 Johannes Hofmann <Johannes.Hofmann@gmx.de>:
Do you have a fast or slow internet connection?
I have FTTH at home and the connection iself is quite fast.
Do you know what kind of proxy you are connecting to?
I'm using this proxy on my machine. <<http://www.acme.com/software/micro_proxy/>> But I modified this to act as daemon and limit the active connection (usually the active connection limit is 4 but proxy itself accepts all connections). I found that my modification for the proxy caused the problem. I had to modify a_Capi_stop_client like this. Client = a_Cache_client_get_if_unique(Key); - if (Client && (force || Client->BufSize == 0)) { + if (Client && force) { /* remove empty entries too */ a_Capi_conn_abort_by_url(Client->Url); After I canceled my modification, downloading didn't stop but I started seeing this kind of warning. ** WARNING **: IO_write, closing with pending data not sent ** WARNING **: "GET http://img.f.hatena.ne.jp/images/fotolife/z/zankuro/20090805/20090805082105.... HTTP/1.1\x0D Regards, furaisanjin
participants (3)
-
corvid＠lavabit.com -
furaisanjin＠gmail.com -
Johannes.Hofmann＠gmx.de