Another (native) Dillo-Win32 patch
If anyone's interested, I've been working on a new patch to add native Win32 support to Dillo 2.x. It needs attention for a more experienced porter and is not yet ready for prime time, but it produces a running dillo.exe and should provide a reasonable base for further efforts. More information is here: http://obeythepenguin.users.sourceforge.net/dillo/ This is a complex patch that introduces several new low-level APIs, mainly sockets-related. There are also some minor fixes to #include paths and other such small changes. Apologies for the long message -- I hope somebody finds this useful :-)
Hi Benjamin, On Fri, Aug 27, 2010 at 08:12:24PM -0400, Benjamin Johnson wrote:
If anyone's interested, I've been working on a new patch to add native Win32 support to Dillo 2.x. It needs attention for a more experienced porter and is not yet ready for prime time, but it produces a running dillo.exe and should provide a reasonable base for further efforts.
More information is here: http://obeythepenguin.users.sourceforge.net/dillo/
This is a complex patch that introduces several new low-level APIs, mainly sockets-related. There are also some minor fixes to #include paths and other such small changes.
Cool! I just had a quick look over the patch and it looks pretty clean. If someone is willing to make it work properly and to maintain the Win32 port, I would like to see it committed to dillo main line. Cheers, Johannes
I'm feeling quite accomplished, then :-) I'll probably go in and work on it some more when I have time -- I'd like to get dpid, etc. to at least compile. I should also put together a README with info on the build process/dependencies since some of them are a little trickier than others. I suspect the HTTP problem is from the clumsy dFcntl() implementation; I'm not sure I implemented non-blocking IO correctly, and Win32 doesn't appear to support close-on-exec at all. I'll take another look at that later, since I really want this thing to work myself. Thanks for the feedback & cheers, ~Benjamin On Sat, 28 Aug 2010 04:15:37 -0400, Johannes Hofmann <Johannes.Hofmann@gmx.de> wrote:
Hi Benjamin,
Cool! I just had a quick look over the patch and it looks pretty clean. If someone is willing to make it work properly and to maintain the Win32 port, I would like to see it committed to dillo main line.
Cheers, Johannes
Thought I should send in an update on Dillo-Win32. It still has a long ways to go before it's usable, but I believe the worst of the low-level changes are done, and if anyone can spare a moment I'd like to get some feedback on the implementation. First, the patch: http://obeythepenguin.users.sourceforge.net/dillo/dillo-r1698-win32-20100829... And now, a quick overview: - I've added two new APIs under dlib/. The first, dFcntl, wraps fcntl on Unix, and simulates it on Windows. The second, dSock, is an abstraction layer over the sockets interface (BSD sockets/Winsock). Among other things, it takes into account that on Windows, socket and file handles are not necessarily interchangeable [works to some degree on NT, but not 9x/Me]. N.B. The existing browser code has been converted to these APIs. - I've implemented an alternate iowatch mechanism using timeouts, to work around non-functioning fltk::add_fd on Windows. This kludge is only enabled when building on Windows; other platforms continue to use the existing, more efficient mechanism. Obviously a better solution in the long run would be to fix FLTK, but that's a project for another time. - I've rewritten a_Dns_dillohost_to_string to use getnameinfo rather than inet_ntop. From what I've seen online, getnameinfo is the preferred function due to protocol(?) independence, among other features. More importantly to my purposes, getnameinfo is cross-platform, whereas inet_ntop is not. - I also added a few other small portability functions and fixes, such as dMkdir in dlib.c; and more explicit relative #include paths, since MinGW seems to prefer those. There are still a few rough edges and probably a few more #ifdef's than I'd like (some deliberate, some from laziness...), but overall I've tried to keep it as clean as possible. This patch should not affect existing Unix behavior; I've been testing it in parallel on OpenBSD, though I have not tried it on others. Anyway, I've already said far too much, so at this point I'll let the code speak for itself. Apologies again for another long post. Thanks in advance for any feedback (and for all received so far), ~Benjamin
Benjamin wrote:
- I've rewritten a_Dns_dillohost_to_string to use getnameinfo rather than inet_ntop. From what I've seen online, getnameinfo is the preferred function due to protocol(?) independence, among other features. More importantly to my purposes, getnameinfo is cross-platform, whereas inet_ntop is not.
It looks like EAI_OVERFLOW is the one for buffer too small. Do you have an idea of what's wrong with the dStr_insert_l() crash?
On 8/30/10, corvid <corvid@lavabit.com> wrote:
It looks like EAI_OVERFLOW is the one for buffer too small. Did I put the wrong constant in there? Thanks for the tip -- this is my first time working with sockets programming, so I'm sure I missed some other things as well.
Do you have an idea of what's wrong with the dStr_insert_l() crash? I didn't get that far until late last night, since it took me a while to figure out iowatch, but it looks like it happens during rendering. I saw there's a zlib inflate (I think) somewhere up the call stack, and that it segfaults at memcpy(), and that's about all I have so far.
Does Dillo use any temporary files when rendering a page? I haven't adjusted paths.cc beyond mkdir() -> dMkdir(), so I wonder if that has something to do with it.
Benjamin wrote:
On 8/30/10, corvid <corvid@lavabit.com> wrote:
Do you have an idea of what's wrong with the dStr_insert_l() crash? I didn't get that far until late last night, since it took me a while to figure out iowatch, but it looks like it happens during rendering. I saw there's a zlib inflate (I think) somewhere up the call stack, and that it segfaults at memcpy(), and that's about all I have so far.
Does Windows let you get a meaningful backtrace?
Does Dillo use any temporary files when rendering a page? I haven't adjusted paths.cc beyond mkdir() -> dMkdir(), so I wonder if that has something to do with it.
Nope. Although it can start trying to do dpi stuff while a page is coming.
On Mon, 30 Aug 2010 13:22:52 -0400, corvid <corvid@lavabit.com> wrote:
Benjamin wrote:
On 8/30/10, corvid <corvid@lavabit.com> wrote:
Do you have an idea of what's wrong with the dStr_insert_l() crash? I didn't get that far until late last night, since it took me a while to figure out iowatch, but it looks like it happens during rendering. I saw there's a zlib inflate (I think) somewhere up the call stack, and that it segfaults at memcpy(), and that's about all I have so far.
Does Windows let you get a meaningful backtrace?
I have the call stack, and I'm not sure how much else. I'm using gdb via Qt Creator, which is less than ideal, but it's the closest thing available to a working front-end. At some point I may try building with Visual C++, since its debugger is reportedly quite good, but for now I'd like to keep it simple and stick to what I have.
Does Dillo use any temporary files when rendering a page? I haven't adjusted paths.cc beyond mkdir() -> dMkdir(), so I wonder if that has something to do with it.
Nope. Although it can start trying to do dpi stuff while a page is coming.
That might be it. What type of DPI stuff, specifically? I have some conceptual understanding and I'm sure I saw it in the docs, but my memory's a little hazy... I should probably look at DPI next (which was my original intention anyway), then if there are still runtime issues, try to address them then. Thanks again, ~Benjamin
Benjamin wrote:
On Mon, 30 Aug 2010 13:22:52 -0400, corvid <corvid@lavabit.com> wrote:
Benjamin wrote:
On 8/30/10, corvid <corvid@lavabit.com> wrote:
Do you have an idea of what's wrong with the dStr_insert_l() crash? I didn't get that far until late last night, since it took me a while to figure out iowatch, but it looks like it happens during rendering. I saw there's a zlib inflate (I think) somewhere up the call stack, and that it segfaults at memcpy(), and that's about all I have so far.
Does Windows let you get a meaningful backtrace?
I have the call stack, and I'm not sure how much else. I'm using gdb via Qt Creator, which is less than ideal, but it's the closest thing available to a working front-end.
What does the call stack look like?
Does Dillo use any temporary files when rendering a page? I haven't adjusted paths.cc beyond mkdir() -> dMkdir(), so I wonder if that has something to do with it.
Nope. Although it can start trying to do dpi stuff while a page is coming.
That might be it. What type of DPI stuff, specifically? I have some conceptual understanding and I'm sure I saw it in the docs, but my memory's a little hazy...
Actually, if you saw inflate and you were coming through the decode.c and cache.c code, I can't immediately think of a way that it would necessarily be a dpi issue...
On Mon, Aug 30, 2010 at 2:53 PM, corvid <corvid@lavabit.com> wrote:
What does the call stack look like?
Here's the backtrace: http://obeythepenguin.users.sourceforge.net/dillo/misc/dStr_insert_l-backtra...
Actually, if you saw inflate and you were coming through the decode.c and cache.c code, I can't immediately think of a way that it would necessarily be a dpi issue...
All right, that makes sense since DPI should report a pipe error instead of segfaulting.
Benjamin wrote:
On Mon, Aug 30, 2010 at 2:53 PM, corvid <corvid@lavabit.com> wrote:
What does the call stack look like?
Here's the backtrace: http://obeythepenguin.users.sourceforge.net/dillo/misc/dStr_insert_l-backtra...
Okay, so the lengths reported for dStr_append/insert_l() are -1163005939, which is not good. When we started in Decode_gzip(), we had 2934 bytes, which sounds normal enough. Looking at z_stream, total_out is a uLong and the dstr code expects an int, but it would be hard to get a very large number there. Does your interface allow you to inspect everything around the inflate call and see what values are good and how they go bad? PS If you like, you can take out the "Accept-Encoding: gzip\r\n" lines in src/IO/http.c to check how much of the rest of dillo is working...
I wrote:
Benjamin wrote:
On Mon, Aug 30, 2010 at 2:53 PM, corvid <corvid@lavabit.com> wrote:
What does the call stack look like?
Here's the backtrace: http://obeythepenguin.users.sourceforge.net/dillo/misc/dStr_insert_l-backtra...
Okay, so the lengths reported for dStr_append/insert_l() are -1163005939, which is not good. When we started in Decode_gzip(), we had 2934 bytes, which sounds normal enough.
Looking at z_stream, total_out is a uLong and the dstr code expects an int, but it would be hard to get a very large number there.
You might try undoing http://hg.dillo.org/dillo/rev/7447d4d53388 and I'll look through the zlib documentation to see whether inflate is supposed to do anything in particular with total_out when there's an error (if that's what's going on).
On Mon, Aug 30, 2010 at 4:40 PM, corvid <corvid@lavabit.com> wrote:
You might try undoing http://hg.dillo.org/dillo/rev/7447d4d53388 and I'll look through the zlib documentation to see whether inflate is supposed to do anything in particular with total_out when there's an error (if that's what's going on).
See for yourself -- it's beautiful: http://obeythepenguin.users.sourceforge.net/dillo/dillo-homepage-win32.png Reverting that change and disabling gzip was just the trick -- thank you so much for that tip! Come to think of it, I had some trouble building zlib (MinGW is always a little difficult), so it's likely a problem with my DLL rather than Dillo or zlib per se. I'm just putting together a README now, and will send a link to the updated patch (and probably a testing build, if anyone wants to play with it) shortly.
Benjamin wrote:
On Mon, Aug 30, 2010 at 4:40 PM, corvid <corvid@lavabit.com> wrote:
You might try undoing http://hg.dillo.org/dillo/rev/7447d4d53388 and I'll look through the zlib documentation to see whether inflate is supposed to do anything in particular with total_out when there's an error (if that's what's going on).
See for yourself -- it's beautiful: http://obeythepenguin.users.sourceforge.net/dillo/dillo-homepage-win32.png
Reverting that change and disabling gzip was just the trick -- thank you so much for that tip! Come to think of it, I had some trouble building zlib (MinGW is always a little difficult), so it's likely a problem with my DLL rather than Dillo or zlib per se.
I'm just putting together a README now, and will send a link to the updated patch (and probably a testing build, if anyone wants to play with it) shortly.
That's great! You shouldn't need both the decode.c change and the http.c change, though. Eventually it will be good for us to get to the very bottom of this segfault problem...
Oops, my last e-mail went to just corvid and not the whole list. Sorry about that :-) Here's what I meant to send below, but first, my new patch page: http://dillo-win32.sourceforge.net/ The latest patch is online there, as well as a pre-compiled build for testing. It turns out the PNG crashes mentioned below were in IO_timeout_cb (at FD_SET), and had nothing to do with libpng/zlib after all -- I think it's just my stupid timeout kludge. Ironically, my homepage is one of the crashing ones, though dillo.org is fine. On the other hand, it's quite usable with image loading disabled. Thanks again for all the advice and encouragement so far. Hope it's been/will be worth it! Cheers, ~Benjamin On Mon, 30 Aug 2010 20:28:45 -0400, corvid <corvid@lavabit.com> wrote:
You shouldn't need both the decode.c change and the http.c change, though. Eventually it will be good for us to get to the very bottom of this segfault problem...
I got a few crashes loading PNG images, so I'm almost certain it's a zlib problem. I botched the install, so it's probably linked incorrectly on my machine. I'm going to leave the changes in for now, just to be on the safe side (and because re-building on MinGW can be quite a tedious process), but I'm re-building zlib/libjpeg/libpng/FLTK and doing one last test build now. If it doesn't crash, I'll assume it's safe to undo those changes. Updated patch and test build available shortly at http://dillo-win32.sourceforge.net/
Benjamin wrote:
It turns out the PNG crashes mentioned below were in IO_timeout_cb (at FD_SET), and had nothing to do with libpng/zlib after all -- I think it's just my stupid timeout kludge. Ironically, my homepage is one of the crashing ones, though dillo.org is fine. On the other hand, it's quite usable with image loading disabled.
I wonder whether it has to do with interrupted image downloading when it gets all of the remote stylesheets for a page and performs a repush. Or, more precisely, when an image has been requested but hasn't started coming back from the server. If you turn on image loading and turn off remote stylesheets, does it work consistently?
On Mon, Aug 30, 2010 at 11:28 PM, corvid <corvid@lavabit.com> wrote:
I wonder whether it has to do with interrupted image downloading when it gets all of the remote stylesheets for a page and performs a repush. Or, more precisely, when an image has been requested but hasn't started coming back from the server.
So what you're saying is, it's trying to fetch stylesheets between requesting an image and receiving it, and that's what's causing the problem?
If you turn on image loading and turn off remote stylesheets, does it work consistently?
It seems to be working so far.
Benjamin wrote:
On Mon, Aug 30, 2010 at 11:28 PM, corvid <corvid@lavabit.com> wrote:
I wonder whether it has to do with interrupted image downloading when it gets all of the remote stylesheets for a page and performs a repush. Or, more precisely, when an image has been requested but hasn't started coming back from the server.
So what you're saying is, it's trying to fetch stylesheets between requesting an image and receiving it, and that's what's causing the problem?
When stylesheets come in, they go through Html_css_load_callback(), and when the last one has been completely received, a_UIcmd_repush() is called to start the page rendering over again. Jorge did the repush code, but it seems to me that dillo continues to receive any partially-received images, whereas for images that haven't started to arrive, the connection is closed and a new connection is made to get the image during the second rendering. So when you mention images and IO callbacks, it comes to mind as a case where something funny could happen...
On Tue, Aug 31, 2010 at 12:57 AM, corvid <corvid@lavabit.com> wrote:
When stylesheets come in, they go through Html_css_load_callback(), and when the last one has been completely received, a_UIcmd_repush() is called to start the page rendering over again. Jorge did the repush code, but it seems to me that dillo continues to receive any partially-received images, whereas for images that haven't started to arrive, the connection is closed and a new connection is made to get the image during the second rendering.
So when you mention images and IO callbacks, it comes to mind as a case where something funny could happen...
Once again you've hit it dead-on before I can even fire up the debugger. Do you have psychic powers or something, or are you just that good? :-) My solution is to have dsock keep a count of active sockets, and rather than repush immediately, start a background loop that waits for all sockets to close before repushing. It seems to work fine on Unix with the regular iowatch mechanism, but I haven't tested that extensively, so if there's problems it could be #ifdef'd for just the timeout mechanism kludge. A few other changes: file paths are now correctly handled on Windows; disabled unnecessary pthreads check in ./configure; gzip compression is re-enabled; and possibly a couple other things I've already forgotten. The latest patch: http://dillo-win32.sourceforge.net/files/dillo-r1698-win32-20100831.diff And an updated build is available from the usual place: http://dillo-win32.sourceforge.net/ Assuming I didn't make too many more stupid mistakes, DPI sounds like the logical next step. Anything in particular I should watch out for there?
Benjamin wrote:
Once again you've hit it dead-on before I can even fire up the debugger. Do you have psychic powers or something, or are you just that good? :-)
:)
DPI sounds like the logical next step. Anything in particular I should watch out for there?
I don't know; we'll find out soon enough...
[Oops, screwed up the mail again. Sorry I keep doing this to you corvid, as if keeping up with all my code wasn't bad enough... -bmj] On Tue, 31 Aug 2010 20:48:03 -0400, corvid <corvid@lavabit.com> wrote:
Benjamin wrote:
Once again you've hit it dead-on before I can even fire up the debugger. Do you have psychic powers or something, or are you just that good? :-)
:)
You *do* have psychic powers, then! I knew it. Thanks again for all your help -- I never expected to get this far so quickly, and I certainly couldn't have done it without you.
DPI sounds like the logical next step. Anything in particular I should watch out for there?
I don't know; we'll find out soon enough...
I suppose so! By the way, I don't know if this patch is/will be quite good enough to merge in, but should this happen -- I think Johannes expressed some interest -- I'll happily maintain it. I can do Windows binary releases and support on the SourceForge site so it's not bothering anyone in main line. Cheers, ~Benjamin
All right, I have it running on Windows 95! Screenshots: http://dillo-win32.sourceforge.net/screenshots/dillo-win95.png http://dillo-win32.sourceforge.net/screenshots/dillo-wikipedia-win95.png It's still a little glitchy (FLTK issues, not Dillo per se), but now Dillo can truly run on any Windows system, for all practical purposes. [Of course, if you can prove people still use Win3.11...] Here's the latest patch: http://dillo-win32.sourceforge.net/files/dillo-r1698-win32-20100902.diff I've also got a full installer up on the SourceForge page, if anyone has a Windows machine and wants to take it for a spin. I've been using it quite a bit myself the last few days, including three straight hours the other night. As long as you don't need DPI, it's quite usable. Speaking of which... there's going to be a few complications, mainly because on Windows, file and socket handles are NOT interchangable. DPI seems to rely heavily on dup2(), for which there's no direct Windows equivalent. I've written up a full description of the problem here: http://dillo-win32.sourceforge.net/dpi.html This is where I'm going to need your expert advice. Jorge, if you have the time, I'd especially like to hear your opinion -- I know you're quite busy, and not a personal fan of the Windows platform, but I feel a well-executed port would ultimately help the project far more than it would hurt anything. Thanks again & cheers, ~Benjamin
participants (3)
-
corvid＠lavabit.com -
Johannes.Hofmann＠gmx.de -
obeythepenguin＠gmail.com