November 2007 - Dillo-dev - lists.mailman3.com

Re: charset conversion issues

by place＠gobigwest.com

Jorge wrote: > > - Reach the meta tag and undo/redo within html.cc. > > Somewhat fragile. > > Whatever I might come up with, some html out there would surely find a way > > to outsmart me. > > This is the one I like most at first sight. > > Considering the charset can be given by HTTP or a META element: > > We can assume ASCII in the html text until the HEAD element is > closed. If there's a charset in the META, then the decoder can be > switched from null to the specified one. > > This approach has the advantage of working both when the > charset comes via HTTP or META (<HEAD> content is ASCII). > > We can even add a text buffer for the HEAD element and append > it to the whole HTML content if the offset is hard to set for the > new decoder. > > Do the problems you found apply to this scheme too? My original code worked a bit like this. It would set a flag to make Html_write_raw() quit early and go back up to Html_write() for re-decoding. (It would just restart from the beginning, though, and I would ignore the error messages about multiple heads and so on.) - At least the title tag has to be called again. - Maybe a future dillo that does things with javascript has already seen some and done things with it -- possibly involving literal strings -- by the time we reach meta. I don't know javascript, though, so I don't know whether that's an actual concern. I'm not quite sure what you're saying about the text buffer, though...

Nov. 28, 2007

Re: charset conversion issues

by jcid＠dillo.org

Hi, On Sat, Nov 24, 2007 at 10:50:18PM +0000, place wrote: > With rough code to look at. > No sense polishing up and completing what I have yet when you'll want changes... OK. > > > Charset in the meta tag. > My options seemed to be: > - Client->Callback returns information to cache. > Probably not supposed to happen. Too big of a change. Agreed. At least to postpone until other approaches prove messier. ;-) > - Reach the meta tag and undo/redo within html.cc. > Somewhat fragile. > Whatever I might come up with, some html out there would surely find a way > to outsmart me. This is the one I like most at first sight. Considering the charset can be given by HTTP or a META element: We can assume ASCII in the html text until the HEAD element is closed. If there's a charset in the META, then the decoder can be switched from null to the specified one. This approach has the advantage of working both when the charset comes via HTTP or META (<HEAD> content is ASCII). We can even add a text buffer for the HEAD element and append it to the whole HTML content if the offset is hard to set for the new decoder. Do the problems you found apply to this scheme too? > - Bail out and start again. > This is what I have now. In fact, not knowing at all what to do, I just > call a_Nav_cancel_expect() and a_Nav_push() right then and there. > Surprisingly, it functions instead of crashing, but I'm probably spilling > resources And Violating Principles. > (And, since a_Nav_push() isn't quite right for the situation even then, > I have to fix the nav stack ptr in Nav_cleanup()) > > > There's a dialog box to set the Content-Type for a page. If you change it > to text/plain, you get source in a proper window (with searching) instead > of the little fltk window. No line numbers, though. The idea of using Dillo for view source is interesting, i'm not sure though. Maybe a fltk widget (like an stripped-down editor as the one in FLTK's test/ directory) which gets utf-8 via a_Capi_get_utf8_data(Url) is also good. > > > Not wanting to mess with TypeDet and TypeHdr until finding out what you > think, I stuck in another one, TypeSet. I'd prefer to try to apply the second method (described above). > I'm directly calling a_Cache_[sg]et_content_type() from just wherever, > which may or may not be all right. Cache should be called from Capi only, but at this stage... > I knew that latin1 != UTF-8, but I was not thinking when I wrote > that Decode code. Must've been thinking about the Unicode code numbers. > FLTK has a workaround in it to deal with fools like me who send it latin1 > or cp1252, but better that I stop being foolish :) > > > I don't have Plain or View Source set up to decode yet. I don't know whether > the cache should know about decoding charsets. Or a Text base class or > something. > > > The real-type-matches-header-type code will have to learn to relax a bit, > now that html might contain all kinds of bytes... -- Cheers Jorge.-

Nov. 28, 2007

patch: decode filter (Re: [Dillo-dev] Implemented a weighted apportionment algorithm for table rowspan.)

by jcid＠dillo.org

On Tue, Nov 27, 2007 at 12:25:13AM +0000, place wrote: > Jorge wrote: > > > Oh, and I just read about multiline headers. It wasn't really clear whether > > > this was new in 1.1, though. Did 1.0 have to handle it as well? > > > > Multiline headers?! > > sorry, I don't remember that off the top of my head. > > To answer my own question, the rfc says "HTTP/1.0 headers may be folded onto > multiple lines if each continuation line begins with a space or horizontal tab." > Doesn't look like Cache_parse_field() handles it, though. Must not have been > written with the spec in one hand. I've been through the SPEC lots of times... Even big browsers, like mozilla can't implement every bit of the SPECs (and they have big manpower). In dillo, there're lots of things that are not implemented, and I'm thankful of the people helping to push it forward. -- Cheers Jorge.-

Nov. 27, 2007

patch: decode filter (Re: [Dillo-dev] Implemented a weighted apportionment algorithm for table rowspan.)

by place＠gobigwest.com

Jorge wrote: > > Oh, and I just read about multiline headers. It wasn't really clear whether > > this was new in 1.1, though. Did 1.0 have to handle it as well? > > Multiline headers?! > sorry, I don't remember that off the top of my head. To answer my own question, the rfc says "HTTP/1.0 headers may be folded onto multiple lines if each continuation line begins with a space or horizontal tab." Doesn't look like Cache_parse_field() handles it, though. Must not have been written with the spec in one hand.

Nov. 27, 2007

patch: decode filter (Re: [Dillo-dev] Implemented a weighted apportionment algorithm for table rowspan.)

by jcid＠dillo.org

On Mon, Nov 26, 2007 at 07:24:34PM +0000, place wrote: > > Yes. Send me this. > > All right. Attached. Committed. > > > > > 2) Chunked Transfer-Encoding running on top of (1) because I thought it might > > > help with gzip latency. It was a can-I-get-this-working-in-principle effort > > > and I will have to rewrite it now. Plus I need to actually read through the > > > http/1.1 spec (what I did was just based on a little web page that > > > summarized the main differences between 1.0 and 1.1 for me). > > > > Hmm, yes. Please rewrite it as simple/clean as possible, and with the SPEC > > in one hand. Once there, please send it. > > > > My experience is that code written to comply with the SPEC is easier > > to debug and polish (there're plenty of cases where wrong behaviour is on > > the server side, and thus we can be sure is not dillo). > > It sounds like somebody's indeed had his share of bitter experience with > being sent hacked-together patches :) I wasn't actually threatening to send > this version; just letting you know what's going on... Yes, you're right. 8-P > > The most annoying part so far was that 1.1 is supposedly able to send > "100 Continue" before you get the real stuff. I haven't seen one actually > do it yet, though. Maybe that's only if you're doing pipelining... > > Oh, and I just read about multiline headers. It wasn't really clear whether > this was new in 1.1, though. Did 1.0 have to handle it as well? Multiline headers?! sorry, I don't remember that off the top of my head. -- Cheers Jorge.-

Nov. 27, 2007

Implemented a weighted apportionment algorithm for table rowspan.

by jcid＠dillo.org

On Mon, Nov 26, 2007 at 03:34:09PM +0000, place wrote: > > Everybody: please enjoy this patch, it was a lot of work! :-) > > if you find a URL where it helps, I'd appreciate some feedback. > > Hurray for better tables! Pages on the wunderground.com site fit together better > than they used to. Thanks for the feedback. (though I didn't find a "Short cell" message in the first clicks). > I currently have: > 1) I finally had the excuse to make decode into a proper filter. That is, > Dstr* as input as well as output. Yes. Send me this. > 2) Chunked Transfer-Encoding running on top of (1) because I thought it might > help with gzip latency. It was a can-I-get-this-working-in-principle effort > and I will have to rewrite it now. Plus I need to actually read through the > http/1.1 spec (what I did was just based on a little web page that > summarized the main differences between 1.0 and 1.1 for me). Hmm, yes. Please rewrite it as simple/clean as possible, and with the SPEC in one hand. Once there, please send it. My experience is that code written to comply with the SPEC is easier to debug and polish (there're plenty of cases where wrong behaviour is on the server side, and thus we can be sure is not dillo). -- Cheers Jorge.-

Nov. 26, 2007

Implemented a weighted apportionment algorithm for table rowspan.

by jcid＠dillo.org

> > Everybody: please enjoy this patch, it was a lot of work! :-) > > if you find a URL where it helps, I'd appreciate some feedback. > > It helps on http://www.heise.de > With the patch the news start on top of the page - as they should. Great! heise.de is very popular. You made my day. > Thanks for your work on this! Most probably the major benefit will come out of the table rendering knowledge gained while working on it, but this time applied to colspanning. -- Cheers Jorge.-

Nov. 26, 2007

DPI infrastructure patches (3rd try)

by jcid＠dillo.org

Hi Diego, On Mon, Nov 26, 2007 at 01:10:28AM +0100, Diego . wrote: > Excuse me the triple mail. I am very sorry for the problems this mail causes Yesterday I skimmed the incoming patches and the two dpi emails trying to find out why you sent a second one. AFAIS they're the same email. !? Now I'm puzzled. Why a third one? If you make slight modifications to the patches, or minor bug fixes, please explain it in the email. -- Cheers Jorge.- PS: No problem if you write to me in spanish.

Nov. 26, 2007

DPI infrastructure patches (3rd try)

by darkspirit5000＠gmail.com

Excuse me the triple mail. I am very sorry for the problems this mail causes Hello and great work. Here i resend 2 patches related to DPI infrastructure. The first patch (dpid_services_support.diff) add the service concept to dpid (like docs sugest) so DPIs names are not related to services names asked by dillo, more than a DPI can serve a service and various DPIs can be installed to serve one service. The patch add new configuration lines in dpidrc, but can work without changes in dpidrc. (backward compatible) The lines in dpidrc have the form: service_name=ending_part_of_path_to_DPI User DPIs are preferred to system DPIs. Example: bookmarks=/bookmarks.dpi cookies=cookies/cookies.dpi downloads=downloads/downloads.dpi file=file/file.dpi ftp=ftp/ftp.filter.dpi https=https/https.filter.dpi data=datauri/datauri.filter.dpi hello=hello/hello.filter.dpi service names ending in '*' match a service asked by dillo starting with the text before the '*' and that not match other service names Example: car=car.dpi cab=cab.dpi ca*=a.dpi a.dpi is used only if dillo ask for a service that start with ca but is not car or cab To test the patch add next line to dpidrc: test=hello/hello.filter.dpi and use the url dpi:/test/ The second path (proto_services.diff) removes ad-hoc bindings for protocols DPIs in dillo. It uses services for protocols handled externaly (internal protocols: http and about; external: file, ftp, https and data). With this 2 features a protocol DPI (mms:, ed2k:, ...) can be programed without changes to dillo. The service names have the form proto.protocol_name. The lines below must be added to dpidrc and dpid need to be patched with first path (dpid_services_support.diff) so protocols works as before. (NOT backward compatible) proto.file=file/file.dpi proto.ftp=ftp/ftp.filter.dpi proto.https=https/https.filter.dpi proto.data=datauri/datauri.filter.dpi With the next line not handled protocols can return an error, a page to download new DPIs or search new protocol DPIs in dillo page using a DPI: proto.*=proto/unknown.filter.dpi This scrip mimics dillo operation #!/bin/sh # read dillo open url command read -d'>' dpip_tag echo "<dpi cmd='send_status_message' msg='ERROR: unsupported protocol' '>" I have rediscover the dpid bug (without and with pach) that ignore a DPI installed in user and system DPI_dirs at the same time. It is not solved so care about it. Diego. ----------- dpid_services_support.diff --------------------------------------------------------- diff -pru dillo2/dpid/dpid.c g/dpid/dpid.c --- dillo2/dpid/dpid.c 2007-10-14 02:09:40.000000000 +0200 +++ g/dpid/dpid.c 2007-11-24 18:19:36.000000000 +0100 @@ -23,6 +23,7 @@ #include <sys/stat.h> #include <sys/wait.h> #include <unistd.h> +#include <ctype.h> #include "dpid_common.h" #include "dpid.h" #include "dpi.h" @@ -138,6 +139,20 @@ void free_plugin_list(struct dp **dpi_at *dpi_attr_list_ptr = NULL; } +/*! Free memory used by the services list + */ +void free_services_list(Dlist *s_list) +{ + int i = 0; + struct service *s; + + for (i=0; i < dList_length(s_list) ; i++) { + s = dList_nth_data(s_list, i); + dFree(s->name); + } +// dList_free(s_list); +} + /*! \todo * Remove terminator and est_terminator unless we really want to clean up * on abnormal exit. @@ -409,6 +424,145 @@ int register_all(struct dp **attlist) return (snum); } +/* + * Compare two FileInfo pointers + * This function is used for sorting directories + */ +static int services_alpha_comp(const struct service *s1, const struct service *s2) +{ + int t; + + t=strcmp(s1->name, s2->name); + if (t>0) + return -1; + else if (t<0) + return 1; + + return 0; +} + +/*! Add services reading a dpidrc file + * each non empty or commented line has the form + * service = path_relative_to_dpidir + * \Return: + * \li Returns number of available services on success + * \li -1 on failure + */ +int fill_services_list(struct dp **attlist, int numdpis, Dlist *services_list) +{ + FILE *dpidrc_stream; + char *line = NULL; + int line_num = 0; + char *p; + char *service, *path; + int i; + struct service *s; + char *user_dpidir = NULL, *sys_dpidir = NULL, *dpidrc = NULL; + + user_dpidir = dStrconcat(dGethomedir(), "/", dotDILLO_DPI, NULL); + if (access(user_dpidir, F_OK) == -1) { + /* no dpis in user's space */ + dFree(user_dpidir); + user_dpidir = NULL; + } + dpidrc = dStrconcat(dGethomedir(), "/", dotDILLO_DPIDRC, NULL); + if (access(dpidrc, F_OK) == -1) { + dFree(dpidrc); + dpidrc = dStrdup(DPIDRC_SYS); + if (access(dpidrc, F_OK) == -1) { + dFree(dpidrc); + dpidrc = NULL; + } + } + if (!dpidrc || (sys_dpidir = get_dpi_dir(dpidrc)) == NULL) + sys_dpidir = NULL; + + if (!user_dpidir && !sys_dpidir) { + ERRMSG("fill_services_list", "Fatal error ", 0); + fprintf(stderr, "\n - Can't find the directory for dpis.\n"); + exit(1); + } + + if ((dpidrc_stream = fopen(dpidrc, "r")) == NULL) { + ERRMSG("fill_services_list", "popen failed", errno); + dFree(dpidrc); + dFree(sys_dpidir); + dFree(user_dpidir); + return (-1); + } + + /* dpidrc parser loop */ + while ((line = dGetline(dpidrc_stream)) != NULL) { + i++; + /* ignore start spaces, commented and empty lines */ + dStrstrip(line); + if (*line == '#' || *line == '\0') + continue; + + service = line; + if ((path = strchr(line, '=')) == NULL) { + printf("malformed line in %s at %d: \'=\' not found\n", dpidrc, line_num); + continue; + } + *path = '\0'; + path++; + + dStrstrip(service); + /* check that service name do not contains spaces */ + if (strchr(service, ' ') != NULL) { + printf("malformed line in %s at %d: service contains spaces\n", dpidrc, i); + continue; + } + if (strlen(service) < 2) { + printf("malformed line in %s at %d: service name too sort\n", dpidrc, i); + continue; + } + /* ignore dpi_dir silently */ + if (strcmp(service, "dpi_dir") == 0) + continue; + + dStrstrip(path); + /* process quotes */ + if (*path == '\"' || *path == '\'') { + p = strrchr(path + 1, *path); + if (p == NULL) { + printf("malformed line in %s at %d: unpaired \'%c\'\n", dpidrc, i, *path); + continue; + } else if (p[1] != '\0') { + printf("malformed line in %s at %d: garbage after \'%c\'\n", dpidrc, i, *path); + continue; + } + path++; + } + + s = dNew(struct service, 1); + /* init services list entry */ + s->name = dStrdup(service); + s->dp_index = -1; + + dList_append(services_list, s); + /* search the dpi for a service by its path */ + for (i = 0; i < numdpis; i++) + if ((p = strstr((*attlist)[i].path, path)) && *(p - 1) == '/' && + strlen(p) == strlen(path)) + break; + /* if the dpi exist bind service and dpi */ + if (i < numdpis) + s->dp_index = i; + dFree(line); + } + fclose(dpidrc_stream); + + dList_sort(services_list, (dCompareFunc)services_alpha_comp); + + dFree(dpidrc); + dFree(sys_dpidir); + dFree(user_dpidir); + + return (dList_length(services_list)); +} + + /*! Initialise the service request socket * \Return: * \li Number of sockets (1 == success) @@ -668,11 +822,13 @@ int register_all_cmd(char *sockdir) stop_active_dpis(dpi_attr_list, numdpis); rm_dpi_sockets(dpi_attr_list, numdpis); free_plugin_list(&dpi_attr_list, numdpis); + free_services_list(services_list); numdpis = 0; numsocks = 1; /* the srs socket */ FD_ZERO(&sock_set); FD_SET(srs, &sock_set); numdpis = register_all(&dpi_attr_list); + fill_services_list(&dpi_attr_list, numdpis, services_list); numsocks = init_all_dpi_sockets(dpi_attr_list, sockdir); return (numdpis); } @@ -697,6 +853,20 @@ char *get_message(int sock, char *dpi_ta return (msg); } +int service_match(const struct service *A, const char *B) +{ + int A_len, B_len, len; + + A_len = strlen(A->name); + B_len = strlen(B); + len = MAX (A_len, B_len); + + if (A->name[A_len - 1] == '*') + len = A_len - 1; + + return(dStrncasecmp(A->name, B, len)); +} + /*! * Send socket path that matches dpi_id to client */ @@ -705,12 +875,18 @@ void send_sockpath(int sock, char *dpi_t int i; char *dpi_id; char *d_cmd; + struct service *serv; dReturn_if_fail((dpi_id = get_message(sock, dpi_tag)) != NULL); - for (i = 0; i < numdpis; i++) - if (strstr(dpi_attr_list[i].id, dpi_id)) - break; + serv = dList_find_custom(services_list, dpi_id, (dCompareFunc)service_match); + + if (serv == NULL || (i = serv->dp_index) == -1) + for (i = 0; i < numdpis; i++) + if (!strncmp(dpi_attr_list[i].id, dpi_id, + dpi_attr_list[i].id - strchr(dpi_attr_list[i].id, '.'))) + break; + if (i < numdpis) { /* found */ if (access(dpi_attr_list[i].path, F_OK) == -1) { diff -pru dillo2/dpid/dpid.h g/dpid/dpid.h --- dillo2/dpid/dpid.h 2007-10-07 00:36:34.000000000 +0200 +++ g/dpid/dpid.h 2007-11-24 18:17:14.000000000 +0100 @@ -45,6 +45,13 @@ struct dp { int filter; }; +/*! bind dpi with service + */ +struct service { + char *name; + int dp_index; +}; + /*! Number of available plugins */ int numdpis; @@ -54,6 +61,9 @@ int numsocks; /*! State information for each plugin. */ struct dp *dpi_attr_list; +/*! service served for each plugin */ +Dlist *services_list; + /*! Set of sockets watched for connections */ fd_set sock_set; @@ -70,6 +80,8 @@ void free_dpi_attr(struct dp *dpi_attr); void free_plugin_list(struct dp **dpi_attr_list_ptr, int numdpis); +void free_services_list(Dlist *s_list); + enum file_type get_file_type(char *file_name); int get_dpi_attr(char *dpi_dir, char *service, struct dp *dpi_attr); @@ -78,6 +90,10 @@ int register_service(struct dp *dpi_attr int register_all(struct dp **attlist); +static int services_alpha_comp(const struct service *s1, const struct service *s2); + +int fill_services_list(struct dp **attlist, int numdpis, Dlist *services_list); + int init_srs_socket(char *sockdir); int init_dpi_socket(struct dp *dpi_attr, char *sockdir); @@ -98,6 +114,8 @@ int register_all_cmd(char *sockdir); char *get_message(int sock, char *dpi_tag); +int service_match(const struct service *A, const char *B); + void send_sockpath(int sock, char * dpi_tag, struct dp *dpi_attr_list); #endif diff -pru dillo2/dpid/main.c g/dpid/main.c --- dillo2/dpid/main.c 2007-10-07 00:36:34.000000000 +0200 +++ g/dpid/main.c 2007-11-24 18:17:14.000000000 +0100 @@ -228,6 +228,7 @@ int main(void) fd_set selected_set; dpi_attr_list = NULL; + services_list = NULL; //daemon(0,0); /* Use 0,1 for feedback */ /* todo: call setsid() ?? */ @@ -258,6 +259,11 @@ int main(void) exit(1); } + /* Init services list */ + services_list = dList_new(8); + /* And get server list */ + fill_services_list(&dpi_attr_list, numdpis, services_list); + /* Remove any sockets that may have been leftover from a crash */ cleanup(sockdir); /* Initialise sockets */ ---------- proto_services.diff ----------------------------------------- diff -pru dillo2/src/capi.c f/src/capi.c --- dillo2/src/capi.c 2007-11-17 15:18:37.000000000 +0100 +++ f/src/capi.c 2007-11-21 04:35:22.000000000 +0100 @@ -240,8 +240,12 @@ static int Capi_dpi_verify_request(Dillo static int Capi_url_uses_dpi(DilloUrl *url, char **server_ptr) { char *p, *server = NULL, *url_str = URL_STR(url); + Dstr *tmp; - if (dStrncasecmp(url_str, "dpi:/", 5) == 0) { + if ((dStrncasecmp(url_str, "http:", 5) == 0) || + (dStrncasecmp(url_str, "about:", 6) == 0)) { + /* Doing nothing returns 0 (url not uses dpi) and server = NULL */ + } else if (dStrncasecmp(url_str, "dpi:/", 5) == 0) { /* dpi prefix, get this server's name */ if ((p = strchr(url_str + 5, '/')) != NULL) { server = dStrndup(url_str + 5, (uint_t)(p - url_str - 5)); @@ -252,16 +256,11 @@ static int Capi_url_uses_dpi(DilloUrl *u dFree(server); server = dStrdup("bookmarks"); } - - } else if (dStrncasecmp(url_str, "ftp:/", 5) == 0) { - server = dStrdup("ftp"); - - } else if (dStrncasecmp(url_str, "https:/", 7) == 0) { - server = dStrdup("https"); - } else if (dStrncasecmp(url_str, "file:", 5) == 0) { - server = dStrdup("file"); - } else if (dStrncasecmp(url_str, "data:", 5) == 0) { - server = dStrdup("datauri"); + } else if ((p = strchr(url_str, ':')) != NULL) { + tmp = dStr_new("proto."); + dStr_append_l(tmp, url_str, p - url_str); + server = tmp->str; + dStr_free(tmp, 0); } return ((*server_ptr = server) ? 1 : 0);

Nov. 26, 2007

Implemented a weighted apportionment algorithm for table rowspan.

by Johannes.Hofmann＠gmx.de

Hi Jorge, On Sun, Nov 25, 2007 at 02:28:55PM -0300, Jorge Arellano Cid wrote: > Hi there, > > A new, weighted apportionment algorithm for table rowspans was > committed. This wasn't implemented in dillo2, and the algorithm > is different from the one in dillo1. > > This produces much better table rendering when rowspan is used > (if there's no rowspan in the table, it's the same as before). > You can notice the rowspan case in the Web because of a stdout > message like this one: > > "Short cell 0, sumRows=66 ih=110" > > It is interesting to compare dillo2.old, dillo2.rowspan, dillo1 > and firefox. You can use: > > http://www.dillo.org/test/rowspan.html > > > Sebastian: I added some comments in rounding-errors.doc, > explained the algorithm, including a formula. I also corrected an > index in the previous section. Please check it in > rounding-errors.html after re-running doxygen, or in the attached > patch (already in CVS). > > Notes: > > * In the test page above, the last table in the second set > shows "cellspacing" is not working. AFAIU this is not implemented > and subject to another patch. > > * The vertical alignment of a cell content is not yet > implemented in dillo2. That's why the cell content is aligned at > the top instead of the middle (default, as firefox does). This is > subject of another patch. > > * The same algorithm may be used for width apportionment. I'm > not sure on this though. > > > Sebastian: are you planning to work on this soon? If not, I can > try to work on the above mentioned problems. Advice welcomed! > > Everybody: please enjoy this patch, it was a lot of work! :-) > if you find a URL where it helps, I'd appreciate some feedback. It helps on http://www.heise.de With the patch the news start on top of the page - as they should. Thanks for your work on this! Johannes

Nov. 25, 2007