- Dillo-dev - lists.mailman3.com

Quoted attribute parsing: summary

by Johannes.Hofmann＠gmx.de

On Wed, Aug 18, 2010 at 12:05:30AM +0100, Jeremy Henty wrote: > > Johannes Hofmann wrote: > > > But for me firefox 3.6.3 shows something given the following HTML > > (as does current dillo): > > > > <div title="foo >hello world</div>dillo is great > > That's Firefox's workaround that I described in my original post: if > it sees EOF while parsing a quoted attribute value (ie. if it *never* > sees a matching quote) then it goes back to the opening quote, > discards it, and parses an unquoted attribute value. So it ends up > parsing your example exactly as it would parse > > <div title=foo >hello world</div>dillo is great > > which gives the same result as vanilla Dillo, but for entirely > different reasons. > > But Firefox only does that if it can't find the matching quote at all; > if you feed it > > <div title="foo >hello world</div>dillo is great [... repeat > 'dillo is great' 10000 times ...]</div><div title="bar"> > > then it matches the second double quote with the first and *all* the > text disappears. Which is exactly what HTML5 says it should do. Of > course vanilla Dillo does *better* than Firefox for this example, but > in the real world I think it does *worse*. JavaScript fragments that > confound Dillo's algorithm are far more common than examples such as > the above that it handles well. > > OK, here's a new proposal: when parsing quoted attribute values, let's > copy Firefox! That would: (a) sensibly handle the missing quotes > examples that people have suggested (which my proposed patch does not > do), (b) handle well-formed JavaScript fragments correctly (which > vanilla Dillo does not do), (c) parse well-formed HTML5 as per the > HTML5 specification, (d) conform to Firefox's established practice, > and (e) not break Reddit! That's 5 wins! > > It's true that we can't expect people to fix their HTML just because > the HTML5 specification says it's broken. And it's even less likely > that they will fix it just because it breaks in Dillo. But it is very > likely that they will fix it if it breaks in Firefox, so copying > Firefox is a good idea, even if you don't care about the HTML5 > specification. > > And, why should we care about edge cases that vanilla Dillo handles > better than Firefox, since those are precisely the cases that people > will fix to keep their Firefox users happy and that we can therefore > expect *not* to see! There's no point in having an algorithm that in > theory is better than Firefox's, because in practice it's not. > > So, why not just copy Firefox? I can't see any downside. I agree. Can you adjust the patch? Then I'd like to wait what Jorge and corvid say, but I think it's best to mimic Firefox. Regards, Johannes

Aug. 18, 2010

Quoted attribute parsing: summary

by onepoint＠starurchin.org

Johannes Hofmann wrote: > But for me firefox 3.6.3 shows something given the following HTML > (as does current dillo): > > <div title="foo >hello world</div>dillo is great That's Firefox's workaround that I described in my original post: if it sees EOF while parsing a quoted attribute value (ie. if it *never* sees a matching quote) then it goes back to the opening quote, discards it, and parses an unquoted attribute value. So it ends up parsing your example exactly as it would parse <div title=foo >hello world</div>dillo is great which gives the same result as vanilla Dillo, but for entirely different reasons. But Firefox only does that if it can't find the matching quote at all; if you feed it <div title="foo >hello world</div>dillo is great [... repeat 'dillo is great' 10000 times ...]</div><div title="bar"> then it matches the second double quote with the first and *all* the text disappears. Which is exactly what HTML5 says it should do. Of course vanilla Dillo does *better* than Firefox for this example, but in the real world I think it does *worse*. JavaScript fragments that confound Dillo's algorithm are far more common than examples such as the above that it handles well. OK, here's a new proposal: when parsing quoted attribute values, let's copy Firefox! That would: (a) sensibly handle the missing quotes examples that people have suggested (which my proposed patch does not do), (b) handle well-formed JavaScript fragments correctly (which vanilla Dillo does not do), (c) parse well-formed HTML5 as per the HTML5 specification, (d) conform to Firefox's established practice, and (e) not break Reddit! That's 5 wins! It's true that we can't expect people to fix their HTML just because the HTML5 specification says it's broken. And it's even less likely that they will fix it just because it breaks in Dillo. But it is very likely that they will fix it if it breaks in Firefox, so copying Firefox is a good idea, even if you don't care about the HTML5 specification. And, why should we care about edge cases that vanilla Dillo handles better than Firefox, since those are precisely the cases that people will fix to keep their Firefox users happy and that we can therefore expect *not* to see! There's no point in having an algorithm that in theory is better than Firefox's, because in practice it's not. So, why not just copy Firefox? I can't see any downside. Regards, Jeremy Henty

Aug. 18, 2010

Quoted attribute parsing: summary

by Johannes.Hofmann＠gmx.de

On Tue, Aug 17, 2010 at 09:13:11PM +0100, Jeremy Henty wrote: > > Johannes Hofmann wrote: > > > I'd agree that we should not make compromises displaying correct > > HTML when trying to deal with buggy HTML. But are the '>' > > characters in the attribute value in the reddit page actually valid? > > Yes, at least according to the HTML5 specification. Indeed, according > to that specification, the only possible parse errors while parsing a > quoted attribute value are (i) EOF, and (ii) a malformed entity > reference. Anything else is valid! > > I doubt that those '>' characters are valid according to SGML, but the > HTML5 specification explicitly states that HTML5 is not an SGML > instance. No popular client has ever parsed HTML as an SGML instance > and servers have been sending non-SGML-compliant "HTML" since forever. > No matter what earlier HTML specifications may have claimed, the > practical reality is that HTML has never been a kind of SGML. > > > The HTML validators at least warn about them. > > Warning about them is probably a good idea, but that's a different > issue from how to handle them. Whatever Dillo should do, its current > behaviour (a) does not conform to HTML5, and (b) breaks Reddit. > > Of course, there's no reason that Dillo *must* conform to HTML5. > Indeed the HTML5 specification is peppered with the lovely phrase > "willful violation", meaning "yes we know this breaks someone else's > specification but we think it's for the best". So it's fine in > principle for Dillo to say "we're going to violate HTML5 because we > think it's for the best", but I think that this particular behaviour > is a bad idea. It violates the HTML5 standard, it deviates (as far as > I can tell) from standard practice, and it breaks an otherwise > perfectly compliant website. If we can think of a useful way to > willful violate standards so as to better handle broken HTML then > let's do it, but I think Dillo is better off without this particular > workaround because it does more harm than good. > > NB: HTML5 is still a work in progress. These bug reports show some of > the discussion of parsing attribute values: > > Bug 9872: "trigger a conformance error when javascript is included > in href attribute" (rejected because there are legitimate use > cases and even if it's sometimes abused it's not the HTML5 > specification's role to police its use) > > http://www.w3.org/Bugs/Public/show_bug.cgi?id=9872 > > Bug 9987: "attribute values should be allowed to contain ambiguous > ampersands ..." (still new) > > http://www.w3.org/Bugs/Public/show_bug.cgi?id=9987 I also played with this some more... If all standard browsers would insist on correctly closed quotes, we could expect web-developers to immediately fix those errors. But for me firefox 3.6.3 shows something given the following HTML (as does current dillo): <div title="foo >hello world</div>dillo is great When we remove the unquoted attribute detection as in your patch, nothing is shown. Are newer firefox versions treating this differently? Regards, Johannes

Aug. 17, 2010

Quoted attribute parsing: summary

by onepoint＠starurchin.org

Johannes Hofmann wrote: > I'd agree that we should not make compromises displaying correct > HTML when trying to deal with buggy HTML. But are the '>' > characters in the attribute value in the reddit page actually valid? Yes, at least according to the HTML5 specification. Indeed, according to that specification, the only possible parse errors while parsing a quoted attribute value are (i) EOF, and (ii) a malformed entity reference. Anything else is valid! I doubt that those '>' characters are valid according to SGML, but the HTML5 specification explicitly states that HTML5 is not an SGML instance. No popular client has ever parsed HTML as an SGML instance and servers have been sending non-SGML-compliant "HTML" since forever. No matter what earlier HTML specifications may have claimed, the practical reality is that HTML has never been a kind of SGML. > The HTML validators at least warn about them. Warning about them is probably a good idea, but that's a different issue from how to handle them. Whatever Dillo should do, its current behaviour (a) does not conform to HTML5, and (b) breaks Reddit. Of course, there's no reason that Dillo *must* conform to HTML5. Indeed the HTML5 specification is peppered with the lovely phrase "willful violation", meaning "yes we know this breaks someone else's specification but we think it's for the best". So it's fine in principle for Dillo to say "we're going to violate HTML5 because we think it's for the best", but I think that this particular behaviour is a bad idea. It violates the HTML5 standard, it deviates (as far as I can tell) from standard practice, and it breaks an otherwise perfectly compliant website. If we can think of a useful way to willful violate standards so as to better handle broken HTML then let's do it, but I think Dillo is better off without this particular workaround because it does more harm than good. NB: HTML5 is still a work in progress. These bug reports show some of the discussion of parsing attribute values: Bug 9872: "trigger a conformance error when javascript is included in href attribute" (rejected because there are legitimate use cases and even if it's sometimes abused it's not the HTML5 specification's role to police its use) http://www.w3.org/Bugs/Public/show_bug.cgi?id=9872 Bug 9987: "attribute values should be allowed to contain ambiguous ampersands ..." (still new) http://www.w3.org/Bugs/Public/show_bug.cgi?id=9987 Regards, Jeremy Henty

Aug. 17, 2010

Quoted attribute parsing: summary

by Johannes.Hofmann＠gmx.de

On Mon, Aug 16, 2010 at 07:21:02AM +0100, Jeremy Henty wrote: > > Prompted by some private conversation with corvid I've been digging > through specs and source code to see what the state of play is. > > The HTML5 specification[1] states that the user agent should consume > text, converting character references until it finds the matching > close quote. If there is no matching close quote (ie. it sees an EOF > first) then it terminates (strictly speaking, it switches to the data > state and reconsumes the EOF, which makes it emit an EOF token). > > Taking out Dillo's bogus attribute value detection as I proposed would > make Dillo parse quoted attribute values as per the HTML5 spec. > > The Hubbub HTML parser library[2] parses quoted attribute values as > per the HTML5 spec. > > Firefox parses quoted attribute values as per the HTML5 spec *except* > that if it sees an EOF then it backs up to the open quote, discards > it, then reparses as though it was expecting an unquoted attribute > value. Otherwise (ie. if it finds the matching close quote) it makes > no attempt to detect a broken attribute value, no matter what content > the attribute value has swallowed up. > > So it seems that the world at large has given up on trying to detect > and correct broken attribute values. I'd agree that we should not make compromises displaying correct HTML when trying to deal with buggy HTML. But are the '>' characters in the attribute value in the reddit page actually valid? The HTML validators at least warn about them. Cheers, Johannes

Aug. 16, 2010

Quoted attribute parsing: summary

by onepoint＠starurchin.org

Prompted by some private conversation with corvid I've been digging through specs and source code to see what the state of play is. The HTML5 specification[1] states that the user agent should consume text, converting character references until it finds the matching close quote. If there is no matching close quote (ie. it sees an EOF first) then it terminates (strictly speaking, it switches to the data state and reconsumes the EOF, which makes it emit an EOF token). Taking out Dillo's bogus attribute value detection as I proposed would make Dillo parse quoted attribute values as per the HTML5 spec. The Hubbub HTML parser library[2] parses quoted attribute values as per the HTML5 spec. Firefox parses quoted attribute values as per the HTML5 spec *except* that if it sees an EOF then it backs up to the open quote, discards it, then reparses as though it was expecting an unquoted attribute value. Otherwise (ie. if it finds the matching close quote) it makes no attempt to detect a broken attribute value, no matter what content the attribute value has swallowed up. So it seems that the world at large has given up on trying to detect and correct broken attribute values. Jeremy Henty [1] http://www.whatwg.org/specs/web-apps/current-work/multipage/ [2] http://www.netsurf-browser.org/projects/hubbub/

Aug. 16, 2010

Debian/Ubuntu packages available

by rogerx＠sdf.org

On Sun, Aug 08, 2010 at 06:54:25PM +0200, Devid Antonio Filoni wrote: >I sent an email time ago with that link, I don't know why dillo.org >was not updated. However there is also >https://edge.launchpad.net/~d.filoni/+archive/dillo+ssl/+packages No problem here... Gentoo is up to date. ;-) -- Roger http://rogerx.freeshell.org/

Aug. 9, 2010

Debian/Ubuntu packages available

by devidfil＠gmail.com

I sent an email time ago with that link, I don't know why dillo.org was not updated. However there is also https://edge.launchpad.net/~d.filoni/+archive/dillo+ssl/+packages Devid On Sun, Aug 8, 2010 at 10:12 AM, Alexander Voigt <Hole.destructor(a)gmx.de> wrote: > Dear Dillo developers and users, > > I found that Devid Antonio Filoni provides Debian/Ubuntu packages of > the current dillo release at > > ? https://launchpad.net/~d.filoni/+archive/dillo/+packages > > Maybe this is worth to mentioned at dillo.org. > > Kind regards, > Alexander Voigt > > _______________________________________________ > Dillo-dev mailing list > Dillo-dev(a)dillo.org > http://lists.auriga.wearlab.de/cgi-bin/mailman/listinfo/dillo-dev >

Aug. 8, 2010

Debian/Ubuntu packages available

by Hole.destructor＠gmx.de

Dear Dillo developers and users, I found that Devid Antonio Filoni provides Debian/Ubuntu packages of the current dillo release at https://launchpad.net/~d.filoni/+archive/dillo/+packages Maybe this is worth to mentioned at dillo.org. Kind regards, Alexander Voigt

Aug. 8, 2010

[patch]: do not let '>' terminate a quoted attribute value

by jcid＠dillo.org

On Sun, Aug 01, 2010 at 12:56:23PM +0100, Jeremy Henty wrote: > > If you point dillo at http://www.reddit.com/ you will see many > javascript fragments. These come from div elements with an > onclick="$(this).vote('<$>votehash</$>', null, event)" attribute. > Dillo thinks that the '>' looks suspiciously like an end tag > character. It looks ahead, sees the '<' character and decides that > its suspicions are correct, so it terminates the tag at the '>', warns > of a missing attribute close quote and renders the rest as HTML. > > This algorithm may have been effective in the early days of the web > but I think it clearly does not work well today. I can't think of a > better way of doing the job so I suggest removing it and instead just > finding the matching close quote in all cases. Patch attached. Effectively, this is a problem. Unfortunately the final-quote approach creates more trouble than it seems. See doc/HtmlParser.txt for an explanation. A simple heuristic that can tackle both cases equally well, would certainly be appreciated. For instance, if there's a missing quote (according to current heuristics), and there's an unmatched '(' in the atribute value, we may try to look for a matching ')' in the vicinity, and if found, look for the closing quote afterwards. If found, we can change the first decision. This approach has low testing cost (O(n) in case of missing quote), and may help discriminate both cases: real missing quote and scripting language. -- Cheers Jorge.-

Aug. 6, 2010