On Mon, Oct 04, 2010 at 02:16:57PM +0200, Johannes Hofmann wrote:

On Mon, Oct 04, 2010 at 07:59:45AM -0400, Jorge Arellano Cid wrote:

...

On Sun, Oct 03, 2010 at 09:16:30PM +0200, Johannes Hofmann wrote:

...

[...] BTW, what do you think about adding the css_compat/ directory to the repo (under doc)? That would give us version control and it would document our progress.

Having the test cases visible as HTML from the site is very helpful IMHO. Unfortunately I don't yet know of a way of browsing raw files from hg web interfaces (it may be doable).

For my own website I have

[hooks] changegroup = hg up

in the .hgrc. So everytime I push to it, the files are updated. We could do the same for the CSS tests.

Yes, it seems fair enough. FWIW, so long we've had the HTML test suite, and the personal test pages each one of us develops. As you explained, the problem is how to automate HTML rendering tests. I hesitate to include an html test-page collection in the tarball because it would grow bigger and bigger and nobody would apply to test each one everytime a relevant change is made! :) I like the CSS checklist we have because it is sorted by property and provides test-cases we can reference by URL, and it lives outside the tarball so its size can grow independently. A fast way to check the tests is to provide an image with the correct rendering in the test page (maybe with an explanation too). It takes more time to prepare though, and each time you update the page you have to also update the screenshot. Quite a burden. I don't have a good solution yet. These days I've been working with Gecko and Dillo. Having the test page showed in both provides for a "dynamic screenshot" from Gecko. This avoids the burden of updating the image, with the advantage of showing how Gecko renders unespecified parts of the SPEC. An HTML test-page, with written explanations of what to check for, and two browsers is the fastest/simplest solution I see so far. (It also has the advantage of almost anyone being able to do the test). If we have it online (i.e. with http URLs), better, and if we had some version control it may be what we need.

...

Besides that I don't see the benefits of having css_compat/ under version control, and our progress is more or less in the hg log, but I see it can be more tightly coupled with the test case if the extended comment makes a reference to it for instance. Please explain me a bit because it may be I don't see the gains because of lack of knowledge on my side!

Basically I like to do almost everything under version control. Just yesterday I almost overwrote your vertical-align.html with my own version! Also just pushing my changes to dillo.org would be faster than copying the files manually. Do you have root on dillo.org? If so, you could install mercurial and I will set up a repo for the CSS tests. We can decide later whether we want to integrate it into the main dillo repo or not.

Not me, but you can ask Andreas Kemnade. He's our savvy sysadmin.

...

OTOH, yes, I see dillo would be in better shape test-wise if we could include an [automated] test framework in the tarball. This wasn't done because it takes some time and needs manpower we didn't have.

At "Henning's company" we are also thinking about automated tests for dillo. My current idea is to create a new view, that writes text that can easily be compared with expected results. That would be more flexible than to compare bitmaps of rendered output, which might give to many false positves. If this works out we can think about integrating the framework into dillo mainline, but currently it's just an idea.

I have some automated tests too (e.g. URL resolver), but rendering is a different beast. That's my thoughts. -- Cheers Jorge.-