extra dependancies vs. testability?

DBIx::Transaction currently only depends on DBI, but for it's unit
tests to run you need a DBI driver available. If SQLite2 is available, it
gives you a nice default DSN to test DBIx::Transaction in a completely
self-contained environment.

The problem I'm facing is that smoke testers (notably CPAN testers
and ppm.activestate.com) can't automatically test this module properly
because the test dependancy on having some sort of DSN isn't made obvious to
them. I'm in a bit of a dilemma here. I'd like my package to be
automatically testable against some sort of "real" DBD driver, but I don't
neccessarily want every person who installs my module to be obliged to
install DBD::SQLite2 alongside it. As nice of a module as it is, it seems
like bloat if they're never going to use it for anything else.

What do you think I should do about this? Bite the bullet and depend
on SQLite2? Create an overly complex mock object that doesn't emulate any
"real" DBD driver's behaviour? Or is there another, better solution?

Thanks,
Tyler

RE: extra dependancies vs. testability?

I haven't used this module myself, but perhaps create two test suites: =
"make test" tests everything you can test without burdensome external =
dependencies, and "make testsql" either depends specifically SQLite2 or =
using a user-provided DSN. The users who want to put more effort into =
testing can invoke the latter. I'm not sure if this is an appropriate =
solution for "smoke testing", though.

Ted Behling

-----Original Message-----
From: Tyler MacDonald [mailto:tyler@yi.org]
Sent: Tuesday, January 10, 2006 9:43 PM
To: dbi-users@perl.org
Subject: extra dependancies vs. testability?


DBIx::Transaction currently only depends on DBI, but for it's unit
tests to run you need a DBI driver available. If SQLite2 is available, =
it
gives you a nice default DSN to test DBIx::Transaction in a completely
self-contained environment.

The problem I'm facing is that smoke testers (notably CPAN testers
and ppm.activestate.com) can't automatically test this module properly
because the test dependancy on having some sort of DSN isn't made =
obvious to
them. I'm in a bit of a dilemma here. I'd like my package to be
automatically testable against some sort of "real" DBD driver, but I =
don't
neccessarily want every person who installs my module to be obliged to
install DBD::SQLite2 alongside it. As nice of a module as it is, it =
seems
like bloat if they're never going to use it for anything else.

What do you think I should do about this? Bite the bullet and depend
on SQLite2? Create an overly complex mock object that doesn't emulate =
any
"real" DBD driver's behaviour? Or is there another, better solution?

Thanks,
Tyler

RE: extra dependancies vs. testability?

Sorry for replying to my own e-mail, but I had another thought. What =
about just failing gracefully if SQLite2 isn't available? If it's =
installed, it gets used; if not, a warning is shown and the dependent =
tests are skipped.=20

-----Original Message-----
From: Ted Behling=20
Sent: Tuesday, January 10, 2006 10:12 PM
To: dbi-users@perl.org
Subject: RE: extra dependancies vs. testability?


I haven't used this module myself, but perhaps create two test suites: =
"make test" tests everything you can test without burdensome external =
dependencies, and "make testsql" either depends specifically SQLite2 or =
using a user-provided DSN. The users who want to put more effort into =
testing can invoke the latter. I'm not sure if this is an appropriate =
solution for "smoke testing", though.

Ted Behling

-----Original Message-----
From: Tyler MacDonald [mailto:tyler@yi.org]
Sent: Tuesday, January 10, 2006 9:43 PM
To: dbi-users@perl.org
Subject: extra dependancies vs. testability?


DBIx::Transaction currently only depends on DBI, but for it's unit
tests to run you need a DBI driver available. If SQLite2 is available, =
it
gives you a nice default DSN to test DBIx::Transaction in a completely
self-contained environment.

The problem I'm facing is that smoke testers (notably CPAN testers
and ppm.activestate.com) can't automatically test this module properly
because the test dependancy on having some sort of DSN isn't made =
obvious to
them. I'm in a bit of a dilemma here. I'd like my package to be
automatically testable against some sort of "real" DBD driver, but I =
don't
neccessarily want every person who installs my module to be obliged to
install DBD::SQLite2 alongside it. As nice of a module as it is, it =
seems
like bloat if they're never going to use it for anything else.

What do you think I should do about this? Bite the bullet and depend
on SQLite2? Create an overly complex mock object that doesn't emulate =
any
"real" DBD driver's behaviour? Or is there another, better solution?

Thanks,
Tyler

Re: extra dependancies vs. testability?

Ted Behling wrote:
> Sorry for replying to my own e-mail, but I had another thought. What
> about just failing gracefully if SQLite2 isn't available? If it's
> installed, it gets used; if not, a warning is shown and the dependent
> tests are skipped.

That's what it does right now. The tests don't depend on SQLite2
per-se, just a DSN that points to a DBI driver that supports transactions
(eg; pretty much everything except MySQL ;-)

But then groups like ActiveState and Debian might end up packaging
this thing up (automatically or not) without ever running the unit tests
properly against their distributions, because "make test" passes by skipping
all the relevant tests. IMHO that's bad Quality Assurance on their part and
mine... so I'm striving for something better.

Another option that occured to me while I was loading the dishwasher
was bundling SQLite2 (or something like it) in my t/ directory. Sure, it
will consume a bit of extra bandwidth when the package is downloaded, but it
won't leave the bloat around once it's tested, installed (or packaged up
into something like a .deb or .ppd), and the source tree is purged. But I
still find myself thinking "there's got to be a better way"...

- Tyler

Re: extra dependancies vs. testability?

At 7:26 PM -0800 1/10/06, Tyler MacDonald wrote:
> Another option that occured to me while I was loading the dishwasher
>was bundling SQLite2 (or something like it) in my t/ directory. Sure, it
>will consume a bit of extra bandwidth when the package is downloaded, but it
>won't leave the bloat around once it's tested, installed (or packaged up
>into something like a .deb or .ppd), and the source tree is purged. But I
>still find myself thinking "there's got to be a better way"...

Actually bundling SQLite is a waste of space for a CPAN distribution,
partly because it is quite large relative to the site of your own
code; like 10-100X.

As for answering your question, I suggest including a small config
file with your distro which your test script looks at. Users can
change that if they are testing an existing database setup of theirs,
and if they don't customize this file, your test script will default
to using SQLite so that the test suite can still test your module
itself.

That's what the current/old Rosetta::Engine::Generic on CPAN does
with its test suite, defaulting to SQLite (3); you can look at its
tests for ideas.

FYI, the not yet released Rosetta rewrite will be different, and
include its own mock database to test with (called
Rosetta::Engine::Native), so that testing the core can be done
completely without any third party database.

Depending on your goals, it might be worthwhile for you to have a
mock database.

But whatever you do, don't bundle third party things like SQLite,
unless your distribution is private/proprietary and the recipients
aren't going to CPAN.

-- Darren Duncan

Re: extra dependancies vs. testability?

Robert Loomans wrote:
> > That's what it does right now. The tests don't depend on SQLite2
> > per-se, just a DSN that points to a DBI driver that supports transactions
> > (eg; pretty much everything except MySQL ;-)
> Errr.... MySQL supports transactions just fine at least since 4.0.

Only if you cast your tables as the appropriate type (eg; InnoDB),
and your distribution hasn't screwed you out of that by setting the
appropriate my.cnf options (eg; skip-innodb is default in debian).

> With something like this, it's more important that the code works than
> inconviencing the packager. The packager will note the dependancy as a
> *build* dependancy and move on.

Agreed.

> For non-packaged builds, I would be more interested in the package
> testing okay than the extra dependancy.
>
> If you really want, make the SQLite tests optional if some environment
> variable is set (eg, SKIP_SQLLITE_TEST).

OK, I like the environment variable thing. That gives me the
opportunity to warn a user installing manually via CPAN that they don't have
to install SQLite2 if they don't want.

I'm going to chew on this for awhile (and get away from the keyboard
before my 5 mo. old daughter chews off my arm :)

Thanks,
Tyler

Re: extra dependancies vs. testability?

On Tue, 10 Jan 2006 20:00:14 -0800, Tyler MacDonald wrote:

Hi Tyler

> OK, I like the environment variable thing. That gives me the
> opportunity to warn a user installing manually via CPAN that they
> don't have to install SQLite2 if they don't want.

Just for the record: When I install a set of modules on a PC not connected=
to
the internet, or install just your module via a CGI script, I /hope/ your=
code
doesn't stop and ask a question assuming someone's watching the command=
line...

--
Cheers
Ron Savage, ron@savage.net.au on 11/01/2006
http://savage.net.au/index.html
Let the record show: Microsoft is not an Australian company

Re: extra dependancies vs. testability?

Ron Savage wrote:
> On Tue, 10 Jan 2006 20:00:14 -0800, Tyler MacDonald wrote:
>
> Hi Tyler
>
> > OK, I like the environment variable thing. That gives me the
> > opportunity to warn a user installing manually via CPAN that they
> > don't have to install SQLite2 if they don't want.
>
> Just for the record: When I install a set of modules on a PC not connected to
> the internet, or install just your module via a CGI script, I /hope/ your code
> doesn't stop and ask a question assuming someone's watching the command line...

Currently, it *does* use the prompt() function supplied by
ExtUtils::MakeMaker, which is supposed to prompt you if you have a
controlling terminal, so long as PERL_MM_USE_DEFAULT is not set.

Not all autobuild systems seem to set that properly though (or there
may be something else going on here, I've asked gozer if he has any tips):

http://ppm.activestate.com/BuildStatus/5.8-windows/windows-5 .8/DBIx-Transaction-0.003.txt

Cheers,
Tyler

Re: extra dependancies vs. testability?

Tyler MacDonald wrote:
> > If you really want, make the SQLite tests optional if some environment
> > variable is set (eg, SKIP_SQLLITE_TEST).
>
> OK, I like the environment variable thing. That gives me the
> opportunity to warn a user installing manually via CPAN that they don't have
> to install SQLite2 if they don't want.
>
> I'm going to chew on this for awhile (and get away from the keyboard
> before my 5 mo. old daughter chews off my arm :)

I've had some time to digest this, and it seems that rather than
coining an entirely new environment variable, I could use DBI_DSN:

- If DBI_DSN is set, use that as the DSN for testing, and don't
depend on SQLite.

- If DBI_DSN is set to an empty string, don't do any testing, and
don't depend on SQLite.

- If DBI_DSN is not set, default to depending on, and testing
against, SQLite.

Cheers,
Tyler

Re: extra dependancies vs. testability?

On Wed, 11 Jan 2006 13:19:41 -0800, Tyler MacDonald wrote:

Hi Tyler

> I've had some time to digest this, and it seems that rather than
> coining an entirely new environment variable, I could use DBI_DSN:

Good!

Here are my votes:

> - If DBI_DSN is set, use that as the DSN for testing, and don't
> depend on SQLite.

Yes.

> - If DBI_DSN is set to an empty string, don't do any testing, and
> don't depend on SQLite.

Yes.

> - If DBI_DSN is not set, default to depending on, and testing
> against, SQLite.

No. People should have to explicitly set DBI_DSN to make it do something.
So, unset should be the same as set to empty string.
--
Cheers
Ron Savage, ron@savage.net.au on 12/01/2006
http://savage.net.au/index.html
Let the record show: Microsoft is not an Australian company

Re: extra dependancies vs. testability?

On Wed, 11 Jan 2006 13:07:18 -0800, Tyler MacDonald wrote:

Hi Tyler

> Currently, it *does* use the prompt() function supplied by
> ExtUtils::MakeMaker, which is supposed to prompt you if you have a
> controlling terminal, so long as PERL_MM_USE_DEFAULT is not set.

Excellent! Much appreciated.

--
Cheers
Ron Savage, ron@savage.net.au on 12/01/2006
http://savage.net.au/index.html
Let the record show: Microsoft is not an Australian company

Re: extra dependancies vs. testability?

Ron Savage wrote:
> > - If DBI_DSN is not set, default to depending on, and testing
> > against, SQLite.
> No. People should have to explicitly set DBI_DSN to make it do something.
> So, unset should be the same as set to empty string.

*whines* But this would break the functionality I was looking for in
the first place: good automated QA being in a somewhat self-contained way.
With SQLite2, the tests can run automatically against a database stored in a
temporary file. I doubt CPAN testers / ppm.activestate.com / etc are going
to be setting DBI_DSN just to test this module. Without *some* DSN to test
against, virtually no testing can happen at all, which means that a "PASS"
on testers.cpan.org doesn't neccessarily mean the code works on that
platform.

- Tyler

Re: extra dependancies vs. testability?

On Wed, 11 Jan 2006 13:45:30 -0800, Tyler MacDonald wrote:

Hi Tyler

I do understand your point, but I think you need to re-think this.

> *whines* But this would break the functionality I was looking for
> in the first place: good automated QA being in a somewhat self-

So far so good.

> contained way. With SQLite2, the tests can run automatically
> against a database stored in a temporary file. I doubt CPAN testers

Still good.

> / ppm.activestate.com / etc are going to be setting DBI_DSN just to
> test this module. Without *some* DSN to test against, virtually no

But that's just it. They are precisely the target users who will install=
SQLite
to help testing, surely? I don't have it installed under Windows (search the=

archives for why I don't trust it, or didn't way back when).

So, this means you have 2 target user groups:

o Serious testers, who, I assume, /will be/ prepared to install a testing
prerequisite

o Other users, who assume the above testers did their self-appointed :-)=
job, or
even if the testers did not test their (the users) platform, are prepared to=

take the risk

> testing can happen at all, which means that a "PASS" on
> testers.cpan.org doesn't neccessarily mean the code works on that
> platform.

It's great to have testing, and more testing, but I wonder if you're not=
placing
too much emphasis on the /end-user/ level testing as against your own=
testing,
and other testers' tests.

Yes, I know you can't test various platforms, that's what the other testers=
are
for.

I have published a number of modules after testing them to my satisfaction,=
but
which do /not/ ship with complex testing, eg modules to be used against
databases, or to be used in CGI scripts.

In other words, you may have to accept that you'll be shipping code which=
has
not yet had an ideal (perfectionistic) level of testing...
--
Cheers
Ron Savage, ron@savage.net.au on 12/01/2006
http://savage.net.au/index.html
Let the record show: Microsoft is not an Australian company

Re: extra dependancies vs. testability?

> > - If DBI_DSN is not set, default to depending on, and testing
> > against, SQLite.
>
> No. People should have to explicitly set DBI_DSN to make it do something.
> So, unset should be the same as set to empty string.

I don't agree.

As a compromise, Tyler, how 'bout you only run the tests in this case if
SQLite is *already* installed? Other modules already do things this way.
eg, look at Log::Log4perl and the tests for LWP, RRD, Log::Dispatch, and
Log::Dispatch::FileRotate: it skips any for which the module doesn't
exist.

Rob

--
Robert Loomans
robert@loomans.org

Re: extra dependancies vs. testability?

On Thu, 12 Jan 2006 09:56:55 +1000, Robert Loomans wrote:

Hi Robert

> As a compromise, Tyler, how 'bout you only run the tests in this
> case if SQLite is *already* installed? Other modules already do
> things this way. eg, look at Log::Log4perl and the tests for LWP,

Yeah, this should be ok.
--
Cheers
Ron Savage, ron@savage.net.au on 12/01/2006
http://savage.net.au/index.html
Let the record show: Microsoft is not an Australian company