How to be a CPAN tester
Barbie
David Golden
Who We Are
- Barbie
- CPAN Tester (2003-2004)
- Co-developer of CPAN-YACSmoke
- Maintainer of CPAN Testers Statistics
- Caretaker for CPAN Testers
- David Golden
- CPAN Tester (2004-present)
- Creator of CPAN-Reporter
- Creator of Vanilla & Strawberry Perl
Who We Are
- Barbie
- CPAN Tester (2003-2004)
- Co-developer of CPAN-YACSmoke
- Maintainer of CPAN Testers Statistics
- Caretaker for CPAN Testers
- David Golden
- CPAN Tester (2004-present)
- Creator of CPAN-Reporter
- Creator of Vanilla & Strawberry Perl
What is CPAN testing?
- Builds on distribution tests (*.t files and TAP)
- Distributed
- Heterogenous
- Closed-loop
- Anarchic (Chaotic?)
Who benefits from CPAN testing?
Who benefits from CPAN testing?
- Authors
- Users
- Corporates
- Small Business
- Individuals
Who benefits from CPAN testing?
- Authors
- Users
- Corporates
- Small Business
- Individuals
- Perl
How does CPAN testing work?
- PAUSE announces a new upload
How does CPAN testing work?
- PAUSE announces a new upload
- Distribution downloaded
How does CPAN testing work?
- PAUSE announces a new upload
- Distribution downloaded
- Users run tests (manual or automated)
How does CPAN testing work?
- PAUSE announces a new upload
- Distribution downloaded
- Users run tests (manual or automated)
- Tests results sent to CPAN Testers mailing list
How does CPAN testing work?
- PAUSE announces a new upload
- Distribution downloaded
- Users run tests (manual or automated)
- Tests results sent to CPAN Testers mailing list
- Results tabulated by CPAN-WWW-Testers
How does CPAN testing work?
- PAUSE announces a new upload
- Distribution downloaded
- Users run tests (manual or automated)
- Tests results sent to CPAN Testers mailing list
- Results tabulated by CPAN-WWW-Testers
- Results available on http://www.cpantesters.org/
How does CPAN testing work?
- PAUSE announces a new upload
- Distribution downloaded
- Users run tests (manual or automated)
- Tests results sent to CPAN Testers mailing list
- Results tabulated by CPAN-WWW-Testers
- Results available on http://www.cpantesters.org/
- Author notified of FAIL and UNKNOWN reports
Types of Reports
- PASS - all the tests in test suite passed
- NA - distribution not installable
- FAIL - one or more tests failed
- UNKNOWN - distribution has no test :(
What to test?
- Correctly formatted distribution uploads
My-Module-0.01.tar.gz
My-Module-0.01.tar.bz2
My-Module-0.01.tgz
My-Module-0.01.zip
My-Module.tar.gz
My-Module-0.01.tar
My-Module-0.01.par
- Format supported by installers and reporting tools
- .tar.bz2 is not currently supported by the installers
- .ppm.tar.gz, .par and other non-source based distributions are ignored
What to report?
- Test using a supported test mechanism
- Include results for failures
- ALWAYS include output of 'perl -V'
- Format Subject correctly
PASS My-Module-0.01 i386-freebsd 5.1-current
PASS My-Module-0.01 i686-linux 2.4.13
My-Module.tar.gz Failed
FAIL My-Module
FAIL My-Module MSWin32-x86-multi-thread 4.0
FAIL My-Module-0.01.tar.gz sun4-solaris 2.8
Two styles of CPAN testing
- "Manual" -- during ordinary module installation
- CPAN + CPAN::Reporter
- CPANPLUS
- "Smoke" -- Automated, mass testing
- CPAN::YACSmoke
- POE::Component::CPAN::YACSmoke
Two styles of CPAN testing
- "Manual" -- during ordinary module installation
- CPAN + CPAN::Reporter
- CPANPLUS
- "Smoke" -- Automated, mass testing
- CPAN::YACSmoke
- POE::Component::CPAN::YACSmoke
CPAN::Reporter
Getting started with CPAN::Reporter
Super quick-start version...
$ cpan
cpan> install CPAN::Reporter
cpan> reload CPAN
cpan> o conf init test_report
Getting started with CPAN::Reporter
- Install distributions as normal
- Now automatically submits the test results
CPAN::Reporter Configuration
- Config file location:
- .cpantester/config.ini
- /home directory (Unix & OS X)
- "My Documents" directory (Windows)
- Created interactively from CPAN::Shell
o conf init test_report
- Just like configuring CPAN itself
- Many option take a list of actions for each grade
send_report=default:ask/yes pass:yes na:no
- Controls interactive prompts and default answers
- Just accept config defaults or else RTFM
CPAN::Reporter Config Options
email_from
cc_author
edit_report
send_report
send_duplicates
smtp_server
email_to
editor
debug
CPAN::Reporter Config Example
[CONFIG]
email_from="John Doe" <johndoe@example.com>
edit_report=default:ask/no pass:no
cc_author=default:yes pass:no
send_report=default:ask/yes pass:yes na:no
send_duplicates=default:no
smtp_server=127.0.0.1
email_to=cpan-testers@perl.org
editor=vi
debug=0
Benefits of CPAN::Reporter
- Adds CPAN test reporting to CPAN.pm
- Allows anyone who installs a distribution to contribute
- Allows testers to edit reports before sending
- Provides selective report submission based on configuration
- Keeps track of subject lines sent to CPAN Testers
- Subject line has grade, module name and perl version
- Avoids duplicate reports if re-testing
The future for CPAN::Reporter
- Include Makefile.PL or Build.PL failures
- Optionally prevent reporting dependencies
- Allow writing reports to a file (Test::Reporter/Mail::File)
- Impliment YACSmoke hooks in CPANPLUS::Internals::Constants::Reports
- Interface for CPAN.pm to pass in test output directly
CPAN::YACSmoke
Getting started with CPAN::YACSmoke
- Install CPAN::YACSmoke
- Amend configuration file (if required)
- Run yacsmoke.pl
- Run yacsmoke.pl via cron or scheduler
CPAN::YACSmoke Configuration
- cpansmoke.ini
- Defaults to local directory
- CLI: perl yacsmoke.pl -c=cpansmoke.ini
CPAN::YACSmoke Config Options
exclude_dists
test_max
fail_max
allow_retries
suppress_extracted
flush_flag
database_type
database_file
list_from
audit_log
config_file
basedir
exclude_dists - distros to be excluded from testing
test_max - distros to be tested (default 100)
fail_max - versions of a distro to be tested (default 3)
allow_retries - retest distros (default UNGRADED and ABORTED)
suppress_extracted - suppress CPANPLUS 'Extracted' messages
flush_flag - used by purge()
database_type - database plugin (default SDBM_File)
database_file - local cpansmoke DB (default cpansmoke.dat)
list_from - source plugin (default Recent)
audit_log - progress log file
config_file - alternate INI file
basedir - base directory for config file,
(defaults to the home directory)
CPAN::YACSmoke Config Example
[CONFIG]
debug=0
verbose=1
cpantest=1
allow_retries=none
suppress_extracted=1
test_max=6
exclude_dists=<<HERE
DBD
Distributed-Process
Term-ReadLine-Gnu
HERE
CPAN::YACSmoke Database Plugins
- SDBM_File (default) (#)
- SQLite (#)
- MySQL, PostgreSQL, ODBC may be considered in the future
- new plugins welcome ... upload yours to CPAN
CPAN::YACSmoke Source Plugins
- Recent (default) (#)
- SmokeDB (#)
- PlainTextList (#)
- NNTP
- NNTPWeb
- Outlook
- WebList
CPAN::YACSmoke CLI
yacsmoke [options] [dist]
Usage: perl yacsmoke.pl
[-t|--test <distribution>]
[-l|--list <source>:<param_name>=<value>]
[-f|--filter <filter>:<filter>:<filter>]
[-a|--audit <audit file>]
[-c|--config <configuration file>]
[-s|--smokedb <smoke database file>]
[-d|--database <database>]
[-h|--help]
CPAN::YACSmoke OO Style
my $smoker = CPAN::YACSmoke->new(%conf);
$smoker->test();
- test() - performs testing
- mark() - prints/returns/updates local grades
- excluded() - prints/returns excluded distributions
- listed() - prints/returns listed distributions
- purge() - removes old graded distributions from local DB
- flush() - selectively cleans the test build directory
Benefits of CPAN::YACSmoke
- Records all distributions tested
- Avoids repeated testing for a known distribution
- If a required prerequisite is NA, then tested distribution can be marked as the same
- Allows retesting when a prerequisite passes
- Tests many distributions in one single smoke run
- Tests without manual intervention (mostly)
- Sends reports without manual intervention
- Introduced $ENV{AUTOMATED_TESTING} environment variable
The future for CPAN::YACSmoke
- Formalise reports, using CPAN::Reporter style
- Allow YACSmoke to use CPAN/CPAN::Reporter or CPANPLUS
- Impliment additional CPAN::Reporter configuration for CPANPLUS
Closing the loop
Closing the loop
- CPAN Testers (main website)
- CPAN Testers statistics
- The ALL NEW ...
CPAN Testers

CPAN Testers Statistics

CPAN Testers Statistics
- http://stats.cpantesters.org/
- Statistical analysis of all uploads and reports
- Highlights environments lacking CPAN Testers support
- Emphasises CPAN Testers contributions
CPAN Testers Statistics

CPAN Testers Statistics
- Last 6 months
- 66-73 regular testers
- 32-50 different platforms
- 16-35 different perls
- Last month (May 2007)
- 1293 uploaded distributions
- 27406 reports submitted in total
- 15968 reports submitted by Chris Williams (BINGOS)
- 4703 reports submitted for i386-netbsd-thread-multi-64int
- 16085 reports submitted for 5.8.8
CPAN Testers Statistics
A Request
- Submit using a recogniseable FROM address
- DAGOLDEN@cpan.org
- "Barbie" <barbie@missbarbell.co.uk>
- "David Golden" <cpantester@example.com>
- "Barbie" <root@a.fake.address>
- Can identify you in the statistical analysis
- Authors know who to respond to
The ALL NEW ...
- Can you guess what it is yet?
CPAN Testers Wiki

The future for CPAN Testers
- Add more documentation to the CPAN Testers Wiki
- Encourage more CPAN Testers to test on other platforms/perls
- Promote CPAN Testers
PRIVACY WARNING
Test reports include information about environment
variables and special Perl variables that could be affecting test
results in order to help module authors interpret the results of the
tests. This includes information about paths, terminal, locale,
user/group ID, installed toolchain modules (e.g. ExtUtils::MakeMaker)
and so on.
These have been intentionally limited to items that should not cause
harmful personal information to be revealed -- it does not include
your entire environment. Nevertheless, if you are concerned about any
disclosure of this information as part of your test report, you are
advised to review this information and, if necessary, edit the report
prior to sending it.
Questions?

Thanks for listening.