CPAN Testers is only made possible with the support of our sponsors.
For more information on sponsoring, please visit the I CPAN Testers website.

Upgrade Notice

The CPAN Testers Blog site has been upgraded since you last accessed the site. Please press the F5 key or CTRL-R to refresh your browser cache to use the latest javascript and CSS files.

News & Views

Posted by Barbie
on 20th July 2010

Having officially announced the launch of CPAN Testers 2.0, we have now entered what David calls a Public Beta. Essentially this is the point where we start to see some real usage of the HTTP transport mechanism.

David Golden has laid out some guidelines of How to join the CPAN Testers 2.0 Public Beta on his blog, together with a Quick HowTo and the previous Beta Test Instructions on the CPAN Testers Wiki. Feel free to add further instructions to the wiki pages to help guide others through the process.

I'm currently setting up monitoring to see how well we are encouraging testers to move across to the HTTP mechanism, and we'll be contacting those that have yet to switch. If you have any questions, please subscribe to the cpan-testers-discuss mailing list and post them there.

 With the big announcement on Monday, that we are now live with CPAN Testers 2.0, the next couple of months are likely to be interesting to say the least. We now have the task of helping people set up their smoker clients to use the HTTP submission process, rather than the current SMTP defaults. In most cases it will just require some changes to existing configuration settings and the possible upgrade to the latest smoker tools. Part of the setup process does involve registering with the Metabase submission service, which initially will be a little bit of a manual process, although will be a more automated process once the web site is up and running. Once correctly configured, you should then be able to continue testing as before.

The beginnings of some howto guides are now available on the CPAN Testers Wiki, which we'll elaborate on over the coming weeks. In the first instance the plan is to move the most prolific testers across to the new submission process. In several cases, these guys were beta testers for us, so are already setup. Those that haven't registered will initially need to contact David Golden, although if you submitted less than 100 reports in June, please wait until the web service is online. The registration process provides a "profile file" that contains your identifying credentials. Using the new web service, existing testers will be able to register and claim addresses which they have previously used to submit reports, and associate them with your "profile", so that you can continue to be identified in the CPAN Testers Leaderboard. The web service will be deployed very shortly, so please be patient, while we get the top testers setup first.

Once the bulk of testers are using the HTTP submission process, the next step will be to contact the less frequent testers. This will mostly be a case of tailing the SMTP submissions and contacting testers individually where possible. The SMTP service will be shut down on or around 31st August, so there should be plenty of time for testers to migrate their current smoker clients to the new system. After this date any report submitted via SMTP will not be processed.

Over the next few months the smoker client tools will be upgraded to use the HTTP submission process by default rather than the SMTP process, thus allowing new testers to automatically be setup to use the new system. In addition, the intention is to provide a mechanism to use the registration web service via an automated API, which will mean that Strawberry Perl installations can automatically be setup to run a CPAN Testers client, and be pre-registered without the user having to do this themselves. The downside to this is that the profile will not be tied to a user that we can necessarily contact, however there are already several submissions each month that use fictious email addresses. The difference with the new registration process, is that at a later date the tester can more actively use their profile file to claim a real email address. Over time we hope this will then become a much more reliable service.

As mentioned in the press release, CT2.0 offers a few advantages for us, one of which is that it doesn't truncate reports. Although, we will be monitoring the reports submitted to ensure that none are unnecessarily large. It does mean that we should be able to parse all the necessary metadata from the report, reducing the number of submissions with conflicting or missing data. As we begin to provide more structured data within the report submission, validation of reports will become much more reliable, and include many more detailed aspects of the testing environment.

One aspect of the testing environments, that often causes confusion, is the operating system used. Currently our understanding of the OS used is mostly drawn from the 'Perl -V' configuration settings, which may not always be correct. In addition, for some OSes the information provided doesn't give a clear enough picture. For example, if the OS is Linux, knowing whether it's RedHat, Debian or SUSE based may have significant benefits to fixing problems, likewise differentiating between Tiger, Leopard and Snow Leopard for Mac OSX. As such, a initiated project aims to provide this information as part of the structured data of a report. As a first step the creation of Devel-Platform-Info is underway, and will provide the metadata required in a consistent format, regardless of platform. This will eventually be then included in the CT2.0 style of reporting, simply as an additional Fact.

We've again had some run-ins with a certain search engine last month, and despite various erroreous claims, we were able to provide logs that disproved them all. The search engine has now finally disabled the aggressive crawling of the CPAN Testers site ... having only taken 6 months of asking! They have asked to be allowed to crawl the website again, having claimed to have fixed the problems, but until I have some free time to properly monitor their bots, they will have to wait.

As mentioned in the press release on Monday, there are two forthcoming conferences which will feature talks concerning CPAN Testers. David Golden will be presenting "Free QA! What FOSS can Learn from CPAN Testers" at OSCON and Barbie will be presenting "CPAN Testers 2.0 : I love it when a plan comes together" at YAPC::Europe.

After several long months of getting CPAN Testers ready for launch, the next month is likely to see activity centred around minor fixes, howtos and helping people migrating to CPAN Testers 2.0. If you do hit any problems, please post them to the cpan-testers-discuss mailing list, where many of the high profile testers and developers reside. After the deadline, we'll be concentrating on some of the new systems that have been waiting in the wings, and taking a more detailed look at providing more structure data in the reports. A new dawn rises.

Posted by Barbie
on 5th July 2010

After 6 months of development work, following 2 years worth of design and preparation, CPAN Testers 2.0 is finally live.

With the rapid growth in CPAN Testers environments and testers over the past few years, the previous method of posting reports to a mailing list had reached a point where the scalability was no longer viable. This was recognised several years ago and discussions for a new system had already begun, with the view that reports should be submitted via HTTP.

At the Oslo QA Hackathon in 2008, David Golden and Ricardo Signes devised the Metabase, with the design work continuing at the Birmingham QA Hackathon in 2009, where David and Ricardo were able to bring others into the thought process to work through potential issues and begin initial coding. A number of releases to CPAN and Github followed, with more people taking an interest in the project.

The Metabase itself is a database framework and web API to store and search opinions from anyone about anything. In the terminology of Metabase, Users store Facts about Resources.  In the Metabase world, each CPAN tester is a User. The Resource is a CPAN distribution. The Fact is the test report. Today that"s just the text of the email message, but in the future it will be structured data.  The Metabase specifies data storage capabilities, but the actual database storage is pluggable, from flat files to relational databases to cloud services, which gives CPAN Testers more flexibility to evolve or scale over time.

Meanwhile the CPAN Testers community was also attracting more and more interest from people wanting to be testers themselves. As a consequence the volume of reports submitted increased each month, to the point that the mail server was struggling to deal with all the mailing lists it hosted. The cpan-testers mailing list was submitting more posts in one day than any other list submitted in a month (in a year in some cases). Robert and Ask, very reasonably, asked if the testers could throttle their submissions down to 5k report posts a day, and set a deadline of 1st March 2010 to switch off the mailing list.

David Golden quickly took on the task to envisage a project plan, and work began in earnest in December 2009. With less than 3 months to the cut-off date, there was a lot of work to do. David concentrated on the Metabase, with Barbie working on ensuring that the current cpanstats database and related websites could move to the Metabase style of reports. Despite a lot of hard work from a lot of people, we unfortunately missed the 1st March deadline. Having throttled report submissions to a more manageable level, and although not complete, the target for HTTP submissions was in sight, Robert and Ask were very understanding and agreed to keep us going a little while longer.

Throughout March and April a small group of beta testers were asked to fire their submissions at the new system. It ironed out many wrinkles and resulted in a better understanding of what we wanted to achieve. The first attempts at retrieving the reports from the Metabase into the cpanstats database began in April, and again highlighted further wrinkles that needed to be addressed. After a month of hard testing and refinement, we finally had working code that went from report submission by a tester, storage into the Metabase, retrieval into the cpanstats database and finally presentation on the CPAN Testers family of websites.

During June the process was silently switched from testing to live, allowing reports to be fed through into the live websites. Due to the ease with which the new style reporting fit into the existing system, the switch largely went unnoticed by the CPAN testers community as well as the Perl community. A considerable success.

The CPAN Testers eco-system is now considerably larger than those early days of simply submitting handwritten reports by email to a mailing list, and the work to get here has featured a cast of thousands. Specifically for CPAN Testers 2.0, the following people have contributed code, ideas and effort to the project over the past six months:

  • Andreas König
  • Apocalypse
  • Ask Bjørn Hansen
  • Barbie
  • Chris Williams
  • Dan Collins
  • David Cantrell
  • David Golden
  • Florian Ragwitz
  • H.Merijn Brand
  • Jon Allen
  • Lars Dɪᴇᴄᴋᴏᴡ 迪拉斯
  • Léon Brocard
  • MW487
  • Nigel Horne
  • Ricardo Signes
  • Richard Dawe
  • Robert Spier
  • Serguei Trouchelle
  • Shlomi Fish
  • Slaven Rezić

Barbie and David would like to thank everyone for their involvement. Without these guys CPAN Testers 2.0 would not have been possible. Thanks to everyone, we can now look forward to another 10 years and more of CPAN Testers.

CPAN Testers now holds over 7.5 million test reports covering nearly 11 years worth of testing Perl distributions. There have been over 1,000 testers in that time, and every single one has helped the CPAN Testers project to be the largest single community supported testing system of any programming language. For a full list of everyone who has contributed, visit the CPAN Testers Leaderboard. A huge thank you to everyone.

With the Metabase now online and live, we can now announce an absolute deadline to close the mailing list. This is currently set as 31st August 2010. After this date all submissions via email will be rejected, and testers will be encouraged to upgrade their testing tools to take advantage of the new HTTP submission system. Many of the high volume testers have already moved to the new system, and we expect nearly everyone else to move in the next month. We will be tailing the SMTP submissions to catch those who haven't switched, such as some of the more infrequent testers, and warn them of the deadline.

More work is planned for CPAN Testers, from further validation and administration of reports, to providing more functionality for alternative analysis and search capabilities. Please check the CPAN Testers Blog for our regular updates.

If you'd like to become a CPAN Tester, please check the CPAN Testers Wiki for details about setting up a smoke testing environment, and join the cpan-testers-discuss mailing list where many of the key members of the project can offer help and advice.

You can find out more about CPAN Testers at two forthcoming conferences. David Golden will be presenting "Free QA! What FOSS can Learn from CPAN Testers" at OSCON and Barbie will be presenting "CPAN Testers 2.0 : I love it when a plan comes together" at YAPC::Europe.

CPAN Testers is sponsored by Birmingham Perl Mongers, and supported by the Perl community.

You can now download the full and complete Press Release from the CPAN Testers Blog. If you have access to further IT news reporting services, please feel free to submit the Press Release to them. Please let us know if you are successful it getting it published.

<< August 2010 (4) June 2010 (3) >>