CPAN Testers is only made possible with the support of our sponsors.
For more information on sponsoring, please visit the I CPAN Testers website.

Upgrade Notice

The CPAN Testers Blog site has been upgraded since you last accessed the site. Please press the F5 key or CTRL-R to refresh your browser cache to use the latest javascript and CSS files.

News & Views

So the 2014 QA Hackathon has drawn to a close, but it is far from the end of the work, particularly for CPAN Testers. You can read several blog posts detailing many aspects of the QA and testing community work done during the hackathon, as well as several aspects of the toolchain, including PAUSE (which saw a lot of collaboration). It does get said often, and it bears repeating, that the QA Hackathons are a vaulable part of the Perl community, and help to drive many projects. Without them it is likely that key elements of the infrastructure we have come to rely on (PAUSE, CPAN, BACKPAN, MetaCPAN, CPAN Testers, CPANTS) would be a long way from being the resourceful, stable and continually developing components we have come to accept. Next year's hackathon will be in Berlin, and I for one am very much looking forward to it.

Aside from the work on the database for CPAN Testers during the hackathon, I did get to make a few releases, but several elements I started during the hackathon, were only completed during the following weeks. One of these CPAN-Testers-WWW-Reports-Query-Report, now enables you to retrieve the Metabase Fact object, JSON or hash representation of a specific report. For those looking at analysing the similarities and differencies between reports, this may make things a little easier, particularly when we start introducing more element facts into the report fact. Currently this only works for reports stored in the Metabase, so those early reports are not currently retrievable .. yet.

I discussed a new command line tool to submit reports with H.Merijn "Tux" Brand, who is keen to run reports in standalone instances. I see this working similar to Garu's cpanm-reporter, and with the common client that Garu has been working on, this could be a nice addition to the submission options. Liz Mattijsen, Tobias Leich (who easily has the coolest gravatar on CPAN) and I talked about how Perl6 distributions could be incorporated into CPAN Testers. There are some subtle differences, but there are also many common factors too. It was interesting to read the Rakudo blog post about Perl 6 and CPAN, as overcoming some of the hurdles potentially facing Perl6 developers are likely to help make CPAN a better place for all of us. The currently proposed solution is a similar approach to how different namespaces are stored in the Metabase. For the time being though, Perl6 distributions are excluded from CPAN Testers, but once we have a Perl6 smoker there is no reason not to include them. I'm not sure how soon that will be, but watch this space.

Andreas König and I once again looked at the way the reports are stored in the Metabase. Andreas had already highlighted the updated date, which is meant to be the date it entered the Metabase, was in actual fact the created date (the date of the testers platform). Along with David Golden, we looked at the code used by the Metabase, but failed to find anything wrong with it. It's hopefully something we can take more time over in the future, however the next priority for the Metabase is getting it moved onto MongoDB and away from SimpleDB. In the meantime, the Generation code, due to time constraints and the lack thereof, has been running using the 2010 version of the Metabase/AWS interface. During the hackathon and the following weeks, I finally upgraded to use the revamped version released in 2012. Although still troublesome to find all the reports, the search interface has been much improved, and now we have a much more reliable feed from the Metabase. This is also in part due to the rewrite of the internals of the Generation code to be more pro-active in find unusual gaps between reports.

I spoke with Neil Bowers during the hackthon too. Neil had suggested some ideas that I'd wanted to included into the CPAN Testers Statistics site. We discussed others during the hackathon, and although I have several notes to work from, it will be a little while yet before I can put aside some time to implement them. Neil has no end to the ideas to help improve CPAN, and I hope he'll be a good sounding board for ideas to incorporate into the site in the future. On the first day of the hackathon he posted about how quickly CPAN Testers test a distribution, after being uploaded to PAUSE. He was surprised to see some reports posted almost instantly. This is largely thanks to Andreas' work to bring the Tier 1 mirrors up to date within 10 seconds of a distribution being successfully uploaded to PAUSE. People such as Chris and Andreas use their own T1 mirrors to feed into their smokers, so as a consequence the reports can appear on the Reports site within one hour of the distribution hitting PAUSE!

I had intended to launch the CPAN Testers Admin site during the hackathon, but didn't get the chance to prepare a launch statement. I did wonder how a launch on 1st April would go down, but elected to wait a little longer. So expect some news on that front very soon.

Just before the hackathon we reach 40 million report submissions. An impressive number by any standards, and gives a compelling argument that Perl & CPAN are still being actively maintained and developed.

John Scoles made a post related to CPAN Testers, highlighting how CPAN Testers showed why a particular test needed a little more thought. Thanks to the wide variety of perls and platforms tested by the CPAN Testers, it provided John with some intriguing failures, and made for an informative look at how hashes called in scalar context are handled differently between versions of Perl and platforms. Prior to the QA Hackathon Neil Bowers posted about his new idea for An author's CPAN dashboard, which he continued to develop during the hackathon too. We even discussed, together with Olaf Alders, whether this was something that should feature in MetaCPAN. For now it won't, but hopefully it can develop further and w'll see what people make of it, once more APIs are fed into the dashboard.

Another month wrapped up, and another QA Hackathon over. Lots to do and plenty of fresh ideas coming. Happy testing.

Once again apologies for the delayed summary this month. The day job has been rather taken up a lot of my time for the past few months, and I've had a few other projects that have needed my attention. Hopefully next month, I'll fair better.

First off for this month, I'd like to advertise the 2013 QA Hackathan, taking place in Lancaster, UK from 12th April to 14th April. This event with be the 6th QA Hackathon, and is looking to as successful as previous years. There are plenty of projects to work on, and plenty of developers willing to pitch in and help out. Plans for CPAN Testers include preparing for the move from AWS. We have an opportunity to clean up the problems of data storage, or rather data search, by moving to a new Metabase, which Dave Golden has been working on. There will plenty of other QA projects that will be worth watching too, so look out for the various blog posts and code releases during and after the event.

David Oswald ended January on the mailing list, with a question about tests for Bytes::Random::Secure on 5.6.*. It highlighted the usefulness of CPAN Dependencies, providing the data to show that pre-requisites were the problem, rather than the distribution itself. It also highlighted why referencing cp5.6.2an at cpxxxan.barnyard.co.uk, could help out those users who need reliable installs of particular distributions for their version of Perl.

The discussion regarding NFS continued into the month, as detailed in my delay summary from last month. Buddy Burden asked about comparing test reports. Although not quite the same, it does follow on from other similar requests about getting at data within reports. There are two problems at the moment, the first is reliably getting at the metadata of a particular report. Although I could open up an API to this data, it doesn't help with the second problem, which is the ability to compare structured data. Currently reports are compiled mostly as a single piece of text. To be able to properly compare reports, it would be better to use structured data. At the last QA Hackathon making this more of a reality was discussed and some code was even released to help push it forward. However, there is still some way to go, and hopefully at the 2013 QA Hackathan we may see some more movement on this. In the meantime, the CPAN Test Analysis site may well provide some of the comparisions you may be looking for.

Matthew Musgrove asked if he could use the same distroprefs with multiple smokers. For those unfamilar with the distroprefs files, these are the files used by testers to help filter out distributions that are problematic when running automated tests. They are also referred to as the ignore lists. These files typically are used with individual smokers and are not shared between them. However, there are a few ways this could be handled. Some use a source code repository (sometimes on GitHub), to sync between their smokers. David Cantrell told us he uses the more traditional method of shell script and rsync. However, you share your distroprefs between smokers, bare in mind that you'll be excluding distributions that may not be a problem on some of your smokers. As such, it may be worth retrying some distributions over time to see whether old issues have been resolved.

David Oswald top and tailed the month with another post asking for help understanding a FAIL for Bytes::Random::Secure. David had identified that the failure was due to Crypt::Random::Seed not being installed, however it was explicitly listed in the prerequisites. David Cantrell pointed to the very long @INC, which some smokers use to avoid installing distributions, but reference the installers build directory. This can be problematic for two reasons. Firstly, long running smokers can blow the length of @INC, such that paths added to the end only get ignored. Secondly, if the installer has a limit for the amount disk space used, it may remove older distributions before running tests. To avoid this, many smokers will automatically stop and start their smokers to keep the @INC at a manageable length.

We passed 28 million reports at the beginning of February (congratulations to Peter John Acklam for his PASS submission of Compress-Zlib-Perl), and passed 29 million (even more congrats to Peter John Acklam for his UNKNOWN submission of Regexp-Assemble) at the beginning of March. I suspect we may well hit 30 million around the time of the 2013 QA Hackathan, which would be rather nice to celebrate while many of us are all together. In February, Thomas Peters (WETERS) held the 6000th unique PAUSE account to upload a distribution. Interesting to see that the number of authors currently submitting to CPAN has stayed pretty constant over the past few years. We're also gaining around 30-40 new active authors every month too.

That's all for now. Hope to see you in Lancaster next month, and I'll do my best to get the next summary out before then too!

So March ended on quite a high, following the 2012 QA Hackathon. With so many key people in one room, it was impressive to see how much got done. You can read reports from myself (parts 1 & 2), David Golden, Ricardo Signes, Miyagawa, Paul Johnson, Ovid and Dominique Dumont, and there were several tweets too, during and after the event, and the wiki also has a Results page. There was a significant number of uploads to PAUSE during and after the event too. And CPAN Testers has benefited hugely from the event.

Arguably the two biggest significant developments during the event were thanks to Breno G. de Oliveira, who not only added support for CPAN Testers within cpanminus, but also began the work on the long desired CPAN::Testers::Common::Client. While working on the former, Breno noticed that there was a lot of common reporting tasks (and differences) performed within CPAN::Reporter and CPANPLUS::YACSmoke. As he wanted to replicate this within his client for cpanminus, he asked whether it would make more sense to wrap this into a separate library, to which David and I were full of encouragement for. Breno set about setting up a GitHub repository and has been doing some fantastic work bring all the reporting together. You can follow his efforts of GitHub, as garu, and hopefully we shall start to see this distribution on CPAN and in Smoker clients soon.

While that may have been the most significant output for CPAN Testers, other parts of the toolchain also made some giant leaps. Having Andreas, David, Ricardo, Schwern and Miyagawa all together meant David's idea of changing the way we look at indexing CPAN, allowed kinks and ideas to be ironed out in minutes rather than weeks. David's idea is to distance the indexing system from the repository. This will allow other repositories, such as the many DarkPAN repositories out there, to use the same system and enable toolchain installers to point to different repositories as needed. Nick Perez is now working on CPAN::Common::Index, which will form the basis for the new system. You can read the details in David's write-up of the event. Hopefully this will be a major step forward to enable CPAN Testers to be used for independent repositories, which has been a request for many years.

In other news we finally announced the sponsorship website and CPAN Testers Fund. Over the past 10 years the CPAN Testers has been funded largely by the developers. In the last 5 years hosting, network and bandwidth has been increasing, with the developers and Birmingham.pm being the principal donors. While this is great, and we really do appreciate this, the bigger CPAN Testers becomes, the more support we need. As such we are hoping to encourage sponsorship from businesses and corporations, especially if they use Perl. If you would like to know more, please get in touch. Many thanks to the Enlightened Perl Organisation for managing the CPAN Testers Fund for us.

On the mailing list there was a discussion about the reports not showing the differences in tests where strings look identical, but where one may contain control characters. Personally I feel trying to fix this in the browser is too late. As we treat the test report as a complete piece of text, we cannot currently isolate the test output to know what control characters to highlight. It could end up confusing the viewer. Andreas also thought that the author should include more appropriate tests in their test suite, and suggested the use of Test::LongString, Test::Differences or Test::HexDifferences.

As we've mentioned a few times, the SQLite database download is no longer as reliable as it once was. Andreas believes that we may have a unusual text pattern that is causing the problem, but whatever it is, it's not something we know how to solve. As the database file is high maintenance, I would like to abandon it sooner rather than later. If you currently consume the SQLite database, please take a look at the new CPAN::Testers::WWW::Reports::Query::Reports release. This now the preferred way to request records from the master cpanstats database. For the summaries, you now also have CPAN::Testers::WWW::Reports::Query::AJAX. If you have any problems or suggestions for improvements, please let me know.

On a final note, please be aware that CPAN Testers will be down for maintenance on Thursday 12th April. We'll try get everything back online as soon as possible. Thanks to Bytemark Hosting for helping us with the disk upgrade.

Back in January 2008 we were celebrating the one millionth post submitted to CPAN Testers. Although that article proclaimed it to be the one millionth report, many initial posts to the mailing list also included discussions and announcements of uploads. It wasn't until I created the Interesting Stats page that we started to see the true picture. However, we only had to wait until March 2008 for the real one millionth report to be posted. Now some 2 years and 7 months later we've had the nine millionth report submitted. It took 9 years to produce 1 million reports, but only a further 2½ years to produce another 8 million reports. The rate at which CPAN Testers has been able to get people involved in the project has been phenomenal. We are now submitting over 500,000 reports a month, so I have no doubt we will pass the 10 millionth mark before the end of the year .. probably just before Christmas :)

In the comments to my nine millionth post on Perl Blogs, John Napiorkowski asked of the comparisons for testing packages in other languages, particularly Python and Ruby. Chris Williams provided some links to the testing setups for those languages, and the sites prove rather interesting. From the perspective of trying to find information about test results CPAN Testers wipes the floor with both, as I found both the Cheese Cake and Firebrigade sites awkward to follow. For the Cheesecake site it seems they are aiming more for a site like CPANTS, which is probably a good first step to encourage a testing culture. While the Firebrigade site seems to have tried to take on the idea of CPAN Testers, but in trying to also be different they've actually made things hard for themselves. I also refute the Ruby claim of "Firebrigade tests every gem ever made on every platform under the sun". On the front page it lists that it only has tests on 45 platforms. CPAN Testers would never make such a bold or false claim, but with over 100 platforms, and 74 alone during October 2010, I think Perl's sun must be much bigger than Ruby's, and CPAN Testers are still only scratching the surface. It will be a long time before any other language can compete with CPAN Testers, and with only 20405 reports in nearly 4 years, the Ruby team have a long way to catch up. CPAN Testers should be immensely proud of the work they have put into the project, whether as a developer, tester or even those with just the odd suggestion to help improve the eco-system. Every contribution has helped to make CPAN Testers worthwhile and valued by the Perl community, as well as respected, imitated and/or envied by other language communities. And we're still improving.

Talking of improvements, there have been several performance improvements to the applications which produce the web pages for several sites. The CPAN Testers Statistics site was suffering from the vast amount of number crunching it performed, and was previously using up as much as 3GB of RAM, and often taking over an hour to produce its results. With a rework to save a snapshot of the data, and restart each time from where we left off, the application now uses less than 1GB RAM and processing takes about 40 minutes. The CPAN Testers Reports page builder has also seen some tweaks, and again the pages have seen a dramatic improvement in build times. Some author pages were taking over an hour to build, but now even RJBS and ADAMK only take 5-15 minutes. There is still room for improvement, and better use of on disk storage is planned.

Still the biggest drain on resources is bzip2. It is a memory and CPU hog at the best of times, and with it holding IO on occasions, it often has a significant impact on other applications. As such I am taking time to review how the bzip2 files are produced. Part of that is to review how often they need to be generated. Tellingly the frequency of any 1 IP to grab the two most popular archive files (uploads.db.bz and cpanstats.db.bz2) are just once a day. Currently the gzip archive of cpanstats.db.gz has a similar popularity. As such over the next few days expect the archive creations to happen in the early hours of the morning (CET), gathering up the previous days stats. Initially I was planning to move the archiving to another server, but with the archives not being in high demand, I will now look at running one complete archive process a day and see how that effects the server performance. If the change in timestamp is likely to cause problems, please me know and I'll see what I can do to help.

In the next few weeks David Golden and I are planning to chat about the future of CPAN Testers. Now that CT2.0 is live, where do we go from here? There are some obvious improvements we can now start to look at, such as expanding the metadata we record, but we have other plans to make CPAN Testers even more reliable and current. Once we've had a chance to discuss the ideas and point them in the right direction, we'll let you know more.

In other news, it is likely that the Preferences site's SSL certificate will fail very soon. For the past 2 years we've been able to qualify for GoDaddy's OpenSource scheme which donates a 1 year certificate for any verified Open Source project. Sadly, despite them considering CPAN Testers an Open Source project for the last 2 years, we have now been rejected for not being an Open Source project! Yes, the response surprised me too, but despite attempts to understand why we no longer qualify, they've now closed the request ticket and have effectively ended the discussion. As such, I'll be looking to purchase a new SSL certificate from another vendor shortly, who hopefully have a better support policy.

I was intrigued to see Yanick Champoux's recent blog post: Generating RT bugs out of CPAN Testers' Reports. Yanick was looking for an effective way submit a test report into his RT queue. Unfortunately I can't add a button to the site as suggested, as at the current time the site doesn't verify that you are the author of a distribution, and opening it up to all would be a nightmare waiting to happen. I did wonder whether this was something that could be added to the Preferences site, but with potentially hundreds of reports coming in, trying to decide whether they are applicable for RT or not could also turn into a nightmare. As such, if you're interested in doing this yourself for your own RT queue, read Yanick's post and see how you get on.

Moving away from CPAN Testers and looking at CPAN, we passed another milestone last month. On 8th October 2010 ETHER became the 5,000th PAUSE user to upload a distribution to CPAN. Although we currently have 8482 PAUSE users (as of 02/11/2010), it is surprising how many have used their ID for other CPAN related activities. After holding the top spot for some considerable time, Adam Kennedy has now been overtaken by Ricardo Signes for the most current distributions attributed to a single PAUSE user. Some years ago it was considered quite a feat to reach 100 distributions, but with 230 active distributions currently to his name, I'm not surprised Ricardo created Dist::Zilla to help him manage them all :)

Finally, we have some more mappings, with 40 new address mappings, of which 22 are for new testers. Until next time, happy testing :)

Posted by Barbie
on 25th May 2010

For some time now I have been promising to publish the list of CPAN Authors who for one reason or another, appear to be uncontactable. These are authors, who when using their PAUSEID @ cpan.org address, generates a rejection of some form. Typically this appears to be for addresses which are set up to be used in redirection, which have since been made obsolete (perhaps from changing jobs or personal domain), but there are a few that are just not setup to receive emails.

I have now created a Missing In Action page, which I will keep updated as I receive bounce backs from the CPAN Testers daily summaries. In some cases bounce backs require verification from a real person, and I do eventually get round to those, but the ones listed are specifically for emails that are outright rejected. If you have not been receiving any CPAN Testers summaries or reports and have been expecting some, please first check this list to ensure that you haven't had your preferences disabled.

If you are on the list, please read the instruction on the page, to learn how to enable your PAUSE email, and to get your CPAN Tester preferences re-enabled.

File Under: pause / prefs
NO COMMENTS