CPAN Testers is only made possible with the support of our sponsors.
For more information on sponsoring, please visit the I CPAN Testers website.

Upgrade Notice

The CPAN Testers Blog site has been upgraded since you last accessed the site. Please press the F5 key or CTRL-R to refresh your browser cache to use the latest javascript and CSS files.

News & Views

Posted by Barbie
on 19th September 2015

One of the first offers of help for getting the CPAN Testers server fixed, was from a guy called Doug Bell. Doug impressed me a lot with that first email, and was the only person who I felt really understood what I was asking for in my post about wanting a successor. Not that others didn't grasp that, but the fact Doug was the only person to send me a CV, proved that he got that this wasn't just a small project, or that I wanted to hand over the keys to someone who wasn't going to make a commitment to the project. Doug credentials impressed me, and his ideas and help since being invited to be involved, only proved he wasn't in this for the short-haul.

Many may know Doug as PREACTION, both on CPAN and IRC. Doug is the current leader of Chicago Perl Mongers, which also amused me, particularly once you know I'm based in Birmingham (UK) and Chicago is our twin city in America, and that I'm the leader of Birmingham Perl Mongers. Cue Twilight Zone theme tune! Doug has worked in web development, systems development and understands SysAdmin responsibilities. An ideal person to look after CPAN Testers, as you really do need a bit of everything. He also, perhaps most importantly, understands scale. CPAN Testers is a huge project, not just in terms of the number of reports (nearly 60 million), but the whole eco-system that is driven across the whole project.

I really needed a successor, as my home life and work life have been taking priority in recent years, and I don't want CPAN Testers to suffer because of that. I believe Doug is ideal to take over the project, and I hope you can give Doug the help and support that most have given me over the past 10 years or so. I'm not walking away completely, and will still be around to offer advice and help where needed, but Doug will be the one to drive the project forward from now on. He has some great ideas to help grow and improve the project, and I for one am really looking forward to seeing them come to fruition.

Doug and I have yet to meet, but hopefully the QA Hackathon 2016 will be the first time that I, along with all the other QA enthusiasts, will be able to sit down and discuss/design/plan his ideas for the future.

Welcome Doug, and thank you.

Posted by Barbie
on 18th September 2015

For over a month the CPAN Testers server has been having problems. In most cases the server locked up for no apparent reason. Sometimes it would work for a few days after a reboot, other times it would bearly last 5 minutes.

With the help of a few people, notably Matt S Trout and Doug Bell, we tried to figure out what was going on. I noted that the 'release' table and SQLite database that is created from it, seemed to be the most fragile for the server, as most times when it was touched, the server seemed to lock up within a few seconds. The table and database itself were used for a project Adam Kennedy created several years agao. These days only the bots and crawlers seemed to be interested. As such, I disabled to see whether that would help. It didn't.

We were getting stuck at every corner it seemed. I got Bytemark, our hosts involved to try and check the disks, but all appeared to be running fine. The SATA disks did seem to be very active, but that's normal for us, as there are alot of disk writes to create the sites. As we've had the current server for nearly two years, Bytemark suggest moving to a new server, which has better CPU and RAM capacity, which we've now completed. The server has now been up for over a week, the sites are building and running without a problem, and best yet the reports site is building faster than it has been for a long time.

Although we still have no clear idea why the cpantesters2 kept falling over (it was even falling over when we were rsync-ing the data to cpantesters3, when nothing else was running on either server!), we suspect there is a faulty CPU or RAM module. There isn't much we can do about that, as there isn't a disk check type feature that can find bad blocks and disable them for CPUs and RAM, at least that I'm aware of. So a move to a new server is about all we could have done.

As a result of all this, we have some ideas to help improve the sites and any future rebuilds, so if we do suffer again, we can recover much quicker. I'm currently documenting how the current systems work, and we have at least 5 projects planned to help improve various parts of the eco-system. Some have been long overdue, others are new and fresh ideas to help take CPAN Testers further into the future. There is a lot to learn about scaling with this project ;)

My sincere thanks to MST, Doug, and Chris at Bytemark, for helping out and getting us back online. It's been a frustrating month or so, and I'm just relieved to have the sites back and stable again. My thanks to everyone who stepped up and offered help and support for the project, it really was very much appreciated. Sorry if I didn't get back to you, or weren't able to take advantage of your assistance. Some of the offers of help were taken advantage of, and a couple are planned for future projects to help grow the project. We'll hopefully have more news about them as we start to design and implement them in the coming months. The future looks promising, and it wouldn't be without everyone's support. Thank you.

It's good to be back!

Sorry October's summary is a bit late, as I had rather a busy few months recently, and finding time to finish this one prove more tricky than normal.

However, first off, a thanks to Gabor Szabo, as one of his screencast featured a brief explanation of how CPAN Testers are intergrated into MetaCPAN. Although anyone familiar with the CPAN Testers Report site, will probably have understood what the numbers mean, the screencast is a handy help to anyone new to MetaCPAN or CPAN to learn more about using MetaCPAN. If anyone else has similar screencasts about CPAN Testers, or wishes to create one, please let me know.

My thanks to Pete Houston for notifying me early in the month that the feed to CPAN Testers had got stuck. Another reason why we really want to move away from Amazon and SimpleDB. This is still being investigated, and hopefully will be a project during the QA Hackathon next year. It also gave me pause to think of better ways the feed and builder could be processing. Thet are both mostly serial process, although there are now 2 builders (distros and authors). However, having worked a lot with MessageQueues over the last year, I'm looking at rewriting both the feed and builders, and again is likely to be my main project during the QA Hackathon in 2015. The ultimate aim, as has been asked on many many occassions, is to try and get as close to real time as we possibly can. On a good day we are less than an hour behind, but being a minute of two behind would be even better.

Karen Etheridge highlighted some problems with the CPAN Testers Admin site. Firstly the login had broken again. I've now permanently fixed this, and uploaded the code to CPAN so the old version doesn't overwrite the new version again. However, Karen also spotted a few other problems. Next up was the find a report by GUID. I'm not quite sure what I tested before, but I suspect I overwrote the working code again :( Again this is now fixed and the code is on CPAN :)

The current CPAN Admin problem is figuring out why, when marking reports for removal, they aren't getting alerted to the tester. They were getting added to the Admin user though, so they weren't lost, but it has highlighted a few cross-reference points that are not being made. Connecting a report to tester, or more imporantly to the tester's current email address is not always straight forward, so the fallback is to email the Admin user. I'm working on the code to reconcile these cross-references better, and in the coming weeks I should get that working properly again. In the meantime if you have marked reports, please bear with me, and I'll get to them as soon as I can. In the process of digging through the report removal process, I also discovered that the Builder mechanism for this was flawed, at least from an Author perspective. I've since corrected this too.

Andreas König noted that several reports from one tester were essentially the same. Breno G. de Oliveira helped to identify the problem, and the reasoning behind it. The reporter for cpanminus, doesn't currently record history as the other smokers, and consequently doesn't filter out repeated attempts by the author to install the same distribution. Breno is planning to work more on the Common Client, and integrate this into the cpanminus Reporter. So if you get similar repeated reports, please be gentle with the tester :)

Subramanian Karunanithi was the casualty of version numbering with CGI-Test. Thanks to some investigation work from Jonas B. Nielsen, he highlighted the latest version was 0.52, but the installed version was 0.104. The problem with version numbering within the installers is that they see 0.104 as Zero Point One Hundred and Four, which is greater than Zero Point Fifty Two. Karen posted a bug report, and Alex Tokarev has now released version 1.000, so all good. However, it does bear repeating that Version Numbers Should be Boring.

Next month we'll have a few notes about LPW, more fixes to the toolchain and some ideas for the future.

During August the CPAN Testers Reports server suffered a problem with the RAID array. Thankfully, it just needed the disk to be re-mirrored. However, this did mean that for a couple of weeks the server, and the websites did seem unresponsive. To lessen the load I reduced the builder processes, and only ran them for a few hours a day. As such, you may not have had your pages updated as frequently. The RAID array is now fully operational, and the pages are now back up to date. Currently reports look to be about an hour behind and the less frequently used pages are less than a day behind.

The new CPAN Testers Admin site is coming along. The bulk of the site is now completed, and work has already started on the integration for CPAN Testers Reports Builder and the CPAN Testers Statistics, to ensure removed reports are accounted for. The first part of the site to be launched will only be looking at historical reports. At some point in the future, I would also like to make it easier to report runaway smokers, so we can (un)block them more pro-actively.

On the mailing list last month, a thread which started the previous month by Nat Goodman, was carried over when Nat mentioning he was using a testing pragma. Steffen Schwigon picked up on this, and asked Nat to elborate, which he did. It seems due to a problem with building pre-requisites, Nat has employed a pragma to help catch when a situation isn't satisfied, in his case when the tools to connect to a MySQL database aren't loaded. It's an interesting use of a pragma and the TAP parser.

Dave Cantrell picked up on the fact that we no longer update the SQLite version of the cpanstats database. As mentioned previously, SQLite has not being storing results correctly for several years. Both Andreas and myself have tried to diagnose the problem without any luck. As the file itself is several gigabytes in size, having this downloaded several times a day was taking up valuable disk IO. As such, the processes to update the file have now been disabled. The recommended way to retrieve these reports is now through the use of CPAN::Testers::WWW::Reports::Query::Reports, such that you can create your own database, and update with only the latest reports, rather than trying to download them all every time.

Neil Bowers asked about getting summary data for a particular distribution. For those interested in this sort of data, there are two distributions you can use, CPAN::Testers::WWW::Reports::Query::AJAX, providing basic data, or Leo Lapworth's more detailed CPAN::Testers::Reports::Query::JSON. Both get the same data, but present it in a slightly differnt format. Rocky Bernstein, also asked how to get the raw reports. However, there is a slight misunderstanding of what a raw report is. In the olden days, we used email, and as such the reports were all in plain text. With CPAN Testers 2.0 we now create structure data. We still display a plain text version, but more of the data is being built as components, to enable better analysis in the future. As such, the reports that appear on the Reports website, are not necessarily the original plain text versions, but rather constructed to look like plain text versions. Thus clicking the 'raw' button doesn't really present the raw email format that it used to be, but a close approximation inside PRE tags. The main reason for this is encoding. HTML handles encoding slightly better than plain text sent to a browser. We have had problems with this in the past, and it has proved more reliable to handle the encoding HTML than plain text. At some point, alongside the 'raw' button, there will be a 'data' button, which will return the report as structured data in JSON format.

Ron Savage highlighted a problem with IO::Pipe and Proc::Fork Tests hanging on Windows. David Golden pointed out that forking on Windows has always been problematic, and can cause unexpected results. The conclusion for Ron was to drop Capture::Tiny, however, if you're coding to be cross platform, it is always worth bearing in mind that Windows and fork() are not the bext of friends.

That's it for this month. Hopefully, we'll have more news about the Admin site next month, and may be even some server news.

If you haven't come across QuestHub, you might want to take a look at the Perl Realm. It's essentially a TODO list of tasks and projects, which can earn you points, if you like to game these sorts of things. There are several stencils to help get you started, and in particular there are now several relating to CPAN Testers. If you're new to the Perl or CPAN Testers community, and want to contribute in some way, the stencils are an ideal starting point. Some are technical, others are more social, but all are to the benefit of Perl. Neil Bowers pointed me to the site in the first place, thanks to his post Make me do some work, via Questhub!, which in turn drew inspiration from a previous CPAN Testers Summary post.

Last month the CPAN::Changes Kwalitee Service saw 40% of CPAN meet the CPAN::Changes::Spec. In the past month, and thanks to Neil and his stencils on QuestHub, that figure has risen to 41%, and is continuing to improve. While the specification is no way necessary to upload your distribution to CPAN, it does help to have some form of standard format for Changes files. As a consequence, Neil Bowers submitted a ticket to include a suggestion in the Perl documentation, which has now been approved and patched.

Reini Urban posted Smoking CPAN in one line on blogs.perl. Although we would recommend more conventional ways of smoking CPAN, this one-liner certainly does the job. Reini had a specific test he was looking, at, but the outcome has been to raise tickets on several distributions, many of which have already been fixed. Testing for specific bugs is not usually what CPAN Testers is about, but in this instance it proved a very productive effort.

After reading several posts and emails over the past few years, and particularly one in the past few months, I felt a post was needed to provide a bit of clarity. There are many ways to get involved with CPAN Testers, aside from being a tester. If you wish to help improve the tools and websites, feel free to take a look at the repos and see how you can help. Let us know your ideas on the mailing list, and by all means ask for guidance and advice there too. CPAN Testers is continually evolving, but it does take time.

The Admin site is coming along. The original framework has been in place for sometime, as it was originally designed before we had CPAN Testers 2.0 in place, but I've had to redesign some of the work with accessing and indexing tester emails. You can follow my progress on QuestHub, as I complete various aspects of coding and testing. A more public repo will also appear soon.

On the mailing list last month, Serguei Trouchelle highlighted some unusual behavior with a tester using CPANPLUS. Chris picked this up and it seems some of the instructions on the Wiki for testing CPANPLUS might not be as complete as they need to be. Hopefully we can clean these up soon, so new testers can ensure their test environments are working correctly. A few people emailed me to say the CPAN Testers Reports site was down. Every so often several backend process are all running at once, and with a few extra Apache processes on top, the disk I/O can be a bit traumatic. As such the websites are usually the first things to suffer. I'm looking at moving some of the websites around soon, to hopefully reduce the impact.

Olivier Mengué posted that he thought the YAML files that are retrieved from the Reports site were invalid. We use YAML::XS, and the real file it reads from is actually a JSON file. The probalem seems to be with the YAML module, as it requires a final newline character at the end of the file. The results of YAML::XS are sent straight out in the HTTP response, and are not written to disk, and don't include a final newline character. I'm not aware of this being a requirement in the YAML spec, so would recommend people switching to YAML::XS or YAML::Syck if they wish to read the YAML files from the Reports site. Or you could just request the JSON file directly.

This week is YAPC::Europe 2013 in Kiev. Sadly I'm not there, but I hope everyone who is making the most of the event. Four talks specifically about testing are being presented, so please check them out if you can.

We're now up to 33 million reports submitted, which is great. But it does means some spring clean is still on going to keep the disk space down. The databases combined now take up around 610GB of storage, and the websites on the main server (including CPAN and BACKPAN) take up another 150GB. I'll be looking at upgrading our disk allocation soon, but it never hurts to do a bit of spring cleaning. The compression of reports is still underway and over 16 million of them are now compressed. More updates for you next month.