CPAN Testers is only made possible with the support of our sponsors.
For more information on sponsoring, please visit the I CPAN Testers website.

Upgrade Notice

The CPAN Testers Blog site has been upgraded since you last accessed the site. Please press the F5 key or CTRL-R to refresh your browser cache to use the latest javascript and CSS files.

News & Views

Posted by Barbie
on 18th September 2015

For over a month the CPAN Testers server has been having problems. In most cases the server locked up for no apparent reason. Sometimes it would work for a few days after a reboot, other times it would bearly last 5 minutes.

With the help of a few people, notably Matt S Trout and Doug Bell, we tried to figure out what was going on. I noted that the 'release' table and SQLite database that is created from it, seemed to be the most fragile for the server, as most times when it was touched, the server seemed to lock up within a few seconds. The table and database itself were used for a project Adam Kennedy created several years agao. These days only the bots and crawlers seemed to be interested. As such, I disabled to see whether that would help. It didn't.

We were getting stuck at every corner it seemed. I got Bytemark, our hosts involved to try and check the disks, but all appeared to be running fine. The SATA disks did seem to be very active, but that's normal for us, as there are alot of disk writes to create the sites. As we've had the current server for nearly two years, Bytemark suggest moving to a new server, which has better CPU and RAM capacity, which we've now completed. The server has now been up for over a week, the sites are building and running without a problem, and best yet the reports site is building faster than it has been for a long time.

Although we still have no clear idea why the cpantesters2 kept falling over (it was even falling over when we were rsync-ing the data to cpantesters3, when nothing else was running on either server!), we suspect there is a faulty CPU or RAM module. There isn't much we can do about that, as there isn't a disk check type feature that can find bad blocks and disable them for CPUs and RAM, at least that I'm aware of. So a move to a new server is about all we could have done.

As a result of all this, we have some ideas to help improve the sites and any future rebuilds, so if we do suffer again, we can recover much quicker. I'm currently documenting how the current systems work, and we have at least 5 projects planned to help improve various parts of the eco-system. Some have been long overdue, others are new and fresh ideas to help take CPAN Testers further into the future. There is a lot to learn about scaling with this project ;)

My sincere thanks to MST, Doug, and Chris at Bytemark, for helping out and getting us back online. It's been a frustrating month or so, and I'm just relieved to have the sites back and stable again. My thanks to everyone who stepped up and offered help and support for the project, it really was very much appreciated. Sorry if I didn't get back to you, or weren't able to take advantage of your assistance. Some of the offers of help were taken advantage of, and a couple are planned for future projects to help grow the project. We'll hopefully have more news about them as we start to design and implement them in the coming months. The future looks promising, and it wouldn't be without everyone's support. Thank you.

It's good to be back!

Sorry October's summary is a bit late, as I had rather a busy few months recently, and finding time to finish this one prove more tricky than normal.

However, first off, a thanks to Gabor Szabo, as one of his screencast featured a brief explanation of how CPAN Testers are intergrated into MetaCPAN. Although anyone familiar with the CPAN Testers Report site, will probably have understood what the numbers mean, the screencast is a handy help to anyone new to MetaCPAN or CPAN to learn more about using MetaCPAN. If anyone else has similar screencasts about CPAN Testers, or wishes to create one, please let me know.

My thanks to Pete Houston for notifying me early in the month that the feed to CPAN Testers had got stuck. Another reason why we really want to move away from Amazon and SimpleDB. This is still being investigated, and hopefully will be a project during the QA Hackathon next year. It also gave me pause to think of better ways the feed and builder could be processing. Thet are both mostly serial process, although there are now 2 builders (distros and authors). However, having worked a lot with MessageQueues over the last year, I'm looking at rewriting both the feed and builders, and again is likely to be my main project during the QA Hackathon in 2015. The ultimate aim, as has been asked on many many occassions, is to try and get as close to real time as we possibly can. On a good day we are less than an hour behind, but being a minute of two behind would be even better.

Karen Etheridge highlighted some problems with the CPAN Testers Admin site. Firstly the login had broken again. I've now permanently fixed this, and uploaded the code to CPAN so the old version doesn't overwrite the new version again. However, Karen also spotted a few other problems. Next up was the find a report by GUID. I'm not quite sure what I tested before, but I suspect I overwrote the working code again :( Again this is now fixed and the code is on CPAN :)

The current CPAN Admin problem is figuring out why, when marking reports for removal, they aren't getting alerted to the tester. They were getting added to the Admin user though, so they weren't lost, but it has highlighted a few cross-reference points that are not being made. Connecting a report to tester, or more imporantly to the tester's current email address is not always straight forward, so the fallback is to email the Admin user. I'm working on the code to reconcile these cross-references better, and in the coming weeks I should get that working properly again. In the meantime if you have marked reports, please bear with me, and I'll get to them as soon as I can. In the process of digging through the report removal process, I also discovered that the Builder mechanism for this was flawed, at least from an Author perspective. I've since corrected this too.

Andreas König noted that several reports from one tester were essentially the same. Breno G. de Oliveira helped to identify the problem, and the reasoning behind it. The reporter for cpanminus, doesn't currently record history as the other smokers, and consequently doesn't filter out repeated attempts by the author to install the same distribution. Breno is planning to work more on the Common Client, and integrate this into the cpanminus Reporter. So if you get similar repeated reports, please be gentle with the tester :)

Subramanian Karunanithi was the casualty of version numbering with CGI-Test. Thanks to some investigation work from Jonas B. Nielsen, he highlighted the latest version was 0.52, but the installed version was 0.104. The problem with version numbering within the installers is that they see 0.104 as Zero Point One Hundred and Four, which is greater than Zero Point Fifty Two. Karen posted a bug report, and Alex Tokarev has now released version 1.000, so all good. However, it does bear repeating that Version Numbers Should be Boring.

Next month we'll have a few notes about LPW, more fixes to the toolchain and some ideas for the future.

Posted by Barbie
on 5th July 2014

For a few months now, Slaven Rezić has been asking me to look at the way the Find A Tester tool keeps bring back an old email address of his, rather than the current one. The problem is the way that Metabase now stores the tester reference for a report. Instead of a specific email address it stores a resource fact, that just happens to be a tester. The tester resource fact then has a number of email addresses associated with it, except it has no way of knowing a tester's preference for which email address is most suitable. As such the list I retrieve, is essentially random. There isn't a good way to distinguish between new and old address.

In my workings on the CPAN Admin site (more news on that very soon), I spotted a gap in the knowledge a tester can tell us about themselves. Their current contact address. In many cases, particularly if they are also a CAPN author, this could be their PAUSE email address (AUTHOR@cpan.org), but equally could be a central email address they prefer to be contacted on. This then enables the tester to keep their contact information current, even if they change jobs. The mechanism behind the scenes now uses this, if available, as the primary addresss to display in the Find A Tester tool, but also now in the "raw" reports on the Reports website, and in the Reports Mailer tool that sends out the summary and report mails to authors.

There is still more to be changed to ensure that the contact address is considered the primary address, but hopefully it will get easier and more reliable to contact testers.

To find out how to update your contact address, please see tomorrow's post :)

So the 2014 QA Hackathon has drawn to a close, but it is far from the end of the work, particularly for CPAN Testers. You can read several blog posts detailing many aspects of the QA and testing community work done during the hackathon, as well as several aspects of the toolchain, including PAUSE (which saw a lot of collaboration). It does get said often, and it bears repeating, that the QA Hackathons are a vaulable part of the Perl community, and help to drive many projects. Without them it is likely that key elements of the infrastructure we have come to rely on (PAUSE, CPAN, BACKPAN, MetaCPAN, CPAN Testers, CPANTS) would be a long way from being the resourceful, stable and continually developing components we have come to accept. Next year's hackathon will be in Berlin, and I for one am very much looking forward to it.

Aside from the work on the database for CPAN Testers during the hackathon, I did get to make a few releases, but several elements I started during the hackathon, were only completed during the following weeks. One of these CPAN-Testers-WWW-Reports-Query-Report, now enables you to retrieve the Metabase Fact object, JSON or hash representation of a specific report. For those looking at analysing the similarities and differencies between reports, this may make things a little easier, particularly when we start introducing more element facts into the report fact. Currently this only works for reports stored in the Metabase, so those early reports are not currently retrievable .. yet.

I discussed a new command line tool to submit reports with H.Merijn "Tux" Brand, who is keen to run reports in standalone instances. I see this working similar to Garu's cpanm-reporter, and with the common client that Garu has been working on, this could be a nice addition to the submission options. Liz Mattijsen, Tobias Leich (who easily has the coolest gravatar on CPAN) and I talked about how Perl6 distributions could be incorporated into CPAN Testers. There are some subtle differences, but there are also many common factors too. It was interesting to read the Rakudo blog post about Perl 6 and CPAN, as overcoming some of the hurdles potentially facing Perl6 developers are likely to help make CPAN a better place for all of us. The currently proposed solution is a similar approach to how different namespaces are stored in the Metabase. For the time being though, Perl6 distributions are excluded from CPAN Testers, but once we have a Perl6 smoker there is no reason not to include them. I'm not sure how soon that will be, but watch this space.

Andreas König and I once again looked at the way the reports are stored in the Metabase. Andreas had already highlighted the updated date, which is meant to be the date it entered the Metabase, was in actual fact the created date (the date of the testers platform). Along with David Golden, we looked at the code used by the Metabase, but failed to find anything wrong with it. It's hopefully something we can take more time over in the future, however the next priority for the Metabase is getting it moved onto MongoDB and away from SimpleDB. In the meantime, the Generation code, due to time constraints and the lack thereof, has been running using the 2010 version of the Metabase/AWS interface. During the hackathon and the following weeks, I finally upgraded to use the revamped version released in 2012. Although still troublesome to find all the reports, the search interface has been much improved, and now we have a much more reliable feed from the Metabase. This is also in part due to the rewrite of the internals of the Generation code to be more pro-active in find unusual gaps between reports.

I spoke with Neil Bowers during the hackthon too. Neil had suggested some ideas that I'd wanted to included into the CPAN Testers Statistics site. We discussed others during the hackathon, and although I have several notes to work from, it will be a little while yet before I can put aside some time to implement them. Neil has no end to the ideas to help improve CPAN, and I hope he'll be a good sounding board for ideas to incorporate into the site in the future. On the first day of the hackathon he posted about how quickly CPAN Testers test a distribution, after being uploaded to PAUSE. He was surprised to see some reports posted almost instantly. This is largely thanks to Andreas' work to bring the Tier 1 mirrors up to date within 10 seconds of a distribution being successfully uploaded to PAUSE. People such as Chris and Andreas use their own T1 mirrors to feed into their smokers, so as a consequence the reports can appear on the Reports site within one hour of the distribution hitting PAUSE!

I had intended to launch the CPAN Testers Admin site during the hackathon, but didn't get the chance to prepare a launch statement. I did wonder how a launch on 1st April would go down, but elected to wait a little longer. So expect some news on that front very soon.

Just before the hackathon we reach 40 million report submissions. An impressive number by any standards, and gives a compelling argument that Perl & CPAN are still being actively maintained and developed.

John Scoles made a post related to CPAN Testers, highlighting how CPAN Testers showed why a particular test needed a little more thought. Thanks to the wide variety of perls and platforms tested by the CPAN Testers, it provided John with some intriguing failures, and made for an informative look at how hashes called in scalar context are handled differently between versions of Perl and platforms. Prior to the QA Hackathon Neil Bowers posted about his new idea for An author's CPAN dashboard, which he continued to develop during the hackathon too. We even discussed, together with Olaf Alders, whether this was something that should feature in MetaCPAN. For now it won't, but hopefully it can develop further and w'll see what people make of it, once more APIs are fed into the dashboard.

Another month wrapped up, and another QA Hackathon over. Lots to do and plenty of fresh ideas coming. Happy testing.

A belated update this month. December proved a busy month preparing the new CPAN Testers Admin site. The site is still being tested, but its full launch will be coming soon. A new testers database is now in place, and the updates to the CPAN Testers Statistics site have been made much more dynamic.

A further update to the Statistics site is thanks to a suggestion from David Golden. On the Statistics of CPAN you can now see the current size of both CPAN and BACKPAN. Current sizes are 17GB and 34GB respectively. I've also added the current size of the static reports site too, 51GB as of now. Meanwhile the complete set of databases comprise 621GB of disk space. Thankfully we still have 1.6TB left on the hard disk, so plenty of remove to move.

My thanks to Andreas König over the past month, as he has been able to monitor performance of the databases, and highlighted a few issues. Since moving to the new server, I'm still having to make a few tweaks to the configs. Please let me know if you notice any part of the websites that appear to have stalled. Sadly the stalls have mostly been due to the way SimpleDB returns its data. As I mentioned in my London Perl Workshop talk, I am not a fan.

Speaking of London Perl Workshop, it was great to meet up with folk again, including Andreas König and Neil Bowers. I also had some discussions with a few people about the proposed search site, which Ben Bullock will hopefully looking at this year. My talk, The Future of CPAN Testers, seemed to go down well, and hopefully we'll keep you posted on the progress of the on going projects throughout the year.

David Yingling posted an interesting blog entry, You Can't Test Everything, But At Least Test What's Important, that briefly mentions how CPAN Testers helped to kick start a series of bugfixes for his Fetchware distribution. If CPAN Testers have helped you, please blog about it, and let me know so I can feature in the summary.

Coming soon is the 2014 QA Hackathon. This will be happening in Lyon, France during March, so expect lots of testing related projects and news to be forthcoming, before, during and after the event.

Until next time, happy testing.