CPAN Testers is only made possible with the support of our sponsors.
For more information on sponsoring, please visit the I CPAN Testers website.

Upgrade Notice

The CPAN Testers Blog site has been upgraded since you last accessed the site. Please press the F5 key or CTRL-R to refresh your browser cache to use the latest javascript and CSS files.

News & Views

Posted by Barbie
on 31st August 2011

Unfortunately the CPAN Testers server has a failing hard disk, which is only allowing READONLY mode access. As such, all CPAN Testers operations are now failing.

The host company has confirmed that ...

a) our disk has a fault
b) we don't have a second RAID disk (even though we're paying for it!)

As such to replace the disk it requires complete data loss!

As you might anticipate this is not making me happy today. I had already started making plans to migrate to a new server, but to do this gradually over the next month. However, it looks like this will need to be sped up.

As things stands the CT server is effectively out of action. I have read access, but will need to wait until the new server is ready before I can copy over everything as it currently stands. Although I have several backups of data, I would rather avoid rebuilding the complete set of data files.

I will keep you updated with further updates as and when I know more or when there is further progress.

Please note that this does not affect the Metabase server, and all reports submitted are being stored successfully. Once the (new/old) CT Server is up and running the feeds from the Metabase server where we left off.

Please bear with us while we get things sorted.

Update: Server is now dead and no longer responding. Awaiting new drives, then will be rebuilding from scratch :(

Update 2011/09/03: HTTP, FTP & rsync all re-enabled for CPAN and BACKPAN. Databases still rebuilding.

File Under: server
3 COMMENTS
Posted by Barbie
on 19th August 2011

Recently Jon 'JJ' Allen sent me some updated graphics with the official new design of the onion for all the CPAN Testers websites. As such I'm now in the process of updating the sites. In most cases you will likely not notice a difference, however, I've taken this opportunity to push out the changes I've been working on for the Blog site. If you haven't already, you may need to refresh your cache to see the latest changes.

Posted by Barbie
on 11th August 2011

Due to the backlog of reports from the Metabase feeds, the reports page builder has been disabled for the time being, to allow more resources to process the feed and the database connections. The feed is currently 4 days behind and needs some breathing space to catch up.

The builder has been performing well and has typically been only 24hrs behind. With the reworking in processing, any backlog incurred now should be processed much more quickly than it has in the past. As such, I shall be running the builder for a couple of hours a day today and tomorrow, so popular distribution should still get updated. However, Infrequently viewed distributions will have to wait until the weekend when I enable the builder properly again.

UPDATE: The feed has now caught up and is now 2hrs behind. The builder is again 24hrs behind, and improving further. This should now mean that from submission, a report should appear on the respective distribution and author pages within a day.

957,044 ... that's 957,044

That's the number of report submissions we saw during the 31 days of July! This biggest monthly submission we've ever had. Just over 40,000 reports more and we would have broken the 1 million barrier. Considering it took 9 and a half years to reach out first million milestone, the fact that we're now seeing nearly 1 million a month is just staggering. I've stopped posting about passing each million mark as its becoming to frequent. You'll have to wait for the 20 millionth report (expected about Christmas 2011 at the current rate) for the next notable post in that regard.

All these new submissions has had one significant effect. They have exposed our stress points. While I have been already working on improving many aspects of the report builder process, I hadn't been looking at the generator side of things, which takes a feed from the Metabase and inserts the reports in a queue for the builder. The generator started to fall behind, and thanks to the great Devel::NYTProf, I ran several passes over the generator code to see where we were exposed. The two noticeable points were the Metabase feed and the database communication.

Although we were already aware of it, the current level of submissions has highlighted how badly Amazon's SimpleDB is able to cope with the volumes of throughput we now have. As a consequence, David Golden has been working on redesigning the Metabase to use a NoSQL database with much better scaling to cope with future volumes. In the coming months, we'll keep you updated of the progress, and I'm sure David will be posting significant challenges on his own blog. In David's own words; "I've written some non-Amazon back-end classes and will be testing them at scale over the next week or two.  If all looks good, I plan to switch over the servers and migrate the legacy data."

The database communication highlighted several areas where single requests were made, where a single large request would be of more benefit by caching the data. There are still more optimisations to be completed, but already the changes I've made have improved the performance. In addition the database has been reconfigured and updated, which has also helped to improve performance.

We're currently seeing a 2-3 day lag of the Metabase feed, which is catching-up slowly, and the reports builder is now roughly 24hrs behind. It's going to be sometime before we are almost realtime, but in the short term we should be able to have all pages displaying their reports with a day of being posted. However, so you know whether your report should have appeared the Report Status page has been updated. The latest report is the most recent report added to the cpanstats database from the Metabase feed, while the oldest page request tells you how old the queue for page building is (note that not all requests are report submissions). Use both to guage whether a particular report may still be waiting to be add to the appropriate page.

In other news I happened to be reading Google+ the other day and spotted a post by Paul Johnson (creator or Devel::Cover), who was able to fix a problem with Devel::Cover thanks to some reports by CPAN Testers. Paul has allowed me to reprint his post:

"CPAN Testers was throwing up a few failures for Devel::Cover. The problem was in a fairly convoluted test whose results were dependant on the number of perl modules being loaded, and that in turn depended on the particular configuration of perl being used.

(The module which might or might not be loaded was Tie::Hash::NamedCapture which provides the behaviour for %+ (new in 5.10), and is used depending on on whether the coverage database is being stored using Data::Dumper or JSON and that depends (before 5.14) on whether or not a JSON module has been installed.)

I thought I had fixed the problem during the Perl QA hackathon in Amsterdam a couple of months ago, but it turns out that I hadn't. The reason was that the API that I had in place (such as it is) for massaging the test output wasn't quite as comprehensive as I thought it was. So I made it more comprehensive, and now I'm expecting far fewer CPAN Testers failures for the next release.

Big thanks to all the CPAN testers for showing me the bugs in my code."

Thanks to Paul for a great example of how CPAN Testers can help. If you have any stories of how CPAN Testers has helped you, please let me know or post on your blog and send me a link.

And finally... YAPC::Europe will be happening 15-17 August, and two specific CPAN Testers talks are to be featured. The first, "Smoking The Onion - Tales of CPAN Testers", will look at some of the hidden corners of CPAN Testers, which can hopefully improve the experience both for authors and users. The second talk, "How CPAN Testers helped me improve my module" by Léon Brocard, takes a look from an author's perspective of how you can use CPAN Testers to help you improve your modules. There are several other Test/TAP related talks too, so if you're going, I hope you can make it along to some of the talks. See you there.

File Under: summary
1 COMMENT
<< September 2011 (4) July 2011 (2) >>