CPAN Testers is only made possible with the support of our sponsors.
For more information on sponsoring, please visit the I CPAN Testers website.

Upgrade Notice

The CPAN Testers Blog site has been upgraded since you last accessed the site. Please press the F5 key or CTRL-R to refresh your browser cache to use the latest javascript and CSS files.

News & Views

Posted by Barbie
on 28th October 2009

With the recent updates to the matrices, there is one new addition that I forgot to mention. Again thanks to Jon 'JJ' Allen's suggestion, you can now download all the matrices as Excel formatted files, such that you can now download and analyse them all in a spreadsheet application that supports the XLS format.

In order to create the new downloads, I have created the new CPAN release, CPAN-Testers-WWW-Statistics-Excel, using the amazing Spreadsheet-WriteExcel. However, as I only use Linux machines now, I have been unable to test these in a real Excel application. If there are any problems loading the files on a Windows machine, I'd be grateful of any feedback, especially as patches.

If you have any suggestions for further improvements to the matrices, or ideas for more matrices, graphs and stats, please let me know.

Posted by Barbie
on 26th October 2009

With thanks to Jon 'JJ' Allen for the suggestion, CPAN Testers now has a new dashboard on the front page of the CPAN Testers Statistics site. 

It's often been asked whether there is a counter of some kind that counts the number of CPAN uploads, thus providing a way to instantly the see the number of distributions on CPAN. As that counter doesn't change very often, I thought including the number of test reports submitted might be a little more dynamic, so you now you get both.

To add a bit of colour, you also get a few selected graphs, hopefully creating a nice snapshot of the current state of CPAN and CPAN Testers.

Posted by Barbie
on 22nd October 2009

Following on from the previous updates, further work has been done on the matrices. The old matrices suffered from seeing mostly a blank table when you opened the page, which isn't really too inviting when you first load a page, and can also give the wrong impression. As such both X & Y axes now include totals, with the Y axis being sorted in descending total order, and the X axis being sorted in order of Perl version. Now each matrix loads with a bit of colour immediately, rather than having to seach for it. An example is the matrix for OS, official Perls for last month, highlighting that Linux really is extremely well tested on more recent versions of Perl.

A further change to the matrices is to provide a widescreen version. Essentially this is for those who have asked to see the full table and print it, without the header and menu navigation layout. Again an example is the matrix for Platforms, all Perls and all reports, which one of the larger tables and would otherwise require a lot of scrolling to see the full table.

Lastly, for this update, a new table has also been introduced for the Monthly statistics, that of Operating Systems. Although the OS matrices show the full picture and a snapshot for the previous full month, it doesn't show how the data has changed each month. This new table highlights better the monthly volumes of reports for each Operating System, and also focuses attention on Opearting Systems which could use a little more attention.

The October Technical Meeting for Birmingham Perl Mongers include a presentation of the new changes, and opened up a discusssion for what other features and statistics could be included in the future. My thanks to JJ and Colin for their ideas and thoughts, as they provided some great suggestions which I shall be working on over the coming weeks. As such, expect even more updates soon :)

Posted by Barbie
on 19th October 2009

Over the last week and weekend, I took a fresh look at the matrices on the CPAN Testers Statistics site, following some suggestions from tester MW487, The matrices as they previous stood, didn't really reveal that much. Plus as the active cells were so spread out, reading each matrix was awkward.

Firstly the numbers for Reports and Distributions were merged into the same cells, thus making it easier to see the corrolation between the number of reports submitted and the number of distributions being tested. In addition the split between the matrices is now based on general OS name and the platform reference in the perl -V block. The reasons for this are twofold; firstly it makes it a little more obvious where testing is lacking, and thus where new testers might want to concentrate their efforts, and secondly for users to highlight Operating Systems and platforms where testing is well supported.

The next step was to provide matrices that covered only officially released perls, as well as all released perls, followed by matrices that highlighted testing within the last month, as well as the whole of CPAN testing. Lastly I've brought back the colour coding, or fading, to further highlight testing strengths and weaknesses.

There a now further plans to expand the matrices, but these are the first major changes. Expect more work on the statistics in the coming months.

Posted by Barbie
on 16th October 2009

In the last post I mentioned that the server was getting hammered by robots. If the CPAN Testers Reports site was the only site served by the DB, this wouldn't be a problem, however in order to build the database and serve several sites, the DB gets quite a demand placed on it. I've tweaked some of the configurations for the DB, such as increasing cache and reducing connection timeouts, but this is only eases the symptoms, and doesn't get to the heart of the problem.

The problem is that of long running queries, particularly for some very complex queries. So this week, I've been taking a look at some of those queries. In some cases I've been able to take advantage of index and summary tables that are now used, thus reducing complexity, and re-evaluating processing to reduce calls too. This has meant changes to the Reports Summary AJAX script, The Uploads Generator and the Statistics Generator, all of which now take less time and have noticeably reduced their load on the server. We're still seeing lots of robot hits, but now they are not taking out the server.

The biggest change has been to the code behind the CPAN Testers Statistics website, by reducing processing time by as much as 20 minutes. It's been the first step to making it easier to produce statistics, and introduce new ones. While reworking the code to reduce DB load, I've also introduced some new features to the site.

The first new feature is the availability of the raw data that is used to generate the graphs. Last month Tim Bunce wanted to update his Perl Myths talk, but had to ask me to produce some fresh graphs for him, rather than be able to generate them himself as he wanted them. At his suggestion, I've now made all the raw data files available as plain text (in CSV format) that produce all the graphs you see on the site. I'm hoping to produce more graphs in the future, but I'm also hoping that others may be able to use the raw data to produce other graphs that highlight different trends too.

The second new feature is the Performance Graph. This shows the number of requests the CPAN Testers Page Builder is processing and the number of reports being submitted each day. The 'requests' is the total number of requests for a page rebuild that have been processed, and 'page' indicates the number of unique requests that have been processed. A request is created when a report is submitted, a new distribution is uploaded or a website user requests a page on the dynamic Reports website, where a previous request exists. A page can either be a distribution page or an author page. Although it may appear that the page processing is low (it's actually producing 6 system files for every unique page request), the fact that the total requests processed is far higher than the number of reports being submitted, does mean that the Page Builder is performing well.

As you can see from the Performance Graph, the site attacks are more evident, but it shows that the Page Builder is coping very well with the ups and downs of the server load. There are more tweaks to come, as I turn my attention to other areas of the site, as well more changes to the CPAN Testers Statistics site, following several suggestions I received over the last few months. Expect more updates soon.

<< November 2009 (5) September 2009 (2) >>