CPAN Testers is only made possible with the support of our sponsors.
For more information on sponsoring, please visit the I CPAN Testers website.

Upgrade Notice

The CPAN Testers Blog site has been upgraded since you last accessed the site. Please press the F5 key or CTRL-R to refresh your browser cache to use the latest javascript and CSS files.

News & Views

Posted by Barbie
on 20th January 2010

Having monitored the logs on the server over the last few days, I've noticed some improvements that can be made to some of the processing. Some of the improvements are in the server code itself and will be introduced alongside some of the changes towards CPAN Testers 2.0.

However, some of the proposed changes are to the files available on the Development site. Specifically the database downloads. As there hasn't been anyone accessing any of the CSV files for the Uploads DB, these will be removed next week. Of the other files, although the GZip and BZip files are accessed, they aren't accessed very often, except for the cpanstats DB. As such the archives for both the Uploads DB and Release DB will only be updated once a day from next week.

These changes are to help reduce the processing load on the server, which in turn will help to improve response times for web pages and allow more processing capacity for parsing reports once we move to the Metabase.

Posted by Barbie
on 18th January 2010

Due to a number of distributions in the last year requiring 5.10, a recent attempt to install some Perl modules on the server failed :( As the server itself was running Debian Etch, which only has 5.8.8 as the latest version of perl, an upgrade was initiated. The server itself has now been upgraded to Debian Lenny, which comes with a default of 5.10.0.

However, all did not go smoothly. Due to the upgrade to 5.10, a number of distributions required reinstallation. While many installed without a fuss, some needed handholding, and while I'm hopeful that all are now installed, there may be one or two oddities that might be seen.

As such, if you come across a web page that isn't loading correctly, or are having problems connecting to the CPAN mirror, or are having problems with some other access you expected to be working fine, please let me (barbie @ cpan.org) know ASAP. Sorry for any inconvenience caused.

File Under: server
3 COMMENTS
Posted by Barbie
on 15th January 2010

If you've suffered any problems accessing any of the sites, the databases, the CPAN mirror, etc from the CPAN Testers server last night, please direct your wrath at Microsoft. Last night the msnbot took out the CPAN Testers server with a dedicated denial of service attack. As a consequence measures are now being put in place to completely ban the msnbot from accessing at least the Reports site, and probably all the sites on the server.

Microsoft in their incompetent wisdom decided to unleash 20-30 bots every few seconds. I know this because I can see the IP addresses in the logs. The ones spotted within a few minutes of rebooting the server this morning to clear the processes were:

65.55.207.50
65.55.207.23
65.55.207.93
65.55.207.25
65.55.207.48
65.55.207.46
65.55.207.72
65.55.207.26
65.55.106.234
65.55.107.179
65.55.207.100
65.55.207.121
65.55.207.30
65.55.207.69
65.55.207.28
65.55.107.180
65.55.207.27
65.55.207.47
65.55.207.21
65.55.207.51
65.55.207.54

It seems their bots completely ignore the rules specified in the robots.txt, despite me setting it up as per their own guidelines on their site, and worst of all they don't talk to each other to see they are accessing the same domain. Most sensible bots, such as those of Google or Majestic 12, will only let one bot at a time crawl a site, as most sensible companies acknowledge that a DOS attack is not good policy. As a consequence I'll now be denying access to anything with the IP matching /^65\.55\.(106|107|207)/. If you discover you fall into that pattern, and are a real person, please let me know.

If anyone from Microsoft ends up reading this, though likely you'll have to do it in person and not via a bot, I now consider you to be no better than a script kiddie trying to bring down a government computer. DOS attacks usually get people charged and arrested. If CPAN Testers was a legal entity, then I might have been able to follow this through. Instead I'm locking the doors, and no longer letting you through.

UPDATE: Microsoft added a whole bunch of IP addresses to those previously listed, to the point I've now had to block any 65.55.*.* address. The others are now getting 403s so maybe eventually they might get the message, though they haven't stopped yet!

File Under: server
2 COMMENTS

Last month CPAN Testers was finally given a deadline to complete the move away from SMTP to HTTP submissions for reports. Or perhaps more accurately to move away from the perl.org servers, as the amount of report submissions has been affecting support of other services to the Perl eco-system. The deadline is 1st March 2010, which leaves just under 2 months for us to move to the CPAN Testers 2.0 infrastructure. Not very long.

David Golden has now put together a plan of action, which is being rapidly consumed and worked on. The first fruits of which has been an update to the CPAN Testers Reports site. The ID previously visible on the site, refering to a specific report, is now being hidden away. The reason for this is that the current ID refers to the NNTP ID that is used on the perl.org NNTP archive for the cpan-testers mailing list. This ID is specific to the SMTP submissions and includes many posts which are not valid reports. As such we will be moving to a GUID as supplied by the Metabase framework, with existing valid SMTP submitted reports being imported into the Metabase. The NNTP ID will eventually be completely replaced by the Metabase GUID across all parts of the CPAN Testers eco-system, including all the databases and websites. As such you will start to see a transition over the next few weeks.

The second change which has now been implemented, is to present the reports via the CPAN Testers Report site and not the NNTP arcive on the perl.org servers. Currently the presentation of a report (e.g. this report for App-Maisha) is accessed via the reports pages for a distribution or an author, but will also be accessible in a similar manner across all the CPAN Testers websites. There are a large batch of early reports that are currently missing from the database, but these are being updated now, and will hopefully be complete within the next few days. If you have any issues with the way the reports are presented, including any broken or missing links from other parts of the site, please let me know.

In all this change, there is one aspect that may worry a few people, and that is the "Find A Tester" application. For the next few months it will still exist, but the plan is to make the Reports site more able to provide tester contact information. In addition to this the testers themselves will soon have the ability to update their own profiles. Initially this will be used to link email addresses to reports and then map those email addresses to a profile held wihtin the Metabase, but in the longer term will be used to help us manage the report submissions better.

David Golden is concentrating on the Client and Metabase parts of the action plan, and I am working on porting the websites and 'cpanstats' database. If you have any free time and would like to help out, please review the action plan, join the cpan-testers-discuss mailing list, and please let us know where you'd like to help. There is a lot of work to be done and the more people involved, the better the spread of knowledge in the longer term.

After David announced the deadline last month, all the testers have throttled back their smoke bots. This saw a dramatic reduction in the number of reports and page being processed, and enabled the Reports Page Builder to catchup with itself, to the point it was frequently having less than a 1000 request waiting. That changed yesterday with the changes to the website, as every page now needs to be updated. It typically takes about 5 days to build the complete site, so this quiet period will help allow the Builder to rebuild the site, without adversely affecting the currently level of report submissions. Expect the site to reach a more managable level of processing some time next week. To help monitor the progress of the builder, a new part of the Reports site, The Status Page, now checks the status of all outstanding request every 15 minutes, providing a 24 hour persepctive and a week long perspective.

A new addition to the family was also launched recently, the CPAN Testers Analysis site, which Andreas König has been working on, to help authors identify failure trends from reports for their distributions. Read more on Andreas' blog.

Last month we had a total of 168 tester addresses submitting reports. The mappings this month included 22 total addresses mapped, of which 2 were for newly identified testers. Another low mapping month, due to work being done on CPAN Testers as a whole.

My thanks this month go to David Golden for finding the time to write an action plan, and his wife for allowing him the time to write it, as well as working on all the other areas involving the CPAN Testers and the Metabase :)

 

Posted by Barbie
on 1st January 2010

Over the last few months Andreas König has been analysing lots of CPAN Testers Reports to understand patterns for why distributions might be failing. The result is the new CPAN Testers Analysis site.

Andreas has written a post entitled "So, you got mixed results from cpantesters?", which expands a little on why he came to create the site. Hopefully it will prove useful to authors and testers alike to identify problem areas, particularly ones that aren't necessarily obvious at first glance.

<< February 2010 (2) December 2009 (4) >>