CPAN Testers is only made possible with the support of our sponsors.
For more information on sponsoring, please visit the I CPAN Testers website.

Upgrade Notice

The CPAN Testers Blog site has been upgraded since you last accessed the site. Please press the F5 key or CTRL-R to refresh your browser cache to use the latest javascript and CSS files.

News & Views

Posted by Preaction
on 9th August 2017

Summary: I will be doing work on the Metabase API on 2017-08-12. Writing test reports may be unresponsive for a few minutes, and there may be bugs. Please let me know if there are any problems submitting test reports.

I have completed the processing script for the new test report format. This was the last step in moving the Metabase API away from Amazon and on to our MySQL cluster for cost and stability reasons: Amazon SimpleDB is too expensive, and its limitations for our purposes outweigh its costs. We have always maintained a copy of the Metabase data in our MySQL database, and there's no real need to continue having two live copies of the same data (especially when one of the copies costs money every time you ask for a piece of data).

This Saturday, 2017-08-12, around 1:00 PM US/Central (18:00 UTC), I will be switching DNS over to the new, backwards-compatible Metabase API which writes to our MySQL database. A few months ago, I asked for testers to try this new API out, and everything went well (thanks to everyone who helped with that). The new API works the same as the old API: No changes are needed for your testers or anyone consuming the minimal data feeds out of the Metabase API (the log.txt view).

Since this is only a DNS change, the downtime for the change should be zero as DNS propagates and your testers are pointed at the new IP address. Since it's possible for me to mess up this change, there may be some downtime. Since all software has bugs, there may be some downtime if any bugs are revealed by all the testers being migrated to the new API.

This change (and all the work around this change) sets up the project for new changes down the road:

  • Speeding up report processing by triggering individual processing jobs
  • Distributing those jobs over multiple machines to improve performance
  • Making test report text available immediately after submission

If you have any questions, you can join the CPAN Testers discussion mailing list, or e-mail me directly.

Since our Fastly caching seems to have been working without issue for most of a year, I've rolled it out to the second-level domain (cpantesters.org). Fastly has also graciously donated us with a wildcard SSL certificate that works for any of our Fastly-configured domains, so now you can access https://cpantesters.org and the SSL certificate on metabase.cpantesters.org has been fixed.

If anyone notices any problems with the site or its DNS over the coming weeks, let me know and I'll fix it.

 

Special thanks to Fastly for continuing to provide an excellent service, free-of-charge, for the CPAN Testers Project.

Posted by Preaction
on 7th February 2017

There was a problem with our moving the database to a remote server, and as a result, some of our page build processes have been timing out. Because of this, the data shown on the website, and by the CPAN Testers Matrix has been growing more incomplete compared to the database. Some of you have reported data disappearing from the matrix, or improbably low numbers of test reports, both of which are caused by this issue.

None of the data in the database is affected, so these pages can be rebuilt. However, since they've been being partially rebuilt for a few months now, the caches are all invalid and must be deleted. Rebuilding these caches has begun, but it will likely take a few days up to a week. During this time, the data shown by the CPAN Testers website and some other consumers will remain incomplete.

As the new CPAN Testers API accesses the database directly, it always has accurate and complete data. If you'd like to help expand the API to include more data, check out our Github repositoryjoin our mailing list, or e-mail me and we can help get you started.

Posted by Preaction
on 9th November 2016

We're going to have a short period of downtime tomorrow to create a snapshot for our new MySQL replication. We're using ZFS, so the downtime should be limited to a few minutes sometime between 12:00 and 17:00 US/Central (UTC-6). Once that is in place, we'll be setting up monitoring for our master-master MySQL cluster and the ZFS file systems they live on. Using ZFS snapshots means we should have a lot fewer problems with backups, and indeed we can start taking daily or even intraday backups.

Over the next few weeks, we'll also be migrating the local website copy of the metabase to InnoDB (hopefully before MyISAM is fully deprecated and removed). Percona has a tool that will allow us to do this while the database is online, so there should be no downtime for this operation however long it ends up taking. Given that we're already in a multi-master cluster, and that we have regular ZFS snapshots, this won't give us that much more protection against downtime, but with MyISAM's deprecation, this is still something we need to do.

We are still catching up on the incoming test reports from our previous downtime, and we should reach equilibrium soon. Tomorrow's downtime should not affect that at all.

Posted by Preaction
on 30th October 2016

CPAN Testers was down for two weeks starting October 16, 2016 and ending today, October 29, 2016. This downtime was caused by a combination of running out of disk space and systemd processes in invalid states forcing a hard reboot resulting in a MySQL table repair. CPAN Testers is back up, and is working through two weeks of report backlog. It should be caught up in a week or so.

 

In our downtime time we:

 

  • Moved the MySQL databases to a new server cluster on an upgraded MySQL
  • Repaired the `metabase` database, which contains 75,000,000 records and weighs 675GB
  • Migrated most databases to InnoDB to prevent future, expensive repairs

 

Over the next few weeks we will be migrating the rest of the databases to InnoDB, which will be done while the site is live now that we have the disk space to do it. We are also setting up failover replication to improve performance and availability, and starting more frequent, zero-downtime backups.

 

I want to thank ServerCentral for hosting our new database servers, and Zach Dykstra from ServerCentral for setting them up and doing basically all the work.

 

Since this last happened a year ago, a lot has changed on the operations side of CPAN Testers, but not much has changed in the applications that make CPAN Testers work. This will be the focus of the next year (and beyond). A lot has changed since CPAN Testers was built, and there's a lot of modernization that can be done to improve what CPAN Testers can offer the Perl community.

 

 

But this is a job bigger than the people we have. To that end, I've started writing a project roadmap, available on Github. This document outlines the parts of CPAN Testers, what they do, and how they could be improved. CPAN Testers is a web application with a lot of data transformation (ETL and SQL reports) and storage. If you'd like to help, or you'd like to learn, join us on our IRC channeljoin our mailing list, or e-mail me with how you'd like to help.