Perl NOC

email issues: .org blocked, now fixed

For about the past 18 hours, you may have experienced issues sending emails to perl.org/cpan.org/pm.org addresses, if those emails contained a reference to any .org URI.  One of the uribl providers we use accidentally added all of .org to their list of things to block.  

This has likely resulted in mail being permanantly rejected, and it will not be resent.   If you sent such an email, you should have received a bounce message from your mail provider. If you are subscribed to any other .org email lists via your cpan.org address, you may wish to double check that you haven't been unsubscribed due to bouncing.

We have disabled the errant uribl provider, and informed them of the issue.  We may not re-enable them after they fix the problem.  (Although, historically, their data has been a very good signal for blocking spam.) 

Attentive network services

One of the great things with Internap was that they were extremely proactive; the few times there was trouble in their network we'd have their mail telling about it at the same time as we got the monitoring alerts.  When we unplugged all the equipment to move it a few months ago we had a voicemail from them when we came up above ground to cell phone coverage.

I'm happy to say that getting ip transit from Phyber is much of the same.  Last week the MSN web crawler did ~10Mbit of traffic crawling one of the perl.org servers for a few days.  When it stopped and the traffic dropped we had an email from Phyber checking in if everything was okay!  We hadn't noticed that the traffic came and went, but they did.  :-)

svn.perl.org briefly down

svn.perl.org will be unavailable for the next 15-25 minutes while we move it to faster storage.

Minor network issue resolved

Some of the switches between us and our uplink didn't play nice together and it made the link effectively only allow each connection to transfer at a couple megabits a second (or less!)

We got it resolved yesterday and now you can download Perl in just a few seconds again.

IP transit from Phyber Communications

As part of the move a couple weeks ago the perl.org internet connection got upgraded from an often somewhat over-saturated 10Mbit connection (hey, that was pretty fast 8 years ago!) to a 100Mbit link generously provided by Phyber Communications.

We can't say enough good things about the insanely effective and proactive Internap NOC, but after years of getting free connectivity from them (it was setup by our friends at Ticketmaster who called in a few favors), it was time to try something new.  We're very grateful that Max Clark and Phyber got us hooked up.

One of the exciting things is that we now have an IPv6 connection, too.  Over the summer we're planning to start running some services on that.

nntp.perl.org and pm.org lists temporarily down

The server running the nntp server and the pm.org lists has gone temporarily insane.  We'll fix it in the morning.

Update: I lucked out that my "maybe this will fix it" attempt I started while going to sleep fixed it, so it was fixed shortly after breaking.

svn.perl.org and rt.cpan.org a little slow tonight

svn.perl.org and rt.cpan.org are both running a little slow right now because we're moving data around on the underlying xen instance, and it's causing IO starvation to the box.  Things should be back to normal later tonight.

Downtime (long) over, move complete!

As you've may have already noticed, perl.org is back up.  The move is complete (all 0.7 miles of it.)  At this point, we believe that everything is working again (at least as well as it was before the move), but it's possible that we missed something.  If you find something, please let us know at webmaster at perl.org, and we'll fix it when we wake up.

Special thanks to IX2 (our awesome colo provider) and Phyber (our new transit provider)!

Downtime / Move Starting Now

We are now beginning the move of the rest of our boxes to our new datacenter.  See you in a few hours.  Watch this space for updates.

Move slightly delayed

The perl.org datacenter move (that was going to happen today) has been postponed while we stabilize our network connection at the new site.  The move may happen tomorrow, Sunday, March 22nd, or later this week.  We'll try and remember to post here before we take things down.

perl.org is moving! (downtime announcement)

perl.org is moving... down the street.

Our wonderful colocation provider, IX2 Networks, is closing down one of their Los Angeles facilities, and we're moving to another one.  We're going to be sharing a cage with YellowBot.  We're looking forward to this move, as the new facility is above ground, our cell phones work, and you can even kind of see daylight if you try.

Since we're going to be physically moving our boxes, this means there will be some downtime and service unavailability.  For a few hours between Thursday, March 19th and Saturday, March 21st, some perl.org (and related) services may not function for you.  (We are being vague because we haven't yet decided when to actually move the hardware.)  We are going to try and minimize downtime as much as we can.  No email will be lost, although it may be delayed.  DNS services will still work fine.

We will update http://log.perl.org as we go.

svn.perl.org moved to a new machine

Tonight, we moved svn.perl.org (and a few other svn vhosts) from real hardware to a virtual Xen box.  You shouldn't notice anything different, but if you do, please let us know.