Page MenuHomePhabricator

LAMP instance becomes 404 a few hours after spawn (reproducible)
Closed, DeclinedPublic

Description

Steps to reproduce:

  1. Create a LAMP instance as per https://wikitech.wikimedia.org/wiki/Help:Lamp_Instance
  2. Check that it serves index.html correctly
  3. Add a few HTML/PHP files and folders
  4. Check that the files are served correctly
  5. Wait a few minutes or hours
  6. For no apparent reason, the server becomes unaccessible, always spitting 404 errors, see for instance: http://i-000008cf.pmtpa-proxy.wmflabs.org

Reproduced 2 times.
Removing all files (except index.html) does not solve the problem.
The only workaround I have found is to delete the instance and create a new one.


Version: unspecified
Severity: critical

Details

Reference
bz54059

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 2:11 AM
bzimport added a project: Cloud-VPS.
bzimport set Reference to bz54059.

Have you checked the access/error logs?

Nothing suspicious in the Apache log, only a warning about favicon not found.

I just ran the simplest possible test of this -- created an instance and let it sit for a few days. Right now it's still serving the "It works!" page correctly.

So... I could use further info about step 3, I think :) Can you suggest a minimal change that I can make on my instance that will provoke this failure?

(Meanwhile I will look at your instance, 8cf.)

Further testing shows that any restart of the apache service produces this failure. However... I'm not sure this is significant. In neither case (my test or yours) is there anything in /etc/apache2/sites-enabled... so I wouldn't really expect them to work in the first place.

I could add a trivial site config to the lamp puppet class, but in general I think it's reasonable for these instance types to require by-hand customization to get pages served.

If you're confident that Apache should work without a config or you think there's an obvious default config that instances should start with, let me know. Bear in mind, though, that any default site will be puppetized such that altering it after the fact will be difficult due to puppet refreshes.

Nicolas, any interest in this? If not I will close.

(In reply to Andrew Bogott from comment #4)

If you're confident that Apache should work without a config or you think
there's an obvious default config that instances should start with, let me
know. Bear in mind, though, that any default site will be puppetized such
that altering it after the fact will be difficult due to puppet refreshes.

Nicolas: ping?

Hi Andrew & Andre,

Thanks for your answers.
Yes, we still really need a server to host our small PHP script.

But if I understand correctly, I first need to setup something in /etc/apache2/sites-enabled otherwise it won't work correctly, right?

If it works after setting up some magic in /etc/apache2/sites-enabled then that's OK, I will try that, and this issue can be closed, thanks!

Cheers!
Nicolas

Citing https://wikitech.wikimedia.org/wiki/Help:Lamp_Instance :
" Once the instance is active you should be able to browse to its root web page by visiting <instancename>.pmtpa-proxy.wmflabs.org "

Is this documentation up-to-date? It seems to contradict "it's reasonable for these instance types to require by-hand customization to get pages served" written by Andrew above.

This may require a doc update, or it may have been resolved by this:

commit 203610a1dd12fe0bb1ca236dab54dc51e71bfb6b
Author: Ori Livneh <ori@wikimedia.org>
Date: Fri Jun 27 12:47:48 2014 -0700

role/labs: touch an empty conf file in sites-local

"Include sites-local/*" will fail if the directory is empty, and the
IncludeOptional directive is only available on Apache 2.3.6 and later.

Change-Id: Ic41f3d85ed9eb21d6676a85b5abc40cb8a804254

Needs checking, in any case

scott.leea wrote:

Tried this out and it keeps displaying the Apache starter page. If you remove /etc/apache2/sites-available/000-default.conf you have to make sure to create another conf file or it won't know where to point to -- I don't know if that was an issue.

Scott, can you try forcing a few puppet runs and a reboot, and verify that Apache survives both of the above? If so then this bug can be closed.

(To manually update puppet: $sudo puppet agent -tv)

scott.leea wrote:

Forced a Puppet run, restarted Apache, and even rebooted the instance. Survived.