I can't recreate this in a clean checkout of MediaWiki-Vagrant. The cache/apt/partial directory exists in the git repository. Your problem may have been a bug in the cache migration code put in place when the directory structure was changed.

Yes, this doesn't happen in a new vagrant installation, but breaks an upgrade attempt of an old one in a really ugly way (in the end I had to migrate everything manually to a new box).

Looking at this closer, the problem is caused by cleaning up the previous apt-cache/** directories before puppet runs on the VM.

We install a file at /etc/apt/apt.conf.d/20shared-cache to configure an alternate cache directory for apt in manifests/base.pp. This file will change to reference the new /vagrant/cache/apt directory as soon as puppet runs on the VM. The vagrant-vbguest plugin however is running before puppet has changed the file to point to the new location and attempting to use apt.

A possible fix would be to remove the code in Vagrantfile that deletes the old directories and instead add puppet code to remove the directories. This would leave the old directory structure intact until puppet has run and reconfigured apt to use the new cache location.

Change 147878 had a related patch set uploaded by BryanDavis:
Defer deleting legacy cache directories

https://gerrit.wikimedia.org/r/147878

After applying the patch, running vagrant reload fails with the same message. However, virtualbox-guest-dkms is now installed on the VM, and the mount fails with a different error message:

/sbin/mount.vboxsf: mounting failed with the error: No such file or directory

Ignore me. I got lost while context switching and ran the commands on the host. The error is still the same as described in comment 0.

This might be only because this box is already in a messed-up state; I can try upgrading another box if you think it will work now.

(In reply to Tisza Gergő from comment #6)

This might be only because this box is already in a messed-up state; I can
try upgrading another box if you think it will work now.

If you have the energy to try and recreate on a "clean" install I'd appreciate the data. If not, I'll try to find some time to setup a test starting from 3c6a6606b5217a2fb16cfde4db31d5a6c8e102c5 and then updating to the current HEAD of master plus the patch.

Not sure if this is really related to the change you mention. The most recent box I am experiencing this on is at I91d51f6deb0b12b795f705cf8895783a11fa8695. Looks like all the old boxes became unusable when I upgraded one of them. Any chance this is somehow related to one of the two vagrant plugins which the setup script installs globally?

The error message after running vagrant reload is slightly different though:

$ vagrant reload
[default] Attempting graceful shutdown of VM...
[default] Clearing any previously set forwarded ports...
[default] Clearing any previously set network interfaces...
[default] Preparing network interfaces based on configuration...
[default] Forwarding ports...
[default] -- 22 => 3335 (adapter 1)
[default] -- 80 => 9095 (adapter 1)
[default] Running 'pre-boot' VM customizations...
[default] Booting VM...
[default] Waiting for machine to boot. This may take a few minutes...
[default] Machine booted and ready!
GuestAdditions versions on your host (4.3.10) and guest (4.2.16) do not match.
Reading package lists...
Building dependency tree...
Reading state information...
linux-headers-3.2.0-56-generic is already the newest version.
The following packages will be upgraded:

dkms

1 upgraded, 0 newly installed, 0 to remove and 112 not upgraded.
E: Archives directory /vagrant/apt-cache/partial is missing. - Acquire (2: No such file or directory)
E: Archives directory /vagrant/apt-cache/partial is missing. - Acquire (2: No such file or directory)
[default] The guest additions on this VM do not match the installed version of
VirtualBox! In most cases this is fine, but in rare cases it can
prevent things such as shared folders from working properly. If you see
shared folder errors, please make sure the guest additions within the
virtual machine match the version of VirtualBox you have installed on
your host and reload your VM.

Guest Additions Version: 4.2.16
VirtualBox Version: 4.3
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

apt-get update

Stdout from the command:

Stderr from the command:

E: Archives directory /vagrant/apt-cache/partial is missing. - Acquire (2: No such file or directory)

(In reply to Tisza Gergő from comment #9)

The error message after running vagrant reload is slightly different though:

$ vagrant reload

[...snip...]

[default] Machine booted and ready!
GuestAdditions versions on your host (4.3.10) and guest (4.2.16) do not
match.

[...snip...]

The following packages will be upgraded:

dkms

1 upgraded, 0 newly installed, 0 to remove and 112 not upgraded.
E: Archives directory /vagrant/apt-cache/partial is missing. - Acquire (2:
No such file or directory)

The output you give seems to show that the vagrant-vbguest plugin is attempting to update your guest's GuestAdditions. This would happen before the puppet run.

The change I mentioned (3c6a6606b5217a2fb16cfde4db31d5a6c8e102c5) was the last commit before Ori added logic to Vagrantfile to move any existing /vagrant/apt-cache directory to /vagrant/cache/apt. My patch is an attempt to give existing hosts that are expecting the cache to be in the old location to at least see empty directories before puppet tells them to use the new location.

I think what has happened (and continues to happen) for you is that the move from /vagrant/apt-cache to /vagrant/cache/apt has happened in the working directory for Vagrant on your local machine but puppet has not run inside the VM to tell it to look for cached apt packages in the new location.

I think if you mkdir -p apt-cache/partial in the MediaWiki-Vagrant checkout directory on your host computer with my patch applied before you vagrant reload or vagrant up then the VM will start. Then you can vagrant provision to fix the apt cache config via puppet.

Change 147878 merged by jenkins-bot:
Defer deleting legacy cache directories

https://gerrit.wikimedia.org/r/147878

I have seen this happen for some other user during the London hackathon (I don't remember who). The fix was to make the old git cache happy by mkdir -p apt-cache/partial and then vagrant up. This is not an ongoing issue for new installs but a horrible inconvenience we inflicted on users from before the consolidation of the guest hosted cache directories.

(In reply to Bryan Davis from comment #12)

The fix was to make the old git cache happy by mkdir -p apt-cache/partial
and then vagrant up.

This does not work for me, presumably because folder sharing is not set up yet at the point where this error happens, so anything done to the folder of the vagrantbox on the host machine does not affect /vagrant on the guest machine.

In T69976#718954, @Tgr wrote:

(In reply to Bryan Davis from comment #12)

The fix was to make the old git cache happy by mkdir -p apt-cache/partial
and then vagrant up.

This does not work for me, presumably because folder sharing is not set up yet at the point where this error happens, so anything done to the folder of the vagrantbox on the host machine does not affect /vagrant on the guest machine.

Me too, same situation (new virtualbox and latest vagrant code but using an old vagrant box). The workaround was to vagrant ssh to the guest, then on the guest

cd /vagrant
sudo mkdir -p apt-cache/partial
exit

Then`vagrant reload` worked,

Installing Virtualbox Guest Additions 4.3.10 - guest version is 4.2.16
Verifying archive integrity... All good.
Uncompressing VirtualBox 4.3.10 Guest Additions for Linux............
...

However, I had further problems

...
An error occurred during installation of VirtualBox Guest Additions 4.3.10. Some functionality may not work as intended.
...
mount -t vboxsf -o uid=`id -u root`,gid=`getent group vagrant | cut -d: -f3` tmp_vagrant-puppet-4_manifests /tmp/vagrant-puppet-4/manifests

I thought vagrant would use NFS, not vboxsf, hmm.

see P187

In T69976#950097, @Spage wrote:

Me too, same situation (new virtualbox and latest vagrant code but using an old vagrant box).

Don't do that. :) The change to the cache structure was made ~6 months ago. If you have a VM that is that old without updates I would suggest deleting it and starting over rather than trying to fix it. I'd actually encourage all MediaWiki-Vagrant users to recreate their VMs from scratch (vagrant destroy -f; vagrant up) at least once a month. If you are using the roles and local puppet customizations all you should lose is accumulated state in the wiki database. Virtual machines are cattle, not pets.

Just for the record, I am getting this on all my boxes after a host system upgrade (nothing in MW-Vagrant changed; must be caused by some change in the Vagrant upstream). sudo mkdir -p /vagrant/cache/apt/partial worked fine in this case.

That directory exists in the rMWVA mediawiki/vagrant repo -- https://phabricator.wikimedia.org/diffusion/MWVA/browse/master/cache/apt/partial/

I wonder if something is trying to get at that directory before it has actually been mounted into the VM? @Tgr can you get a PUPPET_DEBUG=1 VAGRANT_LOG=debug vagrant ... log trace that reproduces the problem?

The directory definitely existed on those boxes before. With the system update, virtualbox got updated and vagrant tried to update the hosted virtualbox plugin accordingly; the error occurred in the process of that. Logfile:

In T69976#1871652, @Tgr wrote:

The directory definitely existed on those boxes before. With the system update, virtualbox got updated and vagrant tried to update the hosted virtualbox plugin accordingly; the error occurred in the process of that. Logfile:

vagrant.log2 KBDownload

I'm not sure it is worth the effort investigating though, it only happens once per box and is easily fixed by creating the directory.

Right. I've seen that before. Here's how it happens:

1. VM is provisioned using our Puppet rules.
2. Puppet changes the apt config to look for cache in /vagrant/cache/apt rather than the default /var/cache/apt location.
3. VM is restarted.
4. Vagrant's plugin for updating VirtualBox runs before any shares are mounted in the VM so it can ensure that vboxsf mounts actually work.
5. The plugin runs apt.
6. apt barfs because it is looking for a directory that is not mounted.

I'm not sure there is much we can do about this one. If we were using a custom base image I guess we could ensure that a /vagrant/apt/partial directory existed in the base image. That tree would normally be hidden by the /vagrant mount but it would make the directory exist when the share was not mounted.

A similar but inverted approach would be to add a separate mount of the mw-vagrant cache/apt directory over the top of the VM's existing /var/cache/apt directory rather than changing Apt's cache location in the first place.

tl;dr instructions to fix if you run into this. ssh into the vagrant instance, run sudo mkdir -p /vagrant/cache/apt/partial, and then run vagrant reload.

If you get sick and tired of this causing you problems, you can globally disable the vbguest plugin by creating a ~/.vagrant/Vagrantfile with this content:

Vagrant.configure("2") do |config|
  # Disable vbguest updates
  config.vbguest.auto_update = false
end

We stopped installing vbguest in rMWVAc8ca59051837: Drop the vbguest plugin and sideload the mediawiki-vagrant plugin for T151928: Vagrant 1.9.0 unable to install local mediawiki-vagrant plugin. Lets close the loop and disable the plugin for anyone who may still have it installed by adding this to our Vagrantfile:

if Vagrant.has_plugin?('vagrant-vbguest')
    config.vbguest.auto_update = false
end

Change 343103 merged by jenkins-bot:
[mediawiki/vagrant] Disable vbguest plugin auto updates

https://gerrit.wikimedia.org/r/343103