Now, Uncaught TypeError: Cannot read property 'status' of null is appearing while trying to open VE in betalabs

jhall wrote:

I've also been seeing this behavior in beta labs today, although it's been difficult to consistently reproduce. I've seen it much more frequently with the automated browser tests than I have with manual testing.

I noticed this today at
http://en.m.wikipedia.beta.wmflabs.org/wiki/0.07511468210381089

Whenever I load VisualEditor, it loads the allergy article instead.

Does beta.wmflabs.org have latest parsoid master? Related is bug 60372 which was fixed in https://gerrit.wikimedia.org/r/#/c/110233/

It seems to be quite easy to consistently reproduce. Just click 'random article' and then load the VisualEditor. I get the contents of a different article 100% of the time. The really weird thing is that the articles that are being loaded don't seem to even exist on Beta Labs.

This is unlikely to be related to bug 60372, as a) the oldid is sent, and b) it only happened sometimes under high request loads.

This does not happen in production (running Parsoid master right now). I'm suspecting a configuration issue in betalabs.

Note that this could also be VE sending the wrong oldid, as that will always take precedence over the page name.

A bit more investigation shows that VE is sending the right oldid.

I think this is caused by Parsoid not finding its config file in the deploy repo. In the production upstart config we are now passing the explicit path to a config file to parsoid, which is checked into conf/wmf/localsettings.js in the deploy repo. Beta labs might still rely on localsettings.js being in src/api, which will no longer work.

I think we'll need to make the config path configurable in the upstart file, and set a different path (ideally conf/wmf-beta or the like) for betalabs.

On deployment-parsoid2 in /var/log/parsoid/parsoid.log

{ [Error: Cannot find module '/srv/deployment/parsoid/conf/wmf/localsettings.js'] code: 'MODULE_NOT_FOUND' }

There is no conf directory but there is a file /srv/deployment/parsoid/localsettings.js which is described in puppet:

manifests/role/parsoid.pp: file { '/srv/deployment/parsoid/localsettings.js':

I introduced it on Jan 2 with https://gerrit.wikimedia.org/r/#/c/104938/

The upstart configuration I wrote ( /etc/init/parsoid.conf ) was pointing to it but now points to ../conf/wmf/localsettings.js.

So I guess the easiest path is to use an env variable such as PARSOID_CONF, set it in /etc/default/parsoid and make it vary in the puppet configuration.

Another symptom of this is if you add or edit a post to a Flow board on betalabs (e.g. http://en.wikipedia.beta.wmflabs.org/wiki/Talk:Flow_QA ) and transclude a page with {{:Article name }}, it pulls in the page from enwiki.

https://gerrit.wikimedia.org/r/#/c/110666/ creates a Debian package and makes the config file path configurable in /etc/defaults/mediawiki-parsoid. We might still want to fix this in puppet temporarily until we move to deb-based installs.

We need this on beta labs at the earliest opportunity. I raised the importance.

Change 111350 had a related patch set uploaded by GWicke:
Bug 60694: Make the config file path configurable

https://gerrit.wikimedia.org/r/111350

Change 111350 merged by Alexandros Kosiaris:
Bug 60694: Make the config file path configurable

https://gerrit.wikimedia.org/r/111350

The puppet agent didn't seem to be running on the parsoid host in beta labs, and after starting that Parsoid was not restarted to reflect the the changes. Fixed that manually, and verified fixed at http://en.wikipedia.beta.wmflabs.org/wiki/0.6884344289426402?veaction=edit.

Verified the fix on Betalabs.