Page MenuHomePhabricator

Reports of secret key generation "hanging" on Windows with PHP 5.3.x
Closed, ResolvedPublic

Description

I've seen a couple reports of the install hanging on the "generating secret key" step. All these reports seem to be windows. See https://www.mediawiki.org/wiki/Thread:Project:Support_desk/Installation_of_V_1.18.2_stopps_while_creating_"secret_key"

Things that need to be determined:

*Is it really hanging, or does it just take a long time to generate "real" randomness. How "long" is a long time
*How common an occurrence is this. Is it just some people, or is it all windows users and we just don't notice because very few of us use windows.

If its really hanging, we should figure out why. If its just taking a really long time, we should figure out if that time is acceptable amount (Obviously, If it takes an hour to generate the secret key, that's not acceptable )

If its going to take such a long time, we should really have some sort of warning to users.


Version: 1.20.x
Severity: normal
OS: Windows Server 2003
URL: https://www.mediawiki.org/wiki/Thread:Project:Support_desk/Installation_of_V_1.18.2_stopps_while_creating_%22secret_key%22
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=44142
https://bugzilla.wikimedia.org/show_bug.cgi?id=55364

Details

Reference
bz35894

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 12:13 AM
bzimport set Reference to bz35894.
bzimport added a subscriber: Unknown Object (MLST).

We should also have them turn on debug output so we can tell what random generator is being used. ie: Is there a mcrypt bug we don't know about, is the openssl bug still applicable to other versions, or is our clock drift implementation taking longer to run on Windows?

Looking at CryptRand, it shouldn't be too hard to make it run standalone on Windows (saving an install of MW in IIS or whatever).. Might be able to ease testing

I wonder if this is something we can unit test?

(In reply to comment #3)

Looking at CryptRand, it shouldn't be too hard to make it run standalone on
Windows (saving an install of MW in IIS or whatever).. Might be able to ease
testing

I wonder if this is something we can unit test?

Actually. I needed random tokens inside of Gareth and couldn't find an acceptable source built into Symfony. So I already ported a version of CryptRand without MediaWiki specific code:
https://github.com/dantman/gareth/blob/master/src/Gareth/GarethWebBundle/Libs/CryptRand.php

Just strip out the 'namespace ...;` and you should have yourself something that works on it's own. Throw an echo inside the debug() method and you should have debug output too.

Doing the simplest of possible tests on PHP 5.3.10:

C:\Users\Reedy\Desktop>e:\php\php.exe CryptRand.php
CryptRand::realGenerate: Generating cryptographic random bytes
CryptRand::realGenerate: mcrypt_create_iv generated 2048 bytes of randomness.
CryptRand::realGenerate: 0 bytes of randomness leftover in the buffer.
4c531f0706c381e2a35a09cde42a59657d4f095fd3b84d921fc899cd96a38c8672c99a5d2b97469af7b248c2f10478bcc13d
32b46ca548ccdcea1daf8ad5035ae9231fbec2bacd389660126563252e99e48af34bae0b26a1aed5305f1ef0df037696cfab
c0136813b114e3e9269b1c3db745ea9d913dc6d9cba92ce862e7665ae87c5d8899559a4a5379bddd4f26edd7ee399a54b468
cd292a654c7fc0b0f6e51f302ad8162f6282c2a25ac6fe3d84ad9234c9d400c69b273d88e9f3747a47b73e02e485789c3968
0409a4763408ac8a8f9ae985398b74417ad01f8beda762ca83eb49d3edb6c65263fe359b89ff61f0ddab5c91fa6a7e396aed
553f0c33756193dae4cfca5bf5149fad3578dafba85cc8031852b75cea97b26793f5818371c61beea90798105e7c227a9859
6210a8b6f28144efe0a5acfc689115b5091f738aa83403c43df0cce1dca1dbc008d3bd2cefb94132a40801657ce4e3f70841
a4028debf3c78c63586820fce6dadf33d1540f7ffbdd0661bc7a5903e7d0761593390587e83f9b7f6d6e581e1cf210c9797f
f4a046cd15db127b96887d4f44254eb0fbd73bee88b5021ea7cf1aee4337ad83a162015f73033fa3ddbdd76ea75760b4d208
94f8db44e05353f0615f2db9304863e4064057392086b778f8ab636f639b61a46d46879cb14e476a6b496874e7fc297c2186
ca76a1d64b88cf77f8345f33ef8deff97bdbd931553acf542a2b1a6aedd99307cd00572be21bceb8577e1fc6eb488b1bd2ad
dc332a9f655cfb2836d71c20bcb2f57e4fa115ebefbca4a09c41e5cfdd473d97ba0561c306a00ed586e43426b71818f49740
22d223601ea5b8fc6d0c3a97f3a34bc8b3b0133808456a8dd548b44d4153e5e48f33f0152bd1edb575b51d1f66d40d568c51
763686a0ac2a078c45f296f0a39f6b1a77b0c0e3a6a4c38596255a88708c27923cdc05b142e7e1f9dd75c23ca855ee588c95
0afbffca1b41be923614fe3e30c3d40ec7c902b9416d8b8a875b510f09cffc22463ad3f0a99953d2733f92a44c37393c70d7
93102e7d4a7aefa04af784dc900ee5653acb787b953810e4dbc58e45bcbd4e6d8fb9784800ac92e14a32965ffc0e5df473cb
c0d4642fa19f0323fe35e4f079611f12539fb307402172b232b076e721853a037b0e45e122a238766ad8ba2bba96fab5cf03
34e5a01b2be6c4107639e0f1e50c2ef5367b0bc58f523b7f0ce4a90430de5aefdfef9d795c0679c92309912da4d0cdf15283
4cb7e9f7f704b3066c659257a0139b4dadf3f1bc568105f2085aa5162a5183eb3d6239e74c7dc21687595962b66b900f97a5
c5a799e13ae0c2a58552fd0c901352bf851c4c8318f19efcd8b49cd8a1d5000fcf3d3f958454f9728e208c9fd034d2181364
4a20762d1a8bb87d0d13dc03883081efa7c9ce588efd16655fc1cebfcc3cbcf2d91bc94bf04d1c1c958e3403c21dbb6f460c
1a3a1f8e914fd439e5bef0075d9b2f5f61fe399ada2ef7cda42d73c6d3dca335bcee4d9190f75b1b2b86e74423c4b7768a8e
797c568989353b15649c00bf4a861c20024fa84ac5d1d5efc7e14719d76c4e281e1e93cc536ef554481ac9cae4396fab4c94
4ddca6713fcf9b17d9507ec688d0d6651f781e9f77aa27ac2e945e5f39b92da28f56913e6361c13cd1d2a028157049318f2e
bb4490387c60595c8feb19c2e77c22726ec09e945229d6cceba4db1ae16bd651bc777b58a9d55d4ac5e488d42eea31ea731b
ce99e03abffe8aec5056d20c507040a82c01e07fab350632a4853bcc3a78bcb9ff3fc3f136ef65174658359f81b91b4ef56e
44bfc64af98999f03934af420f0205d328cfeb7172ec02ffe24c508a629e13bf4758ae6fe36f3fc0d95aa29ad29da7d42eb3
8832e4943427414b3d494e4d3045dcedf02f1ce25361cd9dc8772a5202fce0dd456ab521a161209898a3be1974f5092d6e36
907db813ec7f46eedf0ac4e83357a9955b8a3ec0b90568ed23a75906f49170e6ce32c08332fa53b5a892d0f27693ce6e9eb1
171bee3e5affce8cd8c9032af41398ce1e5cd69d1fecd0f513a1b3e171dba3e384544e191d3efa1b29ec681a8eef0bc71917
350feb93bb7b9423b66dda9e2426dacecd12cdfdfe22375b26deb870945f1750a2c45299c26d4a45165c276c9b00d58ba4b3
86bea197872d94713b972731cc81f1eff5d8d3646eb9e5de0d7c19beb9077dbdfd8288a3e6902047bb73b6ef20d62911eb28
b4e0b24c1b2927dc2715d4b12561d0c556a5389e60f585406febfd3d8ef03f9502df4cf311c81c251d4d8f1fda4a5777057b
7d2ff45f577010df0dda4e068776f7a06ec29aed4a823fd7f3fbc15bacf86cc76b74da50a2ed933d53960f7a1577d669477a
10cc23dab3e338c4491cbd5c5a92f4bafd59ec6611c08dfcebd4c4d95537b2f212e1893d9210a30757381808db6d4ca6c10a
5576ca921b7f020d8bcc35dd6e9b8aef4e0da43c100b0b579fcc4eb153b9b1f3005b371b1c00b192188b9a1f1e0c46dc9627
1b208fa591005d45692617f1de92f46b403761da9f967c1818f3ed2dcab9dfb5b27b9bf3366cfc417c707b33d18b6deb6f9d
0e58b5eaa124256687ecf8d3690ea8764110f0b323606f2056fccb29c66ee973414a65473bc1fed69c6ef125006181fda2a0
fe514d2521d849f2384cb5dbb54a2f1a68fd05a9e479d8ac5d045645458ad93d9ce39b2d33c8918f540497ebb8de312fa5de
f447fc2d33c98999ea97c47a918a751aa32b1ddd907ba6d72bcd8a2bcca1bb6e0ca77abbfd584242354235d651cd2f9db4ba
2651a6dd608c2fde901f5687ecaf9ab73f2e589973fbbb6a6d2e8a0484aa484e35344c726709d7768abb8f2d250558f2
C:\Users\Reedy\Desktop>

echo CryptRand::generateHex( 2147483647 );

C:\Users\Reedy\Desktop>e:\php\php.exe CryptRand.php
CryptRand::realGenerate: Generating cryptographic random bytes
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 1073741825 byt
es) in C:\Users\Reedy\Desktop\CryptRand.php on line 302

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 1073741825 bytes) i
n C:\Users\Reedy\Desktop\CryptRand.php on line 302

Using silly large numbers

C:\Users\Reedy\Desktop>e:\php\php.exe CryptRand.php
CryptRand::realGenerate: Generating cryptographic random bytes
PHP Warning: mcrypt_create_iv(): Cannot create an IV with a size of less than 1 or greater than 214
7483647 in C:\Users\Reedy\Desktop\CryptRand.php on line 302

Warning: mcrypt_create_iv(): Cannot create an IV with a size of less than 1 or greater than 21474836
47 in C:\Users\Reedy\Desktop\CryptRand.php on line 302
CryptRand::realGenerate: mcrypt_create_iv returned false.
CryptRand::realGenerate: /dev/urandom could not be opened.
CryptRand::realGenerate: Falling back to using a pseudo random state to generate randomness.
CryptRand::hashAlgo: Using the whirlpool hash algorithm.
CryptRand::driftHash: Clock drift calculation (time-taken=32.00101852417ms, iterations=36539, time-p
er-iteration=0.87580444249076us)
CryptRand::driftHash: Clock drift calculation (time-taken=32.000780105591ms, iterations=43135, time-
per-iteration=0.7418750459161us)

echo CryptRand::generateHex( 64 );
echo CryptRand::generateHex( 16 );

C:\Users\Reedy\Desktop>E:\php\php.exe CryptRand.php
CryptRand::realGenerate: Generating cryptographic random bytes
CryptRand::realGenerate: mcrypt_create_iv generated 32 bytes of randomness.
CryptRand::realGenerate: 0 bytes of randomness leftover in the buffer.
a2be3683f35deb171184b7daba33f324ecca1e50fe61e2dbeed02104a16aeaf7CryptRand::realGenerate: Generating
cryptographic random bytes
CryptRand::realGenerate: mcrypt_create_iv generated 8 bytes of randomness.
CryptRand::realGenerate: 0

https://bugs.php.net/bug.php?id=55169

"When using the 5.4 snaps calling mcrypt_create_iv with either MCRYPT_DEV_RANDOM or MCRYPT_DEV_URANDOM as source always fails to gather sufficient random data regardless of the specified size."

All seemingly on windows

https://bugs.php.net/bug.php?id=52523

Test script:

The following code should loop forever, but does not do so on Windows (XP and Windows 7):

php -r "$i=0; while( mcrypt_create_iv(16) !== false ) {echo ($i++).'.';};"

Expected result:

Infinite loop.

Actual result:

0.1.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.
Fatal error: mcrypt_create_iv(): Could not gather sufficient random data in Command line code on line 1

I'm marking the milestone field as 1.20. At the very least, the potential fatal error on windows part should be fixed before the 1.20 tarball release.

(In reply to comment #7)

https://bugs.php.net/bug.php?id=55169

"When using the 5.4 snaps calling mcrypt_create_iv with either
MCRYPT_DEV_RANDOM or MCRYPT_DEV_URANDOM as source always fails to gather
sufficient random data regardless of the specified size."

People report problems with 5.3.x

https://bugs.php.net/bug.php?id=52523

Test script:

The following code should loop forever, but does not do so on Windows (XP and
Windows 7):

php -r "$i=0; while( mcrypt_create_iv(16) !== false ) {echo ($i++).'.';};"

MWCryptRand checks for this.

We can make the installer call itself via HTTP(S) to generate the keys - this will allow it to time out/handle fatals. Does this hack worth implementing?

(In reply to comment #9)

MWCryptRand checks for this.

We can make the installer call itself via HTTP(S) to generate the keys - this
will allow it to time out/handle fatals. Does this hack worth implementing?

I think so.

Max, did you implement this? It seems like this would be a good one to have fixed for the 1.20 tarball.

Has anybody seen the problem with

  • a recent MediaWiki version (1.20.x or 1.19.x)?
  • PHP 5.4.x?

I checked installation of 1.20.1 today and the problem still exist. It still has php 5.3.3 on. If I find the time I will update to php 5.4.x these days and check again.

Well I tried, but it does not seem so easy to migrate to php5.4. I replaced all php 5.3 files by those of 5.4, restarted IIS and all wikis (1.16-18) stopped working immediately due to an error :-( . Unfortunately I did not note that error message - something relating mysql it was. I do not have a testserver so I had to act quick :-)

(In reply to comment #15)

Well I tried, but it does not seem so easy to migrate to php5.4.

I can at least try to reproduce your php set up. How are you getting and installing PHP?

I'll see if I can get IIS as well. What sort of configuration are you using for it?

Unfortunately I did not note that error message - something relating mysql it was.

Is it possible that it was the typical silly mistake of copying the php.ini of the new php version but not uncommenting ;extension=mysql.dll ? (you will also need to enable some more extensions, plus setting extension_dir)

I did nothing to the configuration of php in this case. I realy just copied it as I did times before. With some more patience I'll try again. PHP 5.4 was from http://windows.php.net/download . As I see now at the sidebar I need to install some dll from MS also.
What sort of configuration? In which way? It's cgi-fcgi.

It's not clear at all what you did.

Was it using an existing php.ini ? The default values (no php.ini)?

phpinfo() would give that information, as would probably have done the error message, too.

This last issue looks like a php error, though. Nothing on MediaWiki side.

MrBassam wrote:

I managed to pass this bug by stop the call to mcrypt_create_iv in CryptRand.php
Changae the following line , line no:306

if ( function_exists( 'mcrypt_create_iv' )) {

To

if ( function_exists( 'mcrypt_create_iv' ) && 1==2 ) {

Try to run install proces, It will complate, And you may have to change some settings manually

This way you are avoiding the php bug blocking the generation of random data. But you are also not getting a good randomness.

(it would be good if you stated the versions where you were hit by this bug, in order to answer comment 12)

Hi to all. Today I took some time to update to php 5.4.9 successfully (I used the old php.ini) and now the installation process of MW 1.20.2 completes with no error wich makes me very happy. Thanks to all....

  • Bug 55364 has been marked as a duplicate of this bug. ***
  • Bug 44142 has been marked as a duplicate of this bug. ***

Oops! Sorry for the spam! I just wanted to relate upsream PHP bugs instead of local ones.

But now Bugzilla doesn't let me add php bugs, so I'm going to leave them here:

(In reply to comment #20)

I managed to pass this bug by stop the call to mcrypt_create_iv in
CryptRand.php
Changae the following line , line no:306

if ( function_exists( 'mcrypt_create_iv' )) {

To

if ( function_exists( 'mcrypt_create_iv' ) && 1==2 ) {

Try to run install proces, It will complate, And you may have to change some
settings manually

+1 This is a reasonable workaround till this bug is finished. MW will safely fall back to our clock drift based cryptographic random number generator which is still relatively ok.

I'm just mentioning it because it looks like in duplicate bug reports people are picking really bad ways of working around this bug. MCRYPT_DEV_URANDOM should NEVER EVER EVER be changed to MCRYPT_RAND.

Doing that bypasses every cryptographic fallback we have available (including our clock drift code which is still more secure than MCRYPT_RAND) replacing MWCryptRand with an insecure PRNG and directly opening you up to every attack MWCryptRand was created to protect your wiki from.

Quite frankly MCRYPT_RAND should have NEVER been included in PHP. If you look up the PHP source code you'll find that MCRYPT_RAND is a plain wrapper right around PHP's rand().

  • Bug 65363 has been marked as a duplicate of this bug. ***
MaxSem claimed this task.

We use random_bytes() for CSPRNG now.