I can't reproduce either claim:

Please provide a minimal example.

(Just to clear up some confusion: jsub doesn't actually execute the script; it just submits it to the job grid aka SGE/OGS.)

Partially reproduced it.

Using the first script:

local-legobot@tools-login:~/$ jsub ./test.py && while ! job test > /dev/null; do sleep 1; done && cat test.{out,err}
Your job 1933479 ("test") has been submitted
ANSI_X3.4-1968

Second script:

local-legobot@tools-login:~/$ jsub ./test.py && while ! job test > /dev/null; do sleep 1; done && cat test.{out,err}
Your job 1933488 ("test") has been submitted
Talk:Gülen movement

I think this should be a more generic request to make sure the environment on the exec hosts is the same as what someone has when testing in the interactive shell.

In any case, the problem is the following:

valhallasw@tools-login:~$ cat > test.sh
#!/bin/bash
locale
valhallasw@tools-login:~$ chmod +x test.sh
valhallasw@tools-login:~$ ./test.sh
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

valhallasw@tools-login:~$ jsub ./test.sh
valhallasw@tools-login:~$ cat test.out
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

Setting LANG="en_US.UTF-8" (or any other UTF-8 locale) should solve this issue.

Oh, and to reproduce the issues: compare

LANG=C python -c "print u'\xe4'"

to

LANG=en_US.UTF-8 python -c "print u'\xe4'"

(In reply to comment #3)

[...]
Setting LANG="en_US.UTF-8" (or any other UTF-8 locale) should solve this
issue.

It apparently does, because I have:

in ~/.profile, and for me:

But my test account shows "LANG=en_US.UTF-8" interactively, but "jsub locale" gives "LANG=", even after "export LANG". The same occurs if I set the locale to non-"en_US.UTF8" before jsub with "export LANG=de_DE.UTF-8".

My assumption (and fear :-)) is that SGE sources ~/.profile before job execution, which means that there will be a *lot* of confusion on where to configure locales and how they are evaluated.

I don't want to go down that road if it can be avoided. Is it possible to explicitely set the locale in Python? Otherwise we could change jsub so that users can use qsub's "-v" option to set the locale in the environment:

However that does not seem to solve the Python error:

And for the dbreps tool I indeed had to use:

But that is Python 2.7.3 (cf. http://stackoverflow.com/questions/1473577/writing-unicode-strings-via-sys-stdout-in-python, http://pythonhosted.org/kitchen/unicode-frustrations.html, https://wiki.python.org/moin/PrintFails).

I don't know what the situation is for Python 3+.

Ahh, there's another catch.

valhallasw@tools-login:~$ python ./test.py | tee
Traceback (most recent call last):

File "./test.py", line 2, in <module>
  print u"\xe4"

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 0: ordinal not in range(128)

valhallasw@tools-login:~$ PYTHONIOENCODING=utf-8 python ./test.py | tee
ä

but that's painful to say the least.

Python 3 has no issues -- it will just use utf-8 if the LANG says so:

(test.py: print("\xe4") -- remember, str in py3 is unicode in py2)

valhallasw@tools-login:~$ python3 ./test.py | tee
ä

(In reply to comment #5)

[...]
My assumption (and fear :-)) is that SGE sources ~/.profile before job
execution, which means that there will be a *lot* of confusion on where to
configure locales and how they are evaluated.

I don't want to go down that road if it can be avoided. Is it possible to
explicitely set the locale in Python? Otherwise we could change jsub so that
users can use qsub's "-v" option to set the locale in the environment:
[...]

No, we can't as a test on my account with setting LANG to de_DE.UTF-8 in ~/.profile shows:

In bug #48811 we encountered a similar problem: We need "-b y" for binary programs, but "-b y" adds a (login) shell to the call stack:

There is a configuration variable login_shells in sge_conf(5), but I'll need to whip up Toolsbeta in shape to evaluate options.

For the time being I suggest wrapper scripts.

Part of the difficulty is that there is a combinatorial explosion of starting environments depending on more factors than you can shake a stick at (given the gridengine's propensity to try to "guess" at what you're trying to do, and to (silently) add a shell anytime it thinks you need to evaluate shell arguments).

The best rule of thumb is "if you need something specific in your environment, set it explicitly". I would recommend that one /always/ uses a shell wrapper that sets the environment; a simple generic one might be:

#! /bin/bash

export STUFF_I_NEED="foobar"
export PATH="/all:/the/places"
exec "$@"

This will set the STUFF_I_NEED then exec to the program given as argument without needlessly keeping a subshell around. That same script can then be reliably used to launch everything in a reliable way.

I *could* make a globally available script that relies on sourcing, say, .bashrc:

#! /bin/bash

. ~/.bashrc
exec "$@"

Which everyone could then use. I could even have it invoked implicitly by jsub at need.

Is this still a relevant issue?

Left without comment for >six months; reopen if the issue is still relevant.

T121505: Inconsistent locale settings of different grid compute nodes. is somewhat related.

I think the best solution would be to set both the execution environments in the grid and the user environment on bastion to default to UTF8 which is a modern de-facto standard.

In T60784#3439185, @Kotz wrote:

I think the best solution would be to set both the execution environments in the grid and the user environment on bastion to default to UTF8 which is a modern de-facto standard.

I would generally agree with this. If we set the default locale to C.UTF-8 things should mostly work as expected. I wouldd actually be in favor of making C.UTF-8 the default locale across all Foundation servers. I fixed a goofy bug in Striker recently by explicitly setting this locale in it's uwsgi configuration.

I think the best solution would be to set both the execution environments in the grid and the user environment on bastion to default to UTF8 which is a modern de-facto standard.

Having just spent a load of time trying to fix encoding errors when running through the grid, I agree.

The best rule of thumb is "if you need something specific in your environment, set it explicitly".

For what it's worth, this was my fix. Added

export LC_ALL="en_US.UTF-8"

to the top of my jsub script and everything went well again.

Just spent an hour and a half debugging what could be making @Alchimista's bot not work through grid.
I agree with the folk calling for C.UTF-8 as the default locale across the board.

Initial tests looks good:

tools.zhuyifei1999-test@tools-bastion-02:~$ cat unitest.sh 
#! /bin/bash
exec 2>&1
set -x

python2 -c 'print(u"\u1234".encode("utf-8"))'
python2 -c 'print(u"\u1234")'
python3 -c 'print(u"\u1234".encode("utf-8"))'
python3 -c 'print(u"\u1234")'
tools.zhuyifei1999-test@tools-bastion-02:~$ LC_ALL=C bash unitest.sh 
+ python2 -c 'print(u"\u1234".encode("utf-8"))'
ሴ
+ python2 -c 'print(u"\u1234")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u1234' in position 0: ordinal not in range(128)
+ python3 -c 'print(u"\u1234".encode("utf-8"))'
b'\xe1\x88\xb4'
+ python3 -c 'print(u"\u1234")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\u1234' in position 0: ordinal not in range(128)
tools.zhuyifei1999-test@tools-bastion-02:~$ LC_ALL=C.UTF-8 bash unitest.sh 
+ python2 -c 'print(u"\u1234".encode("utf-8"))'
ሴ
+ python2 -c 'print(u"\u1234")'
ሴ
+ python3 -c 'print(u"\u1234".encode("utf-8"))'
b'\xe1\x88\xb4'
+ python3 -c 'print(u"\u1234")'
ሴ

But on grid python2 seems to ignore the locale:

tools.zhuyifei1999-test@tools-bastion-02:~$ rm LOCALE_C.* &> /dev/null; jsub -N LOCALE_C bash unitest.sh; sleep 5; cat LOCALE_C.outYour job 357341 ("LOCALE_C") has been submitted
+ python2 -c 'print(u"\u1234".encode("utf-8"))'
ሴ
+ python2 -c 'print(u"\u1234")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u1234' in position 0: ordinal not in range(128)
+ python3 -c 'print(u"\u1234".encode("utf-8"))'
b'\xe1\x88\xb4'
+ python3 -c 'print(u"\u1234")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\u1234' in position 0: ordinal not in range(128)
tools.zhuyifei1999-test@tools-bastion-02:~$ rm LOCALE_C.UTF-8.* &> /dev/null; jsub -N LOCALE_C.UTF-8 -v LC_ALL=C.UTF-8 bash unitest.sh; sleep 5; cat LOCALE_C.UTF-8.out
Your job 357343 ("LOCALE_C.UTF-8") has been submitted
+ python2 -c 'print(u"\u1234".encode("utf-8"))'
ሴ
+ python2 -c 'print(u"\u1234")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u1234' in position 0: ordinal not in range(128)
+ python3 -c 'print(u"\u1234".encode("utf-8"))'
b'\xe1\x88\xb4'
+ python3 -c 'print(u"\u1234")'
ሴ

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!