[pacman-dev] [PATCH v2 5/8] Avoid problematic use of Python's StringIO.

[Allan McRae]

On 13/10/13 22:15, Andrew Gregory wrote:
> On 10/13/13 at 05:11pm, Allan McRae wrote:
>> On 13/10/13 13:14, Martin Panter wrote:
>>> On 13 October 2013 13:14, Jeremy Heiner <scalaprotractor at gmail.com> wrote:
>>>> On Sat, Oct 12, 2013 at 9:16 PM, Allan McRae <allan at archlinux.org> wrote:
>>>>> On 13/10/13 02:44, Jeremy Heiner wrote:
>>>>>> There was one hiccup in encoding in test sync600: the unicode strings
>>>>>> which work in other tests for some reason cause problems here. This
>>>>>> hiccup is only in 2.7, so a try block is used to fall back to use the
>>>>>> unencoded string on that runtime only.
>>>>>
>>>>> Using "for some reason" in the description is a clear flag for me not to
>>>>> apply the patch.  This tends to result in an underlying issue being
>>>>> missed due to lack of understanding.
>>>>
>>>> Hi, Allan!
>>>> Yeah, I was not happy at all about having to put that exception
>>>> handler in. I tried a crazy number of other workarounds, and in doing
>>>> so I got quite a good understanding. But describing it adequately will
>>>> take a lot of typing. Like several pages. It is absolutely right of
>>>> you to ask, and I'm happy and able to provide that, but I doubt I can
>>>> get to it before tomorrow night. The short answer is that Unicode
>>>> support in Python 2 is fundamentally broken. Fixing that is why Python
>>>> 3 got a major version bump and they had to give up on backwards
>>>> compatibility. Sorry I don't have more time right now to go into all
>>>> the gory details... and I hope the other readers won't be too bored by
>>>> my post when I do.
>>>
>>> Looking at the “sync600” test you mentioned, it has a bunch of file
>>> names with non-ASCII characters. Perhaps you might have better luck
>>> putting a “u” prefix in front of the strings, or using “from
>>> __future__ import unicode_literals”. Be aware that the u"" syntax was
>>> only added back in Python 3.3.
>>>
>>
>> I get the feeling that this should be dealt with in sync600 rather than
>> what this patch does.
>>
>> Comments?
>>
>> Allan
> 
> This is the reason I never submitted my own patchset.  I don't think
> there is a good way to do this.  There doesn't appear to be a way to
> use either a unicode or bytes literal in the test that is compatible
> with both python2 and python3 unless we also drop support for versions
> of python3 < 3.3, which are still supported.  The only solution
> I found was to test for the python version in the test suite.  But,
> given that our build process is tied to python2 anyway until asciidoc
> supports python3 it didn't seem worth it to make the transition until
> we could drop python2 entirely.
> 

Just a note.  Our build process for git is tied to asciidoc, but not for
release tarballs.

I am going to merge all these patches apart from this one and the final
patch.  If a consensus can be found on how to deal with this issue, I
will pull it in - I am not familiar enough with python issues to make
the decision myself.

Allan