Handling s390 libc ABI change in Debian

Post by Aurelien Jarno
glibc 2.19 has changed the libc ABI on s390, more specifically the
setjmp/longjmp functions [1] [2]. Symbol versioning is used to handle
some cases, but it doesn't work when a jmp_buf variable is embedded
into a structure, as it changes the size of the structure. The result
is that mixing programs or libraries built with 2.18 with ones built
with 2.19 do not work anymore, usually they end up with a segmentation
fault. Some persons from this list have experienced that with perl.

That is not true. This is an over generalization of the problem. You
can use libraries built with 2.18 and 2.19 and they work just fine.

The extent of the problem in correct language is listed here:
https://sourceware.org/glibc/wiki/Release/2.19#Packaging_Changes

Post by Aurelien Jarno
We first thought it was limited to a few packages (even if all perl is
already more than that), but as time goes more and more issues are
found. libpng and gauche are also affected, the issue with mono is
also likely due to this ABI change.

That is new information, and it is important for distributions to
relay this information back upstream where the decision for a SO bump
can be made.

Post by Aurelien Jarno
According to upstream [3], the problem is that Debian doesn't do a mass
rebuild, which is the strategy chosen by Red Hat to handle^Wworkaround
this issue. This means some programs might segfault during the upgrade,
or on partially upgraded systems.

I apologize if you took what I wrote to mean that. I did not mean it
was Debian's problem, but rather that Debian suffered the most because
they don't do rebuilds. The two are orthogonal. You face a situation
that is unique to the framework used to build the distribution.

Please engage upstream to champion a SO name bump for libc for

Post by Aurelien Jarno
1) Ignore the issue and just rebuild (binNMU) the packages that seems
affected when we discover them. This means partial upgrades will likely
be broken, and that we might discover some broken packages only after
the jessie release.
2) Rebuild (binNMU) all packages. This means partial upgrades will
likely be broken.
3) Bump the soname of affected packages and rebuild their reverse
dependencies. It is the solution that is currently being implemented for
perl. It clearly won't scale if more broken packages (and even for
libpng) are discovered as it requires a source upload and a transition
handled by the release team. It also means breaking the ABI compatibility
with other distributions.
4) Bump the libc soname to libc.so.6.1 and do a libc transition. This is
probably what upstream should have done instead of breaking the ABI.
This is a huge work though, and this also means breaking the ABI
compatibility with other distributions.
5) Revert the ABI change. This is likely just postponing the problem as
the change is required to support future hardware. This also means
breaking the ABI compatibility with other distributions.
6) simply drop the s390x port and tell users to either use an other
distribution or use Debian on other hardware.
Any opinion? Any other ideas how to handle that?

Option (6) is the nuclear option, and clearly a little excessive given
the situation. If user's install from an installer they get a
perfectly working system. Punishing those users because partial
upgrades don't work seems excessive.

Option (5) postpones the problem until newer s390 hardware arrives.

Option (4) is likely what upstream should have done. Why do you think
it's a huge amount of work? If all s390-supporting distributions agree
then we just change the SO name?

Option (3), (2) or (1) all look like reasonable "fire and forget"
options that require users of s390 to simply move forward with their
installs to full new versions of Debian. It's an option for RHEL
because we don't allow partial-upgrades and we are working with users
to notify them of the breakage during a major RHEL upgrade e.g. RHEL7
to RHEL8.

In my opinion it's not too late to do option (4).

Cheers,
Carlos.

--
To UNSUBSCRIBE, email to debian-s390-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Archive: https://lists.debian.org/CAE2sS1hgdMH4F42iFw-_MrOk1q2+2=***@mail.gmail.com

Aurelien Jarno

2014-07-15 05:20:02 UTC

That is not true. This is an over generalization of the problem. You
can use libraries built with 2.18 and 2.19 and they work just fine.

I agree I probably a bit over exaggerated here, but the problem is real,
breakages do happen, and some persons on this mailing list have already
experienced them.

Post by Carlos O'Donell
https://sourceware.org/glibc/wiki/Release/2.19#Packaging_Changes

This seems to minimize the problem, listing only perl. In practice we
have seen much more breakages, part of them being due to the change of
the __pthread_unwind_buf_t struct.

That is new information, and it is important for distributions to
relay this information back upstream where the decision for a SO bump
can be made.

I can follow up with a list affected packages, but we are slowly
discovering them one by one, so it might takes time. So far we have:

* Mixing modules/libraries built with pre-2.19 and 2.19 libc
- perl
- libpng

* Using libc 2.19 without rebuilding anything:
- gauche
- mono

I apologize if you took what I wrote to mean that. I did not mean it
was Debian's problem, but rather that Debian suffered the most because
they don't do rebuilds. The two are orthogonal. You face a situation
that is unique to the framework used to build the distribution.
Please engage upstream to champion a SO name bump for libc for

I think that would be the correct solution. That said as it is not
something trivial and thus not done often, it's an opportunity to push
for more ABI changes if some others are envisaged in the future.

If we are not able to solve this problem by ensuring all the packages
in the next release are using a consistent ABI, I think it is an option
to consider. It's probably better being honest with the users telling
them "we fail" than shipping a half broken release. Consider that we
have limited human resources to maintain this port.

Post by Carlos O'Donell
Option (5) postpones the problem until newer s390 hardware arrives.
Option (4) is likely what upstream should have done. Why do you think
it's a huge amount of work? If all s390-supporting distributions agree
then we just change the SO name?

It's a huge work for Debian, maybe not for other distribution, as it
basically means we have to rebootstrap everything. This includes manual
bootstrapping of self-dependent languages (haskell, gnat, ...) and
manual handling of some dependencies loop. In addition it's something
which hasn't been done since the libc5 transition, so we might discover
some unexpected issues.

Post by Carlos O'Donell
Option (3), (2) or (1) all look like reasonable "fire and forget"
options that require users of s390 to simply move forward with their
installs to full new versions of Debian. It's an option for RHEL
because we don't allow partial-upgrades and we are working with users
to notify them of the breakage during a major RHEL upgrade e.g. RHEL7
to RHEL8.

Does it mean on RHEL, a full reinstall would be needed? A system being
upgraded from one version to another will transition though partial
ugprades, and we have to ensure that at least the upgrade won't break.

Post by Carlos O'Donell
In my opinion it's not too late to do option (4).

Agreed.

Cheers,
Aurelien

--
Aurelien Jarno GPG: 4096R/1DDD8C9B
***@aurel32.net http://www.aurel32.net
--
To UNSUBSCRIBE, email to debian-s390-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Archive: https://lists.debian.org/***@hall.aurel32.net

Paul Wise

2014-07-15 05:40:01 UTC

Post by Aurelien Jarno
It's a huge work for Debian, maybe not for other distribution, as it
basically means we have to rebootstrap everything. This includes manual
bootstrapping of self-dependent languages (haskell, gnat, ...) and
manual handling of some dependencies loop. In addition it's something
which hasn't been done since the libc5 transition, so we might discover
some unexpected issues.

Helmut Grohne and others have been working on automated rebootstrap of
the Debian archive. The rebootstrap_s390x_gcc49_nobiarch jenkins job
seems to be in good shape currently. That is by no means a full
archive bootstrap, which requires more things.

https://wiki.debian.org/HelmutGrohne/rebootstrap
https://jenkins.debian.net/view/rebootstrap/
https://wiki.debian.org/DebianBootstrap
https://wiki.debian.org/Sprints/2014/BootstrapSprint
--
bye,
pabs

https://wiki.debian.org/PaulWise

Philipp Kern

2014-07-15 07:30:01 UTC

Post by Aurelien Jarno
I can follow up with a list affected packages, but we are slowly
* Mixing modules/libraries built with pre-2.19 and 2.19 libc
- perl
- libpng
- gauche
- mono

I think it's pretty important for perl to keep working as much as is
required for the system to upgrade itself. I'd be a bit less concerned
(aside already broken binary compatibility) if the base system keeps
working during the upgrade.

We could conceivably document the breakage in the release upgrade notes,
as long as updates can complete and suggest a reboot afterwards.

Will it necessarily explode if both libcs are loaded into the same
address space or only if the broken functionality is used? Would
setjmp/longjmp only break if used across libc6/6.1 boundaries? Passing
around an incompatible pthread struct seems bad, though.
If this would work, a re-bootstrap would not necessarily be needed, I
think?

Kind regards and thanks
Philipp Kern

Aurelien Jarno

2014-07-15 08:00:02 UTC

Post by Philipp Kern

Post by Aurelien Jarno
I can follow up with a list affected packages, but we are slowly
* Mixing modules/libraries built with pre-2.19 and 2.19 libc
- perl
- libpng
- gauche
- mono

It might not be easy to ensure the upgrade process works correctly. For
example in mono case, as soon as a new libc is installed, mono stops
working, and installing/upgrading a mono package would fail as mono is
called in the postinst (this is bug#751171). We have to avoid this by
using strict dependencies to make sure the packages are installed in the
right order, but we can't guarantee to detect and handle all cases. That
means some upgrades might break.

Carlos O'Donell

2014-07-15 07:50:01 UTC

That is not true. This is an over generalization of the problem. You
can use libraries built with 2.18 and 2.19 and they work just fine.

I agree I probably a bit over exaggerated here, but the problem is real,
breakages do happen, and some persons on this mailing list have already
experienced them.

Post by Carlos O'Donell
https://sourceware.org/glibc/wiki/Release/2.19#Packaging_Changes

This seems to minimize the problem, listing only perl. In practice we
have seen much more breakages, part of them being due to the change of
the __pthread_unwind_buf_t struct.

That is a change that nobody reported. You're the first to mention it
and that does make it more serious. We have discussed this upstream
and I agree that we need more versioning of the interfaces there to
support the change fully.

That is new information, and it is important for distributions to
relay this information back upstream where the decision for a SO bump
can be made.

I can follow up with a list affected packages, but we are slowly
* Mixing modules/libraries built with pre-2.19 and 2.19 libc
- perl
- libpng

You can never support a mixed-ABI environment with versioning.

You must update all of those packages at once.

The best we could do is warn the user of the incompatibility at
runtime and refuse to load the module via dlopen, or refuse to start
the application at startup.

Post by Aurelien Jarno
- gauche
- mono

This we believe to be pthread issues.

I apologize if you took what I wrote to mean that. I did not mean it
was Debian's problem, but rather that Debian suffered the most because
they don't do rebuilds. The two are orthogonal. You face a situation
that is unique to the framework used to build the distribution.
Please engage upstream to champion a SO name bump for libc for

The problems are worse.

I just tried to simulate this on x86-64 and there are serious problems.

In most libraries you can load multiple different copies and it won't conflict.

Here libc.so.6 and libc.so.7 or libc.so.6.1 all conflict in the same
namespace and worse control aspects of the implementation like TLS. It
doesn't work to bump the SONAME.

We would have to implement a coordination framework amongst all the
SONAME bumped libc's for all of the basic functionality that had to
keep working. That would force future libcs to stay compatible
internally with other libcs and that would be very difficult to
maintain.

I am starting to think that a tooling option to fail to load mixed-ABI
objects is the only option, with user rebuilds happening after that.

I disagree, but I'm not assisting Debian in any way and you have your
own choices to make.

Why do you have to do that? Is it just like for rpm where the
packaging system encodes the SONAME as a dependency? We would also
need a manual bootstrap in Fedora because of this issue.

Yes. For RHEL you always reinstall a new version.

For Fedora you use FedUp (a special initial root disk) to carry out
the upgrade safely (after FedUp downloads it the data it needs
locally).

AFAIK there is no partial upgrade case for Fedora and RHEL.

We still have the problem that user built modules will need to be recompiled.

Post by Carlos O'Donell
In my opinion it's not too late to do option (4).

Agreed.

Unfortunately the more I test this the more impossible this option
seems. The SONAME bump allows two libcs to be loaded at once and it's
not working.

I think the best we can do is diagnose the ABI mixing and refuse to
load or abort a starting program.

Cheers,
Carlos.

Aurelien Jarno

2014-07-15 08:40:02 UTC

That is not true. This is an over generalization of the problem. You
can use libraries built with 2.18 and 2.19 and they work just fine.

I agree I probably a bit over exaggerated here, but the problem is real,
breakages do happen, and some persons on this mailing list have already
experienced them.

Post by Carlos O'Donell
https://sourceware.org/glibc/wiki/Release/2.19#Packaging_Changes

This seems to minimize the problem, listing only perl. In practice we
have seen much more breakages, part of them being due to the change of
the __pthread_unwind_buf_t struct.

That is new information, and it is important for distributions to
relay this information back upstream where the decision for a SO bump
can be made.

I can follow up with a list affected packages, but we are slowly
* Mixing modules/libraries built with pre-2.19 and 2.19 libc
- perl
- libpng

You can never support a mixed-ABI environment with versioning.
You must update all of those packages at once.
The best we could do is warn the user of the incompatibility at
runtime and refuse to load the module via dlopen, or refuse to start
the application at startup.

For perl we handled that using dependencies in the package manager, and
we can probably add some more checks for user modules. That said that do
not scale if we discover more and more affected packages. This is my
fear so far.

Post by Aurelien Jarno
- gauche
- mono

This we believe to be pthread issues.

Yes, this is the pthread issue.

Why do you have to do that? Is it just like for rpm where the
packaging system encodes the SONAME as a dependency? We would also
need a manual bootstrap in Fedora because of this issue.

I assumed that both libc can't be used simultaneously, so that's
basically like bootstrapping a new architecture, except the manual
bootstrapping of self-dependent languages can be done more easily.

Cheers,
Aurelien

Emilio Pozuelo Monfort

2014-07-28 21:00:01 UTC