[cooker-amd64] Re: using and compiling rpms not specifically put together for amd64

[Duncan]

Arie Folger posted <200401231339.33560.afolger@aishdas.org>, excerpted
below,  on Sat, 24 Jan 2004 19:11:59 +0100:

> Can one assume that by compiling SRPMs the result will automatically be a
> program linked to the 64 bit libraries whenever available, does one need a
> special config flag, or does it require that the source code be adapted?
> 
> Also, there are considerably more contrib packages for the i586 than the
> amd64 (IIRC there are none yet for the amd64). Those packages have
> dependencies that aren't resolvable using the amd64 libraries only. Can
> one simply install a 32 bit version alongside the 64 bit version of a
> library, and simply install the i586 package?

Oh... boy..  Someone else embarking upon the learning curve I started on
myself 2-3 months ago.  <g>  I knew, getting into amd64 this early, that
I'd be learning a lot, but expected thru all the frustration, the learning
experience would be fun, and I'd come out the end knowing more about my
new hardware platform of choice, Mandrake and RPM, and Linux as a software
platform, as well as computers in general, and consider it all well worth
it.  How about you? <g>  (If it sounds like I'm saying you have a big job
ahead of you, it's because you do, altho with the full 9.2 for AMD64
release out now, it should be a bit easier and less frustrating  now, tho
without some of the "learning opportunities" <g> as well.)

A single sentence summary would be that the situation is at once simpler
and more complex than you suggest in your question.  However, again,
that's a good opportunity to learn, right?

First, dealing with contribs.  Yes, there ARE contribs out for amd64, now,
altho last I checked they weren't urpmified/hdlistified yet, so you have
to check and download them manually instead of using urpmi to do it. 
Stephan van der Eijk has put together an automated build script that takes
the existing cooker, contribs, and plf srpms, and rebuilds as it
can the amd64 (and other platforms like sparc) binary packages.  

Sometimes these packages rebuild without issue, in which case they are
uploaded to the mirrors (of which there aren't yet so many for the amd64
packages, especially contribs, but..), sometimes there are problems, in
which case the bot logs the errors and Stephan can notify the package
maintainers.  (I'm not sure if he has a bot automatically spamming the
maintainers about their problems or if he does that manually, but..) 
These problems can range from the package being exclusive-arch or
binary-only, examples being binary-only audio/visual codecs and i586
exclusive emulation packages, to rpm spec files that haven't been adjusted
yet to take bi-arch architectures like amd64 into account.

Here's his site, with the various logs and status reports from his
build-bot.  You may or may not find it useful.  I certainly do at times,
as it allows me to confirm that I'm not the ONLY one having trouble
rebuilding a particular RPM.

http://eijk.homelinux.org/build/amd64/

Up under build, you can find the script he uses, and thus, figure out a
bit better what causes the build-logs to be placed in one subdir or
another, if desired.

This is apparently the primary mirror he uploads to, when the build
succeeds, so should have the newest available contribs for amd64, anyway. 
(Main gets built by Mdk, elsewhere, tho it's mirrored there also.)

ftp://anorien.csc.warwick.ac.uk/

On to your library question, the answer to which explains some of the
problems the rebuilds have as well.  Mandrake policy is to keep 64-bit
libraries in their own lib64 path, exactly to allow one to do as you were
suggesting above, install both 32- and 64-bit copies of the same
libraries, without them conflicting, as needed.  One of the SRPM rebuild
problems often encountered is that the SRPM hasn't yet been adjusted to
take that into account.  Rather than using rpm macros to specify library
locations, etc, it has them hardcoded to the standard lib locations, and
the lib64 located libraries it can't find.  Gradually, this is changing as
the rebuild process continues, and the various SRPMs are macroified to be
able to build on all platforms without issue.

That leads into another question you asked.  When building for the
platform you are on, there are no special flags needed, ideally, as the
RPM macro system should take care of everything, automatically adjusting
the target architecture and platform as necessary.  Again, however, that
presupposes that all the macros are set up correctly to do so, and that
isn't yet the case with some SRPMs.  

Other times, the problem building the SRPM is that build-dependencies it
needs aren't yet able to build, due to their own problems.  Or, they can
build, but they can't install, because they depend on other packages that
won't build to run.  (Sometimes, this can just be a wrong dependency in
the RPM.  This is often the case with the various perl script packages and
others that depend on them, such as urpmi, rpmdrake, and the various
other *drake* packages, for example, since perl dependency checking is an
as yet imperfect art in itself, not entirely able to be reliably done,
automatically.  Again, the build-bots mark the error, which then needs
check, and wrong dependencies removed, or corrected as necessary, manually.)

In particular, there's a growing list of *drake* cooker packages now
stalled on perl-GTK2, which won't build, for various reasons, in addition
to a couple bad perl dependencies that may be corrected by now, I'm not
sure, as I have the packages in my urpmi skip-list, since they DO have an
unmet valid dependency on perl-GTK2, of a version that won't build yet for
amd64.  I also have a second stoppage, in that I can't install the latest
sash, because it depends on the latest glibc package, which won't yet
build for amd64, due to some sort of problem with i18n, it would appear,
from Stephan's logs.

Both of those have been blocked for something like two weeks, now, I
think.  There have been continuing *drake* package updates in the mean
time, so I'm now several versions behind, but there's nothing I can do
about it since I'm not a coder or packager yet, and the problems are to
deep for me to figure out and work around on my own.  That's where the
frustration sets in, but at least I know they know about it, and are
working on it, as Stephan's bot logs show I'm not the only one that
couldn't build the problem packages.

The ones that are the real bugger, however, to get working, are the ones,
like the core of the glibc and perl-GTK2 package problems above, that
aren't rpm macro or simple dependency related, and aren't simple rejects
because they're binary-only or exclusive arched packages that won't run in
64 bit amd64 mode.  They have deeper issues, like 64-bit unclean code, or
other assumptions that prove wrong on a platform other than i586.  Code
that isn't 64-bit clean generally comes down to an assumption the original
coder made about the length of 32-bit ints (integers) being the same size
as memory pointers, with the problem being that pointers, on 64-bit apps,
are 64-bits long, while ints are normally still only 32-bit.  There are
other similar cases, but that's by far the most common, from what I've
read.  Obviously, the fix here is to correct the source code, returning
the patch to the application maintainers, if it's not Mandrake, hoping for
inclusion, so Mdk doesn't constantly have to "re-fix" each new version. 
Most application maintainers (aka lead developers) are happy for the help
and include the fix right away.  Others may not.  In any case, tho, that's
far beyond something a bit of srpm package tweaking is going to fix, thus,
far beyond anything I can work around on my own, so I'm dependent on
others to work the fix into the pipeline for me.

Well, that's about it for me, except for some lose ends to wrap up.

1) The contrib packages at the above location should help a lot.

2) If you haven't done so yet, find and configure an updates mirror for
use with urpmi, as there are some significant updates out there for a
number of packages, both security and functional updates.  In particular,
note that the latest AMD64 kernel packages are 2.4.22-27.  Not only do
these close a couple security holes, but release 27 has proven FAR more
stable on my system than release 24 was, and I'm not crashing as much.

3) Unfortunately, the flip side of the coin of the build-bot doing the
stuff that can be done automatically, and Stephan's notifying maintainers
about problems, is that it's the easy problems that have been fixed, with
the real difficult problems (and packages depending on difficult case
problem packages) being the majority of the ones that remain.  Thus, if an
amd64 package isn't already out there for it, as time goes on, it's
meaning the ones that are left are becoming harder and harder for one to
simply up and rebuild one's self, because their the ones that no one else
has been able to fix, just yet.

4) KDE packages deserve special mention, because of their dependency not
only on KDE libs, but on QT libs as well.  The QT3 libs are a special
case, due to their rather strange path on bi-arch systems.  All the qt3
stuff is under /usr/lib/qt3 (NOT /usr/lib64/qt3)  That includes both the
headers, necessary for rebuilding, and the library binaries, under their
own lib64 subdir, thus, /usr/lib/qt3/lib64.  Packages that haven't already
been fixed are thus a bit more difficult than most to get working on your
own.  Not only must you go in and tweak the  various --with switches in
the spec or make file manually, but often times in doing so, you find it
wants to either make that path /usr/lib64/qt3/lib64, or /usr/lib/qt3/lib,
changing either both or neither to the 64, instead of changing only the
latter one.  The easiest way to fix this purely for my own builds, since I
wasn't worried about and had no intention of installing 32-bit qt libs,
was to use two appropriately place symlinks.  the first being
/usr/lib64/qt3, pointing to the qt3 dir under /usr/lib, and a second,
/usr/lib/qt3/lib, pointing to /usr/lib/qt3/lib64.  If I ever accidentally
install 32-bit qt libs, I'll have one **** of a mess trying to clean up
the 32-bit libs that overwrote the 64-bit ones due to the symlinks I
created, but it DID provide a temporary fix for the problem of the srpms
either adjusting both or neither of the libs, when they SHOULD have been
only adjusting the latter one, in the absence of my knowing how to
CORRECTLY fix the rpm macros in question.  Again, this isn't the PROPER
fix for the problem, but in the absence of knowing enough about rpm macros
to figure out the PROPER fix, this was an adequate workaround for me only,
since I wasn't making the resulting packages public.

5) KDE packages again:  Note that the KDE packages now in i586
cooker are newer versions than what we have back here on amd64 cooker, 
KDE 3.2, rather than the KDE 3.1.3+ packages we have.  At one point, I got
VERY frustrated attempting to compile a KDE package, thinking the problem
was the libs path issue above, because the error looked the same
(something about a QT3-mt=multithread capable library being unfound IIRC),
when it was REALLY that I was attempting to build the new package, with
its dependencies on the new library, on my system with the OLD library.  I
spent several hours trying to figure out what was going wrong, and
swearing at the thing, before I figured out my mistake, and very
sheepishly went and downloaded the CORRECT SRPM! <g>

6)  All some SRPM rebuilds need (or did need, when I rebuilt them, they
are probably already available in binary package form now) is the --target
x86_64 switch handed to rpm for the rebuild.  These packages were upgraded
to work with amd64 for the Mdk Corporate-server for AMD64 projects, before
they decided to call the arch amd64, for 9.2.  Now, the two are basic
synonyms, with the automatic target amd64 that rpm attempts to apply from
the site based platform settings set to call x86_64 where necessary, but
some of the earlier packages don't recognize the amd64 form, and need
x86_64 specified as the target specifically.  Then they compile fine. 
Others, such as the KDE packages mentioned above, may need BOTH the
x86_64 target specified, AND some additional tweaking, such as the
symlinks I installed, as mentioned above.  Again, I expect most of these
will already be fixed, however, with binary amd64 rpms available, so you
may not run into a lot of remaining srpms with that problem.

7) A warning on libraries vs application binaries!  Note that while
Mandrake HAS separated the libraries out, so both 32-bit and 64 bit can
normally be installed and running in parallel, that does **NOT** apply to
application binaries, or the /bin dirs they inhabit, themselves.  Thus, be
careful that you don't overwrite 64-bit binary applications, with their
32-bit equivalents, while you are upgrading, or whatever.  A typical
scenario where this might happen would be if you loaded a new urpmi
source, and it found the 32-bit hdlist instead of the 64-bit one (which
may not exist yet).  That happened to me at one point with plf.  It wanted
to upgrade many of my amd64 packages to i586 packages, including
mplayer and other application packages, which would have meant my current
64-bit binaries would have been overwritten by "upgraded" i586 32-bit
binaries.  **NOT** what I had in mind **AT** **ALL**!  Fortunately, I
noticed all those i586 tags where I expected to see amd64s, and canceled
the upgrade.  However, had I just gone ahead and hit enter, I would have
likely been cursing myself for a day or more, trying to straighten out the
mess I'd created!

Hopefully, all this info is of help, and doesn't discourage you, but
rather allows you to pick up the game quickly, and be of help to me, one
of these days!  <g>  Just keep in mind that it's all fun, and we are
learning a lot as we go, and a year or two from now, when everyone else
finally decides 32-bit is going the way of 16-bit and 8-bit before it,
we'll be old hands at the game.  <g>

-- 
Duncan - List replies preferred.   No HTML msgs.
"They that can give up essential liberty to obtain a little
temporary safety, deserve neither liberty nor safety." --
Benjamin Franklin