Arch qualification for buster: call for DSA, Security, toolchain concerns

[Niels Thykier]

Hi,


As part of the interim architecture qualification for buster, we request
that DSA, the security team and the toolchain maintainers review and
update their list of known concerns for buster release architectures.

Summary of the current concerns and issues:
* DSA have announced a blocking issue for armel and armhf (see below)
* Concerns from DSA about ppc64el and s390x have been carried over from
stretch.
* Concerns from the GCC maintainers about armel, armhf, mips, mips64el
and mipsel have been carried over from stretch.

If the issues and concerns from you or your team are not up to date,
then please follow up to this email (keeping debian-***@l.d.o and
debian-***@l.d.o in CC to ensure both parties are notified).

Whilst porters remain ultimately responsible for ensuring the
architectures are ready for release, we do expect that you / your team
are willing to assist with clarifications of the concerns and to apply
patches/changes in a timely manner to resolve the concerns.


List of blocking issues by architecture
=======================================

The following is a summary from the current architecture qualification
table.

armel/armhf:
------------

* Undesirable to keep the hardware running beyond 2020. armhf VM
support uncertain. (DSA)
- Source: [DSA Sprint report]


[DSA Sprint report]:
https://lists.debian.org/debian-project/2018/02/msg00004.html


List of concerns for architectures
==================================

The following is a summary from the current architecture qualification
table.

* Concern for ppc64el and s390x: we are dependent on sponsors for
hardware.
(Raised by DSA; carried over from stretch)

* Concern for armel and armhf: only secondary upstream support in GCC
(Raised by the GCC maintainer; carried over from stretch)

* Concern for mips, mips64el, mipsel and ppc64el: no upstream support
in GCC
(Raised by the GCC maintainer; carried over from stretch)


Architecture status
===================

These are the architectures currently being built for buster:

* Intel/AMD-based: amd64, i386
* ARM-based: arm64, armel, armhf
* MIPS-based: mips, mipsel, mips64el
* Other: ppc64el, s390x

If the blocking issues cannot be resolved, affected architectures are at
risk of removal from testing before buster is frozen.

We are currently unaware of any new architectures likely to be ready in
time for inclusion in buster.

On behalf of the release team,
Niels Thykier

[Florian Weimer]

Post by Niels Thykier
------------
* Undesirable to keep the hardware running beyond 2020. armhf VM
support uncertain. (DSA)
- Source: [DSA Sprint report]

Fedora is facing an issue running armhf under virtualization on arm64:

<https://bugzilla.redhat.com/show_bug.cgi?id=1572916>
<https://gcc.gnu.org/ml/gcc/2018-06/msg00036.html>

Unless the discussion has moved somewhere where I can't follow it, no
one seems to have solid idea what is going on. It's also not clear
that this configuration has substantial vendor or community support.
This makes me concerned that virtualization is a viable path forward
here.

(The discussion on the GCC list started off with a misdirection, sorry
about that. The brief assumption that this was a hardware quirk is
likely quite wrong.)

Post by Niels Thykier
* Concern for mips, mips64el, mipsel and ppc64el: no upstream support
in GCC
(Raised by the GCC maintainer; carried over from stretch)

I'm surprised to read this. ppc64el features prominently in the
toolchain work I do (though I personally do not work on the GCC side).

Post by Niels Thykier
From my point of view, it's absolutely not in the same category as the

MIPS-based architectures.

[Riku Voipio]

Post by Florian Weimer

Post by Niels Thykier
------------
* Undesirable to keep the hardware running beyond 2020. armhf VM
support uncertain. (DSA)
- Source: [DSA Sprint report]

<https://bugzilla.redhat.com/show_bug.cgi?id=1572916>

I think you mean:

https://bugzilla.redhat.com/show_bug.cgi?id=1576593

Post by Florian Weimer
<https://gcc.gnu.org/ml/gcc/2018-06/msg00036.html>
Unless the discussion has moved somewhere where I can't follow it, no
one seems to have solid idea what is going on.

True. Looking at comment #22, the suggestion seems to be that the guest is
doing something wrong, and kvm is being terrible at pinpointing the source.

Post by Florian Weimer
It's also not clear that this configuration has substantial vendor or
community support. This makes me concerned that virtualization is a viable
path forward here.

I understand your concern. It would be surprising if this specific bug doesn't
get found and fixed. It's probably stuck because everyone thinks it's
probably someone elses bug ;)

I still think the armhf vm on arm64 is the only reasonable path forward medium
term. The existing arm64 hw that suport arm32 vm's is still around and
infinitely better than native aarch32 builder hw, and should still be viable
for some time.

Riku

[Steve McIntyre]

Post by Riku Voipio

Post by Florian Weimer

Post by Niels Thykier
------------
* Undesirable to keep the hardware running beyond 2020. armhf VM
support uncertain. (DSA)
- Source: [DSA Sprint report]

<https://bugzilla.redhat.com/show_bug.cgi?id=1572916>

https://bugzilla.redhat.com/show_bug.cgi?id=1576593

Yeah, that looks more likely. I can see that Ard is already trying to
help debug it. But things have gone quiet for a couple of weeks, at
least in the public discussion.

...

Post by Riku Voipio

Post by Florian Weimer
It's also not clear that this configuration has substantial vendor or
community support. This makes me concerned that virtualization is a viable
path forward here.

I understand your concern. It would be surprising if this specific bug doesn't
get found and fixed. It's probably stuck because everyone thinks it's
probably someone elses bug ;)

Yeah, that's my thought too. :-)

Post by Riku Voipio
I still think the armhf vm on arm64 is the only reasonable path forward medium
term. The existing arm64 hw that suport arm32 vm's is still around and
infinitely better than native aarch32 builder hw, and should still be viable
for some time.

Nod. The "fun" thing we see is that quite a few of the biggest AArch64
CPUs are A64-only, but there's still a selection of things available
that I think look OK. I'll post separately in a moment about that...

[Florian Weimer]

Post by Riku Voipio

Post by Florian Weimer

Post by Niels Thykier
------------
* Undesirable to keep the hardware running beyond 2020. armhf VM
support uncertain. (DSA)
- Source: [DSA Sprint report]

<https://bugzilla.redhat.com/show_bug.cgi?id=1572916>

https://bugzilla.redhat.com/show_bug.cgi?id=1576593

Yes, that's right, I copy-pasted the wrong bug URL. It's filed as a
Red Hat Enterprise Linux bug because the Fedora builders run Fedora
VMs on that platform.

[Uwe Kleine-König]

Hello,

Post by Niels Thykier
------------
* Undesirable to keep the hardware running beyond 2020. armhf VM
support uncertain. (DSA)
- Source: [DSA Sprint report]
https://lists.debian.org/debian-project/2018/02/msg00004.html
In short, the hardware (development boards) we're currently using to
build armel and armhf packages aren't up to our standards, and we
really, really want them to go away when stretch goes EOL (expected in
2020). We urge arm porters to find a way to build armhf packages in
VMs or chroots on server-class arm64 hardware.

If the concerns are mostly about the hardware not being rackable, there
is a rackable NAS by Netgear:

https://www.netgear.com/business/products/storage/readynas/RN2120.aspx#tab-techspecs

with an armhf cpu. Not sure if cpu speed (1.2 GHz) and available RAM (2
GiB) are good enough. The machine can run mainline Linux[1]. I think
U-Boot doesn't support this machine in mainline though.

Apart from that the people in #debian-arm (e.g. Sledge) seem to be
positive that at least armhf should be fine to be built on arm64
hardware.

Best regards
Uwe

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm/boot/dts/armada-xp-netgear-rn2120.dts

[Luke Kenneth Casson Leighton]

Post by Uwe Kleine-König
Hello,

Post by Niels Thykier
------------
* Undesirable to keep the hardware running beyond 2020. armhf VM
support uncertain. (DSA)
- Source: [DSA Sprint report]
https://lists.debian.org/debian-project/2018/02/msg00004.html
In short, the hardware (development boards) we're currently using to
build armel and armhf packages aren't up to our standards, and we
really, really want them to go away when stretch goes EOL (expected in
2020). We urge arm porters to find a way to build armhf packages in
VMs or chroots on server-class arm64 hardware.

from what i gather the rule is that the packages have to be built
native. is that a correct understanding or has the policy changed?

Post by Uwe Kleine-König
If the concerns are mostly about the hardware not being rackable, there
https://www.netgear.com/business/products/storage/readynas/RN2120.aspx#tab-techspecs
with an armhf cpu. Not sure if cpu speed (1.2 GHz) and available RAM (2
GiB) are good enough.

no matter how much RAM there is it's never going to be "enough", and
letting systems go into swap is also not a viable option [2]

i've been endeavouring to communicate the issue for many many years
wrt building (linking) of very large packages, for a long, *long*
time. as it's a strategic cross-distro problem that's been very very
slowly creeping up on *all* distros as packages inexorably creep up in
size, reaching people about the problem and possible solutions is
extremely difficult. eventually i raised a bug on binutils and it
took several months to communicate the extent and scope of the problem
even to the developer of binutils:

https://sourceware.org/bugzilla/show_bug.cgi?id=22831

the problem is that ld from binutils by default, unlike gcc which
looks dynamically at how much RAM is available, loads absolutely all
object files into memory and ASSUMES that swap space is going to take
care of any RAM deficiencies.

unfortunately due to the amount of cross-referencing that takes place
in the linker phase this "strategy" causes MASSIVE thrashing, even if
one single object file is sufficient to cause swapping.

this is particularly pertinent for systems which compile with debug
info switched on as it is far more likely that a debug compile will go
into swap, due to the amount of RAM being consumed.

firefox now requires 7GB of resident RAM, making it impossible to
compile on 32-bit systems webkit-based packages require well over 2GB
RAM (and have done for many years). i saw one scientific package a
couple years back that could not be compiled for 32-bit systems
either.

all of this is NOT the fault of the PACKAGES [1], it's down to the
fact that *binutils* - ld's default memory-allocation strategy - is
far too aggressive.

the main developer of ld has this to say:

Please try if "-Wl,--no-keep-memory" works.

now, that's *not* a good long-term "solution" - it's a drastic,
drastic hack that cuts the optimisation of keeping object files in
memory stone dead. it'll work... it will almost certainly result in
32-bit systems being able to successfully link applications that
previously failed... but it *is* a hack. someone really *really*
needs to work with the binutils developer to *properly* solve this.

if any package maintainer manages to use the above hack to
successfully compile 32-bit packages that previously completely ran
out of RAM or otherwise took days to complete, please do put a comment
to that effect in the binutiols bugreport, it will help everyone in
the entire GNU/Linux community to do so.

l.

[1] really, it is... developers could easily split packages into
dynamic-loadable modules, where each module easily compiles well below
even 2GB or 1GB of RAM. they choose not to, choosing instead to link
hundreds of object files into a single executable (or library).
asking so many developers to change their strategy however... yyeah :)
big task, i ain't taking responsibility for that one.

[2] the amount of memory being required for the linker phase of large
packages over time goes up, and up, and up, and up... when is it going
to stop? never. so just adding more RAM is never going to "solve"
the problem, is it? it just *avoids* the problem. letting even
64-bit systems go into swap is a huge waste of resources as builds
that go into swap will consume far more resources and time. so *even
on 64-bit systems* this needs solving.

[John Paul Adrian Glaubitz]

Post by Luke Kenneth Casson Leighton

Post by Niels Thykier
In short, the hardware (development boards) we're currently using to
build armel and armhf packages aren't up to our standards, and we
really, really want them to go away when stretch goes EOL (expected in
2020). We urge arm porters to find a way to build armhf packages in
VMs or chroots on server-class arm64 hardware.

from what i gather the rule is that the packages have to be built
native. is that a correct understanding or has the policy changed?

Native in the sense that the CPU itself is not emulated which is the case
when building arm32 packages on arm64. We're also building i386 packages
on amd64 and we used to build powerpc packages on ppc64 (and we will continue
to do that once the move of powerpc to ports has been completed).

I think that building on arm64 after fixing the bug in question is the
way to move forward. I'm surprised the bug itself hasn't been fixed yet,
doesn't speak for ARM.

Adrian

[Luke Kenneth Casson Leighton]

On Fri, Jun 29, 2018 at 12:06 PM, John Paul Adrian Glaubitz

Post by John Paul Adrian Glaubitz

Post by Luke Kenneth Casson Leighton

Post by Niels Thykier
In short, the hardware (development boards) we're currently using to
build armel and armhf packages aren't up to our standards, and we
really, really want them to go away when stretch goes EOL (expected in
2020). We urge arm porters to find a way to build armhf packages in
VMs or chroots on server-class arm64 hardware.

from what i gather the rule is that the packages have to be built
native. is that a correct understanding or has the policy changed?

Native in the sense that the CPU itself is not emulated which is the case
when building arm32 packages on arm64.

ok. that's clear. thanks john.

Post by John Paul Adrian Glaubitz
I think that building on arm64 after fixing the bug in question is the
way to move forward. I'm surprised the bug itself hasn't been fixed yet,
doesn't speak for ARM.

if you mean ARM hardware (OoO), it's too late. systems are out there
with OoO speculative execution bugs in the hardware (and certainly
more to be found), and they're here to stay unfortunately.

if you mean that buildd on 32-bit systems could be modified to pass
"-Wl,--no-keep-memory" to all linker phases to see if that results in
the anticipated dramatic reduction in memory usage, that's
straightforward to try, nothing to do with ARM themselves.

l.

[John Paul Adrian Glaubitz]

Post by Luke Kenneth Casson Leighton

Post by John Paul Adrian Glaubitz
I think that building on arm64 after fixing the bug in question is the
way to move forward. I'm surprised the bug itself hasn't been fixed yet,
doesn't speak for ARM.

if you mean ARM hardware (OoO), it's too late. systems are out there
with OoO speculative execution bugs in the hardware (and certainly
more to be found), and they're here to stay unfortunately.

How are the speculative execution bugs related in any way to the software
bug which prevents proper KVM emulation of ARM32 binaries on ARM64? Those
are two completely different topics.

I never made any statement regarding these bugs. I merely said that the
best way to build ARM32 packages in the future would be to build them
on ARM64 hardware like we are building i386 packages on amd64.

Post by Luke Kenneth Casson Leighton
if you mean that buildd on 32-bit systems could be modified to pass
"-Wl,--no-keep-memory" to all linker phases to see if that results in
the anticipated dramatic reduction in memory usage, that's
straightforward to try, nothing to do with ARM themselves.

Again: I was talking about building 32-bit packages on 64-bit systems,
i.e. 32-bit userland on a 64-bit kernel. We do that for a lot of architectures,
including mips, powerpc, x86 and in the past also for sparc.

Adrian

[Luke Kenneth Casson Leighton]

On Fri, Jun 29, 2018 at 1:12 PM, John Paul Adrian Glaubitz

Post by John Paul Adrian Glaubitz

Post by Luke Kenneth Casson Leighton

Post by John Paul Adrian Glaubitz
I think that building on arm64 after fixing the bug in question is the
way to move forward. I'm surprised the bug itself hasn't been fixed yet,
doesn't speak for ARM.

if you mean ARM hardware (OoO), it's too late. systems are out there
with OoO speculative execution bugs in the hardware (and certainly
more to be found), and they're here to stay unfortunately.

How are the speculative execution bugs related in any way to the software
bug which prevents proper KVM emulation of ARM32 binaries on ARM64? Those
are two completely different topics.

apologies, i just didn't quite understand your answer, so i was
looking for clarification.

Post by John Paul Adrian Glaubitz

Post by Luke Kenneth Casson Leighton
if you mean that buildd on 32-bit systems could be modified to pass
"-Wl,--no-keep-memory" to all linker phases to see if that results in
the anticipated dramatic reduction in memory usage, that's
straightforward to try, nothing to do with ARM themselves.

Again: I was talking about building 32-bit packages on 64-bit systems,
i.e. 32-bit userland on a 64-bit kernel. We do that for a lot of architectures,
including mips, powerpc, x86 and in the past also for sparc.

would it be possible to try out that linker flag and let the binutils
team know the results? i'd also really like to know.

l.

[Julien Cristau]

[s/debian-ports/debian-arm/]

Post by Uwe Kleine-König
Hello,

Post by Niels Thykier
------------
* Undesirable to keep the hardware running beyond 2020. armhf VM
support uncertain. (DSA)
- Source: [DSA Sprint report]
https://lists.debian.org/debian-project/2018/02/msg00004.html
In short, the hardware (development boards) we're currently using to
build armel and armhf packages aren't up to our standards, and we
really, really want them to go away when stretch goes EOL (expected in
2020). We urge arm porters to find a way to build armhf packages in
VMs or chroots on server-class arm64 hardware.

If the concerns are mostly about the hardware not being rackable, there
https://www.netgear.com/business/products/storage/readynas/RN2120.aspx#tab-techspecs
with an armhf cpu. Not sure if cpu speed (1.2 GHz) and available RAM (2
GiB) are good enough. The machine can run mainline Linux[1]. I think
U-Boot doesn't support this machine in mainline though.

Rackable, while good, is only part of it. The main part is remote
management. I'm not seeing any mention of ipmi or anything like that in
the datasheet?

2G is also way too little memory these days for a new buildd.

Cheers,
Julien

[Uwe Kleine-König]

Hello Julien,

Post by Julien Cristau

Post by Uwe Kleine-König
If the concerns are mostly about the hardware not being rackable, there
https://www.netgear.com/business/products/storage/readynas/RN2120.aspx#tab-techspecs
with an armhf cpu. Not sure if cpu speed (1.2 GHz) and available RAM (2
GiB) are good enough. The machine can run mainline Linux[1]. I think
U-Boot doesn't support this machine in mainline though.

Rackable, while good, is only part of it. The main part is remote
management. I'm not seeing any mention of ipmi or anything like that in
the datasheet?

you can access the serial console, but I don't think there is built-in
support for something IPMI-like.

Post by Julien Cristau
2G is also way too little memory these days for a new buildd.

Then the machine is out, the amount of RAM isn't upgradable.

Best regards
Uwe

[Roger Shimizu]

On Fri, Jun 29, 2018 at 10:04 PM, Uwe Kleine-König

Post by Uwe Kleine-König
Hello Julien,

Post by Julien Cristau

Post by Uwe Kleine-König
If the concerns are mostly about the hardware not being rackable, there
https://www.netgear.com/business/products/storage/readynas/RN2120.aspx#tab-techspecs
with an armhf cpu. Not sure if cpu speed (1.2 GHz) and available RAM (2
GiB) are good enough. The machine can run mainline Linux[1]. I think
U-Boot doesn't support this machine in mainline though.

Rackable, while good, is only part of it. The main part is remote
management. I'm not seeing any mention of ipmi or anything like that in
the datasheet?

you can access the serial console, but I don't think there is built-in
support for something IPMI-like.

Post by Julien Cristau
2G is also way too little memory these days for a new buildd.

Then the machine is out, the amount of RAM isn't upgradable.

I don't think 2GB is not enough for 32-bit machine.

I see armel is already not a candidate for buster [0].
So it seems we can discuss armhf, but no armel at all.
I don't agree with this idea.
And I think we should treat armel and armhf equally.

Both armel and armhf are working fine are millions of boards and
embedded devices, and have stable quality [1].
They deserve the support from a community driven distro.

[0] https://release.debian.org/buster/arch_qualify.html
[1] https://lists.debian.org/debian-arm/2017/11/msg00061.html

Cheers,

[Samuel Thibault]

Post by Roger Shimizu
On Fri, Jun 29, 2018 at 10:04 PM, Uwe Kleine-König

Post by Uwe Kleine-König

Post by Julien Cristau
2G is also way too little memory these days for a new buildd.

Then the machine is out, the amount of RAM isn't upgradable.

I don't think 2GB is not enough for 32-bit machine.

I can say that 2GB is really not enough for a quite-non-small list
of packages. Sure you can add swap, but then e.g. link phases are
agonizingly long.

Samuel

[Gene Heskett]

Post by Samuel Thibault

Post by Roger Shimizu
On Fri, Jun 29, 2018 at 10:04 PM, Uwe Kleine-König

Post by Uwe Kleine-König

Post by Julien Cristau
2G is also way too little memory these days for a new buildd.

Then the machine is out, the amount of RAM isn't upgradable.

I don't think 2GB is not enough for 32-bit machine.

I can say that 2GB is really not enough for a quite-non-small list
of packages. Sure you can add swap, but then e.g. link phases are
agonizingly long.
Samuel

hijacking your thread, my apologies:

And I might add, on the likes of a pi-3b, demands that swap somehow be
migrated to spinning rust or an SSD, else from the descriptions on
binutils above this msg, will likely finish off the best of a u-sd card,
which its booting and running from now. Because I intend to build a new
rt-kernel for that pi, on that pi, the 60 GB SSD is already plugged in
but I've not yet moved swap. And it sounds like it will need far more
than 2GB, so I'm thinking at least 10 would be indicated based on the
binutils comments. Comments otherwise?

Thanks everybody.

[Jonathan Wiltshire]

Rather than apologising for hijacking a thread in the same breath as doing
so, please simply avoid doing so in the first place. The set of lists
in your CC are *really* not appropriate for such a mail.

Thanks.

[Luke Kenneth Casson Leighton]

Post by Samuel Thibault

Post by Roger Shimizu
On Fri, Jun 29, 2018 at 10:04 PM, Uwe Kleine-König

Post by Uwe Kleine-König

Post by Julien Cristau
2G is also way too little memory these days for a new buildd.

Then the machine is out, the amount of RAM isn't upgradable.

I don't think 2GB is not enough for 32-bit machine.

I can say that 2GB is really not enough for a quite-non-small list
of packages.

and that list is only going to get inexorably and slowly bigger, to
the point where even on 64-bit systems at some point someone is going
to notice. 7GB (and climbing) of resident RAM for the linker phase on
firefox to keep it out of swap space *should* be ringing alarm bells
even for amd64 build maintainers.

Post by Samuel Thibault
Sure you can add swap, but then e.g. link phases are
agonizingly long.

sorry if it was not clear, and (to people who have read the analysis
already) apologies for repeating it for the third time, but this is
precisely and exactly why i said that it would be a good idea to
investigate adding "-Wl,--no-keep-memory" to the linker phase of
32-bit builds.

https://sourceware.org/bugzilla/show_bug.cgi?id=22831

linker phases 100% guaranteed to go into thrashing, due to the massive
amount of cross-referencing needed to be done in object-file linking
has been a known problem for over nine years, which nobody really
seems to understand or acknowledge or tackle.

l.

[Jonathan Wiltshire]

Post by Roger Shimizu
I see armel is already not a candidate for buster [0].
So it seems we can discuss armhf, but no armel at all.
I don't agree with this idea.
And I think we should treat armel and armhf equally.

Please review the mail which originated this thread [1] where you will see
that both armel and armhf are affected by DSA's concern. If I understand
correctly, virtualisation of architectures in general is a possible
solution but there are problems in the case of these two.

At the end of the day, if Debian can't reliably run builders for an
architecture we do not do users a service by pretending to be able to
support it in a formal release. A freeze may be for Christmas, but stable
is for at least five+ years.

[John Paul Adrian Glaubitz]

Post by Jonathan Wiltshire

Post by Roger Shimizu
I see armel is already not a candidate for buster [0].
So it seems we can discuss armhf, but no armel at all.
I don't agree with this idea.
And I think we should treat armel and armhf equally.

Please review the mail which originated this thread [1] where you will see
that both armel and armhf are affected by DSA's concern. If I understand
correctly, virtualisation of architectures in general is a possible
solution but there are problems in the case of these two.

I have just talked to a colleague at SUSE about this and apparently Alex Graf from SUSE’s QEMU/KVM team has fixed many bugs regarding ARM32 on ARM64 virtualization. If I understand correctly, SUSE builds ARMv7 on ARM64 without problems.

I have reached out to Alex Graf and asked him for clarification on what possibilities we currently have.

Adrian

[Steve McIntyre]

Post by Julien Cristau

Post by Uwe Kleine-König

Post by Niels Thykier
https://lists.debian.org/debian-project/2018/02/msg00004.html
In short, the hardware (development boards) we're currently using to
build armel and armhf packages aren't up to our standards, and we
really, really want them to go away when stretch goes EOL (expected in
2020). We urge arm porters to find a way to build armhf packages in
VMs or chroots on server-class arm64 hardware.

If the concerns are mostly about the hardware not being rackable, there
https://www.netgear.com/business/products/storage/readynas/RN2120.aspx#tab-techspecs
with an armhf cpu. Not sure if cpu speed (1.2 GHz) and available RAM (2
GiB) are good enough. The machine can run mainline Linux[1]. I think
U-Boot doesn't support this machine in mainline though.

Rackable, while good, is only part of it. The main part is remote
management. I'm not seeing any mention of ipmi or anything like that in
the datasheet?
2G is also way too little memory these days for a new buildd.

Nod - lots of packages are just too big for that now.

So, here's a summary of the current build/porter machines we have for
the arm ports.

armel/armhf *only* build machines
=================================

We're mostly using dev boards to build 32-bit arm stuff yet.

* 7 Marvell Armada XP machines: dev boards supplied in a box, 4GiB
RAM - see http://photos.einval.com/gallery/2014_marvell_buildd
These are nice powerful little machines, but they're a PITA to
manage for power (won't cold boot without a button push) and serial
(USB serial exposed only). We have 6 of these as build boxes and 1
porter box, and I have a spare ready to go into service if
desired. They *don't* have NEON, so we also still have:

* 1 Freescale imx53 dev board as a porter box - old, slow Cortex A8
machine with only 1GiB of RAM. This works better for serial, but
remote power issues again - needs a button push to cold boot. Will
happily retire this once we have NEON available by default.

Each of these dev boards only has support for 1 disk, so disk failures
are painful.

arm64 build machines
====================

These are all more normal computers, with better support for remote
power and serial, DIMM slots, multiple disks, etc.

* APM Mustang (X-Gene 1): officially EOL, but working fine for
now. Normal server-class machine (although supplied in a small
desktop case!) with some onboard server management stuff. We
currently have one of these. We used to have more loaned/hosted by
Linaro, and I've had an offer of more of these if we're
interested. They'll build and run A32 (32-bit instruction set) as
well as A64.

* Gigabyte MP30-AR0 (X-Gene 1): server systems based on the Mustang
core - see
https://b2b.gigabyte.com/Server-Motherboard/MP30-AR0-rev-11#ov
Capable of building/running A32 and A64.

* AMD Seattle (Opteron A1100): officially EOL too, but working
fine. Same as the Softiron 3000, 2U rackmount case. Capable of
building/running A32 and A64. One of these has just been configured
to build armhf only.

Future options
==============

I understand DSA's reluctance to continue supporting dev boards as
build platforms - I've been the one working on some of these machines
in the machine room at Arm, and it's painful when you can't reliably
reboot or get onto the console of crashed machines. We've also had a
spate of disk failures recently which has caused extended downtime.
I'm just in the middle of switching the arm64 machines here to using
SW RAID to mitigate that in future, and that's just not an option on
the dev boards. We want to move away from dev boards for these
reasons, at the very least.

So, at the moment as far as I can see we're happy with our current
arm64 build machines. They are ageing, so obviously I'm continuing to
look out for new options there as well. *But* my priority is new
options for 32-bit building too. Following standard Debian practice,
we want to build *natively* (i.e. not using cross-building or using
hardware emulation). Building 32-bit code on a 64-bit platform should
not be an issue so long as the platform can also execute that 32-bit
code directly.

I am not aware of any 32-bit Arm *server* platforms shipping
today. Some have existed in the past (e.g. Calxeda), but almost
universally people have moved on to 64-bit now. The awkward thing that
is now becoming more common in the arm64 server world is that quite a
few of the vendors are not seeing any value in A32 support so they're
leaving it out of their CPUs. We'll need to be careful about that.

Options I can see today for new publically available machines are
here. I'm sure I'll have missed something obvious - please feel free
to improve on this list!

* Macchiatobin [1] - based on the Marvell 8040 SoC (4-core Cortex
A72), supports both A32 and A64. Standard format (mini-itx) board
mountable in a rack case. DIMM slot supports up to 16GiB, 3 SATA
ports, multiple onboard NICs. Supported in mainline upstream
kernel. Console on USB :-/. Readily available to buy.

* Synquacer [2] - based on the Socionext SC2A11 SoC (24-core Cortex
A53), supports both A32 and A64. Standard format (ATX) board
mountable in a rack case. DIMM slots supports up to 4x16GiB, 2 SATA
ports, onboard NIC. Supported in mainline upstream kernel. I'm
hoping to get some of these machines donated to us from Linaro.

* Qualcomm Centriq [3] - based on Qualcomm's Falkor CPU. Only
supports A64 - no A32 support. All the big features you'd want in a
big expensive server (management, RAM, I/O). Development machines
available, but difficult to get hold of for the general
public. Supported in mainline upstream kernel, some of it
backported for Stretch (9.5) in #896775.

* ThunderX 2 [4] - Cavium's second generation of AArch64 server
CPU. Only supports A64 - no A32 support. All the big features you'd
want in a big expensive server (management, RAM, I/O). Development machines
available, but difficult to get hold of for the general
public. Supported in mainline upstream kernel.

* HiSilicon D05 [5] - HiSilicon's latest AArch64 server CPU and
board. AFAIK only supports A64 - no A32 support. All the big
features you'd want in a big expensive server (management, RAM,
I/O). Development machines available, but difficult to get hold of
for the general public - I've been trying for a while with
HiSilicon. Not sure about upstream kernel support.

While they're on the lower end of this list, I think the Macchiatobin
and Synquacer are probably our best bets *at the moment*, particularly
when considering A32 support. In suitable rack cases with PDU and
serial console, would those work for DSA's needs?

[1] http://macchiatobin.net/
[2] https://www.cnx-software.com/2017/09/24/gigabyte-synquacer-96boards-enterprise-platform-is-powered-by-socionext-sc2a11-24-core-armv8-soc/)
[3] https://www.qualcomm.com/products/qualcomm-centriq-2400-processor
[4] https://www.cavium.com/product-thunderx2-arm-processors.html
[5] http://open-estuary.org/d05/

[Luke Kenneth Casson Leighton]

Post by Steve McIntyre

Post by Julien Cristau
2G is also way too little memory these days for a new buildd.

Nod - lots of packages are just too big for that now.

apologies for repeating it again: this is why i'm recommending people
try "-Wl,--no-keep-memory" on the linker phase as if it works as
intended it will almost certainly drastically reduce memory usage to
the point where it will stay, for the majority of packages, well
within the 2GB limit i.e. within resident RAM.

i'm really not sure why the discussions continue not to take this
into account, repeating the status-quo and accepting "packages are too
big" as if there is absolutely no possible way that this problem may
be solved or even attempted to be solved... ever. i am very confused
by this. perhaps it is down to latency in discussions as new people
contribute (but have signficant delay on reading), i don't know.

Post by Steve McIntyre
Future options
==============
I understand DSA's reluctance to continue supporting dev boards as
build platforms - I've been the one working on some of these machines
in the machine room at Arm, and it's painful when you can't reliably
reboot or get onto the console of crashed machines. We've also had a
spate of disk failures recently which has caused extended downtime.

that is not a surprise to hear: the massive thrashing caused by the
linker phase not being possible to be RAM-resident will be absolutely
hammering the drives beyond reasonable wear-and-tear limits. which is
why i'm recommending people try "-Wl,--no-keep-memory".

... oh, i have an idea which people might like to consider trying.
it's to use "-Wl,--no-keep-memory" on the linker phase of 32-bit
builds. did i mention that already? :) it might save some build
hardware from being destroyed if people try using
"-Wl,--no-keep-memory"!

Post by Steve McIntyre
I'm just in the middle of switching the arm64 machines here to using
SW RAID to mitigate that in future, and that's just not an option on
the dev boards. We want to move away from dev boards for these
reasons, at the very least.

most of them won't have native SATA: very few 32-bit ARM systems do.
GbE is not that common either (so decent-speed network drives are
challenging, as well). so they'll almost certainly be USB-based
(USB-to-SATA, which is known-unreliable), and putting such vast
amounts of drive-hammering through USB-to-SATA due to thrashing isn't
going to help :)

the allwinner A20 and R40 are the two low-cost ARM systems that i'm
aware of that have native SATA.


there is however a new devboard that is reasonably cheap and should
be available really soon: the Rock64Pro (not to be confused with the
Rock64, which does NOT have PCie), from pine64:
https://www.pine64.org/?page_id=61454

it's one of the first *low-cost* ARM dev-boards that i've seen which
has 4GB of RAM and has a 4x PCIe slot. the team have tested it out
with an NVMe SSD and also 4x SATA PCIe cards: they easily managed to
hit 20 Gigabits per second on the NVMe drive (2500 mbytes/sec).

also worth noting, they're working on a 2U rackmount server which
will have i think something insane like 48 Rock64Pro boards in one
full-length case.

the Rock64Pro uses the RK3399 which is a 4-core CortexA53 plus 2-core
CortexA72 for a total 6-core SMP system, all 64-bit.

if anyone would like me to have a word with TL Lim (the CEO of
pine64) i can see if he is willing and able to donate some Rock64Pro
boards to the debian farm, let me know.

l.

[Jonathan Wiltshire]

Post by Luke Kenneth Casson Leighton
apologies for repeating it again: this is why i'm recommending people
try "-Wl,--no-keep-memory" on the linker phase as if it works as
intended it will almost certainly drastically reduce memory usage to
the point where it will stay, for the majority of packages, well
within the 2GB limit i.e. within resident RAM.

[...]

Post by Luke Kenneth Casson Leighton
most of them won't have native SATA: very few 32-bit ARM systems do.
GbE is not that common either (so decent-speed network drives are
challenging, as well). so they'll almost certainly be USB-based
(USB-to-SATA, which is known-unreliable), and putting such vast
amounts of drive-hammering through USB-to-SATA due to thrashing isn't
going to help :)

[...]

Post by Luke Kenneth Casson Leighton
the allwinner A20 and R40 are the two low-cost ARM systems that i'm
aware of that have native SATA.
there is however a new devboard that is reasonably cheap and should
be available really soon: the Rock64Pro (not to be confused with the
https://www.pine64.org/?page_id=61454
it's one of the first *low-cost* ARM dev-boards that i've seen which
has 4GB of RAM and has a 4x PCIe slot. the team have tested it out
with an NVMe SSD and also 4x SATA PCIe cards: they easily managed to
hit 20 Gigabits per second on the NVMe drive (2500 mbytes/sec).
also worth noting, they're working on a 2U rackmount server which
will have i think something insane like 48 Rock64Pro boards in one
full-length case.
the Rock64Pro uses the RK3399 which is a 4-core CortexA53 plus 2-core
CortexA72 for a total 6-core SMP system, all 64-bit.
if anyone would like me to have a word with TL Lim (the CEO of
pine64) i can see if he is willing and able to donate some Rock64Pro
boards to the debian farm, let me know.

None of this addresses the basic DSA requirement of remote management.
Troubling local hands to change a disk once in a while is reasonable; being
blocked waiting for a power cycle on a regular basis is not (and I can't
imagine hosting sponsors are wild about their employees' time being used
for that either).

Development boards just don't cut it any longer.

[Luke Kenneth Casson Leighton]

Post by Jonathan Wiltshire

Post by Luke Kenneth Casson Leighton
also worth noting, they're working on a 2U rackmount server which
will have i think something insane like 48 Rock64Pro boards in one
full-length case.

None of this addresses the basic DSA requirement of remote management.
Troubling local hands to change a disk once in a while is reasonable; being
blocked waiting for a power cycle on a regular basis is not (and I can't
imagine hosting sponsors are wild about their employees' time being used
for that either).

i know exactly what you mean, i've had to deal with data centres.
i'll make sure that TL Lim is aware of this, and will ask him if
there's a way to include remote power-management / power-cycling of
boards in the planned product or if it's already been thought of.

l.

[Luke Kenneth Casson Leighton]

On Fri, Jun 29, 2018 at 8:13 PM, Luke Kenneth Casson Leighton

Post by Luke Kenneth Casson Leighton

Post by Jonathan Wiltshire

Post by Luke Kenneth Casson Leighton
also worth noting, they're working on a 2U rackmount server which
will have i think something insane like 48 Rock64Pro boards in one
full-length case.

None of this addresses the basic DSA requirement of remote management.
Troubling local hands to change a disk once in a while is reasonable; being
blocked waiting for a power cycle on a regular basis is not (and I can't
imagine hosting sponsors are wild about their employees' time being used
for that either).

i know exactly what you mean, i've had to deal with data centres.
i'll make sure that TL Lim is aware of this, and will ask him if
there's a way to include remote power-management / power-cycling of
boards in the planned product or if it's already been thought of.

TL informs me that all the power and reset signals for all 48 of the
RockPro64s tucked into the full-length 2U case are brought out to the
back panel. an MCU (or MCUs) or SBC (or SBCs) may therefore be
connected directly to those in order to provide *individual* remote
power / reset management of all 48 RockPro64s. DIY remote power
management, but it is actual remote power management.

l.

[Luke Kenneth Casson Leighton]

spoke again to TL and asked if pine64 would be willing to look at
sponsorship witn rockpro64 boards (the ones that take 4x PCIe): if
someone from debian were to contact him direct he would happily
consider it.

i then asked him if i could cc him into this discussion and he said he
was way *way* too busy to enter into another mailing list discussion,
so if someone from debian emails me privately, off-list, i will then
cc him and/or put them in touch with him on irc. i can also be
reached on freenode and oftc as "lkcl", if that is easier.

l.

[Florian Weimer]

Post by Luke Kenneth Casson Leighton
that is not a surprise to hear: the massive thrashing caused by the
linker phase not being possible to be RAM-resident will be absolutely
hammering the drives beyond reasonable wear-and-tear limits. which is
why i'm recommending people try "-Wl,--no-keep-memory".

Note that ld will sometimes stuff everything into a single RWX segment
as a result, which is not desirable.

Unfortunately, without significant investment into historic linker
technologies (with external sorting and that kind of stuff), I don't
think it is viable to build 32-bit software natively in the near
future. Maybe next year only a few packages will need exceptions, but
the number will grow with each month. Building on 64-bit kernels will
delay the inevitable because more address space is available to user
space, but that's probably 12 to 18 month extended life-time for
native building.

[Luke Kenneth Casson Leighton]

---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68

Post by Florian Weimer

Post by Luke Kenneth Casson Leighton
that is not a surprise to hear: the massive thrashing caused by the
linker phase not being possible to be RAM-resident will be absolutely
hammering the drives beyond reasonable wear-and-tear limits. which is
why i'm recommending people try "-Wl,--no-keep-memory".

Note that ld will sometimes stuff everything into a single RWX segment
as a result, which is not desirable.

florian, thank you for responding: i've put a copy of the insights
that you give into the bugreport at
https://sourceware.org/bugzilla/show_bug.cgi?id=22831#c16

Post by Florian Weimer
Unfortunately, without significant investment into historic linker
technologies (with external sorting and that kind of stuff),

yes, ah ha! funnily enough the algorithm that i was asked to create
back in 1988 was an external matrix-multiply, i take it you are
talking about the same thing, where linking is done using explicit
load-process-save cycles rather than relying on swap.

Post by Florian Weimer
I don't
think it is viable to build 32-bit software natively in the near
future.

i noted an alternative strategy in the bugreport: if binutils *at
the very least* were to look at the available resident RAM and only
malloc'd and used up to that amount, and kept only a few (or even just
one) object file in memory at a time and did all the linking for that,
it would be infinitely better than the current situation which is
*only going to get worse*.

Post by Florian Weimer
Maybe next year only a few packages will need exceptions, but
the number will grow with each month.

well... that ignores the fact that at some point in the next few
years there will be a package that needs 16 GB of resident RAM to
link. and a few years after that it will be 32 GB. and that's on
64-bit systems. the package's name will probably be "firefox", given
the current growth rate.

does debian *really* want to have to upgrade all 64-bit systems in
the build farm first to 16 GB RAM and then to 32 GB and then to 64
GB?? can the powerpc64 systems and all other 64-bit architectures
even *be* upgraded to 16 GB then 32 GB then 64 GB of RAM??

basically the problems faced by 32-bit systems are a warning shot
across the bows about ld not really being kept up-to-date with the
increases in software complexity that's being thrown at it. it's
*NOT* just about 32-bit.

this problem can basically be faced REactively... or it can be faced
PROactively: the historic linker strategies that you mention are i
feel going to be needed in some years' time *even for 64-bit*.

l.

[W. Martin Borgert]

Post by Uwe Kleine-König
If the concerns are mostly about the hardware not being rackable, there
https://www.netgear.com/business/products/storage/readynas/RN2120.aspx#tab-techspecs

This seems to be out of stock and discontinued, unfortunately.

Anyway, I'm relatively sure, that I can convince my boss to sponsor/donate
both armel and armhf hardware for Debian, if that is of any help. Or arm64
used in "32 bits mode".

[Roger Shimizu]

Dear armel/armhf shakeholders,

I talked to a few people about keeping armel in buster, during 1st and
2nd day in debcamp.
Seems the blocker is just the buildd server hardware, and memory size it has.

Post by W. Martin Borgert

Post by Uwe Kleine-König
If the concerns are mostly about the hardware not being rackable, there
https://www.netgear.com/business/products/storage/readynas/RN2120.aspx#tab-techspecs

This seems to be out of stock and discontinued, unfortunately.

This is still available in amazon:
- https://www.amazon.com/dp/B00MQK14KC

Post by W. Martin Borgert
Anyway, I'm relatively sure, that I can convince my boss to sponsor/donate
both armel and armhf hardware for Debian, if that is of any help. Or arm64
used in "32 bits mode".

I think DSA team prefers armel or armhf real hardware (not just
developing boards).
So it'll be super great if you (or your boss) can kindly sponsor some
armel/armhf hardwares that support to install 4GB memory.

Thanks!

Cheers,

[John Paul Adrian Glaubitz]

Hi Roger!

Post by Roger Shimizu
I talked to a few people about keeping armel in buster, during 1st and
2nd day in debcamp.
Seems the blocker is just the buildd server hardware, and memory size it has.

According to my colleague Alex Graf at SUSE, you can definitely build
ARMv7 on arm64 using chroots. The only problem with chroots is that
"uname -a" shows "armv8" which some userspace applications are stumbling
over.

openSUSE/SLE builds armv7 packages in OBS/IBS using a fully emulated system
using KVM.

As for the hardware, you should watch out for hardware with ARM Cortex Cores.
Alternatively, X-Gene 1. ThunderX, ThunderX2 and Centriq are definitely not
supported.

Adrian

[Steve McIntyre]

[ Dropped cc to debian-ports etc., switched to debian-arm instead ]

Post by John Paul Adrian Glaubitz
Hi Roger!

Post by Roger Shimizu
I talked to a few people about keeping armel in buster, during 1st and
2nd day in debcamp.
Seems the blocker is just the buildd server hardware, and memory size it has.

According to my colleague Alex Graf at SUSE, you can definitely build
ARMv7 on arm64 using chroots. The only problem with chroots is that
"uname -a" shows "armv8" which some userspace applications are stumbling
over.

And a few other problems - see my mail in the other sub-thread.

Post by John Paul Adrian Glaubitz
openSUSE/SLE builds armv7 packages in OBS/IBS using a fully emulated system
using KVM.

Nod.

Post by John Paul Adrian Glaubitz
As for the hardware, you should watch out for hardware with ARM Cortex Cores.
Alternatively, X-Gene 1. ThunderX, ThunderX2 and Centriq are definitely not
supported.

Agreed on the others, but X-Gene 1 works just fine for A32. Not sure
about later cores...

[John Paul Adrian Glaubitz]

Post by Steve McIntyre
[ Dropped cc to debian-ports etc., switched to debian-arm instead ]

Post by John Paul Adrian Glaubitz
As for the hardware, you should watch out for hardware with ARM Cortex Cores.
Alternatively, X-Gene 1. ThunderX, ThunderX2 and Centriq are definitely not
supported.

Agreed on the others, but X-Gene 1 works just fine for A32. Not sure
about later cores...

I should have phrased more carefully: ARM Cortex and X-Gene 1 are the ones that work, the others don’t according to Alex.

As for the GHC bug, it’s probably a good idea to forward this upstream. Sergei Trofimovich from upstream is extremely good at fixing these things. He has fixed many obscure GHC bugs for Debian Ports architectures.

Adrian

[Luke Kenneth Casson Leighton]

Post by Niels Thykier
------------
* Undesirable to keep the hardware running beyond 2020. armhf VM
support uncertain. (DSA)
- Source: [DSA Sprint report]

[other affected 32-bit architectures removed but still relevant]

... i'm not sure how to put this other than to just ask the question.
has it occurred to anyone to think through the consequences of not
maintaining 32-bit versions of debian for the various different
architectures? there are literally millions of ARM-based tablets and
embedded systems out there which will basically end up in landfill if
a major distro such as debian does not take a stand and push back
against the "well everything's going 64-bit so why should *we*
bother?" meme.

arm64 is particularly inefficient and problematic compared to
aarch32: the change in the instruction set resulted in dropping some
of the more efficiently-encoded instructions that extend a 64-bit
program size, require a whopping FIFTY PERCENT instruction-cache size
increase to compensate for, whch increased power consumption by over
15%.

in addition, arm64 is usually speculative OoO (Cavium ThunderX V1
being a notable exception) which means it's vulnerable to spectre and
meltdown attacks, whereas 32-bit ARM is exclusively in-order. if you
want to GUARANTEE that you've got spectre-immune hardware you need
either any 32-bit system (where even Cortex A7 has virtualisattion) or
if 64-bit is absolutely required use Cortex A53.

basically, abandoning or planning to abandon 32-bit ARM *right now*
leaves security-conscious end-users in a really *really* dicey
position.

Post by Niels Thykier
We are currently unaware of any new architectures likely to be ready in
time for inclusion in buster.

debian-riscv has been repeatedly asking for a single zero-impact line
to be included in *one* file in *one* dpkg-related package which would
allow riscv to stop being a NMU architecture and become part of
debian/unstable (and quickly beyond), for at least six months, now.
cc'ing the debian-riscv list because they will know the details about
this. it's really quite ridiculous that a single one-line change
having absolutely no effect on any other architecture whatsover is not
being actioned and is holding debian-riscv back because of that.

what is the reason why that package is not moving forward?

l.

[Adam D. Barratt]

On Fri, 2018-06-29 at 10:20 +0100, Luke Kenneth Casson Leighton wrote:
[...]

 debian-riscv has been repeatedly asking for a single zero-impact
line
to be included in *one* file in *one* dpkg-related package which would
allow riscv to stop being a NMU architecture and become part of
debian/unstable (and quickly beyond), for at least six months, now.
cc'ing the debian-riscv list because they will know the details about
this.  it's really quite ridiculous that a single one-line change
having absolutely no effect on any other architecture whatsover is not
being actioned and is holding debian-riscv back because of that.
 what is the reason why that package is not moving forward?

I assume you're referring to the dpkg upload that's in proposed-updates
waiting for the point release in two weeks time? Please check your
facts before ranting, particularly with such a wide cross-posting.

Also, ttbomk, the dpkg change landing in stable is not likely to
magically lead to the architecture being added to unstable - a decision
which is not the release team's to make or block. Again, please ensure
you've actually done your research.

I'm also getting very tired of the repeated vilification of SRM over
this, and if there were any doubt can assure you that it is not
increasing at least my inclination to spend my already limited free
time on Debian activity.

Regards,

Adam

[Luke Kenneth Casson Leighton]

---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68


On Fri, Jun 29, 2018 at 10:35 AM, Adam D. Barratt

Post by Adam D. Barratt

Post by Luke Kenneth Casson Leighton
what is the reason why that package is not moving forward?

I assume you're referring to the dpkg upload that's in proposed-updates
waiting for the point release in two weeks time?

i don't know: i'm an outsider who doesn't have the information in
short-term memory, which is why i cc'd the debian-riscv team as they
have current facts and knowledge foremost in their minds. which is
why i included them.

Post by Adam D. Barratt
I'm also getting very tired of the repeated vilification of SRM over
this, and if there were any doubt can assure you that it is not
increasing at least my inclination to spend my already limited free
time on Debian activity.

ah. so what you're saying is, you could really do with some extra help?

l.

[Adam D. Barratt]

On Fri, 2018-06-29 at 11:44 +0100, Luke Kenneth Casson Leighton wrote:
[...]

Post by Luke Kenneth Casson Leighton
On Fri, Jun 29, 2018 at 10:35 AM, Adam D. Barratt

Post by Adam D. Barratt

 what is the reason why that package is not moving forward?

I assume you're referring to the dpkg upload that's in proposed-
updates
waiting for the point release in two weeks time?

 i don't know: i'm an outsider who doesn't have the information in
short-term memory, which is why i cc'd the debian-riscv team as they
have current facts and knowledge foremost in their minds.  which is
why i included them.

It would have been wiser to do so *before* stating that nothing was
happening as if it were a fact.

Post by Luke Kenneth Casson Leighton

Post by Adam D. Barratt
I'm also getting very tired of the repeated vilification of SRM over
this, and if there were any doubt can assure you that it is not
increasing at least my inclination to spend my already limited free
time on Debian activity.

 ah.  so what you're saying is, you could really do with some extra
help?

I don't think that's ever been in dispute for basically any core team
in Debian.

That doesn't change the fact that the atmosphere around the change in
question has made me feel very uncomfortable and unenthused about SRM
work. (I realise that this is somewhat of a self-feeding energy
monster.)

Regards,

Adam

[Luke Kenneth Casson Leighton]

On Fri, Jun 29, 2018 at 12:23 PM, Adam D. Barratt

Post by Adam D. Barratt

Post by Luke Kenneth Casson Leighton
i don't know: i'm an outsider who doesn't have the information in
short-term memory, which is why i cc'd the debian-riscv team as they
have current facts and knowledge foremost in their minds. which is
why i included them.

It would have been wiser to do so *before* stating that nothing was
happening as if it were a fact.

true... apologies.

Post by Adam D. Barratt

Post by Luke Kenneth Casson Leighton
ah. so what you're saying is, you could really do with some extra help?

I don't think that's ever been in dispute for basically any core team
in Debian.

:)

Post by Adam D. Barratt
That doesn't change the fact that the atmosphere around the change in
question has made me feel very uncomfortable and unenthused about SRM
work. (I realise that this is somewhat of a self-feeding energy
monster.)

i hear ya.

l.

[Lennart Sorensen]

Post by Luke Kenneth Casson Leighton
in addition, arm64 is usually speculative OoO (Cavium ThunderX V1
being a notable exception) which means it's vulnerable to spectre and
meltdown attacks, whereas 32-bit ARM is exclusively in-order. if you
want to GUARANTEE that you've got spectre-immune hardware you need
either any 32-bit system (where even Cortex A7 has virtualisattion) or
if 64-bit is absolutely required use Cortex A53.

The Cortex A8, A7 and A5 are in order. The A9, A15, A17 etc are out of
order execution. So any 32 bit arm worth using is pretty much always
out of order execution.

For 64 bit, I think the A35 and A53 are in order while the A57, A72 etc
are out of order.

Of course non Cortex designs vary (I think Marvel's JP4 core was out of
order execution for example).

After all, in general in order execution equals awful performance.

[Geert Uytterhoeven]

Hi Lennart,

debian-ports -> debian-arm

On Fri, Jun 29, 2018 at 5:00 PM Lennart Sorensen

Post by Lennart Sorensen

Post by Luke Kenneth Casson Leighton
in addition, arm64 is usually speculative OoO (Cavium ThunderX V1
being a notable exception) which means it's vulnerable to spectre and
meltdown attacks, whereas 32-bit ARM is exclusively in-order. if you
want to GUARANTEE that you've got spectre-immune hardware you need
either any 32-bit system (where even Cortex A7 has virtualisattion) or
if 64-bit is absolutely required use Cortex A53.

The Cortex A8, A7 and A5 are in order. The A9, A15, A17 etc are out of
order execution. So any 32 bit arm worth using is pretty much always
out of order execution.

Are you sure you're not interchanging A8 and A9, cfr. Linux kernel commit
e388b80288aade31 ("ARM: spectre-v2: add Cortex A8 and A15 validation of the
IBE bit")?

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- ***@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

[Lennart Sorensen]

Post by Geert Uytterhoeven
Are you sure you're not interchanging A8 and A9, cfr. Linux kernel commit
e388b80288aade31 ("ARM: spectre-v2: add Cortex A8 and A15 validation of the
IBE bit")?

Yes. That is the main reason the A9 is faster than the A8 at the same
clock speed by quite a bit.

For example http://www.360doc.com/content/12/0806/14/350555_228637047.shtml says:
Cortex-A9 has many advanced features for a RISC CPU, such as speculative
data accesses, branch prediction, multi-issuing of instructions,
hardware cache coherency, out-of-order execution and register renaming.
Cortex-A8 does not have these, except for dual-issuing instructions and
branch prediction.

So the A8 still has branch prediction so it can keep the pipeline fed,
even with in order execution. Spectre isn't really about out-of-order
excution as much as it is about speculative memory accesses which some
in-order execution chips have too.

[Julien Cristau]

Post by Niels Thykier
Hi,
As part of the interim architecture qualification for buster, we request
that DSA, the security team and the toolchain maintainers review and
update their list of known concerns for buster release architectures.

Everyone, please avoid followups to debian-***@lists.debian.org.
Unless something is relevant to *all* architectures (hint: discussion of
riscv or arm issues don't qualify), keep replies to the appropriate
port-specific mailing list.

Thanks,
Julien

[Luke Kenneth Casson Leighton]

Post by Julien Cristau
Unless something is relevant to *all* architectures (hint: discussion of
riscv or arm issues don't qualify), keep replies to the appropriate
port-specific mailing list.

apologies, julien: as an outsider i'm not completely familiar with
the guidelines. the reduction in memory-usage at the linker phase
"-Wl,--no-keep-memory" however - and the associated inherent
slowly-inexorably-increasing size is i feel definitely something that
affects all ports.

it is really *really* tricky to get any kind of traction *at all*
with people on this. it's not gcc's problem to solve, it's not one
package's problem to solve, it's not any one distros problem to solve,
it's not any one port's problem to solve and so on, *and* it's a
slow-burn problem that's taking *literally* a decade to become more
and more of a problem. consequently getting reports and feedback to
the binutils team is... damn hard.

l.

[YunQiang Su]

Post by Niels Thykier
Hi,
As part of the interim architecture qualification for buster, we request
that DSA, the security team and the toolchain maintainers review and
update their list of known concerns for buster release architectures.
* DSA have announced a blocking issue for armel and armhf (see below)
* Concerns from DSA about ppc64el and s390x have been carried over from
stretch.
* Concerns from the GCC maintainers about armel, armhf, mips, mips64el
and mipsel have been carried over from stretch.
If the issues and concerns from you or your team are not up to date,
Whilst porters remain ultimately responsible for ensuring the
architectures are ready for release, we do expect that you / your team
are willing to assist with clarifications of the concerns and to apply
patches/changes in a timely manner to resolve the concerns.
List of blocking issues by architecture
=======================================
The following is a summary from the current architecture qualification
table.
------------
* Undesirable to keep the hardware running beyond 2020. armhf VM
support uncertain. (DSA)
- Source: [DSA Sprint report]
https://lists.debian.org/debian-project/2018/02/msg00004.html
List of concerns for architectures
==================================
The following is a summary from the current architecture qualification
table.
* Concern for ppc64el and s390x: we are dependent on sponsors for
hardware.
(Raised by DSA; carried over from stretch)
* Concern for armel and armhf: only secondary upstream support in GCC
(Raised by the GCC maintainer; carried over from stretch)
* Concern for mips, mips64el, mipsel and ppc64el: no upstream support
in GCC
(Raised by the GCC maintainer; carried over from stretch)

This is a misunderstanding as MIPS company had some unrest in recent half year.
Currently we are stable now, and the shape of gcc upstream is also good.

Post by Niels Thykier
Architecture status
===================
* Intel/AMD-based: amd64, i386
* ARM-based: arm64, armel, armhf
* MIPS-based: mips, mipsel, mips64el

We are plan to drop mips(eb) and keep mipsel/mips64el.

Post by Niels Thykier
* Other: ppc64el, s390x
If the blocking issues cannot be resolved, affected architectures are at
risk of removal from testing before buster is frozen.
We are currently unaware of any new architectures likely to be ready in
time for inclusion in buster.
On behalf of the release team,
Niels Thykier

[Adrian Bunk]

Post by Niels Thykier
...
------------
* Undesirable to keep the hardware running beyond 2020. armhf VM
support uncertain. (DSA)
...

I'd like to get a clear picture regarding the situation of building
armel for buster on arm64, ideally moving it to arm64 hardwre soon.


1. What issues are considered possible problems for moving building
armel from 32bit v7 hardware to 64bit v8 hardware?

ARM has some history of adding new functionality in new versions of
their architecture that gets deprecated in the next version of their
architecture.

The armel baseline (currently armv5te) is low enough that several
of the issues with running armhf code on arm64 are not present
when running armel code on arm64.

If anyone sees potential blockers for building armel on arm64,
especially ones that are not present for building armhf on arm64,
please speak up now.


2. What level of testing has been done before building armhf on arm64?

64bit arm-arm-01 has started participating in building armhf for
unstable and experimental.

I don't want to do less testing for armel than has been done for armhf
prior to doing that, what testing had been done for armhf on arm64
building prior to doing it on a buildd?


3. Starting to build armel on arm64

Depending on the answers to the questions above I would like to
setup building armel on arm64, perhaps on the same arm-arm-01.


Thanks
Adrian

[Steve McIntyre]

Apologies for delayed response - I've been horrendously busy. :-/

Post by Adrian Bunk

Post by Niels Thykier
...
------------
* Undesirable to keep the hardware running beyond 2020. armhf VM
support uncertain. (DSA)
...

I'd like to get a clear picture regarding the situation of building
armel for buster on arm64, ideally moving it to arm64 hardwre soon.

ACK.

Post by Adrian Bunk
1. What issues are considered possible problems for moving building
armel from 32bit v7 hardware to 64bit v8 hardware?
ARM has some history of adding new functionality in new versions of
their architecture that gets deprecated in the next version of their
architecture.
The armel baseline (currently armv5te) is low enough that several
of the issues with running armhf code on arm64 are not present
when running armel code on arm64.
If anyone sees potential blockers for building armel on arm64,
especially ones that are not present for building armhf on arm64,
please speak up now.

I was worried about CP15 barriers, but I've been digging in docs for a
while and can't find anything to back that up for v5.

Post by Adrian Bunk
2. What level of testing has been done before building armhf on arm64?
64bit arm-arm-01 has started participating in building armhf for
unstable and experimental.
I don't want to do less testing for armel than has been done for armhf
prior to doing that, what testing had been done for armhf on arm64
building prior to doing it on a buildd?

Not very much *so far*. I wsan't expecting to find any issues, but at
least two clear issues have shown up so far:

* Alignment fixups. We have these enabled on our v7 buildds, but
there is no support for this at all in the arcm64 kernel. See
#902990 as an example.

* The definitions for MINSIGSTKSZ differ between armhf and arm64 -
see #904385

To get an idea about these and any other problems, I've started a mass
rebuild of the archive as armhf-on-arm64 on three machines at
home. They're currently ~40% of the way through the archive and I
estimate they are going to take another couple of weeks to complete.

The next big problem I can see is in our haskell packages for
armhf. From my build log for haskell-zxcvbn-c_1.0.1-4.log (as an
example):

...
Setting up llvm-3.9 (1:3.9.1-19+b1) ...
Setting up ghc (8.2.2-4) ...
Illegal instruction

update-alternatives: using /usr/bin/ghc to provide /usr/bin/haskell-compiler (haskell-compiler) in auto mode
Illegal instruction
dpkg: error processing package ghc (--configure):
installed ghc package post-installation script subprocess returned error exit status 132
...

I'm going to have a look at that later today.

Post by Adrian Bunk
3. Starting to build armel on arm64
Depending on the answers to the questions above I would like to
setup building armel on arm64, perhaps on the same arm-arm-01.

Possibly. Let's see how things go - I'm looking at sourcing many more
machines too...

[Adrian Bunk]

Post by Steve McIntyre
...
The next big problem I can see is in our haskell packages for
armhf. From my build log for haskell-zxcvbn-c_1.0.1-4.log (as an
...
Setting up llvm-3.9 (1:3.9.1-19+b1) ...
Setting up ghc (8.2.2-4) ...
Illegal instruction
update-alternatives: using /usr/bin/ghc to provide /usr/bin/haskell-compiler (haskell-compiler) in auto mode
Illegal instruction
installed ghc package post-installation script subprocess returned error exit status 132
...
I'm going to have a look at that later today.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864847#15

Post by Steve McIntyre

Post by Adrian Bunk
3. Starting to build armel on arm64
Depending on the answers to the questions above I would like to
setup building armel on arm64, perhaps on the same arm-arm-01.

Possibly. Let's see how things go - I'm looking at sourcing many more
machines too...

Thanks. :-)

cu
Adrian

[Steve McIntyre]

Post by Adrian Bunk

Post by Steve McIntyre
...
The next big problem I can see is in our haskell packages for
armhf. From my build log for haskell-zxcvbn-c_1.0.1-4.log (as an
...
Setting up llvm-3.9 (1:3.9.1-19+b1) ...
Setting up ghc (8.2.2-4) ...
Illegal instruction
update-alternatives: using /usr/bin/ghc to provide /usr/bin/haskell-compiler (haskell-compiler) in auto mode
Illegal instruction
installed ghc package post-installation script subprocess returned error exit status 132
...
I'm going to have a look at that later today.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864847#15

Aha, thanks for the pointer! :-)

[Christoph Biedl]

Adrian Bunk wrote...

Post by Adrian Bunk
I'd like to get a clear picture regarding the situation of building
armel for buster on arm64, ideally moving it to arm64 hardwre soon.

JFTR, I'd appreciate if armel/armhf could continue to be part of a
release.

Post by Adrian Bunk
1. What issues are considered possible problems for moving building
armel from 32bit v7 hardware to 64bit v8 hardware?

Perhaps just babble and FUD: There was (and probably still is) an issue
in powerpc: In a certain package, upstream's compile options for ppc had
higher CPU requirements than what Debian uses for that architecture. As
a result, the buildd (some big IBM POWER box) happily built the package,
but out there on a G4 the code would crash for SIGILL, same when
rebuilding on such a hardware.

Now I'm somewhat afraid this might happen again when packages for
armel/armhf are built on more recent hardware. At the same time, I'd
like to see continued support for these architectures.

If this is a concern, how to solve it? Have some native non-DSA
armel/armhf boxes where volunteers rebuild the archive and hope test
suites will catch such issues?

My 2¢

Christoph

[Gene Heskett]

Post by Christoph Biedl
Adrian Bunk wrote...

Post by Adrian Bunk
I'd like to get a clear picture regarding the situation of building
armel for buster on arm64, ideally moving it to arm64 hardwre soon.

JFTR, I'd appreciate if armel/armhf could continue to be part of a
release.

I'll second that, I'm using the hf variant on some r-pi 3b's. And theres
millions of those around.

Post by Christoph Biedl

Post by Adrian Bunk
1. What issues are considered possible problems for moving building
armel from 32bit v7 hardware to 64bit v8 hardware?

Perhaps just babble and FUD: There was (and probably still is) an
issue in powerpc: In a certain package, upstream's compile options for
ppc had higher CPU requirements than what Debian uses for that
architecture. As a result, the buildd (some big IBM POWER box) happily
built the package, but out there on a G4 the code would crash for
SIGILL, same when rebuilding on such a hardware.
Now I'm somewhat afraid this might happen again when packages for
armel/armhf are built on more recent hardware. At the same time, I'd
like to see continued support for these architectures.
If this is a concern, how to solve it? Have some native non-DSA
armel/armhf boxes where volunteers rebuild the archive and hope test
suites will catch such issues?
My 2¢
Christoph

[John Paul Adrian Glaubitz]

Post by Christoph Biedl
Perhaps just babble and FUD: There was (and probably still is) an issue
in powerpc: In a certain package, upstream's compile options for ppc had
higher CPU requirements than what Debian uses for that architecture. As
a result, the buildd (some big IBM POWER box) happily built the package,
but out there on a G4 the code would crash for SIGILL, same when
rebuilding on such a hardware.

If there are any packages with such an issue, it would be very helpful
if you could open bug reports with the following headers:

User: debian-***@lists.debian.org
Usertags: powerpc

Post by Christoph Biedl
Now I'm somewhat afraid this might happen again when packages for
armel/armhf are built on more recent hardware. At the same time, I'd
like to see continued support for these architectures.

Not really. Unless Matthias Klose makes any changes to the baseline
configuration for gcc, this won't happen. Also, the evolution of
ARMv7 (Debian's armhf) would be ARMv8 and that wouldn't work inside
a 32-bit chroot or KVM environment. So, no, I don't think there is
any risk to run into this problem.

On PowerPC, the issue is usually related to AltiVec or the variations
of 64-Bit PowerPC. There are PowerPC variants by FreeScale that are
64 bit but don't support Altivec, for example. But again, if there
is any package which is affected by this problem, please let us know.

Oh, and in case of FPU instructions, you should make sure your kernel
has math emulation ebaled.

Post by Christoph Biedl
If this is a concern, how to solve it? Have some native non-DSA
armel/armhf boxes where volunteers rebuild the archive and hope test
suites will catch such issues?

It's not a concern on ARM32 for the aforementioned reasons.

Adrian

[Adrian Bunk]

Post by Christoph Biedl
Adrian Bunk wrote...

Post by Adrian Bunk
I'd like to get a clear picture regarding the situation of building
armel for buster on arm64, ideally moving it to arm64 hardwre soon.

JFTR, I'd appreciate if armel/armhf could continue to be part of a
release.

Post by Adrian Bunk
1. What issues are considered possible problems for moving building
armel from 32bit v7 hardware to 64bit v8 hardware?

Perhaps just babble and FUD: There was (and probably still is) an issue
in powerpc: In a certain package, upstream's compile options for ppc had
higher CPU requirements than what Debian uses for that architecture. As
a result, the buildd (some big IBM POWER box) happily built the package,
but out there on a G4 the code would crash for SIGILL, same when
rebuilding on such a hardware.
Now I'm somewhat afraid this might happen again when packages for
armel/armhf are built on more recent hardware. At the same time, I'd
like to see continued support for these architectures.
If this is a concern, how to solve it? Have some native non-DSA
armel/armhf boxes where volunteers rebuild the archive and hope test
suites will catch such issues?

This is not really a concern for a v7 -> v8 build change for armel.
stretch has a v4 baseline and v7 buildds.
buster has a v5 baseline.
I am not saying that baseline violations don't happen on armel (they do),
just that this change is unlikely to make a difference here.

For armhf this is actually a valid concern, especially regarding NEON.
NEON is not in the armhf baseline, and since our current buildd hardware
does not support NEON this catches plenty unconditional NEON usage.
One option for mitigating this problem would be to repurpose one
or more of the current buildd machines (there is even a spare)
for debci.
(Trying to use NEON on armel will give a compile error in any case.)

Post by Christoph Biedl
My 2¢
Christoph

cu
Adrian

[Adam Borowski]

Post by Christoph Biedl
If this is a concern, how to solve it? Have some native non-DSA
armel/armhf boxes where volunteers rebuild the archive and hope test
suites will catch such issues?

The problem is not doing the rebuilds, it's tuits to actually analyze and
report bugs. There are multiple such efforts: reproducible builds use a
diverse bunch of hardware, I run a single Odroid-U2:
(arch:any) https://angband.pl/deb/rebuild.pl
(arch:all) https://angband.pl/deb/rebuild-all.pl
yet didn't have the tuits to go through them in several months.

Also, this machine does have neon so it's not even armhf baseline. And so
many packages compile but don't test. Thus, regressions from building on
arm64 need not just hardware but also manpowers to detect.


Meow.

[John Paul Adrian Glaubitz]

Post by Adam Borowski
Also, this machine does have neon so it's not even armhf baseline. And so
many packages compile but don't test. Thus, regressions from building on
arm64 need not just hardware but also manpowers to detect.

But why should compiling ARMv7 on ARMv8 automatically change the baseline when it’s actually a hardwired configure option in gcc?

By the same logic, lots of the packages we build on ARMv7 machines for armel wouldn’t work on ARMv5.

Plus, we can build on the experience that openSUSE made with building ARMv6/7 on ARMv8. Why are we ignoring that?

Adrian

[Adam Borowski]

Post by John Paul Adrian Glaubitz

Post by Adam Borowski
Also, this machine does have neon so it's not even armhf baseline. And so
many packages compile but don't test. Thus, regressions from building on
arm64 need not just hardware but also manpowers to detect.

But why should compiling ARMv7 on ARMv8 automatically change the baseline
when it’s actually a hardwired configure option in gcc?

There's way too many packages that do compile-time detection.

Post by John Paul Adrian Glaubitz
By the same logic, lots of the packages we build on ARMv7 machines for
armel wouldn’t work on ARMv5.

And many probably don't, but such gear is so weak that a good part of
packages simply have no one running them.

Post by John Paul Adrian Glaubitz
Plus, we can build on the experience that openSUSE made with building
ARMv6/7 on ARMv8. Why are we ignoring that?

How do you propose to do that other than sometimes digging through their
packaging for a patch here and there?


ᛗᛖᛟᚹ

[John Paul Adrian Glaubitz]

Post by Adam Borowski

Post by John Paul Adrian Glaubitz

Post by Adam Borowski
Also, this machine does have neon so it's not even armhf baseline. And so
many packages compile but don't test. Thus, regressions from building on
arm64 need not just hardware but also manpowers to detect.

But why should compiling ARMv7 on ARMv8 automatically change the baseline
when it’s actually a hardwired configure option in gcc?

There's way too many packages that do compile-time detection.

Why has it never been an issue with PowerPC on PowerPC64 kernels, SPARC
on SPARC64 kernels, ARMv5 on ARMv7 kernels, MIPSEL on MIPS64EL kernels,
i386 on amd64 kernels and so on. It's not the first time at all that
we're building with a 32-bit userland on 64-bit kernels.

Post by Adam Borowski

Post by John Paul Adrian Glaubitz
By the same logic, lots of the packages we build on ARMv7 machines for
armel wouldn’t work on ARMv5.

And many probably don't, but such gear is so weak that a good part of
packages simply have no one running them.

I disagree. Previous experience with other architectures as shown above
shows that it does work.

Post by Adam Borowski

Post by John Paul Adrian Glaubitz
Plus, we can build on the experience that openSUSE made with building
ARMv6/7 on ARMv8. Why are we ignoring that?

How do you propose to do that other than sometimes digging through their
packaging for a patch here and there?

SUSE is upstreaming usually everything. Also, I talked to Alex Graf who
is in charge of these things at SUSE and he confirmed me that building
of ARMv7 on ARMv8 is possible. He's also the one who fixed QEMU-KVM
issues with ARMv7 on ARMv8.

Oh, and SUSE also does extensive testing in OpenQA to make sure the
compiled packages actually work. So, I really think we can be confident
that building 32-bit ARM packages on 64-bit machines will actually
work the same way it does and did in the past for other architectures.

If you find a counter-example, I'd be very interested to learn about it,
I love finding new bugs after all :-).

Adrian

[Hideki Yamane]

Hi,

Post by Roger Shimizu
So it'll be super great if you (or your boss) can kindly sponsor some
armel/armhf hardwares that support to install 4GB memory.

From DebConf18 seesion "Building a Debian Derivative: Lessons Learned
and Solutions Found" by Alex Doyle from Cumulus Network, they run
"Jessie on Chromebook" as armel build server

Maybe we can do same thing for armel, it'd be better than using old
NAS with larger memory (I know we're trying to build armel/hf packages
on arm64 but just a FYI).

[Roger Shimizu]

Post by Hideki Yamane
Hi,

Post by Roger Shimizu
So it'll be super great if you (or your boss) can kindly sponsor some
armel/armhf hardwares that support to install 4GB memory.

From DebConf18 seesion "Building a Debian Derivative: Lessons Learned
and Solutions Found" by Alex Doyle from Cumulus Network, they run
"Jessie on Chromebook" as armel build server
http://youtu.be/OPsfX5_YCiQ

Thanks for your info!

Post by Hideki Yamane
Maybe we can do same thing for armel, it'd be better than using old
NAS with larger memory (I know we're trying to build armel/hf packages
on arm64 but just a FYI).

Yes. In latest ARM BoF of DebConf18, Steve (93sam) informed us that
he's trying to rebuild all the armhf archive on arm64 box.
When that's done, we will get to know how many packages need to fix.
If the number of those packages is limited and can be fixed before
buster, we can choose this way for both armhf and armel.
But if the number is huge that we don't have enough manpower to fix,
we need to find real machine (such as rack-mounted armhf/armel NAS
box) to work as buildd.

Looking forward to getting the news update from Steve.

Cheers,

[Adrian Bunk]

Post by Roger Shimizu
...
Yes. In latest ARM BoF of DebConf18, Steve (93sam) informed us that
he's trying to rebuild all the armhf archive on arm64 box.
When that's done, we will get to know how many packages need to fix.
If the number of those packages is limited and can be fixed before
buster, we can choose this way for both armhf and armel.

Note that armel and armhf will have different problem when building
on arm64.

The armel baseline is low enough that some of the issues with running
armhf code on arm64 are not present when running armel code on arm64.

But armel might have different (currently unknown) issues.

Post by Roger Shimizu
But if the number is huge that we don't have enough manpower to fix,
we need to find real machine (such as rack-mounted armhf/armel NAS
box) to work as buildd.
...

There are two problems here:

The first is the remote administration side.

Rack-mounted might not even be necessary, but it must be possible to
remotely reset a hung box without a human going to the machine to press
a button.

The second are the hardware specs.

The current armel/armhf buildds each have 4x 1.6 GHz CPUs [1]
and 4 GM RAM.

Both cpu power and amount of RAM are at the lower side of what is
reasonable for a release architecture, and we cannot regress on
either of that.

We need full support in the upstream kernel.

And it has to be hardware we can rely on until mid-2022.

cu
Adrian

[1] these are Marvell PJ4, comparable speed at same frequency
should be approximately Cortex-A9

[peter green]

Post by Roger Shimizu

Post by Hideki Yamane
Hi,

Post by Roger Shimizu
So it'll be super great if you (or your boss) can kindly sponsor some
armel/armhf hardwares that support to install 4GB memory.

From DebConf18 seesion "Building a Debian Derivative: Lessons Learned
and Solutions Found" by Alex Doyle from Cumulus Network, they run
"Jessie on Chromebook" as armel build server
http://youtu.be/OPsfX5_YCiQ

Thanks for your info!

Post by Hideki Yamane
Maybe we can do same thing for armel, it'd be better than using old
NAS with larger memory (I know we're trying to build armel/hf packages
on arm64 but just a FYI).

Yes. In latest ARM BoF of DebConf18, Steve (93sam) informed us that
he's trying to rebuild all the armhf archive on arm64 box.
When that's done, we will get to know how many packages need to fix.
If the number of those packages is limited and can be fixed before
buster, we can choose this way for both armhf and armel.
But if the number is huge that we don't have enough manpower to fix,
we need to find real machine (such as rack-mounted armhf/armel NAS
box) to work as buildd.

Personal opinion but (as a derivative maintainer) I really think
DSA are being unreasonable here. A buildd does not need to be
some critical 100% uptime server, if it occasionally goes down
to the point where it can't be recovered remotely then so be it.

Arm ports aside I don't think having "server class" hardware
available should be a blocker for getting or maintaining a port
in Debian. It never has been in the past and I don't see what
has changed now.

And I think with a serial console and remote power most issues
can be resolved remotely. We have been running wandboard
quads for raspbian for years and I have never hard a situation
I couldn't get out of with remote power and serial console.

On the subject of building on arm64 hardware we now do this
for some of our builds in raspbian. It *mostly* works but we
do see some testsuite failures that we don't see on the
wandboards (otoh we have some testsuites that pass on the
arm64 box but not the wandboards, go figure) and I have
also been having a build hang with rust recently, but other
than that I haven't noticed any prblems.