Commit 654e1392 authored by Michael Pyne's avatar Michael Pyne
Browse files

rc-file: Add "num-cores-low-mem" and move num-cores* defs to initial-setup.

It is annoying to be maintaining kdesrc-build-setup and --initial-setup
but combining those two can be a subsequent refactoring.

This adds a separate num-cores-low-mem to address the qtwebengine case
and makes both num-cores and num-cores-low-mem into an option generated
during initial setup, and only used by the default config (rather than
any part of the kdesrc-build internals directly).

There is a fallback reference in the code in case there is a usage of
the default configuration file sections (e.g. qt5-build-include) but
this is set conservatively (4 cores, 2 cores during low-mem).

At this point it's almost "just" a configuration convention with a bit
of code in the setup wizard so perhaps it's best not to touch the rest
of the code/docs at all, but I'm happy with where this is at.

I've tested --initial-setup in a Docker container (Fedora 29 with perl
manually installed first) and tested kdesrc-build-setup separately.
parent 27505b67
Pipeline #26801 passed with stage
in 2 minutes and 21 seconds
......@@ -2628,7 +2628,8 @@ due to fixes in the underlying build system.</entry>
<entry>num-cores</entry>
<entry>Cannot be overridden</entry>
<entry>
<para>This option is automatically set by &kdesrc-build; to the number of
<para>This option is defined by &kdesrc-build; (when using the kdesrc-build-setup tool
or <command>kdesrc-build --initial-setup</command>), set to be the number of
available CPUs (as indicated by the external application
<application>nproc</application>). If &kdesrc-build; cannot detect the
number of CPUs, this value is set to 4.</para>
......@@ -2638,6 +2639,31 @@ option's usage. This option was added in version 20.07.</para>
</entry>
</row>
<row id="conf-num-cores-low-mem">
<entry>num-cores-low-mem</entry>
<entry>Cannot be overridden</entry>
<entry>
<para>This option is defined by &kdesrc-build; (when using the kdesrc-build-setup tool
or <command>kdesrc-build --initial-setup</command>), set to be the number of
CPUs that is deemed safe for heavyweight or other highly-intensive modules,
such as <literal>qtwebengine</literal>, to avoid running out of memory
during the build.</para>
<para>The typical calculation is one CPU core for every 2
gigabytes (GiB) of total memory. At least 1 core will be specified,
and no more than <option><link linkend="conf-num-cores">num-cores</link></option>
cores will be specified.</para>
<para>Although this option is intended to support &Qt; modules, you can use it for your
any module in the same way that <option>num-cores</option> is used.</para>
<para>If &kdesrc-build; cannot detect available memory then this value will be
set to 2.</para>
<para>This option was added in version 20.07.</para>
</entry>
</row>
<row id="conf-override-build-system">
<entry>override-build-system</entry>
<entry>Module setting overrides global</entry>
......
......@@ -26,6 +26,7 @@ use File::Copy;
use File::Temp qw/tempfile/;
use File::Basename;
use Cwd qw(abs_path);
use List::Util qw(max min first);
our $VERSION = 0.03; # Not user-visible yet.
......@@ -348,6 +349,25 @@ if (grep /^qt5$/, @chosenModules) {
EOF
}
chomp(my $num_cores = `nproc`);
$num_cores ||= 4;
# Try to detect the amount of total memory for a corresponding option for
# heavyweight modules (sorry ade, not sure what's needed for FreeBSD!)
my $mem_total;
my $total_mem_line = first { /MemTotal/ } (`cat /proc/meminfo`);
if ($total_mem_line && $? == 0) {
($mem_total) = ($total_mem_line =~ /^MemTotal:\s*([0-9]+) /); # Value in KiB
$mem_total = int $mem_total;
}
# 4 GiB is assumed if no info on memory is available, as this will
# calculate to 2 cores. sprintf is used since there's no Perl round function
my $rounded_mem = $mem_total ? (int sprintf("%.0f", $mem_total / 1024000.0)) : 4;
my $max_cores_for_mem = max(1, int $rounded_mem / 2); # Assume 2 GiB per core
my $num_cores_low = min($max_cores_for_mem, $num_cores);
print $output <<EOF;
# Finds and includes *KDE*-based dependencies into the build. This makes
......@@ -366,6 +386,18 @@ print $output <<EOF;
# relative to source-dir by default
build-dir $buildDir
## kdesrc-build sets 2 options which you can use in options like make-options or set-env
# to help manage the number of compile jobs that # happen during a build:
#
# 1. num-cores, which is just the number of detected CPU cores, and can be passed
# to tools like make (needed for parallel build) or ninja (completely optional).
#
# 2. num-cores-low-mem, which is set to largest value that appears safe for
# particularly heavyweight modules based on total memory, intended for
# modules like qtwebengine
num-cores $num_cores
num-cores-low-mem $num_cores_low
# Use multiple cores for building. Other options to GNU make may also be
# set.
make-options -j \${num-cores}
......
......@@ -14,6 +14,15 @@ global
# logs will be kept under this directory as well.
source-dir ~/kde/src
# These values should be set to the number of cores to use during build (if
# in doubt, run "nproc" to see how many cores you have)
num-cores 4
# This is the same as above but used for heavyweight modules like
# qtwebengine, though you can use it for modules yourself. A conservative
# thumbrule is one core for every 2 GiB of total memory.
num-cores-low-mem 2
make-options -j ${num-cores}
end global
......
......@@ -100,7 +100,9 @@ our %defaultGlobalOptions = (
"branch" => "",
"branch-group" => "", # Overrides branch, uses JSON data.
"build-dir" => "build",
"cmake-generator" => "",
"cmake-options" => "",
"cmake-toolchain" => "",
"configure-flags" => "",
"custom-build-command" => '',
"cxxflags" => "-pipe",
......@@ -114,6 +116,9 @@ our %defaultGlobalOptions = (
"make-install-prefix" => "", # Some people need sudo
"make-options" => "",
"module-base-path" => "", # Used for tags and branches
"ninja-options" => "",
"num-cores" => 4, # Used only in rc-file but documented
"num-cores-low-mem" => 2, # Used only in rc-file but documented
"override-build-system"=> "",
"override-url" => "",
"persistent-data-file" => "",
......@@ -172,10 +177,6 @@ sub new
assert_isa($self, 'ksb::Module');
assert_isa($self, 'ksb::BuildContext');
# Make the number of CPUs available to the rc-file by turning it into a pre-set option
my $nproc = int (eval { (filter_program_output(undef, 'nproc'))[0] } // 3) + 1;
$self->setOption('num-cores', $nproc);
return $self;
}
......
......@@ -4,6 +4,7 @@ use 5.014;
use strict;
use warnings;
use File::Spec qw(splitpath);
use List::Util qw(min max first);
use ksb::BuildException;
use ksb::Debug qw(colorize);
......@@ -126,6 +127,31 @@ DONE
}
}
# Return the highest number of cores we can use based on available memory. Requires
# the number of cores we have available
sub _getNumCoresForLowMemory
{
my $num_cores = shift;
# Try to detect the amount of total memory for a corresponding option for
# heavyweight modules (sorry ade, not sure what's needed for FreeBSD!)
my $mem_total;
my $total_mem_line = first { /MemTotal/ } (`cat /proc/meminfo`);
if ($total_mem_line && $? == 0) {
($mem_total) = ($total_mem_line =~ /^MemTotal:\s*([0-9]+) /); # Value in KiB
$mem_total = int $mem_total;
}
# 4 GiB is assumed if no info on memory is available, as this will
# calculate to 2 cores. sprintf is used since there's no Perl round function
my $rounded_mem = $mem_total ? (int sprintf("%.0f", $mem_total / 1024000.0)) : 4;
my $max_cores_for_mem = max(1, int $rounded_mem / 2); # Assume 2 GiB per core
my $num_cores_low = min($max_cores_for_mem, $num_cores);
return $num_cores_low;
}
sub _setupBaseConfiguration
{
my $baseDir = shift;
......@@ -142,6 +168,11 @@ DONE
my $sampleRc = $packages{'sample-rc'} or
_throw("Embedded sample file missing!");
my $numCores = `nproc 2>/dev/null` || 4;
my $numCoresLow = _getNumCoresForLowMemory($numCores);
$sampleRc =~ s/%\{num_cores}/$numCores/g;
$sampleRc =~ s/%\{num_cores_low}/$numCoresLow/g;
$sampleRc =~ s/%\{base_dir}/$baseDir/g;
open my $sampleFh, '>', "$ENV{HOME}/.kdesrc-buildrc"
......@@ -527,6 +558,19 @@ global
include-dependencies true
cmake-options -DCMAKE_BUILD_TYPE=RelWithDebInfo
# kdesrc-build sets 2 options which you can use in options like make-options or set-env
# to help manage the number of compile jobs that # happen during a build:
#
# 1. num-cores, which is just the number of detected CPU cores, and can be passed
# to tools like make (needed for parallel build) or ninja (completely optional).
#
# 2. num-cores-low-mem, which is set to largest value that appears safe for
# particularly heavyweight modules based on total memory, intended for
# modules like qtwebengine
num-cores %{num_cores}
num-cores-low-mem %{num_cores_low}
make-options -j ${num-cores}
end global
......@@ -546,7 +590,4 @@ include %{base_dir}/custom-qt5-libs-build-include
include %{base_dir}/kf5-qt5-build-include
# To change options for modules that have already been defined, use an
# 'options' block
options kcoreaddons
make-options -j4
end options
# 'options' block. See qt5-build-include for an example
......@@ -10,6 +10,8 @@ module-set qt5-set
qttools qtwayland qtwebchannel qtwebsockets qtwebview qtx11extras \
qtnetworkauth qtspeech qtxmlpatterns
# qtwebengine is very different to the rest of Qt. You can try ignoring it if
# you cannot get it to compile by uncommenting the next line.
# ignore-modules qtwebengine
# install path. This *MUST* match your qtdir setting in kdesrc-buildrc!
......@@ -23,11 +25,12 @@ module-set qt5-set
end module-set
# qtwebengine is essentially the Chromium Embedded Framework with Qt bindings
# and has source code of unusually large complexity for the compiler.
# TL;DR: This will eat a *ton* of RAM and can lockup your system if you have a
# lot of CPU cores. qtwebengine is disabled by default but if you enable it
# also ensure you don't outstrip your available RAM with too high of a
# parallelism (-j flag).
# It has unusually complex source codes which require a lot of memory to compile..
#
# This module will eat a *ton* of RAM and can lockup your system if you have a
# lot of CPU cores. qtwebengine uses a lower number of cores to compile by
# default but you can change the setting to -j here (or in your global options)
# to change that.
options qtwebengine
set-env NINJAFLAGS -j4
set-env NINJAFLAGS -j${num-cores-low-mem}
end options
......@@ -39,7 +39,7 @@ syn keyword ksbrcOption contained skipwhite nextgroup=ksbrcStringValue
\ binpath branch build-dir checkout-only cmake-options configure-flags
\ custom-build-command cxxflags dest-dir do-not-compile kdedir
\ libpath log-dir make-install-prefix make-options module-base-path
\ ninja-options
\ cmake-generator cmake-toolchain ninja-options
\ override-build-system override-url prefix qtdir repository
\ revision source-dir svn-server tag remove-after-install
\ qmake-options git-user
......@@ -47,6 +47,7 @@ syn keyword ksbrcOption contained skipwhite nextgroup=ksbrcStringValue
syn keyword ksbrcGlobalOption contained skipwhite nextgroup=ksbrcStringValue
\ branch-group git-desired-protocol git-repository-base http-proxy
\ kde-languages niceness debug-level persistent-data-file set-env
\ num-cores num-cores-low-mem
" MUST BE CONSISTENT WITH ABOVE. Used when a module-set option is used in the
" wrong spot to highlight the error.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment