Commit efc3b427 authored by Michael Pyne's avatar Michael Pyne

Merge remote-tracking branch 'gitlab/master' into make_it_mojo

Mostly a clean merge despite the conflicts, but the test suite showed
that there's still a lot of work to do around cmdline handling with the
mojo-based setup. We can't create a whole web server each time just to
handle --version...

parents d72cf7e2 26c9fc5f
"phabricator.uri" : ""
......@@ -3,7 +3,7 @@ project(kdesrc-build)
# Needed for the docs, but optional. If not found, we will simply
# defer to native CMake handling.
find_package(ECM 5.10.0 NO_MODULE)
find_package(ECM 5.10.0 NO_MODULE QUIET)
......@@ -39,6 +39,7 @@ if (KDESRC_BUILD_INSTALL_MODULES)
# kdesrc-build
1. Install kdesrc-build:
- Copy the kdesrc-build directory (i.e. the directory holding this README)
somewhere where you will be able to find it. The whole directory has to
go together, so don't just copy the script itself.
- Alternately, you can run the `git` command itself to easily install
$ mkdir -p ~/kdesrc
$ cd ~/kdesrc
$ git clone
$ cd kdesrc-build # kdesrc-build is in this directory
2. Setup:
- **For KDE 4 Platform and Plasma Desktop**: If you don't already have a
~/.kdesrc-buildrc, use the kdesrc-build-setup program to make one, or
copy kdesrc-buildrc-sample to ~/.kdesrc-buildrc and make any changes that
you need.
- **For KDE Frameworks ("KF5") / Plasma 5**: Please take a look at
`kdesrc-build-kf5-sample` in the kdesrc-build directory,
which gives a starting point example for a suitable ~/.kdesrc-buildrc.
- the `kdesrc-buildrc-sample` file in the kdesrc-build directory
gives a few hints at what options are available. Also, look at the
documentation at
3. From the kdesrc-build directory, run "./kdesrc-build --metadata-only". This
will download some data that describes the KDE source repositories and their
dependencies, which will help kdesrc-build figure out what to build.
4. Now you can run "kdesrc-build --pretend" to have kdesrc-build go through the
steps that it would perform, but without actually doing them. This is a good
time to make sure that kdesrc-build is set the way you want it.
5. When you're ready, run "./kdesrc-build". Hopefully everything will go well
the first time, and kdesrc-build will be able to download and build all of
the modules that you ask for. :)
Upgrading is simple.
You can delete your old kdesrc-build directory (make sure you don't have any
local changes in there first, or your kdesrc-buildrc file!) and then install
the new version where the old kdesrc-build directory used to be.
In fact, it is recommended to use git to update kdesrc-build itself, so that
kdesrc-build is kept upgraded automatically. This is mostly done in the sample
configuration for KF5, where kdesrc-build is configured to update itself,
however for the best success you should remember to add the kdesrc-build git
repository to your PATH environment variable.
One thing to keep in mind when using kdesrc-build to manage keeping itself
up to date is that updates won't take effect until the *next* time you run
You may want to edit your ~/.kdesrc-buildrc to make sure any new
options are included. You should always read the changes for the new version
however, as sometimes there are slight changes in behavior necessary to adapt
to updates in the source repository. If you are running kdesrc-build from its
git repository, you can use the "git log" command from inside the kdesrc-build
source directory to see the latest changes.
You can use the `./kdesrc-build --version` command to ensure that you have
successfully upgraded kdesrc-build.
A sample configuration file is included. It should work for you for the most
part with only minor tweaking. To use it, just copy it to ~/.kdesrc-buildrc.
Alternatively, a program is included called kdesrc-build-setup, which can be
used to generate a simple configuration (for KDE4-based builds).
## HELP!!!
kdesrc-build includes a limited command-line description with the
--help option.
You can read the kdesrc-build handbook online at
If you have already setup a KDE build environment then kdesrc-build itself can
have its documentation compiled and installed.
In this case the documentation is available in Konqueror (help:/kdesrc-build)
or KHelpCenter. Additionally a man page will be installed, or you can run "man
./kdesrc-build.1" in the "doc/" directory of kdesrc-build's own build
You can also ask for help online on the #kde-devel channel of IRC (
Additionally you can ask for help on the KDE support mailing lists, such as
Finally you can drop me an email at (although I have a job/family
and therefore don't always have time to respond)
You can find updated versions of kdesrc-build online as part of the
extragear/utils module: (You can also see the change log this way)
In fact, it is unlikely that future releases will be made. Though new versions
will be tagged, they will not be released as source tarballs. You should run
from git (it is likely a 'stable' branch will be setup to make this more
If you find a bug, please report it at:
If you have any questions, feel free to let me know:
Michael Pyne <>
This diff is collapsed.
......@@ -4,3 +4,12 @@ kdoctools_create_manpage(man-kdesrc-build.1.docbook 1 INSTALL_DESTINATION ${MAN_
kdoctools_create_manpage(man-kdesrc-build-setup.1.docbook 1 INSTALL_DESTINATION ${MAN_INSTALL_DIR})
install(PROGRAMS kdesrc-build.desktop DESTINATION ${XDG_APPS_INSTALL_DIR})
# Look for asciidoctor for source reference
find_program(ASCIIDOCTOR_PATH asciidoctor)
message(STATUS "Could not find asciidoctor, will not build developer source reference")
......@@ -2961,7 +2961,7 @@ Only display simple help on this script.
<varlistentry id="cmdline-version">
<term><parameter>--version</parameter> (or <parameter>-v</parameter>)</term>
Display the program version.
......@@ -2977,6 +2977,24 @@ prove useful in bug reports or when asking for help in forums or mailing lists.
<varlistentry id="cmdline-initial-setup">
Has &kdesrc-build; perform the one-time initial setup necessary to prepare
the system for &kdesrc-build; to operate, and for the newly-installed &kde;
software to run.
<para>This includes:</para>
<listitem><para>Installing known dependencies (on supported &Linux; distributions)</para></listitem>
<listitem><para>Adding required environment variables to <literal>~/.bashrc</literal></para></listitem>
<listitem><para>Setting up a <link linkend="setup-rcfile">configuration file</link></para></listitem>
<para>Available since version 18.11.</para>
<varlistentry id="cmdline-author">
......@@ -3057,7 +3075,7 @@ Only output warnings and errors.
<varlistentry id="cmdline-verbose">
<term><parameter>--verbose</parameter> (or <parameter>-v</parameter>)</term>
Be very descriptive about what is going on, and what &kdesrc-build; is doing.
# KDE Config File
[Desktop Entry]
Name=KDE Source Builder
Name[bs]=KDE graditelj izvornog koda
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE refentry PUBLIC "-//KDE//DTD DocBook XML V4.5-Based Variant V1.1//EN" "dtd/kdedbx45.dtd" [
<!ENTITY % English "INCLUDE">
<!ENTITY kdesrc-build "<application>kdesrc-build</application>">
......@@ -122,6 +123,21 @@ combining short options into one at this point. (E.g. running
Performs one-time setup for users running &kdesrc-build; on common
distributions. This includes installation of known system dependencies, a
default configuration file setup, and changes to your ~/.bashrc to make the
software installed by &kdesrc-build; accessible.
<option>-p, --pretend</option>
......@@ -732,7 +748,36 @@ kdepim: master
<option>-v, --verbose</option>
Have <command>kdesrc-build</command> start the build process for a module
as soon as the source code has finished downloading. Without this option
<command>kdesrc-build</command> performs all source updates at once and
only then starts with the build process. This option is enabled by default.
Disables asynchronous building of modules. See <option>--async</option> for
a more detailed description. Note that <command>kdesrc-build</command>'s
output will be slightly different in this mode.
# Disable use of external resources by default in the stylesheet
SET(ASCIIDOCTOR_OPTS -a 'webfonts!')
# The most proper way to do this is to have each *.html file individually
# depend upon its *.adoc file, but asciidoctor is quick enough to just
# re-build everything each time any of those files change.
# Note: writes to source directory by default, not build directory!
# Use "git clean -dfx" to easily completely clean a source directory.
I'm working (again) on trying to document some of what this code actually does.
I've really tried hard to get used to Perl's POD format but it's not happening.
NaturalDocs didn't work well either.
I actually switched to Sphinx but it's pretty difficult to have it hyperlink to
(or even understand) Perl code, and the syntax is more annoying than I'd
expected of rST.
So right now I'm going for AsciiDoc, using the AsciiDoctor software (since
AsciiDoc itself carries a 700MB install cost even for its minimal 'base'
release, at least on Ubuntu).
Until I add that into the build chain for kdesrc-build, it should be as simple
as installing it (sudo apt-get install asciidoctor) and then running
"asciidoctor index.adoc ksb/*.adoc" for now.
= kdesrc-build documentation:
Michael Pyne <>
v18.12, 2018-11-22
kdesrc-build is intended to build KDE-based software from its source repository, although it can build
other types of software from its native source control system(s) as well.
This documentation is intended for developers to aid in hacking on kdesrc-build itself, or porting the same concepts
to other build systems if necessary.
== Concepts
Some basic concepts are assumed throughout for brevity.
=== Modules
kdesrc-build uses the "module" as the most granular level of buildable
software. Each module has a name, unique in the list of all modules.
Each module can have a specific source control system plugin (Git,
KDE's git, Subversion, etc.) and a build system plugin (qmake, CMake, KDE's
CMake, autotools, etc.)
=== Build Phases
A module's build process progresses through build phases, which are often
The normal progression is:
. Update
. Uninstall (not normally seen)
. Build system setup and configuration
. Build
. Testsuite (if enabled)
. Install
The update phase can happen concurrently with other modules' build/install
phases, under the theory that the build phase is usually CPU-heavy so it makes
sense to start on subsequent network (IO-heavy) updates while the build
=== Build Context
To group global settings and status together that exist across individual
modules, a "build context" is used, shared across the entire application.
Each module can refer to the global build context.
=== Configuration file (rc-file)
kdesrc-build uses a configuration file (usually abbreviated the `+rc-file+`) to
. The list of modules to build, and
. The dependency order in which to build modules (the order seen in the rc-file), and
. The build or configuration options to use by default or on a per-module
The configuration file is used from `+kdesrc-buildrc+` in the current directory
when kdesrc-build is run, if the file is present. If not, the global rc-file at
`+~/.kdesrc-buildrc+` is used instead.
=== Command line
kdesrc-build uses the command line (seen as "cmdline" in the source and commit
logs) to override the list of modules to build (nearly always still requiring
that any modules built are visible from the rc-file). The command line is also
used to override various options (such as skipping source update phases),
control output verbosity and so on.
In theory every option in the rc-file can be set from the cmdline, and cmdline
entries override and mask any options used by default or read from an rc-file.
=== Module Sets
With the adoption of git, KDE exploded to having hundreds of repositories. It
would be annoying and error-prone to try to manually update the rc-file with
the list of modules to build and the proper ordering.
Because of this, kdesrc-build supports grouping modules into "module sets" of
modules that have common options and a common repository URL prefix, as if the
user had manually entered those modules one by one.
NOTE: This is controlled by the `+git-repository-base+` option to set the URL
prefix, the `+repository+` option to choose one of the defined bases, and the
`+use-modules+` option to list module names.
==== KDE module sets
To support the KDE repositories in particular, a special module set repository
is defined, `+kde-projects+`. Use of this repository enables some extra magic
in the modules that are ultimately defined from such a module set, including
automagic dependency handling and inclusion of modules based on a virtual KDE
project structure.
NOTE: Inclusion of modules is **separate** from dependency handling, which is
also supported!
=== Pretend mode
The user can pass a `+--pretend+` cmdline flag to have kdesrc-build not
actually undertake the more time or resource intensive actions, so that the
user can see what kdesrc-build would do and tweak their cmdline until it looks
correct, and then remove the --pretend flag from there.
This significantly influences the design of the script, both in action and
=== Logs and build output
All build commands are logged to a file (see `+log_command+` in ksb::Util).
This is both to declutter the terminal output and to enable troubleshooting
after a build failure.
The logs are generally kept in a separate directory for each separate run of
kdesrc-build. A "latest" symlink is created for each module name, which points
to the last instance of a script run.
If a build ends in a failure, an error.log symlink is created in the specific
log directory for that module, which points to the specific build phase output
file where the build was determined to have failed.
Sometimes there is no log though (e.g. an internal kdesrc-build failure outside
of log_command)!
Some users prefer to have TTY output. For now the --debug cmdline option is
useful for that, but --debug has a significant amount of other changes as well.
== Basic flow
For each script execution, kdesrc-build generically goes through the following
. Read the cmdline to determine global options, list of module *selectors*
(modules are defined later) and potentially alternate rc-files to use.
. Opens the selected rc-file (chosen on cmdline or based on `+$PWD+`) and reads
in the list of modules and module-sets in the rc-file along with the options
chosen for each.
. Ensures that the KDE git repository metadata is available (containing
dependency information and the virtual project path hierarchy)
. If module selectors are available from the cmdline, creates the build list by
expanding those selectors into the appropriate modules from the rc-file. If no
selectors, uses all module sets and modules from the rc-file.
* Either mode can involve resolving dependencies for KDE-based modules.
. Forks additional children to serve as a way to perform updates and build in
separate processes so that they may proceed concurrently. Once ready, performs
these two steps concurrently:
.. Updates each module in order, and
.. Performs remaining module build steps in order (waiting for the update if
. When all update/build processes are done, displays the results to the user.
== List of Packages
* <<ksb/Module#,ksb::Module>>
= ksb::Module
This is ksb::Module, one of the core classes within kdesrc-build. It represents
any single "buildable" module that kdesrc-build can manage. It acts as a common
interface to the multiple types of build systems and source control management
systems that kdesrc-build supports.
The many options available to the user are managed using setOption/getOption
(but see also the ksb::OptionsBase class that this derives from).
kdesrc-build manages persistent metadata for each module as well, see
The basic description of each method is listed here for ease of reference. See
the source code itself for more detail.
=== Perl integration
These functions are used to integrate into the Perl runtime or for use from
other Perl modules.
* ``new``, creates a new ksb::Module, and sets any provided options.
* ``toString``, for "stringifying" a module into a quoted string.
* ``compare``, for sorting ksb::Modules amongst each other based on name.
These functions are used to configure what the ksb::Module object should do,
change settings, etc.
* ``setModuleSet``, optional, specifies the ksb::ModuleSet this module was
spawned from.
* ``setScmType``, sets the source control plugin (git, svn, kde-projects) based
on the given scmType name. Normally auto-detection is used instead, this
permits manual setup.
* ``buildSystemFromName``, as with ``setScmType``, used to manually set the
build system plugin. This is exposed to the user as *override-build-system*.
* ``setBuildSystem``, like ``buildSystemFromName``, but passes the proper
ksb::BuildSystem directly.
* ``setOption``, sets a configuration option that can be checked later using
getOption. Normally set from user input (cmdline or rc-file) but supports
ways for kdesrc-build to internally override user settings or set hidden
flags for action in later phases. Does not survive beyond the current
* ``setPersistentOption``, sets an option to a string value that will be
read-in again on the next kdesrc-build run and can then be queried again.
* ``unsetPersistentOption``, removes an existing persistent option.
These functions are generally just read-only accessors of information about the
* ``name``, returns the module name. Only one module with a given name can be
present during a build.
* ``buildContext``, returns the ksb::BuildContext (as set when the object
was constructed)
* ``phases``, returns the list of execution phases (update, buildsystem, test,
etc.) that apply to this module in this execution.
* ``moduleSet``, returns the ksb::ModuleSet that was assigned earlier as the
source set. If no module set was assigned, returns a valid (but null) set.
* ``scm``, **autodetects** the appropriate scm plugin if not already done (or
manually set), and then returns the ksb::Updater plugin.
* ``buildSystem``, **autodetects** the appropriate build system plugin if not
already done (or manually set) and then returns the
ksb::BuildSystem|ksb/ plugin.
* ``scmType``, returns the **name** of the scm plugin (as determined by
scm(), which can itself cause an autodetection pass.
* ``buildSystemType``, returns the **name** of the build system plugin (as
determined by buildSystem(), which can itself cause an autodetection pass.
* ``currentScmRevision``, returns a string with an scm-specific revision ID.
Can be a Git-style SHA, SVN-style sequential ID, or something else entirely.
Can case an autodetection of the scm plugin.
==== PATHS
Various path-handling functions. These aren't always easy to tell what they do
just from the method name, sadly.
* ``getSubdirPath``, maps a path from the rc-file (based on the option-name to
pass to getOption) to a potential absolute path (handling tilde expansion
and relative paths). Does not handle colon-separated paths.
* ``getInstallPathComponents``, returns information about the directory the
module should be installed to. See the detailed docs for this method at its
decl, but generally you can just call fullpath today.
* ``getSourceDir``, returns absolute base path to the source directory (not
including dest-dir, module name, or anything else specific to this module).
* ``getLogDir``, returns the base path to use for logs for this module during
this execution. **NOTE** Different modules can have different base paths.
* ``getLogPath``, returns the absolute filename to open() for a log file for
this module based on the given basename. Updates the 'latest' symlink, which
can trigger clean up of old log dirs after all modules are built. Only use
when you're really going to open a log file!
* ``fullpath``, returns the absolute full path to the source or build
directory, including any module name or dest-dir accoutrement. This is the
directory you can git-clone to, cd to for build, etc.
* ``destDir``, returns the 'dest-dir' for the module. dest-dir is effectively
just a way to modify the on-disk module name. It used to be used more heavily
to allow for having multiple build/source directories for a given CVS/SVN
module (varying by branch or tag), but even with git this value may change
for KDE-based repositories to set subdirectories that match KDE project
paths. Supports expanding '$MODULE' or '${MODULE}' sequences to what
otherwise would have been the dest-dir.
* ``installationPath``, as labeled on the tin. Prefers the 'prefix' option but
falls back to 'kdedir' if not set.
* ``getOption``, returns the value of the given named option. If no such option
exists, inherits the same value from the module's build context. If no such
option exists there either, returns an empty string. Option values are used
by this function only exist during this script's execution. There is magic to
permit build jobs that run in a subprocess to feed option changes back to the
parent process.
* accepts an option name, normally as set in the rc-file. Can also accept a
second parameter 'module', to prevent falling back to a global option.
However doing this also permits ``undef`` to be returned so you must check
whether the result is defined.
* Options starting with '#' can only be set internally (i.e. not from rc-file
or cmdline) so this can be used as a way to tag modules with data meant not
to be user-accessible... but this should probably be factored into a
dedicated parallel option stack.
* The combination of module-specific and global options also contains a wee
bit of magic to control things like whether option values combine
("$global-value $module-value" style) or whether a module setting
completely masks a global setting.
* ``getPersistentOption``, similar to ``getOption``, only without the
module/global magic and the append/mask magic, and the subprocess-support
magic. But this function can return options that have been set in a previous
kdesrc-build run. kdesrc-build uses the location of the rc-file to determine
where to look for data from prior runs.
* ``fullProjectPath``, returns the logical module path in the
infrastructure for the module, if it's defined from a kde-projects module
set. E.g. for the 'juk' module, would return 'kde/kdemultimedia/juk'.
* ``isKDEProject``, returns true if the module was sourced from the special
``kde-projects`` module set in the user's rc-file. In this case the module's
``moduleSet()`` function should return a ksb::ModuleSet that is-a
* ``update``, which executes the update (or pretends to do so) using the
appropriate source control system and returns a true/false value reflecting
success. Note this can also throw exceptions and future code is moving more
to this mode of error-handling.
* ``build``, which executes the build **and** install (or pretends to in pretend
mode) using the appropriate build system and returns a true/false value
reflecting success. Can also run the testsuite as part of the build. Note
this can also throw exceptions and future code is moving more to this as the
error-handling mechanism.
* ``setupBuildSystem``, which sets up the build system for the module to permit
``build`` to work, including creating build dir, running cmake/configure/etc.
as appropriate. It is called automatically but will not take any action if
the build system is already established.
* ``install``, which installs (or pretends to install) the module. Called
automatically by ``build``.
* ``uninstall``, which uninstalls (or pretends to uninstall) the module. Not
normally called but can be configured to be called.
* ``applyUserEnvironment``, this adds ``set-env`` module-specific environment
variable settings into the module's build context, called by
``setupEnvironment``. This is needed since $ENV is not actually updated by
ksb::BuildContext until after a new child process is ``fork``'ed.
* ``setupEnvironment``, called by the kdesrc-build build driver, running in a
subprocess, before calling the appropriate update/build/install etc. method.
......@@ -32,36 +32,135 @@
# Adding an option? Grep for 'defaultGlobalOptions' in ksb::BuildContext --mpyne
use FindBin qw($RealBin);
use lib "$RealBin/../share/kdesrc-build/modules";
use lib "$RealBin/modules";
use 5.014; # Require Perl 5.14
use strict;
use warnings;
use Carp;
use Data::Dumper;
use File::Find; # For our lndir reimplementation.
use File::Path qw(remove_tree);
# On many container-based distros, even FindBin is missing to conserve space.
# But we can use File::Spec to do nearly the same.
my $RealBin;
my $modPath;
# The File::Spec calls have to run when parsing (i.e. in BEGIN) to make the
# 'use lib' below work (which itself implicitly uses BEGIN { })
use File::Spec;
# resolve symlinks
my $scriptPath = $0;
for (1..16) {
last unless -l $scriptPath;
$scriptPath = readlink $scriptPath;
die "Too many symlinks followed looking for script" if -l $scriptPath;
use Mojo::IOLoop;
use Mojo::Server::Daemon;
my ($volume, $directories, $script) = File::Spec->splitpath($scriptPath);
use ksb::Debug;
use ksb::Util;
use ksb::Version qw(scriptVersion);
use ksb::Application;
use ksb::UserInterface::TTY;
use web::BackendServer;
$RealBin = File::Spec->catpath($volume, $directories, '');
die "Couldn't find base directory!" unless $RealBin;
use 5.014; # Require Perl 5.14
# Use modules in git repo if running from git dir, otherwise assume
# system install
$modPath = File::Spec->rel2abs('modules', $RealBin);
$modPath = ($RealBin =~ s,/bin/?$,/share/kdesrc-build/modules,r)
unless -d $modPath;
die "Couldn't find modules for kdesrc-build!" unless $modPath;
use lib "$modPath"; # Make ksb:: modules available
sub dumpError
my $err = $@;
open my $fh, '>>', "error-$$.log" or return;
my $time = localtime;
say $fh $time;
say $fh $@;
# When running in a limited environment, we might not be able to load