Bob Foerster [Fri, 17 Dec 2010 15:20:39 +0000 (23:20 +0800)]
Resurrect alternative UIs
The various alternative UIs have been updated to once again be functional
with the latest bitbake internals. Each of the UIs still have much room for
functional improvement.
In particular, they have been updated to:
- interact with the new process based server
- handle the current set of events and notifications fired from the server
and its associated subsystems
Richard Purdie [Wed, 5 Jan 2011 00:54:16 +0000 (00:54 +0000)]
bitbake: build.py logging updates
python tasks calling shell functions using exec_func() would show the log
file as /dev/null. It makes most sense for all the task logging to be setup
centrally by exec_task(), at least with the current code base in Poky.
This commit will need discussion in relation to upstream bitbake and the
IO redirection could be better handled using a context manager (although
task contexts shouldn't ever nest).
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Chris Larson [Wed, 22 Sep 2010 01:11:54 +0000 (18:11 -0700)]
Rework the persist_data API
Rather than having to run .addDomain() and then .getValue(domain, key),
.setValue(domain, key), etc, now it just works as mappings.
As an example:
setValue(domain, key) -> persist[domain][key] = value
It also arranges things so we can construct objects of this type using any
arbitrary filename/path for the sqlite3 database, rather than being so
tightly bound to the metadata.
Chris Larson [Fri, 19 Nov 2010 02:51:51 +0000 (19:51 -0700)]
Revert "persist_data: cache connection and use cursor"
Caching the database connection can cause serious issues if it results in
multiple processes (e.g. multiple tasks) simultaneously using the same
connection.
Yu Ke [Tue, 4 Jan 2011 07:55:33 +0000 (15:55 +0800)]
base.bbclass: add lock file for do_unpack task
This patch intend to fix the random unpack failure of linux-libc-headers-yocto
and linux-yocto.
The root cause of the unpack failure is that: these two recpies has the same URL, thus
has the same dest file during the fetch and unpack phase:
do_fetch : create tar ball ${DL_DIR}/git_git.pokylinux.org.linux-yocto-2.6.37.tar.gz
do_unpack : extract tar ball ${DL_DIR}/git_git.pokylinux.org.linux-yocto-2.6.37.tar.gz
fetch phase is protected by lockfile, so it works fine. but unpack phase is not lock protected,
thus there is race condition like: when linux-yocto do_unpack is extracting the tar ball,
linux-libc-headers-yocto do_fetch starts to create tar ball thus overwrite linux-yocto's
tar ball and cause linux-yocto do_unpack failure
To fix this issue, do_unpack also need to be protected by lock
would only create task-base packages for foo, but not for both foo and bar.
Doing MACHINE=bar bitbake task-base -c cleanall would remove the packages for foo.
The solution is to use MULTIMACH_ARCH as suggested by Richard Purdie.
Only mark fn as clean if it is clean.
This saves us from removing (prematurely added) fn from our clean set
and saves me a few percent of runtime (and misleading debugging output
from remove()).
Previously we emitted two newlines for export and unexport.
One newline for export and unexport is enough (and makes the scripts
look better and a tad smaller).
Signed-off-by: Bob Foerster <robert@erafx.com> Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Signed-off-by: Andreas Oberritter <obi@opendreambox.org> Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
fetchers: Add parameter scmdata=keep to include .git/ and others in generated tarballs.
* Allows generating version information from SCMs during build.
* Note that tar doesn't need to use --exclude '.git', because
git checkout-index doesn't clone the repository.
Signed-off-by: Andreas Oberritter <obi@opendreambox.org> Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Chris Larson [Tue, 14 Dec 2010 15:10:36 +0000 (08:10 -0700)]
Fix PWD issue with new exec_func_shell
The previous attempt was incorrect. The issue isn't that subprocess fails to
set PWD, it's that PWD is in the metadata, inherited from the environment, and
is re-exported, overwriting the actual accurate one in the shell environment
with the old one from the metadata. So, ensure that PWD in the metadata is
not exported.
We can ditch this when the environment handling is reworked (e.g. poky's
commit to do so).
Chris Larson [Fri, 10 Dec 2010 23:53:19 +0000 (16:53 -0700)]
Rename command events, adjust compareRevisions
- Moved the logic for comparing revisions from cooker into command
- Removed 'Cooker' from the event names
- Renamed the 'ExitCode' event into CommandExit, and changed CommandFailed to
be a subclass of CommandExit
Chris Larson [Fri, 10 Dec 2010 15:51:47 +0000 (10:51 -0500)]
Kill the uncaught exception handler
We now wrap the main() in a try/except, ensuring that both the main portion of
bin/bitbake and the UI raising an exception will be shown to the user. For
the server and workers, we can ensure in the server itself that exceptions are
handled correctly.
Chris Larson [Tue, 23 Nov 2010 22:30:45 +0000 (15:30 -0700)]
cooker: use re match, not search in re_match_strings
We want to match the requested pattern at the beginning of the string,
otherwise things behave in an unintuitive manner wrt ASSUME_PROVIDED (e.g.
ASSUME_PROVIDED += "gtk+" will also assume foo-gtk+ is provided), and the user
can always use '.*gtk+' to get the old behavior.
Chris Larson [Tue, 30 Nov 2010 15:25:13 +0000 (08:25 -0700)]
cooker: no cached in progressbar and add ETA
Rather than updating the progress bar based on the recipe being processed
(whether cached or parsed), consider only parsed recipes. This reduces the
instability in progress rate introduced by the cached entries, and allows the
ETA to be resurrected and be a bit more useful.
Chris Larson [Tue, 23 Nov 2010 19:02:40 +0000 (14:02 -0500)]
knotty: drop the ETA from the progressbar for now
Currently, the progress bar is an indication of the processing of our recipes,
which includes loading the cache file, then for each recipe, either adding the
existing cached information to the CacheData or parsing the recipe from disk.
These tasks clearly take different amounts of time, so the ETA is unreliable
today. We'll resurrect this functionality after we revamp the progress
handling, fully incorporating the load of the cache file.
Chris Larson [Tue, 23 Nov 2010 17:46:49 +0000 (11:46 -0600)]
cooker: ensure that the cache sync completes
Without explicitly joining the thread, it's possible for the process to end
(e.g. after a bitbake -p) and kill off the thread without waiting for it to
exit cleanly. So, register the thread join with atexit.
Chris Larson [Sun, 21 Nov 2010 18:59:05 +0000 (11:59 -0700)]
cooker: show progress bar before initializing the cache
This ensures that the time spent loading the cache from disk occurs with the
progress bar up. Though the progress bar stays at 0% during this period, I
think this is an improvement over the multi-second stall which occurred
previously before the progress bar came up. Ideally, we'd integrate cache
loading from disk into the progress display, but this is a first step.
Chris Larson [Fri, 19 Nov 2010 18:46:42 +0000 (11:46 -0700)]
cooker: stop loading the cache for -b
Previously, the cache was actually being loaded from disk twice whenever using
-b or -e -b. This also moves the bb_cache instance into the CookerParser, as
it's not needed by the cooker itself at all.
Chris Larson [Fri, 19 Nov 2010 05:47:36 +0000 (22:47 -0700)]
cache: sync the cache file to disk in the background
This version uses a thread rather than a process, to avoid problems with
waitpid handling. This gives slightly less overall build time reduction than
the separate process for it did (this reduces a -c compile coreutils-native by
about 3 seconds, while the process reduced it by 7 seconds), however this time
is quite insignificant relative to a typical build.
The biggest issue with non-backgrounded syncing is the perceived delay before
work begins, and this resolves that without breaking anything, or so it seems.
Chris Larson [Fri, 19 Nov 2010 05:28:09 +0000 (22:28 -0700)]
Simplify cache syncing
Rather than adding nocache items to the cache, then copying the cache and
removing them to sync it, don't add them in the first place. Also use 'with'
for the cachefile.
Chris Larson [Fri, 19 Nov 2010 03:21:54 +0000 (20:21 -0700)]
Implement parallel parsing support
This utilizes python's multiprocessing module. The default number of threads
to be used is the same as the number of available processor cores, however,
you can manually set this with the BB_NUMBER_PARSE_THREADS variable.
Chris Larson [Wed, 17 Nov 2010 04:55:19 +0000 (21:55 -0700)]
cache: don't write out the cache unnecessarily
If the only recipes's we reparsed this run were those flagged as not to be
cached, there's no point in re-saving the cache, as those items won't be
included anyway.
Chris Larson [Tue, 16 Nov 2010 19:58:52 +0000 (12:58 -0700)]
cache: create and use a RecipeInfo class
This class holds the particular pieces of information about a recipe which are
needed for runqueue to do its job.
By using it, I think we improve code clarity, reduce method sizes, reduce
overuse of primitive types, and prepare for parallel parsing. In addition,
this ditches the leaky abstraction whereby bb.cache attempted to hide the
difference between cached data and a full recipe parse. This was a remnant
from the way things used to be done, and the code using it had to know the
difference anyway. If we choose to reimplement caching of the full recipes,
we can do it in bb.parse, in a completely transparent way.
Bob Foerster [Fri, 19 Nov 2010 20:39:22 +0000 (04:39 +0800)]
Prefer xrange over range for small performance gain.
range() allocates an actual list when called. xrange() is just an iterator
and creates the next range item on demand. This provides a slight
performance increase.
In python 3, range will do what xrange does currently, but the upgrade will
be handled by the 2to3 tool.
